Version 1

Perform a deep dive synopsis in markdown code describing the datasets and input datasetts used by two models in comparison - delving much deeper into real time papers and information on these datasets and ideally find the URL where the dataset or dataset paper can be viewed. Fix the article I have written below with a start on the datasets that were used to train the two models:

Language Models 🗣️

🏆 Bloom sets new record for most performant and efficient AI model in science! 🌸

Comparison of Large Language Models

GPT-3 Datasets 📚

• WebText
• Common Crawl
• BooksCorpus
• English Wikipedia
• Toronto Books Corpus
• OpenWebText

ChatGPT Datasets - Details 📚

• WebText: A dataset of web pages crawled from domains on the Alexa top 5,000 list. This dataset was used to pretrain GPT-2.
  • WebText: A Large-Scale Unsupervised Text Corpus by Radford et al.
• Common Crawl: A dataset of web pages from a variety of domains, which is updated regularly. This dataset was used to pretrain GPT-3.
  • Language Models are Few-Shot Learners by Brown et al.
• BooksCorpus: A dataset of over 11,000 books from a variety of genres.
  • Scalable Methods for 8 Billion Token Language Modeling by Zhu et al.
• English Wikipedia: A dump of the English-language Wikipedia as of 2018, with articles from 2001-2017.
  • Improving Language Understanding by Generative Pre-Training Space for Wikipedia Search
• Toronto Books Corpus: A dataset of over 7,000 books from a variety of genres, collected by the University of Toronto.
  • Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond by Schwenk and Douze.
• OpenWebText: A dataset of web pages that were filtered to remove content that was likely to be low-quality or spammy. This dataset was used to pretrain GPT-3.
  • Language Models are Few-Shot Learners by Brown et al.

Big Science Model 🚀

• 📜 Papers:

  1. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model Paper
  2. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Paper
  3. 8-bit Optimizers via Block-wise Quantization Paper
  4. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Paper
  5. Other papers related to Big Science
  6. 217 other models optimized for use with Bloom

• 📚 Datasets:

Datasets:

1. • Universal Dependencies: A collection of annotated corpora for natural language processing in a range of languages, with a focus on dependency parsing.

• Universal Dependencies official website.

2. • WMT 2014: The fourth edition of the Workshop on Statistical Machine Translation, featuring shared tasks on translating between English and various other languages.

• WMT14 website.

3. • The Pile: An English language corpus of diverse text, sourced from various places on the internet.

• The Pile official website.

4. • HumanEval: A dataset of English sentences, annotated with human judgments on a range of linguistic qualities.

• HumanEval: An Evaluation Benchmark for Language Understanding by Gabriel Ilharco, Daniel Loureiro, Pedro Rodriguez, and Afonso Mendes.

5. • FLORES-101: A dataset of parallel sentences in 101 languages, designed for multilingual machine translation.

• FLORES-101: A Massively Multilingual Parallel Corpus for Language Understanding by Aman Madaan, Shruti Rijhwani, Raghav Gupta, and Mitesh M. Khapra.

6. • CrowS-Pairs: A dataset of sentence pairs, designed for evaluating the plausibility of generated text.

• CrowS-Pairs: A Challenge Dataset for Plausible Plausibility Judgments by Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu, Pascale Fung, and Caiming Xiong.

7. • WikiLingua: A dataset of parallel sentences in 75 languages, sourced from Wikipedia.

• WikiLingua: A New Benchmark Dataset for Cross-Lingual Wikification by Jiarui Yao, Yanqiao Zhu, Ruihan Bao, Guosheng Lin, Lidong Bing, and Bei Shi.

8. • MTEB: A dataset of English sentences, annotated with their entailment relationships with respect to other sentences.

• Multi-Task Evaluation Benchmark for Natural Language Inference by Michał Lukasik, Marcin Junczys-Dowmunt, and Houda Bouamor.

9. • xP3: A dataset of English sentences, annotated with their paraphrase relationships with respect to other sentences.

• xP3: A Large-Scale Evaluation Benchmark for Paraphrase Identification in Context by Aniket Didolkar, James Mayfield, Markus Saers, and Jason Baldridge.

10. • DiaBLa: A dataset of English dialogue, annotated with dialogue acts.

• A Large-Scale Corpus for Conversation Disentanglement by Samuel Broscheit, António Branco, and André F. T. Martins.

• 📚 Dataset Papers with Code

  1. Universal Dependencies
  2. WMT 2014
  3. The Pile
  4. HumanEval
  5. FLORES-101
  6. CrowS-Pairs
  7. WikiLingua
  8. MTEB
  9. xP3
  10. DiaBLa

Deep RL ML Strategy 🧠

The AI strategies are:

• Language Model Preparation using Human Augmented with Supervised Fine Tuning 🤖
• Reward Model Training with Prompts Dataset Multi-Model Generate Data to Rank 🎁
• Fine Tuning with Reinforcement Reward and Distance Distribution Regret Score 🎯
• Proximal Policy Optimization Fine Tuning 🤝
• Variations - Preference Model Pretraining 🤔
• Use Ranking Datasets Sentiment - Thumbs Up/Down, Distribution 📊
• Online Version Getting Feedback 💬
• OpenAI - InstructGPT - Humans generate LM Training Text 🔍
• DeepMind - Advantage Actor Critic Sparrow, GopherCite 🦜
• Reward Model Human Prefence Feedback 🏆

For more information on specific techniques and implementations, check out the following resources:

• OpenAI's paper on GPT-3 which details their Language Model Preparation approach
• DeepMind's paper on SAC which describes the Advantage Actor Critic algorithm
• OpenAI's paper on Reward Learning which explains their approach to training Reward Models
• OpenAI's blog post on GPT-3's fine-tuning process

Version 2: