大模型综述论文笔记6-15-Toy模板网

这篇具有很好参考价值的文章主要介绍了大模型综述论文笔记6-15。希望对大家有所帮助。如果存在错误或未考虑完全的地方，请大家不吝赐教，您也可以点击"举报违法"按钮提交疑问。

Keywords

GPT:Generative Pre-Training

Backgroud for LLMs

Technical Evolution of GPT-series Models

Two key points to GPT’s success are (I) training decoder-onlly Transformer language models that can accurately predict the next word and (II) scaling up the size of language models

Research of OpenAI on LLMs can be roughly divided into the following stages

Early Explorations

大模型综述论文笔记6-15,LLM,论文阅读

Capacity Leap

ICT

Capacity Enhancement

1.training on code data
Codex: a GPT model fine-tuned on a large corpus of GitHub
code
2.alignment with human preference
reinforcement learning from human feedback (RLHF) algorithm

Note that it seems that the wording of “instruction tuning” has seldom
been used in OpenAI’s paper and documentation, which is substituted by
supervised fine-tuning on human demonstrations (i.e., the first step
of the RLHF algorithm).

The Milestones of Language Models

chatGPT(based on gpt3.5 and gpt4) and GPT-4(multimodal)

Resources

大模型综述论文笔记6-15,LLM,论文阅读
Stanford Alpaca is the first open instruct-following model fine-tuned based on LLaMA (7B).
Alpaca LoRA (a reproduction of Stanford Alpaca using LoRA)

model 、data、library

Pre-training

大模型综述论文笔记6-15,LLM,论文阅读

Data Collection

General Text Data:webpages, books, and conversational text
Specialized Text Data:Multilingual text, Scientific text, Code

Data Preprocessing

Quality Filtering

The former approach trains a selection classifier based on highquality texts and leverages it to identify and filter out low quality data.
heuristic based approaches to eliminate low-quality texts through a set of well-designed rules: Language based filtering, Metric based filtering, Statistic based filtering, Keyword based filtering

De-duplication

Existing work has found that duplicate data in a corpus would reduce the diversity of language models, which may cause the training process to become unstable and thus affect the model performance.文章来源地址https://www.toymoban.com/news/detail-688303.html

Privacy Redaction: (PII:personally identifiable information )
Tokenization:(It aims to segment raw text into sequences of individual tokens, which are subsequently used as the inputs of LLMs.) Byte-Pair Encoding (BPE) tokenization; WordPiece tokenization; WordPiece tokenization

到了这里，关于大模型综述论文笔记6-15的文章就介绍完了。如果您还想了解更多内容，请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章，希望大家以后多多支持TOY模板网！