【论文笔记】Gemma: Open Models Based on Gemini Research and Technology-Toy模板网

这篇具有很好参考价值的文章主要介绍了【论文笔记】Gemma: Open Models Based on Gemini Research and Technology。希望对大家有所帮助。如果存在错误或未考虑完全的地方，请大家不吝赐教，您也可以点击"举报违法"按钮提交疑问。

Gemma

日期: March 5, 2024
平台: CSDN, 知乎
状态: Writing

Gemma: Open Models Based on Gemini Research and Technology

谷歌最近放出的Gemma模型【模型名字来源于拉丁文gemma，意为宝石】采用的是与先前Gemini相同的架构。这次谷歌开源了两个规模的模型，分别是2B和7B的版本。【对于个人电脑来说，2B真的要容易运行的多】。在18个基于文本的任务上，有11项胜过其他开源的模型

谷歌在开源社区领域真的做出了巨大的贡献🌼，Transformers, TensorFlow, BERT, T5, JAX, AlphaFold, 以及AlphaCode。每一项对人工智能的发展都起到了推波助澜的作用。

引言

We trained Gemma models on up to 6T tokens of text, using similar architectures, data, and training recipes as the Gemini model family.

分别发布了pre-trained和fine-tuned的checkpoints。

模型架构

【论文笔记】Gemma: Open Models Based on Gemini Research and Technology,论文阅读,gemma,语言模型,transformer

其实通过上面第一张表格就大体就可以计算第二张表格的参数具体是怎么来的

model.layers.0.self_attn.q_proj.weight torch.Size([2048, 2048]) = 4,194,304
model.layers.0.self_attn.k_proj.weight torch.Size([256, 2048]) = 524,288
model.layers.0.self_attn.v_proj.weight torch.Size([256, 2048]) = 524,288
model.layers.0.self_attn.o_proj.weight torch.Size([2048, 2048]) = = 4,194,304
model.layers.0.mlp.gate_proj.weight torch.Size([16384, 2048]) = 33,554,432
model.layers.0.mlp.up_proj.weight torch.Size([16384, 2048]) = 33,554,432
model.layers.0.mlp.down_proj.weight torch.Size([2048, 16384]) = 33,554,432
model.layers.0.input_layernorm.weight torch.Size([2048]) = 2048
model.layers.0.post_attention_layernorm.weight torch.Size([2048]) = 2048

…

model.norm.weight torch.Size([2048])

(4,194,3042+524,2882+33,554,4323+20482)*18+2048=1,981,884,416