X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks论文笔记-Toy模板网

这篇具有很好参考价值的文章主要介绍了X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks论文笔记。希望对大家有所帮助。如果存在错误或未考虑完全的地方，请大家不吝赐教，您也可以点击"举报违法"按钮提交疑问。

Title：X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks

Code

1. Motivation

CLIP这一类方法只能进行图片级别的视觉和文本对齐；

也有一些方法利用预训练的目标检测器进行目标级别的视觉和文本对齐，但是只能编码目标内部的特征，无法有效表达多目标上下文关联；

本文致力于进行多粒度（objects, regions, and images）的视觉文本对齐预训练任务；

X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks论文笔记,多模态,论文阅读

X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks论文笔记,多模态,论文阅读
3. vision-to-text similarity

X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks论文笔记,多模态,论文阅读
4. text-to-vision similarity

5. GT：one-hot

6. cross-entropy loss

X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks论文笔记,多模态,论文阅读

For each visual concept in a mini-batch, we sample an in-batch hard negative text by following $p^{v2t}(V)$ . （与当前视觉特征越接近的文本越可能被采样）
We also sample one hard negative visual concept for each text.
put the pairs as inputs for the fusion module, and then we use xcls, the output [CLS] embedding of the fusion module, to predict the matching probability $p^{match}$ , and the loss is: