Arxiv Dives

Arxiv Dives - LLaVA 🌋 an open source Large Multimodal Model (LMM)

Greg Schoeninger
Jan 7, 2024

What is LLaVA?

LLaVA is a Multi-Modal model that connects a Vision Encoder and an LLM for general purpose visual and language understanding.

Paper: https://arxiv.org/abs/2304.08485

Team: Wisconsin-Madison, Microsoft Research, Columbia University

Release Date: April 17th, 2023

What is Arxiv Dives?

Every Friday at Oxen.ai we host a paper club called "Arxiv Dives" to make us smarter Oxen 🐂 🧠. We believe diving into the details of research papers is the best way to build fundamental knowledge, spot patterns and keep up with the bleeding edge.

If you would like to join live to ask questions or join the discussion we would love to have you! Sign up below 👇

Arxiv Dives with Oxen.ai 📚🤿🐂 · Luma
Hey Nerd, join the Herd!... for a little book/paper review. WHAT TO EXPECT Each week we pick a topic to cover in depth and have open Q/A and discussion. Reading optional 🙃. JOIN THE…

The following are the notes from the live session. Feel free to watch the video and follow along for the full context.

Intro

We as humans navigate and interact with the world around us through both vision and language. Vision can be better at conveying some concepts, where text is better for others. 

Like they say, an image is worth a thousand words.

Traditionally, neural networks for computer vision are quite limited in scope to what they can do.

For example, let’s imagine you are a data scientist who works for an NBA team, and you are tasked with analyzing game footage to find patterns in winning plays.

A data scientist from the 2010s would have to train multiple networks. They would have one network detect where the ball is at all times, another detect the players, and another classify which action the player is taking etc.

Each one of these steps requires a separate labeled dataset and a separate model optimized for the task. This is system would be expensive to build and hard to maintain.

What if instead of 3 separate models and datasets, we could have one model to rule them all? What if we could describe in natural language the task, and the model would dynamically change what it was looking for in the image?

Enter LLaVA 🌋

Let’s prototype this action recognition pipeline without training a model at all. All it takes is some clever prompting, and faith that our model knows something about the game of basketball.

Feel free to follow along or try your own prompts at their live demo of the LLaVA system below.

LLaVA

With the simple prompt below, we have not only created an image classification system, but can chat with the model about the image.

Prompt:

You are looking at a picture of a basketball player, identify if they are shooting, dribbling or passing. Answer with a single classification of "shooting", "dribbling", "passing".

Prototyping a system with prompting gets your from zero-to-hero way faster than training an end to end model. While it might not be perfect, it is amazing the breadth of capabilities this model already has, given the instruction following dataset they introduce in the paper.

Some other fun examples

Identify this artwork

Meal from fridge

Website from drawing

It is a great feeling when you can simply ask a model to do your bidding, and not have to write any additional code to do so.

All of it is Open Source, which means if it doesn’t work perfectly for your use case, you can take the model, add more data, and fine-tune it to your use case.

Let’s dive into how it works.

Core Contributions

The paper makes a couple core open source contributions:

  • Datasets of Image-Text pairs to train on
  • An Open Source Large Multimodal Model (LMM) trained on these datasets

Their best model is a 13B model, but they also have a 7B model that is only 1% lower accuracy from 90% to 89% on ScienceQA according to their ablation studies at the end. Since the papers release, the community has taken these models and fine tuned them to their use cases, and I think there is a lot of room to play and optimize.

A search for "llava" on hugging face at the time of writing returns over 200 models.

Models - Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Visual Instruction Data Generation

While there are large multimodal datasets such as CC and LAION and COCO, none of them focus explicitly on instruction following.

Creating a dataset of instructions and images is time-consuming and not very well defined when asking humans crowd source the task. In order to create the dataset at scale, with limited resources, they use GPT-4 to curate questions that someone might ask of an image.

For the LLaVA dataset they take existing image - caption pairs and image bounding box pairs, and feed them to GPT-4 with an extensive prompt to generate both instructions and conversations about the images.

They generate a few different types of instruction following data.

  1. Conversation
  2. Detailed Description
  3. Complex Reasoning

They give GPT-4 the captions and boxes as text sequences and do not show it the images themselves. This allows it to generate questions, generate answers to those questions, then elaborate into longer form conversation.

An example of an image caption might be:

A group of people standing outside a black SUV

Then GPT-4 can generate a question about the image, given the caption.

What color is the SUV?

They have examples in the appendix of the paper of the prompts used to generate the data.

They generate two main datasets with this paper.

  1. A pre-training dataset of 595k single image, prompt, response triples
  2. An instruction tuned dataset of 158k instruction following conversations given an image.

You can browse both datasets on Oxen.ai as well as download them for your own use.

LilianZhou/LLaVA_pretrain_595k | Datasets at Oxen.ai
This is the dataset for pretraining the Large Language and Vision Assistant(LLaVA), an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.. Contribute to the LilianZhou/LLaVA_pretrain_595k repository by creating an account on Oxen.ai
datasets/LLaVA-Instruct | Datasets at Oxen.ai
LLaVA Visual Instruct 150K is a set of GPT-generated multimodal instruction-following data. It is constructed for visual instruction tuning and for building large multimodal towards GPT-4 vision/language capability.. Contribute to the datasets/LLaVA-Instruct repository by creating an account on Oxen.ai

Visual Instruction Tuning Model

The model architecture is rather simple when looking at the high level building blocks.

The two main building blocks of LLaVA are a Language Transformer (LLM) and a Vision Transformer (ViT). In theory you could take any off the shelf LLM and ViT and zip them together into a LLaVA-like architecture.

In this case, they take the Language Transformer from Vicuna and the Vision Transformer from CLIP (ViT-L/14) and add a trainable projection layer W on top of the image features and text features for the model to learn how to combine them.

The only addition here is the projection matrix W that helps encode the language tokens and the vision tokens into the same latent space. Quite a simple concept, and very few lines to implement in most neural net libraries like PyTorch.

The data is then organized into conversations of the following form.

Where X_instruct may be either an image then a question, or a question then an image, or simply a question later in the sequence.

For example:

You are an AI Visual Assistant who is looking at a single image. Answer the user’s question about the image<STOP>
Human: How many fireworks are in the image?
<STOP>
Assistant: 8<STOP>
Human: Why is the Ox wearing a hat?<STOP>
Assistant: It is new years.<STOP>

The image is broken up into 14x14 patches and fed into the ViT while the text is broken up into tokens and fed into the LLM. The projection layer learns to zip them together and voila! You can instruct the model to extract information from the image.

Stage 1: Pre-training for Feature Alignment

They first perform a large pre-training step on a subset of CC3M dataset which is filtered down to 595k image-text pairs.

We put all the data in Oxen.ai if you want to get a feel for it.

LilianZhou/LLaVA_pretrain_595k | Datasets at Oxen.ai
This is the dataset for pretraining the Large Language and Vision Assistant(LLaVA), an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.. Contribute to the LilianZhou/LLaVA_pretrain_595k repository by creating an account on Oxen.ai

This dataset consists of single turn conversations less complex and is just a set of questions generated directly from the captions, and lacks as much diversity and in-depth reasoning as the smaller instruct dataset.

The questions are generated by GPT-4 and the ground truth responses are simply the original captions.

This pre-training stage uses the larger dataset to make sure the LLM and the ViT embed the images and text into a similar space. During this phase they keep the ViT and the LLM frozen and are simply training the projection weights W in order to align the image features and the text features.

It is much more efficient to only train the projection matrix than train the whole model end to end.

They did an ablation study without the Pre-Training and simply fine-tune on the ScienceQA dataset and find a 5.11% drop in performance without it.

Stage 2: Fine-Tuning End-to-End

Next they perform a fine-tuning on the smaller dataset with full conversational data.

datasets/LLaVA-Instruct | Datasets at Oxen.ai
LLaVA Visual Instruct 150K is a set of GPT-generated multimodal instruction-following data. It is constructed for visual instruction tuning and for building large multimodal towards GPT-4 vision/language capability.. Contribute to the datasets/LLaVA-Instruct repository by creating an account on Oxen.ai

During the fine tuning they keep the visual encoder weights frozen, and continue to update the projection layer as well as the end to end LLM.

They train the model on 8 x A100s. On this hardware setup they can complete the pre-training on 595k image-text pairs in 4 hours and the fine tuning on 158k conversations in 10 hours.

Quantitative Evaluation

They use a text-only GPT-4 to evaluate the quality of the responses. In order to do this they provide GPT-4 the text description of the images, the question, and the generated responses, then have GPT-4 give overall scores from 1 to 10 on helpfulness, relevance, and accuracy of the responses.

I don’t know that it is necessarily a great evaluation metric seeing as they used GPT-4 to generate the training data as well as evaluate. Other models such as OpenFlamingo did not use GPT-4 in their training data. Using GPT-4 to generate training data means that it will be from a distribution that GPT-4 understands, so this evaluation technique feels biased.

It would be interesting to see a different model generate the instruction training data than the one evaluating. For example maybe we could use another open source model to generate training data, and then use GPT-4 to evaluate.

They create a couple small benchmarks to evaluate the models on as well including LLaVA-Bench with 30 images from COCO-Val-2014 and 24 images in the wild. For the 50+ images they have ~150 questions. These benchmarks also feel small, but in general it is hard to quantitatively evaluate these models, and we are all figuring it out as we go 😎.

ScienceQA

ScienceQA is a benchmark of 21k multiple choice questions that include images. There are a diverse set of topic categories and skills.

LilianZhou/ScienceQA | Datasets at Oxen.ai
Science Question Answering(ScienceQA) consists of ~21k multimodal multiple choice questions with a diverse set of science topics and annotations of their answers with corresponding lectures and explanations.. Contribute to the LilianZhou/ScienceQA repository by creating an account on Oxen.ai

Llava performs rather well when fine tuned on this dataset compared to chain of thought prompting on other models. MM-CoT stands for multimodal chain of thought.

Conclusion

This is promising research into using an LLM for synthetic data as well as how easy it can be to link together language and vision models through a simple linear projection.

The fact that the models and datasets are all open source allows the community to iterate on their own problems and have full control over the models without having to wait for the next drop of proprietary models.

It is hard to evaluate these models without multiple choice datasets or human labeling. I would love to explore this dataset a little more, and extend the instructions to other tasks, and really dive into what the model does well in a Practical ML Dive.

Next Up

To find out what paper we are covering next and join the discussion at large, checkout our Discord:

Join the oxen Discord Server!
Check out the oxen community on Discord - hang out with 269 other members and enjoy free voice and text chat.

If you enjoyed this dive, please join us next week!

Arxiv Dives with Oxen.ai · Luma
Hey Nerd, join the Herd!... for a little book/paper review. Make sure to also join our Discord here (https://discord.gg/s3tBEn7Ptg) to share recommendations for future reads and more…

All the past dives can be found on the blog.

Arxiv Dives - Oxen.ai
Each week we dive deep into a topic in machine learning, data management, or general artificial intelligence research. These are notes from a live reading group we do every Friday. Captured for future reference.

The live sessions are posted on YouTube if you want to watch at your own leisure.

Oxen
Each week we dive deep into a topic in machine learning or general artificial intelligence research. The sessions are live with a group of smart Oxen every Friday. Join the discussion: https://lu.ma/oxenbookclub

Best & Moo,

~ The herd at Oxen.ai

Who is Oxen.ai?

Oxen.ai is an open source project aimed at solving some of the challenges with iterating on and curating machine learning datasets. At its core Oxen is a lightning fast data version control tool optimized for large unstructured datasets. We are currently working on collaboration workflows to enable the high quality, curated public and private data repositories to advance the field of AI, while keeping all the data accessible and auditable.

If you would like to learn more, star us on GitHub or head to Oxen.ai and create an account.

GitHub - Oxen-AI/oxen-release: Lightning fast data version control system for structured and unstructured machine learning datasets. We aim to make versioning datasets as easy as versioning code.
Lightning fast data version control system for structured and unstructured machine learning datasets. We aim to make versioning datasets as easy as versioning code. - GitHub - Oxen-AI/oxen-release:…