LLaVA, dissected

How does a language model read? Let's try to understand. In a simple way. A language model never sees letters.

Text is first chopped into tokens: small pieces, often a word or part of a word. The cat sat might become three tokens. Each token is swapped for a list of numbers called an embedding. Picture a lookup table: one row per token, and every row has the same length.

That length is the model's width, and it is fixed. Say the width is four thousand. Then the cat sat becomes three rows of four thousand numbers each, in order. So what the model actually reads is a sequence of vectors, one per token, all the same width.

The words are gone. Inside, there is only this stack of equal rows. The layers let each row borrow from the rows before it. The vector for sat looks back at cat and the.

Widths never change: four thousand in, four thousand out. At the end, the last row is turned into a score for every token in the vocabulary. The highest score wins, or one is sampled. That is the next token.

Then the loop. The new token is appended, embedded into a row of the same width, and the model reads again. One token per pass, until the paragraph is done. Here is the point to keep.

The model never reads text. It reads rows of one fixed width. Anything it reads must arrive as a vector of exactly that width. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is instruction tuning? Let's try to understand. In a simple way. Last time we saw a language model reads vectors and predicts the next token.

That is its whole job in pretraining. Not answering. Just continuing. So ask it, summarize this article, and it may continue with three more instructions.

In its training text, one instruction was often followed by another. The model is not broken. It does exactly what it learned: the most likely continuation. Likely and wanted are different things.

Instruction tuning fixes this with a dataset of pairs: an instruction, and the response a helpful assistant would give. Fine-tuning means training the pretrained model a little more, on just those pairs. The objective does not change. The pair is one text sequence: instruction, then response.

The model still predicts each next token. What changes is which tokens count. The loss is the penalty for wrong predictions. Here it is charged only on response tokens.

Mispredicting a word inside the instruction costs nothing. Read the instruction, get graded on the answer. Say the instruction is five tokens and the response is ten. The loss is summed over ten predictions, not fifteen.

Every training update pushes toward answering, never toward writing more instructions. Same model, same numbers inside, new habit. This is how InstructGPT and ChatGPT were made from plain text models. LLaVA builds on Vicuna, a chat model made this way, with thirteen billion numbers inside.

It follows text instructions. Text only, for now. Next lesson: where those pairs came from. Quick check now.

One question is coming up. Let's see if it clicked.

How does self-instruct data generation work? Let's try to understand. In a simple way. Last lesson, instruction tuning needed pairs: an instruction and a good response.

But writing thousands of pairs by hand is slow and costly. So here is the idea. Humans write only a handful of examples. Call them seeds.

Each seed is one instruction with one careful answer. Now take a strong text model. Paste the seeds into its input and ask it to write more of the same. It has seen the pattern, so it continues it.

Run that prompt again and again, with different seeds each time, and thousands of new pairs roll out. The machine writes the dataset; people wrote only the seeds. Then those machine-written pairs go straight into instruction tuning, the loss on the response tokens, exactly as before. Only the source of the data changed.

This worked in text. Alpaca and Vicuna were both tuned on machine-generated instruction data. And Vicuna, from last lesson, is the chat model LLaVA builds on. One caveat.

The data is only as good as its writer. The LLaVA authors found GPT-4 wrote higher quality examples than ChatGPT, for instance on spatial reasoning. Now the wall. LLaVA needs pairs about pictures: a question about an image, and an answer.

But the strongest writer available, GPT-4, was text-only. It cannot see a single pixel. So how do you get a blind writer to write about a photo? That is the whole puzzle of the next two lessons.

Quick check now. One question is coming up. Let's see if it clicked.

What exactly is a vision encoder? Let's try to understand. In a simple way. Lesson one left us a rule.

Anything the model reads must arrive as a vector of one fixed width. Text gets there through an embedding table. A picture has no such table. A photo is a grid of pixels, each three numbers for red, green and blue.

Fed in raw, there is no vocabulary and no embedding row to look up. So LLaVA borrows an eye: a vision encoder, a network trained separately on images. Here it is the image half of CLIP, a model trained to match pictures with their captions, called ViT-L slash fourteen. Cut the image into a grid of small square patches, like floor tiles.

Each tile becomes one item in a sequence, like a word in a sentence. Layers like those in lesson one then pass information between the tiles, and out comes one feature vector per patch. The paper calls these grid features. A six by six grid gives thirty six vectors.

A finer grid gives more. The count depends on the grid, not on the picture's content. Now the catch. Each vector has a width chosen when CLIP was built.

Vicuna's embeddings have a width chosen when Vicuna was built. The two numbers do not match. LLaVA never retrains this encoder. Its weights stay frozen, a fixed camera speaking one visual dialect.

So we hold visual vectors in the wrong width, and a model that reads only its own. Something must translate. Next lesson. Quick check now.

One question is coming up. Let's see if it clicked.

How does describing a picture to a blind writer work? Let's try to understand. In a simple way. Last lesson ended on a puzzle.

GPT-4, the strongest writer available, reads only text. So how can it write about pictures? LLaVA's answer: do not show it the picture. Describe the picture in words so carefully that the words stand in for the image.

First ingredient: captions. Several of them, each from a different angle. One says a kitchen with a wooden table. Another says two people cooking.

A third mentions sunlight through a window. Second ingredient: bounding boxes. A bounding box is a rectangle around one object, with a label and its position. Person, left of center.

Pot, on the stove. Window, top right. Positions matter. Captions rarely say where things are.

Boxes do. Now a text-only model can work out that the pot sits between the two people. So the picture arrives as a page of text: several captions plus a list of labelled rectangles. To GPT-4, that page is the image.

Now the obvious shortcut. Take one caption, attach the question describe this image briefly, and use the caption as the answer. Instant training pair, zero extra work. The authors judged it too shallow.

Every example has the same shape: one question, one short answer. No variety, and no reasoning about the scene. So instead, the full text stand-in goes to GPT-4 with a few human-written seed examples in context, the self-instruct move from lesson three. What GPT-4 writes back comes next lesson.

Quick check now. One question is coming up. Let's see if it clicked.

How does LLaVA's instruction data work? Let's try to understand. In a simple way. Last time, GPT-4 got captions and boxes instead of pixels, and caption-as-answer was judged too shallow.

What did the authors ask for instead? Three kinds of answers. Type one: conversation. Short questions and answers about the scene.

How many people, what they are doing, where the dog sits relative to the bench. Type two: detailed description. One long, rich paragraph covering the whole image, not a one-line caption. Type three: complex reasoning.

Questions whose answer takes several logical steps. Why might these people be waiting here? The reply has to argue from what the scene shows. Now the counts.

Fifty eight thousand conversation samples. Twenty three thousand detailed descriptions. Seventy seven thousand complex reasoning samples. One hundred fifty eight thousand in total.

Notice the shape. Reasoning is the biggest slice, nearly half. Detailed description is the smallest. The set leans toward answers that think.

All of it sits on roughly eighty thousand photos from COCO, a public image collection. That is about two samples per picture, each asking something different. Here is the part people miss. Humans wrote only the seed examples in the prompt.

Every other question and answer came from the text model. Those seeds are the only human annotation in the dataset. Two writers were used, ChatGPT and GPT-4. Lesson three noted GPT-4 wrote the better spatial reasoning.

The paper never says how many samples each wrote. Data ready: images, questions, answers. Next, the model that reads it. Quick check now.

One question is coming up. Let's see if it clicked.

How does LLaVA's architecture work? Let's try to understand. In a simple way. Two pieces are on the table.

A frozen vision encoder gives one vector per patch. The language model only reads vectors of its own width. The widths do not match. LLaVA's answer is almost embarrassingly small.

One matrix, called W. Every patch vector is multiplied by W, and out comes a vector of exactly the language model's width. That is a linear projection. Linear means nothing but the multiply: no extra layers, no squashing.

Each output number is a weighted sum of the inputs, and W holds those weights. The paper calls the results visual tokens. To the language model they look like word embeddings that happen to come from pixels, sitting in the sequence beside the text. So the whole model is three parts.

CLIP's ViT-L fourteen as the eye, frozen. Vicuna, the chat model with thirteen billion parameters, its learned numbers, as the tongue. W in between. Here is the striking part.

The encoder existed. Vicuna existed. W is the only new parameter in the entire model. Then the language model reads as usual: visual tokens, your question, and it predicts the answer one token at a time.

Same machinery as lesson one. The authors call this lightweight. Earlier systems used heavier connectors, small networks that pick which visual information to pass on. LLaVA leaves those for future work.

Which raises the obvious question. How does one matrix learn to speak the language model's language? That is the training story. Next lesson.

Quick check now. One question is coming up. Let's see if it clicked.

How does LLaVA's first training stage work? Let's try to understand. In a simple way. LLaVA trains in two stages.

Today is stage one, which the paper calls feature alignment, with one narrow job: make W, that single projection matrix, speak the language model's vocabulary before anyone asks it to chat. Both big pieces stay frozen. The CLIP encoder does not change. Vicuna does not change.

Every training update lands on W alone. The data is not the GPT-4 instruction set. It is CC3M, a public pile of images with web captions, filtered down to five hundred ninety five thousand pairs. The filter balances the noun phrases in the captions.

Phrases seen fewer than thirty three times were dropped. Each pair becomes a single turn. A short question asking for a concise description, sampled from eleven phrasings, and the original caption as the answer. Lesson five called this caption-as-answer format too shallow for instruction data.

Here it is fine. Stage one is not teaching reasoning. It is teaching W where each patch vector should land. The budget is tiny.

One epoch, a single pass over the data, batch size one hundred twenty eight, about four hours on eight A100 GPUs. A matrix trains fast when nothing else does. The authors describe the result as a visual tokenizer. A tokenizer turns raw input into vectors the model already understands.

After stage one, W does that for image patches. After stage one, Vicuna still cannot chat about pictures. It can only read them. Teaching it to answer comes next.

Quick check now. One question is coming up. Let's see if it clicked.

How does LLaVA's second training stage work? Let's try to understand. In a simple way. After stage one, W can turn patches into tokens Vicuna accepts.

But it has only produced brief captions. It cannot chat yet. So stage two unfreezes the language model. W and every Vicuna weight now update together.

Only the vision encoder stays frozen. The data is the one hundred fifty eight thousand samples from lesson six: conversations, detailed descriptions and complex reasoning, over about eighty thousand images. Each sample is a multi-turn conversation: a system message, then human and assistant turns, each ending with a stop marker so the model learns where an answer ends. The image appears only in the first turn.

A coin flip decides whether it comes before the first question or after it. Later turns carry just the question. The loss is scored only on the assistant's answer tokens, just like lesson two. The model reads the questions but is never trained to write them.

Picture a three turn chat about one photo. The sequence holds the image, three questions, three answers. Training signal comes from the three answers only, each written with everything before it in view. The budget: three passes over the data, an update step size a hundred times smaller than stage one, about ten hours on eight A100 GPUs.

Stage one took four. Note the division of labor. Stage one taught W the vocabulary. Stage two teaches the whole language model to use it in conversation.

Whether it chats well: next lesson. Quick check now. One question is coming up. Let's see if it clicked.

How does GPT-4 as a blind judge work? Let's try to understand. In a simple way. LLaVA writes free text about pictures, and there is no single right answer to grade against.

So the authors asked another model to grade, one that cannot see the image. Text-only GPT-4 gets the same symbolic stand-in from lesson five: captions and labelled boxes. From that it writes a reference answer to each question. Then the judge sees the question, that textual description, the reference, and LLaVA's answer.

It scores helpfulness, relevance, accuracy and detail from one to ten, and explains why. The relative score is LLaVA's score divided by the reference score. Say the reference earns eight and LLaVA earns six point eight. That is eighty five percent.

LLaVA-Bench COCO uses thirty randomly chosen COCO images, three questions per image, one of each type: ninety questions. With all one hundred fifty eight thousand samples, LLaVA scores eighty five point one. Train on conversation data only and it drops to seventy three point eight. Skip instruction tuning entirely and it collapses to twenty one point five.

The tuning is the whole game. Then a harder set: twenty four images the authors collected themselves, memes, paintings, sketches, sixty questions in all. Here LLaVA scores sixty seven point three. BLIP-2, an earlier model trained on image-text pairs but never on instruction data, gets thirty eight point one.

OpenFlamingo, nineteen point one. One caveat the authors state themselves: the judge never saw the pixels either, and they leave its reliability as an open question. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is the ninety two point five three asterisk? Let's try to understand. In a simple way. Science QA is a school science exam: multiple choice, many questions with a picture or diagram.

Twenty one thousand of them. An answer is simply right or wrong. LLaVA was fine-tuned on it as single turns: question and context in, reasoning plus answer out. Alone, the thirteen billion LLaVA scores ninety point nine two percent.

The previous best, MM-CoT Large, scores ninety one point six eight. So alone, LLaVA does not win. Now the trick. Text-only GPT-4 answers the same questions.

Whenever it disagrees with LLaVA, GPT-4 sees both answers and picks. That pairing scores ninety two point five three. A simpler pairing, using GPT-4 unless it refuses for lack of an image, reaches only ninety point nine seven. Judging disagreements earns the extra point and a half.

So when you see ninety two point five three, remember the asterisk. It is LLaVA plus GPT-4 as tie breaker, not LLaVA alone. The ablations are the real lesson. Skip stage one alignment and train from scratch: accuracy falls to eighty five point eight one, a drop of five point one one.

The biggest loss. Take patch vectors from the encoder's last layer instead of the one before: minus point nine six. The authors say the earlier layer keeps more local detail. Shrinking to seven billion costs one point zero eight.

And writing reasoning before the answer converges faster, but plateaus at eighty nine point seven seven, below answer first. Quick check now. One question is coming up. Let's see if it clicked.

What exactly is the bag-of-patches problem? Let's try to understand. In a simple way. After the scores, the authors listed where LLaVA breaks.

Picture a fridge holding yogurt and strawberries. Ask if there is strawberry-flavored yogurt, and LLaVA sometimes says yes. Their diagnosis: it reads the image as a bag of patches, seeing pieces but not how they relate. Second, resolution.

Reading the brand on that yogurt needs fine detail, and one vector per patch does not carry it. Third, knowledge. Their ramen example needs a restaurant name and side dishes, which may take internet retrieval and other languages. Fourth, the judge.

That eighty five point one score is text-only GPT-4 grading against a GPT-4 reference. Its robustness stays an open question. Fifth, bias. Whatever CLIP, Vicuna, and the generated data carry, LLaVA inherits.

And eighty thousand images is small, so the authors call these early experiments. Now the lineage. LLaVA one point five, October twenty twenty three, swapped the single matrix for a two-layer MLP, two matrices with a bend between, and a three hundred thirty six pixel CLIP encoder. LLaVA-NeXT, a January twenty twenty four blog release, went after resolution: AnyRes feeds up to four times the pixels, across three aspect ratios.

And the road not taken. InstructBLIP, May twenty twenty three, kept BLIP-2's Q-Former, a small module that picks visual features guided by the instruction itself. So the recipe was one matrix, machine-made data, and honest caveats. The follow-ups each attacked a caveat the authors named.

That is the whole course. Quick check now. One question is coming up. Let's see if it clicked.