Stop 11 · Large Language ModelsTopics · 9/9

Multimodal Models (Overview)

In short

Snap a photo of a broken fan and ask 'what's wrong?' Modern models can look at the image, read your question and answer. Multimodal models handle text, images and often audio in one system.

Models that process and generate across multiple modalities (text, images, audio) in a single architecture — increasingly the default for frontier models like GPT-4o and Gemini.

The blob as a tour guide holds up a photo, wears headphones and points while a group listens.

Think of it like this

The tour guide who sees and hears

A good tour guide at a heritage site looks at what you're pointing at, listens to your question and explains in your language. A text-only model is a guide with a blindfold. A multimodal model takes the blindfold off.

Watch it happen

Asking about a photo

User

App

Model

photo + 'what's this?'

1/5The user uploads a picture of a plant leaf with brown spots and asks a question.

The big ideas

What you need to know

Vision encoder + LLM

A common design: an image encoder turns pictures into vectors that are fed into the language model alongside text.

Images cost tokens

Each image is converted into many tokens, so large or many images use up context and money quickly.

Great for documents

Screenshots, receipts, charts and handwritten notes can be read directly, often better than old OCR pipelines.

Still makes things up

Models can misread small text or invent details in images. Check important answers.

See it in code

Walk through the code

python

1# pip install anthropic   (most providers use a similar shape)2import base64, anthropic3 4img = base64.b64encode(open("leaf.jpg", "rb").read()).decode()5client = anthropic.Anthropic()          # reads ANTHROPIC_API_KEY6 7msg = client.messages.create(8    model="<a current vision-capable model>",9    max_tokens=300,10    messages=[{"role": "user", "content": [11        {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": img}},12        {"type": "text", "text": "What might be wrong with this leaf?"},13    ]}],14)15print(msg.content[0].text)

1/3Read the image as base64 and create a client using your API key from the environment.

One message can hold an image part and a text part. Check your provider's docs for current model names.

Save yourself the pain

Common mistakes

Oops

Sending huge 12-megapixel photos for a simple question.

Do this instead

Resize images first. Smaller images use fewer tokens and are usually just as clear to the model.

Oops

Trusting the model to read tiny numbers in a blurry screenshot.

Do this instead

Crop to the important area and ask the model to quote what it reads, then check it.

Your mission

Explain a chart

~20 min

Quick check

In a common multimodal LLM design, how does the language model 'see' an image?

Go deeper

1Try a multimodal model with an image + text prompt and observe its reasoning.
2Learn conceptually how images get tokenized/embedded alongside text.
3Identify a real use case where multimodal input beats a text-only pipeline.