Stop 16 · AI APIs & Cloud AI PlatformsTopics · 4/9

Streaming Responses

In short

A 20-second wait with a spinner feels broken. The same reply appearing word by word feels fast. Streaming doesn't make the model quicker, it makes waiting feel shorter.

Receiving a model's output token by token as it's generated instead of waiting for the full response — essential for responsive chat UIs.

The blob listens excitedly to a small radio while a folded newspaper lies ignored on the table.

Think of it like this

Live commentary vs tomorrow's paper

You could read the full match report in tomorrow's newspaper, or hear the commentator call every ball live on the radio. Same match, very different feeling. Streaming is live commentary: you see tokens the moment the model writes them.

Watch it happen

Server-sent events, one chunk at a time

Browser

Your server

LLM API

POST /chat

1/6The user sends a question. Your server holds the API key, so it makes the LLM call.

The big ideas

What you need to know

Time to first token

Users judge speed by how soon text appears. Streaming cuts that from the full reply time to the first chunk.

It's server-sent events

Providers send a stream of small events over one HTTP response. SDKs parse them for you.

Long replies need it

Very long outputs can hit HTTP timeouts if you wait for the whole thing. Streaming avoids that.

Handle stop and errors

Let users cancel mid-stream, and handle a stream that breaks halfway without losing what arrived.

See it in code

Walk through the code

python

1import anthropic2 3client = anthropic.Anthropic()4 5with client.messages.stream(6    model="claude-opus-5-5",7    max_tokens=1024,8    messages=[{"role": "user", "content": "Write a 4-line poem about monsoon."}],9) as stream:10    for text in stream.text_stream:11        print(text, end="", flush=True)12    final = stream.get_final_message()13 14print("\nTokens used:", final.usage.output_tokens)

1/3Same request as a normal call, but opened as a stream inside a with block.

Streaming with the anthropic SDK. OpenAI and Gemini SDKs have the same idea with stream=True or a stream method.

Save yourself the pain

Common mistakes

Oops

Streaming to your server, then waiting for the full text before replying to the browser.

Do this instead

Forward each chunk to the client as it arrives, or the user sees no benefit.

Oops

Parsing JSON output on every partial chunk.

Do this instead

Wait for the stream to finish before parsing structured output, or use a parser built for partial JSON.

Your mission

Feel the difference

~20 min

Quick check

What does streaming mainly improve?

Go deeper

1Implement a streaming response in a small app and render tokens as they arrive.
2Learn the difference between server-sent events and WebSocket-based streaming for this use case.
3Handle stream interruption (user cancels) gracefully.