Skip to main content
← All projects

Project

Stream Responses End-to-End

Wire streaming all the way from the model API through your server into a browser UI. The pattern that makes AI apps feel alive.

StreamingWebPython~55 min

By the end of this project you'll have a small page where typing a question and hitting enter produces tokens that flow into the page in real time — no buffering, no spinner-then-wall-of-text. The pattern is the same whether you're using SSE, WebSockets, or HTTP streaming.

Step-by-step

The build

  1. 1. Stream from the model API

    Use the SDK's streaming method. Iterate over chunks and print to stdout with flush=True. Confirm tokens arrive as fast as the model produces them.

  2. 2. Build a streaming endpoint

    A small FastAPI / Express route that accepts a prompt and returns text/event-stream. Set Cache-Control: no-cache and X-Accel-Buffering: no so proxies don't hold the response.

  3. 3. Consume the stream in the browser

    Use fetch with a ReadableStream reader (or EventSource if you're using SSE). Append each chunk to a <div> as it arrives.

  4. 4. Handle disconnects

    What if the user closes the tab mid-stream? The server should detect the closed connection and stop billing for tokens it can't deliver. Most frameworks expose a "request cancelled" hook.

  5. 5. Polish the UX

    Add a typing cursor. Disable the input while streaming. Show a "stop generating" button that aborts the request. These are the small affordances that make a stream feel like a real conversation.

What you learned

  • Streaming is end-to-end or it isn't streaming. One buffer in the chain breaks it.
  • Browsers, frameworks, and proxies all want to buffer; you have to opt out at every layer.
  • "Cancellable" is a feature, not a refinement.
Stream Responses End-to-End — AI Fluency School