Multimodal AI is a system that processes and produces more than one type of data, such as text, images, audio, video, and documents, in a single workflow. For a web developer, that definition has a practical edge, because you reach all of it through standard HTTP APIs. You no longer need GPU clusters or custom ML infrastructure to ship a feature that reads a PDF, transcribes a meeting, or answers in spoken audio. The model providers handle the heavy computation. The remaining work is orchestration, streaming, and the user experience around it.
A team ships their first multimodal feature, it demos beautifully, and three weeks later, they're firefighting a request lifecycle nobody designed. Usually, it starts small: a model rejecting a file type in production that it happily accepted in the prototype, or a bill that doubled because no one noticed every request was shipping a full-resolution image, or the same audio file getting transcribed for the fourth time this week. None of it traces back to the model, which is exactly why it blindsides teams. The model was the easy part.
The work crept in while no one was looking. Our AI Gateway data shows that tool-call requests now make up the majority of all tokens flowing through it, up sharply from six months prior. Requests stopped being "text in, text out" and started carrying images, documents, and audio, and the infrastructure assumptions behind "call an LLM" quietly broke underneath them. This piece covers what changed underneath all of it, how the AI SDK handles multiple modalities, what our AI Gateway data reveals about real production usage, the Next.js patterns that hold up under load, and why the cost objection teams still raise is a year out of date.
Copy link to headingMultimodal stopped being a feature tier
The bar for "multimodal" has moved twice in about two years, and both times, teams planned around the old bar. First, it meant a model could accept an image. Then, by late 2025, it meant building an end-to-end product across modalities inside one workflow. Realtime audio APIs went GA. PDFs, images, audio, and video became standard inputs in production pipelines, not preview features behind a ticket.
When every frontier model can see an image, model choice no longer makes or breaks the feature, and routing becomes the deciding factor. Teams that struggle here tend to make the same move: spend weeks picking the perfect model and an afternoon on the plumbing, then spend the next quarter rebuilding the plumbing after the first model rejects an input it was never going to accept. The capability is table stakes now. What a team builds around the capability is the whole game.
The convergence across providers is already close to complete.
Every model in that table clears the bar. The differences live at the edges now, context window, audio latency, maximum image resolution, not in whether a model can take a modality at all. The gap between "text-only" and "multimodal" is no longer a feature tier to design around. It's the baseline, and treating it like a premium capability is where teams tend to waste a month.
Copy link to headingHow multimodal actually works once you're building it
The textbook definition gets in the way more than it helps. For the record, multimodal AI refers to systems that process and integrate multiple data types such as text, images, audio, and video. Where a unimodal model works with one type of data at a time, a multimodal model takes several in the same request and reasons across them. That's accurate, and not particularly useful once you're building against it.
Under the hood, the mechanics are consistent from one provider to the next. Each modality runs through its own encoder that turns it into vectors, a fusion step maps those vectors into a shared representation space, and the model reasons over the combined result. That's the genuinely hard part, and the providers own it. It isn't where developer time goes.
The version that matters when building is simpler: a modality is just another part of a request. Instead of a model that "understands images," picture an HTTP request that happens to carry an image, a paragraph of text, and a PDF in the same payload, with a provider on the other end that knows how to read all three. The providers own the hard computation, turning pixels and waveforms into something a model can reason over. Everything around the request stays the developer's job: assembling the parts, streaming the response back, falling back when a provider can't handle one of the parts, and not paying to do the same work twice. Once modalities are treated as request parts instead of model magic, every pattern in the rest of this piece follows from that one idea.
Copy link to headingThe AI SDK treats modalities as message parts, not separate APIs
We built the AI SDK as a unified TypeScript toolkit across React, Next.js, Vue, Svelte, and Node.js, and the design decision that has aged best is treating modalities as content parts inside a message rather than as separate API surfaces. An image, a text prompt, and a file attachment all live in the same message array passed to generateText or streamText. There is no second SDK for vision and no special client for audio. It's one message shape with different kinds of parts.
import { streamText } from "ai";import { openai } from "@ai-sdk/openai";
const result = streamText({ model: openai("gpt-5.6-sol"), messages: [ { role: "user", content: [ { type: "text", text: "What is in this image?" }, { type: "image", image: imageUrl }, ], }, ],});On the client, the useChat hook converts files into data URLs and sends them along automatically. Image and text content types become multimodal content parts without any manual encoding. On the server, convertToModelMessages translates UI messages into each model's expected format, multimodal parts included.
This is also why provider switching costs one line. The same message structure runs across OpenAI, Anthropic, Google, Mistral, Amazon Bedrock, and others through dedicated provider packages (@ai-sdk/openai, @ai-sdk/anthropic, @ai-sdk/google, and more). Zo's engineering team put it plainly: "AI SDK replaced the custom adapter code. Instead of per-provider implementations with bespoke edge case handling, Zo's engineers got a unified interface for every model, from image support to response format normalization."
Vision is only part of it. The SDK ships generateSpeech for text-to-speech through OpenAI, ElevenLabs, and LMNT, and a .transcription() factory for audio-to-text through Deepgram, AssemblyAI, Rev.ai, and Gladia. For structured data, use streamText with Output.object(...) to stream a schema-shaped result as it is generated, the right fit when multimodal document analysis needs to populate several UI fields progressively.
Today, only image/ and text/ content types get converted into multimodal content parts automatically. Everything else requires manual handling. Audio input processing, as distinct from transcription, and general video input for providers beyond xAI aren't fully documented yet. The SDK's surface is filling in, but teams building document processing across arbitrary MIME types right now should budget for some manual handling.
Copy link to headingWhat 16,000 runtime hours reveal about multimodal infrastructure
The Gateway's runtime breakdown reframes how AI workloads should be billed. A third-party TrueFoundry comparison estimates roughly 10ms of routing latency per AI Gateway request. That number matters because of what it implies. The gateway spends almost all of its time waiting, not computing. Routing a request, verifying credentials, and checking quotas takes milliseconds. Waiting for OpenAI or Anthropic to answer takes seconds.
The data makes it concrete. In the first month of AI Gateway availability, it handled roughly 16,000 total runtime hours. Only 1,200 of those involved actual CPU work. The other 14,800 hours were spent waiting for providers to respond. A workload with that profile bills for idle time under wall-clock billing.
This is the exact profile Fluid compute was built for, because it bills by execution time instead of wall time. An AI call that takes 30 seconds to come back but uses 300ms of compute costs 300ms. The wait is covered through memory reuse, and a single Fluid compute instance handles multiple inference requests at once. The default 300-second execution ceiling, up to 800 seconds on Pro and Enterprise, leaves room for the long-running multimodal jobs that would otherwise time out.
For multimodal work specifically, fallback chains matter most. A primary model that can't take a given input type falls back to one that can, specified as a models array in providerOptions.
providerOptions: { gateway: { models: [ "google/gemini-3.1-pro", "anthropic/claude-sonnet-5", "meta/llama-4-maverick", ], },},Any error triggers the next model in line, whether that's a context limit, an unsupported input, or a provider outage. When a fallback fires, the modelAttempts array in the provider metadata shows every model tried, with both the gateway's normalized name (canonicalSlug) and the provider's own ID (modelId). That metadata turns a silently rejected file type in production from an afternoon of guessing into a five-minute fix.
One constraint worth planning around: images and audio have to be base64 encoded going through AI Gateway. File URL data doesn't work across all models. Some models reject it outright, though others take URLs fine. Encoding is part of routing, not a detail to assume away, so confirm which models in a fallback chain support which mode before shipping.
On pricing, tokens pass through at list price from the upstream providers with no markup. Bring-your-own-key usage adds 0%.
Copy link to headingGamma ships 250 deploys a day across 60 image models
Gamma is a useful reference for teams that assume a multi-provider image pipeline is too much to maintain. Gamma runs one in production that has generated more than 1.5 billion images across 60 models and 20 providers. The team adds a new image model in days using the AI SDK's ImageModelV3 interface, a composable middleware layer for image generation. A new model takes about 30 lines of code, with tracing, cost tracking, and image preprocessing handled for them.
The finding from their work that matters most is that the abstraction layer isn't where the real effort goes. Provider-specific prompt strategy is. Gemini wanted multimodal style references, actual images showing the target aesthetic, while Flux did best with concise text-only prompts. A provider-agnostic SDK handles the plumbing, but prompt strategy still varies by provider, and any framing of a fully provider-neutral multimodal experience undersells that.
Their deploy numbers tell the operational side. More than 250 deployments a day, a median deploy completing in just over seven minutes, and a 99% success rate. Preview deployments let the team try changes safely on every pull request, and Instant Rollbacks make shipping changes to model logic survivable.
Gamma isn't alone, and the applications run across industries. Thomson Reuters built CoCounsel, an AI assistant for attorneys, accountants, and audit teams, on the AI SDK with three developers in two months. It now serves 1,300 accounting firms and is migrating its full codebase onto the SDK, retiring thousands of lines of code across 10 providers into one system. SERHANT. shipped S.MPLE, a Next.js progressive web app that routes work across OpenAI, Claude, and Gemini by task, and scaled from 200 real estate agents to more than 800 without replatforming.
None of this is exotic anymore. Our State of AI survey of 656 application builders found 47% running 2-3 models in production, 18% running four or more, and 63% building primarily in JavaScript or TypeScript. Running multiple models is the normal case, not the advanced one.
Copy link to headingThe five Next.js patterns that actually work for multimodal interfaces
Building multimodal into a Next.js app comes down to five patterns that hold up under load. Each one solves a specific problem in the request lifecycle, and each one comes with a tradeoff worth knowing going in.
Copy link to headingStreaming with useChat and file attachments
The foundational pattern pairs streamText on the server with useChat on the client. The AI SDK hides provider-specific streaming formats behind one interface. An API route at /api/chat/route.ts uses streamText, returns the stream, and useChat() on the client consumes it. The hook takes FileList objects from file inputs directly, so a user can attach an image alongside a text message without any custom upload logic. The tradeoff is giving up fine-grained control over the upload path, fine until resumable uploads or client-side compression are needed, at which point the pattern gets outgrown.
Copy link to headingServer Actions for binary file processing
Server Actions receive form data, pull out uploaded files, and convert them to byte arrays before handing them to a provider. They have direct access to Node.js file system APIs the browser can't touch, and Server Components give progressive enhancement by default, so a form calling a Server Action still submits before JavaScript loads. This breaks down at large files. Once big binaries are in play, a dedicated upload path and a queue beat holding it all in the action.
Copy link to headingStructured output streaming with streamText
For structured multimodal results, use streamText() with an output: Output.object({ schema }) setting. The result exposes partialOutputStream, so the client can render typed fields progressively as they arrive. This is the right pattern when document analysis needs to fill several UI fields at once, pulling a title, summary, key entities, and metadata out of an uploaded PDF and rendering each field as the model produces it. The tradeoff is that streamed partial output can still be incomplete while generation is in progress, so the UI has to tolerate a half-populated state and wait for the final parsed output before treating the object as complete.
Copy link to headingConversation length management
For long conversations, summarizing older messages with a cheap model like Claude Haiku compresses history while keeping recent turns verbatim, then the summary feeds into the system prompt on later requests. The pattern writeup covers the mechanics. Recent state stays intact and context stops growing without bound. The tradeoff is real: summarization loses detail, and the detail it loses is usually the detail a user asks about later.
Copy link to headingProvider-aware fallback routing
This is the patterns version of the Gateway fallback chains covered earlier. A user uploads a video, the primary model doesn't take video, and the gateway routes to one that does without application code changing. The modelAttempts metadata shows which model actually served the request. The thing to watch is silent quality drift, since a fallback model may answer differently than the primary, and the user won't know which one they got.
Copy link to headingWhen to reach for multimodal, and when not to
The most common question is some version of "should this even be multimodal," usually asked after a team has already wired up vision because the model supported it. Multimodal earns its complexity when the input is genuinely non-text and flattening it to text first throws away signal. It doesn't earn its complexity when a plain text call would have answered the question and a modality got added because it could be.
Reach for multimodal when:
The source data is genuinely visual, audio, or document-shaped, and converting it to text first loses information the model needs
The alternative is stitching together a separate OCR, transcription, or vision service and maintaining the glue between them
The user experience depends on the model reasoning over exactly what the user is looking at
Hold off when:
A text-only call answers the question and the image or audio is decorative to the task
The few fields needed can be extracted with a cheaper, narrower tool than a frontier multimodal model
Caching, routing, and billing for the heavier requests aren't decided yet, because adding the modality is the easy 20% and the lifecycle around it is the other 80%
That last point is what separates a demo from a production feature, and it's where the cost conversation usually starts.
Copy link to headingThe cost objection to multimodal is a year out of date
The idea that multimodal AI is too expensive to run in production is about 18 months out of date. The math has moved.
Start with raw token cost. At Anthropic's documented rates, image cost tracks roughly with width times height over 750. A 200×200px image is about 54 tokens, well under a hundredth of a cent on a model priced at $3 per million input tokens. A 3-megapixel 2000×1500px image downscales to about 1,568 tokens on standard models, and runs higher, up to roughly 4,000 tokens, on a high-resolution model like Claude Opus 4.8. Gemini 2.5 Flash takes input at $0.30 per million tokens. OpenAI's cached input tokens run 90% cheaper than standard, $0.40 per million against $4.00. Per-image cost stopped being the expensive part a while ago.
The real cost driver is architectural, not per-token, and this is where the bills actually come from. A naive build sends full-resolution images on every request, reprocesses the same audio more than once, and lets conversation history grow without a ceiling. That generates a large bill at any per-token price. A cost-aware multimodal architecture leans on layered caching instead. It keys prompt caches by model, system prompt, and input, keeps transcript caches that never transcribe the same audio twice, and caches encoded image representations so common inputs aren't reprocessed.
Shopify's production deployment cut GPU usage 40% and brought median latency down from 2 seconds to 500ms with selective field extraction. The technique wasn't custom ML infrastructure. The model was asked for only the fields it needed, and the rest stopped costing anything.
On Vercel, Fluid compute compounds the savings. When a function is waiting on a provider, which the Gateway data shows is most of the time, it's billed at memory-only rates rather than active CPU rates. Falling API prices, layered caching, and compute billing that matches a wait-heavy workload add up to multimodal being cheaper to run than most teams still assume.
Copy link to headingWhere the multimodal surface still has gaps
The AI SDK started as a text tool. Images came next, then audio, speech, and structured output streaming. That growth path left three gaps worth closing.
First, file type handling should extend to every MIME type, not just image/ and text/. The current gap, where anything else needs manual handling, adds friction to exactly the document-processing features teams most want to build.
Second, provider capability detection needs to be a first-class API. Right now, developers have to know which models accept which input types and configure fallback chains around that knowledge. The gateway handles the routing, but the developer still carries the map, and a map that lives in someone's head goes stale.
Third, observability should surface modality-specific metrics from day one. Our AI Gateway observability dashboard already tracks requests by model, time to first token, token counts, and spend. As multimodal grows, teams need to see cost split by input type, image analysis versus text generation versus audio transcription, because a number that isn't visible can't be optimized.
None of these are hypothetical. They come straight from patterns visible across the platform. 47% of surveyed builders run 2-3 models, agentic workloads burn 2.6x more tokens than non-agentic ones, and teams like Gamma run 60 models across 20 providers. Looking across everything in this piece, the real divide comes into focus. It isn't between models that can handle a modality and models that can't, because almost all of them can now. It's between "calling a multimodal API" and "running multimodal in production at scale," and that gap is made almost entirely of infrastructure: fallback routing, wait-aware billing, layered caching, and observability that knows one modality from another. Teams that win treat multimodal as a request-lifecycle problem, not a model problem.
Copy link to headingFAQ
Copy link to headingWhat modalities does the Vercel AI SDK support today?
Text, image input, file attachments, audio transcription (through Deepgram, AssemblyAI, Rev.ai, and Gladia), text-to-speech generation (through OpenAI, ElevenLabs, and LMNT), and structured output streaming. Image and text content types convert into multimodal content parts automatically through the useChat hook. Video understanding is available through xAI's tools with the enableVideoUnderstanding parameter.
Copy link to headingHow does AI Gateway handle multimodal model failures?
It uses fallback chains. A primary model that fails or doesn't support a given input type falls back to the next model in a specified order until a request succeeds or the options run out. Any error triggers the fallback, including context limits, unsupported inputs, and provider outages. The modelAttempts array in the provider metadata shows which models were tried, each entry carrying a canonicalSlug and a modelId.
Copy link to headingWhat are the file size limits for multimodal inputs on Vercel?
Limits vary by provider and model, so file size is best treated as part of routing rather than a fixed Vercel constant. Vercel Workflows supports large multi-step payloads, with per-step limits in the tens of megabytes and per-run limits in the gigabytes. Images and audio sent through AI Gateway have to be base64 encoded.
Copy link to headingWhich multimodal AI model should I start with?
For cost-sensitive workloads, Gemini 2.5 Flash at $0.30 per million input tokens offers strong multimodal capability, with cheaper alternatives available if needed. For the highest image resolution, Claude Opus 4.8 handles images up to 2,576 pixels on the long edge. For real-time audio, OpenAI's GPT-5.6 delivers sub-400ms responses, averaging 320ms. Because of the SDK's provider abstraction, switching between them is a single-line change.