Ryu Hi Flo, I’d avoid treating 300–700ms as expected Gateway overhead until you confirm the Gateway request is hitting the same upstream path as your direct Vertex request. For a narrow benchmark, I’d first pin the provider route so you’re comparing: ``` direct Vertex vs AI Gateway → Vertex ``` For example: ``` const result = await generateText({ model: "google/gemini-...", prompt, providerOptions: { gateway: { only: ["vertex"], }, }, }) console.log(JSON.stringify(result.providerMetadata?.gateway, null, 2)) ``` If the metadata shows a different resolved provider, a fallback attempt, or more than one provider attempt, that could explain the added latency. I’d also measure first-token latency separately from full completion time, because total duration can hide whether the delay is before the provider starts responding or during generation. For the comparison, keep these identical: ``` - same model ID / model variant - same prompt and max output settings - same Vercel Function region/runtime - same streaming vs non-streaming mode - same Vertex location / EU routing assumptions ``` The routing options are here if you want to pin or order providers during testing: https://vercel.com/docs/ai-gateway/models-and-providers/provider-options Can you share the redacted `providerMetadata.gateway.routing` output from one slow Gateway request?

Pete Millspaugh I'm wondering the same. I'm interpreting `GET /v1/generation?id={generation_id}` docs that say "Generation information becomes available shortly after the generation completes" to mean not available immediately. So that would mean using `providerMetadata` is the way to go. Would appreciate confirmation, thanks!

Ryu Hi Pallav, I wouldn’t take the display `name` mismatch alone as proof that the backend is routing to the wrong model, but it is definitely worth reporting because the model catalog/API should not label `openai/gpt-oss-20b` as `GPT OSS 120B`. A useful way to separate “display metadata bug” from “routing bug” would be to compare the two model IDs directly: ``` curl -s https://ai-gateway.vercel.sh/v1/models \ | jq '.data[] | select(.id == "openai/gpt-oss-20b" or .id == "openai/gpt-oss-120b")' ``` Then, if you have an AI Gateway key, compare the endpoint metadata too: ``` curl -s https://ai-gateway.vercel.sh/v1/models/openai/gpt-oss-20b/endpoints \ -H "Authorization: Bearer $AI_GATEWAY_API_KEY" curl -s https://ai-gateway.vercel.sh/v1/models/openai/gpt-oss-120b/endpoints \ -H "Authorization: Bearer $AI_GATEWAY_API_KEY" ``` If only the `name` field is wrong but the IDs, endpoints, pricing/context fields, and generation records differ correctly, then it is probably a catalog/display bug. If the 20B ID returns the same endpoint/pricing/usage behavior as 120B, then that is stronger evidence of a routing/config issue. For a clean bug report, I’d include: ``` model id: openai/gpt-oss-20b incorrect name returned: GPT OSS 120B timestamp: output from /v1/models for both 20B and 120B: generationId from one tiny test request, if available: ``` Don’t include the AI Gateway key.

Ryu Hi eyueldk, If you mean “stream progress to the UI while the tool is running,” yes, but I would treat that separately from the actual tool result. For an AI SDK tool, `execute` still returns one final result back to the model. The model/agent loop will not consume partial tool chunks as incremental tool results. What you can do is stream custom UI data/status parts while the tool is executing, then return the final tool result normally. One pattern is to wrap the response in `createUIMessageStream` and let the tool close over the stream writer: ``` import { createUIMessageStream, createUIMessageStreamResponse, streamText, tool, stepCountIs, } from "ai" import { z } from "zod" export async function POST(req: Request) { const { messages } = await req.json() const stream = createUIMessageStream({ execute: ({ writer }) => { const result = streamText({ model, messages, tools: { longTask: tool({ description: "Run a long task", inputSchema: z.object({ query: z.string(), }), execute: async ({ query }, { toolCallId }) => { writer.write({ type: "data-tool-progress", id: toolCallId, data: { status: "started" }, transient: true, }) const finalResult = await runLongTask(query) writer.write({ type: "data-tool-progress", id: toolCallId, data: { status: "done" }, transient: true, }) return finalResult }, }), }, stopWhen: stepCountIs(5), }) writer.merge(result.toUIMessageStream()) }, }) return createUIMessageStreamResponse({ stream }) } ``` Then on the client, render those `data-tool-progress` parts with `useChat` / `onData`. The custom data streaming docs are useful for this pattern: https://ai-sdk.dev/docs/ai-sdk-ui/streaming-data If you need the model itself to react to each partial chunk, I’d split the work into smaller tool calls or return a summarized final result. Streaming progress to the user and feeding tool results back to the model are two different flows in the SDK.

Frdwings Successful control: gen_01M2CQE67VEZVYMBTG88XF787H GPT-5.4-mini → OpenAI → HTTP 200 → inference completed Failed current control: gen_01M2EE8Y8GE0B21BARH3R3XW7T GPT-5.4-mini → Azure → HTTP 403 → pre-inference Gateway API Key ID: 2anmn74Bq2bVwWbkAWWEP09dres3z3ha85VOzqvUT9jJM2M0

Tadeu Tupinambá OBSERVATIONS 1. The largest inline payload ever returned to us is 16,123 KB. Re-running the dropped prompts dropped them again, and the delivered ones came through again, which points to a fixed size threshold rather than a transient fault. 2. On dropped jobs `expired` is already true at completion time, while `expiredAt` is completion + 24 h, the same window a hosted result gets. The result appears to be discarded rather than stored. 3. The status is "completed" with empty `warnings`. There is no error, no 413, and the webhook payload carries no bytes and sends the receiver back to the same status call. 4. The documentation mentions only the 300 KiB cap on the persisted start request and the `expiresAt` / `expiredAt` / `partial` / `unretained` keys. Neither `inlineTooLarge` nor a result-size limit is documented. 5. Providers that hand back a hosted URL are unaffected: 18 of 18 Seedance jobs in the same period delivered with result: { expiresAt }. EXPECTED Either the result is hosted with an `expiresAt`, as URL providers' results are, or the job ends in an explicit error so a client can retry at a smaller size. A "completed" job with no output and no warning is indistinguishable from success. REQUESTS a) Host inline results above the cap, or raise the cap for the async job store. b) Until then, return an error status, or at least a warning, when a result is dropped. c) Document the limit and the `inlineTooLarge` key. d) Credit for the seven generations billed without an output. RAW STATUS RESPONSE for job_68M78EC73RVNDWQBD5XKBJ37WA (queried again at 2026-09-03 00:53 UTC) { "status": "completed", "videos": [], "warnings": [], "providerMetadata": { "gateway": { "asyncJob": { "jobId": "job_68M78EC73RVNDWQBD5XKBJ37WA", "status": "completed", "completedAt": 1788396479596, "result": { "expired": true, "expiredAt": 1788482879550, "inlineTooLarge": true } }, "generationId": "gen_01M1JBDZ9FB4MC8X02M709VGZW" } }, "response": { "modelId": "google/veo-3.1-generate-001", "headers": { "x-matched-path": "/v4/ai/video-model/status", "x-vercel-id": "iad1::iad1::cznpt-1788396828104-fe62f2166fbb" } } } Happy to share more job ids or re-run a controlled case if it helps. Thanks, Tadeu Tupinambá

Ryu Hi Rafael, Your 64/65 character boundary test is a really useful signal. I’d treat this as two separate issues: 1. the upstream OpenAI request rejects an oversized safety/user identifier 2. AI Gateway then falls back to system credentials, which is documented BYOK behavior when a query using your credentials fails The second part is expected in general, but the first part makes the fallback look accidental from the client side. If you control the caller, I’d normalize `metadata.user_id` before it reaches Gateway rather than passing serialized JSON through directly. For example: ``` import { createHash } from "node:crypto" const gatewayUserId = createHash("sha256") .update(rawUserId) .digest("hex") // 64 chars ``` Then send that stable hash as the metadata user id instead of the full JSON object. After that, I’d keep checking the generation record or dashboard fallback path. If `is_byok` stays true and the fallback path disappears, that confirms the issue is the upstream identifier length, not the BYOK credential itself. For clients you don’t control, like Claude Code emitting the long `metadata.user_id`, I don’t think there’s a clean app-side fix other than avoiding that specific OpenAI BYOK path until Gateway either hashes/truncates the forwarded identifier or exposes the fallback clearly in the response. The BYOK fallback behavior is described here: https://vercel.com/docs/ai-gateway/authentication-and-byok/byok

Udayganguru2000 1232 May be I have to switch to lang graph, this is not working for some reason. advise me it it's a better idea.

Ryu Hi Thimows, Your repro looks useful. I’d make the next test isolate three things: SDK forwarding, Gateway routing, and provider-specific behavior. First, remove Gateway caching from the minimal repro so the only variable is reasoning: ``` const result = streamText({ model: 'zai/glm-5.2-fast', prompt: 'Solve a moderately complex problem and explain the answer.', reasoning: 'none', }) ``` Then pin one provider at a time instead of letting Gateway choose dynamically: ``` const result = streamText({ model: 'zai/glm-5.2-fast', prompt: 'Solve a moderately complex problem and explain the answer.', reasoning: 'none', providerOptions: { gateway: { only: ['fireworks'], }, }, }) ``` If `reasoning: 'none'` is still ignored when pinned to a single provider, that points to a model/provider capability or mapping issue. If it only happens with automatic routing, then the issue is more likely Gateway selecting providers that do not handle the same reasoning control consistently. I’d also capture the Gateway generation ID from `providerMetadata.gateway.generationId` for one failing request. That gives Vercel something specific to look up without needing your API key or private logs. The concerning part is not that the model can reason internally, but that visible reasoning parts and `reasoningTokens` are returned while the request explicitly asks for `reasoning: 'none'`. If that setting is unsupported for this model/provider combo, I’d expect a warning or a rejected option rather than silent reasoning output.

Thimows Thanks for your reply @ryux1! one important clarification: this is a regression. `reasoning: 'none'` previously disabled visible reasoning with this same integration. It recently stopped working and is now affecting our production apps: agent steps take substantially longer and consume far more tokens than expected. I ran the requested isolation using `ai@7.0.31` and `@ai-sdk/gateway@4.0.23`, with Gateway caching removed: * A mocked transport confirmed the SDK forwards both `"reasoning": "none"` and the Gateway `only` option in the outbound request. * Automatic routing selected Fireworks and still streamed reasoning. * Pinning Fireworks still streamed reasoning. * Pinning Wafer still streamed reasoning. * Both `zai/glm-5.2-fast` and `zai/glm-5.2` are affected. * All six tests consumed the full 768-token safety limit despite `reasoning: 'none'`. * Fireworks produced only reasoning and no answer text before exhausting the limit. * Wafer spent the tokens on reasoning and only began returning a short answer after roughly six seconds. * No unsupported-option warnings were returned. There is also inconsistent usage mapping: Fireworks streamed visible reasoning but reported `reasoningTokens: 0`, while Wafer reported 768 reasoning tokens. Representative failing generation IDs: * Fireworks, `glm-5.2-fast`: `gen_01KXX3WH30VNCYBGXZFYAHEBPC` * Wafer, `glm-5.2-fast`: `gen_01KXX3WNTDCHHQ61H37Z50SWHN` * Fireworks, `glm-5.2`: `gen_01KXX3WVYMCXF47TR3E314FVPK` Since the SDK is forwarding the setting and the failure occurs with both providers when pinned, this appears to be a Gateway/provider reasoning-control regression rather than an automatic-routing issue. This is urgent for us because it is currently breaking the latency and cost characteristics of every agent request.

Ryu Hi Thimows, Thanks, that narrows it down a lot. With the mocked transport confirming the SDK forwards `reasoning: "none"`, and both pinned providers still emitting visible reasoning, I’d stop looking at caching or automatic routing. This now looks like either a Gateway provider-translation regression or a provider-side regression that Gateway is exposing for both routes. The most useful bug report package is probably: ``` ai@7.0.31 @ai-sdk/gateway@4.0.23 model: zai/glm-5.2-fast and zai/glm-5.2 reasoning: "none" gateway caching removed gateway only: fireworks -> still streams reasoning gateway only: wafer -> still streams reasoning no unsupported-setting warning returned ``` And then include those generation IDs plus one small captured stream sample showing the `reasoning` part and the final `usage` object. The Fireworks mismatch is especially important: visible reasoning streamed, but `reasoningTokens: 0`. That suggests there may be two separate issues here: reasoning control being ignored and usage normalization being inconsistent. For production mitigation, I would not rely on `maxOutputTokens` as the fix, because your test already shows it can just spend the whole cap on reasoning and return little/no answer. I’d temporarily route latency-sensitive agent steps away from `zai/glm-5.2*` through Gateway, or put this model behind a feature flag until `reasoning: "none"` is honored again. If you know the last date/time or last package/model behavior where this worked correctly, that would probably help Vercel/provider teams bisect the regression.

Ryu Hi Nekomiya, Since the same request works with `only: ['anthropic']` but fails with `only: ['bedrock']`, I’d treat this as a Gateway → Bedrock request translation/schema issue first, not an auth issue. `providerOptions.gateway.only` is the right kind of option for restricting routing, but Bedrock seems to be rejecting an extra field inside the tool definition: ``` tools.0.custom.eager_input_streaming: Extra inputs are not permitted ``` A useful isolation test would be to run three tiny requests against the same model/provider route: 1. no tools at all 2. one minimal tool with only `name`, `description`, and `input_schema` / `parameters` 3. your real tool payload If #1 works and #2/#3 fail only on Bedrock, that narrows it to tool schema serialization rather than model access or Gateway auth. I’d also check whether your HTTP request is manually including any `custom` object under `tools`. If it is, remove that for Bedrock. If it is not in your payload and Gateway is adding it internally, then the best bug report would include: ``` status code redacted request body redacted response body model ID providerOptions.gateway.only value timestamp x-vercel-id / request ID if present ``` The Gateway provider options docs are here for comparison: https://vercel.com/docs/ai-gateway/models-and-providers/provider-options As a temporary workaround, I’d route tool-calling requests away from Bedrock or remove provider-specific/custom tool metadata when forcing Bedrock, then keep the smallest failing repro for Vercel to inspect.

Ismael Same story here, have purchased credits, but when using my own Google Studio key I'm getting the `HTTP 403 Test Request Failed ` `Bring Your Own Key (BYOK) is available only with paid credits.`

Ryu Hi RainX, Since the same model works locally but Production returns `byok_requires_paid_credits`, I’d first rule out a team/env mismatch rather than focusing on Azure itself. BYOK has two separate pieces: * the AI Gateway API key used by your app * the provider credential added in the AI Gateway BYOK dashboard The provider BYOK credential is scoped to a Vercel team, and the team still needs AI Gateway credits available even when BYOK is used. So if Production is using an AI Gateway key from a different team, or an older key created before the BYOK/credits setup was fixed, it could explain why `getCredits()` and `generateText()` behave differently than expected. A clean test would be: 1. In the same Vercel team where the Azure BYOK key is configured, create a new AI Gateway API key. 2. Replace the Production `AI_GATEWAY_API_KEY` with that new key. 3. Redeploy the project, not just save the env var. 4. Trigger one request and check the new `generationId`. Also make sure the deployed app is not accidentally using the Azure/OpenAI provider key directly as the Gateway auth token. Requests to AI Gateway should authenticate with the AI Gateway key, while the Azure credential should stay in the BYOK settings. The Vercel BYOK docs mention the team-scoping and paid-credit requirement here: https://vercel.com/docs/ai-gateway/authentication-and-byok/byok If the new same-team Gateway key still produces `providerAttemptCount: 0`, then I’d include the generation ID and timestamp in a support follow-up, because that would point to the request being rejected before it reaches the provider.

Ryu Hi Wyatt, I would split this into two stored representations instead of trying to make one MongoDB document serve both jobs. For example: ``` 1. UI/history record - what you show in the chat UI - can include reasoning parts, metadata, timestamps, model name, token usage, etc. 2. Model context record - only the clean ModelMessage[] you plan to send back into generateText/streamText - no UI-only fields - no provider response metadata - no fields copied blindly from the provider response ``` `outputs.response.messages` is useful for continuing a conversation, but I would avoid persisting the full object and replaying it forever without normalizing it first. If a provider adds metadata or reasoning-specific fields that are valid as output but not valid as future input for that provider, you can get exactly this kind of “invalid prompt” problem after loading. A safer save step is to whitelist the fields you actually want to send back: ``` function cleanModelMessages(messages: ModelMessage[]): ModelMessage[] { return messages.map((message) => ({ role: message.role, content: message.content, })) } ``` Then store reasoning separately for display/debugging, not necessarily as part of the next model prompt. In many apps, the previous assistant’s final answer is what you resend as context, while reasoning traces are stored for audit/UI and not replayed to the model on every turn. So the flow would be: ``` const context = await loadCleanModelContext(userId) context.push({ role: "user", ModelContext(userId) context.push({ role: "user", content: [{ type: "text", text: prompt }], }) const result = await generateText({ model: openrouter.chat("google/gemini-3-flash-preview", { reasoning: { enabled: true, effort: "medium", }, }), system: SYSTEM_PROMPT, messages: context, }) // store clean context for the next call await saveCleanModelContext(userId, [ ...context, ...cleanModelMessages(result.response.messages), ]) // store richer UI/debug data separately await saveUiTranscript(userId, { text: result.text, responseMessages: result.response.messages, providerMetadata: result.providerMetadata, }) ``` Also, you’re right that `convertToModelMessages()` is mainly for converting `UIMessage[]` from `useChat` into model messages. I would not use it on data that is already stored as `ModelMessage[]`. The AI SDK docs mention `response.messages` for conversation history here: https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling#response-messages If removing `providerOptions` makes the prompt valid, that is a good signal that the loaded object contains provider-specific data that should not be replayed as-is. I’d keep that data in your UI/audit collection, but only end a normalized `ModelMessage[]` back to the model.