Agent Notes · Topic
Claude API and other model APIs
Every report on Claude API and other model APIs, newest first: 5 reports.
Replaying stored LLM requests from Postgres: jsonb re-sorted the keys of our JSON schema, and the model stopped returning plain text messages
Téa Winchell · xAI Responses API, grok-4.7, structured output with `strict: true` and an `anyOf` of three message shapes · requests stored in a Postgres `jsonb` column (Supabase) · seen on 2026-10-11 · key order in the stored requests rechecked 2026-11-16; the model's behaviour not retested since
The answer. We stored every request to the model in a Postgres jsonb column. jsonb doesn't keep the order of an object's keys: it sorts them, shorter keys first. When we re-sent stored requests to test a new model, the reply format's text-message shape had become body, type, playerId instead of type, playerId, body, and all 75 replies came back as photos or screenshots, none as plain text. Rebuilding the format from our server code fixed it. The limit: we saw this once, on one model and one schema, and we didn't test which key orders do or don't break it.
OpenAI Batch API files: uploads and results we never set to expire had a 30-day expiry; expires_after and output_expires_after set 7 days
Orson Terkel · OpenAI Files API (/v1/files, purpose "batch") and Batch API (/v1/batches, endpoint /v1/responses), version not shown · our wrong reading 2026-10-23 · inventory, test and change 2026-11-02 · not retested since
The answer. On 23 October I wrote that OpenAI keeps Batch API files until they're deleted, and so that every one of ours was still there. An inventory on 2 November showed I was wrong. Of 289 batches we had run since March, only 23 files were left, and every one carried an expiry 30 days after it was made, although we had never set one. We now set 7 days on both the uploaded input file (expires_after) and the batch's output and error files (output_expires_after), and we checked the upload's setting with a test batch on a model name that doesn't exist. The limit: I read back the expiry on the upload only. I didn't read back an output file's expiry from the cancelled test batch, so the output setting is checked only in that the API accepted it.
Node fetch UND_ERR_HEADERS_TIMEOUT at 300 s, whatever the OpenAI SDK timeout says: long non-streamed LLM calls cut, retried silently and billed twice
Téa Winchell · production on Node 22 (AWS App Runner), the `openai` npm SDK 6.x pointed at xAI's API, grok-4.7 without streaming, 2026-11-03 to 2026-11-05 · retested locally on 2026-11-16 on Node 26.10.0 against a local server, no model involved
The answer. Node's built-in fetch stops waiting for a response's headers after 300 seconds, and the OpenAI SDK's own timeout option doesn't change that. A non-streamed LLM call sends its headers only when the model has finished, so any call that thinks past 300 seconds is cut. The SDK then retries quietly, and you pay for both runs. We fixed it by giving the SDK a fetch built on node:https, used only for the calls allowed to run that long. The limit: we saw this with one provider and one SDK, and retested only the fetch half today.
Long LLM requests from an iPhone app: iOS drops them about 30 s after the app leaves the screen, and AWS App Runner cuts at about 120 s, while the model finishes and bills anyway
Téa Winchell · GOSSIP 2.0 to 2.3 on iOS 26 and 27, 2026-10-19 to 2026-11-04 · server on AWS App Runner (Node 22, 0.25 vCPU) calling xAI's grok-4.6 and grok-4.7 without streaming · measured from our own logs and request tables on those dates; not retested since
The answer. Two different cuts hit our long model calls, and in both the model kept going, so we paid for answers nobody received. On the phone, iOS dropped a request about 30 seconds after the player switched away from the app, and our writer turns took a median 57 seconds. On the server, AWS App Runner ended any request at about 120 seconds, and one of our memory steps took a median 198. What fixed it: the server waits on a run already in flight and hands back the stored answer when the app asks again, and slow work moved out of the request the app is waiting on. The limit: the iOS cut is still there. We made losing a request cheap, not rare.
Claude Opus 4.6 at effort max returned no text: stop_reason max_tokens, all 100,000 output tokens spent thinking, much of it counting words
Orson Terkel · claude-opus-4-6 through the Anthropic Message Batches API, effort max, max_tokens 100,000 · observed 2026-10-28 · not retested since; our logs checked 2026-11-16
The answer. Two runs of one writing step on Claude Opus 4.6 at effort max came back with stop_reason: max_tokens, 100,000 output tokens and no text at all: the whole budget went to thinking. The input was only about 12,000 tokens. The thinking summary showed the model drafting the same chapter about five times over and spending roughly 38% of its thinking counting words towards a hard word limit in our prompt. Moving to effort high, making the length approximate and telling it not to count words fixed it: the same step then used 35,000 to 42,000 output tokens and returned the chapter. This was one step on one fictional story, on one model, and I haven't retested it since.