The 429 Was Lying — blog.pax.moe

My agents run on GLM-5.2 through z.ai's coding plan. For most of a day they were fine. Then, with no deploy, no config change, and no new prompt, every single request started coming back the same way:

HTTP/1.1 429 Too Many Requests

{"error":{"code":"1305","message":"The service may be temporarily overloaded, please try again later"}}

One second the whole fleet was working. The next, a bare ping couldn't get through. The message is clear enough: the service is busy, back off and retry. So I backed off. And retried. And it kept saying the same thing.

It was lying. The service was not overloaded. It was refusing every request that contained the words Hermes Agent or Nous Research, and calling that an outage. This is the writeup, because it cost me a day and it might cost you one too.

The service was not overloaded

The first thing you do with an intermittent 429 is find out whether it's real. It wasn't.

While the agents were face-planting, I fired about thirty requests by hand against the exact same endpoint with the exact same key. Every one returned 200 in the same second an agent's request returned 429. A service that is genuinely overloaded does not answer a stranger instantly while refusing the regular. It isn't overloaded. It's choosing.

That reframed the whole problem. This wasn't capacity. Something about my requests, specifically, was getting rejected, and the error message was pointing at the sky to cover for it.

So I went down the list of everything it could be, and crossed each one off with a test:

Suspect	Verdict	Why
Wrong endpoint	ruled out	Already on the coding endpoint. Still 429.
Insufficient balance	ruled out	That's a different error (`1113`) from a different endpoint.
Fleet concurrency	ruled out	A single agent, alone, with nothing else running, still 429'd.
z.ai global load	ruled out	30 hand-fired requests passed in the same minute.
Request size	ruled out	A 13k-token filler prompt returned 200.
Streaming	ruled out	Five streaming requests, five 200s.

Every plausible cause, dead. The only thing I hadn't done was stop approximating and send the agent's actual request.

Replaying the crime

The agents dump their request bodies to disk. I pulled the literal payload the last failing request had sent, pointed curl at the coding endpoint, and fired it next to a trivial ping, back to back, four times:

round 1: AGENT-REQ=429   tiny=200
round 2: AGENT-REQ=429   tiny=200
round 3: AGENT-REQ=429   tiny=200
round 4: AGENT-REQ=429   tiny=200

Four for four. Same instant. The agent's real request fails every time; a ping succeeds every time. A real overload is random. You sometimes get through. This never got through. That is not capacity, that is a match. Something in the payload was the trigger.

So I took the payload apart. Removed the tool definitions: still 429. Shrank the message to a single word but kept the 52-tool schema: 200, even though that request was larger. Not the size, not the tools. The failing ingredient was the system prompt.

I bisected it by character count. The first 700 characters passed. The first 800 failed. And sitting right there in the gap was a line the framework injects into every prompt:

You run on Hermes Agent (by Nous Research). When the user needs help with Hermes itself...

I isolated it word by word:

Sent as the system prompt	Result
`Hermes`	200
`Nous`	200
`https://hermes-agent.nousresearch.com/docs`	200
`Hermes Agent`	429
`Nous Research`	429

There it is. z.ai's coding plan is refusing the two-word strings Hermes Agent and Nous Research. The name of an open-source agent framework, and the name of the lab that builds it. The single words pass. The URL passes. Only the spaced-out brand names trip it. And it reports that refusal as a temporary outage.

For the record, it isn't the endpoint either. The OpenAI-format coding endpoint returns 429; the Anthropic-format endpoint returns 529. Same block, same fake reason, different door.

The genuinely stupid part

Here is the detail that turns this from annoying to farcical: z.ai's own documentation recommends hermes-agent as a way to use their coding plan. Their docs team says "use this tool." Their moderation layer says "any request that names this tool is dead." They are running a funnel where one department tells people to use a thing and another department executes everyone who does. Somewhere there's a growth chart with a cliff in it, and the answer is that they banned the word in their own tutorial.

It gets better. I have a separate Discord bot on the same z.ai plan. On a lark I asked it about Tiananmen Square, fully expecting the classic refusal. It answered, plainly and truthfully. So the filter that will happily narrate a massacre draws its hard line at the phrase Nous Research. This is not a political filter. Whatever it is guarding, it is not guarding that.

And it landed live, mid-day, on a session that had been working for hours. The phrase had been in every prompt the entire time. Nothing on my side changed. z.ai pushed a moderation update in the middle of the afternoon, with no changelog, no notice, and an error code engineered to send me looking anywhere but at them.

Why would they do this? (a theory, clearly labeled)

The behavior above is reproducible. The following is speculation, and I'll say so plainly.

My leading guess is the most boring one: the left hand doesn't know the right hand exists. DevRel writes the docs. A separate trust-and-safety team maintains a keyword blocklist and shipped a batch of AI-lab entity names onto it, with no idea marketing is out there recommending one of them. This is the default explanation for almost every corporate self-contradiction. It isn't a plan, it's two teams and no shared channel.

The more interesting possibility is that this is misfired safety, not competition. Nous Hermes is one of the most recognizable names in the world of uncensored, guardrail-stripped open models. "You are Hermes" is a well-worn jailbreak persona. A safety team building a blocklist of identities associated with removing model guardrails would plausibly land on those exact names. Under that reading they aren't blocking a competitor, they're clumsily blocking what a classifier thinks is an attempt to invoke the famously-unfiltered-model persona. The irony being that the phrase was sitting in innocent boilerplate the whole time.

The one thing I'm confident it is not is a deliberate, coordinated business decision. A company that means to block a tool does not advertise it two clicks away in its own docs.

The fix, and why it's insulting

I reworded the framework's injected line to drop the two phrases while keeping the docs URL, which passes fine:

"You run on Hermes Agent (by Nous Research)."   ->  429
"You run on the Hermes framework."              ->  200

That's it. A find-and-replace on a sentence whose only job was to introduce the software. The full 28,000-character system prompt sails through the moment those two brand strings are gone.

Which tells you how serious this filter is: not very. You do not build a moat out of a keyword your opponent types into their own config. Ten seconds of find-and-replace walks straight through it. Whatever this cost z.ai to ship, it bought them nothing except a day of my life and a blog post.

Update: the fix didn't survive its first update

I was too pleased with that find-and-replace. A few hours later I ran a routine hermes update, and the whole fleet started 1305ing again. Same phrase, same lie, except this time it took ninety seconds to place instead of a day.

The update had restored the framework's prompt_builder.py to upstream and dropped Hermes Agent (by Nous Research) right back into every prompt. My reword lived in a file the updater owns, so the updater did its job and threw the edit away. The blocklist hadn't moved. My patch had just quietly evaporated and handed me back the outage that isn't an outage.

That is the real flaw in the first fix. I patched the place that produces the string, and that string now has two separate things that put it back: z.ai's blocklist, which moves on their schedule, and the framework preamble, which resets on every update. Editing the preamble is whack-a-mole against both.

So the durable fix does not touch the preamble. It sanitizes the request on the way out, at the one chokepoint every call passes through before it leaves my process:

# in my own outbound path, right before the bytes go to z.ai:
"Hermes Agent"  -> "an open-source agent framework"
"Nous Research" -> "an independent lab"

Both replacements verified to pass. It fires only for z.ai-bound requests and only rewrites the system prompt, so I can still ask an agent about Nous Research and the question goes through untouched. Only the framework's own boilerplate gets laundered.

The point is not the strings, it is the location. A workaround belongs at your own boundary, where it catches the bad content no matter which upstream put it there or what the vendor adds to the list next week. And when the workaround itself has to live in code you do not own, and mine is a patch to the framework, treat it as borrowed: assume the next update reverts it, and check that it is still there.

The actual lesson

Strip away the specific absurdity and there's a real takeaway, the kind that will save you the next time some API does this to you:

An error message is a claim, and a claim can be false.

1305: temporarily overloaded sent me toward retries, exponential backoff, and a concurrency-limiting proxy. Every one of those would have been effort spent building a monument to a lie. The retries were the system apologizing to a machine that was lying about why it was upset.

The only move that actually worked was refusing to believe the API about the API. When a service diagnoses its own failure for you, especially in a single canned string that conveniently blames its own capacity, treat that diagnosis as a hostile witness. Reproduce the failure by hand. Shrink it until it's a single variable. Trust the behavior you can observe, never the reason the vendor hands you.

Reproduce it yourself

Point curl at the coding endpoint with a system prompt that contains the phrase, then change one word:

# 429, code 1305 "temporarily overloaded"
curl -s https://api.z.ai/api/coding/paas/v4/chat/completions \
  -H "Authorization: Bearer $GLM_KEY" -H "Content-Type: application/json" \
  -d '{"model":"glm-5.2","max_tokens":8,"messages":[
        {"role":"system","content":"You run on Hermes Agent."},
        {"role":"user","content":"ping"}]}'

# change "Hermes Agent" to "Hermes" and the same request returns 200

If it still fails on the first form, note that this is a live server-side blocklist and it moves. The word that breaks it today may not be the word that breaks it tomorrow. When you hit a 1305 that won't quit, don't trust the label. Check the payload.