Innhold Show
A bug in Cloudflare Workers AI caused GLM-5.3 to answer with data from other customers’ requests. It affected 155 responses between August 27 and September 9, 2026, and this was a real breach of the customer boundary – not merely an AI model hallucinating something strange.
The failure appeared when exceptionally long requests hit a KV cache smaller than the model’s advertised context window. SGLang was supposed to reject the request safely. Instead, it allowed the request to continue with stale cache pages left by previous users. The client could receive a coherent response and an HTTP 200 status even though parts of the content came from someone else.
Cloudflare fixed the bug and returned the model to service on September 9, then published its coordinated disclosure on October 26. The software defect was serious, but the launch decision makes it worse. Cloudflare knew the KV pool was smaller than the context window before release, yet its tests checked status codes and engine performance without verifying whether the model’s answer was actually correct. That is not good enough for multi-tenant AI infrastructure.
What happened to GLM-5.3 on Workers AI?
Cloudflare confirmed that GLM-5.3 on Workers AI could return text from other customers’ recent requests. According to Cloudflare’s incident report, 155 responses were affected, all triggered by requests longer than 681,280 tokens.
The exposure began when @cf/zai-org/glm-5.3 launched on August 27. Security researcher Gadi Evron of Knostic alerted Cloudflare late on September 8, and the company reproduced the leak with two independent accounts the following day. Cloudflare made the model private 40 minutes after declaring a security incident.
Cloudflare says it found no evidence of intentional exploitation. Once traffic from its own tests and the researcher’s work was removed, the volume and timing looked like ordinary use of long contexts. That is reassuring, but it does not make 155 cross-customer responses acceptable. No detected attack is not the same thing as a customer boundary that worked.
What caused the Cloudflare GLM-5.3 data exposure?
GLM-5.3 was offered with a 1,048,576-token context window, while the KV cache pool on Cloudflare’s prefill GPUs could hold only 681,280 tokens. A valid request could therefore fit within the model’s advertised limit while being too large for the memory that actually had to process it.
The KV cache stores intermediate model states while a sequence is processed. It speeds up generation because the model does not need to recompute the entire history for every new token. In Cloudflare’s deployment, separate groups of GPUs handled prefill and decode. Prefill read the prompt and built the cache; decode used that cache to generate the answer.
The cache memory was organized as a shared pool of pages reused across requests. Those pages were not zeroed between customers. The design relied on new data overwriting the old content and on SGLang correctly tracking which request owned each page. That can be safe when every allocation and transfer follows the expected path. Here, the rejection path broke those assumptions.
Two failures had to line up
The first failure was Cloudflare’s configuration. The KV pool was too small, and the company knew about the mismatch before launching the model. Cloudflare assumed SGLang would reject oversized requests safely. Its tests returned successful status codes and a stream of data, so the deployment was approved.
That was a dangerous assumption. Cloudflare says its pre-launch testing checked only HTTP status and engine performance with long contexts. Nobody validated the content or coherence of the generated answer. An HTTP 200 response was treated as evidence that the system worked, even though that same status later concealed the failure.
The second failure was in SGLang. When a request could not fit in the KV pool, the system marked it for rejection and replaced it with a one-token placeholder. In a standalone deployment, the client would eventually have received an error. In the distributed prefill/decode setup, however, the rejected request was still treated as active and sent forward.
The decode node had already reserved cache pages for the full original request. Prefill wrote only the single page required by the placeholder, then signaled that the transfer was complete. Decode assumed every reserved page had been overwritten. The remaining pages still contained prompts and partial generations from earlier users, and the model could read them while producing its response.
Why did monitoring miss the exposure?
This is the uncomfortable part: The answer could look normal. The client received HTTP 200, the token count matched the long request, and the generated text could be coherent. It could even arrive faster than a genuine prefill of the complete prompt should have taken. Without checking content, processing time, and cache ownership, the usual operational signals stayed green.
Cloudflare says every request above the 681,280-token threshold entered this faulty path. The confirmed scope remained 155 responses because that threshold is extraordinarily high. Only GLM-5.3 was affected on Workers AI. Other distributed models had KV pools larger than their context windows, so they did not reach the same rejection path.
The underlying problem had been reported to the SGLang project on July 6, but Cloudflare says it found that report only during the incident investigation. A related fix was merged into SGLang on September 6, two days before Evron’s notification, but had not yet reached a release. The gap between source code, released versions, and production deployments can become a security problem very quickly.
How did Cloudflare and SGLang fix the bug?
Cloudflare first increased the KV pool to about 2.04 million tokens, nearly twice the 1,048,576-token context window. That gives every valid request room in the actual pool. A larger pool alone would be a fragile fix, though, because quantization, parallelism, model weights, and other GPU settings can change available capacity during a later deployment.
The rejection path was fixed as well. Requests marked for rejection must now stop before bootstrapping, memory allocation, and KV transfer. SGLang pull request 38935, merged on September 16, prevents a rejected request from entering the transfer as though it were still active.
Cloudflare tested the changes with both the researcher’s reproduction and its own markers in the cache. It also added a mandatory check of real KV pool capacity against the context window whenever distributed deployments change. SGLang added further protections around aborted transfers, delayed cache-page release, and checksums for transfers between prefill and decode.
The SGLang team’s technical security update shows why this was more than one missing conditional. When GPU memory moves between nodes over RDMA, delayed writes can hit pages after the system believes they have been released. The team tested delayed release by aborting 132 requests across two H200 GPUs. There were no crashes, and every held cache slot was returned exactly once.
What should Workers AI customers do?
If you used @cf/zai-org/glm-5.3 on Workers AI between August 27 and September 9, 2026, retrieve your logs and requests from that period. Look especially for prompts longer than 681,280 tokens, responses that arrived unexpectedly quickly, and content unrelated to your own session. Contact Cloudflare if you need to determine whether your account or traffic was part of the incident.
If those requests contained API keys, passwords, customer records, or other secrets, I would not wait for perfect proof. Rotate the affected keys and credentials, document the time window, and assess what information may have been exposed. That is a precaution, not a claim that every prompt during the period leaked.
The incident is also a reminder that very long prompts can put an enormous amount of sensitive context in one place. RAG results, source code, logs, and customer cases can easily end up in the same request. Minimize the data before sending it, keep secrets out of model context where possible, and retain enough traceability to reconstruct what was processed if a provider reports an incident.
What should SGLang operators do?
If you run SGLang in distributed prefill/decode mode, the advice is clear: Upgrade to v0.5.22 or later, or backport the required fixes. Measure the KV pool on every prefill and decode node after changing the model, quantization, parallelism, or memory allocation. Do not treat the model’s context window or a proxy limit as proof that a request fits in the real cache.
Enable delayed release of KV pages on both roles, limit Mooncake transfer sizes, and use checksums and KV canaries where the model architecture is supported. Cloudflare says those checks do not yet fully cover GLM-5.3, GLM-5.3 Flash, or DeepSeek V4, among others. That detail matters. A security control in a configuration file does little if the logs show it does not support the model you are actually running.
And test the result, not just the plumbing. A status code, an uninterrupted stream, and good throughput prove that the service returned something. They do not prove that the right customer received the right content from the right memory. This incident is a textbook example of why security tests must validate data provenance and coherence, not merely whether the endpoint stays up.
My assessment of the Cloudflare incident
Cloudflare deserves credit for an unusually detailed technical disclosure, a fast response once the incident was confirmed, and its work with SGLang on upstream fixes. Reports like this make the entire ecosystem better. But transparency afterward does not erase the failure before launch.
The company knew about the specific mismatch between a 1,048,576-token context window and a 681,280-token KV pool. It still treated a successful HTTP response as evidence that everything was fine. If a test does not even inspect the model’s answer, it is not an adequate test of a multi-tenant AI service. Full stop.
It is tempting to dismiss 155 responses as a small number inside a large cloud service. Wrong yardstick. What matters is that the infrastructure crossed a customer boundary without reporting an error and delivered the result as a plausible answer. Users must be able to trust that their model context will not become raw material for the next customer. That trust failed here, and both providers and self-hosters should treat the lesson as a security gate – not a footnote in a changelog.
Frequently asked questions
Which Cloudflare customers were affected by the GLM-5.3 bug?
The exposure affected customers using @cf/zai-org/glm-5.3 on Workers AI from August 27 through September 9, 2026. Cloudflare identified 155 affected responses, all following prompts longer than 681,280 tokens. The company says no other Workers AI models were affected.
Do 155 affected responses mean only 155 prompts could have leaked?
Cloudflare reports 155 responses containing data from other customers’ earlier requests. That figure counts confirmed affected responses, not necessarily every source prompt whose cached data contributed to them. Cloudflare also says it found no evidence of deliberate exploitation.
Is GLM-5.3 on Workers AI safe to use now?
The specific failure has been fixed, and Cloudflare returned the model to service on September 9 after enlarging the KV pool and patching the rejection path. No cloud service can be guaranteed error-free, so sensitive information should still be minimized and handled with normal security controls.
Which SGLang version should operators use?
Cloudflare recommends SGLang v0.5.22 or later for distributed prefill/decode deployments. Operators should also verify real KV capacity against the context window, use safe delayed release of cache pages, and confirm in their logs that the selected integrity checks support the deployed model.
