13 November 2026

Changing one agent while the fleet keeps running

Kunwar Sangram Singh and Krishna Guda

Your agents are running. What started as one agent calling two tools is now many of them, spread across replicas, reaching systems that matter. That part scaled the way the rest of your platform scaled: add instances, put a gateway in front, keep the authorization decision on the hot path cheap.

Scaling kept working because nothing on that path has to agree with anything else. Each replica answers on its own, holds no shared state, and asks nobody. Add a hundred more and the picture does not change. That property is what let you say yes to the next team, and the one after it.

Then one of them turns out to be doing something it should not.

This is a day two problem, and it only shows up once the fleet is big enough to matter. Standing the fleet up is the half that gets designed. Changing it while it runs is the half that decides whether the agents stay in production.

The change you need is narrow

Nothing is wrong with the fleet. One agent, or one capability of one agent, is reaching something it should not reach. The right response matches the size of the problem: that agent loses that tool, immediately, and every other agent keeps serving.

Immediately is the word that costs something. It has to mean a number you can state, measured from the moment an operator decides rather than from the moment the system gets round to it, and it has to hold under the load that made you notice in the first place.

The blunt response is always available. Stop the fleet, roll the deployment, take the gateway down. It works, and it spends the thing that took longest to earn, which is a set of teams willing to let agents near production at all.

So the capability you need is partition: cut one line of authority out of a running system while the system stays up. That is harder than it sounds, and the reason is the same property that let the fleet scale in the first place.

Why the change does not reach the fleet

A gateway authenticates a request by verifying a JWT signature against a cached key set and checking the standard claims. No network call to the issuer, no lookup, no shared state. That property is why the hot path is fast and why it scales flat, and teams choose it for exactly that reason.

It is also why revocation at the identity provider has no effect on a token that has already been minted. The signature still verifies. exp has not passed. The gateway accepts it, because from the gateway's point of view nothing has changed. Nothing has: it never asked.

Rotating the tenant's client secret does not rescue this either. It invalidates the secret, so no new token can be minted, and it leaves every outstanding token exactly as valid as it was a second earlier.

So if your containment story is "we revoke the grant", your real bound is the access-token lifespan of whichever issuer your largest customer runs. In practice those run from a few minutes to several hours depending on how the issuer was configured, and nobody configured it with your incident response in mind.

Most teams discover this during the incident rather than before it.

What partitioning costs

The way out is an enforcement point that does not consult the token at all. Something checked alongside the authorization decision, reachable by an operator, that can force a deny for one agent and one tool regardless of what credential the caller is holding and regardless of whether that credential is still valid.

That is not a difficult thing to build. The difficulty is that it reintroduces exactly the property offline validation was chosen to avoid. The enforcement point needs state, that state has to reach every replica, and the layer you scaled flat now has a distributed systems problem inside it with a security deadline attached.

Which means you own a new set of questions, and they are the ones worth asking of anyone who claims a containment bound:

How does the decision reach a replica, and how long can that take at worst? If the mechanism polls, the interval is a floor on the worst case rather than a ceiling. A decision landing just after a tick waits most of an interval, plus whatever the fetch costs under load. A poll interval is a cadence. It is not a guarantee, and the two get conflated constantly.

What does a replica do when it cannot reach that state? Serving the last known good answer is reasonable for a while, and at some point a replica that cannot confirm whether anything has been revoked should stop claiming it knows. Where that line sits is a real trade, and crossing it means refusing traffic for a tenant that may be perfectly healthy.

What happens when the state store comes back empty? This is the one that gets missed. A store that restarts with bounded persistence can come back without recent writes. The replica's next refresh succeeds, the replica stays healthy, and every revocation silently disappears. Losing the store fails in the safe direction. The store returning blank does not, and no amount of fail-closed-on-error logic catches it, because nothing errored.

The timeout trap

One specific thing, because it is cheap to get wrong and expensive to find.

Do not derive your staleness threshold from your refresh interval. Making the timeout a small multiple of the poll cadence looks tidy and couples two things that fail together: the conditions that slow your refresh are exactly the conditions that then trip your timeout. One slow cycle can flip a healthy replica into refusing everything. The failure mode you are protecting against becomes the trigger for the protection.

Make it absolute, and pin the decision with a test, because the tidy version is what someone reaches for during the next refactor.

What we would want to see

A published bound on how long a revoked capability keeps working, stated as a worst case rather than a cadence, with the conditions under which it does not hold written next to it. That number is what anyone means by immediately, and it is the one most containment stories leave unsaid.

A bound you cannot state the exceptions to is not a bound. That is the first thing we would check about anyone's containment story, including our own.


We build a Zero Trust platform for agents: authorization and governance for what they do.

Join the waitlist

Tell us who you are and what your agents do.

0/1000

We use your email, name, role and note only to reply about access. They reach us by email and are not stored on this site. To have them deleted, ask in your reply or write to hello@supreplabs.com.