How to Put an AI Agent Into Production and Keep It Up

How-to — 2026-09-16 — by Mahmoud Zalt

Take an AI agent from prototype to production: triggers, limits, logs, an off switch, and the two week shadow run that catches what demos hide.

The demo went well. It ran on your machine, on data you picked, while you watched. Then somebody asked when it goes live and you realised you had no answer for what happens at three in the morning on a Sunday when nobody is looking at it.

That is not a gap in your skills. A prototype only has to work once, in front of you. A production agent has to work when an API is slow, when a token expires, when the trigger fires twice, and when the input is nothing like the three examples you tested with.

The distance between those two states is smaller than people fear and more boring than they hope. There is no exotic infrastructure involved. It is a schedule, a set of hard limits, a place where every run gets written down, and a way to stop the thing without redeploying anything.

That whole layer is what we already built into Sistava. The schedule, the spend caps, the approval gates, the run history and the kill switch come with the account, so you describe the job, connect the accounts it touches, and hire an AI Employee that starts in a state you would otherwise spend a month reaching. If the agent is your product, build it yourself. If the agent is plumbing, do not build the plumbing twice.

At a Glance

4: Things that turn a prototype into production
2 weeks: Shadow run before it touches anything real
1 switch: Every live agent needs an off switch
0: Silent failures you can afford

What does putting an AI agent into production actually mean?

It means the agent starts without you, finishes without you, and leaves enough evidence that you can reconstruct any run later. Nothing about the model changes. What changes is ownership: the schedule owns when it runs, the limits own how far it can go, and the log owns the truth about what happened.

A useful test is the holiday test. If you went away for two weeks with your phone off, would the agent keep doing useful work, and would you be able to tell on your return whether it did? If the honest answer is no, you have a prototype on a timer, which is a different and more dangerous thing.

A trigger you do not press. A cron schedule, a webhook, a new row, a new message. It must be repeat safe, because every trigger fires twice eventually and the second time is always at the worst moment.
Limits it cannot argue with. Maximum steps per task, maximum spend per run and per day, an allowed list of tools, and a timeout. Enforced in code outside the model, never asked for politely in a prompt.
A log you can query. Every run with its input, its decisions, its tool calls, its results, its cost and its outcome. One place, searchable, kept long enough to answer a question from three weeks ago.
An off switch. A single flag that stops new runs immediately, without a code change or a redeploy. If turning it off requires a pull request, it is not off, it is off in twenty minutes.

How do you know an agent is ready to run unattended?

When it has run in shadow mode against real inputs for long enough to meet the weird ones, and you agreed with its decisions often enough to state a number. Shadow mode means it does everything except the last step: it reads live data, decides, drafts the action, and writes it to a log instead of sending it.

Two weeks is a reasonable default because most jobs have a weekly rhythm and you want to see two of them. Read every run in week one. Sample in week two. If you are still surprised by its decisions at the end of week two, it is not ready, and no amount of prompt tuning shortens that clock.

Then go live narrow. One customer segment, one folder, one hour of the day, or a cap of ten actions per run. A narrow launch means your first production mistake is small and recoverable, and it will happen, because the first production mistake always happens.

A worked example: Priya and the first two weeks live

Priya runs operations at a six person homeware brand. Her agent had one job: read every incoming return request, check the order date and the photos, and either approve the refund, ask the customer one clarifying question, or flag it for her. On her laptop it was excellent. Live, it taught her four things in fourteen days.

She ran two weeks of shadow first. The agent read real requests and wrote what it would have done into a spreadsheet. She disagreed with eleven of one hundred and ninety decisions, all of them in the same place: orders where the customer had already emailed twice. So she added a rule that anything with prior contact goes straight to her.

Day two live, the duplicate. The webhook retried and one customer got two identical replies. Fixed by recording the request ID and short circuiting anything already handled.
Day five, the silent stop. A refreshed token broke the email tool. Nothing errored loudly, the agent just did nothing for nine hours. She added an alert on silence, not only on failure.
Day nine, the expensive loop. A malformed photo attachment made the agent retry the same read fourteen times. The step limit caught it, the spend cap would have caught it second, and she was glad both existed.
Day thirteen, the good surprise. Her flag queue had dropped to about six a day from forty. That was the number that justified the whole build, and she only had it because every run was logged.

Note what none of those were. Not one of them was a model problem. Every single failure in her first fortnight lived in the plumbing: retries, tokens, loops and visibility. That is the shape of production agent work, and it is the shape whatever model you pick.

What breaks in production that never breaks in a demo?

Time, duplication, expiry and silence. A demo runs once, immediately, with fresh credentials, while you watch. Production runs a thousand times, sometimes twice at once, with credentials that rot, in front of nobody. Each of those four conditions has a standard fix, and none of the fixes are clever.

The one that catches most people is silence. An agent that crashes is a good agent, because you find out. An agent that quietly returns nothing useful for nine days while the dashboard says green is the expensive kind, and the only defence is alerting when the expected work does not appear.

Production condition	What it looks like	The standard fix
Trigger fires twice	Two identical emails to one customer	Store the trigger ID, exit early if already handled
Credentials expire	Tool fails, agent shrugs and continues	Treat auth errors as fatal, alert, stop the run
Model returns nonsense	A malformed action is passed to a tool	Validate the action shape before any tool is called
Loop gets stuck	The same call repeated until money runs out	Hard step limit plus a repeat detector on arguments
Input drifts	New form field the agent has never seen	Log unrecognised inputs, review them weekly
Nothing happens	Zero runs, zero errors, zero notice	Alert on absence of expected runs, not only on errors
Something goes wrong publicly	A wrong message already sent	Approval gate on anything a customer will read

What will a production AI agent still not do?

It will not notice that the job itself changed. If your refund policy changes on a Monday, the agent keeps applying the old one perfectly and confidently until somebody updates its instructions. Consistency is the feature and the flaw in the same sentence.

It will not hold a plan together across days. On long, multi session tasks, agents lose the thread, repeat work, and keep executing a plan whose premise expired. It will not take responsibility for a decision either, which means anything legal, financial or reputational needs a human name attached before it goes out.

And it will not stay correct without maintenance. Accounts change, APIs version, formats drift. Budget an hour a month per live agent forever, not because something is wrong, but because that is the running cost of anything that touches other systems.

The path from prototype to live

Write the rollback plan before the launch plan — How do you stop it, how do you undo the last hour of its work, and who gets told. If you cannot answer all three, you are not ready to go live.
Move the trigger off your machine — A schedule or a webhook on something that stays up. Then make it repeat safe by recording what has already been handled, because it will fire twice.
Set the hard limits in code — Steps per run, spend per run, spend per day, allowed tools, timeout. Outside the prompt, where the model cannot talk its way past them.
Log every run before the first unattended one — Input, decisions, tool calls, results, cost, outcome, all in one queryable place. You cannot debug what you did not record.
Shadow run for two weeks against real data — The agent decides and drafts but never sends. Read every run in week one, sample week two, and write down how often you disagreed.
Go live narrow, then widen — One segment, one hour, or a cap of ten actions. Widen only after a full week with no surprise in the log.
Alert on failure and on silence — A message when a run fails, and a message when the runs you expected never happened. The second alert is the one that saves you.

Read that list again and notice how little of it is about intelligence. Production is a discipline about failure, not about capability. The agent that wins is rarely the smartest one, it is the one whose owner can answer what happened last Thursday in under a minute.

FAQ

How do I deploy an AI agent to production?

Move the trigger onto something that stays up, put hard limits in code outside the prompt, log every run to one queryable place, and add a flag that stops new runs without a code change. Then shadow run against real data for two weeks before it is allowed to send anything. The deployment itself is ordinary hosting. The readiness is the work.

How long should an AI agent run in shadow mode before going live?

Two weeks is a sensible default, because most business jobs have a weekly rhythm and you want to see two of them. Read every decision in the first week and sample the second. If the agent is still surprising you at the end, extend it rather than launching and hoping.

What should I monitor once an AI agent is live?

Five things: run count against what you expected, failure rate, cost per run, how often a human overrode the agent, and how long each run took. The override rate is the most useful of the five, because a rising override rate tells you the agent is drifting away from the job before anyone complains.

How do I roll back an AI agent that starts behaving badly?

Flip the off switch first, then read the log, then undo. This is why every action an agent takes should be reversible or gated: a draft can be deleted, a sent email cannot. Decide before launch which actions are undoable, and put an approval gate in front of the ones that are not.

Do AI agents need a staging environment?

They need something safer than production, but a full clone is usually overkill for a small team. Shadow mode against live data with sending disabled gives you most of the value at almost no cost, because the failures you care about come from real inputs, not from a synthetic copy of them.

How often do production AI agents break?

Expect something small every few weeks and something real every few months, almost always from a change outside the agent: an expired token, a renamed field, a new form option, an API version. Plan for about an hour of maintenance per live agent per month and it stops feeling like a crisis and starts feeling like upkeep.

If you take one habit from this, take the shadow run. It is unglamorous, it costs almost nothing, and it converts the scariest part of the project, the unknown behaviour of your own agent on real inputs, into a spreadsheet you can read on a Friday afternoon.

Everything else on the list is a smaller version of the same idea. Make the failures visible, make them cheap, and make them yours before they become your customers'.