Demos are easy. Tuesdays are hard.
A demo gets a clean input, a prepared environment, a patient audience, and an engineer standing close enough to intervene.
A customer on Tuesday morning brings stale credentials, partial data, a slow integration, two browser tabs, a production incident, and no interest in how promising the architecture looked last week.
The distance between those two moments is production engineering.
A demo proves possibility.
A good demo answers an important question:
Can this idea create useful behavior at all?
That is real progress. Teams should celebrate it.
But a demo normally avoids the harder questions:
- What happens when an upstream API times out?
- Can the work be retried without duplicating an action?
- What does the user see when evidence is incomplete?
- Which identity is used for a tool call?
- Can one customer's data enter another customer's context?
- How much does the workflow cost at the 95th percentile?
- Can support explain yesterday's failure?
- Can the system be rolled back?
Production readiness begins when those questions become part of the product, not a list for later.
The first hard thing is input reality.
Demo inputs are selected. Production inputs arrive.
They are malformed, duplicated, delayed, contradictory, oversized, multilingual, and occasionally malicious. Security products also ingest attacker-controlled text from alerts, tickets, files, websites, and threat reports.
Every boundary needs:
- schema validation;
- size and rate limits;
- tenant and sensitivity labels;
- idempotency;
- provenance;
- an explicit failure path.
If the system silently drops fields or invents defaults, the demo's fluency becomes an operational liability.
The second hard thing is durable state.
An AI workflow is usually a distributed workflow.
It retrieves context, calls a model, invokes tools, waits for approvals, stores evidence, and publishes a result. Any step can fail after another step has already succeeded.
The production design needs:
- durable workflow state;
- bounded retries;
- idempotency keys;
- timeouts;
- compensating actions;
- cancellation;
- human-visible progress;
- safe resumption after deployment.
A recursive agent loop can be useful for exploration. It is not automatically a reliable execution model.
The third hard thing is authority.
The moment an AI system can change a ticket, disable a user, isolate an endpoint, or update a firewall, quality is no longer the only concern.
The platform must know:
- who requested the action;
- which tenant and case authorize it;
- which policy applies;
- what evidence is required;
- whether approval is needed;
- how the action can be reversed;
- what audit record must remain.
CISA's secure AI guidance emphasizes secure design across development, deployment, and operation. OWASP's agentic guidance similarly treats tools, identity, memory, and unintended actions as system risks.
A prompt asking the model to "be careful" is not an authorization layer.
The fourth hard thing is failure UX.
Production systems should fail legibly.
"Something went wrong" is not enough. The user needs to know whether:
- no evidence was found;
- a source was unavailable;
- an action was denied by policy;
- approval expired;
- the model output failed validation;
- the workflow stopped safely;
- manual work is required.
Failure states are part of the product contract. They shape whether users trust the system and whether support can recover it.
The fifth hard thing is observation.
CPU and memory are necessary but insufficient.
An AI workflow also needs telemetry for:
- end-to-end completion;
- each retrieval, model, and tool step;
- model and prompt version;
- token usage and cost;
- evidence IDs;
- policy decisions;
- approvals;
- retries and fallbacks;
- final outcome and correction.
OpenTelemetry's generative AI conventions help standardize model and agent signals. Sensitive prompts and evidence still need deliberate redaction and access control.
The sixth hard thing is change.
Models change. Prompts change. Search indexes change. Tools change. Customer data changes.
Every significant change should pass:
- a deterministic regression suite;
- representative offline evaluations;
- adversarial tests;
- shadow or limited rollout;
- monitored production release;
- rollback criteria.
The evaluation set should preserve real failures. Otherwise the team pays for the same lesson twice.
My Tuesday test.
Before I call a system production-ready, I want clear answers to ten questions:
- What is the completed unit of work?
- What can fail, and how does each failure appear?
- What state survives a restart?
- Which actions are reversible?
- Where is authorization enforced?
- How are tenant and data boundaries tested?
- Can a result be traced to evidence?
- What are the latency, reliability, and cost budgets?
- How does a human correct or stop the system?
- What happens during provider or integration failure?
The AI security production checklist goes deeper on release gates.
The demo is not dishonest. It is simply answering the first question.
The product begins with all the questions that arrive on Tuesday.