The most consequential failure in an automated workflow can arrive after an external system accepts a write but before the caller records the response. The request looks failed locally. Repeating it may create a second ticket, payment, or message. A reliable system needs a way to establish what happened.
Adding a model does not remove this distributed-systems problem. It adds another source of proposed actions and another place where a confident explanation can outrun the evidence. Build the workflow around durable state and observable effects, then let the agent operate within that structure.
Write down the state before the side effect
For an illustrative inspection-report workflow, retain the task identity, source versions, proposed report, intended destination, approval revision, and external operation identity. Store enough information to resume without reconstructing intent from a conversation transcript.
Use explicit states such as prepared, awaiting approval, ready, submitted, confirmed, failed, and uncertain. A worker claims a ready operation, performs it, and records the result. The transition to confirmed requires evidence that the destination accepted or applied the intended operation at the boundary being reported.
A timeout is evidence that the caller lacks a response. It is not evidence that the destination did nothing. Reconcile a potentially completed write before resubmitting it.
Make retries refer to the same operation
AWS's Builders' Library explains how idempotent API design makes retries safer. Where the destination supports an idempotency key, keep one stable key for the same intended operation. A new random key on each retry asks the destination to treat each attempt as new work.
Bind the operation identity to its payload. If a destination or attachment changes, that is a changed operation requiring the corresponding authorization, not a retry of the old one. Retain the original response and receipt when available so later workers can reconcile without recreating the side effect.
Where the destination lacks a safe deduplication or lookup mechanism, choose a deliberate uncertainty path. Preserve the work, tell the operator which action may have completed, and require reconciliation. Automatic progress is not worth multiplying an irreversible effect.
Use retries selectively
AWS Step Functions documents configurable retries and error handling. The general lesson is to classify failures before applying a retry policy. A transient service failure, a malformed request, a revoked credential, and an ambiguous write require different responses.
- Retry transient, safely repeatable operations with bounded backoff and a deadline.
- Correct invalid inputs through a controlled revision. Do not spend the retry budget submitting the same invalid request.
- Stop on revoked authority and require the appropriate access decision. A fallback tool must not become a route around the denial.
- Reconcile ambiguous external writes using a receipt, stable operation identifier, or authorized destination read.
Preserve approval across interruption without freezing authority
A durable approval record should describe the exact reviewed action and its expiry. On resume, verify that the payload still matches and that the relevant authority still exists. Persistence preserves the evidence of a decision; it does not make that decision valid indefinitely.
Consider a report approved for one recipient before a network outage. After the outage, the recipient list changes. The system should preserve the completed report but return the changed send operation for review. That separation avoids both lost work and stale approval reuse.
Test the moments between steps
Happy-path tests rarely exercise the most useful recovery behavior. Interrupt the worker before submission, after destination acceptance but before local persistence, and while an approval is pending. Deliver the same event twice. Resume after access has been revoked. Inspect the destination and the local record together.
Measure duplicate effects, unreconciled operations, time to recover, and human effort required to reconstruct the run. Define what counts as a duplicate for the actual destination. A second local log entry may be harmless; a second external message may not be.
Turn recovery into a demonstration artifact
For a defense-support prototype, intermittent connectivity is a useful test condition when it matches the intended environment. Demonstrate it with public or synthetic inputs, then show the retained evidence and orderly recovery. Do not imply a disconnected or restricted environment has been supported unless it has actually been tested.
A concise run record should let another operator identify the last confirmed effect and the next permitted action. That is a stronger acceptance artifact than a transcript ending with success. It also makes a handover between people, processes, or model providers less dependent on a single session surviving.
Persist intent, identify operations consistently, reconcile uncertain writes, and recheck authority on resume. Recovery is part of the product behavior.
Continue the conversation
Bring Agent9 one workflow, its operating constraints, and the evidence you need before deployment. Discuss the engineering problem.