On some ordinary morning, the system your order desk, scheduling or invoicing runs on will not answer. The cause may be a cyberattack at your software provider, a faulty update, or an outage at the model provider behind your AI workflow. From that minute, your business either runs a degraded mode your people have rehearsed, or it stops and waits.

Two outages in one summer
In June 2024, cyberattacks hit a software provider to car dealerships that serves more than 15,000 retail locations. A 2024 Associated Press report described dealers writing orders by hand, with stacks of paper awaiting processing. A 2024 Bloomberg News report relayed the provider's word that substantially all dealers had systems back nearly two weeks after the attack, which had disrupted scheduling, repairs, parts deliveries and car sales.
A month later, a faulty security update affected about 8.5 million computers, by the operating system maker's own 2024 estimate. One airline put its cost at as much as $500M over five days, with more than 5,000 flights canceled and its crew-tracking system disrupted, NPR reported in 2024. Neither outage started inside the businesses that bore the cost, and each of those businesses had to keep operating until its systems came back.
Price the day it is down
Expected loss is the probability of an event times its impact. A distributor with $120M revenue sells about $60k an hour over 2,000 working hours a year, at a 25% gross margin. Its order desk stops for 10 selling days, 80 hours, as long as the dealers waited. If 30% of the $4.8M in orders, 4,000 at $1,200 each, go to a competitor, it loses $360k of margin. Re-keying the other 2,800 orders at 15 minutes each takes 700 hours: about $23k at time-and-a-half overtime on the US median wage of $21.53 an hour for customer service representatives. Add the late-delivery credits your customer contracts carry, $45k here, and the outage costs you about $428k, before counting any customer who never comes back.
A degraded mode designed to keep 90% of orders flowing replays them without re-keying: 30% of the 400 orders that wait, about 120, go elsewhere ($36k of margin), reconciliation takes about $4k of staff time, and deliveries largely stay on schedule, so the outage costs about $40k instead of $428k. Budget an estimated $60k to set up the snapshot, form and replay, and roughly $15k a year for upkeep and four rehearsals, or about $27k a year spread over five years. At roughly $390k saved per outage, it pays for itself if an outage that long hits at least once every 14 years. It also covers shorter outages in that workflow.
Two numbers you set in advance
NIST's 2010 contingency planning guide, from the US standards body, defines two targets. The recovery time objective is how long a system can take to recover before the outage harms the processes that depend on it. The recovery point objective is how much recent data you can afford to lose: the point in time to which your records must be recovered.
Set both for each workflow. Your order desk may tolerate two hours without its system; month-end reporting may tolerate three days. The recovery time objective tells you how soon the degraded mode must be running; how long it must hold depends on your provider. The recovery point objective sets how often the records it relies on must be copied.
Design the degraded mode in
Build the degraded mode into the workflow. For the order desk, it has five parts:
- A read-only snapshot. Orders, customers, prices and schedules, refreshed hourly.
- A separate home. Another provider, with no login shared with the main system.
- A manual capture form. Every order carries one ID; the form issues IDs such as D2-0147.
- A named decision maker. One person, with a deputy, declares the mode and ends it.
- A replay into the system. The IDs let it skip what was already entered.
An hourly refresh targets a recovery point objective of one hour, and an alert flags any copy that fails to land: orders since the last good copy are re-entered from the original requests. Over a long outage, prices hold at the last refresh. An AI workflow uses the same parts: if its model provider gives no answer within a set time, such as 30 seconds, the order goes to the capture form under the ID it already carries, so a late answer cannot create a second order.
Recover without double entry
Orders written by hand can be keyed in by several people when the system returns, some twice and some never, and surface weeks later as short shipments or disputed invoices.
The IDs prevent that when the system enforces them. Each order keeps one ID from the moment it is taken, and the system refuses a second order with the same ID, so running the replay twice changes nothing. Then a reconciliation compares the forms with the system: order counts, totals by customer, any ID on one side only, and orders sent just before the outage. Your team clears that short list before the decision maker ends the mode.
Rehearse it every quarter
Until it has run once, nobody knows how long the switch takes. Once a quarter, the decision maker declares the mode for one hour. The team works from the snapshot, captures real orders on the form, then replays and reconciles them. Make one of the four a full day.
Measure three things: minutes from the start to the desk taking orders at 90% of its normal pace, against your recovery time objective; minutes to replay; and the number of exceptions found in reconciliation. A first rehearsal is where gaps show: a snapshot missing credit limits, or a form that lets two people reuse an ID. Each fix goes into the next quarter's run.
When a full mode does not pay
If a workflow can stop for days without breaching a contract or losing a customer, its records restore from a tested backup, and its expected loss is a few thousand dollars a year, a written phone list and a backlog plan are enough. Spend on a full degraded mode where the expected loss pays for it, or where a single two-week outage is more than your business can absorb.
Bring one workflow to a call
A fifteen-minute call takes the one workflow your business cannot run without and sets a first recovery time objective for it with you. You leave with that target and an outline of its first rehearsal; if your current plan already meets the target, you hear that plainly.
Where the snapshot, the form and the replay need building, Engineering builds them into your systems with your team, so your people run the rehearsal themselves.
References
- Grantham-Philips, W., Associated Press. Car dealerships in North America revert to pens and paper after cyberattacks on software provider. Anchorage Daily News, published 2024-06-24.
- Gorrivan, C., Bloomberg News. CDK: 'Substantially All' Car Dealerships Restored After Hack. Transport Topics, published 2024-07-01.
- Weston, D., Microsoft. Helping our customers through the CrowdStrike outage. Microsoft Official Blog, published 2024-07-20.
- Breslow, J., NPR. Delta's CEO says the CrowdStrike outage cost the airline $500 million in 5 days. OPB, published 2024-07-31.
- Swanson, M., Bowen, P., Phillips, A., Gallup, D., and Lynes, D., NIST. Contingency Planning Guide for Federal Information Systems (SP 800-34 Rev. 1). National Institute of Standards and Technology, 2010, updated 2010-11-11.
- US Bureau of Labor Statistics. Customer Service Representatives, Occupational Outlook Handbook. US Department of Labor, May 2025 wage data.
- Wage and Hour Division, US Department of Labor. Overtime Pay. US Department of Labor, no date given.


