02 · Workday
Forty-two customers, thirty-seven minutes
Access restored for 42 Workday customers in 37 minutes. First update at 8 minutes. Then the change-control gap that let it in.
37 minAccess restored
Customers were losing access, and the shape of the fault mattered. It was not one tenant having a bad morning, and it was not the whole platform. A defined set of customers could not get in. Once we had counted them, the number was forty-two. That count is what I put in front of the bridge, because a story without a number invites everyone to guess the size of the event.
I took command and treated the first eight minutes as a deadline, not a hope. The first update named who was hit, what they could not do, what we already knew, what we did not, and when the next update would land. Executives and the customer path heard the same facts. A prettier version for one audience is how trust goes.
The bridge was small on purpose: the owner of access, the person accountable for the change that had landed, support, and communications. Engineers restored the path. I kept the sequence so the channel did not turn into a novel. Service was back in thirty-seven minutes, confirmed on a real login rather than a dashboard that had gone green in one place.
Restore was not the end of the work. The change had been allowed through a gap in change control, and that gap was written down the same day as an action with an owner and a date. A post-incident review that describes the night and does not close the door is a diary.
How I ran it
The order of decisions, written out so the clock is visible.
- 01
I called it a major as soon as access was failing for more than one customer. The alternative was a debate about severity while people were locked out, and that debate does not restore anything.
- 02
Before anyone wrote a narrative, we counted the blast radius. Forty-two customers. If the count was still moving, the update said so. A stable wrong number is worse than an honest moving one.
- 03
The first customer and executive update went at eight minutes. It carried impact, what was known, what was not known, and the time of the next word. Both audiences received the same paragraph.
- 04
One bridge held the access owner, the change that had just landed, support, and communications. I ran the order of decisions. The engineers ran the restore. Speculation stayed off the customer channel.
- 05
Access was brought back and then checked on a customer path: a real sign-in, not a tile that had turned green. A healthy component with a customer still locked out is not a restore.
- 06
At thirty-seven minutes the service was back and the check had passed. The bridge stood down in order — customer note, technical note, then the room — rather than everyone leaving at once and losing the timeline.
- 07
The same day I named the change-control gap that had let the fault in. It had an owner, a date, and a definition of done. Without that, the thirty-seven minutes is a good story and the same incident is still available next month.