01 · Optus
Sixty percent of the towers
A config change took out about 60% of Optus mobile towers. Full service in 40 minutes, with a 40-engineer shift.
40 minNational restore
A configuration change was rolling across the mobile network and towers were dropping with it. This was not a suburb and it was not a single cluster. Roughly 60 percent of the national tower estate was affected, which means customers losing service in more than one state at the same time. I was the incident controller on a 24x7 shift of about forty engineers, and the first job was to make that room one command instead of forty parallel conversations.
The working theory was the change itself. On a night like that, the dangerous move is a second fix written while the first one is still propagating. I froze further change, put radio, core, the person who had pushed the config, field and the vendor on one bridge, and split the floor so each engineer had a region, an alarm queue, or the rollback — not a general invitation to narrate.
The restore path was a rollback of that configuration onto the affected population, watched region by region as alarms cleared. Full service was back in forty minutes. I did not call it restored because one site went green. We waited until the alarm rate had returned toward baseline and a real customer path worked.
The forty minutes is the part people remember. The part that matters the week after is the control: that class of change could no longer leave on one pair of hands. The bridge ends. The peer check stays.
How I ran it
The order of decisions, written out so the clock is visible.
- 01
I declared it a major as soon as the pattern was national, not regional. Naming the incident commander out loud stopped the floor from hunting for one while towers kept falling.
- 02
Everyone who could change the outcome was on one bridge: radio, core, the engineer who had pushed the configuration, field, and the vendor. Side conversations were pulled in. A second call is how you get two restore plans.
- 03
Further change was frozen. The theory was the configuration that had just gone out. Nobody was allowed to ship a compensating fix until we knew what had been pushed, to which population, and at what time.
- 04
Forty engineers is a crowd unless the work is divided. Each person had a slice — a region, an alarm queue, or the rollback itself. I kept the clock and the sentence for executives. They kept the technical work.
- 05
The executive update was one sentence: the share of towers down, what the customer could see, and the time of the next update. There was no root-cause speech while the rollback was still running.
- 06
We rolled the configuration back across the affected population and watched alarms clear region by region. A single healthy site was not enough to say the country was back.
- 07
Restored meant evidence. The alarm rate had to move back toward baseline, a sample of sites had to hold, and a customer path had to complete. That check is what the forty minutes includes.
- 08
The bridge stayed up long enough to see whether the fault rebounded. The review the next day was specific: who had been able to push that change alone, and what peer check would stop the next one.