The night before, my Bluebird driver dropped me at the hotel and asked if I wanted him to take me back to the airport in the morning. We talked it over and agreed on 5am. That gets me there by 6 at the latest, comfortable.
I woke at 4:30 and messaged him on WhatsApp to confirm. He replied straight away that he was already awake, doing his morning prayer.
Good. Sorted.
I packed, brushed my teeth, and was downstairs before 5. The breakfast buffet wasn’t officially open yet, but they let me at it early. I ate something and waited.
5:13. Still no driver. I called him and he told me he was ten minutes away.
I hung up, blocked him, and deleted the conversation. Then I opened Grab.
The car showed up in two minutes.
Delay Technical, 19:30
CX-780 to Hong Kong. Scheduled 08:10. Delay Technical, 19:30.
Eleven hours and twenty minutes.
My itinerary was CX780 to Hong Kong, an eleven hour layover, then CX872 to San Francisco at 01:10 the next morning. That one lands at SFO at 23:15 the previous day, which is my favorite thing about that route.
A 19:30 departure gets me into Hong Kong around 01:30. CX872 leaves at 01:10.
I’d miss it by twenty minutes.
Goddamn.
I asked what “technical” actually meant. Something was broken on the aircraft, they had to fix it, and they were really sorry about it.
They meant it, too. I want to be clear about this before I complain for the next thousand words: nobody I dealt with that day was trying to do a bad job.
The Queue
They’d split the queue in two. Hong Kong only in one line, connecting flights in the other. Mine was the connecting line, which was the shorter of the two, and I was grateful for about ten seconds.
It barely moved. Every person who reached the front turned into a fresh negotiation. New routing, new dates, a hotel, all of it worked out from scratch while everyone behind them stood and waited. They were working, and working hard. They were just doing every step of it manually, with nothing to work from. Each customer was a new problem being solved for the first time.
While I waited, I called the Cathay call center and explained the whole situation. The agent listened, then told me I was already doing the right thing by queuing at the service counter.
Sure. Thanks.
They Already Gave My Seat Away
I finally got to the counter. After some back and forth, they offered me Singapore Airlines instead. Surabaya to Singapore at 4pm, then Singapore to SFO the next morning.
I asked the obvious question. Why not just put me on a different flight to Hong Kong? There were other ways to get me there in time. Keep me on CX872 and I lose nothing.
Because CX872 was full, they said. They knew CX780 was delayed, so they’d already given my premium economy seat to somebody else.
So they broke the first leg, and then used their own delay as the reason to give away my seat on the second one. By the time I got to the counter to ask about a flight I’d already paid for, it was gone.
Fine.
They couldn’t book the replacement right then. They took my WhatsApp number and told me they’d call in 15 minutes.
I went to find somewhere to sit. There was nothing. So I sat on the steel grilles they install around support columns to stop people from ramming luggage carts into them.
They did call, to their credit. I walked back over to the counter and they gave me the whole thing: the 4pm to Singapore was booked, and I was on the next morning’s flight to SFO.
By 9:30 they told me they couldn’t get me a hotel either. CX-780 is a big aircraft, so a few hundred people all needed rebooking and a room on the same morning, and every hotel Cathay had a deal with was already full of them. The only rooms left were far enough out that they were worried about the drive.
So the offer became: sit in the lounge until 4pm.
I’d walked in at 6am. It was now past 9:30, and three and a half hours of queuing, explaining, and waiting had bought me an outcome worse than the ticket I’d already paid for.
Right at that moment the Singapore Airlines agent said the 10am flight had one seat open from a no-show, and did I want it.
Of course I bloody wanted it.
Longer layover in Singapore, but I wouldn’t be stuck in Surabaya for another six hours. The Cathay agent told me someone would be waiting for me on arrival, that they knew I was coming, and that they knew what to do.
Great. I ran for the gate.
I was not the last one to board.
The flight was uneventful. I slept the whole way. Thank god for an eye mask and a neck brace.
Nobody Was Waiting
Nobody was waiting.
I still had the WhatsApp thread with the Surabaya counter, so I asked. They told me to go to the transfer desk.
I walked to the transfer desk. It was closed.
The screen above it said Transfer F was closed and to go to Terminal 2 or Terminal 3. I asked WhatsApp again. They said the Cathay and Singapore Airlines transfer desk. Where exactly? Terminal 4.
Terminal 4 is a bus ride.
I found the bus stop, and a very nice officer there told me there is no Singapore Airlines desk in Terminal 4. So I called the agent back, and she clarified that I just needed the Cathay desk.
Then why did you say Singapore Airlines.
The wait plus the bus took about half an hour. Hard to tell exactly. It felt long.
Terminal 4 is beautiful and it was basically empty. I cleared security again. Then a staff member told me to speak to the police officer. I explained the whole thing to the police officer, who pointed me at a row of empty desks with a phone on one of them.
The sign says the counter opens three hours before departure, and to dial 40815 on the landline to reach the duty supervisor.
That laminated sheet is the closest thing to a runbook I saw all day, and it’s an instruction to call a guy.
So I called. I explained everything again. The man on the phone said someone would come and help me.
I sat and waited. A man eventually arrived. I told him I was the one who called.
He said he had no idea, because he wasn’t the one on the phone.
A Hotel Address on a Blank Boarding Pass
He sorted it out, to be fair. Hotel with breakfast, lunch, dinner, and taxis both ways. Aloft at Novena, four stars, about half an hour from the airport.
He wrote the address down for me on a blank boarding pass, the paper stock they feed into the printer, and pointed me toward the taxis.
I got in a cab. Half an hour later I was at the reception desk and they asked me for the booking slip from the airline.
I had an address on a blank boarding pass.
The taxi driver came in with me, because he hadn’t been paid. The hotel is supposed to pay him. The hotel wasn’t paying him, because they had no record of who I was. They made some calls, decided it was fine, and paid him.
While he was getting paid, he told the receptionist they really need to fix this, because it’s happened to him before. He looked annoyed about having had to stand around waiting with me.
Join the club, pal.
They checked me in. I asked if I could eat. The restaurant was closed. I asked about room service. There is no room service. I asked what my options were, and the answer was the shopping mall down the road.
By then I’d been awake since 4:30am, and the idea of walking to a mall was so exhausting that I chose to just stay hungry and sleep.
The Day Turned
Before I passed out I messaged Jonathan and Adrian to see if they were free.
Adrian brought Ivana and their daughter, and we walked along the river and had dinner at Putien. Lo mee, roast chicken, sweet and sour pork, and the vegetables with beancurd skin.
The baby screamed through most of it. That’s fine. Kids do that.
Adrian, Ivana and I got into it about religion, whether suffering means anything, where morality comes from, whether a country’s progress tracks against how religious it is, divorce, relationships, health. We covered a lot of ground for one dinner. Catching up with old friends is always like this. You skip the small talk entirely, because the context is already there.
Then Jonathan came out and we sat at a Starbucks. His work, his family, my startup journey. I ordered a carrot juice, no added sugar. It was good.
Then I went back to the hotel, showered, and slept.
That morning I’d been standing in front of a departure board doing subtraction. It still turned into a good day.
What Was Actually Broken
Not the plane. Planes break. That’s what maintenance is for, and I’d rather sit in a terminal than fly on a jet somebody rushed.
Here’s what I counted by the end of the day. The call center. The Surabaya counter. The WhatsApp thread. The officer at the bus stop. The police officer at Terminal 4. The voice on the landline. The man who showed up after the landline. The hotel receptionist.
Eight people. Eight full explanations. Not one of them could see what the previous seven had done.
Every single person I dealt with was improvising. Some improvised well, like the Singapore Airlines agent who thought to check for no-shows, and the guy at Terminal 4 who actually solved it. Some improvised badly, like the agent who told me someone would be there to help without telling me which terminal, how to get to it, or what was supposed to happen when I arrived. She wasn’t lying to me. She just had nothing to hand over, so I had to reconstruct all of it myself, from strangers, one hop at a time. None of them were working from a document.
The only thing that carried state across that entire day was me.
Which takes me back to the driver who didn’t show up at 5am. He’s not a worse person than the Grab driver who came in two minutes. Grab has dispatch. One driver falls through, another one gets assigned, and I don’t have to do anything about it. My guy only had himself, so when he didn’t show up there was nothing behind him.
Then I spent the rest of the day dealing with a much bigger version of the same thing.
And I could do it. I speak the languages, I’ve done enough of these trips to know how airports work, I had a working phone and a data plan, and I was awake enough to argue about connection times. It only worked out because I was able to do their job for them.
Now put a 75-year-old with mild cognitive impairment in that chair. Ask them to remember the WhatsApp thread, find the bus to Terminal 4, clear security again, dial an extension on a landline, and repeat the story to the eighth person without dropping a detail.
They’d still be in Terminal 3.
This Is Just a Production Incident
If you write software for a living, you already recognize this.
The outage isn’t the failure. Everything fails eventually. The failure is that your recovery path only exists in the heads of the three people who’ve seen it before, and one of them is asleep and two of them are on vacation.
A runbook is boring. It’s a checklist. Symptom, check this, if that then this, escalate here, and here’s who to tell. It’s the least glamorous document in the repo.
It’s also what decides whether the person on call fixes it in ten minutes or spends the first hour just working out who to ask.
Justin Rhinesmith, our founding engineer at Boon and a good friend, pushes hard on this. Every system, all the time, and he doesn’t let it go. He’s right, and the Cathay day is what made me understand why he’s so annoying about it.
My rule is simple. You get to handle a production issue without a runbook exactly once. That’s the discovery cost, and it’s unavoidable. After that, writing the runbook is part of closing the incident. If it isn’t written down, you’re going to pay the discovery cost again, and next time it’ll be someone with less context paying it at 3am.
What the Runbook Would Say
Big caveat first. I don’t run an airline. I don’t know their staffing, their contracts with ground handlers, which of their systems can talk to which, or what a counter agent in Surabaya is actually allowed to promise me. The thresholds below are my own guesses too, including the six hour cutoff for putting someone in a hotel. There’s a lot of assumption riding on all of this, and I’d expect someone who does this for a living to tell me half of it is impractical.
But I’ve written enough incident procedures to have a rough idea what it should look like. Here’s what I’d have wanted somebody to be holding that morning.
Put everything in one record. Every disrupted passenger gets a case with an ID, and every fact about them goes in it. Not in an agent’s head, not in a WhatsApp thread. Any agent at any station should be able to open it cold and see where I am and what’s already been promised to me. Giving me one dedicated human sounds nicer, but that doesn’t survive three hundred passengers and six staff on shift.
Give me the case number. Cheapest fix on the list, and it deletes most of the day. If I could have shown a number to the officer at the bus stop, to the police officer, and to the man on the phone, then seven of my eight explanations never happen. Airlines already send me a booking reference the second I buy a ticket. Send me one when the trip breaks too.
A reroute is one operation, not two. They released my CX872 seat and left me with nothing for hours. I don’t think holding it was necessarily right, because that seat is scarce and somebody else could use it. But you don’t get to commit half the transaction. Release the old seat at the moment the new one is confirmed, not on a prediction and not four hours early.
Every instruction includes the how. “Someone will be waiting for you” isn’t an instruction. Terminal number, how to get there, roughly how long that takes, the desk or name to ask for, and what happens after that. If nobody is meeting me, say so and give me directions instead. Accurate bad news beats a comfortable promise.
Print the whole thing out. Not a scrap with an address on it. One page with the case number, the new flight details, the hotel and its booking reference, what’s covered and what isn’t, and who to talk to at each stop. Reception wasn’t being difficult with me. They couldn’t find me in any system, so that piece of paper was the only thing either of us had, and all it had on it was an address. Hand the same page to me, to the driver, and to the hotel, and most of that evening doesn’t happen.
CONNECTION BROKEN or CANCELLATION
|
v
[1] OPEN THE CASE (x)
one ID, all state lives in the record,
any agent opens it cold
|
v
[2] GIVE THE PASSENGER THE NUMBER (x)
one reference I can show to anyone
|
v
[3] REROUTE AS ONE OPERATION (x)
old seat released only when the
new one is confirmed
|
+--------+--------+
| |
wait < 6h wait >= 6h
| |
v v
lounge + meals [4] HOTEL (x)
printed page: case no,
booking ref, what's
covered, who to call
| |
+--------+--------+
|
v
[5] HAND OFF TO THE NEXT STATION (x)
they read the case before I land;
I get terminal, route, and a name
|
v
[6] CLOSE: improvised twice -> make it a step (x)
(x) = didn't happen on August 10Six out of six. I got a bed that night, but only because a receptionist made a phone call and decided to trust a man holding a scrap of paper. And I know step 6 never happened, because the taxi driver stood at that same desk and told them this keeps happening to him.
None of that is clever, and it isn’t supposed to be. If a document needs a smart person to interpret it at 6am, it isn’t finished.
Agents Make This Urgent
Here’s what changed my thinking on it.
For years I never prioritized this. There’s always something more urgent to ship, and a runbook only pays off on a bad day that hasn’t happened yet.
Now I can point an agent at a runbook and have it execute the thing. Check the dashboard, run the query, restart the worker, post the status update, escalate if the error rate is still climbing after five minutes. That’s not a hypothetical anymore. We’re doing it.
That changes what a runbook is worth to me. It’s not a doc that sits there waiting for a bad day anymore. I can hand it to an agent and have it actually run.
And an agent can only run a runbook that exists. If your recovery process lives in a senior engineer’s head, you can’t automate it, you can’t test it, and you can’t hand it to anybody. It’s just that guy.
That’s what Cathay had. Eight people, each doing their best from memory, no shared state, no checklist, no handoff. It worked out because I was healthy and stubborn and had a working phone.
Your users won’t always be.