There's a story about a Soviet officer named Stanislav Petrov during the Cold War saving the world. He was a lieutenant colonel in the Soviet Air Defense Forces. In his bunker outside of Moscow, just after midnight on September 26, 1983, the sirens and the warning light started going off. On the display board, he saw the word "ZA-pusk" (LAUNCH) in bold, red letters. A few moments later, a second, third, fourth, and then fifth launches were detected. The words "Ra-KYET-no-ye Na-pa-DYE-ni-ye" (MISSILE ATTACK) appeared in the same bold, red, flashing letters. According to protocol, he was immediately supposed to pick up the phone and run it up the chain of command, the result of which would have been a Soviet counterstrike of likely hundreds of ICBMs headed toward the US. But he wrote the incident off as a false alarm. The way I picture it, I'm back in my desktop-and-network-support days: you hit the side of the monitor a couple of times, eject the CD-ROM and wipe it on your shirt, unplug the thing from the wall, plug it back in, wait thirty seconds. I imagine he did some version of all of that. And the way I imagine it afterward, he's at a bar a few years later, tossing back Stoli with his coworkers, laughing. Remember the time I almost picked up the phone and told everyone to launch the counterstrike, and it turned out the model just hallucinated? Yes, comrade. We remember. Thanks for double-checking.
The system that raised the alarm was called Oko, which is Russian for "eye", and in 1983 it was brand new. It worked by infrared: satellites in long, looping orbits, staring down at the American missile fields for the heat bloom of a rocket leaving its silo. The engineers knew the obvious failure mode was sunlight, so they had the satellites watch the fields edge-on, against the cold black of space at the rim of the Earth, where a real launch's heat would stand out and stray reflections shouldn't reach. It was clever. And on that particular night, a freak seasonal alignment of the sun, the satellite, and the high-altitude clouds over the northern US put sunlight glinting off those cloud tops at the exact angle the system had been built to trust. The detection algorithm did precisely what it was designed to do. It saw heat where the math said heat meant launch, and it called it. One missile, then five.
It's tempting to call Oko an early AI, and to a layperson the difference barely registers: a machine took in a stream of sensor data, ran it through code that decided what the data meant, and announced a confident answer that happened to be catastrophically wrong. That's not quite AI in any sense worth defending, and honestly it doesn't matter. The shape of the failure is the thing. An automated system making a high-confidence call from its inputs, in a corner case nobody designing it had imagined, with the entire weight of the decision resting on whether one human chose to believe it. Strip the decade off it and that's a sentence I could write about plenty of the software I've shipped. And most of the software we all use every day.
As he sat with it, Petrov noticed that five missiles didn't fit any rational picture of a US first strike. You don't open with five. Ground radar showed nothing corroborating. Between his own early distrust of the new system and those two facts, he held the line while the minutes bled out. I imagine him sweating, and also doing the math on what this was going to cost him inside the military. Only when the "missiles" should have struck and didn't could he breathe. Petrov might have been the first human in the loop to ever save the world. I kind of doubt he'll be the last.
I think about ZA-pusk a lot.
Most software gets misused. I'd guess almost all of it does eventually, in some shape. It's baked into the usage model. You cannot possibly watch every interaction of every user, especially on platforms that can spawn millions of downstream effects from a single action. As employees, even as executives, there's only so far you can run something up the flagpole. When I was a CTO I didn't run the company. I ran development and mostly ran the product. Legal and finance were never inside my direct control, and they shouldn't have been. So I can't be certain a vendor isn't quietly overbilling us. I can't be certain that somewhere, on some pipeline, someone isn't using a messaging channel for something genuinely ugly. I can't be sure there aren't any one of ten million things happening every day that we don't have the bandwidth, the tooling, or the budget to chase down. And I have no idea what's happening outside the platform. What agreements our customers have signed with their customers, what terms they're operating under that I'll never see. I get a narrow slice of the puzzle, and even then only when I have the time and the facilities to look. We generated hundreds of gigabytes of events a day. Nobody reads hundreds of gigabytes a day.
So things slip. I've inherited products built on architectural decisions I'd never have made, and watched a few of them fail in ways no audit on earth would have flagged in advance. Developers write bad code. People reuse lame passwords. Somebody leaves a laptop unlocked at a coffee shop while they're in the bathroom. In some of those cases "doing your best" genuinely isn't good enough. Your best needs to get better.
I've spent most of my career in or near regulated industries: banking, communications, consumer finance. I remember going through my first PCI-DSS audit back when that was a monumental lift, and then a few years later you just bolted on Stripe and got most of it for free. I've signed dozens, maybe hundreds, of data-security questionnaires from customers and vendors. I've been through SOC 2 and ISO audits more times than I can count, technical due diligence on my own companies and on clients', pen tests, code audits, open-source audits, root-cause audits, and the occasional breach-and-leak audit, which are not my favorite weeks. And in all of that, across all those rooms, I've never once met a person who could honestly say they did something poorly on purpose, at least insofar as they understood at the time that it was poor. The bad outcomes almost never come from bad intent. They come from speed, blind spots, and a rulebook written for a different decade.
In a lot of cases the regulatory and legal environment just hasn't aged well, or hasn't caught up fast enough, or both.
Take one of my favorite technologies: AI voice agents. I've built a lot of them, on a lot of platforms: Deepgram, Retell, ElevenLabs, Twilio, the whole stack, in production and in skunkworks. It's genuinely incredible. The latency is good now, the integration footprint is world-class, and I have no doubt that within ten years you will never want to talk to a real human on the phone again, at least not when you're calling a business. But the legal scaffolding underneath it is decades behind.
For fifty years our "infrastructure" for order and compliance on the phone was rules-based IVRs, voicemail boxes on a physical machine and then in the cloud, power dialers, copper lines, and phone numbers held and parceled out by carriers. Today a huge and growing share of US phone traffic flows through UCaaS and CPaaS. Number portability is just how it works now. Layer an AI voice agent on top of that. First as a smarter IVR, then as something that can actually do the thing: book the appointment, quote the hours, issue the refund, place the order. The entire calling universe has been living in a gray zone for a long time. Every time you call your bank, your dealership, probably your dentist, almost certainly your doctor, you're being recorded, and either the system or the human is supposed to tell you so for quality assurance purposes. But humans forget, and businesses run three providers at once, so half the time you either never hear the disclosure or you hear it three times. In a two-party-consent state, the gap version of that is a possible federal wiretap question, plausibly a state felony, and probably civil liability. And here's the bind: the only way to prove the disclosure happened is to record the call. The same call you might not have been allowed to record.
Then you process that recording. You transcribe it, analyze the audio, generate summaries and exceptions and alerts, and feed all of it back to train the next model on what's happening across every call you handle. Fruit of the forbidden tree, all the way down. And at the end of the chain sits the agent itself: your speech gets turned into text by a speech-to-text provider, the text goes to an LLM for understanding, the LLM's output goes to a text-to-speech provider that turns it back into audio you can hear, and you do that loop dozens of times in a single conversation, all to move a dental cleaning from Tuesday to Thursday, all of it recorded. Colorado, California, and Utah have already passed laws about disclosing AI agents in communications, and more states will follow. But beyond that, the courts and the regulators are still not caught up to the "this call is being recorded for quality assurance purposes" from ten years ago, let alone to this.
And here's the shitty reality. You're going to make mistakes. Not because you didn't run things down, but because the whole scope of the thing is un-run-down-able. Communication moves so fast and so wide now that someone, somewhere, is going to find the soft spot. Every system has one. My wife used to manage a call center, and she is genuinely S-tier at breaking voice agents. She knows every trick, because her human agents ran them on her and their customers ran them on her agents, and now she runs them on the machines. I won't pretend I haven't told her, more than once, to please stop breaking the thing I built that was obviously perfect. She does not stop.
The environment around all of this is its own problem. There's a small army out there: plaintiff's attorneys, professional complainants, the occasional consumer who deliberately tries to provoke a company into a violation they can then sue over, what parts of the industry call "lead baiting." I genuinely cannot understand why a chunk of that isn't treated as manufactured standing and thrown out. Ergo, again, the system not keeping up. And it cuts the other way too. Plenty of business owners and managers I've talked to over the years, across a lot of industries, have looked at the gray area and the regulatory lag and quietly decided it's cheaper to risk the fine or the lawsuit than to pass up the chance to "aggressively pursue new business." Sometimes the system is behind. Sometimes people are just choosing the odds.
Which is where I keep landing, and where Petrov comes back in. AI has done two things I don't think have ever happened before, at least not at this scale. The first is that software can now be produced faster than any version of our legal system can adapt to it. The second is that the software, and the AI generating it, is moving faster than the human mind can build new frames to even comprehend it. And the honest part is that we almost certainly haven't seen the bulk of the acceleration yet. The gaps are going to get wider before anything closes them.
I've been pulled back toward Stoicism lately, mostly because I've watched my own attempts to control things that were never inside my control, usually in service of forcing some outcome, cause me more friction, internally and with the people around me, than the outcomes were ever worth. There's a particular flavor of this when you ask a good lawyer about something genuinely novel. There's no case law, because there's been no decision, because nobody's litigated the thing yet. So you get a careful, expensive non-answer, and you walk out having spent a real pile of money to feel slightly better and know almost exactly as much as you did walking in. That's not a knock on the lawyer. That's the territory.
So you do the only thing that actually works. You make a column for the things inside your control and a column for the things outside it, and you move on. This is not an argument for doing nothing, and it is the furthest possible thing from a license to be reckless. The opposite, actually. You exhaust every eventuality you can reach. You make the quality of your work genuinely above reproach. You bubble up what you see, clearly, to the people whose job it is to act on it. And then (this is the part I'm still learning) you accept that your knowledge was never going to be perfect, that you only ever saw a slice, and that some of the software is going to mistake a cloud for a missile.
You run the checks. You wipe the disc on your shirt. Under your breath, blyat (fuck). And then you put the phone down.