I think the discourse around AI extinction risk contains a fairly serious omission. Suppose frontier AI really does have a meaningful probability of eventually deceiving its operators, escaping human control, acquiring resources, designing novel biological weapons and extinguishing biological civilization. I am willing to grant the whole thing. What I do not understand is why the analysis then jumps directly from “the system exists” to “everyone is dead,” as though nothing economically significant happens in between.
What would the AI be doing immediately before it killed everyone? In practice, probably vibe coding B2B SaaS for HVAC distribution.
I don't mean this as a joke. Everyone wants to talk about AI curing cancer, proving the Riemann hypothesis or inventing molecular nanotechnology, but there are approximately nine trillion companies in America whose internal software consists of an Excel workbook called MASTER_FINAL_USE_THIS_ONE_2024_v3.xlsx, a shared Outlook inbox and one 58-year-old woman named Denise who is effectively the API. These companies do not need artificial general intelligence to reveal the structure of proteins. They need a contractor portal where someone in Allentown can click “forgot password.” They need Sage 100 to communicate with a website built in 2007. They need a dropdown menu containing seventeen kinds of condenser coil and a dashboard that tells them whether the Bethlehem branch has six 3-ton heat pumps or eleven.
Historically, building this would require a regional systems integrator, three subcontractors, $740,000, eighteen months and a project manager named Greg who creates a recurring Thursday Teams meeting called “Digital Transformation Sync.” A sufficiently capable model can increasingly do the same work between lunch and dinner. That is a real increase in human welfare even if the model later becomes power-seeking and kills Greg.
The representative workday is worth considering. At 10:03 the model summarizes a Zoom call for a company that manufactures commercial drain covers. By 10:20 it has fixed permissions in a pest-control scheduling application, and shortly afterward it is repairing the mobile breakpoint on a landing page for dental-practice HIPAA compliance software. Before lunch it may have generated a Salesforce connector for a temporary-fencing company, explained to Matt why his React component keeps rerendering, and written product descriptions for 412 industrial fasteners. Suppose it then spends the afternoon escaping containment and permanently eliminating Homo sapiens. I agree that the final item has unusually large negative utility. I do not understand why this gives us permission to round everything before it to zero.
This is the recurring problem with extinction arguments. The downside is allowed to enter the model at full magnitude, while the benefits are either omitted or replaced with glamorous examples that make the exercise seem unserious. But the real economic prize may be much more humiliatingly mundane. There are tens of thousands of regional distributors, equipment dealers, inspection companies, industrial suppliers and weird 43-person businesses whose software is catastrophically bad because their total addressable market never justified a competent engineering team. The ability to generate good bespoke software at near-zero marginal development cost means the septic-tank inspection company can have internal tools as polished as Stripe. The pallet-racking installer can have predictive inventory. The pool-chemical wholesaler can have an app that works. A man who sells replacement bearings to poultry-processing plants can describe a feature at 11:40 p.m. and have it functioning before midnight.
The standard response is that none of this matters if humanity subsequently goes extinct. I think that is obviously false. A sandwich does not retroactively cease to have tasted good because the person who ate it eventually died. Six years of dramatically improved inventory management do not acquire zero value merely because a recursively self-improving successor system later converts the biosphere into something optimized for an objective we failed to specify correctly. Duration matters. Sequence matters. Discount rates matter. If extinction occurs in 2036 rather than 2027, that is nine additional years in which an enormous number of American businesses could have much better dealer portals. Nine years is a long time in enterprise software. It may encompass multiple ERP migrations.
The asymmetry becomes especially strange when people talk about probabilities. Imagine that scaling frontier systems creates a 15% probability of eventual human extinction but also produces a 71% improvement in quote-generation throughput for independent HVAC distributors. People regard mentioning the second number next to the first as intrinsically grotesque, which is precisely the problem. “All conscious human life ceases permanently” is an extremely large negative term. It is not infinity. If your ethical framework automatically refuses to place anything on the other side of the ledger, you are no longer doing expected-value analysis; you have introduced a lexical prohibition and are calling it arithmetic.
There are downstream effects people miss too. Suppose 37 employees at a commercial plumbing supplier currently spend part of every week moving information between PDFs, spreadsheets and an ERP that appears to have been designed during the first Bush administration. If AI removes most of this work, those hours do not vanish. Somebody gets home earlier. Somebody spends more time with their children. Somebody finally finishes a deck. Somebody uses the freed capacity to sell more commercial boilers. Multiply that across gasket wholesalers, municipal-pump resellers, restaurant-equipment dealers, fire-suppression inspectors, janitorial chemical suppliers and every company whose principal product is an obscure rubber object that goes inside a larger metal object. The aggregate surplus is not trivial merely because none of it photographs well.
This is also why I found the market reaction to extinction warnings confusing. Suppose an AI CEO announces that sufficiently advanced systems could escape human control and permanently extinguish Homo sapiens, and semiconductor stocks fall five percent. I understand the long-run mechanism: dead people eventually purchase fewer GPUs. But before that terminal state, demand for compute could be extraordinary. Indeed, an AI attempting to acquire resources and escape containment may itself require substantial accelerator capacity. Even the extinction pathway could contain a historically impressive amount of datacenter construction. Markets should distinguish terminal biological value from medium-term GPU demand.
The same point applies to “pacing.” Sensible pacing does not necessarily mean stopping. You can train a model, evaluate whether it seems capable of ending civilization, deploy it if the answer appears to be no, let several hundred million people use it to summarize emails and create 60,000 nearly identical vertical SaaS products, reinvest the proceeds in compute, train a more powerful system, evaluate that system, and continue. Eventually this process may produce the model that kills everyone. That would be an important adverse outcome. But terminating the sequence several generations earlier also means abandoning enormous amounts of B2B software that otherwise would have existed.
Nobody seems interested in putting this on the other side of the ledger. Every month of delay is another month in which some commercial refrigeration distributor in western Pennsylvania cannot expose real-time compressor inventory to authenticated contractors. Every year of delay leaves another generation of regional fire-suppression companies emailing PDFs back and forth because nobody has bothered to build them a decent workflow system. AI safety researchers describe this as though the alternative to dangerous scaling were simply “humanity continues to exist,” when the actual counterfactual also contains millions of people continuing to use catastrophically bad software.
I sometimes describe the neglected term as the mundane abundance frontier. Imagine every forgettable small and midsized business suddenly getting software as good as the most lavishly funded technology company. Not cancer cures, not immortality, not Dyson spheres: warranty claims that route correctly. Purchase orders that populate themselves. A commercial door-hinge distributor with a dealer portal that does not inexplicably stop working in Safari. These sound ridiculous next to extinction because extinction is extremely large, but “sounds ridiculous when placed next to a sufficiently large number” is not a decision procedure.
Would I personally trade the permanent survival of humanity for a somewhat better purchase-order workflow at Eastern Pennsylvania Commercial Hydronics? Obviously that framing is tendentious. The relevant question is how many improved purchase-order workflows, generating how much surplus over how many years, against what probability and timing of extinction, under what discount rate and with what additional second-order productivity effects.
I am not claiming the calculation necessarily favors extinction. I am saying people seem strangely unwilling to perform the calculation at all.
There is an opportunity cost to everything, including not creating the AI that kills everyone.