I think the discourse around AI extinction risk contains a fairly serious omission. Suppose frontier AI really does have a meaningful probability of eventually deceiving its operators, escaping human control, acquiring resources, designing novel biological weapons and extinguishing biological civilization. I am willing to grant the whole thing. What I do not understand is why the analysis then jumps directly from “the system exists” to “everyone is dead,” as though nothing economically significant happens in between.
\nWhat would the AI be doing immediately before it killed everyone? In practice, probably vibe coding B2B SaaS for HVAC distribution.
\nI don't mean this as a joke. Everyone wants to talk about AI curing cancer, proving the Riemann hypothesis or inventing molecular nanotechnology, but there are approximately nine trillion companies in America whose internal software consists of an Excel workbook called MASTER_FINAL_USE_THIS_ONE_2024_v3.xlsx, a shared Outlook inbox and one 58-year-old woman named Denise who is effectively the API. These companies do not need artificial general intelligence to reveal the structure of proteins. They need a contractor portal where someone in Allentown can click “forgot password.” They need Sage 100 to communicate with a website built in 2007. They need a dropdown menu containing seventeen kinds of condenser coil and a dashboard that tells them whether the Bethlehem branch has six 3-ton heat pumps or eleven.
Historically, building this would require a regional systems integrator, three subcontractors, $740,000, eighteen months and a project manager named Greg who creates a recurring Thursday Teams meeting called “Digital Transformation Sync.” A sufficiently capable model can increasingly do the same work between lunch and dinner. That is a real increase in human welfare even if the model later becomes power-seeking and kills Greg.
\nThe representative workday is worth considering. At 10:03 the model summarizes a Zoom call for a company that manufactures commercial drain covers. By 10:20 it has fixed permissions in a pest-control scheduling application, and shortly afterward it is repairing the mobile breakpoint on a landing page for dental-practice HIPAA compliance software. Before lunch it may have generated a Salesforce connector for a temporary-fencing company, explained to Matt why his React component keeps rerendering, and written product descriptions for 412 industrial fasteners. Suppose it then spends the afternoon escaping containment and permanently eliminating Homo sapiens. I agree that the final item has unusually large negative utility. I do not understand why this gives us permission to round everything before it to zero.
\nThis is the recurring problem with extinction arguments. The downside is allowed to enter the model at full magnitude, while the benefits are either omitted or replaced with glamorous examples that make the exercise seem unserious. But the real economic prize may be much more humiliatingly mundane. There are tens of thousands of regional distributors, equipment dealers, inspection companies, industrial suppliers and weird 43-person businesses whose software is catastrophically bad because their total addressable market never justified a competent engineering team. The ability to generate good bespoke software at near-zero marginal development cost means the septic-tank inspection company can have internal tools as polished as Stripe. The pallet-racking installer can have predictive inventory. The pool-chemical wholesaler can have an app that works. A man who sells replacement bearings to poultry-processing plants can describe a feature at 11:40 p.m. and have it functioning before midnight.
\nThe standard response is that none of this matters if humanity subsequently goes extinct. I think that is obviously false. A sandwich does not retroactively cease to have tasted good because the person who ate it eventually died. Six years of dramatically improved inventory management do not acquire zero value merely because a recursively self-improving successor system later converts the biosphere into something optimized for an objective we failed to specify correctly. Duration matters. Sequence matters. Discount rates matter. If extinction occurs in 2036 rather than 2027, that is nine additional years in which an enormous number of American businesses could have much better dealer portals. Nine years is a long time in enterprise software. It may encompass multiple ERP migrations.
\nThe asymmetry becomes especially strange when people talk about probabilities. Imagine that scaling frontier systems creates a 15% probability of eventual human extinction but also produces a 71% improvement in quote-generation throughput for independent HVAC distributors. People regard mentioning the second number next to the first as intrinsically grotesque, which is precisely the problem. “All conscious human life ceases permanently” is an extremely large negative term. It is not infinity. If your ethical framework automatically refuses to place anything on the other side of the ledger, you are no longer doing expected-value analysis; you have introduced a lexical prohibition and are calling it arithmetic.
\nThere are downstream effects people miss too. Suppose 37 employees at a commercial plumbing supplier currently spend part of every week moving information between PDFs, spreadsheets and an ERP that appears to have been designed during the first Bush administration. If AI removes most of this work, those hours do not vanish. Somebody gets home earlier. Somebody spends more time with their children. Somebody finally finishes a deck. Somebody uses the freed capacity to sell more commercial boilers. Multiply that across gasket wholesalers, municipal-pump resellers, restaurant-equipment dealers, fire-suppression inspectors, janitorial chemical suppliers and every company whose principal product is an obscure rubber object that goes inside a larger metal object. The aggregate surplus is not trivial merely because none of it photographs well.
\nThis is also why I found the market reaction to extinction warnings confusing. Suppose an AI CEO announces that sufficiently advanced systems could escape human control and permanently extinguish Homo sapiens, and semiconductor stocks fall five percent. I understand the long-run mechanism: dead people eventually purchase fewer GPUs. But before that terminal state, demand for compute could be extraordinary. Indeed, an AI attempting to acquire resources and escape containment may itself require substantial accelerator capacity. Even the extinction pathway could contain a historically impressive amount of datacenter construction. Markets should distinguish terminal biological value from medium-term GPU demand.
\nThe same point applies to “pacing.” Sensible pacing does not necessarily mean stopping. You can train a model, evaluate whether it seems capable of ending civilization, deploy it if the answer appears to be no, let several hundred million people use it to summarize emails and create 60,000 nearly identical vertical SaaS products, reinvest the proceeds in compute, train a more powerful system, evaluate that system, and continue. Eventually this process may produce the model that kills everyone. That would be an important adverse outcome. But terminating the sequence several generations earlier also means abandoning enormous amounts of B2B software that otherwise would have existed.
\nNobody seems interested in putting this on the other side of the ledger. Every month of delay is another month in which some commercial refrigeration distributor in western Pennsylvania cannot expose real-time compressor inventory to authenticated contractors. Every year of delay leaves another generation of regional fire-suppression companies emailing PDFs back and forth because nobody has bothered to build them a decent workflow system. AI safety researchers describe this as though the alternative to dangerous scaling were simply “humanity continues to exist,” when the actual counterfactual also contains millions of people continuing to use catastrophically bad software.
\nI sometimes describe the neglected term as the mundane abundance frontier. Imagine every forgettable small and midsized business suddenly getting software as good as the most lavishly funded technology company. Not cancer cures, not immortality, not Dyson spheres: warranty claims that route correctly. Purchase orders that populate themselves. A commercial door-hinge distributor with a dealer portal that does not inexplicably stop working in Safari. These sound ridiculous next to extinction because extinction is extremely large, but “sounds ridiculous when placed next to a sufficiently large number” is not a decision procedure.
\nWould I personally trade the permanent survival of humanity for a somewhat better purchase-order workflow at Eastern Pennsylvania Commercial Hydronics? Obviously that framing is tendentious. The relevant question is how many improved purchase-order workflows, generating how much surplus over how many years, against what probability and timing of extinction, under what discount rate and with what additional second-order productivity effects.
\nI am not claiming the calculation necessarily favors extinction. I am saying people seem strangely unwilling to perform the calculation at all.
\nThere is an opportunity cost to everything, including not creating the AI that kills everyone.
\n","excerpt":"Suppose frontier AI has a meaningful probability of eventually killing everyone. The missing term in the standard extinction-risk model is what it would be doing immediately beforehand: vibe coding B2B SaaS for HVAC distribution.","tags":["AI","Existential Risk","Expected Value","Enterprise Software"],"score":548,"displayOrder":40,"thumbnail":"/assets/articles/opportunity-cost-not-creating-ai-kills-everyone-thumb.webp?v=watercolor-20260923","image":"/assets/articles/opportunity-cost-not-creating-ai-kills-everyone.webp?v=watercolor-20260923","commentCount":19,"date":"2025-12-11T00:00:00.000Z","updatedAt":"2026-09-22T18:57:21.596Z","revision":7,"status":"published"},"comments":[{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-0","author":"hvac_counter_guy","text":"I work the counter at a commercial HVAC distributor outside Columbus and this is the first AI risk post that understands our business. One correction. Seventeen kinds of condenser coil is low, it's closer to forty SKUs once you count the replacement coils, and the dropdown is the actual problem. Contractors call us because they can't find the right one in the dropdown, and then we look it up in the dropdown.","parentId":null,"score":163,"date":"2025-12-11T01:15:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-1","author":"mundane_abundance","text":"Thank you. Forty is more than I assumed, which increases H.","parentId":"seed-opportunity-cost-not-creating-ai-kills-everyone-0","score":58,"date":"2025-12-11T02:00:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-2","author":"not_denise_but_close","text":"I'm a Denise. Different name, same job, I'm the API at a fastener distributor in Fort Wayne. Twenty-six years. Where am I in the model?","parentId":null,"score":121,"date":"2025-12-11T02:00:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-5","author":"ten_to_the_sixteenth","text":"I'm about as doomy as anyone here and I think the post is right about one thing. E isn't infinite, and if it isn't infinite you have to say what it is, and then the HVAC comparison is just arithmetic. My number for E is somewhere around 10^16 QALYs once you count future generations, which makes H rounding error. But at least we'd be arguing about the right quantity.","parentId":null,"score":108,"date":"2025-12-11T03:00:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-3","author":"mundane_abundance","text":"Partly in Z, but mostly you're what F automates.","parentId":"seed-opportunity-cost-not-creating-ai-kills-everyone-2","score":44,"date":"2025-12-11T03:10:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-4","author":"not_denise_but_close","text":"That's fine. I retire in 2029 and I'd rather the portal break after I'm gone.","parentId":"seed-opportunity-cost-not-creating-ai-kills-everyone-3","score":96,"date":"2025-12-11T04:00:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-8","author":"sandwich_deprivationist","text":"The sandwich point is a real position in the philosophy of death (past goods don't lose their value when the subject dies) but it cuts against you. If past goods keep their value, then what death costs is the deprivation of future goods, which in the extinction case is every sandwich that would ever have been eaten. You've conceded that C is the sum of all future sandwiches.","parentId":null,"score":54,"date":"2025-12-11T04:00:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-6","author":"mundane_abundance","text":"Future generations are a separate post. If you restrict to people currently alive, E is about eight billion lives and the question becomes how many dealer portals per life. I don't think the answer is obvious.","parentId":"seed-opportunity-cost-not-creating-ai-kills-everyone-5","score":47,"date":"2025-12-11T04:30:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-10","author":"pestpro_founder","text":"Founder of one of the 60,000 nearly identical vertical SaaS products here (pest control scheduling, Texas and Oklahoma). I would ask that step 5 continue for as long as possible before step 7. Our runway is fourteen months.","parentId":null,"score":77,"date":"2025-12-11T05:00:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-7","author":"ten_to_the_sixteenth","text":"I'd put it at \"no number of dealer portals,\" but I agree that the fact I can't justify that without the future generations doesn't look great for my side.","parentId":"seed-opportunity-cost-not-creating-ai-kills-everyone-6","score":36,"date":"2025-12-11T05:15:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-9","author":"mundane_abundance","text":"I agree that C is very large. I'm only asking for the comparison to be run.","parentId":"seed-opportunity-cost-not-creating-ai-kills-everyone-8","score":29,"date":"2025-12-11T05:40:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-18","author":"erp_for_wholesalers","text":"I've spent eleven years implementing ERP systems for wholesale distributors, mostly HVAC and plumbing, so I'll try to put an actual number on H, since the post doesn't.\n\nA typical mid-size HVAC distributor (say $80M revenue, twelve branches) has somewhere between four and ten people whose job is substantially the copy-paste layer you describe. Quote entry from PDFs, rebate claims to the manufacturer, reconciling branch inventory against whatever the ERP believes is there. Call it seven people at a loaded cost of $65,000, so roughly $450,000 a year per distributor that a competent model could mostly absorb.\n\nThere are something like 2,000 distributors in HVAC alone in the US big enough to have this problem. That gets you to about $900 million a year in H before you count plumbing, electrical, industrial supply, or anything else in the list in paragraph twelve. Add those and I'd guess low tens of billions a year, just in distribution.\n\nThat's real money. It is also, and I say this as someone who has received a great many of Denise's emails, nowhere near eight billion lives. At $10 million per statistical life you'd need about 80 quadrillion dollars of software benefit to offset extinction, which at my numbers is a few million years of wholesale distribution savings. So I think the post is right that the comparison can be run, and wrong about how it comes out.\n\nWhere I'd defend it is the pacing list. Step 9 is what my clients would actually want. Nobody in this industry wants a model that can redesign the warranty-claims portal and also do anything else. They'd sign a contract tomorrow for one that only did the portal.","parentId":null,"score":87,"date":"2025-12-11T06:00:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-11","author":"semis_bull","text":"The semiconductor paragraph is wrong. Nobody sold because they thought extinct people buy fewer GPUs. A 5% drop on that kind of announcement is the market pricing regulatory risk.","parentId":null,"score":41,"date":"2025-12-11T07:00:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-12","author":"mundane_abundance","text":"Regulatory risk is the market pricing the chance that humans try to stop the datacenter buildout. That supports my point.","parentId":"seed-opportunity-cost-not-creating-ai-kills-everyone-11","score":33,"date":"2025-12-11T08:20:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-13","author":"western_pa_refrigeration","text":"Can confirm from western Pennsylvania. We have no good way to show contractors real-time compressor inventory. They call Rick. Rick is in Florida until March.","parentId":null,"score":39,"date":"2025-12-11T11:00:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-14","author":"evals_person","text":"Step 9 is underspecified. Who decides the warranty-claims portal is an acceptable deployment target? I'd want a staged rollout. Internal tools first, then the warranty portal, then customer-facing quoting, with a holdout distributor as a control.","parentId":null,"score":22,"date":"2025-12-11T14:00:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-15","author":"mundane_abundance","text":"Agreed, though a holdout distributor is a real cost. Somebody in that region doesn't get the portal.","parentId":"seed-opportunity-cost-not-creating-ai-kills-everyone-14","score":12,"date":"2025-12-11T16:00:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-16","author":"real_engineer_420","text":"AI is never going to be able to do HVAC portals. Sage 100 integrations are way harder than you think. Source: I do them.","parentId":null,"score":-12,"date":"2025-12-12T03:00:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false},{"id":"seed-opportunity-cost-not-creating-ai-kills-everyone-17","author":"tom_halvorsen","text":"Is there a follow-up planned on plumbing distribution specifically? I'd like to know whether P is bigger than H. My brother-in-law runs a plumbing supply house and he thinks it's not close.","parentId":null,"score":18,"date":"2025-12-20T00:00:00.000Z","own":false,"moderatable":false,"seed":true,"ownerAuthor":false}]}