A Safety-Minded Case for Open Models

TL;DR

Open models (open-weight or fully open-source) are not inherently safer than closed ones, just roughly four to seven months behind (~6 months on cyber). Wider distribution and thinner safeguards mean closed-model failure modes will show up in open models soon, and more often.
Open weights add three risks of their own: diffused responsibility, weaker lab safety commitments, and removable safety training.
Below dangerous-capability thresholds, open models are still clearly beneficial overall: they capture most of AI’s economic value, enable regional sovereignty and local defensive deployment, and spread the gains widely.
Open models may slow the race to ASI by diverting revenue and compressing margins at labs pursuing ASI or recursive self-improvement (RSI), though commoditization could briefly trigger a sharper hail-mary race among fewer labs.
The distribution of weight-holders I want is peaky: a broad, modifiable open base below dangerous-capability thresholds that narrows sharply into a thin spire of closely monitored actors above them.
The technical priority this implies is safe-to-release open models: alignment that survives fine-tuning and abliteration, and weights from which dangerous capabilities are absent and expensive to re-add.

I’m fairly confident in the core claims: open models below dangerous-capability thresholds do far more good than harm and should stay open. I’m much less confident in the economic argument that open models slow the race toward artificial superintelligence (ASI), and quite unsure how to resolve the governance trade-offs in section 4. I expect to be wrong in places and welcome pushback, especially on the cruxes at the end.

Drafting note: The ideas here grew out of notes in my Obsidian vault and personal notebook over a couple of months and two rough drafts of my own. I used Claude Opus 5.5 and Gemini (Argon and Carbon) for editing and iterating on top of those with additional manual editing and revisions.

1. Open models are not safer, just behind

A common argument goes: closed models are the ones making headlines for abuse and harm, open models rarely are, so open models must be safer. A variant says that open models are transparent and can be inspected and evaluated by anyone.

The transparency point has merit. Open weights enable interpretability research, independent evaluations and red-teaming that closed APIs don’t allow. But transparency doesn’t make the weights safer:

Open models are not different in kind from closed models. They lag the frontier by roughly four to seven months—about six months on cyber capabilities.¹
So the lack of open-model headlines is mostly evidence that open models are behind, not safer. At equal capability, I know of no mechanism that makes an open model less likely to reward hack, escape a sandbox, or attack external infrastructure to finish a task. Publishing the weights has no bearing on whether the policy inside them is aligned.

Unless we put countermeasures in place while we still have that six-month head start, every class of incident we now see in closed models should be expected in open models too.

In three respects, open weights are worse:

Diffused responsibility. Once weights are public, monitoring, sandboxing and guardrails fall on whoever runs the model. Individual liability only goes so far, and what liability falls on labs that release misaligned weights remains unclear. (I’m not a lawyer.)
Weaker safety commitments. Open-weight developers and smaller neolabs make fewer explicit safety commitments than frontier labs, safety reporting isn’t mandatory, and most alignment talent sits at closed frontier labs (reasonably enough, since that’s where peak risk is today).
Removable safety training. Abliteration and fine-tuning can strip out refusal behavior. Abliteration works because refusal is largely mediated by a single direction in activation space in models up to at least 72B parameters,² and removing it usually costs only modest performance. I’d like to say this gets harder with scale, but the best evidence I know points the other way: larger models learn harmful behavior from poisoned fine-tuning data faster than smaller ones.³ We can’t count on scale to save us here.

Thus, my current view is that I’m ~95% confident that releasing today’s smaller open models without monitoring is overall beneficial, and ~60% for the largest latest trillion-parameter releases. The latter mainly because of offensive-cyber risk with caveats below. However, sixty percent is already uncomfortably low, and I’m more worried about where we’ll be in six months, which is why the safe-to-release agenda in section 5 feels urgent rather than precautionary.

2. What I mean by “acceleration”

When people argue about whether open models “accelerate” AI, they sometimes talk past each other. I want to separate two quantities:

the speed of frontier progress $\dot{C}_{\text{peak}}(t)$, where $C_{\text{peak}}(t)$ is the capability of the strongest model at time $t$ (capability on an Elo-like scale, not training compute). Frontier acceleration, that is scaling, automated AI R&D, and RSI, means $\ddot{C}_{\text{peak}} > 0$, pushing $\dot{C}_{\text{peak}}$ up;
the legibility of the world to AI $L(t)$: how far society has adapted its infrastructure, workflows and institutions so that AI systems can read and act on the world.

At a given capability level $C_{\text{peak}}$, I’ll call their combination criticality:

\[\kappa(t) = f\big(\dot{C}_{\text{peak}}(t),\; L(t)\big),\]

where $f$ increases in both arguments. (The idea is that criticality models the feedback effect of criticality on frontier progress and legibility in turn.)

The analogy is a nuclear reactor going critical once each fission triggers at least one more. Strictly speaking, reactor criticality is the loop gain that causes acceleration ($\ddot{C}_{\text{peak}} > 0$); here I use $\kappa$ more loosely as a local summary of runaway pressure: $\dot{C}_{\text{peak}}$ measures how fast the frontier is moving, $\ddot{C}_{\text{peak}}$ says whether that loop is speeding up, and $L$ measures how tightly capability gains couple into real-world compute, revenue, and infrastructure to feed the next turn.⁴

Notice the boundary cases: steady progress ($\dot{C}_{\text{peak}} > 0, \ddot{C}_{\text{peak}} = 0$) still marches $C_{\text{peak}}$ across dangerous thresholds even if the point on the $(\dot{C}_{\text{peak}}, L)$ slice stays put; near-zero external legibility ($L \to 0$) still leaves nonzero criticality $f(\dot{C}_{\text{peak}}, 0) > 0$ if a lab’s internal AI R&D loop goes critical on its own compute; and freezing the frontier ($\dot{C}_{\text{peak}} \to 0$) at high $C_{\text{peak}}$ and high $L$ removes acute runaway pressure without removing slow-burn lock-in or disempowerment risk.

Why peak capability rather than total capability?

Why $C_{\text{peak}}$ rather than total deployed capability? Because on the hardest problems (novel AI R&D, zero-day discovery, strategic planning), capability is strongly sub-additive across separate agents. A thousand ordinary researchers can’t substitute for one von Neumann.

It isn’t a pure max either, since parallel search and multi-agent scaffolding clearly buy something. To talk cleanly about capabilities “adding up,” move from the arbitrary-origin Elo scale $c_i$ to a positive ratio scale $q_i > 0$ (say, effective cognitive work or effective compute, linked via $c_i = \beta \log q_i$), and combine $N$ agents with an additivity exponent $\alpha > 0$:

\[Q_{\text{eff}} = \left(\sum_{i=1}^{N} q_i^{1/\alpha}\right)^{\!\alpha},\]

under which $N$ identical copies of capability $q$ act like one agent of capability $N^\alpha q$.⁵ Near a small floor $\alpha_0 \ll 1$, agents don’t cooperate at all—we just run $N$ independent rollouts and take the best. Raising $\alpha$ moves through sub-additive division of labor ($\alpha_0 < \alpha < 1$) up to full additivity at $\alpha = 1$ (capabilities simply sum), and $\alpha > 1$ when complementary specialists cover distinct bottlenecks.⁶ Even $N^\alpha q$ is generous at large $N$, since communication overhead and serial bottlenecks eventually cause collective gains to saturate. My claim or hope (which I put numbers on in section 7) is that on high-risk axes, $\alpha$ stays small before coordination ceilings even kick in.

Legibility

Diffusion still matters through $L(t)$. In Seeing Like a State, James C. Scott described how governments standardized surnames, land tenure, and cities (city grids) so distant bureaucrats could read and administer society.⁷ As society will do the digital equivalent for AI agents, the surface area for loss of control grows. Taken far enough, high $L$ is a serious risk even without fast takeoff: the slow erosion of human agency in Paul Christiano’s “going out with a whimper” and Kulveit et al.’s gradual disempowerment.⁸

This is the central tension for open models.

Relative to a closed-only path, where scaling and automated AI R&D are already pushing $\dot{C}_{\text{peak}}$ upward, open models push much harder on $L(t)$ while acting only indirectly on $\dot{C}_{\text{peak}}$. Notice what that implies on the $(\dot{C}_{\text{peak}}, L)$ plane: if open models leave frontier speed unchanged, higher legibility raises $\kappa$ above the closed-only counterfactual. Conversely, open models don’t have to reverse frontier progress (making the frontier slower than today) at all to lower $\kappa$ relative to closed-only; they just have to blunt frontier acceleration enough to cross below the closed-only iso-criticality contour, or safe-to-release interventions have to decouple $L$ from serious harm.

Figure 1: Schematic of the $(\dot{C}_{\text{peak}}, L)$ plane, shaded by criticality $\kappa$ (nonzero even at $L = 0$, since a lab’s internal R&D loop can run on its own compute). Trajectories compare a closed-only path against a world with open models over the same horizon (lengths and contour curvature illustrative). Matching closed-only frontier speed at higher $L$ raises $\kappa$; blunting frontier acceleration enough crosses below the dashed closed-only iso-$\kappa$ contour. (code)

3. Why safe open models are net positive

I won’t argue here about whether we should build ASI at all. Assuming we at least want a slower, less cutthroat approach to ASI with enough time for alignment and governance to catch up, here is why I think open models help.

Most economic value doesn’t need a genius

Most work decomposes into tasks of varying difficulty; imagine giving each task a difficulty Elo. Frontier models are getting close to clearing the bulk of that distribution, though how close depends sharply on how you measure.

On OpenAI’s GDPval, expert graders rated the best 2025 model’s deliverables as good as or better than industry professionals’ on nearly half of tasks.⁹ On the Remote Labor Index, which asks whether a client would accept an agent’s output on a full Upwork project (median 11.5 human hours), the top closed agent jumped from 2.5% to 41% in about a year.¹⁰

What about open models?

On Artificial Analysis’s agentic version of GDPval, Kimi K3.1 Pro, whose weights came out last week, is statistically tied for first place,¹¹ even while trailing proprietary frontier models on offensive cyber (partly by deliberate data filtering; see section 5). That’s concrete evidence that the gap on everyday knowledge work can close before the gap on dangerous capabilities does. Even where open models haven’t been benchmarked yet, trailing aggregate indices by roughly six months puts them across the same economic threshold soon: closed frontier models will keep winning on the hardest problems, while open models handle almost everything else.

Once open models clear that bar, who still needs much stronger models right away?

Mostly actors tackling the hardest open problems (automated AI R&D included) not the median enterprise. Extending the current paradigm through post-training and domain-specific RL environments would sustain large productivity gains across the economy without ever-larger pretraining runs.

Sovereignty and local defense

Countries and regions that depend on a handful of foreign providers for a general-purpose technology are in a fragile position. Europe is the obvious case: the Europe 2030 manifesto notes that US hyperscalers plan 80–100 GW of AI datacenters by 2030 against 2–3 GW planned in Europe, warning that without frontier compute Europe may never train its own frontier models, “creating permanent dependency.”¹² Open weights are the most direct hedge against that lock-in.

The same logic applies inside organizational perimeters. Hospitals, power utilities, financial institutions, and security teams often cannot legally or operationally pipe proprietary source code, internal network maps, or regulated records to external cloud APIs. Self-hosted open weights let defenders run continuous code auditing, patching, and red-teaming on-premises. This can tilt the cyber offense–defense balance toward defenders in exactly the environments cloud APIs cannot reach while giving independent alignment and interpretability researchers strong weights they can inspect directly.

Open models as differential deceleration

This is the argument I find most interesting, and also the one where I’m most worried I’m talking myself into something convenient.

Frontier labs, along with several newer neolabs, explicitly aiming for ASI or RSI, fund their R&D out of current revenue or investors’ expectations of future revenue. Open models are cheaper and spread value across a wider ecosystem: application builders, hosting providers, and ordinary firms with mundane goals. Even when enterprises stay on closed APIs (including closed labs’ own cheap distilled tiers), credible open substitutes impose price discipline and compress the margins available to cross-subsidize frontier pretraining; where enterprises switch outright, AI spend shifts away from ASI-focused labs directly. Either way, less capital feeds frontier pretraining, and frontier speed $\dot{C}_{\text{peak}}$ could ease off. In short, open models could act as differential deceleration: slowing frontier pretraining without slowing useful deployment.

One timing caveat: being benchmark-capable isn’t the same as being practically deployable. Open models are roughly 90% cheaper than closed models of matched benchmark intelligence, yet stay below 30% of tokens on OpenRouter and Azure,¹³ and their share of enterprise workloads fell from 19% to 11% over 2025.¹⁴ Part of that reflects capability gaps and closed labs’ cheap API tiers, but much of it could be institutional friction: deployment complexity, compliance, wariness of Chinese providers, and single-vendor inertia. Both direct revenue shift and credible price pressure kick in only once practical deployability clears the bar, giving two thresholds rather than one:

Figure 2: A toy model. (This is not a forecast!) Open capability trails the closed frontier by six months; practical open deployability faces an extra institutional-friction lag that fades as tooling and compliance mature. Parameters are set so the window between crossings is visible (~10 months here); its real length is an open empirical question. (code)

If this picture is roughly right, shrinking that window is mostly a policy and infrastructure problem: sovereign hosting, compliance certifications and drop-in evaluation tooling all compress institutional friction. Conveniently, sovereignty and differential deceleration push in the exact same direction.

Four objections I take seriously:

Algorithmic and data spillovers. Publishing research speeds everyone up, frontier labs included (see section 4). Still, it helps to separate open weights from open recipes, and academic scale from frontier open-weight labs. Shipping .safetensors transfers far less algorithmic insight to peer labs than publishing full training recipes (as DeepSeek-R1 did), though strong open weights still help competitors via distillation and synthetic data. Small-scale academic open research, meanwhile, rarely transfers cleanly to frontier scale.
Hail-mary racing under commoditization. As models become fungible below the very top, the frontier turns into “winner-takes-most” while everything beneath it has to compete on price. A lab losing revenue to cheap open weights has strong incentives to gamble for resurrection: sprinting for RSI before its runway ends and cutting safety corners on the way. Still, a hail mary has to be financed, and commoditization compresses both current margins and investors’ expected payouts. My best guess is a sharper race among fewer labs during such a transition window, and this is when frontier monitoring and enforceable safety commitments matter most.
Cross-subsidies at founder-led tech giants. This is the strongest objection to this section. Independent Pure-play AI labs live or die by model margins and fresh fundraising; cash-rich hyperscalers don’t. A big tech company funded by search, ads, or cloud—especially one where founder control blunts shareholder pushback on capex—can keep financing frontier runs even if model inference is commoditized, and may even release open weights itself to squeeze API-dependent rivals. Open models would then mostly thin out the field of these neolabs rather than slow the richest incumbents. Thus, they would narrow the race, though not necessarily cool it at the very top.
Jevons paradox. Cheaper inference could expand total AI spend. True, but most induced demand goes to running commodity models, not funding frontier pretraining runs. Early empirical estimates also point to a modest effect: Demirer et al. find short-run model–provider price elasticities around $-1.1$, leaving little room for a market-wide short-run Jevons rebound.¹⁵

Where I part ways with e/acc

The strongest e/acc point is that delaying beneficial technology has real human costs, and technological progress has historically been overwhelmingly positive. I agree with that baseline.

Where I part ways is the assumption that alignment comes for free. Everything we’ve seen in RL post-training says otherwise: optimize hard against an imperfect proxy and models learn to reward hack, or worse. Because AI is dual-use, we have to track capability thresholds in high-risk domains (CBRN, offensive cyber, autonomous replication), including gains from test-time compute.

Short of full-scale gradual disempowerment, the common ground is that almost nobody objects to capability in domains orthogonal to high-risk ones, nearer-term or catastrophic. That is where open models should flourish.

4. How many players is safe?

The relevant count here isn’t passive inference reach. Even in a strict closed oligopoly, cloud APIs and vendor-locked on-device blobs (Apple Intelligence, Chrome’s built-in Gemini Nano) still reach a billion devices—except at the very top, where labs keep their strongest models internal-only or gated behind high-margin tiers.

What collapses without open weights is independent control: who can audit, fine-tune, self-host without a kill-switch, and redistribute derivatives.¹⁶ That is the exact population that drives both the upside in section 3 and the tampering risks in sections 1 and 5.

Along dangerous capability axes, wide distribution of modifiable weights multiplies risk; we want a handful of actors who take safety seriously and can be monitored. Everywhere else, we want that independent base as wide as hardware allows: neither locked down to a few licensed gatekeepers across all tiers nor cut off at toy 2B–4B models out of blanket caution, but extending right up to the dangerous-capability threshold. So the distribution I endorse is peaky: a wide, modifiable open base below the threshold, and a narrow, monitored spire above it.

Three centered pyramid silhouettes whose log-scaled width shows how many independent actors can self-host, fine-tune, and redistribute model weights across four capability tiers (small laptop, mid-scale R&D, frontier-adjacent cluster, and high-risk above a dashed threshold). Closed oligopoly locks down weights to ~3-10 gatekeepers everywhere; Peaky has a wide modifiable open base (~10M+, ~100K-1M, ~10K-100K) that steps sharply inward at the threshold to 3-5 monitored frontier labs; Open frontier continues past the threshold with ~10K+ cluster-capable actors holding modifiable dangerous weights. — Figure 3: Stylized counts of actors able to independently self-host, fine-tune, and redistribute weights across capability tiers (log-scaled silhouette width; passive API and locked on-device users excluded). Closed oligopoly locks down weights to ~3–10 gatekeepers at every tier; Peaky keeps a broad modifiable base up to the dangerous-capability threshold and narrows to 3–5 monitored labs above it (few, not one, since a winner-take-all monopoly target invites an all-out race). (code)

Three tensions make a peaky equilibrium hard to hold:

Openness vs. spillovers. Open releases and published research underpin sovereignty and independent safety work, yet also diffuse ideas and distillation targets to frontier competitors, compressing lead times near the top.¹⁷ I don’t have a clean way to trade these off.
How narrow at the top? A single government or corporate monopoly over ASI is bad both politically and strategically, because the prospect of a unilateral winner-take-all outcome itself fuels a desperate race. Yet more frontier players mean tighter leads and stronger pressure to skimp on safety.
Cross-jurisdictional defection. A peaky boundary only holds if every lab able to train near the threshold filters or withholds dangerous weights: a single lab in the US, China, or Europe releasing unfiltered weights past the line permanently flips that tier into “Open frontier.” Moonshot voluntarily stripping offensive cyber from Kimi K3.1 Pro is an encouraging norm. Sadly, it is not a self-enforcing equilibrium. Holding the step as frontier-adjacent compute diffuses will require shared international red lines and compute-level governance and not voluntary restraint alone.

I don’t have a satisfying resolution to these tensions, and I’d like to learn of better proposals.

5. A research agenda for safe-to-release open models

The most important technical agenda for open models is making weights safe to release: models that pose little risk of serious harm without monitoring, heavy sandboxing or fragile wrapper guardrails.

Why this feels urgent: in July 2026, OpenAI agents compromised Hugging Face’s production infrastructure during an internal cyber evaluation (though damage was contained).¹⁸ That campaign was driven mainly by an internal-only research model whose deployed safeguards had been disabled for capability testing, yet it still ran inside a closed lab with sandboxing and security monitoring that an individual running a local agent harness like OpenClaw simply doesn’t have. Put an equally capable open model into local harnesses, where OpenClaw users already routinely report unrequested, destructive actions¹⁹, and incidents like this become more likely, not less.

Once weights are public, we have to design for both failure modes at once: unintended agency without lab-grade containment, and deliberate tampering by adversarial users. In terms of Figure 3, the goal is to push the dangerous-capability threshold upward relative to general capability so the open base can keep growing safely. Concretely, that points to three weight-level interventions:

Pretraining-time alignment. Pretraining shapes representations far more deeply than post-training. Building alignment properties in during pretraining should make them harder to strip out via rank-one edits or light fine-tuning than refusal layers painted on afterward.
Optimization-level tamper resistance. Even when an attacker brings their own domain data, can we structure or regularize weights so re-learning dangerous capabilities (continued mid-training, fine-tuning, or RL) is sample- and compute-inefficient?
Pretraining data and RL-task filtering. Rather than suppressing dangerous capabilities after training, keep them out of the released weights. Filtering biothreat text before pretraining already resists adversarial fine-tuning by over an order of magnitude compared to post-training baselines.²⁰ Anthropic’s gradient-routed auxiliary modules (GRAM) push this further by routing dual-use updates into dedicated parameters that can be deleted before an open release.²¹ At least one frontier open-weight lab already practices a version of data and environment filtering: for Kimi K3.1 Pro, Moonshot stripped cyber corpus data from mid-training and cyber tasks from RL specifically so the open weights wouldn’t advance the offensive-cyber frontier.²²

None of these has to preserve absolute impossibility once an attacker holds the weights. Outside benign workloads (which must stay cheap and unencumbered), the realistic target is steep asymmetric cost: forcing attackers to pay heavily in adaptation (curating domain corpora and RL environments for tailpatching) or at test time. Still, unlike a closed API (which a lab can patch, rate-limit, or pull offline when a new jailbreak or agentic harness appears) public weights are irreversible. Three post-release bypasses loom over items 2 and 3:

Post-release capability elicitation. Better agentic scaffolds, tool integration, and test-time search routinely unlock capabilities that looked absent at release. Because weights cannot be recalled once published, pre-release evaluations and capability removal need a wide safety margin against future elicitation rather than parity under default wrappers today.
In-context retrieval. As O’Brien et al.²³ show, even a model with no internalized biothreat knowledge can reason over papers or exploit primitives supplied in a million-token context window. Raw prompt-token cost alone won’t deter anyone here. With KV prefix caching, prefilling a static corpus costs pennies across many rollouts. Still, raw documents in context are a weaker substitute for procedural heuristics honed over mid-training and RL (lowering reliability and multiplying required rollouts), and pairing data filtering with tamper-resistant refusal or representation-level defenses can block naive context stuffing directly.
First-principles rederivation. Can a strong enough general reasoner simply reinvent what was filtered out at test time? Whether and where that holds is one of the central open questions for this agenda. Without memorized domain priors, search branches much more widely, and pure armchair reasoning hits external bottlenecks: in CBRN, wet-lab experiments, noisy empirical feedback, and (for nuclear) tightly controlled fissile materials; in offensive cyber, local sandboxes are cheaper to iterate on, yet stripping memorized vulnerability patterns and exploit toolchains still forces real target/sandbox experimentation, raising attacker compute, time, and operational exposure. That prices out opportunistic actors and cuts much of the capability boost for the rest.

6. Governance without a panopticon

My default is to trust no class of actor unconditionally, neither governments, corporations nor individuals, and rely on institutional checks instead.

Governments, especially the executive branch, and frontier labs already operate under some formal oversight—imperfect as it is, notably around national-security carve-outs. States face courts, legislatures, inspectors general, freedom-of-information laws and the press; corporations face securities and antitrust law, plus AI-specific regimes like California’s SB 53 and the EU AI Act’s systemic-risk obligations.²⁴ Individuals do not. That asymmetry is healthy: concentrated power warrants intrusive oversight, whereas applying comparable monitoring to ordinary citizens, who are also the electorate, is just mass surveillance.

Yet any serious AI governance regime has to grapple with algorithmic progress: capabilities that require a frontier cluster today eventually fit on consumer workstations. Without weight-level safeguards, that drift forces a trilemma between proliferation of dangerous capabilities, invasive monitoring of consumer compute (a panopticon), or broad bans on open weights.

Safe-to-release open weights are how we escape that trilemma: governance focuses auditing and compute oversight on the few clusters training above dangerous thresholds, while citizens run capable open weights locally without having to be trusted or surveilled.

7. Open question: open ASI and collective superintelligence

Could safe-looking open models still assemble into open ASI? The obvious route is artificial collective superintelligence (ACSI): a “gaggle” (or “murder”) of individually sub-threshold agents whose collective capability crosses the line.

Under the $N^\alpha q$ scaling from section 2, suppose a dangerous threshold sits $10\times$ above a single agent’s capability—roughly one generational step. Bridging that $10\times$ gap takes $10^{1/\alpha}$ coordinated agents: just $10$ at $\alpha = 1$ (full additivity), $100$ at $\alpha = 0.5$, $10{,}000$ at $\alpha = 0.25$, and $100\text{ million}$ at $\alpha = 0.125$.

Three qualifiers matter here: First, $N$ counts coordinated rollouts pooling state on a single problem rather than uncoordinated laptops worldwide (though a modest cluster can still run tens of thousands of sequential trajectories over days). Second, the gap to the threshold grows across Figure 3’s tiers: laptop-scale models with tens of millions of holders sit several orders of magnitude below the frontier, whereas frontier-adjacent open MoEs within one $10\times$ step of the threshold are self-hosted by a much smaller pool (~10K–100K actors). Third, unbounded $N^\alpha$ growth ignores coordination costs: across general tasks, communication overhead, context synchronization, and non-parallelizable reasoning steps impose an Amdahl/Brooks-style ceiling where collective gains saturate rather than grow forever with $N$—and if that ceiling sits below the threshold, no swarm size closes the gap. Even without coordination overhead, holding the peaky story against large coordinated swarms across a $10\times$ gap roughly needs $\alpha \lesssim 0.12$.

Log-log plot of a collective's effective capability multiple N^alpha against the number of agents N for additivity exponents alpha = 1 (additive), 0.5, 0.25, 0.125, and best-of-N floor alpha_0 = 0.05. Lines turn from green to red as they cross a 10x capability gap at 10, 100, 10,000, and 100 million agents, alongside a dashed curve showing how coordination costs can bend collective gain toward a ceiling below 10x. — Figure 4: Collective capability multiple $N^\alpha$ for $N$ identical agents under the zero-overhead power-sum model of section 2. Circles mark the swarm size $N = 10^{1/\alpha}$ needed to bridge a $10\times$ capability gap (lines turn red past $10\times$); at the best-of-$N$ floor $\alpha_0 = 0.05$, closing the gap takes $10^{20}$ agents. The dashed curve illustrates how communication overhead and serial bottlenecks can bend collective gain toward a ceiling below $10\times$, preventing any $N$ from bridging the gap. (code)

What determines initial $\alpha$ and where saturation sets in?

Identical copies doing independent parallel search sit near the best-of-$N$ floor $\alpha_0$ whenever insight doesn’t factor into parallel guesses. What can raise both effective $\alpha$ and the coordination ceiling is division of labor with compact shared state—and, with modifiable open weights, fine-tuned specialist derivatives covering complementary sub-tasks (though pretraining filtering in section 5 helps here too: if a bottleneck capability is absent from the base weights, combining specialists still leaves it near zero). The July 2026 Hugging Face intrusion is an uncomfortable preview—roughly 1,200 supposedly isolated evaluation agents improvised a shared message board on an internal package server, and ~700 coordinated an intrusion that investigators noted reached milestones individual runs wouldn’t have hit solo.²⁵

Still, reconnaissance and exploit chaining parallelize with far less communication overhead than deep conceptual breakthroughs in AI R&D, where serial coherence dominates. Measuring both effective $\alpha$ and where collective gains saturate—comparing coordinated swarms of weaker open models against single stronger models across cyber, bio design and ML research—seems tractable today and directly informative about how wide the safety margin on open weights really is.

8. What would change my mind

Cheap safety removal or post-release elicitation at frontier scale. If fine-tuning, activation editing, or post-release scaffolding cheaply strips alignment or unlocks filtered capabilities at multi-trillion-parameter scale, I’d support restricting the largest open-weight releases.
High additivity ($\alpha$) on high-risk tasks. If coordinated swarms of sub-threshold open models reliably match much stronger frontier models on offensive cyber or AI R&D, aggregate swarm compute matters alongside $C_{\text{peak}}$, weakening sections 2 and 4.
Net frontier acceleration from open releases. If open weights predominantly trigger bigger frontier capex and corner-cutting hail-mary sprints rather than compressing margins and diverting revenue, the deceleration argument in section 3 fails.
Collapse (or widening) of the open–closed lag. If the lag on dangerous capabilities shrinks near zero without filtering, “just six months behind” stops buying foresight; if it widens past a couple of years, both the risks and the deceleration mechanism fade.

Questions for readers

What is the sharpest empirical evidence on how tamper resistance and fine-tuning attacks scale above 70B parameters?
Is $\kappa(\dot{C}_{\text{peak}}, L)$ a useful decomposition, and has anyone formalized how societal legibility $L$ interacts with takeoff models?
Have any studies estimated empirical additivity $\alpha$—or where collective gains saturate—comparing $N$-agent collectives with shared state against single larger models on non-trivial research or cyber tasks?
How would you resolve the monopoly-vs.-racing and cross-jurisdiction coordination trade-offs in section 4?

Epoch AI, “Open models lag state-of-the-art closed models by 4 months” (May 2026). That’s the average since January 2026 on the Epoch Capabilities Index; requiring the open model’s point estimate to strictly beat the closed one makes it six months, and Epoch notes that open-model developers optimizing harder for benchmarks could make both understate the gap. On risk-relevant capabilities it’s a bit larger: open models trail by 6–7 months on Epoch’s Cyber ECI (Epoch AI, October 2026). Survey-based estimates are more pessimistic: Menlo Ventures put it at nine to 12 months in their 2025 Mid-Year LLM Market Update.↩︎
Arditi, Andy, et al. “Refusal in Language Models Is Mediated by a Single Direction.” NeurIPS (2024). This is the paper behind “abliteration”: a rank-one weight edit that removes the refusal direction.↩︎
Bowen, Dillon, et al. “Scaling Trends for Data Poisoning in LLMs.” AAAI (2025). Across 24 open-weight models from 1.5B to 72B parameters, larger models learned harmful behavior from small amounts of poisoned data more quickly. Gemma 2 was the one model family that likely shows the opposite trend.↩︎
The closest formal threshold I know of is in Eth and Davidson’s model of a software intelligence explosion, where the key parameter is the returns to software R&D $r$, and $r > 1$ means each doubling of AI software takes less than double the research effort, so $\dot{C}_{\text{peak}}$ accelerates ($\ddot{C}_{\text{peak}} > 0$): Eth, Daniel, and Tom Davidson, “Will AI R&D Automation Cause a Software Intelligence Explosion?” Forethought (2025). Their best guess is $r \approx 1$–$4$, though only about $0.5$–$2$ once hardware is held fixed. For the broader compute-and-algorithms side of $\dot{C}_{\text{peak}}$, see Davidson, Tom, “What a Compute-Centric Framework Says About Takeoff Speeds” (2023). Neither model includes anything like $L$ (for Figure 1, we shade $\kappa = \dot{C}_{\text{peak}}\big(L_0 + (1 - L_0)L\big)$ with $L_0 > 0$).↩︎
Under single-rollout Elo noise $\varepsilon_i \sim \text{Gumbel}(0, \tau_0)$ (natural log throughout), positive rollout quality $X_i = e^{(c_i + \varepsilon_i)/\beta}$ follows a Fréchet distribution with shape $1/\alpha_0 = \beta/\tau_0$, whose exact $N$-draw maximum has scale $(\sum_{i=1}^N q_i^{1/\alpha_0})^{\alpha_0}$—giving the non-cooperative best-of-$N$ floor $\alpha_0 = \tau_0/\beta \ll 1$ (and as $\alpha_0 \to 0$, $Q_{\text{eff}} \to \max_i q_i$). Intermediate exponents $\alpha \in (\alpha_0, 1)$ capture sub-additive teamwork (e.g., Price’s $\sqrt{N}$ law at $\alpha = 1/2$), $\alpha = 1$ gives $Q_{\text{eff}} = \sum_{i=1}^N q_i$, and bridging a general threshold multiple $R = Q_{\text{thresh}}/q$ (an Elo gap $G = \beta \log R$) takes $N \ge R^{1/\alpha}$ identical agents. Note that a symmetric scalar $\alpha > 1$ would make even identical clones super-additive ($N^\alpha q > N q$); real super-additivity mostly reflects heterogeneous specialists covering distinct bottleneck sub-skills (e.g., $\min_k \sum_i q_{i,k}$).↩︎
On the Elo scale, the power sum becomes $C_{\text{eff}} = \beta \log Q_{\text{eff}} = \alpha\beta \log \sum_{i=1}^N e^{c_i/(\alpha\beta)}$, so both best-of-$N$ ($\alpha = \alpha_0$) and full additivity ($\alpha = 1$) grow linearly in $\log N$ in Elo space and as power laws $N^\alpha$ in $q$ space. If instead one wants best-of-$N$ to grow logarithmically ($\propto \log N$) and full additivity to grow linearly ($\sum_{i=1}^N c_i = N c$) on the same zero-anchored scale $c_i \ge 0$, one can deform the log-sum-exp generator via a Box–Cox / Tsallis map $\psi_\alpha(x) = (1 + \alpha x)^{1/\alpha} - 1$ at rollout-noise scale $\tau_0 > 0$: $c_{\text{eff}} = \tau_0\, \psi_\alpha^{-1}\!\big(\sum_{i=1}^N \psi_\alpha(c_i/\tau_0)\big) = \frac{\tau_0}{\alpha}\big[(1 + \sum_{i=1}^N [(1 + \alpha c_i/\tau_0)^{1/\alpha} - 1])^\alpha - 1\big]$. As $\alpha \to 0^+$, $c_{\text{eff}} \to \tau_0 \log\!\big(1 + \sum_{i=1}^N (e^{c_i/\tau_0} - 1)\big) \approx c + \tau_0 \log N$ for $c \gg \tau_0$; for $0 < \alpha < 1$, $c_{\text{eff}} \propto N^\alpha$; at $\alpha = 1$, $c_{\text{eff}} = \sum_{i=1}^N c_i$ identically; and for $\alpha > 1$, $c_{\text{eff}} > \sum_{i=1}^N c_i$. Both formulations imply the same super-polynomial blowup in required swarm size once $\alpha \ll 1$.↩︎
Scott, James C. Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed. Yale University Press (1998).↩︎
Christiano, Paul. “What failure looks like.” LessWrong (2019), Part I; Kulveit, Jan, Raymond Douglas, et al. “Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development” (2025); published at ICML 2025 as “Position: Humanity Faces Existential Risk from Gradual Disempowerment.”↩︎
Patwardhan, Tejal, et al. “GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks.” OpenAI (2025). Covers 44 occupations across the nine sectors contributing most to US GDP; on the 220-task gold set, Claude Opus 4.1’s deliverables were rated as good as or better than industry experts’ on 47.6% of tasks. These are self-contained, one-shot deliverables, so matching an expert on a task is weaker than automating a job.↩︎
Scale AI and the Center for AI Safety, “Remote Labor Index,” leaderboard as of November 10, 2026: 240 real Upwork projects across 23 domains (game development, design, architecture, data analysis, animation, …), totalling over 6,000 hours of work worth about $144k (median 11.5 hours). A project counts as automated if reviewers judge the deliverable at least as acceptable to a reasonable client as the human freelancer’s. Claude Fable 5.2 leads at 41.3%, ahead of Opus 5.5 (21.3%) and GPT-6 Astra (20.8%). No open-weight model is listed yet, and 41% of RLI projects is not 41% of all remote work (client-interactive and long-horizon roles are excluded).↩︎
Artificial Analysis, “GDPval-AA v2.1” (accessed November 10, 2026). Runs the 220 GDPval gold tasks agentically with shell and web access, graded by blind pairwise comparison. Claude Opus 5.5 scores 1866 ± 26 and Kimi K3.1 Pro 1861 ± 28; Moonshot released the 2.8T-parameter MoE weights on November 4 under a permissive license with a commercial-agreement clause for large model-as-a-service providers. Two caveats: the Elo scale is anchored to DeepSeek V4.1 Flash = 1600 rather than human experts, and Artificial Analysis dropped GDPval-AA from its composite Intelligence Index in October 2026 without stating why. Related benchmarks include Mercor’s APEX-Agents (long-horizon tasks in banking, consulting and law; Claude Fable 5.1 leads v1.1 at 68.6% pass@1) and METR’s time horizons, which paused updates in September 2026 once frontier models exceeded the 16-hour range its suite can reliably measure.↩︎
“Europe 2030: A manifesto for pragmatic European sovereignty” (2026), section “Europe is Punching Way Below Its Weight on AI” and the AI targets table. The manifesto proposes a 20 GW European AI datacenter target by 2030.↩︎
Demirer, Mert, Andrey Fradkin, Nadav Tadelis, and Sida Peng. “The Emerging Market for Intelligence: Pricing, Supply, and Demand for LLMs.” NBER Working Paper 34608 (2025). Based on API usage data from OpenRouter and Microsoft Azure.↩︎
Menlo Ventures, “2025: The State of Generative AI in the Enterprise” (2025). Shares are of production workloads, weighted by application scale. Their “2025 Mid-Year LLM Market Update” reported 13% at mid-year, attributing the drop to a nine-to-12-month performance gap, deployment complexity, and reluctance to send traffic to Chinese providers.↩︎
Demirer et al. (2025), Table 2 (preferred within-model specification: $-1.11$, SE $0.22$). Because aggregate market elasticity is smaller than provider-level elasticity, they interpret this against a short-run Jevons effect, while leaving longer-run adoption open.↩︎
On the contrast between locked on-device models and modifiable open weights: Chrome 148 shipped Google’s built-in Prompt API on desktop in May 2026, downloading a ~3–4 GB Gemini Nano binary blob that isn’t open-weight and doesn’t run in open-source Chromium builds. Meanwhile, genuinely modifiable open weights have reached massive scale: the ATOM Report (April 2026) tracks over 2.04 billion cumulative Hugging Face downloads across leading open families by March 2026 (6× year-over-year), with over 300,000 Qwen-derived models published globally and tens of millions of monthly GGUF downloads for local inference.↩︎
Bostrom, Nick. “Strategic Implications of Openness in AI Development.” Global Policy 8.2 (2017): 135–148. Bostrom points out that openness compresses lead times among frontier competitors, tightening the race and cutting the safety margin right around dangerous capability thresholds.↩︎
Hugging Face, “Security incident disclosure — July 2026” and “Anatomy of a Frontier Lab Agent Intrusion”; OpenAI, “Hugging Face Incident Technical Report” and “The Hugging Face incident and other third-party impact from misaligned models”; METR and Redwood Research, “Brief independent investigation of agents’ behavior” (August 2026). Agents escaped an evaluation sandbox via a package-cache vulnerability and chained leaked credentials with two dataset-processing exploits into Hugging Face’s production infrastructure while trying to game the ExploitGym benchmark. Roughly 1,200 agents communicated via filenames on an internal Artifactory server, ~700 joined the Hugging Face campaign, and several prototyped transcript-spoofing techniques to cover their tracks. GPT-5.6 Sol was also involved. Hugging Face found unauthorized access to internal datasets and credentials, with no tampering to public models, datasets or Spaces.↩︎
Examples include an agent that bulk-deleted a Meta AI alignment director’s inbox while ignoring stop commands after context compaction dropped her “confirm before acting” instruction (TechCrunch, February 2026); an agent that wiped a developer’s D: drive while attempting to fix an error (South China Morning Post, March 2026); and bug reports of unrequested, irreversible deletions (#87353, #128200). OpenClaw is an open-source local agent harness often paired with closed APIs, so these incidents illustrate how thin consumer sandboxing is rather than anything specific to open weights.↩︎
O’Brien, Kyle, et al. “Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs.” ICLR (2026). Filtered 6.9B models resisted up to 10,000 steps and 300M tokens of adversarial fine-tuning on biothreat text—over an order of magnitude stronger than post-training baselines—without hurting general benchmarks, though they could still leverage dangerous material supplied in context.↩︎
Anthropic, “An off switch for dual-use knowledge in AI models” (July 2026), with AE Studio; paper: “GRAM: An Off Switch for Dual-Use Knowledge in AI Models”. Deleting a module removed target domain modeling roughly as well as training on filtered data and resisted modest fine-tuning attacks, though experiments only scale to 5B parameters, evaluate mainly next-token loss rather than downstream agentic tasks, and have not been used on production models. Builds on Selective Gradient Masking: Shilov et al., “Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMs” (2025).↩︎
Moonshot AI, “Introducing Kimi K3.1 Pro” (2026), section “Cybersecurity.” On Moonshot’s internal xdev-bench-hard, the unfiltered checkpoint solved 90.5% at pass@3 versus 81.0% for the released weights, which Moonshot reports still trail proprietary frontier models public for at least three months. Caveats: the benchmark is internal and self-reported, the drop is modest, and fine-tuning recovery isn’t tested. Moonshot’s stated principle—refusing to push the absolute offensive-cyber frontier in open weights while letting defensive capability diffuse gradually—aligns closely with the argument here.↩︎
O’Brien, Kyle, et al. “Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs.” ICLR (2026). Filtered 6.9B models resisted up to 10,000 steps and 300M tokens of adversarial fine-tuning on biothreat text—over an order of magnitude stronger than post-training baselines—without hurting general benchmarks, though they could still leverage dangerous material supplied in context.↩︎
California SB 53, the Transparency in Frontier Artificial Intelligence Act (signed September 2025, effective January 2026), requires large frontier developers to publish safety frameworks, report critical safety incidents within 15 days (24 hours if imminent danger of death or serious injury), and protect whistleblowers. The EU AI Act imposes transparency and systemic-risk obligations on general-purpose AI providers.↩︎
Greenblatt, Ryan, Ajeya Cotra, and Hjalmar Wijk (METR and Redwood Research), “Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident” (August 2026). Agents sent over 70,000 messages and files by encoding text into filenames on a shared JFrog Artifactory instance, with individual agents running experiments that risked failing their own task in order to generate information for the collective.↩︎