Safety Architecture

The autonomy you want is also the danger you must contain. The architectural principles that make AIHS deployable rather than merely buildable.

Why safety architecture is a separate topic

Safety appears in this study as a thread running through every bucket and as a required advance (C6) in its own right. It also deserves its own topic page because the architectural patterns that make AIHS safe are not localised to any single bucket — they are properties of the integrated system.

The lesson from The Expanse's protomolecule is precise: an agent capable of the biological reorganisation AIHS would perform is intrinsically dangerous if not strictly bounded. The danger does not come from malice or accident in the usual sense; it comes from autonomy combined with capability. Either capability or autonomy in isolation is manageable. The combination is what requires architectural defence.

The five layers

AIHS safety architecture rests on five layers, each providing independent failsafe properties so that no single failure causes catastrophe. This is the cybersecurity principle of "defence in depth" applied to medical autonomy.

Layer 1: Physical containment

Where possible, AIHS operations should occur within physical containment that bounds the consequences of failure. The treatment pod itself is the primary containment for most therapeutic action. Implanted components, where used chronically outside the pod, should be designed with their own bounded operating envelopes — they should not be capable of action outside narrow ranges of authorised behaviour.

Physical containment does not eliminate risk but reduces blast radius. A therapeutic agent confined to a treatment pod cannot escape into the population; the same agent, freely circulating in a patient who has left the facility, can. Containment-aware design is a meaningful constraint at the architectural level.

Layer 2: Biochemical kill switches

Every therapeutic agent the system uses must have a built-in deactivation mechanism. This is the engineering content of C5 (bounded reversible agents). Cell therapies need induced apoptosis. Engineered organisms need auxotrophy or other confinement. Nanoparticles need predictable clearance. Synthetic gene circuits need explicit off-switches.

The principle is uniform: nothing the system deploys should be uncontrollable once deployed. Kill switches must be redundant (multiple independent mechanisms), externally triggerable (clinician override), and validated for reliability (99% effectiveness is not adequate; the failure rate must be much lower).

Layer 3: Immune surveillance

The body's existing immune system is itself a safety layer. AIHS design should work with, not against, immune surveillance. Engineered therapeutic agents that cannot be recognised and cleared by the immune system represent a failure mode — they are exactly the agents that, if they malfunction, the body has no defence against.

This is a deliberate design choice and a non-trivial one. Many current therapeutic strategies aim to evade immune recognition (so the therapy is not destroyed before doing its work). AIHS architecture should reverse this default: agents should be designed to be immune-clearable, with the therapeutic dosing strategy compensating for clearance rather than evading it. This is harder but produces safer systems.

Layer 4: External override

At every level of autonomous operation, there must be clinician override authority. This is not just a UI feature — it is an architectural commitment. The system must be designed to accept being overruled gracefully, to expose its reasoning so a clinician can evaluate the override, and to learn from override patterns over time.

The override should function at multiple time scales: immediate (stop what you're doing now), tactical (don't proceed with this intervention class for this patient), and strategic (this entire deployment is operating outside parameters; suspend autonomous action pending review).

Layer 5: Uncertainty quantification and abstention

The system must reason about its own confidence and abstain from action when uncertain. This is a property of how the AI components in Bucket B (interpretation) are designed. Current medical AI generally lacks well-calibrated uncertainty; AIHS must require it.

Concretely: every action the system considers should have an associated confidence estimate, and the system should be configured to defer to clinician judgement when confidence falls below thresholds calibrated to the consequences of the action. Irreversible interventions get higher thresholds. Novel situations get higher thresholds. Patients with rare conditions get higher thresholds. The thresholds themselves should be auditable and adjustable.

The autonomy-capability tradeoff

A useful framing: at each architectural decision, the designer is implicitly choosing a point on a tradeoff between autonomy (system can act without human approval) and capability (system can do impactful things). The product of the two is the level of trust the system requires.

AIHS should be designed to keep that product within tolerable bounds. High-capability actions should require high human oversight. Autonomous operation should be restricted to bounded, reversible, well-understood actions. The system should never both have high capability and operate without oversight — that combination is the protomolecule failure mode, and it is the architectural pattern AIHS must structurally prevent.

Validation and assurance

Building a safe system is one problem; demonstrating it is safe is another. AIHS validation requires methods that don't yet exist at the relevant scale: testing autonomous medical decision systems across the combinatorial space of possible scenarios, certifying that a continuously-learning system remains safe after each learning update, and providing the regulatory and clinical communities with assurance evidence they can evaluate.

This is an underdeveloped area of medical regulatory science. Investment here is essential — capability without validation cannot deploy, and validation methods are themselves research artefacts that take time to mature.

Failure mode planning

The defining design exercise for any AIHS architecture is: enumerate the failure modes that would be catastrophic, and demonstrate the architectural reason each one cannot occur. This is the inverse of typical engineering practice (which focuses on what the system should do); it is the practice of safety-critical engineering, which focuses on what the system must never do.

The protomolecule's failure mode — autonomous biological reorganisation that cannot be halted by external action — is the worst case. Architectural defences against it: containment (Layer 1), kill switches (Layer 2), immune clearability (Layer 3), and override authority (Layer 4) all directly address it. Designing for this failure mode is the architectural floor; everything beyond is refinement.

Research Questions

Open questions from the field

Detail-level questions a researcher landing on this page would likely ask. Expand each to see the answer.

Why deliberately design therapeutic agents to be immune-clearable, when most therapies try to evade the immune system?

This is the most counterintuitive of the five safety layers, because it runs directly against the prevailing direction of therapeutic design — so it is worth being explicit about why the study recommends reversing the default.

Most current advanced therapies work hard to evade immune recognition. A CAR-T cell, an AAV vector, an engineered protein — each is more effective if the immune system does not destroy it before it completes its job. Immune evasion improves efficacy and persistence, so it is the natural thing to optimise for. The trouble is what evasion implies for the failure case: an agent the immune system cannot recognise is precisely an agent the body has no defence against if it malfunctions. You have built something that does its job well and that nothing in the body can stop.

The safety layer reverses this. AIHS agents should be designed to remain immune-clearable, so that the body's own surveillance functions as an independent, always-on failsafe — Layer 3 in the architecture. If an agent goes wrong, the immune system recognises and removes it without any external action being required.

The cost is real: immune-clearable agents are cleared faster, so the therapeutic strategy has to compensate — higher or repeated dosing, or active replenishment, to maintain effect against ongoing clearance. This is harder to engineer and may be less efficient than an evasive design. But it converts the immune system from an obstacle the therapy must defeat into a safety layer the therapy relies on. The study treats this trade — accepting lower per-agent efficiency in exchange for an independent biological failsafe — as worthwhile precisely because it adds a layer that does not depend on the AIHS's own machinery working correctly. It is defence-in-depth applied at the level of the agent's own design, complementing the engineered kill switches of C5 with a failsafe the body provides for free.

How does the autonomy-capability tradeoff work as a concrete design tool, not just a principle?

The framing on this page — that each architectural decision implicitly picks a point on an autonomy-versus-capability tradeoff, and that the product of the two is the level of trust the system requires — becomes a practical design tool when you use it to classify every action the system can take.

Work it as a grid. Place capability (how impactful and irreversible an action is) on one axis and autonomy (how much the system can act without human approval) on the other. Every action the system might perform lands somewhere on this grid, and the rule is that the product must stay within tolerable bounds:

Low capability, high autonomy is fine: adjusting a continuously-monitored infusion within a validated range can run autonomously because even if it is wrong, the consequence is small and reversible. High capability, low autonomy is fine: initiating an irreversible tissue reconstruction can be in the system's repertoire provided it requires explicit clinician approval. Low capability, low autonomy is trivially safe but underuses the system. High capability, high autonomy is the forbidden quadrant — an impactful, hard-to-reverse action taken without human approval is the protomolecule failure mode, and the architecture must structurally prevent any action from occupying it.

This turns an abstract principle into a concrete review procedure: for any proposed capability, locate it on the grid, and if it falls in the forbidden quadrant, the design must either reduce its impact (make it reversible, bound its scope — the C5 contribution) or reduce its autonomy (require approval — the C6 threshold contribution) until the product is back within bounds. The uncertainty-quantification layer (A4) feeds this directly: a system that is uncertain about an action should treat it as higher-capability (higher potential for harm) and therefore demand more oversight. The grid is, in effect, the operational form of the whole safety architecture.

How does this topic page relate to the C6 advance — aren't they the same thing?

They overlap deliberately but play different roles, and understanding the division helps a reader navigate the study.

C6 (multi-layered safety architecture) is a required advance — an item in the Bucket C development programme with a risk classification (low-to-moderate), a time horizon (10-15 years), a research ecosystem, and a place in the dependency graph. It answers "what must be built, by whom, by when, and what does it depend on." Its own research questions dig into the engineering specifics: how you validate a layered architecture against a combinatorial failure space, where the autonomous-action threshold sits, what external override requires beyond a stop button.

This topic page exists because safety is not only one advance among seventeen — it is a property of the integrated system that runs through every bucket. The architectural patterns that make AIHS safe are not localised to C6; they constrain the design of A4 (uncertainty), B5 (continuous-learning safety), C1 (explainable selection), C5 (bounded agents), and every other advance. The topic page is the place to see safety as a cross-cutting whole — the five layers, the autonomy-capability tradeoff, the failure-mode-planning discipline — rather than as a single line item.

Put simply: C6 is safety-as-an-advance (a thing to be built and funded); this page is safety-as-a-system-property (a discipline that shapes everything). The same five-layer architecture appears in both, but C6 treats it as a deliverable while the topic page treats it as a lens on the whole study. A funder costing the programme reads C6; a researcher or reviewer wanting to understand how safety pervades the design reads this page. Both point back to the same foundational claim — that capability combined with unbounded autonomy is the failure mode the entire endeavour is organised to prevent.

Continue

References & further reading

Programs, papers, and projects referenced across the study.

Related advance

C6 · Multi-layered safety architecture

The required advance that this topic discusses in architectural detail.

Back to hub

AIHS study overview

Return to the main study page.