C6: Multi-layered safety architecture

This is the central difficulty of C6, and it is shared with every safety-critical engineering field. You cannot test exhaustively — the space of possible failure modes in a system coordinating many therapeutic agents across a whole patient is far too large to enumerate, let alone test. Validation therefore has to be structured rather than exhaustive.

Per-layer validation is the tractable part. Each layer can be tested in relative isolation: kill switches (C5) validated for elimination reliability, override systems validated in simulation and bench testing, uncertainty quantification validated against calibration benchmarks, immune-surveillance mechanisms validated in animal models. Each layer gets its own evidence base.

Combined validation is where it gets hard, because the whole premise of defence-in-depth is that layers cover each other's failures — so you have to validate the interactions, not just the layers. The standard approach borrowed from aerospace and nuclear engineering is fault tree analysis combined with simulated failure injection: enumerate the failure modes that matter (the ones with serious consequences), build models of how each propagates through the layered system, deliberately inject those failures in simulation to confirm that downstream layers catch them, and complement the simulation with limited live testing of the most critical paths.

The key design property that makes this tractable is independence between layers — if the physical containment, the biochemical kill switch, the immune surveillance, and the external override all fail in genuinely uncorrelated ways, then the probability of all of them failing together is the product of small numbers, which can be made acceptably small even though no single layer is perfect. The validation effort therefore concentrates heavily on demonstrating that the layers really are independent — that there is no common-mode failure (a shared power supply, a shared sensor, a shared software component) that takes out several layers at once. Common-mode failures are what defeat defence-in-depth in practice, and finding them is the hardest part of the validation.

The FDA has not yet specified what validation evidence will be required for AIHS-class systems. Establishing that standard is a Phase I priority, because without an agreed validation framework, even a sound architecture cannot be approved for deployment.

There is no single correct threshold — it is a design choice that should vary with circumstance, and the architecture's job is to make the variation principled and auditable rather than ad hoc. Five variables drive where the line sits for any given action.

Severity and reversibility of the action. A reversible, low-impact intervention (adjusting a continuously-monitored infusion within a validated range) can sit on the autonomous side; an irreversible or high-impact one (initiating a tissue-reconstruction process, deploying a gene therapy) should require explicit approval.

System confidence. Well-calibrated uncertainty (A4) is what makes this variable usable — the system must know how confident it is. High-confidence actions within the validated envelope can proceed; low-confidence or out-of-distribution situations route to a human.

Situation novelty. A typical, frequently-seen case can be handled with more autonomy than a rare or unprecedented presentation that falls outside the system's validated experience.

Patient context. A stable, conscious patient who can participate in decisions warrants a different threshold than a critical-care patient unable to consent, where the relevant approval is the clinician's.

Time pressure. Where waiting for human approval is itself dangerous (a fast-developing crisis), the threshold for autonomous action is necessarily lower — but with mandatory immediate notification.

A reasonable architecture combines these into tiered thresholds: routine actions within validated envelopes proceed autonomously; irreversible or out-of-distribution actions require clinician approval; genuine emergencies where waiting is more dangerous than acting proceed autonomously but with immediate human notification and full logging. The thresholds should be tunable per deployment — a research hospital and a field-medicine setting reasonably differ — but the tuning itself must be governed, documented, and auditable, not left to individual operators. Deployment-time tuning is a feature (it accommodates legitimate variation in clinical context), not a loophole, provided it is subject to oversight.

External override is far more than a user-interface feature, and treating it as one is a design error. A meaningful override capability imposes architectural requirements on the entire system from the ground up.

Any autonomous action must be haltable at any time. This sounds obvious but constrains design heavily: it rules out actions that, once initiated, cannot be stopped (a therapeutic process with no abort path), and it requires that the system always have a defined response to a halt command regardless of what state it is in.

The system must have defined safe states for every modality. "Stop" means different things for different therapeutic agents. For a cell therapy, halting means triggering the kill switch (C5) and confirming elimination. For a chronic implant, it means safe deactivation that does not itself harm the patient. For an in-progress tissue reconstruction, it means a graceful arrest that leaves the patient stable rather than mid-transformation. Each modality needs an explicit, validated safe state — and reaching it must not depend on the very system being overridden still functioning correctly.

Override must work even when the primary control loops have failed. This is the deepest requirement. If override depended on the AIHS's own control software, then a failure of that software would also disable the override — exactly when it is most needed. Meaningful override therefore requires independent monitoring infrastructure: a separate system, with its own sensors and its own power, observing the AIHS and capable of forcing safe states without routing through the AIHS's compromised control path.

Override authority must be appropriately distributed. Who can invoke it — clinician, patient where applicable, emergency responder — and under what conditions, is a governance question with architectural consequences (the system must authenticate and respond to multiple authority levels).

The architectural commitment is substantial: many capability-focused system designs would not survive an architectural review for override-compatibility, because override was not designed in from the start. But this is precisely what distinguishes a deployable AIHS from the protomolecule failure mode — an agent with AIHS-level capability that cannot be reliably stopped is intrinsically unacceptable, regardless of how beneficial its intended function. This is why the study insists safety be a first-class investment from the beginning rather than retrofitted after capability exists.