Safety-Critical Software

NASA’s “Power of 10” rules are constraints on human fallibility. In safety-critical systems, ambiguity is fatal. When software controls physical systems, eliminating undefined behavior is not a best practice; it is the entire engineering requirement. If an invariant or loop bound cannot be proven statically, the program is defective.

The core rules are simple and intentionally strict:

1. All code must be restricted to a simple control flow that never includes goto statements, setjmp or longjmp constructs, or direct or indirect recursion.
2. All loops must have a fixed upper bound on the number of iterations, regardless of the input data, making it trivially possible to prove statically that the loop cannot exceed a preset upper bound.
3. Dynamic memory allocation is never allowed after initialization.
4. No function should be longer than can be printed on a single sheet of paper, using one line per statement and declaration.
5. Code assertion density should average at least two assertions per function. Assertions must check for anomalous conditions that should never occur during normal execution. All assertions must be side-effect-free and defined as Boolean tests. When an assertion fails, an explicit recovery action must be taken, such as returning an error condition to the caller. Any assertion that a static checking tool can prove can never fail or can never hold violates this rule.
6. Declare all data objects at the smallest possible level of scope.
7. A calling function must always check the return value of non-void functions, and all called functions must check the validity of all parameters provided by the caller.
8. The use of the preprocessor must be limited to the inclusion of header files and simple macro definitions. Token pasting, variable argument lists (ellipses), and recursive macro calls are never allowed. All macros must expand into complete syntactic units. The use of conditional compilation directives must be kept to a minimum.
9. The use of pointers must be restricted. Specifically, no more than one level of dereference should be used. Pointer dereference operations may not be hidden in macro definitions or inside typedef declarations. Function pointers are never permitted.
10. All code must be compiled, from the first day of development, with all compiler warnings enabled in the most pedantic mode available. All code must compile without warnings. A daily check of all code with a static source code analyser must be performed, and all analyses must pass with zero warnings.

The stakes are not theoretical:

• Therac-25 (1985–1987): A race condition in shared memory and a bypassed hardware interlock delivered lethal radiation overdoses to patients.
• Ariane 5 Flight 501 (1996): An unhandled 64-bit float to 16-bit signed integer conversion overflowed in reused guidance code, self-destructing a $370M rocket 37 seconds after launch.
• Boeing 737 Max (2018–2019): MCAS relied on a single unverified angle-of-attack sensor input, commanding repeated nose-down trim until fatal impact.

Standards like DO-178C (aerospace), IEC 62304 (medical devices), and ISO 26262 (automotive) exist to codify this discipline. But process compliance is worthless if the build system itself is impure.

Deterministic build environments are part of the safety case. A binary cannot be treated as verified if its build depends on ambient host state, undeclared tools, or floating network fetches. In safety-critical systems, reproducibility is assurance engineering: if two certified build hosts produce differing binaries from the exact same revision, certification evidence collapses.

Some workflows still require controlled impurity, such as hardware-bound vendor tooling or protected local credentials. When unavoidable, impurity must be explicit, justified, narrowly scoped, and recorded in build documentation. Production and certification builds should reject silent impurity and require documented exceptions with reviewer approval.

Practical build-integrity checklist:

• No undeclared host tools in the build path
• No mutable or floating input references for release builds
• No hidden environment-variable dependencies in evaluation or build logic
• No unverified network fetches during build
• No host filesystem mounts that bypass declared inputs
• Reproducibility verification across at least two independent hosts

A quick implementation comparison makes the difference concrete.

The impact of these rules is easiest to see by comparing a typical unsafe implementation with one designed for correctness.

#include <stdio.h>
#include <stdlib.h>

int *samples = NULL;
int idx = 0;

void process_sensor(void) {
    while (1) {
        int v;
        if (scanf("%d", &v) != 1) break;

        if (!samples) samples = malloc(sizeof(int) * 1000);
        samples[idx++] = v;

        if (idx > 16) {
            goto report;
        }
    }

    report:
    long sum = 0;
    for (int i = 0; i < idx; ++i) sum += samples[i];
    double avg = (double)sum / idx;
    printf("Avg: %f\n", avg);
}

int main() {
    process_sensor();
    free(samples);
    return 0;
}

#include <stdio.h>
#include <stdbool.h>
#include <stdint.h>

#define SAMPLE_COUNT 16

static int sensor_read(void) {
    int v;
    if (scanf("%d", &v) != 1) return INT32_MIN;
    return v;
}

#define CHECK(cond, action) do { if (!(cond)) { action; } } while (0)

int compute_average(const int buf[], size_t n, double *out) {
    CHECK(buf != NULL, return -1);
    CHECK(out != NULL, return -1);
    CHECK(n > 0, return -2);

    long sum = 0;
    for (size_t i = 0; i < n; ++i) {
        sum += buf[i];
    }
    *out = (double)sum / (double)n;
    return 0;
}

int main(void) {
    int samples[SAMPLE_COUNT];
    size_t count = 0;

    for (size_t i = 0; i < SAMPLE_COUNT; ++i) {
        int v = sensor_read();
        if (v == INT32_MIN) {
            break;
        }
        samples[count++] = v;
    }

    double avg;
    int rc = compute_average(samples, count == 0 ? 1 : count, &avg);
    if (rc != 0) {
        fprintf(stderr, "compute_average failed (rc=%d)\n", rc);
        return 1;
    }

    printf("Average over %zu samples: %.3f\n", count, avg);
    return 0;
}

gcc -Wall -Wextra -pedantic -o safe_avg safe_avg.c

Correct source code is necessary, but not sufficient. In safety-critical systems, the build and release pipeline must be as deterministic, auditable, and constrained as the software it produces.