<!-- SPDX-License-Identifier: Apache-2.0 -->
<!-- SPDX-FileCopyrightText: 2026 ndaal GmbH & Co KG, Cologne -->
# Developer Guide
How this workspace is built, verified and extended. For deployment and
day-to-day operation see the
[Administrator guide](Administrator_Guide.md); for using the tools see the
[User guide](User_Guide.md).
## Architecture
Four published crates plus one internal crate:
| Crate | Kind | Responsibility |
| --- | --- | --- |
| `csaf-models` | library | Domain types, the redb/SQLite schema, `DbPool` |
| `csaf-core` | library | Storage, validation, import/export, the updater |
| `csaf-crud` | binary | HTTPS server, HTML UI, REST API |
| `ndaal-csaf-cli` | binary | Command-line import, export, validate |
| `workspace-hack` | internal | Feature unification (`cargo hakari`), never published |
Ten directories sit outside the workspace. The root `Cargo.toml` lists them
in `exclude`; all but `fuzz/` carry their own `[workspace]` table. The same
list also names `vulnerability-lookup-rs`, which does not exist in this
checkout.
- `fuzz-harness/` — the SINGLE definition of every fuzz body, one
`<target>::fuzz_drive(&[u8])` module per target.
- `fuzz/` — `cargo-fuzz` (libFuzzer) targets; thin wrappers.
- `hfuzz/` — honggfuzz targets; thin wrappers over the same bodies.
- `afl/` — `cargo-afl` (AFL++) targets; likewise.
- `bolero-harness/` — bolero harnesses, one body per engine plus Kani.
- `loom-harness/` — concurrency permutation models.
- `callgrind-harness/` — instruction-count benchmarks (iai-callgrind).
- `criterion-harness/` — wall-clock benchmarks (criterion).
- `divan-harness/` — wall-clock benchmarks with allocation counts (divan).
- `dhat-harness/` — heap-usage assertions (dhat), which installs its own
global allocator.
They are isolated for a shared reason and, in three cases, a harder one.
The shared reason: their dependencies must not reach the main `Cargo.lock`,
where `cargo-deny`, `cargo-geiger`, `cargo-machete` and `cargo-vet` would
have to account for them.
The harder reasons are each a build failure, not a preference:
- `loom-harness` — `--cfg loom` is set through `RUSTFLAGS` and propagates to
**every** crate in the compilation; `tokio` gates its whole `net` module
behind `#![cfg(not(loom))]`, so an in-workspace Loom test cannot build.
- `bolero-harness` — `cargo bolero test` compiles the crate with
`-Cpasses=sancov-module`, instrumenting every target in it. Sibling test
binaries that do not link the libFuzzer runtime then fail to link with
`"___sanitizer_cov_trace_switch", referenced from: ...`.
- `callgrind-harness` / `criterion-harness` — a `[[bench]]` with
`harness = false` inside a member crate is built *and run* by the
workspace's own `cargo test --workspace --all-targets` gate. For
callgrind that means shelling out to Valgrind on every test run.
## Toolchain pin and minimum version
Two Rust versions appear in this repository, and they differ on purpose:
- `rust-toolchain.toml` pins Rust `1.99.0` with the `clippy` and `rustfmt`
components. It is the exact version used to develop, test and release.
- `rust-version = "1.95.0"` in the root `Cargo.toml` is the minimum compiler
that a consumer needs to build the published crates.
Do not sync them. The minimum states what a consumer needs; the pin states
what the project builds with, so that code generation and lint results do
not drift between machines.
`rust-toolchain.toml` also lists the six release targets. A newly pinned
toolchain starts with only the host target, so the list is what lets a clean
clone run `release/create_release.sh` without a manual `rustup target add`.
Keep it in step with `TARGETS` and `MSVC_TARGETS` in that script.
The pin does not control every Rust invocation:
- `cargo kani` manages its own pinned nightly and ignores the file.
- `cargo +nightly fuzz`, `cargo +nightly udeps` and `cargo +nightly miri`
select a nightly explicitly, by design.
Change the pin in its own commit and re-run the full gate set. Each clippy
release adds lints and the workspace denies warnings, so a bump is a real
change.
## Verification layers
Each layer catches something the others structurally cannot. None replaces
another.
| Layer | Tool | Answers |
| --- | --- | --- |
| Unit + integration | `cargo test` | Does it do the right thing on the cases we thought of? |
| Property | `proptest` | Does it hold on cases nobody thought of? |
| Fuzz | `cargo-fuzz`, `honggfuzz`, `cargo-afl`, `bolero` | Does untrusted input crash it? |
| Compile-time invariants | `static_assertions` | Did a refactor break a type contract? |
| Concurrency | `loom` | Does it hold on *every* thread interleaving? |
| Model checking | `kani` | Does a counterexample *exist* in a bounded space? |
| Mutation | `cargo-mutants` | Do the tests actually catch a behaviour change? |
| Instruction count | `iai-callgrind` | Did this change cost more instructions? |
| Wall clock | `criterion` | Did the optimisation actually make it faster? |
The last one grades the others. Line coverage proves a line ran; the mutation
score proves a test *notices* when that line misbehaves.
### Running them
```bash
cargo test --workspace --all-targets --all-features # all but doctests
cargo test --workspace --all-features --doc # doctests: separate pass
bash tests/scripts/test_loom.sh # concurrency permutations
QG_RUN_KANI=1 bash tests/scripts/test_cargo_kani.sh
cargo mutants --package csaf-core
# Fuzzing — same body, four engines
cargo +nightly fuzz run fuzz_<target> -- -max_total_time=60
( cd hfuzz && cargo hfuzz run hfuzz_<target> )
( cd afl && cargo afl fuzz afl_<target> )
( cd bolero-harness && cargo bolero test <name> --engine kani )
# Benchmarks
( cd criterion-harness && cargo bench ) # wall clock, runs anywhere
( cd callgrind-harness && cargo bench ) # instructions; needs Valgrind
```
Never gate on `cargo test --lib` alone: it silently skips every integration
test under `crates/*/tests/` and every doctest.
### honggfuzz and AFL++ on macOS
Measured on 2026-10-21 on macOS 26.7.1 (Intel), Apple clang, rustc 1.99,
`honggfuzz` 0.5.62 and `cargo-afl` 0.18.2. Neither engine built out of the
box. Both build now; honggfuzz runs, and AFL++ needs one step from you.
**honggfuzz.** `cargo hfuzz build` failed in three places, in this order.
Each has an environment-variable workaround, so no third-party source
changes:
1. `Makefile: *** Unsupported MAC OS X version.` honggfuzz picks a
crash-reporter object by the macOS major version and its table ends at
13. `CRASH_REPORT` supplies the object it uses for 11 to 13.
2. The C sources fail under the Makefile's `-Werror` with the current Clang:
`-Wc23-extensions` (`= {}`), `-Wstrict-prototypes` and
`-Wc2y-extensions` (case ranges). The `-Wno-` flags below turn those off.
The last two (`-Wno-deprecated-non-prototype`,
`-Wno-unused-but-set-variable`) were added in the same go and were not
shown to be needed.
3. The link fails with `32-bit RIP-relative reference out of range`: the
macOS code declares a 5 GB static table (`g_fuzzer_crash_information`),
and the default linker places it in front of other data. The classic
linker puts it last.
```bash
( cd hfuzz && \
CRASH_REPORT=third_party/mac/CrashReport_Sierra.o \
CFLAGS="-O3 -mtune=native -funroll-loops -Wno-c23-extensions \
-Wno-c2y-extensions -Wno-strict-prototypes -Wno-deprecated-non-prototype \
-Wno-unused-but-set-variable" \
LDFLAGS="-Wl,-ld_classic" \
cargo hfuzz build )
```
That built all 36 targets in 3 minutes 35 seconds on a busy machine, and
`hfuzz_listen_port` ran for 20 seconds with 22,076 executions and no
crash. Apple has deprecated `-ld_classic`; when it is removed, workaround 3
is gone and honggfuzz needs a fix upstream.
**AFL++.** `cargo afl build` first stops with "AFL LLVM runtime was not
built": run `cargo afl config --build` once per rustc version. That in turn
failed with `could not copy directory .../cargo-afl-common-0.18.2/AFLplusplus`:
`cargo-afl` records the registry path of its AFL++ sources when it is
installed, and the unpacked directory had been removed from
`~/.cargo/registry/src`. Fetching the same crate version restores it (a
throw-away project with `cargo-afl-common = "=0.18.2"` and `cargo fetch`;
reinstalling `cargo-afl` does the same). Then `cargo afl build --release`
built all 36 targets, in 33 minutes on a busy machine.
Running them is where macOS stops: AFL++ aborts with `Crash reporter
detected`, and with that check overridden, `shmget() failed`. Both come from
macOS settings (the crash reporter, and the System V shared-memory limits).
`cargo afl system-config` changes them and needs `sudo`, so run it yourself;
settings of this kind do not survive a reboot. No AFL++ run has been made on
this machine. These are macOS findings: the Linux builds were not tried
here.
### Loom is a separate crate, and must stay one
`--cfg loom` is set through `RUSTFLAGS`, which propagates to **every** crate
in the compilation. `tokio` gates its whole `net` module behind
`#![cfg(not(loom))]`, so the moment the flag is set this workspace's
dependency tree stops compiling. An in-workspace Loom test does not run
slowly — it does not build.
The models therefore *mirror* the real disciplines rather than importing
them. That is a real limitation: **a change to `DbPool` or `AppState` that
diverges from its model is not caught automatically.** Each model names the
file and line it mirrors so the pairing survives review.
### Kani proofs state their bounds
A Kani result is a claim about a *bounded* space. The harnesses in
`crates/csaf-core/src/updater.rs` verify the path-traversal guard over names
of four characters drawn from `{'a', '/', '\', '.', NUL}` — 625 concrete
names. That bound is written in the source, because a verification result
without its bound is not a claim anyone can act on.
Every harness carries `kani::cover!` checks. A harness without them is
unreviewed: over-constrained assumptions make every later assertion
trivially true, and a vacuous proof looks exactly like a real one.
## Quality gates
```bash
cargo fmt --all --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo audit --deny warnings --deny unsound --deny unmaintained --deny yanked
cargo deny check
cargo machete --with-metadata
cargo geiger --workspace --all-features
cargo vet check
cargo hakari verify
```
`scripts/quality_gates.sh` is the canonical entry point; `--fast` skips the
live-server suites for a quick inner loop.
**Scope matters more than presence.** A gate that passes without exercising
the surface it claims to is worse than no gate. `cargo geiger` without
`--workspace` scans only the root package and reports nothing about any
member.
**A clean `cargo audit` is not a clean bill.** It reads the RustSec database,
which lags upstream. The `quinn-proto` denial of service fixed in 1.5.26
(GHSA-53rm-773f-4q8c) was published by the upstream project but not yet
imported by RustSec, so `cargo audit` stayed silent while 1.5.25 shipped the
flaw. For any dependency change, also check the upstream repository's own
security advisories and run an OSV scan (`osv-scanner`), as the 1.5.27 refresh
did.
### Clippy configuration lookup
Clippy reads the **nearest** `clippy.toml`, walking up from the crate it is
linting, and uses the first one it finds. It does **not** merge that file
with any file further up the tree.
Every crate directory has its own tracked `clippy.toml`: `csaf-models`,
`csaf-core`, `csaf-crud`, `csaf-cli` and `workspace-hack`. The five files are
identical and contain only the allowances for test code
(`allow-unwrap-in-tests`, `allow-expect-in-tests`,
`allow-indexing-slicing-in-tests`, `allow-panic-in-tests`,
`allow-print-in-tests`, `allow-dbg-in-tests` and `allow-useless-vec-in-tests`).
Two consequences follow:
- The root `clippy.toml` does not apply to those crates. Its
`disallowed-methods`, `disallowed-macros` and stricter complexity
thresholds are shadowed, so to enforce a setting on a crate, put it in
that crate's file.
- A crate without its own file falls back to the root one. Keep the files
tracked: two of them once existed only on one machine, and a clean clone
failed the `-D warnings` clippy gate on a root ban that the developer
tree never saw.
## Web interface design
The HTML interface follows the ndaal design system (`skills/ndaal-design`,
v1.1.0) and the accessibility standard it carries: WCAG 2.2 AA, 44 px
pointer targets, a dual-ring focus indicator, logical CSS properties so
Urdu (right to left) mirrors correctly, and reduced-motion, forced-colours
and increased-contrast support. The look follows sbom.ndaal.eu: a white
header with a hairline border, the skill's measured blue as the accent,
and Bootstrap Icons.
### Files
| Path (under `crates/csaf-crud/`) | Role |
| --- | --- |
| `static/css/tokens/ndaal-tokens.css` | The skill's token file, vendored unchanged. Never edit it |
| `static/css/tokens/csaf-tokens.css` | Only what the skill does not define: mono face, TLP colours, dark status chips |
| `static/css/base/base.css` | Bundled fonts and element defaults; the only place type selectors are allowed |
| `static/css/csaf-ui.css` | Components, classes only |
| `static/js/csaf-ui.js` | Menus, theme apply, confirmation dialog, TLP colouring |
| `static/img/icons.svg` | Sprite built from Bootstrap Icons (MIT) |
| `src/routes/layout.rs` | The page shell, header, theme and language menus |
| `src/routes/ui.rs` | Shared markup helpers: `icon`, `flash`, `flash_open` |
| `src/routes/preferences.rs` | `POST /preferences/theme` |
`wrap_page` links the four stylesheets in cascade order (tokens, product
tokens, base, components) with a `?v=<version>` suffix, because static
files are served with a flat one-day cache and no fingerprint. Add a new
static file in `src/static_files.rs`; there is no directory server.
### Rules the code follows
- **Colour comes from tokens.** A component never uses a raw colour. A
value the skill lacks goes into `csaf-tokens.css` with its measured
contrast ratio in a comment. Verify with
`python3 -I skills/ndaal-design/assets/check-contrast.py` and, for
tokens of ours, the same WCAG formula.
- **Nothing is inline.** The server sends a CSP without `unsafe-inline`
(see the Administrator Guide). `wrap_page` output must contain no
`<style>`, no `<script>` without `src`, no `style=` and no `on…=`
handler; a unit test fails otherwise. The confirmation uses a native
modal `<dialog>`, not `window.confirm`.
- **Status is never colour alone.** `ui::flash` and `ui::flash_open` add
an icon and a screen-reader word, and set `role="alert"` for an error
and `role="status"` for a result. A static fact on a page uses the
`Info` kind, which has no live-region role.
- **Identifiers are not broken or hyphenated.** `.mono` and `code` use
`hyphens: none`; tables that hold them sit in a focusable
`.table-scroll` region labelled by their heading.
- **Every table has a name** (`aria-labelledby` or `caption`), column
headers use `scope="col"` and row headers `scope="row"`.
- **Pass `lang` to anything that speaks.** Chrome strings use
`t(lang, "key")`; a new key goes into `src/i18n_data/en.tsv`, and the
other 48 languages fall back to English until translated. A guard test
walks every route module and fails on a key that `en.tsv` lacks.
### How the web interface is verified beyond unit tests
The UI shell and the response headers sit on request data (cookies,
`Referer`, a tracking id in the URL), so each of the four deeper layers
covers a part of it. They are not interchangeable.
| Layer | Where | What it pins |
| --- | --- | --- |
| Property (`proptest`) | `crates/csaf-crud/tests/proptest_ui.rs` | The theme cookie is found in whichever `cookie` field carries it; the theme is always one of three constants; a message box always has its role, icon and spoken word; a page title can never inject markup |
| Property, in module | `router.rs`, `routes/layout.rs`, `routes/lang.rs` | The header wrapper adds exactly what is missing and keeps the handler's own; the page-title transform is total, idempotent and only inserts; the redirect target is always a safe local path |
| Fuzz (libFuzzer) | `fuzz/fuzz_targets/fuzz_theme_cookie.rs`, `fuzz_return_path.rs`, `fuzz_page_shell.rs` over `fuzz-harness/src/{theme_cookie,return_path,page_shell}.rs` | Arbitrary cookie fields, `Referer` bytes and title/content text never panic and never break the invariants above, in all 49 languages |
| Concurrency (`loom`) | `loom-harness/src/lib.rs` | One settings clone per request is never torn; two clones in one request can straddle a write (a witnessed hazard) |
| Wall clock (`criterion`) | `criterion-harness/benches/ui_render.rs` | `wrap_page` across page sizes and languages, `theme_for` on three cookie jars, `html_escape`, `flash`, the header wrapper alone and through the router |
Run them:
```bash
cargo test -p csaf-crud --all-features --test proptest_ui
PROPTEST_CASES=4096 cargo test -p csaf-crud --all-features # deeper sweep
cd fuzz && cargo +nightly fuzz run fuzz_theme_cookie -- -max_total_time=60
cd loom-harness && RUSTFLAGS="--cfg loom" cargo test --release
cd criterion-harness && cargo bench --bench ui_render
```
These layers earned their place. Between them they found three defects that
example tests had passed:
- `theme_for` read only the first `cookie` header field, so a visitor's theme
was ignored whenever another cookie came first over HTTP/2 (a property
test; the code was unreleased).
- `wrap_page` put its title into `<title>` unescaped, and a tracking id from
the URL reaches that slot (a property test; present in released versions).
- `return_path`, which turns the `Referer` into a redirect target, accepted
`/\evil.example`: `http::Uri` allows the backslash and browsers read `/\`
as `//`, an open redirect (CWE-601) in every release since 1.4.15. The
fuzz target found it on its first seed. The first property test missed it
because uniformly random text almost never starts with `/\`; the property
now draws its inputs from the dangerous prefixes (`//`, `/\`, `\`) before
the random tail, and a shrunk seed is committed in
`crates/csaf-crud/proptest-regressions/`.
Each is pinned as an example test beside its fix. The page-title transform
has no untrusted input of its own (templates call it), but it slices by byte
offset, so the fuzz target feeds it arbitrary text in every language.
### Bruno and Playwright sequences for the web interface
Bruno runs the whole HTTP contract against a running server in folder order
(`seq` orders the files inside a folder); Playwright covers what needs a
real browser. Between them every behaviour above is pinned from outside.
| Folder or spec | Pins |
| --- | --- |
| `test/bruno/collections/headers/` (4) | The six security headers on a page, a static file, a `404` and a `405` |
| `test/bruno/collections/preferences/` (8) | The theme cookie renders `data-theme` (dark, system, hostile value); `POST /preferences/theme` sets the cookie, ignores an unknown value, cannot be used as an open redirect; the old global toggle is gone |
| `test/bruno/collections/ui/` (8) | The icon sprite is inert, the stylesheet and script make no remote request and use no raw colour or `eval`, pages carry no inline code, landmarks and dialog, info boxes announce nothing, dead Bootstrap files are gone |
| `test/bruno/collections/lang/` (10) | Adds the backslash-`Referer` open redirect (09) and a final reset to English (10) |
| `tests/e2e/ndaal-design.spec.ts` | Navigation, theme (including unrelated cookies, no-JavaScript form, open redirect, removed toggle), dialog, CSP, axe in both themes, reflow, 44 px targets, forced colours, assets and message boxes |
| `tests/e2e/language-switcher.spec.ts` | Adds the `Referer` open-redirect cases beside the existing switcher journeys |
```bash
cd test/bruno && bru run collections -r --env development \
--env-var base_url=https://127.0.0.1:8180 \
--env-var api_base=https://127.0.0.1:8180/api/v1 --insecure
BASE_URL=https://127.0.0.1:8180 PLAYWRIGHT_TLS_INSECURE=1 npx playwright test
```
Two Bruno behaviors shaped the sequences. The CLI keeps ONE cookie jar for
the whole run, so a folder that changes a cookie must put it back (the last
request in `lang/` and in `preferences/` does) or every later folder sees it.
And when the jar holds a cookie for the host, Bruno sends the jar INSTEAD of a
`Cookie` header written in the request; `preferences/01` therefore empties the
jar in a pre-request script before the requests that carry their own cookie.
Playwright has no shared jar: every test gets its own context.
Criterion numbers belong to one machine and one session: save a baseline
(`-- --save-baseline before`), change the code, compare (`-- --baseline
before`), and never compare across machines or gate a merge on them. The
cost of the header wrapper is read against the whole request
(`handle_parts_get_page`), not in isolation.
### Lint and test
```bash
npx stylelint "crates/csaf-crud/static/css/**/*.css" # strict config
npx oxlint crates/csaf-crud/static/js/csaf-ui.js
npm run lint:js # eslint
cargo test -p csaf-crud --all-features --lib
BASE_URL=https://127.0.0.1:8180 PLAYWRIGHT_TLS_INSECURE=1 \
npx playwright test tests/e2e/ndaal-design.spec.ts
```
`tests/e2e/ndaal-design.spec.ts` runs axe in both themes on six page
families, checks reflow at 320 px, 44 px targets, the desktop navigation
(the 1.5.23 / 1.5.24 defect), per-browser theme, menu keyboard handling,
the confirmation dialog and that no page raises a CSP violation.
To lint the rendered pages as well, fetch them from a running server and
run `htmlhint` and `markuplint` on the files. The repository's
`href-abs-or-rel` rule is configured as `true`, which reports every
absolute link; the external references on the Info pages are meant to be
absolute.
### Icons
The sprite holds only the icons the pages use. To add one, append its name
to `ICONS` in `scripts/build_icon_sprite.py`, verify the
`bootstrap-icons` tarball against the registry's `dist.integrity`, and run
`python3 -I scripts/build_icon_sprite.py TARBALL
crates/csaf-crud/static/img/icons.svg`. The script copies only `path`,
`circle`, `rect` and `g` elements with a fixed attribute set and aborts on
anything else. A test checks that every `icon("name")` in the route
sources exists in the sprite.
### Recorded divergences from the skill
| Skill | Here | Why |
| --- | --- | --- |
| JetBrains Mono | Roboto Mono | Project rule: Roboto and Roboto Mono, bundled |
| WOFF2 fonts | TTF | `tests/font_glyph_coverage.rs` parses TTF; WOFF2 is a size optimisation |
| Settings as a dialog | `/settings` stays a page | The page already exists and is linked from the header |
| `button` + `aria-expanded` menus | native `<details>` | Works without JavaScript; the script adds Escape and outside-click |
| 44 px checkbox | 24 px box in a 44 px label row | A 44 px native checkbox is a 44 px glyph |
| Theme in `localStorage` | Cookie `csaf_theme` | The server renders `data-theme`, so there is no flash of the wrong theme |
`Settings.theme` stays in the model as the default for browsers without
the cookie; the header's Theme menu replaced the old global toggle.
## Listen ports
**A listen port is accepted iff it is a canonical decimal number in
1025..=65535** (`skills/rust-ip-ports`). Anything that takes a port from
outside the binary goes through `ListenPort`, never a bare `u16`.
| Piece | Where |
| --- | --- |
| The type, the typed error, the rule | `crates/csaf-core/src/listen_port.rs`, a `#[doc(hidden)]` module |
| `listen_port`, `quic_port`, `QuicPort`, `AppConfig::quic_listen_port` | `crates/csaf-core/src/config.rs` |
| The typed refusal `CsafError::ListenPort`, `is_refused_port` | `crates/csaf-core/src/error.rs` |
| Exit status 64 and the help text | `crates/csaf-crud/src/main.rs` (`EXIT_REFUSED_PORT`) |
| The HTTP/3 row of the System Information page | `http3_row` in `crates/csaf-crud/src/routes/info.rs` |
It is a module, not a path dependency on the skill's `listen-port` crate,
because `csaf-core` is published on crates.io and a published crate cannot
depend on an unpublished one. The module is hidden and carries a `//`
comment, not `///`: a doc comment on the `mod` line merges with the file's
own docs and breaks its links under `RUSTDOCFLAGS="-D warnings"`.
How a port gets in, and where it is checked:
- `AppConfig::load` first reads the file as a TOML table and checks
`listen_port` and `quic_port`, so a refusal is a typed
`CsafError::ListenPort { key, source, or_off }` and not a text. Then the
full parse runs, where `ListenPort` deserialises through `try_from = "u16"`
(a plain derive would write 80 into the field). A document that is not
TOML, and a port of the wrong type, are left to that full parse and stay
ordinary `CsafError::Config` errors.
- `main` prints a refusal with `eprintln!` and exits 64, because the config
is loaded before tracing is installed. Every other error keeps exit
status 1. Keep it that way: a script tests for 64.
- The HTTP/3 port is `AppConfig::quic_listen_port()`, never
`listen_port + 1`. That sum in a `u16` overflowed at 65535 and the
release profile has `overflow-checks = true`, so the server aborted at
start. `checked_add` makes "no UDP port above 65535" an answer (`None`).
The derivation is the small `const fn derived_quic_port` so that a Kani
proof can cover every valid port (see the layers below).
Do not parse with `str::parse::<u16>()`: it accepts `+8080` and `08080`.
The method that carries the rule is named `try_new` and not `new`: the
skill measured that cargo-mutants 27.1.0 generates no mutants at all for a
function named `new`, which would leave the one comparison that carries the
rule outside the mutation test.
The layers, and the command for each:
| Layer | File | Run |
| --- | --- | --- |
| Rule, one test per boundary | `crates/csaf-core/tests/listen_port_rules.rs` | `cargo test -p csaf-core --test listen_port_rules` |
| Every `u16`, every decimal 0..=99999 | `listen_port_exhaustive.rs` | `--test listen_port_exhaustive` |
| Property | `listen_port_proptest.rs` and its `.proptest-regressions` pins | `--test listen_port_proptest` |
| serde | `listen_port_serde.rs` | `--test listen_port_serde` |
| Compile-time invariants | `listen_port_static_assertions.rs` | `--test listen_port_static_assertions` |
| Config, typed errors, the derived HTTP/3 port, writing a config back | `config_listen_port.rs` | `--test config_listen_port` |
| Proof for every value (Kani) | `listen_port.rs`: `port_rules_follow_the_contract`; `config.rs`: `the_derived_http3_port_follows_by_one_and_never_overflows`, `a_toml_integer_is_accepted_iff_it_is_inside_the_rule` | `cargo kani -p csaf-core --lib` |
| Wall-clock cost | `criterion-harness/benches/listen_port.rs` | `cd criterion-harness && cargo bench --bench listen_port` |
| Mutation | `listen_port.rs`, and the port functions of `config.rs` | `PROPTEST_CASES=64 cargo mutants -p csaf-core --file crates/csaf-core/src/listen_port.rs --file crates/csaf-core/src/config.rs --re 'listen_port\|ListenPort\|QuicPort\|derived_quic_port\|check_ports\|check_listen_port\|check_quic_port\|port_of_integer\|is_derived\|is_canonical_decimal\|listen_address' -- --test listen_port_rules --test listen_port_exhaustive --test listen_port_proptest --test listen_port_serde --test config_listen_port` |
| Exit status of the real binary | `crates/csaf-crud/tests/test_listen_port_exit.rs` | `cargo test -p csaf-crud --test test_listen_port_exit` |
| Live scan of the running process (nmap, rustscan, lsof) | `tests/scripts/test_listen_ports_live.sh`, its `.bats` suite and `.README.md` | `tests/scripts/test_listen_ports_live.sh` (gate `listen_ports_live`; LIVE, so `--fast` skips it) |
| System Information rows | `http3_row` unit tests; `test/bruno/collections/docs/07-system-information.bru`; `tests/e2e/system-information.spec.ts` | the Bruno and Playwright commands above |
| Fuzz | `fuzz-harness/src/listen_port.rs`, wrappers `fuzz_listen_port`, `hfuzz_listen_port`, `afl_listen_port` | `cargo +nightly fuzz run fuzz_listen_port -- -max_total_time=60` |
The Kani proofs state the claim for every value, not for samples. They
verify in seconds. The derivation proof is worth its place: with the
original `listen.get() + 1` put back, it fails with "attempt to add with
overflow", so it would have stopped the 65535 crash. Kani manages its own
nightly and ignores `rust-toolchain.toml`.
The benchmark measures what the rule costs at the granularity a caller uses
it: parsing a port text (tens of nanoseconds), the `u16` path, the
`quic_port` forms, and `AppConfig::load`, the whole start-up path (about a
millisecond, nearly all of it the file read and the TOML parse). Measured
on a busy machine (load average 168–315), so read them as orders of
magnitude; compare two versions with `--save-baseline` and `--baseline` on
a quiet one, and never gate a merge on them.
The live test looks at the real binary from the outside, on `127.0.0.1`
only. It starts it about twenty times from a config file in a private
directory, with a clean environment. Thirteen configs are refused:
`listen_port` 0, 22, 80, 443, 1024, 65536, -1 and a text, and `quic_port` 80,
1024, 65536, -1 and a text. Seven are accepted: HTTP/3 derived at 1025, 8080,
65534 and 65535, `quic_port = "off"`, an explicit `quic_port`, and
`quic_port = 65535` beside TCP 65535. For a refusal it checks the exit status
(64, or 1 for a text where a number belongs), that the message names the key
and the rule, that the app never printed a listening line, that it created
nothing, and that nmap and rustscan see the same port state before and after.
For an accepted start it checks `GET /api/v1`, nmap `-sV` and rustscan, `lsof`
for the sockets the process owns (one TCP listener, and exactly the expected
UDP socket: `listen_port` + 1, none at 65535, none with `"off"`, the chosen
one when set), that nothing the app owns is open in 1-1024, and that the ports
are closed again after SIGTERM.
Three limits, all measured. nmap cannot scan UDP without root, so HTTP/3 is
checked with `lsof` only. The test sends no `--tlsv1.3`: the `curl` that macOS
ships (SecureTransport) cannot negotiate TLS 1.3 and fails with error 4 before
any request, which is how the first run reported 14 FAIL for a healthy app;
the TLS version belongs to `testssl_endpoints`. The app installs no SIGTERM
handler, so it ends with status 143, which the test accepts as well as 0. The
run after that fix: 163 PASS, 0 FAIL, 4 SKIP (the four samples that are not
scannable port numbers). The bats suite beside the script (60 tests, 58 run
and 2 skip by design) runs it against a fake app and proves that three bugs
are reported as a FAIL: serving on 1024, an unexpected UDP socket, and a
refusal that leaves state behind. The
evidence (every scanner output, `lsof` line and verdict) goes to
`documentation/ip-ports/<UTC timestamp>/`, which is git-ignored.
Mutation testing found real gaps in the first draft of the tests, which is
what it is for. Of 45 mutants of the port code, 11 are unviable (the type has
no `Default`, so "return the default" does not compile), 28 were caught at
first and 6 were missed:
- `QuicPort::is_derived` returning `true` or `false` (2): nothing wrote a
config back, so nothing noticed that an unset `quic_port` would be saved as
`"off"`. Closed by the serialisation tests in `config_listen_port.rs`.
- Two comparisons in `port_of_integer` (2): the tests checked the key of a
refusal for `-1`, never which refusal it was. Closed by tests that assert a
minus sign is "not a decimal" and a huge number is "above 65535".
- `< 0` to `<= 0` in the same guard (1): equivalent, because the guard is only
reached when `u16::try_from` fails and 0 never fails it. Checked over
140,010 values with no difference, and excluded in `.cargo/mutants.toml`
with that reasoning.
- The body of a Kani proof replaced with `()` (1): the proofs exist only
under `cfg(kani)`, so no test build contains them. Excluded for these two
files only; the proofs are checked by showing they fail on the original bug.
A second run of just the affected mutants confirmed the four real gaps are
closed. That leaves 32 viable mutants and 32 caught. Tests were only added
between the runs, so what the first run caught stays caught. The run uses
`PROPTEST_CASES=64` to keep each mutant's test pass short, and one baseline
build alone took 16 to 36 minutes on a busy machine, so allow hours.
The process test starts the real binary. For the 65535 case it waits for
the server's own `HTTP/3 is off` line with a two-minute deadline and does
not sleep for a fixed time: a cold debug start takes about five seconds to
its first log line, and a fixed grace period killed it first.
The fuzz body asserts more than "does not panic": an accepted text is
exactly the canonical decimal of its value, and `QuicPort::from_text`
agrees with `ListenPort::parse` for every text except `off` (any case) and
`0`.
`fuzz/Cargo.toml` declares its own `[workspace]`, as every other standalone
harness does. Without it `cargo fuzz` fails in a git worktree that sits
inside another checkout, because Cargo walks up to the outer manifest.
## Shell scripts
Every `.sh` follows the ndaal canonical form and must pass three gates:
```bash
bash -n script.sh
shfmt -d -i 0 -bn -ci -sr script.sh
shellcheck -o all script.sh
```
Each script has a sibling `*.bats`. Two rules learned the hard way and now
asserted there:
- **Bound anything that shells out.** `test_cargo_inspect.sh` once wedged for
17.5 hours holding the cargo build lock, blocking every other cargo command
in the workspace. Unbounded analysis turns one tool hanging into a silent
stall of the whole pipeline.
- **`grep -c` prints `0` *and* exits 1** on no match, so
`$(grep -c … || printf '0')` yields the two-line string `"0\n0"` and every
downstream numeric comparison breaks. Use `|| true`.
## Hardening
Release builds layer allocator, arithmetic, linker and reproducibility
mitigations — see [Binary hardening](wiki/en/concepts/binary-hardening.md)
for the full picture and the verification commands.
The one thing to internalise: **Cargo does not merge `rustflags`.** A
`RUSTFLAGS` environment variable silently replaces everything in
`.cargo/config.toml`, so a release job that sets it for
`--remap-path-prefix` must repeat the linker flags in the same variable or
the shipped binary quietly loses them.
## Dependency refresh
Releases 1.5.26 and 1.5.27 were dependency refreshes: `cargo update` of
`Cargo.lock`, then the gates. 1.5.26 moved twelve direct dependencies whose
caret requirements in `Cargo.toml` already allowed the new versions; 1.5.27
moved 55 packages and raised the `self_update` requirement to match. What a
refresh leaves behind:
- **A changelog entry that says what was checked and what was not.** Each
advisory is recorded with how its reachability was established: read from
the sources and the cargo feature set, or reproduced against a running
server. The 1.5.26 and 1.5.27 entries both say "not reproduced". Keep that
distinction; do not write "not affected" for something you only read.
- **One advisory per bumped crate that has no published flaw**, scored CVSS
0.0 (1.5.27 names `ndaal-sa-2026-867` to `-909`), so the release record
lists every crate that moved.
- **A VEX statement for advisories that do not reach csaf-crud**: 1.5.27 uses
`ndaal-sa-2026-887` for four `rustls` advisories, 1.5.26 uses
`ndaal-sa-2026-761` for four `quinn-proto` ones.
- **Test counts** in `README.md` and its four translations, which state the
numbers (1.5.27: 314 unit tests, 634 in total), and the embedded copy of
the README.
- **The embedded `CHANGELOG.md`**, re-synced; see
[Documentation](#documentation).
The 1.5.27 entry also says which bumped crates are not in the shipped build:
`zip` 9 and `zipsign-api` 0.2.2, because no `archive-zip` or `signatures`
feature is enabled.
If the build host cannot run newly built programs, see
[Newly built programs hang on macOS](#newly-built-programs-hang-on-macos).
## Release pipeline
`release/create_release.sh` builds both binaries for the six release targets
and packages them under `release/v<version>/`. It takes no options and reads
the version from `cargo metadata`. This section only orients you; the
script's contract and the reason for each build route are in
[create_release.README.md](../release/create_release.README.md).
The build route per target is fixed, not a preference. Linux and
`x86_64-pc-windows-gnu` build with `cargo zigbuild`. Both Apple targets build
with the native Apple toolchain, because zig's linker mis-parses
`-dead_strip` when it links `redb`. `aarch64-pc-windows-msvc` builds with
`cargo xwin`, compiling with Apple clang and linking with Homebrew LLVM's
`lld-link`.
For every archive and every `.deb` and `.rpm` package the script writes the
five hash sidecars. It then writes the five aggregate manifests and
`SHA256SUMS`, and verifies them before it finishes.
Before building, the script exits with status 1 if `cargo-zigbuild`, `zig`,
`cargo-xwin`, `cargo-deb`, `cargo-generate-rpm`, `shasum`, `sha3sum`, `b3sum`
or `python3` is missing. Homebrew LLVM and Apple clang are checked only when
the last build stage, `aarch64-pc-windows-msvc`, starts, so confirm them
before you begin. Only the Podman smoke test of the packages is optional: it
is skipped when `podman` is missing or its machine is unreachable.
Plan for hours, not minutes. After a compiler change every target recompiles
from cold, and the 1.5.23 release took roughly three hours of wall time on a
loaded machine. That is an observation, not a guarantee.
`release/create_release.bats` pins that the `aarch64-pc-windows-msvc`
`csaf-crud` build does not pass `--no-default-features`. That flag drops the
default `quic` feature and with it HTTP/3, and it did so silently up to
1.5.23.
### Publishing a release
The script builds and packages; it does not upload. It leaves every sidecar
under `release/v<version>/`, but do **not** attach the per-archive sidecars
(`<archive>.tar.gz.sha-256` and the other four families) to the GitLab
release. An archive's name is a prefix of its sidecars' names, and the
`self_update` crate takes the first asset whose name contains the target
triple and the binary name. With the sidecars attached, `--self-update`
downloaded a sidecar instead of the archive and failed with a checksum
mismatch.
Attach everything else: the archives, the `.deb` and `.rpm` packages with
their sidecars, the five aggregate manifests and `SHA256SUMS`. That is 66
assets for 1.5.23. The unit test
`archive_name_is_a_prefix_of_every_sidecar_so_sidecars_must_not_be_release_assets`
in `crates/csaf-core/src/updater.rs` pins the hazard.
## Documentation
Any change to a document under `documentation/` that is also embedded in the
binary must be mirrored into `crates/csaf-crud/embedded_docs/`. `info.rs`
serves those copies via `include_str!`, and the copies are kept in step by
hand. `crates/csaf-crud/tests/embedded_docs_in_sync.rs` makes drift a test
failure — without it, a stale copy compiles, passes every test, and serves
the previous release's changelog.
### The wiki and its bundles
`documentation/wiki/` holds the codebase wiki in five languages: `en/` is
canonical and authored first, `de/`, `fr/`, `es/` and `it/` mirror it page for
page under the same file names. A translated page changes the prose and the
`title` value of the frontmatter, and nothing else: the other frontmatter
values, the heading structure, every code span and every `[[link]]` stay as
in English. Each language also has a bundle, `CodeWiki_<lang>.docx`, `.odt`
and `.pdf`, for offline reading. No script in the repository builds them. The
recipe: join `overview.md`, `codemap.md`, `glossary.md`, `index.md`, then
`concepts/`, `entities/` and `patterns/` in file-name order (not `log.md`),
drop each page's frontmatter, and start with one `# ndaal_public_csaf_crud --
CodeWiki (<Language>)` heading, because `markdown2pdf.sh` takes the PDF title
from the first H1. Then `pandoc <bundle>.md -f gfm+smart -t docx` (and
`-t odt`) and `bash markdown2pdf.sh -f 3u -l <lang> -o CodeWiki_<lang>.pdf
<bundle>.md`, which validates the PDF as PDF/A-3U with veraPDF.
Pandoc cannot be installed with Homebrew on an Intel Mac (no bottles, and the
source build fails). The renders of 2026-10-21 used the release binary
`pandoc-3.12.1-x86_64-macOS.zip` from github.com/jgm/pandoc, checked against
the SHA-256 that GitHub publishes for it.
## Hung GitLab push on macOS
On macOS, `git push` to GitLab can hang if `git-credential-osxkeychain` is
blocked waiting for a keychain approval. `git ls-remote` still works because
the project is readable anonymously, so a push that hangs with no hook output
is a credential-helper problem, not a network one. Answer the pending
keychain approval and push again.
## Newly built programs hang on macOS
On 2026-10-19 every cargo command that had to run a freshly compiled
executable hung on the macOS build host: build scripts, tests and clippy.
`ps` showed `build-script-build` in state `S` at 0% CPU for minutes to hours,
while `XprotectService` used about 375% CPU and `syspolicyd` about 65%. A
hello-world program built with `rustc` did not start within 20 seconds. The
macOS malware scan of new executables is the likely cause; that is inferred
from the process list, not proven. The same check passed on 2026-10-20, so
run it before queuing a long gate:
```bash
printf 'fn main(){println!("hello");}\n' > hello.rs
rustc hello.rs && ./hello
```
If it does not print within 20 seconds, do not start the gates on the host.
Commands that run no new executable still work: `cargo update`,
`cargo metadata`, `cargo tree`, `cargo audit`, `cargo deny check`,
`cargo fmt --check` and `osv-scanner`.
To run the gates anyway, use a Linux container (Podman, `rust:1.99.0`,
`linux/amd64`). The 1.5.27 gates ran this way: `cargo clippy --workspace
--all-targets --all-features -- -D warnings` and 634 tests passed, and the
doctest pass found none.
- Mount `git archive HEAD` plus any changed `Cargo.toml` and `Cargo.lock`
into a volume, and keep `CARGO_HOME` and `CARGO_TARGET_DIR` in a second
volume.
- Set `RUSTUP_TOOLCHAIN=1.99.0-x86_64-unknown-linux-gnu`. Without it,
`rust-toolchain.toml` makes rustup install the six cross targets.
- Pass `--locked`, and set `CARGO_PROFILE_DEV_DEBUG=0` and
`CARGO_PROFILE_TEST_DEBUG=0` to save disk.
- The image needs `rustup component add clippy`.
- Debian's gcc 14 segfaulted at random while compiling SQLite's `sqlite3.c`
(two of three runs). Retry the gate when the log says `internal compiler
error`, and never retry on a real compile or test error.
- Delete the volumes afterwards.
The container does not compile the macOS-only (`objc2`) or Windows-only
(`windows-sys`, `windows-targets`) crates. Build those platforms before
tagging. In the changelog and in any release record, say which gates ran and
where; never write "tests passed" for a run that could not happen.
Do not stop `XprotectService` or `syspolicyd`. They are macOS security
services. Stop only your own hung `cargo` and `build-script-build` processes.
A reboot is the other way out.
## Contribution checklist
- [ ] Test written **first**, and watched to fail for the expected reason
- [ ] `cargo fmt --all` and `clippy -D warnings` clean
- [ ] `cargo test --workspace --all-features` plus the `--doc` pass
- [ ] New untrusted-input surface has a fuzz target
- [ ] New shared mutable state has a Loom model
- [ ] Shell scripts pass all three gates and have a sibling `.bats`
- [ ] SPDX header on every new file that supports comments
- [ ] `CHANGELOG.md` updated, and `embedded_docs/` re-synced if touched
- [ ] Five checksum sidecars regenerated for any changed artefact