# nixos-router — Specs

Working name: `nixos-router`. A NixOS-based distribution for routers/firewalls, delivered as a flake exporting NixOS modules and bootable images (not a nixpkgs fork).

This file is the one place for what the product is: decisions, how each part works (sections marked *confirmed* are built and tested), and, at the end, what is not done yet.

## Decisions

| Area | Decision |
|---|---|
| Audience | Own homelab first; keep structure clean enough to publish later. |
| Config model | Opinionated `router.*` option schema compiled to systemd-networkd + nftables, with build-time assertions. |
| Hardware | x86_64 UEFI. aarch64 stays possible (the flake exposes both systems; avoid x86-only assumptions); a NanoPi R2S SD card image was built on 2026-11-01 and dropped. |
| v1 features | IPv6 + prefix delegation, VLANs / multiple LANs, WireGuard. |
| Deferred | Multi-WAN failover, QoS (CAKE/SQM). (The web UI, an MCP server and a REST API exist.) |
| Updates (v1) | Push from workstation (`nixos-rebuild --target-host` / colmena) + commit-confirm rollback timer. |
| Updates (later) | **A/B image updates** with automatic rollback. |
| Footprint | Minimal closure, read-only root, no Nix daemon/compiler on device, small persistent `/var`. |

## Architecture

1. **`router.*` schema** — users describe intent (wan, lan, vlans, zones, port forwards, wireguard); modules compile it to:
   - systemd-networkd (DHCPv4/v6, DHCPv6-PD, VLANs, bridges, PPPoE later)
   - nftables zone-based ruleset (generated, not `networking.firewall`)
   - DHCP / DNS / RA services
2. **Build-time validation** — Nix assertions reject lock-out or inconsistent configs (no management access rule, overlapping subnets, zone referencing missing interface); generated ruleset checked with `nft -c`.
3. **Images** — x86_64: installer ISO (`router-setup`) and installed NixOS; the read-only A/B image module exists but is not the product (see below).
4. **Deployment** — deploy a prebuilt closure (`nix copy` + `switch-to-configuration`); never evaluate or build on the router. This keeps the device light and leads naturally to A/B updates.

## Implications

- No on-device evaluation: all designs must work from a prebuilt closure.
- Flake exposes `x86_64-linux` and `aarch64-linux` outputs from the start; hardware specifics (bootloader, kernel, NIC naming) live in separate `boards/` modules.
- IPv6 PD fan-out to multiple VLANs needs stable per-LAN subnet-ID allocation; this is the hardest part of the schema.

## Default zone policy (confirmed)

- WAN: deny inbound (except established/related, required ICMPv6, explicit port forwards).
- LAN: allow outbound to WAN.
- Other zones (e.g. IoT): isolated from each other and from LAN unless a rule opens them.

## Management access (confirmed)

- Key-only SSH, reachable from LAN zones (trusted, non-WAN, non-isolated zones). No GUI in v1.
- Build-time assertion fails if no management path exists.
- Isolated zones (e.g. IoT) get no router access beyond DHCP/DNS/NDP.

## Services (confirmed)

- DHCP: Kea (DHCPv4 + DHCPv6 incl. PD).
- DNS: Unbound.
- RA: systemd-networkd (or radvd if networkd falls short).

## Repo layout (confirmed)

```
flake.nix
modules/     # router.* schema and service modules
tests/       # nixosTest VM tests
images/      # image/ISO builds
examples/    # sample configs
```

## Status

**WireGuard done**: `wireguard` VM test (5 VMs: ISP/provider, router, laptop, remote site, LAN host) covers handshakes, road-warrior DNS/SSH over IPv4 and IPv6, zone rules between tunnels and LAN, site-to-site routing without NAT or transfer net, provider tunnel with masquerade and IPv6 transfer net, and the kill switch (kill switch and masquerade mutation-tested). Not covered: preshared keys, `mtu`, tunnels with several peers, key rotation.

**VLANs + zones done**: `vlans` VM test (5 VMs: ISP, router, untagged `main`, VLAN 20 `iot`, VLAN 30 `cams`) covers VLAN devices, per-VLAN DHCP/gateway, opt-in internet, DNS vs SSH per LAN, intra-zone forwarding across VLANs, one-directional inter-zone rules and the destination restriction (each mutation-tested). Not covered yet: IPv6 on VLAN LANs (only the PD LAN has IPv6), port forwards from the WAN.

**PPPoE done**: `pppoe` VM test = the same topology with a fake ISP running `pppoe-server` (VLAN 7, CHAP, password containing a quote) — covers session, NAT, MTU, MSS clamp (mutation-tested) and WAN-deny. `basic` remains the DHCPv4 variant.

**IPv6 + prefix delegation done (single /64)**: `router.wan.ipv6.enable` and `router.lans.<name>.ipv6.pd` (at most one LAN, asserted). The VM test now covers the router's DHCPv6 address and delegated /64, SLAAC on the client, routing without NAT66, and WAN-deny over IPv6 (mutation-tested). Still open: RDNSS, other LANs are IPv4-only, DHCPv6 reservations.

**Phase 1 (MVP) skeleton done**: flake, `router.*` schema (wan, lans, dns.forwarders), networkd, nftables (zone-less for now: WAN/LAN), Kea DHCPv4, Unbound, assertions. `nix build .#checks.x86_64-linux.basic` runs a 3-node VM test (upstream/router/client) covering WAN DHCP, LAN DHCP+DNS, NAT, WAN-deny and LAN SSH.

Interactive lab: `nix run .#lab` (x86_64) opens tmux with one console pane per VM plus the test-driver REPL (`router.shell_interact()`, `client.succeed(...)`, `test_script()`). The REPL pane also shows all VM output interleaved.

Learned along the way:
- The default policy now explicitly answers ICMP echo on LANs only; the WAN gets only ICMP error types and ND.
- In NixOS tests `fail(...)` also passes on "command not found": install probe tools explicitly and add a positive control per negative check.
- Unbound answers `.test` locally (RFC 6761) and validates DNSSEC; tests using fake domains need both `local-zone ... transparent` and `domain-insecure` (both verified necessary; set in the test, not the module).
- `modules/router/lib.nix` has IPv4 helpers (parse/format/netmask/cidr/overlap). `ipv4.subnet` is now derived from `address` + `prefixLength` (read-only). Assertions also cover pool bounds, pool containing the router address, and overlapping LANs. IPv6 helpers (prefix arithmetic for PD subnet-IDs) are still to do.
- Checks: `basic`, `pppoe`, `onestick`, `rollback`, `dns`, `vlans` and `wireguard` (VM tests), `example` (the example system builds), `lib` (helper unit tests), `assertions` (broken configs must fail with exact messages). Not yet covered: NPTv6.

## IPv6 design (confirmed)

- **WAN**: mirrors the existing router's uplink config — networkd `DHCP` (v6), `IPv6AcceptRA = true`, `KeepConfiguration = yes`, `dhcpV6Config = { WithoutRA = "solicit"; UseDelegatedPrefix = true; }`. The ISP sends RAs (default route) without DHCPv6 flags, so the router solicits on its own.
- **Delegation**: the ISP delegates a single **/64**, which cannot be split. Exactly one LAN may claim it via an explicit flag (`router.lans.<name>.ipv6.pd = true`); an assertion rejects more than one.
- **LANs**: SLAAC only (RA, no DHCPv6). All other LANs/VLANs are **IPv4-only** for now (ULA or NPTv6 can come later).
- **Firewall**: inet table covers v6; WAN→LAN forwarding stays denied, no NAT66.
- **Test**: fake ISP on `upstream` = Kea DHCPv6 (IA_NA + IA_PD /64 from a /48 pool) + networkd RA sender (default route only) + a static route for the delegated prefix back to the router.
- **Known gap**: no RDNSS in RAs yet (the router's global LAN address is not known at eval time); clients still get DNS via DHCPv4.

## PPPoE uplink (confirmed)

- **IPv4 via PPPoE** (`router.wan.ipv4.type = "pppoe"`), carried on an optional tagged **VLAN** on the WAN link (`pppoe.vlan`); native DHCPv6/PD stays on the untagged link.
- `wan.interface` is the physical link; the derived `wan.ipv4.interface` (`pppoe-wan` or the link) is where NAT, MSS clamping and the IPv4 forwarding rule apply. IPv6 forwarding keeps using the physical link.
- pppd (`pppoe.so`) runs as `router-pppoe.service`, ordered after the VLAN device. Credentials: `username` inline, password from `pppoe.passwordFile` (type forbids Nix-store paths); the options file is generated into a root-only runtime dir. **Deployment must provide that file** on persistent storage (or via agenix/sops).
- **MTU 1492 + TCP MSS clamp** (1452) in both directions on the PPP device; RFC 4638 not implemented.

## VLANs and zones (confirmed)

- **VLAN devices** are created by the router: `router.lans.<name> = { interface = <trunk>; vlan = <id>; }` makes `<trunk>.<id>` via networkd. A trunk may also carry one untagged LAN. Assertions: duplicate device, device name > 15 chars, WAN interface reused as trunk.
- **Zones** are an attribute on the LAN (`zone`, default = the LAN's own name); LANs sharing a zone may forward to each other, traffic **between zones is denied** unless a rule in `router.firewall.rules` allows it (`from`, `to`, `protocol`, `ports`, optional IPv4 `destination` that must lie inside a LAN of the destination zone). Rules are one-directional; return traffic via conntrack. Assertions reject unknown zones, from == to, ports without tcp/udp, bad/out-of-zone/host-bit destinations.
- **Internet is opt-in per LAN** (`internet`, default false) — this replaces the earlier "LAN allow-out" default; any LAN that should reach the WAN must set it.
- Router services stay per LAN: DNS and DHCP always, SSH only with `allowManagement`.
- Kea subnet IDs are derived from the network address, so adding or removing LANs does not shift IDs (and orphan persistent leases).

## MCP server (confirmed)

- **Endpoint**: `POST /mcp` on the web UI (MCP Streamable HTTP, JSON-RPC 2.0, single request/response; no streaming). It is served on the same LANs and tunnels as the UI, with the UI's certificate (self-signed unless ACME is set up: the client must trust it).
- **Tokens** (System → API access): `rtr_…`, shown once, stored as SHA-256 in `/var/lib/router/webui/tokens.json`. Scopes `read` and `change`; an `apply` scope is reserved. A session cookie is not a token. A wrong token costs a second; a foreign `Origin` is refused (DNS rebinding).
- **Tools**: read — `get_status`, `get_settings` (with an etag), `get_pending_changes`, `list_interfaces`, `list_dhcp_leases`, `get_update_status`, `get_apply_log`, `check_settings`; change — `update_settings` (JSON merge patch, refused when the etag is stale), `discard_changes`. Resources: `router://settings`, and `router://options` (every router option with type, default and description, from `nixosOptionsDoc`, so an assistant knows what settings exist).
- **Editing is safe by construction**: edits only write `settings.json`, like the Advanced page. They stay *unapplied*; a person reviews them (Changes page, a diff against the confirmed settings) and applies them in the web UI, with commit-confirm. Not editable: `system.sshKeys`, `system.stateVersion`, the schema `version`, any other `system` key besides hostName and timeZone; every `…File` path must be under the state directory (so a patch cannot point the PPPoE password file at `/etc/shadow`).
- **`check_settings`** runs `router-check.service` (as root, outside the web UI's sandbox): `nix eval` of the system's `drvPath`, which runs the option types and all assertions without building. The verdict is in `/var/lib/router/check/result.json` with the etag of the settings it evaluated.
- **Audit**: every tool call (token name, tool, arguments, error) is appended to `/var/lib/router/webui/audit.log` (rotated at 1 MB) and shown on the API access page.
- **`get_logs`** (read): `journalctl` with arguments built in `logs.go` (no shell): `unit` (required: `kernel` or one of `router-*`, `kea-dhcp4-server`, `unbound`, `systemd-networkd`, `systemd-resolved`, `systemd-timesyncd`, `nftables`, `ddclient`, `acme-*`, `sshd`; not the journal as a whole), `lines` (≤ 1000), `since`/`until`, `priority`, `grep`, `boot`. The output is capped at 256 KB.
- **Token expiry**: a token is made for 1, 7, 30 (default) or 90 days, or without limit; an expired token is refused like an unknown one. The API access page shows when each ends.
- **Edits through the API are flagged**: after `update_settings` the overview's pending banner and the Changes page say which token made the changes, as long as the saved settings are still that edit (the note holds the settings' etag, so a later edit in the web UI or an apply makes it stale).
- **options.json** (`router://options`) has no `declarations` (the source's store path): the file is the same for every build of a release, so a router that rebuilds itself offline already has it.
- **Not done**: applying or confirming (needs the `apply` scope), metric history.

## REST API (confirmed)

- `/api/v1/…` with an OpenAPI 3.1 description at `GET /api/openapi.json` (no token: it describes the interface only). Same tokens, scopes, origin check, limits and audit log as MCP.
- **One implementation**: each operation runs an MCP tool (`apiRoutes` in `api.go` maps method and path to a tool; query parameters or `If-Match` and the body become its arguments), and `TestOpenAPI` checks that every tool has an operation. Status, settings (`ETag` header), `PATCH /settings` (merge patch, `If-Match`; 409 when stale), `settings/check`, `changes` (GET the diff as text, DELETE to discard), `interfaces`, `dhcp-leases`, `update`, `apply-log`, `logs`, `options`.
- **Settings areas** (`/api/v1/settings/router/firewall`, `.../router/lans/<name>`, `.../system`, down to a single option): `GET` reads, `PUT` replaces, `PATCH` merges (objects), `DELETE` removes, with the settings' `ETag`/`If-Match` and the same limits as the whole-document edit (`editSettings`). Values are checked against a JSON Schema made from the option documentation (`schema.go` reads the types nixosOptionsDoc describes in words, e.g. "null or integer between 1 and 4094"; a type it does not know becomes "anything", so it never refuses a valid setting): a wrong type, a value out of range or a key that is not an option is refused with its place. The OpenAPI document has these paths (three levels deep) with their schemas; lists are replaced whole (elements have no paths). Tested with the real option file (`ROUTER_OPTIONS_JSON`, `LIVE_SETTINGS_JSON` for a router's own settings) and in `apply`.
- **Typed responses**: every tool returns a Go struct (`apitypes.go`); `typeschema.go` derives the OpenAPI response schemas from those types by reflection (`json` tags, `doc` and `enum` tags), so the document cannot drift from what is sent. `TestResponsesMatchTheirSchemas` checks real responses against the generated schemas. The settings in answers use the typed settings schema when the option documentation is present. Hand-written reflection, not a library: the web UI is stdlib-only (`vendorHash = null`).
- **API reference** page in the web UI (`/apireference`, linked from API access): Redoc (pinned `redoc@2.5.2` bundle, fetched by `pkgs/router-webui/package.nix`, served from the UI's own address) rendering the generated `/api/openapi.json`. Only that page gets a nonce for Redoc's styles and `worker-src blob:` in its CSP. The spec's own text explains authentication, saved-not-applied, ETags/`If-Match`/409 and errors; every operation has a curl sample. Established tooling instead of a hand-built page (swaggo rejected: spec is generated at runtime from the MCP tools and the option docs). No "try it" form.
- Errors are `{"error": "…"}` with 400 (refused or bad argument), 401, 403 (scope), 409 (stale ETag), 500.

## Interface names (confirmed)

- **Cards**: `router.interfaces.<name>.mac` names the card with that (permanent) MAC address `<name>` from the next boot on, via a systemd `.link` file; `wan.interface` and `lans.*.interface` use the new names, so the web UI, nftables, Kea and the logs show `wan`, `lan`, not `enp2s0`.
- **VLAN devices**: `lans.<l>.deviceName` (and `wan.ipv4.pppoe.vlanName`) name the VLAN device (`dmz` instead of `lan.1002`); the default stays `<interface>.<vlan>`.
- **Only after a reboot**: a device that is up cannot be renamed and the configuration refers to the new name, so `router-apply` compares `/etc/router/interface-names` (cards and VLAN devices with their names) between the running and the new system. If a card gets, loses or changes its name, or an existing VLAN device changes its name, it only builds, sets the system profile and `switch-to-configuration boot` (no commit-confirm, no live switch); the web UI shows "starts at the next reboot". New VLAN devices (also named ones) are created live as before.
- **Names**: lower case letters, digits and dashes, start with a letter, at most 15 characters, and not what the kernel or udev hand out (`eth0`, `wlan0`, `enp2s0`, `ppp0`, ...), checked by an assertion and in the web UI (`chosenName`).
- **Risk**: a wrong MAC leaves the new system without its LAN after the reboot and there is no automatic rollback (the previous generation is in the boot menu). The web UI only offers cards that are present, with their MAC read from the machine.

## Port forwards and NAT (confirmed)

- Filtering and translation are separate. `router.firewall.rules` only filter and may use the reserved source zone `wan` (no LAN/tunnel may use that zone name; `wan` is never a destination). Rule fields: port ranges (`"9000-9002"`), `sources` allow-list (v4/v6 CIDRs), `rateLimit` (`N/second|minute|hour`, per source via nft meter), `log`, `natOnly` (only connections a DNAT rule translated).
- `router.firewall.nat` is the translation table: `type` `dnat` (IPv4 only, one host `toAddress` in the `to` zone, tcp/udp, optional `toPort`, `hairpin` from `wan`: NAT loopback for LANs/tunnels with `internet = true`; masquerades so replies return via the router), `snat` (`toAddress`) and `masquerade`, matching `from`, `sources`, `destination` (default: the router's own addresses), protocol and ports.
- The web UI's NAT table can also add the matching filter rule ("also allow", a `natOnly` rule). Schema 1 (`migrations/1.jq`) converts the old rules with `dnat` into a NAT row plus such a rule.
- Safeguards: a rule from `wan` must name a destination, an explicit protocol and its ports, so a typo cannot expose a whole zone.
- IPv6 has no NAT: it is opened only by explicit pinholes (`from = "wan"`, IPv6 `destination`). The prefix is dynamic, so v6 destinations are not checked against the zone.
- Tested in `basic`/`pppoe` (source preserved, toPort, ranges, sources, rate limit, v6 pinhole, hairpin, log lines) and in `assertions`; mutation-tested (source match, hairpin masquerade, rate limit).

## DNS accounts, dynamic DNS and certificates

- `router.dnsProviders` holds accounts at DNS providers by name (`type`, `username`, `server`, `secretFile`); the token is a private file (web UI: `/var/lib/router/dns-secrets/NAME`), never in settings or the store. `router.ddns.provider` and `router.acme.provider` refer to one. Schema 2 (`migrations/2.jq`) moves the old `ddns.protocol/server/username/passwordFile` into an account named after the protocol.
- `router.acme`: a Let's Encrypt certificate for the web UI through `security.acme` (lego), DNS-01 only (no open port, wildcards work), with providers that have one token: cloudflare, desec, digitalocean, duckdns, hetzner. Staging by default; `acceptTerms` must be set. The web UI serves `/var/lib/acme/<first name>/{fullchain,key}.pem` when present (reloaded when the files change, so renewals need no restart) and its self-signed certificate otherwise.
- Tested: assertions, the web UI forms (Go and VM), the certificate source and the migration. **Not tested: an order against Let's Encrypt** (needs a pebble test server or a real zone).

## Deployment and rollback (confirmed)

- **Router on a stick**: `wan.interface` may also carry LANs, but only for PPPoE on a VLAN with IPv6 disabled (the untagged link is then free for a LAN; native IPv6 needs an untagged uplink). In that mode only the PPP device is an uplink in the firewall, so LAN traffic can never match WAN rules; the WAN VLAN is created from the LAN's/trunk's networkd unit. A LAN VLAN may not reuse the PPPoE VLAN id.
- **Commit-confirm** (`router.rollback`, on by default): `router-arm [time]` starts a transient timer; `router-rollback` re-activates the confirmed generation (`/nix/var/nix/profiles/system`, else `/run/booted-system`) with `switch-to-configuration test`; `router-confirm` stops the timer, sets the profile and the boot default. Deploy with `nix run .#deploy -- <flake-attr> <user@host> [--yes]`: build, arm, `nixos-rebuild test --target-host`, check SSH, confirm. A lost connection means the timer rolls back by itself; a reboot also returns to the confirmed generation.
- `systemd.network.wait-online` waits only for the LAN devices (not the uplink) by default: an unplugged uplink must not block boot or activation, while VLAN devices still exist before Kea starts (anyInterface broke this; caught by `vlans`).
- `examples/desktop` + `nixosConfigurations.example` is the deployable starting point (single NIC `enp2s0` on the new box (antigua calls its NIC `eno1`), PPPoE VLAN 1001); it is built as a flake check. Secrets stay on the device (`/var/lib/router/pppoe-password`).
- Open for real hardware: Nix is still on the device (`router-confirm` uses `nix-env`); no image/read-only root yet.

## Local DNS and DHCP reservations (confirmed)

Needed to replace the user's hand-written router (host `antigua`, dnsmasq). Services such as k3s, Docker and Gitea stay on other hosts, so the router has no input rules for them.

- `router.dns.domain`: local domain, authoritative on the router (`static` zone; unknown names answer NXDOMAIN, never forwarded).
- `router.dns.hosts`: name to IPv4/IPv6 addresses; short names live in the domain, full names override public DNS for that name only (split horizon). Reverse (PTR) records are generated.
- `router.dns.overrides`: like hosts but wildcard (Unbound `redirect` zone).
- `router.dns.forwardZones`: zone to resolvers; these and the local domain are `domain-insecure` (private zones under possibly signed parents).
- `lans.<n>.dhcp.reservations.<hostname> = { mac; address; }`: Kea host reservations; the hostname is published in DNS. Assertions: valid MAC/hostname, address inside the LAN and not the router's, no duplicate MACs/addresses, no clash with `dns.hosts`.
- Tunnels with `dns = true` answer on the LAN addresses too: Unbound's access control admits what the tunnel's peers route (`allowedIPs`), so a site-to-site peer without a transfer address can use the resolver.
- Tests: `dns` (VM), site-to-site DNS in `wireguard`, and assertion cases; mutation-tested.
- Still missing for the `antigua` migration: IPv6 over the PPP session (ipv6cp, DHCPv6-PD on the PPP device), an extra static ULA address with RA on the LAN, tunnels without own addresses/fully trusted tunnels (partly covered), and the PPP device name (`wan` there, `pppoe-wan` here: `ddclient` must follow).

## WireGuard (confirmed)

- **Use cases**: road warrior (listening, dual-stack), site-to-site (no transfer network), and a provider VPN the router dials into (dual-stack transfer net, masquerade). All via `router.wireguard.<name>`.
- **Firewall**: a tunnel is a zone member like a LAN (`zone`, `internet`, `allowManagement`, plus `dns`, `masquerade`, `killSwitch`), so `router.firewall.rules` cover it. Safer defaults than LANs: management, DNS and internet are off. A tunnel's peers never see each other (no same-device forwarding).
- **Routing**: by destination only — a peer's `allowedIPs` become routes (no default-route tunnels; `/0` is rejected). Provider-style "policy routing" is exactly this; source-based selectors (`ip rule`) are not implemented.
- **Kill switch** (default on): forwarded packets for a tunnel's `allowedIPs` are dropped if they would leave via an uplink (placed before the established-accept, so flows that were in the tunnel are cut too).
- **Addresses** are explicit (`addresses` list, empty for site-to-site); nothing generated. No IPv6 internet for road warriors (the delegated /64 belongs to a LAN; NPTv6 not implemented).
- **Keys**: `privateKeyFile` (and `presharedKeyFile`) outside the Nix store, handed to the unprivileged systemd-networkd via `LoadCredential`, so files can stay root-only 0600. Deployment must provide them.
- **Endpoint names**: networkd resolves a hostname in a peer's `endpoint` when it creates the device and retries (0.2 s doubling to 25.6 s) until it resolves, so boot order is fine (seen on the real router: four failures, then success once PPPoE was up). It never resolves again, so a peer's new address (dynamic DNS) is not noticed. `router.wireguardEndpointRefresh` (default `1min`, null = off) runs `router-wireguard-reresolve` (a timer), which sets the endpoint again with `wg set` for named peers that have no handshake in the last 135 s (a working tunnel is not touched). Tested in `wireguard` (a peer that never answers, a changing /etc/hosts, a working tunnel left alone).

## Replacing `antigua` (the user's hand-written router)

> `examples/home` (the hand-written Nix port of antigua's router role), its `home` check, `installer-home` and the WireGuard import script were removed once the installer and web UI could configure the router (`settings.json`); the notes below describe that port and stay for the reasoning. The PPPoE username and everything else now live in `settings.json` (Network page).

Source: the user's infra repo (host `antigua`, NixOS, dnsmasq + networkd + pppd). It is also a k3s/Docker/Gitea host today; **those services will stay on antigua / move elsewhere, so the router does not need input rules for them or trusted container interfaces.** The new box is set up next to antigua and tested by swapping the two.

Done: PPPoE on VLAN 1001 over the shared NIC (`eno1`, LAN untagged), DNS records/domain/forward zones/DHCP reservations/DNS for site-to-site peers, WireGuard (freifunk dial-out with masquerade, family site-to-site with PSK, mobile-clients road warrior), MSS clamp.

Open (in rough order):
1. ~~**IPv6 over the PPP session**~~ **done**: `wan.ipv6.transport = "ppp"` (default `native`) enables IPv6CP in pppd, RA + DHCPv6-PD via networkd on the PPP device, PD to the LAN, DHCPv6 replies accepted on the PPP device and the IPv6 MSS clamped to MTU-60 (1432). Allowed on a shared link (single NIC). Tested in `onestick` (ISP with radvd and a minimal DHCPv6-PD responder, since Kea cannot serve point-to-point links); mutation-tested (MSS clamp, DHCPv6 input rule). `ipv6cp-use-ipaddr` is set as on antigua.
2. ~~**ULA on the LAN**~~ **done**: `lans.<n>.ipv6.address` (a ULA /64, e.g. `fd2c:cdef:cde9::1/64`) is assigned, announced via SLAAC (explicit `[IPv6Prefix]`, because networkd announces only the delegated prefix when PD is on) and advertised as RDNSS; Unbound listens on it. Build-time checks: valid IPv6 CIDR, /64, fc00::/7. Tested in `onestick` (client SLAAC ULA, ping and DNS over the ULA, RA captured on the wire) and assertion cases; mutation-tested.
3. ~~**Tunnel trust and addressing**~~ **done**: tunnels may have no address; the new `wireguard.<n>.sourceAddresses` sets the source (`PreferredSource`) for router-originated traffic into an addressless tunnel (e.g. DNS to Troisdorf; tested, mutation-tested). Antigua's blanket `trustedInterfaces` is spelled out as per-tunnel `allowManagement`/`dns` plus zone rules in `examples/home`. The default device name `wg-<name>` must fit 15 characters (caught by an assertion: `mobile-clients` became `mobile`).
4. ~~**PPP device name**~~ **done**: `ddclient` in `examples/home` uses `config.router.wan.ipv4.interface` (`pppoe-wan`) instead of a hard-coded name. DDNS has since moved into the module (`router.ddns`, a web UI page; tested in `ddns`): `examples/home` can drop its `services.ddclient` for it.
5. **Credentials**: the PPPoE password is assigned by the ISP (cannot be rotated) and is plaintext in antigua's repo/store; never copy it into a repo again. Here it is read from `/var/lib/router/pppoe-password` (delivered with `router-install --secrets`). The PPPoE username in `examples/home` is a placeholder.
6. **Services that stay on antigua** (k3s, Gitea, Syncthing, WoW, Traefik) were reachable from the WAN because antigua *was* the router. On the new router they need explicit port forwards to the host's address (a NAT row on the Firewall page); antigua's address on the LAN also changes (k3s nodeIP and the `antigua.mch` record point at 192.168.168.254 today). **Open: needs the host's new address and the list of public ports.**
7. `examples/home` (`nixosConfigurations.home`, `packages.installer-home`) ports antigua's routing role; its toplevel builds as the `home` check. Not VM-testable as a whole (needs the real ISP); its pieces are covered by the component tests.

## Installer and image (in progress)

- **Delivery order**: installer ISO first (set up a second box next to antigua), `image.repart` disk image + A/B updates later.
- `router-install [--yes] [--secrets DIR] <disk>`: GPT with an ESP (label `boot`) and an ext4 root (label `nixos`), `nixos-install --system <prebuilt toplevel>` (nothing is built on the box), secrets copied to `/var/lib/router`. The layout contract is `nixosModules.installed-layout` (labels, systemd-boot without EFI variables so it boots via the removable fallback path, generic initrd modules, redistributable firmware); target configs import it.
- `lib.mkInstaller { configuration }` builds the ISO for a given `nixosSystem`; the live system runs sshd with that system's root keys. `packages.x86_64-linux.installer-example` is the example's ISO.
- Test `installer` (VM): the nixpkgs installer-test pattern (an installer VM writes `target.qcow2`, a second VM boots it with UEFI); checks refusal of bad input/mounted disks, the fallback boot file, router services up, secrets owned by root with 0700/0600. Mutation-tested (secret permissions, mounted-disk guard).
- Status: `nix build .#installer-example` produces a 1.6 GB ISO. **Not yet tested**: booting that ISO on real firmware (the VM test exercises `router-install`, not the live ISO), the NIC name in the example, and installing over an existing system.
- Build: `nix build .#installer-example`, write `result/iso/*.iso` to a USB stick, boot the new box (UEFI), `scp -r secrets root@<box>:` then `router-install --secrets /root/secrets /dev/<disk>`.

## A/B image product: considered, not pursued

Considered 2026-10-30: moving the product to the A/B dm-verity appliance image (`modules/image`) with a
runtime renderer for `settings.json`. Dropped in favour of the current approach (installed NixOS, on-device
builds with `router-apply`, signed Nix-cache updates). Reasons: a read-only store cannot build a new system, so
the choice was a Go reimplementation of ~2300 lines of Nix module logic, or Nix with a writable overlay on the
box, which makes every post-update boot wait for a rebuild and leaves the settings-derived system outside
verity. The image module and its test (`checks.image`) stay as they are for anyone who wants a fixed image.

## Local zone (confirmed)

- **`local` zone**: `firewall.rules` may have `to = "local"`, the router itself (SSH, the web UI, ...): those rules go on the `input` chain. From `wan` they need a protocol and ports (no destination); `natOnly` is refused; `local` cannot be a zone name. A port forward on the same port takes the traffic first, so it never reaches `local`. Checked by `tests/assertions.nix` (`local*`); the generated rules are covered by a subtest of `basic` (the named sender reaches the router's port, others and unopened ports do not, the LAN-only rule works).

## Package upgrades (`router-upgrade`, 2026-11-08)

Upgrading nixpkgs (the kernel and every package) on the router, without a new release of nixos-router. Step 1 (CLI and setting) and step 2 (web UI page) are done; the notice of a new NixOS release follows.

- **Setting** `router.update.channel`: `unstable` or a release like `26.05` (branch `nixos-26.05`); null: the packages that came with the installed release. Read from the saved `settings.json` when upgrading; applying settings does not change nixpkgs.
- **`router-upgrade check`**: the branch's newest revision from `<channels.nixos.org>/<branch>/git-revision` (the last one Hydra's tests passed), against the running one (`nixos-version --revision`). Writes `/var/lib/router/upgrade/check.json`.
- **`preview [--force]`**: locks a copy of the configuration flake to that revision (`nix flake lock --override-input nixos-router/nixpkgs`), evaluates it and lists the packages of the running system that have another version in the new system's derivations, how much `nix build --dry-run` would download, and how many derivations would be built here. Nothing is downloaded: evaluation fetches the nixpkgs source only. Approximate: a rebuild without a new version is not listed. Refuses a NixOS release older than the running one. A changed `linux` sets `rebootNeeded`. Result in `preview.json`.
- **`download`**: fetches and builds it (needs 3 GiB free), so a flaky uplink does not matter at `apply`.
- **`apply [timeout]`**: takes the previewed lock into the configuration directory and runs `router-apply` (commit-confirm). The directory as it was is kept in `config.pre-update` like for updates: a rollback or a failed build puts the lock back. Refused if the configuration flake changed since the preview.
- **Pin**: `nixpkgs-pin.json` in the configuration directory (channel, revision, flake reference with `narHash`). `router-update` locks it again after it installs a new release of nixos-router, so a release does not undo an upgrade; the reference is found in the store by its hash, offline.
- **Web UI page** `/upgrade` ("Packages", under System): the channel as a select (none, `unstable`, the running release; a configured one that is none of these stays listed; saved to `settings.json` and the confirmed settings, so it never shows as unapplied), buttons Check, Preview (with "look anyway" when up to date), Download now and Upgrade. The preview shows the version table, download size, local builds (a warning above 50) and a reboot note for a new kernel. The page asks for a step in `upgrade/request.json` and starts `router-upgrade.service`, which runs `router-upgrade --request` (not restarted or stopped by a switch, like `router-apply`); it shows `check.json`/`preview.json` (not those of another channel, nor a check from before the last upgrade) and `upgrade/last.log`, which the unit tees (a finished one-shot unit has no journal to find). Download and Upgrade need a preview and no change waiting for confirmation; a request without a `channel` field is refused rather than clearing it.
- **Module options** (for tests): `routerSettings.upgrade.channelsUrl`, `routerSettings.upgrade.nixpkgsRef` (`%rev%`).
- **Test** (`apply`): the channel is a `file://` directory and the "newer" nixpkgs is the installed one under a `path:` reference with the same `lastModified` and `rev`: a really different nixpkgs cannot be built offline (any version change rebuilds the NixOS tools). Covers check, preview (including the package-list rules), download, an unconfirmed upgrade rolled back with its lock and pin, a confirmed one, and a failing build.

## Not done yet (2026-11-01, after 0.12.0)

Remote access
- **MCP and REST: an `apply` scope**: build, activate and confirm. A token with it changes the running system, so it needs its own design (probably a short expiry and a confirmation the token cannot give).
- **More read tools**: metrics history, WireGuard status (handshakes, transfer), certificate status.
- **REST API**: the rest of what the web UI does (apply and confirm, metrics, updates); the web UI could use the same handlers.
- **GraphQL** on top of it, possibly: one query for the overview and for related settings.
- Rule for all of it: changes go through `settings.json` and `router-apply` (commit-confirm), never straight to the live system; tokens have scopes; every call is logged.

Firewall and networking
- **Host-based routing on 80/443**: decide by TLS server name (or the `Host` header) between the router's web UI and a host behind it. It needs an SNI proxy on the router and loses the client address unless the host takes the PROXY protocol. Until then, expose the web UI and SSH on their own ports with `wan -> local` rules.
- **WireGuard**: live status (handshake, transfer) in the web UI; a tunnel with `dns = true` but no address.
- **IPv6** for forwards: a pinhole for a host behind the router is possible by hand (`wan -> lan` with an IPv6 `destination`), nothing helps with finding the host's delegated address. Also open: other LANs are IPv4-only (PD serves one LAN), DHCPv6 reservations, NPTv6.
- **Multi-WAN failover**, **QoS** (CAKE/SQM): deferred, not started.

Installer and web UI
- Installer: name the network cards (`router.interfaces`) during `router-setup`; booting the ISO on real firmware, installing over an existing system and the NIC name in the example are untested.
- A pebble-based ACME test (certificate issuance is only checked by evaluation).
- Monitoring beyond the dashboard (node exporter, nftables counters).
- Web UI statistics are read-only history of a few metrics; longer retention and per-LAN traffic are not there.

Hardware
- **aarch64**: no board is supported. The NanoPi R2S image (U-Boot, extlinux, first-boot password on the console) was built and then dropped; booting it was never tried. Open concerns if it comes back: 1 GB of RAM for on-device rebuilds, USB NIC naming, `router-update` and the release tooling for `aarch64-linux`.

On the user's own router (settings and infrastructure, not code)
- Port forwards for antigua's public names exist for 80 and 443 (hairpin on); the list of other public ports of antigua (k3s, Gitea, Syncthing, WoW, Traefik) is still unknown.
- The WireGuard tunnel `family` has `allowManagement = true` and open firewall rules to the LAN; narrow them when the intended rules are known.
- No time zone is set (UTC); the wifi card is unused. (`truenas.home.stinnesbeck.com` in `dns.hosts` stays: old things in the cluster still refer to it.)

## Phases

1. **MVP** — flake with `router.*` schema: single WAN (DHCP), single LAN, NAT, basic firewall, DHCP + DNS. VM tests. *Done.*
2. **Real-world features** — IPv6 PD, VLANs, zone policy, port forwards, WireGuard. *Done.*
3. **Images and hardware** — installer ISO, `router-setup`, signed updates. *Done for x86_64;* read-only images were considered and dropped (see above), aarch64 was tried and dropped.
4. **Operations** — commit-confirm rollback, signed updates, backups, DDNS and certificates. *Done.* Monitoring beyond the dashboard and multi-WAN failover: not done.
5. **Web UI and remote access** — pages for every setting, themes, MCP server, REST API. *Done* (0.12.0); see "Not done yet".

Package upgrades (see above)
- **New NixOS release**: look for newer `nixos-YY.MM` branches and say so; mark releases the installed nixos-router was not tested with.
- **Tested revisions**: a signed manifest in the release naming the nixpkgs revisions the tests passed on, as a gate in front of the channel's tip.
- **Reset**: go back to the nixpkgs that came with the release (`channel = null` and remove the pin).