nixos-router/PLAN.md

nixos-router — Plan

Working name: nixos-router. A NixOS-based distribution for routers/firewalls, delivered as a flake exporting NixOS modules and bootable images (not a nixpkgs fork).

Decisions

Area	Decision
Audience	Own homelab first; keep structure clean enough to publish later.
Config model	Opinionated router.* option schema compiled to systemd-networkd + nftables, with build-time assertions.
Hardware	x86_64 UEFI first. aarch64 is a planned target; avoid x86-only assumptions.
v1 features	IPv6 + prefix delegation, VLANs / multiple LANs, WireGuard.
Deferred	Multi-WAN failover, QoS (CAKE/SQM), web UI.
Updates (v1)	Push from workstation (nixos-rebuild --target-host / colmena) + commit-confirm rollback timer.
Updates (later)	A/B image updates with automatic rollback.
Footprint	Minimal closure, read-only root, no Nix daemon/compiler on device, small persistent /var.

Architecture

1. router.* schema — users describe intent (wan, lan, vlans, zones, port forwards, wireguard); modules compile it to:
   • systemd-networkd (DHCPv4/v6, DHCPv6-PD, VLANs, bridges, PPPoE later)
   • nftables zone-based ruleset (generated, not networking.firewall)
   • DHCP / DNS / RA services
2. Build-time validation — Nix assertions reject lock-out or inconsistent configs (no management access rule, overlapping subnets, zone referencing missing interface); generated ruleset checked with nft -c.
3. Images — x86_64 first (installer ISO + raw disk), aarch64 later. Read-only root (erofs/squashfs store), small persistent /var.
4. Deployment — deploy a prebuilt closure (nix copy + switch-to-configuration); never evaluate or build on the router. This keeps the device light and leads naturally to A/B updates.

Implications

• No on-device evaluation: all designs must work from a prebuilt closure.
• Flake exposes x86_64-linux and aarch64-linux outputs from the start; hardware specifics (bootloader, kernel, NIC naming) live in separate boards/ modules.
• IPv6 PD fan-out to multiple VLANs needs stable per-LAN subnet-ID allocation; this is the hardest part of the schema.

Default zone policy (confirmed)

• WAN: deny inbound (except established/related, required ICMPv6, explicit port forwards).
• LAN: allow outbound to WAN.
• Other zones (e.g. IoT): isolated from each other and from LAN unless a rule opens them.

Management access (confirmed)

• Key-only SSH, reachable from LAN zones (trusted, non-WAN, non-isolated zones). No GUI in v1.
• Build-time assertion fails if no management path exists.
• Isolated zones (e.g. IoT) get no router access beyond DHCP/DNS/NDP.

Services (confirmed)

• DHCP: Kea (DHCPv4 + DHCPv6 incl. PD).
• DNS: Unbound.
• RA: systemd-networkd (or radvd if networkd falls short).

Repo layout (confirmed)

flake.nix
modules/     # router.* schema and service modules
boards/      # hardware-specific modules (x86_64 first, aarch64 later)
tests/       # nixosTest VM tests
images/      # image/ISO builds
examples/    # sample configs

Status

WireGuard done: wireguard VM test (5 VMs: ISP/provider, router, laptop, remote site, LAN host) covers handshakes, road-warrior DNS/SSH over IPv4 and IPv6, zone rules between tunnels and LAN, site-to-site routing without NAT or transfer net, provider tunnel with masquerade and IPv6 transfer net, and the kill switch (kill switch and masquerade mutation-tested). Not covered: preshared keys, mtu, tunnels with several peers, key rotation.

VLANs + zones done: vlans VM test (5 VMs: ISP, router, untagged main, VLAN 20 iot, VLAN 30 cams) covers VLAN devices, per-VLAN DHCP/gateway, opt-in internet, DNS vs SSH per LAN, intra-zone forwarding across VLANs, one-directional inter-zone rules and the destination restriction (each mutation-tested). Not covered yet: IPv6 on VLAN LANs (only the PD LAN has IPv6), port forwards from the WAN.

PPPoE done: pppoe VM test = the same topology with a fake ISP running pppoe-server (VLAN 7, CHAP, password containing a quote) — covers session, NAT, MTU, MSS clamp (mutation-tested) and WAN-deny. basic remains the DHCPv4 variant.

IPv6 + prefix delegation done (single /64): router.wan.ipv6.enable and router.lans.<name>.ipv6.pd (at most one LAN, asserted). The VM test now covers the router's DHCPv6 address and delegated /64, SLAAC on the client, routing without NAT66, and WAN-deny over IPv6 (mutation-tested). Still open: RDNSS, other LANs are IPv4-only, DHCPv6 reservations.

Phase 1 (MVP) skeleton done: flake, router.* schema (wan, lans, dns.forwarders), networkd, nftables (zone-less for now: WAN/LAN), Kea DHCPv4, Unbound, assertions. nix build .#checks.x86_64-linux.basic runs a 3-node VM test (upstream/router/client) covering WAN DHCP, LAN DHCP+DNS, NAT, WAN-deny and LAN SSH.

Interactive lab: nix run .#lab (x86_64) opens tmux with one console pane per VM plus the test-driver REPL (router.shell_interact(), client.succeed(...), test_script()). The REPL pane also shows all VM output interleaved.

Learned along the way:

• The default policy now explicitly answers ICMP echo on LANs only; the WAN gets only ICMP error types and ND.
• In NixOS tests fail(...) also passes on "command not found": install probe tools explicitly and add a positive control per negative check.
• Unbound answers .test locally (RFC 6761) and validates DNSSEC; tests using fake domains need both local-zone ... transparent and domain-insecure (both verified necessary; set in the test, not the module).
• modules/router/lib.nix has IPv4 helpers (parse/format/netmask/cidr/overlap). ipv4.subnet is now derived from address + prefixLength (read-only). Assertions also cover pool bounds, pool containing the router address, and overlapping LANs. IPv6 helpers (prefix arithmetic for PD subnet-IDs) are still to do.
• Checks: basic, pppoe, onestick, rollback, dns, vlans and wireguard (VM tests), example (the example system builds), lib (helper unit tests), assertions (broken configs must fail with exact messages). Not yet covered: NPTv6.

IPv6 design (confirmed)

• WAN: mirrors the existing router's uplink config — networkd DHCP (v6), IPv6AcceptRA = true, KeepConfiguration = yes, dhcpV6Config = { WithoutRA = "solicit"; UseDelegatedPrefix = true; }. The ISP sends RAs (default route) without DHCPv6 flags, so the router solicits on its own.
• Delegation: the ISP delegates a single /64, which cannot be split. Exactly one LAN may claim it via an explicit flag (router.lans.<name>.ipv6.pd = true); an assertion rejects more than one.
• LANs: SLAAC only (RA, no DHCPv6). All other LANs/VLANs are IPv4-only for now (ULA or NPTv6 can come later).
• Firewall: inet table covers v6; WAN→LAN forwarding stays denied, no NAT66.
• Test: fake ISP on upstream = Kea DHCPv6 (IA_NA + IA_PD /64 from a /48 pool) + networkd RA sender (default route only) + a static route for the delegated prefix back to the router.
• Known gap: no RDNSS in RAs yet (the router's global LAN address is not known at eval time); clients still get DNS via DHCPv4.

PPPoE uplink (confirmed)

• IPv4 via PPPoE (router.wan.ipv4.type = "pppoe"), carried on an optional tagged VLAN on the WAN link (pppoe.vlan); native DHCPv6/PD stays on the untagged link.
• wan.interface is the physical link; the derived wan.ipv4.interface (pppoe-wan or the link) is where NAT, MSS clamping and the IPv4 forwarding rule apply. IPv6 forwarding keeps using the physical link.
• pppd (pppoe.so) runs as router-pppoe.service, ordered after the VLAN device. Credentials: username inline, password from pppoe.passwordFile (type forbids Nix-store paths); the options file is generated into a root-only runtime dir. Deployment must provide that file on persistent storage (or via agenix/sops).
• MTU 1492 + TCP MSS clamp (1452) in both directions on the PPP device; RFC 4638 not implemented.

VLANs and zones (confirmed)

• VLAN devices are created by the router: router.lans.<name> = { interface = <trunk>; vlan = <id>; } makes <trunk>.<id> via networkd. A trunk may also carry one untagged LAN. Assertions: duplicate device, device name > 15 chars, WAN interface reused as trunk.
• Zones are an attribute on the LAN (zone, default = the LAN's own name); LANs sharing a zone may forward to each other, traffic between zones is denied unless a rule in router.firewall.rules allows it (from, to, protocol, ports, optional IPv4 destination that must lie inside a LAN of the destination zone). Rules are one-directional; return traffic via conntrack. Assertions reject unknown zones, from == to, ports without tcp/udp, bad/out-of-zone/host-bit destinations.
• Internet is opt-in per LAN (internet, default false) — this replaces the earlier "LAN allow-out" default; any LAN that should reach the WAN must set it.
• Router services stay per LAN: DNS and DHCP always, SSH only with allowManagement.
• Kea subnet IDs are derived from the network address, so adding or removing LANs does not shift IDs (and orphan persistent leases).

Port forwards and NAT (confirmed)

• Filtering and translation are separate. router.firewall.rules only filter and may use the reserved source zone wan (no LAN/tunnel may use that zone name; wan is never a destination). Rule fields: port ranges ("9000-9002"), sources allow-list (v4/v6 CIDRs), rateLimit (N/second|minute|hour, per source via nft meter), log, natOnly (only connections a DNAT rule translated).
• router.firewall.nat is the translation table: type dnat (IPv4 only, one host toAddress in the to zone, tcp/udp, optional toPort, hairpin from wan: NAT loopback for LANs/tunnels with internet = true; masquerades so replies return via the router), snat (toAddress) and masquerade, matching from, sources, destination (default: the router's own addresses), protocol and ports.
• The web UI's NAT table can also add the matching filter rule ("also allow", a natOnly rule). Schema 1 (migrations/1.jq) converts the old rules with dnat into a NAT row plus such a rule.
• Safeguards: a rule from wan must name a destination, an explicit protocol and its ports, so a typo cannot expose a whole zone.
• IPv6 has no NAT: it is opened only by explicit pinholes (from = "wan", IPv6 destination). The prefix is dynamic, so v6 destinations are not checked against the zone.
• Tested in basic/pppoe (source preserved, toPort, ranges, sources, rate limit, v6 pinhole, hairpin, log lines) and in assertions; mutation-tested (source match, hairpin masquerade, rate limit).

Deployment and rollback (confirmed)

• Router on a stick: wan.interface may also carry LANs, but only for PPPoE on a VLAN with IPv6 disabled (the untagged link is then free for a LAN; native IPv6 needs an untagged uplink). In that mode only the PPP device is an uplink in the firewall, so LAN traffic can never match WAN rules; the WAN VLAN is created from the LAN's/trunk's networkd unit. A LAN VLAN may not reuse the PPPoE VLAN id.
• Commit-confirm (router.rollback, on by default): router-arm [time] starts a transient timer; router-rollback re-activates the confirmed generation (/nix/var/nix/profiles/system, else /run/booted-system) with switch-to-configuration test; router-confirm stops the timer, sets the profile and the boot default. Deploy with nix run .#deploy -- <flake-attr> <user@host> [--yes]: build, arm, nixos-rebuild test --target-host, check SSH, confirm. A lost connection means the timer rolls back by itself; a reboot also returns to the confirmed generation.
• systemd.network.wait-online waits only for the LAN devices (not the uplink) by default: an unplugged uplink must not block boot or activation, while VLAN devices still exist before Kea starts (anyInterface broke this; caught by vlans).
• examples/desktop + nixosConfigurations.example is the deployable starting point (single NIC enp2s0 on the new box (antigua calls its NIC eno1), PPPoE VLAN 1001); it is built as a flake check. Secrets stay on the device (/var/lib/router/pppoe-password).
• Open for real hardware: Nix is still on the device (router-confirm uses nix-env); no image/read-only root yet.

Local DNS and DHCP reservations (confirmed)

Needed to replace the user's hand-written router (host antigua, dnsmasq). Services such as k3s, Docker and Gitea stay on other hosts, so the router has no input rules for them.

• router.dns.domain: local domain, authoritative on the router (static zone; unknown names answer NXDOMAIN, never forwarded).
• router.dns.hosts: name to IPv4/IPv6 addresses; short names live in the domain, full names override public DNS for that name only (split horizon). Reverse (PTR) records are generated.
• router.dns.overrides: like hosts but wildcard (Unbound redirect zone).
• router.dns.forwardZones: zone to resolvers; these and the local domain are domain-insecure (private zones under possibly signed parents).
• lans.<n>.dhcp.reservations.<hostname> = { mac; address; }: Kea host reservations; the hostname is published in DNS. Assertions: valid MAC/hostname, address inside the LAN and not the router's, no duplicate MACs/addresses, no clash with dns.hosts.
• Tunnels with dns = true answer on the LAN addresses too: Unbound's access control admits what the tunnel's peers route (allowedIPs), so a site-to-site peer without a transfer address can use the resolver.
• Tests: dns (VM), site-to-site DNS in wireguard, and assertion cases; mutation-tested.
• Still missing for the antigua migration: IPv6 over the PPP session (ipv6cp, DHCPv6-PD on the PPP device), an extra static ULA address with RA on the LAN, tunnels without own addresses/fully trusted tunnels (partly covered), and the PPP device name (wan there, pppoe-wan here: ddclient must follow).

WireGuard (confirmed)

• Use cases: road warrior (listening, dual-stack), site-to-site (no transfer network), and a provider VPN the router dials into (dual-stack transfer net, masquerade). All via router.wireguard.<name>.
• Firewall: a tunnel is a zone member like a LAN (zone, internet, allowManagement, plus dns, masquerade, killSwitch), so router.firewall.rules cover it. Safer defaults than LANs: management, DNS and internet are off. A tunnel's peers never see each other (no same-device forwarding).
• Routing: by destination only — a peer's allowedIPs become routes (no default-route tunnels; /0 is rejected). Provider-style "policy routing" is exactly this; source-based selectors (ip rule) are not implemented.
• Kill switch (default on): forwarded packets for a tunnel's allowedIPs are dropped if they would leave via an uplink (placed before the established-accept, so flows that were in the tunnel are cut too).
• Addresses are explicit (addresses list, empty for site-to-site); nothing generated. No IPv6 internet for road warriors (the delegated /64 belongs to a LAN; NPTv6 not implemented).
• Keys: privateKeyFile (and presharedKeyFile) outside the Nix store, handed to the unprivileged systemd-networkd via LoadCredential, so files can stay root-only 0600. Deployment must provide them.

Replacing antigua (the user's hand-written router)

examples/home (the hand-written Nix port of antigua's router role), its home check, installer-home and the WireGuard import script were removed once the installer and web UI could configure the router (settings.json); the notes below describe that port and stay for the reasoning. The PPPoE username and everything else now live in settings.json (Network page).

Source: the user's infra repo (host antigua, NixOS, dnsmasq + networkd + pppd). It is also a k3s/Docker/Gitea host today; those services will stay on antigua / move elsewhere, so the router does not need input rules for them or trusted container interfaces. The new box is set up next to antigua and tested by swapping the two.

Done: PPPoE on VLAN 1001 over the shared NIC (eno1, LAN untagged), DNS records/domain/forward zones/DHCP reservations/DNS for site-to-site peers, WireGuard (freifunk dial-out with masquerade, family site-to-site with PSK, mobile-clients road warrior), MSS clamp.

Open (in rough order):

1. IPv6 over the PPP session done: wan.ipv6.transport = "ppp" (default native) enables IPv6CP in pppd, RA + DHCPv6-PD via networkd on the PPP device, PD to the LAN, DHCPv6 replies accepted on the PPP device and the IPv6 MSS clamped to MTU-60 (1432). Allowed on a shared link (single NIC). Tested in onestick (ISP with radvd and a minimal DHCPv6-PD responder, since Kea cannot serve point-to-point links); mutation-tested (MSS clamp, DHCPv6 input rule). ipv6cp-use-ipaddr is set as on antigua.
2. ULA on the LAN done: lans.<n>.ipv6.address (a ULA /64, e.g. fd2c:cdef:cde9::1/64) is assigned, announced via SLAAC (explicit [IPv6Prefix], because networkd announces only the delegated prefix when PD is on) and advertised as RDNSS; Unbound listens on it. Build-time checks: valid IPv6 CIDR, /64, fc00::/7. Tested in onestick (client SLAAC ULA, ping and DNS over the ULA, RA captured on the wire) and assertion cases; mutation-tested.
3. Tunnel trust and addressing done: tunnels may have no address; the new wireguard.<n>.sourceAddresses sets the source (PreferredSource) for router-originated traffic into an addressless tunnel (e.g. DNS to Troisdorf; tested, mutation-tested). Antigua's blanket trustedInterfaces is spelled out as per-tunnel allowManagement/dns plus zone rules in examples/home. The default device name wg-<name> must fit 15 characters (caught by an assertion: mobile-clients became mobile).
4. PPP device name done: ddclient in examples/home uses config.router.wan.ipv4.interface (pppoe-wan) instead of a hard-coded name. DDNS has since moved into the module (router.ddns, a web UI page; tested in ddns): examples/home can drop its services.ddclient for it.
5. Credentials: the PPPoE password is assigned by the ISP (cannot be rotated) and is plaintext in antigua's repo/store; never copy it into a repo again. Here it is read from /var/lib/router/pppoe-password (delivered with router-install --secrets). The PPPoE username in examples/home is a placeholder.
6. Services that stay on antigua (k3s, Gitea, Syncthing, WoW, Traefik) were reachable from the WAN because antigua was the router. On the new router they need explicit port forwards to the host's address (a NAT row on the Firewall page); antigua's address on the LAN also changes (k3s nodeIP and the antigua.mch record point at 192.168.168.254 today). Open: needs the host's new address and the list of public ports.
7. examples/home (nixosConfigurations.home, packages.installer-home) ports antigua's routing role; its toplevel builds as the home check. Not VM-testable as a whole (needs the real ISP); its pieces are covered by the component tests.

Installer and image (in progress)

• Delivery order: installer ISO first (set up a second box next to antigua), image.repart disk image + A/B updates later.
• router-install [--yes] [--secrets DIR] <disk>: GPT with an ESP (label boot) and an ext4 root (label nixos), nixos-install --system <prebuilt toplevel> (nothing is built on the box), secrets copied to /var/lib/router. The layout contract is nixosModules.installed-layout (labels, systemd-boot without EFI variables so it boots via the removable fallback path, generic initrd modules, redistributable firmware); target configs import it.
• lib.mkInstaller { configuration } builds the ISO for a given nixosSystem; the live system runs sshd with that system's root keys. packages.x86_64-linux.installer-example is the example's ISO.
• Test installer (VM): the nixpkgs installer-test pattern (an installer VM writes target.qcow2, a second VM boots it with UEFI); checks refusal of bad input/mounted disks, the fallback boot file, router services up, secrets owned by root with 0700/0600. Mutation-tested (secret permissions, mounted-disk guard).
• Status: nix build .#installer-example produces a 1.6 GB ISO. Not yet tested: booting that ISO on real firmware (the VM test exercises router-install, not the live ISO), the NIC name in the example, and installing over an existing system.
• Build: nix build .#installer-example, write result/iso/*.iso to a USB stick, boot the new box (UEFI), scp -r secrets root@<box>: then router-install --secrets /root/secrets /dev/<disk>.

A/B image product: considered, not pursued

Considered 2026-10-30: moving the product to the A/B dm-verity appliance image (modules/image) with a runtime renderer for settings.json. Dropped in favour of the current approach (installed NixOS, on-device builds with router-apply, signed Nix-cache updates). Reasons: a read-only store cannot build a new system, so the choice was a Go reimplementation of ~2300 lines of Nix module logic, or Nix with a writable overlay on the box, which makes every post-update boot wait for a rebuild and leaves the settings-derived system outside verity. The image module and its test (checks.image) stay as they are for anyone who wants a fixed image.

Open items

• Web UI (stats and possibly configuration) — to be discussed later. Considerations: read-only stats first is cheap; config editing must write declarative config (Nix/JSON) that feeds the module, not mutate the live system.

Phases

1. MVP — flake with router.* schema: single WAN (DHCP), single LAN, NAT, basic firewall, DHCP + DNS. NixOS VM tests (nixosTest) in CI.
2. Real-world features — IPv6 PD, VLANs, zone policy, port forwards, WireGuard.
3. Images and hardware — installer ISO, read-only root, aarch64.
4. Operations — commit-confirm rollback, monitoring (node exporter, nftables counters), A/B image updates; multi-WAN failover.
5. Optional — web UI (stats, then configuration).