nixos-router/PLAN.md

nixos-router — Plan

Working name: nixos-router. A NixOS-based distribution for routers/firewalls, delivered as a flake exporting NixOS modules and bootable images (not a nixpkgs fork).

Decisions

Area	Decision
Audience	Own homelab first; keep structure clean enough to publish later.
Config model	Opinionated router.* option schema compiled to systemd-networkd + nftables, with build-time assertions.
Hardware	x86_64 UEFI first. aarch64 is a planned target; avoid x86-only assumptions.
v1 features	IPv6 + prefix delegation, VLANs / multiple LANs, WireGuard.
Deferred	Multi-WAN failover, QoS (CAKE/SQM), web UI.
Updates (v1)	Push from workstation (nixos-rebuild --target-host / colmena) + commit-confirm rollback timer.
Updates (later)	A/B image updates with automatic rollback.
Footprint	Minimal closure, read-only root, no Nix daemon/compiler on device, small persistent /var.

Architecture

1. router.* schema — users describe intent (wan, lan, vlans, zones, port forwards, wireguard); modules compile it to:
   • systemd-networkd (DHCPv4/v6, DHCPv6-PD, VLANs, bridges, PPPoE later)
   • nftables zone-based ruleset (generated, not networking.firewall)
   • DHCP / DNS / RA services
2. Build-time validation — Nix assertions reject lock-out or inconsistent configs (no management access rule, overlapping subnets, zone referencing missing interface); generated ruleset checked with nft -c.
3. Images — x86_64 first (installer ISO + raw disk), aarch64 later. Read-only root (erofs/squashfs store), small persistent /var.
4. Deployment — deploy a prebuilt closure (nix copy + switch-to-configuration); never evaluate or build on the router. This keeps the device light and leads naturally to A/B updates.

Implications

• No on-device evaluation: all designs must work from a prebuilt closure.
• Flake exposes x86_64-linux and aarch64-linux outputs from the start; hardware specifics (bootloader, kernel, NIC naming) live in separate boards/ modules.
• IPv6 PD fan-out to multiple VLANs needs stable per-LAN subnet-ID allocation; this is the hardest part of the schema.

Default zone policy (confirmed)

• WAN: deny inbound (except established/related, required ICMPv6, explicit port forwards).
• LAN: allow outbound to WAN.
• Other zones (e.g. IoT): isolated from each other and from LAN unless a rule opens them.

Management access (confirmed)

• Key-only SSH, reachable from LAN zones (trusted, non-WAN, non-isolated zones). No GUI in v1.
• Build-time assertion fails if no management path exists.
• Isolated zones (e.g. IoT) get no router access beyond DHCP/DNS/NDP.

Services (confirmed)

• DHCP: Kea (DHCPv4 + DHCPv6 incl. PD).
• DNS: Unbound.
• RA: systemd-networkd (or radvd if networkd falls short).

Repo layout (confirmed)

flake.nix
modules/     # router.* schema and service modules
boards/      # hardware-specific modules (x86_64 first, aarch64 later)
tests/       # nixosTest VM tests
images/      # image/ISO builds
examples/    # sample configs

Status

WireGuard done: wireguard VM test (5 VMs: ISP/provider, router, laptop, remote site, LAN host) covers handshakes, road-warrior DNS/SSH over IPv4 and IPv6, zone rules between tunnels and LAN, site-to-site routing without NAT or transfer net, provider tunnel with masquerade and IPv6 transfer net, and the kill switch (kill switch and masquerade mutation-tested). Not covered: preshared keys, mtu, tunnels with several peers, key rotation.

VLANs + zones done: vlans VM test (5 VMs: ISP, router, untagged main, VLAN 20 iot, VLAN 30 cams) covers VLAN devices, per-VLAN DHCP/gateway, opt-in internet, DNS vs SSH per LAN, intra-zone forwarding across VLANs, one-directional inter-zone rules and the destination restriction (each mutation-tested). Not covered yet: IPv6 on VLAN LANs (only the PD LAN has IPv6), port forwards from the WAN.

PPPoE done: pppoe VM test = the same topology with a fake ISP running pppoe-server (VLAN 7, CHAP, password containing a quote) — covers session, NAT, MTU, MSS clamp (mutation-tested) and WAN-deny. basic remains the DHCPv4 variant.

IPv6 + prefix delegation done (single /64): router.wan.ipv6.enable and router.lans.<name>.ipv6.pd (at most one LAN, asserted). The VM test now covers the router's DHCPv6 address and delegated /64, SLAAC on the client, routing without NAT66, and WAN-deny over IPv6 (mutation-tested). Still open: RDNSS, other LANs are IPv4-only, DHCPv6 reservations.

Phase 1 (MVP) skeleton done: flake, router.* schema (wan, lans, dns.forwarders), networkd, nftables (zone-less for now: WAN/LAN), Kea DHCPv4, Unbound, assertions. nix build .#checks.x86_64-linux.basic runs a 3-node VM test (upstream/router/client) covering WAN DHCP, LAN DHCP+DNS, NAT, WAN-deny and LAN SSH.

Interactive lab: nix run .#lab (x86_64) opens tmux with one console pane per VM plus the test-driver REPL (router.shell_interact(), client.succeed(...), test_script()). The REPL pane also shows all VM output interleaved.

Learned along the way:

• The default policy now explicitly answers ICMP echo on LANs only; the WAN gets only ICMP error types and ND.
• In NixOS tests fail(...) also passes on "command not found": install probe tools explicitly and add a positive control per negative check.
• Unbound answers .test locally (RFC 6761) and validates DNSSEC; tests using fake domains need both local-zone ... transparent and domain-insecure (both verified necessary; set in the test, not the module).
• modules/router/lib.nix has IPv4 helpers (parse/format/netmask/cidr/overlap). ipv4.subnet is now derived from address + prefixLength (read-only). Assertions also cover pool bounds, pool containing the router address, and overlapping LANs. IPv6 helpers (prefix arithmetic for PD subnet-IDs) are still to do.
• Checks: basic, pppoe, onestick, rollback, vlans and wireguard (VM tests), example (the example system builds), lib (helper unit tests), assertions (broken configs must fail with exact messages). Not yet covered: NPTv6.

IPv6 design (confirmed)

• WAN: mirrors the existing router's uplink config — networkd DHCP (v6), IPv6AcceptRA = true, KeepConfiguration = yes, dhcpV6Config = { WithoutRA = "solicit"; UseDelegatedPrefix = true; }. The ISP sends RAs (default route) without DHCPv6 flags, so the router solicits on its own.
• Delegation: the ISP delegates a single /64, which cannot be split. Exactly one LAN may claim it via an explicit flag (router.lans.<name>.ipv6.pd = true); an assertion rejects more than one.
• LANs: SLAAC only (RA, no DHCPv6). All other LANs/VLANs are IPv4-only for now (ULA or NPTv6 can come later).
• Firewall: inet table covers v6; WAN→LAN forwarding stays denied, no NAT66.
• Test: fake ISP on upstream = Kea DHCPv6 (IA_NA + IA_PD /64 from a /48 pool) + networkd RA sender (default route only) + a static route for the delegated prefix back to the router.
• Known gap: no RDNSS in RAs yet (the router's global LAN address is not known at eval time); clients still get DNS via DHCPv4.

PPPoE uplink (confirmed)

• IPv4 via PPPoE (router.wan.ipv4.type = "pppoe"), carried on an optional tagged VLAN on the WAN link (pppoe.vlan); native DHCPv6/PD stays on the untagged link.
• wan.interface is the physical link; the derived wan.ipv4.interface (pppoe-wan or the link) is where NAT, MSS clamping and the IPv4 forwarding rule apply. IPv6 forwarding keeps using the physical link.
• pppd (pppoe.so) runs as router-pppoe.service, ordered after the VLAN device. Credentials: username inline, password from pppoe.passwordFile (type forbids Nix-store paths); the options file is generated into a root-only runtime dir. Deployment must provide that file on persistent storage (or via agenix/sops).
• MTU 1492 + TCP MSS clamp (1452) in both directions on the PPP device; RFC 4638 not implemented.

VLANs and zones (confirmed)

• VLAN devices are created by the router: router.lans.<name> = { interface = <trunk>; vlan = <id>; } makes <trunk>.<id> via networkd. A trunk may also carry one untagged LAN. Assertions: duplicate device, device name > 15 chars, WAN interface reused as trunk.
• Zones are an attribute on the LAN (zone, default = the LAN's own name); LANs sharing a zone may forward to each other, traffic between zones is denied unless a rule in router.firewall.rules allows it (from, to, protocol, ports, optional IPv4 destination that must lie inside a LAN of the destination zone). Rules are one-directional; return traffic via conntrack. Assertions reject unknown zones, from == to, ports without tcp/udp, bad/out-of-zone/host-bit destinations.
• Internet is opt-in per LAN (internet, default false) — this replaces the earlier "LAN allow-out" default; any LAN that should reach the WAN must set it.
• Router services stay per LAN: DNS and DHCP always, SSH only with allowManagement.
• Kea subnet IDs are derived from the network address, so adding or removing LANs does not shift IDs (and orphan persistent leases).

Port forwards (confirmed)

• Implemented by extending router.firewall.rules with a reserved source zone wan (no LAN/tunnel may use that zone name; wan is never a destination).
• Extra rule fields: dnat (IPv4 only, single host destination, tcp/udp), toPort, port ranges ("9000-9002"), sources allow-list (v4/v6 CIDRs), rateLimit (N/second|minute|hour, per source via nft meter), log, hairpin (NAT loopback for LANs/tunnels with internet = true; masquerades so replies return via the router).
• Safeguards: a rule from wan must name a destination, an explicit protocol and its ports, so a typo cannot expose a whole zone.
• IPv6 has no NAT: it is opened only by explicit pinholes (from = "wan", IPv6 destination). The prefix is dynamic, so v6 destinations are not checked against the zone.
• Tested in basic/pppoe (source preserved, toPort, ranges, sources, rate limit, v6 pinhole, hairpin, log lines) and in assertions; mutation-tested (source match, hairpin masquerade, rate limit).

Deployment and rollback (confirmed)

• Router on a stick: wan.interface may also carry LANs, but only for PPPoE on a VLAN with IPv6 disabled (the untagged link is then free for a LAN; native IPv6 needs an untagged uplink). In that mode only the PPP device is an uplink in the firewall, so LAN traffic can never match WAN rules; the WAN VLAN is created from the LAN's/trunk's networkd unit. A LAN VLAN may not reuse the PPPoE VLAN id.
• Commit-confirm (router.rollback, on by default): router-arm [time] starts a transient timer; router-rollback re-activates the confirmed generation (/nix/var/nix/profiles/system, else /run/booted-system) with switch-to-configuration test; router-confirm stops the timer, sets the profile and the boot default. Deploy with nix run .#deploy -- <flake-attr> <user@host> [--yes]: build, arm, nixos-rebuild test --target-host, check SSH, confirm. A lost connection means the timer rolls back by itself; a reboot also returns to the confirmed generation.
• systemd.network.wait-online waits only for the LAN devices (not the uplink) by default: an unplugged uplink must not block boot or activation, while VLAN devices still exist before Kea starts (anyInterface broke this; caught by vlans).
• examples/desktop + nixosConfigurations.example is the deployable starting point (single NIC eno1, PPPoE VLAN 1001); it is built as a flake check. Secrets stay on the device (/var/lib/router/pppoe-password).
• Open for real hardware: Nix is still on the device (router-confirm uses nix-env); no image/read-only root yet.

WireGuard (confirmed)

• Use cases: road warrior (listening, dual-stack), site-to-site (no transfer network), and a provider VPN the router dials into (dual-stack transfer net, masquerade). All via router.wireguard.<name>.
• Firewall: a tunnel is a zone member like a LAN (zone, internet, allowManagement, plus dns, masquerade, killSwitch), so router.firewall.rules cover it. Safer defaults than LANs: management, DNS and internet are off. A tunnel's peers never see each other (no same-device forwarding).
• Routing: by destination only — a peer's allowedIPs become routes (no default-route tunnels; /0 is rejected). Provider-style "policy routing" is exactly this; source-based selectors (ip rule) are not implemented.
• Kill switch (default on): forwarded packets for a tunnel's allowedIPs are dropped if they would leave via an uplink (placed before the established-accept, so flows that were in the tunnel are cut too).
• Addresses are explicit (addresses list, empty for site-to-site); nothing generated. No IPv6 internet for road warriors (the delegated /64 belongs to a LAN; NPTv6 not implemented).
• Keys: privateKeyFile (and presharedKeyFile) outside the Nix store, handed to the unprivileged systemd-networkd via LoadCredential, so files can stay root-only 0600. Deployment must provide them.

Open items

• Web UI (stats and possibly configuration) — to be discussed later. Considerations: read-only stats first is cheap; config editing must write declarative config (Nix/JSON) that feeds the module, not mutate the live system.

Phases

1. MVP — flake with router.* schema: single WAN (DHCP), single LAN, NAT, basic firewall, DHCP + DNS. NixOS VM tests (nixosTest) in CI.
2. Real-world features — IPv6 PD, VLANs, zone policy, port forwards, WireGuard.
3. Images and hardware — installer ISO, read-only root, aarch64.
4. Operations — commit-confirm rollback, monitoring (node exporter, nftables counters), A/B image updates; multi-WAN failover.
5. Optional — web UI (stats, then configuration).