nixos-router/PLAN.md

nixos-router — Plan

Working name: nixos-router. A NixOS-based distribution for routers/firewalls, delivered as a flake exporting NixOS modules and bootable images (not a nixpkgs fork).

Decisions

Area	Decision
Audience	Own homelab first; keep structure clean enough to publish later.
Config model	Opinionated router.* option schema compiled to systemd-networkd + nftables, with build-time assertions.
Hardware	x86_64 UEFI first. aarch64 is a planned target; avoid x86-only assumptions.
v1 features	IPv6 + prefix delegation, VLANs / multiple LANs, WireGuard.
Deferred	Multi-WAN failover, QoS (CAKE/SQM), web UI.
Updates (v1)	Push from workstation (nixos-rebuild --target-host / colmena) + commit-confirm rollback timer.
Updates (later)	A/B image updates with automatic rollback.
Footprint	Minimal closure, read-only root, no Nix daemon/compiler on device, small persistent /var.

Architecture

1. router.* schema — users describe intent (wan, lan, vlans, zones, port forwards, wireguard); modules compile it to:
   • systemd-networkd (DHCPv4/v6, DHCPv6-PD, VLANs, bridges, PPPoE later)
   • nftables zone-based ruleset (generated, not networking.firewall)
   • DHCP / DNS / RA services
2. Build-time validation — Nix assertions reject lock-out or inconsistent configs (no management access rule, overlapping subnets, zone referencing missing interface); generated ruleset checked with nft -c.
3. Images — x86_64 first (installer ISO + raw disk), aarch64 later. Read-only root (erofs/squashfs store), small persistent /var.
4. Deployment — deploy a prebuilt closure (nix copy + switch-to-configuration); never evaluate or build on the router. This keeps the device light and leads naturally to A/B updates.

Implications

• No on-device evaluation: all designs must work from a prebuilt closure.
• Flake exposes x86_64-linux and aarch64-linux outputs from the start; hardware specifics (bootloader, kernel, NIC naming) live in separate boards/ modules.
• IPv6 PD fan-out to multiple VLANs needs stable per-LAN subnet-ID allocation; this is the hardest part of the schema.

Default zone policy (confirmed)

• WAN: deny inbound (except established/related, required ICMPv6, explicit port forwards).
• LAN: allow outbound to WAN.
• Other zones (e.g. IoT): isolated from each other and from LAN unless a rule opens them.

Management access (confirmed)

• Key-only SSH, reachable from LAN zones (trusted, non-WAN, non-isolated zones). No GUI in v1.
• Build-time assertion fails if no management path exists.
• Isolated zones (e.g. IoT) get no router access beyond DHCP/DNS/NDP.

Services (confirmed)

• DHCP: Kea (DHCPv4 + DHCPv6 incl. PD).
• DNS: Unbound.
• RA: systemd-networkd (or radvd if networkd falls short).

Repo layout (confirmed)

flake.nix
modules/     # router.* schema and service modules
boards/      # hardware-specific modules (x86_64 first, aarch64 later)
tests/       # nixosTest VM tests
images/      # image/ISO builds
examples/    # sample configs

Status

VLANs + zones done: vlans VM test (5 VMs: ISP, router, untagged main, VLAN 20 iot, VLAN 30 cams) covers VLAN devices, per-VLAN DHCP/gateway, opt-in internet, DNS vs SSH per LAN, intra-zone forwarding across VLANs, one-directional inter-zone rules and the destination restriction (each mutation-tested). Not covered yet: IPv6 on VLAN LANs (only the PD LAN has IPv6), port forwards from the WAN.

PPPoE done: pppoe VM test = the same topology with a fake ISP running pppoe-server (VLAN 7, CHAP, password containing a quote) — covers session, NAT, MTU, MSS clamp (mutation-tested) and WAN-deny. basic remains the DHCPv4 variant.

IPv6 + prefix delegation done (single /64): router.wan.ipv6.enable and router.lans.<name>.ipv6.pd (at most one LAN, asserted). The VM test now covers the router's DHCPv6 address and delegated /64, SLAAC on the client, routing without NAT66, and WAN-deny over IPv6 (mutation-tested). Still open: RDNSS, other LANs are IPv4-only, DHCPv6 reservations.

Phase 1 (MVP) skeleton done: flake, router.* schema (wan, lans, dns.forwarders), networkd, nftables (zone-less for now: WAN/LAN), Kea DHCPv4, Unbound, assertions. nix build .#checks.x86_64-linux.basic runs a 3-node VM test (upstream/router/client) covering WAN DHCP, LAN DHCP+DNS, NAT, WAN-deny and LAN SSH.

Interactive lab: nix run .#lab (x86_64) opens tmux with one console pane per VM plus the test-driver REPL (router.shell_interact(), client.succeed(...), test_script()). The REPL pane also shows all VM output interleaved.

Learned along the way:

• The default policy now explicitly answers ICMP echo on LANs only; the WAN gets only ICMP error types and ND.
• In NixOS tests fail(...) also passes on "command not found": install probe tools explicitly and add a positive control per negative check.
• Unbound answers .test locally (RFC 6761) and validates DNSSEC; tests using fake domains need both local-zone ... transparent and domain-insecure (both verified necessary; set in the test, not the module).
• modules/router/lib.nix has IPv4 helpers (parse/format/netmask/cidr/overlap). ipv4.subnet is now derived from address + prefixLength (read-only). Assertions also cover pool bounds, pool containing the router address, and overlapping LANs. IPv6 helpers (prefix arithmetic for PD subnet-IDs) are still to do.
• Checks: basic, pppoe and vlans (VM tests), lib (helper unit tests), assertions (broken configs must fail with exact messages). Not yet covered: WireGuard, port forwards, WAN-originated rules.

IPv6 design (confirmed)

• WAN: mirrors the existing router's uplink config — networkd DHCP (v6), IPv6AcceptRA = true, KeepConfiguration = yes, dhcpV6Config = { WithoutRA = "solicit"; UseDelegatedPrefix = true; }. The ISP sends RAs (default route) without DHCPv6 flags, so the router solicits on its own.
• Delegation: the ISP delegates a single /64, which cannot be split. Exactly one LAN may claim it via an explicit flag (router.lans.<name>.ipv6.pd = true); an assertion rejects more than one.
• LANs: SLAAC only (RA, no DHCPv6). All other LANs/VLANs are IPv4-only for now (ULA or NPTv6 can come later).
• Firewall: inet table covers v6; WAN→LAN forwarding stays denied, no NAT66.
• Test: fake ISP on upstream = Kea DHCPv6 (IA_NA + IA_PD /64 from a /48 pool) + networkd RA sender (default route only) + a static route for the delegated prefix back to the router.
• Known gap: no RDNSS in RAs yet (the router's global LAN address is not known at eval time); clients still get DNS via DHCPv4.

PPPoE uplink (confirmed)

• IPv4 via PPPoE (router.wan.ipv4.type = "pppoe"), carried on an optional tagged VLAN on the WAN link (pppoe.vlan); native DHCPv6/PD stays on the untagged link.
• wan.interface is the physical link; the derived wan.ipv4.interface (pppoe-wan or the link) is where NAT, MSS clamping and the IPv4 forwarding rule apply. IPv6 forwarding keeps using the physical link.
• pppd (pppoe.so) runs as router-pppoe.service, ordered after the VLAN device. Credentials: username inline, password from pppoe.passwordFile (type forbids Nix-store paths); the options file is generated into a root-only runtime dir. Deployment must provide that file on persistent storage (or via agenix/sops).
• MTU 1492 + TCP MSS clamp (1452) in both directions on the PPP device; RFC 4638 not implemented.

VLANs and zones (confirmed)

• VLAN devices are created by the router: router.lans.<name> = { interface = <trunk>; vlan = <id>; } makes <trunk>.<id> via networkd. A trunk may also carry one untagged LAN. Assertions: duplicate device, device name > 15 chars, WAN interface reused as trunk.
• Zones are an attribute on the LAN (zone, default = the LAN's own name); LANs sharing a zone may forward to each other, traffic between zones is denied unless a rule in router.firewall.rules allows it (from, to, protocol, ports, optional IPv4 destination that must lie inside a LAN of the destination zone). Rules are one-directional; return traffic via conntrack. Assertions reject unknown zones, from == to, ports without tcp/udp, bad/out-of-zone/host-bit destinations.
• Internet is opt-in per LAN (internet, default false) — this replaces the earlier "LAN allow-out" default; any LAN that should reach the WAN must set it.
• Router services stay per LAN: DNS and DHCP always, SSH only with allowManagement.
• Kea subnet IDs are derived from the network address, so adding or removing LANs does not shift IDs (and orphan persistent leases).

Open items

• Web UI (stats and possibly configuration) — to be discussed later. Considerations: read-only stats first is cheap; config editing must write declarative config (Nix/JSON) that feeds the module, not mutate the live system.

Phases

1. MVP — flake with router.* schema: single WAN (DHCP), single LAN, NAT, basic firewall, DHCP + DNS. NixOS VM tests (nixosTest) in CI.
2. Real-world features — IPv6 PD, VLANs, zone policy, port forwards, WireGuard.
3. Images and hardware — installer ISO, read-only root, aarch64.
4. Operations — commit-confirm rollback, monitoring (node exporter, nftables counters), A/B image updates; multi-WAN failover.
5. Optional — web UI (stats, then configuration).