nixos-router/PLAN.md

nixos-router — Plan

Working name: nixos-router. A NixOS-based distribution for routers/firewalls, delivered as a flake exporting NixOS modules and bootable images (not a nixpkgs fork).

Decisions

Area	Decision
Audience	Own homelab first; keep structure clean enough to publish later.
Config model	Opinionated router.* option schema compiled to systemd-networkd + nftables, with build-time assertions.
Hardware	x86_64 UEFI first. aarch64 is a planned target; avoid x86-only assumptions.
v1 features	IPv6 + prefix delegation, VLANs / multiple LANs, WireGuard.
Deferred	Multi-WAN failover, QoS (CAKE/SQM), web UI.
Updates (v1)	Push from workstation (nixos-rebuild --target-host / colmena) + commit-confirm rollback timer.
Updates (later)	A/B image updates with automatic rollback.
Footprint	Minimal closure, read-only root, no Nix daemon/compiler on device, small persistent /var.

Architecture

1. router.* schema — users describe intent (wan, lan, vlans, zones, port forwards, wireguard); modules compile it to:
   • systemd-networkd (DHCPv4/v6, DHCPv6-PD, VLANs, bridges, PPPoE later)
   • nftables zone-based ruleset (generated, not networking.firewall)
   • DHCP / DNS / RA services
2. Build-time validation — Nix assertions reject lock-out or inconsistent configs (no management access rule, overlapping subnets, zone referencing missing interface); generated ruleset checked with nft -c.
3. Images — x86_64 first (installer ISO + raw disk), aarch64 later. Read-only root (erofs/squashfs store), small persistent /var.
4. Deployment — deploy a prebuilt closure (nix copy + switch-to-configuration); never evaluate or build on the router. This keeps the device light and leads naturally to A/B updates.

Implications

• No on-device evaluation: all designs must work from a prebuilt closure.
• Flake exposes x86_64-linux and aarch64-linux outputs from the start; hardware specifics (bootloader, kernel, NIC naming) live in separate boards/ modules.
• IPv6 PD fan-out to multiple VLANs needs stable per-LAN subnet-ID allocation; this is the hardest part of the schema.

Default zone policy (confirmed)

• WAN: deny inbound (except established/related, required ICMPv6, explicit port forwards).
• LAN: allow outbound to WAN.
• Other zones (e.g. IoT): isolated from each other and from LAN unless a rule opens them.

Management access (confirmed)

• Key-only SSH, reachable from LAN zones (trusted, non-WAN, non-isolated zones). No GUI in v1.
• Build-time assertion fails if no management path exists.
• Isolated zones (e.g. IoT) get no router access beyond DHCP/DNS/NDP.

Services (confirmed)

• DHCP: Kea (DHCPv4 + DHCPv6 incl. PD).
• DNS: Unbound.
• RA: systemd-networkd (or radvd if networkd falls short).

Repo layout (confirmed)

flake.nix
modules/     # router.* schema and service modules
boards/      # hardware-specific modules (x86_64 first, aarch64 later)
tests/       # nixosTest VM tests
images/      # image/ISO builds
examples/    # sample configs

Status

Phase 1 (MVP) skeleton done: flake, router.* schema (wan, lans, dns.forwarders), networkd, nftables (zone-less for now: WAN/LAN), Kea DHCPv4, Unbound, assertions. nix build .#checks.x86_64-linux.basic runs a 3-node VM test (upstream/router/client) covering WAN DHCP, LAN DHCP+DNS, NAT, WAN-deny and LAN SSH.

Interactive lab: nix run .#lab (x86_64) opens tmux with one console pane per VM plus the test-driver REPL (router.shell_interact(), client.succeed(...), test_script()). The REPL pane also shows all VM output interleaved.

Learned along the way:

• The default policy now explicitly answers ICMP echo on LANs only; the WAN gets only ICMP error types and ND.
• In NixOS tests fail(...) also passes on "command not found": install probe tools explicitly and add a positive control per negative check.
• Unbound answers .test locally (RFC 6761) and validates DNSSEC; tests using fake domains need both local-zone ... transparent and domain-insecure (both verified necessary; set in the test, not the module).
• Subnets are given explicitly (subnet + address) for now; a small CIDR helper lib is needed before IPv6 PD / VLAN subnet-ID allocation.
• Not yet covered: IPv6, VLANs, WireGuard, formal zones, a test for the assertions themselves (verified manually via nix eval).

Open items

• Web UI (stats and possibly configuration) — to be discussed later. Considerations: read-only stats first is cheap; config editing must write declarative config (Nix/JSON) that feeds the module, not mutate the live system.

Phases

1. MVP — flake with router.* schema: single WAN (DHCP), single LAN, NAT, basic firewall, DHCP + DNS. NixOS VM tests (nixosTest) in CI.
2. Real-world features — IPv6 PD, VLANs, zone policy, port forwards, WireGuard.
3. Images and hardware — installer ISO, read-only root, aarch64.
4. Operations — commit-confirm rollback, monitoring (node exporter, nftables counters), A/B image updates; multi-WAN failover.
5. Optional — web UI (stats, then configuration).