nixos-router/PLAN.md

# nixos-router — Plan

Working name: `nixos-router`. A NixOS-based distribution for routers/firewalls, delivered as a flake exporting NixOS modules and bootable images (not a nixpkgs fork).

## Decisions

| Area | Decision |
|---|---|
| Audience | Own homelab first; keep structure clean enough to publish later. |
| Config model | Opinionated `router.*` option schema compiled to systemd-networkd + nftables, with build-time assertions. |
| Hardware | x86_64 UEFI first. aarch64 is a planned target; avoid x86-only assumptions. |
| v1 features | IPv6 + prefix delegation, VLANs / multiple LANs, WireGuard. |
| Deferred | Multi-WAN failover, QoS (CAKE/SQM), web UI. |
| Updates (v1) | Push from workstation (`nixos-rebuild --target-host` / colmena) + commit-confirm rollback timer. |
| Updates (later) | **A/B image updates** with automatic rollback. |
| Footprint | Minimal closure, read-only root, no Nix daemon/compiler on device, small persistent `/var`. |

## Architecture

1. **`router.*` schema** — users describe intent (wan, lan, vlans, zones, port forwards, wireguard); modules compile it to:
   - systemd-networkd (DHCPv4/v6, DHCPv6-PD, VLANs, bridges, PPPoE later)
   - nftables zone-based ruleset (generated, not `networking.firewall`)
   - DHCP / DNS / RA services
2. **Build-time validation** — Nix assertions reject lock-out or inconsistent configs (no management access rule, overlapping subnets, zone referencing missing interface); generated ruleset checked with `nft -c`.
3. **Images** — x86_64 first (installer ISO + raw disk), aarch64 later. Read-only root (erofs/squashfs store), small persistent `/var`.
4. **Deployment** — deploy a prebuilt closure (`nix copy` + `switch-to-configuration`); never evaluate or build on the router. This keeps the device light and leads naturally to A/B updates.

## Implications

- No on-device evaluation: all designs must work from a prebuilt closure.
- Flake exposes `x86_64-linux` and `aarch64-linux` outputs from the start; hardware specifics (bootloader, kernel, NIC naming) live in separate `boards/` modules.
- IPv6 PD fan-out to multiple VLANs needs stable per-LAN subnet-ID allocation; this is the hardest part of the schema.

## Default zone policy (confirmed)

- WAN: deny inbound (except established/related, required ICMPv6, explicit port forwards).
- LAN: allow outbound to WAN.
- Other zones (e.g. IoT): isolated from each other and from LAN unless a rule opens them.

## Management access (confirmed)

- Key-only SSH, reachable from LAN zones (trusted, non-WAN, non-isolated zones). No GUI in v1.
- Build-time assertion fails if no management path exists.
- Isolated zones (e.g. IoT) get no router access beyond DHCP/DNS/NDP.

## Services (confirmed)

- DHCP: Kea (DHCPv4 + DHCPv6 incl. PD).
- DNS: Unbound.
- RA: systemd-networkd (or radvd if networkd falls short).

## Repo layout (confirmed)

```
flake.nix
modules/     # router.* schema and service modules
boards/      # hardware-specific modules (x86_64 first, aarch64 later)
tests/       # nixosTest VM tests
images/      # image/ISO builds
examples/    # sample configs
```

## Status

**WireGuard done**: `wireguard` VM test (5 VMs: ISP/provider, router, laptop, remote site, LAN host) covers handshakes, road-warrior DNS/SSH over IPv4 and IPv6, zone rules between tunnels and LAN, site-to-site routing without NAT or transfer net, provider tunnel with masquerade and IPv6 transfer net, and the kill switch (kill switch and masquerade mutation-tested). Not covered: preshared keys, `mtu`, tunnels with several peers, key rotation.

**VLANs + zones done**: `vlans` VM test (5 VMs: ISP, router, untagged `main`, VLAN 20 `iot`, VLAN 30 `cams`) covers VLAN devices, per-VLAN DHCP/gateway, opt-in internet, DNS vs SSH per LAN, intra-zone forwarding across VLANs, one-directional inter-zone rules and the destination restriction (each mutation-tested). Not covered yet: IPv6 on VLAN LANs (only the PD LAN has IPv6), port forwards from the WAN.

**PPPoE done**: `pppoe` VM test = the same topology with a fake ISP running `pppoe-server` (VLAN 7, CHAP, password containing a quote) — covers session, NAT, MTU, MSS clamp (mutation-tested) and WAN-deny. `basic` remains the DHCPv4 variant.

**IPv6 + prefix delegation done (single /64)**: `router.wan.ipv6.enable` and `router.lans.<name>.ipv6.pd` (at most one LAN, asserted). The VM test now covers the router's DHCPv6 address and delegated /64, SLAAC on the client, routing without NAT66, and WAN-deny over IPv6 (mutation-tested). Still open: RDNSS, other LANs are IPv4-only, DHCPv6 reservations.

**Phase 1 (MVP) skeleton done**: flake, `router.*` schema (wan, lans, dns.forwarders), networkd, nftables (zone-less for now: WAN/LAN), Kea DHCPv4, Unbound, assertions. `nix build .#checks.x86_64-linux.basic` runs a 3-node VM test (upstream/router/client) covering WAN DHCP, LAN DHCP+DNS, NAT, WAN-deny and LAN SSH.

Interactive lab: `nix run .#lab` (x86_64) opens tmux with one console pane per VM plus the test-driver REPL (`router.shell_interact()`, `client.succeed(...)`, `test_script()`). The REPL pane also shows all VM output interleaved.

Learned along the way:
- The default policy now explicitly answers ICMP echo on LANs only; the WAN gets only ICMP error types and ND.
- In NixOS tests `fail(...)` also passes on "command not found": install probe tools explicitly and add a positive control per negative check.
- Unbound answers `.test` locally (RFC 6761) and validates DNSSEC; tests using fake domains need both `local-zone ... transparent` and `domain-insecure` (both verified necessary; set in the test, not the module).
- `modules/router/lib.nix` has IPv4 helpers (parse/format/netmask/cidr/overlap). `ipv4.subnet` is now derived from `address` + `prefixLength` (read-only). Assertions also cover pool bounds, pool containing the router address, and overlapping LANs. IPv6 helpers (prefix arithmetic for PD subnet-IDs) are still to do.
- Checks: `basic`, `pppoe`, `vlans` and `wireguard` (VM tests), `lib` (helper unit tests), `assertions` (broken configs must fail with exact messages). Not yet covered: NPTv6.

## IPv6 design (confirmed)

- **WAN**: mirrors the existing router's uplink config — networkd `DHCP` (v6), `IPv6AcceptRA = true`, `KeepConfiguration = yes`, `dhcpV6Config = { WithoutRA = "solicit"; UseDelegatedPrefix = true; }`. The ISP sends RAs (default route) without DHCPv6 flags, so the router solicits on its own.
- **Delegation**: the ISP delegates a single **/64**, which cannot be split. Exactly one LAN may claim it via an explicit flag (`router.lans.<name>.ipv6.pd = true`); an assertion rejects more than one.
- **LANs**: SLAAC only (RA, no DHCPv6). All other LANs/VLANs are **IPv4-only** for now (ULA or NPTv6 can come later).
- **Firewall**: inet table covers v6; WAN→LAN forwarding stays denied, no NAT66.
- **Test**: fake ISP on `upstream` = Kea DHCPv6 (IA_NA + IA_PD /64 from a /48 pool) + networkd RA sender (default route only) + a static route for the delegated prefix back to the router.
- **Known gap**: no RDNSS in RAs yet (the router's global LAN address is not known at eval time); clients still get DNS via DHCPv4.

## PPPoE uplink (confirmed)

- **IPv4 via PPPoE** (`router.wan.ipv4.type = "pppoe"`), carried on an optional tagged **VLAN** on the WAN link (`pppoe.vlan`); native DHCPv6/PD stays on the untagged link.
- `wan.interface` is the physical link; the derived `wan.ipv4.interface` (`pppoe-wan` or the link) is where NAT, MSS clamping and the IPv4 forwarding rule apply. IPv6 forwarding keeps using the physical link.
- pppd (`pppoe.so`) runs as `router-pppoe.service`, ordered after the VLAN device. Credentials: `username` inline, password from `pppoe.passwordFile` (type forbids Nix-store paths); the options file is generated into a root-only runtime dir. **Deployment must provide that file** on persistent storage (or via agenix/sops).
- **MTU 1492 + TCP MSS clamp** (1452) in both directions on the PPP device; RFC 4638 not implemented.

## VLANs and zones (confirmed)

- **VLAN devices** are created by the router: `router.lans.<name> = { interface = <trunk>; vlan = <id>; }` makes `<trunk>.<id>` via networkd. A trunk may also carry one untagged LAN. Assertions: duplicate device, device name > 15 chars, WAN interface reused as trunk.
- **Zones** are an attribute on the LAN (`zone`, default = the LAN's own name); LANs sharing a zone may forward to each other, traffic **between zones is denied** unless a rule in `router.firewall.rules` allows it (`from`, `to`, `protocol`, `ports`, optional IPv4 `destination` that must lie inside a LAN of the destination zone). Rules are one-directional; return traffic via conntrack. Assertions reject unknown zones, from == to, ports without tcp/udp, bad/out-of-zone/host-bit destinations.
- **Internet is opt-in per LAN** (`internet`, default false) — this replaces the earlier "LAN allow-out" default; any LAN that should reach the WAN must set it.
- Router services stay per LAN: DNS and DHCP always, SSH only with `allowManagement`.
- Kea subnet IDs are derived from the network address, so adding or removing LANs does not shift IDs (and orphan persistent leases).

## Port forwards (confirmed)

- Implemented by extending `router.firewall.rules` with a reserved source zone `wan` (no LAN/tunnel may use that zone name; `wan` is never a destination).
- Extra rule fields: `dnat` (IPv4 only, single host destination, tcp/udp), `toPort`, port ranges (`"9000-9002"`), `sources` allow-list (v4/v6 CIDRs), `rateLimit` (`N/second|minute|hour`, per source via nft meter), `log`, `hairpin` (NAT loopback for LANs/tunnels with `internet = true`; masquerades so replies return via the router).
- Safeguards: a rule from `wan` must name a destination, an explicit protocol and its ports, so a typo cannot expose a whole zone.
- IPv6 has no NAT: it is opened only by explicit pinholes (`from = "wan"`, IPv6 `destination`). The prefix is dynamic, so v6 destinations are not checked against the zone.
- Tested in `basic`/`pppoe` (source preserved, toPort, ranges, sources, rate limit, v6 pinhole, hairpin, log lines) and in `assertions`; mutation-tested (source match, hairpin masquerade, rate limit).

## WireGuard (confirmed)

- **Use cases**: road warrior (listening, dual-stack), site-to-site (no transfer network), and a provider VPN the router dials into (dual-stack transfer net, masquerade). All via `router.wireguard.<name>`.
- **Firewall**: a tunnel is a zone member like a LAN (`zone`, `internet`, `allowManagement`, plus `dns`, `masquerade`, `killSwitch`), so `router.firewall.rules` cover it. Safer defaults than LANs: management, DNS and internet are off. A tunnel's peers never see each other (no same-device forwarding).
- **Routing**: by destination only — a peer's `allowedIPs` become routes (no default-route tunnels; `/0` is rejected). Provider-style "policy routing" is exactly this; source-based selectors (`ip rule`) are not implemented.
- **Kill switch** (default on): forwarded packets for a tunnel's `allowedIPs` are dropped if they would leave via an uplink (placed before the established-accept, so flows that were in the tunnel are cut too).
- **Addresses** are explicit (`addresses` list, empty for site-to-site); nothing generated. No IPv6 internet for road warriors (the delegated /64 belongs to a LAN; NPTv6 not implemented).
- **Keys**: `privateKeyFile` (and `presharedKeyFile`) outside the Nix store, handed to the unprivileged systemd-networkd via `LoadCredential`, so files can stay root-only 0600. Deployment must provide them.

## Open items

- **Web UI** (stats and possibly configuration) — to be discussed later. Considerations: read-only stats first is cheap; config editing must write declarative config (Nix/JSON) that feeds the module, not mutate the live system.

## Phases

1. **MVP** — flake with `router.*` schema: single WAN (DHCP), single LAN, NAT, basic firewall, DHCP + DNS. NixOS VM tests (`nixosTest`) in CI.
2. **Real-world features** — IPv6 PD, VLANs, zone policy, port forwards, WireGuard.
3. **Images and hardware** — installer ISO, read-only root, aarch64.
4. **Operations** — commit-confirm rollback, monitoring (node exporter, nftables counters), A/B image updates; multi-WAN failover.
5. **Optional** — web UI (stats, then configuration).