nixos-router/PLAN.md

# nixos-router — Plan

Working name: `nixos-router`. A NixOS-based distribution for routers/firewalls, delivered as a flake exporting NixOS modules and bootable images (not a nixpkgs fork).

## Decisions

| Area | Decision |
|---|---|
| Audience | Own homelab first; keep structure clean enough to publish later. |
| Config model | Opinionated `router.*` option schema compiled to systemd-networkd + nftables, with build-time assertions. |
| Hardware | x86_64 UEFI first. aarch64 is a planned target; avoid x86-only assumptions. |
| v1 features | IPv6 + prefix delegation, VLANs / multiple LANs, WireGuard. |
| Deferred | Multi-WAN failover, QoS (CAKE/SQM), web UI. |
| Updates (v1) | Push from workstation (`nixos-rebuild --target-host` / colmena) + commit-confirm rollback timer. |
| Updates (later) | **A/B image updates** with automatic rollback. |
| Footprint | Minimal closure, read-only root, no Nix daemon/compiler on device, small persistent `/var`. |

## Architecture

1. **`router.*` schema** — users describe intent (wan, lan, vlans, zones, port forwards, wireguard); modules compile it to:
   - systemd-networkd (DHCPv4/v6, DHCPv6-PD, VLANs, bridges, PPPoE later)
   - nftables zone-based ruleset (generated, not `networking.firewall`)
   - DHCP / DNS / RA services
2. **Build-time validation** — Nix assertions reject lock-out or inconsistent configs (no management access rule, overlapping subnets, zone referencing missing interface); generated ruleset checked with `nft -c`.
3. **Images** — x86_64 first (installer ISO + raw disk), aarch64 later. Read-only root (erofs/squashfs store), small persistent `/var`.
4. **Deployment** — deploy a prebuilt closure (`nix copy` + `switch-to-configuration`); never evaluate or build on the router. This keeps the device light and leads naturally to A/B updates.

## Implications

- No on-device evaluation: all designs must work from a prebuilt closure.
- Flake exposes `x86_64-linux` and `aarch64-linux` outputs from the start; hardware specifics (bootloader, kernel, NIC naming) live in separate `boards/` modules.
- IPv6 PD fan-out to multiple VLANs needs stable per-LAN subnet-ID allocation; this is the hardest part of the schema.

## Default zone policy (confirmed)

- WAN: deny inbound (except established/related, required ICMPv6, explicit port forwards).
- LAN: allow outbound to WAN.
- Other zones (e.g. IoT): isolated from each other and from LAN unless a rule opens them.

## Management access (confirmed)

- Key-only SSH, reachable from LAN zones (trusted, non-WAN, non-isolated zones). No GUI in v1.
- Build-time assertion fails if no management path exists.
- Isolated zones (e.g. IoT) get no router access beyond DHCP/DNS/NDP.

## Services (confirmed)

- DHCP: Kea (DHCPv4 + DHCPv6 incl. PD).
- DNS: Unbound.
- RA: systemd-networkd (or radvd if networkd falls short).

## Repo layout (confirmed)

```
flake.nix
modules/     # router.* schema and service modules
boards/      # hardware-specific modules (x86_64 first, aarch64 later)
tests/       # nixosTest VM tests
images/      # image/ISO builds
examples/    # sample configs
```

## Status

**WireGuard done**: `wireguard` VM test (5 VMs: ISP/provider, router, laptop, remote site, LAN host) covers handshakes, road-warrior DNS/SSH over IPv4 and IPv6, zone rules between tunnels and LAN, site-to-site routing without NAT or transfer net, provider tunnel with masquerade and IPv6 transfer net, and the kill switch (kill switch and masquerade mutation-tested). Not covered: preshared keys, `mtu`, tunnels with several peers, key rotation.

**VLANs + zones done**: `vlans` VM test (5 VMs: ISP, router, untagged `main`, VLAN 20 `iot`, VLAN 30 `cams`) covers VLAN devices, per-VLAN DHCP/gateway, opt-in internet, DNS vs SSH per LAN, intra-zone forwarding across VLANs, one-directional inter-zone rules and the destination restriction (each mutation-tested). Not covered yet: IPv6 on VLAN LANs (only the PD LAN has IPv6), port forwards from the WAN.

**PPPoE done**: `pppoe` VM test = the same topology with a fake ISP running `pppoe-server` (VLAN 7, CHAP, password containing a quote) — covers session, NAT, MTU, MSS clamp (mutation-tested) and WAN-deny. `basic` remains the DHCPv4 variant.

**IPv6 + prefix delegation done (single /64)**: `router.wan.ipv6.enable` and `router.lans.<name>.ipv6.pd` (at most one LAN, asserted). The VM test now covers the router's DHCPv6 address and delegated /64, SLAAC on the client, routing without NAT66, and WAN-deny over IPv6 (mutation-tested). Still open: RDNSS, other LANs are IPv4-only, DHCPv6 reservations.

**Phase 1 (MVP) skeleton done**: flake, `router.*` schema (wan, lans, dns.forwarders), networkd, nftables (zone-less for now: WAN/LAN), Kea DHCPv4, Unbound, assertions. `nix build .#checks.x86_64-linux.basic` runs a 3-node VM test (upstream/router/client) covering WAN DHCP, LAN DHCP+DNS, NAT, WAN-deny and LAN SSH.

Interactive lab: `nix run .#lab` (x86_64) opens tmux with one console pane per VM plus the test-driver REPL (`router.shell_interact()`, `client.succeed(...)`, `test_script()`). The REPL pane also shows all VM output interleaved.

Learned along the way:
- The default policy now explicitly answers ICMP echo on LANs only; the WAN gets only ICMP error types and ND.
- In NixOS tests `fail(...)` also passes on "command not found": install probe tools explicitly and add a positive control per negative check.
- Unbound answers `.test` locally (RFC 6761) and validates DNSSEC; tests using fake domains need both `local-zone ... transparent` and `domain-insecure` (both verified necessary; set in the test, not the module).
- `modules/router/lib.nix` has IPv4 helpers (parse/format/netmask/cidr/overlap). `ipv4.subnet` is now derived from `address` + `prefixLength` (read-only). Assertions also cover pool bounds, pool containing the router address, and overlapping LANs. IPv6 helpers (prefix arithmetic for PD subnet-IDs) are still to do.
- Checks: `basic`, `pppoe`, `onestick`, `rollback`, `dns`, `vlans` and `wireguard` (VM tests), `example` (the example system builds), `lib` (helper unit tests), `assertions` (broken configs must fail with exact messages). Not yet covered: NPTv6.

## IPv6 design (confirmed)

- **WAN**: mirrors the existing router's uplink config — networkd `DHCP` (v6), `IPv6AcceptRA = true`, `KeepConfiguration = yes`, `dhcpV6Config = { WithoutRA = "solicit"; UseDelegatedPrefix = true; }`. The ISP sends RAs (default route) without DHCPv6 flags, so the router solicits on its own.
- **Delegation**: the ISP delegates a single **/64**, which cannot be split. Exactly one LAN may claim it via an explicit flag (`router.lans.<name>.ipv6.pd = true`); an assertion rejects more than one.
- **LANs**: SLAAC only (RA, no DHCPv6). All other LANs/VLANs are **IPv4-only** for now (ULA or NPTv6 can come later).
- **Firewall**: inet table covers v6; WAN→LAN forwarding stays denied, no NAT66.
- **Test**: fake ISP on `upstream` = Kea DHCPv6 (IA_NA + IA_PD /64 from a /48 pool) + networkd RA sender (default route only) + a static route for the delegated prefix back to the router.
- **Known gap**: no RDNSS in RAs yet (the router's global LAN address is not known at eval time); clients still get DNS via DHCPv4.

## PPPoE uplink (confirmed)

- **IPv4 via PPPoE** (`router.wan.ipv4.type = "pppoe"`), carried on an optional tagged **VLAN** on the WAN link (`pppoe.vlan`); native DHCPv6/PD stays on the untagged link.
- `wan.interface` is the physical link; the derived `wan.ipv4.interface` (`pppoe-wan` or the link) is where NAT, MSS clamping and the IPv4 forwarding rule apply. IPv6 forwarding keeps using the physical link.
- pppd (`pppoe.so`) runs as `router-pppoe.service`, ordered after the VLAN device. Credentials: `username` inline, password from `pppoe.passwordFile` (type forbids Nix-store paths); the options file is generated into a root-only runtime dir. **Deployment must provide that file** on persistent storage (or via agenix/sops).
- **MTU 1492 + TCP MSS clamp** (1452) in both directions on the PPP device; RFC 4638 not implemented.

## VLANs and zones (confirmed)

- **VLAN devices** are created by the router: `router.lans.<name> = { interface = <trunk>; vlan = <id>; }` makes `<trunk>.<id>` via networkd. A trunk may also carry one untagged LAN. Assertions: duplicate device, device name > 15 chars, WAN interface reused as trunk.
- **Zones** are an attribute on the LAN (`zone`, default = the LAN's own name); LANs sharing a zone may forward to each other, traffic **between zones is denied** unless a rule in `router.firewall.rules` allows it (`from`, `to`, `protocol`, `ports`, optional IPv4 `destination` that must lie inside a LAN of the destination zone). Rules are one-directional; return traffic via conntrack. Assertions reject unknown zones, from == to, ports without tcp/udp, bad/out-of-zone/host-bit destinations.
- **Internet is opt-in per LAN** (`internet`, default false) — this replaces the earlier "LAN allow-out" default; any LAN that should reach the WAN must set it.
- Router services stay per LAN: DNS and DHCP always, SSH only with `allowManagement`.
- Kea subnet IDs are derived from the network address, so adding or removing LANs does not shift IDs (and orphan persistent leases).

## Port forwards (confirmed)

- Implemented by extending `router.firewall.rules` with a reserved source zone `wan` (no LAN/tunnel may use that zone name; `wan` is never a destination).
- Extra rule fields: `dnat` (IPv4 only, single host destination, tcp/udp), `toPort`, port ranges (`"9000-9002"`), `sources` allow-list (v4/v6 CIDRs), `rateLimit` (`N/second|minute|hour`, per source via nft meter), `log`, `hairpin` (NAT loopback for LANs/tunnels with `internet = true`; masquerades so replies return via the router).
- Safeguards: a rule from `wan` must name a destination, an explicit protocol and its ports, so a typo cannot expose a whole zone.
- IPv6 has no NAT: it is opened only by explicit pinholes (`from = "wan"`, IPv6 `destination`). The prefix is dynamic, so v6 destinations are not checked against the zone.
- Tested in `basic`/`pppoe` (source preserved, toPort, ranges, sources, rate limit, v6 pinhole, hairpin, log lines) and in `assertions`; mutation-tested (source match, hairpin masquerade, rate limit).

## Deployment and rollback (confirmed)

- **Router on a stick**: `wan.interface` may also carry LANs, but only for PPPoE on a VLAN with IPv6 disabled (the untagged link is then free for a LAN; native IPv6 needs an untagged uplink). In that mode only the PPP device is an uplink in the firewall, so LAN traffic can never match WAN rules; the WAN VLAN is created from the LAN's/trunk's networkd unit. A LAN VLAN may not reuse the PPPoE VLAN id.
- **Commit-confirm** (`router.rollback`, on by default): `router-arm [time]` starts a transient timer; `router-rollback` re-activates the confirmed generation (`/nix/var/nix/profiles/system`, else `/run/booted-system`) with `switch-to-configuration test`; `router-confirm` stops the timer, sets the profile and the boot default. Deploy with `nix run .#deploy -- <flake-attr> <user@host> [--yes]`: build, arm, `nixos-rebuild test --target-host`, check SSH, confirm. A lost connection means the timer rolls back by itself; a reboot also returns to the confirmed generation.
- `systemd.network.wait-online` waits only for the LAN devices (not the uplink) by default: an unplugged uplink must not block boot or activation, while VLAN devices still exist before Kea starts (anyInterface broke this; caught by `vlans`).
- `examples/desktop` + `nixosConfigurations.example` is the deployable starting point (single NIC `eno1`, PPPoE VLAN 1001); it is built as a flake check. Secrets stay on the device (`/var/lib/router/pppoe-password`).
- Open for real hardware: Nix is still on the device (`router-confirm` uses `nix-env`); no image/read-only root yet.

## Local DNS and DHCP reservations (confirmed)

Needed to replace the user's hand-written router (host `antigua`, dnsmasq). Services such as k3s, Docker and Gitea stay on other hosts, so the router has no input rules for them.

- `router.dns.domain`: local domain, authoritative on the router (`static` zone; unknown names answer NXDOMAIN, never forwarded).
- `router.dns.hosts`: name to IPv4/IPv6 addresses; short names live in the domain, full names override public DNS for that name only (split horizon). Reverse (PTR) records are generated.
- `router.dns.overrides`: like hosts but wildcard (Unbound `redirect` zone).
- `router.dns.forwardZones`: zone to resolvers; these and the local domain are `domain-insecure` (private zones under possibly signed parents).
- `lans.<n>.dhcp.reservations.<hostname> = { mac; address; }`: Kea host reservations; the hostname is published in DNS. Assertions: valid MAC/hostname, address inside the LAN and not the router's, no duplicate MACs/addresses, no clash with `dns.hosts`.
- Tunnels with `dns = true` answer on the LAN addresses too: Unbound's access control admits what the tunnel's peers route (`allowedIPs`), so a site-to-site peer without a transfer address can use the resolver.
- Tests: `dns` (VM), site-to-site DNS in `wireguard`, and assertion cases; mutation-tested.
- Still missing for the `antigua` migration: IPv6 over the PPP session (ipv6cp, DHCPv6-PD on the PPP device), an extra static ULA address with RA on the LAN, tunnels without own addresses/fully trusted tunnels (partly covered), and the PPP device name (`wan` there, `pppoe-wan` here: `ddclient` must follow).

## WireGuard (confirmed)

- **Use cases**: road warrior (listening, dual-stack), site-to-site (no transfer network), and a provider VPN the router dials into (dual-stack transfer net, masquerade). All via `router.wireguard.<name>`.
- **Firewall**: a tunnel is a zone member like a LAN (`zone`, `internet`, `allowManagement`, plus `dns`, `masquerade`, `killSwitch`), so `router.firewall.rules` cover it. Safer defaults than LANs: management, DNS and internet are off. A tunnel's peers never see each other (no same-device forwarding).
- **Routing**: by destination only — a peer's `allowedIPs` become routes (no default-route tunnels; `/0` is rejected). Provider-style "policy routing" is exactly this; source-based selectors (`ip rule`) are not implemented.
- **Kill switch** (default on): forwarded packets for a tunnel's `allowedIPs` are dropped if they would leave via an uplink (placed before the established-accept, so flows that were in the tunnel are cut too).
- **Addresses** are explicit (`addresses` list, empty for site-to-site); nothing generated. No IPv6 internet for road warriors (the delegated /64 belongs to a LAN; NPTv6 not implemented).
- **Keys**: `privateKeyFile` (and `presharedKeyFile`) outside the Nix store, handed to the unprivileged systemd-networkd via `LoadCredential`, so files can stay root-only 0600. Deployment must provide them.

## Replacing `antigua` (the user's hand-written router)

Source: the user's infra repo (host `antigua`, NixOS, dnsmasq + networkd + pppd). It is also a k3s/Docker/Gitea host today; **those services will stay on antigua / move elsewhere, so the router does not need input rules for them or trusted container interfaces.** The new box is set up next to antigua and tested by swapping the two.

Done: PPPoE on VLAN 1001 over the shared NIC (`eno1`, LAN untagged), DNS records/domain/forward zones/DHCP reservations/DNS for site-to-site peers, WireGuard (freifunk dial-out with masquerade, family site-to-site with PSK, mobile-clients road warrior), MSS clamp.

Open (in rough order):
1. **IPv6 over the PPP session**: antigua runs `+ipv6` and DHCPv6-PD on the PPP device itself (not on an untagged link); the module only knows native IPv6 on the uplink. Needs ipv6cp in pppd, networkd DHCPv6/RA on the PPP device, PD to the LAN.
2. **ULA on the LAN** (`fd2c:cdef:cde9::1/64`, announced by RA): needs an extra static address / ULA option per LAN.
3. **Tunnel trust and addressing**: antigua trusts `family`, `freifunk`, `mobile-clients` fully and gives two of them no address; check the schema covers this (`allowManagement`, `dns`, `internet` per tunnel; `addresses` may be empty?).
4. **PPP device name**: `wan` on antigua, `pppoe-wan` here; `ddclient` (Cloudflare, uses the interface address) must follow, or make the name configurable.
5. **Secrets**: the PPPoE credentials are plaintext in antigua's repo/store (rotate the password); here they are read from `/var/lib/router/pppoe-password`.
6. DDNS (ddclient) is not part of the module yet; configure it in the host config for now.

## Installer and image (in progress)

- **Delivery order**: installer ISO first (set up a second box next to antigua), `image.repart` disk image + A/B updates later.
- `router-install [--yes] [--secrets DIR] <disk>`: GPT with an ESP (label `boot`) and an ext4 root (label `nixos`), `nixos-install --system <prebuilt toplevel>` (nothing is built on the box), secrets copied to `/var/lib/router`. The layout contract is `nixosModules.installed-layout` (labels, systemd-boot without EFI variables so it boots via the removable fallback path, generic initrd modules, redistributable firmware); target configs import it.
- `lib.mkInstaller { configuration }` builds the ISO for a given `nixosSystem`; the live system runs sshd with that system's root keys. `packages.x86_64-linux.installer-example` is the example's ISO.
- Test `installer` (VM): the nixpkgs installer-test pattern (an installer VM writes `target.qcow2`, a second VM boots it with UEFI); checks refusal of bad input/mounted disks, the fallback boot file, router services up, secrets owned by root with 0700/0600. Mutation-tested (secret permissions, mounted-disk guard).
- Status: `nix build .#installer-example` produces a 1.6 GB ISO. **Not yet tested**: booting that ISO on real firmware (the VM test exercises `router-install`, not the live ISO), the NIC name in the example, and installing over an existing system.
- Build: `nix build .#installer-example`, write `result/iso/*.iso` to a USB stick, boot the new box (UEFI), `scp -r secrets root@<box>:` then `router-install --secrets /root/secrets /dev/<disk>`.

## Open items

- **Web UI** (stats and possibly configuration) — to be discussed later. Considerations: read-only stats first is cheap; config editing must write declarative config (Nix/JSON) that feeds the module, not mutate the live system.

## Phases

1. **MVP** — flake with `router.*` schema: single WAN (DHCP), single LAN, NAT, basic firewall, DHCP + DNS. NixOS VM tests (`nixosTest`) in CI.
2. **Real-world features** — IPv6 PD, VLANs, zone policy, port forwards, WireGuard.
3. **Images and hardware** — installer ISO, read-only root, aarch64.
4. **Operations** — commit-confirm rollback, monitoring (node exporter, nftables counters), A/B image updates; multi-WAN failover.
5. **Optional** — web UI (stats, then configuration).