App 2.41.0, server 2.9.0: recordings reach the server

A finished signal recording goes to an outbox and is sent gzipped with
the car's token (PUT /v1/logs/{name}); the service retries every 2
minutes while the car is offline. The server keeps the newest 200 on a
new volume (LOGS_DIR); reading takes a token of its own (LOGS_TOKEN).
fetch-logs in the dev shell fetches new ones into logs/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LhzAVaUyqcw5PkGjz4mWW6
This commit is contained in:
2026-10-27 10:22:40 +01:00
co-authored by Claude Opus 5.5
parent 16e6f349da
commit eebe840b6a
20 changed files with 840 additions and 13 deletions
+23
View File
@@ -25,6 +25,8 @@ Environment only; the server refuses to start when a required value is missing.
| `FORGE_URL`, `FORGE_REPO`, `FORGE_TOKEN` | optional; app updates (below). The token only needs to read the repository |
| `MAP_TILES_URL` | a TileJSON naming the vector tiles the map's roads come from; default OpenFreeMap's `https://tiles.openfreemap.org/planet`, `off` switches maps off, otherwise https only |
| `MAP_CACHE_DIR` | where fetched map tiles are kept across restarts (the manifest mounts a volume there); empty keeps them in memory only |
| `LOGS_DIR` | where the files the car sends from its Logs tab are kept (the manifest mounts a volume there); empty switches logs off |
| `LOGS_TOKEN` | optional, at least 24 characters and not the `INGEST_TOKEN`; reads those files (below). Empty: the car can send, nobody can read |
| `LISTEN_ADDR` | default `:8080` |
## Endpoints
@@ -43,6 +45,7 @@ with (403 otherwise); the shared token works for any.
- `PUT /v1/charging-sessions/{sessionId}` — a finished charging session (AC or DC, as far as the car was awake) with its curve.
- `GET /v1/consumption/history/{installationId}` — the car reads its own totals back.
- `GET /v1/health` — `{"status":"ok"}`.
- `PUT /v1/logs/{name}` — a file from the car's Logs tab (below).
A malformed request is a 400 naming the field; a database failure is a 500 without detail.
@@ -79,6 +82,26 @@ and is kept, and the car hears 503 with `Retry-After: 10` and gets the tile on i
404 when maps are off. The log says `map tile fetched` with the size, never the tile.
Map data © OpenStreetMap contributors (ODbL), tiles © OpenMapTiles, served by OpenFreeMap.
## Logs from the car
So a signal recording reaches the owner without a USB stick. The car sends each one when it ends
(again later when it is offline) with its own token: `PUT /v1/logs/{name}`, plain UTF-8 text,
gzipped (`Content-Encoding: gzip`) or not; at most 2 MB as sent and 8 MB unpacked. The name is
the app's (`signal-recording-20261027-085240.txt`: lower-case letters, digits and dashes,
`.txt`); the same name again replaces the file, so a retry is harmless. 204 when stored. The
newest 200 files are kept, none older than 180 days.
Reading takes `LOGS_TOKEN`, which can do nothing else, and the car's token cannot read:
```sh
curl -H "Authorization: Bearer $LOGS_TOKEN" https://car.k8s.stinnesbeck.com/v1/logs # {logs: [{name, bytes, received}]}, newest first
curl -H "Authorization: Bearer $LOGS_TOKEN" https://car.k8s.stinnesbeck.com/v1/logs/<name> # the text
```
`fetch-logs` in the dev shell fetches the new ones into `logs/` (token from
`~/.config/nilsmg4/logs-token`). 404 when reading is off. The log says `log received` with the
name and size, never the contents.
## Settings from a phone
The ABRP credentials are long to type on the car. With pairing on, the car (with its own token)
+4
View File
@@ -21,6 +21,7 @@ import (
// /v1/settings/…, /settings settings typed on a phone (handoff_http.go)
// GET /v1/app/… app updates from the forge (releases.go)
// POST /v1/map/roads the roads of one map tile (maps.go)
// /v1/logs/… files from the Logs tab, read with LOGS_TOKEN (logs.go)
//
// A car authenticates with the shared INGEST_TOKEN or with its own token from pairing
// (pairing_http.go).
@@ -34,6 +35,8 @@ type API struct {
Releases *Releases
// Maps serves the roads for the Overview's map from OpenStreetMap vector tiles (maps.go); nil is off.
Maps *Maps
// Logs keeps the files the car sends from its Logs tab (logs.go); nil is off.
Logs *Logs
// PublicURL is the server's address as a phone reaches it, for the pairing QR code;
// derived from the request when empty.
PublicURL string
@@ -54,6 +57,7 @@ func (a *API) Routes() http.Handler {
a.handoffRoutes(mux)
a.releaseRoutes(mux)
a.mapRoutes(mux)
a.logRoutes(mux)
return mux
}
+248
View File
@@ -0,0 +1,248 @@
package main
import (
"bytes"
"compress/gzip"
"crypto/subtle"
"errors"
"io"
"log/slog"
"net/http"
"os"
"path/filepath"
"regexp"
"sort"
"strings"
"sync"
"time"
"unicode/utf8"
)
// Logs keeps the text files the car sends from its Logs tab (signal recordings) on a volume,
// so they reach the owner without a USB stick. The car sends with its ingest token; reading
// needs LOGS_TOKEN, a token of its own, so neither the car's token can read them nor this one
// write anything.
//
// PUT /v1/logs/{name} one file from the car (ingest token); the same name again replaces it
// GET /v1/logs the files, newest first (LOGS_TOKEN)
// GET /v1/logs/{name} one file (LOGS_TOKEN)
//
// The newest maxLogFiles files younger than maxLogAge are kept. A file's contents never go to
// the server log, only its name and size.
type Logs struct {
Dir string
// ReadToken is LOGS_TOKEN; empty switches reading off (the car can still send).
ReadToken string
mu sync.Mutex
now func() time.Time
}
const (
maxLogWireBytes = 2 << 20 // as sent, gzip or not
maxLogBytes = 8 << 20 // as stored
maxLogFiles = 200
maxLogAge = 180 * 24 * time.Hour
)
// logName is what a file may be called: the app's own names, e.g. signal-recording-20261027-085240.txt.
var logName = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,79}\.txt$`)
// NewLogs keeps the files in dir, which must exist.
func NewLogs(dir, readToken string) *Logs {
return &Logs{Dir: dir, ReadToken: readToken, now: time.Now}
}
func (a *API) logRoutes(mux *http.ServeMux) {
if a.Logs == nil {
return
}
mux.HandleFunc("PUT /v1/logs/{name}", a.putLog)
mux.HandleFunc("GET /v1/logs", a.Logs.list)
mux.HandleFunc("GET /v1/logs/{name}", a.Logs.get)
}
func (a *API) putLog(w http.ResponseWriter, r *http.Request) {
if _, ok := a.authenticate(r); !ok {
writeJSON(w, http.StatusUnauthorized, map[string]string{"error": "unauthorized"})
return
}
name := r.PathValue("name")
if !logName.MatchString(name) {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "name must be lower-case letters, digits and dashes, ending in .txt"})
return
}
text, status, msg := readLogBody(r)
if status != 0 {
writeJSON(w, status, map[string]string{"error": msg})
return
}
if err := a.Logs.save(name, text); err != nil {
slog.Error("log not stored", "name", name, "error", err)
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "internal server error"})
return
}
slog.Info("log received", "name", name, "bytes", len(text))
w.WriteHeader(http.StatusNoContent)
}
// readLogBody is the request's text, un-gzipped when it says so; a status and message when it
// is too large, not gzip after all, or not UTF-8 text.
func readLogBody(r *http.Request) ([]byte, int, string) {
wire, err := io.ReadAll(io.LimitReader(r.Body, maxLogWireBytes+1))
if err != nil {
return nil, http.StatusBadRequest, "body could not be read"
}
if len(wire) > maxLogWireBytes {
return nil, http.StatusRequestEntityTooLarge, "body is too large"
}
text := wire
switch strings.ToLower(r.Header.Get("Content-Encoding")) {
case "", "identity":
case "gzip":
zr, err := gzip.NewReader(bytes.NewReader(wire))
if err != nil {
return nil, http.StatusBadRequest, "body is not gzip"
}
text, err = io.ReadAll(io.LimitReader(zr, maxLogBytes+1))
if err != nil {
return nil, http.StatusBadRequest, "body is not gzip"
}
default:
return nil, http.StatusUnsupportedMediaType, "only gzip or no encoding"
}
if len(text) > maxLogBytes {
return nil, http.StatusRequestEntityTooLarge, "body is too large"
}
if !utf8.Valid(text) {
return nil, http.StatusBadRequest, "body is not UTF-8 text"
}
return text, 0, ""
}
// save writes the file whole (a temporary file renamed into place), then prunes.
func (l *Logs) save(name string, text []byte) error {
l.mu.Lock()
defer l.mu.Unlock()
tmp, err := os.CreateTemp(l.Dir, ".incoming-*")
if err != nil {
return err
}
defer os.Remove(tmp.Name()) // gone after the rename; cleans up after a failure
if _, err := tmp.Write(text); err != nil {
tmp.Close()
return err
}
if err := tmp.Close(); err != nil {
return err
}
if err := os.Rename(tmp.Name(), filepath.Join(l.Dir, name)); err != nil {
return err
}
l.prune()
return nil
}
type logEntry struct {
Name string `json:"name"`
Bytes int64 `json:"bytes"`
Received time.Time `json:"received"`
}
// entries are the stored files, newest first.
func (l *Logs) entries() ([]logEntry, error) {
dir, err := os.ReadDir(l.Dir)
if err != nil {
return nil, err
}
var out []logEntry
for _, e := range dir {
if !e.Type().IsRegular() || !logName.MatchString(e.Name()) {
continue
}
info, err := e.Info()
if err != nil {
continue
}
out = append(out, logEntry{Name: e.Name(), Bytes: info.Size(), Received: info.ModTime().UTC()})
}
sort.Slice(out, func(i, j int) bool {
if !out[i].Received.Equal(out[j].Received) {
return out[i].Received.After(out[j].Received)
}
return out[i].Name > out[j].Name
})
return out, nil
}
// prune removes all but the newest maxLogFiles and anything older than maxLogAge.
func (l *Logs) prune() {
entries, err := l.entries()
if err != nil {
return
}
cutoff := l.now().Add(-maxLogAge)
for i, e := range entries {
if i >= maxLogFiles || e.Received.Before(cutoff) {
if err := os.Remove(filepath.Join(l.Dir, e.Name)); err != nil && !errors.Is(err, os.ErrNotExist) {
slog.Warn("old log not removed", "name", e.Name, "error", err)
}
}
}
}
// mayRead accepts LOGS_TOKEN only, compared in constant time; main refuses a short one.
func (l *Logs) mayRead(w http.ResponseWriter, r *http.Request) bool {
if l.ReadToken == "" {
writeJSON(w, http.StatusNotFound, map[string]string{"error": "reading logs is off (no LOGS_TOKEN)"})
return false
}
supplied, ok := strings.CutPrefix(r.Header.Get("Authorization"), "Bearer ")
if !ok || subtle.ConstantTimeCompare([]byte(supplied), []byte(l.ReadToken)) != 1 {
writeJSON(w, http.StatusUnauthorized, map[string]string{"error": "unauthorized"})
return false
}
return true
}
func (l *Logs) list(w http.ResponseWriter, r *http.Request) {
if !l.mayRead(w, r) {
return
}
l.mu.Lock()
entries, err := l.entries()
l.mu.Unlock()
if err != nil {
slog.Error("logs not listed", "error", err)
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "internal server error"})
return
}
writeJSON(w, http.StatusOK, map[string]any{"logs": append([]logEntry{}, entries...)})
}
func (l *Logs) get(w http.ResponseWriter, r *http.Request) {
if !l.mayRead(w, r) {
return
}
name := r.PathValue("name")
if !logName.MatchString(name) {
writeJSON(w, http.StatusNotFound, map[string]string{"error": "no such log"})
return
}
l.mu.Lock()
text, err := os.ReadFile(filepath.Join(l.Dir, name))
l.mu.Unlock()
if errors.Is(err, os.ErrNotExist) {
writeJSON(w, http.StatusNotFound, map[string]string{"error": "no such log"})
return
}
if err != nil {
slog.Error("log not read", "name", name, "error", err)
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "internal server error"})
return
}
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.WriteHeader(http.StatusOK)
_, _ = w.Write(text)
}
+212
View File
@@ -0,0 +1,212 @@
package main
import (
"bytes"
"compress/gzip"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
const testLogsToken = "logs-read-token-0123456789abcdef"
func newLogsAPI(t *testing.T) (*API, string) {
t.Helper()
api, _ := newTestAPI()
dir := t.TempDir()
api.Logs = NewLogs(dir, testLogsToken)
return api, dir
}
func gzipped(t *testing.T, text string) []byte {
t.Helper()
var b bytes.Buffer
zw := gzip.NewWriter(&b)
if _, err := zw.Write([]byte(text)); err != nil {
t.Fatal(err)
}
if err := zw.Close(); err != nil {
t.Fatal(err)
}
return b.Bytes()
}
func putLog(t *testing.T, h http.Handler, name, token string, body []byte, encoding string) *httptest.ResponseRecorder {
t.Helper()
req := httptest.NewRequest("PUT", "/v1/logs/"+name, bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+token)
req.Header.Set("Content-Type", "text/plain; charset=utf-8")
if encoding != "" {
req.Header.Set("Content-Encoding", encoding)
}
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
return rec
}
func TestTheCarSendsALogAndTheOwnerReadsIt(t *testing.T) {
api, _ := newLogsAPI(t)
h := api.Routes()
const name = "signal-recording-20261027-085240.txt"
text := "Signal recording, 37 captures\nlistening to 108 assistance properties\n"
if rec := putLog(t, h, name, testToken, gzipped(t, text), "gzip"); rec.Code != http.StatusNoContent {
t.Fatalf("put: %d %s", rec.Code, rec.Body)
}
rec := do(t, h, "GET", "/v1/logs", testLogsToken, "")
if rec.Code != 200 {
t.Fatalf("list: %d %s", rec.Code, rec.Body)
}
var list struct {
Logs []logEntry `json:"logs"`
}
if err := json.Unmarshal(rec.Body.Bytes(), &list); err != nil || len(list.Logs) != 1 || list.Logs[0].Name != name ||
list.Logs[0].Bytes != int64(len(text)) {
t.Fatalf("list: %v %s", err, rec.Body)
}
rec = do(t, h, "GET", "/v1/logs/"+name, testLogsToken, "")
if rec.Code != 200 || rec.Body.String() != text || !strings.HasPrefix(rec.Header().Get("Content-Type"), "text/plain") {
t.Fatalf("get: %d %q", rec.Code, rec.Body)
}
// Sent again (a retry), it replaces itself.
if rec := putLog(t, h, name, testToken, []byte("again\n"), ""); rec.Code != http.StatusNoContent {
t.Fatalf("put again: %d", rec.Code)
}
if rec := do(t, h, "GET", "/v1/logs/"+name, testLogsToken, ""); rec.Body.String() != "again\n" {
t.Fatalf("replaced: %q", rec.Body)
}
}
func TestLogTokensDoNotCross(t *testing.T) {
api, _ := newLogsAPI(t)
h := api.Routes()
name := "signal-recording-1.txt"
if rec := putLog(t, h, name, testLogsToken, []byte("x"), ""); rec.Code != http.StatusUnauthorized {
t.Fatalf("the read token wrote: %d", rec.Code)
}
if rec := putLog(t, h, name, testToken, []byte("x"), ""); rec.Code != http.StatusNoContent {
t.Fatalf("put: %d", rec.Code)
}
for _, path := range []string{"/v1/logs", "/v1/logs/" + name} {
for _, token := range []string{"", testToken, testLogsToken + "x"} {
if rec := do(t, h, "GET", path, token, ""); rec.Code != http.StatusUnauthorized {
t.Fatalf("GET %s with %q: %d", path, token, rec.Code)
}
}
}
}
func TestReadingIsOffWithoutALogsToken(t *testing.T) {
api, _ := newLogsAPI(t)
api.Logs.ReadToken = ""
h := api.Routes()
if rec := putLog(t, h, "a.txt", testToken, []byte("x"), ""); rec.Code != http.StatusNoContent {
t.Fatalf("put: %d", rec.Code)
}
if rec := do(t, h, "GET", "/v1/logs", "", ""); rec.Code != http.StatusNotFound {
t.Fatalf("list: %d", rec.Code)
}
}
func TestBadLogsAreRefused(t *testing.T) {
api, dir := newLogsAPI(t)
h := api.Routes()
for _, c := range []struct {
name, encoding string
body []byte
want int
}{
{"..%2Fescape.txt", "", []byte("x"), http.StatusBadRequest},
{"Upper.txt", "", []byte("x"), http.StatusBadRequest},
{"no-suffix", "", []byte("x"), http.StatusBadRequest},
{".hidden.txt", "", []byte("x"), http.StatusBadRequest},
{"a.txt", "gzip", []byte("not gzip"), http.StatusBadRequest},
{"a.txt", "br", []byte("x"), http.StatusUnsupportedMediaType},
{"a.txt", "", []byte{0xff, 0xfe, 0x00}, http.StatusBadRequest},
{"a.txt", "", make([]byte, maxLogWireBytes+1), http.StatusRequestEntityTooLarge},
{"a.txt", "gzip", gzipped(t, strings.Repeat("a", maxLogBytes+1)), http.StatusRequestEntityTooLarge},
} {
if rec := putLog(t, h, c.name, testToken, c.body, c.encoding); rec.Code != c.want {
t.Errorf("%s %s: %d, want %d (%s)", c.name, c.encoding, rec.Code, c.want, rec.Body)
}
}
if files, _ := os.ReadDir(dir); len(files) != 0 {
t.Fatalf("stored: %v", files)
}
if rec := do(t, h, "GET", "/v1/logs/missing.txt", testLogsToken, ""); rec.Code != http.StatusNotFound {
t.Fatalf("missing: %d", rec.Code)
}
}
func TestOldAndSurplusLogsArePruned(t *testing.T) {
dir := t.TempDir()
l := NewLogs(dir, testLogsToken)
// The new file gets the real time as its own, so the others are placed before it.
now := time.Now()
old := filepath.Join(dir, "old.txt")
if err := os.WriteFile(old, []byte("x"), 0o600); err != nil {
t.Fatal(err)
}
if err := os.Chtimes(old, now.Add(-maxLogAge-time.Hour), now.Add(-maxLogAge-time.Hour)); err != nil {
t.Fatal(err)
}
for i := 0; i < maxLogFiles+5; i++ {
p := filepath.Join(dir, fmt.Sprintf("f-%03d.txt", i))
if err := os.WriteFile(p, []byte("x"), 0o600); err != nil {
t.Fatal(err)
}
at := now.Add(-time.Duration(maxLogFiles+5-i) * time.Minute)
if err := os.Chtimes(p, at, at); err != nil {
t.Fatal(err)
}
}
if err := l.save("new.txt", []byte("y")); err != nil {
t.Fatal(err)
}
entries, _ := l.entries()
if len(entries) != maxLogFiles || entries[0].Name != "new.txt" {
t.Fatalf("%d kept, first %s", len(entries), entries[0].Name)
}
if _, err := os.Stat(old); !os.IsNotExist(err) {
t.Fatal("the old one is still there")
}
if _, err := os.Stat(filepath.Join(dir, "f-000.txt")); !os.IsNotExist(err) {
t.Fatal("the oldest surplus one is still there")
}
}
func TestLogsConfig(t *testing.T) {
base := map[string]string{"INFLUX_URL": "http://i", "INFLUX_TOKEN": "t", "INFLUX_ORG": "o", "INGEST_TOKEN": testToken}
env := func(extra map[string]string) func(string) string {
return func(k string) string {
if v, ok := extra[k]; ok {
return v
}
return base[k]
}
}
for _, c := range []struct {
extra map[string]string
ok bool
}{
{map[string]string{}, true},
{map[string]string{"LOGS_DIR": "/l"}, true},
{map[string]string{"LOGS_DIR": "/l", "LOGS_TOKEN": testLogsToken}, true},
{map[string]string{"LOGS_TOKEN": testLogsToken}, false},
{map[string]string{"LOGS_DIR": "/l", "LOGS_TOKEN": "short"}, false},
{map[string]string{"LOGS_DIR": "/l", "LOGS_TOKEN": testToken}, false},
} {
if _, err := loadConfig(env(c.extra)); (err == nil) != c.ok {
t.Errorf("%v: %v", c.extra, err)
}
}
}
+23 -1
View File
@@ -36,6 +36,8 @@ type config struct {
forgeToken string
mapTilesURL string
mapCacheDir string
logsDir string
logsToken string
}
// loadConfig reads the environment and fails closed: a missing InfluxDB setting or a
@@ -55,6 +57,8 @@ func loadConfig(getenv func(string) string) (config, error) {
forgeToken: getenv("FORGE_TOKEN"),
mapTilesURL: getenv("MAP_TILES_URL"),
mapCacheDir: getenv("MAP_CACHE_DIR"),
logsDir: getenv("LOGS_DIR"),
logsToken: getenv("LOGS_TOKEN"),
}
if c.listenAddr == "" {
c.listenAddr = ":8080"
@@ -89,6 +93,17 @@ func loadConfig(getenv func(string) string) (config, error) {
return config{}, errors.New("MAP_TILES_URL must be an https URL or \"off\"")
}
}
// Logs from the car are optional (LOGS_DIR); reading them needs a token of their own, as
// long as the ingest token, and never the ingest token itself.
if c.logsToken != "" && c.logsDir == "" {
return config{}, errors.New("LOGS_TOKEN is set but LOGS_DIR is not")
}
if c.logsToken != "" && len(c.logsToken) < minIngestTokenLength {
return config{}, fmt.Errorf("LOGS_TOKEN must be at least %d characters", minIngestTokenLength)
}
if c.logsToken != "" && c.logsToken == c.ingestToken {
return config{}, errors.New("LOGS_TOKEN must differ from INGEST_TOKEN")
}
if c.approvalPass != "" && len(c.approvalPass) < minApprovalPasswordLength {
return config{}, fmt.Errorf("APPROVAL_PASSWORD must be at least %d characters", minApprovalPasswordLength)
}
@@ -127,6 +142,13 @@ func main() {
}
api.Maps = NewMaps(cfg.mapTilesURL, cfg.mapCacheDir, mapUserAgent)
}
if cfg.logsDir != "" {
if err := os.MkdirAll(cfg.logsDir, 0o700); err != nil {
slog.Error("logs directory not usable", "error", err)
os.Exit(1)
}
api.Logs = NewLogs(cfg.logsDir, cfg.logsToken)
}
srv := &http.Server{
Addr: cfg.listenAddr,
@@ -142,7 +164,7 @@ func main() {
go func() {
slog.Info("listening", "addr", cfg.listenAddr, "bucket", cfg.influxBucket,
"pairing", pairing.Enabled(), "pairedDevices", len(devices), "appUpdates", releases.enabled(),
"maps", api.Maps != nil)
"maps", api.Maps != nil, "logs", api.Logs != nil, "logsReadable", api.Logs != nil && cfg.logsToken != "")
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
slog.Error("server failed", "error", err)
os.Exit(1)