App 2.39.1, server 2.8.0: map roads from OpenFreeMap, on demand
The public Overpass answered 429/502 and needs ~40 s a tile while the car gives up after 8 s, so the server cancelled its queries and the car's retries added load. Now the server fetches only the tiles a car asks for, as OpenStreetMap vector tiles from OpenFreeMap (MAP_TILES_URL), decodes the transportation layer with a small reader of its own, and keeps each tile 30 days on a volume (MAP_CACHE_DIR). A fetch finishes even after the car has stopped waiting; the car hears 503 with Retry-After and gets the tile on its next try. App: OpenFreeMap/OpenMapTiles attribution on the map, retry after 15 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LhzAVaUyqcw5PkGjz4mWW6
This commit is contained in:
Claude Opus 5.5
+18
-12
@@ -23,7 +23,8 @@ Environment only; the server refuses to start when a required value is missing.
|
||||
| `APPROVAL_PASSWORD` | optional, at least 12 characters; switches pairing on (see below) |
|
||||
| `PUBLIC_URL` | the address a phone uses, for the pairing QR code; derived from the request if empty |
|
||||
| `FORGE_URL`, `FORGE_REPO`, `FORGE_TOKEN` | optional; app updates (below). The token only needs to read the repository |
|
||||
| `OVERPASS_URL` | the Overpass API the map's roads come from; default `https://overpass-api.de/api/interpreter`, `off` switches maps off, otherwise https only |
|
||||
| `MAP_TILES_URL` | a TileJSON naming the vector tiles the map's roads come from; default OpenFreeMap's `https://tiles.openfreemap.org/planet`, `off` switches maps off, otherwise https only |
|
||||
| `MAP_CACHE_DIR` | where fetched map tiles are kept across restarts (the manifest mounts a volume there); empty keeps them in memory only |
|
||||
| `LISTEN_ADDR` | default `:8080` |
|
||||
|
||||
## Endpoints
|
||||
@@ -60,18 +61,23 @@ download cut short`. The forge's address and token never appear in the log.
|
||||
|
||||
## Map roads
|
||||
|
||||
`POST /v1/map/roads` `{z, x, y}`, with the car's token: the drivable roads (motorway to
|
||||
residential) of one web-mercator tile at zoom 14, for the app's "Grid with map" style. The car
|
||||
names the tile, not its position, and in the body, so it stays out of access logs. The answer is
|
||||
`POST /v1/map/roads` `{z, x, y}`, with the car's token: the drivable roads (motorway to minor
|
||||
streets) of one web-mercator tile at zoom 14, for the app's "Grid with map" style. The car names
|
||||
the tile, not its position, and in the body, so it stays out of access logs. The answer is
|
||||
`{z, x, y, extent: 4096, ways: [{c, p: [x, y, …]}]}`: per road its class (0 motorway and trunk,
|
||||
1 primary and secondary, 2 tertiary and unclassified, 3 residential and living streets) and its
|
||||
points in tile units, simplified to about 1.5 m; gzipped when the car accepts it (20–30 KB a tile
|
||||
in a city before that). The roads come from OpenStreetMap through `OVERPASS_URL`, one request at
|
||||
a time and at most one a second, with a User-Agent naming this service, and are kept in memory
|
||||
for 30 days (up to 3000 tiles); the car keeps its own copy too. 400 for anything but zoom 14 or
|
||||
a tile outside it, 503 when Overpass is busy or unreachable (not cached, the car tries again a
|
||||
minute later), 404 when maps are off. The log says `map tile fetched` with the car's installation
|
||||
and the size, never the tile. Map data © OpenStreetMap contributors, ODbL.
|
||||
1 primary and secondary, 2 tertiary, 3 minor streets) and its points in tile units, simplified to
|
||||
about 1.5 m; gzipped when the car accepts it (10–15 KB a tile in a city before that).
|
||||
|
||||
The roads are fetched **only for the tiles a car asks for**: the OpenStreetMap vector tile
|
||||
(OpenMapTiles schema) of that square from the TileJSON at `MAP_TILES_URL` (by default
|
||||
OpenFreeMap: free, no key, no request limits; its template moves with each weekly build and is
|
||||
looked up every 6 hours), decoded by a small reader of our own (`mvt.go`, the `transportation`
|
||||
layer only). Each tile is fetched once and kept for 30 days in `MAP_CACHE_DIR` (and the last 300
|
||||
in memory); at most two fetches run at once. A request waits up to 6 s; a slower fetch goes on
|
||||
and is kept, and the car hears 503 with `Retry-After: 10` and gets the tile on its next try.
|
||||
400 for anything but zoom 14 or a tile outside it, 503 when the tile host fails (not cached),
|
||||
404 when maps are off. The log says `map tile fetched` with the size, never the tile.
|
||||
Map data © OpenStreetMap contributors (ODbL), tiles © OpenMapTiles, served by OpenFreeMap.
|
||||
|
||||
## Settings from a phone
|
||||
|
||||
|
||||
+1
-1
@@ -32,7 +32,7 @@ type API struct {
|
||||
Handoff *Handoff
|
||||
// Releases relays the app's releases from the private forge to the car (releases.go).
|
||||
Releases *Releases
|
||||
// Maps serves the roads for the Overview's map from OpenStreetMap (maps.go); nil is off.
|
||||
// Maps serves the roads for the Overview's map from OpenStreetMap vector tiles (maps.go); nil is off.
|
||||
Maps *Maps
|
||||
// PublicURL is the server's address as a phone reaches it, for the pairing QR code;
|
||||
// derived from the request when empty.
|
||||
|
||||
+19
-11
@@ -34,7 +34,8 @@ type config struct {
|
||||
forgeURL string
|
||||
forgeRepo string
|
||||
forgeToken string
|
||||
overpassURL string
|
||||
mapTilesURL string
|
||||
mapCacheDir string
|
||||
}
|
||||
|
||||
// loadConfig reads the environment and fails closed: a missing InfluxDB setting or a
|
||||
@@ -52,7 +53,8 @@ func loadConfig(getenv func(string) string) (config, error) {
|
||||
forgeURL: getenv("FORGE_URL"),
|
||||
forgeRepo: getenv("FORGE_REPO"),
|
||||
forgeToken: getenv("FORGE_TOKEN"),
|
||||
overpassURL: getenv("OVERPASS_URL"),
|
||||
mapTilesURL: getenv("MAP_TILES_URL"),
|
||||
mapCacheDir: getenv("MAP_CACHE_DIR"),
|
||||
}
|
||||
if c.listenAddr == "" {
|
||||
c.listenAddr = ":8080"
|
||||
@@ -75,16 +77,16 @@ func loadConfig(getenv func(string) string) (config, error) {
|
||||
if c.forgeURL != "" && !strings.HasPrefix(c.forgeURL, "https://") {
|
||||
return config{}, errors.New("FORGE_URL must be an https URL")
|
||||
}
|
||||
// The map's roads come from an Overpass API: the public one unless another is named, none
|
||||
// with "off". Only https, like the forge.
|
||||
switch c.overpassURL {
|
||||
// The map's roads come from vector tiles named by a TileJSON: OpenFreeMap unless another is
|
||||
// named, none with "off". Only https, like the forge.
|
||||
switch c.mapTilesURL {
|
||||
case "":
|
||||
c.overpassURL = DefaultOverpassURL
|
||||
c.mapTilesURL = DefaultMapTilesURL
|
||||
case "off":
|
||||
c.overpassURL = ""
|
||||
c.mapTilesURL = ""
|
||||
default:
|
||||
if !strings.HasPrefix(c.overpassURL, "https://") {
|
||||
return config{}, errors.New("OVERPASS_URL must be an https URL or \"off\"")
|
||||
if !strings.HasPrefix(c.mapTilesURL, "https://") {
|
||||
return config{}, errors.New("MAP_TILES_URL must be an https URL or \"off\"")
|
||||
}
|
||||
}
|
||||
if c.approvalPass != "" && len(c.approvalPass) < minApprovalPasswordLength {
|
||||
@@ -116,8 +118,14 @@ func main() {
|
||||
releases := &Releases{ForgeURL: cfg.forgeURL, Repo: cfg.forgeRepo, Token: cfg.forgeToken}
|
||||
api := &API{Store: store, IngestToken: cfg.ingestToken, Pairing: pairing, Handoff: NewHandoff(),
|
||||
Releases: releases, PublicURL: cfg.publicURL}
|
||||
if cfg.overpassURL != "" {
|
||||
api.Maps = NewMaps(cfg.overpassURL, mapUserAgent)
|
||||
if cfg.mapTilesURL != "" {
|
||||
if cfg.mapCacheDir != "" {
|
||||
if err := os.MkdirAll(cfg.mapCacheDir, 0o700); err != nil {
|
||||
slog.Error("map cache directory not usable", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
api.Maps = NewMaps(cfg.mapTilesURL, cfg.mapCacheDir, mapUserAgent)
|
||||
}
|
||||
|
||||
srv := &http.Server{
|
||||
|
||||
+263
-195
@@ -12,53 +12,69 @@ import (
|
||||
"math"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Maps serves the roads around the car for the Overview's map (the "Grid with map" style),
|
||||
// taken from OpenStreetMap through an Overpass API and cached here, so the car only ever
|
||||
// talks to this server and asks Overpass nothing itself.
|
||||
// Maps serves the roads around the car for the Overview's map (the "Grid with map" style). It
|
||||
// fetches only the tiles a car asks for, as OpenStreetMap vector tiles (OpenMapTiles schema,
|
||||
// by default from OpenFreeMap), keeps their roads and stores them, so each tile is fetched
|
||||
// once and the car only ever talks to this server.
|
||||
//
|
||||
// POST /v1/map/roads {"z":14,"x":…,"y":…}
|
||||
// → {"z","x","y","extent":4096,"ways":[{"c":class,"p":[x,y,x,y,…]}]}
|
||||
//
|
||||
// The car names a web-mercator tile at zoom 14 (about 1.5 km across in central Europe), not
|
||||
// its position, and the request goes in the body so the tile stays out of access logs. Points
|
||||
// are tile-local in 0..4096 (they may lie a little outside: a road is kept up to the first
|
||||
// point beyond the edge), simplified to about 1.5 m. Classes: 0 motorway and trunk, 1 primary
|
||||
// and secondary, 2 tertiary and unclassified, 3 residential and living streets.
|
||||
// its position, and in the body so the tile stays out of access logs. Points are tile-local
|
||||
// in 0..4096 (a little beyond at the edges), simplified to about 1.5 m. Classes: 0 motorway and
|
||||
// trunk, 1 primary and secondary, 2 tertiary, 3 minor streets (residential, unclassified,
|
||||
// living streets).
|
||||
//
|
||||
// Overpass is asked one tile at a time, at most once a second; a tile is kept for 30 days,
|
||||
// up to mapCacheTiles of them. The tile's coordinates never appear in the log.
|
||||
// A fetch runs to its end even when the car stops waiting (the car gives up after a few
|
||||
// seconds), so its next try finds the tile. Tiles are kept in memory (the last mapMemoryTiles)
|
||||
// and, with a cache directory, on disk for mapCacheAge. The tile's coordinates never appear
|
||||
// in the log.
|
||||
type Maps struct {
|
||||
OverpassURL string
|
||||
UserAgent string
|
||||
Client *http.Client
|
||||
// TileJSONURL names the vector tiles: a TileJSON document whose first "tiles" template is
|
||||
// used (OpenFreeMap moves its template with every weekly build).
|
||||
TileJSONURL string
|
||||
// CacheDir keeps fetched tiles across restarts; empty keeps them in memory only.
|
||||
CacheDir string
|
||||
UserAgent string
|
||||
Client *http.Client
|
||||
|
||||
mu sync.Mutex
|
||||
tiles map[mapTile]mapEntry
|
||||
|
||||
fetchMu sync.Mutex // one Overpass request at a time
|
||||
lastFetch time.Time
|
||||
gap time.Duration
|
||||
mu sync.Mutex
|
||||
memory map[mapTile]mapEntry
|
||||
inflight map[mapTile]*mapFlight
|
||||
template string
|
||||
tmplAt time.Time
|
||||
// slots limits fetches running at once.
|
||||
slots chan struct{}
|
||||
}
|
||||
|
||||
const (
|
||||
mapZoom = 14
|
||||
mapExtent = 4096
|
||||
mapCacheTiles = 3000
|
||||
mapCacheAge = 30 * 24 * time.Hour
|
||||
maxOverpassBytes = 32 << 20
|
||||
mapZoom = 14
|
||||
mapExtent = 4096
|
||||
mapMemoryTiles = 300
|
||||
mapCacheAge = 30 * 24 * time.Hour
|
||||
maxTileBytes = 8 << 20
|
||||
mapFetchTime = 60 * time.Second
|
||||
templateAge = 6 * time.Hour
|
||||
// The simplification tolerance in tile units (about 0.37 m each at 52° N).
|
||||
mapTolerance = 4.0
|
||||
// DefaultOverpassURL is the public instance of the Overpass API.
|
||||
DefaultOverpassURL = "https://overpass-api.de/api/interpreter"
|
||||
// mapUserAgent names this service to Overpass, as its usage policy asks.
|
||||
mapUserAgent = "nils-mg4-server/2.7 (private car dashboard; roads for an in-car map)"
|
||||
// DefaultMapTilesURL is OpenFreeMap's TileJSON: free OpenStreetMap vector tiles, no key.
|
||||
DefaultMapTilesURL = "https://tiles.openfreemap.org/planet"
|
||||
// mapUserAgent names this service to the tile host.
|
||||
mapUserAgent = "nils-mg4-server/2.8 (private car dashboard; roads for an in-car map)"
|
||||
)
|
||||
|
||||
// mapWait is how long a request waits for a fetch: less than the car's own 8 s.
|
||||
var mapWait = 6 * time.Second
|
||||
|
||||
type mapTile struct{ Z, X, Y int }
|
||||
|
||||
type mapEntry struct {
|
||||
@@ -66,12 +82,19 @@ type mapEntry struct {
|
||||
at time.Time
|
||||
}
|
||||
|
||||
// roadClasses maps OSM's highway values to the four classes the car draws.
|
||||
type mapFlight struct {
|
||||
done chan struct{}
|
||||
body []byte
|
||||
err error
|
||||
}
|
||||
|
||||
// roadClasses maps the OpenMapTiles transportation classes to the four the car draws; the
|
||||
// rest (service, track, path, rail, ferry…) are left out.
|
||||
var roadClasses = map[string]int{
|
||||
"motorway": 0, "motorway_link": 0, "trunk": 0, "trunk_link": 0,
|
||||
"primary": 1, "primary_link": 1, "secondary": 1, "secondary_link": 1,
|
||||
"tertiary": 2, "tertiary_link": 2, "unclassified": 2,
|
||||
"residential": 3, "living_street": 3,
|
||||
"motorway": 0, "trunk": 0,
|
||||
"primary": 1, "secondary": 1,
|
||||
"tertiary": 2,
|
||||
"minor": 3,
|
||||
}
|
||||
|
||||
type roadTile struct {
|
||||
@@ -87,92 +110,227 @@ type road struct {
|
||||
P []int `json:"p"`
|
||||
}
|
||||
|
||||
func NewMaps(overpassURL, userAgent string) *Maps {
|
||||
return &Maps{OverpassURL: overpassURL, UserAgent: userAgent, gap: time.Second}
|
||||
func NewMaps(tileJSONURL, cacheDir, userAgent string) *Maps {
|
||||
return &Maps{TileJSONURL: tileJSONURL, CacheDir: cacheDir, UserAgent: userAgent}
|
||||
}
|
||||
|
||||
func (m *Maps) client() *http.Client {
|
||||
if m.Client != nil {
|
||||
return m.Client
|
||||
}
|
||||
return &http.Client{Timeout: 60 * time.Second}
|
||||
return &http.Client{Timeout: mapFetchTime}
|
||||
}
|
||||
|
||||
// tileBounds is the tile's south, west, north and east edge in degrees.
|
||||
func tileBounds(t mapTile) (s, w, n, e float64) {
|
||||
scale := math.Exp2(float64(t.Z))
|
||||
lon := func(x float64) float64 { return x/scale*360 - 180 }
|
||||
lat := func(y float64) float64 { return math.Atan(math.Sinh(math.Pi*(1-2*y/scale))) * 180 / math.Pi }
|
||||
return lat(float64(t.Y + 1)), lon(float64(t.X)), lat(float64(t.Y)), lon(float64(t.X + 1))
|
||||
// Tile returns the tile's roads as JSON and whether it came from a cache. Waiting ends with
|
||||
// ctx; the fetch itself goes on and is kept.
|
||||
func (m *Maps) Tile(ctx context.Context, t mapTile) ([]byte, bool, error) {
|
||||
if body, ok := m.cached(t); ok {
|
||||
return body, true, nil
|
||||
}
|
||||
m.mu.Lock()
|
||||
if m.inflight == nil {
|
||||
m.inflight = map[mapTile]*mapFlight{}
|
||||
m.slots = make(chan struct{}, 2)
|
||||
}
|
||||
f, running := m.inflight[t]
|
||||
if !running {
|
||||
f = &mapFlight{done: make(chan struct{})}
|
||||
m.inflight[t] = f
|
||||
go m.run(t, f)
|
||||
}
|
||||
m.mu.Unlock()
|
||||
select {
|
||||
case <-f.done:
|
||||
return f.body, false, f.err
|
||||
case <-ctx.Done():
|
||||
return nil, false, errors.New("still fetching")
|
||||
}
|
||||
}
|
||||
|
||||
// tilePoint is a position in the tile's own units: 0..extent across, y downwards.
|
||||
func tilePoint(t mapTile, lat, lon float64) (int, int) {
|
||||
scale := math.Exp2(float64(t.Z)) * mapExtent
|
||||
x := (lon + 180) / 360 * scale
|
||||
r := lat * math.Pi / 180
|
||||
y := (1 - math.Log(math.Tan(r)+1/math.Cos(r))/math.Pi) / 2 * scale
|
||||
return int(math.Round(x - float64(t.X)*mapExtent)), int(math.Round(y - float64(t.Y)*mapExtent))
|
||||
func (m *Maps) run(t mapTile, f *mapFlight) {
|
||||
defer func() {
|
||||
m.mu.Lock()
|
||||
delete(m.inflight, t)
|
||||
m.mu.Unlock()
|
||||
close(f.done)
|
||||
}()
|
||||
m.slots <- struct{}{}
|
||||
defer func() { <-m.slots }()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), mapFetchTime)
|
||||
defer cancel()
|
||||
raw, err := m.fetch(ctx, t)
|
||||
if err == nil {
|
||||
f.body, err = buildTile(t, raw)
|
||||
}
|
||||
if err != nil {
|
||||
f.err = err
|
||||
return
|
||||
}
|
||||
m.store(t, f.body)
|
||||
slog.Info("map tile fetched", "ways", strings.Count(string(f.body), `"c":`), "bytes", len(f.body))
|
||||
}
|
||||
|
||||
// overpassQuery asks for the drivable roads in the tile, with a small margin so that roads
|
||||
// crossing its edge reach a point beyond it.
|
||||
func overpassQuery(t mapTile) string {
|
||||
s, w, n, e := tileBounds(t)
|
||||
dy, dx := (n-s)*0.03, (e-w)*0.03
|
||||
box := fmt.Sprintf("%.6f,%.6f,%.6f,%.6f", s-dy, w-dx, n+dy, e+dx)
|
||||
return "[out:json][timeout:50];way[\"highway\"~\"^(motorway|motorway_link|trunk|trunk_link|primary|primary_link|" +
|
||||
"secondary|secondary_link|tertiary|tertiary_link|unclassified|residential|living_street)$\"](" + box +
|
||||
");out geom(" + box + ");"
|
||||
func (m *Maps) cachePath(t mapTile) string {
|
||||
return filepath.Join(m.CacheDir, fmt.Sprintf("%d_%d_%d.json", t.Z, t.X, t.Y))
|
||||
}
|
||||
|
||||
type overpassResult struct {
|
||||
Elements []struct {
|
||||
Type string `json:"type"`
|
||||
Tags map[string]string `json:"tags"`
|
||||
Geometry []*struct {
|
||||
Lat float64 `json:"lat"`
|
||||
Lon float64 `json:"lon"`
|
||||
} `json:"geometry"`
|
||||
} `json:"elements"`
|
||||
func (m *Maps) cached(t mapTile) ([]byte, bool) {
|
||||
m.mu.Lock()
|
||||
e, ok := m.memory[t]
|
||||
m.mu.Unlock()
|
||||
if ok && time.Since(e.at) <= mapCacheAge {
|
||||
return e.body, true
|
||||
}
|
||||
if m.CacheDir == "" {
|
||||
return nil, false
|
||||
}
|
||||
info, err := os.Stat(m.cachePath(t))
|
||||
if err != nil || time.Since(info.ModTime()) > mapCacheAge {
|
||||
return nil, false
|
||||
}
|
||||
body, err := os.ReadFile(m.cachePath(t))
|
||||
if err != nil || !json.Valid(body) {
|
||||
return nil, false
|
||||
}
|
||||
m.remember(t, body, info.ModTime())
|
||||
return body, true
|
||||
}
|
||||
|
||||
// buildTile turns Overpass's answer into the tile the car reads. A missing point (outside
|
||||
// the clipped geometry) splits a road in two.
|
||||
func (m *Maps) store(t mapTile, body []byte) {
|
||||
m.remember(t, body, time.Now())
|
||||
if m.CacheDir == "" {
|
||||
return
|
||||
}
|
||||
tmp := m.cachePath(t) + ".tmp"
|
||||
if err := os.WriteFile(tmp, body, 0o600); err != nil {
|
||||
slog.Warn("map tile not stored", "error", err)
|
||||
return
|
||||
}
|
||||
if err := os.Rename(tmp, m.cachePath(t)); err != nil {
|
||||
slog.Warn("map tile not stored", "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// remember keeps the tile in memory, dropping the oldest when full.
|
||||
func (m *Maps) remember(t mapTile, body []byte, at time.Time) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if m.memory == nil {
|
||||
m.memory = map[mapTile]mapEntry{}
|
||||
}
|
||||
if _, ok := m.memory[t]; !ok && len(m.memory) >= mapMemoryTiles {
|
||||
var oldest mapTile
|
||||
var first time.Time
|
||||
for k, e := range m.memory {
|
||||
if first.IsZero() || e.at.Before(first) {
|
||||
oldest, first = k, e.at
|
||||
}
|
||||
}
|
||||
delete(m.memory, oldest)
|
||||
}
|
||||
m.memory[t] = mapEntry{body: body, at: at}
|
||||
}
|
||||
|
||||
func (m *Maps) get(ctx context.Context, u string, limit int64) ([]byte, error) {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("User-Agent", m.UserAgent)
|
||||
resp, err := m.client().Do(req)
|
||||
if err != nil {
|
||||
// The URL holds the tile: name the failure, not the request.
|
||||
if errors.Is(err, context.DeadlineExceeded) || errors.Is(ctx.Err(), context.DeadlineExceeded) {
|
||||
return nil, errors.New("tile host timed out")
|
||||
}
|
||||
return nil, errors.New("tile host could not be reached")
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 1<<16))
|
||||
return nil, fmt.Errorf("tile host answered %d", resp.StatusCode)
|
||||
}
|
||||
body, err := io.ReadAll(io.LimitReader(resp.Body, limit+1))
|
||||
if err != nil {
|
||||
return nil, errors.New("tile host answer could not be read")
|
||||
}
|
||||
if int64(len(body)) > limit {
|
||||
return nil, errors.New("tile host answer is too large")
|
||||
}
|
||||
return body, nil
|
||||
}
|
||||
|
||||
// tileTemplate is the current {z}/{x}/{y} URL from the TileJSON, looked up every few hours.
|
||||
func (m *Maps) tileTemplate(ctx context.Context) (string, error) {
|
||||
m.mu.Lock()
|
||||
tmpl, at := m.template, m.tmplAt
|
||||
m.mu.Unlock()
|
||||
if tmpl != "" && time.Since(at) < templateAge {
|
||||
return tmpl, nil
|
||||
}
|
||||
body, err := m.get(ctx, m.TileJSONURL, 1<<20)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var doc struct {
|
||||
Tiles []string `json:"tiles"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &doc); err != nil || len(doc.Tiles) == 0 {
|
||||
return "", errors.New("tile index has no tiles")
|
||||
}
|
||||
tmpl = doc.Tiles[0]
|
||||
u, err := url.Parse(strings.NewReplacer("{z}", "0", "{x}", "0", "{y}", "0").Replace(tmpl))
|
||||
if err != nil || u.Scheme != "https" || u.Host == "" ||
|
||||
!strings.Contains(tmpl, "{z}") || !strings.Contains(tmpl, "{x}") || !strings.Contains(tmpl, "{y}") {
|
||||
return "", errors.New("tile index names no usable https template")
|
||||
}
|
||||
m.mu.Lock()
|
||||
m.template, m.tmplAt = tmpl, time.Now()
|
||||
m.mu.Unlock()
|
||||
return tmpl, nil
|
||||
}
|
||||
|
||||
func (m *Maps) fetch(ctx context.Context, t mapTile) ([]byte, error) {
|
||||
tmpl, err := m.tileTemplate(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
u := strings.NewReplacer("{z}", strconv.Itoa(t.Z), "{x}", strconv.Itoa(t.X), "{y}", strconv.Itoa(t.Y)).Replace(tmpl)
|
||||
raw, err := m.get(ctx, u, maxTileBytes)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// Some hosts send the tile gzipped whatever the request said.
|
||||
if len(raw) > 2 && raw[0] == 0x1f && raw[1] == 0x8b {
|
||||
zr, err := gzip.NewReader(bytes.NewReader(raw))
|
||||
if err != nil {
|
||||
return nil, errors.New("tile is not readable")
|
||||
}
|
||||
raw, err = io.ReadAll(io.LimitReader(zr, maxTileBytes+1))
|
||||
if err != nil || len(raw) > maxTileBytes {
|
||||
return nil, errors.New("tile is not readable")
|
||||
}
|
||||
}
|
||||
return raw, nil
|
||||
}
|
||||
|
||||
// buildTile keeps the roads of the vector tile's transportation layer, in the car's format.
|
||||
func buildTile(t mapTile, raw []byte) ([]byte, error) {
|
||||
var res overpassResult
|
||||
if err := json.Unmarshal(raw, &res); err != nil {
|
||||
return nil, fmt.Errorf("overpass answer: %w", err)
|
||||
lines, err := decodeRoads(raw)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := roadTile{Z: t.Z, X: t.X, Y: t.Y, Extent: mapExtent, Ways: []road{}}
|
||||
for _, el := range res.Elements {
|
||||
class, ok := roadClasses[el.Tags["highway"]]
|
||||
if el.Type != "way" || !ok {
|
||||
for _, l := range lines {
|
||||
pts := simplify(l.points, mapTolerance)
|
||||
if len(pts) < 2 {
|
||||
continue
|
||||
}
|
||||
var run [][2]int
|
||||
flush := func() {
|
||||
if pts := simplify(run, mapTolerance); len(pts) >= 2 {
|
||||
flat := make([]int, 0, len(pts)*2)
|
||||
for _, p := range pts {
|
||||
flat = append(flat, p[0], p[1])
|
||||
}
|
||||
out.Ways = append(out.Ways, road{C: class, P: flat})
|
||||
}
|
||||
run = nil
|
||||
flat := make([]int, 0, len(pts)*2)
|
||||
for _, p := range pts {
|
||||
flat = append(flat, p[0], p[1])
|
||||
}
|
||||
for _, g := range el.Geometry {
|
||||
if g == nil {
|
||||
flush()
|
||||
continue
|
||||
}
|
||||
x, y := tilePoint(t, g.Lat, g.Lon)
|
||||
if n := len(run); n > 0 && run[n-1] == [2]int{x, y} {
|
||||
continue
|
||||
}
|
||||
run = append(run, [2]int{x, y})
|
||||
}
|
||||
flush()
|
||||
out.Ways = append(out.Ways, road{C: l.class, P: flat})
|
||||
}
|
||||
return json.Marshal(out)
|
||||
}
|
||||
@@ -216,100 +374,11 @@ func segmentDistance(p, a, b [2]int) float64 {
|
||||
if dx == 0 && dy == 0 {
|
||||
return math.Hypot(px-ax, py-ay)
|
||||
}
|
||||
u := math.Max(0, math.Min(1, ((px-ax)*dx+(py-ay)*dy)/(dx*dx+dy*dy)))
|
||||
u := ((px-ax)*dx + (py-ay)*dy) / (dx*dx + dy*dy)
|
||||
u = max(0, min(1, u))
|
||||
return math.Hypot(px-ax-u*dx, py-ay-u*dy)
|
||||
}
|
||||
|
||||
// Tile returns the tile's roads as JSON, from the cache or from Overpass, and whether it
|
||||
// came from the cache.
|
||||
func (m *Maps) Tile(ctx context.Context, t mapTile) ([]byte, bool, error) {
|
||||
if body, ok := m.cached(t); ok {
|
||||
return body, true, nil
|
||||
}
|
||||
m.fetchMu.Lock()
|
||||
defer m.fetchMu.Unlock()
|
||||
// Another request may have fetched it while this one waited.
|
||||
if body, ok := m.cached(t); ok {
|
||||
return body, true, nil
|
||||
}
|
||||
if wait := m.gap - time.Since(m.lastFetch); wait > 0 {
|
||||
select {
|
||||
case <-time.After(wait):
|
||||
case <-ctx.Done():
|
||||
return nil, false, ctx.Err()
|
||||
}
|
||||
}
|
||||
m.lastFetch = time.Now()
|
||||
raw, err := m.fetch(ctx, t)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
body, err := buildTile(t, raw)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
m.store(t, body)
|
||||
return body, false, nil
|
||||
}
|
||||
|
||||
func (m *Maps) cached(t mapTile) ([]byte, bool) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
e, ok := m.tiles[t]
|
||||
if !ok || time.Since(e.at) > mapCacheAge {
|
||||
return nil, false
|
||||
}
|
||||
return e.body, true
|
||||
}
|
||||
|
||||
// store keeps the tile, dropping the oldest when the cache is full.
|
||||
func (m *Maps) store(t mapTile, body []byte) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if m.tiles == nil {
|
||||
m.tiles = map[mapTile]mapEntry{}
|
||||
}
|
||||
if len(m.tiles) >= mapCacheTiles {
|
||||
var oldest mapTile
|
||||
var at time.Time
|
||||
for k, e := range m.tiles {
|
||||
if at.IsZero() || e.at.Before(at) {
|
||||
oldest, at = k, e.at
|
||||
}
|
||||
}
|
||||
delete(m.tiles, oldest)
|
||||
}
|
||||
m.tiles[t] = mapEntry{body: body, at: time.Now()}
|
||||
}
|
||||
|
||||
// fetch asks Overpass. The query goes in the body, so its errors never carry the tile.
|
||||
func (m *Maps) fetch(ctx context.Context, t mapTile) ([]byte, error) {
|
||||
form := url.Values{"data": {overpassQuery(t)}}.Encode()
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, m.OverpassURL, strings.NewReader(form))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
req.Header.Set("User-Agent", m.UserAgent)
|
||||
resp, err := m.client().Do(req)
|
||||
if err != nil {
|
||||
return nil, errors.New("overpass could not be reached")
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 1<<16))
|
||||
return nil, fmt.Errorf("overpass answered %d", resp.StatusCode)
|
||||
}
|
||||
raw, err := io.ReadAll(io.LimitReader(resp.Body, maxOverpassBytes+1))
|
||||
if err != nil {
|
||||
return nil, errors.New("overpass answer could not be read")
|
||||
}
|
||||
if len(raw) > maxOverpassBytes {
|
||||
return nil, errors.New("overpass answer is too large")
|
||||
}
|
||||
return raw, nil
|
||||
}
|
||||
|
||||
func (a *API) mapRoutes(mux *http.ServeMux) {
|
||||
mux.HandleFunc("POST /v1/map/roads", a.ingest(a.mapRoads))
|
||||
}
|
||||
@@ -332,17 +401,16 @@ func (a *API) mapRoads(w http.ResponseWriter, r *http.Request, c caller, body []
|
||||
if *req.X < 0 || *req.X >= limit || *req.Y < 0 || *req.Y >= limit {
|
||||
return invalid("x and y must be within the zoom level")
|
||||
}
|
||||
t := mapTile{Z: *req.Z, X: *req.X, Y: *req.Y}
|
||||
data, cached, err := a.Maps.Tile(r.Context(), t)
|
||||
// The car waits a few seconds at most; the fetch goes on without it.
|
||||
wait, cancel := context.WithTimeout(r.Context(), mapWait)
|
||||
defer cancel()
|
||||
data, _, err := a.Maps.Tile(wait, mapTile{Z: *req.Z, X: *req.X, Y: *req.Y})
|
||||
if err != nil {
|
||||
slog.Warn("map tile not available", "installation", who(c), "error", err)
|
||||
writeJSON(w, http.StatusServiceUnavailable, map[string]string{"error": "map data not available, try again later"})
|
||||
slog.Warn("map tile not available yet", "installation", who(c), "error", err)
|
||||
w.Header().Set("Retry-After", "10")
|
||||
writeJSON(w, http.StatusServiceUnavailable, map[string]string{"error": "map data not available yet, try again shortly"})
|
||||
return nil
|
||||
}
|
||||
if !cached {
|
||||
slog.Info("map tile fetched", "installation", who(c), "ways", strings.Count(string(data), `"c":`),
|
||||
"bytes", len(data))
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
if strings.Contains(r.Header.Get("Accept-Encoding"), "gzip") {
|
||||
var buf bytes.Buffer
|
||||
|
||||
+246
-97
@@ -12,50 +12,140 @@ import (
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A tile in central Berlin.
|
||||
var testTile = mapTile{Z: 14, X: 8802, Y: 5373}
|
||||
|
||||
// fakeOverpass answers every query with a primary road (with a gap in its geometry), a
|
||||
// residential street with a needless middle point, and a footway the car does not draw.
|
||||
func fakeOverpass(t *testing.T, calls *atomic.Int32, status int) *httptest.Server {
|
||||
t.Helper()
|
||||
s, w, n, e := tileBounds(testTile)
|
||||
lat := func(f float64) float64 { return s + (n-s)*f }
|
||||
lon := func(f float64) float64 { return w + (e-w)*f }
|
||||
answer := fmt.Sprintf(`{"elements":[
|
||||
{"type":"way","tags":{"highway":"primary"},"geometry":[
|
||||
{"lat":%f,"lon":%f},{"lat":%f,"lon":%f},null,{"lat":%f,"lon":%f},{"lat":%f,"lon":%f}]},
|
||||
{"type":"way","tags":{"highway":"residential"},"geometry":[
|
||||
{"lat":%f,"lon":%f},{"lat":%f,"lon":%f},{"lat":%f,"lon":%f}]},
|
||||
{"type":"way","tags":{"highway":"footway"},"geometry":[
|
||||
{"lat":%f,"lon":%f},{"lat":%f,"lon":%f}]}]}`,
|
||||
lat(0.1), lon(0.1), lat(0.2), lon(0.2), lat(0.6), lon(0.6), lat(0.7), lon(0.7),
|
||||
lat(0.5), lon(0.1), lat(0.5), lon(0.5), lat(0.5), lon(0.9),
|
||||
lat(0.3), lon(0.3), lat(0.4), lon(0.4))
|
||||
return httptest.NewTLSServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
|
||||
calls.Add(1)
|
||||
if r.Method != http.MethodPost || !strings.Contains(r.UserAgent(), "nils-mg4-server") {
|
||||
t.Errorf("request %s %q", r.Method, r.UserAgent())
|
||||
}
|
||||
_ = r.ParseForm()
|
||||
if q := r.PostForm.Get("data"); !strings.Contains(q, "out geom(") || !strings.Contains(q, "residential") {
|
||||
t.Errorf("query %q", q)
|
||||
}
|
||||
if status != http.StatusOK {
|
||||
http.Error(rw, "busy", status)
|
||||
return
|
||||
}
|
||||
_, _ = rw.Write([]byte(answer))
|
||||
}))
|
||||
// ---------- A minimal vector-tile encoder for the tests ----------
|
||||
|
||||
func pbVarint(v uint64) []byte {
|
||||
var b []byte
|
||||
for v >= 0x80 {
|
||||
b = append(b, byte(v)|0x80)
|
||||
v >>= 7
|
||||
}
|
||||
return append(b, byte(v))
|
||||
}
|
||||
|
||||
func mapsAPI(overpass *httptest.Server) http.Handler {
|
||||
func pbKey(num, wire int) []byte { return pbVarint(uint64(num<<3 | wire)) }
|
||||
|
||||
func pbBytes(num int, data []byte) []byte {
|
||||
return append(append(pbKey(num, 2), pbVarint(uint64(len(data)))...), data...)
|
||||
}
|
||||
|
||||
func pbUint(num int, v uint64) []byte { return append(pbKey(num, 0), pbVarint(v)...) }
|
||||
|
||||
func pbPacked(num int, vs []uint32) []byte {
|
||||
var d []byte
|
||||
for _, v := range vs {
|
||||
d = append(d, pbVarint(uint64(v))...)
|
||||
}
|
||||
return pbBytes(num, d)
|
||||
}
|
||||
|
||||
func zz(v int) uint32 { return uint32((v << 1) ^ (v >> 31)) }
|
||||
|
||||
// lineCmds encodes line strings as MoveTo/LineTo commands with relative moves.
|
||||
func lineCmds(lines ...[][2]int) []uint32 {
|
||||
var out []uint32
|
||||
x, y := 0, 0
|
||||
for _, l := range lines {
|
||||
out = append(out, 1|1<<3, zz(l[0][0]-x), zz(l[0][1]-y))
|
||||
x, y = l[0][0], l[0][1]
|
||||
out = append(out, uint32(2|(len(l)-1)<<3))
|
||||
for _, p := range l[1:] {
|
||||
out = append(out, zz(p[0]-x), zz(p[1]-y))
|
||||
x, y = p[0], p[1]
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
type testFeature struct {
|
||||
class int // index into the values below
|
||||
geomType uint64
|
||||
geometry []uint32
|
||||
}
|
||||
|
||||
// testVectorTile has a water layer (ignored) and a transportation layer with the given extent.
|
||||
func testVectorTile(extent uint64) []byte {
|
||||
values := []string{"primary", "minor", "service", "motorway"}
|
||||
features := []testFeature{
|
||||
// A primary road with a needless middle point.
|
||||
{0, 2, lineCmds([][2]int{{100, 100}, {200, 200}, {300, 300}})},
|
||||
// A minor street in two pieces.
|
||||
{1, 2, lineCmds([][2]int{{0, 500}, {400, 500}}, [][2]int{{600, 500}, {900, 520}})},
|
||||
// A service road and a motorway as a polygon: neither is drawn.
|
||||
{2, 2, lineCmds([][2]int{{10, 10}, {20, 20}})},
|
||||
{3, 3, lineCmds([][2]int{{10, 10}, {20, 20}})},
|
||||
}
|
||||
var layer []byte
|
||||
layer = append(layer, pbUint(15, 2)...)
|
||||
layer = append(layer, pbBytes(1, []byte("transportation"))...)
|
||||
for _, f := range features {
|
||||
var feat []byte
|
||||
feat = append(feat, pbPacked(2, []uint32{0, uint32(f.class)})...)
|
||||
feat = append(feat, pbUint(3, f.geomType)...)
|
||||
feat = append(feat, pbPacked(4, f.geometry)...)
|
||||
layer = append(layer, pbBytes(2, feat)...)
|
||||
}
|
||||
layer = append(layer, pbBytes(3, []byte("class"))...)
|
||||
for _, v := range values {
|
||||
layer = append(layer, pbBytes(4, pbBytes(1, []byte(v)))...)
|
||||
}
|
||||
layer = append(layer, pbUint(5, extent)...)
|
||||
|
||||
water := append(pbBytes(1, []byte("water")), pbUint(5, 4096)...)
|
||||
return append(pbBytes(3, water), pbBytes(3, layer)...)
|
||||
}
|
||||
|
||||
// ---------- A fake tile host ----------
|
||||
|
||||
type fakeHost struct {
|
||||
*httptest.Server
|
||||
tiles atomic.Int32
|
||||
status int
|
||||
delay time.Duration
|
||||
gzip bool
|
||||
}
|
||||
|
||||
func newFakeHost(t *testing.T) *fakeHost {
|
||||
t.Helper()
|
||||
h := &fakeHost{status: http.StatusOK}
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("GET /planet", func(w http.ResponseWriter, r *http.Request) {
|
||||
fmt.Fprintf(w, `{"tilejson":"3.0.0","tiles":["%s/planet/build1/{z}/{x}/{y}.pbf"]}`, h.URL)
|
||||
})
|
||||
mux.HandleFunc("GET /planet/build1/{z}/{x}/{y}", func(w http.ResponseWriter, r *http.Request) {
|
||||
h.tiles.Add(1)
|
||||
if !strings.Contains(r.UserAgent(), "nils-mg4-server") {
|
||||
t.Errorf("user agent %q", r.UserAgent())
|
||||
}
|
||||
time.Sleep(h.delay)
|
||||
if h.status != http.StatusOK {
|
||||
http.Error(w, "busy", h.status)
|
||||
return
|
||||
}
|
||||
body := testVectorTile(4096)
|
||||
if h.gzip {
|
||||
var buf bytes.Buffer
|
||||
zw := gzip.NewWriter(&buf)
|
||||
_, _ = zw.Write(body)
|
||||
_ = zw.Close()
|
||||
body = buf.Bytes()
|
||||
}
|
||||
_, _ = w.Write(body)
|
||||
})
|
||||
h.Server = httptest.NewTLSServer(mux)
|
||||
return h
|
||||
}
|
||||
|
||||
func mapsAPIWith(host *fakeHost, cacheDir string) http.Handler {
|
||||
api, _ := newTestAPI()
|
||||
api.Maps = NewMaps(overpass.URL, mapUserAgent)
|
||||
api.Maps.Client = overpass.Client()
|
||||
api.Maps.gap = 0
|
||||
api.Maps = NewMaps(host.URL+"/planet", cacheDir, mapUserAgent)
|
||||
api.Maps.Client = host.Client()
|
||||
return api.Routes()
|
||||
}
|
||||
|
||||
@@ -63,15 +153,8 @@ func tileBody(t mapTile) string {
|
||||
return fmt.Sprintf(`{"z":%d,"x":%d,"y":%d}`, t.Z, t.X, t.Y)
|
||||
}
|
||||
|
||||
func TestMapTileRoadsAreClassifiedSplitAndSimplified(t *testing.T) {
|
||||
var calls atomic.Int32
|
||||
overpass := fakeOverpass(t, &calls, http.StatusOK)
|
||||
defer overpass.Close()
|
||||
h := mapsAPI(overpass)
|
||||
if rec := do(t, h, "POST", "/v1/map/roads", "", tileBody(testTile)); rec.Code != 401 {
|
||||
t.Fatalf("without the car's token: %d", rec.Code)
|
||||
}
|
||||
rec := do(t, h, "POST", "/v1/map/roads", testToken, tileBody(testTile))
|
||||
func decodeAnswer(t *testing.T, rec *httptest.ResponseRecorder) roadTile {
|
||||
t.Helper()
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("%d %s", rec.Code, rec.Body)
|
||||
}
|
||||
@@ -79,39 +162,100 @@ func TestMapTileRoadsAreClassifiedSplitAndSimplified(t *testing.T) {
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Extent != 4096 || got.X != testTile.X || got.Y != testTile.Y {
|
||||
return got
|
||||
}
|
||||
|
||||
// ---------- Tests ----------
|
||||
|
||||
func TestVectorTileRoadsAreClassifiedSplitAndSimplified(t *testing.T) {
|
||||
lines, err := decodeRoads(testVectorTile(4096))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(lines) != 3 || lines[0].class != 1 || lines[1].class != 3 || lines[2].class != 3 {
|
||||
t.Fatalf("lines %+v", lines)
|
||||
}
|
||||
if fmt.Sprint(lines[1].points) != "[[0 500] [400 500]]" || fmt.Sprint(lines[2].points) != "[[600 500] [900 520]]" {
|
||||
t.Fatalf("pieces %v %v", lines[1].points, lines[2].points)
|
||||
}
|
||||
// Another extent is scaled to 4096.
|
||||
half, err := decodeRoads(testVectorTile(8192))
|
||||
if err != nil || fmt.Sprint(half[0].points[2]) != "[150 150]" {
|
||||
t.Fatalf("scaled %v %v", half, err)
|
||||
}
|
||||
body, err := buildTile(testTile, testVectorTile(4096))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(string(body), `{"c":1,"p":[100,100,300,300]}`) {
|
||||
t.Fatalf("the straight road keeps only its ends: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMalformedVectorTilesAreRefused(t *testing.T) {
|
||||
good := testVectorTile(4096)
|
||||
for name, raw := range map[string][]byte{
|
||||
"cut short": good[:len(good)-7],
|
||||
"bad wire type": {0x1f, 0x00},
|
||||
"length too long": {0x1a, 0x7f, 0x01},
|
||||
} {
|
||||
if _, err := decodeRoads(raw); err == nil {
|
||||
t.Errorf("%s: accepted", name)
|
||||
}
|
||||
}
|
||||
if lines, err := decodeRoads(nil); err != nil || len(lines) != 0 {
|
||||
t.Fatalf("an empty tile is an empty map: %v %v", lines, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMapTilesAreFetchedOnceAndKeptOnDisk(t *testing.T) {
|
||||
host := newFakeHost(t)
|
||||
host.gzip = true // some hosts gzip whatever was asked
|
||||
defer host.Close()
|
||||
dir := t.TempDir()
|
||||
h := mapsAPIWith(host, dir)
|
||||
if rec := do(t, h, "POST", "/v1/map/roads", "", tileBody(testTile)); rec.Code != 401 {
|
||||
t.Fatalf("without the car's token: %d", rec.Code)
|
||||
}
|
||||
got := decodeAnswer(t, do(t, h, "POST", "/v1/map/roads", testToken, tileBody(testTile)))
|
||||
if got.Extent != 4096 || got.X != testTile.X || got.Y != testTile.Y || len(got.Ways) != 3 {
|
||||
t.Fatalf("tile %+v", got)
|
||||
}
|
||||
// The primary road in two pieces, the street straightened to its ends, no footway.
|
||||
if len(got.Ways) != 3 || got.Ways[0].C != 1 || got.Ways[1].C != 1 || got.Ways[2].C != 3 {
|
||||
t.Fatalf("ways %+v", got.Ways)
|
||||
}
|
||||
if len(got.Ways[2].P) != 4 {
|
||||
t.Fatalf("the straight street keeps only its ends: %v", got.Ways[2].P)
|
||||
}
|
||||
// Tile-local: 0.1 of the way in is about 410 units from the west edge.
|
||||
if x := got.Ways[0].P[0]; x < 400 || x > 420 {
|
||||
t.Fatalf("x %d", x)
|
||||
decodeAnswer(t, do(t, h, "POST", "/v1/map/roads", testToken, tileBody(testTile)))
|
||||
// A restart: a new server with the same directory asks the host nothing.
|
||||
decodeAnswer(t, do(t, mapsAPIWith(host, dir), "POST", "/v1/map/roads", testToken, tileBody(testTile)))
|
||||
if n := host.tiles.Load(); n != 1 {
|
||||
t.Fatalf("tile fetched %d times", n)
|
||||
}
|
||||
}
|
||||
|
||||
// Cached: the second car asking costs Overpass nothing.
|
||||
if rec := do(t, h, "POST", "/v1/map/roads", testToken, tileBody(testTile)); rec.Code != 200 {
|
||||
t.Fatalf("%d", rec.Code)
|
||||
func TestASlowTileIsFinishedForTheCarsNextTry(t *testing.T) {
|
||||
previous := mapWait
|
||||
mapWait = 50 * time.Millisecond
|
||||
defer func() { mapWait = previous }()
|
||||
host := newFakeHost(t)
|
||||
host.delay = 300 * time.Millisecond
|
||||
defer host.Close()
|
||||
h := mapsAPIWith(host, "")
|
||||
rec := do(t, h, "POST", "/v1/map/roads", testToken, tileBody(testTile))
|
||||
if rec.Code != 503 || rec.Header().Get("Retry-After") == "" {
|
||||
t.Fatalf("the car should hear 'try again': %d", rec.Code)
|
||||
}
|
||||
if calls.Load() != 1 {
|
||||
t.Fatalf("overpass asked %d times", calls.Load())
|
||||
time.Sleep(600 * time.Millisecond)
|
||||
decodeAnswer(t, do(t, h, "POST", "/v1/map/roads", testToken, tileBody(testTile)))
|
||||
if n := host.tiles.Load(); n != 1 {
|
||||
t.Fatalf("tile fetched %d times", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMapTilesAreGzippedWhenTheCarAcceptsIt(t *testing.T) {
|
||||
var calls atomic.Int32
|
||||
overpass := fakeOverpass(t, &calls, http.StatusOK)
|
||||
defer overpass.Close()
|
||||
host := newFakeHost(t)
|
||||
defer host.Close()
|
||||
req := httptest.NewRequest("POST", "/v1/map/roads", strings.NewReader(tileBody(testTile)))
|
||||
req.Header.Set("Authorization", "Bearer "+testToken)
|
||||
req.Header.Set("Accept-Encoding", "gzip")
|
||||
rec := httptest.NewRecorder()
|
||||
mapsAPI(overpass).ServeHTTP(rec, req)
|
||||
mapsAPIWith(host, "").ServeHTTP(rec, req)
|
||||
if rec.Code != 200 || rec.Header().Get("Content-Encoding") != "gzip" {
|
||||
t.Fatalf("%d %v", rec.Code, rec.Header())
|
||||
}
|
||||
@@ -126,84 +270,89 @@ func TestMapTilesAreGzippedWhenTheCarAcceptsIt(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestMapRequestsAreChecked(t *testing.T) {
|
||||
var calls atomic.Int32
|
||||
overpass := fakeOverpass(t, &calls, http.StatusOK)
|
||||
defer overpass.Close()
|
||||
h := mapsAPI(overpass)
|
||||
host := newFakeHost(t)
|
||||
defer host.Close()
|
||||
h := mapsAPIWith(host, "")
|
||||
for _, body := range []string{`{}`, `{"z":15,"x":1,"y":1}`, `{"z":14,"x":-1,"y":1}`,
|
||||
`{"z":14,"x":1,"y":16384}`, `not json`} {
|
||||
if rec := do(t, h, "POST", "/v1/map/roads", testToken, body); rec.Code != 400 {
|
||||
t.Errorf("%s: %d", body, rec.Code)
|
||||
}
|
||||
}
|
||||
if calls.Load() != 0 {
|
||||
t.Fatal("a bad request reached Overpass")
|
||||
if host.tiles.Load() != 0 {
|
||||
t.Fatal("a bad request reached the tile host")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABusyOverpassIsATemporaryFailureAndNotCached(t *testing.T) {
|
||||
func TestAFailingHostIsATemporaryFailureAndNotCached(t *testing.T) {
|
||||
var logged bytes.Buffer
|
||||
previous := slog.Default()
|
||||
slog.SetDefault(slog.New(slog.NewTextHandler(&logged, nil)))
|
||||
defer slog.SetDefault(previous)
|
||||
|
||||
var calls atomic.Int32
|
||||
overpass := fakeOverpass(t, &calls, http.StatusTooManyRequests)
|
||||
defer overpass.Close()
|
||||
h := mapsAPI(overpass)
|
||||
host := newFakeHost(t)
|
||||
host.status = http.StatusBadGateway
|
||||
defer host.Close()
|
||||
h := mapsAPIWith(host, t.TempDir())
|
||||
for i := 0; i < 2; i++ {
|
||||
if rec := do(t, h, "POST", "/v1/map/roads", testToken, tileBody(testTile)); rec.Code != 503 {
|
||||
t.Fatalf("%d", rec.Code)
|
||||
}
|
||||
}
|
||||
if calls.Load() != 2 {
|
||||
t.Fatalf("a failure was cached: %d calls", calls.Load())
|
||||
if n := host.tiles.Load(); n != 2 {
|
||||
t.Fatalf("a failure was cached: %d fetches", n)
|
||||
}
|
||||
// Where the car is stays out of the log.
|
||||
out := logged.String()
|
||||
if !strings.Contains(out, "overpass answered 429") || strings.Contains(out, "8802") ||
|
||||
if !strings.Contains(out, "tile host answered 502") || strings.Contains(out, "8802") ||
|
||||
strings.Contains(out, "5373") {
|
||||
t.Fatalf("log:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMapsAreOffWithoutOverpass(t *testing.T) {
|
||||
func TestATileIndexWithoutHttpsIsRefused(t *testing.T) {
|
||||
index := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"tiles":["http://tiles.example.com/{z}/{x}/{y}.pbf"]}`))
|
||||
}))
|
||||
defer index.Close()
|
||||
api, _ := newTestAPI()
|
||||
api.Maps = NewMaps(index.URL, "", mapUserAgent)
|
||||
api.Maps.Client = index.Client()
|
||||
if rec := do(t, api.Routes(), "POST", "/v1/map/roads", testToken, tileBody(testTile)); rec.Code != 503 {
|
||||
t.Fatalf("%d", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMapsAreOffWithoutATileSource(t *testing.T) {
|
||||
api, _ := newTestAPI()
|
||||
if rec := do(t, api.Routes(), "POST", "/v1/map/roads", testToken, tileBody(testTile)); rec.Code != 404 {
|
||||
t.Fatalf("%d", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOverpassURLConfiguration(t *testing.T) {
|
||||
func TestMapTilesURLConfiguration(t *testing.T) {
|
||||
base := map[string]string{"INFLUX_URL": "http://influx", "INFLUX_TOKEN": "t", "INFLUX_ORG": "o",
|
||||
"INGEST_TOKEN": "0123456789abcdef01234567"}
|
||||
load := func(overpass string) (config, error) {
|
||||
load := func(tiles string) (config, error) {
|
||||
return loadConfig(func(k string) string {
|
||||
if k == "OVERPASS_URL" {
|
||||
return overpass
|
||||
if k == "MAP_TILES_URL" {
|
||||
return tiles
|
||||
}
|
||||
return base[k]
|
||||
})
|
||||
}
|
||||
if c, err := load(""); err != nil || c.overpassURL != DefaultOverpassURL {
|
||||
t.Fatalf("default: %q %v", c.overpassURL, err)
|
||||
if c, err := load(""); err != nil || c.mapTilesURL != DefaultMapTilesURL {
|
||||
t.Fatalf("default: %q %v", c.mapTilesURL, err)
|
||||
}
|
||||
if c, err := load("off"); err != nil || c.overpassURL != "" {
|
||||
t.Fatalf("off: %q %v", c.overpassURL, err)
|
||||
if c, err := load("off"); err != nil || c.mapTilesURL != "" {
|
||||
t.Fatalf("off: %q %v", c.mapTilesURL, err)
|
||||
}
|
||||
if _, err := load("http://overpass.example.com/api/interpreter"); err == nil {
|
||||
if _, err := load("http://tiles.example.com/planet"); err == nil {
|
||||
t.Fatal("plain http accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTilePointsAndSimplification(t *testing.T) {
|
||||
s, w, n, e := tileBounds(testTile)
|
||||
if x, y := tilePoint(testTile, n, w); x != 0 || y != 0 {
|
||||
t.Fatalf("north-west corner %d,%d", x, y)
|
||||
}
|
||||
if x, y := tilePoint(testTile, s, e); x != 4096 || y != 4096 {
|
||||
t.Fatalf("south-east corner %d,%d", x, y)
|
||||
}
|
||||
func TestSimplification(t *testing.T) {
|
||||
line := [][2]int{{0, 0}, {10, 1}, {20, 0}, {30, 40}}
|
||||
got := simplify(line, 4)
|
||||
if len(got) != 3 || got[1] != [2]int{20, 0} {
|
||||
|
||||
+278
@@ -0,0 +1,278 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"math"
|
||||
)
|
||||
|
||||
// The roads of a Mapbox Vector Tile (spec 2.1) in the OpenMapTiles schema: the line features
|
||||
// of the "transportation" layer whose "class" the car draws (roadClasses), scaled to
|
||||
// mapExtent. A minimal protobuf reader, enough for that and nothing more; anything malformed
|
||||
// is an error, never a guess.
|
||||
|
||||
type roadLine struct {
|
||||
class int
|
||||
points [][2]int
|
||||
}
|
||||
|
||||
var errBadTile = errors.New("vector tile is malformed")
|
||||
|
||||
// pbReader walks protobuf fields.
|
||||
type pbReader struct {
|
||||
b []byte
|
||||
i int
|
||||
}
|
||||
|
||||
func (r *pbReader) done() bool { return r.i >= len(r.b) }
|
||||
|
||||
func (r *pbReader) varint() (uint64, error) {
|
||||
var v uint64
|
||||
for shift := uint(0); shift < 64; shift += 7 {
|
||||
if r.i >= len(r.b) {
|
||||
return 0, errBadTile
|
||||
}
|
||||
c := r.b[r.i]
|
||||
r.i++
|
||||
v |= uint64(c&0x7f) << shift
|
||||
if c < 0x80 {
|
||||
return v, nil
|
||||
}
|
||||
}
|
||||
return 0, errBadTile
|
||||
}
|
||||
|
||||
// field reads one field's key and, for length-delimited ones, its bytes.
|
||||
func (r *pbReader) field() (num int, wire int, val uint64, data []byte, err error) {
|
||||
key, err := r.varint()
|
||||
if err != nil {
|
||||
return 0, 0, 0, nil, err
|
||||
}
|
||||
num, wire = int(key>>3), int(key&7)
|
||||
switch wire {
|
||||
case 0:
|
||||
val, err = r.varint()
|
||||
case 1:
|
||||
if r.i+8 > len(r.b) {
|
||||
return 0, 0, 0, nil, errBadTile
|
||||
}
|
||||
r.i += 8
|
||||
case 2:
|
||||
var n uint64
|
||||
n, err = r.varint()
|
||||
if err == nil {
|
||||
if n > uint64(len(r.b)-r.i) {
|
||||
return 0, 0, 0, nil, errBadTile
|
||||
}
|
||||
data = r.b[r.i : r.i+int(n)]
|
||||
r.i += int(n)
|
||||
}
|
||||
case 5:
|
||||
if r.i+4 > len(r.b) {
|
||||
return 0, 0, 0, nil, errBadTile
|
||||
}
|
||||
r.i += 4
|
||||
default:
|
||||
return 0, 0, 0, nil, errBadTile
|
||||
}
|
||||
return num, wire, val, data, err
|
||||
}
|
||||
|
||||
// packed reads a repeated uint32 field, packed or not.
|
||||
func packed(wire int, val uint64, data []byte, into []uint32) ([]uint32, error) {
|
||||
if wire == 0 {
|
||||
return append(into, uint32(val)), nil
|
||||
}
|
||||
if wire != 2 {
|
||||
return nil, errBadTile
|
||||
}
|
||||
r := pbReader{b: data}
|
||||
for !r.done() {
|
||||
v, err := r.varint()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
into = append(into, uint32(v))
|
||||
}
|
||||
return into, nil
|
||||
}
|
||||
|
||||
// decodeRoads returns the drawable roads of the tile.
|
||||
func decodeRoads(tile []byte) ([]roadLine, error) {
|
||||
r := pbReader{b: tile}
|
||||
for !r.done() {
|
||||
num, _, _, data, err := r.field()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if num != 3 { // layers
|
||||
continue
|
||||
}
|
||||
lines, isRoads, err := decodeLayer(data)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if isRoads {
|
||||
return lines, nil
|
||||
}
|
||||
}
|
||||
return nil, nil // a tile without roads (sea, desert) is an empty one
|
||||
}
|
||||
|
||||
type rawFeature struct {
|
||||
tags []uint32
|
||||
geomType uint64
|
||||
geometry []uint32
|
||||
}
|
||||
|
||||
func decodeLayer(layer []byte) ([]roadLine, bool, error) {
|
||||
r := pbReader{b: layer}
|
||||
var name string
|
||||
var keys []string
|
||||
var values []string // only string values matter for "class"
|
||||
var features []rawFeature
|
||||
extent := uint64(4096)
|
||||
for !r.done() {
|
||||
num, wire, val, data, err := r.field()
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
switch num {
|
||||
case 1:
|
||||
name = string(data)
|
||||
case 2:
|
||||
f, err := decodeFeature(data)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
features = append(features, f)
|
||||
case 3:
|
||||
keys = append(keys, string(data))
|
||||
case 4:
|
||||
values = append(values, valueString(data))
|
||||
case 5:
|
||||
if wire != 0 || val == 0 {
|
||||
return nil, false, errBadTile
|
||||
}
|
||||
extent = val
|
||||
}
|
||||
}
|
||||
if name != "transportation" {
|
||||
return nil, false, nil
|
||||
}
|
||||
scale := float64(mapExtent) / float64(extent)
|
||||
var lines []roadLine
|
||||
for _, f := range features {
|
||||
if f.geomType != 2 { // LINESTRING
|
||||
continue
|
||||
}
|
||||
class, ok := -1, false
|
||||
for i := 0; i+1 < len(f.tags); i += 2 {
|
||||
k, v := int(f.tags[i]), int(f.tags[i+1])
|
||||
if k >= len(keys) || v >= len(values) {
|
||||
return nil, false, errBadTile
|
||||
}
|
||||
if keys[k] == "class" {
|
||||
class, ok = roadClasses[values[v]]
|
||||
if !ok {
|
||||
class = -1
|
||||
}
|
||||
}
|
||||
}
|
||||
if class < 0 {
|
||||
continue
|
||||
}
|
||||
parts, err := lineGeometry(f.geometry, scale)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
for _, p := range parts {
|
||||
lines = append(lines, roadLine{class: class, points: p})
|
||||
}
|
||||
}
|
||||
return lines, true, nil
|
||||
}
|
||||
|
||||
func decodeFeature(data []byte) (rawFeature, error) {
|
||||
var f rawFeature
|
||||
r := pbReader{b: data}
|
||||
for !r.done() {
|
||||
num, wire, val, d, err := r.field()
|
||||
if err != nil {
|
||||
return f, err
|
||||
}
|
||||
switch num {
|
||||
case 2:
|
||||
if f.tags, err = packed(wire, val, d, f.tags); err != nil {
|
||||
return f, err
|
||||
}
|
||||
case 3:
|
||||
f.geomType = val
|
||||
case 4:
|
||||
if f.geometry, err = packed(wire, val, d, f.geometry); err != nil {
|
||||
return f, err
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(f.tags)%2 != 0 {
|
||||
return f, errBadTile
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
|
||||
// valueString is a Value's string_value, "" for any other kind.
|
||||
func valueString(data []byte) string {
|
||||
r := pbReader{b: data}
|
||||
for !r.done() {
|
||||
num, _, _, d, err := r.field()
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
if num == 1 {
|
||||
return string(d)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// lineGeometry follows the MoveTo/LineTo commands of a (multi)line string.
|
||||
func lineGeometry(cmds []uint32, scale float64) ([][][2]int, error) {
|
||||
var parts [][][2]int
|
||||
var cur [][2]int
|
||||
x, y := 0, 0
|
||||
zig := func(v uint32) int { return int(int32(v>>1) ^ -int32(v&1)) }
|
||||
at := func() [2]int {
|
||||
return [2]int{int(math.Round(float64(x) * scale)), int(math.Round(float64(y) * scale))}
|
||||
}
|
||||
for i := 0; i < len(cmds); {
|
||||
id, count := cmds[i]&7, int(cmds[i]>>3)
|
||||
i++
|
||||
switch id {
|
||||
case 1, 2: // MoveTo, LineTo
|
||||
if i+2*count > len(cmds) {
|
||||
return nil, errBadTile
|
||||
}
|
||||
for k := 0; k < count; k++ {
|
||||
x += zig(cmds[i])
|
||||
y += zig(cmds[i+1])
|
||||
i += 2
|
||||
if id == 1 {
|
||||
if len(cur) >= 2 {
|
||||
parts = append(parts, cur)
|
||||
}
|
||||
cur = [][2]int{at()}
|
||||
} else {
|
||||
if p := at(); len(cur) == 0 || cur[len(cur)-1] != p {
|
||||
cur = append(cur, p)
|
||||
}
|
||||
}
|
||||
}
|
||||
case 7: // ClosePath, not in a line; ignored
|
||||
default:
|
||||
return nil, errBadTile
|
||||
}
|
||||
}
|
||||
if len(cur) >= 2 {
|
||||
parts = append(parts, cur)
|
||||
}
|
||||
return parts, nil
|
||||
}
|
||||
Reference in New Issue
Block a user