How to manage a fleet of weather stations
A weather station is a perfect example of a device nobody wants to visit. It sits on a mast, a roof or a hilltop, it runs on a small solar panel, and it has to keep reporting for years. The software on it still needs to change: a new sensor driver, a fix for a bad reading, a different upload interval. This is how teams manage a whole network of stations from one place.
The shape of a station
Most stations are a small Linux computer, often a Raspberry Pi, with a few sensors on its pins or on a USB bus. The software is usually two or three small programs: one that reads the sensors, one that stores or uploads the readings, and sometimes a small local web page. That is exactly what a multi-service release describes.
{"services": {
"reader": {"command": "./reader/run.sh", "cwd": "volume"},
"uploader": {"command": "./uploader/run.sh", "dependsOn": ["reader"]},
"page": {"command": "./page/run.sh", "port": 8080, "healthPath": "/health", "expose": true}}}The volume setting keeps the readings between updates, so a new release never throws away what a station has collected.
Updates that survive bad weather
Stations lose power and network all the time. A release is staged next to the old one and only becomes current after the new services pass their health check. If the power goes in the middle of an update, the station comes back on the old or the new release, never in between. If a service fails its check, the device goes back to the previous release by itself.
Roll out in steps
Do not update 200 stations at once. Mark a few stations on your own roof as canaries, then send a release to them first and widen it to 5, 25, 50 percent and finally everyone. If a canary reports a problem, you stop and nothing else has changed.
tuna push weather --notes "new wind sensor driver" --percent 5Know which station went quiet
Every station checks in on its own interval. A station that stops checking in turns up as silent, and you can get a notification instead of finding out from a gap in the data a week later. Groups and tags (for example by region, or by hardware revision) make it easy to filter the map of problems down to the stations that matter.
Look at a station without driving there
Each device can have a public address that shows its local web page, even behind a mobile modem, so you can open the station's own status page from your desk. For the rare case where you need a shell, the remote terminal is off by default, needs consent each time, and every session is recorded.
Settings that differ per site
Altitude, station name, upload endpoint: each station needs a few values of its own. Set defaults for the fleet and let any device override its own. The overrides are marked, so you can see which stations are special.