Tomas Kracmar
0a4fb7b55e
Includes: - Backend services: ingestion (:8001), weather API (:8002), gateway (:8003), billing (:8004) with BTCPay integration - Shared asyncpg pool, TimescaleDB hypertable, Redis, Mosquitto MQTT - React frontend: Dashboard (MapLibre) and Messaging (chat UI) - Bridge daemon for Pi + Meshtastic (Serial/TCP T-Deck support) - Production Docker Compose, Nginx reverse proxy, ops scripts - DEPLOY.md with step-by-step deployment guide
146 lines
5.0 KiB
Markdown
146 lines
5.0 KiB
Markdown
# Data Flow
|
|
|
|
This document describes how environmental data moves from the sensor to the user's web browser, and how control/commands flow in the opposite direction.
|
|
|
|
## 1. Sensor Reading & Local Storage
|
|
|
|
**Frequency**: Every 60 seconds (configurable)
|
|
**Actor**: Enviro-Node firmware
|
|
|
|
1. MCU wakes from deep sleep (or remains active if interval is short)
|
|
2. Sensors are powered on, stabilized, and read
|
|
3. Raw readings are calibrated and packaged into a compact binary format
|
|
4. The packet is appended to a local ring buffer in SPI flash or SD card
|
|
5. A "data ready" flag is set for the Meshtastic module
|
|
|
|
### Data Packet Structure (Enviro-Node Local)
|
|
```c
|
|
typedef struct {
|
|
uint32_t timestamp_unix;
|
|
int16_t temperature_c; // 0.01°C resolution
|
|
uint16_t humidity_percent; // 0.01% resolution
|
|
uint32_t pressure_pa;
|
|
uint16_t wind_speed_ms; // 0.1 m/s resolution
|
|
uint16_t wind_direction; // degrees
|
|
uint16_t pm25_ugm3;
|
|
uint16_t pm10_ugm3;
|
|
uint16_t gas_resistance_kohm;
|
|
uint8_t node_id[6]; // Meshtastic Node ID
|
|
uint16_t crc16;
|
|
} enviro_packet_t;
|
|
```
|
|
|
|
## 2. Mesh Transmission
|
|
|
|
**Frequency**: Every 5-15 minutes (configurable, power-dependent)
|
|
**Actor**: Meshtastic firmware + custom module
|
|
|
|
1. The custom module requests one or more packets from the local buffer
|
|
2. Packets are encoded into a Meshtastic `DATA` payload on a dedicated environmental channel
|
|
3. The packet is broadcast into the mesh with `want_ack = false` (fire-and-forget for efficiency)
|
|
4. If an infrastructure node is within range (direct or multi-hop), it receives the packet
|
|
5. If no ACK or route is available, the packet remains in the buffer for the next transmission window
|
|
|
|
### Channel Strategy
|
|
- **Primary Channel**: Standard Meshtastic LongFast for relaying user messages
|
|
- **Secondary Channel**: Custom `KOSMO_ENV` channel for environmental data (can use different frequency slot or SF to avoid congesting primary channel)
|
|
|
|
## 3. Bridge Ingestion
|
|
|
|
**Actor**: Infrastructure Node Bridge Daemon
|
|
|
|
1. The bridge daemon listens to Meshtastic packets via the serial/protobuf API
|
|
2. It filters for packets on the `KOSMO_ENV` channel or with a specific portnum
|
|
3. Valid environmental packets are decoded and wrapped in a JSON envelope:
|
|
```json
|
|
{
|
|
"type": "enviro_reading",
|
|
"node_id": "!a1b2c3d4",
|
|
"received_at": "2026-04-12T09:15:00Z",
|
|
"hop_count": 2,
|
|
"payload": {
|
|
"timestamp": 1744446900,
|
|
"temperature_c": 18.50,
|
|
"humidity_percent": 62.30,
|
|
...
|
|
}
|
|
}
|
|
```
|
|
4. The envelope is published to the cloud MQTT broker topic: `kosmo/ingest/enviro`
|
|
|
|
## 4. Cloud Ingestion
|
|
|
|
**Actor**: Backend Ingestion Service
|
|
|
|
1. The ingestion service subscribes to `kosmo/ingest/#`
|
|
2. On receiving a message:
|
|
- Validate JSON schema
|
|
- Verify `node_id` is registered and active
|
|
- Write raw payload to TimescaleDB hypertable `enviro_readings`
|
|
- Update node `last_seen` timestamp in PostgreSQL
|
|
- If the node has a backlog, trigger a "sync complete" notification (optional)
|
|
|
|
### Database Schema (Simplified)
|
|
```sql
|
|
-- TimescaleDB
|
|
CREATE TABLE enviro_readings (
|
|
time TIMESTAMPTZ NOT NULL,
|
|
node_id TEXT NOT NULL,
|
|
temperature_c DOUBLE PRECISION,
|
|
humidity_percent DOUBLE PRECISION,
|
|
pressure_pa DOUBLE PRECISION,
|
|
wind_speed_ms DOUBLE PRECISION,
|
|
wind_direction SMALLINT,
|
|
pm25_ugm3 DOUBLE PRECISION,
|
|
pm10_ugm3 DOUBLE PRECISION,
|
|
gas_resistance_kohm DOUBLE PRECISION
|
|
);
|
|
SELECT create_hypertable('enviro_readings', by_range('time'));
|
|
|
|
-- PostgreSQL
|
|
CREATE TABLE nodes (
|
|
id UUID PRIMARY KEY,
|
|
mesh_node_id TEXT UNIQUE NOT NULL,
|
|
name TEXT,
|
|
location GEOGRAPHY(POINT, 4326),
|
|
hardware_revision TEXT,
|
|
installed_at TIMESTAMPTZ,
|
|
last_seen TIMESTAMPTZ,
|
|
is_active BOOLEAN DEFAULT true
|
|
);
|
|
```
|
|
|
|
## 5. Web Dashboard Display
|
|
|
|
**Actor**: Web Dashboard (React)
|
|
|
|
1. User loads the dashboard
|
|
2. Frontend queries `/api/v1/weather/latest` and `/api/v1/weather/history`
|
|
3. API service fetches aggregated data from TimescaleDB
|
|
4. Frontend renders:
|
|
- Map markers with latest readings
|
|
- Time-series charts (temperature, wind, etc.)
|
|
- Node health indicators (battery, signal strength, last seen)
|
|
|
|
## 6. Command Flow (Web to Node)
|
|
|
|
For configuration updates or remote diagnostics:
|
|
|
|
1. Admin sends a command via web admin panel (e.g., "change reporting interval to 10 min")
|
|
2. API validates admin permissions
|
|
3. Command is queued in the message gateway for the specific node
|
|
4. Infrastructure node picks up the command via MQTT
|
|
5. Bridge daemon injects the command as a Meshtastic admin packet
|
|
6. Enviro-node receives and applies the config update
|
|
7. Acknowledgment (if requested) flows back through the same path
|
|
|
|
## Data Retention Policy
|
|
|
|
| Data Type | Storage Location | Retention |
|
|
|-----------|-----------------|-----------|
|
|
| Raw sensor readings | Enviro-Node flash | 30-90 days (ring buffer) |
|
|
| Ingested readings | TimescaleDB | 2 years raw, then downsampled |
|
|
| Downsampled aggregates | TimescaleDB | Indefinite |
|
|
| Mesh messages | PostgreSQL | 90 days |
|
|
| Audit logs | PostgreSQL | 1 year |
|