feat: initial KosmoConnect platform v0.1
Includes: - Backend services: ingestion (:8001), weather API (:8002), gateway (:8003), billing (:8004) with BTCPay integration - Shared asyncpg pool, TimescaleDB hypertable, Redis, Mosquitto MQTT - React frontend: Dashboard (MapLibre) and Messaging (chat UI) - Bridge daemon for Pi + Meshtastic (Serial/TCP T-Deck support) - Production Docker Compose, Nginx reverse proxy, ops scripts - DEPLOY.md with step-by-step deployment guide
This commit is contained in:
145
docs/architecture/data-flow.md
Normal file
145
docs/architecture/data-flow.md
Normal file
@@ -0,0 +1,145 @@
|
||||
# Data Flow
|
||||
|
||||
This document describes how environmental data moves from the sensor to the user's web browser, and how control/commands flow in the opposite direction.
|
||||
|
||||
## 1. Sensor Reading & Local Storage
|
||||
|
||||
**Frequency**: Every 60 seconds (configurable)
|
||||
**Actor**: Enviro-Node firmware
|
||||
|
||||
1. MCU wakes from deep sleep (or remains active if interval is short)
|
||||
2. Sensors are powered on, stabilized, and read
|
||||
3. Raw readings are calibrated and packaged into a compact binary format
|
||||
4. The packet is appended to a local ring buffer in SPI flash or SD card
|
||||
5. A "data ready" flag is set for the Meshtastic module
|
||||
|
||||
### Data Packet Structure (Enviro-Node Local)
|
||||
```c
|
||||
typedef struct {
|
||||
uint32_t timestamp_unix;
|
||||
int16_t temperature_c; // 0.01°C resolution
|
||||
uint16_t humidity_percent; // 0.01% resolution
|
||||
uint32_t pressure_pa;
|
||||
uint16_t wind_speed_ms; // 0.1 m/s resolution
|
||||
uint16_t wind_direction; // degrees
|
||||
uint16_t pm25_ugm3;
|
||||
uint16_t pm10_ugm3;
|
||||
uint16_t gas_resistance_kohm;
|
||||
uint8_t node_id[6]; // Meshtastic Node ID
|
||||
uint16_t crc16;
|
||||
} enviro_packet_t;
|
||||
```
|
||||
|
||||
## 2. Mesh Transmission
|
||||
|
||||
**Frequency**: Every 5-15 minutes (configurable, power-dependent)
|
||||
**Actor**: Meshtastic firmware + custom module
|
||||
|
||||
1. The custom module requests one or more packets from the local buffer
|
||||
2. Packets are encoded into a Meshtastic `DATA` payload on a dedicated environmental channel
|
||||
3. The packet is broadcast into the mesh with `want_ack = false` (fire-and-forget for efficiency)
|
||||
4. If an infrastructure node is within range (direct or multi-hop), it receives the packet
|
||||
5. If no ACK or route is available, the packet remains in the buffer for the next transmission window
|
||||
|
||||
### Channel Strategy
|
||||
- **Primary Channel**: Standard Meshtastic LongFast for relaying user messages
|
||||
- **Secondary Channel**: Custom `KOSMO_ENV` channel for environmental data (can use different frequency slot or SF to avoid congesting primary channel)
|
||||
|
||||
## 3. Bridge Ingestion
|
||||
|
||||
**Actor**: Infrastructure Node Bridge Daemon
|
||||
|
||||
1. The bridge daemon listens to Meshtastic packets via the serial/protobuf API
|
||||
2. It filters for packets on the `KOSMO_ENV` channel or with a specific portnum
|
||||
3. Valid environmental packets are decoded and wrapped in a JSON envelope:
|
||||
```json
|
||||
{
|
||||
"type": "enviro_reading",
|
||||
"node_id": "!a1b2c3d4",
|
||||
"received_at": "2026-04-12T09:15:00Z",
|
||||
"hop_count": 2,
|
||||
"payload": {
|
||||
"timestamp": 1744446900,
|
||||
"temperature_c": 18.50,
|
||||
"humidity_percent": 62.30,
|
||||
...
|
||||
}
|
||||
}
|
||||
```
|
||||
4. The envelope is published to the cloud MQTT broker topic: `kosmo/ingest/enviro`
|
||||
|
||||
## 4. Cloud Ingestion
|
||||
|
||||
**Actor**: Backend Ingestion Service
|
||||
|
||||
1. The ingestion service subscribes to `kosmo/ingest/#`
|
||||
2. On receiving a message:
|
||||
- Validate JSON schema
|
||||
- Verify `node_id` is registered and active
|
||||
- Write raw payload to TimescaleDB hypertable `enviro_readings`
|
||||
- Update node `last_seen` timestamp in PostgreSQL
|
||||
- If the node has a backlog, trigger a "sync complete" notification (optional)
|
||||
|
||||
### Database Schema (Simplified)
|
||||
```sql
|
||||
-- TimescaleDB
|
||||
CREATE TABLE enviro_readings (
|
||||
time TIMESTAMPTZ NOT NULL,
|
||||
node_id TEXT NOT NULL,
|
||||
temperature_c DOUBLE PRECISION,
|
||||
humidity_percent DOUBLE PRECISION,
|
||||
pressure_pa DOUBLE PRECISION,
|
||||
wind_speed_ms DOUBLE PRECISION,
|
||||
wind_direction SMALLINT,
|
||||
pm25_ugm3 DOUBLE PRECISION,
|
||||
pm10_ugm3 DOUBLE PRECISION,
|
||||
gas_resistance_kohm DOUBLE PRECISION
|
||||
);
|
||||
SELECT create_hypertable('enviro_readings', by_range('time'));
|
||||
|
||||
-- PostgreSQL
|
||||
CREATE TABLE nodes (
|
||||
id UUID PRIMARY KEY,
|
||||
mesh_node_id TEXT UNIQUE NOT NULL,
|
||||
name TEXT,
|
||||
location GEOGRAPHY(POINT, 4326),
|
||||
hardware_revision TEXT,
|
||||
installed_at TIMESTAMPTZ,
|
||||
last_seen TIMESTAMPTZ,
|
||||
is_active BOOLEAN DEFAULT true
|
||||
);
|
||||
```
|
||||
|
||||
## 5. Web Dashboard Display
|
||||
|
||||
**Actor**: Web Dashboard (React)
|
||||
|
||||
1. User loads the dashboard
|
||||
2. Frontend queries `/api/v1/weather/latest` and `/api/v1/weather/history`
|
||||
3. API service fetches aggregated data from TimescaleDB
|
||||
4. Frontend renders:
|
||||
- Map markers with latest readings
|
||||
- Time-series charts (temperature, wind, etc.)
|
||||
- Node health indicators (battery, signal strength, last seen)
|
||||
|
||||
## 6. Command Flow (Web to Node)
|
||||
|
||||
For configuration updates or remote diagnostics:
|
||||
|
||||
1. Admin sends a command via web admin panel (e.g., "change reporting interval to 10 min")
|
||||
2. API validates admin permissions
|
||||
3. Command is queued in the message gateway for the specific node
|
||||
4. Infrastructure node picks up the command via MQTT
|
||||
5. Bridge daemon injects the command as a Meshtastic admin packet
|
||||
6. Enviro-node receives and applies the config update
|
||||
7. Acknowledgment (if requested) flows back through the same path
|
||||
|
||||
## Data Retention Policy
|
||||
|
||||
| Data Type | Storage Location | Retention |
|
||||
|-----------|-----------------|-----------|
|
||||
| Raw sensor readings | Enviro-Node flash | 30-90 days (ring buffer) |
|
||||
| Ingested readings | TimescaleDB | 2 years raw, then downsampled |
|
||||
| Downsampled aggregates | TimescaleDB | Indefinite |
|
||||
| Mesh messages | PostgreSQL | 90 days |
|
||||
| Audit logs | PostgreSQL | 1 year |
|
||||
Reference in New Issue
Block a user