kosmo-connect/docs/architecture/data-flow.md

Data Flow

This document describes how environmental data moves from the sensor to the user's web browser, and how control/commands flow in the opposite direction.

1. Sensor Reading & Local Storage

Frequency: Every 60 seconds (configurable) Actor: Enviro-Node firmware

1. MCU wakes from deep sleep (or remains active if interval is short)
2. Sensors are powered on, stabilized, and read
3. Raw readings are calibrated and packaged into a compact binary format
4. The packet is appended to a local ring buffer in SPI flash or SD card
5. A "data ready" flag is set for the Meshtastic module

Data Packet Structure (Enviro-Node Local)

typedef struct {
    uint32_t timestamp_unix;
    int16_t  temperature_c;     // 0.01°C resolution
    uint16_t humidity_percent;  // 0.01% resolution
    uint32_t pressure_pa;
    uint16_t wind_speed_ms;     // 0.1 m/s resolution
    uint16_t wind_direction;    // degrees
    uint16_t pm25_ugm3;
    uint16_t pm10_ugm3;
    uint16_t gas_resistance_kohm;
    uint8_t  node_id[6];        // Meshtastic Node ID
    uint16_t crc16;
} enviro_packet_t;

2. Mesh Transmission

Frequency: Every 5-15 minutes (configurable, power-dependent) Actor: Meshtastic firmware + custom module

1. The custom module requests one or more packets from the local buffer
2. Packets are encoded into a Meshtastic DATA payload on a dedicated environmental channel
3. The packet is broadcast into the mesh with want_ack = false (fire-and-forget for efficiency)
4. If an infrastructure node is within range (direct or multi-hop), it receives the packet
5. If no ACK or route is available, the packet remains in the buffer for the next transmission window

Channel Strategy

• Primary Channel: Standard Meshtastic LongFast for relaying user messages
• Secondary Channel: Custom KOSMO_ENV channel for environmental data (can use different frequency slot or SF to avoid congesting primary channel)

3. Bridge Ingestion

Actor: Infrastructure Node Bridge Daemon

1. The bridge daemon listens to Meshtastic packets via the serial/protobuf API
2. It filters for packets on the KOSMO_ENV channel or with a specific portnum
3. Valid environmental packets are decoded and wrapped in a JSON envelope:

   {
     "type": "enviro_reading",
     "node_id": "!a1b2c3d4",
     "received_at": "2026-04-12T09:15:00Z",
     "hop_count": 2,
     "payload": {
       "timestamp": 1744446900,
       "temperature_c": 18.50,
       "humidity_percent": 62.30,
       ...
     }
   }
   

4. The envelope is published to the cloud MQTT broker topic: kosmo/ingest/enviro

4. Cloud Ingestion

Actor: Backend Ingestion Service

1. The ingestion service subscribes to kosmo/ingest/#
2. On receiving a message:
   • Validate JSON schema
   • Verify node_id is registered and active
   • Write raw payload to TimescaleDB hypertable enviro_readings
   • Update node last_seen timestamp in PostgreSQL
   • If the node has a backlog, trigger a "sync complete" notification (optional)

Database Schema (Simplified)

-- TimescaleDB
CREATE TABLE enviro_readings (
    time TIMESTAMPTZ NOT NULL,
    node_id TEXT NOT NULL,
    temperature_c DOUBLE PRECISION,
    humidity_percent DOUBLE PRECISION,
    pressure_pa DOUBLE PRECISION,
    wind_speed_ms DOUBLE PRECISION,
    wind_direction SMALLINT,
    pm25_ugm3 DOUBLE PRECISION,
    pm10_ugm3 DOUBLE PRECISION,
    gas_resistance_kohm DOUBLE PRECISION
);
SELECT create_hypertable('enviro_readings', by_range('time'));

-- PostgreSQL
CREATE TABLE nodes (
    id UUID PRIMARY KEY,
    mesh_node_id TEXT UNIQUE NOT NULL,
    name TEXT,
    location GEOGRAPHY(POINT, 4326),
    hardware_revision TEXT,
    installed_at TIMESTAMPTZ,
    last_seen TIMESTAMPTZ,
    is_active BOOLEAN DEFAULT true
);

5. Web Dashboard Display

Actor: Web Dashboard (React)

1. User loads the dashboard
2. Frontend queries /api/v1/weather/latest and /api/v1/weather/history
3. API service fetches aggregated data from TimescaleDB
4. Frontend renders:
   • Map markers with latest readings
   • Time-series charts (temperature, wind, etc.)
   • Node health indicators (battery, signal strength, last seen)

6. Command Flow (Web to Node)

For configuration updates or remote diagnostics:

1. Admin sends a command via web admin panel (e.g., "change reporting interval to 10 min")
2. API validates admin permissions
3. Command is queued in the message gateway for the specific node
4. Infrastructure node picks up the command via MQTT
5. Bridge daemon injects the command as a Meshtastic admin packet
6. Enviro-node receives and applies the config update
7. Acknowledgment (if requested) flows back through the same path

Data Retention Policy

Data Type	Storage Location	Retention
Raw sensor readings	Enviro-Node flash	30-90 days (ring buffer)
Ingested readings	TimescaleDB	2 years raw, then downsampled
Downsampled aggregates	TimescaleDB	Indefinite
Mesh messages	PostgreSQL	90 days
Audit logs	PostgreSQL	1 year