Skip to content

Fleet Management & Device Clouds

Module 4-02 covered securing and rolling out an OTA update. This module is about everything a fleet of deployed devices needs beyond the update mechanism itself: identity, telemetry ingestion at scale, remote configuration, and the operational visibility to know a fleet is healthy without physically touching any device.

Device identity: the foundation everything else depends on

Every device in a fleet needs a stable, unique identity that survives firmware updates and doesn't depend on anything that can be spoofed or duplicated by accident. A common, sound pattern:

/* device identity, provisioned once at manufacturing time (module 4-08),
   never regenerated by application firmware */
typedef struct {
    uint8_t  device_uuid[16];     /* globally unique, provisioned at manufacture */
    uint8_t  cert_chain[512];      /* device certificate, signed by the fleet's CA (module 4-02) */
    char     fw_version[16];
} device_identity_t;

Deriving identity from something like a MAC address alone is a common mistake — MAC addresses can collide across manufacturing lots from different chip vendors, get reused after RMA, or simply aren't guaranteed unique across every radio a device might carry. A properly provisioned UUID plus a certificate the fleet's own CA issued is the pattern that scales to real fleet sizes and survives hardware component swaps during repair.

Telemetry: sampling and reporting policy, not just "send everything"

A fleet of thousands of devices reporting raw sensor data continuously overwhelms both the network and the backend, and burns power on devices that could be sleeping instead (module 3-05/4-07). Real fleet telemetry design distinguishes report cadence from sample cadence:

typedef struct {
    float value;
    uint32_t timestamp_ms;
} sample_t;

/* device samples frequently, locally, but only REPORTS a summary —
   this is the same idea as module 3-10's ring buffer, extended with
   aggregation before transmission rather than sending every raw sample */
typedef struct {
    float min, max, sum;
    uint32_t count;
} telemetry_summary_t;

void telemetry_summary_add(telemetry_summary_t *s, float value) {
    if (s->count == 0 || value < s->min) s->min = value;
    if (s->count == 0 || value > s->max) s->max = value;
    s->sum += value;
    s->count++;
}

float telemetry_summary_mean(const telemetry_summary_t *s) {
    return s->count > 0 ? s->sum / s->count : 0.0f;
}

Reporting {min, max, mean, count} once a minute instead of every raw sample cuts payload size and radio-on time dramatically while still surfacing anomalies (a min/max outside expected range) that a pure average would hide.

Remote configuration: versioned, validated, and rollback-capable

Pushing configuration changes to a fleet has the same integrity requirements as pushing firmware (module 4-02) — an invalid or malformed configuration pushed to thousands of devices is a fleet-wide incident, not a single-device bug:

typedef struct {
    uint32_t config_version;
    uint32_t report_interval_ms;
    uint32_t sample_interval_ms;
    float    alert_threshold;
} device_config_t;

/* validate before ever applying — the same principle as module 4-02's
   verify-before-install for firmware, applied to configuration */
int validate_config(const device_config_t *cfg) {
    if (cfg->report_interval_ms < cfg->sample_interval_ms) return -1;  /* nonsensical */
    if (cfg->alert_threshold < 0.0f) return -2;
    if (cfg->sample_interval_ms == 0) return -3;   /* would busy-loop */
    return 0;
}

Modeling telemetry aggregation and config validation

Both are pure logic, verified without any real fleet backend. Compiled and run with gcc:

#include <stdio.h>
#include <assert.h>
#include <math.h>
#include <stdint.h>

/* telemetry_summary_t / telemetry_summary_add / telemetry_summary_mean
   and device_config_t / validate_config as above */

int main(void) {
    telemetry_summary_t s = {0};
    float readings[] = {20.0f, 22.5f, 19.0f, 25.0f, 21.0f};
    for (int i = 0; i < 5; i++) telemetry_summary_add(&s, readings[i]);

    assert(s.count == 5);
    assert(fabsf(s.min - 19.0f) < 0.001f);
    assert(fabsf(s.max - 25.0f) < 0.001f);
    assert(fabsf(telemetry_summary_mean(&s) - 21.5f) < 0.001f);

    device_config_t good = { 1, 60000, 1000, 30.0f };
    device_config_t bad_order = { 1, 1000, 60000, 30.0f };   /* report faster than sample */
    device_config_t bad_zero  = { 1, 60000, 0, 30.0f };

    assert(validate_config(&good) == 0);
    assert(validate_config(&bad_order) == -1);
    assert(validate_config(&bad_zero) == -3);

    printf("telemetry aggregation + config validation model OK\n");
    return 0;
}

Traps in fleet management at scale

  • Identity tied to something mutable or reusable: as above — MAC-based identity is the single most common mistake here.
  • No cardinality limits on telemetry: a fleet-wide backend that accepts arbitrary-cardinality tags (e.g. a raw error message string as a metric label) can degrade or crash the telemetry pipeline itself once enough devices are reporting — this is an operational fleet-scale failure mode that doesn't exist at prototype scale with one device.
  • Config push with no staged rollout: exactly module 4-02's canary argument applies to configuration changes as much as firmware — an invalid threshold pushed fleet-wide can trigger false alerts (or silence real ones) across every device simultaneously.
  • No offline/disconnected behavior defined: a device that assumes connectivity is always available and blocks indefinitely trying to report telemetry, instead of buffering locally and retrying, effectively stops functioning during any network outage even if its actual sensing/control function doesn't depend on connectivity at all.

How It Actually Works

Why MAC-address-based identity fails at fleet scale specifically: a MAC address is assigned by whichever radio/networking chip vendor supplied a given batch of parts, drawn from that vendor's allocated address block — a fleet sourcing the same MCU/radio combination across multiple manufacturing lots, or worse, from two different vendors' compatible parts as a supply-chain substitution, has no guarantee those blocks don't overlap or that a vendor's own allocation process was error-free at the scale of millions of parts (documented collisions exist in the wild). Worse, MAC addresses are typically stored in the radio chip's own fuses/EEPROM, not the main MCU's — a board repair that swaps a failed radio module changes the device's "identity" without any application code decision at all, silently breaking any backend record keyed on that value. A provisioned UUID, by contrast, is generated and burned in specifically as an identity field (often alongside the device certificate, in the same manufacturing step as module 4-08's factory test) independent of any component that might later be swapped during repair — the identity is a deliberate manufacturing artifact, not a side-effect of which radio chip happened to be soldered on.

Why aggregating telemetry before transmission is a real bandwidth/energy trade and not just data reduction for its own sake: transmitting data over a radio (module 4-04's link budget) costs energy roughly proportional to payload size and, more significantly, to radio-on time — bringing a radio out of its low-power state, establishing a connection or transmission window, and returning to sleep carries fixed overhead cost regardless of how much actual data rides along, similar in kind to a flash sector's fixed erase cost from module 2-07 dominating over the actual bytes written. A device sampling every second but transmitting a summary once a minute pays that fixed radio overhead once per 60 samples instead of once per sample — a 60x reduction in the dominant cost — while {min, max, mean, count} still preserves the statistical shape of that minute's data (an anomalous spike is visible in max even though it's smoothed out of mean), which is exactly why the aggregate is chosen to include min/max and not just an average.

Why config validation must run identically to firmware verify-before-apply, mechanically: from the receiving device's perspective, an incoming configuration blob and an incoming firmware image are both just untrusted bytes arriving over a network connection, parsed by code that has no inherent guarantee the sender validated them correctly (a backend bug, a truncated transfer, or a malicious actor with access to the update channel are all indistinguishable failure modes from the device's point of view). A sample_interval_ms == 0 accepted without validation isn't a hypothetical — it's a divisor or loop-bound elsewhere in the firmware that, given that specific value, produces a divide-by-zero fault or an unbounded busy-loop the instant the new configuration is applied, on every device in the fleet simultaneously if pushed without staging — which is precisely why validate_config runs the identical logical role as esp_ota_end()'s image validation in module 2-05: the last checkpoint before untrusted external input becomes live device behavior.

Cheat sheet

Concept Detail
Device identity Provisioned UUID + fleet-CA-signed certificate — never a MAC address alone
Telemetry aggregation Report summaries (min/max/mean/count) on an interval, not every raw sample
Config validation Validate before applying, same principle as firmware verify-before-install
Staged config rollout Canary/staged pattern from module 4-02 applies to configuration, not just firmware
Cardinality limits Fleet-scale telemetry backends need bounded label/tag cardinality to stay operational
Offline behavior Devices must buffer and retry, not block, during connectivity loss
Verification here Aggregation/validation logic compiled/run with gcc; real fleet backend/network behavior not represented

Exercise

Extend the telemetry summary with an alert_triggered flag set when any individual sample (not just the aggregate mean) crosses a device_config_t.alert_threshold, and make telemetry_summary_add take the threshold as a parameter so it can flag this at sample time rather than only after aggregation. Write assertions for: no sample crossing the threshold (no alert), one brief spike crossing it that the mean alone would hide (alert still fires), and confirm the aggregate min/max correctly reflect the spike even though the alert and the aggregation are now computed together. Compile and run with gcc.