The engineering bottleneck with assembled stacks
Hardware teams post telemetry from a fleet and inherit a streaming stack sized for a stock exchange: brokers, stream processors, and a schema registry to record that a sensor said 21.4 degrees. The fleet is four hundred devices sending every five minutes, and the platform’s operational weight exceeds the product it feeds by an order of magnitude.
Architecture on the Software Lifecycle Cloud
Ingest is a write pattern: the function checks the device key, stamps arrival time beside the device’s own timestamp, and appends the reading under the device key with the reading time as sort key. Clocks lie and networks retry, so arrival order is never trusted — queries sort by device time, and duplicate device timestamps overwrite idempotently. Aggregations (hourly means, last-state) are worker jobs writing summary tables the product reads.
Late data is normal data
Devices go offline in tunnels, on ships, in barns. A store keyed by device and reading time absorbs a week-late batch as naturally as a live trickle: the rows land in order, summaries recompute over their windows, and nothing about the pipeline treats the event as an exception. Designing for lateness up front is the difference between a telemetry store and a telemetry incident generator.
- Per-device keys mean a compromised device is one revocation, not a fleet event
- Last-known-state is one read of the newest row per device — the query every dashboard actually runs
- Summaries are worker jobs over the raw table, so new aggregations can be backfilled from history
Where this pattern stops
Sub-second alerting over millions of devices per minute is streaming work with streaming tools. The honest boundary is volume and latency: product fleets reporting on human timescales belong here, where the infrastructure is a table and the operating team is whoever reads this page — not a platform group.
Frequently asked questions
With per-device API keys on the route, minted and rotated as declared gateway keys. Each key maps to a device id in a devices table, so provisioning, revocation, and “which firmware is this” live with the fleet data.
They are written under the device’s own timestamp, so series read correctly regardless of arrival order; duplicate timestamps overwrite by key. Summary jobs recompute affected windows, which is a bounded, ordinary operation.
Alert rules and channels are stack resources, and threshold checks run in the aggregation workers. For the timescales this pattern serves — minutes, not milliseconds — that covers the alerting products actually promise customers.
Related capabilities, architectures, and comparisons
Ship your stack on the Software Lifecycle Cloud.
Closed preview on dedicated European infrastructure. Bring a repository or start from a blueprint.