How telemetry streams work
A telemetry stream is a named sequence of datapoints.
A datapoint is a typed, timestamped data payload. A datapoint can be optionally tagged with several key-value pairs.
A telemetry stream may contain text data, numerical data, perception data, navigation data, log files, or any other of Formant’s supported stream types. It is the core building block of the Formant data platform.
The full list of stream types is as follows:
|Numeric||An integer or float value||No|
|Numeric set||A set of labeled integer or float values||No|
|Bitset||A set of labeled bits||No|
|Location||Latitude and longitude||No|
|Health||Sent by the Formant agent to the reserved stream ||No|
|Battery||Percentage, voltage, current, and charge||No|
|File||A raw file of any type||Yes|
|Image||A compressed image file bytestring (jpeg or png)||Yes|
|Video||A video file (mp4)||Yes|
|Point cloud||A raw point cloud||Yes|
|Localization||A unified localization datapoint with odometry, map, an array of point clouds, a path, and a goal||Yes|
|Json||A JSON-encoded string||Yes|
|Transform tree||A raw transform tree||Yes|
See the Formant agent's datapoint ingestion schema for a more detailed reference: https://github.com/FormantIO/formant/blob/826322aca53a6cc112346d171729f07e17e62e58/protos/model/v1/datapoint.proto#L16
Ingestion from a device
The process by which telemetry data goes from a device to the cloud is called ingestion.
The primary method of ingesting telemetry data is via the Formant agent. The Formant agent runs on each device in your organization and is responsible for delivering telemetry data to the cloud.
The agent has built-in mechanisms to add guarantees to the ingestion process. The most important ones to understand are buffering and throttling.
Mechanism #1: Buffering
The Formant agent maintains four ring buffers that persist to disk.
Each buffer stores in-flight datapoints that have been received by the Formant agent, but have yet to be delivered to the cloud. Datapoints from default, hardware, ROS, and API sources all flow through these buffers. The four buffers are:
- Datapoint buffer
- Asset buffer
- Datapoint on-demand buffer
- Asset on-demand buffer
We'll go through what each of those mean, but first, let's define what datapoints are defined as assets, and what on-demand streams are.
Assets vs Datapoints
Some data is too large to send around raw in batch ingestion requests. For these, we upload the data to storage and only send around the URL. These datapoints are considered assets.
The asset buffers store the following stream types:
- Point cloud
- Transform tree
By contrast, the datapoint buffers store the remaining types:
- Numeric set
Refer to the table toward the beginning of this document for a reference with descriptions.
Streams can be configured to be either "live" (default) or "on-demand."
Live streams will upload data when they can. If the device has an internet connection, and the Formant agent has datapoints for a live stream, the device will be attempting to upload these datapoints.
On-demand streams upload data from a device to the cloud on request. This request is usually done manually by a user of the Formant web application, but can also be done using an API call.
Let's define those four types of buffers now that we have full context:
- Datapoint buffer : Non-asset datapoints for live streams
- Asset buffer : Asset datapoints for live streams
- Datapoint on-demand buffer : Non-asset datapoints for on-demand streams
- Asset on-demand buffer : Asset datapoints for on-demand streams
Each buffer's size on disk is individually configurable. When a buffer overflows, data is overwritten by newer datapoints in a FIFO process. Datapoints inside a buffer are also uploaded to the cloud FIFO.
Mechanism #2: Throttling
The Formant agent throttles incoming datapoints at a configurable stream rate of up to 20Hz, with 5Hz being the default stream rate. What does this mean?
When the agent receives a new datapoint, it checks if at least
1/(stream rate) seconds have elapsed since the previous datapoint was successfully received on this stream. If not enough time has passed, the datapoint is throttled, which means dropped.
So, if datapoints are being sent to the agent at a rate of 10Hz, and the stream rate is 5Hz (the default), every other datapoint will be dropped.
Throttling is an easy knob in Formant to control bandwidth allocation and reduce the number of datapoints being uploaded, without having to change anything about how your robot-side application works.
Updated 10 months ago