How telemetry streams work

Telemetry Streams

A telemetry stream is a named sequence of datapoints.

A datapoint is a typed, timestamped data payload. A datapoint can be optionally tagged with several key-value pairs.

A telemetry stream may contain text data, numerical data, perception data, navigation data, log files, or any other of Formant’s supported stream types. It is the core building block of the Formant data platform.

The full list of stream types is as follows:

TextA stringNo
NumericAn integer or float valueNo
Numeric setA set of labeled integer or float valuesNo
BitsetA set of labeled bitsNo
LocationLatitude and longitudeNo
HealthSent by the Formant agent to the reserved stream $ Values are unknown, operational, offline, or errorNo
BatteryPercentage, voltage, current, and chargeNo
FileA raw file of any typeYes
ImageA compressed image file bytestring (jpeg or png)Yes
VideoA video file (mp4)Yes
Point cloudA raw point cloudYes
LocalizationA unified localization datapoint with odometry, map, an array of point clouds, a path, and a goalYes
JsonA JSON-encoded stringYes
Transform treeA raw transform treeYes

See the Formant agent's datapoint ingestion schema for a more detailed reference:

Ingestion from a device

The process by which telemetry data goes from a device to the cloud is called ingestion.

The primary method of ingesting telemetry data is via the Formant agent. The Formant agent runs on each device in your organization and is responsible for delivering telemetry data to the cloud.

The agent has built-in mechanisms to add guarantees to the ingestion process. The most important ones to understand are buffering and throttling.

Mechanism #1: Buffering

The Formant agent maintains four ring buffers that persist to disk.

Each buffer stores in-flight datapoints that have been received by the Formant agent, but have yet to be delivered to the cloud. Datapoints from default, hardware, ROS, and API sources all flow through these buffers. The four buffers are:

  • Datapoint buffer
  • Asset buffer
  • Datapoint on-demand buffer
  • Asset on-demand buffer

We'll go through what each of those mean, but first, let's define what datapoints are defined as assets, and what on-demand streams are.

Assets vs Datapoints

Some data is too large to send around raw in batch ingestion requests. For these, we upload the data to storage and only send around the URL. These datapoints are considered assets.

The asset buffers store the following stream types:

  • File
  • Image
  • Video
  • Point cloud
  • Localization
  • Json
  • Transform tree

By contrast, the datapoint buffers store the remaining types:

  • Text
  • Numeric
  • Numeric set
  • Bitset
  • Location
  • Health
  • Battery

Refer to the table toward the beginning of this document for a reference with descriptions.

On-demand streams

Streams can be configured to be either "live" (default) or "on-demand."

Live streams will upload data when they can. If the device has an internet connection, and the Formant agent has datapoints for a live stream, the device will be attempting to upload these datapoints.

On-demand streams upload data from a device to the cloud on request. This request is usually done manually by a user of the Formant web application, but can also be done using an API call.

Buffer types

Let's define those four types of buffers now that we have full context:

  • Datapoint buffer : Non-asset datapoints for live streams
  • Asset buffer : Asset datapoints for live streams
  • Datapoint on-demand buffer : Non-asset datapoints for on-demand streams
  • Asset on-demand buffer : Asset datapoints for on-demand streams

Each buffer's size on disk is individually configurable. When a buffer overflows, data is overwritten by newer datapoints in a FIFO process. Datapoints inside a buffer are also uploaded to the cloud FIFO.

Mechanism #2: Throttling

The Formant agent throttles incoming datapoints at a configurable stream rate of up to 20Hz, with 5Hz being the default stream rate. What does this mean?

When the agent receives a new datapoint, it checks if at least 1/(stream rate) seconds have elapsed since the previous datapoint was successfully received on this stream. If not enough time has passed, the datapoint is throttled, which means dropped.

So, if datapoints are being sent to the agent at a rate of 10Hz, and the stream rate is 5Hz (the default), every other datapoint will be dropped.

Why throttle?

Throttling is an easy knob in Formant to control bandwidth allocation and reduce the number of datapoints being uploaded, without having to change anything about how your robot-side application works.