How telemetry streams work

Telemetry Streams

A telemetry stream is a named sequence of datapoints.

A datapoint is a typed, timestamped data payload. A datapoint can be optionally tagged with several key-value pairs.

A telemetry stream may contain text data, numerical data, perception data, navigation data, log files, or any other of Formant’s supported stream types. It is the core building block of the Formant data platform.

The full list of stream types is as follows:

Type

Description

Asset?

Text

A string

No

Numeric

An integer or float value

No

Numeric set

A set of labeled integer or float values

No

Bitset

A set of labeled bits

No

Location

Latitude and longitude

No

Health

Sent by the Formant agent to the reserved stream $.agent.health. Values are unknown, operational, offline, or error

No

Battery

Percentage, voltage, current, and charge

No

File

A raw file of any type

Yes

Image

A compressed image file bytestring (jpeg or png)

Yes

Video

A video file (mp4)

Yes

Point cloud

A raw point cloud

Yes

Localization

A unified localization datapoint with odometry, map, an array of point clouds, a path, and a goal

Yes

Json

A JSON-encoded string

Yes

Transform tree

A raw transform tree

Yes

See the Formant agent's datapoint ingestion schema for a more detailed reference: https://github.com/FormantIO/formant/blob/826322aca53a6cc112346d171729f07e17e62e58/protos/model/v1/datapoint.proto#L16

Ingestion from a device

The process by which telemetry data goes from a device to the cloud is called ingestion.

The primary method of ingesting telemetry data is via the Formant agent. The Formant agent runs on each device in your organization and is responsible for delivering telemetry data to the cloud.

The agent has built-in mechanisms to add guarantees to the ingestion process. The most important ones to understand are buffering and throttling.

Mechanism #1: Buffering

The Formant agent maintains four ring buffers that persist to disk.

Each buffer stores in-flight datapoints that have been received by the Formant agent, but have yet to be delivered to the cloud. Datapoints from default, hardware, ROS, and API sources all flow through these buffers. The four buffers are:

  • Datapoint buffer
  • Asset buffer
  • Datapoint on-demand buffer
  • Asset on-demand buffer

We'll go through what each of those mean, but first, let's define what datapoints are defined as assets, and what on-demand streams are.

Assets vs Datapoints

Some data is too large to send around raw in batch ingestion requests. For these, we upload the data to storage and only send around the URL. These datapoints are considered assets.

The asset buffers store the following stream types:

  • File
  • Image
  • Video
  • Point cloud
  • Localization
  • Json
  • Transform tree

By contrast, the datapoint buffers store the remaining types:

  • Text
  • Numeric
  • Numeric set
  • Bitset
  • Location
  • Health
  • Battery

Refer to the table toward the beginning of this document for a reference with descriptions.

On-demand streams

Streams can be configured to be either "live" (default) or "on-demand."

Live streams will upload data when they can. If the device has an internet connection, and the Formant agent has datapoints for a live stream, the device will be attempting to upload these datapoints.

On-demand streams upload data from a device to the cloud on request. This request is usually done manually by a user of the Formant web application, but can also be done using an API call.

Buffer types

Let's define those four types of buffers now that we have full context:

  • Datapoint buffer : Non-asset datapoints for live streams
  • Asset buffer : Asset datapoints for live streams
  • Datapoint on-demand buffer : Non-asset datapoints for on-demand streams
  • Asset on-demand buffer : Asset datapoints for on-demand streams

Each buffer's size on disk is individually configurable. When a buffer overflows, data is overwritten by newer datapoints in a FIFO process. Datapoints inside a buffer are also uploaded to the cloud FIFO.

Mechanism #2: Throttling

The Formant agent throttles incoming datapoints at a configurable stream rate of up to 20Hz, with 5Hz being the default stream rate. What does this mean?

When the agent receives a new datapoint, it checks if at least 1/(stream rate) seconds have elapsed since the previous datapoint was successfully received on this stream. If not enough time has passed, the datapoint is throttled, which means dropped.

So, if datapoints are being sent to the agent at a rate of 10Hz, and the stream rate is 5Hz (the default), every other datapoint will be dropped.

Why throttle?

Throttling is an easy knob in Formant to control bandwidth allocation and reduce the number of datapoints being uploaded, without having to change anything about how your robot-side application works.


Did this page help you?