Streaming image and video data

The post_image method can be used to send image or video data to the Formant Agent.

Posting a single image datapoint to telemetry

import cv2
from formant.sdk.agent.v1 import Client as FormantClient

if __name__ == "__main__":
    fclient = FormantClient()

    image = cv2.imread("/path/to/example.jpg")
    encoded = cv2.imencode(".jpg", image)[1].tostring()

    fclient.post_image("example.image", encoded)

By default, post_image accepts JPEGs. To send a PNG image, add the keyword argument content_type="image/png", e.g. fclient.post_image("example.image", encoded, content_type="image/png")

Streaming video to telemetry or teleop

The Formant Agent can automatically encode high-frequency image streams to H264 video data. It can also accept a raw H264 stream directly.

🚧

Consider using a hardware video stream

Hardware streams can automate the process of reading and streaming USB and IP cameras.

See Telemetry hardware streams.

Streaming from a USB camera

import sys
import cv2
from formant.sdk.agent.v1 import Client as FormantClient

if __name__ == "__main__":
    fclient = FormantClient(ignore_throttled=True, ignore_unavailable=True)

    cap = cv2.VideoCapture(0)  # usb cam may be on a different index: try 1, 2 ...
    if cap is None:
        sys.exit()

    while True:
        _, image = cap.read()
        encoded = cv2.imencode(".jpg", image)[1].tostring()
        fclient.post_image("usb_cam", encoded)

When encoding is turned on in device telemetry or teleop configuration for the stream "usb_cam", the JPEG stream sent with post_image in this script will be automatically encoded into an H264 stream by the Formant Agent and streamed as telemetry and/or teleop data.

If you have direct access to a raw H264 stream and wish to ingest it, use content_type="video/h264".

Configuration

The stream that is sent on can be configured in both telemetry and teleop configuration for a device.

Telemetry image API stream configuration

No configuration is required for images. Consider using video to preserve bandwidth unless the images are meant to be single frame shots.

Telemetry video API stream configuration

  1. Navigate to Settings->Devices
  2. Locate the name of the stream
  3. Make sure "enable encoding" is toggled on in the stream's configuration.
  4. Save
12421242
  1. Navigate to Settings->Streams
  2. Find the stream in the list of streams for your organization.
  3. Make sure the "type" dropdown is set to "video".
  4. Save
14361436

Teleop video API stream configuration

  1. Navigate to Settings->Devices
  2. Add a new stream under the "Image" section
  3. Enter the name of the stream
  4. Make sure "enable encoding" is toggled on
  5. Save
10321032