Add a teleoperation image stream

This guide will teach you how to add image and video streams to your teleoperation interface in Formant.

You can add image or video streams via ROS, a direct hardware connection, or the Formant Agent SDK.

Step 1: Navigate to device teleoperation settings

  1. In Formant, in the upper-left corner, open the menu and click Settings.
  2. Click Devices, and then click on the device you want to configure.
  3. Click on the Teleoperation tab to open the teleoperation configuration page.

Step 2: Configure an image stream

ROS image stream

Formant supports two ROS data types: sensor_msgs/Image and sensor_msgs/CompressedImage. To save bandwidth,Image topics are encoded to H.264 video. CompressedImage topics can be ingested either frame-by-frame as images, or encoded to H.264.

  1. In the teleoperation configuration page, scroll down to Image and click the plus (+) icon.
  2. Formant will populate the Image configuration window with ROS topics found on your device which have a data type of sensor_msgs/Image or sensor_msgs/CompressedImage:

Click one of these topics to add it as a video stream.

If you don't see the stream you want:

  1. Click Add ROS topic.
  2. Configure the ROS topic:
PropertyUsage
Topic nameEnter the name of the image ROS topic you want to add to your teleoperation view.
Topic typeProvide the appropriate image topic type: sensor_msgs/Image or sensor_msgs/CompressedImage.
Enable encoding (CompressedImage topics only)If set to YES, Formant will encode the incoming image stream to H.264. Image streams are encoded to H.264 by default.
Quality of the video (encoded streams only)Choose the resolution of the encoded video stream. Higher video quality will use more data.
  1. Click Done, and proceed to Step 3.

Camera stream

You can ingest an image stream directly from a camera. Formant supports USB, Camera Serial Interface (CSI), and IP cameras.

USB and CSI

  1. In the teleoperation configuration page, scroll down to Image and click the plus (+) icon.
  2. Click Add from hardware. This will open the camera configuration window:
  1. Configure your camera stream:
PropertyUsage
Hardware typeSet this to usb.
Hardware path to USB cameraEnter the hardware path to your camera. This field should auto-populate with detected USB devices.
Quality of the videoChoose the resolution of the encoded video stream. Higher video quality will use more data.

When using a USB or CSI camera, Format will attach to the camera stream with the closest dimensions to the preferred video quality specified above.

The frame rate is controlled by the environment variable FORMANT_VIDEO_FRAMERATE, which has a default value of 24.

You can see the native video formats of your camera with the command v4l2-ctl. Here's an example:

$ v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'YUYV'
        Name        : YUYV 4:2:2
                Size: Discrete 640x480
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.042s (24.000 fps)
                        Interval: Discrete 0.050s (20.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.100s (10.000 fps)
                        Interval: Discrete 0.133s (7.500 fps)
                        Interval: Discrete 0.200s (5.000 fps)
        Index       : 1
        Type        : Video Capture
        Pixel Format: 'MJPG' (compressed)
        Name        : Motion-JPEG
                Size: Discrete 640x480
                        Interval: Discrete 0.017s (60.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.042s (24.000 fps)
                        Interval: Discrete 0.050s (20.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.100s (10.000 fps)
                        Interval: Discrete 0.133s (7.500 fps)
                        Interval: Discrete 0.200s (5.000 fps)

Formant will avoid attaching to the MJPG stream unless you set environment variable FORMANT_PREFER_MJPG to true.

  1. Click Done, and proceed to Step 3.

IP cameras

Formant supports IP cameras via the Real-Time Streaming Protocol (RTSP), Real-Time Messaging Protocol (RTMP), and UDP, as well as the ONVIF protocol.

  1. In the teleoperation configuration page, scroll down to Image and click the plus (+) icon.
  2. Click Add from hardware. This will open the camera configuration window:
  1. Configure your camera stream:
PropertyUsage
Hardware typeSet this to ip.
RTSP URI of IP cameraEnter the URI of your camera in the following format:
RTSP:
rtsp://<ip>:<port>/H264
RTMP:
rtmp://<ip>:<port>/steam/live
Encoding requiredSet this to YES if your camera outputs a format other than H.264.

Formant requires incoming H.264 video to be encoded with the baseline or baseline constrained profile. If Formant detects a different profile, or a non-supported set of encoding parameters, it will try to re-encode the video. If this fails, you must enable this property.

Proper encoding of your video prior to ingestion by the Formant agent (allowing you to set this property to NO) will save computation resources.
Supports ONVIFSet this to YES if your camera supports the ONVIF protocol. Formant will be able to connect to your camera automatically.

If you specify a fully qualified address in RTSP URI of IP camera, this property will be ignored, as its functionality will already be fulfilled.
UsernameEnter the username to access your camera.
PasswordEnter the password to access your camera.
  1. Click Done, and proceed to Step 3.

API stream

You can use the Formant Agent SDK to capture images and video on your device and ingest it with the Formant agent.

You can post images and videos, and control their encoding, using the post_image() function. For more information and examples on ingesting images and video using the Agent SDK, see Streaming image and video data.

For a complete reference of the Agent SDK, see Agent SDK Reference.

Once you have written an application which captures images or video on your device:

  1. In the teleoperation configuration page, scroll down to Image and click the plus (+) icon.
  2. Click Add from API. This will open the API stream configuration window:
  1. Configure your API stream:
PropertyUsage
Image stream nameEnter the name of the stream to which you are posting image or video data (stream parameter in post_image() function).
Enable encodingIf set to YES, Formant will encode the incoming image stream to H.264.
Quality of the video (encoded streams only)Choose the resolution of the encoded video stream. Higher video quality will use more data.
  1. Click Done, and proceed to Step 3.

Step 3 (optional): Further camera configuration via environment variables

In addition to the configuration in Step 2, you can use Formant environment variables to control a wide range of camera settings. For a full list of environment variables available in Formant, see Environment variables for the agent.

Once you've configured your environment variables, proceed to Step 4.

Step 4: Add an image stream to your teleoperation view

  1. Scroll to the top of the teleoperation configuration window. Video streams which have been added under the Image section can be added to the Primary View or Sidebar Views by clicking the dropdown arrow:
  1. Click Save to confirm your teleoperation settings.

Step 5: Verify your work in teleoperation mode

  1. In your device view, in the upper-right corner, click the three dots >> Teleop, or enter Shift + T.
  2. Verify that your video stream is working properly:

πŸ“˜

If you notice an issue with this page or need help, please reach out to us! Use the 'Did this page help you?' buttons below, or get in contact with our Customer Success team via the Intercom messenger in the bottom-right corner of this page, or at [email protected].