Video Configuration for Teleoperation

Three ways to add video to Formant:

1. ROS Image stream

• Two data types: sensor_msgs/Image or sensor_msgs/CompressedImage
  Can upload frame by frame, or choose to encode it. Encoding saves bandwith as video is uploaded in compressed chunks instead of frame-by-frame. For this reason, sensor_msgs/Image must be encoded.

2. API stream

Works exactly like the ROS stream when sending images. The only additional feature is that you have the option to send h.264 video directly if you'd prefer to manage the encoding yourself. For more information, see Streaming image and video data with the Agent SDK.

📘

Encoding Quality with ROS or API

When connecting a ROS image topic, you will be given the option to choose a video quality. In the case of ROS, it will used a fixed bitrate regardless of the frame rate. That means the video quality will be higher with fewer frames.

For example, if choosing 720p, the bitrate might be fixed to 4,500kbps. At 30 FPS, that means each frame is allotted just 150kb (4,500/30). But at 5 FPS, they get 900kb.

If you are publishing at a low framerate, you should override the environment variable FORMANT_VIDEO_FRAMERATE to get the most optimal results.

3. Hardware stream

There are two types: USB (including CSI) cameras and IP cameras.

USB Cameras

When using a USB camera, Format will attach to the hardware stream with the closest dimensions to the preferred quality. For framerate, we will look for the environment variable FORMANT_VIDEO_FRAMERATE, which is defaulted to 24.

For example, you can see the native formats with the very helpful command v4l2-ctl.

Here's an example:

$ v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'YUYV'
        Name        : YUYV 4:2:2
                Size: Discrete 640x480
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.042s (24.000 fps)
                        Interval: Discrete 0.050s (20.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.100s (10.000 fps)
                        Interval: Discrete 0.133s (7.500 fps)
                        Interval: Discrete 0.200s (5.000 fps)
        Index       : 1
        Type        : Video Capture
        Pixel Format: 'MJPG' (compressed)
        Name        : Motion-JPEG
                Size: Discrete 640x480
                        Interval: Discrete 0.017s (60.000 fps)
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.042s (24.000 fps)
                        Interval: Discrete 0.050s (20.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.100s (10.000 fps)
                        Interval: Discrete 0.133s (7.500 fps)
                        Interval: Discrete 0.200s (5.000 fps)

Formant will avoid attaching to the MJPG stream. However, if you'd like to attach to this stream any way, you can set it in the environment variable FORMANT_PREFER_MJPG.

IP Cameras

The IP camera can connect with a protocol of RTSP, RTMP, or UDP (although RTSP is the primary choice, RTMP is supported mostly for DJI drones).

If you are using the ONVIF camera, it will be discoverable by Formant in the Hardware IP box. Checking the "Supports ONVIF" toggle will allow Formant to automatically connect with the right port to the RTSP stream. If you are using a fully qualified RTSP stream address (e.g., rtsp://192.168.1.110:554/h264), the "Supports ONVIF" toggle will be ignored since the connection is already specified.

❗️

Video Stream Encoding

If you are not using an h264 video stream, you will need to check the "Encoding required" toggle.

If using h264, we require it to be encoded with the "baseline" or "baseline constrained" profile. If you are not using one of these profiles, or have other non-supported encoding parameters, Formant will try it's best to re-encode in an acceptable Formant. If Formant is unable to, you must enable "Encoding required."

It's much better if your video is already encoded in an accepted manner, as this will save compute resources by Formant's agents.

📘

Environment Variables

Camera settings can be managed more directly and granularly through environment variables on your device. For a full list of environment variables and their settings, see the page here.