Video Configuration for Teleoperation
Three ways to add video to Formant:
1. ROS Image stream
- Two data types:
Can upload frame by frame, or choose to encode it. Encoding saves bandwith as video is uploaded in compressed chunks instead of frame-by-frame. For this reason,
sensor_msgs/Imagemust be encoded.
2. API stream
Works exactly like the ROS stream when sending images. The only additional feature is that you have the option to send h.264 video directly if you'd prefer to manage the encoding yourself. For more information, see Streaming image and video data with the Agent SDK.
Encoding Quality with ROS or API
When connecting a ROS image topic, you will be given the option to choose a video quality. In the case of ROS, it will used a fixed bitrate regardless of the frame rate. That means the video quality will be higher with fewer frames.
For example, if choosing 720p, the bitrate might be fixed to 4,500kbps. At 30 FPS, that means each frame is allotted just 150kb (4,500/30). But at 5 FPS, they get 900kb.
If you are publishing at a low framerate, you should override the environment variable
FORMANT_VIDEO_FRAMERATEto get the most optimal results.
3. Hardware stream
There are two types: USB (including CSI) cameras and IP cameras.
When using a USB camera, Format will attach to the hardware stream with the closest dimensions to the preferred quality. For framerate, we will look for the environment variable
FORMANT_VIDEO_FRAMERATE, which is defaulted to 24.
For example, you can see the native formats with the very helpful command
Here's an example:
$ v4l2-ctl --list-formats-ext ioctl: VIDIOC_ENUM_FMT Index : 0 Type : Video Capture Pixel Format: 'YUYV' Name : YUYV 4:2:2 Size: Discrete 640x480 Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps) Index : 1 Type : Video Capture Pixel Format: 'MJPG' (compressed) Name : Motion-JPEG Size: Discrete 640x480 Interval: Discrete 0.017s (60.000 fps) Interval: Discrete 0.033s (30.000 fps) Interval: Discrete 0.042s (24.000 fps) Interval: Discrete 0.050s (20.000 fps) Interval: Discrete 0.067s (15.000 fps) Interval: Discrete 0.100s (10.000 fps) Interval: Discrete 0.133s (7.500 fps) Interval: Discrete 0.200s (5.000 fps)
Formant will avoid attaching to the MJPG stream. However, if you'd like to attach to this stream any way, you can set it in the environment variable
The IP camera can connect with a protocol of RTSP, RTMP, or UDP (although RTSP is the primary choice, RTMP is supported mostly for DJI drones).
If you are using the ONVIF camera, it will be discoverable by Formant in the Hardware IP box. Checking the "Supports ONVIF" toggle will allow Formant to automatically connect with the right port to the RTSP stream. If you are using a fully qualified RTSP stream address (e.g.,
rtsp://192.168.1.110:554/h264), the "Supports ONVIF" toggle will be ignored since the connection is already specified.
Video Stream Encoding
If you are not using an h264 video stream, you will need to check the "Encoding required" toggle.
If using h264, we require it to be encoded with the "baseline" or "baseline constrained" profile. If you are not using one of these profiles, or have other non-supported encoding parameters, Formant will try it's best to re-encode in an acceptable Formant. If Formant is unable to, you must enable "Encoding required."
It's much better if your video is already encoded in an accepted manner, as this will save compute resources by Formant's agents.
Camera settings can be managed more directly and granularly through environment variables on your device. For a full list of environment variables and their settings, see the page here.
Updated 4 months ago