Real-Time Connections with WebRTC

Formant's Data SDK allows custom Formant Modules or your own websites to establish and utilize real-time connections to devices over the internet or locally. Real-time connections provide a direct link between the browser and the device without having to route through a back-end server. This dramatically reduces latency, which is crucial for streaming media or teleoperation.

The underlying technology used to establish connections is called WebRTC. It's a web standard for real-time rich media and data applications.

After a web application connects to a device, custom data channels can be created between the application and a Formant device. Data channels are bi-directional and have configurable reliability. Any data payload can be sent over custom data channels.

How to establish remote Real-Time Connections

The Data SDK provides a very easy way to establish a real-time connection. All you need to do is call startRealtimeConnection() on the device object. Review this snippet for an example:

import { Authentication, Fleet } from "@formant/data-sdk";
import "@formant/ui-sdk-joystick";
import "@formant/ui-sdk-realtime-player";
import "./style.css";

// When the user clicks "connect"
el("button").addEventListener("click", async () => {
    try {
        // hide intro
        el("section").style.display = "none";
        el("#log").style.display = "block";

        // start connecting
        log("Waiting for authentication ...");
        if (!(await Authentication.waitTilAuthenticated())) {
            log("Not Authenticated");
            return;
        }

        // get current device from url context
        const device = await Fleet.getCurrentDevice();

        // connect to realtime
        log("Currently looking at <b>" + device.name + "</b>");
        log("Getting a realtime connection ... ");
        await device.startRealtimeConnection();

        // Create the custom data channel
        const channel = await device.createCustomDataChannel("example_channel");

        // Send data from the custom web application to the robot every 200ms
        setInterval(() => {
            channel.send(
                JSON.stringify({
                    value: `JSON message from the web application sent at ${Date.now()}`,
                })
            );
        }, 200);

        // Listen to data from the robot and log it to the screen
        channel.addListener((message) => {
            log(`Received JSON message from the robot: ${message}`);
        });
    } catch (e) {
        log((e as Error).message);
    }
});

function log(msg: string) {
    el("#log").innerHTML = msg + "<br>" + el("#log").innerHTML;
}

function el(selector: string) {
    return document.querySelector(selector) as HTMLElement;
}

document.body.style.display = "block";

Requirements for Local Real-Time Connections

To enable local WebRTC connections (when you know the IP of your device and you're on the same local network), you must have completed the following:

1. Install the Formant Agent on your device (see here for instructions)
2. Install the Formant P2P Adapter on your device(see below for instructions)
3. Set up WebRTC on your own server or website (see below for instructions)

Installing the Formant P2P Adapter on your Device

github.com

GitHub - FormantIO/formant-p2p-adapter: A P2P adapter for Formant

The P2P adapter runs on your device and will automatically handle the WebRTC signalling and communication with the Formant agent. The easiest way to install it will be to upload the adapter to Formant.

On your computer, in whichever install directory you choose (typically home), perform the following steps:

1. Clone the P2P Adapter repository (linked above): git clone https://github.com/FormantIO/formant-p2p-adapter.git && cd formant-p2p-adapter
2. Generate a certificate: ./gen_cert.sh
3. Optional: if running from a remote network, because it's a "cross-origin resource," browsers will not allow a p2p connection unless explicitly listed. To do so, replace line 10 of main.py with the domain name of your external web server.
4. Run ./create_adapter_zip.sh
5. In Formant, upload formant-p2p-adapter.zip to the Adapters section in the main application settings. (see Adapters for more information). Enter ./start.sh in the "Exec Command" box.

6. Configure your device to use the P2P Adapter (again, see Adapters for instructions).

Using the Data SDK to establish a Local Network Peer-to-Peer Connection

If one peer is the device itself, the other peer is your external server (either embedded in Formant as a custom view/module, or your own destination). We've provided an API in the Data SDK to easily establish the p2p connection with your device without having to go through all the signalling and handshaking involved in WebRTC.

📘

Don't forget

1. Remember, you must enable the host of your Data SDK code in the CORS exceptions in your P2P Adapter above.

2. WebRTC requires HTTPS. Don't forget to generate SSL certificates for your device.

Working with a PeerDevice in the Data SDK

The Data SDK will allow you to establish a P2P connection with your device. To do this, you'll use the Fleet object (an overview can be found here).

import { Fleet } from "@formant/data-sdk";

const device = await Fleet.getPeerDevice("https://localhost:8000");

For a how-to example in which we establish a realtime connection and draw video on the screen, see the below recpie:

RTC Capabilities with your Device object

Once you have a Device as in the example above, some additional things you can do are the following:

• Status: getRealtimeStatus(): "connected" | "connecting" | "disconnected"
• isInRealtimeSession(): Promise<boolean>
• addRealtimeListener(listener: RealtimeListener) and removeRealtimeListener(listener: RealtimeListener)

See the full Device documentation here.