Real-Time Connections with WebRTC

Formant's Data SDK allows custom Formant Modules or your own websites to establish and utilize real-time connections to devices over the internet or locally. Real-time connections provide a direct link between the browser and the device without having to route through a back-end server. This dramatically reduces latency, which is crucial for streaming media or teleoperation.

The underlying technology used to establish connections is called WebRTC. It's a web standard for real-time rich media and data applications.

After a web application connects to a device,custom data channels can be created between the application and a Formant device. Data channels are bi-directional and have configurable reliability. Any data payload can be sent over custom data channels.

Requirements for Real-Time Connections

To enable WebRTC connections, you must have completed the following:

1. Install the Formant Agent on your device (see here for instructions)

Optionally, if you want to enable P2P WebRTC over a local network only:

1. Install the Formant P2P Adapter on your device(see below for instructions)
2. Set up WebRTC on your own server or website (see below for instructions)

Installing the Formant P2P Adapter on your Device

github.com

GitHub - FormantIO/formant-p2p-adapter: A P2P adapter for Formant

The P2P adapter runs on your device and will automatically handle the WebRTC signalling and communication with the Formant agent. The easiest way to install it will be to upload the adapter to Formant.

On your computer, in whichever install directory you choose (typically home), perform the following steps:

1. Clone the P2P Adapter repository (linked above): git clone https://github.com/FormantIO/formant-p2p-adapter.git && cd formant-p2p-adapter
2. Generate a certificate: ./gen_cert.sh
3. Optional: if running from a remote network, because it's a "cross-origin resource," browsers will not allow a p2p connection unless explicitly listed. To do so, replace line 10 of main.py with the domain name of your external web server.
4. Run ./create_adapter_zip.sh
5. In Formant, upload formant-p2p-adapter.zip to the Adapters section in the main application settings. (see Adapters for more information). Enter ./start.sh in the "Exec Command" box.

6. Configure your device to use the P2P Adapter (again, see Adapters for instructions).

Using the Data SDK to establish a Peer-to-Peer Connection

If one peer is the device itself, the other peer is your external server (either embedded in Formant as a custom view/module, or your own destination). We've provided an API in the Data SDK to easily establish the p2p connection with your device without having to go through all the signalling and handshaking involved in WebRTC.

📘

Don't forget

1. Remember, you must enable the host of your Data SDK code in the CORS exceptions in your P2P Adapter above.

2. WebRTC requires HTTPS. Don't forget to generate SSL certificates for your device.

Working with a PeerDevice in the Data SDK

The Data SDK will allow you to establish a P2P connection with your device. To do this, you'll use the Fleet object (an overview can be found here).

import { Fleet } from "@formant/data-sdk";

// If remotely over the internet
const device = await Fleet.getCurrentDevice();
await device.startRealtimeConnection();

// If on local network
const device = await Fleet.getPeerDevice("https://<local IP>:8000");
await device.startRealtimeConnection();

For a how-to example in which we establish a realtime connection and draw video on the screen, see the below recpie:

RTC Capabilities with your Device object

Once you have a Device as in the example above, some additional things you can do are the following:

• Status: getRealtimeStatus(): "connected" | "connecting" | "disconnected"
• isInRealtimeSession(): Promise<boolean>
• addRealtimeListener(listener: RealtimeListener) and removeRealtimeListener(listener: RealtimeListener)

See the full Device documentation here.