You can configure your teleoperation view to send the location of a mouse click to a ROS topic or API endpoint. You can use mouse click streams to create image-based UIs which can, for example, determine which pixel was clicked on the screen, or send locations on a map.

The position of the mouse click on the screen will be sent as a float between 0 and 1 in the (x,y) dimensions, indexed from the top-left corner of the image. Mouse click coordinates will be sent if the click occurs within the image frame. Clicks outside of the image frame will be ignored.

This guide will teach you how to add a mouse click stream to your teleoperation configuration.

Prerequisites

Add a teleoperation image stream

Mouse clicks work on top of teleoperation image streams. You must first create a teleoperation image stream before you can configure a mouse click stream.

To configure a teleoperation image stream, see Add a teleoperation image stream.

Set your teleoperation view to Fit Image

Mouse clicks can be transmitted accurately only when the teleoperation view is set to Fit Image.

To set your teleoperation view to Fit Image:

1. In Formant, in the upper-left corner, open the menu and click on the Operate tab.
2. Click on the device you want to teleoperate.
3. In the bottom-right corner, click the three dots. If you see Fit Image as an option, click it. If you see Expand Image, you are already in Fit Image mode and can proceed to Step 1.

Step 1: Navigate to device teleoperation settings

1. In Formant, in the upper-left corner, open the menu and click Settings.
2. Click Devices, and then click on the device you want to configure.
3. Click on the Teleoperation tab to open the teleoperation configuration page.

Step 2: Select a destination stream

First, we'll choose the channel which should receive the mouse click data. This depends on whether we're intending to send the data to a ROS topic, or to an application written in the Formant Agent SDK.

ROS

When configured for a ROS stream, the click will be sent as a geometry_msgs/PointStamped message.

1. In the teleoperation configuration page, scroll down to Mouse Click and click the plus (+) icon.
2. Click Add ROS topic.
3. Enter the name of the ROS topic to which you want to send mouse click position data.
4. Click Done, and proceed to Step 3.

API

You can also handle mouse clicks through the Agent SDK.

1. In the teleoperation configuration page, scroll down to Mouse Click and click the plus (+) icon.
2. Click Add from API.
3. Enter the name of the function which handles mouse clicks. For an example of how to handle a mouse click stream in the Agent SDK, see GitHub: handle_mouse_clicks.py.
4. Click Done, and proceed to Step 3.

📘

For a complete reference for the Agent SDK, see Agent SDK Reference.

Step 3: Select an image stream

Now that we've designated where our mouse click data is going, we will choose the image stream on which we are clicking.

1. Click the Mouse Click stream you just created in Step 2.
2. Select the image stream you want to use as the basis for your mouse click data. Click on the Video stream text box to drop down all available image streams.
3. Click Done.
4. Click Save to confirm your teleoperation settings.

👋

If you notice an issue with this page or need help, please reach out to us! Use the 'Did this page help you?' buttons below, or get in contact with our Customer Success team via the Intercom messenger in the bottom-right corner of this page, or at [email protected].