Obtain images from a camera mounted on the robotic manipulator

[maria3791]

Is it possible to obtain images from a camera mounted on the robotic manipulator, as shown in the figure below? I would like to get these images from the manipulator's camera to process them using the OpenCV library. If possible, do you have any examples of how to do this?

[petercorke]

Hi @maria3791, thank you for your interest in RTB, and apologies for the very long delay in replying. Various life events got in the way, and since May this year I've been doing a lot of work across RTB, Swift and SpatialGeometry.

Not directly, I'm afraid. RTB's graphical backends (Swift and PyPlot) display the robot but don't render images from a viewpoint on it that you can hand to OpenCV. There are two things that get partway there:

1. Synthetic image points (a good fit for visual servoing). The Machine Vision Toolbox's CentralCamera projects 3D points and wireframes into pixel coordinates from any camera pose, so you can mount a camera on the end-effector with a tool transform and project your scene into it:

import roboticstoolbox as rtb
from spatialmath import SE3
from machinevisiontoolbox import CentralCamera

panda = rtb.models.Panda()
cam = CentralCamera.Default()               # intrinsic parameters
T_cam = SE3.Tz(0.05)                        # camera pose relative to the end-effector

Tc = panda.fkine(q) * T_cam                 # camera pose in the world frame
uv = cam.project_point(P, pose=Tc)          # P is 3xN world points, uv is 2xN pixels

This gives you image-plane coordinates (and cam.plot_point and cam.plot_wireframe draw them), but it is not a rendered image of a textured scene.

2. Camera view in Swift. env.set_camera_pose(position, look_at) moves Swift's viewing camera, so you can look through the manipulator's camera pose, and env.screenshot("name") saves the view as a PNG (downloaded by the browser). That's useful for illustration but it isn't a Python-side image array.

For true camera images (textures, lighting, depth, noise) you need a simulator with a rendering engine such as CoppeliaSim, MuJoCo or Isaac. A CoppeliaSim connector is something I'd like to have in the toolbox: see #675. Collaboration is welcome if you're interested.

If you hit a problem with any of the above, please post an Issue rather than a discussion thread. Thanks for the question.