Hardware components | ||||||
 |
| × | 1 | |||
Hand tools and fabrication machines | ||||||
 |
| |||||
This case study demonstrates how to enhance the grasping accuracy of robotic arms using a visual system, successfully enabling a humanoid robot to perform dual-arm grasping rather than being limited to single-arm operations.
IntroductionToday, various types of humanoid robots are available in the market, covering a range of fields, including the service and medical industries. These robots are increasingly integrated into our daily lives and work due to their intelligent and automated features. However, despite the significant potential shown by existing humanoid robots in specific application scenarios, improving their operational precision, adaptability, and versatility remains a critical challenge in robotics development. Addressing this need, this case study explores the integration of visual systems with robotic arm technology to enhance the autonomous operational capabilities of humanoid robots in complex environments, particularly focusing on tasks that require precise grasping and manipulation.
ObjectiveBy combining the OpenCV algorithm, STag marker-based visual system, and Mercury X1 wheeled humanoid robot, this study aims to achieve precise grasping of various objects of different shapes and sizes, thereby improving sorting efficiency and accuracy, and fully leveraging the capabilities of humanoid robots through dual-hand cooperation.
Mercury X1 is an advanced humanoid robot developed by Elephant Robotics, specifically designed to handle various automation tasks. It features 19 DOF (with 7 DOF per arm), offering high flexibility and adaptability in task execution. Equipped with a wheeled mobile base driven by high-performance direct-drive motors, Mercury X1 can move stably in complex environments and has a battery life of up to 8 hours, making it suitable for both personal and commercial applications.
The robot employs a high-performance main controller system powered by NVIDIA Jetson Xavier, providing robust computational support for processing complex algorithms, including visual ranging, sensor fusion, localization and mapping, obstacle detection, and path planning. Additionally, the Mercury X1's mobile base is equipped with LiDAR, ultrasonic sensors, and a 2D vision system, enabling highly perceptive environmental interaction.
The myCobot Pro Adaptive Gripper is designed to pick up objects of any shape securely without releasing them. It is ideal for completing a wide range of applications and can be quickly deployed into production without the need for robotics expertise. This gripper is a key component of a highly flexible and reliable robotic system.
The myCobot Pro Camera Flange is a camera module that captures images using a USB-B data cable.
OpenCV is an open-source library used for image processing and computer vision, playing a critical role in this case study. The project would not have been possible without OpenCV. The robot's camera uses OpenCV to analyze collected visual data, allowing for the recognition and positioning of objects. OpenCV algorithms enable the robot to identify the shape, size, and precise coordinates of objects, which are crucial for accurate grasping and manipulation. By providing the robot with the coordinates of an object, precise grasping is achieved.
S-Tag is a highly reliable marking system designed to provide accurate marker recognition in visually challenging environments. These markers are used to identify objects and positions within the operating environment of the Mercury X1 robot. Even in low-light conditions or when the line of sight is obstructed, S-Tag ensures that the robot can accurately recognize target objects through its camera system.
Through the application of these technologies, the Mercury X1 humanoid robot can perform complex tasks such as autonomous navigation, object recognition, and precise manipulation, all of which are essential capabilities for modern automation and intelligent systems.
pymycobot is a Python library used to control the robotic arms and end-effectors (such as grippers) of the Mercury X1 robot. It allows developers to precisely control the arm's angles, coordinates, and movement modes, including interpolation and refresh modes. This library provides the robot with high flexibility and customizability, enabling it to perform complex grasping and manipulation tasks while adapting to various operational requirements.
Project ImplementationPreparationBefore starting, it's essential to ensure that the robotic arms are correctly positioned at their zero point. You can calibrate the zero position using the following steps:
1) Release the Joint Motors:
Use the release command to disengage the joint motors. (Important: After releasing, hold the joints to prevent the arms from falling and causing damage!)
ml.release_all_servos()
mr.release_all_servos()2) Manually Return the Arms to Zero Position:
Drag the robotic arms back to the zero position manually.
3) Lock the Motors:
Send the command to lock the motors in place.
ml.focus_all_servos()
mr.focus_all_servos()4) Check the Zero Position:
Use the command to get the joint angles and verify the current position.
ml.get_angles()
mr.get_angles()If the returned joint angles are approximately `[0, 0, 0, 0, 90, 0]`, the zero position is considered correct.
5) Zero Position Calibration:
If the joint angles at the zero position significantly deviate from `[0, 0, 0, 0, 90, 0]`, the joints need to be recalibrated.
for i in range(1,7):
ml.set_servo_calibration(i)
mr.set_servo_calibration(i)After calibration, read the joint information again. If the returned angles are `[0, 0, 0, 0, 90, 0]`, the calibration is successful.
ml.get_angles()
mr.get_angles()With the preparation complete, you can proceed to implement the functional parts of the project.
Camera and Gripper Installation
The installation of the camera and gripper must correspond to the hand-eye matrix for visual recognition. A hand-eye calibration matrix for the Mercury X1's camera and gripper has already been prepared. If any changes are made to the setup, re-calibration of the hand-eye matrix will be necessary.
Steps:
1. Return the Robotic Arms to the Zero Position:
Move the robotic arms back to the zero position using the following commands:
ml.over_limit_return_zero()
mr.over_limit_return_zero()2. Install the Camera:
First, install the camera, paying careful attention to its orientation.
3. Install the Gripper:
After the camera is installed, proceed to install the gripper, also ensuring that it is oriented correctly.
Following these steps ensures that the camera and gripper are correctly aligned with the pre-calibrated hand-eye matrix, enabling accurate visual recognition and manipulation. If any adjustments are made, remember to recalibrate the hand-eye matrix to maintain precision.
Vision Algorithm ModuleIn this phase, we will use the `camera_detect` package, which encapsulates various methods related to camera usage. This package simplifies the recognition of STag markers, eliminating the need for manual hand-eye calibration calculations. Below are the key interfaces and their functions for using the package:
Key Interfaces and Functions:
1. Object Initialization:
obj = camera_detect(Camera_ID, mark_size, mtx, dist, direction)● Parameters:
● `camera_ID`: The ID of the camera.
● `mark_size`: The edge length of the STag marker in millimeters.
● `direction`: `0` for the left arm, `1` for the right arm.
● Function: Initializes the camera and robotic arm.
2. Open Camera Loop:
obj.camera_open_loop()● Function: Displays the camera feed.
3. STag Marker Identification Loop:
obj.stag_identify_loop()● Function: Displays the camera feed and identifies the STag marker, printing its camera coordinates.
4. Get STag Marker Coordinates in Robotic Arm Frame:
coords = obj.stag_robot_identify(direction)● Parameters:
● `direction`: `0` for the left arm, `1` for the right arm.
● Returns: The coordinates of the STag marker relative to the robotic arm.
● Function: Calculates the STag marker's coordinates in the robotic arm's reference frame.
5. Vision-Based Grasping:
obj.vision_trace(catch_mode, direction)● Parameters:
● `catch_mode`: `0` for horizontal grasping, `1` for vertical grasping.
● `direction`: `0` for the left arm, `1` for the right arm.
● Function: Displays the camera feed, identifies the STag marker, and prints its coordinates in the camera's reference frame.
Example Code Using `camera_detect` Package:
The following code demonstrates how to use the `camera_detect` package to initialize the camera, recognize STag markers, and perform vision-based grasping with the robotic arms:
import numpy as np
from pymycobot import Mercury
from camera_detect import camera_detect # Import the STag recognition package
if __name__ == "__main__":
# Load camera configuration file
camera_params = np.load("camera_params.npz")
mtx, dist = camera_params["mtx"], camera_params["dist"]
# Set up ports for left and right arms
ml = Mercury("/dev/left_arm")
mr = Mercury("/dev/right_arm")
arm = 0 # 0 for left arm, 1 for right arm
catch_mode = 0 # 0 for horizontal grasping, 1 for vertical grasping
# Camera IDs need to be adjusted according to the actual setup
left_camera_id = 3
right_camera_id = 6
stag_size = 32
# Create vision recognition objects for both arms
ml_obj = camera_detect(left_camera_id, stag_size, mtx, dist, arm)
mr_obj = camera_detect(right_camera_id, stag_size, mtx, dist, arm)
# Move both arms to a horizontal observation position for grasping
mr_obj.vision_trace(catch_mode, mr)
ml_obj.vision_trace(catch_mode, ml)This code provides a streamlined approach to using the `camera_detect` package for vision-based control of robotic arms. It initializes the cameras and robotic arms, and then performs STag marker recognition and grasping operations.
Steps for Hand-Eye Calibration:
1. Data Collection:
● Collect multiple sets of hand-eye data, which include the pose (position and orientation) of the robot's end effector at different locations and the pose of the feature points as seen by the camera.
2. Establishing Motion Equations:
● Formulate motion equations to determine the transformation relationship between the camera coordinate system and the robot end effector coordinate system.
3. Solving the Transformation Matrix:
● The goal is to solve for the transformation matrix that describes the spatial transformation relationship between the camera coordinate system and the robot end effector (hand) coordinate system.
The following Python code provides an example of solving the transformation matrix by converting between rotation matrices and Euler angles, and then constructing a homogeneous transformation matrix.
Code Example for Solving Transformation Matrix
import numpy as np
# Convert rotation matrix to Euler angles
def CvtRotationMatrixToEulerAngle(pdtRotationMatrix):
pdtEulerAngle = np.zeros(3)
pdtEulerAngle[2] = np.arctan2(pdtRotationMatrix[1, 0], pdtRotationMatrix[0, 0])
fCosRoll = np.cos(pdtEulerAngle[2])
fSinRoll = np.sin(pdtEulerAngle[2])
pdtEulerAngle[1] = np.arctan2(-pdtRotationMatrix[2, 0],
(fCosRoll pdtRotationMatrix[0, 0]) + (fSinRoll pdtRotationMatrix[1, 0]))
pdtEulerAngle[0] = np.arctan2((fSinRoll pdtRotationMatrix[0, 2]) - (fCosRoll pdtRotationMatrix[1, 2]),
(-fSinRoll pdtRotationMatrix[0, 1]) + (fCosRoll pdtRotationMatrix[1, 1]))
return pdtEulerAngle
# Convert Euler angles to rotation matrix
def CvtEulerAngleToRotationMatrix(ptrEulerAngle):
ptrSinAngle = np.sin(ptrEulerAngle)
ptrCosAngle = np.cos(ptrEulerAngle)
ptrRotationMatrix = np.zeros((3, 3))
ptrRotationMatrix[0, 0] = ptrCosAngle[2] * ptrCosAngle[1]
ptrRotationMatrix[0, 1] = ptrCosAngle[2] ptrSinAngle[1] ptrSinAngle[0] - ptrSinAngle[2] * ptrCosAngle[0]
ptrRotationMatrix[0, 2] = ptrCosAngle[2] ptrSinAngle[1] ptrCosAngle[0] + ptrSinAngle[2] * ptrSinAngle[0]
ptrRotationMatrix[1, 0] = ptrSinAngle[2] * ptrCosAngle[1]
ptrRotationMatrix[1, 1] = ptrSinAngle[2] ptrSinAngle[1] ptrSinAngle[0] + ptrCosAngle[2] * ptrCosAngle[0]
ptrRotationMatrix[1, 2] = ptrSinAngle[2] ptrSinAngle[1] ptrCosAngle[0] - ptrCosAngle[2] * ptrSinAngle[0]
ptrRotationMatrix[2, 0] = -ptrSinAngle[1]
ptrRotationMatrix[2, 1] = ptrCosAngle[1] * ptrSinAngle[0]
ptrRotationMatrix[2, 2] = ptrCosAngle[1] * ptrCosAngle[0]
return ptrRotationMatrix
# Convert coordinates to a homogeneous transformation matrix (x, y, z, rx, ry, rz) where rx, ry, rz are in radians
def Transformation_matrix(coord):
position_robot = coord[:3]
pose_robot = coord[3:]
# Convert Euler angles to rotation matrix
RBT = CvtEulerAngleToRotationMatrix(pose_robot)
# Position vector as a column matrix
PBT = np.array([[position_robot[0]],
[position_robot[1]],
[position_robot[2]]])
# Concatenate rotation matrix and position vector to form the transformation matrix
temp = np.concatenate((RBT, PBT), axis=1)
# Append [0, 0, 0, 1] as the last row to complete the homogeneous transformation matrix
array_1x4 = np.array([[0, 0, 0, 1]])
matrix = np.concatenate((temp, array_1x4), axis=0)
return matrixTo control the dual robotic arms, we will use the `pymycobot` library, specifically the `Mercury` class. Each arm will be controlled through an instance of the `Mercury` class for clarity and ease of use.
Initial Setup
from pymycobot import Mercury
# Create objects for controlling the left and right arms
ml = Mercury('/dev/left_arm')
mr = Mercury('/dev/right_arm')Function to Move the Arm to Specific Coordinates
# Move the robotic arm's end-effector to the specified coordinates
def send_base_coords(coords, speed):
"""
Function to move the robotic arm's end-effector to the specified coordinates.
Parameters:
coords : list
List of coordinates [x, y, z, rx, ry, rz] (length 6).
speed : int
Speed of movement (range: 1 ~ 100).
"""
ml.send_base_coords(coords, speed)Example Code for Grasping a Target Object
The following code demonstrates how to control the robotic arm to recognize a target object and perform a grasping operation.
if __name__ == "__main__":
# Set camera ID
camera = UVCCamera(5, mtx, dist)
# Open the camera
camera.capture()
# Set the observation point for the left arm
origin_anglesL = [-44.24, 15.56, 0.0, -102.59, 65.28, 52.06, 23.49]
# Set the gripper movement mode (0 for default mode)
ml.set_gripper_mode(0)
# Set the tool reference coordinate system
Tool_LEN = 100 # Example length, adjust as needed
ml.set_tool_reference([0, 0, Tool_LEN, 0, 0, 0])
# Set the end coordinate system to tool
ml.set_end_type(1)
# Set the movement speed
sp = 40
# Move to the observation point
ml.send_angles(origin_anglesL, sp)
# Wait for the robotic arm to finish moving
waitl()
# Refresh the camera interface
camera.update_frame()
# Capture the current frame
frame = camera.color_frame()
# Detect the angles and IDs of QR codes in the frame
(corners, ids, rejected_corners) = stag.detectMarkers(frame, 11)
# Calculate the object's position in the camera coordinate system
marker_pos_pack = calc_markers_base_position(corners, ids, MARKER_SIZE, mtx, dist)
# Get the current coordinates of the robotic arm
cur_coords = np.array(ml.get_base_coords())
# Convert the angle values to radians
cur_bcl = cur_coords.copy()
cur_bcl[-3:] *= (np.pi / 180)
# Convert the object's coordinates from the camera coordinate system to the base coordinate system
fact_bcl = Eyes_in_hand_left(cur_bcl, marker_pos_pack)
# Prepare the target coordinates for the robotic arm
target_coords = cur_coords.copy()
target_coords[0] = fact_bcl[0]
target_coords[1] = fact_bcl[1]
target_coords[2] = fact_bcl[2] + 50 # Adjust height above the object
# Move the robotic arm to the position above the QR code
ml.send_base_coords(target_coords, 30)
# Wait for the robotic arm to finish moving
waitl()
# Open the gripper
ml.set_gripper_value(100, 100)
# Move the robotic arm downward along the Z-axis to grasp the object
ml.send_base_coord(3, fact_bcl[2], 10)
# Wait for the robotic arm to finish moving
waitl()
# Close the gripper
ml.set_gripper_value(20, 100)
# Wait for the gripper to close
time.sleep(2)
# Lift the gripper with the object
ml.send_base_coord(3, fact_bcl[2] + 50, 10)Left arm
Right arm
Summary
If you had a Mercury X1, what would you use it for? Leveraging its key features as a humanoid robot, how would you push its capabilities? If you have any creative ideas or projects you'd like to see realized, we welcome your thoughts and suggestions in the comments below. Your likes and comments are the greatest support for us!
Comments