Robotic Haptics by 3D Magnetic Sensor

Introduction

Creating a robotic hand that mimics human touch is a fascinating journey that combines the art of engineering with the science of sensory technology. For our latest project, we decided to build a robotic haptic hand using Infineon’s 3D hall-effect sensors and the XMC1400 Kit 2Go, all programmed with Arduino IDE and the results were visualized with Python.

Our design focuses on three robotic fingers, each equipped with sensors embedded in a silicone layer, which contains magnets at the fingertips. This setup is designed to imitate the feel and function of human skin. When pressure is applied to the fingertips, the magnets move within the silicone, and the sensors detect this movement. The data collected by these sensors allows us to measure and visualize the pressure on a serial plotter, enabling the robotic hand to adjust its grip as needed.

We used Infineon’s 3D Magnetic Sensor Library to expand our capabilities. This library helps us track the X, Y, and Z coordinates of the magnetic field from three sensors. To make the data easier to understand, we added code to calculate the magnitude vector of each sensor. This helps us monitor the pressure and movements of each finger in a more intuitive way.

By combining these technologies, we've created a system that brings us one step closer to developing a robotic hand that can interact with its environment in a more human-like manner.

Hardware Setup

Schematics

Before and after Silicone Coverage

Step by Step Finger Preparation

Software & Code

To get started with this project, download the Arduino IDE to program the XMC1400 2Go kit and interface with the sensors. This environment allows for seamless writing, compiling, and uploading of code to the microcontroller. Additionally, download Thonny, or any other user-friendly Python IDE, to write and execute the visualization code.

Code Arduino

Includes and Namespace

#include "TLx493D_inc.hpp"
using namespace ifx::tlx493d;

This segment includes the header file for the TLx493D sensor library and imports the ifx::tlx493d namespace for easier access to its classes and functions.

Object Initialization

TLx493D_W2B6 dut1(Wire, TLx493D_IIC_ADDR_A0_e);
TLx493D_W2B6 dut2(Wire, TLx493D_IIC_ADDR_A0_e);
TLx493D_W2B6 dut3(Wire, TLx493D_IIC_ADDR_A0_e);

Here, three instances of the TLx493D_W2B6 sensor class are created, each assigned to the same I2C address TLx493D_IIC_ADDR_A0_e initially.

Calibration Constants

Since our magnets are not placed at the centre of our X, Y, Z-plane, we need to calibrate the position of the sensors to bring them as close as possible to zero.

const int CALIBRATION_SAMPLES = 10;
double xOffset1 = 0, yOffset1 = 0, zOffset1 = 0;
double xOffset2 = 0, yOffset2 = 0, zOffset2 = 0;
double xOffset3 = 0, yOffset3 = 0, zOffset3 = 0;

These variables and constants are then for storing calibration offsets and the number of calibration samples.

Setup Function

void setup() {
    Serial.begin(115200);
    delay(3000);

    Serial.println("Starting setup...");

    dut1.setPowerPin(5, OUTPUT, INPUT, HIGH, LOW, 0, 250000);
    dut2.setPowerPin(4, OUTPUT, INPUT, HIGH, LOW, 0, 250000);
    dut3.setPowerPin(8, OUTPUT, INPUT, HIGH, LOW, 0, 250000);

    Serial.println("Power Pins set");

    if (dut1.begin(true, false, false, true)) {
        dut1.setIICAddress(TLx493D_IIC_ADDR_A2_e);
        dut1.printRegisters();
        Serial.println("DUT1 initialized");
    } else {
        Serial.println("DUT1 initialization failed");
    }

    if (dut2.begin()) {
        dut2.setIICAddress(TLx493D_IIC_ADDR_A1_e);
        dut2.printRegisters();
        Serial.println("DUT2 initialized");
    } else {
        Serial.println("DUT2 initialization failed");
    }

    if (dut3.begin()) {
        dut3.printRegisters();
        Serial.println("DUT3 initialized");
    } else {
        Serial.println("DUT3 initialization failed");
    }

    Serial.println("Setup done.");

    calibrateSensors();

    Serial.println("Calibration completed.");
}

• Serial Communication Initialization: It starts the serial communication at a baud rate of 115200 and introduces a delay.
• Power Pin Configuration: Each sensor is assigned a power pin with specific configurations.
• Sensor Initialization: Each sensor is initialized using the begin() method. If initialization is successful, the I2C address is set, and the register values are printed. If initialization fails, a corresponding message is printed.
• Sensor Calibration: The calibrateSensors() function is called to calibrate the sensors.

Calibration Function

void calibrateSensors() {
    double sumX1 = 0, sumY1 = 0, sumZ1 = 0;
    double sumX2 = 0, sumY2 = 0, sumZ2 = 0;
    double sumX3 = 0, sumY3 = 0, sumZ3 = 0;

    for (int i = 0; i < CALIBRATION_SAMPLES; ++i) {
        double temp;
        double valX, valY, valZ;

        dut1.getMagneticFieldAndTemperature(&valX, &valY, &valZ, &temp);
        sumX1 += valX;
        sumY1 += valY;
        sumZ1 += valZ;

        dut2.getMagneticFieldAndTemperature(&valX, &valY, &valZ, &temp);
        sumX2 += valX;
        sumY2 += valY;
        sumZ2 += valZ;

        dut3.getMagneticFieldAndTemperature(&valX, &valY, &valZ, &temp);
        sumX3 += valX;
        sumY3 += valY;
        sumZ3 += valZ;

        delay(10);
    }

    xOffset1 = sumX1 / CALIBRATION_SAMPLES;
    yOffset1 = sumY1 / CALIBRATION_SAMPLES;
    zOffset1 = sumZ1 / CALIBRATION_SAMPLES;

    xOffset2 = sumX2 / CALIBRATION_SAMPLES;
    yOffset2 = sumY2 / CALIBRATION_SAMPLES;
    zOffset2 = sumZ2 / CALIBRATION_SAMPLES;

    xOffset3 = sumX3 / CALIBRATION_SAMPLES;
    yOffset3 = sumY3 / CALIBRATION_SAMPLES;
    zOffset3 = sumZ3 / CALIBRATION_SAMPLES;
}

This function collects several samples of the magnetic field readings from each sensor, sums them up, and calculates the average to determine the zero-offset.

Loop Function

void loop() {
    double temp1 = 0.0, temp2 = 0.0, temp3 = 0.0;
    double valX1 = 0, valY1 = 0, valZ1 = 0, valX2 = 0, valY2 = 0, valZ2 = 0, valX3 = 0, valY3 = 0, valZ3 = 0;

    dut1.getMagneticFieldAndTemperature(&valX1, &valY1, &valZ1, &temp1);
    dut2.getMagneticFieldAndTemperature(&valX2, &valY2, &valZ2, &temp2);
    dut3.getMagneticFieldAndTemperature(&valX3, &valY3, &valZ3, &temp3);

    valX1 -= xOffset1;
    valY1 -= yOffset1;
    valZ1 -= zOffset1;

    valX2 -= xOffset2;
    valY2 -= yOffset2;
    valZ2 -= zOffset2;

    valX3 -= xOffset3;
    valY3 -= yOffset3;
    valZ3 -= zOffset3;

    double mag1 = sqrt(pow(valX1, 2) + pow(valY1, 2) + pow(valZ1, 2));
    double mag2 = sqrt(pow(valX2, 2) + pow(valY2, 2) + pow(valZ2, 2));
    double mag3 = sqrt(pow(valX3, 2) + pow(valY3, 2) + pow(valZ3, 2));

    Serial.print(mag1); Serial.print("\t");
    Serial.print(mag2); Serial.print("\t");
    Serial.println(mag3);

    delay(300);
}

• Sensor Data Collection: The magnetic field and temperature values are read from each sensor. In our case we don't need the temperatures.
• Offset Subtraction: The calibration offsets are subtracted to get the calibrated values.
• Magnitude Calculation: The magnitude of the magnetic field vector is calculated for each sensor.
• Serial Printing: The magnitudes are printed out for visualization in the Serial Plotter.

Serial Plotter:

When you press the robotic hand's fingers, which contain sensors and a magnet in a silicone structure, the sensors detect changes in the magnetic field caused by the deformation of the silicone. These changes are shown as spikes in the graph on the Serial Plotter. The height of each spike corresponds to the strength of the haptic feedback or pressure you apply; higher spikes mean stronger pressure, and lower spikes mean lighter pressure. This visualization helps you see how hard the robotic hand is pressing.

Visualizing with Python

In this part we are going to walk through the Python code that helped me visualize the real time data collected from Arduino IDE. Here we used Thonny, but you can use any development environment you are familiar with! Let's go!

Importing Libraries

import serial
import time
import matplotlib.pyplot as plt
import numpy as np
from matplotlib.colors import LinearSegmentedColormap

• serial: Used to handle serial communication.
• time: Provides time-related functions.
• matplotlib.pyplot: A plotting library for creating static, animated, and interactive visualizations.
• numpy: A library for numerical computations.
• matplotlib.colors.LinearSegmentedColormap: Used to create a custom colormap.

Setting Up Serial Connection

ser = serial.Serial('COM5', 115200)  # Replace 'COM5' with your actual serial port
time.sleep(2)

• serial.Serial('COM5', 115200): Initializes the serial connection on COM5 port with a baud rate of 115200. Change 'COM5' to the actual serial port in use.
• time.sleep(2): Pauses for 2 seconds to allow the serial connection to stabilize.

Sensor and Image Setup

sensor_positions = [(1, 6), (3.7, 8.8), (5.8, 9.2)]  # Adjust these based on actual positions
hand_image = plt.imread('./hand_image1.png')  # Replace with the path to your hand image

• sensor_positions: List of x, y coordinates denoting sensor positions on a hand image.
• hand_image: Loads an image of a hand from the specified file path.

Plot Setup

fig, ax = plt.subplots()
ax.imshow(hand_image, extent=(0, 10, 0, 10), alpha=0.7)  # Adjust the extent to match your heatmap size

• fig, ax = plt.subplots(): Creates a figure and axis for plotting.
• ax.imshow(hand_image, extent=(0, 10, 0, 10), alpha=0.7): Displays the hand image as a background with transparency set to 0.7.

Custom Colormap

colors = [(1, 0, 0, 0.1), (1, 0, 0, 0.5), (1, 0, 0, 1)]  # Light/transparent red to mid-red to very bright red
n_bins = 100  # Discretize into 100 steps
cmap_name = 'light_to_bright_red'
cm = LinearSegmentedColormap.from_list(cmap_name, colors, N=n_bins)

• colors: Defines a gradient of colors from light red to bright red.
• n_bins: Number of discrete color steps.
• cmap_name: Name of the custom colormap.
• cm: Creates a LinearSegmentedColormap object for color mapping.

Heatmap Update Function

def update_heatmap(sensor_values):
    ax.clear()
    ax.imshow(hand_image, extent=(0, 10, 0, 10), alpha=0.5)
    data_min = 0
    data_max = 14
    sensor_values = np.array(sensor_values)
    normalized_values = (sensor_values - data_min) / (data_max - data_min)
    for (x, y), value in zip(sensor_positions, normalized_values):
        ax.scatter(x, y, color=cm(value), s=300, edgecolor='k')
plt.draw()
plt.pause(0.01)

• Clears the axis.
• Displays the hand image as a background with less transparency.
• Normalizes sensor values to a [0, 1] range for colormap application.
• Plots the normalized sensor values at specified positions as circles on the hand image.
• plt.draw(): Redraws the current figure.
• plt.pause(0.01): Brief pause to update the plot.

Real-Time Data Reading and Plotting

try:
    plt.ion()  # Turn on interactive mode
    while True:
        data = ser.readline().decode('utf-8').strip()  # Read the serial data
        if data:
            print(f"Received: {data}")
            sensor_values = list(map(float, data.split("\t")))  # Parse values
            update_heatmap(sensor_values)
except KeyboardInterrupt:
    print("Exiting...")
finally:
    ser.close()  # Close the serial connection

• plt.ion(): Enables interactive mode for live updates.
• while True:: Infinite loop to continuously read serial data.
• ser.readline().decode('utf-8').strip(): Reads a line of serial data, decodes it from UTF-8, and strips any surrounding whitespace.
• print(f"Received: {data}"): Prints received data.
• list(map(float, data.split("\t"))): Splits the data by tab characters and converts each value to a float.
• update_heatmap(sensor_values): Updates the heatmap with new sensor values.
• except KeyboardInterrupt:: Handles user interruption to exit the loop cleanly.
• finally: ser.close(): Closes the serial connection when exiting the loop.

Outcomeof the Visualization:

Real Life Demo

Farewell

As we conclude our journey in creating this robotic haptic hand, we are thrilled by the progress we've made. This project has demonstrated how advanced sensor technology and innovative programming can come together to create a more intuitive, human-like robotic interaction.