AI-Driven Dynamic Soft Gripper

Project: Dynamic Soft Gripper with Object Detection

Our project aims at building intelligent and self adaptable robotic soft gripper that self adjusts grip strength based on real time object detection.

It will help in proper object handling and making industrial robots more safe and efficient. This will remove the problem of using multiple grippers and manually changing grip strength to handle different objects.

Use of AI based object detection will help in determining object surface texture, material hardness, shape and size, delicacy, center of gravity. This will help the gripper to apply enough strength to pick and place item precisely and efficiently without damage.

This device and set of designed grippers will be mounted on SCARA or UARM robot for testing and debugging.

Currently the grippers that are used in industries have a fixed grip strength that is needed to be changed manually if we need to pick objects with different tensile strength or compressibility. Also this process is quite unefficient and unsafe. For example in motherboard assembly line the robot or the gripper that is handeling fixing of mouse and keyboard ports cannot be used to handle processors as this require much less and precise strength comparatively.

It is absolutely required and is useful for example this could bring a revolutionary change in assembly line and manufacturing of building PCB and hardware boards, by placing delicate components securely and accurately. It can also be used in automotive industry as one single robot with one gripper can be used for assembling car components, including heavy metal components and lightweight plastic components.

This hardware development can make the robotics industry more efficient and precise.

Understanding the Project Scope

Our project is ambitious and has the potential to significantly impact the robotics industry. Here's a breakdown of the key components:

• Hardware: Soft gripper design, actuator selection, sensor integration, and SCARA / UARM robot integration.
• Software: Object detection algorithm (using OpenCV and AI), grip force control, and overall system integration.
• AI: Machine learning model development for object property estimation (texture, hardness, shape, size, delicacy, center of gravity).

Potential Challenges and Solutions

Soft Gripper Design:

• Material Selection: Consider materials with high elasticity, durability, and tactile sensitivity.
• Actuator Choice: Pneumatic, hydraulic, or electric actuators can be used based on required force and speed.
• Sensor Integration: Force/pressure sensors, tactile sensors, and position sensors are crucial for grip control.

Object Detection:

• Algorithm Selection: OpenCV's features like blob detection, contour detection etc were tested for initial object detection.
• Deep Learning: For complex object properties estimation, consider convolutional neural networks (CNNs) or other deep learning architectures.
• Data Acquisition: Used kaggle dataset of objects with varying properties for training the AI model. (still in progress and updation)

Grip Force Control:

• Sensor Fusion: Combine data from force/pressure sensors and object property estimations to determine optimal grip force.
• Control Algorithm: Development of a control algorithm to adjust grip force in real-time based on sensor readings.

System Integration:

• Hardware-Software Interface: Ensure seamless communication between the gripper, sensors, and control system.
• SCARA / UARM Robot Integration: Coordinate gripper movements with the robot's kinematics.

Potential Resources

• OpenCV: For image processing and object detection tasks.
• TensorFlow/PyTorch: For deep learning model development.
• Robotics Toolboxes: MATLAB or Python-based toolboxes for kinematics and dynamics calculations.
• Simulation Software: Simulate gripper behavior and object interactions before physical prototyping.
• 3D Printing: For rapid prototyping of gripper components.

Next Steps

Work on silicon grippers.

Integrate sensors on the silicon gripper.

update for different object and sizes.

sim=require'sim'

function initialize(maxVel,maxAccel,maxJerk,maxTorque)
    motorHandles={-1,-1,-1,-1}
    for i=1,4,1 do
        motorHandles[i]=sim.getObject('../motor'..i)
        sim.setJointTargetForce(motorHandles[i],maxTorque)
        sim.setObjectFloatParam(motorHandles[i],sim.jointfloatparam_maxvel,maxVel)
    end
    auxMotor1=sim.getObject('../auxMotor1')
    auxMotor2=sim.getObject('../auxMotor2')

    maxVelVect={maxVel,maxVel,maxVel,maxVel}
    maxAccelVect={maxAccel,maxAccel,maxAccel,maxAccel}
    maxJerkVect={maxJerk,maxJerk,maxJerk,maxJerk}
end

function moveToPosition(newMotorPositions,synchronous,maxVel,maxAccel,maxJerk)
    local _maxVelVect={}
    local _maxAccelVect={}
    local _maxJerkVect={}
    if not maxVel then
        maxVel=maxVelVect[1]
    end
    if not maxAccel then
        maxAccel=maxAccelVect[1]
    end
    if not maxJerk then
        maxJerk=maxJerkVect[1]
    end
    for i=1,4,1 do
        _maxVelVect[i]=math.max(1*math.pi/180,math.min(maxVel,maxVelVect[i]))
        _maxAccelVect[i]=math.max(1*math.pi/180,math.min(maxAccel,maxAccelVect[i]))
        _maxJerkVect[i]=math.max(1*math.pi/180,math.min(maxJerk,maxJerkVect[i]))
    end
    local op=sim.ruckig_nosync
    if synchronous then
        op=-1
    end
    local params = {
        joints = motorHandles,
        targetPos = newMotorPositions,
        maxVel = _maxVelVect,
        maxAccel = _maxAccelVect,
        maxJerk = _maxJerkVect,
        flags = op,
    }
    sim.moveToConfig(params)
end

function enableSuctionCup(enable)
    if enable then
        sim.writeCustomStringData(gripperHandle,'activity','on')
    else
        sim.writeCustomStringData(gripperHandle,'activity','off')
    end
end

function sysCall_thread()
    modelBase=sim.getObject('..')
    gripperHandle=sim.getObject('../uarmVacuumGripper')
    pickupPart=sim.getObject('../pickupPart')
    sim.setObjectParent(pickupPart,-1,true)

    local maxVelocity=45*math.pi/180 -- rad/s
    local maxAcceleration=40*math.pi/180 -- rad/s^2
    local maxJerk=80*math.pi/180 -- rad/s^3
    local maxTorque=10 -- kg*m^2/s^2

    initialize(maxVelocity,maxAcceleration,maxJerk,maxTorque)

    while true do
        enableSuctionCup(false)
        -- Synchronous operation of the individual joints:
        moveToPosition({180*math.pi/180,59*math.pi/180,84*math.pi/180,180*math.pi/180},true)
        moveToPosition({180*math.pi/180,52.5*math.pi/180,84*math.pi/180,180*math.pi/180},true)
        enableSuctionCup(true)
        moveToPosition({180*math.pi/180,59*math.pi/180,84*math.pi/180,180*math.pi/180},true)
        moveToPosition({90*math.pi/180,104*math.pi/180,60*math.pi/180,90*math.pi/180},true)
        -- Asynchronous operation of the individual joints:
        moveToPosition({180*math.pi/180,59*math.pi/180,84*math.pi/180,180*math.pi/180},false)
        moveToPosition({180*math.pi/180,52.5*math.pi/180,84*math.pi/180,180*math.pi/180},true)
        enableSuctionCup(false)
        moveToPosition({180*math.pi/180,59*math.pi/180,84*math.pi/180,180*math.pi/180},true)
        moveToPosition({90*math.pi/180,104*math.pi/180,60*math.pi/180,90*math.pi/180},false)
    end
end

function sysCall_joint(inData)
    if inData.handle==auxMotor1 then
        local t2=-sim.getJointPosition(motorHandles[2])+104*math.pi/180
        local t3=sim.getJointPosition(motorHandles[3])-59.25*math.pi/180
        error=t3-t2-inData.pos
    end
    if inData.handle==auxMotor2 then
        local t3=sim.getJointPosition(motorHandles[3])-59.25*math.pi/180
        error=-t3-inData.pos
    end
    local ctrl=error*20
    
    local maxVelocity=ctrl
    if (maxVelocity>inData.maxVel) then
        maxVelocity=inData.maxVel
    end
    if (maxVelocity<-inData.maxVel) then
        maxVelocity=-inData.maxVel
    end
    local forceOrTorqueToApply=inData.maxForce

    local outData={}
    outData.vel=maxVelocity
    outData.force=forceOrTorqueToApply
    return outData
end
sim=require'sim'

function sysCall_init() 
    modelBase=sim.getObject('..')

    s=sim.getObject('../sensor')
    l=sim.getObject('../loopDummyA')
    l2=sim.getObject('../loopDummyB')
    b=sim.getObject('../link2_dyn')
    suctionPadLink=sim.getObject('../connection')
    local gripperBase=sim.getObject('..')

    infiniteStrength=true
    maxPullForce=3
    maxShearForce=1
    maxPeelTorque=0.1

    sim.setLinkDummy(l,-1)
    sim.setObjectParent(l,b,true)
    m=sim.getObjectMatrix(l2)
    sim.setObjectMatrix(l,m)
end

function sysCall_cleanup() 
    sim.setLinkDummy(l,-1)
    sim.setObjectParent(l,b,true)
    m=sim.getObjectMatrix(l2)
    sim.setObjectMatrix(l,m)
end 

function sysCall_sensing() 
    parent=sim.getObjectParent(l)
    local on=sim.readCustomStringData(modelBase,'activity')=='on'
    if not on then
        if (parent~=b) then
            sim.setLinkDummy(l,-1)
            sim.setObjectParent(l,b,true)
            m=sim.getObjectMatrix(l2)
            sim.setObjectMatrix(l,m)
        end
    else
        if (parent==b) then
            -- Here we want to detect a respondable shape, and then connect to it with a force sensor (via a loop closure dummy dummy link)
            -- However most respondable shapes are set to "non-detectable", so "sim.readProximitySensor" or similar will not work.
            -- But "sim.checkProximitySensor" or similar will work (they don't check the "detectable" flags), but we have to go through all shape objects!
            index=0
            while true do
                shape=sim.getObjects(index,sim.object_shape_type)
                if (shape==-1) then
                    break
                end
                local val=sim.getObjectInt32Param(shape,sim.shapeintparam_respondable)
                if (shape~=b) and (val~=0) and (sim.checkProximitySensor(s,shape)==1) then
                    -- Ok, we found a respondable shape that was detected
                    -- We connect to that shape:
                    -- Make sure the two dummies are initially coincident:
                    sim.setObjectParent(l,b,true)
                    m=sim.getObjectMatrix(l2)
                    sim.setObjectMatrix(l,m)
                    -- Do the connection:
                    sim.setObjectParent(l,shape,true)
                    sim.setLinkDummy(l,l2)
                    break
                end
                index=index+1
            end
        else
            -- Here we have an object attached
            if (infiniteStrength==false) then
                -- We might have to conditionally beak it apart!
                result,force,torque=sim.readForceSensor(suctionPadLink) -- Here we read the median value out of 5 values (check the force sensor prop. dialog)
                if (result>0) then
                    breakIt=false
                    if (force[3]>maxPullForce) then breakIt=true end
                    sf=math.sqrt(force[1]*force[1]+force[2]*force[2])
                    if (sf>maxShearForce) then breakIt=true end
                    if (torque[1]>maxPeelTorque) then breakIt=true end
                    if (torque[2]>maxPeelTorque) then breakIt=true end
                    if (breakIt) then
                        -- We break the link:
                        sim.setLinkDummy(l,-1)
                        sim.setObjectParent(l,b,true)
                        m=sim.getObjectMatrix(l2)
                        sim.setObjectMatrix(l,m)
                    end
                end
            end
        end
    end
end

Note for evaluator - We worked with software part but due to budget and time constraints we could not complete the hardware implementation fully, but we would definitely be pushing boundaries moving further on this project.