This is my second attempt at an interactive humanoid robot. I'm coming back to this after years :-)
The robot is fully interactive, it will have moving parts and it will interact with the environment and people using AI. The robot will recognize people and their emotions, age, gender, etc. It recognize the voice and you will be able to talk with him.
All parts are printed on 3d printers and cut on laser and cnc.
I started building C3PO by creating a project in Fusion 360.
Unfortunately, I don't remember where I got the robot model to print on a 3D printer. Anyway, I made a lot of changes there. But thank you if anyone recognizes your model.
Then I printed everything and cut it on a laser and CNC plotter.
On the video some parts printed and some part cutted:
When the robot was already assembled, I started writing an application to operate it. Everything was written in C#. A very large part was written using the OpenCV wrapper for the .Net platform - OpenCvSharp. Creating applications using deep learning with the OpenCV library is pure pleasure :-).
In the application I use two AI\deep learning frameworks: YOLO and Caffe.
YOLO is responsible for detection hands, and Caffe for recognizing heads , age, gender and emotions.
I created some of the deep learning models by myself when I was getting to know deep learning world (by maaany years).
At the end of the project I added voice recognition and voice syntesis.
To summarize how my robot works:
- recognizes heads and selects the person who is closest to him (I do it as simply as possible - I measure the surface area of the detected heads. In the future I will use OAK-D from Luxonis),
- then turns his head towards that person,
- recognizes the age and gender of this person,
- if he doesn't see the hand, he asks the person to move back - of course, he calls Mrs./Mr./My dear child, etc... using the previously identified age and gender,
- at this moment it is waiting for a voice command. If we give the "Hands Up" command, the robot will raise its hands up. If we give the command "Follow me", the robot will move its arms just like the person in front of it.
There are many commands.
I'm currently working on a dictionary so you can talk to him.
In "Code" section I present the most important parts of my code (I use Visual Studio Community 2019 and C#. All tasks run in threads. ).
I will now be working on a new version of C3PO and I have a very ambitious plan to create a blog describing the creation of this robot from start to finish.
When I finish the software, I will share it as well.
I'll let you know where you can find this blog ;-)
Thank you for taking the time to read this post. See you next time.
Comments