AI Job Coach

Many people with disabilities struggle with obtaining jobs because they do not have the tailored job skills and life skills training they need. A few examples of job training skills would be cashier duties, cleaning tables, and putting silverware sets together. A few examples of life skills are crossing the street, waiting in line, and doing laundry. However, the number of skills is so numerous, specialized, and unknown that it would take teachers months or even years to develop the custom training needed for each person. Due to the large number of job openings as a result of COVID, there is a large need for people to fill these types of positions. Teachers need a way to rapidly develop this training as well as provide one on one feedback to the students in real time.

I consulted with a leading life skills teacher on how to address this problem. She pointed to research, https://files.eric.ed.gov/fulltext/ED605910.pdf, that proved that video modeling is a successful method to teach these skills. However, it involves hours of teacher preparation and one on one instruction to be effective. This is where an AI-based coach could save a teacher months or even years of time and accelerate the learning for students.

AI Job Coach analyzes images of the students' progress on tasks and makes a determination if the task is completed correctly. For example, if the task is to properly set a dinner table at a restaurant the correct table setting might look like this

However, if the placement of the dinner table looked something like the image below the AI Job Coach would determine that the placement was incorrect and provide feedback to the student:

To get started I followed the Installation Instructions — Ryzen AI Software 1.1 documentation (amd.com). However, AMD IPU Device was not showing up in Device Manager, but I discovered that I needed to enable IPU in the BIOS after I read Running LLM on AMD NPU Hardware. I was able to finish the installation, but when I tried to use any of the object detection models, I kept getting "Error loading ...\anaconda3\envs\YOLOv8Env\Lib\site-packages\torch\lib\fbgemm.dll" or one of its dependencies." I ran conda install pytorch from the command prompt and the problem went away one time, but I had other dependencies errors. Then when I ran it again, I got the same error. So, I was not able to overcome this error, so I was unable to complete the project using the Ryzen AI software and AMD UM790 Pro.