AI personal assistance solution - AI HarmonyCare

Project About

AI HarmonyCare is software that utilizes advanced AI models for facial expression recognition and email/calendar retrieval & summarization. It offers two main features:

1. Recommend break based on the real-time user emotion via facial expression recognition

2. Interact with an LLM agent to get a summary of interested latest emails and today's calendar events updates

With the AMD Ryzen AI PC, the powerful CPU and NPU enable seamless parallel processing of frequent facial recognition tasks alongside the Llama 3 Large Language Model (LLM). This setup, combined with Retrieval-Augmented Generation (RAG) capabilities using a vector database like ChromaDB, ensures efficient and intelligent system performance

Setup

Setting up Email client (Thunderbird) and Thunderlink

1. Setup thunderbird, tested with version 115.*.

2. Set up account, ensure that pop3 configuration is selected.

3. Get cb_thunderlink, download the cb_thunderlink_windows.zip, unzip and follow instructions at cb_thunderlink installation under Windows (bouchier.be), briefly,

- Extract the zip file

- Open CMD in administrator mode

- cd <Path to extracted folder>

- cb_thunderlink.exe register

- Install the cb_thunderlink in thunderbird, by going to thunderbird setting -> extensions -> install add on from file and select `cb_thunderlink.xpi` from the extracted zip folder.

- Test that everything is installed successfully by copying the thunderlink/cb_thunderlink and opening it in browser.

Setting up Ryzen AI Driver and Ryzen SW

1. Download ryzen-ai-sw., install following the steps in the link for [AMD Ryzen AI NPU driver installation steps]

2. Download RyzenAI-SW examples by following the clone steps in [AMD Ryzen AI software Platform Github]

3. Note the downloaded, unzipped path of the 2 folders. These will be needed in the next steps.

Setting up conda env

1. Select a directory and clone the AI HarmonyCare Github Repository

git clone https://github.com/ludwigHoon/Project_KAI.git

2. At the cloned repository, update the file PROJECT_KAI\KAI_backend\setup_link.bat

- At line 1, replace it with the directory of RyzenAI transformers example.

- At line 2, replace it with the directory of the extracted ryzenai-sw.

e.g.: The final output is something similar to:

SET AMD_DIR=C:\github\RyzenAI-SW\example\transformers
SET RYZENAI_SW_DIR=C:\Users\kai\Downloads\ryzen-ai-sw-1.1

Setup Huggingface_hub

1. Create a huggingface account and create a new access token.

2. Edit the PROJECT_KAI\run.bat in line 1, replace <your_token> with the token you obtained from hugginface.

SET HF_TOKEN=<YOUR_TOKEN>

Running the application

1. Ensure you have completed all setup steps above.

2. At the AI HarmonyCare Github Repository clone directory, locate the file named "run.bat"

3. At the terminal, Execute the "run.bat" and 2 command prompts will be launched

4. Wait patiently to let LLM models load (Note: first-time run may take longer time)

5. Access the GUI via a web browser using the link http://localhost:8000/

Note: Make sure you allow all the permissions needed in web broswer

Note: All videos are unlisted on YouTube as this editor only allows video links. LOL

See Running the application-only Demo:

Emotion/Facial expression-based break recommendation demo:

Accessing the latest email and calendar info demo:

Future works:

1. Testing and identifying the most suitable model, while there are many AWQ-quantised models on huggingface, most were quantised using autoawq and the weights were not easily transferred. At the point of this writing, llama3-8b is one of the most capable models, though the speed of generation is somewhat disappointing.

2. Reducing the loading time for the model. The model takes a long time to load, reducing the time needed can be achieved by packaging the quantised model more efficiently.

3. Training and using a more accurate and suitable model to determine if the worker is fatigue/stressed. This can help to ensure that the notification for workers to take a