Sight Buddy - AI For Your Eyes

California Vision thanks all of the sponsors who contributed to the Build2Gether 2.0 Superbox including Nordic, DFRobot, Blues, Seeed Studio, ARM, and Adafruit!

Sight Buddy is a suite of tools that enables the use of AI features on embedded devices. Elbow Donut is an adaptive haptic product to enhance dexterity that I have made since 2019. Build2Gether 2.0 allowed me to think about how to bring these concepts together and I am grateful for the support of all the sponsors!

Video of Blues Notecard and XIAO syncing:

Nordic nRF52840DK - Great Documentation and Tutorials

Part of the challenge during this contest was navigating a steep learning curve with the components. Nordic offers training courses through their Nordic Dev Academy which are all free and include excellent code for stepping through the coursework. Nordic has nrfCloud for hosting projects and collecting data, and also appears to be supporting Memfault for embedded projects.

The nRF52840DK is an impressive board and I recommend it. The repository used in the Fundamentals course was very helpful and I'm frustrated that a better demonstration didn't result from using it. I found a great case for the board that covers the edges and the 3MF file is in the Github repository. There are many sophisticated features with this kit including NFC and cryptography modules that I plan to explore in the future. Programming for Matter, Zigbee, and Zephyr are all discussed in the Nordic documentation as interoperable development frameworks.

XIAO ESP32S3 Runs Hot!

The XIAO ESP32S3 is very impressive in general and the fact that it can launch a wifi-enabled webcam server by simply plugging in to a USB-C is amazing. The device gets remarkably hot with this camera and MCU setup and so I recommend caution if using the device as a wearable. I have also seen photos of the XIAO setup melting PLA housings when streaming and capturing images. I included several CAD files for housings in the GitHub repository and will be experimenting more with the heat dissipation effects of available 3D printing resins.

The code included for the ESP32S3 is from Espressif and has many optimization choices. The webcam server it launches also includes facial recognition and learning specific faces. The camera may be upgraded (will have a longer ribbon) for better quality. This chip seems like a perfect component to deliver a 2.4 GHz access point and also run a security camera with features like person detection.

Unihiker - SBC with Rockchip processor Offers a great developer experience!

The Unihiker device is an amazing value at $70 for a SBC with accelerometer and well-designed dashboard directory experiences. Bluetooth connectivity is a strong feature and the device can connect to the internet via a browser when plugged or Wi-Fi. Unihiker can serve as a hotspot. This feature enables you to tether Unihiker to your phone for internet, which then enables the hotspot feature for other devices to connect to Unihiker.

I recommend using the USB port to run a UVC camera to start because it is a serious challenge to achieve video output from any other type of connection. Before this contest I definitely thought plugging in a female 4-pin I2C or SPI connector would be enough to have immediate video and that just did not happen for me. I would like to use advanced features such as MQTT more and had some success pushing images from the Unihiker to an S3 bucket on Cubbit.io.

Grove Vision AI Module V2 Kit

This is an impressive component that did an excellent job of running facial detection models on the device. The models can be loaded from Sensecraft AI. It was a challenge for me to stream video using the ESP32S chip and so I used the streaming feed shown on the Sensecraft site. I attempted to train an existing model to identify playing cards and was not successful. Many of the models such as People Detection are outstanding and deploying those models to report on the status of social gatherings is one future goal for Sight Buddy. This kit could be worn at social gatherings and give the user updates on demand regarding the number of people in the room, whether they are facing towards them, etc. The chip can be programmed using AT and I think could do many of the same things as a ESP32 chip with further programming.

Blues Notecarrier A and Notecard Cellular

The Blues Notecarrier-A and Cellular Notecard are very exciting products. I was able to update the Notecarrier firmware and was able to start sending messages to Notehub.io. The overall platform is a strong solution for using Cloud API features for AI services while outside the home. It is possible to efficiently encode image data so that the data allotment can call Gemini 30-50 times per day to describe an image.

I must improve with data structures overall and JSON in particular. I struggled with the serial connection between the MCUs and the Notecarrier and was unable to realize the promise of these components for this submission. I am currently designing a housing for the Notecarrier and will update the project repository when it is finished. The global coverage of the Cellular Notecard is amazing and can keep you connected anywhere in the world with 5G. If that isn't enough there is always Starnote!

DFR Gravity

Cubbit.io = Low Cost S3 Egress

This project is still a work in progress, I was able to execute image capture on the Unihiker from an image stream broadcast by the XIAO ESP32S in the form of a webcam server. This was an unexpected solution and I will continue to optimize the solution architecture that includes components, Cubbit.io as an S3 platform, and Google Gemini to analyze images.

Elbow Donut Haptic Interfaces

Serial connections are not trivial and ensuring that data routes correctly between devices is difficult. There were some challenges navigating custom connectors between manufacturers and the UVC camera option was the most reliable. I was unable to integrate my Elbow Donut concept as I desired and will continue working to deliver more features in the Sight Buddy AI Vision Suite. Elbow Donut objects were initially created as a mobile solution to leaning on hard surfaces, and platinum silicone is the material of choice. This allows the objects to be dishwasher-safe and some formulations are optimized with anti-fungal properties.

I currently sell ergonomic objects in my Etsy store and would like to make cases for embedded devices that enable users with limited dexterity to improve their grip. The Unihiker silicone case resulted in a much better experience than using the device without a cover. I recently started to use a 4K resin 3D printer and plan to extend the designs included in the repository. Sensors can also be placed inside Elbow Donut haptic interfaces to learn more about the force vectors that occur for users with normal and abnormal dexterity.