Automatic Calorie Tracking Scale

For the past couple of months, I have been frequently logging my meals on HealthifyMe to track my caloric goals. It is a great tool to ensure that you are having a well-balanced diet.

However, it gets a little tiresome over time to open the app and keep logging the same items over and over with a few adjustments in quantity. To ease my pain, I decided to build a system that would automatically log my meals when kept on the weighing scale. This machine uses an image classifier to identify the foods kept on it and logs them within HealthifyMe.

In this post, I shall explain how I went through building this project with step-by-step instructions. We shall start with connecting our RaspberryPi Zero to the weighing scale to fetch weight values and move on to setting up a wireless ESP32 Cam to capture food images. Once we have a collection of varied food images, we will move on to training a small classifier online and exporting the model to a quantized version which will allow us to run it locally on our Pi Zero using TFLite. With the food classification model set up, we will go on to build a food logger API that will allow us to log our food and the respective quantity to HealtifhyMe.

By the end of this post, you will have an end-to-end system set up in your kitchen, that will help you track your calorie intake precisely using a weighing scale, a camera, and Healthify Me.

Prerequisites

• Basic operating knowledge of Raspberry Pi
• Soldering skill
• Experience in working with hardware modules
• Experience with Python Programming

Setting up the weighing scale

Our first task is to fetch the measurement values from our weighing scale. To do this we will open our weighing scale and solder some Jumper cables to our load sensor.

Open the weighing scale by unscrewing the bolts on the rear. Within you should notice the LCD display and the PCB attached to the front panel at the bottom. Notice the four wires (black, red, white, green) that are connected from the load sensor to the PCB. We need to attach our jumper cables to these wires.

Detach the PCB from the front panel. At the bottom, you should notice the four wires from the load cell being attached to the PCB. We shall solder our jumper cables over here.

Open the ends of 4 jumper cables using a wire cutter as shown below. I have used the same color coding for the wires as the ones attached to the load sensor. This will come in handy later when we attach them to the HX711 module.

Solder the ends of these wires to the existing connections as shown in the image below.

Setting up the HX711 module

Our next step involves fetching the load sensor values within the Raspberry Pi Zero via the HX711 module. The HX711 module is an ADC converter that converts analog signals from our sensor to digital signals for our Pi Zero to read. We need to use this module as an interface to interpret the signals from the load sensor.

Connect the Load Sensor to the HX711 module in the following manner.

If your HX711 module does not have header pins, you can solder it directly to the slots present. However, I would recommend soldering the header pins to the module and using female port connectors to easily attach/detach to/from the module as shown below.

Now connect the HX711 module to your Raspberry Pi as shown below.

Your final setup should look something like this with the HX711 module and Raspberry PI connected.

Setting up the Raspberry within the Weighing Scale

Now that we have set up our Pi Zero to capture weight values from the weighing scale, it is time to place it within the weighing scale. We will place the Raspberry Pi Zero and the HX711 module within the weighing scale along with a 5V battery to power the Pi. A setup like this will allow our scale to be easily portable.

First, let's power the raspberry with a 5V 16340 Li-Ion Battery and a 16340 Single Battery Shield. To do this we need to connect the 5V and GND pin from the battery shield to the Pi Zero via the header pins ( 2 and 6 ) using jumper cables as shown below.

Insert a battery in the battery shield and turn on the switch at the top. If done right you should see the green led blinking on the Pi Zero.

Now we can place these devices within the weighing scale and arrange it in a way so that it does not disrupt the current functioning of the device. The following image shows my setup.

I have used some tapes to stick the devices to the scale and made sure the wires are well arranged so that the lid can be closed properly.

Once done, switch on the raspberry and close the weighing scale by screwing the cover back on.

Setting up the camera

To log our meals we need to identify the meal we are having. We will do this using a camera to capture images and passing it to an image classifier within our raspberry. Let's go on and set up a camera to fetch images.

For this project, we are going to use the ESP32 Cam. The ESP32 Cam is a wireless camera that can transmit images via an in-built server. The ESP32 Cam consists of a module and a camera that has to be attached to it.

To set up the ESP32 Cam, we will require a USB to TTL UART serial converter which will help us interface with the ESP32 Cam and set it up via our machine.

Please follow this tutorial to set up your ESP32 Cam with your WiFi. Once done you can check the images by calling the /capture endpoint on your module's IP.

Note: Comment out the following line before burning the program to ensure the camera captures the images in the highest resolution.

//  s->set_framesize(s, FRAMESIZE_QVGA);

Now that we have to ESP32 Cam set up, we need to power it using a battery. We are going to use another 5V 16340 Li-Ion Battery and a 16340 Single Battery Shield to power our camera. Insert the battery in the battery shield and plug the USB to TTL UART serial converter within the USB Port on the Battery shield. You can also power the ESP32 Cam using just the 3.3v and GND ports. I have made use of the serial converter as it was a much easier option and worked out of the box.

Note that you do not need to connect the UOT and UOR pins anymore as we won't be watching any logs. Once set up, press the reset button on the ESP32 Cam and access your camera IP to ensure it works.

To make the camera more portable I packaged the camera along with the battery within a disposable cardboard box and pointed the lens out through a rectangle hole I created at the bottom.

Now that we have the camera ready, we need to arrange our final setup.

The Kitchen Setup

With the wiring done and the modules connected, I arranged my setup on a kitchen counter.

This is the final placement for my project. I have attached the ESP32 Cam to the underside of my kitchen cabinet so that the camera faces the scale.

The main idea of this setup is to have the weighing scale below the camera so that it can capture images of the food.

You can use the webserver to test the position of the camera.

Fetching weight values from the machine

Now that you have set up your camera and weight sensor, we can move on to setting up the weight readings. Login to your pi device via ssh.

Note: You can read how to ssh into or use your raspberry pi by following this tutorial.

Download the hx711.py file to your project. Now create a main.py file and import the HX711 module within it. Add the following code within it.

import RPi.GPIO as GPIO
from hx711 import HX711

referenceUnit = 1 # Calculate based on weight readings
hx = HX711(5, 6) # Ports 5 & 6 to interact with the module
hx.set_reading_format("MSB", "MSB")
hx.set_reference_unit(referenceUnit)
hx.reset()
hx.tare()

while True:
    try:
        value = hx.get_weight() # Read value from sensor
        print(value)

    except (KeyboardInterrupt, SystemExit):
        GPIO.cleanup()

This code basically fetches weight values from your load sensor via the hx711 module and prints them to the console. If everything is set up fine you should see some large values being printed.

We need to calculate a reference unit within our code so that the machine detects the weight accurately. To do this, run the main.py file and add an object to the weighing scale. Note the value from the LCD display on the weighing scale. Also, note the values being printed on the console. You might notice numbers in a certain range i.e 99000 - 99700 depending on the weight of the object. Take a median value from the range i.e 11350 and divide it by the weight of the object in gms. eg. If the weight of the object is 200gms and the values being printed on the console are around 99350, then we shall do 99350 / 200 = 496.75. This will be our reference unit. Replace the existing reference value with 496 and rerun the main.py file. After adding some items to the weighing scale you will notice the console prints the correct weight value in gms. You can tweak the reference unit until the weight values are printed accurately.

Capturing images to the device

Let's add the functionality to click and store pictures whenever something is placed on the scale. The following code will be used to store the pictures in the images directory. Ensure that you replace the CAMERAIP in the code, with the IP assigned to the ESP32Cam module.

import os
import time
import requests
IMAGES_FOLDER = 'images'

if not os.path.exists(IMAGES_FOLDER): os.mkdir(IMAGES_FOLDER)

def capture():
    img =  requests.get("http://<CAMERAIP>/capture").content
    img_name = int(time.time())
    img_path = f"images/{img_name}.jpg"
    with open(img_path) as file:
        file.write(img)
    return img_path

In the while loop, add the following code.

while True:
    try:
        value = hx.get_weight()
        print(value)

        while value > 10:
            img_path = capture()
            print(f"{int(value)}gm")

            time.sleep(0.2)
            value = hx.get_weight_A()

    except (KeyboardInterrupt, SystemExit):
        GPIO.cleanup()

This code captures an image whenever the weight sensor detects a weight greater than 10gm. When we run our program and place something on our sensor, you will notice images appear in the images directory.

Creating a classifier

We now collect images for all the items we wish to identify by adding them to the scale for at least 10 seconds. Move your food item around a bit so that the image dataset is a bit diversified. Different sizes, lighting conditions, the quantity goes a long way while training a classifier.

After capturing the images, the next step in our process is to train a classifier to recognize these items. To train this dataset we will avoid using any complex deep learning frameworks or state-of-the-art models. Instead, we can achieve our goal easily by using an online machine learning tool that will train the model within our browser.

Transfer the images folder to your Desktop PC using a file transfer command like scp. Head over to Teachable Machine and choose an Image project. Add the images for each class within the UI.

Important: Ensure that you name your food items as close to their names on the HealthifyMe app, as these names will be used on the app to search for the items.

Now click on the Train Model button.

That's it. You now have a trained classifier to detect your foods. Export the model to a Tensorflow lite format (Floating Point). You should now have a converted_tflite.zipfile on your system containing labels and model files within.

Extract the archive over to your project folder on Pi Zero. If done right you should have the converted_tflite folder in your hx711py folder.

Setting up TF Lite

To run our model on PiZero we need to use a lighter version of TensorFlow called TFLite. TFLite is designed for devices with limited resources like Mobile phones and IoT gadgets. To install TFLite on our PiZero, we need to build the executable from the source. The instructions for the same are mentioned here.

Alternatively, you can also download the wheel I built for this project here. You can easily install it using the pip install command.

pip install tflite_runtime-2.7.0-cp37-cp37m-linux_armv6l.whl

Also, make sure you have NumPy and Pillow installed.

pip install numpy pillow

Download the classfier.py file from Github and add it to your project. The Classifier class within, will let us import our model and run an inference.

Now let's load our model and try to classify an image.

from classifier import Classifier

img_path = "images/test.jpg"
classfier = Classifier('converted_tflite/model_unquant.tflite', 'converted_tflite/labels.txt')
label = classfier.infer(img_path)
print(label)

If configured correctly you should have the label of the image printed out.

Edit the main.py file to accommodate these changes. Import and initialize the classifier as shown above. Then add the following inference code within the while loop. This code will call the classifier whenever an object is placed on the weighing scale.

while value > 10:
    img_path = capture()
    label = classifier.infer(img_path)
    print(f"{label} : {int(value)}gm")

    time.sleep(0.2)
    value = hx.get_weight_A()

Run the main.py file and add an item on the scale. The model should now detect and print the food label on the console.

Logging the items within HealthifyMe

The last step in this process is to enter the food item along with the weight within HealthifyMe. This part is a bit tricky as HealthifyMe does not provide a public API for use. To get around this issue I had to manually monitor my app traffic using Charles proxy and find the right APIs. You can follow my earlier post to do the same.

After you get Charles set up to track your HealthifyMe API, the next step involves getting your credentials from the app, so that our script can make requests on behalf of the app. Log in to your account and make a few requests from the app by opening screens. The requests should appear within Charles. Explore the headers of these requests to find your API key. It should be available within the authorization header. It should start with ApiKey followed by your phone number and will look something like this: ApiKey +91xxxxxxxx03:c2cdaa65c8eab20c8d86f4453f81108ecc111f2. Also, note down the vc and auth_user_id present in the URL parameters.

Next, download the food_logger.py file and add it to your project. The food_logger.py file requires the API key, User ID, and VC that we have retrieved above. Add these values with the ones we fetched above.

One thing to note is that HealthifyMe logs the food items within the breakfast/snack/lunch/dinner bracket. I have added a few conditions to automatically pick the right bracket based on the time of the day.

if 7 <= hour < 11:
        meal_type = 'B'
    elif 12 <= hour < 15:
        meal_type = 'L'
    elif 19 <= hour < 22:
        meal_type = 'D'

Let's import the food_logger module within our main.py and add it to our while loop.

import food_logger

...
...


while True:
    try:
        value = hx.get_weight()
        counter = 0
        print(value)

        while value > 10:
            img_path = capture()
            label = classifier.infer(img_path)
            print(f"{label} : {int(value)}gm")

            counter += 1
            if counter == 3:
                food_logger.log(label, value)

            time.sleep(0.2)
            value = hx.get_weight()

    except (KeyboardInterrupt, SystemExit):
        GPIO.cleanup()

Notice the use of the counter variable added to the loop. The counter is added to ensure that the food is logged is only once i.e on the third inference call of the object.

Test our machine

Our small weighing scale is now ready for a test. Run the main.py file in python. Once the tare is done and the program is ready to receive weights, add an item on the scale. Open the HealthifyMe app and verify if the item has appeared within the Breakfast/Lunch/Dinner section.

Your calorie weighing scale is now ready. You can run this program as a service so that it automatically starts when the raspberry boots up. This way you easily begin logging your meals on the app when you start the machine.

We can further improve the food classification feature by using a Siamese network to dynamically add a new food item with few images. This model would not require re-training for adding new food items. Given the ease of implementation, this project could ideally be integrated in every food weighing scale and installed in every home to help people track their foods.

If you liked this project and wish to read more of such content please do consider following me on Twitter. I frequently delve into the internals of software and journal my learnings on this blog, so make sure you also subscribe to it.

You can find the code for this post on Github.

References

• "Build A Digital Raspberry Pi Scale (With Weight Sensor HX711)". Raspberry Pi Tutorials, 2021, https://tutorials-raspberrypi.com/digital-raspberry-pi-scale-weight-sensor-hx711/. Accessed 30 June 2021.
• "Tatobari/Hx711py". Github, 2021, https://github.com/tatobari/hx711py. Accessed 30 June 2021.
• Digital Kitchen Scale ; How to Disassemble, Repair and Manufacture It ! Part 2. www.youtube.com, https://www.youtube.com/watch?v=u_MJ3uQg4YY. Accessed 30 June 2021.
• “Charles Proxy: Debug Your Android Traffic.” TechTalk, 15 May 2021, https://techtalk.digitalpress.blog/charles-proxy/.
• Teachable Machine. https://teachablemachine.withgoogle.com/. Accessed 30 June 2021.
• "Python Quickstart | Tensorflow Lite". Tensorflow, 2021, https://www.tensorflow.org/lite/guide/python. Accessed 30 June 2021.
• How to Setup Python Script Autorun As a Service in Ubuntu 18.04 | Learn Technology With Websofttechs For Free. 28 Feb. 2020, https://websofttechs.com/tutorials/how-to-setup-python-script-autorun-in-ubuntu-18-04/.
• IDE, Ai-Thinker. "Ai-Thinker ESP32-CAM In The Arduino IDE – Robot Zero One". Robot Zero One, 2019, https://robotzero.one/esp32-cam-arduino-ide/. Accessed 12 Sept 2021.