Getting Started with the TDA4VM Edge AI Starter Kit

The TDA4VM Edge AI Starter Kit from Texas Instruments is a ARM-based Linux development board for developing smart camera applications, robots, and other intelligent machines. It comes ready out-of-the-box with available AI demo applications in a pre-built Linux SD card image, but users can also access the source code to build their own version of the Linux image to add customization such as new hardware to the device tree.

As the name of the development board implies, it is powered by the TDA4VM processor. The TDA4VM is a dual Arm® Cortex®-A72 system on chip (SoC) and C7x DSP. It also has deep-learning, vision, and multimedia hardware accelerators built into it. TI's TDA4VM processors are a part of their Jacinto™ 7 lineup designed deliver efficient edge-computing performance for automotive, industrial, and IoT devices.

As I've been dabbling with developing AI applications and ML models lately, I was curious to branch out and try other hardware and SDK setups. The TDA4VM Edge AI Starter Kit is interesting because is it comes preloaded with an assortment of AI deep learning models you can run demos of out-of-the-box. Personally, the human pose estimation deep learning model caught my attention as I've been wanting to build a project around that to help with my yoga practice (because it's surprisingly hard to just "feel" if you're doing the pose correctly).

So to get started, I'm going to fire-up a TDA4VM Edge AI Starter Kit with the default Linux image TI provides for it and run the human pose estimation demo to get an idea of how to works so I can integrate it into my own yoga pose judgement application.

Hardware Setup

In terms of hardware, I used the following:

• Monitor with HDMI port (DisplayPort is also an option with the TDA4VM Edge AI Starter Kit)
• LogiTech USB webcam (Raspberry Pi CSI-2 camera is also an option)
• Ethernet connection to my router
• MicroUSB cable connection to host PC

The Linux OS provided by TI for the TDA4VM board will live on and be booted from an SD card, so SW1 need to be set to SD boot mode (which was how the switches happened to be set on my TDA4VM board out of the box).

Finally, the TDA4VM board requires a USB C 20V/3A power supply wall adapter with a maximum current limit output of 5A. I picked up a cheap 60W USB C charging brick from Amazon that worked perfectly (linked in the material list above).

NOTE: Be sure to not connect the USB C power supply to the TDA4VM until the imaged SD card is installed.

Prep the SD Card

Download the latest or desired version of the embedded Linux image for the TDA4VM Edge AI Starter Kit board here.

Flash the image onto the SD card using an application such as balenaEtcher.

Install the SD card in the TDA4VM board on the bottom side with the top side of the SD card facing the bottom side of the TDA4VM PCB (also facing up relative to the benchtop). I mention this because it's a little counter intuitive since the norm is for the bottom of the SD card to face the PCB in other development boards.

Boot the TDA4VM

Open the preferred serial terminal application targeting the 3rd indexed serial port (there will be 4 total) that the TDA4VM created with a baud rate of 115200 (for example, if COM1 - COM4 appears, choose COM3).

Once booted up, the TDA4VM will output an HDMI signal with a desktop that states the user login information (the username and password) to log in with on the serial terminal from the host PC. The default username and password are root.

Run the ifconfig command and/or ping commands to verify the network connection:

root@tda4vm-sk:/opt/edge_ai_apps# ifconfig
root@tda4vm-sk:/opt/edge_ai_apps# ping google.com

Run the Deep Learning Demo Application

Even though I knew I wanted to look the human pose estimation deep learning model, I was still curious about what all was available to me by default in this Linux image for the TDA4VM so I just started by running the list (ls) command in the /opt/edge_ai_apps directory:

root@tda4vm-sk:/opt/edge_ai_apps# ls -la
total 3924
drwxr-xr-x 13 root root    4096 Feb  2 20:10 .
drwxr-xr-x 18 root root    4096 Dec 18 07:57 ..
drwxr-xr-x  8 root root    4096 Dec 26 14:27 .git
-rw-r--r--  1 root root     203 Dec 26 14:27 .gitignore
drwxr-xr-x  2 root root    4096 Dec 26 14:27 .vscode
-rw-r--r--  1 root root    2937 Dec 26 14:27 LICENSE
-rw-r--r--  1 root root 3855040 Dec 26 14:27 PROCESSOR-SDK-LINUX-SK-TDA4VM_manifest.html
drwxr-xr-x 10 root root    4096 Dec 26 14:27 apps_cpp
drwxr-xr-x  2 root root    4096 Dec 26 14:27 apps_python
drwxr-xr-x  2 root root    4096 Dec 26 14:27 configs
drwxr-xr-x  5 root root    4096 Dec 26 14:27 data
drwxr-xr-x  2 root root    4096 Dec 26 14:27 docker
drwxr-xr-x  5 root root    4096 Dec 26 14:27 docs
-rwxr-xr-x  1 root root    9387 Dec 26 14:27 download_models.sh
-rw-r--r--  1 root root   72348 Dec 26 14:27 edge_ai_apps_manifest.html
-rwxr-xr-x  1 root root    4643 Dec 26 14:27 init_script.sh
drwxr-xr-x  3 root root    4096 Dec 26 14:27 internal_docs
drwxr-xr-x  4 root root    4096 Dec 26 14:27 scripts
-rwxr-xr-x  1 root root    2559 Dec 26 14:27 setup_script.sh
drwxr-xr-x  2 root root    4096 Dec 26 14:27 tests

The demo applications can be run in either C++ (cpp) or Python by calling the edgeai script in the target language and specifying the target deep learning model to use with it.

All of the available deep learning models are located in the /opt/edge_ai_apps/configs directory:

root@tda4vm-sk:/opt/edge_ai_apps# ls -la ./configs/    
total 92
drwxr-xr-x  2 root root  4096 Dec 26 14:27 .
drwxr-xr-x 13 root root  4096 Feb  2 20:10 ..
-rw-r--r--  1 root root 14204 Dec 26 14:27 app_config_template.yaml
-rw-r--r--  1 root root  1246 Dec 26 14:27 display_and_encode_example.yaml
-rw-r--r--  1 root root  1032 Dec 26 14:27 face_detection.yaml
-rw-r--r--  1 root root  4495 Dec 26 14:27 gst_plugins_map.yaml
-rw-r--r--  1 root root   966 Dec 26 14:27 http_src_example.yaml
-rw-r--r--  1 root root  1041 Dec 26 14:27 human_pose_estimation.yaml
-rw-r--r--  1 root root  1200 Dec 26 14:27 image_classification.yaml
-rw-r--r--  1 root root  1224 Dec 26 14:27 imx390_cam_example.yaml
-rw-r--r--  1 root root  1593 Dec 26 14:27 multi_input_multi_infer.yaml
-rw-r--r--  1 root root  1286 Dec 26 14:27 object_detection.yaml
-rw-r--r--  1 root root  1014 Dec 26 14:27 ov5640_sensor_example.yaml
-rw-r--r--  1 root root  1358 Dec 26 14:27 remote_display.yaml
-rw-r--r--  1 root root  1118 Dec 26 14:27 rpiV2_cam_example.yaml
-rw-r--r--  1 root root  1110 Dec 26 14:27 rtsp_src_example.yaml
-rw-r--r--  1 root root  1276 Dec 26 14:27 semantic_segmentation.yaml
-rw-r--r--  1 root root  1210 Dec 26 14:27 shared_model_instance.yaml
-rw-r--r--  1 root root  1510 Dec 26 14:27 single_input_multi_infer.yaml

So I changed directories into the Python demo application directory (/opt/edge_ai_apps/apps_python) and ran the Python Edge AI application with the human pose estimation deep learning model.

root@tda4vm-sk:/opt/edge_ai_apps# cd ./apps_python/
root@tda4vm-sk:/opt/edge_ai_apps/apps_python# ./app_edgeai.py ../configs/human_pose_estimation.yaml

To my surprise, the demo ran with a canned video and didn't appear to have needed the webcam to run out-of-the-box.

While the getting started guide from TI says that the demos are pulling frames from a USB webcam to pass through the models, they are actually pulling frames from prerecorded video files included in the TDA4VM's default Linux image. The following snippet of output from the serial console of the TDA4VM while the demo was running confirmed this:

filesrc location=/opt/edge_ai_apps/data/videos/video_0001_h264.mp4

Looking at the Python app script, app_edgeai.py and edge_ai_class.py I realized the input parameters were being loaded from elsewhere. This was confirmed by the Demo Configurations File and Data Flows documentation page on TI's website which explained that the input, output, and model file parameters are loaded in by the YAML format demo config file.

The YAML format demo config file defines input sources, models, outputs and the flows which defines how everything is connected (ie - which input source is routed through which model, and where the model sends its output to). So I took a look at the config file for the human pose estimation model (located in /opt/edge_ai_apps/configs/human_pose_estimation.yaml):

title: "Human Pose Estimation Demo"
log_level: 2
inputs:
    input0:
        source: /dev/video2
        format: jpeg
        width: 1280
        height: 720
        framerate: 30
    input1:
        source: /opt/edge_ai_apps/data/videos/video_0001_h264.mp4
        format: h264
        width: 1280
        height: 720
        framerate: 30
        loop: True
    input2:
        source: /opt/edge_ai_apps/data/images/%04d.jpg
        width: 1280
        height: 720
        index: 0
        framerate: 1
        loop: True
models:
    model0:
        model_path: /opt/model_zoo/ONR-KD-7060-human-pose-yolox-s-640x640
        viz_threshold: 0.6
outputs:
    output0:
        sink: kmssink
        width: 1920
        height: 1080
    output1:
        sink: /opt/edge_ai_apps/data/output/videos/output_video.mkv
        width: 1920
        height: 1080
    output2:
        sink: /opt/edge_ai_apps/data/output/images/output_image_%04d.jpg
        width: 1920
        height: 1080

flows:
    flow0: [input1,model0,output0,[320,180,1280,720]]

By default, this model is configured to potentially use three different input sources: a USB webcam indexed as /dev/video2 (input0), a preloaded video file (input1), or preloaded image files (input2). There is only the one option for the model file itself, then there are also three different options for output sources: the display connected to the TDA4VM board (output0), a new video file (output1), or new image file(s) (output2).

Then the last thing defined in the demo config file is the flow, which by default connects the preloaded video file (input1) to the human pose estimation model, and connects the model's output to the display connected to the TDA4VM board (output0). It also defines the size of the output image frames for the display.

Since I wanted to see this demo run with a video stream pulled from the USB webcam I have connected to the TDA4VM, all that I needed to do was change the flow to pull from input0 instead of input1:

flows:
    flow0: [input0,model0,output0,[320,180,1280,720]]

I also double-checked that my Logitech Brio webcam was showing up at /dev/video2 (otherwise I'd had to update the video index in human_pose_estimation.yaml as well):

root@tda4vm-sk:/opt/edge_ai_apps# v4l2-ctl --list-devices
TI-CSI2RX (platform:4500000.ticsi2rx):
        /dev/media0

TI-CSI2RX (platform:4510000.ticsi2rx):
        /dev/media1

vxd-dec (platform:vxd-dec):
        /dev/video0

vxe-enc (platform:vxe-enc):
        /dev/video1

Logitech BRIO (usb-xhci-hcd.0.auto-1.1):
        /dev/video2
        /dev/video3
        /dev/video4
        /dev/video5
        /dev/media2

I then reran the demo:

root@tda4vm-sk:/opt/edge_ai_apps# cd ./apps_python/
root@tda4vm-sk:/opt/edge_ai_apps/apps_python# ./app_edgeai.py ../configs/human_pose_estimation.yaml

And the model worked great!

At this point, the next step is to take a look at the post-processing step to determine how I want to handle the output from the human pose estimation model for my yoga pose judgement application.

Yoga Pose Post-Processing

The human pose estimation model works by applying a “visualization” path to the post-processing module for each image frame which overlays the keypoints on the human body (the dots) and connects them with lines to draw the pose.

These lines that create the pose then need to be correlated to correct and incorrect yoga poses:

The post-processing routines for the Python demo applications are located in the /opt/edge_ai_apps/apps_python/post_process.py script.

Taking a look at the post processing route for the human pose estimation, the PostProcessPoseEstimation class, we see the logic for identifying the overlayed the keypoints on the human body (the dots) identified by the DNN and then connecting them with lines to draw the pose.

They also happen to be color coded so it should be fairly straightforward to group the different body lines together for a given pose then apply some simply geometry to determine if the lines correlate to the correct position of a pose. But I'm going to have to determine the correct geometry for each pose so this project is going to have to be a multipart series (see next post here).