Published June 6, 2023 © GPL3+

Practicing Yoga with AI: Human Pose Estimation on the TDA4VM

This project shows how I used TI's SK-TDA4VM and Edge AI Cloud tool to train a human pose estimation ML model to judge my yoga practice.

IntermediateFull instructions provided3 hours1,979

Practicing Yoga with AI: Human Pose Estimation on the TDA4VM

Things used in this project

Hardware components

Texas Instruments SK-TDA4VM Edge AI starter kit

Nekteck 60W USB C Charger

Audio / Video Cable Assembly, Ultra Slim RedMere HDMI to HDMI

Ethernet Cable, Cat6a

USB-A to Micro-USB Cable

Webcam, Logitech® HD Pro

Story

As an individual that spends entirely too much time literally hunched over a keyboard, I took an interest in yoga in few years back in an attempt to help my poor neck and back. I quickly learned however that it's surprisingly difficult to be able to tell if you're holding a pose correctly, so I started filming myself.

It was comical how I felt so sure that I was holding a pose perfectly, only to watch back the video on my phone to see that I looked more like Creed doing "the perfect cartwheel".

Then with the uptick of machine learning and AI in the maker realm, I saw the human pose estimation deep learning (DL) models popping up. So naturally, my first thought was the idea that I could somehow deploy this with my yoga practice.

Since I'm not a machine learning expert by any means (I'm old enough that machine learning wasn't even an elective course yet during my undergrad and grad years), it was important that I could find a pre-built/trained human pose estimation model and simply rerun calibration inference on it, versus creating or training one from scratch. This is when I found the TDA4VM Edge AI Starter Kit and Edge AI SDK from Texas Instruments.

The repository of pre-built DL models ready for use out-of-the-box in TI's Model Zoo contained a 6D pose estimation model based on YOLOX that takes a single image input and infers the pose of each object without any intermediate representation or complex post-processing like refinement. Furthermore, the models in TI's Model Zoo are optimized for performance and accuracy on the TDA4VM chip via the TIDL tools compiling them for hardware acceleration in the C7x DSP blocks of the TDA4VM processor.

So my objective was to take this pose estimation YOLOX model and compile it with calibration images of myself in various yoga poses, then create an algorithm in the post processing function to determine if the skeleton drawn over the pose from a new live image captured by the webcam was in the "correct" pose.

Compile Model in TI Edge AI SDK

First things first, I needed to compile the pose estimation YOLOX model from Model Zoo with calibration images of myself doing various yoga poses. I found that the Model Analyzer tool in TI's Edge AI SDK was the easiest way to accomplish this step.

Model Analyzer in TI's Edge AI SDK is a free cloud-based tool that allows you to emulate their hardware like the TDA4VM to compile and deploy a deep learning model using industry standard runtime engines from a web browser. I found this super handy for figuring out how to navigate my way around the file structure and my custom implementation of a pre-trained model in my application in the Jupyter notebooks in the cloud EVM.

Overall, what I did was simply compile the human pose estimation model from Model Zoo for the TDA4VM by creating an Onnx runtime using the tidl_model_import_onnx library to generate artifacts and offload the supported portion of the human pose estimation model to the TDA4VM's C7x DSP.

I then passed it various images of myself in different poses (screenshots from my aforementioned videos) to get keypoint data back for my post processor algorithm. Since I'm using a pre-trained model, I found that it was important to make sure both the images used in the calibration inference and execution inference runs were the appropriate size the model was expecting.

In the case of the YoloX model I used in this project, that turned out to be 640x640. I learned this the hard way when the skeleton for the human pose would be drawn either too big or too small and in some random spot in the image instead of over the body of the subject. I was initially using images that were way too large to the skeleton was tiny and placed in the upper lefthand corner of the image, but it was in the right shape:

Another pitfall that's easy to hit is making sure you call the right inference type with the EP_List variable. When compiling the model and passing it calibration images, the TIDLCompilationProvider needs to be called. While inference for new data from a regular run of the final application needs to call TIDLExecutionProvider. This is easy to miss as a copy+paste error (which is what I did), and is how I discovered that trying to compile a model twice will kill the kernel.

I made sure to make the directory the compiled model artifacts went to was easy to find so I could zip it up from a terminal shell and download it from the Jupyter notebook to my PC:

output_dir = '/home/root/notebooks/custom-artifacts/onnx/yolox_s_pose_ti_lite_49p5_78p0.onnx'

Another cool thing in Model Analyzer, is that you can run benchmarking tools on your compiled DL model to verify/visual the hardware acceleration on the TDA4VM chip:

Note: if you create any custom files in Model Analyzer, be sure to download them to your PC on a regular basis as they will automatically be deleted after 30 days of inactivity.

Deploying ML Model on SK-TDA4VM

Transitioning my custom model artifacts from Model Analyzer in Edge AI SDK to my SK-TDA4VM board was pretty straightforward. I simply wrapped up the Jupyter notebook that I had created in the cloud along with my custom model artifacts from my compiling of the human pose estimation model, and uploaded them all together as a compressed archive to a Jupyter notebook on my SK-TDA4VM board where I was able to do the final integration.

There is a great getting started guide on TI's website that covers how to prep the SK-TDA4VM board with all of the necessary hardware and how to flash an SD card with the pre-built Linux image for it. There is a note cautioning against the known bug with BalenaEtcher (1.7.2 at the time of writing this), but I found another issue not noted.

After I flashed my first SD card and went to boot my SK-TDA4VM board, I would get a timeout error and the boot would just hang indefinitely:

[    4.373780] debugfs: Directory 'pd:239' with parent 'pm_genpd' already present!
[    4.393520] ALSA device list:
[    4.396482]   No soundcards found.
[    5.404306] mmc0: SDHCI controller on 4fb0000.mmc [4fb0000.mmc] using ADMA 64-bit
[    5.412801] Waiting for root device PARTUUID=871090a6-02...
[    5.457911] mmc0: error -110 whilst initialising SD card
[    5.588945] mmc0: error -110 whilst initialising SD card
[    5.725134] mmc0: error -110 whilst initialising SD card
[    5.862363] mmc0: error -110 whilst initialising SD card

After much panic and distress thinking I had somehow damaged a pin in the actual SD card slot, I discovered that the firmware in the pre-built image just doesn't get along with all SD card brands. Once I switched to a SanDisk SD card, all was fine and the board booted as expected.

Note: I only thought to try this as this is not the first time I've been burned by this sameissue in the embedded Linux world. I've just decided that I'm exclusively buying and using SanDisk SD cards from now on...

There is a bit of environment prep necessary in the form of a few Python packages that need to be installed, which include the runtime engine being used (ONNX in my case) and Munkres. I also liked having progress bars show in my notebooks, so I installed tqdm as well:

root@tda4vm-sk:/opt/edgeai-gst-apps# pip3 install tqdm
root@tda4vm-sk:/opt/edgeai-gst-apps# pip3 install onnx
root@tda4vm-sk:/opt/edgeai-gst-apps# pip3 install munkres

I also found that TIDL tools is not installed by default in the pre-build image, but it's just a matter of simply cloning the matching version repository:

root@tda4vm-sk:/opt/edgeai-gst-apps# cd ../
root@tda4vm-sk:/opt# git clone https://github.com/TexasInstruments/edgeai-tidl-tools.git
root@tda4vm-sk:/opt# cd ./edgeai-tidl-tools/
root@tda4vm-sk:/opt/edgeai-tidl-tools# git checkout master

In order to save myself the worry of checking if the versions match, I just pulled the latest pre-built image from the TI Linux downloads page and matched it to the master branch of the TIDL tools repository. I also figured out (the hard way of course) that the version of TIDL tools in Edge AI SDK usually always matched the master branch of the TIDL tools repository. So if you compiled your model there like I did, the latest version of the pre-built image for the SK-TDA4VM board is what you need to use as well.

There are a couple of options for getting files onto the SK-TDA4VM board: you can either use a file transfer command such as scp to transfer them from a host PC or launch Jupyter notebook and upload them from a connected web browser on a host PC.

I chose the latter since I had already zipped all of my custom artifacts and notebook up in Edge AI SDK to download it onto my host PC, so it'd be easy to upload the compressed archive via Jupyter (then unzip it with a terminal).

Use ifconfig to see TDA4VM's IP address then launch Jupyter notebook:

root@tda4vm-sk:/opt/edgeai-gst-apps# jupyter-notebook --allow-root --no-browser --ip=192.168.x.xxx

As usual, the terminal will print out a web address for you to navigate to the notebook to from a web browser on a host PC where you can upload files as usual:

I had everything zipped up in yoga_pose_judge.tar.gz and unzipped it in the default /opt/edgeai-gst-apps/ directory.

I did have to make a few updates to the notebook to remove the steps for compiling the model and pointing to the new locations of the model and model artifacts since it is difference than in the EVM in Edge AI SDK:

tidl_tools_path = '/opt/edgeai-tidl-tools'
output_dir = '/opt/edgeai-gst-apps/yoga_pose_judge/model-artifacts/onnx/yolox_s_pose_ti_lite_49p5_78p0.onnx'
onnx_model_path_TDA4VM = '/opt/edgeai-gst-apps/yoga_pose_judge/model/yolox_s_pose_ti_lite_49p5_78p0.onnx'
onnx.shape_inference.infer_shapes_path(onnx_model_path_TDA4VM, onnx_model_path_TDA4VM)
    
# model compilation options
compile_options = {
    'tidl_tools_path' : tidl_tools_path,
    'artifacts_folder' : output_dir,
    'tensor_bits' : 8, 
    'accuracy_level' : 1,
    'object_detection:meta_arch_type': 6,
    'object_detection:meta_layers_names_list': f'/opt/edgeai-gst-apps/yoga_pose_judge/modelyolox_s_pose_ti_lite_metaarch.prototxt',
}

I also added the code to capture live images from the USB webcam, resize the image to 640x640 required by the DL model, run inference on the new image, then put it through the pose judgement post processing algorithm:

capture = cv2.VideoCapture(2)
time.sleep(0.1)
(success, reference) = capture.read()

cv2.imwrite('/opt/edgeai-gst-apps/yoga_pose_judge/captured_images/live_image.jpg',reference)

capture.release()

im = Image.open(r"/opt/edgeai-gst-apps/yoga_pose_judge/captured_images/live_image.jpg")
width, height = im.size

left = (width/2) - (height/2)
right = (width/2) + (height/2)
top = height - height
bottom = height

im1 = im.crop((left, top, right, bottom))
newsize = (640, 640)
im1 = im1.resize(newsize)
im1.save("/opt/edgeai-gst-apps/yoga_pose_judge/captured_images/live_image_resized.jpg")

Post Processing Pose Judgement Algorithm

The pose estimation post processing algorithm from utils.py is pulling the determined keypoint data output from the DL 6D pose estimation model and creating 17 keypoints for the subject found in a frame. Each of these keypoints have an X and Y coordinate, then it is simply drawing a line between each point based on the assigned color value to denote a give body part type (arm, leg, body, head).

I decided the easiest way to implement my yoga pose "judgement" algorithm was to calculate the slope of each line for a given body part then correlate a set of slopes to a pose with determined tolerances to say if the pose is correct or not.

slope = (y2 - y1)/(x2 - x1)

Next Steps

For now, I'm just printing these slope values of the lines of the drawn skeleton to the terminal and saving them to a CSV file since it'll take some time to correlate the slope values to a pose. AND this needs to be done for each pose, so needless to say it'll take some time but the concept is working!

Since the different parts of the body are assigned different color lines to be drawn with, each body part will have its own slope calculation for each pose that I can hopefully use as basic building blocks to assign to new poses as I go.

1 / 2

For example in the poses for upward dog and pigeon pose are similar in that the back/body and one leg are in the same positions so once I correlate the slope values for the body and leg for one, I can copy+paste it to the other.

Overall, this will be a fun tool to keep working on and ever iterating on! Hopefully my yoga skills will improve along with it!

#!/usr/bin/env python
# coding: utf-8

# In[1]:


import os
import re
import sys
import cv2
import tqdm
import onnx
import math
import copy
import shutil
import platform
import itertools

import numpy as np
import onnxruntime as rt
import ipywidgets as widgets
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches

from pathlib import Path
from munkres import Munkres
from numpy.lib.stride_tricks import as_strided
from IPython.display import Markdown as md
from PIL import Image, ImageFont, ImageDraw, ImageEnhance
from scripts.utils import imagenet_class_to_name, download_model, loggerWritter, get_svg_path, get_preproc_props, single_img_visualise, vis_pose_result


# In[2]:


def preprocess_for_onnx_pose_estimation(image_path, size, mean, scale, layout, reverse_channels, pad_color=114, pad_type="center"):
    # Step 1
    # read the image using openCVimport json_tricks as json
    img = cv2.imread(image_path)
    
    # Step 2
    # convert to RGB
    img = img[:,:,::-1]
    
    # Step 3    
    # Most of the onnx models are trained using
    # 512x512 images. The general rule of thumb
    # is to scale the input image while preserving
    # the original aspect ratio so that the
    # longer edge is 512 pixels, and then
    # pad the scaled image to 512x512
    
    size = (size,size) if not isinstance(size, (list,tuple)) else size
    desired_size = size[-1]
    old_size = img.shape[:2] # old_size is in (height, width) format

    ratio = float(desired_size)/max(old_size)
    new_size = tuple([int(x*ratio) for x in old_size])

    # new_size should be in (width, height) format
    img = cv2.resize(img, (new_size[1], new_size[0]))

    delta_w = size[1] - new_size[1]
    delta_h = size[0] - new_size[0]

    if pad_type=="corner":
        top, left = 0, 0
        bottom, right = delta_h, delta_w
    else:
        delta_w = size[1] - new_size[1]
        delta_h = size[0] - new_size[0]
        top, bottom = delta_h//2, delta_h-(delta_h//2)
        left, right = delta_w//2, delta_w-(delta_w//2)


    img = cv2.copyMakeBorder(img, top, bottom, left, right, cv2.BORDER_CONSTANT,
        value=pad_color)
    
    # Step 4
    # Apply scaling and mean subtraction.
    # if your model is built with an input
    # normalization layer, then you might
    # need to skip this
    if mean is not None and scale is not None:
        img = img.astype('float32')
        for mean, scale, ch in zip(mean, scale, range(img.shape[2])):
            img[:,:,ch] = ((img.astype('float32')[:,:,ch] - mean) * scale)
            
    # Step 5
    if reverse_channels:
        img = img[:,:,::-1]
    
    # Step 6
    img = np.expand_dims(img,axis=0)
    img = np.transpose(img, (0, 3, 1, 2))
    
    return img, top, left, ratio


# In[4]:


calib_images = [
    '/home/root/notebooks/side_bend_pose/image-640x640_0.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_1.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_2.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_3.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_4.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_5.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_6.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_7.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_8.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_9.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_10.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_11.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_12.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_13.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_14.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_15.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_16.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_17.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_18.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_19.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_20.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_21.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_22.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_23.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_24.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_25.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_26.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_27.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_28.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_29.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_30.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_31.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_32.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_33.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_34.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_35.png',
    '/home/root/notebooks/side_bend_pose/image-640x640_36.png',
]

output_dir = '/home/root/notebooks/custom-artifacts/onnx/yolox_s_pose_ti_lite_49p5_78p0.onnx'
onnx_model_path_EdgeAIcloud = '/home/root/notebooks/prebuilt-models/8bits/kd-7060_onnxrt_coco_edgeai-yolox_yolox_s_pose_ti_lite_49p5_78p0_onnx/model/yolox_s_pose_ti_lite_49p5_78p0.onnx'
onnx.shape_inference.infer_shapes_path(onnx_model_path_EdgeAIcloud, onnx_model_path_EdgeAIcloud)

log_dir = Path("logs").mkdir(parents=True, exist_ok=True)

# stdout and stderr saved to a *.log file.  
with loggerWritter("logs/custon-model-onnx"):
    
    # model compilation options
    compile_options = {
        'tidl_tools_path' : os.environ['TIDL_TOOLS_PATH'],
        'artifacts_folder' : output_dir,
        'tensor_bits' : 16, 
        'accuracy_level' : 1,
        'advanced_options:calibration_frames' : len(calib_images), 
        'advanced_options:calibration_iterations' : 3, # used if accuracy_level = 1   
        'object_detection:meta_arch_type': 6,
        'object_detection:meta_layers_names_list': f'/home/root/notebooks/prebuilt-models/8bits/kd-7060_onnxrt_coco_edgeai-yolox_yolox_s_pose_ti_lite_49p5_78p0_onnx/model/yolox_s_pose_ti_lite_metaarch.prototxt', 
    }

# create the output dir if not present
# clear the directory
os.makedirs(output_dir, exist_ok=True)
for root, dirs, files in os.walk(output_dir, topdown=False):
    [os.remove(os.path.join(root, f)) for f in files]
    [os.rmdir(os.path.join(root, d)) for d in dirs]


# In[5]:


# create & compile model with compile options specified above 
so = rt.SessionOptions()
EP_list = ['TIDLCompilationProvider','CPUExecutionProvider']
sess = rt.InferenceSession(onnx_model_path_EdgeAIcloud ,providers=EP_list, provider_options=[compile_options, {}], sess_options=so)


# In[6]:


input_details = sess.get_inputs()


# In[7]:


label = 'ONR-KD-7060-human-pose-yolox-s-640x640'
pad_color = 128 if 'ae' in label and 'yolo' not in label else 114
pad_type = "corner" if 'yolox' in label else "center"
size = 640
mean = [0.0, 0.0, 0.0]
scale = [1.0, 1.0, 1.0]
layout = 0
reverse_channels = True


# In[8]:


# run inference for each calibration image 
for num in tqdm.trange(len(calib_images)):
    image_name = calib_images[num]
    print('label = ', label)
    print('pad_color = ', pad_color)
    print('pad_type = ', pad_type)
    print('image_name = ', image_name)
    processed_image, top, left, ratio = preprocess_for_onnx_pose_estimation(image_name, size, mean, scale, layout, reverse_channels, pad_color, pad_type)
    
    print('processed_image', processed_image)
    print('top', top)  
    print('left', left)
    print('ratio', ratio)
    
    if not input_details[0].type == 'tensor(float)':
        processed_image = np.uint8(processed_image)

    image_size = processed_image.shape[3]    
    print('image_size = ', image_size)
    out_file=None
    output=None
    output = list(sess.run(None, {input_details[0].name : processed_image})) 
    print('output = ', output)


# In[9]:


EP_list = ['TIDLExecutionProvider','CPUExecutionProvider']


# In[10]:


sess = rt.InferenceSession(onnx_model_path_EdgeAIcloud ,providers=EP_list, provider_options=[compile_options, {}], sess_options=so)


# In[11]:


input_details = sess.get_inputs()


# In[15]:


from scripts.utils import single_img_visualise

image_name = '/home/root/notebooks/side_bend_pose/image-640x640_28.png'
processed_image, top, left, ratio = preprocess_for_onnx_pose_estimation(image_name, size, mean, scale, layout, reverse_channels, pad_color, pad_type)

if not input_details[0].type == 'tensor(float)':
    processed_image = np.uint8(processed_image)

image_size = processed_image.shape[3]    
output = list(sess.run(None, {input_details[0].name : processed_image}))[0]


# In[16]:


# post processing 
get_ipython().run_line_magic('matplotlib', 'inline')
output_image = single_img_visualise(output, image_size, image_name, out_file, top, left, ratio, udp=True, thickness=2, radius=5, label=label)

# plot the outut using matplotlib
plt.rcParams["figure.figsize"]=20,20
plt.rcParams['figure.dpi'] = 200 # 200 e.g. is really fine, but slower
plt.imshow(output_image)
plt.show()


# In[14]:


from scripts.utils import plot_TI_performance_data, plot_TI_DDRBW_data, get_benchmark_output
stats = sess.get_TI_benchmark_data()
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(10,5))
plot_TI_performance_data(stats, axis=ax)
plt.show()

tt, st, rb, wb = get_benchmark_output(stats)
print(f'Statistics : \n Inferences Per Second   : {1000.0/tt :7.2f} fps')
print(f' Inference Time Per Image : {tt :7.2f} ms  \n DDR BW Per Image        : {rb+ wb : 7.2f} MB')


# In[ ]:

#!/usr/bin/env python
# coding: utf-8

# In[1]:


import os
import re
import sys
import cv2
import tqdm
import onnx
import math
import copy
import time
import shutil
import platform
import itertools

import numpy as np
import onnxruntime as rt
import ipywidgets as widgets
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches

from pathlib import Path
from munkres import Munkres
from numpy.lib.stride_tricks import as_strided
from IPython.display import Markdown as md
from PIL import Image, ImageFont, ImageDraw, ImageEnhance
from utils import single_img_visualise


# In[2]:


def preprocess_for_onnx_pose_estimation(image_path, size, mean, scale, layout, reverse_channels, pad_color=114, pad_type="center"):
    # Step 1
    # read the image using openCVimport json_tricks as json
    img = cv2.imread(image_path)
    
    # Step 2
    # convert to RGB
    img = img[:,:,::-1]
    
    # Step 3    
    # Most of the onnx models are trained using
    # 512x512 images. The general rule of thumb
    # is to scale the input image while preserving
    # the original aspect ratio so that the
    # longer edge is 512 pixels, and then
    # pad the scaled image to 512x512
    
    size = (size,size) if not isinstance(size, (list,tuple)) else size
    desired_size = size[-1]
    old_size = img.shape[:2] # old_size is in (height, width) format

    ratio = float(desired_size)/max(old_size)
    new_size = tuple([int(x*ratio) for x in old_size])

    # new_size should be in (width, height) format
    img = cv2.resize(img, (new_size[1], new_size[0]))

    delta_w = size[1] - new_size[1]
    delta_h = size[0] - new_size[0]

    if pad_type=="corner":
        top, left = 0, 0
        bottom, right = delta_h, delta_w
    else:
        delta_w = size[1] - new_size[1]
        delta_h = size[0] - new_size[0]
        top, bottom = delta_h//2, delta_h-(delta_h//2)
        left, right = delta_w//2, delta_w-(delta_w//2)


    img = cv2.copyMakeBorder(img, top, bottom, left, right, cv2.BORDER_CONSTANT,
        value=pad_color)
    
    # Step 4
    # Apply scaling and mean subtraction.
    # if your model is built with an input
    # normalization layer, then you might
    # need to skip this
    if mean is not None and scale is not None:
        img = img.astype('float32')
        for mean, scale, ch in zip(mean, scale, range(img.shape[2])):
            img[:,:,ch] = ((img.astype('float32')[:,:,ch] - mean) * scale)
            
    # Step 5
    if reverse_channels:
        img = img[:,:,::-1]
    
    # Step 6
    img = np.expand_dims(img,axis=0)
    img = np.transpose(img, (0, 3, 1, 2))
    
    return img, top, left, ratio


# In[3]:


tidl_tools_path = '/opt/edgeai-tidl-tools'
output_dir = '/opt/edgeai-gst-apps/yoga_pose_judge/model-artifacts/onnx/yolox_s_pose_ti_lite_49p5_78p0.onnx'
onnx_model_path_TDA4VM = '/opt/edgeai-gst-apps/yoga_pose_judge/model/yolox_s_pose_ti_lite_49p5_78p0.onnx'
onnx.shape_inference.infer_shapes_path(onnx_model_path_TDA4VM, onnx_model_path_TDA4VM)
    
# model compilation options
compile_options = {
    'tidl_tools_path' : tidl_tools_path,
    'artifacts_folder' : output_dir,
    'tensor_bits' : 8, #8,
    'accuracy_level' : 1,
    'object_detection:meta_arch_type': 6,
    'object_detection:meta_layers_names_list': f'/opt/edgeai-gst-apps/yoga_pose_judge/modelyolox_s_pose_ti_lite_metaarch.prototxt',
}


# In[4]:


label = 'ONR-KD-7060-human-pose-yolox-s-640x640'
pad_color = 128 if 'ae' in label and 'yolo' not in label else 114
pad_type = "corner" if 'yolox' in label else "center"
size = 640
mean = [0.0, 0.0, 0.0]
scale = [1.0, 1.0, 1.0]
layout = 0
reverse_channels = True


# In[5]:


so = rt.SessionOptions()


# In[6]:


EP_list = ['TIDLExecutionProvider','CPUExecutionProvider']


# In[7]:


sess = rt.InferenceSession(onnx_model_path_TDA4VM ,providers=EP_list, provider_options=[compile_options, {}], sess_options=so)


# In[8]:


input_details = sess.get_inputs()


# In[9]:


capture = cv2.VideoCapture(2)
time.sleep(0.1)
(success, reference) = capture.read()

cv2.imwrite('/opt/edgeai-gst-apps/yoga_pose_judge/captured_images/live_image.jpg',reference)

capture.release()


# In[10]:


im = Image.open(r"/opt/edgeai-gst-apps/yoga_pose_judge/captured_images/live_image.jpg")
width, height = im.size

left = (width/2) - (height/2)
right = (width/2) + (height/2)
top = height - height
bottom = height

im1 = im.crop((left, top, right, bottom))
newsize = (640, 640)
im1 = im1.resize(newsize)
im1.save("/opt/edgeai-gst-apps/yoga_pose_judge/captured_images/live_image_resized.jpg")


# In[13]:


#image_name = '/opt/edgeai-gst-apps/yoga_pose_judge/side_bend_pose/image-640x640_28.png'
image_name = '/opt/edgeai-gst-apps/yoga_pose_judge/captured_images/live_image_resized.jpg'
processed_image, top, left, ratio = preprocess_for_onnx_pose_estimation(image_name, size, mean, scale, layout, reverse_channels, pad_color, pad_type)

if not input_details[0].type == 'tensor(float)':
    processed_image = np.uint8(processed_image)

image_size = processed_image.shape[3]    
output = list(sess.run(None, {input_details[0].name : processed_image}))[0]


# In[14]:


# post processing 
out_file=None
get_ipython().run_line_magic('matplotlib', 'inline')
#print(output)
output_image = single_img_visualise(output, image_size, image_name, out_file, top, left, ratio, udp=True, thickness=2, radius=5, label=label)

# plot the outut using matplotlib
plt.rcParams["figure.figsize"]=20,20
plt.rcParams['figure.dpi'] = 200 # 200 e.g. is really fine, but slower
plt.imshow(output_image)
plt.show()


# In[ ]: