PocketTrainer

You want to do sports?

But you are not sure on how to do your exercises correctly? That's exactly what PocketTrainer stands for!

During the ongoing pandemic, we didn't just want to sit on our couch. Exercising is essential and so it happened that we watched different videos on YouTube. Now, however, it wasn't always clear if that was a good execution. The gyms were closed and we didn't have our own personal trainer as students either - someone who helps us working out and provides essential feedback.

So what do you need?

All you need is a smartphone and the app PocketTrainer. With this app, you get real-time feedback on whether you're doing your exercise correctly.

What is the basic idea?

You choose an exercise from a trainer on your smartphone or tablet. While your chosen trainer is performing an exercise, you are filmed and a digital twin is created out of your body movements. Your digital twin is compared with the trainer and evaluated as to whether you are doing the exercise well.

Here you can see it as a short demovideo (it´s in german): https://videos.simpleshow.com/pDhre7IMCg

How the app home screen looks like:

Explanation video how to use the app:

First, choose an exercise from a range of different workouts. Now place your smartphone (or tablet) a few meters in front of you so that the camera can capture you well. The app guides you to the best position. The workout starts and you see the trainer perform a training pose. Try to imitate this pose. Here you should see on your smartphone that your movements are being imitated by your digital twin. If the pose is not quite right, you will get visual feedback and the video will pause. Try to correct your posture. As soon as your posture corresponds to the trainer, the video continues and you can learn the next pose.

In the second tab of the app (Video) you have the option to upload a video of yourself. Then the person from the uploaded video is superimposed over a previously selected video. This gives you the opportunity to compare yourself with your idol. For example, you want to learn a soccer trick from Ronaldo. Then you film yourself and put yourself over Ronaldo's video with the help of the app, and the differences are revealed. If there are any at all.

How does the app work from a technical point of view:

First, a picture of the user is taken every 100ms via the smartphone camera. This image is transferred to PoseNet. PoseNet is a pre-trained model of Tensorflow, which returns coordinates of the keypoints (as can be seen in the image above). For example, you get the coordinates of the right knee in the image. If all these coordinates are available, vectors are then formed, so that one receives, for example, the vector between the right knee and the right ankle. At the same time, every 100ms a picture is taken from the explanatory video, which is also transferred to PoseNet, and in this way the vectors of the trainer are also obtained. Finally, the vectors of the user and the trainer are compared via the angles.

Own CNN:

We have also developed our own CNN model for the detection of keypoints in a taken image. However, this is not used in the app yet because we have not been able to collect enough training images yet. The goal here is to train the model so that it is more suitable for fitness purposes than PoseNet. The CNN was developed in a Jupyter notebook and can be viewed via Git.