Practical Pose Estimation on the Go

Arm pose tracking has a wide range of practical applications in a variety of industries. In healthcare, this technology can be leveraged for physical therapy exercises, enabling real-time monitoring and correction of patients' movements to assist in a speedy rehabilitation. Additionally, in virtual reality and gaming, arm pose tracking helps create an immersive user experience by allowing players to control and interact with virtual environments through natural hand gestures, enhancing the overall gameplay. Moreover, in industrial settings, the precise tracking of arm poses can optimize manufacturing processes and ensure the safety and efficiency of manual tasks.

Although it has the potential to be useful, arm pose tracking is not readily available to most consumers due to a number of limitations associated with existing systems. These systems typically rely on cameras that are not practical for portable applications, which limits the widespread adoption of this technology in everyday consumer products. The need for high-resolution cameras with complex processing algorithms makes it difficult to integrate arm pose tracking into devices that require mobility, such as smartphones or wearable gadgets.

Another option is to use arm pose tracking systems that rely on a large number of inertial measurement units (IMUs) or markers that must be attached to the user's body. This results in a cumbersome and impractical setup for regular use. The need for external attachments or markers can limit the ease of use and convenience, making it difficult for consumers to integrate such systems seamlessly into their daily lives.

To unlock the potential of arm pose tracking in everyday life, less cumbersome options are needed. One possible solution may be on the horizon thanks to the work of a trio of researchers at Carnegie Mellon University. They have developed a technique that they call SmartPoser that requires only the off-the-shelf smartphone and smartwatch that many people already own. By leveraging sensors in these devices, the team has demonstrated how the positions of arm joints can be located in three-dimensional space with reasonably good accuracy. And importantly, the system is highly portable and practical for use on the go.

The researchers’ novel approach used the ultra-wideband (UWB) sensor that is being included in an increasing number of portable smart devices. This sensor makes it possible to precisely measure the distance between two devices — in this case, a smartphone and a smartwatch. This information is fused with data from inertial measurement units (IMUs) in both devices to provide orientation and acceleration metrics.

The absolute measurements from the UWB sensor, in conjunction with the relative measurements from the IMUs provide the raw data that is needed to determine the pose of the arm that the smartwatch is strapped to. These data sources are fed into the SmartPoser software, which leverages a recurrent neural network to predict the three-dimensional joint locations of the wrist, elbow, and shoulder. While this algorithm primarily ran on a laptop to facilitate development and debugging, it was also deployed to an Apple iPhone 12 Pro, where inference times were observed to be less than one millisecond, proving that SmartPoser is capable of fully mobile operation.

A small user study, consisting of 10 participants, was conducted to assess the system’s performance. After a short calibration process, the participants were instructed to perform a number of typical, daily activities. By comparing the SmartPoser predictions with ground-truth observations made with an Azure Kinect, it was discovered that the new method has a median error of just over 4 inches.

The researchers envision SmartPoser being used for gaming, safety training, context-aware user interfaces, and more in the future. To help move the ball forward, they have open-sourced their data and trained model and made them available for anyone to download.