What is Time of Flight markerless motion capture? What sensor do we prefer, and where to get it?
We told the Qinematic story about sensors for recording, measuring and visualising whole-body human movement (kinematics) in a previous blog post. Read the post if you want a taste of what it takes to make a great product that does what it claims to do, without the ‘bling’ and the ‘buzz’ factors outweighing the purpose - to get an accurate picture of how people move, so we can help them move well, and thereby move more.
Qinematic does this by using affordable, off-the-shelf Time of Flight (TOF) sensors, and we choose them very carefully so that the raw signal is optimal, and for a great price. FYI - we have rejected many other sensors due to inferior performance. We started with Prime Sense, then moved to the XBox Kinect, then the Kinect Azure and now the Orbbec Femto Bolt. They just get better, and smaller with time. They well and truly outperform all single 2D camera (RGB) systems, including our own Moovment.Pro mobile app (available on App Store and Google Play).
Video: The depth cloud is created from IR light reflected from the body. This provides adequate data to construct a 3D image of the person. The image is created 30 times per second. The Qinematic tracking algorithms can then identify key body parts for analysis, measurement and visualisation.
Time of Flight (ToF) sensors are devices that measure the distance between an object and a sensor by calculating the time it takes for a signal to travel from the sensor to the object and back. ToF sensors can use different types of signals, such as light, sound, or radio waves, depending on the application and the environment. ToF sensors have many advantages over other types of distance sensors, such as high accuracy, fast response, low power consumption, and wide field of view.
The Femto Bolt from Orbbec is based on the Kinect Azure sensor, and is now Qinematic’s preferred sensor.
One of the most well-known applications of ToF sensors is the Kinect sensor, a motion controller peripheral for Xbox video game consoles. The Kinect sensor was first introduced by Microsoft in 2010 as a revolutionary device that could track the movements and gestures of players without requiring any physical controllers. The Kinect sensor used a ToF camera that projected infrared light onto the scene and captured the reflected light with an infrared detector. By measuring the phase difference between the emitted and received light, the ToF camera could create a depth map of the scene, which could then be used to recognize human bodies and their poses.
The first-generation Kinect sensor was based on technology from an Israeli company called PrimeSense, which was acquired by Apple in 2013. The PrimeSense technology used a structured light approach, which means that it projected a pattern of infrared dots onto the scene and analyzed the distortion of the pattern caused by the depth of the objects. This approach had some limitations, such as sensitivity to ambient light, interference from other devices, and low resolution.
Prime Sense was our first sensor for prototyping at Qinematic.
Generation 1 Kinect, based on the Prime Sense sensor.
Image from: Merrouche, F et al (2020) Fall detection based on shape deformation, November 2020, Multimedia Tools and Applications 79(3).
The second-generation Kinect sensor was launched in 2013 along with the Xbox One console. This version used a different ToF technology that was developed by Microsoft itself. The new ToF camera used a modulated infrared light source that emitted pulses of light at a high frequency. The ToF camera then measured the time delay between the emission and detection of each pulse, which was proportional to the distance of the object. This approach improved the accuracy, resolution, and robustness of the depth sensing, and also enabled new features such as heart rate detection and facial expression recognition.
Generation 2 Kinect - much improved, and chosen by Qinematic for its amazing performance for a great price.
The Kinect sensor was not only a popular accessory for Xbox gamers, but also a powerful tool for researchers, developers, and hobbyists who wanted to explore new possibilities with computer vision and human-computer interaction. The Kinect sensor was widely used for applications such as 3D scanning, gesture recognition, augmented reality, robotics, education, art, and entertainment. Microsoft also released software development kits (SDKs) for Windows and Azure platforms that allowed developers to create custom applications using the Kinect sensor. The body tracking SDK was fine for games but not accurate enough for Qinematic, so created our own finely-tuned skeletal tracking algorithms. Some other companies seemingly used the SDK tracking, and to avoid the joints jumping around, simply smoothed the data to make it look nice, albeit inaccurate.
However, despite its technological achievements and its impact on various fields, the Kinect sensor did not achieve commercial success as a gaming device. The majority of the games developed for Kinect were casual and family-oriented titles that did not appeal to the core Xbox gamers. Microsoft eventually removed the requirement for Kinect to be connected to Xbox One in 2014, and stopped bundling it with new consoles in 2016. In late 2017, Microsoft announced that it would discontinue production of both versions of Kinect sensors. The timing was terrible for Qinematic - we had just released our commercial version of the software. It was lab-tested as 97% accurate compared to a 6-camera marker-based system, and we had successfully achieved medical device status. Fortunately, there were around 50m sensors in the market after, so supply was possible for some years.
Nevertheless, Microsoft continued to invest in ToF technology and its applications. In 2019, Microsoft introduced Azure Kinect sensor to enable cloud-based artificial intelligence solutions using Azure services such as computer vision, speech, and cognitive service. Azure Kinect sensor was aimed at enterprise customers who wanted to create innovative solutions for healthcare, retail, manufacturing, education, and other industries. Qinematic did just that, introducing new voice and gesture recognition features for health and fitness providers.
The Kinect Azure sensor, with improved performance and robust design.
In 2023, Microsoft announced that it would also end production of the Azure Kinect sensor. However, this was not the end of Microsoft's ToF technology, as it would continue to be available through its partner ecosystem. Microsoft now works with Orbbec, a 3D depth camera manufacturer, which launched cameras based on Microsoft's ToF depth technology. Orbbec is now Qinematic’s preferred provider of sensors to customers who subscribe to the Moovment suite of software and services. For as little as $395 + postage, anyone can buy an impressive TOF sensor for human movement analysis using Moovment SaaS.
The Femto Bolt is very compact and stylish sensor from Orbbec, with the same technical excellence as the Kinect Azure sensor from Microsoft (discontinued).
In conclusion, ToF sensors are amazing devices that can measure distance with high precision and speed. The Kinect sensor was one of the most prominent examples of how ToF sensors can enable natural and intuitive interaction with computers and games. Although Microsoft stopped producing its own ToF products, it still supported its partners and customers who wanted to use its ToF technology for various applications and industries.