Towards passive heart health monitoring via smartphone camera

Heart rate (HR), one of the cardinal vital signs, is a dynamic indicator of physiological status, influenced by everything from activity, to stress, to acute and chronic illness. Further, resting heart rate (RHR) is a key biomarker of cardiovascular health and long-term health risk. A higher RHR and increases in RHR over time are associated with major adverse cardiovascular events and all-cause mortality.

Wearables, such as Fitbit devices and the Pixel Watch, have made it possible to track these health markers throughout our daily lives. However, there is room to improve their adoption, especially in low-resource environments and among those most at risk for cardiovascular disease. Smartphones present a unique opportunity to broaden access to health tracking — today, around five billion people already own a device with powerful sensors capable of monitoring their health. In 2022, we demonstrated using smartphones for on-demand HR measurement via a finger placed over the camera, and subsequent Google research considered how the signal detected during that measurement could help predict cardiovascular disease.

A technological breakthrough designed for everyone

Like wearables, pulse oximeters, and our previous work, PHRM measures HR via photoplethysmography (PPG), i.e., by sensing the fluctuation in how light interacts with the skin each time blood pulses through it. We developed an on-device software pipeline that processes 8-second facial video clips and uses computationally-efficient temporal shift convolutional neural networks to predict HR along with a confidence score. The pipeline further aggregates HR predictions over the day and leverages confidence scores and Kalman filtering to estimate a daily RHR.