You finish your workout, look at your pulse and find a number that is very reassuring in its precision: 427 calories burned. Four hundred and twenty-seven, not “more or less a little”. The problem is that that figure may have a lot more imagination than it lets on. A new study published in PLOS One compared four popular smartwatches with a laboratory measurement and found consistent errors in estimating energy expenditure. With a further complication: the error increased as the percentage of body fat increased.
The authors from Florida International University tested the Apple Watch Series 8, Garmin Forerunner 955, Samsung Galaxy Watch 5 and Fitbit Sense 2. Apple showed the smallest average deviation; Garmin and Samsung systematically overestimated calories. Fitbit returned much more erratic results, including some values so extreme that they had to be treated separately in the analysis. In short, the small number on the display remains useful as an indication. As accounting to the cent of our metabolism, decidedly less.
Four smartwatches against a machine that measures breathing
The test involved 58 Hispanic adults between 18 and 50 years old, with an average age of 23 years: 31 women and 27 men. Twenty-five participants had a body mass index of 30 or higher. They all did a short session on a recumbent bike: five minutes of rest, ten minutes of exercise alternating between moderate and vigorous intervals, then another five minutes of recovery.
The different devices were on the wrists at the same time. The reference energy consumption was measured by a COSMED K5, a metabolic analyzer that detects oxygen consumed and carbon dioxide produced through breathing. It is indirect calorimetry: decidedly less practical to take to the supermarket, much more suitable for understanding how much energy the body is really using.
Smartwatches have to achieve the same result by taking a much longer ride. The optical sensor under the case uses photoplethysmography to estimate heart rate; accelerometers and other sensors record movement; the software then combines these signals with personal information such as age, gender, height and weight. The end result goes through proprietary algorithms that researchers, of course, can’t open to see what’s going on inside.
And some gears, at least during this test, creaked.
Apple makes fewer mistakes, Garmin and Samsung tend to add calories
After quality checks, 52 measurements were usable for Apple, 51 for Garmin, 50 for Samsung, and 44 for Fitbit. The average deviation from calorimetry was approximately 21.6 excess calories for Apple, 68.6 for Garmin and 56.8 for Samsung. In all three cases the average overestimation was statistically significant, according to the results published by the researchers.
On such a short protocol, almost 69 calories donated by the wrist are not exactly metabolic pocket change.
Fitbit deserves its own parenthesis. By eliminating the results considered implausible, the average difference dropped to just 3.1 calories and was not statistically significant. Too bad that seven measurements, about 13% of Fitbit tests, exceeded the value detected by the laboratory system by 450%. One session produced no estimates. Leaving the outliers in, the average bias rose to 128.6 calories. An apparently excellent average, therefore, hid a dispersion that was anything but reassuring.
Looking at the absolute percentage error, i.e. how much the estimate moved away from the reference value regardless of the direction of the deviation, the medians of the four devices were roughly between 15 and 25%. None of the four, during this test, therefore transformed the wrist into a metabolic laboratory.
The higher the body fat, the worse the estimates
The most interesting result, however, concerns who wore the watch. In statistical models, higher body fat percentage was associated with larger errors across all four brands, although the increase varied from device to device.
The authors point to several possible explanations, without being able to establish which is responsible for the result. Fatty tissue and tissue movement around the wrist can affect the quality of the optical signal; the algorithms may also have been developed on samples that represent certain body types less well. However, their structure is proprietary, so attributing the error to a specific cause would be going beyond the available data.
The study also looked for a possible effect of skin color. He did not find a statistically robust one, but here caution must be attached to the result: all the participants belonged to the Fitzpatrick phototypes III, IV or V and only four were classified in type V. The researchers themselves warn that the sample does not allow us to close the question.
It doesn’t mean the smartwatch is useless
The study measures a very precise situation: a single session, on a recumbent exercise bike, in the laboratory and on a young and specific sample. We don’t know if the same errors appear while running, walking, lifting weights, or throughout the day. Furthermore, the training mode of the watches remained active even during the initial five minutes of rest and the final five minutes of recovery, an element indicated among the limits by the authors themselves. The results cannot therefore be automatically transferred to every smartwatch, activity or person.
However, there is a rather cumbersome precedent. Already in 2017 a study published on Journal of Personalized Medicine and coordinated by researchers from Stanford University had tested seven wrist devices on 60 people: heart rate was generally detected quite well, while the estimate of energy expenditure was much weaker. Even Stanford, presenting the results, underlined that even the best device then showed an average error of 27%. Models and algorithms have changed quite a bit since then. The calorie problem, evidently, did not evaporate along with the older generations of smartwatches.
Jason Kostrna, lead author of the new work, summarized the practical consequence quite simply: that number should not be treated as an exact measurement. Florida International University also draws attention above all to the risk of using it to decide how much to eat or to construct a calorie deficit with mathematical precision.
The smartwatch can continue to tell if we moved more or less, follow the progress of training and return useful personal trends. The authors themselves recognize that an error on the absolute value does not exclude the possibility of following relative changes over time. Those 427 calories, however, deserve a few decimals less in our confidence. The display is precise. Metabolism, unfortunately for those who love round accounts, continues to be much less cooperative.
