Google tests phone photos as a screen for insulin resistance

Why it matters

PhotoScan could make richer metabolic-risk screening cheaper than clinical imaging, but the evidence comes from controlled research cohorts rather than real-world consumer use.

Google Research (@GoogleResearch) on August 17th highlighted PhotoScan, a deep learning method that estimates body composition from smartphone images and uses those estimates to help classify insulin resistance. Google's central claim is unusually specific: in a clinical research cohort, a model using PhotoScan data performed nearly as well as one supplied with measurements from a dual-energy X-ray absorptiometry scanner, or DXA. The announcement on X arrived months after the underlying research was first submitted on March 27th and revised on April 6th.

The comparison needs careful parsing. PhotoScan did not diagnose insulin resistance directly from a casual photograph, and the phone camera did not reproduce every output of a DXA scan. The researchers first estimated three body-composition measures: total body-fat percentage, the ratio of abdominal to hip-area fat, and the ratio of visceral to subcutaneous fat. A separate classifier then combined those estimates with demographic inputs to predict insulin-resistance status.

In the research paper, the model using age, sex and body mass index produced an area under the receiver operating characteristic curve, or AUROC, of 0.692. Adding PhotoScan's estimates raised that result to 0.760. Replacing the photo-derived measurements with actual DXA data produced an AUROC of 0.773. The small gap underpins Google's comparison with DXA, while the 0.760 score still leaves substantial room for classification error. (arxiv.org)

What PhotoScan actually sees

PhotoScan uses frontal and side views of a participant along with sex, height and weight. That last detail matters because the system is not drawing its conclusions from image pixels alone. BMI is calculated from the supplied height and weight, then combined with features extracted by the image model.

Google's researchers built PhotoScan in two stages. They pretrained a ResNet-50 neural network using data from 35,323 UK Biobank participants. The training process converted three-dimensional MRI data into paired two-dimensional projections, with corresponding DXA measurements serving as reference values. The researchers then fine-tuned the model on a separate cohort of 677 people who had smartphone imagery and DXA data. (arxiv.org)

In that fine-tuning cohort, PhotoScan estimated body-fat percentage with a mean absolute error of 2.15 percentage points. A model based on demographic information alone had an error of 3.44 points, according to the paper. PhotoScan also estimated fat-distribution measures that BMI cannot represent, including the balance between abdominal and lower-body fat.

The researchers tested the frozen model on the independent MetabolicMosaic cohort of 132 participants. Body-fat percentage error remained similar at 2.13 points. That consistency supports the case that PhotoScan learned features that transferred beyond its fine-tuning sample. The independent cohort remained modest, particularly for the downstream insulin-resistance test. (arxiv.org)

The DXA comparison has a narrow meaning

The insulin-resistance analysis included 195 visit-level data sets: 47 classified as insulin resistant and 148 classified as insulin sensitive or having impaired sensitivity. The researchers used HOMA-IR, a calculation based on fasting insulin and glucose, as the reference label. They separated records from the same participant during cross-validation to reduce leakage between training and evaluation sets.

PhotoScan's strongest useful signal was the estimated ratio of abdominal to hip-area fat. That measure showed a stronger and more consistent relationship with insulin resistance than total body-fat percentage in the study. The result matches the research team's broader thesis: where fat is stored may reveal metabolic risk that a weight-and-height calculation misses.

The study design still differs from the way a consumer would take photographs at home. The fine-tuning process used selected frontal and lateral frames extracted from participant videos, with controlled poses and camera placement. The paper describes imaging with minimal clothing so the model can observe body shape and fat distribution. Performance from casually framed photographs, loose clothing, varied lighting or different cameras was not established by the reported results.

Google also presented PhotoScan as research rather than a consumer or clinical product. The paper supports testing in larger populations and deployment conditions; it does not establish that a phone photograph can replace a scan, blood test or clinician's assessment for an individual patient.

The strategic direction is clear. Google has been building methods that turn common consumer sensors into health-research instruments, including work using wearables and routine blood biomarkers to predict insulin resistance and camera-based heart-rate measurement. PhotoScan extends that approach from physiological signals to body shape. (research.google)

A phone-based body-composition screen would be cheaper and easier to distribute than medical imaging. It would also require collecting highly sensitive body photographs and biometric inferences at scale. Any path from research study to product will depend on whether the model holds up outside controlled imaging sessions and whether Google can make that data collection acceptable to patients, clinicians and regulators.