People are bad at reporting what they eat. That's a problem for dietary research

science.org

265 points by XzetaU8 a day ago


wnorris510 - 19 hours ago

I've done research in this space for many years at Google AI and now at SnapCalorie. The thing I find interesting is how confident people are in their ability to estimate portion size visually, and in truth how wrong they all are.

We published in CVPR (top peer reviewed academic conference for computer vision) and people are on average off by 53% and even trained professionals are still off by 40%. Basically if you want to have a higher level of accuracy you need to use a food scale or something that measure the volume of food, people just can't estimate portion sizes visually.

Oils, cooking fats, hidden ingredients are what people are most concerned about but they actually add far less error to people's tracking than portion. Nutrition5k is the paper we published if you want to check out more details on the breakdown of error most people get when tracking.

aziaziazi - 20 hours ago

For those that "track and weight everything" (how ?) do you manage ?:

- sauces you make yourself? I often mix some different oils, mustard, seeds, miso, bit of leamon juice and spices… but weighting and logging everything will take 3x the time to do the sauce itself

- different cooking time in one receive : oignons going first, tomato sauce in the middle and parsley at the end (but still cook a bit with residual heat)

- Leftovers nutrients decrease with time

- counting how much you take of a meal shared with others, especially when you serves yourself multiple time

- different species/cultivation methods like the rustic small and dense cucumber from your neighbor and the spongy one from the supermarket in January

I have the feeling that might have been easy at some point in my life when I lived alone and mostly eat packaged food and raw vegetable that looked like clones but not when I share my meal, cook a lot more raw un-barcoded aliments and gained confidence to dose "by the eye" without recipes.

everdrive - 20 hours ago

A lot of people seem to have a purely emotional relationship with resources which logic doesn't seem to be able to penetrate. Food and finances seem similar here. For years I tried to get my wife to stick to a grocery budget. That is, we have $n per week for all groceries. She'd blow badly over the limit every time. "But we needed [food]" or "These were toiletries, so they don't _count_ as groceries." Ultimately we never had an real success sticking to a grocery budget, and ultimately the solution was me working towards better paying jobs.

This feels a LOT like weight loss. Calories in --> calories out is conceptually very simple, but in practice more people struggle with it than not. It certainly cannot be the case that they struggle with the concept; they struggle with actually putting it into action. Lying to themselves, twisting themselves into philosophical knots, and probably most often, conceding to their cravings. Food acquisition is one of out more basic drives, so it should probably be no surprise that people struggle to intellectualize it.

(as and aside, there are also people who wrongly believe that calories in --> calories out is a flawed concept because not all people have the same metabolism, or not all calories are equal. Both of these are true, but none of them actually negate the premise. For whatever your metabolism, and whatever your category of calories, fewer calories will still produce weight loss. It may feel unfair that someone doesn't have to work as hard as you to produce the same result, but this is actually true in all areas of life. Now that said, improving the quality of your calories is very important, and should not be ignored -- but it also does not negate the premise.)

TypingOutBugs - 21 hours ago

I try calorie count with My Fitness Pal and holy shit it’s a lot of effort. Eat out and you’re screwed (estimated at best). When you include sauces and oils etc it’s really hard to be accurate in the best of times, and it’s just a pain to keep on top of. Best option is to avoid any so you don’t have to count.

I imagine almost everyone will add bad data in a study at some point with the best of intentions.

JoeAltmaier - 19 hours ago

People are bad at reporting ANYTHING. Exercise, food, sex, grooming. Just ask a lawyer or anybody trying to get a story out of somebody.

This should be a fundamental understanding of anybody asking people anything. That scientists imagine there's some accurately-reporting population of subjects for their experiment is an example of the breathtaking naivete of scientists.

lm28469 - 21 hours ago

> Is coffee good for you? What about wine or chocolate? Scientists trying to answer these questions

There is a virtually infinite amount of cofounding variables, genetics, meal timing, fitness level, sedentarity, &c. . It's a 80/20 type of problem, do the 80, forget about the 20, you'll never be able to get your answers anyways.

If you look and feel like shit you're most likely eating like shit. If you look and feel good a glass of wine every now and then or a bite of chocolate after dinner won't do much.

BoxFour - 21 hours ago

Does this actually pose an issue for most studies?

This seems like it would be an issue for any studies relying on absolute food consumption being accurate. Most studies I come across frame their findings in relative terms (likely for this very reason): Individuals who engage in more of X compared to their peers show a correlation with outcome Y.

For example, if you’re trying to determine whether morning coffee consumption correlates with longevity it doesn’t seem particularly relevant if you believe everyone is underreporting their food intake, as the article implies; it's a relative comparison.

Sure, those findings often get twisted into clickbait headlines like “X is the secret to a longer life!” but that’s more a popular science problem than an issue with dietary research itself.

damnesian - 20 hours ago

This is why sleep studies are conducted in clinics, not left to patients to self-report. they want accurate data? They will need to conduct a real study, portion the meals out themselves, give people a schedule.

cainxinth - 18 hours ago

In my experience, people are especially bad at understanding how calorific alcohol is. Carbs and protein are generally 4 calories per gram. Alcohol is 7 calories per gram. Only fat is more energy dense at 9 calories per gram.

I can recall in the aughts when there was a major low carb food trend and Bacardi had a popular ad campaign around the fact that their rum had no carbs, basically marketing it as the smarter option for people watching their weight -- even though all unflavored hard liquor has no carbs and is still incredibly calorific.

alexfromapex - 21 hours ago

I thought this was generally known that people are bad at reporting most things about themselves. It's a good argument in favor of wearables or other smart monitors, if anyone expects to do actual rigorous research it needs to be objective.

f1shy - 21 hours ago

Just a couple of days ago, I wrote I automatically flag any submission with any kind of "dietary" studies. I'm not saying there is no one study well done, but doing it well, is just (almost) impracticable. Not only the people have literally no idea what they eat, they forget and misreport, also a human living normal life in the society has just TOO MANY variables. There is no way to keep the other variables like, sport, social interaction, stress and such out of the study.

drchiu - 20 hours ago

Another way to look at it is that tracking what you eat is very difficult. Currently trying to lose a few pounds and doing calorie tracking. Practically carry a scale and a calorie tracking app with me. About once a day there's still some "estimation" involved due to the fact that all the ingredients are mixed together.

UomoNeroNero - 17 hours ago

Have you ever wondered why it’s such a struggle for a diabetic to manage blood sugar levels in a sensible way? Here’s the answer. I assure you that anyone with diabetes is forced (and the word "forced" doesn’t fully convey the mental burden involved) to maintain an almost obsessive level of awareness about what they eat. There’s no comparison to someone simply “on a diet.”

I guarantee you, it’s an incredibly complex task. Unless one adopts a monastic approach of always eating exactly the same carefully measured meals at home, the challenge is constant.

If one day a system based on vision and AI could accomplish this task (and it can't, it’s impossible), it could charge any price and have millions of users.

DannyBee - 20 hours ago

"> Is coffee good for you? What about wine or chocolate? Scientists trying to answer these questions"

These are dumb questions to ask in the first place, because the "you" and "good" here are too personal for any general answer to be useful to most people. Unfortunately, this is not just lazy writing that took complex questions and simplified them to the point of uselessness - we really are asking these kinds of questions :(

Most of this doesn't generalize to populations the size of the world in the way something like "physics" does, because, for starters, we aren't very deterministic or very homogeneous at large scale.

Instead, you end up with millions to tens of millions of people in a subgroup particularly affected or unaffected by something because of genetic variation, etc.

Any reasonable scientist knows this. Instead, the main reason to try to answer these questions framed like this seems to be either to get funding, or to make headlines.

Sometimes we can answer extreme versions of this question (IE it seems data suggests alcohol is fairly universally bad for almost any person, definition of bad, and amount), but that's pretty rare. This then gets used as a "success" to do more poorly designed and thought out studies.

Just because we want to know things doesn't mean we should use mechanisms that we know don't work and produce mostly useless results. This is true even when we don't have lots of mechanisms that do work or produce useful results.

It's much slower and much more expensive, but what we learn is at least more useful.

It's really hard, slow, and expensive to answer questions about particle physics - this doesn't mean we revert to asking atoms to self-report their energy levels and publishing headlines about how "larger atoms that move around more live longer" or whatever based on the results. Instead, we accept that it will hard, slow, and expensive, and therefore, we better get started if we want to ever get somewhere.

dredmorbius - 11 hours ago

Points of measurement are a challenging issue.

One of the ... beyond annoying ... aspects of our track-everything-individuals-do-and-utilise-it-against-them contemporary information ecology is that it is so painfully difficult to make use of that information for personal advantage.

In the specific case of food intake, it should be reasonably trivial to aggregate purchase information, at grocery stores, restaurants, and online deliveries, and at least arrive at a reasonable baseline of total consumption. Rather than having to fill out a food diary from memory with uncertain measurments, one can rely on grocery and menu receipts directly.

This is more useful for those who live alone or shop for themselves (a large fraction of the population, but far from complete). It's based on the general principle that you tend to eat what you buy. There's some error imposed by food acquired elsewhere (shared at work, school, from friends, etc.), and of tossed food, but what you'll arrive at is over time a pretty accurate record of intake.

I'm surprised that such methods aren't more widely used or reported in both dietary management and research.

My own personal experience has been that I've been most successful in dietary management when 1) I have direct control over shopping and 2) I focus far more on what I eat than how much, though some of the latter applies. If I'm aware that specific foods are deleterious to goals (highly-processed, junk foods, high-caloric / high-sugar liquids, etc), then the most effective control point at minimum decision cost is at the store. If you don't buy crisps, chips, biscuits, fizzy drinks, ice cream, and the like, it's not at the house for you to consume.

I'm well aware that there are circumstances in which this is difficult to arrange, sometimes with friends or roommates, more often with families. I'll only say that clearly expressing terms and boundaries is tremendously useful here.

dacox - 15 hours ago

It's an open secret that most nutrition research is of extremely low quality - almost all relying on decades old self reported nutritional questionnaires.

Sometimes dozens of these studies get wrapped up and analyzed together, and we headlines that THING IS BAD with a hazard ratio of like 1.05 (we figured out smoking was bad with a hazard ratio that was like 3! - you need a really good signal when you are analyzing such low quality data)

3abiton - 4 hours ago

This could be the reason why nutrition "science" is so fragmented. There is no way to ensure the process is being conducted correctly.

Havoc - 19 hours ago

It’s also just really hard unless you live off packaged meals or only eat thing that are isolated.

Something like a curry cooked in kitchen and shared among a family is a complete black box as to who got how many calories. Maybe one person got a different ratio of rice to curry. Or this family likes a sweeter type of curry etc

thefz - 21 hours ago

Everything needs to be weighted on a precise scale, every ingredient and not just the macros. On top of that the reported nutrition values on labels can be wrong by a large margin so for not whole foods, we introduce an error.

This is why calorie counting is ballparking to get us a general idea, and not a precise science.

jmount - 18 hours ago

I tried to use some dietary research as data examples in machine learning training courses. After running into self reporting (which I naively thought would be the exception, not the rule) I changed the my use of such sources to class discussions around reporting such as the following: https://www.vox.com/science-and-health/2018/9/19/17879102/br...

FredPret - 20 hours ago

We're far too generous with what we allow to be called "science".

There is no dietary research, because you can't pull off an unbiased dietary study over a meaningful period of time. Practical and ethical problems abound.

Maybe one day we can simulate n=10mm people from the neck down for a period of 30 years, and feed half of them bacon and half of them beans, but even that will have the major problems of being a simulation and that only from the neck down.

Read the original "fat = heart attacks" studies by Ancel Keys from the 1950's. I've done free online 5-minute long data science tutorials with more statistical rigor.

lt_snuffles - 19 hours ago

I feel bad for suggesting this, but what about using prison population for researching dietary science? Every single part of their life is controlled. As long as it's humane (stuff like coffee vs no coffee).

throw78311 - 19 hours ago

I've heard about using food tracking apps as a planner instead. Instead of logging what you ate, you add what you PLAN to eat for the day, and adjust accordingly to fulfill the nutrition requirements.

zahlman - 18 hours ago

While the article may have been published today, putting "self-reported food intake" into a search engine shows me that this is not at all a new finding. I would have considered it common knowledge, even. The entire reason people can conjure the mental image of a stereotypical obsessed dieter, weighing every morsel of food and looking it up in tables, is because everyone else has barely any idea what their intake looks like.

giantg2 - 20 hours ago

Seems like a decent use of AI if we could use something like Glass to scan the label and plate to estimate calories and stuff. You could even record those portions to be used for audit.

qingcharles - 14 hours ago

I remember when I was researching illegal medical experimentation on prisoners in the USA, I found a quote from a researcher saying "one of the reasons we prefer to use prisoners is because we know exactly what they do and what they eat every single day."

dennis_jeeves2 - 20 hours ago

There is no problem with dietary research. The 'problem' is by design.

Both people doing the research and people funding the research know very well that what the flaw of this approach is, but just chose to do the shoddy job that they do because it brings in money. If it's not by design then there is a worse conclusion - the researchers/funders are incompetent. It's most likely a mix of incompetence/corruption.

Fricken - 15 hours ago

The thing is we're all experts at eating food, we've all been doing it our whole lives. You'd think in that time one would have cultivated an intuition about whether they are eating too much or too little regardless of the nutrition information.

SoftTalker - 16 hours ago

Any research or studies based on self-reporting or interviews should be very suspect. Most people will answer or report what they think the researcher wants to hear, or what they think will make them "look good" even if the responses are anonymous.

pards - 21 hours ago

> many studies of nutritional epidemiology that try to link dietary exposures to disease outcomes are founded on really dodgy data

I wonder if the data are always skewed in a particular direction. For example, do people typically underreport junk food and overreport salads? Or do they omit entire meals? Or snacks?

arionhardison - 13 hours ago

I had some success here by gamifying the reporting process and breaking it into sessions of compliance .. "just do this for ever".

jfbaro - 20 hours ago

I was wondering if there’s a way to automatically measure calorie intake—like some kind of biosensor that could be worn on the body. Companies are investigating this I bet!

pluto_modadic - 14 hours ago

This reminds me, out of the one app I *actually want* to be addictive, food tracking apps NEVER are.

amelius - 20 hours ago

Can't this be solved with camera plus AI? I'd be surprised if some startup isn't already working on it.

Refusing23 - 20 hours ago

thats a problem for research that relies on food questionaires and thats been known for a while, and probably why there's even a thing called 'the french paradox' and so on.

but i get it

it's expensive to do properly, and so its not really done that often, and when it happens there's usually only a few participants.

worthless-trash - 5 hours ago

My wife takes pictures of -everything- she eats, I'm sure she's not alone in this behavior. I can't eat until the photo is taken, and its a good one.

They just need to tap these people.

Its a real problem when I'm hungry.

agos - 19 hours ago

and let's not forget the garbage data you find in the database used by all the calorie counting apps, which make it a chore and a challenge even when you weigh everything

Mountain_Skies - 20 hours ago

The benefits of the Minnesota Starvation Study were that both food intake and physical activity could be accurately tracked. If we had a draft and there were conscientious objectors, would similar studies be possible as alternative service? I suspect that our ethical concerns now are greater than they were back then, so maybe it wouldn't be possible to conduct.

Iulioh - 21 hours ago

I'm basically tracking anything that i eat with...too much precision

I always wondered if i could volunteer for this types of studies somehow

idontwantthis - 16 hours ago

My successful weight loss approach is to give up on counting calories and focus entirely on my own weight.

I set a goal weight for the week, and if I weigh more than that, I don’t eat (or eat just vegetables). I’ve lost 5 kg so far and I maintained the weight loss between thanksgiving and new year’s despite spikes on the holidays.

It’s taught me what proper portions feel like so I don’t have a desire to rebound as soon as I take a break.

shadowtree - 19 hours ago

As someone who just lost 35 pounds through clean dieting:

All calorie/portion numbers on packaged food are off by 10-20%. I set MyFitnessTracker to 1.5k calories (deficit for my build) and for weeks nothing would budge - even with strict portion control and weighing everything, plus 800 extra cals spent through exercise.

Once I went to "1250" calories, I started losing weight. Went from to 205 to 175 pounds.

With packaged food I mean anything like cream cheese, various sides, etc. - not pre-built meals (I assume those would be off by 50%).

What weighing your food really does, is reveal how shockingly little you actually should be eating. I switched to small plates for all meals, as using the normal large ones was pointless and slightly de-motivating.

But yeah, it's just calories. No matter what you eat.

paul7986 - 18 hours ago

Have used chatGPT for about a year to count my daily calorie in-take .

Since I eat out daily at fairly healthy places (Cava, Panera, chic fil a only grilled nuggets & fruit cups, Jersey Mikes number 7 mini, noodles & company, MOD pizza) GPT knows their menu & each items calories. Upon getting the food I just tell GPT what I'm about to eat each time and it counts & retains and calculates through the day.

In doing so as adult male (late 40s) 5'10 175 my body has gotten used to eating 1500 to 2k calories a day. Do weigh myself daily to ensure I'm not gaining as I do have a cheat day once a week.

I understand the sodium content is higher then if I cooked at home but I'm focused on maintaining a fit look & counting calories along with a few weekly gym visits I think keeps me as I seek.

FollowingTheDao - 19 hours ago

The bigger problem with dietary research is that it does not take human diversity and genetics into account.

See : Genetics, Nutrition, and Health: A New Frontier in Disease Prevention

https://www.tandfonline.com/doi/abs/10.1080/27697061.2023.22...

fortran77 - 20 hours ago

There's have been several studies, well researched and cited, where people who claim to be "diet resistant" are given metabolic markers "double labeled water" that will accurately show caloric intake.

For example:

https://www.nejm.org/doi/full/10.1056/NEJM199212313272701

and

https://journals.physiology.org/doi/full/10.1152/ajpendo.200...

In the NEJM article they note that every single person who claimed to be "diet resistant" was lying about food intake.

> The main finding of this study is that failure to lose weight despite a self-reported low caloric intake can be explained by substantial misreporting of food intake and physical activity. The underreporting of food intake by the subjects in group 1 even occurred 24 hours after a test meal eaten under standardized conditions. In contrast, values for total energy expenditure, resting metabolic rate, thermic effect of food, and thermic response to exercise were comparable with those of obese subjects in group 2 who did not report a history of diet resistance.

and

> In addition to their greater degree of misreporting, the subjects in group 1 used thyroid medication more often, had a stronger belief that their obesity was caused by genetic and metabolic factors and not by overeating, and reported less hunger and disinhibition and more cognitive restraint than did the subjects in group 2. Subjects presenting for weight-control therapy who had these findings in association with a history of self-reported diet resistance would clearly convey the impression that a low metabolic rate caused their obesity.

Calories-in/Calories-out is true for everyone, and everyone can lose weight by putting down his fork.

varispeed - 20 hours ago

Ideally we should find where in the brain is the calorie counter and just expose its value through an endpoint and have an app to call it.

guerrilla - 21 hours ago

I just weight and scan erything. The only problem is eating out. Mobile apps make this very easy today. They should be using them and scales that automatically report, with photo documentation, etc. Skip self-reporting and go straight to self-measuring.