Jury told that Meta, Google 'engineered addiction' at landmark US trial
techxplore.com305 points by geox 4 hours ago
305 points by geox 4 hours ago
They're not afraid of the idea of programming people.
When I worked there every week there would be a different flyer on the inside of the bathroom stall door to try to get the word out about things that really mattered to the company.
One week the flyer was about how a feed video needed to hook the user in the first 0.2 seconds. The flyer promised that if this was done, the result would in essence have a scientifically measurable addictive effect, a brain-hack. The flyer was to try to make sure this message reached as many advertisers as possible.
It seemed to me quite clear at that moment that the users were prey. The company didn't even care what was being sold to their users with this brain-reprogramming-style tactic. Our goal was to sell the advertisers on the fact that we were scientifically sure that we had the tools to reprogram our users brains.
Another way of describing this - they find people lose interest almost immediately, and so if you want to actually show a consumer something new, you have to get to the point with your ad.
I'm not sure that's a fair characterization of a policy that promotes ads that hook the user within the first 200ms.
200ms isn't enough time for significant information to be transmitted to a person and for them to process it. You don't 'get to the point' in 200ms.
That means that the way to the user's brain and attention is with some irritating little jingle, a picture of a bunny beating a drum, cartoon bears wiping their asses with toilet paper, a picture of a caveman salesman or a picture of an absolutely artifical thing that looks like food but isn't. Stuff that stands out as unnatural.
But that isn't enough. You gotta pair it with spaced reeitition. Let them think about this every time they take a shit in the office. Hammer them with the same shrill sounds and garish images on every commercial break. Or after every couple of songs they're trying to listen to on youtube. Or in institials that are algorithmically optimized to pop up in their feed as they mindlessly scroll looking for gossip about their neigbhours to scratch that social group animal itch in all of us.
Exactly, 200ms is rather different than 'get to the point.' Here is a 'reaction speed test' site: https://reactiontimetest.net/ for somebody who doesn't intuit what 200ms is like.
You will likely be unable to click the screen in response to a box turning green faster than 200ms. To hook somebody on something within 200ms is largely appealing to casino like stuff where every single jingle, color, flash of light, and other aspect of their games is carefully researched in order to maximize addiction on a subconscious level.
...which has been known for at least a century
Here is an important difference. A century ago, the predator (seller) and the prey (buyer) were on equal evolutionary terms. Each generation of humans on either side of the transaction came into the world, learned to convince, learned to resist, then passed, and some balance was maintained. In this century, corporations and algorithms don't die, but the targets do. This means that the non-human seller is continuously, even immortally, learning, adapting and perfecting how to manipulate. The target, be it adult, adolescent, or child, is, and will be ever increasingly, at a severe disadvantage.
Ah yes because trade secrets were never a thing at any of these companies. The companies always shut down when it's founding members died wiping out all the knowledge it had built up.
That is to say organizations have always had this edge on individuals.
Right, because we know that parents never pass down useful skills or life tips to their children, like skepticism of propaganda and advertising, and instead send their children into the world like sheep into a lion's den.
There might come a day when advertising is too flawless for a human mind to resist it, but we're not there yet.
Most of everybody thinks their behaviors and decisions are not meaningfully influenced by advertising. Companies spend literally trillions of dollars running ads. One side is right, one side is wrong.
And advertising largely relies on this ignorance of its effects, or otherwise most of everybody would go to much greater lengths to limit their exposures to such, and governments would be more inclined to regulate the ad industry as a goal in and of itself.
Advertising is just companies saying "This is what you can purchase from me - it's awesome - please consider purchasing it". I have managed hundreds of millions in ad spend for major brands. None of them rely on weird ad magic to persuade people secretly - just showing off different aspects of the product or service.
And only recently could be optimized in real time, individually, for each target. I remember when there was a big moral scare about "subliminal advertising". People were appalled that an ad on TV could manipulate you without your awareness. That is 100% the business model of modern social media advertising.
It's not embedded in a specific ad, but the entire operation of the promotion algorithms.
Users as prey is a terrifying but not unrealistic narrative. Thanks for sharing.
The only business that call customers "users" are software development and drug dealers :)
Facebook employees may be the easiest prey to program
If something as crude as flyers in bathroom stalls is effective
FB uses its addict money to pay those employees. I assume the pay is what’s effective. Actually a good business model. Pay employees to improve how addictive your drug is, get more money from addicts, and use that to pay your employees more money, completing the loop.
But then drugs being profitable isn’t really news
It also says a lot if that's the most effective way vs normal ways of disseminating the info.
I haven't worked at FAANG so maybe I'm out the loop, but flyers on bathroom stalls seems bizarre, like almost less of a corporate action and more of a personal one (like you might get for unionisation), but with all the messaging of corporate, like something you'd see in a company memo.
Like I say, maybe everyone else is accustomed to this idea, but if you have any pictures of them I think a lot of people would be interested in seeing it, unless I'm misunderstand what it is
It started as Testing on the Toilet, which was an effort to get people to actually care about unit-testing their code and software quality and writing maintainable code that doesn't break in 6 months. Later was expanded to Learning on the Loo, general tips and tricks, and then Testing on the Toilet became Tech on the Toilet. It's been going on for a good 20 years now, so that's about 1000 articles (they change them out weekly) and there aren't really 1000 articles you can write about unit testing.
The insight is actually pretty similar to Google's core business model: when you're going to the bathroom, there isn't a whole lot else you're doing, so it's the perfect time to put up a 2-3 minute read to reinforce a message that you want people to hear but might not get attention for otherwise.
It's not really a FAANG thing. I bet you've seen the memes about X days without a serious accident, or without stopping the production line. It's the equivalent in a restroom or a urinal: A place you can make sure people see key information. You can find this in many industrial sites. A call center might have reminders of core principles for how to close calls quickly, or when to escalate. A lab might have safety tips. A restaurant will remind you of hand washing. An industrial site of some important safety tip or two.
While I've not seen this in every single place I've worked, it's very common.
You're right that it was just other employees who decided what to print there. But I don't think that absolves the company (Facebook) really... Everything a company does is just things that its people do! Nothing about the flyer was outside the parameters of the job of its maker. Their job was to make the company money by helping advertisers maximize ad revenue, and that's exactly what they were doing.
Facebook had a serious internal propaganda arm when I was there. Couldn't manage to get floor length stall walls in most of the bathrooms, but every stall had a weekly newsletter about whatever product stuff.
Every high traffic flat space on the wall would be covered with a poster, most of them with designs lifted from US WWII propaganda, many hard to tell if satire or not. I was surprised there was never one about carpooling with der füher.
I can say at Google we usually just had engineering tip posters in the washrooms they were usually very insightful and just written by other engineers at the company.
Stuff like how to reduce nesting logic, how to restructure APIs for better testing, etc.
People usually like them. I can't say I've seen what the parent post described so I imagine it's "the other" FAANG mentioned here.
Yep, I frankly thought Testing On The Toilet was pretty great.
That and nice washing toilets.
Alright. I may object to the wording, but ... isn't what you described also a good website? I am aware of how much propaganda Google uses too, e. g. "engage the user" - you see that on youtube "leave a like". They are begging people to vote. Not for the vote, but to engage him. I saw this not long ago on Magic Arena by Wizards of the coast. They claim "your feedback is valuable" but you can only vote up or vote down. That's not feedback - that is lying to the user to try to get the user to make a reaction and tell others about it. I just don't really see the difference. You describe it that they manipulate people, but ANY ad-department of a company uses propaganda and manipulates people. Look in a grocery, how many colours are used in the packaging. Isn't ALL of this also manipulative?
Google doesn't beg you for likes. Channels beg you for likes because it's one of the metrics they are stack–ranked by. Someone will lose, and they don't want it to be them.
The public got a peek at it with Cambridge Analytica creating hundreds of thousands of personality profiles, they then used to create Trump's MAGA army of flying monkeys. The Democrats could have done something about it, and made it illegal, but instead they just decided to build there own armies of flying monkeys. Why? Because both sides are bought and paid for by the same rich parasites trying to reprogram us.
did democrats create flying monkey armies? I haven't seen anywhere near as much Democrat propaganda as Republican, which is probably why they keep losing. Only recently, once Republican policies came into effect and people experienced their consequences, did Republican votes decline.
To me this is simply a consequence of the capitalist mode of production.
Yes, because governments are so restrained in their use of propaganda.
What it is is the consequence of the power existing. 200 years ago nobody was arguing about how to hook people in the first 0.2 seconds of video, but it's not because nobody would have refused the power it represents if offered. They just couldn't have it. It's humans. People want this power over you. All of them.
To be fair, it is basically one and the same. I doubt most people railing against capitalism are actually against private property. They probably dislike corporatism which only exists as an extension of the government. Very very few of us voluntarily gave up our right to hold people personally responsible for their actions, but this is forced on everyone on behalf of business interests. The corporate vale is materialized from government alone.
> I doubt most people railing against capitalism are actually against private property. They probably dislike corporatism which only exists as an extension of the government.
I really don't know. In my experience, it can about private property when talking about housing, it is about markets when talking salaries and work conditions, and it's just about having no idea of what capitalism even is and just vaguely pointing at economics the vast majority of the time.
"Capitalism" can be safely replaced with "the illuminati" or "Chem trails" in the vast majority of complaints I hear and read and the message would ultimately make as much sense. There's not a lot of how or why capitalism doesn't work, but by God there sure is a lot of what it seemingly does wrong.
> All of them.
At least an unhealthy amount of them. I have no desire to have power over people, except I would like it if my kids actually listened to me...
Well adjusted people so not want that power over other people
It's sociopaths and narcissists which want it.
And as Atlas667 pointed out, it's also a direct consequence from a capitalistic world view, where it has replaced your morals.
This is not in relationship to state propaganda. Multiple things can cause abhorrent behavior, and just because we've identified something as problematic doesn't inherently imply that other unrelated examples are any better.
"Well adjusted people so not want that power over other people"
There are certainly well adjusted people that would like to fix things they feel are inefficiencies or issues in their government, especially when those issues are directly related to their areas of expertise. Thinking well adjusted people wouldn't want to be in a position of power is exactly how you ensure that only bad people end up with power.
Power seekers acquire power, not knowledge seekers. This is from Plato’s The Republic so about as old as it gets.
We've always had sociopaths and narcissists, and if you're looking to "capitalism" as the reason why they exist, you're in out-and-out category error territory, not-even-wrong territory. Now that this power exists to be had, human beings are racing to acquire it. If you think you can fix that by "fixing capitalism" you are completely wasting your efforts.
So if that’s not the answer, what is? Should we just throw our hands in the air and say that technology has defeated our monkey brains, and there’s no going back?
Given that these tendencies are not evenly distributed throughout the population, you can have structures that leverage the large mean to mitigate the worst tendencies of the extreme tails. Given that the natural state of things is that power begets more power, these are harder to build and maintain, but it can be done. In particular, Democracies and Republics are major historical examples of this.
Is your name Epson? Because you’re really good at projecting.
Your comment speaks droves about you, not humanity.
History contains abundant, well-documented cases of ordinary people participating in atrocities without coercion. Most people will act decently in low-pressure environments and will act badly under certain incentives, authority structures, or group dynamics. There is no way to know what a person's threshold is until it's tested, but it can be assumed that most people have a low threshold.
Parent and was implying “all” humans crave this power over others. This is patently false.
“Most” people won’t act badly to attain this power, “some” will. Being placed into a position and choosing harm is not the same as pursuing it.
That is absolutely against the evidence, but yes people do like to think they are naturally righteous and good.
That may be true but I think the unspoken assumption in your comment is that somehow, without capitalism, greed magically melts away. How do you explain the constant extreme rampant corruption in communist and socialist countries over 100 years if not from GREED?
I know that it doesn't. Greed will be ever-present, yes, but that doesn't mean that it's a one-way ratchet. It's something we have to keep fighting against all the time. Greed starts out as a driver of progress, then eventually becomes an impediment to progress. The other constant there is progress! No dam will block a river forever.
Capitalism or consumerism, a never ending offer and demand for goods, material or immaterial?
Large portions of the tech sector thrive off the attention economy. If your goal as a product is to have someone spend hours a day everyday engaged with your product, and you focus on a data driven approach to maximize the time spent on the app, then you’ll create something not dissimilar to addiction.
Supposedly the people working for these companies are "the brightest of the bright" but if they didn't even notice that this was what they were contributing to, what kind of intelligence is even that? Not everyone working there could possibly be so socially inept that they didn't realize what they helped building right? Or are we chalking it down to just missing morals? I feel like I'm missing something here to properly understand why people ended up working for these companies in the first place, even before it started making the news.
Its basically impossible for them to not notice. I know someone who is a software engineer for lockheed. He told me that back in the 90s he wrote a bunch of software for a missile. He wasn't told that is what he was working on, it was all classified, and part of that is you only know what you need to know. But from the specifications and how the math worked, it was very clear to him that it was a surface to air missile. After the fact, it was confirmed that is what he was working on.
Google and Meta are surely more open than a classified missile project. So it would really be beyond the pale for someone to not realize that what they are working on is an additive platform, sure I am willing to bet they didn't say "Addictive" and instead cleaned it up in tidy corporate product management lingo, "highly engaged users" or something like that. But its just impossible.
It’s simpler than that. Engagement increases are the perpetual goal. The vector is constant. Relative motion is all that matters.
Nobody would talk about whether the product is now “addictive” because that suggests crossing a finish line to completion, and we can’t ever be done.
It is interesting that Software Engineering as it's practitioners like to call it, is unregulated.
If you want to be an accountant, lawyer, surveyor et cetera, one has to learn about ethics, and violating ones professional institute's code of ethics may result in you being unable to practice in future.
Professional engineers are required to consider the interests of the public in their work, have an obligation to reject unethical or harmful instructions and are regulated by their professional organization to support competency and address malpractice. Much of this was driven over the past 50-100 years as society determined that they wanted things built by engineers to not kill people or have material deficiencies following construction.
From my understanding, software engineers are a long away out from this still but perhaps we'll get there once the dust settles on more of these sorts of lawsuits.
The dust will never settle because once people try to regulate they can basically move software engineering in its whole somewhere else. Something great about being active in multiple places is the fact that these companies have leverage. There's not just a cost advantage to having amazon in luxembourg, just employ a few thousands (10 000 jobs are linked to amazon in luxembourg) and you can block votes in europe (because of veto power). 10K jobs is nothing for amazon but is 2% of all jobs in luxembourg.
Same way amazon being big in india isn't just great because of the vast talent pool and 'low' costs in India (even if many if most indian programmers are subpar, they got over a billion people), they basically ensure that the government in India can never turn against Amazon, because these jobs are concentrated in a specific region and India isn't a unified state. Amazon can try many getting into many different things in India without having the risk associated some small foreign company breaking into India would have.
> basically move software engineering in its whole somewhere else.
You don't think that is true in other professions? You don't think I could get my accounts done in India, or a bridge designed in China? The regulatory environment in my country would still apply. Your answer is just exceptionalism
There's no need to have software engineering be regulated. It'd be a restriction/deterrent at the wrong level.
In order to fix this we need the individuals in charge to be held legally accountable without hiding behind a corporation.
In the software industry management rarely ever listens to concerns brought up by engineering even if it's technical concerns.
It's extremely embarrassing that my (American) employer refers to me as a "software engineer" when in fact I dropped out of the university computer engineer program and can not legally call myself an engineer in my country.
I would just as soon call myself a software doctor or software lawyer. Or software architect.
We don't even need formal regulation to start — just honest internal conversation. I work in tech and most teams I've been part of never once discussed the ethical implications of what we were building. Not because people are evil, but because the incentive structure doesn't reward asking "should we?" — only "can we ship it?"
The gap isn't education, it's accountability. Engineers building engagement loops know exactly what they're doing. They just don't have a professional body that can revoke their license for it.
Your comment is saying two very different things?
> We don't even need formal regulation to start — just honest internal conversation
> They just don't have a professional body that can revoke their license for it.
What internal conversations could lead to a professional body that can revoke anyone's license? I'm sorry, but your comment doesn't make much sense.
Edit: Dammit, I realize now I think I fell for vibebait, leaving for posterity so others don't fall into the same trap.
Fair point — I contradicted myself. What I meant is: the first step doesn't require waiting for regulation (just have the conversation). But long-term, some form of professional accountability would help. Those are two different timescales, not alternatives. I wrote it badly.
And no, not vibebait — just a poorly structured comment from a guy with a fever typing on his phone.
I am amazed that I’ve never considered this before
Have been surveying Computer Science courses at university with my son recently. All the ones we looked at had a compulsory ethics module which shows the direction things are headed at least.
I wonder how many programmers working today are coming through universities though? I'm self-taught, most of my programmers friends are as well, same with most of my colleagues back when I worked. I can remember maybe the name of 3-4 people in total, out of maybe ~30 or so, who went to university for computer science before they started working.
In my experience CompSci ethics modules are about hacking or mishandling user data or code theft... i.e. things that companies don't want their employees doing.
I've yet to see an ethics module that covers ethics from the perspective of ethics over profit.
Whereas an accountant is taught that they should resign rather than get involved in unethical practices, like profit manipulation for example. I interview people with ethics questions. I discussed them frequently when training.
I refused the pressure to be unethical when I was pushed, even when I knew I would be fired (which I was). I was able to discuss it with old mentors, who made time to meet with me, even when I hadn't worked at their company for years.
Lastly I disclosed why I was fired at interview for a new job (without the confidential details), and was hired partly on the strength of it by a person who had been through much the same.
And I didn't learn it at University, I learnt it on my professional qualification, that was around 3 years long and was postgraduate level, although had non-degree based entry routes for technicians. It also required a wide range of supervised experience.
This was not at all the ethics program that was taught in my university computing ethics course. They did indeed cover the societal and moral responsibility of software developers. This was way back in 2002.
Mine had one over a decade ago. After graduating, the industry decided that developing everything we just got done establishing was unethical, was the hot topic to innovate for the next decade. I never worked at any of those places and still got burned ethically in much more indirectly unethical product streams in the finance and insurance sectors. To be honest, if there is really good money to be made at this point, there is a safe bet that if you dig deep enough, there is an unethical core to it. Most of my peers assuaged themselves with some variant of "I'm a programmer, not an ethicist, and philosophy doesn't put food on my table. So sadly, the problem seems much more systemic and a priori to the capitalistic optimization function.
This is a comment that my reaction is different based on your age. If you're older, I'd be more disappointed. If you're young, I'd be more sympathetic. However, the careers mentioned by GP all require schooling where those ethics courses can be taught. In "Software Engineering", so many people are self taught or taken boot camps without formal schooling. The SE title is just a joke to me knowing that it is so overused and given to people that clearly are not trained as an engineer.
Maybe we should have Gavin Belson's Tethics be more widely taught???
Whereas accountants, lawyers, civil engineers and surveyors have to do postgraduate training with their institute to become chartered.
Interestingly many accountants in the UK never did a degree (very many more did a degree in something unrelated), but came through the technician route of evening, weekend or day release study. Many do their chartered training at weekends.
We have separate words for intelligence and wisdom for a reason.
Intelligence is not particularly correlated to ethics or morality. Probably sounds obvious when I say it directly, but it is clearly something that you have banging around in the back of your mind. Bring it forward out of the morass of unexamined beliefs so you can see that it is clearly wrong, and update the rest of your beliefs that are implicitly based on the idea that intelligence somehow leads to some particular morality as appropriate.
Because nobody is clocking in and willfully contributing to the addiction machine. They're completing an 8-point ticket to integrate a new scroll-tracking library, or a 5-point ticket to send an extra parameter to the logging system. When there's thousands of people working on a product, nobody feels like they're doing anything impactful.
> Because nobody is clocking in and willfully contributing to the addiction machine.
Are people really not aware of what the company's overall mission, product and impact is? I'm finding that hard to believe. If you accept employment at Facebook, regardless of what department you're in, you know exactly what kind of company you're contributing your time, energy and effort into.
I joined Google Analytics in 2018 and had no idea that Analytics really meant "Tracking and Remarketing" until about 3 weeks into the role. At that point, what're you going to do, quit? I knew it wasn't what I wanted to do, but it took two years to get out cleanly.
> At that point, what're you going to do, quit?
Yes? Why not? If I'd join a company and figured out what I did actually harmed more than helped, I'd leave that place, absolutely. I'm a software engineer, even with the lowest possible position in a random company I'd earn better than most people in the country and live a better life, even just the bottom 30% of earners in software in the country (not counting outsourcing obviously). Especially at that time it was very easy to find new jobs.
Good for you. I've got a family and no other source of income.
You think Google is the single company out there who is willing to employ you? How come?
Edit: Thinking about it, your comment actually made me more frustrated than I realized. I've been poor enough to having to be homeless at some points in my life, and yes, I've worked for immoral companies too, because I need food and shelter. But once you move up in life to the comfy jobs like software engineering, you don't have any excuse anymore that it's just about "feeding your family" when you literally have a sea of jobs available. It might be an excuse you tell yourself to justify your reasoning for getting paid more, but if you truly did care about it, you'd make a different choice, and still be able to feed your family, and I'm almost confident your family would be OK with you making that choice too, unless they also lack the empathy you seem to be missing.
You were homeless and didn't have a choice, so now obviously you're qualified to give assurances that essentially, "it is unlikely that your family will starve", right? /s
And if you're wrong, and shit hits the fan for whatever reason, who's going to fix that? You? No, he's going to have to fix that, because nobody else is going to step in.
It's easy to tell others that it's going to be OK, but put your money where your mouth is. Put $1M in a fund that he can access should he no longer be able to find employment. Then he'll have absolute certainty that it's going to be OK.
Something tells me you're not going to do that. Something tells me that what you would do if shit hits the fan, is you're going to tell him that he should find solace in the fact that while he's working for 1/5th of his former total comp, putting in more hours at the same time, seeing his kids less, not putting his kids through private school to give them the best chance at the best education, that, at least, some kid out there isn't watching 6-7 videos on the tablet that their parents use to do less parenting.