School cell phone bans and student achievement
nber.org75 points by harias 7 hours ago
75 points by harias 7 hours ago
Back in the late 90s, when I first entered the video game industry to work (when it was quite scruffy, countercultural, and populated by some pretty odd people), one of the first things I encountered was a new co-worker who, next to his giant tower of used Mountain Dew cans, had a black and white TV in his cubicle. This struck me as very odd at that moment in time - as I understood things, obviously the point of work was supposed to be that it was a place where you worked, not a place where you watched TV. (Now, granted, everyone else was playing the recently released Diablo on their work PCs during lunch in network mode, and we were a game studio after all, so my reaction wasn't totally coherent). Still, no one else had a TV, and that guy was young and single with no work-life balance, he was a recent transplant, and it still seemed unusual at the time.
Fast forward 28 years later, and now everyone has an amazing TV in their pocket at all times when they commute, sit in their work space, go out for coffee or lunch, or go sit down in the bathroom, all with a near infinite collection of video via youtube, netflix, and even massive amounts of porn. How little did I know. And that's to say nothing of texting and twitter and reddit and instant messaging and discord and ...
Several years ago, I was working on a college campus, and there were giant corporate-flavored murals beside some of the city blocks students walked, full of happy multicultural clip art people and exciting innovative technological innovation, and adorned with the message, "Imagine a borderless world!" Clearly that message was meant to be rhetorical, not a call to reflection, critique, or reevaluation. There did not seem to be the suggestion that one might imagine the borderless world and then, having done so, decide it was a problem to be corrected.
I wonder a lot, these days, if we're not deep into a Chesterton's Fence situation, where we have to rediscover the hard way the older wisdom about having separate spheres with separate hard constraints and boundaries on behaviors, communities, and communication pathways to facilitate all sorts of important activities that simply don't happen otherwise - something like borders and boundaries as a crucial social technology, specifically about directing attention productively. Phones and tablets are, in their own Turing complete way, portals to a borderless world that pierces the older intentional classroom boundaries.
For certain tasks for me, having a movie running while I'm working is more productive. It gives something to take your attention when you have to wait for something without getting sucked in to endless scrolling.
In my first job out of university in the 80s, I spent all one night playing Knight Lore on the Spectrum with friends. I failed to get up the next morning. My boss drove across Leeds and to bang on the door to see if I was alright. I needed that job so I stopped playing computer games.
In the 90s a later boss called me out for spending my days attached to the Slashdot firehose. I had sort-of known that it was a wasteful time sink, so I resolved to completely stop using the social media of its time, and have avoided most incarnations of it ever since (but here I am).
As a scouter working with teenagers, I feel that most kids with a supportive backgrounds will tame this beast for themselves eventually, so I hate to make hard "no phones" rules. I would rather they come to terms with this addiction for themselves. I know that some simply won't finish school without strong guidance, but delaying exposure to this might just be worse in the long term.
As someone who graduated from high school in 2025 I completely agree with this. I am glad I had to work it out on my own, and I don't think this is a place that a school should take control. If I had to figure this out along with the stress of college, I don't know if I would be able to handle it. I also think that it has helped with my overall time management skills and prioritizing my time.
I know not everyone will have the same experience as me, but I just feel like learning to manage it on my own was overall beneficial for me in the end.
You could even argue that society is incapable of not running into these cycles of building wisdom and losing it. Our minds are differential.. things that are here have less value, we seek newness no matter what.
The older wisdom was that you worked on the farm with your husband and children for your entire life, breastfeeding while you peeled the potatoes, putting down your spindle to comfort a crying child. Millers lived in the mill; even blacksmiths lived at their smithies. Except for rituals, separate spheres with separate hard constraints was a novelty of the Satanic mills where the Victorian proletariat toiled.
They still had clear boundaries. They slept in the sleeping place and at the sleeping time, they worked at the working place and at the working time. See, they didn't have smartphones to fiddle with in bed.
> they didn't have smartphones to fiddle with in bed
This is solvable for people who want to. We have a dedicated charging station in our house for all electronic devices. Before bed, all of those devices get put there. Including me and my wife's phones.
This definitely is the way to do it. I have started keeping my phone in my living room at night instead of my bedroom, but am still bad about doing this every night. Phones are addictive and it is mentally hard to break out of the addiction. It is essentially a "you just have to do it" situation, but "just do it", while technically simple, is still difficult if you're addicted.
https://www.ou.org/holidays/the_thirty_nine_categories_of_sa... outlines those "clear boundaries" in detail from a pre-medieval or early medieval perspective.
I had an early experience with a Palm III and a cell modem strapped to it. It was intoxicating. I still find the pull of the phone to be very strong sometimes. It's an ongoing battle to maintain a healthy relationship with it. Such a useful tool, but also a massive time suck if you let it.
> rediscover the hard way the older wisdom about having separate spheres with separate hard constraints and boundaries on behaviors
This is something I also believe. Thanks for saying it.
I've been thinking and reflecting a lot on what I've been calling for myself "generative constraint". It's sure as heck not something that is the same for everyone, but I think we all have a set of them that might help us be our best person.
We've universalized constraints and expansivenesses in a way that seems really poor judgement. And yes, there is a capitalist critique in this too, as any good theory should have :)
I think of it as "introducing friction". There's a lot things that we do now which is largely as a result of frictionless ease of doing it. Smartphones and social media are the obvious one, but it applies to many technology/digital driven behaviour (pay with face id/touch and people end up consuming more for instance). And it's no surprise to me that what works for a lot of people is putting their phone somewhere else in their house. Essentially introducing artificial friction.
My slightly cynical view is for many of us we're more often lazy than not and default to doing the most frictionless thing. Introduce friction and very quickly I find it forces you to think about what you're actually doing
I am also older and I see that my kids don't have certain things that I perceived as disadvantages at the time but may have helped develop useful habits. These things include quiet and boredom, which helped with focus; lack of ready answers or information, which may have helped imagination or generative reasoning.
I think we can recreate these things if and when we need to, but that recreation may be for the elites. I heard an interview with a professor who said he had to reintroduce Socratic exams to get around chat bots and the fact that kids now have very poor handwriting. At an elite school you can do that.
Your Phone Isn't a Drug. It's a Portal to the Otherworld. https://news.ycombinator.com/item?id=46115659
Article about smartphones being bad? Right to the top.
Generic comment that would fit in the comment section of any of those articles? Right to the top.
I get baited into reading these posts and comments every day - why can't I stop? Probably for the same reason these posts and comments get up votes.
Some very important context that the researchers don't mention: during the same period that they are claiming test scores improved because of phone bans, Florida changed the way they administer standardised tests. Starting in 2024, they switched from doing one end-of-year assessment and started administering more frequent tests throughout the year in order to better gauge a student's progress and provide a tighter feedback loop. (source: https://www.educationadvanced.com/blog/florida-standardized-...)
It's much more likely that simply changing the way they administer these tests had a more significant impact on test scores than phone bans.
> It's much more likely that simply changing the way they administer these tests had a more significant impact on test scores than phone bans.
Why do you think that's more likely?
Put yourself in the student's shoes: instead of being required to rote memorise every detail and hold that in your head until the end of the year, you are now only required to be assessed at the time that you are learning the material. Do you think you'd fare better on that type of test, or a test done months after you actually studied the material?
One of the first things they teach you in educational research is that standardised test scores are significantly impacted based on how the tests are administered and what the test is actually assessing.
A good student would do well regardless, a bad student would do bad regardless. Cell phones might help a bad student do a little less bad, but only a little.
For the middle, it really depends on the material covered. if it's cumulative, then results might not change as much. if it's "learn and forget", then it might be testing the wrong incentives.
> if it's "learn and forget", then it might be testing the wrong incentives
The thing I find interesting is that when most people talk about standardised tests, they are talking about assessments that benchmark how much trivial knowledge about a given subject one has, and this has been the standard for most of the history of the American education system. I would argue that this is a flawed way to measure a student's literacy–in any subject for that matter.
I would actually frame "learn and forget" as "learn and adapt" because I would much rather a student forget a piece of trivial knowledge, but still have the ability to figure it out on their own with the right resources than a student who can tell you the colour Benjamin Franklin wore on his 15th birthday, but couldn't explain the effects of imperialism on societies.
For much of history, we have incentivised rote memorisation of trivial knowledge and accidentally de-valued critical thinking and problem solving skills. Do you remember the backlash that schools got from _parents_ when schools started implementing Common Core in an attempt to get students to think more abstractly? While I scoff at them, I genuinely don't blame parents for coming to the conclusion that we should just do math "the way we used to do it", but I can't help but point out that this is leading to the exact decline in general literacy that we have seen in public schools over the years. Now when you start comparing the educational attainment of students in public schools vs. private schools this becomes a who other conversation that cell phones can't even begin to explain.
I'm not particularly old yet, in my mid-thirties, but I reacted like someone much older when I learned kids are allowed to carry around their phones all day at school.
Back in my day (when we walked to school uphill both ways), we weren't allowed to carry around basic flip phones. They had to be in our locker and only used before or after school.
When and why did it become acceptable for much more distracting and stimulating devices to be allowed in class?
I was a classroom teacher from 1994-2019, so I watched the transition through the advent of phones until just before Covid. It's not as simple as it seems, for a few reasons.
One, there's the very real pressure from parents to be able to contact their kids when they need to. In the US, regular school shootings have made this a complicated issue to navigate.
Also, it requires much more consistency from school staff than most people realize. If it's top down and not supported by just about everyone, then many teachers and staff find themselves in endless battles. It takes more consistency and clarity of vision, and consistent enforcement than many schools are capable of.
Last, the devices students carry with them are often more capable and reliable than school based technology. So when students need to look something up, it's easy for them to just pull out their device.
Super-addictive devices in a society that's prioritizing many of the wrong things is a hard thing to manage. How many of you would give up your tech salaries to make $40-60k to take on these issues?
When parents themselves also became addicted and decided it was easier to give their kids phones than to parent them.
I think it is precisely because they are more distracting. When the most addictive thing in phones was the snake game, kids did not bother to insist in using their phones all the time. Now, when you try to tell a pupil to put the phone away, it often results in a huge arguments, so eventually teachers gave up.
no seizing of phones, no detention/disciplary action? It's not even about the phones at that point, it's just general disrespect to staff. What changed overtime?
Or maybe it was always this way and I simply had a better environment?
In the 90's only drug dealers had pagers and cell phones, at least in the eyes of the board of education. If you were caught with one you'd be expelled.
Yeah I'm not that far out of high school but my school in the late 00's had a library policy on phones. You can keep them in your pocket, but don't bring them out during school. Otherwise they get taken for the class time, and it escalates from there.
This included recess and pretty much extended to all non-calculator electronic devices, but it was generally more lax when you weren't disrupting someone. I couldn't imagine brazenly having my phone out while a teacher was talking unless it was an emergency.
We weren't allowed to have any of several different individual devices the functions of which are present in a smartphone. Banning that stuff was more-or-less uncontroversial. Obviously kids in an ordinary classroom shouldn't have instant cameras, and video recorders, and audio recorders, and Walkmen, and radios, and game boys, and TVs, and flashlights, and...
Now we have devices that are all of those things in one and parents will fight you if you try to keep kids from having or using them. Go figure.
What's baffling is why so many more people started thinking all those devices were OK when they're combined into one device. Like, not much of this is novel, we could have had devices that did most of the relevant things a smartphone does, in class. But we didn't because of fucking course they weren't permitted.
I really don't understand why the parents would fight for them. My theoretical kid is there to learn, enforce any reasonable rules that can disrupt that goal.
It's also in general a good way to form work habits for future aspects. Be it college, a job, military, etc. You can't fight over having your phone out to your boss. You can do it to your professor, but that's your $20k/yr tuition talking.
>You can't fight over having your phone out to your boss
Give me a company phone or you don't get this rule. I'm not using my phone for work if I can't have it out during work.
I use it 99% for work related things during work, though, with the 1% being happy birthday texts or something similar
This is kind of a weird example to begin with on a forum mostly populated by software engineers, because I'd find it very weird if a manager ever objected to someone using their personal phone at a SWE or similar office job, but I'd guess that the sets of jobs where a "boss" would object to someone using their phone during work (but still getting their work done) and those that would potentially have a company phone are mostly disjoint... a complete prohibition on using your phone seems like entry-level retail job type rules. excepting corner-case stuff like some very high security facility where you wouldn't even be allowed to bring any outside electronics in.
Parents really like the convenience and the feeling of safety they get when their kid has a phone. If they have to change school pickup plans they'd much rather text their kid than call and leave a message with the school office and then hope that the office gets the message to their kid.
We're so used to being able to get in touch with our family members at all times that it feels really unnerving when your kid isn't immediately accessible.
And the parents who complain think that their kids aren't the ones who are addicted to their phones.
That's why these bans needed to happen at the state or school district level - expecting individual teachers to have to spend their time arguing with parents and kids over cell phones was just not realistic.
It became "acceptable" because the teachers and admin were already on their phones constantly. I went to grade school from 2005-2017, when iPhones came around the adults got them years before kids did, I had numerous teachers that would sit on their phones half the class.
Never. There has never been a time when it was OK to use a phone in class. What happened is A) Some kids do take their phone out and play with them and either get caught or not B) Something happens and kids record it aka school fight videos. C) giant moral panic that has very little basis in reality.
I think it's less about what's okay and more about enforcement. It does seem like post pandemic schools lost all their teeth.
Having grown up in the "no cellphones allowed at school" age, and now having a kid, I'm super glad that my local school district is finally banning phones.
There's always going to be exceptions but speaking for myself there's no way I'd be able to resist the allure of a cellphone in class.
I wonder if there's a hidden confounding selection bias here, i.e. "ability of the school to ban phones". This is probably easier in less chaotic schools where the students listen to the teachers, say.
In our local schools, they don't make *teachers* responsible for enforcing the bans. Students have to keep their phones in a Yondr pouch. If they're caught with their phone it will be confiscated (and require a parent to pick it up), and the administration will give also give additional consequences such as being banned from extracurriculars or school activities like Prom.
My student tells me that in practice many students don't keep their phone in the pouch, but they are very careful about how and when they use them. Many teachers have a "don't ask, don't tell" policy - if I don't see you using the phone, and it's not disruptive, then they don't care.
If the teachers and schools cannot implement a phone ban because the students won't listen to them, it might be time to reassess what their purpose is.
Ever watch The Wire? The school in that show is a very accurate depiction of what a... middling-bad inner city school is like. There are tons of ones worse than it (they may have shied away from one of that sort either because it didn't suit the story, or because they thought too many viewers would think they were exaggerating)
There are schools where the administrators are too busy dealing with violence to have time for much else.
These sorts of schools already make kids pass through metal detectors on their way in so phones can just be confiscated at that point.
A lot of schools implement this by having the students put their phones in a locked pouch when they enter the building in the morning. https://www.overyondr.com/phone-locking-pouch
This generally takes it out of the hands of individual classroom teachers.
Two years after the imposition of a student cell phone ban, student test scores in a large urban school district were significantly higher than before, David N. Figlio and Umut Özek find in The Impact of Cell Phone Bans in Schools on Student Outcomes: Evidence from Florida (NBER Working Paper 34388).
Are students really allowed to be on their phones during class at a lot of schools?
When I was in high school, we didn't have smartphones, but we had game boys, flip phones, and graphing calculators that could play games.
If we were ever caught playing with any of these things we got in trouble. That seemed sufficient at the time, but is that not the case anymore?
iPads are required in some public school classrooms, in place of textbooks! There no "lockdown mode" that the teacher can enable, to lock to the apps/websites related to the lesson. It's INSANE.
You have to admit that it's quite clever how they approximate phone use:
> Our identification strategy relies upon our ability to calculate school-specific measures of smartphone activity that we can attribute to students, rather than adults in the building. To do so, we use detailed smartphone activity data from Advan between January 2023 and December 2024 that we link to LUSD schools using point-of-interest coordinates.13 In particular, we focus on the average number of unique smartphone visits (pings) between 9am and 1pm on school days (a common time frame that elementary, middle, and high schools in LUSD are all in session during school days) in the last two months of the 2022-23 school year (right before the ban took effect) and the first two months of the 2023-24 and 2024-25 school years.14 To disentangle student activity from the smartphone activity of teachers/staff, we subtract the average number of unique smartphone visits between 9am and 1pm on teacher workdays (in the same school year) from the same average on regular school days.
It's beyond obvious that they should be paying attention to the classroom and not their screen. Future generations will equate our screentime addictions to smoking and drinking. Just putting it down doesn't work. It needs to be out of reach and in certain locations, taken away from us entirely for the betterment of humanity.
Since when is a study needed to confirm that enabling a dopamine addiction, especially in developing minds, is a bad idea? Isn't our own direct experience as adults/parents struggling with said addictions enough?
Correlation/Causation issues as others have pointed out.
But also, complete inability of schools to adapt.
Refusing to adapt to the reality that is, students will be living their entire lives with these devices, and that they should be working out ways to ensure student productivity despite their existence, is not the same thing as success.
Teachers are inherently lazy. Its one of their more human qualities. But really they need to adapt, or fail and be replaced.
There was a kid in my class in highschool. We had a school that permitted laptops, one of the first near us, but situationally. Teachers could exclude, or instruct the student to not use the laptop for periods during class. However policy was that students were allowed to use the laptop any time they could use a workbook. This kid was the only one who both had access to a laptop and was willing to risk damaging it by bringing it to school.
Math class with this kid was:
1. He plays games on his laptop unless the teacher was looking, in which case he would be solving problems in excel or notepad. Proficient alt tab user.
2. At the end of class, he would copy out all the answers from the back of the textbook to his workbook, and he would hand that in.
English was different. In english the teacher built a relationship with him and engaged him directly. If he was unresponsive he might be forced back to attention somehow, asked a direct question about the text, but that was true of a lot of the students. The entire class was a discussion on book content. When he used the laptop, he was using it to write notes because he was engaged through positive reinforcement. If the teacher caught him playing, the teacher would on those rare occasions, engage him about the game. Often, he had finished his assigned reading\tasks early and simply drifted over. In that case he was left to play because he wasn't disrupting anyone.
Phone bans are a crutch for lazy, uninterested educators. Kids need to be prepared to live in a world with these things in their pockets. The correct dopamine reward feedback loops are not going to be built by banning them entirely. And being better at rote learning and regurgitating ancient course material isn't a strong indicator, if it was even an indicator, of better student outcomes.
This is an area where hacker news shows its weakness. We have:
1. A chart showing a very low increase (1-2 percent)
2. Nothing to control scores rising in every school in America in the last school year (due to reduction of COVID effects).
3. Scores not moving immediately after the ban, but only after the start of a new school year, which means a new cohort of students muddying the data.
Yet the data fits people's biases here (regardless whether it's right or wrong), so the celebrate it and add anecdotes and explanations why it's true.
There's no study that's good enough for HN.
I don't think I've ever seen a science or research article posted here that didn't immediately get picked apart for this or that in the comment section. The methodology is flawed. The data is flawed. The conclusions cannot be drawn. There are confounding variables not accounted for. The sources are questionable. It's become a trope at this point. Either our commenters' standards are way too high, or all of science reporting is deeply flawed.
No study is perfect – research is and has always been expensive, and playing devil's advocate while seeing the arc of promising research is one of the fundamental skills of reading and doing research.
No study should be good enough for HN. If a single non-obviously-flawed study is enough to convince you to something, then you can be convinced of anything and everything under the sun.
One study can find any effect it's looking for.
A study shouldn't move consensus. A study finding an effect is a signal that more studies should be done.
Once they are done, and people who know their stuff pour through them and reach some consensus is the sort of bar that needs to be crossed for a reasonable non-expert to 'follow the science'.
And sometimes those experts get it wrong, and accepting that degree of uncertainty is part of it.
Ok here is the crucial part of the paper:
It's a difference in differences design, using individual-level test scores and de-seasonalized data (p. 13). Their wording is:
> Y_igst is the outcome of interest for student i in grade g in school s in time period t, HighAct_s is an indicator for high pre-ban smartphone activity schools, D_t is a series of time period dummies (t = 0 indicates the first period after the ban took effect), δ_s is school fixed effects, and θ_g is grade fixed effects. In this setting, β_t are the parameters of interest, reflecting the difference in the outcome of interest between treatment and comparison schools for each period, with the period before the ban serving as the omitted category, holding grade level constant.
To me some modeling choices seem a bit heavy-handed, but I'm not an economist and could not do better.
what it means is that this paper shows probable causality and models a lot of interesting features. it is most definitely not flawed.
i think the tough thing is that 0.6 percentage points gain for the average student is quite small. it's actually less than you gain by studying for 1h for the SAT, which is probably about 0.9 percentage points, depending on how you interpret college board's research (it recommends 20h of studying). that is to say, if students studied one fucking hour for the FAST, they would probably get a bigger benefit on it than all the time they get back not looking at their phones throughout two years of school.
so whatever cell phone use (1) in school (2) causes, it causes a small effect on test scores.
you would have to pick some other objective criteria, for example mental health assessment, for maybe a larger effect, or seek a larger treatment, perhaps a complete ban of cell phones period, to observe a larger effect.
To be fair, all those details are in the paper. And a 1-2 percent increase does not seem low to me for such a measure.
There is another major factor that could cause scores to rise globally: ChatGPT. It is now good enough that it can explain topics to students at home, like a private tutor.
The lack of control and cohort following are legitimate criticisms. The effect size is not. Even a single-digit percentage increase over a single year from policy treatment is incredibly impressive when we open the door to cumulative effects.
> Yet the data fits people's biases
It does. But it also fits priors, particularly those we've seen documented when it comes to teens and social media.
I'd go further and say its a global weakness and unbelievably destructive. The bulk of current discourse today is:
1. Read a headline/tweet/instagram.
2. Decide whether or not it fits in your worldview.
3. Move forward with the confidence that you are better informed than everyone else who agrees/disagrees with it.
You see it everywhere on all sides of all beliefs.
It didn't use to be like this. We used to read articles, we used to read common news sources, we use to not have media overrun with bad actors who know exactly what to say to get the most engagement and solidify people in their own world views.
It's all over HN and I could have hoped there'd be more willingness to say "let me consider the contents and the source before deciding if I accept it". That attitude is just lost and I don't think it will be regained and I think it's the reason we are all in a death spiral.
When was it not like this, though? I think people are rosey about the past here. A small educated set was different in the past but probably the bulk of the population has always done something like this - now you can hear them online easier.
We didn't always have bad actors directly injecting rage-bait into our blood streams.
Sure we did. Or is that also just us conforming to our preferred worldview of the past?
Not as tuned for engagement as now, but we had to have yellow journalism laws for a reason too. There's always been lots of propaganda and manipulation and bad actors in journalism.
I don't think you are being totally fair to the paper, but you do point out something that drives me crazy: local freakouts about year-to-year changes in math tests scores. It's like people don't realize that these 8th grade students are not the same as those.
Yea, it's strange that the line didn't move quickly. I would give grace for a couple weeks to a few months, but next year? The timing feels really disconnected.
The increase in scores is really small. It’s 1.1 percentage points.
My interpretation is, the pandemic is a root cause of lower test scores for many reasons, one reason is that kids started using cell phones way more during the pandemic, and that new stuff on the phone (TikTok, let’s be real) causes lower test scores. Reducing usage during school is addressing a real problem, but it’s one of many real problems, and some are way bigger.
I think you're correct, but another piece of this is that students (especially in HS, but probably also in MS) have realized they can accomplish most assigned tasks in a fraction of the time using online resources (whether copies of old tests, agentic AI, or other sites), so they lean on their phone for this. A big piece of the missing equation here is the fact that home PC/laptop use has also been consistently decreasing, in favor of phones & tablets, causing phones themselves to be even more indispensable for youth.
I read a position paper last week suggesting the solution to this is to take a zero tolerance policy in the classroom and move all course testing back to pencil & paper / bluebooks. I would support that (as a parent of two current high schoolers).
Funnily enough, as a parent, I'd have loved for my (young-ish at the time) kids to have been on old-school pencil and paper stuff during the pandemic.
Tear-out worksheet books or a weekly trip to the schools to grab a packet of physical papers with the week's lessons and work (or, hell, send the buses around to drop them off) would have been SO MUCH easier to manage and help with than all the online horse-shit.
Like, I truly think my 80s and 90s classrooms would have been better prepared to deal with the pandemic than the modern computerized ones. You'd think it'd be the other way around, but from what I saw, no. It's just so much harder to keep track of what's going on in several different computer programs, than a stack of paper and a couple books for each kid.
I think there is a typo in the paper, that was carried over to the article. These two sentences appear to contradict one another as written:
> Interestingly, we observe significantly improved student test scores in the second year of the ban (about 2-3 percentiles higher than the year before the ban) when suspensions revert to pre-ban levels.
> Overall, we show that student test scores improved by 0.6 percentiles, with the ban increasing spring test scores 1.1 percentiles in the second year relative to the spring test right before the ban took effect.
Instead, I think the 1.1 percentile gain should be about the first year, and a 2-3 percentile gain by the second year. That is consistent with the graph.
But yes, a fairly small gain. I agree that much of the gain could be recovering from losses during the pandemic. Also the FAST is a new test that started in the 2022-2023 school year, so some of this could also be due to students and teachers adjusting to the new test and improving over time.
There are of course bigger problems, but it seems like phones are one of those "attainable" problems for a school to fix.
Schools aren't exactly much better equipped to make sure parents don't both need to work 50 hours to survive, nor bring housing prices down. They can barely pay their teachers to begin with.
When you're dealing with large populations (here, the study include 230,065 students--a very large number), even small shifts due to some treatment can be significant. It is very hard to generate top-down policy interventions that shift the mean of a population in significant ways: if this treatment effect (banning phones) is real, 1.1 points represents a very big policy win that can easily be applied elsewhere. The devil is in the details, however: they exclude some recent data based on the pandemic, but baseline off of 2022-2023, which was still in the throes of the pandemic. The data they show looks to have around a 0.5-1 sigma variation in percentile from 2022-2024, so the shift from the baseline of around 1 to 4 definitely looks significant, but it will be interesting to see if sticks over time.