AI boosted homework scores, then exam scores dropped: study

It's not so clear that AI is an amplifier.. The paper has some fascinating analysis on this topic:

"At the other end of the distribution, AI students who spend more than 65 minutes on their homework receive homework and exam scores similar to those of non-AI students, suggesting that these students do not use generative AI for homework assignments. However, this group consists entirely of students who adopted generative AI no more than Öve months. Six months after adoption, no AI student spends more than 65 minutes completing their homework (see Figure A5). This is consistent with the gradual process of learning how to use AI tools. It also suggests that AI crowds out the highest level of e§ort."

"Interestingly, in the range of 50-65 minutes, the median and the interquartile range of exam scores of AI and non-AI students are similar. This implies that, in the range where AI students and non-AI students have overlapping homework times, students who spend the same amount of time completing homework on average receive similar exam scores."

"This pattern shows that students who spend the same amount of time on homework learn similarly, with or without generative AI. In other words, generative AI reduces time spent learning for the majority of AI students but not learning efficiency for those who spend the same time studying as the non-AI students."

The scary thing is that 81% of AI users in this study were determined to be "outsourcing" their homework to the LLMs - and the rate increases the more exposure they had to LLMs.

The "slightly higher" performance is based on statistically insignificant samples (between 4 and 20 students, depending on the context, out of the total population of 26,000): https://bsky.app/profile/benjaminjriley.bsky.social/post/3mt...

This is kind of damning, isn't it? "Similar or slightly higher" for spending the _same time_ means it's not, in fact, helping. At the same time as being extremely damaging to a significant number of students who, of course, use the AI to cheat. Because everything about it makes cheating easy.

> I believe AI is basically an amplifier of bad and good.

Same can be said of technology in general tbh.

You aren’t being cynical, you’re arguing with disingenuous entities with money on the line. We already know it’s used primarily for the negative case. Everyone who ever intended High School or College knows this.

The dangerous thing about amplifying is that bad people are often more willing to amplify their activities, because they don't care about the negative effects. We are seeing this now as AI companies (and large companies of all kinds) rush to secure whatever advantages they can, regardless of the negative externalities. Meanwhile people who actually care about doing the right thing get trampled.

We need to shift the incentives by adding ruinous penalties for things that are currently quite commonplace if they are done by large players. Some dude training his own AI on his own computer can scrape and train. The fine for OpenAI or Meta using a single copyrighted book without permission should be in the tens or hundreds of millions.

What we're seeing currently in our society is a "loophole inversion" where the rules have an effect mainly via their loopholes. The most profitable activity is to find loopholes and exploit them as frenetically as possible to gain as much advantage as you can before the loophole is closed, or get people hooked on the loophole so it's retroactively legalized. Entities that are big enough to do this are big enough because they have lots of money behind them. Entities doing the same kinds of things without lots of money are not really doing much harm. So the best approach is to adopt a "sliding scale" in which even tiny violations by wealthy actors result in penalties enormously greater than fairly large violations by small players.

It's a familiar story. If you copy/paste Wikipedia and turn it in, you learn nothing. If you skim Wikipedia, hit up the references, the consult other sources, then synthesize your own thought, you probably get farther faster than you would without it.

We could probably cook up thousands of examples of the same problem (YouTube DIY tutorials, GPS navigation, etc.)

This is my thoughts as well! It makes smarter people smarter and dumb people dumber!

> Homework should be optional material for self study

At Caltech, homework was assigned but had no bearing on your grade. The grades were based on the midterm and final exams.

But not mastering the homework usually resulted in flunking the exams. There were "retch" sessions after each homework assignment that was staffed by a grad student, and the purpose was to help the students understand the homework problems. I knew only one person (Hal Finney) who was so smart he didn't need to do the homework.

I learned the hard way that the path to success was:

1. never miss a lecture, no matter what

2. take notes by hand during lecture

3. do the homework on time, and make sure you understand every problem. Take advantage of the retch sessions.

And that worked for me.

Unlimited retakes would be an enormous amount of work for professors/TAs/teachers.

Optional homework is often a disaster. At best, students would do it right before an exam and the goal of education is not to just pass exams. They’d probably still get a lower score than if they did the homework when they were supposed to.

What I think is better is to have a due date, but just make the maximum 10% each day it is late. So after 2 days, the highest score you could receive would be 80%.

I liked that system because it gave some flexibility with deadlines while still encouraging you to turn things in on time.

What you described about exams is like how I approach most certification exams: go through a test bank of questions until I get 80% then take the real test.

The study was done in China, where making the class easier to improve pass rates isn't really a thing, as the Gaokao operates more like a stack ranking where it doesn't matter much how well you did in an absolute sense, only that not too many others who did better than you are competing for the same spot. Unlimited retakes are possible in theory, except each costs you a year of your life and except for some extreme cases of repeat test-takers, most people are going to give up after just one or two bad results. Homework is definitely not optional, but going home from school is. (Due to the hukou system, children often have to go to school far from where their parents work, so boarding schools with teacher-supervised self-study are common.) I don't see how holding back for individual classes is supposed to work considering scheduling conflicts.

> F students will realize they have to study, start actually learning and then pass on the second time

Do you have evidence for this beyond your friends (who were accepted into college)?

It's all cute but naively glances over what this all boils to - competition.

Education system, contrary to popular belief/name, isn't tailored to educate but to select winners and losers which then will be picked on the job market.

That's why it seems absurd when you think about it as an institution that aims to educate. That's because that isn't the real purpose of it. The purpose is to stratify and classify early.

Schools do not operate in a vacuum; they serve as credentialing gatekeepers for a hyper-competitive capitalist job market. If everyone could easily retake exams until they got an A, grades would lose their primary utility for employers and universities: differentiation. Society relies on schools to provide a neat hierarchy of candidates that for one reason or another thrived in difficult environment of adolescent schooling.

The system often prioritizes compliance, endurance of boredom, and social maneuvering over actual critical thinking precisely because those traits align with corporate hierarchies.

> If you try and force kids to study when they don't want to "for their own good" you're not actually helping them. Everyone needs to find their own path. Let people fail.

this rhetoric is pretending to be an alternative to coercion. IMO the ideas you are talking about are well trodden and are still coercion nonetheless.

> Homework scores once predicted exam performance

As with most training the journey is the point, not the destination.

That said, I think smart use of AI could help. It could explain concepts in a way that might help you understand better, it could probe your knowledge in a more dynamic way by tailoring questions, and so on. This requires the AI be restrained by some harness, not free to write down the answers for you.

I've tried stuff like this, but the problem is that you don't know if it's answers are correct, and more importantly, you don't know if its questions are on base or not. Plus its too easy to go off on tangents, especially when you ask it questions to clarify. You can get some more success by writing down a framework beforehand, but openended dialog is still not a good way to learn with AI, from my experience.

I mean, everyone I know has been using socratic dialogue for months to aid understanding, this seems like a variation of that. The issue is that if the answer is too easy to obtain it's hard to have the discipline not to cheat.

Flipped classroom was a popular method when I was teaching math. Some students found it very helpful. The trouble was getting the students to do the reading before class. If students came in unprepared they struggled to keep up.

I think for my ADHD child-self, this would have been a huge improvement. I didn’t need help learning, I needed help doing homework.

They’re using flipped learning even in universities now.

This is typical at the college/university level - when I attended 20-25 years ago, we were expected to learn the material through readings outside of class, and then class time was spent on discussion, application, questions, and working through problems.

IMHO there's a lot to recommend this style. A lot of students learn better when they can wrestle with the material at their own pace, on their own time, in their own setting. It quells a lot of anxiety about "I'm not following what the teacher is saying, am I stuped, will I look bad in front of all my peers?" And then class time can be spent identifying holes in your knowledge and getting instant feedback from an expert, which is where they are most useful.

Flipped classroom method?

Interesting. Ages?

as a fun aside here from the abstract it does come down to how you use it:

> AI users who maintain similar homework completion time as non-AI users experience small learning losses.

If the pure completion time of homework goes drops significantly with an LLM tool, I suspect these students are spending the extra time turning the material over in different ways to internalize, which is interesting.

Maybe. As the nature of assistive tools changes, so does pedagogy. Calculators are the easy example since most of us are (still) familiar. But there are others that were just as pronounced -- e.g. log tables and slide rules were transformational and instrumental at one time, and you've (probably) never used them. If you had used slide rules to solve your homework problems then were excluded from using them on exams, you'd see a similar effect.

AI is a big change; pedagogy is going to change too.

I see this as a memorize-vs-understand fallacy many repeat.

It is really a combination: you need to memorize a large corpus of information to operationalize knowledge — understanding a bunch of theorems in mathematics does not help much (even if you are able to prove them when you see them) if you can't remember the boundary conditions they hold under.

Or having good understanding of foreign language grammar won't help you if you do not memorize words that you need to express your thoughts.

Good literature will show you some of life's challenges and potentially let you think through them in a non-stressful situation (other than "I've got to finish the last 200 pages by Monday" ;)).

I wish that was the main way countries understood education

School has always been about creating compliant workers, not educating people. They will double down on "performing the right things" and "morals" while the economy will continue on its K shaped path as AI gets more capable.

To extend this analogy:

Using the forklift as a spotter and to assist in loading weights increased gains. Then again, a human can do all those things, and provide real human connection.

Which was pretty much the whole damn premise of needing a teacher to begin with. What a surprise!

Meetings are still like exams, if you can't craft a line of thinking on the spot it doesn't come off well.

You don't need a final answer but you can't just answer "I don't know" otherwise you risk being slimed (shout out to anyone who still remembers "You Can't Do That on Television")

Every conversation you have with a person is an "exam." Not being able to extemporaneously express yourself or reason analytically without running to Claude seems undesirable.

In my professional career, my clients value the depth of ready knowledge that helps me use my tools with particular insight and finesse -- a kind of knowledge that exams demonstrated far more effectively than homework when I was back in school.

The exams aren't the thing that matter, they're an experiment to measure the thing that matters. Which, presumably, still matters at work!

Stand-alone bottom-up observations (even if valid) obscure the fundamental problem that AI (or "AI" if you like) is inducing systemic change across human civilization.

If cohorts are graduating into the workforce with decreasing stand-alone skills, that implies the economic value-add of labour is shifting to AI, which implies broadly falling standards of living, falling political power, and an ongoing transfer of power and wealth to capital.

If students are taking shortcuts with AI when their brains have the highest capacity for learning they may be permanently weakening their prospects. It doesn't seem likely that whatever "AI management skills" they incidentally learn will make up for, over their lifetime, the economic value loss they suffer due to reduced cognitive skills.

On an individual basis, in the next few years, this might not matter. Across decades, economies and polities, it will.

Your career is a continual exam and the assignments are just parts of it.

Your employer gave you your job based on your exam results.

Your employers want those assignments done. That’s why they pay you. School assignments belong in the trash can and you pay for someone to look at them before throwing them away.

Even if we grant that education is to prepare for employment, which a lot of people disagree with, it does not follow education should do the same thing as what's done in employment.

evaluating/reviewing your LLM's output/code/reasoning is an exam

unless you're just blindly passing it on for someone else to deal with

One of the main reasons safety standards are written in blood.

(This was originally posted in response to https://news.ycombinator.com/item?id=49389565, which has a different article. We've since merged the threads.)

They left 20% of the 87,000 students as a control group. Is that not enough to allow for differences in environment, etc?

I got graded homework even in a few college courses and thought it was dumb, but then I realized it's always in the low-tier classes where they can't trust the students to study on their own. Those who skip homework hurt themselves by doing poorly on exams, but that's not enough to stop them. This ruins the curve in ways the instructor can't really fix unless they're prepared to fail half the class, which they aren't.

This is the answer. Many (all?) European schools don’t give homework (this is partly because of the way the grad programs work). It’s on the students to study in order to pass the (difficult) exams.

Around here kids get homework in kindergarten. It's absurd.

The often underworked (most are autograding these days and reading old powerpoints rather than do real curriculum development) and (for the quality) overpaid teachers/professors will mass riot against this because it means they might have to actually teach. We might even get yet more anti-boy treatment than already exists among them as a social class in retaliation.

Please also abolish PowerPoint and force oral exams while you are at it.

Teachers are also luddites as a social class. Math teachers cried about calculators yet everyone of them who refused to adapt did disservice to their students (especially anyone doing statistics). My state's education system is utter trash despite being well off economically because of strong teachers unions keeping them from undergoing even a tiny bit of scrutiny.

They do not deserve the social credit/grace that they get. They are ultimately gatekeepers, whose extremely biased decisions in grading/treatment of their flocks decide which kid grows up on the streets and which kid grows up to eat lobster thermador every day.

Teachers also should not be disciplinarians. Children with discipline problems bad enough to disrupt a whole class should be thrown out of the class room as basically the only thing a teacher does for "discipline". The "right to education for the shit kids means we can't do that" framing is such trash for those who want an education and will inevitably be disrupted by a small minority of terrible students.

Also it's interesting that one of the only other things Max Stirner wrote about besides philosophy was education and his myriad problems with it (given his history as a school teacher).

https://theanarchistlibrary.org/library/max-stirner-the-fals...

So your complaint is...it's boring?

The point is to train the brain via repitition and iteration.

If your goal is entertainment, I'm not sure what you are looking for.

Everytime a student says this—it's because they're procrastinating on their work. The hardest part about anything new is starting it. Once you build some momentum, interest and focus tends to build as well.There certainly are some homework assignments that are boring.

A child today can absolutely be successful at doing their homework if they have time to... sit down and write, or draw, or do crafts or things that require them to focus. It's not like Screens exist, therefore no attention span.

What's happening is people are only fixating on screens, and foregoing these slower forms of media. That's akin to saying all children will be obese now that fast food is around. Of course, obesity rates will sky rocket if all you eat is fast food. But if a kid grows up to have homecooked meals, and eat with their parent(s) then they won't. It's as simple as that.

Homework isn't boring.....its the social media and addictive games that completely are destroying our attention span.

After I detox from online games, I can go back to learning new "boring" stuff. There are nerds(myself included) who find learning fun, and using AI boring because it doesn't scratch the learning itch.

Learning how to do "boring" things is probably one of the more important skill a child can learn, most things worth doing tend to be kinda boring at least at some point.

Yep, learning is hard work.

I remember reading about people talking how writing and reading books leads to cognitive decline because people won't need to remember things anymore.

Pretty sure the way social media is designed contributes more to short attention spans and atrophy of the cerebral cortex.

Not doubting this one bit but is there actually any research linking AI use to Alzheimer's? I think that takes long-term studies.

> Yes, using a calculator leads to worse outcomes in your learning.

Does it? https://www.jstor.org/stable/749255

You do sometimes get to unlock it by being allowed to use calculators in more advanced classes

The important thing is that I could learn to use a flint and steel pretty easily if I wanted to. I don't need to carry out that particular task, but my ability to adapt to new problems means I can carry out more relevant tasks when needed. The most useful part of school and university for me wasn't that I learned the difference between a cirrocumulus and a cirrostratus cloud in geography class. It's that I learned how to learn.

If we let children cheat their way through school and justify it by saying they don't really need to think for themselves anymore, then we're going to end up with a generation of adults who struggle to take on any difficult tasks at all. I suppose your counter-argument to that will be "but we can just hand over everything to the AI" but that's an essentially an argument that humans should be considered obsolete altogether. I sincerely hope that that's not a path we end up going down (although I'm genuinely scared that it is).

These are students aged 12-18. Even if we ignore all the other negative externalities of AI and assume it will be ubiquitous and cheap, kids need repetition and exercises to make inferences and develop intuition.

Sure - you can ask AI to, for instance, calculate how much money you'll have given a fixed principal and a compounding interest rate. But we need to get a kid to the point where they even understand what the question to be answered is. If they're using AI all along the way, they are developing less understanding of what fractions and exponential growth are.

Will it stop the really smart and motivated kids? Of course not. But at population scale there will be big consequences.

You need some baseline knowledge to be able to gauge the correctness of answers you get from AI and to be able to ask the right questions. Removing homework and exams seems very short-sighted.

We could also drop school altogether and just make sure every kid has an LLM-connected device that's voice driven. They wouldn't even need to learn to read and write -- that's old school.

update: this would make universal education free! because MegaCorp would happily give/support the devices free of charge (in exchange for ads being inserted into the responses)

Sure. Let's not think too hard about what it means that general thinking and creativity are the skills we've stopped practicing this time. It will be fine.

(You may be making the same point as me or the opposite point, I can't tell what's sarcasm anymore. I try to make mine obvious.)

> We also don't know how to use flint and steel to start fires any more.

Tell me you weren't a Boy Scout without telling me you weren't a Boy Scout.

you know we let social media companies cause massive harm over 2 decades; but clearly, this will be deifferent!

Why one would care? Look at your comment from two days ago.

https://news.ycombinator.com/item?id=49363319

> I hope we'll figure out some equitable society before that, but I'm not going to bet on it, so I'll do what I can to be on the side of the winners.

Obviously people care if they think they might lose out.

Is it going to be this hopes-and-prayers about egalitarianism/ah screw it I hope I get to oppress and dominate the majority? Or all of us sipping wine on the beach for eternity?

https://news.ycombinator.com/item?id=49366313

You gotta pick a lane with your rah-rah-ism.

What is a doctor? The person recommending next course of action based on best practices. What is the best practice? Is it a globally decided on thing, a locally tested thing that has proven to be better? Is the global thing just the most popular because someone wrote it on reddit and it got millions of views (but isn't actually good).

If our society is organized on just delegating all authority to "the best answer" we can drop all jobs - and only the surgeon has some authority over the machine. When we have a surgery robot, then they can go to. And at some point we'll be recommending and forcing surgery for athletes foot because some kind of best practice or viral post going out of control and a bunch of openclaw bots randomly decide to make tons of recommendations online for amputation for athletes foot and GPT 8.6 agrees this is the way to go.

OR we can somehow decide there is a boundary, there is some room for human memory and decisions.

This is too simplistic. There are disciplines, like Anatomy, where you're basically just memorizing bones, glands, and other parts of the human body. For organic chemistry, students need to memorize most of the krebs cycle diagram for their exams (because doing so and drawing it lets them reference it later).

Thinking is infinitely more complex than "critical thinking" vs. rote functions. You want students to get to critical, eventually, but the lower orders on bloom's taxonomy serve important roles, too.

Course design built around multiple choice questions, essays, and a lot of the historical assessments that we're used to are largely not great for learning in the first place—–they're used because they can be auto-graded or score assigned quickly. AI, conversely, is kind of revealing the limitations of these outmoded learning design practices.

Think of the meme going around right now from Milennials to Gen Z: "I wrote a 5 page essay on a book I never read!" <- They're saying this to AI generated essays as if there's any difference in the critical thinking that comes out of an exercise like that (read: absolutely none).

AI is a new tech—so of course people are still figuring out the limits of it. In 10 years a lot of the paradigm will be set. Until then, students are unfortunately kind of in the thick of it.

How do you differentiate between mastery of a topic and mastery of using an AI in an exam setting?

Open book tests and allowed cheat sheets already solve the memorization-is-not-required gap. It seems like allowing use of AI pushes that closer to thinking-is-not-required. And if no thinking is required, then how can you prove personal mastery?

Mastery requires memorization.

You can't play an instrument without muscle memory.

Why are commercial airline pilots required to spend hundreds of hours in simulators? So that when things go wrong they don't have time to think, they just react. And their reaction will be the correct course of action because that's what they have drilled on.

How many times have you gone "Oh, this new thing I have just encountered reminds me of this other thing?". How can you make such leaps without knowing that other thing?

Good exams don't just test memorization

> to show mastery of a topic

How do you show mastery of a topic where you neither understand fundamentals nor have skills to do a task without help?

Mastery is about demonstrating skills. Skills take practice, hands down.

AI can help us build better tools, provide tutoring, answer broad ranging questions, help relate concepts, and do reasoning for us. All of those things can be good in that they can help us acquire skills faster, e.g. with better practices and exercises. But it cannot replace actual time spent doing a thing.

Wanna learn knitting? You're going to have to knit some stuff.

Completely agree.

Absolute waste of time getting babies to walk now that we have cars and electric scooters!

A lot of expertise is actually knowing things about the domain. You can't make a good judgement call on a topic without getting relevant knowledge in your head to make that decision. You can't even lookup the relevant knowledge without knowing what knowledge is important to make the decision, so reference material is not enough.

Memorizing stuff is good for understanding. Of course memorizing without understanding doesn't really help, which is what a lot of student last-minute "cramming" boils down to. But without concrete and memorized knowledge, any abstract reasoning skills become quickly impossible.

A good analogy to caching and memory lookups. You store in your head only the most valuable immediately necessary information, the rest can be outsourced.

This is not, in fact, fine. The point of education is to learn things, and these results show that kids are not learning things as effectively when they use LLMs.

"You don’t need to be able to memorize something for an exam to show mastery of a topic."

Yes you do???

One more step towards eliminating the value of labor entirely. What "work" will people be doing on a daily basis, and what makes it valuable?

so how _do_ you show mastery of a topic?

> You don’t need to be able to memorize something for an exam to show mastery of a topic.

Yes, you do need to memorize and know things about the subject in order to be a master of it. What is the definition of "mastery" that you are using? What are the Things being memorized?

Definitely not fine. People are graduating from Computer Science programs literally unable to write a line of code without AI assistance.

These people have fallen into OpenAI and Anthropic's trap and are illiterate without the assistance of coin-operated slop generators.

It's critical to understand the fundamentals on your own and that is different from AI being used in public. Sure school doesn't have as much value as it used to but it was always more about social credit or a certification that you took the time and effort to understand the world under pressure (obviously not true for all disciplines) or goals attached preferably in group settings.

This is something that AI cannot replace. Viewing things as memorizing is the wrong take.

Way before AI, the problem was very similar - universities do not teach practically useful stuff. Its been a permanent complaint about the education system, really, at least for the last 50 years.

AI just gave the students a way to dodge the slog.

But university was never intended to teach the bleeding edge. That would really be impossible in practice. Pre-phd, it is supposed to teach ways to efficiently attack a problem. you can take a bunch of "play courses", and succeeding at any of those requires pretty much only that one skill.

Nowadays, the challenge they face, is to keep teaching "problem attack methods" in a way that can't be trivialized by AI.

Though fundamentally, if you go to university, and evade learning the one thing you can learn there - thats your loss.