AI Tutoring with Khanmigo in a Two-Year School Experiment
edworkingpapers.com26 points by bryan0 2 hours ago
26 points by bryan0 2 hours ago
https://www.pangram.com/history/2e681dd5-5fbb-4eb8-8562-973b...
This is page 4-11
The abstract is also AI generated. Good god people.
A very honest take. They tried something, it showed a very small improvement, and they acknowledged it. Acknowledging that the cause appears to be a lack of engagement agrees with my own experience. You can give students great tools but that doesn’t make them interested in math. Khanmigo might be a good tutor but it won’t help unless you’re chaining kids to a desk and not letting them leave until they finish some quota of problems.
It probably isn't a good tutor, I'm using frontier SOTA models and whenever I ask them to explain anything they don't consider my question and don't display any understanding but just spam out unrelated simplistic metaphors, can only imagine how much worse the models the kids had access to were, LLMs don't have understanding and can't fake it either, there's no conception of how a memory might be formed or something might be understood, without that qualia it can't be a good teacher it can only output text that looks like it might be a good explanation until you read it
Agreed, I need to do a lot of prompt crafting when some math academy thing is explained poorly by them.
What sometimes helps is to ask them to make something interactive and visual. Like 3B1B does
A good tutor builds an understanding of the learners current mental model then tries to broaden it with targeted materials (text, diagrams, videos), selected exercises, the Socratic method ... All LLMs can do is produce text that statistically looks like tutoring. AI has a lot of uses but tutoring is not a great fit. I think a well written textbook even would be better than AI because the author can have a general understanding of where the target audience is usually at and where they're expected to end up.
Conclusion: "The binding constraint appears to be engagement: realizing the promise of AI tutoring will require getting students to use it, not just giving them access."
> The binding constraint appears to be engagement: realizing the promise of AI tutoring will require getting students to use it, not just giving them access.
This seems backward. Engaging students should be the primary goal, not AI tutoring. Great teachers engage students. Perhaps AI can somehow aid teachers in becoming better? [Though, I doubt it]
Maybe the AI can help them not quit in their first two years, when they realize what the job really entails. It’s a profession powered by idealism but the long-timers often have the cynicism of former idealists. Not always! But it’s a tough gig that attracts dreamers.
I have found this with my kids as well. I have an AI writing tutor that they use but only one of the two actually likes engaging with it. The other gets easily disengaged.
Some kids will be able to learn at a record pace with AI, others will need to be brought along in the usual ways via good teaching that fosters engagement.
My son does not have a great opinion of any edtech -- AI or not. I would wager that if you asked any kid his age they'd rather have better air conditioning than another expensive bit of software they hate using.
My casual interpretation is that most kids didn't want to use the AI tutor. The authors estimate that time spent using the AI tutor is as effective as time spent studying without AI assistance, which is around one fourth as effective as studying with a real tutor. The real tutors are obviously quite expensive, comparatively.
So, if some kids do prefer studying with this AI tutor, that's as good as studying without it. Whichever approach gets them to put the study time in.
> The authors estimate that time spent using the AI tutor is as effective as time spent studying without AI assistance, which is around one fourth as effective as studying with a real tutor.
I find that surprising. I would expect the AI tutor to add noticeable value over studying static written material.
blah blah blah.