Book: The Emerging Science of Machine Learning Benchmarks
mlbenchmarks.org136 points by jxmorris12 5 days ago
136 points by jxmorris12 5 days ago
Very cool book. I think a reason why ML has seen so much progress despite benchmark overfitting/abuse is that results are "regularized" by real world applications and the Lindy effect. Methods, or research, that abuse benchmarks aren't adopted by follow-up research so they tend not to survive. And they aren't adopted because people try them but then find out that they don't generalize to other/newer benchmarks. So the system works not because of specific benchmarks, but because of how the community as a whole deals with benchmarks.
This is the actual link to reach the book. There is no navigation link back to the index on the shared link.
If I'm recall correctly, this was also a keynote at MDS24? That was also a great talk, Hardt is an excellent speaker.
A little rule I live by is that if Moritz Hardt writes it, I will read it
Why is that?
You honestly don't know of Moritz Hardt?
Why so snarky? I also didn't know who he was:
I'm a director at the Max Planck Institute for Intelligent Systems. Prior to joining the institute, I was Associate Professor for Electrical Engineering and Computer Sciences at the University of California, Berkeley. My research contributes to the scientific foundations of machine learning and algorithmic decision making with a focus on social questions.[0]
Also simply knowing of him doesn't answer the question.
added to my reading list :)
Read the preface.
1. It sounds like this book can be summarized in a practical blog post or a series of posts
2. Is using the term crisis so many times really necessary?