I got fooled by AI-for-science hype - here's what it taught me
A physicist's firsthand account of how AI's impact on science has been overhyped. Real advancements are limited due to lack of reproducibility and survivorship bias in published results.
tl;dr
A Princeton physics PhD switched to studying machine learning for science, expecting revolutionary results. Instead, he found that AI methods performed much worse than advertised, comparisons were unfair, and the field suffers from survivorship bias since negative results are never published.
My Thoughts
Opposite to the hype created online, it seems that real advancements made by AI in different science fields is quite limited, basically it lacks reproducibility and it has fall in the hype wheel. It does not mean that AI can revolutionize, but real impact and frequency should still be challenged.
As the author notes:
AI might eventually prove useful for certain applications related to solving PDEs, but I currently don’t see much reason for optimism. I’d like to see a lot more focus on trying to match the reliability of numerical methods and on red teaming AI methods; right now, they have neither the theoretical guarantees nor empirically validated robustness of standard numerical methods.
This is my personal commentary on the original article. Please read the original article for the full context.