Peer review is suffocating science
HTML-код
- Опубликовано: 22 сен 2024
- You know peer review, right?
It’s the way academics check each other’s research papers.
It ensures that only the good ones are published and prevents the bad ones from getting through.
Right?
Wrong.
Peer review does precisely the opposite of what you think it does.
It prevents the good papers from being published, and ensures that only the bad ones get through.
Peer review is suffocating science.
If we want to reverse the stagnation of science over the last 50 years, then we’ve got to get rid of peer review.
-
I highly recommend you read Adam Mastroianni’s www.adammastro... splendid article The rise and fall of peer review www.experiment...
I first heard Adam’s ideas about peer review in his conversation Adam Mastroianni on Peer Review and the Academic Kitchen www.econtalk.o... with Russ Roberts russroberts.info/ on EconTalk russroberts.in...
Why has there been no progress in physics since 1973?
• article lasttheory.com...
• audio lasttheory.com...
• video lasttheory.com...
Scientific papers:
• The journal Nature began to require peer review in 1973 newscience.sub...
• Millions of academic articles are published every year blog.cdnscience...
• Some scientists simply make stuff up research.uh.ed...
• Fraudulent studies make it into respectable journals like Science www.science.or... Nature retractionwatc... and The Lancet www.bmj.com/co...
Physicists:
• Isaac Newton en.wikipedia.o...
• Albert Einstein’s en.wikipedia.o... four papers en.wikipedia.o... published in 1905
• Max Planck’s en.wikipedia.o... principle that science progresses one funeral at a time en.wikipedia.o...
The Wolfram Physics Project www.wolframphy...
• Stephen Wolfram www.stephenwol...
• Jonathan Gorard www.wolframphy...
My projects:
• The Last Theory lasttheory.com/
• Open Web Mind www.openwebmin...
Image of Adam Mastroianni www.adammastro... by permission from Adam Mastroianni
-
The Last Theory lasttheory.com/ is hosted by Mark Jeffery markjeffery.com/ founder of the Open Web Mind www.openwebmin...
Prefer to listen to the audio? Search for The Last Theory in your podcast player, or listen at lasttheory.com...
The full article is at lasttheory.com...
Kootenay Village Ventures Inc.
Science is not the only field plagued by the primacy of the status quo and the suffocating gate-keeping of certain academics. Join a writers' group and listen to 13 people describing how they would have written your short story, and how much better it would be. When everything is reducible to money and when the only pathway to success is through a group of staid, egotistical bumblers, then we're all in serious trouble. (Ed Witten, I am not talking about you.)
Ha, yes, thanks Greg! That's a great point of comparison: I've always avoided writers' groups like the plague!
I do get the point, but I dont think it is enough to suffocate science as a whole. As you rightly said in the video, people still can get their ideas out on the web using youtube, or blogs or other means. And if the ideas are good enough, people would surely notice them and it would get attention.
Yes, you’re right, Jwalin, people _can_ get their ideas out there on the web, and that’s a wonderful thing. So much better than the pre-Internet days of Einstein!
But there’s still extreme prejudice against anything that’s not peer reviewed. If peer review were abolished, or at least dismissed as behind the times, then truly new ideas wouldn’t be at such a disadvantage.
I’ve seen this repeatedly with Wolfram Physics: scientists and reporters, even bloggers and podcasters, dismiss it out of hand, complaining that it’s not peer reviewed. This failure to be open to these new ideas will set The Wolfram Physics Project back maybe 10 years.
It’d be good to change this closed-mindedness.
Thanks for the comment!
Well, isn't it almost happening? I mean as you know there are multiple rather science online outlets you can publish your ideas without peer review. Also, there is the Wild Wild web where you can put out anything, and if it sticks, well it does. The remaining thing is the idea of "recognition" and "credit" to receive for your resume and career advancement. Well, that does come around these days through your viewership online! Isn't it the new peer review? The only problem is that most of these peers are not really that educated, but hey, they can have opinion by their power of clicks. Anyhow, I am saying traditional peer review is only useful for traditional media. Once someone is established his/her expertise through a formal Ph.D. degree, can publish pretty much anything, either traditionally peer-reviewed or peer-reviewed. What is left is the power of prediction and explanation of an idea. If someone can explain things that are either not explainable in current theories or does it in more straightforward way with less arbitrary parameters, then it really doesn't matter these days where that idea has been published first.
Yes, absolutely, it is sort of happening! I've found there's still extraordinary acceptance of peer review, though, and prejudice against anything that hasn't been peer reviewed. I've received many messages telling me that Wolfram Physics must be dismissed because it hasn't been peer reviewed, rather than on its merits. I hope this changes. Thanks for your thoughts, Reza!
This is the way.
The way for cranks and grifters, yes.
I thought peer review was about seeing if the results can be replicated? You do have to have results that can be replicated in the first place though.
Sadly, no, it's not about replication. Research is rarely replicated, since there's little incentive for scientists to re-do research from scratch (and get very little attention, either for proving or disproving an existing research), only to do original research (that's what gets the attention!) Thanks for the comment!
It's about seeing whether the experiment is described well enough to know whether you can replicate the results. I.e., not "can the results be replicated" but "can the experiment be replicated." It prevents papers that are so bad that you can't even figure out whether they're good or bad. It doesn't prevent bad papers per se.
DARN RIGHT. The whole notion of this post is nonsense.
I feel this is a much wider problem than in just the Sciences. Academia in general seems beyond saving at this point. The Humanities fell a long time ago. Yes Mark, peer review has to go. Open Web Mind, all the way!
I hear you. Thanks, as ever, for the comments!
I got a kick out of that post...
@@gregvondare Not too painful, I hope?🤭
@@Terpsichore1 -- I was indulging in a personal joke. Since Terpsichore is the muse of dance, getting a "kick" out of her... Well, you know.
@@gregvondare Gotcha! Nice. Not much “kicking” in my line of dance, so went right over my head. 🩰
LK99 superconductor past peer review, but web boffins were fairly quick in disproving it.
I don't think the LK99 superconductor paper did pass peer review, Roger. I think it was published on arxiv.org without any peer review.
But that makes it a good example of how science could work _without_ peer review. As you say, the room-temperature superconductivity claim was reasonably quickly disproved.
Seems like this is exactly the way things _should_ work!
Dang. Compelling arguments. Even the mRNA lady who won the Nobel prize was lambasted for a decade before her ideas where excepted.
Let's not forget poor Boltzmann 😭
Thanks, and yes, it's strange to see that peer review is proving so counterproductive, but even stranger to see the scientific community being so unscientific in their assessment of its merits!
Beautifully stated! I (for the sake of the point of the video) was met with somewhat encouraging and scathing rebuttals to my own thoughts on the universe when i originally posited them to arivx, before (ultimately) anyone with an email who I thought may understand (based on their area of research, i.e. game theory and beyond) what I was suggesting. My ultimate conclusion; based on my own social status and other variables was that these things are maximally and minamally seperated based on a number sociological variables. I think that aligns sadly or inconsequentially, hilariously or serendipitously with that which you are exporing. It is all (dramatically), none the less true! Godspeed my good friend!
Thank you! It's good (and instructive) to hear your own experience with this.
You can always put your paper in the arXiv, it's getting so called publish. If it's a good paper, it's a good paper.
Yes, I completely agree. Platforms like arXiv are the future of scientific publishing. Not to mention RUclips, podcasts, the open web… let a thousand flowers bloom!
@@lasttheoryDoesn't this expose a basic flaw in your argument then? No young scientist needs "permission" to put their work on ArXiV. If peer review is not forced on anyone to publish their work, then why do we need to "end it"? As it stands, educated readers can choose to either give content in peer-reviewed journals priority and trust, and read that first, or they could choose to ignore that and just read randomly selected articles from a preprint server, or blogs, or RUclips. Many probably do just this, though I imagine some use peer review to prioritize their time. I think it's inaccurate to hold peer review to the standard of being a perfect system to prevent bad content getting published. It is simply an established, though imperfect, way of putting an indicator of quality against a work of research prior to actually reading the content. Surely this is merely curation - which you are free to disagree with and of course should not be considered definitive - but not really censorship in any sense?
@@billyaxon I agree, peer review is not censorship. Anyone can publish anything on the web, at arXiv or anywhere else, and that's a seriously good thing.
But what if peer review, beyond being merely imperfect, actually does more harm than good? What if it gives too much credibility to papers that are deeply flawed, while preventing much better research from being considered? Wouldn't you want to end peer review if that were the case?
Many good points! I'll be following the Open Web Mind project.
Thanks Neale, that's great! I'll be launching my new Open Web Mind channel over the next couple of days. Really excited to get this out there!
Asking as an outsider, is this one of those things where there's a near-consensus but nothing ever happens? I mean how much of your life have you wasted listening to "Dr Krgebgk Odnfbg is the distinguished Monsanto Koch Brothers Fraggalicious Chair of Gardening. At Princeton they..."
That's never gonna end.
I always found it hilarious with something like physics because the person at the podium is very obviously qualified to be there. They act as though they're trying to convince an audience of hardcover skeptics and debunkers
Yes, I kinda think there's too much consensus in science today. It'd take a real mindset shift to allow anyone to publish anything, but I think it'd be really healthy.
Nobody is preventing you from publishing your papers exactly as Newton and Einstein did. If you're having tired old scientists pay your research, and you're coming up with stuff that most of those tired old scientists disagree with, then yeah, you're going to have trouble. Go do what Newton and Einstein did - get a job, and do your science on the side.
And while Wolfram's stuff is very cool and might indeed prove to be a true model of the universe, but I'm not sure it's "science" yet. As far as I can tell, it's not even to the point that String Theory is.
I think we agree on these things, Darren! You're right, anyone can publish anything on the web, and that's a good thing.
You're right, anyone can do science on the side. Indeed, Stephen Wolfram is doing exactly what you suggest: he has a job, as CEO of Wolfram Research, and he's doing physics on the side.
And yes, there's a long way to go before Wolfram Physics makes novel, testable predictions, but I think it's way ahead of String Theory, at least in the sense that it would survive Occam's Razor.
@@lasttheory I think String Theory made predictions, but the predictions just wound up not matching measurements. It'll be fun to see what Wolfram is doing come to fruition. I liked the idea since NKOS and I've seen a couple of science fiction stories based around Quantum Graph Theory (as it's called there). (Greg Egan being the primary author of those, IIRC.)
AI will be huge in the peer review process.
The machine learning systems I've used have been a little superficial. Sure, they'll detect unscientific _language,_ but the point is to detect unscientific _research._ Do you think AI can get there?
@@lasttheory Absolutely. AI can already reason. Within a few years AI will be able to read and understand the paper, run the calculations, make suggestions, pulling from its vast memory.
@@aunumever Strange times... I'm fascinated by the intersection of AI and computational physics. We'll see what it comes up with!
@@aunumeverAI cannot reason beyond whatever can be easily deduced from the prompt. It can show a proof of the Pythagorean theorem, with no guarantee that it's correct, but it can't construct a novel proof. AI that can reason should be able to prove theorems it hasn't been trained on. No AI can do that and nothing even comes close.
I share your optimism but LLMs are either a dead end or the first step of many many more. I think we're a long way away from AI that can understand things.
That's well stated, thanks Alex. I wish everyone understood that LLMs are, well, large language models, next-word predictors, not AIs.
"Jesus f#@ing Christ!" Did somebody pay you off? LOL 🙂It seems that there's a new paradigm in town and it's called the "wild west" approach.
I think there's something to be said for a bit of the Wild West in science! Proving ideas requires precision; coming up with ideas in the first place requires imagination!
@@lasttheory Definitely I do agree with the proving/imagination statement. The "Wild West" paradigm is fine as long as one doesn't "decree their stuff as gospel."
@@williamschacht Yes, absolutely, I agree, thanks William!
@@lasttheory "Proving ideas requires precision; coming up with ideas in the first place requires imagination!" Correct. Real scientists have both, and you have neither.
@@nataliearcadia Ah, but I'm _not_ a real scientist! I'm not a scientist at all. I'm a writer and a humble maker of videos, trying to make sense of all this. With a _little_ precision and a _little_ imagination, I hope.