I do get the point, but I dont think it is enough to suffocate science as a whole. As you rightly said in the video, people still can get their ideas out on the web using youtube, or blogs or other means. And if the ideas are good enough, people would surely notice them and it would get attention.
Yes, you’re right, Jwalin, people _can_ get their ideas out there on the web, and that’s a wonderful thing. So much better than the pre-Internet days of Einstein! But there’s still extreme prejudice against anything that’s not peer reviewed. If peer review were abolished, or at least dismissed as behind the times, then truly new ideas wouldn’t be at such a disadvantage. I’ve seen this repeatedly with Wolfram Physics: scientists and reporters, even bloggers and podcasters, dismiss it out of hand, complaining that it’s not peer reviewed. This failure to be open to these new ideas will set The Wolfram Physics Project back maybe 10 years. It’d be good to change this closed-mindedness. Thanks for the comment!
@@lasttheoryyou're right, but since most papers didn't contain enough information to judge the validity, a wave of plausible looking but essentially fake stuff would hit us. Per review doesn't prevent fraud, but it makes it much harder, because experts of that particular field look at it. Who would bother to look at tons and tons of non reviewed papers?
@@testboga5991 These are good questions, thanks. If you shift your thinking to a post-peer-review world, things look very different. First, we wouldn't have the false positives of: "It's been peer reviewed, so it must be _kinda_ right." I don't think peer review merely fails to catch fraud, I think it _enables_ fraud by adding a false stamp of approval to papers that really haven't been checked. Second, your question of who would bother to look at untold numbers of non-peer-reviewed papers could be applied to peer-reviewed papers, too. There are far too many published for anyone to keep up with, so you have to be selective. Think ahead to that post-peer-review world, and it's like asking: who would bother to look at untold numbers of RUclips videos? No one would. There are far too many RUclips videos for anyone to watch them all. But no one would need to. There are mechanisms for surfacing the good ones: algorithmic mechanisms, crowdsourcing mechanism, and the good old-fashioned mechanism where your trusted colleague at another university tells you that you really should take a look at this new theory. That's the way it currently works with peer review, and that's the way it'd continue to work post-peer-review.
Science is not the only field plagued by the primacy of the status quo and the suffocating gate-keeping of certain academics. Join a writers' group and listen to 13 people describing how they would have written your short story, and how much better it would be. When everything is reducible to money and when the only pathway to success is through a group of staid, egotistical bumblers, then we're all in serious trouble. (Ed Witten, I am not talking about you.)
Yes, exactly. Peer review is, well, your peers reviewing your work. That's a good thing! The formalization and entrenchment of today's officious brand of peer review is not.
Well, isn't it almost happening? I mean as you know there are multiple rather science online outlets you can publish your ideas without peer review. Also, there is the Wild Wild web where you can put out anything, and if it sticks, well it does. The remaining thing is the idea of "recognition" and "credit" to receive for your resume and career advancement. Well, that does come around these days through your viewership online! Isn't it the new peer review? The only problem is that most of these peers are not really that educated, but hey, they can have opinion by their power of clicks. Anyhow, I am saying traditional peer review is only useful for traditional media. Once someone is established his/her expertise through a formal Ph.D. degree, can publish pretty much anything, either traditionally peer-reviewed or peer-reviewed. What is left is the power of prediction and explanation of an idea. If someone can explain things that are either not explainable in current theories or does it in more straightforward way with less arbitrary parameters, then it really doesn't matter these days where that idea has been published first.
Yes, absolutely, it is sort of happening! I've found there's still extraordinary acceptance of peer review, though, and prejudice against anything that hasn't been peer reviewed. I've received many messages telling me that Wolfram Physics must be dismissed because it hasn't been peer reviewed, rather than on its merits. I hope this changes. Thanks for your thoughts, Reza!
Beautifully stated! I (for the sake of the point of the video) was met with somewhat encouraging and scathing rebuttals to my own thoughts on the universe when i originally posited them to arivx, before (ultimately) anyone with an email who I thought may understand (based on their area of research, i.e. game theory and beyond) what I was suggesting. My ultimate conclusion; based on my own social status and other variables was that these things are maximally and minamally seperated based on a number sociological variables. I think that aligns sadly or inconsequentially, hilariously or serendipitously with that which you are exporing. It is all (dramatically), none the less true! Godspeed my good friend!
I don't think the LK99 superconductor paper did pass peer review, Roger. I think it was published on arxiv.org without any peer review. But that makes it a good example of how science could work _without_ peer review. As you say, the room-temperature superconductivity claim was reasonably quickly disproved. Seems like this is exactly the way things _should_ work!
Sadly, no, it's not about replication. Research is rarely replicated, since there's little incentive for scientists to re-do research from scratch (and get very little attention, either for proving or disproving an existing research), only to do original research (that's what gets the attention!) Thanks for the comment!
It's about seeing whether the experiment is described well enough to know whether you can replicate the results. I.e., not "can the results be replicated" but "can the experiment be replicated." It prevents papers that are so bad that you can't even figure out whether they're good or bad. It doesn't prevent bad papers per se.
I feel this is a much wider problem than in just the Sciences. Academia in general seems beyond saving at this point. The Humanities fell a long time ago. Yes Mark, peer review has to go. Open Web Mind, all the way!
Yes, I completely agree. Platforms like arXiv are the future of scientific publishing. Not to mention RUclips, podcasts, the open web… let a thousand flowers bloom!
@@lasttheoryDoesn't this expose a basic flaw in your argument then? No young scientist needs "permission" to put their work on ArXiV. If peer review is not forced on anyone to publish their work, then why do we need to "end it"? As it stands, educated readers can choose to either give content in peer-reviewed journals priority and trust, and read that first, or they could choose to ignore that and just read randomly selected articles from a preprint server, or blogs, or RUclips. Many probably do just this, though I imagine some use peer review to prioritize their time. I think it's inaccurate to hold peer review to the standard of being a perfect system to prevent bad content getting published. It is simply an established, though imperfect, way of putting an indicator of quality against a work of research prior to actually reading the content. Surely this is merely curation - which you are free to disagree with and of course should not be considered definitive - but not really censorship in any sense?
@@billyaxon I agree, peer review is not censorship. Anyone can publish anything on the web, at arXiv or anywhere else, and that's a seriously good thing. But what if peer review, beyond being merely imperfect, actually does more harm than good? What if it gives too much credibility to papers that are deeply flawed, while preventing much better research from being considered? Wouldn't you want to end peer review if that were the case?
Dang. Compelling arguments. Even the mRNA lady who won the Nobel prize was lambasted for a decade before her ideas where excepted. Let's not forget poor Boltzmann 😭
Thanks, and yes, it's strange to see that peer review is proving so counterproductive, but even stranger to see the scientific community being so unscientific in their assessment of its merits!
You have a point that peer review has issues, but mostly you're attacking a straw man. You're description of how it works is wrong. As long as you're willing to publish in smaller journals, your ideas will get published as long as they are technically sound. The big shots defending their theories only happens in the big journals because the big shots don't review for the lesser ones.
Right, yes, that's a good point, that you can usually publish in smaller journals even if the bigger ones block you. I'm trying to make a wider argument, though. Ideas like Stephen Wolfram's framework for physics simply don't fit into the paradigm by which scientific papers are judged by journals large _or_ small. If he had submitted his introduction, let alone _all_ the material he published on the web, to _any_ journal, it would have been rejected. This lends a patina of respectability to small ideas that are publishable and inspires suspicion of bigger ideas that aren't (I can' tell you how often I've heard "It hasn't been peer-reviewed" as a reason to reject Wolfram's ideas wholesale). And it's the paradigm-breaking ideas like this that are the _most_ important to get out there.
Nobody is preventing you from publishing your papers exactly as Newton and Einstein did. If you're having tired old scientists pay your research, and you're coming up with stuff that most of those tired old scientists disagree with, then yeah, you're going to have trouble. Go do what Newton and Einstein did - get a job, and do your science on the side. And while Wolfram's stuff is very cool and might indeed prove to be a true model of the universe, but I'm not sure it's "science" yet. As far as I can tell, it's not even to the point that String Theory is.
I think we agree on these things, Darren! You're right, anyone can publish anything on the web, and that's a good thing. You're right, anyone can do science on the side. Indeed, Stephen Wolfram is doing exactly what you suggest: he has a job, as CEO of Wolfram Research, and he's doing physics on the side. And yes, there's a long way to go before Wolfram Physics makes novel, testable predictions, but I think it's way ahead of String Theory, at least in the sense that it would survive Occam's Razor.
@@lasttheory I think String Theory made predictions, but the predictions just wound up not matching measurements. It'll be fun to see what Wolfram is doing come to fruition. I liked the idea since NKOS and I've seen a couple of science fiction stories based around Quantum Graph Theory (as it's called there). (Greg Egan being the primary author of those, IIRC.)
Asking as an outsider, is this one of those things where there's a near-consensus but nothing ever happens? I mean how much of your life have you wasted listening to "Dr Krgebgk Odnfbg is the distinguished Monsanto Koch Brothers Fraggalicious Chair of Gardening. At Princeton they..." That's never gonna end. I always found it hilarious with something like physics because the person at the podium is very obviously qualified to be there. They act as though they're trying to convince an audience of hardcover skeptics and debunkers
Yes, I kinda think there's too much consensus in science today. It'd take a real mindset shift to allow anyone to publish anything, but I think it'd be really healthy.
The machine learning systems I've used have been a little superficial. Sure, they'll detect unscientific _language,_ but the point is to detect unscientific _research._ Do you think AI can get there?
@@lasttheory Absolutely. AI can already reason. Within a few years AI will be able to read and understand the paper, run the calculations, make suggestions, pulling from its vast memory.
@@aunumeverAI cannot reason beyond whatever can be easily deduced from the prompt. It can show a proof of the Pythagorean theorem, with no guarantee that it's correct, but it can't construct a novel proof. AI that can reason should be able to prove theorems it hasn't been trained on. No AI can do that and nothing even comes close. I share your optimism but LLMs are either a dead end or the first step of many many more. I think we're a long way away from AI that can understand things.
I think there's something to be said for a bit of the Wild West in science! Proving ideas requires precision; coming up with ideas in the first place requires imagination!
@@lasttheory Definitely I do agree with the proving/imagination statement. The "Wild West" paradigm is fine as long as one doesn't "decree their stuff as gospel."
@@lasttheory "Proving ideas requires precision; coming up with ideas in the first place requires imagination!" Correct. Real scientists have both, and you have neither.
@@nataliearcadia Ah, but I'm _not_ a real scientist! I'm not a scientist at all. I'm a writer and a humble maker of videos, trying to make sense of all this. With a _little_ precision and a _little_ imagination, I hope.
I do get the point, but I dont think it is enough to suffocate science as a whole. As you rightly said in the video, people still can get their ideas out on the web using youtube, or blogs or other means. And if the ideas are good enough, people would surely notice them and it would get attention.
Yes, you’re right, Jwalin, people _can_ get their ideas out there on the web, and that’s a wonderful thing. So much better than the pre-Internet days of Einstein!
But there’s still extreme prejudice against anything that’s not peer reviewed. If peer review were abolished, or at least dismissed as behind the times, then truly new ideas wouldn’t be at such a disadvantage.
I’ve seen this repeatedly with Wolfram Physics: scientists and reporters, even bloggers and podcasters, dismiss it out of hand, complaining that it’s not peer reviewed. This failure to be open to these new ideas will set The Wolfram Physics Project back maybe 10 years.
It’d be good to change this closed-mindedness.
Thanks for the comment!
@@lasttheoryyou're right, but since most papers didn't contain enough information to judge the validity, a wave of plausible looking but essentially fake stuff would hit us. Per review doesn't prevent fraud, but it makes it much harder, because experts of that particular field look at it. Who would bother to look at tons and tons of non reviewed papers?
@@testboga5991 These are good questions, thanks. If you shift your thinking to a post-peer-review world, things look very different.
First, we wouldn't have the false positives of: "It's been peer reviewed, so it must be _kinda_ right." I don't think peer review merely fails to catch fraud, I think it _enables_ fraud by adding a false stamp of approval to papers that really haven't been checked.
Second, your question of who would bother to look at untold numbers of non-peer-reviewed papers could be applied to peer-reviewed papers, too. There are far too many published for anyone to keep up with, so you have to be selective.
Think ahead to that post-peer-review world, and it's like asking: who would bother to look at untold numbers of RUclips videos? No one would. There are far too many RUclips videos for anyone to watch them all. But no one would need to. There are mechanisms for surfacing the good ones: algorithmic mechanisms, crowdsourcing mechanism, and the good old-fashioned mechanism where your trusted colleague at another university tells you that you really should take a look at this new theory. That's the way it currently works with peer review, and that's the way it'd continue to work post-peer-review.
Science is not the only field plagued by the primacy of the status quo and the suffocating gate-keeping of certain academics. Join a writers' group and listen to 13 people describing how they would have written your short story, and how much better it would be. When everything is reducible to money and when the only pathway to success is through a group of staid, egotistical bumblers, then we're all in serious trouble. (Ed Witten, I am not talking about you.)
Ha, yes, thanks Greg! That's a great point of comparison: I've always avoided writers' groups like the plague!
1:40 This isn't peer review, this is regulation. 1:22 This is peer review.
Yes, exactly. Peer review is, well, your peers reviewing your work. That's a good thing! The formalization and entrenchment of today's officious brand of peer review is not.
Well, isn't it almost happening? I mean as you know there are multiple rather science online outlets you can publish your ideas without peer review. Also, there is the Wild Wild web where you can put out anything, and if it sticks, well it does. The remaining thing is the idea of "recognition" and "credit" to receive for your resume and career advancement. Well, that does come around these days through your viewership online! Isn't it the new peer review? The only problem is that most of these peers are not really that educated, but hey, they can have opinion by their power of clicks. Anyhow, I am saying traditional peer review is only useful for traditional media. Once someone is established his/her expertise through a formal Ph.D. degree, can publish pretty much anything, either traditionally peer-reviewed or peer-reviewed. What is left is the power of prediction and explanation of an idea. If someone can explain things that are either not explainable in current theories or does it in more straightforward way with less arbitrary parameters, then it really doesn't matter these days where that idea has been published first.
Yes, absolutely, it is sort of happening! I've found there's still extraordinary acceptance of peer review, though, and prejudice against anything that hasn't been peer reviewed. I've received many messages telling me that Wolfram Physics must be dismissed because it hasn't been peer reviewed, rather than on its merits. I hope this changes. Thanks for your thoughts, Reza!
Beautifully stated! I (for the sake of the point of the video) was met with somewhat encouraging and scathing rebuttals to my own thoughts on the universe when i originally posited them to arivx, before (ultimately) anyone with an email who I thought may understand (based on their area of research, i.e. game theory and beyond) what I was suggesting. My ultimate conclusion; based on my own social status and other variables was that these things are maximally and minamally seperated based on a number sociological variables. I think that aligns sadly or inconsequentially, hilariously or serendipitously with that which you are exporing. It is all (dramatically), none the less true! Godspeed my good friend!
Thank you! It's good (and instructive) to hear your own experience with this.
LK99 superconductor past peer review, but web boffins were fairly quick in disproving it.
I don't think the LK99 superconductor paper did pass peer review, Roger. I think it was published on arxiv.org without any peer review.
But that makes it a good example of how science could work _without_ peer review. As you say, the room-temperature superconductivity claim was reasonably quickly disproved.
Seems like this is exactly the way things _should_ work!
This is the way.
The way for cranks and grifters, yes.
I thought peer review was about seeing if the results can be replicated? You do have to have results that can be replicated in the first place though.
Sadly, no, it's not about replication. Research is rarely replicated, since there's little incentive for scientists to re-do research from scratch (and get very little attention, either for proving or disproving an existing research), only to do original research (that's what gets the attention!) Thanks for the comment!
It's about seeing whether the experiment is described well enough to know whether you can replicate the results. I.e., not "can the results be replicated" but "can the experiment be replicated." It prevents papers that are so bad that you can't even figure out whether they're good or bad. It doesn't prevent bad papers per se.
DARN RIGHT. The whole notion of this post is nonsense.
I feel this is a much wider problem than in just the Sciences. Academia in general seems beyond saving at this point. The Humanities fell a long time ago. Yes Mark, peer review has to go. Open Web Mind, all the way!
I hear you. Thanks, as ever, for the comments!
I got a kick out of that post...
@@gregvondare Not too painful, I hope?🤭
@@Terpsichore1 -- I was indulging in a personal joke. Since Terpsichore is the muse of dance, getting a "kick" out of her... Well, you know.
@@gregvondare Gotcha! Nice. Not much “kicking” in my line of dance, so went right over my head. 🩰
You can always put your paper in the arXiv, it's getting so called publish. If it's a good paper, it's a good paper.
Yes, I completely agree. Platforms like arXiv are the future of scientific publishing. Not to mention RUclips, podcasts, the open web… let a thousand flowers bloom!
@@lasttheoryDoesn't this expose a basic flaw in your argument then? No young scientist needs "permission" to put their work on ArXiV. If peer review is not forced on anyone to publish their work, then why do we need to "end it"? As it stands, educated readers can choose to either give content in peer-reviewed journals priority and trust, and read that first, or they could choose to ignore that and just read randomly selected articles from a preprint server, or blogs, or RUclips. Many probably do just this, though I imagine some use peer review to prioritize their time. I think it's inaccurate to hold peer review to the standard of being a perfect system to prevent bad content getting published. It is simply an established, though imperfect, way of putting an indicator of quality against a work of research prior to actually reading the content. Surely this is merely curation - which you are free to disagree with and of course should not be considered definitive - but not really censorship in any sense?
@@billyaxon I agree, peer review is not censorship. Anyone can publish anything on the web, at arXiv or anywhere else, and that's a seriously good thing.
But what if peer review, beyond being merely imperfect, actually does more harm than good? What if it gives too much credibility to papers that are deeply flawed, while preventing much better research from being considered? Wouldn't you want to end peer review if that were the case?
Dang. Compelling arguments. Even the mRNA lady who won the Nobel prize was lambasted for a decade before her ideas where excepted.
Let's not forget poor Boltzmann 😭
Thanks, and yes, it's strange to see that peer review is proving so counterproductive, but even stranger to see the scientific community being so unscientific in their assessment of its merits!
You have a point that peer review has issues, but mostly you're attacking a straw man. You're description of how it works is wrong. As long as you're willing to publish in smaller journals, your ideas will get published as long as they are technically sound. The big shots defending their theories only happens in the big journals because the big shots don't review for the lesser ones.
Right, yes, that's a good point, that you can usually publish in smaller journals even if the bigger ones block you. I'm trying to make a wider argument, though. Ideas like Stephen Wolfram's framework for physics simply don't fit into the paradigm by which scientific papers are judged by journals large _or_ small. If he had submitted his introduction, let alone _all_ the material he published on the web, to _any_ journal, it would have been rejected. This lends a patina of respectability to small ideas that are publishable and inspires suspicion of bigger ideas that aren't (I can' tell you how often I've heard "It hasn't been peer-reviewed" as a reason to reject Wolfram's ideas wholesale). And it's the paradigm-breaking ideas like this that are the _most_ important to get out there.
Nobody is preventing you from publishing your papers exactly as Newton and Einstein did. If you're having tired old scientists pay your research, and you're coming up with stuff that most of those tired old scientists disagree with, then yeah, you're going to have trouble. Go do what Newton and Einstein did - get a job, and do your science on the side.
And while Wolfram's stuff is very cool and might indeed prove to be a true model of the universe, but I'm not sure it's "science" yet. As far as I can tell, it's not even to the point that String Theory is.
I think we agree on these things, Darren! You're right, anyone can publish anything on the web, and that's a good thing.
You're right, anyone can do science on the side. Indeed, Stephen Wolfram is doing exactly what you suggest: he has a job, as CEO of Wolfram Research, and he's doing physics on the side.
And yes, there's a long way to go before Wolfram Physics makes novel, testable predictions, but I think it's way ahead of String Theory, at least in the sense that it would survive Occam's Razor.
@@lasttheory I think String Theory made predictions, but the predictions just wound up not matching measurements. It'll be fun to see what Wolfram is doing come to fruition. I liked the idea since NKOS and I've seen a couple of science fiction stories based around Quantum Graph Theory (as it's called there). (Greg Egan being the primary author of those, IIRC.)
Many good points! I'll be following the Open Web Mind project.
Thanks Neale, that's great! I'll be launching my new Open Web Mind channel over the next couple of days. Really excited to get this out there!
Asking as an outsider, is this one of those things where there's a near-consensus but nothing ever happens? I mean how much of your life have you wasted listening to "Dr Krgebgk Odnfbg is the distinguished Monsanto Koch Brothers Fraggalicious Chair of Gardening. At Princeton they..."
That's never gonna end.
I always found it hilarious with something like physics because the person at the podium is very obviously qualified to be there. They act as though they're trying to convince an audience of hardcover skeptics and debunkers
Yes, I kinda think there's too much consensus in science today. It'd take a real mindset shift to allow anyone to publish anything, but I think it'd be really healthy.
AI will be huge in the peer review process.
The machine learning systems I've used have been a little superficial. Sure, they'll detect unscientific _language,_ but the point is to detect unscientific _research._ Do you think AI can get there?
@@lasttheory Absolutely. AI can already reason. Within a few years AI will be able to read and understand the paper, run the calculations, make suggestions, pulling from its vast memory.
@@aunumever Strange times... I'm fascinated by the intersection of AI and computational physics. We'll see what it comes up with!
@@aunumeverAI cannot reason beyond whatever can be easily deduced from the prompt. It can show a proof of the Pythagorean theorem, with no guarantee that it's correct, but it can't construct a novel proof. AI that can reason should be able to prove theorems it hasn't been trained on. No AI can do that and nothing even comes close.
I share your optimism but LLMs are either a dead end or the first step of many many more. I think we're a long way away from AI that can understand things.
That's well stated, thanks Alex. I wish everyone understood that LLMs are, well, large language models, next-word predictors, not AIs.
"Jesus f#@ing Christ!" Did somebody pay you off? LOL 🙂It seems that there's a new paradigm in town and it's called the "wild west" approach.
I think there's something to be said for a bit of the Wild West in science! Proving ideas requires precision; coming up with ideas in the first place requires imagination!
@@lasttheory Definitely I do agree with the proving/imagination statement. The "Wild West" paradigm is fine as long as one doesn't "decree their stuff as gospel."
@@williamschacht Yes, absolutely, I agree, thanks William!
@@lasttheory "Proving ideas requires precision; coming up with ideas in the first place requires imagination!" Correct. Real scientists have both, and you have neither.
@@nataliearcadia Ah, but I'm _not_ a real scientist! I'm not a scientist at all. I'm a writer and a humble maker of videos, trying to make sense of all this. With a _little_ precision and a _little_ imagination, I hope.