Is there a reproducibility crisis in science? - Matt Anticole
HTML-код
- Опубликовано: 12 сен 2024
- View full lesson: ed.ted.com/less...
Published scientific studies can motivate research, inspire products, and inform policy. However, recent studies that examined dozens of published pharmaceutical papers managed to replicate the results of less than 25% of them - and similar results have been found in other scientific disciplines. How do we combat this crisis of scientific irreproducibility? Matt Anticole investigates.
Lesson by Matt Anticole, animation by Brett Underhill.
Psychology is the first thing that came to mind. Alot of experiments can't be reproduced, but students are still learning about them in schools like they're valid.
Fucking Freud.
theguyshadows I know right.
EveryDayIsAGoodDay I'm a psychology student myself man! Every damn experiment we learn is made by some old guy, the experiment is never reproduced and bam we take it as it is. To be honest psychology is not critical at all. Not sure about the other science fields though, maybe you're right, but I'm speaking from experience.
Freud didn't do experiments. He just made theories and because his theories were so important to the development of the field of psychology as a whole, we still learn about him. Nobody pretends that his theories have any scientific validity, just that they have historical importance I guess. I think it makes sense to learn about Freud himself and how he had an influence on the history of psychology, but it's such a waste of time to dedicate so much of psychology classes to learning about the details of his theories. Same with Erik Erikson and Carl Jung, they all just sat there and came up with theories.
***** I would love to see the scientific evidence that supports Freud's theories and why this evidence compels you.
2:11
If a study cannot be reproduced because it is unclear what the original study was or what the original study group did (as suggested in the video) then the original study should not be considered scientific.
More like, the study can not be considered entirely valid, until reproduced/verified. If it's of any interest and hasn't been verified yet, then whoever is interested in making use of it should be responsible for reproducing results. There isn't and never was any other way.
Klin-Klin No the point is reproducing results is impossible because it is unclear how the original experiment was undertaken.
When they say it's unclear what the original study did, that usually happens with the very specific details. As an example, a published paper could describe that researchers brought group 1 into the lab every morning and fed them 40g of raw broccoli and repeated this every day for 14 days. But we don't know what time of day this was, whether the participants were told not to eat before or after, whether they drank water with it, and other details that seem irrelevant but may effect the reproducibility of that study.
+monkiram Thats why its very annoying to me when I talking about studies to other people. There is many people that takes the study who not gives any those mini important details and people holds them as a solid fact.
agreed
universities should make courses for first and second years students to replicate studies.
this way students can have hand on experience in doing research and those research would be verified.
Sami Mas oh yeah this is already happening - I've spent sooo many hours in the lab and for many other degrees it's a daily thing! Although, a lot of times at university level it tends be heavily based on the previous work of current professors and I'm not entirely convinced they actually take our data that seriously
potatoes
kenichi 2233 mashed or roasted?
Roasted
kenichi 2233 i can have some too?
This is outstanding and very important. Thank you for doing it. Unfortunately, publishing negative results, if you can even do it, is not likely to lead to grant funding. Without funding, you have no science. Thus, negative studies or confirmatory studies to confirm reproducibility will continue to be underemphasized (and typically just lacking) in our current model of science and research
I honestly think we should find ways for grant funding to be decoupled from the research at hand. Maybe publishing methods and having those methods approved by peer institutions would be a good way to get funding.
Perhaps each time new research gets funding several other institutions get funding for their reproducibility trials automatically.
@@danielsykes7558 You are very sweet but apparently naive. Who is it that provides the funding? The corporations and pharmaceutical companies. Why would they incentivise / fund study that will cause their newest product to be NOT be produced and sold?
@@EtreTocsin he is not naive at all - it’s a case of regulation and law.
@@EtreTocsinit would literally have to be the government. It exists to brute force people into doing things for the public good when there is no personal incentive.
@@EtreTocsinfree market can’t solve the worlds problems
"If you thought science was certain, well, that was just an error on your part." ~Richard Feynman, scientist, Nobel Prize winner
The media and the educational system always makes it look like it's certain. I never knew richard feynman even said that. It isn't that popular of an opinion
But this shouldn't be an excuse to defend soft sciences and diss hard sciences. Thanks to hard sciences, we are able to put man on the moon.
@@cagribaba4464 And thanks to hard science, we are now in an irreversible course towards climate change. Your point?
Can't emphasize enough how well done andto the point this video is.
And it's true - it really comes down to the fact that there's no motivation in the scientific community to cross-verify anything. As said, it's actually a detriment to one's career to do that, as it's a delay to career progress. Conferencies and journals are simply not interested to publish validation studies.
0:39 Idk why but the facepalm cracked me up😂
Potato Shadow It was perfectly timed. Props to the animator. 😃
It is very relatable, isn't it?
me to. it was hilarious!!! i watched that part like 5 times...:p
anyway, good thoughts in this video...some things to think about...
Thank you for not focusing solely on psychology. This is a problem for all emperical research.
Thankfully math related subjects are not affected
Comp Sci master race
@@tannergordon8302 I keep seeing adverts that misrepresent sociological ideas as computational ones. There's a whole industry definable solely as often claiming - without explaining why - "building computers that function as advertised has ceased to be feasible". Its "work" advances outward from that point. It's a whole industry. It's called "cyber-security" like 1984. Its the activity definable as "applying the belief that it doesn't to exist". CompSci bods are mostly going to have a really surreal thousand years navigating the treacheries and retardations of that "industry".
Psychology doesn't even qualify to be part of the problem.
@@wliaputs Actually, they are. Some errors in proofs are ridiculously hard to spot. There's definitely more than one false paper out there that nevertheless managed to pass peer review.
Conclusion: Take EVERYTHING we read with a grain of salt.
And truth will always find a way, sooner or later.
That's too broad. I'd say check the sources credibility and research the issue before you come to a conclusion, otherwise you're inferring that all information is equally valid which isn't true.
I think it's "a pinch of salt"
Micah Miranda I am gonna take that with a grain of salt if you don't mind 😋
even the most trusted sources can go a bit overboard sometimes as the vid suggest
Good topic, great points brought up... but needs TONS MORE DETAIL!
For instance, on the subtopic of improving reproducability alone: How can we incentivize more raw data availability? Why isn't it done already? Why don't papers detail their techniques sufficiently already? Why why why why why!?!? DETAILS PLEASE!!!!!
Because it's a corrupt system that scientists don't want to meddle with. Everyone knows what's going on, scientist have the cheese and the knive to solve the issue. But they won't and you won't hear many details anywhere.
Because they don't want their idea stolen and someone else claiming credit for it. And because if general public can replicate and use it on their own then the one with the initial discovery won't make bank. Greed.
Higgins2001 They do publish papers with very descriptive techniques. Not that many read their papers, besides others in their field. Thus the manner of speaking is more precise yet somewhat confusing to the untrained.
+Higgins2001 The full lesson can be found in the description.
If they give all the incite of their ideas, how do they make money from it, and whats the incentive for others to chase other discoveries. Many scientist love their work but money makes shit happen.
Team work is important
It helps to put the blame on someone else :
agree?
The fame, funding, and career maintenance motives certainly bias the way work gets done and produce problems. I'm starting to wonder if we'll look back at today like we do the middle ages. The church used to fund most research. Nowadays it seems to be corporations that fund most of it.
Honestly, science should be curious people coming together, and no one should have their livelihood based on certain results. This requires that people everywhere have more freetime and that we have strong peer review institutions.
We need perhaps more funding to duplicate research rather than to do original research. And that funding should be independent of corporate interests.
Easy solution: Get Honours students, Masters students or students in their late studies to prove or disprove recently published papers as one of their assignments... and then make Mexico pay for it.
We were actually assigned to do this by writing a Critique on a peer-reviewed published article as part of our Academic Writing course.
The thing is, it's hard to "prove" or "disprove" anything in research papers because scholar papers themselves aren't trying to "prove" anything. They only suggest that their hypothesis or "theory" is *probably* correct by showing a high probability (using statistical methods)
No paper will include a full list of their collected data/calculations; instead they conclude with their calculated results. You can only do so much with these results, and as a reader will have to assume that the researchers aren't lying/didn't make a mistake.
2:33 anyone else think of a trump reference
+Z zz lol, Ur joking right, right?
It was actually part of my first year's assignment in Psychology to reproduce the result a famous study that was shown to accept the alternative hypothesis :)
Making it an "assignment" as if they were some high school students would just have them go through the motions. Why waste such time and effort on something you're not passionate about?
0:56 that guy went on to create the powerpuff girls
I thought the same thing
“I think that we shall have to get accustomed to the idea that we must not look upon science as a ‘body of knowledge,’ but rather as a system of hypotheses, or as a system of guesses or anticipations that in principle cannot be justified, but with which we work as long as they stand up to tests, and of which we are never justified in saying that we know they are ‘true’ or ‘more or less certain’ or even ‘probable.’ -Karl Popper
Honestly, the best thing to learn is how to remove or even just heavily suppress your personal biases(Theological, Political, Racial, ect,..) when conducting Scientific research of any kind.
that would be very ideal, but in reality, I don't think that would be possible. I'm no physiology major, but hat's just my opinion
When conducting research? Thats what review boards, blinding, and meta analyses are for.
It's not really a significant problem is science so i don't see why you would say it's the "best thing to learn"
Personal bias is bad enough, but there's also money being dangled out there by organizations who _only_ want their biases validated.
@@crimsonmask3819 yes. Also, this comment reminds me of the quote from the Three-Body Problem: "Should philosophy guide experiments, or should experiments guide philosophy?"
I had seniors in lab who wouldn't publish exact measurements of their data so no one could replicate it, in case they needed to sell it for patent.
If all "government funded" research was forced into the public domain and researchers couldn't patent said research, a lot of these problems would go away.
I remember verlisium did a video on this. worth a watch for more detail.
Sophie Matthews yup
Veritasium, yeah.
Juan Pablo Mina
yeah couldnt remember the name lol
link??
+Sophie Matthews Veristablium
Looks like the Powerpuff Girls got a new plant.
Powerpuff Girls vs Chomper...
Hi Ted ed, I hope you see this. I wanted to suggest that you cite the resources you use in the description box as many of your viewers are students probably researching for a paper or something. It would be helpful to get the resources so that we can cite them too. Thank you!
Psychology gets a bad wrap because it's more of a heuristic. The brain is very complex and hard to understand. When a psychologist seeks to understand human behavior it's like trying to figure out the inner workings of a complex math equation knowing only a few inputs and an output. They do good work though so take it easy on them, they have a very difficult job.
Unfortunately, I find that many that go into that field is underqualified to begin with. Those that really succeed (the 1%ers) will most likely succeed in other areas of study, but can't say the same with the others (99%ers).
This is a biased opinion though so lol
I find some Irony in a video talking about potential dangers of non-replicated scientific studies...that uses scientific studies to back up its argument like at 3:25. I don't know what they would use instead but I still find it a little funny.
1:08 is that samurai jack
Mystery Man No, it's Professor Utonium!!
TJ Simmons they are the same person . You know the devil in the power puff girls ? He is the reincarnation of haku in samurai jack
If you're not questioning science, you're treating it like religion, not science. Scientists are not gods and we should not have faith in them.
well said.
Exactly. Too often it is forgotten that skepticism is supposed to be central to the scientific method.
And i love that folks never doubt "science" yet theres plenty wrong.
Have a better method?
@@ExatedWarrioryes, the scientific method, which is better than blindly trusting everything that gets "science" stamped on it
Question everything
I always wondered if there should be a Federal Beaural of Scientific Integrity which would go through and look at all published papers, in the USA at least, and help flag any that are outstandingly false by replication if possible or by review.
Collin Bruce Unfortunately that is subjected to bias by who is checking. Ex. Human induced climate change, people tend to look for biased research inline with their views/agenda as some sort of proof.
you really don't wanna do that, You can corrupt people easily, especially when you give them power to decide which scientist is getting funding that year.
See this is criticism I need to hear.
If you're curious, the Italian lab is under the Gran Sasso mountain in Abruzzo, Italy. I live a hundred kilometers from that lab.
"Real science is more than just static textbooks"
Tell that to every science teacher in elementary and middle school
Exactly! Even my high school chemistry teacher was so dogmatic about methods and style. I remember when she taught us how to calculate Formal Charge and Lewis dot structure problems, she just messed up and then ended the class saying do 4-5 problems from the text you'll understand. 😆
It's a bit stunning how this video glosses over how severe this issue is. Whatever the motivations, this calls to question the entire body of modern science, all that is based on prior papers. In law, there's the concept of the fruit of the poisonous tree. This is far worse because nobody has been minding the store for a great many years, while we've been assured it's all super sound.
Whenever there is a need for certain results, the science will usually go out the window. The answer in our predicament is to always look for several replication studies of any previous conclusion. Most people don't care enough and most youtube videos don't include this either
Great animation. Would watch again just for that.
we all need to show more love and compassion for our human counterparts
While reviewing a paper, I was literally told by a senior scientist: "Don't correct that much and don't be so strict. Otherwise our own papers will get a lot closer look and more criticism". Combine that with the fact that our institute (part of one of the largest scientific societies in Europe and in the world) has a papers per person quota and you get the point...
Funny thing is, chasing KPIs in the real world (companies) has been proven to be really bad. I wonder when the academic society will realize that. Then again, many people there (especially the old farts) never worked anything practical and remotely useful in their entire life. Their whole existence will be put in jeopardy. We can't have that!
for science as in (physics, math, chemistry and biology) there is no such thing as risk of losing time, as physicist and mathematicians who created concept of time and is not even subjected to the change of anyone's liking and just say they are not valid without even studying them. getting weird results is expected as scientist keep their curiosity and ambitions in understanding the world while these people who say science is not valid are just people who may just be jealous of the level of smartness and criticise science without even understanding them. this contradictory behaviour is just toxic and non-productive whereas those who say maths is not useful and inaccurate are just being ridiculous.
So isn't the reproducibility issue, in part, cancelled out by the fact that if new studies are build upon incorrect older studies, eventually the results don't add up and you can track it back to what may have been false? Sure it's a waste of time, money and other resources but it does prevent false information from being acted upon in the long run (?)
2:50
I understood that Popeye reference!
Anyone else notice the Little Shop of Horrors reference with the carnivorous plants on the White House?
My chem teacher told me neutrinos can travel faster than light smh
Any sort of medical study is just in it's own separate paradigm of accuracy from the rest of science. There are just far too many unknown factors, as well as the giant fatal flaw of relying heavily on people's testimonials about their condition and past history.
...And this is why I like maths. Prove your theorem in maths, and it generally stays proved. Although statisticians do like to wobble things :)
In theory, yes. But from time to time there are errors in proofs that are ridiculously hard to spot. There's definitely more than one false paper out there that nevertheless managed to pass peer review.
How does mathematics ever "prove" a scientific theorem (other than a math theorem)? Math can be used to describe observed data or behaviors, but I don't see how it ever proves a theorem.
except Flossing. we never check that
Then how can we be sure to trust the scientific literature?
Sadly, most literature these days says whatever the source of funding wants it to say and not what real science dictates.
Look at the money. Don't trust a pro-meat paper funded by McD. Also don't trust a paper funded by Beyond eat.
The tricky part is that they usually hide behind a proxy organization, so it's not that easy to spot.
so, the takeaway is that we should encourage reproducibility, not that we should disregard science as a useful tool for learning new things.
Math: I don't have such weaknesses
Maths has the weakness of human error and understandability, just like anything else.
This is why it bothers me when people worship science as a god. Science is only as good as the scientists that use it and well you know humans.
How would you verify hypotheses if not through the scientific method?
The scientific method isn't foolproof. It's manmade formula that exists to try to explain everything. In the end, most of it is all a guess. The big clue is manmade. The human intellect is highly limited in the grand scheme of the universe and the unknown.
Heavy Gun Blade
Agreed, the scientific method has its limitations. However, the backbone of science has been the scientific method. The reason we live in such a technologically advanced world is because it has been the modus operandi of research and development. So, how would _you_ verify hypotheses if not through the scientific method?
*****
Induction and deduction are important elements in the development of hypotheses, and so are important elements of the scientific method as well. However, the scientific method has the additional rigor of testing and experimentation.
At NASA, every component is put through a battery of stress tests, each system is tested for proper functioning in the conditions expected before launch. This is important because there are often phenomeon that were previously unknown that need to be taken into account. If a person suggested launching a spacecraft on nothing more than the logic of induction and deduction, they'd be laughed out of the organization.
Induction and deduction are important elements in philosophy, but without testing and experimentation, their conclusions can't be verified.
The scientific method isn't perfect but it's the best method we got.
this is why we need to start teaching newer and latest textbooks than the 10 -20 year old science textbooks in high school
This would seem to be a reason not to rushing the results of experiments and put them into text books. Really, there is very little from the past 50 or even 80 years that need to be in a high school text book. I am reasonable sure that "the lack up to date text books" is the reason why US high school science is crap.
Should start teaching ancient knowledge since they weren't dumb enough to believe people stuck to the bottom of a sphere are somehow not standing upside down.
maham meher all your doing is feeding the textbook industry. They do revisions but they're either reshuffling, adding more pictures, lastly adding new content worth a few pages. Don't believe me? Try buying a college textbook a 3rd edition isn't that different from a 5th edition..
Really interesting: Everyone can make errors... But we shouldn't discourage ourselves because of that! Thomas Edison once said: "I didn't fail, I just found 10000 wrong solutions"
and then he nicked all of Tesla's ideas and made them his own ...
@@lakestreet3951 true
1:59 Genius.
Flourishing always follows after crises
Very important.
This is outstanding and very important. Thank you for
doing it. Unfortunately, publishing negative results, if
you can even do it, is not likely to lead to grant
funding. Without funding, you have no science. Thus,
negative studies or confirmatory studies to confirm
reproducibility will continue to be underemphasized
(and typically just lacking) in our current model of
science and research
I always wondered if there should be a Federal
Beaural of Scientific Integrity which would go through
and look at all published papers, in the USA at least,
and help flag any that are outstandingly false by
replication if possible or b review.
Team work is important
It helps to put the blame on someone else:
agree?
Like the Little Shop reference...
Unfortunately it is quite common in science for results to be non reproducible. As a biological scientist myself I have seen several cases of that. Interestingly it depends on the publishing scientist and the group behind that scientist. For example pretty much all of the results of Canadian researchers I have been working on were plausible and reproducible to some extent. On the other hand I did work on a few cell biology projects based on an Austrian researcher Manuela Baccarini from the Max Perutz Labs in Vienna. Her work was a classic example of non plausible research and likely fraud.
So it depends on the field of research and the underlying papers.
Woah! I've seen this too. Thanks to PubPeer, some karma is being dished :)
With the study done in 1998, you have to consider that only like 3 years later, Jan Hendricks Schön’s fabrication of most of his papers was discovered which forced the peer review processed to be reviewed itself. It made the process significantly more rigorous but also more accurate.
No replications published, huh? The greatest physical law is, I propose, the second law of (classical) thermodynamics, expressed as " it is not possible to construct a system that exchanges heat with a single reservoir and delivers net work". The classical second law depends exactly upon reproductions of experiments in which the failure to produce net work is confirmed.
Now, in number theory, Colatz' conjecture is considered to have been proven because billions of sequential calculations from the initial number (index) have all led to 1, which is what Colatz conjectured. Clausius ' mode of thought lives!
Charles A Berg
I dont see nobody talking about episthemology and that is a shame :( the main problem (or at least from my point of view) is in the reliability of the scientific mehtod. Maybe we should criticize and rebuild what we think it is the scientific method.
A big part of the scientific method is reproducability. The problem is we're not using that part enough.
1:59 is just cruel to headphone users, haha. I paused the video to kill that fly before I realized it wasn't real...
loved the concept on which is the video of Based, very nice video
We, as a society, need to get out of the arrogant mindset that we have it all figured out. We continue to make incredible advancements in science and technology, but when considering things like this, we still have a small window of confident knowledge about the world. Long story short, everyone from our scientists, to our politicians, to our common readers and youtube-watchers needs to stay teachable and be ready to admit when we are wrong. Let's write our beliefs "in pencil", not in stone.
Does anyone know where I can find newly published science papers??
Do you have access to any journals, like through college? If so, Web of Knowledge should work, if not, .... well, Sciencehub is love, Sciencehub is life
Ohhhh thx
arxiv
+Technition
If you can't find them, you'll certainly not be able to understand them.
Frank Schneider that is straight up BS
The publish or perish model is the main root of this problem. The funding of govertnment is the second main causr
Nice Little Shop of Horrors reference :D
0.35 i know that feel
at 1:02 I spot an Audrey II!
1:58
Literally I'm so stupid. I smacked my headphones off because I thought there was a mosquito near me lmao
***** Especially when it's night and there's no light to see it.
Love the animation!
Perhaps there should be a government initiative that solely funds replications of important studies.
Oh good! More government waste and graft!
Providing funding for original research is actually one of the things that government is good at.
First, no, no it isn't. In fact, it's horrible at it, and causes more waste and graft.
Second, and I understand that you likely don't care about such things, but it isn't one of the jobs the founders set out for our Federal Government.
Third, are you one of those "Without the Government, you couldn't build/invent/create that" crowd? Because if so, there is no further point attempting to have a dialog with you, as if so, you do not care about the reasons a system was set up, and why it is important to stick to the principles the system was founded on, thus you will dream up all kinds of other tasks the Federal Government was not supposed to stick it's nose into.
"Second, and I understand that you likely don't care about such things, but it isn't one of the jobs the founders set out for our Federal Government."
Which country are you from?
Why, the same one that gave the world the tool we are now using to communicate, of course. Thank you for affirming my supposition that you do not care about the jobs set forth for my Government by it's founding fathers.
...and no, the irony of the fact that DARPA did a lot of the groundbreaking research to develop the inter-webs is not lost on me, but I would posit that without Governmental interference...er, funding of "important" studies, private industry and private research would have developed it anyway, and most likely with much less waste that is inherent to all such redistribution of funds from the citizenry (or subjects for non-U.S) by the Government under threat of imprisonment and/or fine.
AKA a rehashing of John Ioannidis 2005 essay.
This is why it bothered me so much when people would say "trust the science"
Trust is the antithesis of science. Science is about doubt!
this is 100% the reason why im not wanting to go into scientific resesrch. unti these biases are fixed and science becomes less about money and more about humanity, im gonna stick to working in museums.
that little shop reference though...
That little shop of horrors reference though.
I thought the issue with neutrinos was that a squirrel ate one of the cables and got electrocuted in the process?
even ted can't dodge teh trump memes dis gonna be huuuuuuuggeee
your clickbait game is weak TED
Well the title says what's in the box. No clickbait anyway
djancak Woah, do mean me? Where is this coming from?
That bee made me drop my phone and swat around my head XD
Interestingly enough, traditionally one of the worst culprits, Psychology is leading the charge to fix these problems
How do Psychology is leading to fix the problem? I didn't get it.
that is so, so wrong. Physics is 100% in the lead followed by chemistry. Psychology is worse than ecology and ecology is really, really bad with reproducibility.
yeahh.. John oliver from Last Week Tonight had already talked about it..
3:56 legit thought my iphone alarm went off
The crux of the problem is that science doesn't operate the same way as regular society, except that it does and has to interact with it that way
The wife and I make our own experimentals back in home.
We found exciting results :)
Thats cool 0u0
remind me of certain video from veritasium
This is so fun to watch.
well on the bright side I've seen videos on this subject before :-)
Does anyone know the reference/source of the fact about less than 25% of pharmaceutical studies are not reproducible?
0:53 Samurai Jack?
Or Professor Utonium.
Veritasium's video on this was better IMO
we need a protocol.
Like the scientific method? It exists
That's more of an idea/method than a protocol.
Scientific discoveries should lead public policies? Someone tell this to Trump. 😕
sore loser
L Galicki you dare question the god emperor?
Where did ted go
beautiful video
did anyone else thought about the EM drive when watching this ?
Little Shop of Horrors reference..nice.
didn't know that about spinach!! Go to supermarket and buy some!! lol
Very interesting. And the animations are funny too.
dat face palm, though
Proffessor was pulled right out of powerpuffGirls.
Ok, but was the study on how often mistakes are caught reproduced?
Bless Audrey2 👌🌿