There's a lot of risk and relatively little benefit to debunking falsified research so Simonsohn, Simmons, and Nelson deserve a lot of credit for doing so. Unfortunately, promotions and tenure decisions are not going to be based on keeping one's profession honest; promotions are based on publishing highly cited research which is why this kind of fraud will continue and likely go undetected almost always.
Yes, unfortunately there seems to be a number of arguably broken mechanisms in our modern knowledge production machine. It spans many fields it seems, and the undue influence of money, ideology, narrative, upholding prestige, etc. all seem to have wormed itself throughout. You may or may not agree that "peer review" for example is compromised. There are examples of major gatekeeping, like in who gets assigned to review certain papers. When the prestigious, high impact publications assign peer reviewers with conflicts of interest that may even go so far as being part of a cabal to repress certain lines of inquiry, discredit certain researchers who attempt to publish alternative frameworks to a subject. Bret Weinstein says something like (paraphrasing) we use peer review as a heuristic as to whether we should even bother with a paper, though this has created the potential for a rent-seeking niche... if an interested party can gain influence over who reviews the papers, or how reviewers are incentivized/disincentivized to treat certain submissions/arguments, then you can use the chokepoint to gatekeep the field... if peer review is not true peer review, then we need "review by peers" as having other experts and those of adjacent domains review your work, or even collaborate, then that will tend to catch errors, refine arguments and even produce new insights... it just doesn't need to take the form as it's practiced now, we can setup alternative means of review by peers, do it transparently and allow people to determine for themselves whether the process is fit for purpose.
@@profdc9501Oh yes, this was quite the shock when I went into grad school and got a better look behind the curtains. Professors generally receive zero education on how to effectively teach, and ability to teach is generally not even really a question. At least in my department, feedback from grad students talking to prospective professors is taken into account, but that doesn’t help a whole lot because we also don’t get educated on science informed teaching and what that looks like, so you only have a few overachiever grad students that go out of their way to learn how to teach effectively. And for the professors, they have plenty on their plate from research alone, and the university only cares about their about your ability to get grants. Students don’t avoid going to a university because of bad professors, they’re choosing based on status and location more than anything else, so actual quality of teaching is left up to the discretion of the professor
This scandal is in behavioral. But it is by no means confined to it. You live in a world where math and time is racist. Do not deny the threat science is under
The biggest take away that I have from this scandal is that instead of de facto bans on publication of Reproduction Studies, there should almost be an _Automatic_ Publication of reproduction studies. "You are reproducing ? Great. If you find the same results, we'll publish a short paper of your findings (methodology section limited to "where we differed from the original," conclusions section limited to "here's the data, a few charts, our p-value" and maybe a bit more, no "future work" section, and other such shortenings). If you get a _different_ result? You get a more comprehensive publication, to explain why you believe you got different results, and how you think others could improve procedures to get more conclusive results (supporting whatever the correct conclusion happens to be)." After all, that would effectively expose data fudging, and encourage _aggressive_ attempts to disprove the hypothesis among reproduction investigators (which should be the default in proper science)
Yes! Reproducing studies needs to become a respected line of work, especially for every one where they cannot reproduce the original author’s suspicious results with the same methods.
And then you end up creating a circle jerk of specialists who run around in circles disapproving one another. That's already done in politics and see how great it's doing.
Honestly I think anything that receives over 250- 1000 citations should be heavily heavily scrutinised and resources should be put back into verifying that work
The most successful researcher I worked for was p-hacking like crazy. Thousands of citations, generous grants. I don't believe anything that comes out of that lab, because I've seen how the sausage is made.
Ugh, that’s horrifying. My only hope from this public reveal would be that more people would start investigating source data, duplicating studies, and publishing results as a rule so that peer review would start to get better to catch more cheaters.
"Don't overgeneralize bro! Just trust shit bro! Sure it's not repeated and it's totally possible for people in academia to be more subtle liars than the people in these videos and never, ever get caught. However, just trust shit bro!" - half the commenters here
The point is that sociology and psychology aren't science.Their foundations lie in subjective prejudices, not in objective facts. They are a branch of the human "sciences", which are not sciences. It's even wrong calling these discrimations "sciences"...
What you can say is that the constant pursuit of funding and prestige is a strong incentive to lie for the sake of getting published. It also doesn’t help that the paywalling of scientific literature makes it much harder for people to audit these things. It’s not quite as bad a closed source vs open source code, but there’s still a barrier to getting access to this data to audit it in the first place. If all you have to do is clear the peer review process to be home free, then there’s an incentive to tailor your data towards meeting that specific challenge, a la “teaching to the test”, instead of having your study be rigorous enough to pass any and all scrutiny.
Good points. I'd add the the burden of peer review is just way way too big to expect it to catch something like this. I've done lots of peer review in humanities and based on how much unpaid time it takes I literally can't imagine how someone in sciences or social sciences would sit down with ever piece of data and go through with a fine toothed comb. It's just not possible unless you were paid to do nothing but peer review as a full time job. Realistically you're just going to give the data the smell test to make sure it doesn't look insane and otherwise trust that your colleagues aren't simply making stuff up. The argument that this is "peer review at work" is unfortunately more correct than that person realized...after ten years some motivated people might catch stuff maybe. Yeah that's about right.
As a former associate editor of a journal, a reviewer of dozens of articles, and an author of dozens of articles, I don't trust what I read in journals. I just don't trust the peer review process. One can get published, regardless of the review of the paper, if the author is well-known especially to the editor, if the result is one that the journal believes will be highly cited (regardless of its veracity), or even if the journal simply doesn't provide the reviewers or editors sufficient leeway to reject articles that are suspicious. There is little incentive to address these issues as it benefits those who game the system and in turn make the decisions about how the system works. In short, caveat lector.
- It also doesn’t help that the paywalling of scientific literature makes it much harder for people to audit these things. That's a huge part of the reason why they paywall it. Scientists and institutions, through the MSM, were making claims and people were looking at the sources saying 'yeah, that's not what the science says'. So they paywalled it (often needing not just $$ but credentials for access) to ensure those pesky commoners can't come to their own conclusions. The reason for the mistrust in academia is well earned.
Academia deserves the scorn it gets, but that doesn’t mean it’s not worthwhile saving it, unfortunately as an institution it needs great reform and purging of bad faith actors as well as a lot of humbling. After all, the scientific method is an honest pursuit of knowledge purged from pride and preconceived notions. It’s a hard pill to swallow, bit it’s the truth.
For the most part, researchers in academia are honest and ethical about their work, collecting data honestly and correctly following the Scientific Method. Unfortunately, especially with higher-end journals, some bad actors will do anything to get a paper published due to the money incentive of certain results. It's a challenge that is slowly gaining steam on challenging, with thousands of researchers retesting old results and seeing if there's something off about various papers, including their methods and/or conclusions.
The scientific method is independent of the journal system and academic bureaucracy. Frankly a large portion of academic conventions tend to stifle good science just to protect their rituals.(Not all is without practical merit, but much of it is hogwash.)
@@mizarimomochi4378 You think so? How naive are you. Even in school it already starts with hidden notes on exams. You think this is getting better later when the chances of getting caught are far less than the teacher looking over your shoulder?
No question the research scandal encompassing Francesca Gino and Dan Ariely is important. Their work was taken all over the world - believed and then revealed as false. Your video was and is important, and the field does require better guardrails. We are fortunate the Data Colada researchers stepped up to the plate.
When you title a video "Academia is Broken", don't be surprised when people use it to justify their mistrust of academic/scientific research in general.
@@plno2443 you have got a point, but do understand that RUclips itself is publish or perish. If he wouldn’t have named it that, it wouldn’t have even reached 100k views; good chance. What’s the use in *that*? In then end, the morons are acceptable collateral.
Shouldn't they be mistrustful of academia and research? This fraud isn't something that occurred in a vacuum. The replication crisis was first raised in like 2013 and has been increasingly observed in a variety of fields, including hard sciences, medicine, engineering, etc. It hasn't been solved or even mitigated.
Hi, I think it is important that you alert us to these issues. I recently stumbled onto this channel and I find your thoughts helpful. I am shocked that the peer review process is not vigorous enough to catch these problems. Thank you for breaking these things down for someone like me.
To trust the science is to be dogmatic, whereas to doubt the science is to be scientific. Has anyone here read Descartes? Now he was an excellent scientist. And what did he do in his Meditations? Have a read and find out.
I have a PhD in Math, even though one can not publish fake math, many mathematicians get around it by publishing theorems with very long hypothesis that restrict this and that parameter in an equation.
I took 3rd year engineering math and the youngish prof teaching us said for a PhD you basically find a paper where someone solves an equation where x = 2 and just solve it for x = 3. I think that's the same point, math is math, you're not going to "discover" or invent a new theorem so just manipulate what's out there and hopefully find an interesting result.
@@TheJhtlag What you say might true for engineering math. But pure math PhD is all about finding and proving new theorems. There are thousands of papers published each year, you can take a look.
As a behavioral scientist, you should work to understand why people distrust vaccine and climate data rather than just dismiss them. Consider, it's not just this one incident at Harvard, but the Stanford president was forced to resign due to falsified data. I've spent 40 years in R&D and it's not just isolated to a few cases. I've refused a direct order to falsify data in order to get more funding for a project. The peer review process is nearly a joke. I've seen papers rejected by one set of reviewers as trivial and unimportant yet accepted unchanged by another set of reviewers. Then when you move to things like vaccine and climate change and add in the politization and cherry picking of data, it's easy to see why the general public has a distrust. I don't think it's a distrust of science in general, but a distrust of science when associated with big money and big politics -- ie people can no longer trust that what they are being told is actual science.
It's called 'humans being human'. Corruption and entropy exists at every level, within every vertical of society. However, where it's most impactful are the verticals entrusted with truth making and information sharing. Oh and there is also evidence of Pfizer at least lying about their trials.
I've been fired from a job for asking how data is collected or whether or not the data is collected should be retaken because it seems to be cherry-picked (though I did not exactly use those words). You get the idea from working in this field that to some this is just a game to be played and won rules be damned.
I couldn’t agree more. When science is driven by money and political agenda, it’s hard to be trusted. Not to mention, when people question such science in open dialog, they got labeled badly along with phrases like trust the science…
Only a tiny amount of money is spent on vaccines compared to other medications. And there is much more money in companies that oppose climate action than in companies that support it. Although I agree that this is a behavioural science problem, it cannot be explained by the simple logic of money.
Ok, but in the end how do we catch "cheaters" that manipulate their database? In the Gino case the data had clear outliers, but a "good" cheater could easily hide these outliers or manipulate the entire data to seem natural. If that was the case it would never get caught. I was a scientist in Brazil and I got out preciselly because of things like these. And I'm not talking about "high visibility" papers from Harvard, which means nobody is checking. And even though I have several contacts in academia here, there is zero interest in catching cheaters, exactly because they make the whole of academia seem bad (and this atitude may confirm most of them are). In the end, I can't blame regular people for doubting, if even academics are doubting high visibility, well published articles from Harvard teachers. If we don't solve this problem quick I guess most people will be "anti-vaxxers" in the future, and rightly so.
I think the mentality of a lot of cheaters is that they are smarter than everyone else, and therefore don't believe they're going to be caught because they're too clever. So they don't take the effort to cover their tracks like someone who is genuinely afraid of being caught would. Researchers who understand the risks of being caught and that other people can catch on to fraud probably would try other ways to advance their career, or at least not produce so many high profile results as to raise suspicion. I am guessing careless cheaters have strong narcissistic tendencies that convinces them that they can get away with such behavior.
Ideally, you'd have other researchers reproducing the study independently. This way you'd have multiple datasets and you'd inevitably find anomalies in fraudulent work by comparison. Unfortunately, proposals to do so often go unfunded, and thus never happen.
@fourdimensionalgirlfriend There's a reason sociology and similar fields have such massive problems replicatability. I have zero faith in most published studies these days. I have no way of knowing if things are falsified so even if the study is 100% legit, I still can't trust it.
I love how ironic it is for the academia and the research community in general, where although this whole fiasco with the fake data will most likely be a stain for a while now (or at least it should be so that the fraction of researchers who are willing to falsify their results for the sake of their own career progression will be caught sooner than later), it is great that there are still researchers out there that are going through all of the possible lengths in order to expose these issues, even when it comes to renowned researchers in different fields. Yet in light of all of that, there are still people who will either try to defend those people or exaggerate the findings, not realizing that in principle they are doing the exact same thing that those researchers did but in different context, by trying to falsify further the content that covers the scandal related with falsifying xD
Will you also be breaking down the resignation of Stanford president over a very similar false data reporting? Seems so similar and you did a great job in the first video!
creators who reads comment are criminally underrated imho, and for keeping it clean and healthy, thank you for that. i was one of the millions who randomly stumbled upon your channel from your last video too, cheers and have a great day. it was honestly a really good topic, and this follow-up video is also interesting!
This really shows how important truth and honesty is in science. As soon as the truth is subordinated to any other purpose -- personal gain, social justice, or anything else -- the entire institution is called into question. I get that you would prefer if regular people would partition this malpractice case out from other fields or people, but that's not how this works. We need to do better to preserve the sanctity of science from those who would twist it to their narrow self interest, or genuinely risk the collapse of the entire edifice. Maybe a good start would be fixing the incentives so that replication happens more often.
Tbh, that's not the point..i read a lot of journals for master thesis and tbh with you, journals that were published never show you the details on how the researchers conduct the experiments, they only show you the result which quite frankly ridiculous in my opinion, that's why people like her think they could get away with this type of manipulation, because people never actually look at the full research that was conducted..
1. For an interesting case, look up the case of Peter G. Schultz, the missing notebook, the blackmail attempt, the expanded genetic code, and the glycosylated amino acid. 2. "These bad ideas fell through the safety net" How many bad ideas were caught by the safety net? How many good ideas went through at the same time? Without these numbers, there's no case.
dude, nobody is against getting cured or getting the shots. people have legitimate concers over health side effects, and they are rightly so. especially after these past 3 years.
I absolutely hate the division between "soft science" and "hard science" in discussions by laypeople. It's utterly reductive and doesn't help anything, especially when blanket applied to entire fields. Take psychology for instance, or even its sub-disciplines like your own behavioural science: some research is going to be "soft science-y" necessarily, by the nature of its topic or research method, but other stuff can be very "hard science-y" because the specific things it is looking at is dealing with very strict, clear numbers. And in all cases I still agree with your point in this video; just because something is soft science does not mean it isn't important.
I'm a hard science phd and I gotta say soft sciences seem to be way more susceptible to this level of fraud. Just look at how quickly so many claims of working "room temperature superconductivity" get immediately destroyed or rescinded from journals within weeks or even months. Frankly I don't think the social sciences have anywhere near the same level of rigor which is why egregious cases like these go undetected for years.
You’re crying about people discounting science because they extrapolate from your video that the research process is broken. However, you titled your video as clickbait “Academia is BROKEN.” Hmmmm…. I wonder why people are over extending the intended message of that video… Congratulations, you are part of the misinformation problem.
Brother.. can’t be calling people out for taking this scandal as evidence that other areas of academia are broken when you title your video “Academia is BROKEN”
@@anannon8384well then maybe he shouldn't have titled it "ACADEMIA IS BROKEN" if he didn't mean it. Whether he likes it or not, he's hit the nail square on the head.
@@bobchipman4473 Perhaps it is, but people need to provide more evidence than just this one case. If you know of another, please let us know! I have enjoyed learning about academic misdeeds from the Bobby Broccoli RUclips channel.
During peer review you would have no chance to spot this fraudulent behavior, because usually you do not get to see the dataset. Luckily there are some trends towards publishing the raw data along the paper and also pre-register experiments that are designed by one group and executed by another.
Just a note on 'how it's supposed to work', science _is not_ equivalent or limited to the current practice of {phd researcher} -> research -> paper -> submit paid journal -> peer review -> publish. Vigilante science absolutely is science, and it's great science. If the current practices in behavioral research are producing faked results, as a behaviorist you should know the incentives misaligned, so I'm confused why you would characterize the current research cycle as 'how it's supposed to work', from a behavioral point of view 'supposed to' is a hopelessly naive, basically nonsensical concept.
It really grinds my gears when people say something which boils down to: "We don't understand this phenomenon very well and that's why we should stop studying it."
Following the replication crisis in the 2010's my personal stance is and always will be - Does this topic have a well designed meta study? - Does this topic at least have a decent replication study? If neither of those checkboxes are true, I completely ignore the findings until they are. Too much statistical incompetence and fraud in academia.
On the carbon footprint subject you mentioned, what does your community think of the extensive mining caused by current battery technology? There is a parallel thing with the subject of space travel. You don't launch spaceships to do certain things that are really far away, as you can actually get there faster by waiting until technology catches up to the task. Seems like current battery technology, and magnet materials are the same. We need to do much better on that before widely adapting or we do more damage in the mean time. Wish it was not so, I like planet earth but we are causing a lot of verifiable harm while not really affecting carbon emissions much so far.
Seems worse than irresponsible to try to do nothing when the world is literally falling apart in front of our eyes, and the attitude you espouse is so, so easy to abuse by dishonest actors who don't actually care about environmental damage at all.
Ironically I'd say "softer" non-experimental studies are more difficult to fake (and less likely to be faked) if they rely on public data. Since the data is public, it's easier to spot data forgery in a way that's more difficult to do for an experiment where the only way to check is to get access to the original survey software account.
The reasoning about proof of an unreliable peer review system not casting a measure of doubt on other fields using the same or equivalent system is not sound. If peer review can fail in behavioral science journals, it can fail the same way in e.g. medical or vaccinology journals.
given how I found your video after finishing a Bobbybrocoli video, a guy who's published several videos about fake data in physics, the idea that hard science is somehow immune to forgery hit me as particularly funny
My philosophy? Replicate it, or didn’t happen. Findings aren’t facts until they’ve been replicated. Pre-replication, findings are just ‘huh, could be true, dunno. Does it replicate?’
Agree one should not generalize between fields of science , on good and bad behavior . EXCEPT , lying should have one consequence . Career over ! ( the Pres of Stanford in stepping down BUT he remains a Professor ! WHY )
Dude, bravo. If I had to put myself into one of the pro/anti vaccination camps, I'm more against their use than for it, so the way you handled those people using this as a form of ammo was stellar. You did not dismiss the claim and you did not dismiss the people that believe it, you only pointed out how their line of reasoning was flawed (and rightly so, behavioral science has nothing to do with medical science.) And again, thank you for that. That was the most respectful, and in my opinion, best way to handle that situation.
“The work of science has nothing whatever to do with consensus. Consensus is the business of politics. Science, on the contrary, requires only one investigator who happens to be right, which means that he or she has results that are verifiable by reference to the real world.” Michael Crichton, 2003
Trust science, not always the scientists. Scientists are people, and they are influenced by peers, culture, politics, popularity, money, prestige, etc, etc. They are people and they are just as fallible as anyone else. They are not the science. Being cautious of the ‘science’ isn’t the same as being cautious of the people performing the science or being cautious of the media reporting the science, which is a whole other problem.
I really have to direct a question towards you then, how, perchance, are we supposed to trust the "science" when it is being done by the "scientists" with entirely false hypotheses that data is being faked to prove? Or studies that slide through review with little to no pushback as long as the buzz words are nice and viral? (Replication crisis)
Academia exists on a presumption of academic integrity being intact. Given the broadness of human nature, it should come as no shock that there are bad actors lurking within academia itself. The danger here is that academia has long been less about the research and more about the outcome in terms of economic potential, because of this, bad actors are far more inclined to 'fake it till you make it' as frankly, without a funding grant, 10 years of academic integrity gets you working as a cleaner.
This exactly. I swear the same discussion arises whenever a cheater is caught in any field, whether it's something as high-level and respected as academia or as low-level as gaming. People are people. If there is an opening for fraud, some fraction of people will take advantage of it. Navel-gazing about "the state of X endeavor" beyond that is to me sort of pointless. No, the existence of cheaters, however high-profile, does not mean the entire field is nothing but cheaters. No, your personal belief that most people are honest doesn't mean there are not severe systemic issues that make things much worse than they need to be. Just focus on how to improve things.
Are results in behavioural sciences ever reproduced? It seems very time consuming and costly with little benefit to reproduce a study which surveyed 500 people
The thing is people aren't allow to question those other supposedly 100% settled science.. there a whole machine preventing and subverting such public discourse on the internet.
To dismiss people's trust in big companies and the politicians associated with them might be a mistake. Even if the data fiasco isn't related to the vaccine discourse, one can see why some people would rather question "the science".
Thanks for defending the integrity of some of us working in academia. We will work to improve the peer review process. This is my promise as a Editor for several journals
Don't worry you did great, and this is also a great video. As a researcher in a different field than you, I found fake data in a paper at least one time per year... In my first review paper, almost half of the papers didn't have de data, or they faked several points... I asked myself how I had so many problems with reviewers and some papers with those faked data were published like that. But I work in a small university in a country without research history.
No system/process is perfect. The valid criticism of a system / process should be looked at as an opportunity to fix the flaws so that we have less faulty system and not as an argument to dismantle or reject it completely.
>1 bad scientist Well 2 is all it took to structure the entire child support/visitation program and create the foundation for "family court" as well. Let that sink in
Have you ever had a discussion with Bret and Heather at Darkhorse? They in my view make a pretty good case that peer review is broken as a safeguard across a lot of fields. The behavioral issues in this case apply elsewhere, possibly more so where there is big money involved.
Agreed. I like B & H's idea that peer review is more often used as gatekeeping. I get the impression from this 'reply' vid that perhaps he's not ready to hear their critique just yet - but probably needs to. IMO this problem is WAAAY bigger than these papers. Thanks for your comment. I hope it gets more visibility.
@@gnoelalexmay agreed that he may not be ready yet. The use of the term anti-vaxxer to describe critics of one group of vaxxs hinted in that direction.
@@Midlife-Adventures Absolutely 💯. Use of labels like 'anti-vaxxer', 'climate denier', 'covidiot' etc should really not be accepted in serious discussion, but unfortunately they are used without a thought. I can't remember if he mentioned the Sokal squared hoax in his previous video 🤔 but that would be an interesting addition to the subject.
@@gnoelalexmayunfortunately (as you are well aware of I’m sure) this is all a result of insidious mental conditioning and subversive entities setting the parameters in a dimension most people cannot even perceive.
I am going into a data science but I am so inspired by all the digital tools we have now that didnt exist ten years ago. I foresee a whole community of amateur scientists will be tearing through mountains of older studies and looking for markers of fake data because the barrier of entry to doing it has become so low with the advancement of technology.
For the record, I have never included any of these studies in any of my research, papers or teaching. They never seemed credible to me. Their findings are NOT common knowledge, because many of us shun low-quality BE studies. It's the BE folks, who live in a lower-grade scientific bubble and shill their work, even when it's bunk. I've seen BE folks tout Dan Ariely papers that have the credibility of a small pilot study, as if it were truth--while ignoring the work of Prochaska with studies that build on sample sizes in the hundreds of thousands. Even the moderators of BE groups encourage people to share low-grade pop psychology and some nonsense too. I blame the BE community, and NOT behavioral science in general, as many in my field see BE as the absurdly popular practice which is new to behavioral science, and the least mature of many behavioral science practices. Also, this is how peer review works. Few journals have people with the quantitative skills required to test data, so it's up to practitioners who fail to replicate to report this. Eventually, nonsense will be picked up by the meta analyses.
I loved the section of the video where you talk about behavioral science as "soft" science compared to "hard" sciences. As a Psychology student, this topic has always bothered me. Friends: all areas of human study are important and I am very grateful if you decided to study physics or chemistry or any other so-called "hard" science, let's work together to deepen knowledge about the universe and nature! (Sorry, my English is not perfect).
I think a unifying principle of science as a human endeavor, whether for "soft" or "hard" science, is the need for intellectual honesty. This is to be candid about one's findings, both their strengths and limitations, so that the presumption can be that one's publication can be trusted to faithfully represent the work performed. In practice, I think that research has to have significant spin put on the results to be competitive with other publications for space in prestigious journals, and this spin often approaches the edge of misrepresentation or outright falsification without stepping over the line. Therefore the presumption is typically that the research is being presented in the most favorable way possible and one has to determine the actual significance and impact of the results, with the end result being that it is usually not worth it to read a paper and take the effort to assess this.
@@profdc9501I think a unifying factor for all human endeavor isn’t that there needs to be honesty, but a way for anyone to do their own research and a means to share your findings. Scientific research can be replicated. Dishonesty is inevitable, but what makes science great is that you can debunk claims yourself if you can gather strong evidence. If something is inherently unprovable, then it’s no science at all. The problem is, a soft science can be very subjective with a lot of qualitative data or interpretation of quantitative data, unlike hard sciences which have almost exclusively quantitative data. Everyone can agree 2+2=4, but not everyone can agree on things like country with best quality of life; to circumvent, we interpret cost of living and wealth. Soft sciences also have a lot of variables that are hard or “unethical” (what does it mean to be unethical anyways?) to isolate, so research can have wildly different results or even different interpretations with the same data.
@@czpiaor I agree that 'soft' refers to less concrete, 'hard' data. The above comment is why I prefer the terms social sciences and material science. Plus it's more descriptive-- even though, yes, psychology happens in the material brain. Rigorous experimentation happens in all sciences, even if some data is more 'hard' than others.
Bad “debunk” video. You don’t even address the point of the comments. It’s pretty obvious that Geno is just a symptom of a larger issue. Academia is full of p-hackers and data manipulators who get away with it due to the incentive structure of research. Studies that are shocking and/or agree with the preconceived academic consensus are way more likely to get published and funded. This is especially relevant in the social sciences where results are more subjective. You can see the whole field is just propping up the current political ideology of academia. People need to realize this isn’t just an “isolated incident”.
Really appreciate your perspective. Completely agree about how vital this is... I have a very dim view of our species, and it gets worse the more I learn. Open mind, clear eyes. Far too rare for how our species perceives itself traditionally, and that hubris will be our downfall.
Pointing out the flaws in academia is VERY MUCH related to flaws in other areas of academia. Drawing a parallel between Pfizer data and this researcher isn't completely unrelated. The same systems for verification (and lack there-of) will cause the same flaws regardless of what field it is in.
It's not remotely the same system. During normal times the requirements to prove a drug is safe are absolutely absurd, cost a fortune, take years and most do not survive the process. The level of scrutiny that was done for the Covid vaccines was on a whole nother level though. We never saw that many human test subjects and we never had results as scrutinized. The fact that we got 3 good vaccines that worked on hundreds of millions of people with no statistically significant number of significant side effects is a modern day miracle and comparing it to the garbage you see in research paper review and publication is about as equivalent as seeing someone blow their fingers off with a firework and drawing a parallel to the Apollo program.
The issue is not that people are questioning the vaccine. The issue is you not recognizing the obvious potential for poorly researched vaccines, with a large incentive to lie about them, being pushed upon billions of people based on studies done by a field that has questionable scientific rigor.
This has been an issue with academia for decades if not centuries. It was typically only accessible to the wealthy elite and those within it were incentivized to flaunt their intellectual prowess by out-doing others in their fields. The ignorance and pomp that plagued science was even worse back before the great shakeups caused by people like Max Planck. It's still a problem today and continues to be the grime that slows open intellectual thought to this day. Science is less about the objective truth and more about what the current groupthink is and what viewpoints organizations are directing their funding. 'Left' or 'Right' are meaningless, this is just human nature as it's been, likely for millennia.
Unbroken system fallacy is a really nasty form of gaslighting. If you find a case of cheating, cheating was caught, the system is working. If you can't find cheating, obviously the system works.
The majority of our problems are not caused by humanity not knowing what needs to be done, but rather by people not being willing to do what needs to be done. Behavioral science matters.
The comment section of RUclips is one of the most toxic places in the universe. Lol While it’s okay to be mad about this stuff, having complete distrust in science is really scary in our modern world. Thank you for this
You aren't supposed to trust science, that's antithetical to the scientific method, you are supposed to question and test it as much as reasonably possible. The amount of people in this world who have chosen to make a pseudo-religion around our modern (extremely limited, ever changing) understanding of the world is very exasperating, though not really surprising, considering actual religion is very popular.
@@ethanwilliams1880 that’s not what I mean To those have no field in science or don’t know how it’s done, you kinda have to. It’s good to have skeptics in science but in many hard sciences like astronomy or climatology or biology theories have stood the test of time. It’s dumb to go over and say it’s “fake” because you disagree with it. You gotta prove it. And conspiracy theories aren’t the way. Let me put it in a different way If your doctor says you have cancer do you a) hear out his advice and take action b) say he’s in it on the money, don’t trust him, and “do your own research” The situation with Harvard was different, it was skeptics in that field who exposed that specific person with data and reasoning. Biologists who try to disprove evolution or climatologists who try to disprove climate change fail immediately because they cherry-pick or misinterpret facts. It’s not the same. Also for all sakes and purposes I’m an atheist
Who's fault is it that people distrust modern science? if you blame social media and disinformation, YOU are the problem because you don't get it. My doctor told me it's perfectly fine to take an experimental vaccine that was rushed. I got my MMR, tetanus, dyptheroa, hep A and B, etc vaccines. But I was skeptical of the COVID vaccines. I was told by people like you that I was a conspiracy theorist, that I should "trust the science", even though I made arguments that were the SAME arguments people like you made to remain skeptical 10 years ago! It was the one that said wait hold on, why are we rushing to push this out, but the response I got from the other side was almost RELIGIOUS in its response. Trust the science. That mantra was beaten over my head and reinforced by moderation policies on social media. Now who's fault was it? You're telling me I was the ignorant anti science guy for refusing to get it? Yeah... no thanks. I trust the science when it isn't kicked out the door with faulty info. Thanks.
Whoever said "Trust the science" doesn't use scientific method. Always question and never go along with the crowd, just because it is popular or being pushed. The scientific method is proving if something is so, not just because your professor or a billionaire tells you it.
6:40 I completely agree that the scandal discussed in the previous video is not a reason to discredit behavioral science or science generally, and I also see no problem in principle with you or anyone continuing to rely on and use good science in their work, but my big question is, how will you know the next science you rely on and use is a valid one? By your own admission, you have, obviously unknowingly, used the bad science in your work before, and you didn’t know it, and you wouldn’t have realized it had these “vigilantes” not uncovered it. So, given the current state of academia where the peer review process and other safeguards that are supposed to have filtered this kind of stuff didn’t work, at least in some cases (cases involving well-known, high profile studies too), how will you have reasonable comfort about the validity of the next science that you rely on and use, let alone compete confidence in it? This isn’t meant as a personal criticism or attack, but I am genuinely curious because short of doing the kind of work that these vigilantes did, which I expect you will not have the time or capacity to do (at least with respect to every single science that you will be using), and which most people out there don’t either, how does one continue to rely on and use the science from the practical standpoint, even if one were’t out to discredit the whole concept of science and the scientific method in their very roots?
5:25: Why not? The argument that the utter lack of accountability and scientific rigor in soft sciences is *NOT* an endorsement for spending more money on them. That's like arguing that the solution to people being defrauded with cryptocurrencies & NFTs is to spend more money on them. You know about the replication crisis, it's not just *this* study, it's the entire field. Because the real problem is not that peer review doesn't work, it's that correlations riddled with externalities, backed up by nothing more than P-values *aren't proof*. Without empirical, quantifiable, objectively measurable evidence, a peer review is nothing more than a rain dance. And the argument that we have problems which arise from behavior does not redeem the paucity of evidence the field can produce. That's like arguing that we need to fund Miss Cleo because predicting the future would be so very useful.
Thanks for your excellent analysis and presentation. I am more of a “hard science” guy but completely agree that the scientific disciplines of data integrity and logical analysis should apply equally to the soft sciences…maybe more so.
You make a good point that Behavioral Science is not 'hard science', but the problem is that the general public often perceives it to be. This could be one motivation for a small number of researchers to falsify data to try to make their findings more robust. It's similar to the 'replication crisis' - the expectations from Behavioral Science research are unrealistic. When I read research papers, it is clear that most (if not all) the findings are only 'suggestive', and to be honest, I'm actually surprised when something DOES replicate. That is what 'suggestive' means. Psychology research is far different from other scientific fields as you point out, but there is motivation to try to make 'findings' more concrete than they really are.
I would group the soft sciences in with the Humanities. That’s not meant to diminish their value but it is important to keep in mind that just because you’re using the scientific method that doesn’t mean you’re gonna get a lot of scientifically valid answers. If gravity failed 1 time out of every 200 we’d be having some major problems…
Speaking as an average, unremarkable, ordinary person, it’s not so literal for most of us. It’s an overall justified mistrust of academia, media, government…
How did she get to be a professor? I'd warrant it has nothing to do with her academic prowess, but rather how friendly she was to the existing quackademia.
I'm very close with people who are lifelong academics. It's basically a system that trades time and money for respect. There is no meritocracy to speak of. If you SPEND time and money, you're BUYING respect. You don't actually have to learn or know anything, some of these people are the dumbest people I've ever talked to. You just need to put in the time and pay them the tuition fees. It's sort of like Scientology, the more money you put in the higher your thetan level goes. Then I watch these people go out into the real world and directly use that purchased respect to gain leadership roles in organizations that they then run off of a cliff.
hay big tip. don't look at the comments.
Whoops 😬
it's pretty important for yt content creators to engage with their comment stream
@@victorfinberg8595 it's important but it's also mentally draining for them
Hahaha great comment
IMMEDIATELY FAILED.
There's a lot of risk and relatively little benefit to debunking falsified research so Simonsohn, Simmons, and Nelson deserve a lot of credit for doing so. Unfortunately, promotions and tenure decisions are not going to be based on keeping one's profession honest; promotions are based on publishing highly cited research which is why this kind of fraud will continue and likely go undetected almost always.
promotion is based on your fame and your appeal to investors. None else.
@@dongshenghan1473 When it comes to hiring professors, their ability to educate isn't even number two on the priority list of qualities.
Yes, unfortunately there seems to be a number of arguably broken mechanisms in our modern knowledge production machine. It spans many fields it seems, and the undue influence of money, ideology, narrative, upholding prestige, etc. all seem to have wormed itself throughout.
You may or may not agree that "peer review" for example is compromised. There are examples of major gatekeeping, like in who gets assigned to review certain papers. When the prestigious, high impact publications assign peer reviewers with conflicts of interest that may even go so far as being part of a cabal to repress certain lines of inquiry, discredit certain researchers who attempt to publish alternative frameworks to a subject. Bret Weinstein says something like (paraphrasing) we use peer review as a heuristic as to whether we should even bother with a paper, though this has created the potential for a rent-seeking niche... if an interested party can gain influence over who reviews the papers, or how reviewers are incentivized/disincentivized to treat certain submissions/arguments, then you can use the chokepoint to gatekeep the field... if peer review is not true peer review, then we need "review by peers" as having other experts and those of adjacent domains review your work, or even collaborate, then that will tend to catch errors, refine arguments and even produce new insights... it just doesn't need to take the form as it's practiced now, we can setup alternative means of review by peers, do it transparently and allow people to determine for themselves whether the process is fit for purpose.
@@profdc9501Oh yes, this was quite the shock when I went into grad school and got a better look behind the curtains. Professors generally receive zero education on how to effectively teach, and ability to teach is generally not even really a question. At least in my department, feedback from grad students talking to prospective professors is taken into account, but that doesn’t help a whole lot because we also don’t get educated on science informed teaching and what that looks like, so you only have a few overachiever grad students that go out of their way to learn how to teach effectively. And for the professors, they have plenty on their plate from research alone, and the university only cares about their about your ability to get grants. Students don’t avoid going to a university because of bad professors, they’re choosing based on status and location more than anything else, so actual quality of teaching is left up to the discretion of the professor
Yep. The incentives to hit home runs easily are there. But, the fact many professionals don't take the easy road is helpful.
Pete: It looks like this scandal happened in behavioral science, not the hard sciences
Marc Tessier-Levigne: Hold my photoshopped Western Blot.
It's so awful it's funny 🤣😂
still a soft science
@@sciencewithskilldog How is neurology a soft science?
This scandal is in behavioral. But it is by no means confined to it. You live in a world where math and time is racist. Do not deny the threat science is under
@@WhereinTruthLies idk brain tissue is pretty gooey
The biggest take away that I have from this scandal is that instead of de facto bans on publication of Reproduction Studies, there should almost be an _Automatic_ Publication of reproduction studies.
"You are reproducing ? Great. If you find the same results, we'll publish a short paper of your findings (methodology section limited to "where we differed from the original," conclusions section limited to "here's the data, a few charts, our p-value" and maybe a bit more, no "future work" section, and other such shortenings). If you get a _different_ result? You get a more comprehensive publication, to explain why you believe you got different results, and how you think others could improve procedures to get more conclusive results (supporting whatever the correct conclusion happens to be)."
After all, that would effectively expose data fudging, and encourage _aggressive_ attempts to disprove the hypothesis among reproduction investigators (which should be the default in proper science)
Yes! Reproducing studies needs to become a respected line of work, especially for every one where they cannot reproduce the original author’s suspicious results with the same methods.
And then you end up creating a circle jerk of specialists who run around in circles disapproving one another.
That's already done in politics and see how great it's doing.
Honestly I think anything that receives over 250- 1000 citations should be heavily heavily scrutinised and resources should be put back into verifying that work
@@xyzyzx1253 agreed. I honestly wonder if "peer review" shouldn't be considered the epitome of scientific validity, but "convincingly replicated"
I am afraid that the reproduced study become fraud too. How can we ensure that the reprodruced study is independent from the original paper and such?
The most successful researcher I worked for was p-hacking like crazy. Thousands of citations, generous grants. I don't believe anything that comes out of that lab, because I've seen how the sausage is made.
Ugh, that’s horrifying. My only hope from this public reveal would be that more people would start investigating source data, duplicating studies, and publishing results as a rule so that peer review would start to get better to catch more cheaters.
and you would be branded as "antivaxxer" because of it....the world we live in
Beaks and feathers in some sausages…
Name some names, so people won't cite the crap.
"Don't overgeneralize bro! Just trust shit bro! Sure it's not repeated and it's totally possible for people in academia to be more subtle liars than the people in these videos and never, ever get caught. However, just trust shit bro!" - half the commenters here
Whatever you do, don't look into data fabrication in the medical research field, it'll depress you.
Tommy Shelby looks up: “Already broken.”
One extremely serious consequence is that this erodes the public trust in scientists in general.
The point is that sociology and psychology aren't science.Their foundations lie in subjective prejudices, not in objective facts.
They are a branch of the human "sciences", which are not sciences. It's even wrong calling these discrimations "sciences"...
Oh no, the population of Idiocracy will not trust some narcissistic data manipulators inbetween their gooning sessions
If there’s no incentive to do robust peer review it’s not gonna get done.
What you can say is that the constant pursuit of funding and prestige is a strong incentive to lie for the sake of getting published. It also doesn’t help that the paywalling of scientific literature makes it much harder for people to audit these things. It’s not quite as bad a closed source vs open source code, but there’s still a barrier to getting access to this data to audit it in the first place. If all you have to do is clear the peer review process to be home free, then there’s an incentive to tailor your data towards meeting that specific challenge, a la “teaching to the test”, instead of having your study be rigorous enough to pass any and all scrutiny.
Good points. I'd add the the burden of peer review is just way way too big to expect it to catch something like this. I've done lots of peer review in humanities and based on how much unpaid time it takes I literally can't imagine how someone in sciences or social sciences would sit down with ever piece of data and go through with a fine toothed comb. It's just not possible unless you were paid to do nothing but peer review as a full time job. Realistically you're just going to give the data the smell test to make sure it doesn't look insane and otherwise trust that your colleagues aren't simply making stuff up. The argument that this is "peer review at work" is unfortunately more correct than that person realized...after ten years some motivated people might catch stuff maybe. Yeah that's about right.
As a former associate editor of a journal, a reviewer of dozens of articles, and an author of dozens of articles, I don't trust what I read in journals. I just don't trust the peer review process. One can get published, regardless of the review of the paper, if the author is well-known especially to the editor, if the result is one that the journal believes will be highly cited (regardless of its veracity), or even if the journal simply doesn't provide the reviewers or editors sufficient leeway to reject articles that are suspicious. There is little incentive to address these issues as it benefits those who game the system and in turn make the decisions about how the system works. In short, caveat lector.
People have kind of known this for years. They just got caught
- It also doesn’t help that the paywalling of scientific literature makes it much harder for people to audit these things.
That's a huge part of the reason why they paywall it. Scientists and institutions, through the MSM, were making claims and people were looking at the sources saying 'yeah, that's not what the science says'. So they paywalled it (often needing not just $$ but credentials for access) to ensure those pesky commoners can't come to their own conclusions.
The reason for the mistrust in academia is well earned.
The funny part is that on the Harvard emblem it says in Latin "Veritas", which means "the truth".😂😂😂
Academia deserves the scorn it gets, but that doesn’t mean it’s not worthwhile saving it, unfortunately as an institution it needs great reform and purging of bad faith actors as well as a lot of humbling. After all, the scientific method is an honest pursuit of knowledge purged from pride and preconceived notions. It’s a hard pill to swallow, bit it’s the truth.
For the most part, researchers in academia are honest and ethical about their work, collecting data honestly and correctly following the Scientific Method. Unfortunately, especially with higher-end journals, some bad actors will do anything to get a paper published due to the money incentive of certain results. It's a challenge that is slowly gaining steam on challenging, with thousands of researchers retesting old results and seeing if there's something off about various papers, including their methods and/or conclusions.
The scientific method is independent of the journal system and academic bureaucracy. Frankly a large portion of academic conventions tend to stifle good science just to protect their rituals.(Not all is without practical merit, but much of it is hogwash.)
Some of the blame is external, publish-or-perish comes from university rankings, something pioneered by newspapers and magazines.
@@mizarimomochi4378 You think so? How naive are you. Even in school it already starts with hidden notes on exams. You think this is getting better later when the chances of getting caught are far less than the teacher looking over your shoulder?
Do we save or do we go full apocalyptist and see what we can rebuild on the other side?
No question the research scandal encompassing Francesca Gino and Dan Ariely is important. Their work was taken all over the world - believed and then revealed as false. Your video was and is important, and the field does require better guardrails. We are fortunate the Data Colada researchers stepped up to the plate.
Ariely is also involved ?
@@Hexanitrobenzene His mug was shown briefly as a co author in some of her publications. He has his own replication problems.
When you title a video "Academia is Broken", don't be surprised when people use it to justify their mistrust of academic/scientific research in general.
Agreed. Creates over generalized accusation as title to get clicks, and then is shocked that commentators are overgeneralizing.
This is an excellent point.
@@plno2443 you have got a point, but do understand that RUclips itself is publish or perish. If he wouldn’t have named it that, it wouldn’t have even reached 100k views; good chance. What’s the use in *that*? In then end, the morons are acceptable collateral.
Shouldn't they be mistrustful of academia and research? This fraud isn't something that occurred in a vacuum. The replication crisis was first raised in like 2013 and has been increasingly observed in a variety of fields, including hard sciences, medicine, engineering, etc. It hasn't been solved or even mitigated.
absolutely. Chasing clicks like that has consequences.
Hi, I think it is important that you alert us to these issues. I recently stumbled onto this channel and I find your thoughts helpful. I am shocked that the peer review process is not vigorous enough to catch these problems. Thank you for breaking these things down for someone like me.
Because it's "peer review." You scratch my back and I'll scratch yours.
To trust the science is to be dogmatic, whereas to doubt the science is to be scientific. Has anyone here read Descartes? Now he was an excellent scientist. And what did he do in his Meditations? Have a read and find out.
I appreciate your fighting the good fight as difficult as it is for you. This is the essential underpinning of science.
Safe and effective was nudging, applied behavioral psychology. When experience shows up theory, revise or reject the theory.
I have a PhD in Math, even though one can not publish fake math, many mathematicians get around it by publishing theorems with very long hypothesis that restrict this and that parameter in an equation.
Can you elaborate what you mean? I didn't understand your point.
I took 3rd year engineering math and the youngish prof teaching us said for a PhD you basically find a paper where someone solves an equation where x = 2 and just solve it for x = 3. I think that's the same point, math is math, you're not going to "discover" or invent a new theorem so just manipulate what's out there and hopefully find an interesting result.
@@TheJhtlag What you say might true for engineering math. But pure math PhD is all about finding and proving new theorems. There are thousands of papers published each year, you can take a look.
As a behavioral scientist, you should work to understand why people distrust vaccine and climate data rather than just dismiss them. Consider, it's not just this one incident at Harvard, but the Stanford president was forced to resign due to falsified data. I've spent 40 years in R&D and it's not just isolated to a few cases. I've refused a direct order to falsify data in order to get more funding for a project. The peer review process is nearly a joke. I've seen papers rejected by one set of reviewers as trivial and unimportant yet accepted unchanged by another set of reviewers. Then when you move to things like vaccine and climate change and add in the politization and cherry picking of data, it's easy to see why the general public has a distrust. I don't think it's a distrust of science in general, but a distrust of science when associated with big money and big politics -- ie people can no longer trust that what they are being told is actual science.
It's called 'humans being human'. Corruption and entropy exists at every level, within every vertical of society. However, where it's most impactful are the verticals entrusted with truth making and information sharing. Oh and there is also evidence of Pfizer at least lying about their trials.
I've been fired from a job for asking how data is collected or whether or not the data is collected should be retaken because it seems to be cherry-picked (though I did not exactly use those words). You get the idea from working in this field that to some this is just a game to be played and won rules be damned.
I couldn’t agree more. When science is driven by money and political agenda, it’s hard to be trusted. Not to mention, when people question such science in open dialog, they got labeled badly along with phrases like trust the science…
Only a tiny amount of money is spent on vaccines compared to other medications. And there is much more money in companies that oppose climate action than in companies that support it. Although I agree that this is a behavioural science problem, it cannot be explained by the simple logic of money.
Just admit you're an anti vaxxer 🤡
Ok, but in the end how do we catch "cheaters" that manipulate their database? In the Gino case the data had clear outliers, but a "good" cheater could easily hide these outliers or manipulate the entire data to seem natural. If that was the case it would never get caught.
I was a scientist in Brazil and I got out preciselly because of things like these. And I'm not talking about "high visibility" papers from Harvard, which means nobody is checking. And even though I have several contacts in academia here, there is zero interest in catching cheaters, exactly because they make the whole of academia seem bad (and this atitude may confirm most of them are).
In the end, I can't blame regular people for doubting, if even academics are doubting high visibility, well published articles from Harvard teachers. If we don't solve this problem quick I guess most people will be "anti-vaxxers" in the future, and rightly so.
I think the mentality of a lot of cheaters is that they are smarter than everyone else, and therefore don't believe they're going to be caught because they're too clever. So they don't take the effort to cover their tracks like someone who is genuinely afraid of being caught would. Researchers who understand the risks of being caught and that other people can catch on to fraud probably would try other ways to advance their career, or at least not produce so many high profile results as to raise suspicion. I am guessing careless cheaters have strong narcissistic tendencies that convinces them that they can get away with such behavior.
You offer to pay them more to not publish their paper and see if they take it.
Ideally, you'd have other researchers reproducing the study independently. This way you'd have multiple datasets and you'd inevitably find anomalies in fraudulent work by comparison. Unfortunately, proposals to do so often go unfunded, and thus never happen.
"People are right to be anti vaxxers" 🤡
@fourdimensionalgirlfriend There's a reason sociology and similar fields have such massive problems replicatability. I have zero faith in most published studies these days. I have no way of knowing if things are falsified so even if the study is 100% legit, I still can't trust it.
I just finished college and I would love more places talking about Behaviour Science. Thank you for being one of those places!
I love how ironic it is for the academia and the research community in general, where although this whole fiasco with the fake data will most likely be a stain for a while now (or at least it should be so that the fraction of researchers who are willing to falsify their results for the sake of their own career progression will be caught sooner than later), it is great that there are still researchers out there that are going through all of the possible lengths in order to expose these issues, even when it comes to renowned researchers in different fields. Yet in light of all of that, there are still people who will either try to defend those people or exaggerate the findings, not realizing that in principle they are doing the exact same thing that those researchers did but in different context, by trying to falsify further the content that covers the scandal related with falsifying xD
Behavioral "science" - we need a more formal delineation between hard, soft, pseudo, and humanities. Different degrees, different everything
Will you also be breaking down the resignation of Stanford president over a very similar false data reporting? Seems so similar and you did a great job in the first video!
Thanks for both videos. Let’s clean up science and give people some trust.
Influence is one thing. Manipulation is another.
>we lied
>give us more money
lmao
creators who reads comment are criminally underrated imho, and for keeping it clean and healthy, thank you for that. i was one of the millions who randomly stumbled upon your channel from your last video too, cheers and have a great day. it was honestly a really good topic, and this follow-up video is also interesting!
This really shows how important truth and honesty is in science. As soon as the truth is subordinated to any other purpose -- personal gain, social justice, or anything else -- the entire institution is called into question. I get that you would prefer if regular people would partition this malpractice case out from other fields or people, but that's not how this works. We need to do better to preserve the sanctity of science from those who would twist it to their narrow self interest, or genuinely risk the collapse of the entire edifice. Maybe a good start would be fixing the incentives so that replication happens more often.
I am shocked by the amateur nature of her imputations. I work as retail data analyst and even me can see clearly that something was off
😆
Tbh, that's not the point..i read a lot of journals for master thesis and tbh with you, journals that were published never show you the details on how the researchers conduct the experiments, they only show you the result which quite frankly ridiculous in my opinion, that's why people like her think they could get away with this type of manipulation, because people never actually look at the full research that was conducted..
This also happens in hard sciences. Just look at Jan Hendrik Schoen.
1. For an interesting case, look up the case of Peter G. Schultz, the missing notebook, the blackmail attempt, the expanded genetic code, and the glycosylated amino acid.
2. "These bad ideas fell through the safety net" How many bad ideas were caught by the safety net? How many good ideas went through at the same time? Without these numbers, there's no case.
dude, nobody is against getting cured or getting the shots. people have legitimate concers over health side effects, and they are rightly so. especially after these past 3 years.
I absolutely hate the division between "soft science" and "hard science" in discussions by laypeople. It's utterly reductive and doesn't help anything, especially when blanket applied to entire fields. Take psychology for instance, or even its sub-disciplines like your own behavioural science: some research is going to be "soft science-y" necessarily, by the nature of its topic or research method, but other stuff can be very "hard science-y" because the specific things it is looking at is dealing with very strict, clear numbers. And in all cases I still agree with your point in this video; just because something is soft science does not mean it isn't important.
I'm a hard science phd and I gotta say soft sciences seem to be way more susceptible to this level of fraud. Just look at how quickly so many claims of working "room temperature superconductivity" get immediately destroyed or rescinded from journals within weeks or even months. Frankly I don't think the social sciences have anywhere near the same level of rigor which is why egregious cases like these go undetected for years.
You’re crying about people discounting science because they extrapolate from your video that the research process is broken. However, you titled your video as clickbait “Academia is BROKEN.” Hmmmm…. I wonder why people are over extending the intended message of that video… Congratulations, you are part of the misinformation problem.
Good job explaining away the illusion that this was somehow a peer review win.
You just disproved yourself with the Stanford video.
If the peer review process is broken, it is broken. Also in the 'real science'.
When you call your academic field your "industry" the problem is obvious to me.
This video is akin to pouring gasoline on a garbage fire.
Brother.. can’t be calling people out for taking this scandal as evidence that other areas of academia are broken when you title your video “Academia is BROKEN”
Uh, yeah he can. The title is hyperbolic and designed for social media. People who over-extrapolate do so on their own error.
@@anannon8384well then maybe he shouldn't have titled it "ACADEMIA IS BROKEN" if he didn't mean it. Whether he likes it or not, he's hit the nail square on the head.
@@bobchipman4473 Perhaps it is, but people need to provide more evidence than just this one case. If you know of another, please let us know! I have enjoyed learning about academic misdeeds from the Bobby Broccoli RUclips channel.
oh wow that's correct
Thanks for taking on the comments. Letting them live without comment is like...not checking on bad data in social science. Great videos! Thanks.
During peer review you would have no chance to spot this fraudulent behavior, because usually you do not get to see the dataset.
Luckily there are some trends towards publishing the raw data along the paper and also pre-register experiments that are designed by one group and executed by another.
Side effect of SOCIAL problems.
Just a note on 'how it's supposed to work', science _is not_ equivalent or limited to the current practice of {phd researcher} -> research -> paper -> submit paid journal -> peer review -> publish. Vigilante science absolutely is science, and it's great science.
If the current practices in behavioral research are producing faked results, as a behaviorist you should know the incentives misaligned, so I'm confused why you would characterize the current research cycle as 'how it's supposed to work', from a behavioral point of view 'supposed to' is a hopelessly naive, basically nonsensical concept.
Watched those videos and absolutely loved them, but subscribed after this one. Thank you for your work!
It really grinds my gears when people say something which boils down to: "We don't understand this phenomenon very well and that's why we should stop studying it."
Right. They spend millions on them and it seems like a scam to me
Following the replication crisis in the 2010's my personal stance is and always will be
- Does this topic have a well designed meta study?
- Does this topic at least have a decent replication study?
If neither of those checkboxes are true, I completely ignore the findings until they are. Too much statistical incompetence and fraud in academia.
On the carbon footprint subject you mentioned, what does your community think of the extensive mining caused by current battery technology? There is a parallel thing with the subject of space travel. You don't launch spaceships to do certain things that are really far away, as you can actually get there faster by waiting until technology catches up to the task. Seems like current battery technology, and magnet materials are the same. We need to do much better on that before widely adapting or we do more damage in the mean time. Wish it was not so, I like planet earth but we are causing a lot of verifiable harm while not really affecting carbon emissions much so far.
Seems worse than irresponsible to try to do nothing when the world is literally falling apart in front of our eyes, and the attitude you espouse is so, so easy to abuse by dishonest actors who don't actually care about environmental damage at all.
If I ever feel too happy I read RUclips's comments and it fixes that right up.
Ironically I'd say "softer" non-experimental studies are more difficult to fake (and less likely to be faked) if they rely on public data. Since the data is public, it's easier to spot data forgery in a way that's more difficult to do for an experiment where the only way to check is to get access to the original survey software account.
The reasoning about proof of an unreliable peer review system not casting a measure of doubt on other fields using the same or equivalent system is not sound.
If peer review can fail in behavioral science journals, it can fail the same way in e.g. medical or vaccinology journals.
given how I found your video after finishing a Bobbybrocoli video, a guy who's published several videos about fake data in physics, the idea that hard science is somehow immune to forgery hit me as particularly funny
He didn't say "immune", he just said that it was _harder_
My philosophy? Replicate it, or didn’t happen. Findings aren’t facts until they’ve been replicated. Pre-replication, findings are just ‘huh, could be true, dunno. Does it replicate?’
Agree one should not generalize between fields of science , on good and bad behavior .
EXCEPT , lying should have one consequence . Career over !
( the Pres of Stanford in stepping down BUT he remains a Professor ! WHY )
The rapid rate at which the screens change as well as how fast you are speaking make your posting a challange to process.
If the video is going too fast for you, you can set your playback speed to 0.75, 0.5, or even 0.25 in the video settings.
Dude, bravo. If I had to put myself into one of the pro/anti vaccination camps, I'm more against their use than for it, so the way you handled those people using this as a form of ammo was stellar. You did not dismiss the claim and you did not dismiss the people that believe it, you only pointed out how their line of reasoning was flawed (and rightly so, behavioral science has nothing to do with medical science.) And again, thank you for that. That was the most respectful, and in my opinion, best way to handle that situation.
I'm from the "hard sciences", chemistry, and you give me a whole new appreciation for behavioural science
The rage that academic fraud gets out of people is just deserts. Rage being an irrational emotion does nothing to change that.
I would argue that extreme rage is the rational response to the constant flood of lies.
Oh what am I doing with these gasoline-soaked rags? Why nothing!
“The work of science has nothing whatever to do with consensus. Consensus is the business of politics. Science, on the contrary, requires only one investigator who happens to be right, which means that he or she has results that are verifiable by reference to the real world.” Michael Crichton, 2003
Lessons from this scandal apply to all science- healthy sceptic attitudes- no arguments of authority just authority of arguments.
Trust science, not always the scientists. Scientists are people, and they are influenced by peers, culture, politics, popularity, money, prestige, etc, etc. They are people and they are just as fallible as anyone else. They are not the science. Being cautious of the ‘science’ isn’t the same as being cautious of the people performing the science or being cautious of the media reporting the science, which is a whole other problem.
I really have to direct a question towards you then, how, perchance, are we supposed to trust the "science" when it is being done by the "scientists" with entirely false hypotheses that data is being faked to prove? Or studies that slide through review with little to no pushback as long as the buzz words are nice and viral? (Replication crisis)
Academia exists on a presumption of academic integrity being intact. Given the broadness of human nature, it should come as no shock that there are bad actors lurking within academia itself. The danger here is that academia has long been less about the research and more about the outcome in terms of economic potential, because of this, bad actors are far more inclined to 'fake it till you make it' as frankly, without a funding grant, 10 years of academic integrity gets you working as a cleaner.
This exactly.
I swear the same discussion arises whenever a cheater is caught in any field, whether it's something as high-level and respected as academia or as low-level as gaming. People are people. If there is an opening for fraud, some fraction of people will take advantage of it. Navel-gazing about "the state of X endeavor" beyond that is to me sort of pointless. No, the existence of cheaters, however high-profile, does not mean the entire field is nothing but cheaters. No, your personal belief that most people are honest doesn't mean there are not severe systemic issues that make things much worse than they need to be. Just focus on how to improve things.
Are results in behavioural sciences ever reproduced? It seems very time consuming and costly with little benefit to reproduce a study which surveyed 500 people
I will sooner embrace the study of dog-gone dogging at dog parks.
Humanity never fails to disappoint
**sigh**
Have a biscuit, Potter.
The thing is people aren't allow to question those other supposedly 100% settled science.. there a whole machine preventing and subverting such public discourse on the internet.
To dismiss people's trust in big companies and the politicians associated with them might be a mistake. Even if the data fiasco isn't related to the vaccine discourse, one can see why some people would rather question "the science".
Thanks for defending the integrity of some of us working in academia. We will work to improve the peer review process. This is my promise as a Editor for several journals
Do you publish replication studies?
That should be step one in the so-called soft sciences after the replication crisis.
@@inthefade of course yes! I always make sure my students explain all the methods in detail so people can replicate the study.
Don't worry you did great, and this is also a great video.
As a researcher in a different field than you, I found fake data in a paper at least one time per year... In my first review paper, almost half of the papers didn't have de data, or they faked several points... I asked myself how I had so many problems with reviewers and some papers with those faked data were published like that. But I work in a small university in a country without research history.
No system/process is perfect. The valid criticism of a system / process should be looked at as an opportunity to fix the flaws so that we have less faulty system and not as an argument to dismantle or reject it completely.
if behavioral science is vital, i think humanity is lost
>1 bad scientist
Well 2 is all it took to structure the entire child support/visitation program and create the foundation for "family court" as well.
Let that sink in
Have you ever had a discussion with Bret and Heather at Darkhorse? They in my view make a pretty good case that peer review is broken as a safeguard across a lot of fields.
The behavioral issues in this case apply elsewhere, possibly more so where there is big money involved.
Agreed.
I like B & H's idea that peer review is more often used as gatekeeping. I get the impression from this 'reply' vid that perhaps he's not ready to hear their critique just yet - but probably needs to.
IMO this problem is WAAAY bigger than these papers.
Thanks for your comment. I hope it gets more visibility.
@@gnoelalexmay agreed that he may not be ready yet. The use of the term anti-vaxxer to describe critics of one group of vaxxs hinted in that direction.
@@Midlife-Adventures Absolutely 💯.
Use of labels like 'anti-vaxxer', 'climate denier', 'covidiot' etc should really not be accepted in serious discussion, but unfortunately they are used without a thought.
I can't remember if he mentioned the Sokal squared hoax in his previous video 🤔 but that would be an interesting addition to the subject.
@@gnoelalexmayunfortunately (as you are well aware of I’m sure) this is all a result of insidious mental conditioning and subversive entities setting the parameters in a dimension most people cannot even perceive.
I am going into a data science but I am so inspired by all the digital tools we have now that didnt exist ten years ago. I foresee a whole community of amateur scientists will be tearing through mountains of older studies and looking for markers of fake data because the barrier of entry to doing it has become so low with the advancement of technology.
Great point!
"this is not my area of expertise" - phrase never ever uttered by *those* commentators.
Cheating occurs according to the measurements and incentives used for the work. Any thing can be
pencil- whipped.
For the record, I have never included any of these studies in any of my research, papers or teaching. They never seemed credible to me. Their findings are NOT common knowledge, because many of us shun low-quality BE studies. It's the BE folks, who live in a lower-grade scientific bubble and shill their work, even when it's bunk. I've seen BE folks tout Dan Ariely papers that have the credibility of a small pilot study, as if it were truth--while ignoring the work of Prochaska with studies that build on sample sizes in the hundreds of thousands.
Even the moderators of BE groups encourage people to share low-grade pop psychology and some nonsense too. I blame the BE community, and NOT behavioral science in general, as many in my field see BE as the absurdly popular practice which is new to behavioral science, and the least mature of many behavioral science practices.
Also, this is how peer review works. Few journals have people with the quantitative skills required to test data, so it's up to practitioners who fail to replicate to report this. Eventually, nonsense will be picked up by the meta analyses.
🤓
The most important three words in this video are: "Behavior Science *INDUSTRY*".
I loved the section of the video where you talk about behavioral science as "soft" science compared to "hard" sciences. As a Psychology student, this topic has always bothered me. Friends: all areas of human study are important and I am very grateful if you decided to study physics or chemistry or any other so-called "hard" science, let's work together to deepen knowledge about the universe and nature!
(Sorry, my English is not perfect).
I think a unifying principle of science as a human endeavor, whether for "soft" or "hard" science, is the need for intellectual honesty. This is to be candid about one's findings, both their strengths and limitations, so that the presumption can be that one's publication can be trusted to faithfully represent the work performed. In practice, I think that research has to have significant spin put on the results to be competitive with other publications for space in prestigious journals, and this spin often approaches the edge of misrepresentation or outright falsification without stepping over the line. Therefore the presumption is typically that the research is being presented in the most favorable way possible and one has to determine the actual significance and impact of the results, with the end result being that it is usually not worth it to read a paper and take the effort to assess this.
I think he meant "easy" and hard, lol
@@cadthunkinI think he actually means “soft” science, where the facts aren’t black and white.
@@profdc9501I think a unifying factor for all human endeavor isn’t that there needs to be honesty, but a way for anyone to do their own research and a means to share your findings. Scientific research can be replicated. Dishonesty is inevitable, but what makes science great is that you can debunk claims yourself if you can gather strong evidence. If something is inherently unprovable, then it’s no science at all.
The problem is, a soft science can be very subjective with a lot of qualitative data or interpretation of quantitative data, unlike hard sciences which have almost exclusively quantitative data. Everyone can agree 2+2=4, but not everyone can agree on things like country with best quality of life; to circumvent, we interpret cost of living and wealth. Soft sciences also have a lot of variables that are hard or “unethical” (what does it mean to be unethical anyways?) to isolate, so research can have wildly different results or even different interpretations with the same data.
@@czpiaor I agree that 'soft' refers to less concrete, 'hard' data. The above comment is why I prefer the terms social sciences and material science. Plus it's more descriptive-- even though, yes, psychology happens in the material brain. Rigorous experimentation happens in all sciences, even if some data is more 'hard' than others.
the emphasis on individual carbon footprint gaslights how polluting industrial transit and manufacturing are
Bad “debunk” video. You don’t even address the point of the comments. It’s pretty obvious that Geno is just a symptom of a larger issue. Academia is full of p-hackers and data manipulators who get away with it due to the incentive structure of research. Studies that are shocking and/or agree with the preconceived academic consensus are way more likely to get published and funded. This is especially relevant in the social sciences where results are more subjective. You can see the whole field is just propping up the current political ideology of academia.
People need to realize this isn’t just an “isolated incident”.
Heh, you watch Dark Horse podcast too? Many of us knew this for awhile. Just glad more academics are aware.
Really appreciate your perspective. Completely agree about how vital this is... I have a very dim view of our species, and it gets worse the more I learn.
Open mind, clear eyes. Far too rare for how our species perceives itself traditionally, and that hubris will be our downfall.
Pointing out the flaws in academia is VERY MUCH related to flaws in other areas of academia. Drawing a parallel between Pfizer data and this researcher isn't completely unrelated. The same systems for verification (and lack there-of) will cause the same flaws regardless of what field it is in.
It's not remotely the same system.
During normal times the requirements to prove a drug is safe are absolutely absurd, cost a fortune, take years and most do not survive the process.
The level of scrutiny that was done for the Covid vaccines was on a whole nother level though. We never saw that many human test subjects and we never had results as scrutinized.
The fact that we got 3 good vaccines that worked on hundreds of millions of people with no statistically significant number of significant side effects is a modern day miracle and comparing it to the garbage you see in research paper review and publication is about as equivalent as seeing someone blow their fingers off with a firework and drawing a parallel to the Apollo program.
The issue is not that people are questioning the vaccine.
The issue is you not recognizing the obvious potential for poorly researched vaccines, with a large incentive to lie about them, being pushed upon billions of people based on studies done by a field that has questionable scientific rigor.
This has been an issue with academia for decades if not centuries. It was typically only accessible to the wealthy elite and those within it were incentivized to flaunt their intellectual prowess by out-doing others in their fields. The ignorance and pomp that plagued science was even worse back before the great shakeups caused by people like Max Planck. It's still a problem today and continues to be the grime that slows open intellectual thought to this day.
Science is less about the objective truth and more about what the current groupthink is and what viewpoints organizations are directing their funding. 'Left' or 'Right' are meaningless, this is just human nature as it's been, likely for millennia.
When did “trust” or “faith” in science become a thing? If you’re not skeptical of it, it’s religion or ideology.
Unbroken system fallacy is a really nasty form of gaslighting.
If you find a case of cheating, cheating was caught, the system is working.
If you can't find cheating, obviously the system works.
Nice insight!
The majority of our problems are not caused by humanity not knowing what needs to be done, but rather by people not being willing to do what needs to be done. Behavioral science matters.
The comment section of RUclips is one of the most toxic places in the universe. Lol
While it’s okay to be mad about this stuff, having complete distrust in science is really scary in our modern world.
Thank you for this
You aren't supposed to trust science, that's antithetical to the scientific method, you are supposed to question and test it as much as reasonably possible. The amount of people in this world who have chosen to make a pseudo-religion around our modern (extremely limited, ever changing) understanding of the world is very exasperating, though not really surprising, considering actual religion is very popular.
@@ethanwilliams1880 that’s not what I mean
To those have no field in science or don’t know how it’s done, you kinda have to. It’s good to have skeptics in science but in many hard sciences like astronomy or climatology or biology theories have stood the test of time. It’s dumb to go over and say it’s “fake” because you disagree with it. You gotta prove it. And conspiracy theories aren’t the way.
Let me put it in a different way
If your doctor says you have cancer do you
a) hear out his advice and take action
b) say he’s in it on the money, don’t trust him, and “do your own research”
The situation with Harvard was different, it was skeptics in that field who exposed that specific person with data and reasoning.
Biologists who try to disprove evolution or climatologists who try to disprove climate change fail immediately because they cherry-pick or misinterpret facts. It’s not the same.
Also for all sakes and purposes I’m an atheist
Who's fault is it that people distrust modern science?
if you blame social media and disinformation, YOU are the problem because you don't get it.
My doctor told me it's perfectly fine to take an experimental vaccine that was rushed.
I got my MMR, tetanus, dyptheroa, hep A and B, etc vaccines. But I was skeptical of the COVID vaccines. I was told by people like you that I was a conspiracy theorist, that I should "trust the science", even though I made arguments that were the SAME arguments people like you made to remain skeptical 10 years ago!
It was the one that said wait hold on, why are we rushing to push this out, but the response I got from the other side was almost RELIGIOUS in its response. Trust the science. That mantra was beaten over my head and reinforced by moderation policies on social media.
Now who's fault was it? You're telling me I was the ignorant anti science guy for refusing to get it?
Yeah... no thanks. I trust the science when it isn't kicked out the door with faulty info. Thanks.
Whoever said "Trust the science" doesn't use scientific method. Always question and never go along with the crowd, just because it is popular or being pushed. The scientific method is proving if something is so, not just because your professor or a billionaire tells you it.
6:40 I completely agree that the scandal discussed in the previous video is not a reason to discredit behavioral science or science generally, and I also see no problem in principle with you or anyone continuing to rely on and use good science in their work, but my big question is, how will you know the next science you rely on and use is a valid one? By your own admission, you have, obviously unknowingly, used the bad science in your work before, and you didn’t know it, and you wouldn’t have realized it had these “vigilantes” not uncovered it. So, given the current state of academia where the peer review process and other safeguards that are supposed to have filtered this kind of stuff didn’t work, at least in some cases (cases involving well-known, high profile studies too), how will you have reasonable comfort about the validity of the next science that you rely on and use, let alone compete confidence in it?
This isn’t meant as a personal criticism or attack, but I am genuinely curious because short of doing the kind of work that these vigilantes did, which I expect you will not have the time or capacity to do (at least with respect to every single science that you will be using), and which most people out there don’t either, how does one continue to rely on and use the science from the practical standpoint, even if one were’t out to discredit the whole concept of science and the scientific method in their very roots?
5:25: Why not? The argument that the utter lack of accountability and scientific rigor in soft sciences is *NOT* an endorsement for spending more money on them. That's like arguing that the solution to people being defrauded with cryptocurrencies & NFTs is to spend more money on them. You know about the replication crisis, it's not just *this* study, it's the entire field.
Because the real problem is not that peer review doesn't work, it's that correlations riddled with externalities, backed up by nothing more than P-values *aren't proof*. Without empirical, quantifiable, objectively measurable evidence, a peer review is nothing more than a rain dance.
And the argument that we have problems which arise from behavior does not redeem the paucity of evidence the field can produce. That's like arguing that we need to fund Miss Cleo because predicting the future would be so very useful.
Thanks for your excellent analysis and presentation. I am more of a “hard science” guy but completely agree that the scientific disciplines of data integrity and logical analysis should apply equally to the soft sciences…maybe more so.
Happens more than it should. Very good presentation.
You make a good point that Behavioral Science is not 'hard science', but the problem is that the general public often perceives it to be. This could be one motivation for a small number of researchers to falsify data to try to make their findings more robust. It's similar to the 'replication crisis' - the expectations from Behavioral Science research are unrealistic. When I read research papers, it is clear that most (if not all) the findings are only 'suggestive', and to be honest, I'm actually surprised when something DOES replicate. That is what 'suggestive' means. Psychology research is far different from other scientific fields as you point out, but there is motivation to try to make 'findings' more concrete than they really are.
I would group the soft sciences in with the Humanities. That’s not meant to diminish their value but it is important to keep in mind that just because you’re using the scientific method that doesn’t mean you’re gonna get a lot of scientifically valid answers. If gravity failed 1 time out of every 200 we’d be having some major problems…
Speaking as an average, unremarkable, ordinary person, it’s not so literal for most of us. It’s an overall justified mistrust of academia, media, government…
how does it feel that your whole profession is a scam?
Well Explained Pete, looking for our first talk
How did she get to be a professor? I'd warrant it has nothing to do with her academic prowess, but rather how friendly she was to the existing quackademia.
I'm very close with people who are lifelong academics. It's basically a system that trades time and money for respect. There is no meritocracy to speak of. If you SPEND time and money, you're BUYING respect. You don't actually have to learn or know anything, some of these people are the dumbest people I've ever talked to. You just need to put in the time and pay them the tuition fees.
It's sort of like Scientology, the more money you put in the higher your thetan level goes.
Then I watch these people go out into the real world and directly use that purchased respect to gain leadership roles in organizations that they then run off of a cliff.
*How* prevalent do you feel/think/estimate academic fraud is?
Some estimates I’ve seen have been as much as 40%. 4/10 papers have some kind of fraud or at least poor methodology.
@@PeteJudo1 Thanks :)