TL;DW: crinacle.com/2021/07/06/why-most-ratings-suck-an-analysis/ for the deja vu bois yes, this is a reupload, originally uploaded on 1st November 2021 on the original crinacle channel before it was aliven't. RIP to OG crinacle :(
@@diazzsama It seems that we both disagree but you're massively more aggressive about it. Crin himself warns us all not to take what he says at face value, it's his opinion and his only what makes the list after all. So avoid being surprised not if but WHEN you disagree with him.
I think a large portion of this is due to people having grown up with the school grading system, where anything below 60 is a F. So when they try to grade something on a 10 point scale, they automatically put the average at 7. And maybe other factors like trying to make more affiliate sales push that up further.
I agree 100% with your assessment. If I want to know more about an IEM or whatever, I will watch as many vids on it as possible and read as many articles about it as well. From that, you can learn what sonic characteristics reviewers like and don’t like. I have done this way and have come out of it mostly well. While this method is no guarantee that you’ll find what you want, it’s better than relying on the opinion of one or two. In other words, watching/reading multiple vids and articles reduces the chance of buying something you may not like.
Hearing is especially individualistic. What may be shouty for one person may be perfect for someone with less sensitivity to those frequencies. Also comfort is different for people. Usually if something is just really bad everyone will agree on it.
@@brandonabbott8093 I have found that there are some that always say this is the next big thing. I like negative reviews actually. Taken with a grain of salt.
Crin with his disclaimers reminds me of the Messiah scene from movie Life of Brian. Crin says "I'm not the Messiah" and the rest of us say "he is the Messiah, only the real Messiah would say he is not the Messiah".
This is the video that made me a fan of Crin. As a consumer who fell for that faulty rating system and bought some crap stuff before, I learned the hard way on what the rating actually means.
You're absolutely right, and that's the reason I'm so glad to have found you. I need a reviewer who doesn't hesitate to catalog headphones as trash if they are.
One of the more extreme case of this is like AliExpress store reviews. So this is how it goes: DO NOT BUY from stores blow 95% positive feedback. yes. below 95% is BAD. meaning the lowest score you should be seeing when buying product from a store is 4.8/5 star. Problem : 99% of stores are around 95%, which makes this number nothing more than clear scam indication. yes, same problem is also present in every single website with reviews. so what 4 star or 5 star means is not awful / scam, nothing more and not "it's good"
There are other factors not mentioned. Reviews rarely seem to be adjusted for time. With advances in technology, particularly with a newer type of technology, which hasn't matured, a product may get 5 stars but within a year may go down to 4 stars and within a couple of years be 3 stars but because reviews were done upon release, they do not reflect newer technology or in some cases, firmware updates, which can fix issues and sometimes dramatically improve a product. I realise that the examples I am about to give are camera but this is where my expertise is, whereas I just have a bit of audio knowledge. The Fujifilm X-H1 was released to so so reviews, with people criticising the autofocus. Four months later, Fujifilm released a firmware update, which fixed most of the issues, but the vast majority of reviewers had moved on, so the review were out of date but people checking out the reviews, are going to see the outdated review and go by that. One the other end of the scale you have a Sony a7rII from 2015, which is rated at 90% on DP review but technology has moved on so much it is completely meaningless but the review still stand.
I am not so sure about time and improvements. Digital went through rapid progress and you needed a new system every year and mobile phones changes exponential. But I have a fast i7 computer from 2013 with internal cards 15 years old using a 12 month 4K screen. My mobile phone is getting old and possible needs a new battery that I can't fit but has a 13Mp camera that I like. The current top model (10 times as much) has a 13Mp camera that I don't like. Zoom, wide, triple lens, etc don't make up for it. For several years I have searched for a more modern replacement for my speakers. Not to get better just to not do worse and in being more modern hopefully more change of maintenance and repairs. Prices have risen five or six times higher than inflation compared to what I paid decades ago. Some with near perfect hearing mine say the modern speakers is far better. Aging ears over 40 years might not hear that detail. But clearly inferior models are costing far more than I could afford. Then we have the new update problem. Much of hi-fi in streaming, digital servers, DSP, life style and room correction requires an app. The OS updates and the app will fail. The new product needs a new app and that will not run on any phone more than a month old. Then the steaming service and your choice list goes bust and you lost it all. Your internet radio uses a package that your choice of radio stations does not support and blocks (BBC national radio is blocking a number on internet radio software and devices in 2023 where you had all your options in one place will need different less friendly ways to get them. Installing a software update bricks your equipment. Features you used disappear on the next update and you have no use for the multi-thousand (£,$,€) system anymore or have to spend out of devices you don't ever want to get nearly back what had worked.
If you’re running out of ideas for videos, maybe you could start getting into how and why iems are designed/tuned the way they are. Especially how tuning (with sound tubes and screens to decrease air flow) can affect the performance of drivers. There’s vanishingly little about any of this anywhere and it’s a fascinating subject.
Hi Crin, great video and I agree, ratings on average tend to be high. One thing I'd like to add is on the selective reviewing. Sure if they don't publish a review we don't know, but can't we just find out? You have a large dataset of reviews (the largest), you could pull the ratings from the other sites (which you already did), and match them up to yours (a simple vlookup on a spreadsheet should work). From there, you'll see what's missing as well as what you rated those at (see if they really are below avg) and maybe find other stats too. I think this would be interesting data to look at and could possibly even be a follow up video.
Exactly the point I was going to make, it it wasn’t already in the comments. Other review sites cover a smaller number of products, and likely ones readers are most likely to search for, meaning they may naturally have fewer lower-quality products in their dataset than day someone who, say, reviews airline headphones for fun. And if this were true, removing from Crin’s distribution the reviews for products his competitors didn’t cover would cause his average rating to increase as well. FWIW, a follow-up analysis that controls for selection bias would likely still lead to the same damning assessment of his competitors.
Thank you for putting things in perspective and cudos to the disclaimers you put on top of your headphone and IEM ranking lists, - the fact that you explain the science behind your rating system and still emphasize that in the end it's subjective, makes me trust you a lot more than someone who claims to be 100% correct on everything.
Hey crin. Its good that you will upload all of you old channel videos but my advise would be to keep them coming at regular scheduled/intervals so that Algorithm will bless you & your channel will grow in subs get more eyes. 👍
Enjoy how genuine you are about your opinion it's really a breath of fresh air. Also I've just stumbled upon a Google extension that brings likes and dislikes back to RUclips, maybe it's something you would like to shine some light on for other people. Since like and dislike is a form of review as well as comments.
you realize this applies just as well to the argument for the flat earth right? "You can't personally see it, so don't trust it!" I can't see carbon dioxide either, so I guess the greenhouse effect is a myth. I also haven't ever been to space, so I don't trust that for a fucking second. This is one of those lines that sounds good, but it literally only works if you have massive double standards otherwise if fails comically badly.
@@robonator2945 hmmm I see your point, maybe I could've phrased it better. I think I meant more in the subjective realm of things, not scientific truths. But it's late and I can't think words straight so maybe I'll edit op in the morning lmfao.
Think reviews for audio is extra important the prices are so high and physical locations with test demos are extremely limited. Another important factor is company warranty or ez of contacting them.
I think for better or worse, peoples most instinctive thought process about the ten point scale in the US anyway is to translate to Grades, i.e. 75 is average, 60 is borderline failing, etc. This basically just turns the 10 point scale to a 5 point scale but I think that is why.
Yeah, I like the idea behind the rating system on the website. I like having the products change and evolve with a percentage representation that you do. Keep it up Crin!
I just wanted to comment about giveaway iems i received about a year ago. It was my first nice iem's. I have gotten a lot of use out of them and i am just blown away by them. 10+hours and no fatigue. Crinnacle says its super bassy which just means its normal bassy. I LOVE it. Fiio Crinnacle is the iem but anything crinn is got to be good
I'm one of those guys who say - if I don't like it, I just won't waste my time on it. There are so many good headphones that it makes no sense to dig through the trash. Better to enjoy testing something interesting. Therefore, the ratings are skewed in the positive direction. On the other hand, the ranking system does not take time into account. After a year or two, I would give the headphones a noticeably lower score due to the progress made by Chinese manufacturers. Which, by the way, are also being improved. Also, the rating system rarely takes into account direct comparisons. So ratings are an artificial system that has more of a binary purpose: good or bad. And who is higher - under a huge question. SyncerTech channel.
You mentioning computer reviews made me thing of Gamer's Nexus (they're not the only one but the most virulent I'd say). More so at the end where you stress that reviewers should cater to their audiences I think you're absolutely right. And sorry for dropping the channel I did earlier (don't know if it's a problem or not, if it is I'll edit) but I have a lot of respect for them exactly for that reason, they don't shy away from calling out manufacturers or vendors etc, in a industry they love. And that's how it should be. The interwebs as a whole is a wonderful tool, especially nowadays, for making money. And for communication : for ppl with the same hobby but different culture. It's not TV, ads should be ads, there are spaces for that. If I want a friendly face selling me shit, I don't need to start my computer and search for things to find that. I like my youtubers and review sites, but I feel these are less and less relatable, and worse, trustworthy
Honestly, i wish there was a standard in grading something, since this is a product, we can set standard based on the specification of the product while also rely on human review. (Example such as driver size, the KHz range, or anything else) it would be not that perfect, but it was a ground base for reviewing the product more detail. It would make so much easier to "judge" the product we are buying while also knowing that if the person reviewing has certain bias toward the product.
Excellent video Crinacle. ⭐️ ⭐️ ⭐️ Funny thing is, I had always assumed that your rating system was based on 5 ⭐️s!! I have never trusted online reviews - for the very reasons you mentioned. Backing up your comments with mathematical proof works for me. 👍
suppose the problems with most critics is that they have money riding on it, quite a lot of places want to give things a boosted rating so they boost sales for commission, affiliate programs or their own sales. That is why watching people like Crin, Dankpods, Linus (non sponsored vids) is so helpful. 🙂
ratings are supposed to be a bell curve based on the average at the time, not a exponential curve trying to convince you to click their affiliate link!
This is why I find comparisons most useful on any review. Though I am aware of the "new is better" bias, it usually isn't as bad as a point system bias. Also if Crin's rankings are normally distributed, the Monarch Mk2 is impressive (statistically speaking) at the #1 spot of S tier (1%) and 2 star (3%).
reminds me of when i was in the military and we had annual performance reports with a 1-5 scoring scale. you get this form, fill it out with the bullet points of whatever impressive sounding things you did that year, and it gets submitted. it got absurdly inflated. like 85% of people got a 5/5. if you got less than a 5, you clearly screwed up somewhere. like a 4 either meant you sucked at your job or you got a DUI or something. the only 1/5 i ever heard of had one bullet scribbled on it. "convicted of manslaughter." the intent of the system was for 3/5 to be average. in practice, 3/5 probably meant you were arrested at some point. they "fixed" it by making 4 and 5 limited. each unit only had so many 4's and 5's they could give out, so they had to be selective of who gets them. the trade-off being that it was based on the number of people in the unit, so a smaller unit means you had fewer slots, and if you were a smaller unit that was just a detachment of a larger unit, you had to compete with the main unit for their slots, and they might have no idea what you do all day, so you'll never get them.
Having spent over 20 years in the wine & spirits industry, this issue is not just reserved for the audio industry. I can tell you that for certain. It is pervasive. When producers/manufacturers are “consulted” about their products by the industry publications during trade shows and other events, it’s hard to imagine that bias will not be introduced. You combine that with advertising dollars spent and recency bias and what you get is an ocean of mediocre to pretty good wines that rate 90/100. I mean it was getting ridiculous 7 years ago when I left the business. Now everything that doesn’t rate at least 90 points is summarily dismissed. And the craziest part is that a normie sees this 90 point rating for a $12 bottle of wine and expects it to be nearly as good as the $100 or $1000 bottle that rates 93?!?!? It’s a problem. I’d say that wine and audio are in the same realm of human experience. They are meant to be enjoyable. They can take the sting out of a particularly shitty day. And they can help you relax (in moderation). So what we need more of is HONESTY and less of is inflated “ratings”. People are spending their hard earned cash on these things. Like many, I do not have a ton of disposable income to throw around and if every headphone/iem is at least a 4 star/83 then how the hell am I supposed to know what to buy? This problem is pervasive and it needs to be addressed. Thank you for doing just that Crin.
Hey, this is why I always spend days/weeks looking at reviews for headphones/earbuds and I can never decide. There's too many that are put in a way too positive light.
Some reviewers comment on the A/B comparison requested. If they don't have both set up they can't do a fair comparison. Audio memory is not very accurate. I guess this is correct. I do know I have been impressed on listening to some systems and immediately forgot others as uninteresting but I could not rank them. At a certain point I feel that they are different but which is better is much harder to work out. Good here, not so good there. In isolation would I ever notice or worry about it. But that does extend to clearly widely different systems that when with seriously listening are clearly not in the same league. But I probably could live with it and 80% of the time not want to switch off and switch on the better system. I would however switch quickly from a mono system to stereo, from any earbud I have heard to a decent comfortable headphone at a fraction of the cost and mute the TV to listen through nearly any audio system that might be termed hi-fi. For the poor audiophile it is the nagging case of thinking a small change of component would be a better fit and make a huge difference even when they know that for 20% improvement they have to spend four times as much. But yes a poor choice in matching components can be bettered for no more cash with a better mix of separates - the research is at least very time consuming.
One could say that a rating system in audiphilia is inherently broken since nobody shares the exact same acoustic experience due to differences in biology, acoustic variables and of course musical preferences. In my opinion, an acceptable result may be brought forth only through direct comparison with products around the same price and with similar products (soundwise) of higher or lower -financially defined- tiers.
When I see a 3 star review in What Hifi I know the manufacturer just hasn't bought enough ads from them. I much prefer reviews without ratings in favor of those with measurements and a subjective review. Erin Audio Corner does that really well for speaker reviews. Because you have the data you can also usually see what he mentions in his subjective review, and if you know that isn't an issue for you then you can take that into account. He also doesn't hold punches if a product is really garbage.
bro, great video, your method of ratings makes them much more useful. Another reason for positive bias, is that they typically review good gear, but the ratings are useless as you point out
Very insightful. been on different review sites but you and gizaudio recommendation and ratings works for me. thank you for that saves me a lot of money and I don't have to blind purchase anymore.
You're definitely right about the downsides with affiliate links, incentivization of future review products and all that but I think most of these sites are trying to approach the concept of objective scores from a different perspective than you. If we do in fact want an average score of 5.0/10, (in my head at least) that requires products to be scored relative either to the absolute worst or best products, or a consensus on what is truly the most 'average' product (and average quality of a product category typically changes over time). It's not really feasible to have to re-adjust every score once there is a new highest or lowest standard for a product, which is what would be required to keep those scores accurate. Also credit where credit is due for major sites, to my knowledge The Verge is fairly good at at least handing out scores that aren't all 7, 8, or 9, even if they aren't as objectively scored (and maybe that's the key?), and I've seen plenty of 5, 6, even 3 and 4s from them in the past.
Comparative rating systems are necessary in competitive environments, but using the worst and best products to define everything else in the middle will exaggerate quality differences, which may seem useful in overall rankings, but comparing minutiae becomes incredibly quickly convoluted, especially if price is a factor that influences rating. Even though price is the first factor a buyer considers imo it complexifies ratings in a way that each individual buyer should be doing on their own. What the grading systems you are criticizing fail at is by considering the bottom of their scale as being nothing, unobtainable except by random objects that are nothing close to IEMs or Headphones, while 100 is nearly obtainable by a great product, on top of the failures you mentioned. I am not a statistician by far, but hearing that you care about that shows your passion.
I agree 99.999% but let me play devil's advocate. 1. We would not necessarily expect a 'normal distribution' of quality from mature product categories. So maybe most headphones are decent enough to be good for the average user. 2. Headphones that get reviewed a lot are on average more notable in some way so maybe that skews things towards the positive. There is a lot of unreviewed crap. 3. Smaller audiophile publications are less likely to review crap, so I wouldn't expect a lot of failing scores. But I agree that their consistency isn't any better And I agree that for enthusiasts of anything, reading typical reviews isn't going to provide much useful subjective assessment information. And your points are all well taken.
DON'T TRUST THE RATINGS, TRUST THE RANDOM CHINESE MAN ON RUclips I'm joking btw, this is the best channel when it comes to audio, as a producer I almost went all crazy about picking headphones and a dac, but this guy made me really take a different, simple approach, good channel!
I hate star rating systems and refuse to contribute to the problem now. Especially around service reviews for things like hotels, delivery services, transport, food, etc. Most use a 5 star system. If you get average service and provide a 2.5/5, you'll end up getting angry emails and calls. There's no room for exceptional service. You have to leave room for the exceptions. Since you're using a database, if I were in your shoes Crin, I'd make the tier list computed based on prior reviews. Provide a sorting mechanisms for reviews best to worst, and calculate the ratings each time the order moves.
What Hi-FI have subjective ratings for HDMI cables, talking about colour gamut, and warmth, and that kind of nonsense. And that tells me all I need to know about What Hi-Fi reviews.
"...oh you sweet summer child..." shows how much you know... i was born during winter :) love me some crin - his delivery is dryer than lint and he's as funny as a mircofiber cloth is fuzzy. cheers mate!
I find it all largely circles back to our current world of on demand services and metrics people see in everyday life. You never want a rideshare below 4.7 rating. If you have a good ride you are expected to rate 5 stars because you didn't die and made it there. If your food wasn't stolen and made it you rate 5 stars. If you rate 4 stars in an app you have a mandatory, "OMG WHAT WENT WRONG!?!" We're in a conditioned world where people don't actually rate things in a granular way.
For a long time I occasionally used What hifi to list items to consider buying. The thing is to try, and listen, and then decide. If your going to buy on line without listening at least read the reviews from different media to give yourself a fighting chance of buying something that suits your taste in sound, and budget (not the best way...... But I've done this myself, and in general got want I wanted at my budget).
Most people read reviews because they are looking for reasons to buy something so reviewers are not rewarded for bad ratings in the same way they are for good ones.
I agree with PCMag. Tbh, since building my first PC, this is the first time I heard about PCMag thanks(?) to your channel. The fact that I havent heard of it from friends means nobody really listens to them.
Since you can only either have or not have a product, its review should essentially boil down to whether or not the product is good enough to buy. There’s no in-between. You can’t “7/10” buy a product, it’s either buy or skip. The problem is people don’t want to read the review and make an informed buy/skip decision based on whether the product fits their specific needs, they want to be spoonfed an easy answer from the review.
What i think is they should have community rating and not just the website rating. Probably some headphones would be lower but the ones that truly deserve a higher rating would be well rated.
Dope editorial, made me think about his hamburger review, I did not agree with his #1 pick. Made me wonder if he made that as a meme video just to prove a point.....
i want to ask... last year when sg opened up, i went there.... i was hoping to eat black/white carrot cake.... where are all the carrot cake stalls???.... it feels like almost all of them disappeared.... have they come back?
Crin! Your rating system's distribution has five distinct modes because you are reluctant to grade things with a plus or a minus!! Just look at how there's a big spike at every letter!!!! It seems even you have your own biases :)
The selective reviewing thing is a great thing to bring up because even What Hifi has basically no Dan Clark Audio headphones and I would argue that they are very good
I appreciate how you rate things though, especially compared to many others in this space. I always use your input to gauge things vs other reviews. We each have preferences and I find that Josh Valour has very similar preferences to me, however, I use every source I can to compare before making a purchase. Unfortunately, I don't know of a high-end dealer nearby so I am unable to audition a lot of these audio products before purchasing them. I bought a Fiio K9 pro AMK model and the Arya stealth shortly after release. I absolutely love the Aryas and only have one complaint. They can sound metallic on some female vocals. I was mostly able to fix that through minor EQ though at their price point I was hoping they would not need any EQ. That said it is mostly when paired with the Fiio k9 pro. The k9 pro is one of the clearest all in one units I've heard but I should have known pairing an ultra-clear unit with a mid-fi to high-fi planner magnetic may not be the best pairing. I also have an iFi Hip Dac which I love and that paired with the Aryas is honestly wonderful. It helps to remove a bit of the harshness in female vocals but listing to strings and classical music with the k9 pro is insane. The level of clarity and reproduction of tone is the most true to life I've ever heard a set of headphones or speakers get to being there live. Being a classically trained musician that really blew me away and made me decide to keep the K9 pro. I think the balance of the overall system is most important and I hope to be able to attend a can jam or another event with dealers there sometime to try more gear in person. There is obviously a huge difference in price and feature set to my hardware now but I love everything I have. Please do keep being objective and keep reminding us that it's important to read between the lines on reviews, and most importantly not to base a purchasing decision on only one person's opinion or review.
what if one was to use a *relative* grading system, where products are compared against each other and put either in front or behind another product, then update the grades of each product on placement. And to make these more accurate, there can be "padding" applied by how close or far the product is from its comparison (so an almost as good product would have a closer rating to the one right above compared to one that's marked nowhere as good) An example scenario: List of headphones - A really good headphone [10/10] - decent headphone [4/10] - an okay headphone [3.3/10] - a terrible headphone [0/10] > The "Good set of headphones" is much better than the "decent headphones", but nowhere as good as "A really good headphone" Modified list of headphones - A really good headphone [10/10] - Good set of headphones [6/10] - decent headphones [3.5/10] - an okay headphone [3/10] - a terrible headphone [0/10] > The "manageable headphones" are far better than "a terrible headphone" and just shy of passing "an okay headphone" Modified modified list of headphones - A really good headphone [10/10] - Good set of headphones [6.2/10] - decent headphones [3.6/10] - an okay headphone [3/10] - manageable headphones [3/10] - a terrible headphone [0/10] etc. The ratings should be done by actual math (instead of just guessing like I'm doing), but that's the basics. I mean I know that it won't be easy comparing one headphone to another if they're meant for two completely different markets, but that may be solved by having multiple pages of the list that list off the headphone ratings according to each market sector.
What this means is that most products are actually pretty good and get the job done. Zero out of ten means the product doesn’t work, ten out of ten means absolute perfection with nothing to criticise. The majority of products will score seven or eight of ten because they work and they don’t have any major failings.
If I had a rating website I would do a 10 point scale and only rate them between 5-10, but I would try to keep each with 20% of the products (20% of the products between 5.0 and 6.0, etc.)
While I understand the problem with incentivized reviews, I personally think the higher than average ratings could be a sign of headphones getting to a point where, on average, headphones are great. Purchasing a headphone, speaker, synth, monitor, hardware that rates higher than 8.5 is typically a VERY expensive endeavour. Looking at Crinacle's ranking list, any headphone ranking A- or higher is personally out of my price range ($59,000 - 350), possibly aside from the ATH-R70 ($350) & HD600 ($400), everything else is $900 and up. So, for the average consumer, average might be different, and a website that can garner millions of views, might have a rating system that appeals to more people, hence better on average. (for those who may not know: "popular" isn't the most LIKED, its the most "dealt with", the song most people don't mind listening to.) The same ranking list "value rating" has many B's, some A, and even some S's, therefore skewing the average rating up. When you look at the median rating its actually between a D and E. So, there is an argument that Crinacle could be negatively skewed. MKBHD says it all the time, most "pro" gear is unnecessary for most people. Devils advocate: Headphones can be tested objectively and therefore should be normally distributed. Counter: Audio Technology may have reached a maturity where a "decent" headphone is attainable for a price the "lower than average person" can buy, finally. i.e. headphones made today are just by default, "decent" you have to try to fuck up, and BOY oh BOY! do some brands try.
For the smaller sites, this can be considered a problem. For the big sites your math is “wrong”. They are not using the full scale, nor should they. Normies will not see 5/10 and think, ah average probably good enough for me. They will see “bad”. Rating the average around a 7/10 or 4/5 psychologically speaking will mean average to most people, even though numerically it shouldn’t. So whilst it definitely makes these scores less useful especially a 5 point scale with no decimal, it does a good job at what it’s trying to achieve, communicate to an uninformed buyer what they can expect. A bigger problem is them rating below average products as average or higher in their custom scales.
9:36 That's basicly how videogame reviews also work nowadays. Years ago 7/10 meant pretty good, but now the scale starts at 7 and games will be put to shame for it getting it.
Well this is so true and very glad to hear this because i also read reviews from these sites...finally...i know the truth, but there is one more thing, there are tons of youtubers doing the same shit...getting free stuffs, promoting the hell out of it, and you always hear...got it for free to review but by no means its my own opinion.....just hyping stuff is s great job of youtubers but finally there is someone who reveals the truth
I think in a lot of industries, mass produced products are rarely ‘trash’ or truly horrible to the point of granting a low rating. Most of the things in the world are ‘pretty good’, 7/10, nothing special but not bad either. So rather than the problem of ‘not using the whole scale’, not using all the scores like a normal distribution is simply a reflection of real life. In the modern days rarely anything is truly trash.
I used to follow the video game site GameTrailers and they never gave a game a full 100 (highest given was a 98). It's unfortunate that they shut down due to low traffic.
Whenever I look for reviews for a headset or something, it are always those named-and-shamed websites that come up first on Google. As someone who knew decades ago already not to trust IGN and kin with their video game ratings, I know to cross-reference and seek differing opinions across review sites, forums and buddies. Also engage that 'common sense' feature with any claims made. To stick with the video game rating systems, Angry Joe's channel uses a '5/10 is average' system, with 6/10 above average. This means that by the time you hit 8/10, it's already pretty darn good, 9/10 hits epic levels and 10/10 basically never happens. Always felt rather sensible as a rating system to me.
So statistically when asking others or yourself trying to make a thoughtful choice. As an example O-5-10 scale doesn't do this. Try 2-7- 16 makes them ask if it's not fabulous but it's not bad, maybe 7 ok but definitely not 16 and not 2. Or could be a 4 but not a seven so it's a 2
I think that manufacturers that care about their products (and I firmly believe that at most of them do care at some degree) on a long run will profit from critical review because of useful information that can actually take into account, not only sales data.
True. According to many ratings the Melomania 1+ is "the best sounding TWS" but according to me, it's dogshit and losing against my CCA C10 in every category, and that thing can be used wireless too, which STILL sounds better than the Melomania 1+..
How about a scale with no scores which has a side with professional use thoughts and the other with consumer taste . A scale vertically with two pointing arrows
![[#2024MAMA] ROSÉ (로제), Bruno Mars - APT. | Mnet 241122 방송](http://i.ytimg.com/vi/dgGqD28J6aQ/mqdefault.jpg)
TL;DW: crinacle.com/2021/07/06/why-most-ratings-suck-an-analysis/
for the deja vu bois yes, this is a reupload, originally uploaded on 1st November 2021 on the original crinacle channel before it was aliven't. RIP to OG crinacle :(
Press f to pay respects.
infinite deja vu
Didn't see it so thanks I think
Aha I was wondering the whole time, bc the ign meme isn’t anywhere nowadaysxD
at least make a giveaway!!
Support the supporters who support the people who support you!
When your rig is ranked B 😥, but it's on Crinacle's scale 😎
My iems are B- 😥, but it's 2 stars 😎
They are the Tin HiFi T4
@@t1995roger10 tfw A+ tone but C+ technical (I can barely hear cymbals please send EQ)
When your headphones only have 3 stars, but on crinacle value rating
@@fastgecko5799 Meanwhile in hear them a bit too clearly with my BLON BL-03. Wonder if something's wrong with my ears.
@@diazzsama It seems that we both disagree but you're massively more aggressive about it.
Crin himself warns us all not to take what he says at face value, it's his opinion and his only what makes the list after all.
So avoid being surprised not if but WHEN you disagree with him.
This video was literally unwatchable with eyes closed! 7/10 © IGN
"A miserable experience for even the biggest masochist" 8.3/10 Headfonics
"A travesty to have both eyes and ears while experiencing this" 7.5/10 Soundguys
"Absolutely appalling, my wife and daughter cried and screamed in agony." 5✩ WhatHifi.
"This video made me realise what holocaust vicitms went through!" 9.1/10 IGN
“I wasn’t able to watch this video. It was stuck buffering.” 8.9/10 IGN
Crinacle: *telling me every video to not trust him*
Me: "I trust this man with my life"
Rewatching to feed the algorithm.
The algorithm is pleased by your offering. The fourth version of ectoplasmus laporidae is clearly the superior iteration. 8/10.
Replying to feed the algorithm
Hay Doug I'm doing this for the waffle house and the algorithm juice 🧃
i dumped my girlfriend for the algorithm
This worked btw. I got a recommend 2 days later
I think a large portion of this is due to people having grown up with the school grading system, where anything below 60 is a F. So when they try to grade something on a 10 point scale, they automatically put the average at 7. And maybe other factors like trying to make more affiliate sales push that up further.
When being an 8/10 is now below average :(
That's not the same for a lot of the world
@@kainan613 ok. It’s the way it is in America, and all sites referenced by Crin in this video are American.
@@michaelmacvittie6977 Doesn't mean their users are. Crin isn't
@@kainan613 Ok. It’s Americans making an American system. Get used to it.
Exactly the reason to look at multiple reviews. Or find a reviewer you have the most in common with.
When the overwhelmingly vast majority of review sites act like this, it makes that pretty meaningless. Unfortunately.
I agree 100% with your assessment. If I want to know more about an IEM or whatever, I will watch as many vids on it as possible and read as many articles about it as well. From that, you can learn what sonic characteristics reviewers like and don’t like. I have done this way and have come out of it mostly well. While this method is no guarantee that you’ll find what you want, it’s better than relying on the opinion of one or two. In other words, watching/reading multiple vids and articles reduces the chance of buying something you may not like.
Hearing is especially individualistic. What may be shouty for one person may be perfect for someone with less sensitivity to those frequencies. Also comfort is different for people. Usually if something is just really bad everyone will agree on it.
@@brandonabbott8093 I have found that there are some that always say this is the next big thing. I like negative reviews actually. Taken with a grain of salt.
That's why I usually don't look at ratings and instead read/watch the review itself.
Crin with his disclaimers reminds me of the Messiah scene from movie Life of Brian. Crin says "I'm not the Messiah" and the rest of us say "he is the Messiah, only the real Messiah would say he is not the Messiah".
crinacle review list is the one true list we should trust
LISAN AL-GAIB. AS IT WAS WRITTEN
This is the video that made me a fan of Crin. As a consumer who fell for that faulty rating system and bought some crap stuff before, I learned the hard way on what the rating actually means.
What Hi-Fi be like:
Brad Pitt ⭐⭐⭐⭐⭐
Heinrich Himmler ⭐⭐⭐⭐
Adolf H ⭐⭐⭐⭐
Kinda... not same and messed up in a way, but what you're saying is definitely not false, just a weird way of saying it.
You're absolutely right, and that's the reason I'm so glad to have found you. I need a reviewer who doesn't hesitate to catalog headphones as trash if they are.
One of the more extreme case of this is like AliExpress store reviews.
So this is how it goes: DO NOT BUY from stores blow 95% positive feedback. yes. below 95% is BAD.
meaning the lowest score you should be seeing when buying product from a store is 4.8/5 star.
Problem : 99% of stores are around 95%, which makes this number nothing more than clear scam indication.
yes, same problem is also present in every single website with reviews.
so what 4 star or 5 star means is not awful / scam, nothing more and not "it's good"
There are other factors not mentioned. Reviews rarely seem to be adjusted for time. With advances in technology, particularly with a newer type of technology, which hasn't matured, a product may get 5 stars but within a year may go down to 4 stars and within a couple of years be 3 stars but because reviews were done upon release, they do not reflect newer technology or in some cases, firmware updates, which can fix issues and sometimes dramatically improve a product.
I realise that the examples I am about to give are camera but this is where my expertise is, whereas I just have a bit of audio knowledge. The Fujifilm X-H1 was released to so so reviews, with people criticising the autofocus. Four months later, Fujifilm released a firmware update, which fixed most of the issues, but the vast majority of reviewers had moved on, so the review were out of date but people checking out the reviews, are going to see the outdated review and go by that. One the other end of the scale you have a Sony a7rII from 2015, which is rated at 90% on DP review but technology has moved on so much it is completely meaningless but the review still stand.
I am not so sure about time and improvements. Digital went through rapid progress and you needed a new system every year and mobile phones changes exponential. But I have a fast i7 computer from 2013 with internal cards 15 years old using a 12 month 4K screen. My mobile phone is getting old and possible needs a new battery that I can't fit but has a 13Mp camera that I like. The current top model (10 times as much) has a 13Mp camera that I don't like. Zoom, wide, triple lens, etc don't make up for it.
For several years I have searched for a more modern replacement for my speakers. Not to get better just to not do worse and in being more modern hopefully more change of maintenance and repairs. Prices have risen five or six times higher than inflation compared to what I paid decades ago. Some with near perfect hearing mine say the modern speakers is far better. Aging ears over 40 years might not hear that detail. But clearly inferior models are costing far more than I could afford.
Then we have the new update problem. Much of hi-fi in streaming, digital servers, DSP, life style and room correction requires an app. The OS updates and the app will fail. The new product needs a new app and that will not run on any phone more than a month old. Then the steaming service and your choice list goes bust and you lost it all. Your internet radio uses a package that your choice of radio stations does not support and blocks (BBC national radio is blocking a number on internet radio software and devices in 2023 where you had all your options in one place will need different less friendly ways to get them. Installing a software update bricks your equipment. Features you used disappear on the next update and you have no use for the multi-thousand (£,$,€) system anymore or have to spend out of devices you don't ever want to get nearly back what had worked.
I like this studio setup much better!)) It looks so genuine and real, I think some platforms and channels should never get redesigned)
If you’re running out of ideas for videos, maybe you could start getting into how and why iems are designed/tuned the way they are. Especially how tuning (with sound tubes and screens to decrease air flow) can affect the performance of drivers. There’s vanishingly little about any of this anywhere and it’s a fascinating subject.
That makes sense soooooo much. I would have never thought about it if it wasn’t for this video. Thank you!
Hi Crin, great video and I agree, ratings on average tend to be high.
One thing I'd like to add is on the selective reviewing. Sure if they don't publish a review we don't know, but can't we just find out? You have a large dataset of reviews (the largest), you could pull the ratings from the other sites (which you already did), and match them up to yours (a simple vlookup on a spreadsheet should work). From there, you'll see what's missing as well as what you rated those at (see if they really are below avg) and maybe find other stats too. I think this would be interesting data to look at and could possibly even be a follow up video.
Exactly the point I was going to make, it it wasn’t already in the comments.
Other review sites cover a smaller number of products, and likely ones readers are most likely to search for, meaning they may naturally have fewer lower-quality products in their dataset than day someone who, say, reviews airline headphones for fun.
And if this were true, removing from Crin’s distribution the reviews for products his competitors didn’t cover would cause his average rating to increase as well.
FWIW, a follow-up analysis that controls for selection bias would likely still lead to the same damning assessment of his competitors.
Cant wait till all the old stuff is up on here, love rewatching it ngl
Salute for this mans for being humble and truthful AF
Agreed. Crin rocks!
And then he make the rating him self.. what a men?
Thank you for putting things in perspective and cudos to the disclaimers you put on top of your headphone and IEM ranking lists, - the fact that you explain the science behind your rating system and still emphasize that in the end it's subjective, makes me trust you a lot more than someone who claims to be 100% correct on everything.
Hey crin. Its good that you will upload all of you old channel videos but my advise would be to keep them coming at regular scheduled/intervals so that Algorithm will bless you & your channel will grow in subs get more eyes. 👍
New here, what happened to the old channel ?
@@readyforlol same question
Enjoy how genuine you are about your opinion it's really a breath of fresh air. Also I've just stumbled upon a Google extension that brings likes and dislikes back to RUclips, maybe it's something you would like to shine some light on for other people. Since like and dislike is a form of review as well as comments.
GIVE THIS GUY THE MOST VIEWS!!!
He gives us honesty!
Never full trust anything you can't experience yourself lol
you realize this applies just as well to the argument for the flat earth right? "You can't personally see it, so don't trust it!" I can't see carbon dioxide either, so I guess the greenhouse effect is a myth. I also haven't ever been to space, so I don't trust that for a fucking second.
This is one of those lines that sounds good, but it literally only works if you have massive double standards otherwise if fails comically badly.
@@robonator2945 hmmm I see your point, maybe I could've phrased it better. I think I meant more in the subjective realm of things, not scientific truths. But it's late and I can't think words straight so maybe I'll edit op in the morning lmfao.
With audio, this becomes even harder as one persons great is another persons okay
Trust frequency response
@@feelingevaporated2912 Saying that, my fave iem to date is the Tin Hifi t4 which has quite a higher treble. Some reviewers didn't like it
Think reviews for audio is extra important the prices are so high and physical locations with test demos are extremely limited. Another important factor is company warranty or ez of contacting them.
I think for better or worse, peoples most instinctive thought process about the ten point scale in the US anyway is to translate to Grades, i.e. 75 is average, 60 is borderline failing, etc. This basically just turns the 10 point scale to a 5 point scale but I think that is why.
Precise and reasonable. Maximum respect for you crinacle. Huge audio and statistic knowledge hahhaha. Cheers 🍻
Yeah, I like the idea behind the rating system on the website. I like having the products change and evolve with a percentage representation that you do. Keep it up Crin!
I just wanted to comment about giveaway iems i received about a year ago. It was my first nice iem's. I have gotten a lot of use out of them and i am just blown away by them. 10+hours and no fatigue. Crinnacle says its super bassy which just means its normal bassy. I LOVE it. Fiio Crinnacle is the iem but anything crinn is got to be good
I'm one of those guys who say - if I don't like it, I just won't waste my time on it. There are so many good headphones that it makes no sense to dig through the trash. Better to enjoy testing something interesting. Therefore, the ratings are skewed in the positive direction. On the other hand, the ranking system does not take time into account. After a year or two, I would give the headphones a noticeably lower score due to the progress made by Chinese manufacturers. Which, by the way, are also being improved. Also, the rating system rarely takes into account direct comparisons. So ratings are an artificial system that has more of a binary purpose: good or bad. And who is higher - under a huge question. SyncerTech channel.
😮😮😝😊😅
I can predict the answer from whathifi to this. "We only review the best of best".
You mentioning computer reviews made me thing of Gamer's Nexus (they're not the only one but the most virulent I'd say). More so at the end where you stress that reviewers should cater to their audiences
I think you're absolutely right. And sorry for dropping the channel I did earlier (don't know if it's a problem or not, if it is I'll edit) but I have a lot of respect for them exactly for that reason, they don't shy away from calling out manufacturers or vendors etc, in a industry they love. And that's how it should be. The interwebs as a whole is a wonderful tool, especially nowadays, for making money. And for communication : for ppl with the same hobby but different culture. It's not TV, ads should be ads, there are spaces for that. If I want a friendly face selling me shit, I don't need to start my computer and search for things to find that.
I like my youtubers and review sites, but I feel these are less and less relatable, and worse, trustworthy
Honestly, i wish there was a standard in grading something, since this is a product, we can set standard based on the specification of the product while also rely on human review. (Example such as driver size, the KHz range, or anything else) it would be not that perfect, but it was a ground base for reviewing the product more detail.
It would make so much easier to "judge" the product we are buying while also knowing that if the person reviewing has certain bias toward the product.
Excellent video Crinacle. ⭐️ ⭐️ ⭐️
Funny thing is, I had always assumed that your rating system was based on 5 ⭐️s!!
I have never trusted online reviews - for the very reasons you mentioned.
Backing up your comments with mathematical proof works for me.
👍
suppose the problems with most critics is that they have money riding on it, quite a lot of places want to give things a boosted rating so they boost sales for commission, affiliate programs or their own sales. That is why watching people like Crin, Dankpods, Linus (non sponsored vids) is so helpful. 🙂
ratings are supposed to be a bell curve based on the average at the time, not a exponential curve trying to convince you to click their affiliate link!
This is why I find comparisons most useful on any review. Though I am aware of the "new is better" bias, it usually isn't as bad as a point system bias. Also if Crin's rankings are normally distributed, the Monarch Mk2 is impressive (statistically speaking) at the #1 spot of S tier (1%) and 2 star (3%).
reminds me of when i was in the military and we had annual performance reports with a 1-5 scoring scale. you get this form, fill it out with the bullet points of whatever impressive sounding things you did that year, and it gets submitted. it got absurdly inflated. like 85% of people got a 5/5.
if you got less than a 5, you clearly screwed up somewhere. like a 4 either meant you sucked at your job or you got a DUI or something.
the only 1/5 i ever heard of had one bullet scribbled on it. "convicted of manslaughter."
the intent of the system was for 3/5 to be average. in practice, 3/5 probably meant you were arrested at some point.
they "fixed" it by making 4 and 5 limited. each unit only had so many 4's and 5's they could give out, so they had to be selective of who gets them.
the trade-off being that it was based on the number of people in the unit, so a smaller unit means you had fewer slots, and if you were a smaller unit that was just a detachment of a larger unit, you had to compete with the main unit for their slots, and they might have no idea what you do all day, so you'll never get them.
4/5 is top 75% ??
Would love to see that bell curve justified in class.
Having spent over 20 years in the wine & spirits industry, this issue is not just reserved for the audio industry. I can tell you that for certain. It is pervasive. When producers/manufacturers are “consulted” about their products by the industry publications during trade shows and other events, it’s hard to imagine that bias will not be introduced. You combine that with advertising dollars spent and recency bias and what you get is an ocean of mediocre to pretty good wines that rate 90/100. I mean it was getting ridiculous 7 years ago when I left the business. Now everything that doesn’t rate at least 90 points is summarily dismissed. And the craziest part is that a normie sees this 90 point rating for a $12 bottle of wine and expects it to be nearly as good as the $100 or $1000 bottle that rates 93?!?!? It’s a problem. I’d say that wine and audio are in the same realm of human experience. They are meant to be enjoyable. They can take the sting out of a particularly shitty day. And they can help you relax (in moderation). So what we need more of is HONESTY and less of is inflated “ratings”. People are spending their hard earned cash on these things. Like many, I do not have a ton of disposable income to throw around and if every headphone/iem is at least a 4 star/83 then how the hell am I supposed to know what to buy? This problem is pervasive and it needs to be addressed. Thank you for doing just that Crin.
please make a video about eartips maybe?(best for titan s pls)
Hey, this is why I always spend days/weeks looking at reviews for headphones/earbuds and I can never decide. There's too many that are put in a way too positive light.
Some reviewers comment on the A/B comparison requested. If they don't have both set up they can't do a fair comparison. Audio memory is not very accurate.
I guess this is correct. I do know I have been impressed on listening to some systems and immediately forgot others as uninteresting but I could not rank them. At a certain point I feel that they are different but which is better is much harder to work out. Good here, not so good there. In isolation would I ever notice or worry about it.
But that does extend to clearly widely different systems that when with seriously listening are clearly not in the same league. But I probably could live with it and 80% of the time not want to switch off and switch on the better system. I would however switch quickly from a mono system to stereo, from any earbud I have heard to a decent comfortable headphone at a fraction of the cost and mute the TV to listen through nearly any audio system that might be termed hi-fi.
For the poor audiophile it is the nagging case of thinking a small change of component would be a better fit and make a huge difference even when they know that for 20% improvement they have to spend four times as much. But yes a poor choice in matching components can be bettered for no more cash with a better mix of separates - the research is at least very time consuming.
One could say that a rating system in audiphilia is inherently broken since nobody shares the exact same acoustic experience due to differences in biology, acoustic variables and of course musical preferences. In my opinion, an acceptable result may be brought forth only through direct comparison with products around the same price and with similar products (soundwise) of higher or lower -financially defined- tiers.
When I see a 3 star review in What Hifi I know the manufacturer just hasn't bought enough ads from them. I much prefer reviews without ratings in favor of those with measurements and a subjective review. Erin Audio Corner does that really well for speaker reviews. Because you have the data you can also usually see what he mentions in his subjective review, and if you know that isn't an issue for you then you can take that into account. He also doesn't hold punches if a product is really garbage.
bro, great video, your method of ratings makes them much more useful. Another reason for positive bias, is that they typically review good gear, but the ratings are useless as you point out
Very insightful. been on different review sites but you and gizaudio recommendation and ratings works for me. thank you for that saves me a lot of money and I don't have to blind purchase anymore.
4:24 I like that the videos have a lot of personality, makes them interesting, and dont feel 'dry'
You're definitely right about the downsides with affiliate links, incentivization of future review products and all that but I think most of these sites are trying to approach the concept of objective scores from a different perspective than you. If we do in fact want an average score of 5.0/10, (in my head at least) that requires products to be scored relative either to the absolute worst or best products, or a consensus on what is truly the most 'average' product (and average quality of a product category typically changes over time). It's not really feasible to have to re-adjust every score once there is a new highest or lowest standard for a product, which is what would be required to keep those scores accurate. Also credit where credit is due for major sites, to my knowledge The Verge is fairly good at at least handing out scores that aren't all 7, 8, or 9, even if they aren't as objectively scored (and maybe that's the key?), and I've seen plenty of 5, 6, even 3 and 4s from them in the past.
What are your opinions of rtings? I've been using it and it seems reliable.
I find them good for specs. But lacking for actual use cases. I would never only use them as a solo source.
Comparative rating systems are necessary in competitive environments, but using the worst and best products to define everything else in the middle will exaggerate quality differences, which may seem useful in overall rankings, but comparing minutiae becomes incredibly quickly convoluted, especially if price is a factor that influences rating. Even though price is the first factor a buyer considers imo it complexifies ratings in a way that each individual buyer should be doing on their own.
What the grading systems you are criticizing fail at is by considering the bottom of their scale as being nothing, unobtainable except by random objects that are nothing close to IEMs or Headphones, while 100 is nearly obtainable by a great product, on top of the failures you mentioned. I am not a statistician by far, but hearing that you care about that shows your passion.
Thank God. I never trust your rating and reviews. And I knew I am right!
I agree 99.999% but let me play devil's advocate. 1. We would not necessarily expect a 'normal distribution' of quality from mature product categories. So maybe most headphones are decent enough to be good for the average user. 2. Headphones that get reviewed a lot are on average more notable in some way so maybe that skews things towards the positive. There is a lot of unreviewed crap. 3. Smaller audiophile publications are less likely to review crap, so I wouldn't expect a lot of failing scores. But I agree that their consistency isn't any better
And I agree that for enthusiasts of anything, reading typical reviews isn't going to provide much useful subjective assessment information. And your points are all well taken.
DON'T TRUST THE RATINGS, TRUST THE RANDOM CHINESE MAN ON RUclips
I'm joking btw, this is the best channel when it comes to audio, as a producer I almost went all crazy about picking headphones and a dac, but this guy made me really take a different, simple approach, good channel!
isn’t he singaporean
Surely the affiliate links provide an incentive to rate things higher as well
even though this is a reupload, this video covers such a big issue with modern review systems so it deserves a rewatch
I hate star rating systems and refuse to contribute to the problem now. Especially around service reviews for things like hotels, delivery services, transport, food, etc. Most use a 5 star system. If you get average service and provide a 2.5/5, you'll end up getting angry emails and calls. There's no room for exceptional service. You have to leave room for the exceptions.
Since you're using a database, if I were in your shoes Crin, I'd make the tier list computed based on prior reviews. Provide a sorting mechanisms for reviews best to worst, and calculate the ratings each time the order moves.
Before: Why you SHOULDN'T trust me
Now: Why you SHOULD trust me
What Hi-FI have subjective ratings for HDMI cables, talking about colour gamut, and warmth, and that kind of nonsense.
And that tells me all I need to know about What Hi-Fi reviews.
I love your honesty Crin. Still want you opinion on best IEM under $200 though.
go blind on the heavily discounted AKG N5005.
@@711jastin Wired
"...oh you sweet summer child..."
shows how much you know... i was born during winter :)
love me some crin - his delivery is dryer than lint and he's as funny as a mircofiber cloth is fuzzy. cheers mate!
I find it all largely circles back to our current world of on demand services and metrics people see in everyday life. You never want a rideshare below 4.7 rating. If you have a good ride you are expected to rate 5 stars because you didn't die and made it there. If your food wasn't stolen and made it you rate 5 stars. If you rate 4 stars in an app you have a mandatory, "OMG WHAT WENT WRONG!?!" We're in a conditioned world where people don't actually rate things in a granular way.
For a long time I occasionally used What hifi to list items to consider buying. The thing is to try, and listen, and then decide. If your going to buy on line without listening at least read the reviews from different media to give yourself a fighting chance of buying something that suits your taste in sound, and budget (not the best way...... But I've done this myself, and in general got want I wanted at my budget).
This and Dunkeys "Game Critics" video should be required viewing in the educational system.
I appreciate your sponsors are shops and patrons, and not the manufacturers, which is what killed forums for me.
Most people read reviews because they are looking for reasons to buy something so reviewers are not rewarded for bad ratings in the same way they are for good ones.
Your star rating for IEM pricing reminds me a lot of the Michelin Star system for restaurants.
I agree with PCMag. Tbh, since building my first PC, this is the first time I heard about PCMag thanks(?) to your channel. The fact that I havent heard of it from friends means nobody really listens to them.
Since you can only either have or not have a product, its review should essentially boil down to whether or not the product is good enough to buy.
There’s no in-between. You can’t “7/10” buy a product, it’s either buy or skip.
The problem is people don’t want to read the review and make an informed buy/skip decision based on whether the product fits their specific needs, they want to be spoonfed an easy answer from the review.
What HiFI is the meme of rating websites. Whatever you look for, it's mostly a 5 stars product.
Thanks for video, Crin😗👍
What i think is they should have community rating and not just the website rating. Probably some headphones would be lower but the ones that truly deserve a higher rating would be well rated.
Dope editorial, made me think about his hamburger review, I did not agree with his #1 pick. Made me wonder if he made that as a meme video just to prove a point.....
i want to ask... last year when sg opened up, i went there.... i was hoping to eat black/white carrot cake.... where are all the carrot cake stalls???.... it feels like almost all of them disappeared.... have they come back?
I have found that rtings seems to be unafraid of giving a product any score between 0 and 10
Crin! Your rating system's distribution has five distinct modes because you are reluctant to grade things with a plus or a minus!! Just look at how there's a big spike at every letter!!!! It seems even you have your own biases :)
I mainly look for frequency response of the IEM’s and driver specifications
The selective reviewing thing is a great thing to bring up because even What Hifi has basically no Dan Clark Audio headphones and I would argue that they are very good
I appreciate how you rate things though, especially compared to many others in this space. I always use your input to gauge things vs other reviews. We each have preferences and I find that Josh Valour has very similar preferences to me, however, I use every source I can to compare before making a purchase.
Unfortunately, I don't know of a high-end dealer nearby so I am unable to audition a lot of these audio products before purchasing them. I bought a Fiio K9 pro AMK model and the Arya stealth shortly after release. I absolutely love the Aryas and only have one complaint. They can sound metallic on some female vocals. I was mostly able to fix that through minor EQ though at their price point I was hoping they would not need any EQ. That said it is mostly when paired with the Fiio k9 pro. The k9 pro is one of the clearest all in one units I've heard but I should have known pairing an ultra-clear unit with a mid-fi to high-fi planner magnetic may not be the best pairing. I also have an iFi Hip Dac which I love and that paired with the Aryas is honestly wonderful. It helps to remove a bit of the harshness in female vocals but listing to strings and classical music with the k9 pro is insane. The level of clarity and reproduction of tone is the most true to life I've ever heard a set of headphones or speakers get to being there live. Being a classically trained musician that really blew me away and made me decide to keep the K9 pro.
I think the balance of the overall system is most important and I hope to be able to attend a can jam or another event with dealers there sometime to try more gear in person. There is obviously a huge difference in price and feature set to my hardware now but I love everything I have. Please do keep being objective and keep reminding us that it's important to read between the lines on reviews, and most importantly not to base a purchasing decision on only one person's opinion or review.
Flashback! I've spammed this video on many discussions when people reference What HiFi. Its a gem! 👌
what if one was to use a *relative* grading system, where products are compared against each other and put either in front or behind another product, then update the grades of each product on placement. And to make these more accurate, there can be "padding" applied by how close or far the product is from its comparison (so an almost as good product would have a closer rating to the one right above compared to one that's marked nowhere as good)
An example scenario:
List of headphones
- A really good headphone [10/10]
- decent headphone [4/10]
- an okay headphone [3.3/10]
- a terrible headphone [0/10]
> The "Good set of headphones" is much better than the "decent headphones", but nowhere as good as "A really good headphone"
Modified list of headphones
- A really good headphone [10/10]
- Good set of headphones [6/10]
- decent headphones [3.5/10]
- an okay headphone [3/10]
- a terrible headphone [0/10]
> The "manageable headphones" are far better than "a terrible headphone" and just shy of passing "an okay headphone"
Modified modified list of headphones
- A really good headphone [10/10]
- Good set of headphones [6.2/10]
- decent headphones [3.6/10]
- an okay headphone [3/10]
- manageable headphones [3/10]
- a terrible headphone [0/10]
etc.
The ratings should be done by actual math (instead of just guessing like I'm doing), but that's the basics. I mean I know that it won't be easy comparing one headphone to another if they're meant for two completely different markets, but that may be solved by having multiple pages of the list that list off the headphone ratings according to each market sector.
What this means is that most products are actually pretty good and get the job done. Zero out of ten means the product doesn’t work, ten out of ten means absolute perfection with nothing to criticise. The majority of products will score seven or eight of ten because they work and they don’t have any major failings.
In the review world 1-6 is Negative, 7-8 is Neutral and only 9-10 is Positive
If I had a rating website I would do a 10 point scale and only rate them between 5-10, but I would try to keep each with 20% of the products (20% of the products between 5.0 and 6.0, etc.)
Crin, please make a review video of the Tangzu Heyday.
While I understand the problem with incentivized reviews, I personally think the higher than average ratings could be a sign of headphones getting to a point where, on average, headphones are great. Purchasing a headphone, speaker, synth, monitor, hardware that rates higher than 8.5 is typically a VERY expensive endeavour. Looking at Crinacle's ranking list, any headphone ranking A- or higher is personally out of my price range ($59,000 - 350), possibly aside from the ATH-R70 ($350) & HD600 ($400), everything else is $900 and up. So, for the average consumer, average might be different, and a website that can garner millions of views, might have a rating system that appeals to more people, hence better on average. (for those who may not know: "popular" isn't the most LIKED, its the most "dealt with", the song most people don't mind listening to.) The same ranking list "value rating" has many B's, some A, and even some S's, therefore skewing the average rating up. When you look at the median rating its actually between a D and E. So, there is an argument that Crinacle could be negatively skewed.
MKBHD says it all the time, most "pro" gear is unnecessary for most people.
Devils advocate: Headphones can be tested objectively and therefore should be normally distributed.
Counter: Audio Technology may have reached a maturity where a "decent" headphone is attainable for a price the "lower than average person" can buy, finally. i.e. headphones made today are just by default, "decent" you have to try to fuck up, and BOY oh BOY! do some brands try.
For the smaller sites, this can be considered a problem. For the big sites your math is “wrong”. They are not using the full scale, nor should they. Normies will not see 5/10 and think, ah average probably good enough for me. They will see “bad”. Rating the average around a 7/10 or 4/5 psychologically speaking will mean average to most people, even though numerically it shouldn’t.
So whilst it definitely makes these scores less useful especially a 5 point scale with no decimal, it does a good job at what it’s trying to achieve, communicate to an uninformed buyer what they can expect.
A bigger problem is them rating below average products as average or higher in their custom scales.
9:36
That's basicly how videogame reviews also work nowadays.
Years ago 7/10 meant pretty good, but now the scale starts at 7 and games will be put to shame for it getting it.
Well this is so true and very glad to hear this because i also read reviews from these sites...finally...i know the truth, but there is one more thing, there are tons of youtubers doing the same shit...getting free stuffs, promoting the hell out of it, and you always hear...got it for free to review but by no means its my own opinion.....just hyping stuff is s great job of youtubers but finally there is someone who reveals the truth
I think in a lot of industries, mass produced products are rarely ‘trash’ or truly horrible to the point of granting a low rating. Most of the things in the world are ‘pretty good’, 7/10, nothing special but not bad either. So rather than the problem of ‘not using the whole scale’, not using all the scores like a normal distribution is simply a reflection of real life. In the modern days rarely anything is truly trash.
I used to follow the video game site GameTrailers and they never gave a game a full 100 (highest given was a 98). It's unfortunate that they shut down due to low traffic.
People usually just think of the school grading system, so they think 7/10 is average.
Whenever I look for reviews for a headset or something, it are always those named-and-shamed websites that come up first on Google. As someone who knew decades ago already not to trust IGN and kin with their video game ratings, I know to cross-reference and seek differing opinions across review sites, forums and buddies. Also engage that 'common sense' feature with any claims made.
To stick with the video game rating systems, Angry Joe's channel uses a '5/10 is average' system, with 6/10 above average. This means that by the time you hit 8/10, it's already pretty darn good, 9/10 hits epic levels and 10/10 basically never happens. Always felt rather sensible as a rating system to me.
So statistically when asking others or yourself trying to make a thoughtful choice. As an example O-5-10 scale doesn't do this. Try 2-7- 16 makes them ask if it's not fabulous but it's not bad, maybe 7 ok but definitely not 16 and not 2. Or could be a 4 but not a seven so it's a 2
You hit the nail with the ign comparison. 6/10 is good, but people just riot it's not a 10
I think that manufacturers that care about their products (and I firmly believe that at most of them do care at some degree) on a long run will profit from critical review because of useful information that can actually take into account, not only sales data.
Smooth ad read
i bought me the shr840 because of your website and i really love them. much love
True. According to many ratings the Melomania 1+ is "the best sounding TWS" but according to me, it's dogshit and losing against my CCA C10 in every category, and that thing can be used wireless too, which STILL sounds better than the Melomania 1+..
How about a scale with no scores which has a side with professional use thoughts and the other with consumer taste . A scale vertically with two pointing arrows