We've been working hard lately at posting more investigations, documentaries, and deep-dives! Check out some other ones below! Intel CPU fabrication factory documentary in Arizona: ruclips.net/video/IUIh0fOUcrQ/видео.html AMD lab tour & documentary in Texas: ruclips.net/video/7H4eg2jOvVw/видео.html The Downfall of EK Water Blocks: ruclips.net/video/6VjYFdHMC3A/видео.html& And we're working on more! Support this content by grabbing something on the store at store.gamersnexus.net/ or by throwing a few bucks at us on Patreon at www.patreon.com/gamersnexus HUGE THANKS to our fact checkers & peer reviewers. Find them below! Aris (Hardware Busters & Cybenetics): @HardwareBusters & www.cybenetics.com/ Roman (Der8auer): @der8auer Elmor Labs: www.elmorlabs.com/ (they sell great tools for PCs)
From what I've managed to observe. Everything was fine until the eleventh generation processors. Before e-cores and pcie 5.0 appeared. Or companies don't want to pay highly qualified employees because the columns in the spreadsheet don't match in the management. And they think that everyone is going to be replaced and go to the streets.
As I predicted, when this whole 12VHPWR fiasko blew up: its a connector designed and built for the bare minimum of "scraping by" in terms of overhead. I suspect this was a pennies on the dollar decision, making the CHEAPEST possible connector to do the job. They could have taken the old 8pin PCI plug, added the extra 4 and sense wires... and STILL have a way more reliable and redundant design with marginal overhead. Considering theres been literal DECADES of connector development, know-how etc... this just seems LAUGHABLY undercooked (no pun intended). I truly hope who ever designed and signed off on this... does not work at PCI-SIG anymore.
If it was ready for daylight it would have been on the ARC cards, not nvidia's. Intel WARNED PEOPLE about this head of time as they were directly involved in creating it.
I like MSI's solution. Their PCIE-5 PSUs come with a 12VHPWR socket and a 12VHPWR cable with yellow pins. Essentially, if you can still see yellow after plugging in, it's not plugged in all the way. You have to push until you can no longer see yellow. Look up A750GL or A850GL.
For its faults, MSI has some product managers we've met who are very build conscious / ease-of-installation focused. Awesome to see those voices occasionally winning amongst all the marketing!
You know the reporting will be next level (for RUclips) when they outright disclose the outside reviewers of their content they are presenting. Kudos for that
This wasnt even reduction in temperature. More like +300c if you dicerolled wrong or plugged the socket in loose without double checking. Though I wonder what the resistance difference is between regular pcie vs hvpower
Incredible video. Absolutely groundbreaking and amazing journalistic work here. A true embodiment of what the SPJ Code of Ethics stands for. Keep on keeping on, GN. Much love from Seattle
I think I speak for the community when I say: Thank you for your support to GN ❤ Some of us have financial difficulties and are unable to properly compensate GN for the work that they do. We appreciate you.
Wow - massive donation. Thank you for that. It is sincerely appreciated and will go straight back into the next one. Also, it's been years since I was out in Seattle (PAX Prime), but I love it out there! Great area.
These videos are exactly why I'm proud to be a paying supporter/member of this channel! Thank you Steve for your incredible integrity :) And also for signing my sweet modmat - I'll be breaking that out again in a week for another PC I'm building for a friend :)
I've had so many people tell me that my own card is all fixed now that the previous discussions had settled. I'm so glad you decided to revisit this Steve because the dialogue needs to be revisited to address this horrible spec. Even the basic nomenclature is completely backwards in the white paper. 😂
If only there already was a connector able to deliver 300W in the same footprint as the PCIe 8pin, widely proven and already in mass production for all modular PSUs... Wait a minute
They could have just kept the old pitch size of the old PIC-E power plugs and use the pin layout from 12VHPWR, so 6x12V and 6x GND plus their sense pin thingy. The way I see it the smaller size is the main culprit.
GN feels like modern day Sherlock Holmes going after criminals and I'm here for it. Impressive work all around. As a phd level scientist in medicine, I appreciate your efforts for transparency and reproducibility. Its refreshing to see rigorous science being applied in the consumer tech world. Cheers!
@@joshschmidt8784 because sherlock holmes solved murder cases through deduction and didn't expose dodgy electrical engineering by asking some industry experts.
@@RegenTonnenEnte I think he was declaring his expertise in the field of discussion and I imagine paying a well deserved compliment to GN. But maybe English isn't your first language.
I can’t believe how lucky we are to have GN. I feel like this level of quality, detail, professionalism, is something I never expected. Like this is real journalism, science, and dedication FROM A RUclips CHANNEL!! Love you Steve and Team ❤
Consumer electronics are required to use lead-free solder. AFAIK the only place you can use leaded solder in mass production is healthcare, aviation and military equipment.
Yeah it's a real shame IMO, the tiny bit of lead in solder helps so much more than it hurts. If govts would be willing to properly dispose of stuff this would not be an issue & we'd have much sounder electrical connections on our electronics. Thankfully you can still get leaded solder for personal use, it'll be a real sad day when they outlaw leaded solder across the board. Lead free introduces so many issues into what should be a very easy task of soldering a connection together!
Soldering with lead free alert is not that much more difficult. We do it all day, every day in the industry. With the proper prep that is always needed for long lived solder connections it is not that different.
@@robr640 It's not down to the government to actually put it in the correct bin, or take it to the correct waste collection facility to make sure it's disposed of effectively. They can provide all the safe disposal sites they want. Joe Bloggs is still going to f**k it in the bin though. It's a consumer conscience problem. They don't care.
I've worked a lot with high amperage/low voltage electronics. I knew from the beginning that those pcb adapters were going to be a problem from the moment they were announced. We did a lot of experimentation with high current lipos years ago and we constantly worried about the wired connectors we used, and those were from reputable, professional industry/military companies that have been making high current connectors for decades.
That's really what blows my mind. Hobby RC electronics for example have proven that more appropriate consumer grade connectors are possible and even available quite cheaply as of recent years. Paralleling up small friction fit connections is rarely a better solution then one solid low resistance one. Some even contain sense pins already. And on top of that silicone wire with its flexible jacket and high strand counts would practically eliminate the bend strain caused by these atrociously stiff meshed/bundled PCI power cables. It even comes in all sorts of fun colors for your customization desires haha!
In addition to the original ampacity compared to the latest, along with less connection contact area, a mess. Any power engineer should have seen this coming. I did. Most did. All of this root cause is for those that lack any basic understanding. The root cause is obvious, not enough. Any connector that blows will cause a cascade. Any of these other things that might have been listed are just sensationalism. The how and the why all will point to the real root, ampacity and quality connections. There simply isn't enough. The mechanics and quality of the connector just make it worse. Delving into that is nothing more than entertainment. Yup, a crack at Steve who blatantly said everyone was wrong. Basics are enough in most cases. Just jumped to the end to get this comment out. Test and test, build and build, technician. Leave the design to those who do. Comment on it, but stop throwing shade. Best Buy at it's best.
This looks like a gigantic mess. I am not an engineer, But when I saw them change the PCIe 8 pin to 12vhpwr connector? Why? I didn't understand why they were trying to push more current through a smaller connector.
That's the best part! It is a gigantic mess! Saw you edited your comment: We do address why they did that (at face value) in this piece, if you're genuinely curious what their reasons were!
@@GamersNexus yes why one reason it's to reduce space the other maybe more power in one connector for biggest cards instead of 4x8 pin pcie they do 2x 12VHPWR also big mistake in testing products after fabrication
The big selling point was the 12vhpwr connector being able to “talk” to the power supply better than existing connections. I’ve been into computers north of a decade and have seen the shift away from Molex (not a bad thing) to PCIE power connectors and now this. This connector is plagued by the same “engineering” principles that have caused people to loathe getting a new car. It’s great on paper but a train wreck in practice.
@@jonathanjones7751 That’s the most ridiculous part because most cables have the sense pins wired within the male connector to use 600W - the sense pins on the GPU and the PSU actually don’t connect
@@jonathanjones7751 This power connector is cutting-edge-right up there as the new design wave! Just like Starliner and the Titan submersible! 🚀🌊 What could possibly go wrong? 😅
Excellent video. One thing that I've been wondering about ever since the 12VHPWR connector was announced is: Why the PCI-SIG didn't switch to a blade type contact connector design. Blade type connectors have been the de-factor industry standard in high current applications for decades, with nominal ratings well over 15A for a single contact and some of them going up to 100A per contact in some of the off-the-shelf designs that have been available for quite some time.
@@hexarith you mean like Mini-Fit Sr at 50A per contact 😀 Given all the other connectors are Mini-Fit Jr that would be the obvious option IMHO. Given the new plug anyway means new PSU, a 20V option would give the headroom for 1000W in the GPU on two pins and while the individual pins are larger as you only need two it will be similar sized.
Then NVidia wouldn't have owned it and been able to take the credit. They tried to "push the envelope" by being cheap, and surprise surprise, engineers "overbuild" things for a reason. "Overbuilding" is about knowing everything's not going to be perfect. It's reminiscent of Apple's butterfly keyboards.
Steve, I've been watching you ever since some of your very first videos. I cannot express my gratitude for you and your team's contribution to this industry and journalism as a whole. Keep on keeping on!
In 15+ years of PCIe power, 3.3% of users have ever had a failure. In less than one Nvidia generation 4% had failures. That's huge! I'll go check my 4090 again 👀
Yep, that's something that need to be re-iterated. Plus it should also be noted that of the PCIe 8 pin failures, most of them are due to PSU manufacturers having daisy chained cables. The connector itself isn't at fault.
Ehm... That's how percentages work all right. If statistics is correct, then difference is actually not as big... Reasons for failure rate are important though (for older connector daisy-chaining is a thing) But other than that. 3.3% in 15+ years and 3.3% in 1 year is basically same percentage. Granted, you for sure have larger sample size with more lasting measurement. And 4% is not that much larger than 3.3%... Hard to say anything on statistical significance of this difference, though.
@@DimkaTsv Not necessarily. the 3.3% includes failures shortly after the product was made and long term failures that took a long time to happen, the new one has only been around for a year so 4% fail in the short term and an unknown additional amount that fail after a longer period of use is not counted in that 4%.
I've been saying for two years now: This plug cannot handle 50 amps, and the proper way to have handled this was to change power supply standards and GPU standards to power GPUS off of 48 volts instead of 12 volts. This would reduce the current by 75% for a given number of watts, and would completely eliminate the overheating and failure of these plugs. When I first started saying this, people mocked me, asking if I thought I knew more than the engineers at Nvidia. Plainly, on this subject, I did. Based on almost 50 years in electronics, I knew immediately on seeing this connector that it would not hold up at 50 amps of current draw. And renaming the plug is not going to help. These plugs need to go away, but they're not going to. So, if you're buying a 4090 or 5090, keep a fire extinguisher handy.
Issue with 48V solution is that almost everything else in your PC is powered by 12V. Meaning PSU manufacturers will need to build 2 circuits for 12V and 48V separately... And, then, how do you even divide power budget? Remember times when PSU almost never were able to provide rated wattage on 12V line because 5V eaten significant part? You suggest similar story. Not to say it will break almost all intercompatability (even if with adapter cables) that PSU spec currently has. [And will increase risk of user error of building system in a way that will send 48V instead of 12V... But this one can be avoided by unique connectors. Returning to point of intercompatability]
People assume Nvidia Engineers are also Electricians. Well these people are downright morons to think that. Maybe Nvidia Engineers are similar to an Electrician but on a sub level. Engineer doesnt mean your an expert in multiple fields with the project your building. Most of these stupid shenanigans are the simple result of companies attempting to find a cheaper, cut - the - corners type scenario to make more on their margins. You can bet Nvidia pushed and backed this stupid connector.
Yes higher voltage transmission will be more efficient and will require thinner cables (hence newer 800V EVs), but CPU EPS is also 12V so what do you do with it? You would either have to make a new 4th rail on PSU side (requires buying a brand new more expensive PSU), or you would entirely replace 12V with 48V, which would require both new PSU and new GPU/mother boards with more expensive VRMs to handle 48V to ~1V conversion. Both solutions would generate a ton of ewaste
@@ThunderingRoarYou handle it the same way USB cables do. The base voltage is 12V for the connector, and through some handshake the PSU, cable and card can switch to a higher voltage mode. Sure it takes a bit more hardware to make it work, but making circuits that support 50A ain't cheap either. Backwards and forwards compatible. No extra e-waste. Gradual rollout is possible. And low power devices can stick with 12V where it's absolutely fine. Obviously this would require a bunch of coordination in the industry, but this is far from impossible to make work...
Your contribution to the PCMR community is simply beyond measure. You are the best. The amount of work you do and the help we receive as a result exceed all expectations. Thanks Steve and the whole team.
The craziest thing about this is that we could just use a re-branded eps cable like they use in servers. The safety margin on them is so big it could hold the same amount of power as a 12 pin and still have a larger safety margin.
Awesome research, it’s so cool that you guys worked hard and now are able to afford making this type of content! GN deserves every bit of success it has and then some more. Also love that Aris and other expertises were used. Thanks Patrick, Steve and Team! Looking forward to the next awesome content.
"THIS IS BAD" Thank you Steve EXCELLENT coverage of a potentially multi level catastrophic event that could have been exasperated if the cards were in a more affordable price range. The initial buy-in cost of these cards greatly kept them out of reach of budget volume amounts of retail sales.
@@GamersNexus speaking of coolers, I'm very intrigued by the Thermalright Burst Assassin 120 Evo Dark - it looks like an attempt to rival the NH-U12A at a much more reasonable price. Could we get a review on that?
I work as an electrician in germany. I mostly saw this kind of failure in outlets which were used to their maximum capacity at regular intervals. They used normal 230V sockets which are rated for 16A for their forklift chargers. What they didnt know was, that these plugs were actually rated for 10A continuous operation. At first nothing will happen. But everyday the socket was overloaded and got warm during operation. When it cooled down again a little bit of condensation formed inside the connector and the resistance got a little bit higher. This cycle continued for while until the socket failed completely and caught on fire. So the actual failure was not that the connector was bad, but it got overloaded everyday and the cycle between cold and warm increased the resistance until the failure accured. I could also imagine that this factor can also come into play with the 12VHPWR. But I cant understand why they would put such a high load on such a small connector. And the fact that you have to be extra careful to not stress the pins while installing these is a huge design oversight. Just look at an XT60/90 connector and how easy it is to connect those. Would be nice if they would build some kind of failsafe into the connector such as a PTC thermistor. If that would be implemented in the future you could give the user a message that the connection is bad before the connector melts and shut down the card into a low Power mode.
Apparently, cable design is too hard, advanced a technology, for a manufacturer of computer parts from the world's highest-valued company, such as NVIDIA.
"But I cant understand why they would put such a high load on such a small connector" I think that the answer to this is pretty simple: they wanted to go the Apple route with just one small, sleek & sexy connector to look "high end" engineering be damned. Thing is you can't cheat physics (something you know even better than I do) and well it's come to bite them in the ass.
This video is the perfect example of why this channel is my go to for information you guys triple check every thing before making a statement to be as accurate as humanly possible thanks for all your hard work over the years
I still can't believe they approved that retention clip design. It's one of the tiniest most pitiful clips I've ever seen, and it's for a thick, stiff bundle of high-power cables that have to run into the side of a graphics card. You can tell just from looking at it that it's destined to waste hundreds of thousands of hours in cumulative troubleshooting and RMAs.
The more I look at 12VHPWR / 12V 2×6, the more I think to myself that the connector should had been compression-based or a _really_ thick ribbon cable. With that many small attachments, shoe-horning it into a PCI-like formfactor with smaller conductors seems to be the root problem. Whereas, if it was a compression-attached solution it would require a couple of screws, but it would be _guaranteed_ to be seated well or if a giant ribbon, would afford more (and firm) contact. The execution was bad, and PCI-SIG should feel bad.
@@OGSumo *_Damn_* right. Thumbscrews would had been baller to have and yet another accessory. Combine this with special shapes or ribbing that affix onto ratcheting teeth and the connector would _never_ come undone unless a tab were pulled down, and it would had _stayed._
I have a background (degree) in computer engineering and I very much enjoyed my class specifically on optics and how to design cables.... Great topic. Now, as a design engineer your task is to understand the spec, design the product to meet the spec, and to work with certification to verify performance. If we want to take a step back I can give you 4-5 main things I would investigate or inquire further for why this happened. This is a broader term.... Computer design engineering puzzle.... And I think many people failed to do their jobs along the way. 1. Molex failed to design connectors and cables that were able to operate correctly. They had to be redesigned and my instincts tell me that they will have to be revised again. 2. ATX, PCI-E, Nvidia, and Intel all failed to verify the spec appropriately. Specs are designed to be upgraded, improved. However, you should be able to have a standard that works the vast majority of the time and the reason specs exist is to have a reliable product in manufacturing. This is because of engineering calculations needing to have a reliable tolerance. However you want to cut it, the spec(s) are not reliable and need to be revised..... Again. (Sidenote: imagine if this was ANSI and it was a screw on the space shuttle.... Netflix has great documentaries about this) 3. Manufacturers had the issue of demand from the designers of the products (Nvidia, Corsair, Asus, etc.) which led to extremely terrible quality control due to a variety of factors. The common mantra is "ship it and we'll figure it out later" because the timeline of sales matters way more to the business mindset than the certification and engineering concerns. Think about profit margin vs. replacement costs. Often you will see inferior products added onto quality designs due to availability because at that specific moment the product was already manufactured and engineering is told to update their design to allow it. Certification is told they have to accept it through simple analysis. And this is what leads to Boeing having their massive issues with the dreamliner. 4. No one considered the basic functionality of the product and NOBODY did long term testing for fatigue failures. Things were done in such a brief timeline that it was not possible to test them thoroughly, which is more clear when you step back and think about why this all happened. 5. No one really cared or understood what bend radius means in the design requirements sense. You can even see this from the actual pictures in the spec. Computer cases have been designed in such a way for how long? No one thought it was a good idea to run every single design constraint on this cable+connector on an open bench scenario where there aren't things jammed into some small enclosure. The stack up of massive connector on the card into massive connector on the cable into computer literally does not work with bend radius (cable) design guidelines. Talk to a person who works with sheet metal and ask them about bend radius and why it matters. Ask them about different types of metal and how the bend requirements change based on the material itself that you're using. Now.... Consider that the vast majority of phone cables fail because the weight of the wire itself and the relatively small size of the connector. Flip that on its head.... Now you have a massive connector trying to jam itself into a small space. It doesn't work when you need (in both situations) proper bend radius to prevent signal and connector failures. How we fix this as a whole...... A. You need a minimum clearance and bend radius guideline for CASE MANUFACTURERS. B. You need power supply manufacturers to meet a specific quality standard for clean power signals and standardization of what the cables need to do (this is from ATX spec) C. You need motherboards, video cards, and OEMs to follow the actual specs and not mess things up by trying to push limits (waves at Nvidia) D. You need to design things in such a way that they aren't designed to fail. Step back and think about 6-pin and 8-pin connectors. You have a clip and it's attached on one side to keep a power cable from disconnecting and sparking on the board or somewhere else. Now, you have 2x6 or 2x8 connectors, even 24-pin connectors which have been used for a long time to send signals and various power connections from the PSU to the board reliably for years, decades. All of those connectors were not this big and didn't require this much power. It's a massive tolerance issue and the entire thing is centered around this very cheap clip that supposedly going to magically hold down this massive duct-taped connector design to the card while being jammed into place because the computer door won't shut and the weight of everything alone is too heavy to keep it in place. You should have 2-4 clips. Not just one. Fix the spec... Fix the connector.... Fix the quality issues..... Then maaaaaybe it'll be reliable.
A possible solution might be actually hiring engineers instead of calling engineer whoever has a job in tech regardless of their background... 🤷♂ Call me crazy, but I feel like the industry is trying to cheap out on skills, materials and QA while still trying to keep that aura of "we've got the best minds" when clearly either is not true or these people do not sleep enough to take poised and well reasoned decisions...
Too bad you had to pay extra for those "AI cores" that still aren't being utilized ..... Those could have been at least $50 cheaper without the wasted AI cores or more powerful by making those million+ transistors used in something like more shader cores, TMUs and ROPs and thus closer to actually competing with a 4090
@longjohn526 cool thing it doesn't compete with the 4090 and was never supposed to. AMD didn't bother making a competitor to that because no one would buy it. The XTX competes with the 4080, and beats it for cheaper in everything but ray tracing. Which to many of us is a joke
As an engineer. When the 4090 was released... it intrigued me for work load. But refused to put one in my system cause of the potential fire hazard. The math on the pin density never mathed for 600w. It only maths to 480w. And that leaves no overhead for the voltage droop, spikes, etc... There needs to be a lawsuit!
I like that the 12vhpwr cables from seasonic just uses two six pin pcie plugs on the psu side. Which for me shows that a six pin pcie plug could easily provide 300 watts. And I’m sure they still have a safety margin.
I was just talking about the shit show around this adapter with a friend a couple of days ago! My thanks to the GN team for the show to go along with my breakfast!
I've never been a fan of this 12vHPWR mess. If I upgraded to a card using it, honestly? I'd trust it more if I hardwired it by soldering wires to the thing, rather than using 12vHPWR.
Listened to this on a drive home from work and got so confused on all the wires, definitely one i need to sit and watch. Great research and reporting as always!
this problem is something that will easily become more prevalent with future GPUs that almost certainly are going to pull over 600Watts (theoretically we could see GPUs that pull 1000Watts within a few years from now), so if this 'connector issue' isn't fixed then we will be seeing way more than just 4% of high-end GPU owners reporting this issue
@@miken3963 If you want more pixels, you'll have to make more horsepower. The 4090 barely does simracing on triple 1440p, and is totally inadequate for triple 4K. As long as pixelcount increases faster than the Chipmakers can make smaller more efficient processors, the GPU size and powerconsumption has to increase in order to make more power.
Good luck trying to keep your circuit breakers from popping with a single machine pulling +1700 watts from a single outlet. People seem to forget that whole rooms (sometimes more) are serviced by a singular 15 amp (sometimes 20 amp) breaker. A 15 amp breaker in the USA gives you 1800 watts capability. Taking your hypothetical +1000 watt GPU and giving it a +300 watt cpu when combined with the power draw of all subsystems, fans, drives etc you're looking at +1300 watts from the PSU. So a 1300 watt system would require about 1500 watts from the wall with a 90% efficient PSU. I'm not aware of a PSU that is that efficient at 1300 watts but we're just making assumptions at this point. Point is every time you crank that system up it's going to pop a breaker unless you have nothing else running on that circuit. If CPUs continue to increase in watt demands you could easily surpass the ability for a 15 amp circuit to provide sufficient power. What then?
@@ToolofSociety In germany you can hook up 16A at 230V per circuit. I can run my 1200W sim rig, my 2000W pizza oven and some powertools at the same time no problem. Though i can refrain from using my angle grinder while baking Pizza if that means I get a GPU that runs a 12288x2560 Resolution for my Racing sims.
The very first time I saw this connector I thought "wow that's really small for 600w." How this was signed off by any of the PCI SIG members, let alone Nvidia actually _use_ it is beyond me. I've been building PCs for almost 3 decades and I've seen Molex and 8 pins melt if they don't get a solid connection. For anyone to think pushing 4X the power through a connector that's even smaller is a good idea is simply challenging the laws of physics. Especially when it could easily have been made much more robust while being barely bigger, and still be notably smaller than 2 8pins, let alone 3.
The stupid thing is that Molex have the Mini-Fit Sr range of plug sockets which is rated at 50A per pin, so while bigger you only need two pins for 600W, so actually no bigger.
Well, USB C can do 240w so the size isn’t actually an issue, as other comments have pointed out it’s more the amperage. I hope I’m not mistaken lol but IMO 24 pin motherboard power connections bother me and I like small connections
@@raypav USB C 240w uses 48V at 5A max while 12VHPWR uses 12V at 50A max. Heat generated is I²R, so the increase from 5A to 50A leads to almost 100x increase in heat generation
It was complete for the units we tested. The conclusion was that it was a mix of improper insertion for the units we tested and the failures of that era plus design oversights, and this expanded as more cable designs came out later to trigger this piece (which could not have existed at the time, because many of these changes didn't exist yet).
I know, understand and am cognizant of the fact that these hour-long deep dives take an insane amount of man-hours. But my god they are excellent and I can't get enough of them. So excellent, in fact, that I'm holding out on the Intel factory tour video for a weekend evening viewing.
Couldn't they use a more server-grade-alike solution with ZIF latches or side connectores yada yada yada the thousands of low cost solutions that servers used for years now instead of stuffing 500W+ in a classic-molex-ish connector that sticks out "beautifully" at the center point of the VGA making cable management a nightmare. 😢
@nemezzyyzz yeah, probably something like "we don't really care about gamer grade hardware anymore, just but this byproduct of our datacenter hardware"
@5urg3x That seems the most appropriate solution. Even some AIBs made some cableless VGA+MoBo combos, I just don't like that they implemented it in a way where it kinda blocks ITX designs (if the whole GPU designs would become these cable-less design with the server-alike slide-in power supply connectors.) But the power slot could be moved to the second slot, since most VGAs, even the ITXs ones, are dual slot nowadays. And I'd wish for a modular connector design, not integrating the power connector on the motherboard, but mounted on the case MoBo tray instead. (man the whole ATX spec is too old, it works, but it's old, and it's not like the oddly overly integrated NUC-alikes with CPU cooling capped at ~100W that will save us lol.)
@@5urg3x power through the PCIE slot is limited, unless a complete redesign is done (which is long overdue). we can not draw enough power through the slot so we have to have additional power connectors.
If you need to pull more power from a smaller connector and need to invent a new standard, use 48V. As long as electricity was studied, the formula Power=Voltage*"Intensity of current" (also know as "current") really makes that a simple thing, and for backward compatibility they could sell a 12v to 48v boost converter.
You'd just have tons of cheap convertors burning out as they can't handle what they're rated for, cheap poorly made crap invading PC space even more. They could have just used an XT90 connector. Not too big, and can handle 90A.
I bought a new 4090 FE (12V-2x6 revision), and believe it or not, I managed to push/break 1 pin of the GPU connector by just plugging the supplied octopus cable the first time. I wanted to hear "the click" so used some force. Does not inspire confidence for sure.
Very thorough research and reporting on this one. Thanks to the GN team and the professionals that have worked on aspects of this or even just providing validation of the findings on this whole spec issue.
12VHPWR is the connector nobody asked for and nobody needs. It’s change for the sake of change and does not bring anything to the table worth the problems it also brings to the table. PCI SIG and Nvidia need to ditch this connector and go back to the 2x3 and 2x4 connectors that have worked reliably for years now.
I have been using a 4090 undervolted shortly after it's launch. It eats around 300W. I have an older Corsair 750W TX-M PSU, to which I custom made a 12VHPWR with a local electrician guy that does this as a side income. Cable is oversized and rigid AF, it's secured in the slot so much, it feels like it will never come out. :D It was a really good buy. Quite happy I didn't bother at the end with 3rd party adapters.
12:44 the survey is not normalized for time. Micro-Fit (6 and 8 pins) connectors have been used for video cards for around 2 decades, whereas 12VHPWR at the time of the survey had only been out for what, 2 years? Normalized for time, that means we are looking at about 0.2% and 2%, respectively. A staggering difference in failure rate.
Fixing what was never broken was why I never decided to get any 30/40 series FE nor anything past my 4070. The risk should never be above zero off the bat nor, well, ever.
@@UnimportantAcc The 4070 Super still only draws less than 250W, so it's nowhere near the limit of the connector and has a much higher safety margin compared to the 4090. Should be fine.
I think any cables coming out make cards ugly. I wish we had dropped these crappy connectors years ago and just put an XT90 connector on the end or bottom of the card somewhere so it could be tucked away.
we do that constantly tho. and at similar scales. USB C can support between 25 and 250 watts depending on spec yet the connector remains the same as does the crosssection of the wires used (with some variation) The issue is not "smaller can not support more power" the issue is "Smaller might require more skill to properly plug in/operate" the 12VHPWR connector is just not designed to be used by the average human. Its "safety" the seating of the clamp is very easy to miss, when compared to the old connector which was both audible as well as haptic due to more material being in the clamp. The old 6 and 8 pin was also bigger with less pins making each pin easier to see and therefore gaps more obvious. The 12VHPWR connector was developed in a lap for lap use and never ever saw the eyes of someone who knows how people are. Its like car engeniers and mechanics. Some are hated by the mechanic, others are mechanics as well as engeniers.
I bought my gigabyte 4090 weeks before this whole high power cable nonsense became public. I am furious that I had to take multiple stops including buying a vertical mount, now buying a $70 plug adapter just so I can feel safe that my $2,000 GPU isn't going to light on fire
Somebody rally the Nvidia fans to come tell everybody that it's all "user error" and let us know they they never had a problem with their 12VHPWR cable so it's no big deal.
They will just read off the next line of Narcissist's prayer "That didn't happen. And if it did, it wasn't that bad. And if it was, that's not a big deal. And if it is, that's not my fault. And if it was, I didn't mean it. And if I did, you deserved it." Just replace "my" with Nvidia
Oh FCK yeah dude I'm sitting on a 12th gen Intel build ATM with a CORSAIR RM850 CP-9020235-NA 850W PSU. I've been seeing yours and others' coverage of 12VHPWR and honestly has kept me from upgrading my RTX 3080 in part because the entire circus around cramming more watts on less copper seems just awful. Thanks for making this video - I still have to figure out what the hell to do moving forward, but hopefully all this sorts out when I go to upgrade!
@@IntegerOfDoom The spec doesn't cover the materials used on the connector if I understood this excellent coverage correctly. The use of substandard materials or manufacturing is a risk for any power standard.
While not the root cause, I'm amazed how many people ignore the spikes of melting connectors in the summer and winter. AC Units and Heater blocks running on the same loop to your electrical box can also cause an under-volt on the GPU and the GPU try's to draw even more power. (mainly if you set your GPU over 110% power on something like MSI Afterburner. Also explains why we get a low on returns and then in Summer and Winter, 12VHPWR issues all of a sudden come back.
I remote trouble shot one of these not being plugged in(friend thought her new gfx card was broken) almost lots a friend as she swore the plug was in.. I have a pic of like .5mm gap and the card wouldn't even power up, she said it took all her might to seat it fully after I convinced her that it wasn't seated.. these plugs are super flawed
As a PC enthusiast, I have been against these new connectors since the start. Our 8 pins and 6 pins were just fine and draping your beautiful sleeved cables across your case interior is one of my favorite steps of a build. The cards are already huge, stop saving PCB space and just bring our OG cables back. Three and four PCIE cables look awesome.
Tension on the cable is a big problem. Bending the cable/wires close to the connector can be or not be an issue depending on various factors. Both of my builds, a 4090 and a 4080 Super have bends that are pretty close to the connector. This is needed to clear the side panel. The reason I don't have an issue, is because I was very careful with how I formed the bend, and I do not have any stress on the cable and connector. I ensured that the cable was not pulled tight, and it sits naturally in the position I need it to be. But I have a ton of experience in high reliability, safety critical, wiring and cable routing. So these kinds of things are just second nature to me when routing and dressing the cables in my builds. Most people do not have this experience and often do not think through routing and variables as diligently.
cards with two 8-pin connectors can pull a total of 375w. that should be enough for ANY card and is already not easy to cool well without a high noise level
They need to move away from using multi pin connections to delivery high amperage. Instead they could use a more simpler and flexible 2 pin connection like the ones I use that can easily deliver 600w+ of power. Better yet move from 12v to a higher voltage like 24v or even 48v and been able to deliver more power at less amps, it's the amperage that can cause the melting in connections.
Steve, an interesting part you did not pay attention to is that the problem is not only in the 16-pin Connector but also in the board's design. Nvidia changed the power design at the transition of 3090ti to 4090. As far as I know, 3090ti had 3 separate power lines and with problems with one of them, the card turned off. 4090 has all the lines work as one, so the card continues to work even when some of the power pins in the Connector are not connected as they should or are overloaded by power because others are not connected or have poor contact. Nvidia can solve the problem, at least partially, by adding protection against overload to each of the 6 power lines of the connector or at least 3 lines for 2 pins each. When at least one of the contacts is not connected or transmits excess power, the video card should turn off. Please like it so that Steve can see this.
I just bought the C8 last week and absolutely love it. A tip though, the rear fan spot does not fit Lian li 140 infinity due to them having slightly more bulk than some other fans and the spot not having much additional space for bulkier fans. I wish Antec had added just a millimeter or 2 more space width wise to accommodate different companies fans or to account for different tolerances between fans. Now back to you Steve.
We've been working hard lately at posting more investigations, documentaries, and deep-dives! Check out some other ones below!
Intel CPU fabrication factory documentary in Arizona: ruclips.net/video/IUIh0fOUcrQ/видео.html
AMD lab tour & documentary in Texas: ruclips.net/video/7H4eg2jOvVw/видео.html
The Downfall of EK Water Blocks: ruclips.net/video/6VjYFdHMC3A/видео.html&
And we're working on more! Support this content by grabbing something on the store at store.gamersnexus.net/ or by throwing a few bucks at us on Patreon at www.patreon.com/gamersnexus
HUGE THANKS to our fact checkers & peer reviewers. Find them below!
Aris (Hardware Busters & Cybenetics): @HardwareBusters & www.cybenetics.com/
Roman (Der8auer): @der8auer
Elmor Labs: www.elmorlabs.com/ (they sell great tools for PCs)
PROUDLY SPONSORED BY VALVE
From what I've managed to observe. Everything was fine until the eleventh generation processors. Before e-cores and pcie 5.0 appeared. Or companies don't want to pay highly qualified employees because the columns in the spreadsheet don't match in the management. And they think that everyone is going to be replaced and go to the streets.
This is absolutely incredible work. Fantastic insight. Extensively detailed investigation.
I have an idea. Just freaking solder the wires to it. There, fixed!
Kinda hilarious title from the people who famously just said that if you have issues with 12VHPWR it is user error.
As I predicted, when this whole 12VHPWR fiasko blew up: its a connector designed and built for the bare minimum of "scraping by" in terms of overhead. I suspect this was a pennies on the dollar decision, making the CHEAPEST possible connector to do the job. They could have taken the old 8pin PCI plug, added the extra 4 and sense wires... and STILL have a way more reliable and redundant design with marginal overhead. Considering theres been literal DECADES of connector development, know-how etc... this just seems LAUGHABLY undercooked (no pun intended). I truly hope who ever designed and signed off on this... does not work at PCI-SIG anymore.
Or switched to mini fit senior connector range which is rated at 50A per pin so you only need 2 pins for 600W
If it was ready for daylight it would have been on the ARC cards, not nvidia's. Intel WARNED PEOPLE about this head of time as they were directly involved in creating it.
I blame DEI heir engineers and managers.
He does. It's John. John sucks.
@@markburton5292 That is one of the stupidest things I have ever seen
I like MSI's solution. Their PCIE-5 PSUs come with a 12VHPWR socket and a 12VHPWR cable with yellow pins. Essentially, if you can still see yellow after plugging in, it's not plugged in all the way. You have to push until you can no longer see yellow. Look up A750GL or A850GL.
I just built a PC for someone with one of these and I was impressed.
Actual 5.0 compliant PSU released along the timeline of the 4090. This was the way 👍
For its faults, MSI has some product managers we've met who are very build conscious / ease-of-installation focused. Awesome to see those voices occasionally winning amongst all the marketing!
I like how Asus does it with their btf connection..... An extra part of PCB for the power.
You know the reporting will be next level (for RUclips) when they outright disclose the outside reviewers of their content they are presenting. Kudos for that
"Did you achieve a 5 degree Celsius reduction in temperature?"
"Yes."
"What did it cost you?"
"Everything."
This wasnt even reduction in temperature. More like +300c if you dicerolled wrong or plugged the socket in loose without double checking.
Though I wonder what the resistance difference is between regular pcie vs hvpower
Incredible video. Absolutely groundbreaking and amazing journalistic work here. A true embodiment of what the SPJ Code of Ethics stands for.
Keep on keeping on, GN.
Much love from Seattle
Coke much? 😅
I think I speak for the community when I say:
Thank you for your support to GN ❤
Some of us have financial difficulties and are unable to properly compensate GN for the work that they do. We appreciate you.
Wow - massive donation. Thank you for that. It is sincerely appreciated and will go straight back into the next one. Also, it's been years since I was out in Seattle (PAX Prime), but I love it out there! Great area.
Can you send me $100.00 I need to buy food.
These videos are exactly why I'm proud to be a paying supporter/member of this channel!
Thank you Steve for your incredible integrity :)
And also for signing my sweet modmat - I'll be breaking that out again in a week for another PC I'm building for a friend :)
Thank you for the support! And glad that the modmat is getting use!
@@GamersNexus
My brain unit gets use when I watch your content
I've had so many people tell me that my own card is all fixed now that the previous discussions had settled. I'm so glad you decided to revisit this Steve because the dialogue needs to be revisited to address this horrible spec. Even the basic nomenclature is completely backwards in the white paper. 😂
If only there already was a connector able to deliver 300W in the same footprint as the PCIe 8pin, widely proven and already in mass production for all modular PSUs...
Wait a minute
The one for the Outlet in the wall.
They could have just kept the old pitch size of the old PIC-E power plugs and use the pin layout from 12VHPWR, so 6x12V and 6x GND plus their sense pin thingy. The way I see it the smaller size is the main culprit.
Nvidia DC GPUs even already use EPS for power which I think is what you're getting at!
But then how are the poor power supply companies supposed to make money?
Did Nvidia ever explain why they made this cable standard. I dont get it.
GN feels like modern day Sherlock Holmes going after criminals and I'm here for it. Impressive work all around. As a phd level scientist in medicine, I appreciate your efforts for transparency and reproducibility. Its refreshing to see rigorous science being applied in the consumer tech world. Cheers!
What a clumsy boast and misrepresentation of sherlock holmes.
@@RegenTonnenEnteI would love to hear your breakdown of your comment.
@@joshschmidt8784 because sherlock holmes solved murder cases through deduction and didn't expose dodgy electrical engineering by asking some industry experts.
@@joshschmidt8784 got deleted by YT... well basically he just isn't and I fail to see *any* parallel.
@@RegenTonnenEnte I think he was declaring his expertise in the field of discussion and I imagine paying a well deserved compliment to GN. But maybe English isn't your first language.
I can’t believe how lucky we are to have GN. I feel like this level of quality, detail, professionalism, is something I never expected. Like this is real journalism, science, and dedication FROM A RUclips CHANNEL!!
Love you Steve and Team ❤
Very kind words. Thank you. We will keep trying to live up to them!
Consumer electronics are required to use lead-free solder. AFAIK the only place you can use leaded solder in mass production is healthcare, aviation and military equipment.
The amount of e-waste made by lead free solder because of misguided policy is staggering.
Yeah it's a real shame IMO, the tiny bit of lead in solder helps so much more than it hurts. If govts would be willing to properly dispose of stuff this would not be an issue & we'd have much sounder electrical connections on our electronics. Thankfully you can still get leaded solder for personal use, it'll be a real sad day when they outlaw leaded solder across the board. Lead free introduces so many issues into what should be a very easy task of soldering a connection together!
Soldering with lead free alert is not that much more difficult. We do it all day, every day in the industry. With the proper prep that is always needed for long lived solder connections it is not that different.
@@robr640 It's not down to the government to actually put it in the correct bin, or take it to the correct waste collection facility to make sure it's disposed of effectively. They can provide all the safe disposal sites they want. Joe Bloggs is still going to f**k it in the bin though. It's a consumer conscience problem. They don't care.
@@robr640I'm glad they don't mass produce products with brain damage included
Grabbing my popcorn now! Thanks Steve.
Same 🍿
I'll grab the 12VHPWR to pop the popcorn.
Maybe we should just put all the longer videos into a playlist called "popcorn"
@@GamersNexus I'll watch the whole playlist!
@@GamersNexus please!
I've worked a lot with high amperage/low voltage electronics. I knew from the beginning that those pcb adapters were going to be a problem from the moment they were announced. We did a lot of experimentation with high current lipos years ago and we constantly worried about the wired connectors we used, and those were from reputable, professional industry/military companies that have been making high current connectors for decades.
That's really what blows my mind. Hobby RC electronics for example have proven that more appropriate consumer grade connectors are possible and even available quite cheaply as of recent years. Paralleling up small friction fit connections is rarely a better solution then one solid low resistance one. Some even contain sense pins already. And on top of that silicone wire with its flexible jacket and high strand counts would practically eliminate the bend strain caused by these atrociously stiff meshed/bundled PCI power cables. It even comes in all sorts of fun colors for your customization desires haha!
In addition to the original ampacity compared to the latest, along with less connection contact area, a mess. Any power engineer should have seen this coming. I did. Most did. All of this root cause is for those that lack any basic understanding. The root cause is obvious, not enough. Any connector that blows will cause a cascade. Any of these other things that might have been listed are just sensationalism. The how and the why all will point to the real root, ampacity and quality connections. There simply isn't enough. The mechanics and quality of the connector just make it worse. Delving into that is nothing more than entertainment. Yup, a crack at Steve who blatantly said everyone was wrong. Basics are enough in most cases. Just jumped to the end to get this comment out. Test and test, build and build, technician. Leave the design to those who do. Comment on it, but stop throwing shade. Best Buy at it's best.
This looks like a gigantic mess. I am not an engineer, But when I saw them change the PCIe 8 pin to 12vhpwr connector? Why? I didn't understand why they were trying to push more current through a smaller connector.
That's the best part! It is a gigantic mess! Saw you edited your comment: We do address why they did that (at face value) in this piece, if you're genuinely curious what their reasons were!
@@GamersNexus yes why
one reason it's to reduce space
the other maybe more power in one connector for biggest cards
instead of 4x8 pin pcie they do 2x 12VHPWR
also big mistake in testing products after fabrication
The big selling point was the 12vhpwr connector being able to “talk” to the power supply better than existing connections. I’ve been into computers north of a decade and have seen the shift away from Molex (not a bad thing) to PCIE power connectors and now this. This connector is plagued by the same “engineering” principles that have caused people to loathe getting a new car. It’s great on paper but a train wreck in practice.
@@jonathanjones7751 That’s the most ridiculous part because most cables have the sense pins wired within the male connector to use 600W - the sense pins on the GPU and the PSU actually don’t connect
@@jonathanjones7751 This power connector is cutting-edge-right up there as the new design wave! Just like Starliner and the Titan submersible! 🚀🌊
What could possibly go wrong? 😅
When the design spec calls for tight tolerances, but corner cutting is your business model:
Excellent video.
One thing that I've been wondering about ever since the 12VHPWR connector was announced is: Why the PCI-SIG didn't switch to a blade type contact connector design. Blade type connectors have been the de-factor industry standard in high current applications for decades, with nominal ratings well over 15A for a single contact and some of them going up to 100A per contact in some of the off-the-shelf designs that have been available for quite some time.
@@hexarith you mean like Mini-Fit Sr at 50A per contact 😀 Given all the other connectors are Mini-Fit Jr that would be the obvious option IMHO. Given the new plug anyway means new PSU, a 20V option would give the headroom for 1000W in the GPU on two pins and while the individual pins are larger as you only need two it will be similar sized.
yeah that is the weirdest part to me
Then NVidia wouldn't have owned it and been able to take the credit. They tried to "push the envelope" by being cheap, and surprise surprise, engineers "overbuild" things for a reason. "Overbuilding" is about knowing everything's not going to be perfect. It's reminiscent of Apple's butterfly keyboards.
Ring terminal and a screw
Steve, I've been watching you ever since some of your very first videos. I cannot express my gratitude for you and your team's contribution to this industry and journalism as a whole. Keep on keeping on!
Thank you for sticking around for so long! It's been a long journey. Still love it!
In 15+ years of PCIe power, 3.3% of users have ever had a failure. In less than one Nvidia generation 4% had failures. That's huge! I'll go check my 4090 again 👀
Yep, that's something that need to be re-iterated. Plus it should also be noted that of the PCIe 8 pin failures, most of them are due to PSU manufacturers having daisy chained cables. The connector itself isn't at fault.
Ehm... That's how percentages work all right.
If statistics is correct, then difference is actually not as big... Reasons for failure rate are important though (for older connector daisy-chaining is a thing)
But other than that. 3.3% in 15+ years and 3.3% in 1 year is basically same percentage. Granted, you for sure have larger sample size with more lasting measurement.
And 4% is not that much larger than 3.3%... Hard to say anything on statistical significance of this difference, though.
0.22% vs 4%, that's ~18.2x the failure rate (3.3/15=0.22)
@@DimkaTsv Not necessarily. the 3.3% includes failures shortly after the product was made and long term failures that took a long time to happen, the new one has only been around for a year so 4% fail in the short term and an unknown additional amount that fail after a longer period of use is not counted in that 4%.
@@coopercummings8370 how YEAR since this shit plug is with us from 2020 ??
I've been saying for two years now: This plug cannot handle 50 amps, and the proper way to have handled this was to change power supply standards and GPU standards to power GPUS off of 48 volts instead of 12 volts. This would reduce the current by 75% for a given number of watts, and would completely eliminate the overheating and failure of these plugs.
When I first started saying this, people mocked me, asking if I thought I knew more than the engineers at Nvidia. Plainly, on this subject, I did. Based on almost 50 years in electronics, I knew immediately on seeing this connector that it would not hold up at 50 amps of current draw. And renaming the plug is not going to help.
These plugs need to go away, but they're not going to. So, if you're buying a 4090 or 5090, keep a fire extinguisher handy.
Issue with 48V solution is that almost everything else in your PC is powered by 12V.
Meaning PSU manufacturers will need to build 2 circuits for 12V and 48V separately... And, then, how do you even divide power budget? Remember times when PSU almost never were able to provide rated wattage on 12V line because 5V eaten significant part? You suggest similar story.
Not to say it will break almost all intercompatability (even if with adapter cables) that PSU spec currently has. [And will increase risk of user error of building system in a way that will send 48V instead of 12V... But this one can be avoided by unique connectors. Returning to point of intercompatability]
People assume Nvidia Engineers are also Electricians. Well these people are downright morons to think that. Maybe Nvidia Engineers are similar to an Electrician but on a sub level. Engineer doesnt mean your an expert in multiple fields with the project your building. Most of these stupid shenanigans are the simple result of companies attempting to find a cheaper, cut - the - corners type scenario to make more on their margins. You can bet Nvidia pushed and backed this stupid connector.
Yes higher voltage transmission will be more efficient and will require thinner cables (hence newer 800V EVs), but CPU EPS is also 12V so what do you do with it?
You would either have to make a new 4th rail on PSU side (requires buying a brand new more expensive PSU), or you would entirely replace 12V with 48V, which would require both new PSU and new GPU/mother boards with more expensive VRMs to handle 48V to ~1V conversion. Both solutions would generate a ton of ewaste
Mini-Fit Sr enters the room at 50A per pin 😂
@@ThunderingRoarYou handle it the same way USB cables do. The base voltage is 12V for the connector, and through some handshake the PSU, cable and card can switch to a higher voltage mode. Sure it takes a bit more hardware to make it work, but making circuits that support 50A ain't cheap either.
Backwards and forwards compatible. No extra e-waste. Gradual rollout is possible. And low power devices can stick with 12V where it's absolutely fine.
Obviously this would require a bunch of coordination in the industry, but this is far from impossible to make work...
Your contribution to the PCMR community is simply beyond measure.
You are the best.
The amount of work you do and the help we receive as a result exceed all expectations.
Thanks Steve and the whole team.
You guys are a beacon of trust and knowledge in an environment ridden with shady and negligent companies.
Thanks heaps for this Steve & GN team. I've been really curious and worried about this overall issue. You guys are doing fucking brilliant work.
Thanks so much for the kind words! We're really trying to focus on gradual improvement of the formats!
The craziest thing about this is that we could just use a re-branded eps cable like they use in servers. The safety margin on them is so big it could hold the same amount of power as a 12 pin and still have a larger safety margin.
Bingo!
my RTX A6000 actually has an 8 pin EPS socket on it! no reason why they couldn't do that on consumer GPUs too
Awesome research, it’s so cool that you guys worked hard and now are able to afford making this type of content! GN deserves every bit of success it has and then some more. Also love that Aris and other expertises were used. Thanks Patrick, Steve and Team! Looking forward to the next awesome content.
"THIS IS BAD"
Thank you Steve
EXCELLENT coverage of a potentially multi level catastrophic event that could have been exasperated if the cards were in a more affordable price range. The initial buy-in cost of these cards greatly kept them out of reach of budget volume amounts of retail sales.
I've spent some time designing connectors in the past and my first thought when I came across this connector was, "What in the actual fuck is this?"
Feels like everything sucks now.
Not everything! Cases and coolers have been awesome lately!
@@GamersNexus If only there was good new stuff to put in those cases and cool with the coolers
@@LivingParable693 It's simple - just buy any Sapphire, Powercolor or XFX GPU. It's safe and it works.
@@GamersNexusso basically the only thing you "don't need" to run your PC doesn't sucks...
@@GamersNexus speaking of coolers, I'm very intrigued by the Thermalright Burst Assassin 120 Evo Dark - it looks like an attempt to rival the NH-U12A at a much more reasonable price. Could we get a review on that?
Or how to make NVIDIA angry right before RTX 5000 launch :D
dont worry - they can wipe their tears with billions of dollars
@@traiges414 more like trillions 😭
5090: Hold my beer 4090!
@@Tra-vis nvidia really needs the ubisoft treatment
angr? they got lucky houses did not ended as ashes and people died
I work as an electrician in germany. I mostly saw this kind of failure in outlets which were used to their maximum capacity at regular intervals. They used normal 230V sockets which are rated for 16A for their forklift chargers. What they didnt know was, that these plugs were actually rated for 10A continuous operation. At first nothing will happen. But everyday the socket was overloaded and got warm during operation. When it cooled down again a little bit of condensation formed inside the connector and the resistance got a little bit higher. This cycle continued for while until the socket failed completely and caught on fire.
So the actual failure was not that the connector was bad, but it got overloaded everyday and the cycle between cold and warm increased the resistance until the failure accured.
I could also imagine that this factor can also come into play with the 12VHPWR.
But I cant understand why they would put such a high load on such a small connector. And the fact that you have to be extra careful to not stress the pins while installing these is a huge design oversight. Just look at an XT60/90 connector and how easy it is to connect those.
Would be nice if they would build some kind of failsafe into the connector such as a PTC thermistor. If that would be implemented in the future you could give the user a message that the connection is bad before the connector melts and shut down the card into a low Power mode.
Apparently, cable design is too hard, advanced a technology, for a manufacturer of computer parts from the world's highest-valued company, such as NVIDIA.
Main question is do we need that much high power on low quality crap micro component filled pcbs designed by
greedy idiots ?
@@Miparwo they only made a few trillion on ai bullsh1t... They don't have the spare cash to engineer a proper cable.
"But I cant understand why they would put such a high load on such a small connector"
I think that the answer to this is pretty simple: they wanted to go the Apple route with just one small, sleek & sexy connector to look "high end" engineering be damned. Thing is you can't cheat physics (something you know even better than I do) and well it's come to bite them in the ass.
@@primodragoneitaliano This is also my guess. It's all about the aesthetic.
This video is the perfect example of why this channel is my go to for information you guys triple check every thing before making a statement to be as accurate as humanly possible thanks for all your hard work over the years
I love how one timestamp is "CHAOS" 🤣
I still can't believe they approved that retention clip design. It's one of the tiniest most pitiful clips I've ever seen, and it's for a thick, stiff bundle of high-power cables that have to run into the side of a graphics card. You can tell just from looking at it that it's destined to waste hundreds of thousands of hours in cumulative troubleshooting and RMAs.
The more I look at 12VHPWR / 12V 2×6, the more I think to myself that the connector should had been compression-based or a _really_ thick ribbon cable. With that many small attachments, shoe-horning it into a PCI-like formfactor with smaller conductors seems to be the root problem. Whereas, if it was a compression-attached solution it would require a couple of screws, but it would be _guaranteed_ to be seated well or if a giant ribbon, would afford more (and firm) contact.
The execution was bad, and PCI-SIG should feel bad.
Bring back the thumbscrews! VGA was a pain to swap in and out, but when it was in it was *locked in*
One or two XT90 connectors would deliver 480/960W with a wide margin. Proven reliable.
@@OGSumoand it's not like you're plugging in and out your gpu power cable every day
@@OGSumo *_Damn_* right. Thumbscrews would had been baller to have and yet another accessory. Combine this with special shapes or ribbing that affix onto ratcheting teeth and the connector would _never_ come undone unless a tab were pulled down, and it would had _stayed._
Oh boy here we go, another exhilarating 60 minutes of a deep dive that I don't truly understand but I watch because it's GN. Thanks Steve.
I have a background (degree) in computer engineering and I very much enjoyed my class specifically on optics and how to design cables.... Great topic.
Now, as a design engineer your task is to understand the spec, design the product to meet the spec, and to work with certification to verify performance.
If we want to take a step back I can give you 4-5 main things I would investigate or inquire further for why this happened. This is a broader term.... Computer design engineering puzzle.... And I think many people failed to do their jobs along the way.
1. Molex failed to design connectors and cables that were able to operate correctly. They had to be redesigned and my instincts tell me that they will have to be revised again.
2. ATX, PCI-E, Nvidia, and Intel all failed to verify the spec appropriately. Specs are designed to be upgraded, improved. However, you should be able to have a standard that works the vast majority of the time and the reason specs exist is to have a reliable product in manufacturing. This is because of engineering calculations needing to have a reliable tolerance. However you want to cut it, the spec(s) are not reliable and need to be revised..... Again.
(Sidenote: imagine if this was ANSI and it was a screw on the space shuttle.... Netflix has great documentaries about this)
3. Manufacturers had the issue of demand from the designers of the products (Nvidia, Corsair, Asus, etc.) which led to extremely terrible quality control due to a variety of factors. The common mantra is "ship it and we'll figure it out later" because the timeline of sales matters way more to the business mindset than the certification and engineering concerns. Think about profit margin vs. replacement costs. Often you will see inferior products added onto quality designs due to availability because at that specific moment the product was already manufactured and engineering is told to update their design to allow it. Certification is told they have to accept it through simple analysis. And this is what leads to Boeing having their massive issues with the dreamliner.
4. No one considered the basic functionality of the product and NOBODY did long term testing for fatigue failures. Things were done in such a brief timeline that it was not possible to test them thoroughly, which is more clear when you step back and think about why this all happened.
5. No one really cared or understood what bend radius means in the design requirements sense. You can even see this from the actual pictures in the spec.
Computer cases have been designed in such a way for how long? No one thought it was a good idea to run every single design constraint on this cable+connector on an open bench scenario where there aren't things jammed into some small enclosure. The stack up of massive connector on the card into massive connector on the cable into computer literally does not work with bend radius (cable) design guidelines. Talk to a person who works with sheet metal and ask them about bend radius and why it matters. Ask them about different types of metal and how the bend requirements change based on the material itself that you're using. Now.... Consider that the vast majority of phone cables fail because the weight of the wire itself and the relatively small size of the connector. Flip that on its head.... Now you have a massive connector trying to jam itself into a small space. It doesn't work when you need (in both situations) proper bend radius to prevent signal and connector failures.
How we fix this as a whole......
A. You need a minimum clearance and bend radius guideline for CASE MANUFACTURERS.
B. You need power supply manufacturers to meet a specific quality standard for clean power signals and standardization of what the cables need to do (this is from ATX spec)
C. You need motherboards, video cards, and OEMs to follow the actual specs and not mess things up by trying to push limits (waves at Nvidia)
D. You need to design things in such a way that they aren't designed to fail.
Step back and think about 6-pin and 8-pin connectors. You have a clip and it's attached on one side to keep a power cable from disconnecting and sparking on the board or somewhere else.
Now, you have 2x6 or 2x8 connectors, even 24-pin connectors which have been used for a long time to send signals and various power connections from the PSU to the board reliably for years, decades. All of those connectors were not this big and didn't require this much power. It's a massive tolerance issue and the entire thing is centered around this very cheap clip that supposedly going to magically hold down this massive duct-taped connector design to the card while being jammed into place because the computer door won't shut and the weight of everything alone is too heavy to keep it in place.
You should have 2-4 clips. Not just one.
Fix the spec... Fix the connector.... Fix the quality issues..... Then maaaaaybe it'll be reliable.
A possible solution might be actually hiring engineers instead of calling engineer whoever has a job in tech regardless of their background... 🤷♂
Call me crazy, but I feel like the industry is trying to cheap out on skills, materials and QA while still trying to keep that aura of "we've got the best minds" when clearly either is not true or these people do not sleep enough to take poised and well reasoned decisions...
GN videos should be added to IMBD. The production quality is always top notch.
the sole reason i went with a 7900XTX for this gen
Same here, that and the extra vram is nice.
same, i don't use Rt anyway. no reason to pay $1700 -$2200 for a headache.
Too bad you had to pay extra for those "AI cores" that still aren't being utilized ..... Those could have been at least $50 cheaper without the wasted AI cores or more powerful by making those million+ transistors used in something like more shader cores, TMUs and ROPs and thus closer to actually competing with a 4090
Yes me too, i'm using 7900XXX
@longjohn526 cool thing it doesn't compete with the 4090 and was never supposed to. AMD didn't bother making a competitor to that because no one would buy it. The XTX competes with the 4080, and beats it for cheaper in everything but ray tracing. Which to many of us is a joke
15:48 Shaky table makes 4090 nervous
As an engineer. When the 4090 was released... it intrigued me for work load. But refused to put one in my system cause of the potential fire hazard. The math on the pin density never mathed for 600w. It only maths to 480w. And that leaves no overhead for the voltage droop, spikes, etc... There needs to be a lawsuit!
I was saying what de8ueurs was saying from day one. It made zero sense.
^ This. Spot on!
I like that the 12vhpwr cables from seasonic just uses two six pin pcie plugs on the psu side. Which for me shows that a six pin pcie plug could easily provide 300 watts. And I’m sure they still have a safety margin.
This is investigative journalism at its finest: FAIR. TRANSPARENT & ACCURATE. Because NO ONE else wants to do the dirty work...ain't that the truth!
I was just talking about the shit show around this adapter with a friend a couple of days ago! My thanks to the GN team for the show to go along with my breakfast!
I've never been a fan of this 12vHPWR mess. If I upgraded to a card using it, honestly? I'd trust it more if I hardwired it by soldering wires to the thing, rather than using 12vHPWR.
Listened to this on a drive home from work and got so confused on all the wires, definitely one i need to sit and watch. Great research and reporting as always!
It's amazing how 8 pin cables have existed for well over a decade and had zero issues that justified these terrible connectors.
this problem is something that will easily become more prevalent with future GPUs that almost certainly are going to pull over 600Watts (theoretically we could see GPUs that pull 1000Watts within a few years from now), so if this 'connector issue' isn't fixed then we will be seeing way more than just 4% of high-end GPU owners reporting this issue
I believe at this point it's more appropriate to call it "space heater with a graphic output" rather than "GPU"...
@@miken3963 If you want more pixels, you'll have to make more horsepower. The 4090 barely does simracing on triple 1440p, and is totally inadequate for triple 4K. As long as pixelcount increases faster than the Chipmakers can make smaller more efficient processors, the GPU size and powerconsumption has to increase in order to make more power.
Good luck trying to keep your circuit breakers from popping with a single machine pulling +1700 watts from a single outlet.
People seem to forget that whole rooms (sometimes more) are serviced by a singular 15 amp (sometimes 20 amp) breaker. A 15 amp breaker in the USA gives you 1800 watts capability.
Taking your hypothetical +1000 watt GPU and giving it a +300 watt cpu when combined with the power draw of all subsystems, fans, drives etc you're looking at +1300 watts from the PSU.
So a 1300 watt system would require about 1500 watts from the wall with a 90% efficient PSU. I'm not aware of a PSU that is that efficient at 1300 watts but we're just making assumptions at this point. Point is every time you crank that system up it's going to pop a breaker unless you have nothing else running on that circuit.
If CPUs continue to increase in watt demands you could easily surpass the ability for a 15 amp circuit to provide sufficient power. What then?
@@ToolofSociety In germany you can hook up 16A at 230V per circuit. I can run my 1200W sim rig, my 2000W pizza oven and some powertools at the same time no problem. Though i can refrain from using my angle grinder while baking Pizza if that means I get a GPU that runs a 12288x2560 Resolution for my Racing sims.
@@ToolofSociety Be Quiet! makes a 80+ Platinum 1500W PSU up to 93% efficiency.
The very first time I saw this connector I thought "wow that's really small for 600w."
How this was signed off by any of the PCI SIG members, let alone Nvidia actually _use_ it is beyond me. I've been building PCs for almost 3 decades and I've seen Molex and 8 pins melt if they don't get a solid connection. For anyone to think pushing 4X the power through a connector that's even smaller is a good idea is simply challenging the laws of physics. Especially when it could easily have been made much more robust while being barely bigger, and still be notably smaller than 2 8pins, let alone 3.
The stupid thing is that Molex have the Mini-Fit Sr range of plug sockets which is rated at 50A per pin, so while bigger you only need two pins for 600W, so actually no bigger.
"How this was signed off by any of the PCI SIG members, let alone Nvidia actually use it is beyond me."
One word : greed.
Well, USB C can do 240w so the size isn’t actually an issue, as other comments have pointed out it’s more the amperage. I hope I’m not mistaken lol but IMO 24 pin motherboard power connections bother me and I like small connections
@@raypav USB C 240w uses 48V at 5A max while 12VHPWR uses 12V at 50A max.
Heat generated is I²R, so the increase from 5A to 50A leads to almost 100x increase in heat generation
Glad to see you finally follow up on this. I always felt that the initial conclusion of "user error" was lacking and incomplete.
It was complete for the units we tested. The conclusion was that it was a mix of improper insertion for the units we tested and the failures of that era plus design oversights, and this expanded as more cable designs came out later to trigger this piece (which could not have existed at the time, because many of these changes didn't exist yet).
I love these deep dive videos. I work in manufacturing so it's refreshing to know that other companies screw up just as often as the one I work for.
I know, understand and am cognizant of the fact that these hour-long deep dives take an insane amount of man-hours. But my god they are excellent and I can't get enough of them. So excellent, in fact, that I'm holding out on the Intel factory tour video for a weekend evening viewing.
Couldn't they use a more server-grade-alike solution with ZIF latches or side connectores yada yada yada the thousands of low cost solutions that servers used for years now instead of stuffing 500W+ in a classic-molex-ish connector that sticks out "beautifully" at the center point of the VGA making cable management a nightmare. 😢
" no, f you, give me money" Jen-Hsun Huang probably
@nemezzyyzz yeah, probably something like "we don't really care about gamer grade hardware anymore, just but this byproduct of our datacenter hardware"
Or just deliver the power thru the slot…
@5urg3x That seems the most appropriate solution. Even some AIBs made some cableless VGA+MoBo combos, I just don't like that they implemented it in a way where it kinda blocks ITX designs (if the whole GPU designs would become these cable-less design with the server-alike slide-in power supply connectors.) But the power slot could be moved to the second slot, since most VGAs, even the ITXs ones, are dual slot nowadays. And I'd wish for a modular connector design, not integrating the power connector on the motherboard, but mounted on the case MoBo tray instead. (man the whole ATX spec is too old, it works, but it's old, and it's not like the oddly overly integrated NUC-alikes with CPU cooling capped at ~100W that will save us lol.)
@@5urg3x power through the PCIE slot is limited, unless a complete redesign is done (which is long overdue). we can not draw enough power through the slot so we have to have additional power connectors.
If you need to pull more power from a smaller connector and need to invent a new standard, use 48V. As long as electricity was studied, the formula Power=Voltage*"Intensity of current" (also know as "current") really makes that a simple thing, and for backward compatibility they could sell a 12v to 48v boost converter.
You'd just have tons of cheap convertors burning out as they can't handle what they're rated for, cheap poorly made crap invading PC space even more. They could have just used an XT90 connector. Not too big, and can handle 90A.
I bought a new 4090 FE (12V-2x6 revision), and believe it or not, I managed to push/break 1 pin of the GPU connector by just plugging the supplied octopus cable the first time. I wanted to hear "the click" so used some force. Does not inspire confidence for sure.
Very thorough research and reporting on this one. Thanks to the GN team and the professionals that have worked on aspects of this or even just providing validation of the findings on this whole spec issue.
i'm now trained to check out the scenery in these videos to locate new GN store products. You've got me trained well, thanks Steve
12VHPWR is the connector nobody asked for and nobody needs. It’s change for the sake of change and does not bring anything to the table worth the problems it also brings to the table. PCI SIG and Nvidia need to ditch this connector and go back to the 2x3 and 2x4 connectors that have worked reliably for years now.
There was a time when engineers knew how to select the appropriate connector for a particular load.
I think this is the earliest I have started watching a GN video.
same
Truly going above and beyond, this has to be one of the very best tech channels on the internet.
This is brilliant! The amount of research, fact checking and stuff. I love it! Thank you GN Team
I have been using a 4090 undervolted shortly after it's launch. It eats around 300W. I have an older Corsair 750W TX-M PSU, to which I custom made a 12VHPWR with a local electrician guy that does this as a side income. Cable is oversized and rigid AF, it's secured in the slot so much, it feels like it will never come out. :D It was a really good buy. Quite happy I didn't bother at the end with 3rd party adapters.
12:44 the survey is not normalized for time. Micro-Fit (6 and 8 pins) connectors have been used for video cards for around 2 decades, whereas 12VHPWR at the time of the survey had only been out for what, 2 years? Normalized for time, that means we are looking at about 0.2% and 2%, respectively. A staggering difference in failure rate.
Yes, we talked about this.
@@GamersNexus ah I must have missed it! Thanks for covering this issue again.
Tonight I am skipping the movies to watch this gem. Thanks Steve & GN Team!
The quality of this video is off the charts! Thank you!
Unbelievably such a high quality video
Fixing what was never broken was why I never decided to get any 30/40 series FE nor anything past my 4070. The risk should never be above zero off the bat nor, well, ever.
4070 super here... got that cursed adapter 😭
@@UnimportantAcc The 4070 Super still only draws less than 250W, so it's nowhere near the limit of the connector and has a much higher safety margin compared to the 4090. Should be fine.
@@VexingRaven good point. i still check my connector every now and then xd
Uhhh...I'll just keep getting cards with and using three 8pin GPU connectors, TYVM!
Same
TYVM has actually been replaced with HTYVM.
@@whlewis9164 What about HTYTYMVVM?
Same here, but I'm sticking with 2x8. I refuse to use a GPU over 225W TDP. I need a GPU not a space heater.
reason why I bought a 4070. Couldn't risk my house for a 4080 Super
And all the while, the EPS connector sits by idle, able to handle 300 watts, but being used only the MB.
38:15 - Definitely a place to have had a (sic) in the script.
Deep dives like this are _why_ you guys deserve our money. Keep on keeping on, GN!
GN "we brought a half dozen experts in to review our product"
The entire electronics industry *surprised Pikachu face*
Me with my 7900xtx with 3 pcie 8 pins cables, that makes my card look badass 😜
i agree i like the looks of 3pcie 8 pins aswell rather than a stupid octopus cable that splits to 4 pcie
I'm gonna hang tight on my 7900 XTX for as long as I can 😂
Which model do you have? I got the Red Devil personally. I love the thing. And I agree, the 3x8pins with custom extension cables *chef's kiss*
I think any cables coming out make cards ugly. I wish we had dropped these crappy connectors years ago and just put an XT90 connector on the end or bottom of the card somewhere so it could be tucked away.
Pushing more power through a smaller connector. Seriously, what could go wrong?
we do that constantly tho. and at similar scales.
USB C can support between 25 and 250 watts depending on spec yet the connector remains the same as does the crosssection of the wires used (with some variation)
The issue is not "smaller can not support more power" the issue is "Smaller might require more skill to properly plug in/operate"
the 12VHPWR connector is just not designed to be used by the average human. Its "safety" the seating of the clamp is very easy to miss, when compared to the old connector which was both audible as well as haptic due to more material being in the clamp. The old 6 and 8 pin was also bigger with less pins making each pin easier to see and therefore gaps more obvious.
The 12VHPWR connector was developed in a lap for lap use and never ever saw the eyes of someone who knows how people are. Its like car engeniers and mechanics. Some are hated by the mechanic, others are mechanics as well as engeniers.
I bought my gigabyte 4090 weeks before this whole high power cable nonsense became public. I am furious that I had to take multiple stops including buying a vertical mount, now buying a $70 plug adapter just so I can feel safe that my $2,000 GPU isn't going to light on fire
shoulda just got a cool leather jacket like Jensen.
Gigabyte specifically had a pcb cracking issue, but I see you already know that.
Thanks Steve & rest of the team!
Excellent coverage, I haven't gotten a chance to follow this *HOT* topic in a while.
And the scrolling timeline on the bottom?
Chefs kiss.
Somebody rally the Nvidia fans to come tell everybody that it's all "user error" and let us know they they never had a problem with their 12VHPWR cable so it's no big deal.
They will just read off the next line of Narcissist's prayer
"That didn't happen.
And if it did, it wasn't that bad.
And if it was, that's not a big deal.
And if it is, that's not my fault.
And if it was, I didn't mean it.
And if I did, you deserved it."
Just replace "my" with Nvidia
Oh FCK yeah dude I'm sitting on a 12th gen Intel build ATM with a CORSAIR RM850 CP-9020235-NA 850W PSU. I've been seeing yours and others' coverage of 12VHPWR and honestly has kept me from upgrading my RTX 3080 in part because the entire circus around cramming more watts on less copper seems just awful. Thanks for making this video - I still have to figure out what the hell to do moving forward, but hopefully all this sorts out when I go to upgrade!
*More watts on copper clad aluminimum.
@@IntegerOfDoom The spec doesn't cover the materials used on the connector if I understood this excellent coverage correctly. The use of substandard materials or manufacturing is a risk for any power standard.
Did I just hear "wet the mating surfaces"? Steve plz
Blame the standard terminology.
Well, when you have that many partial insertions, the mating surfaces may get a bit wet.
While not the root cause, I'm amazed how many people ignore the spikes of melting connectors in the summer and winter.
AC Units and Heater blocks running on the same loop to your electrical box can also cause an under-volt on the GPU and the GPU try's to draw even more power. (mainly if you set your GPU over 110% power on something like MSI Afterburner.
Also explains why we get a low on returns and then in Summer and Winter, 12VHPWR issues all of a sudden come back.
Thanks for the catch up. The next iteration is going to be fun.
Just when you thought it was over....
I remote trouble shot one of these not being plugged in(friend thought her new gfx card was broken) almost lots a friend as she swore the plug was in.. I have a pic of like .5mm gap and the card wouldn't even power up, she said it took all her might to seat it fully after I convinced her that it wasn't seated.. these plugs are super flawed
yall want my melted 4090? still has the cable mod adapter melted onto it
As a PC enthusiast, I have been against these new connectors since the start. Our 8 pins and 6 pins were just fine and draping your beautiful sleeved cables across your case interior is one of my favorite steps of a build. The cards are already huge, stop saving PCB space and just bring our OG cables back. Three and four PCIE cables look awesome.
Thank you and keep holding the industry to a high standard!
Or HEAR ME OUT! What if…we just use 8pin connectors? I have a 4080 that has 2x8pin holes ready to have connectors mounted…
Stop making sense.🤣
@@biloki3079yeah basically when designing these connectors they threw logic out the window. Great job nvidia!
@@justcynical6590 Right?? They created a solution to a problem that didn't exist, and actually created a real problem in the process. 🙄
Tension on the cable is a big problem.
Bending the cable/wires close to the connector can be or not be an issue depending on various factors.
Both of my builds, a 4090 and a 4080 Super have bends that are pretty close to the connector. This is needed to clear the side panel.
The reason I don't have an issue, is because I was very careful with how I formed the bend, and I do not have any stress on the cable and connector. I ensured that the cable was not pulled tight, and it sits naturally in the position I need it to be.
But I have a ton of experience in high reliability, safety critical, wiring and cable routing. So these kinds of things are just second nature to me when routing and dressing the cables in my builds.
Most people do not have this experience and often do not think through routing and variables as diligently.
Did ya warm the cable up first 😝
@@UnimportantAcc A little preheating to prepare for the final heat treatment? 😆
Power overwhelming!
Executor?!
@@GamersNexus For Aiur!
@@broodwarjc1517 My life for Aiur
You must construct additional pylons.
The merging is complete.
Thanks Steve! Love y'alls long form reports lately. Watched the Intel tour twice, thanks for giving me something informative to watch this monday!
Phenomenal as always. Thanks on behalf of the community.
PCI-SIG used the same process to develop 12VHPWR as USB-IF uses for all their standards, a room full of monkeys slamming their faces into keyboards.
All this to save a handful of cents in production... There was no real problem to solve…
cards with two 8-pin connectors can pull a total of 375w.
that should be enough for ANY card and is already not easy to cool well without a high noise level
I don't comment often, but Steve and team knocked it out of the park!
You guys have really stepped your game up, and it was already really really good.
They need to move away from using multi pin connections to delivery high amperage. Instead they could use a more simpler and flexible 2 pin connection like the ones I use that can easily deliver 600w+ of power. Better yet move from 12v to a higher voltage like 24v or even 48v and been able to deliver more power at less amps, it's the amperage that can cause the melting in connections.
8-Pin for the win!
get us back out trusty 8pin connector
Steve, an interesting part you did not pay attention to is that the problem is not only in the 16-pin Connector but also in the board's design. Nvidia changed the power design at the transition of 3090ti to 4090. As far as I know, 3090ti had 3 separate power lines and with problems with one of them, the card turned off. 4090 has all the lines work as one, so the card continues to work even when some of the power pins in the Connector are not connected as they should or are overloaded by power because others are not connected or have poor contact.
Nvidia can solve the problem, at least partially, by adding protection against overload to each of the 6 power lines of the connector or at least 3 lines for 2 pins each. When at least one of the contacts is not connected or transmits excess power, the video card should turn off.
Please like it so that Steve can see this.
I just bought the C8 last week and absolutely love it. A tip though, the rear fan spot does not fit Lian li 140 infinity due to them having slightly more bulk than some other fans and the spot not having much additional space for bulkier fans. I wish Antec had added just a millimeter or 2 more space width wise to accommodate different companies fans or to account for different tolerances between fans. Now back to you Steve.