Why CUDA "Cores" Aren't Actually Cores, ft. David Kanter
HTML-код
- Опубликовано: 22 авг 2024
- We talk about NVIDIA CUDA Cores vs. AMD Stream Processors and why neither is actually a "core," featuring David Kanter of Real World Tech.
Ad: Buy the Corsair Dark Core SE on Amazon (goo.gl/3psBno)
Check out Real World Tech here: www.realworldt...
Read David's article on TBR: www.realworldt...
David Kanter of Real World Tech (formerly Microprocessor Report) joins us to discuss why CUDA "Cores" aren't actually cores, later explaining the differences between CPU & GPU cores, stream processors vs. CUDA cores, and more. The discussion follows GPU architecture and explains the building blocks of a GPU, with details on streaming multiprocessors (SMs), multiply-add operations, texture map units (TMUs), and so forth.
This is a great opportunity to learn from an expert about GPU architecture basics, and to help demystify some of the marketing language used in the industry.
We have a new GN store: store.gamersne...
Like our content? Please consider becoming our Patron to support us: / gamersnexus
** Please like, comment, and subscribe for more! **
Follow us in these locations for more gaming and hardware updates:
t: / gamersnexus
f: / gamersnexus
w: www.gamersnexus...
Host: Steve Burke
Expert: David Kanter
Video: Andrew Coleman
Links to Amazon and Newegg are typically monetized on our channel (affiliate links) and may return a commission of sales to us from the retailer. This is unrelated to the product manufacturer. Any advertisements or sponsorships are disclosed within the video ("this video is brought to you by") and above the fold in the description. We do not ever produce paid content or "sponsored content" (meaning that the content is our idea and is not funded externally aside from whatever ad placement is in the beginning) and we do not ever charge manufacturers for coverage.
Our thanks to David Kanter for joining us. It's old now, but we think you might really like his article explaining tile-based rasterization on NVIDIA GPUs: www.realworldtech.com/tile-based-rasterization-nvidia-gpus/
Check out our DDR5 memory news: ruclips.net/video/evBKIfBeGJk/видео.html
Back-order a GN Modmat here for the next round! We sold out already. Again. store.gamersnexus.net/
Thank you, David Kanter! I'm very interested in machine learning and have been messing with Tensorflow recently. It's amazing to see stuff like random forests, which were invented way ahead of their time, finally being used for things like neural networks. Good stuff, would love to see more videos like this.
And the GPU is throttling for sure, because the draw distance is dynamically reducing.
The FPS are rock solid though, good job.
Draw distance is a bit low, did you forget to set it to high in settings? Tho hair physics are spot on, are you using TressFX?
Error989 Is the exaggerated levels of depth of field.
Rendered using N64
They had to turn the view distance down to reach 60FPS at 4K.
They had to reduce draw distance to get better out of the box thermals.
Steve HairWorks
David sounds like the kind of guy who could take a few empty soda cans, some plastic spoons + duck tape and produce a 1080ti in the garage.
someone get that man a cape!
OR" rebuilt Radeon AMD Vega to... R.A.V.2.
Ggchb Gig ghb David needs help wiping his own ass. Show me something he has done in the last year? 2 years? 5 years? Ok, lets make it easy, EVER?
Tan Tan hes done more than you
joshcogaming proof?
THIS! Steve, this is the type of content I love. Pretty please, keep this kind of content coming!
Frank R If you're smart enough to keep up with this, you're smart enough to know why this cannot be the regular content.
Sorry to say but it's views that pay the bills and thus making it accessible to the majority is financially the best option. Leaning either way too often kills revenue.
I discovered David Kanter through Arstechnica (one of my favorite tech/science websites) and learned a lot about chip engineering. While that website tends to cater to a rather technical demographic, Gamers Nexus has come close to matching their attention to detail. There is a lot of science behind Steve's content and his audience (based off his current subscription total) is far from the majority that visits, say LTT.
So that could imply that the Corsair Dark Core SE does not contain an actual core either? /mindblown
Brought to you by the Corsair Dark Lane on a Vector Unit SE!
Doesn't quite have the same ring to it.
It really has stream processors.
So if bob owns a orchard with apple trees and its over 400 apples per tree.....(does math) omg bob has over a million cores on his farm, hope its a crypto farm
no its a apples farm... xD
some computer mice have actually an embedded microcontroller. these run their firmware what is a tiny bit of software. that does probably some stuff to smoothen your mouse movements and to detect when you lift up your mouse. this mouse could actually have a working processor in it. nothing powerful but it executes commands and processes sensor data
The inner machinations of my mind are an enigma!
-Patrick Star
Atilolzz poetic
**Bill Fagerbakke
Really loving the technical topics GN has been presenting lately
Steve's hair flowing majestically in the wind.
As a Computer Science PhD student I find this channel to be an amazing source of technical information that, at least for bleeding edge hardware, can't be found anywhere else. Keep up the great work!
i think i am to stupid for this type of content......will watch anyway :)
Sebastian wait until the discussion about CGI real time DXR gaming technology.
it's two
Same, but it excites me to have a lane to further my understanding of computers and a direction to look for it also.
Basically a core is a self sufficient & independent structure that can access memory and other things on it's own. What the guy is saying is that he doesn't consider CUDA cores as true cores because they are not independent from the rest of the structure. A vector execution unit (which is what a CUDA "core" really is) is the part of the chip that traces 3D vectors, angles, magnitudes etc... it does the math to move & transform 3D objects. 3D graphics is mostly vector math. A vector is essentially a position in 3D space with a direction & an amplitude. Think of it as an arrow pointing in some direction. It's the fundamental structure of all 3D graphics.
In shader programming, in order to create the illusion of lighting (lit/unlit objects), you have to multiply or in simpler terms; join two or more colors together. These colors are stored as floating point numbers or decimal numbers. Once all the math is done and the final color is defined, the pixel is drawn on the screen. Now before it actually becomes a pixel, during the math process it's actually a vector stored inside a matrix, not a pixel.
That's about as simple as I can explain his multiplication process.
The most interesting thing about CUDA cores and SMs (at least in my opinion) is that one SM can render multiple pixels at once, by having it's different CUDA cores calculating with different values. But since there is only one instruction fetch and decode unit all CUDA cores must be state locked (meaning calculating the same instructions). This may result in wired behavior when running shaders with many branches, since all CUDA cores may run all branches, but discard the results of the calculation. This is why it can be faster to calculate everything the shader could need and then multiplying it by one or zero instead of using an if, since both parts of the if statement may get exicuted if the different pixels need to take different branches. This is something every graphics programmer should know, but many don't. Also that's why it's actually not that bad to market with the amount of CUDA cores, since that directly corelates to the amount of pixels that can be drawn simultaneously. (Please note that this is simplified you don't need to tell me that some details are not perfectly correct :p )
i think you left out to much detail but yeah
A lot of graphics programmers are familiar with branchless programming. It's some what relevant to CPU optimisation as well were missed-branch predictions cost so much performance.
How is always calculating what each branch needs first more efficient than letting one branch calculate (while the others are on hold), and then the others doing their thing? It seems equal to me, except that in the case where an entire warp takes a single branch, using the second method, you save the time of doing the calculations for the other branch. Given that a warp is only 32 ''threads', the odds of everyone taking the same branch sometimes aren't that bad.
listen man, you left out some details. I want to speak to your manager.
@@sjh7132 Because splitting the work across cores isn't free. When the GPU is executing code inside of an if statement, it needs to enable and disable its execution flag for the cores that aren't going to execute this cycle. Enabling/disabling the execution flag takes time. Additionally, doing so breaks the pipelining process, giving you less than 1 instruction per cycle before and after the execution flag is changed.
(Instructions usually take several cycles to execute, the reason why cores achieve 1 instruction per cycle is because they're pipelined)
If you have short if statements, the enable/disable issues are a significant fraction of the time spent executing the whole if statement.
If the if and the else statement both have very long bodies but are doing something similar, branchless programming can often mean combining them into one body. For a CPU it doesn't matter, for a GPU it's twice as fast in warps that execute both sides of the if.
Of course, if you're just wrapping a texture access with an if-statement, it's better to keep it an if than access the texture every single time (Unless you expect the if to happen a very high percentage of the time, but that's up to you)
This guy knows what the hell he's talking about!
Everyone likes to forget about the GeForce 8 Series where 'CUDA Cores' had yet to receive their name and they were just called Streaming Processors. In the same way that Hyperthreading is often called an Intel thing but SMT is an AMD thing, yet they are one-and-the-same. Hyperthreading is just marketing.
Need to watch it again, I'm damn lost
This was immensely interesting. I love the series of interviews you have been doing lately, it's great.
Steve can you do technical analysis just like his some time in the future? It’s always great to get sources from multiple people. As shown at the beginning. I love in depth info of how computers work transistor level.
You really Hammered that answer out !!
Sorry but had to it . !! lol !!
Videos like this are why you guys are my favorite tech channel! Keep them up! Educating the community is so valuable, I'm glad you guys take the time to get into the real details!
Videos like these are the reason I come to you. Also because you are honest about your reviews. All hail tech Jesus.
This is good content. These technical videos are a great change of pace compared to the usual content GN produces. Love it.
Awesome video, I love this discussion format. This is a really interesting topic, explained simply enough to make sense to me, but just far enough over my head to get me to do some additional reading. I love it! I would really enjoy more videos like this. o/
More David plz, always a treat. I really miss the old TechReport podcast days.
Someone's trying real hard to tell us a *knock knock* joke
I like this (minus the hammering in the background). Let's have more expert-interviews where they explain something in detail!
I really like all those interviews you are putting out lately
This video aged really well. Thanks for making these Videos, Steve.
I loved this talk, so much great information packed into 17min!
Neat. I still don't get it, but I'm not a microprocessor engineer so what are you gonna do.
in short what GPUs do most of is a Fuse-Multiply-aggregate instruction the take 2 values in binary put them together using the "ADD" instruction then multiplies them together then adds it to the value in the holding place that it got the data from until the FMA stream is done and does it very fast as it does not need anything but the FMA instruction for them to use the other instructions done are done with less units or more that nVidia and AMD do not tell the public how many are in that block of their GPU's block diagram
@Jordan Rodrigues how does that relate to the fuse-multiply-aggregate instruction on the instruction set of the GPU?
II agree they are more open about what they are doing but you most have misread or not understood what i meant. Both nVidia, AMD, ARM and Intel use some version of this instruction. The programmer never gets to see the binary most times as programming using APIs is a lot quicker and a lot more compatible with more hardware than programming using binary code. Now how they do it is different in APIs from it on a GPU to it on a CPU due to what is being "called" but it is the same operation being done. It is all math so you only have add, subtract, and store. That is all all microprocessor can do when the instructions and data is on the microprocessor.
Love these highly detailed videos!
I really like this type of content; in depth technical descriptions of how GPUs are excellent at linear algebra!
HAMMER TIME !, awesome video btw, keep it coming!.
Thank you! So many questions answered! And then some! I appreciate videos like this Steve, keep up the great work, love your channel!
just found this video and for someone who has a bachelor degree in computer science this makes a hell of a lot more sense than the usual jargon that is usually thrown around.
In short a "core" is usually made up of:
1. A set of circuits that handle integer instructions (full number arithmetic/logic) or "Arithmetic Logic Unit".
2. A set of circuits that handle floating point instructions (decimal place arithmetic) or "Floating Point Unit".
3. An "Instruction Decoder" which directs the instruction to the right set of circuits (ALU or FPU) and converts the instruction into the electrical signals that go into that set of circuits.
4. lastly we have the "clock" which turns "on" and "off" at a set speed to differentiate between one instruction and the next.
In this video he says the GPUs use a core setup where there are still only one "Instruction Decoder" and "clock" but a large number of FPUs designed for specific instructions.
Thus what Nvidia calls the shader core is not the whole package but just the FPU that actually computes the decoded instructions. By limiting the core to only handle a small set of instructions and only use floating point numbers they can make these FPUs much smaller and cram many more of them into the same area.
roax206 you forgot the fetch unit and the store unit, also modern cores have other units like branch prediction
I like the use of "knocking on heavens door" as your backtrack throughout the video.
Best content GN has ever made! Thank you so much! Need more of this.
Finally a video that goes into detail. Would have loved some more insight into Bulldozer.
Kanter's deep-dive article on Bulldozer is on his site: www.realworldtech.com/bulldozer/. Lots of block diagrams and detailed analysis comparing BD to previous AMD and Intel microarchitectures. Bulldozer is an interesting design, but several of its features were mistakes in hindsight that they "fixed" for Zen. e.g. Zen has normal write-back L1d cache vs. Bulldozer's weird 16k write-through L1d with a 4k write-combining buffer.
Having 2 weak integer cores means that single-threaded integer performance is not great on Bulldozer-family. Zen uses SMT (the generic name for what Hyperthreading is) to let two threads share one wide core, or let one thread have all of it if there isn't another thread.
Zen also fixed the higher latency of instructions like `movd xmm, eax` and the other direction, which is like 10 cycles on Bulldozer (agner.org/optimize/ ) vs. about 3 on Intel and Zen. Steamroller improved this some but it's still not great. Kinda makes sense that 2 cores sharing an FP/SIMD unit would have higher latency to talk to it, and it's not something that normal code has to do very often. Although general-purpose integer XMM does happen for integer FP conversions if the integer part doesn't vectorize.
During filming a construction worker is performing many hammer to nail operations
i always thought it was strange that a GPU could have thousands of cores while CPUs are lucky to reach double digits
also, knock knock
Valfaun who's there?
Daniel Bryant Me a nugger
Daniel Bryant someone who wants them nerds off their roof, perhaps
Best analogy I heard is that a CPU is a room with a few geniuses that solve complex problems while GPUs are like gymnasium filled with teens trying to solve for x.
More like a gym-ful of teens told to go paint one picket each on a fence.
Thanks! This exactly the video i needed. This CUDA core thing was driving me nuts.
This is some good background noise. Thank you for that! :)
Ivo Silva ambient noise lvl set to 100%
Interesting chat. I love these little talks you have with people now and then, please keep that up!
No matter if it's random cool people or industry insiders, it's always interesting to hear people who know what they're talking about, well, talk! About stuff they know! :)
Love this this kind of technical discussion. Keep them coming.
That guy with hammer is really annoying 😠
Steve looking over the edge to check if the guy is within spitting range before realizing he's on camera and proceeding to nod in agreement.
Omg. I thought maybe there was something wrong with my receiver. I know it's not their fault but holy crap is that distracting.
At least he had parents who gave a damn unlike you who acts like an asshole for no reason.
That's not a hammer, it's the audience banging their heads against the wall trying to follow the explanation!
As a C++ Programmer (dabbled in GPGPU) I found this video very interesting!
Easy to fallow, great interview!
hahahahaahah
I love these intelligent discussions/interviews/explainations. Please continue doing these GN. 😁
Great and informative as usual. Thanks for the knowledge!
Great video! Loved the depth of it! Want more!
I like this type of content. Same as the one with primitive shaders. Keep it up!
Niceeeee :D
Also some point that were not mentioned:
Programming on a gpu is much harder than multi threading (use every core of a cpu) due to their physical properties
And even for multi cpu, most of the time parralelising a workload is not so easy
I'd like Steve and GN to do a multi-episode detailed analysis of graphics architectures since the beginning of PCs. I've been PC gaming since the final days of vertex and pixel shaders (2005 & 2006), and the switch to unified shaders rightfully took the PC gaming world by storm. Interestingly enough, the first unified shader GPU - Xenos in the 360 - more or less were directly related to the pixel shaders found in the Radeon X1000 series.
very nice, I really enjoy it, I wish more talks with this guy
I think a little clearer explanation is to point out that all the threads in a warp / workgroup share a single program counter. If two sides of a branch are taken by different threads then you wind up with branch divergence and the threads have to serially execute the two branches individually. Following that serial execution the thread scheduler can then reconverge the threads so they all run in parallel again. But basically if you're not branching then the performance of all the cuda "cores" behaves the same as if they were all full-fledged "cores"
great info in this video. Thanks!
Huge thanks for publishing technical material like this! It's really nice to get behind all the marketing nonsense to get a better understanding of the underlying technology.
Thank you for a great piece of content. This is exactly why I subscribed in the first place, keep it up!
I feel this is a rather subjective definition based problem.
For a more "layman" example, think about an apartment complex versus a mansion.
Both are a single building and can be equal size.
But they have different number of "homes" within them.
The question then is, is "core" a building or a home.
One can further equate each processing lane or whatever the proper term is to a room, where in theory both building and home are a collection of and therefore a home qualifies as a "core" as a collection of "rooms" or processing lanes. But then we have studios, which is in effect a home of a single room.
Edit: perhaps it is "fair" to say the building is a CPU/GPU, a home is a core while rooms are processing lanes or whatever. But then, we once again arrive at the fact that there ARE homes of a single room. There is no reason, as far as definition goes, why we can't have a "core" of a single processing lane. I mean all else aside, if I had a "stupid" calculator that can ONLY do the fused multiply add function on the single "CUDA" core that it process, does that mean it have no core at all even if all its function is done on a single unit as opposed to distributed on a whole PCB etc?
On the one hand, you can argue that each CUDA is part of a core, as only a processing lane or whatever the correct term is.
On the other, you can argue that each CUDA is a stripped down core, down to a single processing lane(or whatever) because GPU "core" doesn't need all the functionality of a CPU core so overtime, they "devolved" from full fledged cores to what they are now.
Personally, I am much more annoyed by Nvidia mis-appropriation of the term "Max-Q" which have NOTHING to do with the actual meaning of the word in its originality as well as the "standard" "4K" which is entirely a marketing term that have no proper definition (2160p is what people should use, for what is commonly considered "4K" and what people SHOULD use. But 4k sounds "twice as good" as 2160p so here we are). Because those are more clear-cut. Something like this... I don't think it's clear-cut at all because a proper analysis would require considering the development history of both the components and word. So no, I'm not saying you are wrong. But you are also not "right", not unless you write a 200 page study on the historical evolution of the term and GPUs and note down the correlation every step of the way.
Wow this video is so excellent, David is amazing!
Thanks for this one! Learned alot
This video teaches more about technology than a entire semester of tech class.
Very cool, knowledge is power. This video opens up a whole different perspective on how GPU's and CPU's differ but yet the same. Very interesting. Thumbs up!
We are at the brink of advancement in computing technology. The only problem is making the computing technology smaller without affecting temps and transfer. If we ever start using gallium nitride as a core component and replacement of silicon then we might be looking at the next jump in hardware evolution. So exciting.
Very informative. Thanks!!
Best. Channel! This contend is amazing!
Half of me loves the complicated content because I can learn from it, the other half hates it because it's hard to understand.
1. So that's THE David Kanter, amazing
2. As far as I know (before watching this video) the CUDA "cores" are more like ALU than complete core. Nice deeper explanation here.
Please makes these interviews much longer. When you are already interviewing someone who knows about that low level stuff that you should try to get even more information out of them.
Good interview. Keep going!
Please do more technical content like this.
A very interesting piece of content. Goof stuff, Steve. Keep em coming :)
To the guy hammering away in the background "We're trying to shoot a *bleep*ing video!"
Nice! I suggest the next video should be about FPGA‘s and why they perform better for ray tracing.
thanks for another excellent and informative video Im watching it with my newborn son Loki
Tomos Saggers congrats on the baby!
thanks dude take care
Super interesting conversation!
How many years in seclusion at the top of a mountain does one have to spend to achieve such deep understanding of not only CPU but also GPU architecture?
I strongly disagree with the title / thesis of the argument. CPUs operate at a standard 1 operation per second, so Hz is a measurement of FLOPS. CUDA cores / Shader Cores are identical. They execute 1 FLOP per cycle.
Now, in 1997, Intel came out with MMX, allowing a CPU to setup a 64bit register with two 32bit floats, and execute 2 FLOPs in a single clock cycle. MMX. Note that videocards predate MMX, where MMX didn't become common place until into the 21st century and software support was minimal for a long time.
MMX doesn't make any sense in GPUs, because GPUs are already designed to be parallel instead of serial. Hence, they use the same phrase "core" which meant exactly what core has always meant: something that executed one operation per cycle.
======
Shader cores are true, honest to goodness, "cores" and meet the proper definition. Both CPU and GPUs have their own unique performance penalties when used improperly.
~ GPUs have the requirement that the entire shader must be executing the same code. As a penalty, if two cores in the same 32-core warp take different branches of an if statement, both must be executed in serial. They run at 1-2GHz.
~ CPUs have the penalty that if you access memory from two cores, they must wait a few clock cycles to reach consensus. If an if statement is mispredicted, many clock cycles are wasted as it un-does the work it did. They have special SIMD instructions that act on wide registers. They run at 4-5GHz.
Both are true honest cores. BOTH have severe penalties for using if statements improperly. GPUs' defining characteristic is that it runs one code on all of the cores. CPUs' defining characteristic is high GHz.
Obviously, because a GPU is only ever designed to run one shader code, no, they didn't give every CUDA core its own instruction processing unit. Not because they couldn't, but because it's already forced that they're all running the same code anyway! It doesn't make it any less of a core.
Good interview technique!!!
i wish everyone listened to their guests as good!!!
👏👏😁😁🖒🇺🇸☯♾
Loved the video, very informative
love these kinds of videos
sick no-scope @ 8:04
Cool stuff, thanks for the content!
This is the exact piece of information that i was looking for
I love the guy hammering in the back ground
Can you talk about how the fundamental "core" design changed when going from Fermi to Maxwell? I believe this was one of the paradigm shifts for CUDA.
I want more in depth content like this.
The legendary Kanter! Whooooooo!
I think Windel from Level1Techs would like to talk to this guy
I guess CUDA Floating Point Registers just didn't have the right ring to the marketing guys.
2:47 steve mesmerized by his dream coming true
Steve looks so happy he can talk about one of his sponsors hardware
I look forward to an interview where the guest looks at the camera as much as Steve seems to want them to.
Bold content. Much appreciated. 10/10.
What a smart dude! David Kanter for president. lol.
I’m a ML Guy and ended up here now I’m core audience thanks for the great video
The problem is blurring the lines between cores and processors in marketing dept.
Someone was hammering for sure.
Whenever I want to fall asleep fast, I watch this video. Ambien ain't got shit on this
Right. Right... It's a lane of a vector execution unit. I totally get it now.
I literally got lost past 1:30 hahahaha
need to learn more about gpu architecture before re-watching this vid
lol i thought someone was hammering outside my house
Love such content!