0:00:00 Typical problems during game development 0:08:40 Rasterization 0:16:18 Sight with one eye 0:24:23 Perspective projection 0:35:48 Clipping 0:46:38 Triangles 0:52:40 Faster rasterization 1:00:40 Projection in 3D 1:03:24 Projecting a cheesecake onto a dish 1:08:05 Choosing primitives 1:14:43 LERP, except we don't do that here 1:22:10 Buffer layouts, shaders 1:34:47 Scanlines 1:42:40 Communication between CPU and GPU 1:52:35 The Z Buffer 2:05:50 Z Fighting 2:08:00 When Z Buffering doesn't work 2:10:10 How does translucency work? - Alpha Blending 2:21:25 Solutions for rendering opaque and translucent objects 2:30:56 Current implementation in the game 2:39:30 Chosen solution: Splitting the object 2:45:52 Q&A
1:42:40 Communication between CPU and GPU 1:52:35 The Z Buffer 2:05:50 Z Fighting 2:08:00 When Z Buffering doesn't work 2:10:10 How does translucency work? - Alpha Blending 2:21:25 Solutions for rendering opaque and translucent objects 2:30:56 Current implementation in the game 2:39:30 Chosen solution: Splitting the object 2:45:52 Q&A
One of the important thing of solo game development is Animation. So dissapointed that there are no detailed talk of 3+ hours on internet that explains how to implement animation, vertex by vertex, triangle by trainangle etc. Only closest is by David Rosen of Wolfire games who made Overgrowth; but, it is more like principles rather than low level implementation.
2:43:26 "Then, we will render it twice." My jaw dropped because I immediately realized the rest of the process and how simple it is. This is such an "aha" moment, not unlike ones from puzzle games.
Yeah, that's technically wrong. Light always goes in a straight line, it's spacetime that curves. It's like when two people walk perfectly parallel in a straight line from the equator to the north pole over the surface of a sphere, they appear to turn toward each other until the collide at the pole but they're really going straight and it's the geometry that curves. That said, he clearly understands physics, what he said about QED destructive interference in light paths was spot on so it was probably just a case of putting it colloquially/informally.
@@biggiefrosty It does seem pretty topsy-turvy I'll grant. It's about something called inertial / non-inertial frames of reference and equivalence. And field theory. Don't take my word for it though, look it up (make sure your source is reputable). Edit: with regard to your planet question. Look up the "einstein procession of the perihelion of Mercury". There are bound to be some documentaries out there, it's a great story and it settles the matter.
Thank you for this, it has filled in so many missing links in my understanding of 3D graphics. I finally have a framework where I can start thinking about it all sensibly. Thanks again.
Slowly catching up with your videos. Such a treasure trove of useful tips, insights and knowledge. I do a bit of programming myself, mainly for engineering purposes (C, C++, python etc.); even though I'll probably never code a game or a game engine, I still enjoy listening to how the experts and masters of the trade like you tackle or think about certain problems! There's so much to learn! Thanks for your time and dedication to sharing your thoughts and experiences.
A drop of water got on my screen while watching this and I could see the RGB of the pixels behind it as I moved the screen around. Felt like a surreal practical example of light rendering through a transparent surface…
Great video, I missed this part of the stream. This video has helped me understand how 3D graphics work much better than I previously did! Thank you so much for spreading this information
In such a primitive case when you render objects penetrated into *flat* (water) plane you can split them w/o discard operation in the shader by just modifying camera ViewProjection matrix so that the near clip plane (on the first drawing operation) fits the water plane, so all the triangle clipping will automatically be performed by the rasterizer. On the second draw operation you have to modify the camera ViewProjection matrix so that the far clip plane match the water plane. In this approach you don't have to modify the transparent shaders at all and this won't decrease their performance (which happens with the by-pixel-discard approach). If you are paranoid by the rasterization cracks in the model due to the floating point imprecision you also can tweak the matrices to ensure a little overlapping of the top and lower parts or hide them by other vfx-es (splashes, foam on the water). Also note that the View matrix should be modified only twice per frame independently on the number of the objects to draw. I've did so in a couple of mobile projects without lowering the performance (except for the second pass for each half-underwater transparent object)
As someone with little knowledge of game dev or software in general I am impressed at how Jonathan seems to encourage understanding deep structure / generalizable properties of an ontology AKA knowing the deep shit about shit, as part of the learning process. Increasingly rare style of teaching because of the state of our complex systems which is too long to sketch out here. Thank you Jonathan, you've opened up a new door for me.
Thanks for this amazing explanation. I was going to watch stream but I'm in different timezone. Thanks for also uploading it to RUclips. It really helps!
2:11:50 If you sprinkled in some cocoa, that's the colour you'd get :D Pretty close! I have learned so much from this stream, thank you so much. We need more teachers that have spent years since childhood tinkering in their craft. The important thing to remember is: if you only have MHz processors to work with, you necessarily have to make things fast and efficient! We should learn from that, not sneer at it.
I wish you would do more videos like these! Thank You! I was always curious about this but never looked it up. It was very interesting, educative and entertaining.
I remember reading some paper which collected all sorts of methods for the solving paint-order problem, with a footnote that went something like "of course we could just store the depth at each pixel but that would be way too expensive"
4 года назад+9
Jon understands rendering so well that he can explain it to a 5yo. And by 5yo I mean my dumb self. I felt in the loop for most of the time which surprised me.
Thanks Jonathan for helping explain something I've found daunting to understand as a Game Dev. Respect for putting this knowledge out there for the community.
I liked that I came with exactly the similar idea of cutting the object twice and applying different shaders before the final ending. Its like the most converging simple solution.
Thank you! This is indeed very usefull, as are all your explanations. I really appreciate your stance on the importance of understanding the fundamentals. You and Casey have really helped me understand more about programming and computers. This is also a bit of a curse to be honest because I’m not in the gaming industry unfortunately (yet ;) ) - I took the path of least resistance and ended up in e-commerce and web-tech - and I’m realising how disturbingly bad performance is in applications in this business. I’ve been trying to raise awareness in my company at least about the importance of understanding the hardware and not just blindly follow OOP practices, even for “managed” languages like Java, but I get no traction at all so far. I mean sure, it might sound scarily complicated, but to me it’s very much more complicated to try to understand and follow the multitude of OOP patterns, practices and whatnot as they are based on a bunch of different peoples mental models and they don't always fit together or work well for cases outside the ones they were designed for. Anyway, thanks for all the eye-openers is what I’m getting at :)
I think it's good to use what OOP tools you find useful, but that in many cases convenient can be confused for useful. Managing lifetimes of stuff with destructors can be incredibly valuable, but doesn't necessarily need to be a default. Understanding how many of these tools impact your code long term is incredibly complicated, even for experienced programmers. I always recommend "game programming patterns", can find it free online. It has tons of really useful patterns presented in a very intuitive way. Only learning book I've read cover to cover.
@@Bozemoto Well, sure OOP is a collection of mental models basically that you can use as tools to design your code. You have to learn these mental models and how and when to use them. OOP is difficult to master and there are many models within OOP you need to learn. Designing for the hardware is no less difficult, contains no less mental models you need to learn and is equally difficult to master. So I mean, why not choose the latter, as you also get the benefit of more performant applications.
@@Bozemoto Sure. With mental model I mean the things you have to imagine when talking about something. We can start with the “lower level” ones like encapsulation, for instance. This is meant to prevent “outside” objects to mess with an objects internal state. Already here we have to deal with the mental model of “internal state” and what that means. Then we have things like singleton, factory, builder, observer, etc etc. Factory for instance is a tool, or mental model, to help you manage objects, which in them selves are mental models (what is an object and why do you need one? Because of encapsulation etc). So a lot of the OOP patterns, or mental models, are there to help me manage the other mental models. So I kind of see OOP as trying to solve the problem of programming and it just goes round and round. Instead, I should be using programming to solve my problem, which is manage the inventory in a warehouse, or simulate physics or whatever, and those problems are usually mostly about data and transforming the data somehow.
I think you see it in a wrong way. Usually you have to ask yourself what kind of problem you are trying to solve. Often performance does not matter. Or even memory leaks. If it does not bother anyone except yourself, it's probably fine to ignore it. Don't take my words too serious tho. It is important to have nice code, amazing performance, no bugs, leaks or whatsoever if this is the problem that needs to be solved. In games, or let's say engine work, this is important. In one time project that will never be touched again, probably not. A website that needs to look amazing and perform okayish, probably not as well. So, back to it, what problem needs to be solved is the question that you should ask. It is also fine to change your job if all those criteria matters to you
This is incredibly enlightening, but every time he said "you can't do this because your math will become inaccurate" I was left with SO MANY burning questions.
47:40 I think its because a triangle is the simplest shape that can make any other shape (okay maybe not perfect circles if you limit how small thay can be, but you get my point).
You know what's funny? I see him talk, I see code that does math, and it all makes sense, but as soon as I try to learn from math people who just use the syntax of math formulas and math jargon, it feels like complete gibberish that's way over my head.
1) Feed the plane equation of the water into your shader. 2) In your vertex shader calculate the signed distance from plane and interpolate into your fragment shader (a vector dot product and scalar subtraction) 3) draw everything transparent intersecting the water discarding fragments above the plane using this signed distance, 4) draw water, 5) draw all the things intersecting the water discarding the fragments below the water (a simple depth test here could suffice if you write the water to the depth buffer but the complimentary discard would be guaranteed to avoid cracks (the complimentary fragment invariance you were concerned about)) This is quite trivial to code. A more fully general system might involve any transparent planes in a BSP sorting structure. There's the secondary issue of drawing the water inside the transparent objects which really should displace the water and not have it inside their volume. This can be fixed using the stencil buffer to rasterize the inside of the rock intersection with the water to the stencil buffer and reject water samples there, but I would suggest this is a secondary effect and doesn't really needing fixing unless you care about water that might be more opaque. If you insist on fixing this though, you must in the above sequence insert: 3.4) mask off all color writes 3.5) draw water to depth buffer only 3.6) on depth pass draw outside of all intersecting objects with stencil increment 3.7) on depth pass draw inside of all intersecting objects with stencil decrement 3.8) turn color writes back on 4.0) draw the water stencil testing to draw to only stencil == 0 on depth pass, 5) as above. This assumes your objects are manifold hulls and you clear stencil buffer each frame. But again masking the stuff INSIDE the objects is probably overkill, but you might take some pride in accomplishing this detail.
Yes. Think of it this way: For each vertex of the triangle, you can draw a line, from the eye point, that intersects that vertex. Every point further out on each of these lines (and closer too!) projects to the same point in screen space. Thus, any triangle made from *any* of the points on those 3 lines, will project to the same triangle in screen space.
I live in the film / VFX universe of things - and finally I can pinpoint what makes this so similar to the gaming universe: All we actually do is create tons of tiny, logical triangles to fill a screen. Thanks for the insight! Sub.
So Basically, in the quantum world, photons will "take" every path to "their destination" even the crazy ones you saw him draw. All those paths arrive to your eye. What you see is a combination. The thing is, the weird paths have equal and opposite weird paths. This causes them to cancel out to 0. The only paths that don't have this are the straight paths. That's my attempt at explaining this. I think The Science Asylum explains it better.
I have a few issues with this, mainly how rasterization is explained in terms of physical analogies (like you would raytracing). The earlier a learner understands that everything is completely fake and all is made by projecting points and going from there, the better, IMHO.
1:17:00 Can someone give me a hint how he determines the coordinates of this point, given the 3 other points ? Or does he just mean, for any given point you can check if it is inside the triangle ? I am a bit confused. EDIT: Maybe he means that the 3 points of the triangle allow him to know every point that fills the triangle and that is not possible for the quad ?
For every point, given the lengths of the perpendicular arrows Jon drew and the triangle point coordinates, you can calculate the coordinates for the point the arrows lead to. Do you want the actual calculation? In reality, you definitely choose the points and get their coordinates in some other way. I think the main point is that you calculate the world coordinates of pixels you draw during rasterization and can use it in pixel/fragment shaders. Jon explains some of that during the rasterization part. Indeed, this is not possible for quads unless you make great restrictions on the quads, most importantly to force them to lie in a single plane.
Dude, I love this so much. I was wondering how the heck worldspace to screenspace transformations worked like 2 weeks ago. I also walked away knowing how to implement a triangle fill efficiently. This was perfect timing.
You don't need trigonometry to get position on screen just algebra. It's the intersection of line of sight with the line of the screen. In the frame of reference where the screen is axis aligned, it's (offset / distance) * scale.
Humans ask questions also as a way to feel included and socialise. I was watching a streamer and honestly there was random questions every 3 to 4 seconds.
Beginner question that do not matter in large scale. Maybe you should make a video to give decent answers for default values of those for the beginners so that beginners don't procrastinate on trying to find answer to question that doesn't matter in the long run and can actually start working on their project. The which graphics API is good question, the time they invest in learning the API matters and if they make wrong choice it will take too much time that could be spend on actually producing things. So time to Triangle would be most important metric to optimize for and available on multiple platforms would be beneficial so that learning it would be great. I use Vulkan and I don't recommend that for anyone's first project.
So even though raytracing is more expensive, it might be easier to code because you know what object is first in line of sight and transparent objects just modify the colour the object behind it?
Why a quad (which is a 2D object) somehow makes multiple planes is not well explained. He muttered something briefly about what sounded like floating point rounding errors, so im going yo assume this is the reason, but im not 100% sure and it should have been made more explicit.
With a triangle, if you move any of the points in their depth coordinate (further away or closer to the camera) it doesn't distort the shape's flatness, it just changes the angle of the plane the triangle is aligned with. With a quad, if you start moving points further away or closer, for example moving a corner forwards in space, there's no longer a flat plane that all the points sit on, the shape has to bend to accomodate. Any 3 points of that bent quad will be on a plane, but those planes aren't guaranteed be aligned with each other in any sensible way. So you can't reason about where in 3D space a particular co-ordinate within that quad will be without breaking the shape down.
I sometimes wonder how good/bad would be to have three Z values instead of one: Z, minZ and maxZ with a simple min-max search going on while rasterizing transparent objects. If there is no transparency minZ and maxZ would be just copy of Z and thus we can indicate the special case easily. Then if the two values differ from Z, because there were transparent objects drawn, then we can know that there were already a raster / pixel drawn at this location with transparency BEFORE we are drawing at this location. We can also know if we are the closest to the camera as of current order or farthest or "in-between" and the shader would get this information alongside the current color. I know it can still generate artifacts, but no need for sorting this way and artifacts are mostly just off-colors when transparents are behind transparents in more than two layers and even then likely there is some quite normal approximation happening at least.
Understanding pinhole cameras encapsulates a lot of the confusing knowledge about how we see. I just wish Textbooks didnt just show a picture of the camera and say yep, that's what it looks like and therefore that *waves hands* is how it works. I understand the point where all the light passes through (the "eye" in the drawings) and the "screen" as two parts of a virtual eye. Then the aspect ratio makes more intuitive sense. So the drawing is a virtual eye's view that we're putting on the screen. Our real eyes can see it at any distance from the screen and it still works because all the math was done for the virtual eye's parameters. Edit: it's amazing looking at poor explanations I wrote months ago. I see that it's nearly unintelligible, yet simultaneously I still see the meaning I thought I was conveying clearly
I've been coding my first 3d-engine, it's made with Powerpoint VBA (don't ask, its a demoscene wild demo flex). and it is visualized by drawing triangle shapes using the shape-system of office programs. So in a way it doesn't draw pixels as such, only triangles, but after implementing all the occlusion culling and simple normal angle lighting systems it dawned on me that creating a proper Z-buffer using triangles will be most likely impossible without a proper per pixel z-buffer. As I've been sorting the triangles in screen by the middle point of the polygon there is considerable z-fighting on some 3d-models. But moving away from the shapes-system defeats the joke, I might need an old-school 3d-modeler to tell me how to model for such engines..
You could also do something like Quake or DOOM where the geometry is actually sorted in a way that only the visible polygons are rendered in order. But you have to cut the polygons and sort them.
I've always been interested in 3d graphics even though I couldn't even write hello world. This helps me understand a bit more. One day, I'd like to write a program that makes a 3D cube, that by itself would feel magical to me.
you should remake stephen's sausage roll and call it Jonathan's cheesecake, and use a spatula instead of a fork. As always, the video was a great way to fill out holes in my technical knowledge from a high level perspective that i can use as a technical game designer to make better decisions as a designer from understanding the technical requirements. Thanks
Jonathan, I admire your craft and commitment to game development! I also admire and follow Daniel Benmergui! You just explain with the 3D to 2D what I was asking and looking in books years ago (2002-2003), I make in that time in Quick Basic a 3D render (only "meshes" lines), but I use a way to get the 2D position completely diferent, It uses the screen size but not the camara distance to the window... Is more complicated, but I don't know if it had the problem of the object get through the virtual screen. Were I can find another ways to represent or obtain 3D position into 2D? there is a book or something that explore this possibilities? Thxs!!! I'm not a native speaker. I loved several of your talks about game design and farming of the player in some games!
as you were giving your explanation of how rasterisation of 3d on to a 2d plane (screen) works (1:37:44), it struck me that if gpus were to actually take, and instead of stepping down the screen or filling a line like in your software renderer... imagine if all of the gpu knew about all of the geometry the scene would need to render, then per pixel, massively parallel, it did the similar triangle thing to find all the intersections with geometries on the ray cast from of camera pos, out through the FOV where the screen is, coming out toward the world. then, for each intersection with a triangle that ray cast intersects, the triangles involved would be rendered from front to back (like with the z-buffer) discarding everything further, cept the transparent ones go the slower path of having to go from back to front. this technique (I dunno if it's ever been suggested before), of casting from the FOV out toward the screen was interesting to me because it could simplify problems like the one you outlined, and potentially add the ability to render much more in parallel than happens now... really enjoyed the discussion :)
If I understand your proposal, the problem I see is that you then have to iterate over every triangle for every pixel, rather than just iterating over every triangle once, and only iterating over the pixels that roughly cover that triangle.
@@wessmall7957 good point. I supposed that there would be some sort of way to partition the space so that ... yeah that would be somewhat complicated. however, I suppose with time hardware will have increased so much that it may be possible to do a full lookup for each pixel because there are like 2^20 parallel processing unis or something :)
Is this project only using z buffers for depth effects? Your project doesn't look like it requires advanced graphics solutios so it might be beneficial to use software tricks from the 90s and early 2000s so you don't have to check every single pixel. Then again a modern CPU & GPU has no trouble dealing with this
Regarding the CPU v GPU analogy. The Bruce Lee quote "I fear not the man who has practiced 10,000 kicks once, but I fear the man who has practiced one kick 10,000 times." fits pretty well. CPU can do any kick imaginable and also backflips and other tricks, but it can do them e.g. ~8x2x5,000,000,000 times per second (minus fetch times). GPU can do only a couple different kicks, but it does them ~76x128x2,200,000,000 times a second. That is about 267:1 win for the GPU. The GPU cache latency might be higher than CPU but since the fetch sizes are much bigger and more uniform, it doesn't necessarily matter at all.
2:45 Jonathan - I have a simple idea how to fix this. JUST make that grid lines not above (on) the water, BUT below, on the floor (under water), I mean- only in the water, the grid lines should be painted exceptionally on the bottom (floor) of the water. Then it will be easer to quick define where crystal or stone lies (on which line and column) Because now it is a little confusing... I have to FORCE my brain to see that lines are on the top of surface (water). It is not intuitive. This is why in some angles, brain see that stone or crystal takes two places in the same time. In fact on the first time my brain immediately assumed that lines are on the bottom (because it is logic to see that floor/bottom is painted, instead of believe that lines are floating magically on the top of water) Greetings from Poland. BTW: I love Witness and Braid. Thank you so much for these awesome games, and your hard work.
But when the lines are on top of solid, floor tiles, should those lines be in different heights? I mean, you solutin sure is a practical one for the graphics engine prespective, but from the UX, I believe you just changed one problem for another.
@@riibbert change of height only for water area. Solid floor should be untouched. solid floor has lines on the bottom on ground. You can walk on it and the lines are under your feet. but on the water block you cannot walk on it, but you fall down by one block down, so the lines also should "fall" down and be on bottom of water block. otherwise it will be hard for me to force my brain all the time to figure out on which line exactly things lie. this calculation takes more time, that my proposition which gives more intuitive look.
can we talk about how good those rgb guesses were for cream and brown? I know he's done this a long time and has done a lot of rgb tuning most likely and that's not the point of what he's saying but oh my god those were good. Perfect cheesecake and beer bottle
The model used in most 3d engines is not exactly correct. You can see why if you look at some long horizontal thing in front of you, for instance your monitor. In your field of view, the central part of the monitor is bigger, because it's closer to the eye. The corners of the monitor are farther from the eye, so it's not exactly a rectangle in your eye. It's subtle and hard to notice, because the monitor takes relatively small angular area in your view field. But you can notice the implications of it when in the 3d scener something gets really close to the camera, like faces or gun muzzles. You would see it very differently if these objects got close to your eye
What are the stylistic freedom of ray tracing, comparing to traditional polygomic + shaders? Haven't dive into it yet but all examples I saw have the exact same look
I don't quite understand what "stylistic freedom" means, but ray tracing makes sense because it adheres more to the nature of our real universe and how light works in it than conventional rasterizers do.
It's not mature yet, but as it always happens once it's more accessible you're gonna see a lot of creative ways of using the technology that weren't "intended".
0:00:00 Typical problems during game development
0:08:40 Rasterization
0:16:18 Sight with one eye
0:24:23 Perspective projection
0:35:48 Clipping
0:46:38 Triangles
0:52:40 Faster rasterization
1:00:40 Projection in 3D
1:03:24 Projecting a cheesecake onto a dish
1:08:05 Choosing primitives
1:14:43 LERP, except we don't do that here
1:22:10 Buffer layouts, shaders
1:34:47 Scanlines
1:42:40 Communication between CPU and GPU
1:52:35 The Z Buffer
2:05:50 Z Fighting
2:08:00 When Z Buffering doesn't work
2:10:10 How does translucency work? - Alpha Blending
2:21:25 Solutions for rendering opaque and translucent objects
2:30:56 Current implementation in the game
2:39:30 Chosen solution: Splitting the object
2:45:52 Q&A
2:11:20 Jon guesses a very decent rgb value for “cream color”
1:42:40 Communication between CPU and GPU
1:52:35 The Z Buffer
2:05:50 Z Fighting
2:08:00 When Z Buffering doesn't work
2:10:10 How does translucency work? - Alpha Blending
2:21:25 Solutions for rendering opaque and translucent objects
2:30:56 Current implementation in the game
2:39:30 Chosen solution: Splitting the object
2:45:52 Q&A
@@MrTheSaxon Thanks
It's time we start giving awards to people who take time to write timestamps under long videos.
Thank you!
@@zhulikkulik You made my day :D You're welcome
For people interested in this topic, John Carmack has an excellent presentation about the Principles of Light and Rendering.
Do you mind linking the presentation? :)
@Caleb Rollins thanks!
@Caleb Rollins cheers
Loved it, I think Carmack has that insane ability to explain complext topics to all levels at the same time.
One of the important thing of solo game development is Animation. So dissapointed that there are no detailed talk of 3+ hours on internet that explains how to implement animation, vertex by vertex, triangle by trainangle etc. Only closest is by David Rosen of Wolfire games who made Overgrowth; but, it is more like principles rather than low level implementation.
2:43:26
"Then, we will render it twice."
My jaw dropped because I immediately realized the rest of the process and how simple it is. This is such an "aha" moment, not unlike ones from puzzle games.
based on the time you posted this, we may have had that same moment at the same time, haha
A cheesecake so nice, we'll render it twice
This is lovely. I wish Jon had more time to teach us newbs glorious wisdom like this
I fell asleep watching this and when I woke up I had a basic idea of what a shader is and how it works. No joke.
no joke, just lies
Heard these kinds of explanations so many times, and yet still feel amazed at the beauty of the maths behind it all
"Remember how mom said don't sit too close to the TV? Well... rip"
My brother and I grew up playing split screen on a 13 inch CRT…the only place to sit is CLOSE 😂
"light goes in a straight line when gravity isn't too strong", weird gravitational lensing puzzles confirmed
Yeah, that's technically wrong. Light always goes in a straight line, it's spacetime that curves. It's like when two people walk perfectly parallel in a straight line from the equator to the north pole over the surface of a sphere, they appear to turn toward each other until the collide at the pole but they're really going straight and it's the geometry that curves.
That said, he clearly understands physics, what he said about QED destructive interference in light paths was spot on so it was probably just a case of putting it colloquially/informally.
FECU! That’s a strange way to put it. If you call that straight then you’d have to also say that Earth moves straight in its orbit around the sun
@@biggiefrosty it's all perspective, in the light's perspective, no one is moving
@@biggiefrosty It does seem pretty topsy-turvy I'll grant. It's about something called inertial / non-inertial frames of reference and equivalence. And field theory. Don't take my word for it though, look it up (make sure your source is reputable).
Edit: with regard to your planet question. Look up the "einstein procession of the perihelion of Mercury". There are bound to be some documentaries out there, it's a great story and it settles the matter.
Double-slit experiment: light doesn't go straight after the slit
Thank you for this, it has filled in so many missing links in my understanding of 3D graphics.
I finally have a framework where I can start thinking about it all sensibly.
Thanks again.
Slowly catching up with your videos. Such a treasure trove of useful tips, insights and knowledge. I do a bit of programming myself, mainly for engineering purposes (C, C++, python etc.); even though I'll probably never code a game or a game engine, I still enjoy listening to how the experts and masters of the trade like you tackle or think about certain problems! There's so much to learn! Thanks for your time and dedication to sharing your thoughts and experiences.
A drop of water got on my screen while watching this and I could see the RGB of the pixels behind it as I moved the screen around. Felt like a surreal practical example of light rendering through a transparent surface…
"The guy with the molotov cocktail IS the GPU driver..."
I fucking died XD
Great video, I missed this part of the stream. This video has helped me understand how 3D graphics work much better than I previously did! Thank you so much for spreading this information
In such a primitive case when you render objects penetrated into *flat* (water) plane you can split them w/o discard operation in the shader by just modifying camera ViewProjection matrix so that the near clip plane (on the first drawing operation) fits the water plane, so all the triangle clipping will automatically be performed by the rasterizer. On the second draw operation you have to modify the camera ViewProjection matrix so that the far clip plane match the water plane. In this approach you don't have to modify the transparent shaders at all and this won't decrease their performance (which happens with the by-pixel-discard approach). If you are paranoid by the rasterization cracks in the model due to the floating point imprecision you also can tweak the matrices to ensure a little overlapping of the top and lower parts or hide them by other vfx-es (splashes, foam on the water). Also note that the View matrix should be modified only twice per frame independently on the number of the objects to draw.
I've did so in a couple of mobile projects without lowering the performance (except for the second pass for each half-underwater transparent object)
This is a great idea. Just goes to show that there's always another trick to learn...
Hello Jonathan,
it's very kind of you that you grant viewers an insight into how you do things and why. Much appreciated!
There are so many channels on YT that answer those easy questions. Not so many that force you to think about the hard ones. Thanks for this Jon.
Came for a programming lesson, stayed for the physics lesson.
Thank you the full deeper explanation even for beginners. Keep up doing awesome work Sir!!!
This was really interesting and informative. Thanks for taking the time to talk about it.
I'm just watching this for the looting stickfigure drama deeplore.
That and the importance of cheesecake in the rendering of frames in 3D games
44:15
Looting is popular nowadays.
@@fluffy6923 fire
the looting comparisons were killing me 🤣 "the tv we looted isn't as nice as the tv they're looting"
As someone with little knowledge of game dev or software in general I am impressed at how Jonathan seems to encourage understanding deep structure / generalizable properties of an ontology AKA knowing the deep shit about shit, as part of the learning process. Increasingly rare style of teaching because of the state of our complex systems which is too long to sketch out here. Thank you Jonathan, you've opened up a new door for me.
You're such an inspiration. I'll definitely make time to watch this whole video by the weekend. Thank you for making this.
Thank you for having recorded this and spent the time! I learned a lot. Why triangles now makes sense.
Thanks for this amazing explanation. I was going to watch stream but I'm in different timezone. Thanks for also uploading it to RUclips. It really helps!
2:11:50 If you sprinkled in some cocoa, that's the colour you'd get :D Pretty close!
I have learned so much from this stream, thank you so much. We need more teachers that have spent years since childhood tinkering in their craft. The important thing to remember is: if you only have MHz processors to work with, you necessarily have to make things fast and efficient! We should learn from that, not sneer at it.
I wish you would do more videos like these! Thank You! I was always curious about this but never looked it up. It was very interesting, educative and entertaining.
I remember reading some paper which collected all sorts of methods for the solving paint-order problem, with a footnote that went something like "of course we could just store the depth at each pixel but that would be way too expensive"
Jon understands rendering so well that he can explain it to a 5yo.
And by 5yo I mean my dumb self.
I felt in the loop for most of the time which surprised me.
Thanks Jonathan for helping explain something I've found daunting to understand as a Game Dev. Respect for putting this knowledge out there for the community.
I liked that I came with exactly the similar idea of cutting the object twice and applying different shaders before the final ending. Its like the most converging simple solution.
Thank you! This is indeed very usefull, as are all your explanations. I really appreciate your stance on the importance of understanding the fundamentals. You and Casey have really helped me understand more about programming and computers. This is also a bit of a curse to be honest because I’m not in the gaming industry unfortunately (yet ;) ) - I took the path of least resistance and ended up in e-commerce and web-tech - and I’m realising how disturbingly bad performance is in applications in this business. I’ve been trying to raise awareness in my company at least about the importance of understanding the hardware and not just blindly follow OOP practices, even for “managed” languages like Java, but I get no traction at all so far. I mean sure, it might sound scarily complicated, but to me it’s very much more complicated to try to understand and follow the multitude of OOP patterns, practices and whatnot as they are based on a bunch of different peoples mental models and they don't always fit together or work well for cases outside the ones they were designed for. Anyway, thanks for all the eye-openers is what I’m getting at :)
I think it's good to use what OOP tools you find useful, but that in many cases convenient can be confused for useful. Managing lifetimes of stuff with destructors can be incredibly valuable, but doesn't necessarily need to be a default. Understanding how many of these tools impact your code long term is incredibly complicated, even for experienced programmers.
I always recommend "game programming patterns", can find it free online. It has tons of really useful patterns presented in a very intuitive way. Only learning book I've read cover to cover.
@@Bozemoto Well, sure OOP is a collection of mental models basically that you can use as tools to design your code. You have to learn these mental models and how and when to use them. OOP is difficult to master and there are many models within OOP you need to learn. Designing for the hardware is no less difficult, contains no less mental models you need to learn and is equally difficult to master. So I mean, why not choose the latter, as you also get the benefit of more performant applications.
@@rayboblio Could you give an example of a mental model so I know what you're referring to a bit better.
@@Bozemoto Sure. With mental model I mean the things you have to imagine when talking about something. We can start with the “lower level” ones like encapsulation, for instance. This is meant to prevent “outside” objects to mess with an objects internal state. Already here we have to deal with the mental model of “internal state” and what that means. Then we have things like singleton, factory, builder, observer, etc etc. Factory for instance is a tool, or mental model, to help you manage objects, which in them selves are mental models (what is an object and why do you need one? Because of encapsulation etc). So a lot of the OOP patterns, or mental models, are there to help me manage the other mental models. So I kind of see OOP as trying to solve the problem of programming and it just goes round and round. Instead, I should be using programming to solve my problem, which is manage the inventory in a warehouse, or simulate physics or whatever, and those problems are usually mostly about data and transforming the data somehow.
I think you see it in a wrong way. Usually you have to ask yourself what kind of problem you are trying to solve. Often performance does not matter. Or even memory leaks. If it does not bother anyone except yourself, it's probably fine to ignore it. Don't take my words too serious tho. It is important to have nice code, amazing performance, no bugs, leaks or whatsoever if this is the problem that needs to be solved. In games, or let's say engine work, this is important. In one time project that will never be touched again, probably not. A website that needs to look amazing and perform okayish, probably not as well.
So, back to it, what problem needs to be solved is the question that you should ask.
It is also fine to change your job if all those criteria matters to you
This is incredibly enlightening, but every time he said "you can't do this because your math will become inaccurate" I was left with SO MANY burning questions.
Jon Blow is always fun to watch
I can not thank you enough for these long standing video game series. If i had the time they would mo doubt spur my interest into development.
Jonathon Blow giving his best Quantum Electro Dynamics lessons. I'm lovin it!
It's amazing he take so much time to explain stuff to some randos on the internet, while he must be busy on this endless amount of projects.
It's good that this kind of content is made. Even though it's been done a couple times before. It's always nicer with a more modern video.
This subject is actually part of the content in one of my software engineering classes! Very well explained, a good supplement! Thank you!
47:40 I think its because a triangle is the simplest shape that can make any other shape (okay maybe not perfect circles if you limit how small thay can be, but you get my point).
You know what's funny? I see him talk, I see code that does math, and it all makes sense, but as soon as I try to learn from math people who just use the syntax of math formulas and math jargon, it feels like complete gibberish that's way over my head.
1) Feed the plane equation of the water into your shader. 2) In your vertex shader calculate the signed distance from plane and interpolate into your fragment shader (a vector dot product and scalar subtraction) 3) draw everything transparent intersecting the water discarding fragments above the plane using this signed distance, 4) draw water, 5) draw all the things intersecting the water discarding the fragments below the water (a simple depth test here could suffice if you write the water to the depth buffer but the complimentary discard would be guaranteed to avoid cracks (the complimentary fragment invariance you were concerned about)) This is quite trivial to code. A more fully general system might involve any transparent planes in a BSP sorting structure. There's the secondary issue of drawing the water inside the transparent objects which really should displace the water and not have it inside their volume. This can be fixed using the stencil buffer to rasterize the inside of the rock intersection with the water to the stencil buffer and reject water samples there, but I would suggest this is a secondary effect and doesn't really needing fixing unless you care about water that might be more opaque. If you insist on fixing this though, you must in the above sequence insert: 3.4) mask off all color writes 3.5) draw water to depth buffer only 3.6) on depth pass draw outside of all intersecting objects with stencil increment 3.7) on depth pass draw inside of all intersecting objects with stencil decrement 3.8) turn color writes back on 4.0) draw the water stencil testing to draw to only stencil == 0 on depth pass, 5) as above. This assumes your objects are manifold hulls and you clear stencil buffer each frame. But again masking the stuff INSIDE the objects is probably overkill, but you might take some pride in accomplishing this detail.
That triangle cheesecake that you haven´t eating make my day. Love from Argentina Jonathan!!!
so excited to see this released. love your games.
This is awesome, I am laughing and learning at the same time. Thank you so much for your time.
I have a question. Besides degenerate cases, will a triangle in 3D space project to a similar triangle in screen space? Thanks.
Yes. Think of it this way: For each vertex of the triangle, you can draw a line, from the eye point, that intersects that vertex. Every point further out on each of these lines (and closer too!) projects to the same point in screen space. Thus, any triangle made from *any* of the points on those 3 lines, will project to the same triangle in screen space.
I live in the film / VFX universe of things - and finally I can pinpoint what makes this so similar to the gaming universe: All we actually do is create tons of tiny, logical triangles to fill a screen. Thanks for the insight! Sub.
Anyone have a link to that Feynman lecture mentioned at 20:55?
prump If you haven’t found it by now it’s probably this one. ruclips.net/video/P9nPMFBhzsI/видео.html
Is there a continuation of this? I'd love to see this problem being tackled in the code.
Really enjoyed this. Even the parts I didn't understand (most of it)
Thanks, I knew about rendering but wasn't sure about the best technique that was used nowadays (including for storing the data), this helps a lot!
"the game reifies the cheesecake as a concept." Love it.
What's the deal with the light moving in weird paths at 20:00? is he talking about the way photons act as waves or is it something else?
So Basically, in the quantum world, photons will "take" every path to "their destination" even the crazy ones you saw him draw. All those paths arrive to your eye. What you see is a combination. The thing is, the weird paths have equal and opposite weird paths. This causes them to cancel out to 0. The only paths that don't have this are the straight paths.
That's my attempt at explaining this. I think The Science Asylum explains it better.
I have a few issues with this, mainly how rasterization is explained in terms of physical analogies (like you would raytracing). The earlier a learner understands that everything is completely fake and all is made by projecting points and going from there, the better, IMHO.
Very fun video. The solution proposed at the end is the cherry on top of the cheesecake.
Very interesting! Wrote my own 3d Engine in pygame thanks to this
1:17:00 Can someone give me a hint how he determines the coordinates of this point, given the 3 other points ? Or does he just mean, for any given point you can check if it is inside the triangle ? I am a bit confused.
EDIT: Maybe he means that the 3 points of the triangle allow him to know every point that fills the triangle and that is not possible for the quad ?
For every point, given the lengths of the perpendicular arrows Jon drew and the triangle point coordinates, you can calculate the coordinates for the point the arrows lead to.
Do you want the actual calculation? In reality, you definitely choose the points and get their coordinates in some other way. I think the main point is that you calculate the world coordinates of pixels you draw during rasterization and can use it in pixel/fragment shaders. Jon explains some of that during the rasterization part.
Indeed, this is not possible for quads unless you make great restrictions on the quads, most importantly to force them to lie in a single plane.
I really liked the shaoline temple analogy :D 1:48:42
Congratulations with 50k subs =) Amazing videos and streams.
Dude, I love this so much. I was wondering how the heck worldspace to screenspace transformations worked like 2 weeks ago. I also walked away knowing how to implement a triangle fill efficiently. This was perfect timing.
What API do you use for the sharpie?
2:12:0 those colors (0.9, 0.83, 0.8) wich is E6D4CC in hex, and that is a combination of white and red
This was wonderful. Thank-you Mr. Blow
You don't need trigonometry to get position on screen just algebra. It's the intersection of line of sight with the line of the screen. In the frame of reference where the screen is axis aligned, it's (offset / distance) * scale.
Humans ask questions also as a way to feel included and socialise.
I was watching a streamer and honestly there was random questions every 3 to 4 seconds.
Beginner question that do not matter in large scale. Maybe you should make a video to give decent answers for default values of those for the beginners so that beginners don't procrastinate on trying to find answer to question that doesn't matter in the long run and can actually start working on their project. The which graphics API is good question, the time they invest in learning the API matters and if they make wrong choice it will take too much time that could be spend on actually producing things. So time to Triangle would be most important metric to optimize for and available on multiple platforms would be beneficial so that learning it would be great.
I use Vulkan and I don't recommend that for anyone's first project.
I mean, y = mx + b is incredibly useful for functional representations in calculus, which then extrapolates to physics and all sorts of other fields.
So even though raytracing is more expensive, it might be easier to code because you know what object is first in line of sight and transparent objects just modify the colour the object behind it?
Not quite on that last bit, but otherwise yet.
Raytracing is easier and more accurate, looks more good. But its not fast.
Greate video. I'm not sure how I got here, but I definitely need to check more of your content.
Any good textbooks for computer graphics that teach you this stuff from zero and give you project ideas?
yeah I would love to know this too!
Why a quad (which is a 2D object) somehow makes multiple planes is not well explained. He muttered something briefly about what sounded like floating point rounding errors, so im going yo assume this is the reason, but im not 100% sure and it should have been made more explicit.
In 3D space, 2 points are guaranteed to be on a line, 3 points guaranteed to be on a plane, 4 points aren't guaranteed to be on anything.
With a triangle, if you move any of the points in their depth coordinate (further away or closer to the camera) it doesn't distort the shape's flatness, it just changes the angle of the plane the triangle is aligned with. With a quad, if you start moving points further away or closer, for example moving a corner forwards in space, there's no longer a flat plane that all the points sit on, the shape has to bend to accomodate. Any 3 points of that bent quad will be on a plane, but those planes aren't guaranteed be aligned with each other in any sensible way. So you can't reason about where in 3D space a particular co-ordinate within that quad will be without breaking the shape down.
I sometimes wonder how good/bad would be to have three Z values instead of one: Z, minZ and maxZ with a simple min-max search going on while rasterizing transparent objects. If there is no transparency minZ and maxZ would be just copy of Z and thus we can indicate the special case easily.
Then if the two values differ from Z, because there were transparent objects drawn, then we can know that there were already a raster / pixel drawn at this location with transparency BEFORE we are drawing at this location. We can also know if we are the closest to the camera as of current order or farthest or "in-between" and the shader would get this information alongside the current color. I know it can still generate artifacts, but no need for sorting this way and artifacts are mostly just off-colors when transparents are behind transparents in more than two layers and even then likely there is some quite normal approximation happening at least.
Understanding pinhole cameras encapsulates a lot of the confusing knowledge about how we see. I just wish Textbooks didnt just show a picture of the camera and say yep, that's what it looks like and therefore that *waves hands* is how it works.
I understand the point where all the light passes through (the "eye" in the drawings) and the "screen" as two parts of a virtual eye. Then the aspect ratio makes more intuitive sense.
So the drawing is a virtual eye's view that we're putting on the screen. Our real eyes can see it at any distance from the screen and it still works because all the math was done for the virtual eye's parameters.
Edit: it's amazing looking at poor explanations I wrote months ago. I see that it's nearly unintelligible, yet simultaneously I still see the meaning I thought I was conveying clearly
Well, not a surprise that he is looting the flatscreen TV, look at the wavelengths he is emitting.
I've been coding my first 3d-engine, it's made with Powerpoint VBA (don't ask, its a demoscene wild demo flex). and it is visualized by drawing triangle shapes using the shape-system of office programs. So in a way it doesn't draw pixels as such, only triangles, but after implementing all the occlusion culling and simple normal angle lighting systems it dawned on me that creating a proper Z-buffer using triangles will be most likely impossible without a proper per pixel z-buffer. As I've been sorting the triangles in screen by the middle point of the polygon there is considerable z-fighting on some 3d-models. But moving away from the shapes-system defeats the joke, I might need an old-school 3d-modeler to tell me how to model for such engines..
16:49 But light always goes in straight lines??
World: lights on fire, Jonathan Blow: here’s how to represent a cheesecake using triangles
As someone who knows nothing about programming, this is a really cool and helpful insight! Appreciate it!
You could also do something like Quake or DOOM where the geometry is actually sorted in a way that only the visible polygons are rendered in order. But you have to cut the polygons and sort them.
The thing for cancel the pixel is called discard in GLSL.
Gave me inspiration to finally scratch some small 3d redactor myself, vearyi gouujd!
I've always been interested in 3d graphics even though I couldn't even write hello world. This helps me understand a bit more. One day, I'd like to write a program that makes a 3D cube, that by itself would feel magical to me.
do you want to do it? just follow this tutorial of opengl on learnopengl.com/, you will learn how to draw elements, move around the world, etc.
Thanks for sharing! This is gold.
Late to comment, but thanks for taking the time to do this Jonathan.
you should remake stephen's sausage roll and call it Jonathan's cheesecake, and use a spatula instead of a fork.
As always, the video was a great way to fill out holes in my technical knowledge from a high level perspective that i can use as a technical game designer to make better decisions as a designer from understanding the technical requirements. Thanks
Jonathan, I admire your craft and commitment to game development! I also admire and follow Daniel Benmergui! You just explain with the 3D to 2D what I was asking and looking in books years ago (2002-2003), I make in that time in Quick Basic a 3D render (only "meshes" lines), but I use a way to get the 2D position completely diferent, It uses the screen size but not the camara distance to the window... Is more complicated, but I don't know if it had the problem of the object get through the virtual screen. Were I can find another ways to represent or obtain 3D position into 2D? there is a book or something that explore this possibilities? Thxs!!! I'm not a native speaker. I loved several of your talks about game design and farming of the player in some games!
GLSL documentation: "you can discard a fragment"
JB be like: KILL THE PIXEL
as you were giving your explanation of how rasterisation of 3d on to a 2d plane (screen) works (1:37:44), it struck me that if gpus were to actually take, and instead of stepping down the screen or filling a line like in your software renderer... imagine if all of the gpu knew about all of the geometry the scene would need to render, then per pixel, massively parallel, it did the similar triangle thing to find all the intersections with geometries on the ray cast from of camera pos, out through the FOV where the screen is, coming out toward the world. then, for each intersection with a triangle that ray cast intersects, the triangles involved would be rendered from front to back (like with the z-buffer) discarding everything further, cept the transparent ones go the slower path of having to go from back to front. this technique (I dunno if it's ever been suggested before), of casting from the FOV out toward the screen was interesting to me because it could simplify problems like the one you outlined, and potentially add the ability to render much more in parallel than happens now...
really enjoyed the discussion :)
If I understand your proposal, the problem I see is that you then have to iterate over every triangle for every pixel, rather than just iterating over every triangle once, and only iterating over the pixels that roughly cover that triangle.
@@wessmall7957 good point. I supposed that there would be some sort of way to partition the space so that ... yeah that would be somewhat complicated. however, I suppose with time hardware will have increased so much that it may be possible to do a full lookup for each pixel because there are like 2^20 parallel processing unis or something :)
Is this project only using z buffers for depth effects? Your project doesn't look like it requires advanced graphics solutios so it might be beneficial to use software tricks from the 90s and early 2000s so you don't have to check every single pixel. Then again a modern CPU & GPU has no trouble dealing with this
31:11 everything clicked at this moment, like i actual understand how world coords become screen coords now
does anyone have the link for the implementation of the fix?
Regarding the CPU v GPU analogy. The Bruce Lee quote "I fear not the man who has practiced 10,000 kicks once, but I fear the man who has practiced one kick 10,000 times." fits pretty well. CPU can do any kick imaginable and also backflips and other tricks, but it can do them e.g. ~8x2x5,000,000,000 times per second (minus fetch times). GPU can do only a couple different kicks, but it does them ~76x128x2,200,000,000 times a second. That is about 267:1 win for the GPU. The GPU cache latency might be higher than CPU but since the fetch sizes are much bigger and more uniform, it doesn't necessarily matter at all.
6:37 did he just make minecraft noises while drinking juice ?
2:45 Jonathan - I have a simple idea how to fix this.
JUST make that grid lines not above (on) the water, BUT below, on the floor (under water),
I mean- only in the water, the grid lines should be painted exceptionally on the bottom (floor) of the water.
Then it will be easer to quick define where crystal or stone lies (on which line and column)
Because now it is a little confusing... I have to FORCE my brain to see that lines are on the top of surface (water). It is not intuitive.
This is why in some angles, brain see that stone or crystal takes two places in the same time.
In fact on the first time my brain immediately assumed that lines are on the bottom (because it is logic to see that floor/bottom is painted, instead of believe that lines are floating magically on the top of water)
Greetings from Poland.
BTW:
I love Witness and Braid. Thank you so much for these awesome games, and your hard work.
But when the lines are on top of solid, floor tiles, should those lines be in different heights? I mean, you solutin sure is a practical one for the graphics engine prespective, but from the UX, I believe you just changed one problem for another.
@@riibbert change of height only for water area. Solid floor should be untouched.
solid floor has lines on the bottom on ground. You can walk on it and the lines are under your feet.
but on the water block you cannot walk on it, but you fall down by one block down, so the lines also should "fall" down and be on bottom of water block.
otherwise it will be hard for me to force my brain all the time to figure out on which line exactly things lie. this calculation takes more time, that my proposition which gives more intuitive look.
Great content and camera upgrade.
Does anyone know which feynaman video is he talking about?
can we talk about how good those rgb guesses were for cream and brown? I know he's done this a long time and has done a lot of rgb tuning most likely and that's not the point of what he's saying but oh my god those were good. Perfect cheesecake and beer bottle
The model used in most 3d engines is not exactly correct. You can see why if you look at some long horizontal thing in front of you, for instance your monitor. In your field of view, the central part of the monitor is bigger, because it's closer to the eye. The corners of the monitor are farther from the eye, so it's not exactly a rectangle in your eye. It's subtle and hard to notice, because the monitor takes relatively small angular area in your view field. But you can notice the implications of it when in the 3d scener something gets really close to the camera, like faces or gun muzzles. You would see it very differently if these objects got close to your eye
What are the stylistic freedom of ray tracing, comparing to traditional polygomic + shaders? Haven't dive into it yet but all examples I saw have the exact same look
I don't quite understand what "stylistic freedom" means, but ray tracing makes sense because it adheres more to the nature of our real universe and how light works in it than conventional rasterizers do.
It's not mature yet, but as it always happens once it's more accessible you're gonna see a lot of creative ways of using the technology that weren't "intended".
@@Radgerayden-ist thanks for the reply i think i saw you a lot on love2d discord lol
@@slmjkdbtl I'm there yes, happy to find a fellow lover in the wild :)
1:13:00 "THe reason we use triangles is because 3 points determine a plane"
graph paper ❤
I tell all my students to get some and use it for note taking. game changer