Holy SHIT dude. You underplayed that debug massively, I imagine you were losing your mind at moments. Thank you for this devlog, I don't think I've ever seen one run through a debug process so well. It's really highlighted to me the importance of testing on as much hardware as possible, what a wild bug
Yea I was sweating for a bit there 😅 Ultimately I really don't think I would have been able to resolve it without having access to an AMD chip myself, which is scary when you eventually have to support tons of hardware you don't actually own
I literally experienced this Camera bug yesterday and spend the whole day looking for any information on the forums, documentations, but could not find anything! As I was just going crazy trying to understand why I get my FPS halved with every camera I have in the scene I stumbled randomly upon your video. While this bug continues to exist I'm glad I found that i'm not tripping and it's actually a bug and not a mistake on my end. Subscribed and will keep an eye on your videos. Great work.
Thanks for the video, it is really interesting to watch! As for crash, it seems that the shader has some faulty logic that calls an invalid instruction on the GPU or blocks further execution. I had the same problem when I accidentally went outside of the StructuredBuffer size in my shader (and, to note, not exactly one item over - it caused the fault after hundreds of non-existing collection items - similar to how this happens in C on CPU) because half data type was treated as float by the compiler, thus breaking everything. The error message actually does not specify that the GPU was physically disconnected. What actually happens, your GPU is being reset as it got stuck or received an error while executing your shader code. This is quite simple to reproduce: try creating a shader with an infinite for loop, it should give you the same result. However, in your case it seems that the problem is reproducible only on the AMD architecture which is not suprising as the execution pipepile of your shader code differs between NVIDIA and AMD. The code you've shown could be investigated even further as it has functions called "GetGrayscaleDistanceFactor" and "Grayscale" - they also might be the cause. Eventually, from the code snippet you've provided it bothers me whether the zero value for normal and viewDir might cause this problem as they should never be zero length vectors.
Congratulations for fixing your bug. I just want you to know that I, as part of your community of subscribers, appreciate your persistence, creativity, dedication, and generosity in sharing your experience that inspires people.
I recommend using the profiler for future problems like this. You can determine what the problem is faster bc u have updated information on your hardware the moment the crash happens.
This has to be the weirdest bug I've ever seen, it's so random and unexplainable. Why only on AMD specifically? Why in that particular place? And PHYSICALLY REMOVING THE GPU? HOW? Biggest unsolved mystery ever lmao
That camera fix is ingenious. Just to ensure I'm understanding correctly, what ur doing is similar to moving the camera to above the player getting the texture then moving it back to the player except the camera isn't moving at all just it's viewport.
Sort of, instead of moving the main camera I still keep the overhead camera around (but disabled) and send it's projection matrix to the shader after the main camera sets its projection matrix. Basically it's like // Unity Code SetProjectionMatrix(mainCamera) // My Code, after Unity and right before drawing SetProjectionMatrix(disabledOverheadCamera)
There had to be some issue in that calculation that crashed the game. Most likely comming from the material in view (on lighthouse for example). Maybe IN.positionWS was missing/broken for some reason, maybe _WorldSpaceCameraPos had some issue, etc. Anyways, if you've checked the code and didn't find nothing, then check that spot where it's happening and check the asset involved.
I've gone through the frame debugger and checked every variable, and I'm truly at a loss. The only thing I can think of is that its a red herring and the real issue is still lurking somewhere else, but testing was pretty conclusive that removing that one line was the only change that would resolve the crash.
@@kylebanks Yeah, I absolutely understand your fix. It's just that the code is so simple at that place that it has to be a data error. Or, that code is part of something bigger and this calculation has pushed it over the edge memory-wise or something like that. Anyways, if you'll ever revisit and find the actual issue, please mention it in your devlog. I'm bothered by not knowing :D
@@GonziHere If this is a data error, then why is the issue only reproducible on AMD GPUs? And why is the error code so strange (and is present in other games)? I'm also very interested to know the root cause
Been quite a while since I checked in on this game, cool to see how far it's come. It looks WAY more polished (as expected) and like something I could see gaining a pretty substantial following.
Wouldn't necessarily recommend it unless you're willing to tinker and explore the available Unity source code to fix things yourself, but overall it actually hasn't been too bad at all
@@kylebanks I'm working on a DOTS prototype, updating to their laster (pre-65) version broke every single component Baker because they decided to change up the syntax, fixing 30+ files just to add one extra line wasn't fun. There was also no documentation update so I would need to browse the forum for answer which wasn't fun. I also love tinkering with pre release stuffs but damn this is stressful
@@saito853 I have been using multi-threading with unity, I wouldn't want to use dots because it seems unstable. Multi-threading is very delicate, definitely wouldn't trust pre release stuff.
3 on-site, 2 off-site. Backups are too important not to utilize them. The title of this video scared me, I'm truly looking forward to games full release.
@@kylebanksDo whatever works for you, I guess. I learned years ago when I first got into IT work to never trust a singular backup or hard drive. Since the rise and ransomware sometime ago always off site back up as a spare with disconnected on site backups. It only takes losing everything once.
Okay, I think this devlog stressed me out more than it should have :D Good that you're some kind of shader wizard and were able to find and fix the bug! I probably would have gone crazy in that situation... x) Was your AMD GPU an OnBoard chip?
Ha sorry to stress you out man! Yea it's a low power integrated chip, I guess some gaming laptops have them because they consume much less battery than the gaming chip for trivial GPU tasks
Did you try further mess with the values within the broken function, before removing it, to see if any different or even basic values still crash it? So like, getting the very exact line and variable that does it? That could help further understand the issue. You could also try isolate the problem to its own file with everything entirely stripped out of it that does not make it crash.
The "device removed" error is badly worded, because it is usually a GPU driver crash. But the OS sees it as the device being removed because the driver is no longer there.
About the lighthouse-crash bug, did you check texture streaming pool size? I know that's how my games can slow down substantially. I can't say that is the issue (I've never faced an issue like this) but that could be a possibility.
I didn't, but given that removing the one line (unrelated to any textures) resolved it, and the total memory footprint remains the same, I don't think that's it
I have no idea why that crash happens, but since you can reproduce it so reliably and you have access to the source code I wonder if it would be worth reporting to AMD so they could look into it, and hopefully fix it if it's a driver/firmware issue.
Maybe, but the issue was caused by a few simple arithmetic operations while disabling hundreds or even thousands of other operations didn't solve anything
@@kylebanks yep to be fair to AMD they are working on making their drivers compatible for tons of other applications plus the way they operate can be very different from nvidia cards I would recommend asking someone at unity if they have come across any similar issues
How can you bear all the time to run Unity and start working on the game I don't have all that patience even with a small project Any tips if you have to rev the engine to start
Some days it's really not fun or enjoyable, but I think you need to build some discipline and just do it. Usually once I get going I find it enjoyable again and get into a rhythm, the hardest part is just getting started
Pure stubbornness and deadline stress 🙃 More seriously, some days it is super demotivating. But discipline and learning some tricks for debugging issues where you don't know what's going on can be a big help. That "binary search" technique can be a big help when you have a hard crash that can't rely on logs/debuggers/etc.
Unity most incompatible with Intel Graphic Minor compatible issues with AMD Fully Compatible with Nvidia and developers of unity never really cared about it it is a engine side problem bro you gonna be fine with other engines also lately flax engine had problems with intel gpu they fixed it in new version when users asked them they are small team but Unity big company doesn't care!!!! I remember lots of crashes from Intel users on game i made with unity also i had a decent system that time still unity even was crashing on my own system!
Thanks for watching
I wish I had even half your patience. Seriously incredible work.
Haha thank you. I think it's just stubbornness more than anything
What about my patience? 9 year of developing one game full time alone… 😅
bro's debugging has O(log n) time complexity
4:33
That's the most demented bug I've seen in a while
I don't disagree :/
Still no idea what the problem was
Holy SHIT dude.
You underplayed that debug massively, I imagine you were losing your mind at moments. Thank you for this devlog, I don't think I've ever seen one run through a debug process so well. It's really highlighted to me the importance of testing on as much hardware as possible, what a wild bug
Yea I was sweating for a bit there 😅
Ultimately I really don't think I would have been able to resolve it without having access to an AMD chip myself, which is scary when you eventually have to support tons of hardware you don't actually own
This looks like an amazing game! Can't wait to see the demo at PAX East!
Thanks! I'm gutted I can't be there personally but you can find Farewell North at the Mooneye booth :)
Really love your devlog style. Just shows all the troubles you get (I guess not all) to make things work.
Thank you, really appreciate it :)
I literally experienced this Camera bug yesterday and spend the whole day looking for any information on the forums, documentations, but could not find anything! As I was just going crazy trying to understand why I get my FPS halved with every camera I have in the scene I stumbled randomly upon your video. While this bug continues to exist I'm glad I found that i'm not tripping and it's actually a bug and not a mistake on my end. Subscribed and will keep an eye on your videos. Great work.
No way! Ha what are the odds... Well, welcome to the channel 😊
Thanks for the video, it is really interesting to watch!
As for crash, it seems that the shader has some faulty logic that calls an invalid instruction on the GPU or blocks further execution. I had the same problem when I accidentally went outside of the StructuredBuffer size in my shader (and, to note, not exactly one item over - it caused the fault after hundreds of non-existing collection items - similar to how this happens in C on CPU) because half data type was treated as float by the compiler, thus breaking everything. The error message actually does not specify that the GPU was physically disconnected. What actually happens, your GPU is being reset as it got stuck or received an error while executing your shader code. This is quite simple to reproduce: try creating a shader with an infinite for loop, it should give you the same result. However, in your case it seems that the problem is reproducible only on the AMD architecture which is not suprising as the execution pipepile of your shader code differs between NVIDIA and AMD. The code you've shown could be investigated even further as it has functions called "GetGrayscaleDistanceFactor" and "Grayscale" - they also might be the cause. Eventually, from the code snippet you've provided it bothers me whether the zero value for normal and viewDir might cause this problem as they should never be zero length vectors.
Congratulations for fixing your bug. I just want you to know that I, as part of your community of subscribers, appreciate your persistence, creativity, dedication, and generosity in sharing your experience that inspires people.
Thanks so much, the support really means a lot
Excellent devlog! I was really with you in the bug hunting :D Keep it up ❤💥
Thanks! Glad you enjoyed
Props to you for getting this demo up man. Nothing scares me more than hearing my game has crashed. Trying to fix those issues is a NIGHTMARE
Thanks man, definitely not the fun part of making games 😅
An epic journey you are on dude. I remember watching the game come together in it's early days - it's come a long way! Almost there man!
Thanks man
Really interesting video. Love these kind of videos with weird problems getting solved. Hope PAX goes well!
Thank you!
really loving this dev loging style.. informative as always
Glad you like them! ☺️
Thank you for updating the title. Now I don't feel click-baited, and feel like I can click it.
I never changed the title 🤔
Looks great! Can't wait to play the new demo!
Hope you enjoy it!
I recommend using the profiler for future problems like this. You can determine what the problem is faster bc u have updated information on your hardware the moment the crash happens.
You can't profile a hard crash, the profiler goes down with unity before debug information is available
This has to be the weirdest bug I've ever seen, it's so random and unexplainable.
Why only on AMD specifically? Why in that particular place? And PHYSICALLY REMOVING THE GPU? HOW?
Biggest unsolved mystery ever lmao
Great work which not easy to debug stuff. At least you got it done and clean up some codes.
Always nice to tidy things up ☺️
Beautiful looking game dude! Amazing work.
Thanks a ton!
very excited for the full release
thank you :)
That camera fix is ingenious. Just to ensure I'm understanding correctly, what ur doing is similar to moving the camera to above the player getting the texture then moving it back to the player except the camera isn't moving at all just it's viewport.
Sort of, instead of moving the main camera I still keep the overhead camera around (but disabled) and send it's projection matrix to the shader after the main camera sets its projection matrix. Basically it's like
// Unity Code
SetProjectionMatrix(mainCamera)
// My Code, after Unity and right before drawing
SetProjectionMatrix(disabledOverheadCamera)
Really entertaining video. GReat work on the demo!
Thanks Valentin!
What if you added a pile of raked autumn leaves you could jump in? Like the stick throwing mini game.
Love it
Happy to see you managed to get the version running for PAX! ☺
Thanks so much Darenn, loving the Leuer updates!
@@kylebanks Oh thanks!
Good job! Looking for such a bug must be hard... Shaders are "scarecrow" for me... It's like another level of programming :D
Haha yea they can be pretty mysterious at times for sure
Well done! There's a sweetness to engine bugs making you find more efficient solutions in the end. 🤓
There really is :)
There had to be some issue in that calculation that crashed the game. Most likely comming from the material in view (on lighthouse for example). Maybe IN.positionWS was missing/broken for some reason, maybe _WorldSpaceCameraPos had some issue, etc.
Anyways, if you've checked the code and didn't find nothing, then check that spot where it's happening and check the asset involved.
I've gone through the frame debugger and checked every variable, and I'm truly at a loss. The only thing I can think of is that its a red herring and the real issue is still lurking somewhere else, but testing was pretty conclusive that removing that one line was the only change that would resolve the crash.
@@kylebanks Yeah, I absolutely understand your fix. It's just that the code is so simple at that place that it has to be a data error.
Or, that code is part of something bigger and this calculation has pushed it over the edge memory-wise or something like that.
Anyways, if you'll ever revisit and find the actual issue, please mention it in your devlog. I'm bothered by not knowing :D
@@GonziHere If this is a data error, then why is the issue only reproducible on AMD GPUs? And why is the error code so strange (and is present in other games)? I'm also very interested to know the root cause
Ur game is looking AMAZING 💪
Thank you ☺️
Awesome video! Excited to try this game at Pax East!
Hope you enjoy!
Been quite a while since I checked in on this game, cool to see how far it's come. It looks WAY more polished (as expected) and like something I could see gaining a pretty substantial following.
Thanks, welcome back :)
This man need a medal (and probably some therapy sessions) for using the bi-weekly Unity beta releases on a production-ready game.
Wouldn't necessarily recommend it unless you're willing to tinker and explore the available Unity source code to fix things yourself, but overall it actually hasn't been too bad at all
@@kylebanks I'm working on a DOTS prototype, updating to their laster (pre-65) version broke every single component Baker because they decided to change up the syntax, fixing 30+ files just to add one extra line wasn't fun. There was also no documentation update so I would need to browse the forum for answer which wasn't fun.
I also love tinkering with pre release stuffs but damn this is stressful
@@saito853 I have been using multi-threading with unity, I wouldn't want to use dots because it seems unstable. Multi-threading is very delicate, definitely wouldn't trust pre release stuff.
3 on-site, 2 off-site. Backups are too important not to utilize them. The title of this video scared me, I'm truly looking forward to games full release.
Oh don't worry about that, this game's got version control for days
@@kylebanksDo whatever works for you, I guess. I learned years ago when I first got into IT work to never trust a singular backup or hard drive. Since the rise and ransomware sometime ago always off site back up as a spare with disconnected on site backups. It only takes losing everything once.
i bless you with my thanks and graditude
Okay, I think this devlog stressed me out more than it should have :D
Good that you're some kind of shader wizard and were able to find and fix the bug! I probably would have gone crazy in that situation... x)
Was your AMD GPU an OnBoard chip?
Ha sorry to stress you out man! Yea it's a low power integrated chip, I guess some gaming laptops have them because they consume much less battery than the gaming chip for trivial GPU tasks
Did you try further mess with the values within the broken function, before removing it, to see if any different or even basic values still crash it? So like, getting the very exact line and variable that does it? That could help further understand the issue.
You could also try isolate the problem to its own file with everything entirely stripped out of it that does not make it crash.
The "device removed" error is badly worded, because it is usually a GPU driver crash. But the OS sees it as the device being removed because the driver is no longer there.
Yea that's kind of what I figured, and the error that shows up inside unity is much better worded, but it was pretty confusing for a minute there
About the lighthouse-crash bug, did you check texture streaming pool size? I know that's how my games can slow down substantially. I can't say that is the issue (I've never faced an issue like this)
but that could be a possibility.
I didn't, but given that removing the one line (unrelated to any textures) resolved it, and the total memory footprint remains the same, I don't think that's it
omg it looks so bad from the editor but so good when you actually play (that’s how you know you are a good developer)
The cursed lighthouse
Not for nothing my dude but YOU are amazing, thank you very much. Simplest thing I've ever downloaded I think
Thanks so much :)
Keep up the good work been here since ep1
Thanks for all the support
What did you do to improve your textures and shadows?
New devlog!
hype!
Binary search by turning code is my favourite debug tool too!!! 😅
Never fails 👌
I have no idea why that crash happens, but since you can reproduce it so reliably and you have access to the source code I wonder if it would be worth reporting to AMD so they could look into it, and hopefully fix it if it's a driver/firmware issue.
At least you did not have a crash during a big live stream! 😅
The thought of that stresses me out big time
@@kylebanks I survived, so would you 💪
It works, earned a sub
Thanks!
Packaging errors are always the worst.
They really are
looking great
Thanks!
a the device lost error the nemesis of all renderer devs
Its a driver bug that effects AMD drivers crashing
when they get certain load in dirx has been prevalent for ages
Maybe, but the issue was caused by a few simple arithmetic operations while disabling hundreds or even thousands of other operations didn't solve anything
@@kylebanks yep to be fair to AMD they are working on making their drivers compatible for tons of other applications
plus the way they operate can be very different from nvidia cards
I would recommend asking someone at unity if they have come across any similar issues
Yep when some Unity bug like that happens the best way is just to bypass Unity code and fix it your own. (It’s always faster from my experience XD)
I noticed you have multiple scenes loaded. are they additively added and active or what is their status?
They're additive. Check my last devlog, it's all about how I setup world streaming in Unity ☺️
Will there be a version for Linux/Steam Deck?
I'm assuming it hit a strange edge case with that specific geometry
Must have, yea, I just can't figure out what it could have been
I would have torn my hair out at that crash. Great work fixing it!
How can you bear all the time to run Unity and start working on the game
I don't have all that patience even with a small project
Any tips if you have to rev the engine to start
Some days it's really not fun or enjoyable, but I think you need to build some discipline and just do it. Usually once I get going I find it enjoyable again and get into a rhythm, the hardest part is just getting started
"Amd doesn't have driver issues".
how do you not get demotivated from big game crashes like this
Pure stubbornness and deadline stress 🙃
More seriously, some days it is super demotivating. But discipline and learning some tricks for debugging issues where you don't know what's going on can be a big help. That "binary search" technique can be a big help when you have a hard crash that can't rely on logs/debuggers/etc.
That's why I'm afraid starting making games xD
There's a lot to games but if you just take it one thing at a time it's much less daunting 😊
Awesome
thanks :)
HEYY DUDEEE I am one of your fans. Can we make some 2d animations with ur game? I am interested to work with u
Wow amazing, whenever I get a bug that is because of unity i just pack my bags and give up
Haha that's the spirit 👌
Don’t do pre-releases, kids
switch to the engine given by the heaven themselves, godot, before another crash
nice
Unity most incompatible with Intel Graphic
Minor compatible issues with AMD
Fully Compatible with Nvidia
and developers of unity never really cared about it
it is a engine side problem bro you gonna be fine with other engines
also lately flax engine had problems with intel gpu they fixed it in new version when users asked them they are small team but Unity big company doesn't care!!!!
I remember lots of crashes from Intel users on game i made with unity also i had a decent system that time still unity even was crashing on my own system!
(ﾟοﾟ人))