I have been saying for a long time now that having 1000+ players on a server is not just a matter of the meshing/networking/server performance - the existing infrastructure/services IN GAME cannot support that much traffic at all - the wait times to use hangars/terminals/etc will be insane as you will end up at the end of very large lines constantly. stations and cities and such will need to be WAAAAY BIGGER to accommodate that population.
Don't forget that there won't *usually* be the huge spikes in population doing the exact same thing at the same time in the same few locations. i.e. Most people will not all be all logging on together as the server comes up. The trick will be to get the player population evenly distributed over a bunch of different shards (star systems).
@@stormwolf3255 There will definitely be occasions where this does happen. An org logging in for a scheduled event, people logging back in after a server update or crash, people gathering in one area in anticipation for a schedule official event like IAE. There are a lot of cases where a sudden spike can and will naturally happen.
Their focus on elevators was always hilarious to me, there is reason that ZERO train station, airports or any similar place has elevators as the main way to travel to something. Even the super deep soviet subway stations don't have elevators as the main way to get down there, even though it would be faster than the escalators, it just isn't feasible to transport the amount of people necessary.
@@generalbandege1184 That's why you have subway and stations, and then mixed with normal pavements for walking, also as alternatives so you always have 2 means of reaching any place.
The soviets didn't need to spawn hangars on top of eachother in instanced parallel universes so elevators didn't solve a necessary problem like they do in star citizen.
Elevators have long been a loading trick for games (you get in the elevator, assets load in and out). I know SC has some robust asset streaming, but I would be surprised if there wasn't still some good old-fashioned elevator loading going on.
I really gotta question the guy's mental health. He spent the time downloading the PTU version of the fame. What is it, 150 GB? Then he spent 5 hours in a line. Then he decides to blow the whole test and run off in gleeful euphoria? A nutter, for sure.
@@fwdcnorac8574 average gamer more like. Self destruct was the ONLY way to end that test anyway, gotta test how the game handles that many people being around an effects heavy explosion.
@fwdcnorac8574 The troll may not have spent 5 hours in line. Morph said he spent 5 hours trying to set this up. That guy could have been hanging out for an hour or less and jumped in on the action. No doubt he was laughing super hard at all the bodies floating, but that ended the test so it's unfortunate since they finally got on the ship.
@@fwdcnorac8574 In Elite Dangerous a troll spent around a week doing hundreds of tedious hyperspace jumps, just so he could ambush an expedition of lighly armored and unarmed explorer ships at the center of the galaxy (must have taken a minute tops to blow them all up). Never underestimate the drive of people with mental health issues.
A 1000 player server is a Good Problem to have. If its a problem you can go play that other game with a 1000 players.... Oh wait... There is no other game with a 1000 players.
@@sandman1347 yes, this should be the alarming part. For the past year people were saying "server meshing is all we need!" And this shows how much more work is needed beyond it. Even the most positive estimates would be at least another 5 years before that many players can actually play long term.
It makes me wonder how many game systems will need to be re-worked AGAIN once they get server meshing added to the game. We have years of work built around a 100 player cap, now we're adding 10x that amount of people.
@@SA80TAGE not really, there was none of this that existed before and when they started there was no game engine that was any better, cry engine was peak at the time and still had a ton of issues, the problem is its not easy to work in and most of the old manuals etc are gone, they relied upon some of the original engineers but its a lot of work rebuilding everything. Currently there is no game engine that can do what they're doing, unreal engine, unit, godot, NMS own engine their devs created also wouldn't work. In reality the scope broke things far more, originally it wasn't anything overly interesting then changed to sort of like starfield is currently and now we are here..
Let’s estimate how much CIG might pay in salaries per year based on the $700 million over 12 years. First, calculate the average yearly funding: Average yearly funding= 700,000,000 ÷ 12 = 58,333,333.33 per year So, CIG receives roughly $58.3 million per year. Now, considering the rough salary ranges and estimating how many employees they might have, we'll attempt a rough calculation. Assume: CIG employs around 500 people (a reasonable estimate based on reports). If the average salary per employee is around $100,000 (to account for a mix of junior to senior positions). Total salary cost=500×100,000=50,000,000 Therefore, CIG could be paying approximately $50 million per year in salaries, leaving around $8.3 million for other operational costs like software, development, marketing, and server upkeep. This is a rough estimate based on the information provided.
The fact that so many gameplay systems and fundamentals were designed without SM, as it didn’t even exist, presents a problem. We can all tell that this isn’t going to be a simple matter of changing a few values to magically allow the current ‘game’ to work with hundreds, if not thousands, of people. I foresee many more years of reworking the core systems and gameplay to make this be a smooth experience
I argue about that point. The core systems are already in the rework for SM since a few years, and that's why we can see result today because of those reworks. What needs rework now are some features/gameplay that sit on top of it (Missions, Socials, Mass transit, Dynamic economy, etc). There is work, for sure, but many more years seems very very very pessimistic. 4.0 will be the first iteration on SM, and it's planned to be release later this year (but we all know it will be a bit later aha). So a few months of work left at least for static SM support and all the gameplay? Though I do agree that improvements will be necessary on the path to SC 1.0
I was arguing about this on discord - for hangars / habs / elevators it would be enough to have it "instanced" in case of "higher than expected load". Nobody wants to sit 10 minutes in hangar, waiting for take off permission / wait 10 minutes in lobby for a free hangar.
@@BeSk9991 Yes, at the same time I could understand some reasonable wait - 5min tops - however, I would need to know what is happening. If there is a queue they need to show me or tell me somehow that I'm X in line for take-off or whatever. Otherwise, with current systems I'm sitting there not sure if there is a bug or a queue or maybe the server is crashing. This is needed because they won't be making 1000 hangar doors around every city just in case 1000 people show up at the same time. Some sort of queue system is needed and it needs to be visible for the player. This would work exactly the same as traffic lights with a counter till next change. They are proven to relax drivers and actually make the traffic smoother. We need that!
That's what exhausts me the most about this game: the constant "reworking" or better saying, doing things the way they should've thought before doing anything. How many reworks of the actual systems that are actually beign developed right now, will have in the future, to accomodate the brainfarts CR will have?, because oh now we have 200milion more dollars to spend so we're going to rework everything because, idk, not releasing the game ever, to keep selling ships at astronomical prices?
Do you ever consider making a throwback video if you ever feel burnt out? It feels like a lifetime ago I was watching your space engineer tutorial videos on how to make the skyke MKIII.
About performance, if the problem was just the amount of geometry because of all the player models, the GPU would be the bottleneck and at 100% load, not the CPU. The fact that it's the CPU means that either something goes wrong during preprocessing, or the data transfer from CPU to GPU isn't fast enough. And because you were maxed out on all cores, not just one, it has to be preprocessing. You are using the DX11 renderer, which can only ever use a single thread for CPU to GPU data transfer. So if that was the problem, only a single CPU core would be stuck at 100% and bottleneck everything. Now, I'm getting into speculation territory here, but there are really only two preprocessing steps that both take significant resources and are exclusive to character models - physics simulation for things like cloth and hair, and "skinning" - the process of calculating what a character should look like given a standard t-posing model, and a pose. The good news is that, if my theory is correct, this is definitely solvable. Skinning can be done on the GPU, which is much faster than doing it on CPU, and the physics sim should be possible to disable for things that matter too much. You don't really see whether the hair of a dude 100 meters away is waving correctly. But the final possibility would be that whatever uses all those CPU cycles is not related to rendering, but the renderer ends up waiting around for it because it is not properly separated. This is a very frequent problem in older game engines, and if it's also the case here, CIG would need to make major modifications to the engine to make it better. So I really hope that's not what this is.
What I figured out reading SC comments over time is that there is a lot of people who like argue about "server meshing" and unable to formulate what "server" is at the same time ))) Do you know exactly what client CPU do in SC? I bet you have no even slightest idea ) Concider there is like ~120000-150000 "entities" shown in r_displayinfo. From what we know "entity" is an object which have a mesh, some physical properties, some gameplay properties and it's state constantly synchronised with server. Entity is a players's body, helmet, chest armour, every clip on that armour, his backpack and it's content. All these "entities" they are not magicaly moving "with player", they position is calculated and synchronised with server. Imagine synchronising 150000 objects like this with server every server frame, this is kind of a lot non-GPU work.
@@alexpetrov8871 ... I doubt you figured out much correctly because your comment does not indicate that you have properly read what @CdrSonan wrote. He is not talking about server meshing or the speed at which the server runs. His point is about the slowdown of rendered video frames on the client. What the client CPU cores are doing is entirely irrelevant. Why? Because they should not be doing what they are doing now because they are bottlenecking the whole system (as shown in the video, just watch it again and again and again). The code should be changed and either make CPU cores do less, or off-load more to the GPU. Just read his CdrSonan's command a few times over with your local grey matter setting turned on. Eventually you will see the light, it's not that hard to comprehend. And of course rendering code should be entirely separate from networking code. Not doing so is asking for all kinds of trouble and then some.
@@alexpetrov8871 Don't jump to conclusions. I've made games in both Unreal Engine and Unity, including a complete custom rendering pipeline in Unity once. So also that isn't CryEngine or StarEngine, I think I have an idea of what I'm talking about. Directly tying rendering to network updates would be insanely stupid, because it would instantly increase your display latency by your ping, and losing even a single packet would cause a freeze for several seconds until the timeout is reached. That's why, on modern games, the renderer runs on a separate thread from the game logic, and networking again runs on a separate thread from that. This way, the main thread can continue to update things like physics or animations even if the network thread is waiting around for some packet to arrive, and likewise, the renderer can continue to push out frames even when the main thread is running at a lower tick rate. When this separation works, you will still get high FPS with excessive CPU load, and instead it is only going to cause things like physics glitches or animation stuttering. The problem is that games used to be designed for single, or few-core processors, where it was better to tie game ticks to FPS and save on the overhead of using several threads. This is especially the case for console games, and the reason many of them still have a 60FPS cap. Pushing more FPS would also increase the tick rate of the main thread and throw everything out of whack. Anyway, in a lot of game engines, the separated pipeline was more or less "tacked on", so the separation isn't perfect. (looking at you, Unity...) Only networking has always been put on a separate thread because otherwise things fall apart pretty much immediately.
@@CdrSonan >I've made games in both Unreal Engine and Unity, including a complete custom rendering pipeline in Unity once. Didn't it came to you then that the problem of high CPU load is not with rendering at all?
Suggestion for how to organize getting 200+ people in your 890; have groups of 10-20 meet at different landmarks on a moon and pick them up group by group. This way the trash wont build up too fast in an area, and perhaps people will log in in various places then fly to you. Also allows you to monitor the incremental build up of server lag on you as player count increases
This was a fantastic video, and excellently put together. I imagine CIG's QA director probably loved this feedback because it hits on a lot of really good points and has actual constructive criticism about where the design needs to go to support CIG's next milestone of 1,000 players per shard (which is quite the leap from the 150 they set about two years ago). In any case, this was very well put together, has actual proper concerns that the design and networking team can look into, and actually provides workable feedback that should help move the game in the right direction. Well done.
I disagree, I don’t think it was good. 1000 isn’t necessity their target, it’s just a number they know will break the current tech. Even if it were that, it’s unlikely you’d have that many people in one place, if there is two systems
LOL, seeing all those bodies floating away from the 890 made my loot goblin tendencies kick in. I'm glad the testing is showing progress. Pyro will be crazy next year. Thanks for the video.
It was a super dick move to blow up the ship, but I bet the data from having all those players on different clients dying at the same time was invaluable.
They said during the tests they are still rewriting ATC, Transit, and Mission Broker services to work with the new messaging system and why the 1k tests degrades from overloading those services. Even on 1k player at nearly 900 players movement tracking and doors were still good with active NPCs. Once enough people hammered ATC and the transit services ran long enough server FPS stayed at 5 sim FPS and barely responsive doors/seats the rest of the test until they killed it. The fact they tried 1k again means they think they might be able to hit that eventually once all the other loose ends are tied up. We had people in Mantis/Antares stopping people from QTing away from cities or crashing into ship gatherings by the commons and attacking people by stations. People were getting stream sniped too. The 3:500 and 4:500 tests worked the best if not better than live. The 6:1000 started solid but degraded fast and they said to expect it, but they needed the server message queue data loaded exercised even though they knew other things would fail. Things went so well they decided to leave the 4:500 mesh running to see how it held up over time to see if there were memory leaks or other slow burner failures that take time to cascade. Flying in QT between planets we didn't get knocked out of QT at every server border like the last test. Just a 'left channel local' and frame skips. It was encouraging but I have to wonder what value is getting 1000 player meshes because no GPU in existence can real-time render that many high poly models let alone light them with something as process intense as raytracing and global illumination eventually. They have some serious culling and optimization work to do and probably should have leaned on depth and bump map shaders more instead of putting geometry into every little crack and panel line of their models. It makes nice closeups and renders but most of the time you aren't going to notice it. The 890j is a few thousand entities alone and each entity can be thousands to hundreds of thousands of polygons. When I did FPS game stuff making player models 3-5k polygons total was the ceiling in the Quake/Doom/Unreal Tournament engine days and now 10-15k is considered getting too heavy. I don't think there is any single object model under 10-20k poly in SC lol.
Looks to me like we want server meshing with 50-100 player caps for Pyro launch for now, they can figure out the much higher player counts later, it’s not the priority.
Surprised you didn't mention the queues to leave hangars. Because there's only a handful of hangar doors, and people don't always leave immediately, landing zones get easily congested. I spent like 30 minutes just trying to leave Area18 and there weren't even many people (because they all went to NB lol). Each spot in the queue was taking 5+ minutes, meaning people were taking on average 5 minutes to leave their hangar after they got clearance.....which is wild. And you could definitely see the potential for abuse there, too. It'd only take like 9 people to completely lock down a particular hangar size at a space port.
@@QuotidianStupidity There's 4 Landing Zones. If SC hits 1,000 players (hopefully they're going for more per system), that's still huge backup potential, even after a server start. They're meant to be hubs, people use them for trade and eventually habitation and more missions. It's not enough.
Still a far away but from these tests, CIG need to consider scaling and improving the infrastructure surrounding large player counts, this was very apparent during this test. Imagine when base building goes live and you just have massive organisations that take hundreds if not, thousands of players right off the bat in one area. Long way to go for CIG but will be an amazing experience. These tests did ran very well and am very impressed with what CIG has shown us just need to consider and remove these blockers that are preventing us from going forward...
HUD in general causes mayor FPS drops.. and usually the letters are too small for me to properly read even.. mayor improvements possible there (both on readability/use-ability and performance)
All I'll say is Ollie Had a better time with the 500 player test and did ok with the 1000 player test but he moved away from people as they was trying to kill him. the problem II see with having 1000 players all together in one shard in Stanton is you only need 50 PVP hard players to make 900 leave . At the moment we dont run into the ( die hard kill everything ) as much
It's not the constant delays and blockers that annoy me.. Its CR and others at CIG teasing deadlines only to more often than not get changed. I love the enthusiasm to a point but after being a backer for 11 years now, the same old song and dance gets very, very old. Server meshing is an amazing but of tech, no doubt. If it takes a bit longer, so be it. 4.0 to me is just a reminder of how far we have to go still. Remember when there were going to be 100 solar systems? Which of course also got changes..
@QuotidianStupidity There is nothing "Irrelevant" about it. Many people backed the game at that time based on such things. It was a funding goal and people payed good money that will never see it. If anything, your comment about irrelevancy is just you not understanding the history of this games development and advertised features.
@@PCPAyLOAD no, it seems you don’t understand the history. You are quoting things that are no longer relevant because the scope change. They will still be working on getting more systems in as long as they are still developing the game
@QuotidianStupidity You literally just made my point for me lol. I'm sorry you don't understand basic sales ethics. But, I'll let you have the last word because something tells me you can't help yourself
@@PCPAyLOADNah, you completely missed the point. You know shit about the development of this game, your comment about backers not ever going to see certain things proves this. Try again.
i dont value my time and even i wouldnt spend 5 hours to just stand on a ship to help out a company who constantly lies to us and which iv payed money to.
This, especially the performance problems that hit the CPU just for players existing, shows how many fundamental problems SC still has engine-wise that have to be tediously fixed for the game to become what it is meant to be. I hate to say it but we are still quite a few years away from that.
Games always been spaghetti code, case and point the million ways elevators break when they change or add features that should not even touch those parts of the code.
@@SnowTerebi He may not be, but I am, and he is right. SC engine has many problems that should be addressed first before server meshing. With modern processors and decent optimization, you should be able to support 1000 players before you even need to start meshing. Server meshing is an expensive technology (cost per player), it doesn't scale as gracefully as simply having more powerful single servers. As it stands right now, SC engine does not appear to be able to properly do multithreading. I don't see the point of trying to mesh servers BEFORE fully utilizing multithreading & AVX instructions. Furthermore, no point on supporting high player counts if you overload client code to a point where game runs at 20fps. These are fundamental problems that needs to be fixed before meshing. If you fix these problems, but don't have meshing, you can still put forth a pretty decent and interesting game with traditional instancing. Furthermore, your game design needs to account for the fact that you simply can't have infinite players on small space. When you are doing server meshing, a player still needs to see player and interact with players on another server mesh near boundary regions. At that point you lose any computational advantage you get from meshing in the first place. Entire idea about meshing is that there shouldn't be too many players near mesh boundaries that can overload individual instances. So, it doesn't provide any solution for too many players on small region. It is a good solution for having entire game universe processed by multiple servers as long as players are decently separated along the universe. Lastly, I think RSI's approach the designing a game like SC was wrong from the start. You need your core technologies ready before you even start making a game like this. And you need your entire game design to revolve around the technical limitations of your core technology. They went the other direction and that puts them on a really-really troublesome position. Only thing that prevents SC from being another scam is the ridiculously high amount of funding they received.
I think the fundamental problem with Star Citizen is going to be server costs. Without a subscription model or continuous store sales, I just don't see how CIG will keep large numbers of servers running in the long term. They're probably burning 10's of thousands or more each month.
@@QuotidianStupidity Well they have 1200 staff and nothing is really happening. I feel a lot smaller properly talented/directed/motivated team would accomplish more. Also once the game is released (if that will ever happen) then what will be the use for of all that staff as I question that already at present.. They're going to have to downsize eventually, even tho I don't generally like that sort of thing, but not sure why they needed so many staff in the first place... Cost must be off the roof, hence the continuous sales this year I guess...
@@vfgdfg firstly the people they have are working on things that cannot be implemented into the game at this point. For instance the planet devs have been working on Nyx for SC, but also Odin and some others for SQ42. The Ai teams have made some great leaps forward, but most players would never know it because the current server tick rate is too low. Hopefully server meshing will fix that. Then you have the whole other game in SQ42 that’s been made, the teams involved in mocap etc. once Squadron is out, there’s another two games to make. CIG is not just here to deliver SC and be done with things
1000 players in a single instance is incredibly impressive for any game but it is especially impressive for a game of this complexity. The only other game that sorta gets close is PlanetSide that can do a 1000 players in a server but those maps and the interactions that take place are way less complex. That is why pretty much all mmo's have different instances because otherwise the game would just crash/down to 1fps. The fact that it runs this well is very impressive. Look forward to see them improve and make it even more amazing.
@@wudimusic Developing game in alpha. When I was alpha/beta testing past games(planetside,SWG,Eve Online,Earth and Beyond,etc.) the new tech was also complex and impressive for it's time. Nobody had much fun either except the super testing geeks. I'm just playing,testing,reporting bugs, and enjoying the ride with Star Citizen. Never done this before. You should find a finished game to play if you have issue.
@@blackworldtraveler3711The Games i play and enjoy have nothing to do with the state Star Citizen is in. And neither games you metioned, have been in alpha for 10 Years or were in a better state back then. I should know, because i pledged to SC over 9 years ago. I played it for years, but now i hop into SC every few month only to have it not work and revisit in a few month after that, for the same experience. They ´re at a point where they can´t stop the Ship-selling & feature-creep. Because the Spice (Money) must flow.
@@ryanmitchell4044You don't lick cake mix then claim the cake tastes terrible. Let them make it first, then judge. Tech comes before balance and "fun." Welcome to game development of the most ambitious kind.
How do you know it was 500, when you can´t even see the number of players online anymore? Also, SC never is or was smooth, at best it´s "sometimes less laggy"
@@MathiasBronneshas visto el numero en un menú. No hemos visto 500. Tampoco hemos visto múltiples "fragmentos" con misiones activas o jugadores matando IA en un búnker. Aún está muy lejos está tecnología funcinando
This has been my concern about serve meshing especially static it doesn’t matter how many severs you have if one dedicated server can’t handle the traffic of the shard number it’s too big.
Loads of the (somewhat) smaller issues you had - party management, hab spawning, and performance with so many players in the same area are 100% blockers to having server meshing at 1000 players, however, it doesnt stop them doing exactly what they did and test the tech. I can imagine that even when / if they get it running perfectly (which, it sounds like, other than the above mentioned it did work to a certain degree) they wont release it until they have improved all those other bits to handle it. Having the server meshing tech working is for sure a higher priority than a party managment system, seeing as most folks play with maybe 5 - 10 people at the most. This was a great video - and very likely a fantastic insight for devs - please keep em coming!
Mind you, we are testing the static server meshing. Dynamic meshing should be able to solve this problem assigning multiple servers to a higher traffic area
I've been saying this for years, but everyone told me that Im whiny, toxic or that I don't understand game development. Every time someone mentions server meshing, I respond with - good luck with talking to everyone through discord, or buying a ship on new deal
There are so many scaling issues, lets say how many players are gonna be required to run a Carrack then if we are to believe some crew are npcs how many npcs per person will that be and how expensive server wise would those crew be per person....
Your observations are a surprise to nobody including CIG. They're not intending for nor surprised at the possibility of 100+ people lining up to purchase ships at New Deal at this stage of development of SM. Scalability of in game services was not the point of these SM tests. As for you being whiny, that's just your personality and that's ok.
@@gk.4102 If you consider pointing out massive, fundamental, core problems to be "whining", then I sure hope there are some really, really whiny people in leadership positions in CIG.
Honestly, the bugs and desync make Star Citizen look like an early access game that's only been in development for a few years. If you had no knowledge of the development, it might be quite a shock to learn that this project has been in development for 12 years with almost a billion dollars in funding.
@@mightyfineincredible2252 yeah they totally announced server meshing done in 2016? 2016 was the forst time afaik that they TALKED about it. The first time they implied a release date was 2020. So 4 years late. Typical chris roberts date. But why would you lie. Just a hater
@@mightyfineincredible2252 they could have published this game long time ago but then it would've been something more like StarField. Only a few landing points on few planets, loading screens etc. but since they want to make it something so much better, its gonna take more time obviously
Seriously. They were supposed to release Pyro back in 2020. They just admitted in an SCL this year that the devs haven't even decided how the law system will work in Pyro. Are you kidding me?
@@QuotidianStupidity By having like, so few hangars in a space station, that you will need +8hours before taking off if there is more than 100 players on it and is the last on the line ?
@@karakiri283 that isn’t a clear sign of “having no idea what they need for the end game”. It’s a sign that it’s not right for now, but it is going to be very easy to make changes to stations and ports. Until they know what the limit is for each shard, the popularity of each sector, planet and station within that, player habits and what the underlying tech for ATC replacement etc can handle then there will always be areas that are not fit for purpose whilst they are deliberately stress testing the tech by increasing the player count so dramatically.
you pulled off an absolute banger of a test, especially in a QA environment. I've been in games where it was difficult to get a 100 people to coordinate.
The main problem CIG has had since this game's inception was not making server meshing the architectural priority from the start. They knew from what was functionally day 1 that the end goal was MMO with a world with a ton of active players in one "server". They could have built their own engine from the ground up to support this (they later down the line realized they needed to make their own engine anyways), and then designed the rest of the game on top of that. Instead they spent **years** on flashy simulation features that no one really asked for (super detailed ship damage models that don't work 90% of the time, wild amounts of character customization that no one asked for, expanding the map size to be huge before it could actually support enough players to not make the huge map feel empty, the list goes on). Star Citizen currently is an amalgamation of tech demos, some of which are admittedly very cool, but their entire structure of this game is fundamentally preventing them from ever reaching a finished product. It seems almost every patch they give us flashy new features with little actual utility, don't get us any closer to a game that is actually playable and enjoyable, all while kicking the fundamental technical problems down the road. I'm starting to think that this game is not architecturally salvageable and if they ever want it to be a real, playable MMO with thousands of players in the same server-mesh, with good UX, and enjoyable gameplay loops (yk the things that actually matter in a video game, not accurately modeled hair physics), they may genuinely need to tear a lot of the game down, start again with the goal of MMO servers, and then re-layer things on in order of importance, lest this game be doomed to remain a forever tech demo. Lots of cool snippets here and there, but nothing coherent and complete enough to actually enjoy and play regularly. - someone who has spent hundreds of dollars on this game, and have been watching it since day 1.
The issue was this level of fidelity with FPS and planets was not part of the scope.. The FPS game play was supposed to be limited and not AAA. When that was rethought and the planet tech came on board everything changed. This is context that you arent realizing. And when you start with 8 people Im sure building an engine was not a affordable consideration. No current game with this level of fidelity supports 1000 players on a sever or world. Period. So people need to temper their expectations and be realistic here. Cramming players in a single zone like this is edge case at best. SC will have limitations, however that should not be considered a failure or unplayable. This isnt WOW with low poly models and minor physics to worry about... And planetside also is low poly by todays standard and really is just a battle plane.
Someone within CIG prioritized hunger, showering and actual shitting mechanics (no joke, that's coming with pyro) over the basic fundamentals. Guess who that was! And guess how many people are defending that decision. XD
That's a lot of words coming from someone who doesn't seem to know very much about the project Over the last decade they have implemented piece by piece new underlying tech that enabled them to get to implement server meshing right now (OCS, SSOCS, PES etc.) They chose Cryengine as the engine for this game because they also got to hire some of the guys who literally built that engine from the ground up at Crytek, so they could rebuild the engine to support SC's vision, quite a few of these people are still working at CIG rn Also, what map size are you talking about? The planet sized maps? You think they should have made extremely tiny planets/moons or what? Also there has been no focus on an "extremely detailed ship damage model" until the last 2 years with the Maelstrom system, and it's still in development and hasn't been implemented in any form in the PU
i dont even know if i want 1000 player servers anymore. what a nightmare the tech behind this is. maybe it's time to cut losses and focus on making a good game instead of focusing on the scaling all the time. The Port Ollisar style of gameplay was cool anyway, felt more connected and personal.... now we're here with instanced hangars and copy pasta locations everywhere to accommodate the masses. when scale overtakes individuality, things lose meaning. e.g. Jump Town.. invented by players, ONE specific spot. Now they copy pasted it to multiple places. also fleet battles suck anyway. 1v1s or 3v3 is where its at. where is the point of scale, when everything will end up in "instanced-mass-theme-park"-style gameplay anyway (and doesnt even work)?
The communication problem you encountered after the crash is the single reason I think whatever voip Spectrum inevitably includes needs to be its own desktop client, separate from the SC client itself.
"Star Citizen Server Meshing WORKS, but..." Is like saying you have a racecar....but every time you get into it it catches fire. Nice to see an actual video on this in practice
It’s a matter of choice: Drive the change vs wait for something big happen and miss the chance to been part or it! Either are good, just be sure to pick the best one for you!
This game is hilarious. It's such a massive clusterfuck that it's almost like watching house burning down veeeeeeery slowly irl. Fans still act like everything is fine but it doesn't change the fact shit's burning down;D
Yeah dude it's totally gonna crumble and fail in the next 2 years just like everyone said in 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024 etc. 😂 Meanwhile not a single AAA developer making billions every year is able to replicate even a single tech that SC has at the same scale Maybe the only exception being MSFS 2024 with their Earth, but we'll see to what extent
@@gian.4388 Yeah dude, it's totally gonna change the gaming world like fanboys were saying... in 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024 etc. ;] Meanwhile bunch of AAA games already surpassed it when it comes to technology. And MSFS will be an actual game... You know... With a RELEASE date and all;] When's that happening for star citizen?
There is a related factor that seems to be glossed over at present - the limited number of regional and localized servers. Currently, it appears that SC servers aren't dynamically enforcing and capping or adjusting the client-side draw and render distances of global objects and effects - based on server load. If it is I'm yet to see it. The consequence is clear - even players with very powerful rigs get extremely poor performance. Take my situation for example - I'm 6k miles (9.7k km) from the nearest SC server. Any lag in my connection impacts my group, and all players on the server. This issue exponentially compounds problems with server meshing, persistence etc. It impacts all network related traffic especially streaming. More client-side objects would help reduce traffic. Dynamically enforcing local view and render distances based on server load and. kicking players with high ping times and jitter would help a lot. I'll take one for the team. But ultimately we need more regional and localized servers and a lot less streaming data.
If we all lived in Japan, we could do it. Infrastructure there is insane. I have 5gig symmetrical at my house but I know most people have shit internet and how far people are from each other and servers.
not the infrastructure hes talking about... hes talking about how the game wasnt designed to handle this many people. the most obvious being that there arent enough habs to spawn in or only being one elevator etc... the game was designed for a smaller population, throwing a thousand people into that will cause backups in a lot of places. there will be lines to access stores, elevators, asops, etc... the problems they are having here have nothing to do with your internet lol
@@magicalfungi3206 no shit. Server infrastructure works best when you look at physical location, physical hardware and software/coding. And everything in between. If you have an amazing server but someone in china is using it, it won’t be a great experience.
@@busterscruggs_sc can you read? It’s what I said. The bottleneck at the end is physical. Not code. They can make the best code on the planet but it won’t work with a 300ms
How they could fix the issues you raised: 1. Clear up the UI mechanics (like they are doing for ship MDFs) so you can turn on/off party members, or have a proximity for certain distances for short/long range toggle on when they appear on your UI. 2. Have the habs instanced like the hangars where the game world creates duplicates and instanced habs. Gateway the doors like hangars. 3. Dynamic server meshing (my hope is the game will adapt and create more servers/spaces when the replication layer can see bottlenecks in the DGS like you experienced.) More servers in smaller spaces like Hub areas are probably key for overall improvements. The game has to be optimized and Im sure CIG know of these constraints.
We can't expect server meshing to be perfect from the beginning, do you remember when we could not stay connected to a server for 10 minutes without experiencing a 30k? Now 30ks are much less common than before, they only happen from time to time even on servers with 100 players on. The idea behind server meshing is great, give it time, it will resolve a lot of issues and allow 1000 players and more at every location.
@@dragoncpu2226I'd go one step more and say even for a finished game it shouldn't have anything that charges a grand for. But again, no one is forcing you to do so. And in term of SC they are mainly gathering the funding from whoever can throw that amount of money away rather than selling some product.
So the problem is that it's too realistic when you try to funnel hundreds of people through a facility and into a ship? It's behaving how it should, it's a space sim.
This game ate three sized 8SCU containers after I loaded them with all my best stuff and transported the crates to a new space station. As soon as I sent the elevator down to store everything, all 3 crates just disappeared themselves. All loot gone, and it took me 40 minutes to move/load everything!! Fuck your drain and water analogy, I’m done wasting time on this project. They should have called the game Project 2050 which represents the real GAME RELEASE DATE. Fucking A. Going out for a moto ride now! Cya CHRIS!!!! Good riddance! 😂
I've been convinced for a while that CIG is going to end up separating people into more shards (when they're in the same area) and instancing more content in the final experience. Mountains of physics and detailed dynamic surfaces coupled with hundreds of people and the expectation of being able to have functional interaction/combat is a tall task. It's not impossible, but like with anything, the more variables you add to it - the higher the potential for issues.
i’m interested to see how they change city layouts and increasing the capacity of the transit system to account for the increase in average/peak players
Star citizen is at a rate on getting the epic fail stamp approval from me. And this comes from a person who has been sitting here since the beginning. The problem here, it went from an idea... to going total apeshit... and here we are 12 years later lol. You can't even get your network code to work proper...
What are you talking about, the test was for 500 players and it was fantastic. The stress test which is to take it to failure was for them to learn, and improve just as they did from tests 1 & 2
Moreover stating “you can’t even get your netcode to work properly” is meaningless. Netcode actually work, otherwise you couldn’t login and move around station. The fact here is they are improving the netcode to support more ppl, reduce lag, rubberband, increase stability and so on. For a car 🚗 to work you must let that engine to stay on, THEN you can add HP, reduce fuel consumption, reduce noise and so on. And it’s exactly what they are doing, the only difference is that they let you sneak inside the labs instead of simply wait in the shop to see the final product.
@@MartinDlabaja just elaborate better your meaning of "doesn't even work". Because for me it's thing that is completely broken, instead SC it's working, with flaws, but it's working, and it's in active development... so, for what I can see.. it's working rigth now
Thanks for mentioning that the point of the 1000 player test was not to see if it would work, but to see how it would break. Honestly, I'm surprised it worked as well as it did. While I agree that it's a good sign that they're testing, I wouldn't be too worried if they had periods when they weren't testing. If they feel they have sufficient information to have benchmarks and make changes, then it makes sense for them to shut it down while they make those changes as the additional information isn't really necessary. Then, after the changes are made, they can test again, get metrics, and see when and how that build breaks.
It also has the knock on effect that people dont get "tired"/"used" to server meshing if they keep server meshing as a bit of a mythical thing, odds are they will have more people show up to these tests, which helps gather more data
Even in THIS test, when they went about 500 players. It took 30secs-1+minutes just to get a response for turning on my ship. That's just not good enough to surpass 500 players.
If they thought it was anywhere ready it would be on PTU not on tech preview . They know it’s broken they must want to see where pand why. IMHO they just need to do 300per shard just to shut up some doubters , make it work and then slowly up it while optimizing the actual game including bed counts or at the very least que people in using the time it takes them to leave the habs.
@@robertchandler2063 I choose to remain skeptical on the scalability of this model, for now. 1000 people on the same shard of static meshed servers for the Stanton system is a good goal. They will definitely need to get Dynamic Meshing in soon to handle bottleneck areas and load balancing needs. Not to mention when events like Xenothreat pop up, it needs it's own server.
Remember when star citizen was going to let players host their own multiplayer servers, and it would have probably been 100 or so people in a server? I bet that tech would have been working by now...
That was a promise early on, but as a software engineer i knew this was NEVER going to be possible to run the whole PU for the majority of people. I have hosed multiplayer servers of my own for things like 7 days to die or ark. In these i had maybe 20 people as a peak. Star Citizen is right now, probably 500x more playable space than 1 ark server so this means the only way private individuals willing to run a huge cloud cluster of servers which would be pretty expensive...and this is just to run santon. So the only private server we may ever get would be to host a single system or just run arena commmander
@@MrBlackjimrogan yeah, that's because the scope of the game changed dramatically since the original pitch. We were originally going to have landing areas we could go to and explore, but not whole planets. The first person shooter stuff was not going to be as prevalent either. As the feature creep set in, it became clear that no normal computer would be able to host this monstrosity of a game. Sadly, CIG have really put themselves in it now too. They need to invent completely new technology to manage the enormous server requirements, and that's expensive. Once (if) they get the technology working, they need to maintain many expensive servers to run that tech. All of this adds up to a very expensive game to run, and they will need a constant flow of cash to keep the game running. That puts the players in a crap position, because it forces CIG to adopt predatory monetisation tactics to keep the lights on. It's all connected, and every year it gets harder to support in my opinion.
@@TKanal3 freelancer is still going strong, 21 years after it released. It's a fantastic time investment, and is one of the best space games of all time. That has exactly the style of multiplayer that star citizen originally pitched. 100 or so players in a server, the ability to host your own server, and mod support.
@@funkyschnitzel absolutely correct, the one thing that is good about the server meshing and general approach of where we are now is it is scalable up and inevitably when player numbers drop back down. Im pretty certsin that so long as squadron 42 sells well we should have at least 5 years of money come in when that drops
They should have 2 versions of the game. A super stable version (maybe no persistent hangars) but super stable, then a "nightly" build for those who dont mind the instability but want the bells & whistles. Problem is, I dont think a wholly solid version exists.
Morph isn't quite fair in his explanation or testing. It does "work". The goal of SM is to use multiple dedicated game servers (DGS) to distribute server load. Each DGS will have a logical limit, meaning just like any game with the same amount of processing, it's only going to handle 100-200 players. Putting everyone on a single DGS is not the result of SM not working... its the result of Dynamic server meshing not existing to mesh more DGS and split the load. If you watched any other streamer at the time, you would've noticed that SaltEMike had a low desync, 30 fps, reactive ai dgs on that same shard. Summary? Yes it's "working" However RMQ is still a bottleneck and basic network optimization still sucks. Most of the issues aren't a server meshing problem.
I'm not an expert but gathering bunch of people in specific location is not something current version of server meshing is suppose to help with. I mean, it will help with performance in other locations on the other X servers where your group isn't located but it won't help to your group. Pretty sure what is supposed to help here is "dynamic" server meshing, where extra server gets assigned specifically to the overcrowded location on the fly, meaning the bar with 100 players in it would get it's own server assigned in real time, with border being probably located somewhere in the close corridors while the rest of the landing zone would be running on other server...and that tech is proly still light years away.
Can’t be light years away… simply because that is a measure of distance and not time.😅 Joking apart, your assumption are indeed true, but to achieve that you must consolidate the static meshing, with a solid message queue system. And imo they’re on the right track…
They’ve gone from 100 to 500 in a short space of time, and they are still working on the system. Given this was a stress test, why would you assume such a thing? That’s very negative for no reason
I mean, back back out in the olden days, Star Citizen was at the same point, just instead of 100 players it was sub-50, and instead of 1000, it was 100. Similarily, a LONG time back far before even Star Citizen, people once assumed the Video Game Industry was only for nerds, and a niche market - now its a behemoth large enough to swallow the music and movie industries. Just because something is hard, doesnt mean its impossible.
A video I’d like to see from you is “an architect reviews star citizen Habs” This would be cool to see Howd you’d design a hab to fit in the same space and provide real world solutions to problems such as not enough elevators and not enough habs.
Definitely need to make Space stations possible home locations to spread out the player base as a temporary fix for over crowding. But I’d like to see a rework of home areas or city additions to planets.
Thanks for using your unique status to organize a community test like this - I really hope CIG got good data and that it's a wakeup call about everything that needs to change.
Literally clicked on this video and started crying like a bitch before it started. Hey, those systems also went from unlandable surfaces with max 2 locations or so for some planets to FULL PLANETS. Moron
I enjoyed the video. I don't "test" anymore since we're just so far away from what I consider a "game" that I don't want to burn out before it's here. So it's great to see videos like this to "keep my eye on the game". There's such a long road behind and ahead of us.
There is a lot of unused space at these space stations. We just need more hangars and lobbies added to that empty space. It is hilarious that nobody at CIG thought of this beforehand, tho 😂
Or maybe they know this and are giving priority to build the backbone to support that later. Like in chess… sometimes the best move must be kept for later time… just build the ground for that decisive move.
Always a high quality experience watching your videos. I trusted Ben after the first test because I know how new, better solutions can create a clusterfuck when just implemented in development. Happy that I did. I can see 1000 players working at some point with how improvements are going. (Tho i suspect improvements wont be as massive as from 3->4)
Hey Morph. I know this is an odd request but i would LOVE to see a video on how you would make and furnish and house on the planet reach from halo. Like you did for star citizen. I love the compound/houses you can see in the winter contingency mission.
While people say threre are not enough hangars/terminals etc...We are still in 1 system. The testing tested crazy things. If we did not know a test was happening we wouldnt have everyone congregating in 1 location for a start.
I think what you're seeing is that CIG expects that when they add more star systems, player density is going to be around the same as it is now, in other words around 100-200 players per star system. Otherwise all this time we've been hearing about the networking team getting servers ready to handle 1000+ players per shard, we should have also been seeing more elevators being added to habs, UI improvements to handle more players in a party, etc. Instead we're seeing progress in pretty much everything else but that, which means that CIG thinks it's unlikely that we'll see 1000 players all spawning in say NB at the same time.
Server meshing is cool, but your point on the non mesh systems that cause issues is something CIG needs to address. I know it gets old when I talk about ATMO, but we are the best test for these systems. The Daymar Rally has 1000+ people across multiple servers and last year was a nightmare, not enough habs, not able to launch because of too large of a party, and just the general issues you get with that many people being in one spot. On top of all that we have the organization aspect, We spend months getting ready for the rally and have to have a massive team to make it happen, while a lot can't be fixed in game due to the nature of the event, tools beyond the party system would be so helpfull.
They either need to add more terminals in cities to allow more people using them all simultaneously, or they just need to make cities their own servers with less people per server, and once you get into your hangar ready to leave, that is when you hop over to the larger servers. Then again, once SC will actually see such large numbers, and even now, there are and will be more cities and stations one will be able to spawn at.
One thing about the "few elevators"... something that always bothers me is how fast the doors will close again. Like you can't really "hold the lift" for someone coming down the hallway.
I definitely agree on the idea of a long PTU cycle for 4.0. Sadly, though, as we've seen.....as the PTU goes on, less people are there to test. So after the initial couple weeks, testing a big server config could be practically pointless when it won't ever get back to near capacity. It's something CIG has struggled with for a long time now, and could be a big reason 4.0 especially will launch with a lot of issues. CIG should probably, as many have recommended, do testing "events" during that long PTU cycle to occasionally draw people in and re-stress the environment. They might even have to offer some kind of reward. But they really do NEED to do this, or it's likely they'll "think" some issues are fixed while it's in reality just less people loading the PTU.
![An Architect Reviews the RSI Polaris Capital Ship [Star Citizen]](http://i.ytimg.com/vi/AdtdVoi_53Q/mqdefault.jpg)
I have been saying for a long time now that having 1000+ players on a server is not just a matter of the meshing/networking/server performance - the existing infrastructure/services IN GAME cannot support that much traffic at all - the wait times to use hangars/terminals/etc will be insane as you will end up at the end of very large lines constantly. stations and cities and such will need to be WAAAAY BIGGER to accommodate that population.
Don't forget that there won't *usually* be the huge spikes in population doing the exact same thing at the same time in the same few locations. i.e. Most people will not all be all logging on together as the server comes up. The trick will be to get the player population evenly distributed over a bunch of different shards (star systems).
Don't forget that interactions will be handled by the RMQ system
@@stormwolf3255 A lot of people work 9-5s and will want to play after work (like the OP) without issues.
@@stormwolf3255 There will definitely be occasions where this does happen. An org logging in for a scheduled event, people logging back in after a server update or crash, people gathering in one area in anticipation for a schedule official event like IAE. There are a lot of cases where a sudden spike can and will naturally happen.
@@Morphologis That's how it is in every MMO. I don't think that the devs simply never thought about this
Their focus on elevators was always hilarious to me, there is reason that ZERO train station, airports or any similar place has elevators as the main way to travel to something. Even the super deep soviet subway stations don't have elevators as the main way to get down there, even though it would be faster than the escalators, it just isn't feasible to transport the amount of people necessary.
Remember that hangars on stations or spaceport can be miles away from eachother.
@@generalbandege1184 That's why you have subway and stations, and then mixed with normal pavements for walking, also as alternatives so you always have 2 means of reaching any place.
@@cryonim Would you consider a one hour journey to get to your ship in order to take off a fun activity?
The soviets didn't need to spawn hangars on top of eachother in instanced parallel universes so elevators didn't solve a necessary problem like they do in star citizen.
Elevators have long been a loading trick for games (you get in the elevator, assets load in and out). I know SC has some robust asset streaming, but I would be surprised if there wasn't still some good old-fashioned elevator loading going on.
when the guy detonated ur 890, props to you for not punching your monitor, especially after 5 hours of work
Could have set a record for a mass death.
I really gotta question the guy's mental health. He spent the time downloading the PTU version of the fame. What is it, 150 GB? Then he spent 5 hours in a line. Then he decides to blow the whole test and run off in gleeful euphoria? A nutter, for sure.
@@fwdcnorac8574 average gamer more like. Self destruct was the ONLY way to end that test anyway, gotta test how the game handles that many people being around an effects heavy explosion.
@fwdcnorac8574 The troll may not have spent 5 hours in line. Morph said he spent 5 hours trying to set this up. That guy could have been hanging out for an hour or less and jumped in on the action. No doubt he was laughing super hard at all the bodies floating, but that ended the test so it's unfortunate since they finally got on the ship.
@@fwdcnorac8574 In Elite Dangerous a troll spent around a week doing hundreds of tedious hyperspace jumps, just so he could ambush an expedition of lighly armored and unarmed explorer ships at the center of the galaxy (must have taken a minute tops to blow them all up). Never underestimate the drive of people with mental health issues.
"The game itself may have major blockers". May? MAY?
Positive vibes.
To be honest, players were intentoinally crashing the server. Ive heard Server meshing works fine when players tend to their own business.
That was really downplayed, wasn't it? I'm no expert but I see years of work still needed here.
A 1000 player server is a Good Problem to have. If its a problem you can go play that other game with a 1000 players.... Oh wait... There is no other game with a 1000 players.
@@sandman1347 yes, this should be the alarming part. For the past year people were saying "server meshing is all we need!" And this shows how much more work is needed beyond it. Even the most positive estimates would be at least another 5 years before that many players can actually play long term.
It makes me wonder how many game systems will need to be re-worked AGAIN once they get server meshing added to the game. We have years of work built around a 100 player cap, now we're adding 10x that amount of people.
CiG have made this game in the most ass-backwards way possible ngl.
@@SA80TAGE not really, there was none of this that existed before and when they started there was no game engine that was any better, cry engine was peak at the time and still had a ton of issues, the problem is its not easy to work in and most of the old manuals etc are gone, they relied upon some of the original engineers but its a lot of work rebuilding everything.
Currently there is no game engine that can do what they're doing, unreal engine, unit, godot, NMS own engine their devs created also wouldn't work.
In reality the scope broke things far more, originally it wasn't anything overly interesting then changed to sort of like starfield is currently and now we are here..
@@SA80TAGE True. But they've made almost a billion dollars in the process. The joke is on us.
@@BigBobBlazer 700 mill in 12 years is not a lot btw and most of that paid for salaries and sq42
Let’s estimate how much CIG might pay in salaries per year based on the $700 million over 12 years. First, calculate the average yearly funding:
Average yearly funding= 700,000,000 ÷ 12 = 58,333,333.33 per year
So, CIG receives roughly $58.3 million per year. Now, considering the rough salary ranges and estimating how many employees they might have, we'll attempt a rough calculation.
Assume:
CIG employs around 500 people (a reasonable estimate based on reports).
If the average salary per employee is around $100,000 (to account for a mix of junior to senior positions).
Total salary cost=500×100,000=50,000,000
Therefore, CIG could be paying approximately $50 million per year in salaries, leaving around $8.3 million for other operational costs like software, development, marketing, and server upkeep.
This is a rough estimate based on the information provided.
The fact that so many gameplay systems and fundamentals were designed without SM, as it didn’t even exist, presents a problem. We can all tell that this isn’t going to be a simple matter of changing a few values to magically allow the current ‘game’ to work with hundreds, if not thousands, of people. I foresee many more years of reworking the core systems and gameplay to make this be a smooth experience
I argue about that point. The core systems are already in the rework for SM since a few years, and that's why we can see result today because of those reworks.
What needs rework now are some features/gameplay that sit on top of it (Missions, Socials, Mass transit, Dynamic economy, etc). There is work, for sure, but many more years seems very very very pessimistic. 4.0 will be the first iteration on SM, and it's planned to be release later this year (but we all know it will be a bit later aha). So a few months of work left at least for static SM support and all the gameplay?
Though I do agree that improvements will be necessary on the path to SC 1.0
I was arguing about this on discord - for hangars / habs / elevators it would be enough to have it "instanced" in case of "higher than expected load".
Nobody wants to sit 10 minutes in hangar, waiting for take off permission / wait 10 minutes in lobby for a free hangar.
@@BeSk9991 Yes, at the same time I could understand some reasonable wait - 5min tops - however, I would need to know what is happening. If there is a queue they need to show me or tell me somehow that I'm X in line for take-off or whatever. Otherwise, with current systems I'm sitting there not sure if there is a bug or a queue or maybe the server is crashing. This is needed because they won't be making 1000 hangar doors around every city just in case 1000 people show up at the same time. Some sort of queue system is needed and it needs to be visible for the player. This would work exactly the same as traffic lights with a counter till next change. They are proven to relax drivers and actually make the traffic smoother. We need that!
@@jeefuji when was this game originally promised to come out? I think you're paying overly optimistic
That's what exhausts me the most about this game: the constant "reworking" or better saying, doing things the way they should've thought before doing anything. How many reworks of the actual systems that are actually beign developed right now, will have in the future, to accomodate the brainfarts CR will have?, because oh now we have 200milion more dollars to spend so we're going to rework everything because, idk, not releasing the game ever, to keep selling ships at astronomical prices?
Do you ever consider making a throwback video if you ever feel burnt out? It feels like a lifetime ago I was watching your space engineer tutorial videos on how to make the skyke MKIII.
one of these days SE2 will roll around and we can relive those good ol days. damn i miss SE lol
I miss SE but I hate how it never became stable for multiplayer. It was always buggy as shit in MP
@@taienv remember that, as we watch SC improve.
About performance, if the problem was just the amount of geometry because of all the player models, the GPU would be the bottleneck and at 100% load, not the CPU.
The fact that it's the CPU means that either something goes wrong during preprocessing, or the data transfer from CPU to GPU isn't fast enough.
And because you were maxed out on all cores, not just one, it has to be preprocessing. You are using the DX11 renderer, which can only ever use a single thread for CPU to GPU data transfer. So if that was the problem, only a single CPU core would be stuck at 100% and bottleneck everything.
Now, I'm getting into speculation territory here, but there are really only two preprocessing steps that both take significant resources and are exclusive to character models - physics simulation for things like cloth and hair, and "skinning" - the process of calculating what a character should look like given a standard t-posing model, and a pose.
The good news is that, if my theory is correct, this is definitely solvable. Skinning can be done on the GPU, which is much faster than doing it on CPU, and the physics sim should be possible to disable for things that matter too much. You don't really see whether the hair of a dude 100 meters away is waving correctly.
But the final possibility would be that whatever uses all those CPU cycles is not related to rendering, but the renderer ends up waiting around for it because it is not properly separated. This is a very frequent problem in older game engines, and if it's also the case here, CIG would need to make major modifications to the engine to make it better. So I really hope that's not what this is.
What I figured out reading SC comments over time is that there is a lot of people who like argue about "server meshing" and unable to formulate what "server" is at the same time ))) Do you know exactly what client CPU do in SC? I bet you have no even slightest idea ) Concider there is like ~120000-150000 "entities" shown in r_displayinfo. From what we know "entity" is an object which have a mesh, some physical properties, some gameplay properties and it's state constantly synchronised with server. Entity is a players's body, helmet, chest armour, every clip on that armour, his backpack and it's content. All these "entities" they are not magicaly moving "with player", they position is calculated and synchronised with server. Imagine synchronising 150000 objects like this with server every server frame, this is kind of a lot non-GPU work.
@@alexpetrov8871 ... I doubt you figured out much correctly because your comment does not indicate that you have properly read what @CdrSonan wrote. He is not talking about server meshing or the speed at which the server runs. His point is about the slowdown of rendered video frames on the client. What the client CPU cores are doing is entirely irrelevant. Why? Because they should not be doing what they are doing now because they are bottlenecking the whole system (as shown in the video, just watch it again and again and again). The code should be changed and either make CPU cores do less, or off-load more to the GPU. Just read his CdrSonan's command a few times over with your local grey matter setting turned on. Eventually you will see the light, it's not that hard to comprehend.
And of course rendering code should be entirely separate from networking code. Not doing so is asking for all kinds of trouble and then some.
@@alexpetrov8871 Don't jump to conclusions. I've made games in both Unreal Engine and Unity, including a complete custom rendering pipeline in Unity once. So also that isn't CryEngine or StarEngine, I think I have an idea of what I'm talking about.
Directly tying rendering to network updates would be insanely stupid, because it would instantly increase your display latency by your ping, and losing even a single packet would cause a freeze for several seconds until the timeout is reached.
That's why, on modern games, the renderer runs on a separate thread from the game logic, and networking again runs on a separate thread from that. This way, the main thread can continue to update things like physics or animations even if the network thread is waiting around for some packet to arrive, and likewise, the renderer can continue to push out frames even when the main thread is running at a lower tick rate. When this separation works, you will still get high FPS with excessive CPU load, and instead it is only going to cause things like physics glitches or animation stuttering.
The problem is that games used to be designed for single, or few-core processors, where it was better to tie game ticks to FPS and save on the overhead of using several threads. This is especially the case for console games, and the reason many of them still have a 60FPS cap. Pushing more FPS would also increase the tick rate of the main thread and throw everything out of whack.
Anyway, in a lot of game engines, the separated pipeline was more or less "tacked on", so the separation isn't perfect. (looking at you, Unity...) Only networking has always been put on a separate thread because otherwise things fall apart pretty much immediately.
@@CdrSonan >I've made games in both Unreal Engine and Unity, including a complete custom rendering pipeline in Unity once.
Didn't it came to you then that the problem of high CPU load is not with rendering at all?
@@alexpetrov8871 Read the last paragraph of my comment. Or my answer explaining it.
Suggestion for how to organize getting 200+ people in your 890; have groups of 10-20 meet at different landmarks on a moon and pick them up group by group. This way the trash wont build up too fast in an area, and perhaps people will log in in various places then fly to you.
Also allows you to monitor the incremental build up of server lag on you as player count increases
This was a fantastic video, and excellently put together. I imagine CIG's QA director probably loved this feedback because it hits on a lot of really good points and has actual constructive criticism about where the design needs to go to support CIG's next milestone of 1,000 players per shard (which is quite the leap from the 150 they set about two years ago). In any case, this was very well put together, has actual proper concerns that the design and networking team can look into, and actually provides workable feedback that should help move the game in the right direction.
Well done.
QA director. 😂
I disagree, I don’t think it was good.
1000 isn’t necessity their target, it’s just a number they know will break the current tech.
Even if it were that, it’s unlikely you’d have that many people in one place, if there is two systems
LOL, seeing all those bodies floating away from the 890 made my loot goblin tendencies kick in. I'm glad the testing is showing progress. Pyro will be crazy next year. Thanks for the video.
Next year, hah...
"next year"
First time?
havent been around long eh?
Yes it will be "crazy" but in what what way that is the question.
More like 3 years
It was a super dick move to blow up the ship, but I bet the data from having all those players on different clients dying at the same time was invaluable.
They said during the tests they are still rewriting ATC, Transit, and Mission Broker services to work with the new messaging system and why the 1k tests degrades from overloading those services. Even on 1k player at nearly 900 players movement tracking and doors were still good with active NPCs. Once enough people hammered ATC and the transit services ran long enough server FPS stayed at 5 sim FPS and barely responsive doors/seats the rest of the test until they killed it. The fact they tried 1k again means they think they might be able to hit that eventually once all the other loose ends are tied up.
We had people in Mantis/Antares stopping people from QTing away from cities or crashing into ship gatherings by the commons and attacking people by stations. People were getting stream sniped too. The 3:500 and 4:500 tests worked the best if not better than live. The 6:1000 started solid but degraded fast and they said to expect it, but they needed the server message queue data loaded exercised even though they knew other things would fail. Things went so well they decided to leave the 4:500 mesh running to see how it held up over time to see if there were memory leaks or other slow burner failures that take time to cascade. Flying in QT between planets we didn't get knocked out of QT at every server border like the last test. Just a 'left channel local' and frame skips.
It was encouraging but I have to wonder what value is getting 1000 player meshes because no GPU in existence can real-time render that many high poly models let alone light them with something as process intense as raytracing and global illumination eventually. They have some serious culling and optimization work to do and probably should have leaned on depth and bump map shaders more instead of putting geometry into every little crack and panel line of their models. It makes nice closeups and renders but most of the time you aren't going to notice it. The 890j is a few thousand entities alone and each entity can be thousands to hundreds of thousands of polygons. When I did FPS game stuff making player models 3-5k polygons total was the ceiling in the Quake/Doom/Unreal Tournament engine days and now 10-15k is considered getting too heavy. I don't think there is any single object model under 10-20k poly in SC lol.
8:08 The Stair-jumper getting back into line like he didnt just commit insubordination during a *Morphologist Role Call* 😂😅
Looks to me like we want server meshing with 50-100 player caps for Pyro launch for now, they can figure out the much higher player counts later, it’s not the priority.
Great to see some progress! Any progress with server meshing is good progress at this rate. Exciting times
Surprised you didn't mention the queues to leave hangars. Because there's only a handful of hangar doors, and people don't always leave immediately, landing zones get easily congested. I spent like 30 minutes just trying to leave Area18 and there weren't even many people (because they all went to NB lol). Each spot in the queue was taking 5+ minutes, meaning people were taking on average 5 minutes to leave their hangar after they got clearance.....which is wild. And you could definitely see the potential for abuse there, too. It'd only take like 9 people to completely lock down a particular hangar size at a space port.
In a typical game you wouldn’t have everyone trying to get ships and leave hangers all at the same time
@@QuotidianStupidity There's 4 Landing Zones. If SC hits 1,000 players (hopefully they're going for more per system), that's still huge backup potential, even after a server start. They're meant to be hubs, people use them for trade and eventually habitation and more missions. It's not enough.
@@DustinHarms I agree, I was just pointing that aspect out
@@QuotidianStupidity Gotcha, hard to tell with YT comments
@@DustinHarms it would help if I’d told you! 😂
Still a far away but from these tests, CIG need to consider scaling and improving the infrastructure surrounding large player counts, this was very apparent during this test. Imagine when base building goes live and you just have massive organisations that take hundreds if not, thousands of players right off the bat in one area. Long way to go for CIG but will be an amazing experience. These tests did ran very well and am very impressed with what CIG has shown us just need to consider and remove these blockers that are preventing us from going forward...
HUD in general causes mayor FPS drops.. and usually the letters are too small for me to properly read even..
mayor improvements possible there (both on readability/use-ability and performance)
All I'll say is Ollie Had a better time with the 500 player test and did ok with the 1000 player test but he moved away from people as they was trying to kill him.
the problem II see with having 1000 players all together in one shard in Stanton is you only need 50 PVP hard players to make 900 leave .
At the moment we dont run into the ( die hard kill everything ) as much
Sounds like pyro would be more fun true anarchy.
yeah indeed, for such higher numbers we need Pyro, so pvers go there while stanton remains as a new-player friendly system, with less risk-reward
@@seveneternal7988Assuming the PVPers actually do go to Pyro and don't just stay in Stanton to get lots of easy kills on newbies.
@@seveneternal7988 maintains the problem, that why would you go to pyro if all piracy targets are stil in Stanton.
for pyro it would be split so if its 1000 players total its 500 in pyro and 500 in stanton type deal
Good job on this. Loads of issues found that most did not.this should hopefully have helped cig a lot
It's not the constant delays and blockers that annoy me.. Its CR and others at CIG teasing deadlines only to more often than not get changed. I love the enthusiasm to a point but after being a backer for 11 years now, the same old song and dance gets very, very old. Server meshing is an amazing but of tech, no doubt. If it takes a bit longer, so be it. 4.0 to me is just a reminder of how far we have to go still. Remember when there were going to be 100 solar systems? Which of course also got changes..
100 systems was when there was not planetary landing. It’s a completely irrelevant point
@QuotidianStupidity There is nothing "Irrelevant" about it. Many people backed the game at that time based on such things. It was a funding goal and people payed good money that will never see it. If anything, your comment about irrelevancy is just you not understanding the history of this games development and advertised features.
@@PCPAyLOAD no, it seems you don’t understand the history. You are quoting things that are no longer relevant because the scope change. They will still be working on getting more systems in as long as they are still developing the game
@QuotidianStupidity You literally just made my point for me lol. I'm sorry you don't understand basic sales ethics. But, I'll let you have the last word because something tells me you can't help yourself
@@PCPAyLOADNah, you completely missed the point. You know shit about the development of this game, your comment about backers not ever going to see certain things proves this. Try again.
i dont value my time and even i wouldnt spend 5 hours to just stand on a ship to help out a company who constantly lies to us and which iv payed money to.
This, especially the performance problems that hit the CPU just for players existing, shows how many fundamental problems SC still has engine-wise that have to be tediously fixed for the game to become what it is meant to be. I hate to say it but we are still quite a few years away from that.
Out of the 1000 employees 900 seem to be marketing😂 seriously though I'm losing faith in this spaghetti code
What's your experience that backs your statement? Are you an engine engineer or something?
@@SnowTerebi his experience is: i backed 3 years ago and i know all believe me.
Games always been spaghetti code, case and point the million ways elevators break when they change or add features that should not even touch those parts of the code.
@@SnowTerebi He may not be, but I am, and he is right. SC engine has many problems that should be addressed first before server meshing. With modern processors and decent optimization, you should be able to support 1000 players before you even need to start meshing. Server meshing is an expensive technology (cost per player), it doesn't scale as gracefully as simply having more powerful single servers. As it stands right now, SC engine does not appear to be able to properly do multithreading. I don't see the point of trying to mesh servers BEFORE fully utilizing multithreading & AVX instructions. Furthermore, no point on supporting high player counts if you overload client code to a point where game runs at 20fps. These are fundamental problems that needs to be fixed before meshing. If you fix these problems, but don't have meshing, you can still put forth a pretty decent and interesting game with traditional instancing.
Furthermore, your game design needs to account for the fact that you simply can't have infinite players on small space. When you are doing server meshing, a player still needs to see player and interact with players on another server mesh near boundary regions. At that point you lose any computational advantage you get from meshing in the first place. Entire idea about meshing is that there shouldn't be too many players near mesh boundaries that can overload individual instances. So, it doesn't provide any solution for too many players on small region. It is a good solution for having entire game universe processed by multiple servers as long as players are decently separated along the universe.
Lastly, I think RSI's approach the designing a game like SC was wrong from the start. You need your core technologies ready before you even start making a game like this. And you need your entire game design to revolve around the technical limitations of your core technology. They went the other direction and that puts them on a really-really troublesome position. Only thing that prevents SC from being another scam is the ridiculously high amount of funding they received.
I think the fundamental problem with Star Citizen is going to be server costs. Without a subscription model or continuous store sales, I just don't see how CIG will keep large numbers of servers running in the long term. They're probably burning 10's of thousands or more each month.
Income from SQ42
+the cost of the ridiculous amount of employees they have....
@@vfgdfg why do you think it’s ridiculous?
@@QuotidianStupidity Well they have 1200 staff and nothing is really happening. I feel a lot smaller properly talented/directed/motivated team would accomplish more. Also once the game is released (if that will ever happen) then what will be the use for of all that staff as I question that already at present.. They're going to have to downsize eventually, even tho I don't generally like that sort of thing, but not sure why they needed so many staff in the first place... Cost must be off the roof, hence the continuous sales this year I guess...
@@vfgdfg firstly the people they have are working on things that cannot be implemented into the game at this point. For instance the planet devs have been working on Nyx for SC, but also Odin and some others for SQ42.
The Ai teams have made some great leaps forward, but most players would never know it because the current server tick rate is too low. Hopefully server meshing will fix that.
Then you have the whole other game in SQ42 that’s been made, the teams involved in mocap etc. once Squadron is out, there’s another two games to make.
CIG is not just here to deliver SC and be done with things
1000 players in a single instance is incredibly impressive for any game but it is especially impressive for a game of this complexity.
The only other game that sorta gets close is PlanetSide that can do a 1000 players in a server but those maps and the interactions that take place are way less complex.
That is why pretty much all mmo's have different instances because otherwise the game would just crash/down to 1fps. The fact that it runs this well is very impressive.
Look forward to see them improve and make it even more amazing.
Yeah, complexity of tech is impressive and amazing!
But is it fun?
@@wudimusic
Developing game in alpha.
When I was alpha/beta testing past games(planetside,SWG,Eve Online,Earth and Beyond,etc.) the new tech was also complex and impressive for it's time. Nobody had much fun either except the super testing geeks.
I'm just playing,testing,reporting bugs, and enjoying the ride with Star Citizen. Never done this before.
You should find a finished game to play if you have issue.
@@blackworldtraveler3711The Games i play and enjoy have nothing to do with the state Star Citizen is in.
And neither games you metioned, have been in alpha for 10 Years or were in a better state back then.
I should know, because i pledged to SC over 9 years ago.
I played it for years, but now i hop into SC every few month only to have it not work and revisit in a few month after that, for the same experience.
They ´re at a point where they can´t stop the Ship-selling & feature-creep.
Because the Spice (Money) must flow.
@@wudimusiclooks like you triggered a white knight by asking if it’s fun. How dare you care about having fun in a video game!
@@ryanmitchell4044You don't lick cake mix then claim the cake tastes terrible. Let them make it first, then judge. Tech comes before balance and "fun." Welcome to game development of the most ambitious kind.
i was on server 080, 500 players on 4dgs. it was PERFECT. no interaction delays, no lag, no problem. it was smooth.
How do you know it was 500, when you can´t even see the number of players online anymore?
Also, SC never is or was smooth, at best it´s "sometimes less laggy"
@@wudimusic i had someone add me as friend so they could check.
@@wudimusicyou are wrong, it was smooth. Get over it
@@MathiasBronneshas visto el numero en un menú. No hemos visto 500. Tampoco hemos visto múltiples "fragmentos" con misiones activas o jugadores matando IA en un búnker. Aún está muy lejos está tecnología funcinando
This has been my concern about serve meshing especially static it doesn’t matter how many severs you have if one dedicated server can’t handle the traffic of the shard number it’s too big.
Loads of the (somewhat) smaller issues you had - party management, hab spawning, and performance with so many players in the same area are 100% blockers to having server meshing at 1000 players, however, it doesnt stop them doing exactly what they did and test the tech. I can imagine that even when / if they get it running perfectly (which, it sounds like, other than the above mentioned it did work to a certain degree) they wont release it until they have improved all those other bits to handle it.
Having the server meshing tech working is for sure a higher priority than a party managment system, seeing as most folks play with maybe 5 - 10 people at the most.
This was a great video - and very likely a fantastic insight for devs - please keep em coming!
Mind you, we are testing the static server meshing. Dynamic meshing should be able to solve this problem assigning multiple servers to a higher traffic area
When do they start testing this?
@@DarkSpaceStudios soon™
@@SaucerX 😂
@@DarkSpaceStudios given the delays...probably 5 years
Dynamic server meshing will just move the bottleneck to the replication layer if everybody gathers in one place.
It will be so cool when it works to see people in lines waiting for the tram or the rental panel, etc. wow, so cool go CIG make it happen
I've been saying this for years, but everyone told me that Im whiny, toxic or that I don't understand game development.
Every time someone mentions server meshing, I respond with - good luck with talking to everyone through discord, or buying a ship on new deal
There are so many scaling issues, lets say how many players are gonna be required to run a Carrack then if we are to believe some crew are npcs how many npcs per person will that be and how expensive server wise would those crew be per person....
You get called toxically whiny because those are the definition of good problems, but your grumpy👁️ was never meant to see it that way 😂
Your observations are a surprise to nobody including CIG. They're not intending for nor surprised at the possibility of 100+ people lining up to purchase ships at New Deal at this stage of development of SM. Scalability of in game services was not the point of these SM tests.
As for you being whiny, that's just your personality and that's ok.
@@gk.4102 exactly what Im talking about^
@@gk.4102 If you consider pointing out massive, fundamental, core problems to be "whining", then I sure hope there are some really, really whiny people in leadership positions in CIG.
Shoot when there was more than 350 people the wheels start coming off the wagon. They didn't even get close to 1000 so very much so a long ways away
Honestly, the bugs and desync make Star Citizen look like an early access game that's only been in development for a few years. If you had no knowledge of the development, it might be quite a shock to learn that this project has been in development for 12 years with almost a billion dollars in funding.
The game was testing new tech whilst deliberately trying to break it.
I must admit, it’s exciting watching this technology get rolled out and tested in real time.
This game needs another 10 years in the oven
I mean I think its still gonna take many years but this is such a strange comment to make on such a insane achievement
@@TKanal3Only 8 years late
@@mightyfineincredible2252 yeah they totally announced server meshing done in 2016? 2016 was the forst time afaik that they TALKED about it. The first time they implied a release date was 2020. So 4 years late. Typical chris roberts date. But why would you lie. Just a hater
@@mightyfineincredible2252 they could have published this game long time ago but then it would've been something more like StarField. Only a few landing points on few planets, loading screens etc. but since they want to make it something so much better, its gonna take more time obviously
Oh it will be, don't worry about that.
I just want to point out that static server meshing isn't meant to handle concentrations of players. That's for future dynamic meshing!
This honestly isnt a good sign. Its clear they have no idea what they need for the level of players they want end-game
Seriously. They were supposed to release Pyro back in 2020. They just admitted in an SCL this year that the devs haven't even decided how the law system will work in Pyro. Are you kidding me?
How is that clear?
@@QuotidianStupidity By having like, so few hangars in a space station, that you will need +8hours before taking off if there is more than 100 players on it and is the last on the line ?
@@karakiri283 that isn’t a clear sign of “having no idea what they need for the end game”.
It’s a sign that it’s not right for now, but it is going to be very easy to make changes to stations and ports.
Until they know what the limit is for each shard, the popularity of each sector, planet and station within that, player habits and what the underlying tech for ATC replacement etc can handle then there will always be areas that are not fit for purpose whilst they are deliberately stress testing the tech by increasing the player count so dramatically.
@@QuotidianStupidity bullshit, a poor excuse for them not doing proper level design to begin with. Reworks costs many millions and much time.
you pulled off an absolute banger of a test, especially in a QA environment. I've been in games where it was difficult to get a 100 people to coordinate.
The main problem CIG has had since this game's inception was not making server meshing the architectural priority from the start. They knew from what was functionally day 1 that the end goal was MMO with a world with a ton of active players in one "server". They could have built their own engine from the ground up to support this (they later down the line realized they needed to make their own engine anyways), and then designed the rest of the game on top of that. Instead they spent **years** on flashy simulation features that no one really asked for (super detailed ship damage models that don't work 90% of the time, wild amounts of character customization that no one asked for, expanding the map size to be huge before it could actually support enough players to not make the huge map feel empty, the list goes on). Star Citizen currently is an amalgamation of tech demos, some of which are admittedly very cool, but their entire structure of this game is fundamentally preventing them from ever reaching a finished product. It seems almost every patch they give us flashy new features with little actual utility, don't get us any closer to a game that is actually playable and enjoyable, all while kicking the fundamental technical problems down the road.
I'm starting to think that this game is not architecturally salvageable and if they ever want it to be a real, playable MMO with thousands of players in the same server-mesh, with good UX, and enjoyable gameplay loops (yk the things that actually matter in a video game, not accurately modeled hair physics), they may genuinely need to tear a lot of the game down, start again with the goal of MMO servers, and then re-layer things on in order of importance, lest this game be doomed to remain a forever tech demo. Lots of cool snippets here and there, but nothing coherent and complete enough to actually enjoy and play regularly.
- someone who has spent hundreds of dollars on this game, and have been watching it since day 1.
The issue was this level of fidelity with FPS and planets was not part of the scope.. The FPS game play was supposed to be limited and not AAA. When that was rethought and the planet tech came on board everything changed. This is context that you arent realizing. And when you start with 8 people Im sure building an engine was not a affordable consideration. No current game with this level of fidelity supports 1000 players on a sever or world. Period. So people need to temper their expectations and be realistic here. Cramming players in a single zone like this is edge case at best. SC will have limitations, however that should not be considered a failure or unplayable. This isnt WOW with low poly models and minor physics to worry about... And planetside also is low poly by todays standard and really is just a battle plane.
Someone within CIG prioritized hunger, showering and actual shitting mechanics (no joke, that's coming with pyro) over the basic fundamentals. Guess who that was! And guess how many people are defending that decision. XD
Best comment 👍
That's a lot of words coming from someone who doesn't seem to know very much about the project
Over the last decade they have implemented piece by piece new underlying tech that enabled them to get to implement server meshing right now (OCS, SSOCS, PES etc.)
They chose Cryengine as the engine for this game because they also got to hire some of the guys who literally built that engine from the ground up at Crytek, so they could rebuild the engine to support SC's vision, quite a few of these people are still working at CIG rn
Also, what map size are you talking about? The planet sized maps? You think they should have made extremely tiny planets/moons or what?
Also there has been no focus on an "extremely detailed ship damage model" until the last 2 years with the Maelstrom system, and it's still in development and hasn't been implemented in any form in the PU
In 2012 it was still standard MMO instancing with fake on-rails animations between instances. They should have sticked with that.
Definitely needs a unique verification code to self destruct your ship, so it is not just ‘backspace’ and then it is game over.
i dont even know if i want 1000 player servers anymore. what a nightmare the tech behind this is. maybe it's time to cut losses and focus on making a good game instead of focusing on the scaling all the time. The Port Ollisar style of gameplay was cool anyway, felt more connected and personal.... now we're here with instanced hangars and copy pasta locations everywhere to accommodate the masses. when scale overtakes individuality, things lose meaning. e.g. Jump Town.. invented by players, ONE specific spot. Now they copy pasted it to multiple places. also fleet battles suck anyway. 1v1s or 3v3 is where its at. where is the point of scale, when everything will end up in "instanced-mass-theme-park"-style gameplay anyway (and doesnt even work)?
(I was not involved)
I love seeing the progress and people actually trying to organize to test it. Thanks
Morphologis out here being a an actual better beta tester than a normal player
The communication problem you encountered after the crash is the single reason I think whatever voip Spectrum inevitably includes needs to be its own desktop client, separate from the SC client itself.
"Star Citizen Server Meshing WORKS, but..." Is like saying you have a racecar....but every time you get into it it catches fire. Nice to see an actual video on this in practice
I have a rule that I don't play games until first patch after full release. 😂
A solid rule that will prevent headaches in the long run.
It’s a matter of choice:
Drive the change vs wait for something big happen and miss the chance to been part or it!
Either are good, just be sure to pick the best one for you!
This game is hilarious. It's such a massive clusterfuck that it's almost like watching house burning down veeeeeeery slowly irl.
Fans still act like everything is fine but it doesn't change the fact shit's burning down;D
It's a house of cards built upon a Jenga block, built on sand.
just stop following it.
@@lostvayne9146 Why would I? I just said - It's hilarious
Yeah dude it's totally gonna crumble and fail in the next 2 years just like everyone said in 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024 etc. 😂
Meanwhile not a single AAA developer making billions every year is able to replicate even a single tech that SC has at the same scale
Maybe the only exception being MSFS 2024 with their Earth, but we'll see to what extent
@@gian.4388 Yeah dude, it's totally gonna change the gaming world like fanboys were saying... in 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024 etc. ;]
Meanwhile bunch of AAA games already surpassed it when it comes to technology.
And MSFS will be an actual game... You know... With a RELEASE date and all;] When's that happening for star citizen?
There is a related factor that seems to be glossed over at present - the limited number of regional and localized servers.
Currently, it appears that SC servers aren't dynamically enforcing and capping or adjusting the client-side draw and render distances of global objects and effects - based on server load. If it is I'm yet to see it.
The consequence is clear - even players with very powerful rigs get extremely poor performance.
Take my situation for example - I'm 6k miles (9.7k km) from the nearest SC server. Any lag in my connection impacts my group, and all players on the server. This issue exponentially compounds problems with server meshing, persistence etc. It impacts all network related traffic especially streaming. More client-side objects would help reduce traffic.
Dynamically enforcing local view and render distances based on server load and. kicking players with high ping times and jitter would help a lot. I'll take one for the team.
But ultimately we need more regional and localized servers and a lot less streaming data.
If we all lived in Japan, we could do it. Infrastructure there is insane.
I have 5gig symmetrical at my house but I know most people have shit internet and how far people are from each other and servers.
Which means if the server fps is on par with what latency does. You probably have realy smooth experience as seen in many videos from testers.
not the infrastructure hes talking about... hes talking about how the game wasnt designed to handle this many people. the most obvious being that there arent enough habs to spawn in or only being one elevator etc... the game was designed for a smaller population, throwing a thousand people into that will cause backups in a lot of places. there will be lines to access stores, elevators, asops, etc... the problems they are having here have nothing to do with your internet lol
@@magicalfungi3206 no shit. Server infrastructure works best when you look at physical location, physical hardware and software/coding. And everything in between. If you have an amazing server but someone in china is using it, it won’t be a great experience.
It doesn’t matter if everyone has 1 billion gigabit internet. CIG servers can’t handle that much data to 1000 players each individually
@@busterscruggs_sc can you read? It’s what I said. The bottleneck at the end is physical. Not code.
They can make the best code on the planet but it won’t work with a 300ms
How they could fix the issues you raised:
1. Clear up the UI mechanics (like they are doing for ship MDFs) so you can turn on/off party members, or have a proximity for certain distances for short/long range toggle on when they appear on your UI.
2. Have the habs instanced like the hangars where the game world creates duplicates and instanced habs. Gateway the doors like hangars.
3. Dynamic server meshing (my hope is the game will adapt and create more servers/spaces when the replication layer can see bottlenecks in the DGS like you experienced.) More servers in smaller spaces like Hub areas are probably key for overall improvements.
The game has to be optimized and Im sure CIG know of these constraints.
We can't expect server meshing to be perfect from the beginning, do you remember when we could not stay connected to a server for 10 minutes without experiencing a 30k? Now 30ks are much less common than before, they only happen from time to time even on servers with 100 players on. The idea behind server meshing is great, give it time, it will resolve a lot of issues and allow 1000 players and more at every location.
Your positive attitude will not be appreciated here son.
Charging 1ks$ of money for a game that doesnt work is just insane to me. But keep coping. I dont even play this game just stumbled upon this vid. lol
@@dragoncpu2226I'd go one step more and say even for a finished game it shouldn't have anything that charges a grand for.
But again, no one is forcing you to do so. And in term of SC they are mainly gathering the funding from whoever can throw that amount of money away rather than selling some product.
@@dragoncpu2226 SC is $45 bucks
It's been over a decade all ready, "give it more time"....lol
So the problem is that it's too realistic when you try to funnel hundreds of people through a facility and into a ship? It's behaving how it should, it's a space sim.
This game ate three sized 8SCU containers after I loaded them with all my best stuff and transported the crates to a new space station.
As soon as I sent the elevator down to store everything, all 3 crates just disappeared themselves.
All loot gone, and it took me 40 minutes to move/load everything!!
Fuck your drain and water analogy, I’m done wasting time on this project.
They should have called the game Project 2050 which represents the real GAME RELEASE DATE.
Fucking A.
Going out for a moto ride now!
Cya CHRIS!!!! Good riddance! 😂
Have a good cry about it and we'll see you next week.
@@Apav1x Still out riding. Shits a blast.
I've been convinced for a while that CIG is going to end up separating people into more shards (when they're in the same area) and instancing more content in the final experience. Mountains of physics and detailed dynamic surfaces coupled with hundreds of people and the expectation of being able to have functional interaction/combat is a tall task. It's not impossible, but like with anything, the more variables you add to it - the higher the potential for issues.
I was part of this test! Next time let's do something a bit more "scientific" and not waste hours standing around.
i’m interested to see how they change city layouts and increasing the capacity of the transit system to account for the increase in average/peak players
Star citizen is at a rate on getting the epic fail stamp approval from me. And this comes from a person who has been sitting here since the beginning.
The problem here, it went from an idea... to going total apeshit... and here we are 12 years later lol.
You can't even get your network code to work proper...
What are you talking about, the test was for 500 players and it was fantastic.
The stress test which is to take it to failure was for them to learn, and improve just as they did from tests 1 & 2
Moreover stating “you can’t even get your netcode to work properly” is meaningless.
Netcode actually work, otherwise you couldn’t login and move around station.
The fact here is they are improving the netcode to support more ppl, reduce lag, rubberband, increase stability and so on.
For a car 🚗 to work you must let that engine to stay on, THEN you can add HP, reduce fuel consumption, reduce noise and so on.
And it’s exactly what they are doing, the only difference is that they let you sneak inside the labs instead of simply wait in the shop to see the final product.
@@HārokkuKyaputen why are you protecting something that doest even work so hard I wonder?
@@MartinDlabaja just elaborate better your meaning of "doesn't even work".
Because for me it's thing that is completely broken, instead SC it's working, with flaws, but it's working, and it's in active development... so, for what I can see.. it's working rigth now
Thanks for mentioning that the point of the 1000 player test was not to see if it would work, but to see how it would break. Honestly, I'm surprised it worked as well as it did.
While I agree that it's a good sign that they're testing, I wouldn't be too worried if they had periods when they weren't testing. If they feel they have sufficient information to have benchmarks and make changes, then it makes sense for them to shut it down while they make those changes as the additional information isn't really necessary. Then, after the changes are made, they can test again, get metrics, and see when and how that build breaks.
It also has the knock on effect that people dont get "tired"/"used" to server meshing
if they keep server meshing as a bit of a mythical thing, odds are they will have more people show up to these tests, which helps gather more data
Just soft pedaling the fact that it doesnt work. Classic bs.
8:04 I only can imagine what whas going on in a globalchat audio channel at this moment.
Even in THIS test, when they went about 500 players. It took 30secs-1+minutes just to get a response for turning on my ship.
That's just not good enough to surpass 500 players.
If they thought it was anywhere ready it would be on PTU not on tech preview . They know it’s broken they must want to see where pand why. IMHO they just need to do 300per shard just to shut up some doubters , make it work and then slowly up it while optimizing the actual game including bed counts or at the very least que people in using the time it takes them to leave the habs.
Good theory. But you cant scale forcing a bus through a hairline fracture.
@@robertchandler2063 I choose to remain skeptical on the scalability of this model, for now.
1000 people on the same shard of static meshed servers for the Stanton system is a good goal. They will definitely need to get Dynamic Meshing in soon to handle bottleneck areas and load balancing needs.
Not to mention when events like Xenothreat pop up, it needs it's own server.
Weird that you had that and others didn’t.
Nah, I recorded my entire sessions [3:500, 6:1000, 4:500].
It never took 30 seconds to do anything. Quit exaggerating lmao.
I do love the massive diversity of gear and looks. No two players looked the same.
Always love your videos Morph! Always keeping the quality high
Thanks for keeping it real. Some people are truly delusional when it comes to what can and can’t be done with their tech as it stands.
Remember when star citizen was going to let players host their own multiplayer servers, and it would have probably been 100 or so people in a server?
I bet that tech would have been working by now...
Yeah and it would not be a good time investment and would not be a good game neither.
That was a promise early on, but as a software engineer i knew this was NEVER going to be possible to run the whole PU for the majority of people. I have hosed multiplayer servers of my own for things like 7 days to die or ark. In these i had maybe 20 people as a peak. Star Citizen is right now, probably 500x more playable space than 1 ark server so this means the only way private individuals willing to run a huge cloud cluster of servers which would be pretty expensive...and this is just to run santon.
So the only private server we may ever get would be to host a single system or just run arena commmander
@@MrBlackjimrogan yeah, that's because the scope of the game changed dramatically since the original pitch.
We were originally going to have landing areas we could go to and explore, but not whole planets.
The first person shooter stuff was not going to be as prevalent either. As the feature creep set in, it became clear that no normal computer would be able to host this monstrosity of a game.
Sadly, CIG have really put themselves in it now too. They need to invent completely new technology to manage the enormous server requirements, and that's expensive. Once (if) they get the technology working, they need to maintain many expensive servers to run that tech.
All of this adds up to a very expensive game to run, and they will need a constant flow of cash to keep the game running.
That puts the players in a crap position, because it forces CIG to adopt predatory monetisation tactics to keep the lights on.
It's all connected, and every year it gets harder to support in my opinion.
@@TKanal3 freelancer is still going strong, 21 years after it released. It's a fantastic time investment, and is one of the best space games of all time.
That has exactly the style of multiplayer that star citizen originally pitched. 100 or so players in a server, the ability to host your own server, and mod support.
@@funkyschnitzel absolutely correct, the one thing that is good about the server meshing and general approach of where we are now is it is scalable up and inevitably when player numbers drop back down. Im pretty certsin that so long as squadron 42 sells well we should have at least 5 years of money come in when that drops
They should have 2 versions of the game. A super stable version (maybe no persistent hangars) but super stable, then a "nightly" build for those who dont mind the instability but want the bells & whistles. Problem is, I dont think a wholly solid version exists.
So it doesn't work...
Morph isn't quite fair in his explanation or testing. It does "work".
The goal of SM is to use multiple dedicated game servers (DGS) to distribute server load. Each DGS will have a logical limit, meaning just like any game with the same amount of processing, it's only going to handle 100-200 players. Putting everyone on a single DGS is not the result of SM not working... its the result of Dynamic server meshing not existing to mesh more DGS and split the load.
If you watched any other streamer at the time, you would've noticed that SaltEMike had a low desync, 30 fps, reactive ai dgs on that same shard.
Summary? Yes it's "working" However RMQ is still a bottleneck and basic network optimization still sucks. Most of the issues aren't a server meshing problem.
@@ZaneDragonBornthank you for clarifying that.
"working" 😂😂😂
It works but it's not ready for implementation.
@@Dominus-Noctis 😆
This a great video morph. Really highlights the progress and the work still to be done. Territory we have not been in before.
Star Citizen is going to crash on itself like a tiny black hole and disappear from our reality ...
I'm not an expert but gathering bunch of people in specific location is not something current version of server meshing is suppose to help with. I mean, it will help with performance in other locations on the other X servers where your group isn't located but it won't help to your group.
Pretty sure what is supposed to help here is "dynamic" server meshing, where extra server gets assigned specifically to the overcrowded location on the fly, meaning the bar with 100 players in it would get it's own server assigned in real time, with border being probably located somewhere in the close corridors while the rest of the landing zone would be running on other server...and that tech is proly still light years away.
Can’t be light years away… simply because that is a measure of distance and not time.😅
Joking apart, your assumption are indeed true, but to achieve that you must consolidate the static meshing, with a solid message queue system.
And imo they’re on the right track…
Lol i screamed in chat that someone self destructed the 890 but none lissent 🤣 (zimboy) check chat lol 11:15
Getting this to work smoothly on live with thousands of players all doing their own thing at once looks like a pipe dream at this point.
They’ve gone from 100 to 500 in a short space of time, and they are still working on the system.
Given this was a stress test, why would you assume such a thing? That’s very negative for no reason
I mean, back back out in the olden days, Star Citizen was at the same point, just instead of 100 players it was sub-50, and instead of 1000, it was 100.
Similarily, a LONG time back far before even Star Citizen, people once assumed the Video Game Industry was only for nerds, and a niche market - now its a behemoth large enough to swallow the music and movie industries. Just because something is hard, doesnt mean its impossible.
A video I’d like to see from you is “an architect reviews star citizen Habs”
This would be cool to see Howd you’d design a hab to fit in the same space and provide real world solutions to problems such as not enough elevators and not enough habs.
If you needed a Star Citizen reality check, You havent been following very long.
I hope you made a spectrum comment/IC ticket going over your points. You have some great one regarding large parties.
It's "queue", not "que". There's "cue", but that has a different meaning.
Definitely need to make Space stations possible home locations to spread out the player base as a temporary fix for over crowding. But I’d like to see a rework of home areas or city additions to planets.
i got Nord VPN just because your SG1 shout out, that show is a Culkt classic!
Thanks for using your unique status to organize a community test like this - I really hope CIG got good data and that it's a wakeup call about everything that needs to change.
I’m losing hope for this game
Was not expecting the Stargate SG-1 shoutout. Such an amazing show.
Thank God starfield's an offline single player RPG.
Everyone was pooping on it but it isn't that bad even for a single player
starshit lol
Good stuff, mate. Exciting times!
12 years and still only 2% of the planetary systems done.
Imagine showing up to a video 3 minutes after release only to whine.
True but those 2% systems are 100x better than any other game right now. I would be happy with just Stanton right now.
stay mad
Literally clicked on this video and started crying like a bitch before it started. Hey, those systems also went from unlandable surfaces with max 2 locations or so for some planets to FULL PLANETS. Moron
So build thousands of car bodies before the engine so people can look at them, then realize the engine doesn't fit the car and rebuild everything?
I enjoyed the video. I don't "test" anymore since we're just so far away from what I consider a "game" that I don't want to burn out before it's here. So it's great to see videos like this to "keep my eye on the game". There's such a long road behind and ahead of us.
I mean, it's not going to take as long as you think
Bro spent half the video talking about issues with setting up his little record no one cares about as if that's the point of this test 💀💀
Well it’s a great way to test something like this?
There is a lot of unused space at these space stations. We just need more hangars and lobbies added to that empty space. It is hilarious that nobody at CIG thought of this beforehand, tho 😂
Or maybe they know this and are giving priority to build the backbone to support that later.
Like in chess… sometimes the best move must be kept for later time… just build the ground for that decisive move.
CIG future relies on this tech to work, as they want to license it for other mmo's.
I mean obviously this game will be very different 5 years from now but yes atm they need the cash
Always a high quality experience watching your videos. I trusted Ben after the first test because I know how new, better solutions can create a clusterfuck when just implemented in development. Happy that I did. I can see 1000 players working at some point with how improvements are going. (Tho i suspect improvements wont be as massive as from 3->4)
It was a success where they got their data. So what's your point?
Hey Morph. I know this is an odd request but i would LOVE to see a video on how you would make and furnish and house on the planet reach from halo. Like you did for star citizen. I love the compound/houses you can see in the winter contingency mission.
While people say threre are not enough hangars/terminals etc...We are still in 1 system.
The testing tested crazy things. If we did not know a test was happening we wouldnt have everyone congregating in 1 location for a start.
I think what you're seeing is that CIG expects that when they add more star systems, player density is going to be around the same as it is now, in other words around 100-200 players per star system. Otherwise all this time we've been hearing about the networking team getting servers ready to handle 1000+ players per shard, we should have also been seeing more elevators being added to habs, UI improvements to handle more players in a party, etc. Instead we're seeing progress in pretty much everything else but that, which means that CIG thinks it's unlikely that we'll see 1000 players all spawning in say NB at the same time.
Server meshing is cool, but your point on the non mesh systems that cause issues is something CIG needs to address. I know it gets old when I talk about ATMO, but we are the best test for these systems. The Daymar Rally has 1000+ people across multiple servers and last year was a nightmare, not enough habs, not able to launch because of too large of a party, and just the general issues you get with that many people being in one spot. On top of all that we have the organization aspect, We spend months getting ready for the rally and have to have a massive team to make it happen, while a lot can't be fixed in game due to the nature of the event, tools beyond the party system would be so helpfull.
They either need to add more terminals in cities to allow more people using them all simultaneously, or they just need to make cities their own servers with less people per server, and once you get into your hangar ready to leave, that is when you hop over to the larger servers. Then again, once SC will actually see such large numbers, and even now, there are and will be more cities and stations one will be able to spawn at.
You can self destruct the ship from the battle bridge on the lower deck so it may not have been him.
One thing about the "few elevators"... something that always bothers me is how fast the doors will close again. Like you can't really "hold the lift" for someone coming down the hallway.
I suspect client performance tuning will happen once the system is in place, and working properly on the server and network side.
I definitely agree on the idea of a long PTU cycle for 4.0.
Sadly, though, as we've seen.....as the PTU goes on, less people are there to test. So after the initial couple weeks, testing a big server config could be practically pointless when it won't ever get back to near capacity. It's something CIG has struggled with for a long time now, and could be a big reason 4.0 especially will launch with a lot of issues.
CIG should probably, as many have recommended, do testing "events" during that long PTU cycle to occasionally draw people in and re-stress the environment. They might even have to offer some kind of reward. But they really do NEED to do this, or it's likely they'll "think" some issues are fixed while it's in reality just less people loading the PTU.