Steve, That IS how it works! I’m retired now, but while I was working our data centers recovered from at least two ransomware attacks using robust backups. The affected systems are offline while the system is rebuilt, which takes many hours, even days. The trouble with paying ransom is the hackers remain in the systems, even if they make good on giving the systems back.
Also, many times these are timed. So, that means this could have been triggered months ago and when triggered, it sets a date in future and when that date comes it takes the system down. Which means the backups are corrupted so they need to scrub through the backups to be sure the virus isn't there. Also, most municipalities, hospitals and schools have ransomware insurance. The insurance company needs to investigate to be sure there wasn't something that happened that violates the terms of the insurance. It is a whole thing, not as easy as you would think it should be.
True. However, restorations are after the fact. The whole problem stems for incorrect security policies. If security was done right, very little to none of the machines would have been affected.
I work for a large MSP, and the way we usually deal with that is to rebuild everything from scratch, decrypt the data with the key (which does almost always work), make sure there's nothing executable in the decrypted data, and put the now clean data onto the newly built servers. It can take weeks to rebuild things.
Systems like you described already exist. The problem is that they're only "worth millions" to someone that's already being attacked. It's never worth that to them beforehand.
Offline backups already exist. Banks use them. The real problem is they don’t hire cybersecurity professionals based on skill. Instead they hire people based on communication skills and certificates.
Backups are not usually handled by the cybersecurity team, for the record. Sometimes they are, but often they're handled by the IT team or if a company is big enough to separate IT and infrastructure/devops/systems administration/whatever they call it at that company by the infrastructure team. Security should be involved, but sometimes office politics get in the way of how things should work. Most medium sized and larger companies used online backups for most of the time such things have been possible, but vendors are pushing cloud backups and all kinds of other things and I'm sure some people are making the wrong decisions. Restoring from online backups in the event of a ransomware attack that has compromised your entire network is not always straightforward, either. Because the backups may themselves be compromised, depending on how long the ransomware was on your network before activating. It can also be that you restore a system only for it to get re-compromised if you haven't fully cleaned up your network. A major company has competent and experienced pros to do this right, but a local government? Ehhhh I don't think they're paying enough to hire people who are competent at this stuff.
I used to work for the contractor that supported all of Wayne County's computer systems. If this company did their job which I doubt they will have full back ups of every computer in Wayne County. Resolving this problem should be a simple as restoring backups from the day before. But knowing this company terminated my services considering how experienced I was this was a very stupid thing to do on their part. It was all about them skimping on reimbursements. Triggered by the bribery between them and Wayne county officials. We shall see how this turns out I have hopes for Wayne County's Full recovery😇
There ARE such "offline" backup systems. That's how some organizations are able to restore their systems without paying ransoms. They still lose some small amount of data (such as a day's worth of transactions). The question is whether or not the org is willing to pay the cost of those systems.
The other issue is, even IF offline backups exist, the time it takes to restore can easily be weeks or even months depending on the organization. It becomes a balance of equities which is it cheaper to pay the ransom, or cheaper to rebuild the systems from backups. The more "offline" the backups are, generally the longer they take to restore ironically enough.
The biggest issues are how long would a system restoration take. And can they immediately crash the system again. Or is the hack one that can allow them to hide a virus or place a backdoor hidden in the OS that will leave the system vulnerable to any on of zero day exploits.
Don't forget there's also the question of how long ago was the system compromised. Many of these types of attack compromise a system and then wait weeks or even months to trigger. Thus ensuring all recent backups are also compromised, and making their ransom demand even more pressing.
The issue with a lot of backups is that if you are compromised that deeply, they usually get your backups as well. Usually due to bad internal security practices. Many counties have no dedicated cybersecurity guy, let alone team.
As an IT Professional it's ridiculously easy to protect yourself from this problem, lots of products make it easy the most popular and expensive one of course being Datto. You're right that many counties don't have dedicated cybersecurity teams, but they can and it's not even "that expensive" compared to what you might think. If you aren't big enough to budge a security team you can hire a Managed Service Provider to take care of that stuff. I've worked for many over the years I've had fire departments, police departments, ems departments (I think they're private but not sure), county clerks offices, and several public or private schools as clients. Also law firms, medical entities, and financial institutions. I believe if you CAN budget, you should have a dedicated internal team, but for those who are to small MSP's are actually a good option. I'll never be the guy who says "you should out source that" but if you have nobody... you should hire someone for security. Honestly the people in charge of the budget should be liable for this, there's not having enough money to hire a team of expensive professionals, and there's not having the will to do your job and make sure you're not exposing citizens to undue risk.
@nebula_rasa It's possible, but counter productive as the things such a system can do become so limited it reduces it's value in performing it's tasks. Now obviously that kind of means the system isn't something worth protecting anymore, it's a balancing act but it cannot be said to be axiomatically true that there is no perfect defence.
@@nebula_rasa Your right, it's not perfect. Problem I see is that they don't even TRY until it's too late. I always look at it like "The Club" (for your car), sure if a thief really wants your car they're going to spend the time to cut the club off and steal your car. But it will deter at least some people who are just looking for a quick payout and will move on to a car without it.
@@ValleyMansonOfficialand how long ago did the virus get in there? Also, every single computer would have to be 100% wiped before installing the backups to make sure it’s safe.
The City of Tulsa recently used $109 million from a “sinking fund” to cover the loss of that amount when the City erroneously wired $109,000,000 into a counterfeit bank account. I wonder if these municipalities and government agencies are using the pharmaceutical industry tactics to settle known problems down the line because actually fixing them at the outset would be logical and ethical.
The protocol is to wire a few cents first, then verify it is received. And then wire the rest to the same account using the same wiring method. So you lose an extra $40 wiring fee, but that is well worth it.
I am a systems engineer and have worked in IT of over 30 years If you store the backups offsite or in the cloud you can relatively quickly restore everything to a working state. Also as long as the firewall itself is not compromised in which case you would need to reset it to factory defaults and reset the config (which shouldn't take too long if you kept backups of the config.) you can be back up and running in a few hours for critical systems and a few days for less critical things. Of course it all depends on what you spend on it. Back when I did consulting I always asked my customers when they had sticker shock "How much is it worth to you if you lose it and can't get it back?" Most of the time they would say something like "I would be out of business." My next comment would be "Then money isn't the problem."
I'm an IT Director and backups is a part of disaster recovery. If they don't have that in place then people should be fired. I have multiple backups including cloud planning for the worse case scenario.
Happened here in Curry County, OR. I live in the County seat of Gold Beach, OR - population around 3,000. They wiped out the county's entire digital footprint & affiliated organizations: courts, police, sheriff, title office, city, everything - even the county hospital, etc. Royal Ransomware got us here, heard they got Texas & elsewhere. Held the county hostage for months - total nightmare, except for rotten criminals including drunks, targeters (terrorizing witnesses, whistle-blowers, victim-witnesses), and traffickers working for those higher up behind it.
The Scranton Pa school district recently paid $25,000 in ransom and $50,000 to a computer doctor to resolve the attack and still hasn't fully corrected it completely. They lost Internet and phone service. Classes were taught the old fashion way .
Still cheaper than paying a cyber security specialists annual salary. The real question is what data did the attacks gain access to. The cost of such data loss/theft may be an unforeseen consequence. However, that is almost always the afterthought of bean counters because it isn't tangible.
There are offline backups. The industry moved from backing up to tapes (similar to cassettes and VHS, but different) to online hard drives. Now there is a movement to have "immutable" and "vaulted" backups that are deployed in an air-gapped manner.
The same thing happened to Patelco Credit Union a couple months ago in California. A lot of people had all their eggs in one basket and they couldn’t access their accounts for about a month while things were being restored. The ransom was NOT paid.
The weird thing about these armies of federal, state and local government bureaucrats, is that all of their work can be done on a laptop by one lady named Sue working from home in her pajamas.
Record a video of you trying to pay them, have them explain in the video they're unable to take the money due to a system error and that it isn't the fault of the person who is trying to pay their taxes. It won't prevent head ache's but it will save your bacon for being late. That stinks. I work in IT, if they told me an excuse like that I'd mention that then ask "Why don't you have a paper backup in place for when your system is down? Doesn't that seem awfully negligent? I understand if the amount is wrong once you're able to access it you'll need to notify me and have me come down within 48-72 hours but saying you "can't" when you won't just seems problematic". I'm fine with a man or woman tells me no, when the government tells me no to doing their job, I get argumentative.
Steve I work for government IT and have recovered from ransomeware twice now. I was back up in less than an hour because everything is virtualized and nothing was exfiltrated. Services were not completely up for about 3 hours though as we have to verify that the backdoor or dropper isn't installed on any of the servers otherwise it will just happen again. But with virtualization the backups are back up with in a matter of minutes. The problem is they dwell in your network for weeks or months now and you need to make sure that your backups are clean. I am a small local government however which is why it didn't take long I could see this county taking a day or two to fully restore services. The great thing about virtualization is the other virtual servers aren't deleted or shut off either just removed from the network so the evidence is not destroyed.
Just yesterday while doing some work on the west side of Detroit I had them call 911 for an emergency at the back of the building. They were unable to connect with 911 from the landline and it took me another 10 minutes to connect with 911 on my cell phone. So this story actually answers a lot. There is also, just so you know two solar flares that just occurred in the last 48 hours that will hit today between about 5:30 and 7:00 p.m. today. Or 10:30 and midnight Greenwich Mean Time, and the second one is supposed to hit sometime tomorrow but there's the effects are supposed to be gone by the 6th. Those could cause some issues as well
Similar to Wayne County, Michigan, Columbus, Ohio experienced a significant ransomware attack on July 18, 2024. The attack, carried out by the group Rhysida, caused major disruptions to city services. The city is still in the process of recovering from the incident.
It's your tax dollars that fund your under-funded governments so that they can't function properly. Pay more taxes and you might get the government that you want and expect. Or are you one of those who prefer no government at all thinking you can get along just fine without it?
There are several issues. But the primary is money - it’s not that off site and frequent backup systems don’t exist, it’s that these places are usually underfunded as it is. Security, redundancy and backup is expensive, and in the case of for example hospitals there are many companies who provide services off site - so even if the hospital security is top notch, the service providers might not be. At the end of the day, someone has to sit down and say “do we want to handle 15% less cases/patients/customers forever, when we’re already behind, in order to have security which won’t be needed and will only slow us down 99,90% of the time?” Now the security professionals know what they would answer, but it might not be the same answer as an administrator or CEO might give. And then even if you choose security, it takes a lot of time to work out systems and procedures that don’t end up introducing even more issues. The most expensive part is not even all the IT stuff. It’s the constant training, vigilance and focus on it by every single employee. Every employee is a gaping security hole waiting to be exploited. Even and sometimes especially IT employees. Source: I work in IT at a hospital.
I was IT for 30 yrs. My question has always been, why do you need internet access if everything is on an intranet?! The CEO and CFO were big internet users and usually the ones getting compromised and the ones axing the intranet only policy.
A year or two ago in Ireland some hackers basically shut down our entire health service. They had cyber security experts on the news saying the government should pay the random haha. And they did! Seems like a serious problem...!
Backup systems definitely exist. The issue is that most people responsible for allocating funds for most any kind of organization are generally not well versed in IT and people's in charge often balk at spending money for a backup that they "might" need.
What you mention with offline backups, absolutly possible. The issue is it cost money. The company I work with finds it hard enough for clients to install dev servers, to verify updates don't break anything on prod at the very least.
Yeah, that is how it works providing your backups are setup and configured correctly. The process of wiping everything from all servers and workstations and restoring it all isn't exactly an easy task, but definitely cheaper and better than paying ransom to criminals.
As a computer programmer I say it is possible to create a backup system that keeps a separate isolated log of data transactions and sustains a known good backup. The good backup can be brought up to date with the isolated log. And it is also possible to isolate the data on a separate system than that used to transact business, and have a backup of the system so that if the system is infected you simply delete and re install the backup system and you are all done. The problem is that many of these systems are very old, some running on no longer supported Windows systems and hackers know the hacks that have been discovered since that OS was last updated. When an operating system announces it’s cut off date for support, that is when every government system using that operating system should be updating their software.
In theory. The forensic teams would have to go through and determine what the scope of the breach is. However, city clerks maintain their voter roles and the paper copies of the edits, additions, and deletions would make it possible to ho back to a backup and reprocess the forms since the backup. Not to start a conspiracy, but it would be a reasonable target to do some voter role shenanigans in Wayne County and a select group of predominantly Democrat leaning counties/cities to get the state to go red because of the population density of the areas. Even if not successful, the formerly discredited voter fraud claims would suddenly have a place to point to as a credible example. Either way, this could open up the can of worms because it could swing the state to red, will convince others that fraud is possible, solidify the people that already believe the conspiracy to be unconvinced in the face of overwhelming evidence to the contrary, and some will never believe a government investigation of the government.
There are often offline backups available in these situations. The problem is restoration takes a significant amount of time. Not something that gets done in hours. It's a days or weeks long process to restore everything. You need to do a forensic analysis to determine when the hacker intruded the systems which is often weeks or months before they executed the encryption and asked for ransom. So you need to disconnect everything and restore things, but also make sure you don't just restore systems with the malware that was on the systems when they were backed up.
Backups that get pushed securely to an off-site location can be immune to ransomware. Thing is, you have then download all that data to restore. Depending on inter/intranet speeds, this could take a LONG time. That could be why they are still offline while they restore the backups. Another issue is that hackers can sit in the compromised system undetected for a period of time before they decide to attack. This means once you restore, you need to verify the restored system is not compromised before taking that system into production.
Just a note that hackers usually don't just attack when they first get in. They rifle through the newly hacked sucker looking for the important info and the backups. Once gathered it's then that they spring the trap - encrypting the files. It's only after this that the saps are notified of their new misfortune. I've seen networks that were hacked 6 months prior to taking over the network. Most of these hackers are patient.
The true issue isn't that you need a backup but that your operating system is compromised. That means accessing any information is not working so you need to defeat that before loading backup information.
The problem is these attacks are not execute day-of-infection. They wait MONTHS. Now the infection is so deep in the backups you can't get rid of it without losing millions of records.
My employer does exactly what you're describing, of course for a fee. We can restore data to yesterday pretty quickly, then they'd just need to figure out what was processed today and needs to be re-entered. A lot of organizations choose to risk it and think they can manage this stuff in house.
It is exactly the way you said you'd think it is albeit the servers that I know of are Intel. The admin should be able to swap in another hot swappable unit and have everything up and running in the time it takes to drive to the facility. If the network administrator is on site it's already up and running. Otherwise they may try remote fix however that could be the back door a hacker accessed.
My first reaction to this story is we need to change the laws. We need to increase the penalties for these actions. With the computer era the seriousness of those crimes has exponentially grown and so should the penalties! Minimum 20 years in jail for for these crimes. There is no fine that will do justice for this.
I used to work at the Henry Ford library and museum as their in-house computer support. They have an extensive backup system where the backup tapes are Fedex out too a storage facility daily. It worked fantastic and tell the person that supported that hardware wanted my job then he sabotaged it. I'm hoping he repaired it when he got my job. You might ask how did I lose the two prior jobs they're both self-explanatory. I am an extremely honest person and contractors do not like honest people because they can interfere with their profit flow.
I have not worked in government information technology since the 1980s. We did do backups then and had audit trails of the work that was done by people on remote terminals. The world wide web may be a marvelous thing, but there is no real way to secure a computer that is attached to it. Even if you can't get past all the hardware and software lockouts, humans that have access are still a weak spot. When I was working in company IT, uptime was stressed. I remember writing a backup routine for a corporate database, the backup seemed to run fine, but when I asked to test it by doing a restore, I was refused by the manager because it would affect uptime for the users. I had no actual assurance that the backup ran correctly. We waste trillions on military hardware that sits around and does nothing, and yet spend very little on tracking down and eliminating cyber criminals that are attacking our data systems Every Day.
Wayne County, MI. That's where my 1968 Ford Galaxie 500 convertible, my 2019 Ford Ranger, *and* my 2022 Ford Bronco were built. 🙂 The Wayne Assembly Plant.
Banks have had it for years it's a government requirement. When I worked for a financial institution in the 80s, we had a hot site four states away. If we went down within an hour the hot side had us running at full capacity
Steve, they SCAN checks to deposit them nowadays. That is why I always ask for a receipt so if the county can't scan the check, I still have proof. Unfortunately, if the computers are down they CAN'T print receipts!
Anybody else immediately think “County Election Board”? Whether such an attack on Election Day would affect voting or not, it is exactly the kind of thing that could be used to sow distrust in the results. Perfect timing, too: long enough lead time to be useful as a dry run; not a lot of time for the County to patch up security and thoroughly test it.
"There must be a way to back everything up in case this happens..." There is. The problem is what you later said--it costs money to do that. And higher ups see it as a needless expense, thus a cost-savings they can remove from the budget to either come under budget or make them look like they're saving the company/government money.
For those drunk drivers, I would think they went cold turkey with their habit after that ordeal. I mean seriously, what are the odds, a fire and no backup….
Steve, not only that IS how it work, my job is to do exactly that. There is a computer that periodically checks if there is a ransomware attack. If nothing is detected, the vault opens up and data moves in. At no point is that data exposed to a compromised system or network. No..does it cost a lot of money. Yes. Everyone chokes on the cost of prevention until they see the bill for recovery.
That is how it works, it's called Disaster Recovery Steve. They can literally have everything uploaded to a cloud and pulled back up from alternative CPUs... These state entities are not investing in operational continuity plans.
If people can't pay their taxes due to the county's poor cyber security, they should not be penalized. There should be a grace period for some time after it gets fixed.
I am one of those cyber security experts who would be tasked with mitigating the vulnerabilities through one or several preventative or detective measures to ensure that weaknesses like this can not be exploited. I can tell you that in our profession we don't typically have the authority to require that the safeguards are put in place and can only raise the awareness to IT management who in many cases ignore the computer threats in favor of getting getting software and OS revisions out the door even if they have missing patches and the software has not been security scanned or approved to go into production. In addition, a "disaster recovery plan", to insure continuous operation (although they are well known in the industry) are rarely implemented since they cost extra and organizations would rather pay the price in what they think is the unlikely event of an outage or just buy insurance to cover the expenses. I could go on and on since I've seen it all before, but you get the idea.
Restoring backups is only part of the issue, First, they have to identify HOW the attackers got in, patch it, then verify that the vulnerability has been mitigated. They then have to restore the backups in an offline environment and verify that it doesn't re-open the vulnerability. Only then can start bringing services back online. This process can take weeks, depending on the situation.
Modern computer systems use storage technologies that utilize Copy On Write (COW) to protect data, and can restore data back to the known good time/date. Besides that, the data is usually backed up offsite.
Steve, there are people who get together to discuss the types of issues related to information/cyber security, and they have consulting firms who will help to secure the network infrastructure. The problem is that the cities and private sector organizations often choose to skimp on cyber security. Their "risk appetite", or how much risk they choose to accept, is too great. They fail to see the need for security awareness training and audits, and as a result their systems are left vulnerable and they end up being exploited. Their disaster recovery plans are outdated or ineffective. Same thing happened in other big cities and counties... Atlanta and Baltimore in addition to healthcare systems. As technology continues to evolve, cybersecurity is going to become even more important and until these local governments and private sectors accept that then these ransomware attacks and data breaches are going to continue to occur more frequently.
The average American can barely afford to feed and shelter themselves, much less pay for any health care. How can they pay for the cost of effective cyber security? That's why companies don't incur that expense - because we, the people, can't pay for it in the needs we buy, forget about the wants.
First problem: How often do you make backups, and how far back do your backups go. An attack can sit on your system for weeks or even months before it is triggered. The backups might all contain the malware, ready to reactivate at any moment. Even if they don't, there's some point between your last good backup and now that will be "lost". Figuring out what data is safe and what data is potentially infected is basically a forensic investigation. Second problem: How low does the infection go? You run a program on your computer, and let's assume you have Windows 10 because a lot of people do. Programs you run are run as a "user" account. But occasionally you do something on your computer that Windows will pop up this fullscreen "are you sure you want to do that?" sort of message. That's usually "elevated" privileges, you're going to do something as administrator. A corporate/enterprise/government/agency level computer will have more permissions in between these, but there are others at higher levels. One layer above that is system privileges-that's the level of privilege your antivirus, crowdstrike, kernel-level anti-cheat, etc. runs. Kind of above that is the kernel/driver level where your video card, network card, mouse driver, and DRM protections live. I say kind of above that because system level is similar and there's overlap both directions. Potentially above that might be something called Hypervisor level which is the thing that controls "virtual servers" if your organization has those. Above Hypervisor is BIOS or UEFI level (the latter is technically correct, but people often call it the former.) Above UEFI level-yes, there's an above UEFI level-is system management controller level. Every Intel machine made in the past *shrug* has something called a System Management Controller (SMC) which contains a little 486-class CPU all its own, and it runs a secret (not very secure) OS complete with web server inside your computer. If something infects THAT, you're screwed, because that's the part of the computer that rewrites your CPU's individual instructions. OH YEAH, that's the final layer-your CPU itself, the chip that does the "computer thinking thing" is actually like a little computer inside your computer running a program telling it how to be a CPU. It's … more efficient that way, or something. But that program could be modified! So: How deep does the infection go? At what level was your computer infected? Supposedly there's a layer of protection at each and every level to stop you from attacking deeper and deeper into the computer … but attacks have gone all the way down to the SMC layer that we know about, and a fairly recent Mac vulnerability exploited things that were part of the Apple Silicon CPU that nobody knew existed outside of Apple and presumably the NSA, who might have had that information stolen from them or something. Even if your backups were perfect and made five minutes before the exploit happened, you have to figure out what was attacked and how deeply. Because if the attack is deep enough, good luck disinfecting the system. If anyone is still reading, I'm SHOCKED, but absolutely we know how to prevent these kinds of attacks: STOP PUTTING THE SYSTEMS CONTAINING THE DATA ON THE FLIPPIN' INTERNET. Attacks are still possible but they start to require spies and people breaking into buildings or convincing some idiot to pick up a USB stick on the street and plug it in or … something. There are ways to actually isolate data communication between systems so that a generic "infect windows and encrypt files" attack can't hurt the machines with the important data on them, but the technology "sounds old" (it is old) "isn't up to date", "costs too much", "isn't very convenient", "requires more money and resources", you get the idea. It's like your doctor telling you to stop eating three cake-frosted donuts and two triple espressos for "breakfast" every day to become healthier. Your doctor can know it's the right thing to do. Your doctor can tell you it's the right thing to do. The doctor can try and plead with you to stop doing that because you're freaking diabetic and on insulin with failing pancreas and liver, stop with the freaking donut breakfast OMG!! But you're the one that decides what you're going to eat today, not the doctor.
Offline backups are a thing, but they aren't always helpful in these situations. The bad actors occasionally sit in a compromised network for weeks or months before they pull the trigger on the ransomware. It's not uncommon to restore a system from a backup and then have it immediately start encrypting itself again. Sometimes it's a choice between going back a month or 2 or paying the ransom, and guess which choice most places will make?
That is how it works. Every sane organization does offline backups & has restoration procedures. We're dealing with government here though so they probably don't do it.
Oh, my god! Computers are down and no one can cope. What a disaster! That means having to go back to using pen and paper again! You will actually have to go there in person to conduct business. Can people even read and write anymore? OLD SCHOOL RULES! 50 years ago no one person could shut down an entire city. Regress with progress.
There are systems that copy/mirror data to two or more sites and systems and the lag time is relatively low. If one site/system goes down, loses power, or otherwise locks up the backup system(s) can take over and things continue to work. Of course, that's a simplistic view and additional protections and security measures are needed to overcome the possiblty of replicating a virus from the infected/compromised system to the parralel system(s). That is why you want to continue to have backups in addition to the mirroring system (s).
CONSPIRACY THEORY TIME. What if this affects the Voter roles... 'And theres no backup that includes the last 6 months of registered voters... 😮. I bet the media wont ask about it...
Probably done intentionally so 'they' can now walk off with all that property tax $ with no records 😂😢 Time for an independent audit of that city/town imho.
Those types of backup systems totally exist. Nobody wants to pay for them or the expertise to maintain them. I’d be happy to make Wayne county immune to these types of attacks, but I guarantee you they won’t pay a reasonable salary, won’t purchase the appropriate hardware and software, and wouldn’t give me the authority to make necessary procedural and policy changes.
How could they NOT have a robust recovery plan for a county this large!? Oh it's WAYNE County.... Virtual networking, full backup once a week and incrementals done every day, stored off site. And configuration documentation stored on paper in a recovery manual, not in a pdf file. Would take a competent IT staff 2 days.... run 2, 12hr shifts, half the time. But even with all that, it would still take Wayne County three weeks.
As an IT guy with 40 years of skin in the game, what can I say? Phools will be phished, sadly. Idiots are pretty much the standard vector for such attacks and it's heartbreaking.
Lol how do you figure that for easy? Every system requires internet this isn't a factory it's a series of databases that talk to other databases, as well as remote communication tools, etc. This is why you're supposed to follow the 3-2-1 principal in IT, from Veams website just because I'm lazy and didn't want to figure out the best way to say it. Maintain three copies of your data: This includes the original data and at least two copies. Use two different types of media for storage: Store your data on two distinct forms of media to enhance redundancy. Keep at least one copy off-site: To ensure data safety, have one backup copy stored in an off-site location, separate from your primary data and on-site backups. You take the network hardware offline, the servers offline; bring in the backups and within 48 hours you should be back online.
@@GeorgeWashingtonLaserMusket I do exactly this, as a home user. The ONLY thing I don't do is keep the backup "off-site", but then again there's really no need to, the most I'm going to lose is progress on the latest video game I'm playing. You would think a business, or a government, who's data is much more important then my latest game save would be doing this.
That would require people to physically be at the air gapped system's location. That means the court would need to travel to the prison, or vis versa, to enter or look up data.
@@GeorgeWashingtonLaserMusket That is all in theory. I have dealt with several security breaches and they were all by professional organizations that move to cripple your backups before they cripple the network. Private fiber circuits are availible. We use them for franchises and multi branch companies. Most cyber threats take there time when they first get access they position themselves to do maximum damage. Ransomware is a multi billion dollar industry many run by actual companies over seas. Remove email, and internet from the equation and your networks only risk at the point would be an internal onsite attack.
That system type does exist so some ransomware installs silently then waits potentially months before activating so that any backups are also infected.
![FendiDa Rappa - Clock Dat ft. Shamar Marco [Official Music Video]](http://i.ytimg.com/vi/3NBU8KHdxYY/mqdefault.jpg)
Steve, That IS how it works! I’m retired now, but while I was working our data centers recovered from at least two ransomware attacks using robust backups. The affected systems are offline while the system is rebuilt, which takes many hours, even days. The trouble with paying ransom is the hackers remain in the systems, even if they make good on giving the systems back.
Yeah, why aren't their IT guys driving to the local bank to retrieve their LTO tapes??
Also, many times these are timed. So, that means this could have been triggered months ago and when triggered, it sets a date in future and when that date comes it takes the system down. Which means the backups are corrupted so they need to scrub through the backups to be sure the virus isn't there. Also, most municipalities, hospitals and schools have ransomware insurance. The insurance company needs to investigate to be sure there wasn't something that happened that violates the terms of the insurance. It is a whole thing, not as easy as you would think it should be.
True. However, restorations are after the fact. The whole problem stems for incorrect security policies. If security was done right, very little to none of the machines would have been affected.
@@finkelmana True.
I work for a large MSP, and the way we usually deal with that is to rebuild everything from scratch, decrypt the data with the key (which does almost always work), make sure there's nothing executable in the decrypted data, and put the now clean data onto the newly built servers. It can take weeks to rebuild things.
No one invests into cybersecurity until something bad happens
Even then they tend to think they can just buy equipment to be safe instead of more gollies to keep things up and stop more shots.
Nobody helps us when they attack us, millions lost to those hackers from us poor people.
Welcome to our World.
Systems like you described already exist. The problem is that they're only "worth millions" to someone that's already being attacked. It's never worth that to them beforehand.
Offline backups already exist. Banks use them. The real problem is they don’t hire cybersecurity professionals based on skill. Instead they hire people based on communication skills and certificates.
DEI = Didn't earn it
Backups are not usually handled by the cybersecurity team, for the record. Sometimes they are, but often they're handled by the IT team or if a company is big enough to separate IT and infrastructure/devops/systems administration/whatever they call it at that company by the infrastructure team. Security should be involved, but sometimes office politics get in the way of how things should work.
Most medium sized and larger companies used online backups for most of the time such things have been possible, but vendors are pushing cloud backups and all kinds of other things and I'm sure some people are making the wrong decisions.
Restoring from online backups in the event of a ransomware attack that has compromised your entire network is not always straightforward, either. Because the backups may themselves be compromised, depending on how long the ransomware was on your network before activating. It can also be that you restore a system only for it to get re-compromised if you haven't fully cleaned up your network.
A major company has competent and experienced pros to do this right, but a local government? Ehhhh I don't think they're paying enough to hire people who are competent at this stuff.
@@bdp-racing And lowball pay and benefits. IT has always been an expensive black box to the business people; a cost to be reduced.
Not only that but these things cost money and nobody wants to pay for it
"The real problem is"
...they're not willing to pay the cost of avoidance until they've already been compromised.
I used to work for the contractor that supported all of Wayne County's computer systems. If this company did their job which I doubt they will have full back ups of every computer in Wayne County. Resolving this problem should be a simple as restoring backups from the day before. But knowing this company terminated my services considering how experienced I was this was a very stupid thing to do on their part. It was all about them skimping on reimbursements. Triggered by the bribery between them and Wayne county officials. We shall see how this turns out I have hopes for Wayne County's Full recovery😇
That assumes the lockout happened as soon as they took over and didn’t wait to make sure the virus was in the backups.
There ARE such "offline" backup systems. That's how some organizations are able to restore their systems without paying ransoms. They still lose some small amount of data (such as a day's worth of transactions). The question is whether or not the org is willing to pay the cost of those systems.
every real organization uses offline backups, going back at least 6 years.
The other issue is, even IF offline backups exist, the time it takes to restore can easily be weeks or even months depending on the organization. It becomes a balance of equities which is it cheaper to pay the ransom, or cheaper to rebuild the systems from backups. The more "offline" the backups are, generally the longer they take to restore ironically enough.
The biggest issues are how long would a system restoration take. And can they immediately crash the system again. Or is the hack one that can allow them to hide a virus or place a backdoor hidden in the OS that will leave the system vulnerable to any on of zero day exploits.
Don't forget there's also the question of how long ago was the system compromised. Many of these types of attack compromise a system and then wait weeks or even months to trigger. Thus ensuring all recent backups are also compromised, and making their ransom demand even more pressing.
@@cgi2002 That's true, but if you're just pushing the data, rather than imaging a drive, it's less likely to be a problem
The issue with a lot of backups is that if you are compromised that deeply, they usually get your backups as well. Usually due to bad internal security practices. Many counties have no dedicated cybersecurity guy, let alone team.
BTW… I use Arch
As an IT Professional it's ridiculously easy to protect yourself from this problem, lots of products make it easy the most popular and expensive one of course being Datto.
You're right that many counties don't have dedicated cybersecurity teams, but they can and it's not even "that expensive" compared to what you might think. If you aren't big enough to budge a security team you can hire a Managed Service Provider to take care of that stuff.
I've worked for many over the years I've had fire departments, police departments, ems departments (I think they're private but not sure), county clerks offices, and several public or private schools as clients. Also law firms, medical entities, and financial institutions.
I believe if you CAN budget, you should have a dedicated internal team, but for those who are to small MSP's are actually a good option. I'll never be the guy who says "you should out source that" but if you have nobody... you should hire someone for security. Honestly the people in charge of the budget should be liable for this, there's not having enough money to hire a team of expensive professionals, and there's not having the will to do your job and make sure you're not exposing citizens to undue risk.
They should 2 back-ups On-site and off-site physical, all my IT contacts I have setup for both.
@@bdp-racing No one cares about your backwards version of linux
@@justindionne8744 Hot sites are expensive. Cold sites take time to get online and need good internal practices to be useful.
No excuse for cyber attacks beyond not caring enough about network security.
This is objectively untrue. Every security professional will tell you differently. There is no such thing as a perfect defense.
@@nebula_rasa*"laughs in offline backups"*
@nebula_rasa It's possible, but counter productive as the things such a system can do become so limited it reduces it's value in performing it's tasks. Now obviously that kind of means the system isn't something worth protecting anymore, it's a balancing act but it cannot be said to be axiomatically true that there is no perfect defence.
@@nebula_rasa Your right, it's not perfect. Problem I see is that they don't even TRY until it's too late. I always look at it like "The Club" (for your car), sure if a thief really wants your car they're going to spend the time to cut the club off and steal your car. But it will deter at least some people who are just looking for a quick payout and will move on to a car without it.
@@ValleyMansonOfficialand how long ago did the virus get in there? Also, every single computer would have to be 100% wiped before installing the backups to make sure it’s safe.
The City of Tulsa recently used $109 million from a “sinking fund” to cover the loss of that amount when the City erroneously wired $109,000,000 into a counterfeit bank account. I wonder if these municipalities and government agencies are using the pharmaceutical industry tactics to settle known problems down the line because actually fixing them at the outset would be logical and ethical.
$191,900 odd dollars September 24 2024 Fox 24 not 109 million and September 25th
The protocol is to wire a few cents first, then verify it is received. And then wire the rest to the same account using the same wiring method. So you lose an extra $40 wiring fee, but that is well worth it.
Another example of "it'll never happen to me" along with intuitional incompetence.
I am a systems engineer and have worked in IT of over 30 years If you store the backups offsite or in the cloud you can relatively quickly restore everything to a working state. Also as long as the firewall itself is not compromised in which case you would need to reset it to factory defaults and reset the config (which shouldn't take too long if you kept backups of the config.) you can be back up and running in a few hours for critical systems and a few days for less critical things. Of course it all depends on what you spend on it.
Back when I did consulting I always asked my customers when they had sticker shock "How much is it worth to you if you lose it and can't get it back?" Most of the time they would say something like "I would be out of business." My next comment would be "Then money isn't the problem."
I remember when everything ran on paper forms. This wouldn't be much of a problem.
I'm an IT Director and backups is a part of disaster recovery. If they don't have that in place then people should be fired. I have multiple backups including cloud planning for the worse case scenario.
I'll bet the Wayne County Road Commission systems were untouched. You can't mess with perfection! 🤣🤣🤣
Happened here in Curry County, OR. I live in the County seat of Gold Beach, OR - population around 3,000. They wiped out the county's entire digital footprint & affiliated organizations: courts, police, sheriff, title office, city, everything - even the county hospital, etc. Royal Ransomware got us here, heard they got Texas & elsewhere. Held the county hostage for months - total nightmare, except for rotten criminals including drunks, targeters (terrorizing witnesses, whistle-blowers, victim-witnesses), and traffickers working for those higher up behind it.
The Scranton Pa school district recently paid $25,000 in ransom and $50,000 to a computer doctor to resolve the attack and still hasn't fully corrected it completely. They lost Internet and phone service. Classes were taught the old fashion way .
Do you think it was Dwight Schrute that hacked them? 🤪 ... Sorry, I couldn't help myself when you mentioned Scranton.
Still cheaper than paying a cyber security specialists annual salary. The real question is what data did the attacks gain access to. The cost of such data loss/theft may be an unforeseen consequence. However, that is almost always the afterthought of bean counters because it isn't tangible.
Any ransomware attacker worth their salt will encrypt the backups too.
There are offline backups. The industry moved from backing up to tapes (similar to cassettes and VHS, but different) to online hard drives. Now there is a movement to have "immutable" and "vaulted" backups that are deployed in an air-gapped manner.
The same thing happened to Patelco Credit Union a couple months ago in California. A lot of people had all their eggs in one basket and they couldn’t access their accounts for about a month while things were being restored. The ransom was NOT paid.
The weird thing about these armies of federal, state and local government bureaucrats, is that all of their work can be done on a laptop by one lady named Sue working from home in her pajamas.
and can be undone by Viktor from a laptop on an island
Ben hiding on the mic that is hiding in back of Steve
The first thing I thought of when I saw this was something about voting.
My county won't accept a check if they can't verify the amount due on the computer. And even then they have trouble calculating the amount haha
Record a video of you trying to pay them, have them explain in the video they're unable to take the money due to a system error and that it isn't the fault of the person who is trying to pay their taxes. It won't prevent head ache's but it will save your bacon for being late.
That stinks. I work in IT, if they told me an excuse like that I'd mention that then ask "Why don't you have a paper backup in place for when your system is down? Doesn't that seem awfully negligent? I understand if the amount is wrong once you're able to access it you'll need to notify me and have me come down within 48-72 hours but saying you "can't" when you won't just seems problematic".
I'm fine with a man or woman tells me no, when the government tells me no to doing their job, I get argumentative.
I have meet with our Parish Govt and they now have a everything backed up off line. This way they can scrub the computers and reload.
"To err is human; to really foul things up requires a computer".
Ben - Very tough to see. Just behind Steve's head, resting atop the short mic.
Steve I work for government IT and have recovered from ransomeware twice now. I was back up in less than an hour because everything is virtualized and nothing was exfiltrated. Services were not completely up for about 3 hours though as we have to verify that the backdoor or dropper isn't installed on any of the servers otherwise it will just happen again. But with virtualization the backups are back up with in a matter of minutes. The problem is they dwell in your network for weeks or months now and you need to make sure that your backups are clean. I am a small local government however which is why it didn't take long I could see this county taking a day or two to fully restore services. The great thing about virtualization is the other virtual servers aren't deleted or shut off either just removed from the network so the evidence is not destroyed.
Just yesterday while doing some work on the west side of Detroit I had them call 911 for an emergency at the back of the building. They were unable to connect with 911 from the landline and it took me another 10 minutes to connect with 911 on my cell phone. So this story actually answers a lot. There is also, just so you know two solar flares that just occurred in the last 48 hours that will hit today between about 5:30 and 7:00 p.m. today. Or 10:30 and midnight Greenwich Mean Time, and the second one is supposed to hit sometime tomorrow but there's the effects are supposed to be gone by the 6th. Those could cause some issues as well
Similar to Wayne County, Michigan, Columbus, Ohio experienced a significant ransomware attack on July 18, 2024. The attack, carried out by the group Rhysida, caused major disruptions to city services. The city is still in the process of recovering from the incident.
This happened in Flint a few weeks back, computers are down, can't pay bills online right now.
Just another example of how incompetent government has become ❗
It's your tax dollars that fund your under-funded governments so that they can't function properly. Pay more taxes and you might get the government that you want and expect. Or are you one of those who prefer no government at all thinking you can get along just fine without it?
Or how much computers expand incompetence.
There are several issues.
But the primary is money - it’s not that off site and frequent backup systems don’t exist, it’s that these places are usually underfunded as it is. Security, redundancy and backup is expensive, and in the case of for example hospitals there are many companies who provide services off site - so even if the hospital security is top notch, the service providers might not be.
At the end of the day, someone has to sit down and say “do we want to handle 15% less cases/patients/customers forever, when we’re already behind, in order to have security which won’t be needed and will only slow us down 99,90% of the time?”
Now the security professionals know what they would answer, but it might not be the same answer as an administrator or CEO might give.
And then even if you choose security, it takes a lot of time to work out systems and procedures that don’t end up introducing even more issues.
The most expensive part is not even all the IT stuff. It’s the constant training, vigilance and focus on it by every single employee.
Every employee is a gaping security hole waiting to be exploited. Even and sometimes especially IT employees.
Source: I work in IT at a hospital.
I was IT for 30 yrs.
My question has always been, why do you need internet access if everything is on an intranet?!
The CEO and CFO were big internet users and usually the ones getting compromised and the ones axing the intranet only policy.
To have a Back-Up the Government just needs MORE of YOUR MONEY! Just a 50 or 60 Million should do it.
So since we now rely on computers for everything we do negotiate with terrorists? That's progress for you.
A year or two ago in Ireland some hackers basically shut down our entire health service. They had cyber security experts on the news saying the government should pay the random haha. And they did! Seems like a serious problem...!
In a swing state? In an election year? How is this possible?
Dems corrupt everything they touch
yes
@@ValleyMansonOfficial
The marxists that stole the last election are hard at work to do more of the same in November
Backup systems definitely exist. The issue is that most people responsible for allocating funds for most any kind of organization are generally not well versed in IT and people's in charge often balk at spending money for a backup that they "might" need.
What you mention with offline backups, absolutly possible. The issue is it cost money. The company I work with finds it hard enough for clients to install dev servers, to verify updates don't break anything on prod at the very least.
what good is an offline backup when the app that needs the data is locked out. offline backups are a clue that the IT department are clueless.
Yeah, that is how it works providing your backups are setup and configured correctly. The process of wiping everything from all servers and workstations and restoring it all isn't exactly an easy task, but definitely cheaper and better than paying ransom to criminals.
As a computer programmer I say it is possible to create a backup system that keeps a separate isolated log of data transactions and sustains a known good backup. The good backup can be brought up to date with the isolated log. And it is also possible to isolate the data on a separate system than that used to transact business, and have a backup of the system so that if the system is infected you simply delete and re install the backup system and you are all done. The problem is that many of these systems are very old, some running on no longer supported Windows systems and hackers know the hacks that have been discovered since that OS was last updated.
When an operating system announces it’s cut off date for support, that is when every government system using that operating system should be updating their software.
Could this impact the voter rolls in the clerk's office?
Are they not maintained by the Secretary of State's office?
Boy we have some smart people in here, great question!
We have nothing to worry about because former President Obama said the U.S. election system is the most secure and reliable in the whole wide world.
Thats what i was thinking about as soon as i read the headline this morning. Kinda weird to happen in biggest county for state a month b4 elections.
In theory. The forensic teams would have to go through and determine what the scope of the breach is. However, city clerks maintain their voter roles and the paper copies of the edits, additions, and deletions would make it possible to ho back to a backup and reprocess the forms since the backup. Not to start a conspiracy, but it would be a reasonable target to do some voter role shenanigans in Wayne County and a select group of predominantly Democrat leaning counties/cities to get the state to go red because of the population density of the areas. Even if not successful, the formerly discredited voter fraud claims would suddenly have a place to point to as a credible example. Either way, this could open up the can of worms because it could swing the state to red, will convince others that fraud is possible, solidify the people that already believe the conspiracy to be unconvinced in the face of overwhelming evidence to the contrary, and some will never believe a government investigation of the government.
Very few businesses will see the paying more for mostly the same functionality as reasonable. They can but they don’t.
Thanks Steve!
There are often offline backups available in these situations. The problem is restoration takes a significant amount of time. Not something that gets done in hours. It's a days or weeks long process to restore everything. You need to do a forensic analysis to determine when the hacker intruded the systems which is often weeks or months before they executed the encryption and asked for ransom. So you need to disconnect everything and restore things, but also make sure you don't just restore systems with the malware that was on the systems when they were backed up.
The county won’t be able service you anymore.
Backups that get pushed securely to an off-site location can be immune to ransomware. Thing is, you have then download all that data to restore. Depending on inter/intranet speeds, this could take a LONG time. That could be why they are still offline while they restore the backups. Another issue is that hackers can sit in the compromised system undetected for a period of time before they decide to attack. This means once you restore, you need to verify the restored system is not compromised before taking that system into production.
I hope we are giving more than we are given.
Just a note that hackers usually don't just attack when they first get in. They rifle through the newly hacked sucker looking for the important info and the backups. Once gathered it's then that they spring the trap - encrypting the files. It's only after this that the saps are notified of their new misfortune. I've seen networks that were hacked 6 months prior to taking over the network. Most of these hackers are patient.
risk you take when you decide you want to have everything accessible over teh interwebz.
The true issue isn't that you need a backup but that your operating system is compromised. That means accessing any information is not working so you need to defeat that before loading backup information.
The problem is these attacks are not execute day-of-infection. They wait MONTHS. Now the infection is so deep in the backups you can't get rid of it without losing millions of records.
6:41 vendor:"sure when can we test it?" gov:"never"
Could you imagine being released from jail and they say,,cant do it,have to wait another day or so? lmao
My employer does exactly what you're describing, of course for a fee. We can restore data to yesterday pretty quickly, then they'd just need to figure out what was processed today and needs to be re-entered. A lot of organizations choose to risk it and think they can manage this stuff in house.
It is exactly the way you said you'd think it is albeit the servers that I know of are Intel. The admin should be able to swap in another hot swappable unit and have everything up and running in the time it takes to drive to the facility. If the network administrator is on site it's already up and running. Otherwise they may try remote fix however that could be the back door a hacker accessed.
Ohs noes! Government shut down! Whatever shall we do?
Do you get that government shutdowns negatively affect US?
My first reaction to this story is we need to change the laws. We need to increase the penalties for these actions. With the computer era the seriousness of those crimes has exponentially grown and so should the penalties! Minimum 20 years in jail for for these crimes. There is no fine that will do justice for this.
Except a lot of these hackers are not on US soil or anywhere the US has an influence, so you're only going to deter home grown criminals.
Most of the ransomware attacks come from countries we are not on good terms with.
There probably is a way to have a back-up. But most government offices will probably not want to fund it.
I used to work at the Henry Ford library and museum as their in-house computer support. They have an extensive backup system where the backup tapes are Fedex out too a storage facility daily. It worked fantastic and tell the person that supported that hardware wanted my job then he sabotaged it. I'm hoping he repaired it when he got my job.
You might ask how did I lose the two prior jobs they're both self-explanatory. I am an extremely honest person and contractors do not like honest people because they can interfere with their profit flow.
I have not worked in government information technology since the 1980s. We did do backups then and had audit trails of the work that was done by people on remote terminals.
The world wide web may be a marvelous thing, but there is no real way to secure a computer that is attached to it. Even if you can't get past all the hardware and software lockouts, humans that have access are still a weak spot.
When I was working in company IT, uptime was stressed. I remember writing a backup routine for a corporate database, the backup seemed to run fine, but when I asked to test it by doing a restore, I was refused by the manager because it would affect uptime for the users. I had no actual assurance that the backup ran correctly.
We waste trillions on military hardware that sits around and does nothing, and yet spend very little on tracking down and eliminating cyber criminals that are attacking our data systems Every Day.
They didn’t keep an offline backup 🤦♂️
Wayne County, MI. That's where my 1968 Ford Galaxie 500 convertible, my 2019 Ford Ranger, *and* my 2022 Ford Bronco were built. 🙂 The Wayne Assembly Plant.
LOL, you think people care about what you have and where they were built?
Banks have had it for years it's a government requirement. When I worked for a financial institution in the 80s, we had a hot site four states away. If we went down within an hour the hot side had us running at full capacity
Steve, they SCAN checks to deposit them nowadays. That is why I always ask for a receipt so if the county can't scan the check, I still have proof. Unfortunately, if the computers are down they CAN'T print receipts!
Shut the government down
Anybody else immediately think “County Election Board”? Whether such an attack on Election Day would affect voting or not, it is exactly the kind of thing that could be used to sow distrust in the results. Perfect timing, too: long enough lead time to be useful as a dry run; not a lot of time for the County to patch up security and thoroughly test it.
"There must be a way to back everything up in case this happens..."
There is. The problem is what you later said--it costs money to do that. And higher ups see it as a needless expense, thus a cost-savings they can remove from the budget to either come under budget or make them look like they're saving the company/government money.
If trade security for convenience, you deserve neither.
I don't even write checks now that I can pay online.
For those drunk drivers, I would think they went cold turkey with their habit after that ordeal. I mean seriously, what are the odds, a fire and no backup….
Steve, not only that IS how it work, my job is to do exactly that. There is a computer that periodically checks if there is a ransomware attack. If nothing is detected, the vault opens up and data moves in. At no point is that data exposed to a compromised system or network. No..does it cost a lot of money. Yes. Everyone chokes on the cost of prevention until they see the bill for recovery.
That is how it works, it's called Disaster Recovery Steve. They can literally have everything uploaded to a cloud and pulled back up from alternative CPUs... These state entities are not investing in operational continuity plans.
If people can't pay their taxes due to the county's poor cyber security, they should not be penalized. There should be a grace period for some time after it gets fixed.
I am one of those cyber security experts who would be tasked with mitigating the vulnerabilities through one or several preventative or detective measures to ensure that weaknesses like this can not be exploited. I can tell you that in our profession we don't typically have the authority to require that the safeguards are put in place and can only raise the awareness to IT management who in many cases ignore the computer threats in favor of getting getting software and OS revisions out the door even if they have missing patches and the software has not been security scanned or approved to go into production. In addition, a "disaster recovery plan", to insure continuous operation (although they are well known in the industry) are rarely implemented since they cost extra and organizations would rather pay the price in what they think is the unlikely event of an outage or just buy insurance to cover the expenses. I could go on and on since I've seen it all before, but you get the idea.
Restoring backups is only part of the issue, First, they have to identify HOW the attackers got in, patch it, then verify that the vulnerability has been mitigated. They then have to restore the backups in an offline environment and verify that it doesn't re-open the vulnerability. Only then can start bringing services back online. This process can take weeks, depending on the situation.
Modern computer systems use storage technologies that utilize Copy On Write (COW) to protect data, and can restore data back to the known good time/date. Besides that, the data is usually backed up offsite.
Money will buy a facsimile that is so perfect that it is indistinguishable from real.
6:00 Then they compare it to the millions it costs to have that redundancy and say "eh, security is not worth the cost".
someone probably unplugged the servers for the court house trying to charge their phone 🤳
Steve, there are people who get together to discuss the types of issues related to information/cyber security, and they have consulting firms who will help to secure the network infrastructure. The problem is that the cities and private sector organizations often choose to skimp on cyber security. Their "risk appetite", or how much risk they choose to accept, is too great. They fail to see the need for security awareness training and audits, and as a result their systems are left vulnerable and they end up being exploited. Their disaster recovery plans are outdated or ineffective. Same thing happened in other big cities and counties... Atlanta and Baltimore in addition to healthcare systems. As technology continues to evolve, cybersecurity is going to become even more important and until these local governments and private sectors accept that then these ransomware attacks and data breaches are going to continue to occur more frequently.
The average American can barely afford to feed and shelter themselves, much less pay for any health care. How can they pay for the cost of effective cyber security? That's why companies don't incur that expense - because we, the people, can't pay for it in the needs we buy, forget about the wants.
First problem: How often do you make backups, and how far back do your backups go. An attack can sit on your system for weeks or even months before it is triggered. The backups might all contain the malware, ready to reactivate at any moment. Even if they don't, there's some point between your last good backup and now that will be "lost". Figuring out what data is safe and what data is potentially infected is basically a forensic investigation.
Second problem: How low does the infection go? You run a program on your computer, and let's assume you have Windows 10 because a lot of people do. Programs you run are run as a "user" account. But occasionally you do something on your computer that Windows will pop up this fullscreen "are you sure you want to do that?" sort of message. That's usually "elevated" privileges, you're going to do something as administrator. A corporate/enterprise/government/agency level computer will have more permissions in between these, but there are others at higher levels.
One layer above that is system privileges-that's the level of privilege your antivirus, crowdstrike, kernel-level anti-cheat, etc. runs. Kind of above that is the kernel/driver level where your video card, network card, mouse driver, and DRM protections live. I say kind of above that because system level is similar and there's overlap both directions. Potentially above that might be something called Hypervisor level which is the thing that controls "virtual servers" if your organization has those. Above Hypervisor is BIOS or UEFI level (the latter is technically correct, but people often call it the former.) Above UEFI level-yes, there's an above UEFI level-is system management controller level. Every Intel machine made in the past *shrug* has something called a System Management Controller (SMC) which contains a little 486-class CPU all its own, and it runs a secret (not very secure) OS complete with web server inside your computer. If something infects THAT, you're screwed, because that's the part of the computer that rewrites your CPU's individual instructions.
OH YEAH, that's the final layer-your CPU itself, the chip that does the "computer thinking thing" is actually like a little computer inside your computer running a program telling it how to be a CPU. It's … more efficient that way, or something. But that program could be modified!
So: How deep does the infection go? At what level was your computer infected? Supposedly there's a layer of protection at each and every level to stop you from attacking deeper and deeper into the computer … but attacks have gone all the way down to the SMC layer that we know about, and a fairly recent Mac vulnerability exploited things that were part of the Apple Silicon CPU that nobody knew existed outside of Apple and presumably the NSA, who might have had that information stolen from them or something.
Even if your backups were perfect and made five minutes before the exploit happened, you have to figure out what was attacked and how deeply. Because if the attack is deep enough, good luck disinfecting the system.
If anyone is still reading, I'm SHOCKED, but absolutely we know how to prevent these kinds of attacks: STOP PUTTING THE SYSTEMS CONTAINING THE DATA ON THE FLIPPIN' INTERNET. Attacks are still possible but they start to require spies and people breaking into buildings or convincing some idiot to pick up a USB stick on the street and plug it in or … something. There are ways to actually isolate data communication between systems so that a generic "infect windows and encrypt files" attack can't hurt the machines with the important data on them, but the technology "sounds old" (it is old) "isn't up to date", "costs too much", "isn't very convenient", "requires more money and resources", you get the idea.
It's like your doctor telling you to stop eating three cake-frosted donuts and two triple espressos for "breakfast" every day to become healthier. Your doctor can know it's the right thing to do. Your doctor can tell you it's the right thing to do. The doctor can try and plead with you to stop doing that because you're freaking diabetic and on insulin with failing pancreas and liver, stop with the freaking donut breakfast OMG!! But you're the one that decides what you're going to eat today, not the doctor.
Offline backups are a thing, but they aren't always helpful in these situations. The bad actors occasionally sit in a compromised network for weeks or months before they pull the trigger on the ransomware. It's not uncommon to restore a system from a backup and then have it immediately start encrypting itself again. Sometimes it's a choice between going back a month or 2 or paying the ransom, and guess which choice most places will make?
That is how it works. Every sane organization does offline backups & has restoration procedures. We're dealing with government here though so they probably don't do it.
6:38 that is how it’s “supposed “ to work.😂
I read that title as Cybertruck 3 times before realizing i'm an idiot hahahaha
Oh, my god! Computers are down and no one can cope. What a disaster! That means having to go back to using pen and paper again! You will actually have to go there in person to conduct business. Can people even read and write anymore? OLD SCHOOL RULES! 50 years ago no one person could shut down an entire city. Regress with progress.
There are systems that copy/mirror data to two or more sites and systems and the lag time is relatively low. If one site/system goes down, loses power, or otherwise locks up the backup system(s) can take over and things continue to work. Of course, that's a simplistic view and additional protections and security measures are needed to overcome the possiblty of replicating a virus from the infected/compromised system to the parralel system(s). That is why you want to continue to have backups in addition to the mirroring system (s).
CONSPIRACY THEORY TIME.
What if this affects the Voter roles... 'And theres no backup that includes the last 6 months of registered voters...
😮. I bet the media wont ask about it...
Probably done intentionally so 'they' can now walk off with all that property tax $ with no records 😂😢
Time for an independent audit of that city/town imho.
The problem with backups is that they will likely be infected by whatever malware the scammers are using.
Here's a thought, do it the old way before it was all on computer's and when the system is better, have it all digitized again.
Those types of backup systems totally exist. Nobody wants to pay for them or the expertise to maintain them. I’d be happy to make Wayne county immune to these types of attacks, but I guarantee you they won’t pay a reasonable salary, won’t purchase the appropriate hardware and software, and wouldn’t give me the authority to make necessary procedural and policy changes.
How could they NOT have a robust recovery plan for a county this large!? Oh it's WAYNE County....
Virtual networking, full backup once a week and incrementals done every day, stored off site. And configuration documentation stored on paper in a recovery manual, not in a pdf file.
Would take a competent IT staff 2 days.... run 2, 12hr shifts, half the time.
But even with all that, it would still take Wayne County three weeks.
As an IT guy with 40 years of skin in the game, what can I say? Phools will be phished, sadly. Idiots are pretty much the standard vector for such attacks and it's heartbreaking.
Prefessional hackers wait a backup cycle or to to assure that the backups are also compromised.
The easiest approach for this would be an air gapped system. The crucial data and networks should not be connected to the internet.
Lol how do you figure that for easy? Every system requires internet this isn't a factory it's a series of databases that talk to other databases, as well as remote communication tools, etc.
This is why you're supposed to follow the 3-2-1 principal in IT, from Veams website just because I'm lazy and didn't want to figure out the best way to say it.
Maintain three copies of your data: This includes the original data and at least two copies.
Use two different types of media for storage: Store your data on two distinct forms of media to enhance redundancy.
Keep at least one copy off-site: To ensure data safety, have one backup copy stored in an off-site location, separate from your primary data and on-site backups.
You take the network hardware offline, the servers offline; bring in the backups and within 48 hours you should be back online.
@@GeorgeWashingtonLaserMusket I do exactly this, as a home user. The ONLY thing I don't do is keep the backup "off-site", but then again there's really no need to, the most I'm going to lose is progress on the latest video game I'm playing. You would think a business, or a government, who's data is much more important then my latest game save would be doing this.
That would require people to physically be at the air gapped system's location. That means the court would need to travel to the prison, or vis versa, to enter or look up data.
@@brianorca Private fiber circuits are availible. We use them for franchises and multi branch companies
@@GeorgeWashingtonLaserMusket That is all in theory. I have dealt with several security breaches and they were all by professional organizations that move to cripple your backups before they cripple the network. Private fiber circuits are availible. We use them for franchises and multi branch companies. Most cyber threats take there time when they first get access they position themselves to do maximum damage. Ransomware is a multi billion dollar industry many run by actual companies over seas. Remove email, and internet from the equation and your networks only risk at the point would be an internal onsite attack.
Why don't we ever hear about the IT department being fired for negligence/incompetence when a hack occurs on their watch?
Old school paper backups should be a thing.
That system type does exist so some ransomware installs silently then waits potentially months before activating so that any backups are also infected.