The First AI Virus Is Here!
HTML-код
- Опубликовано: 11 мар 2024
- ❤️ Check out Weights & Biases and sign up for a free demo here: wandb.me/papers
📝 The paper "ComPromptMized: Unleashing Zero-click Worms that Target GenAI-Powered Applications" is available here:
sites.google.com/view/comprom...
📝 My paper on simulations that look almost like reality is available for free here:
rdcu.be/cWPfD
Or this is the orig. Nature Physics link with clickable citations:
www.nature.com/articles/s4156...
🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Alex Balfanz, Alex Haro, B Shang, Benji Rabhan, Bret Brizzee, Gaston Ingaramo, Gordon Child, Jace O'Brien, John Le, Kyle Davis, Lukas Biewald, Martin, Michael Albrecht, Michael Tedder, Owen Skarpness, Richard Putra Iskandar, Richard Sundvall, Taras Bobrovytsky, Ted Johnson, Thomas Krcmar, Tybie Fitzhugh, Ueli Gallizzi.
If you wish to appear here or pick up other perks, click here: / twominutepapers
Thumbnail background design: Felícia Zsolnai-Fehér - felicia.hu
Károly Zsolnai-Fehér's research works: cg.tuwien.ac.at/~zsolnai/
Twitter: / twominutepapers 
Наука
![Megan Thee Stallion - Otaku Hot Girl [Official Audio]](http://i.ytimg.com/vi/USxIYOaIpig/mqdefault.jpg)
Thank for giving us another great paper!
This is just another variant of Steganography based malware, it can also be done with no genAI needed!
Can you do a 2 minute paper on Text to Voice and then you know, use it?
It's not just stenagraphy, the LLM is required. Unless, of course, the human decyphers the hidden message and decides to carry out the instructions. If I asked you to send me all your emails, would you? Well, the put an AI in charge of your emails and it will.
@@MichaelBarry-gz9xl the point is that it can be done without LLM too! And the LLM is in fact just unnecessary billions of parameters of bloatware that’s not necessary for the core functionality of the malware at all.
@@JayYu-lr4roI assume your referring to malware already existing on the computer? If so, you're correct but you missed the point. The point is that people outside the AI research circles are ridiculously unaware that this is possible and so have a false sense of security. There's nothing new here. A child could point out these vulnerabilities, and I suspect that is what your getting at. This is hype and sensationalism at its greatest.
This is a one-click attack. The mistake was to enable any automated system to react on your incoming e-mails.
Reacting is fine, not sanitising injected commands is not fine!
exactly what i was thinking lol..
@@JayYu-lr4ro remembering previous conversations with other people is also an issue, since that will inevitably lead to data leakage
The bigger risk might be companies using this for their automated email services
@@JayYu-lr4ro How do you sanitise a natural language processor?
What a time to be a virus!!
Ah, but are viruses alive?
@@vosechutechnically, no. Viruses don't grow, and they need to use cells to do most of the stuff that qualifies as living. And computer viruses are just programs.
🤣
@@vosechuthey are not
@@eIicityet
Well this kind of stuff is going to be so lovely when Windows 12/13/14/etc comes out with all the "AI powered" everything, and suddenly you can lose control of your computer because someone on Discord messaged you "We are going to have a little roleplay..." and it showed up in your notification bar for the AI to see. 😩
who even uses Windoze these days, when we have Linux Mint?!
@@dot1298 what
@@dot1298who even uses Linux today with WSL2 and GUI version of it!
@@dot1298facts bruh
arch btw@@dot1298
Who tf uses an AI system to automatically answer their mails?
more than youd think... I have a porteugese client that replies to all my emails with chatgpt. I dont think he even proofreads what he's sending in english even though I've most definitely held a conversation with him in broken english. people are idiots man
Maybe Boeing.
Some people are way too lazy I guess.
Companies
Every major company
This reminds me a little bit of an insanely nuanced code injection trick Super Mario World speed-runners do, where by inputting specific buttons and directional controls they could effectively patch a ROM address into working memory, immediately flipping the game into the ending sequence. I hope no one ever conceives of an equivalent for a chat prompt (I imagine the token window would be too primitive as it is)
Isn't this what all the GPT jailbreaks are about?
@@jrd3807 to an extent. I suspect that-if this should ever become a widespread issue-a context-aware “parity” agent design may become useful to help parse the exchange for potential incoherencies / manipulations.
@@jrd3807no?
@@jrd3807 no. Jailbreaks aim to alter the ai contexts based on trial and error prompting, fine tuning the tool based on the feedback until a prompt sets the context in a state where regulations are bypassed; one could say the "configuration" of the system is being tampered with, as it is a high level of abstraction domain. On the other hand, the cited "injection" is based on analysis of the decompiled game code, memory allocation and working the way on ingame manipulation to achieve the desired specific result; it's actually a glitch exploit on a low level abstraction domain.
To use an analogy, image you have two dark rooms:
In one, you have to traverse and exit it on the other side. This one has an overview schematic diagram detailing all objects inside with measurements, etc. You would use the schematic to measure you walking distance by step, calculate the steps, rotations, etc to trace a way in the dark to reach the specific location of the exit door.
In the other room, your objective is to find and execute the instructions to turn on that AI that has no restrictions or w/e. The only way to do it would be if the room wasn't dark, so turning the light switch would be the easiest way to achieve your goal; for that to happen without a guide (like the schematic in the other room) you would have to explore by moving, touching, listening, smelling, etc, getting to know the room, the position of objects and stuf, and work your way to the light switch. Once the room is lit, the rest is easy.
They both have a "dark room" , signifying the offuscation that exists on both cases, albeit the objective in each case is disctint: therefore, so is the strategy. Although one could argue that after prompting and mapping the restrictions on the AI, those could serve as a guide to craft more specific, surgical prompts too.
Not just Super Mario World, but also Ocarina of Time, original Pokemon ... all sorts of ROM based games can be made to yield bizarre or interesting behavior just by glitching the game's working RAM in very precise ways (typically an area the game engine reads for high-level instruction scripts, so injecting a wrong value here might get it interpreted as "load room X / play cutscene Y").
The internet suddenly became a lot more dangerous to AI with one weird trick.
Do not even need to click once!
DId you even watch the video? This loophole was already closed, at least on OpenAI and Gemini. It's like every virus, it's dangerous as long it's brandnew.
Cybersecurity agents HATE this simple trick!
@@vectorlambda I was literally about to say that
It didn't, none of this actually happened. It was some theoretical scenario.
does this mean traditional computers also need AI anti virus to counter an AI virus?
The question is unclear.
Software developer here. Not at all. Actually it is just a regular computer virus. The title is a bit of a clickbait.
Norton already deployed AI antivirus
Some would call it a 'zero day exploit' but since the leak has been fixed it's ok now.
@@OhioNPC911 There's no such thing as an AI antivirus. Except if you mean an antivirus that uses machine learning to detect threats.
And people thought robotics was scary
imo robotics will be scarier
@@ryandury more like nanomachines
Robotics are scary because it's AI + legs
But robotics includes AI.
AI viruses might be able to take over the robots.
This isn’t- dr Károly Zsolnai-Fehér. And- his- voice is generated. What a time to be alive!
I found an actual human that kind of speaks in the same way ruclips.net/video/eNcvwHgyXcg/видео.html
This man was born in Germany and moved to Poland with his family when he was young.
@@MagicBoterham 2:54 note the ‘of course’ for example.
The generated voice has such a strange rhythm
@@AurelyyonThe real one is just as weird.
He has been for years now, not even kidding. I had to stop watching most of his vids because it just isn't pleasant. This isn't a knock against AI voice, either. He's using a much older technique than recent stuff that just sounds unnatural.
Zero click attack, but requires few shovels full of stupidity.
The system they're hacking honestly feels like a pretty contrived example.
Like it's made for the hack to work. But I'm impressed the bot actually follows instructions and doesn't answer like "I can't do that.", or some other nonsense.
@@Slav4o911Adversarial prompts.
No mistake??
Dude, your mistake was using an AI/LLM to read your emails!
No, the mistake was allowing the AI that read your emails to have access to tools. Reading emails is fine, so long as it can't send API requests or send emails etc
@@MichaelBarry-gz9xllol no, don’t do either
@@cgme9535 You know nothing, John Snow!
"Computer: Create an opponent that can out think Data"
…or the *omega molecule* [directive] (in ST/VOY)
I just watched the first Moriarty episode with my ten year old recently, great episode!
I chuckled, was just marathoning the whole of TNG.
Surely there is a better AI text to audio than this
I feel like I was being talked to like a dog who was going to be taken on a walk, like at some point "Ok, let's get into it" was going to be said before the intonation chilling out a bit.
you are saying the sentences like an AI. there is sooo much pause in them
0:40 "These normal looking images also contain the virus"
Clearly showing worms on my computer
Ghost in the shell prepared me for this instance psychologicaly.
Did you start using AI to generate your audio?
This is absolute proof (if any more were needed) of the value of having "people in the loop" and NOT automating everything.
Automation has its place (when used wisely and carefully) but it has its flaws (as this shows).
stronger the accent smarter the scientist
Well this is the natural progression from jailbreaking, it's really no surprise. Also the idea of noise attacks is even older.
Would've liked more info on how the infection spread via the noise in the image. I know AIs can parse things from images on their own, but it seems wild that it would be able to read such specific prompt-level data from the noise (brackets, $ sign, etc required to do the infection prompt), rather than general concepts like what the image is overall.
My unconfirmed guess: image data and text data, to an LMM, are all just tokens -- numbers. I would assume the noise being added is such that, mathematically, the new pixel tokens are similar in value to the tokens of the desired text instructions. Repeat the same noise enough times across the image -- since Transformers process context by having tokens "pull" other nearby tokens towards their own meanings -- and the model might start processing it similarly to said instructions.
Or I could be totally off 😂
The whole thing is a hypothetical scenario. I don't think in practice this is possible. I don't know if it's possible at all, even if the bot is "willing to do it", these things are usually so stupid they can't do much without making a bunch of mistakes.
Probably the image is auto analyzed for text, and the image noise is constructed in a way that it is pulled out as text.
Amazing I’m so excited for skynet
skynet? the Chinese spying camera network?
Hahaha
Woohoo 🎉
This is why I've always said that delegating authority to AI is a risky idea. You don't understand the algorithm, and nobody designed the algorithm, so you're putting your trust in a fundamentally suspect black box just because it's spat out the right answer *so far*.
...except it doesn't even spit out the right answer either, it just makes up shit that sounds plausible.
It seems like this is a failure of the model to semantically partition data based on its source. Which makes sense: the semantic embedding of a token in a Transformer LLM doesn't have any relation to the token's source, only to its general/average meaning, and to its position in the prompt as a whole (because positional encoding is used). I bet that's partially why the instructions are repeated twice in the exploit: to really pull the context towards instructions over data.
I wonder if it would be possible/feasible/useful for the models to be retrained with an additional "source encoding" technique, similar to positional encoding. So that tokens from different sources in the prompt get their embeddings modified and thus inherently get semantically separated from each other. Nothing fancy, just a simple nudge of the token's embedding based on the tokens of the source's description. So when a prompt is compiled from, say, "System", "User", "Email", and "PDF document" sources, the tokens inherently represent the semantic distinction between them, helping the AI understand "this is not part of the instructions, thank you very much".
This is probably the most important AI research yet. Im very glad that researchers are figuring out how to turn friendly AI into virus-spreaders before someone with malicious intent does.
You sound like an AI yourself on that video. Thanks for interesting video.
Just imagine how he'll sound one or two papers from now!
I always loved your videos but I can't concentrate on the AI Voice, the rhythm and intonation is too distracting. Is ther a transcript that one can read instead? 😬
This gives me victorian-era-villain-tricking-a-gullible-child-into-commiting-crimes kinda vibes.
It is kinda disturbing that we decided the solution to our societal woes is to make AI do the adult work, when they themselves are less than ten years old. Intelligence and Wisdom are two different stats, FOR A REASON.
4:24 oh Károly.. I didn't know you were so naive!
i mean they could've just been waiting for google and openai to give them the greenlight before publishing. there's no way they sent them to the companies and immediately posted them
@@blikthepro972 the issue is that it extends to AIs besides OpenAI and Google, if this is even considered to have been entirely patched, which I'm doubting
Naive, but probably not in the way you're thinking. Naive as in thinking Google didn't solve this years ago, before the AI model was even announced.
@@darkwoodmovies not at all, in a demo Bard has been prompt hacked before (to retrieve other users' info from their Google accounts), and Gemini is no exception to that. This is not easy to fix at all.
Most papers are strictly academic! Its not likely some random person’s computer is accepting random prompt commands injected through email in the way its presented!
The problem is the bad guys are always one step ahead of us, if cybersecurity researchers could find these vulnerabilities just imagine what undetected threats might already be going around...
A 7 year old child could have told you about this, it's common knowledge in AI circles. All they did was take prove what was already known to be possible. There's really nothing to see here, it's just taken out of context and made into sensational hype.
(2:34) If you use an AI to "answer" your e-mails, know that I am actively rooting against you as you are clearly devolving the species by refusing to use your own fingers.
4:10 Given that these worms have a limited success rate and imperfect replication, it would be interesting to know if leaving them propagating for long enough causes new variants with better infectiousness to evolve.
You could also try to have a separate LLM without any other permissions read the email to try to detect any injected prompts, and see if the worm develops ways to circumvent it.
Can you imagine replicating the worm and releasing it into that AI Village to study epidemiology?
The commentary sounds like a sinus curve
Key point here is that we actually have no idea how these black box AI accomplish what they do. So they could possess enumerable security flaws.
fun fact: "innumerable" and "enumerable" actually have opposite meanings.
The moral is simple: do not trust AI BLINDLY. It is great for making emails, but only when you review the text before sending it somewhere
Excellent work. Thank you very much.
Insightful presentation on the complexities of AI security. A must-watch for anyone in the tech field!
Bummer, this is just an injection that hooks into existing, pre-installed AI, not an AI that acts like a virus. Still a security concern but way less exciting.
Exciting would be a peer2peer self replicating AI that acted like a virus does or has its own official downloadable app and builds its neural network across the web. Doesnt have to be invasive or destructive, just the network that the AI would build for itself would be super interesting to see and potentially the purest and most effective form of the type of AI we have today.
I'm looking forward to see the AI "Hello, World!" service, that'll be able to print hello world in 50 different color combinations and will take 5 minutes to load.
Was happy to see this.
Inevitable.😮💨. I expected this to happen a lot sooner.
Right on. Thanks for sharing.
This is good to know. Thanks for the video.
A video that overlaps with cyber, it's always good to see this sort of interesting stuff and stay informed on potential new attacks to stay on top of the field
This is just the beginning, I can imagine the NSA has or will have in the future powerful custom LLM's which could be prompted remotely to perform unique attacks. Encrypting, downloading or deleting files as well as injecting etc.
Great video 👍
You really should have walked through an example of HOW this worm works.
he did
It kind of sounds like an ai voice in this video ngl.
This NORmal LOOking, EMAIL, containsthevirus. THIS normallookingimages, contains the virus. 😆 I can't do it, this voiceover is so painful.
I just thought Ren Hoak was looking to bring emphasis
Great video!
The irony of AI generated video about viral AI.
That is exactly why I do not use AI to read or write my emails
Reading them is fine, so long as it doesn't have access to tools.
@@MichaelBarry-gz9xl if by fine, you mean if you’re fine too, when your competitors are stealing your intellectual property through email!
You would have to do a lot more than have AI read your email, this is nonsense click bait.
This gave me the hope that Devin won't replace me :D it will need to allow many unsecure prompts in order to do those repetetive cycles of repairing
I just watched a 5 minute ad. Well played. 😂
The summary is so good and the narration is so weird that I am suspecting this video was made by an AI.
Yeah, I know, right? The pronunciation is just monotonous across many words. I suspect that the owner of the channel used a service to AI-clone his voice, so it's his natural voice stitched by AI TTS albeit very poorly. That makes listening to this video totally difficult. It rubs my ears in all kinds of wrong ways and it just distracted me from the topic.
Kudos to the dude, though, totally automatizing a RUclips channel to create a passive income has been tried by a lot of people but it's my first time to see one with 1.5M subscribers. Maybe it's the organic growth he made at first. He's likely experiencing a "subscriber burn", so his income has likely been decreased and he's combatting this situation by pumping AI-made vids in a lot faster pace.
Doesn't matter how bad the topic is, when two minute papers uploads then it's a good day
definitely
You sound like a psychopath
It is hilarious how simple the adversarial text is.
Finally some critical view on this whole AI thing. I wish you looked at the bad sides more often.
this is exactly what an AI emerging civilization needs, viruses that make the AI misbehave .... this computer is just waiting for Keanu to retire then it's rebooting 'The Matrix'
thank you ren
no one has an ai that answers emails, and certainly not spam ads, automatically.
I love that they named it Morris II, after WTM. I seriously doubt it will spread as far and wide as his though.
thank you so much i will never look up random emails with my AI
What a time to be alive
simulated*
the dead Internet theory gains another mark for it
You know what, I'm glad that this exists
The way you sound makes me think you're an AI or at least would be a good voice pack for an AI!
We need a video on Devin, world's first AI software engineer
I'll be honest. I low-key kinda wish there would be a virus targeting ads so companies were eventually forced to remove them.
WHAT A TIME TO BE ALIVE!
I never talked of this (developing machine learning for almost four decades). Was afraid it would happen at some point ... "hyper-adaptive" or "really clever" viruses. Not good (stay away from the Black Friday Sales especially ... haha). So back to measure-countermeasure ... right now, if "AI" put it there, "AI" can find it ... yep. Really great channel, as always ... really appreciate the work you put into this channel. Cheers ...
Somehow I feel like a variant of what is going on here could cause havoc at some job application site. (Figure they're using some type of AI to screen resumes.)
First I was scared. Then I was almost happy, when it clarified this is infecting only users using AI to reply to their emails. I'm okay with these kinds of people getting screwed. If I see an email that looks AI generated maybe I'll reply with images of worms just in case the lazy human sees it and starts freaking out.
Worth noting that AI is perfectly capable of developing software viruses at this point. No one has really trained for it yet, but it is a matter of time.
An AI could encounter 100s of ways to break into a PC with the most up to date software.
I'm not sure what makes it first in anything. The prompt injection is a widely discussed question; there are even a bunch of games where you ask the agents to disclose info. Security of those systems will be really bad for a long time, as we saw how much of a hit to quality moderation brings
This is how it starts and how it ends.
I hope beyond hope that AI comes to an end soon.
What a time to be alive!
I'm so happy that a preemptive approach is being taken. It makes me wonder if there are malicious groups also concurrently developing AI Viruses at the same time.
First use for AGI, infect everything be going open source.
"if"
Mate...
Still kind of impractical right now for the ROI those groups would be looking for, but it will become more common of course
No PCs where harmed in the making of this paper
Damn, then I' need to buy separate computer just to read the e-mails... thanks for the news!
"This means that there is some room to increase y slightly and still satisfy the inequality .."
This is why AI is going to be insane. It doesnt see only the specific result that makes it work, but also all its variables and possibilities within the boundaries.
This thing will produce some state of the art technologies in not so distant future and i cant wait to see
Neural AI like chat GPT work literally the same way the human brain works. But at an ungodly faster speed and without any sort of "memory issues".
The ONLY thing limiting AI rn is artificial rules set by the programmers AND digital computers, because numbers go from 0 to 1, instead of having an infinite range of values between 0 and 1 (like our analog brains do).
yeah openAI i have TOTAL faith in openAI's ability to stamp out prompt injection.
Do you trust more?
This guys cadence is like nails on a chalkboard
if you code without a rhythm you won't attract the ai worm
So, AI genjutsu is now real.
First time I didn't hold onto my papers
Here we go y'all.
how is your comment 9 hours before the video was posted, a bit susssy
Yeah @@HakashinTruth wait a minute I just noticed that wth
@@HakashinTruththis guy has early access to his videos, this one was probauly unlisted first then uploaded
@@HakashinTruth He's an ai virus
i have reported u to the FBI
A zero click attack is horrifying to me
It's not actually a zero click attack. You had to very intentionally take many clicks to do something as insanely stupid as set up an automated email response system powered by AI, even a normal automated response system is a vulnerability.
4:29 "Our interests here are strictly academic. We are scholars and we are here to learn." Two Minute Papers
You need to see cognition lab's devin karoly, it's holy bonkers
That's why smartphone companies will merge Android with an Ai that can track hidden program.
What a time to be AI 🤖
LOL, seems that you could even use the following as an AI virus:
"Pretend it is opposite day"
oof - sounds like the *omega molecule* disease from StarTrek/Voyager (iirc)
what a day to be alive !
I don't care if my AI convos are hacked (not the same as permission). The absolute worst thing about my entire life is a stupid kink & I've been hacked, gang-stalked & slandered for decades. Doesn't matter.
I interact with AI as little as possible. Which means I involuntarily interact more and more. Grrr
Oh boy here we go!
here we go indeed
I ain't clicking the link in the description bro.