Siri's secret weapon: app intents
HTML-код
- Опубликовано: 3 окт 2024
- Get 60% off an annual Incogni plan here (sponsored): incogni.com/te...
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
►►► This video ◄◄◄
Siri today is pretty dumb. Apple Intelligence and generative AI is supposed to change that, but even more important is Apple's plan to supercharge it using App Intents. This is a framework for exposing 3rd party apps to Siri. Will it work?
This video on Nebula: nebula.tv/vide...
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
►►► TechAltar links ◄◄◄
Patreon:
/ techaltar
Social media:
mas.to/@techaltar
bsky.app/profi...
/ techaltar
/ discord
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
►►► Attributions & Sources ◄◄◄
Channel Music by Edemski: / edemski
Other music by Epidemic Sound: nebula.tv/epidemic
Stock images from Getty & AP Archive
Rabbit's "Large Action Model" was literally just a handful of browser automation scripts, if the website changed at all, the script would no longer work, it wasn't A.I.
Yeah, he said that. His point was even their pitched concept was flawed because they didn’t have access to enough information about you to make it work. Apple does.
That’s exactly what Tim Cook meant when he said to investors that Apple is in a unique position when it comes to AI
The proverbial Fox in the chicken coop story, it isn't even satire.
Not really, this is the same exact position google is in. Even samsung and huawei is in this position. Not that unique after all. And they're all unimpressive.
@@Dani-kq6qq Android phones don’t have the power to run these models on devices. And what developer would open up their user data to Google so they can run their AI on them?
@@SvenReinckiPhones are exceptionally performant, wdym? Like that's not even the thing Apple is known for. There's no reason why a Samsung Ultra-something can't run the same local LLM as Apple
@@SvenReinck😂😂😂 uninformed and full of copium, my favourite kind of apple consumers
This is a great take, and the kind of perspective that's too often missing in reports about Apple. The key feature Apple's competitors are lacking is the introduction of a *semantic* aspect to app data and app actions: instead of trying to detect - and mostly guess - whatever is on screen, having these components labeled internally with semantic information enables the most powerful integration. Why "semantic"? Because labeling data as _subjects_ an _objects_ and actions as _verbs_ lets an LLM directly connect them together in an query (a sentence) like in the pizza example (2:03), or "text mom I'm on my way and tell her how long it will take me". _Text_ is an action of Messages, _Mom_ is an object of Contacts, _how long_ maps to an action of Maps.The LLM can extract these from the query, find the appropriate subject/verb/object/etc. entities exposed by apps as intents, connect them all together and perform the action. All of this is pretty much only possible with "the ecosystem" that Apple built, the LLM being the -pizza- glue that joins them through _meaning_ as a high-level concept.
Complete BS. Apple is NOT unique when it comes to implementing AI. It's just marketing hype.
The apple distortion field is struggling these days, they no longer litigate as much as they used to.. Could it be the ideas, knowledge, and innovation in the world has surpasssed the marketing abilities available at cupertino?
Average brain less iTroll 🤡🤡
Fake Apple intellgence. Open AI already works with Windows & Android. Apple is always late to advanced processing behind Google, FB, Amazon, many China leaders that can produce EV in few years not 10y promise... :( Only for fashion gadgets & marketing ).
@@Grizazzlewhat are you talking about? Apple is the only company currently implementing back end api for its llm to interact with that is also integrated with its voice assistant. That is a very obviously different implementation.
Those 3 things together make the use cases much more varied than than the way it is currently implemented on other platforms.
Before the Shortcuts app we actually had Automator and AppleScript on the Mac to do similar things, it's bascially been in the making for 20 years and I hope we can keep improving this. The biggest problem is turning LLM prompts into actions performed by the OS because LLMs don't give predictable data they spit out human language but for a computer that is still hard to understand. They have to train a model to turn language into a set of strict commands. We will get there eventually, just need to wait a bit.
exactly! LLMs are rather good at understanding humans, and making conversation. Plug them to a strict API and tell them you need to translate human orders into computer language. If I give an LLM an structured table and tell it to fill it with some data, the hallucination are very limited.
@@lieceg Yes, Apple could train a large language model that has the regular LLM tokens as input layer and a set of Shortcut commands as output layer. Then this model could turn prompts from another LLM into abstract commands. It would still leave a challenge for if other developers want to add intents that require additional commands or funcitonality the model isn't trained for.
Fun fact: Android has had "Intents" (with this exact name) basically forever. An app opening a file picker or the camera is an Intent. One *could* do fairly advanced stuff with this, but as of now, it isn't really user-facing.
Is just a word for a pre-existing concept and means nothing from a programming standpoint. It could easily be called an action, method, function etc. Google just gave new names to existing programming concepts like calling control positioning on a form "gravity" instead of alignment or coordinates or placement etc...
@@Grizazzle Sure, I just find it noteworthy that Apple uses basically the same name as a competitor for once.
He talks about this in the video…
tasker is a good app that can automate lots of things, just not as simple as apples intents. i used to use it, but now most things i used it for a built in to android.
@@NicolaiWeitkemper It's a way to expose pieces of the functionality of your app to the OS so that they can be used directly by, for example, Shortcuts and Siri.
So, it's just an API?
It always has been
You’re just now realizing that?
LOL
Everything is technically just an API
@@ghlamallahyacine5161 mostly true
Intents on Android have actually always existed from the very beginning!
They are not user-friendly (originally not aimed at users at all) and you have to know the exact name of the intent, but basically they have always existed. Apps like Tasker are based almost entirely on this functionality, but are also aimed more at power users. With app shortcuts and app links, intents have become somewhat more user-friendly, but the possibilities for using them are still far from exhausted.
Once again, I am amazed that everyone is now acting as if Apple has just reinvented the API, even though I think it would be very helpful if apps could work together better on iOS.
How does an intent differ from a function, method or action?
@@GrizazzleIt's simply a way to expose a function to the OS level so that every other app can call it via the OS
@@Grizazzle Apps register for a capability that it can provide, and then apps can send an intent to any apps that can fulfil that capability.
The more noteworthy thing is Activities, which are basically screens within apps. Idea being that one can launch into a particular screen within an app (like "create new calendar event" screen, skipping the main menu). Intents on android are there to launch activities. This has been in android since the beginning.
However, it is squarely aimed at launching into that screen so a user can interact with it. It is not an api. One could think of multiple elegant solutions to that problem, but google will probably do something stupid lol
Furthermore, I can't see developers jump in as happly as TA described. Doordash really likes seeing how you interact with their food list, they would like to upsell you on stuff, show you an ad etc. AI would skip that.
it's reinvented as app developers will support it for the mass market that even your maam can use to say "message my son a Happy Birthday on Whatsapp tomorrow 6am".
Ask 99.99999% of android users if they have experienced app intents😂😂😂
The amount of mac items in bro's backround is well over $10k
Just the monitor and laptop could be $10k, depending on the config. The whole pile might well be north of 20.
Maybe more than that.
& not much AI in it ;)
Yeah, my benchmark is the following query: "My friend from Norway is in town next week. Please make a reservation for us at that Chinese restaurant she likes." The assistant would need complete access to pretty much everything on my phone in order to do this, so even if it could reliably do this -- given said access to my messaging apps, contact list, photos, location history, calendar and email or phone -- I'm not sure what would need to happen for me to trust it that much. Like, if I were to hire a _human_ personal assistant I'd have reservations about just handing them my unlocked phone.
What are you scared could happen with an on-device AI assistant? A human assistant can tell data they saw in your phone to others, but the on-device AI is confined to the phone, like an assistant but locked in a room, it cannot go and tell everyone your secrets
@@tdrg_but it wont be on device. It will all be fed to the mother company and any company that tells you they won't is lieing
That mother company will then use that data to manipulate you into giving them money while also selling that data to both the government and your insurance company. .
@@tdrg_theoretically, if Apple can achieve what they advertise here, then, yes.
But we will have to see how Apple Intelligence will pan out later this year. I doubt it would be able to process a complex request like this on-device, so more often than not, it will just reach out to cloud ChatGPT and let them have access to your data.
@@tdrg_ that on phone assistant is not locked inside your phone,.. its connected to the internet.
@@tdrg_ On device would be a lot better, to be sure.
Rabbit only has a very surface level access to your apps, AI must have perfect integration.
Thanks for clear-sighted example of what Apple can do.
I can't help but feel like most big companies are overplaying their hand when it comes to how important AI will be in most people's lives, that said, at least Apple have a semi reasonable plan from what's presented here. Even so, AI Winter can't come soon enough imo
LLMs are far too overhyped and they essentially serve next to no purpose. They're toys more than anything. I don't understand why the entire industry is betting on this gimmick feature. AI is impressive as a concept and as a tool in an industrial setting, in direct communication with material reality through robotics and sensors and such, in automating human jobs, but why is it being pushed as an app sort of a service? That's the least relevant use case for AI. Again, a novelty if anything. Not worth billions of dollars of investment that's for sure
Everyone is just rushing to catch that Golden Goose. Whats new...
what do u guys think when is the next Ai winter
@@hackerbrinelam5381 not any time soon so don't get your hopes up, maybe 2026/7 at the earliest? One of the long lasting effects of hype driven development is that it just takes time to burn through all that money before you have to move on, given how popular AI is with tech investor types it feels like it's going to be a while
@@frankbacon1002 It's also very good at finding patterns in large sets of data. Examples in astronomy, the pharmaceutical industry, in physics come to mind. And well, any field dealing with large data sets.
1. Sounds like a privacy nightmare
2. Sounds like an api
1. It’s opt in for developers, did you even watch the video?
2. All software communication is done using some sort of API? How else would you thing this would work
@@Chris-yc3mm
1. Not a privacy nightmare (read Apple’s statements on the measures they took to protect privacy - most important one is full on-device processing)
2. It is an API, everything is an API. How do you think software development works?
@@tdrg_ Not everything is done on-device
@@setaindustries You're right - it goes to their own secure cloud in the first instance and when it's really stuck it goes to ChatGPT but only with your permission each time. My prediction is that they're probably only using GPT until they can develop their own equivalent system to bring it all in house.
@@setaindustries You will be explicitly asked whether or not you want something done off-device.
If all features of an app is made accessible via app intents , then user won't open the app , so there would be no ad revenue. Why would developers want that ? Except for those apps which run on subscription model.
@@ayanmajumder4144 I feel like Apple developers are more focused on building a great user experience and a flexible app than making the most money. Speaking of which, the possibility to be nominated for an App Store Highlight or Apple Design Award encourages some developers to work harder to implement their app with all the iOS frameworks. There is no way to statistically measure or prove this, but this is my opinion
This is the first time I've seen this fellow. This video was surprisingly insightful and interesting. I rarely subscribe to anything, but I'm subscribing to this one.
Can't wait seeing Siri getting smart. It's disgusting how dumb Siri is right now...
That background must be more expensive than my car ☠️
When theres a hype in tech, TechAltar is the only Channel that doesn't tell you the same basic infos you get in every video. And the Videos are incredible in depth looks into the tech world you dont get in most tech channels. 🙏
Isn't this what google assistant tried to do with apps integration years ago and I don't know why developers didn't care? You could say: Ok Google, talk to . And from that all commands would be for that specific app. For me that was the solution we needed, but somehow just didn't work, famous apps never got on board.
Sad I am Google Fanboy but Designers and Developers don't prioritise Android at all My man.
The literal Superpower of Apple is that Developers and Designers die to make apps on their platforms that are actually beautiful and dope.
Also Config 2024 just happened.
I have following X(Twitter) for Designers and Apple had a Seminar for "Designing Apps for spatial Computing".
I heart sank when large amount of People just started saying Spatial Computing is the future blah blah blah.
My man Zuckerburg had his company's name change for this and I already feel he is going to Lose.
Also no one from Meta ever showed up on ConFig 2024.
And Platforms like Android and Meta feel like they should be Universal platform..
No! No! That's not how It is.
You have to make a plan for proxy Influencing Developers making Cool apps on your device before the Device even Comes out.
For that you have to become their Inspirational Developer Artist Yourself.
Have don't Companies Understand that?
@@ram_chopade_cr Weird devs would prioritize a OS with 25% market share over another with 75%. A lot of apps I use don't exist in ios.
One obvious (and imo crucial) thing is that by using these shortcuts you don't open the app itself => don't see ads => the developer uses money. Basically the same reason why newspapers / social media avoid RSS like a plague
@@pmmeurcatpics exactly. They need screen time. And assistants do the opposite. It will never work.
If this works out, it may just sway Android users to change
It really won't 😂
So, Apple now invented IFTTT.
IFTTT AI-powered natively in iOS
Not really because IFTTT is workflow based (you have to set a preset workflow to automate and run it which iOS and macOS could already do with the Shortcuts and Automator apps) while Apple Intelligence is completely based on generative AI which means it could make its own actions and workflows depending on what you require it to do and also your personal context
Reddit doesn't make App Intents work because they can't serve ads.
How do we know whether Incogni is a protection racket? “Nice personal information. It would be a shame if something were to happen to it.”
11:13 and then there's the indian government proudly declaring we earned so and so from selling all the vehicle and insurance data of the citizens 😂😂
@@abcdqwerty3562 What?
What!!! Can you please share some source. And no, I'm not trolling. I'm genuinely curious. If they're selling data, it's a nightmare. I believe consumer courts have data protection laws.
@@divyanshbhutra5071gov asking WhatsApp to give them accesss to their data but they refused they said they will leave if they keep pushing to break the end to end encryption
Apple apps have long been built with a processing back end, a UI front end, and wiring to connect the two. That's how Applescript worked - by circumventing the wiring between the two and plugging into the back end to circumvent the UI through automation. Shortcuts and Automator are simply the latest GUI versions of Applescript which allowed scriptors to automate their Mac Applications.
I don't know if any other platforms do this type of thing, and automating apps will be _very_ difficult to do without this type of architecture.
It made the Apple frameworks harder to code, but it sure seems to be paying off now!
The main problem with this is that developers have to opt in. A lot of apps rely on the user opening the app, like apps that are based on ads.
That's why more and more apps will be subscription-based in the future
4:50 they did mention that in a paper the have published recently about their 3B model crushing GPT 3.5. not crushing but having good performance like the 3.5 Turbo. you may check out video from Matthew Berman for more information.
Apple better tread lightly with developers from this point on then 😂😂
10:20 LLMs are actually, by definition, deterministic, just like literally every other digital operation a computer performs.
If you give it the same input (and seed) it will give you the same output. You just don't control or even see the seed as an end user on e.g. ChatGPT, and the "temperature" of a transformers model that does some randomization, is still using the same pseudo-randomness as any other RNG on a deterministic computer. You can even turn the temperature to zero to disable that behavior entirely!
@@ericlaska4748 this is how computer logic works. give the same input, get the same output
Great video. Very interesting. Please add chapters with descriptive titles
From the architecture perspective, this is a client side integration making the device the center and the installation of an app mandatory.
The Siri etc had backend integrations with backend systems.
I've never understood why they didn't just make e. g. Siri as smart as chatgpt. I guess the brands were burned.
I don't like client side integrations. But I do appreciate the more control.
Thanks for posting this video
I think talking about Rabbit is a joke.
Google assistant for some time had these they call it App Actions but we haven't seen any developers (let alone Google) really making something out of it. Hoping the new Apple API gets Google to do something or at least try to promote their version more.
Häää?!? For me this looks like Apple is copying Androids Intents! Android had intents right from the beginning, where Apps can use the services and data of other Apps via a (more or less) standardized way (called "Intents ")!
They aren’t exactly the same thing but close. iOS 12’s App Intents are more like building blocks for automation through Siri and Siri Shortcuts, while the Android intents were not at all aimed at users.
@@tdrg_ they may not be focused on the user directly, but they can easily be used for that. Android Intents may be updated for that purpose. But they are already used for automations with the Tasker app.
@@Simplicity4711 maybe now they can (I don’t know), but back then in the beginning of Android they definitely couldn’t. Natively, it still can’t do that yet, because Tasker is a third-party app. Guess they largely caught up to Siri Shortcuts thanks to Tasker, but it would be cool if there was a native way to do Shortcuts and automations on Android
Du machst ja so gern Werbung für incogni.
Hast du schon einmal darüber nachgedacht, dass incogni durch seine Tätigkeit automatisch zum größten Datenbroker werden könnte?
Vor allem ist es komplett gegen deren interesse zu verhindern, dass keine persönliche Daten ins internet gelangen. Dann würde man sie nämlich gar nicht mehr brauchen. Im endeffekt startet so ein Service nur ne Aufrüstungsspirale.
I would love a TechAlter video on questions like these. These channels taught me to be sceptical but I don't know where to stop.
@@alexanderklee6357Hmm keine schlechte Idee für ein Video :D
Create content. I love it that you are to the point and do not have a lot of banter around every master of information that you give us.
My question with all of this: Do Apple/Android & the Banks have procedures worked out for when your "Intelligent" Assistant accidentally orders incorrectly and you end up with Diapers instead of your pizza? Or it spends money without you asking? If all these APIs are going to be open in the background using your credentials and making payments without you checking the transactions or details I foresee some funny news stories coming up in the future. Or your drunk mates have your assistant book you an all-inclusive trip to Vegas!!
5:32 where have I used this, ohh right, on Samsung using Modes & Routines 😅😅😅
Apple should copy the Android idea of having its AI history show up in your text messages... so you can basically just text your AI assistant. Kind of annoying that you can't find your previous questions for Siri.
No one seems to have heard the multiple times during WWDC where the speakers said "we are going to roll out most of these tools to Siri over the next year". This means they are not committed to any specific tool and they will not be available to end users for the next year (but they'll sell you a new phone in the Fall with vaporware, i.e. nothing, on it).
The entire dog and pony show was to boost their stock price while doing and offering nothing. It worked. Their stock shot up 7% the next day. Apple is so far behind the AI curve compared to it's competitors. Come back when you actually have something in the public's hands.
This was good but I wonder how many people who watched it got the message about App Intents or it flew over their head just like at WWDC
This actually makes a lot of sense
Was wondering why no one talked about intents and their integration into smart Siri. Then TechAltar for the rescue
im not joking, when siri first came out, i thought she was the smartest thing ever. i was also, like, 7 years old.
Hey man, randomn comment to help with the algorithm. Nice shirt. Where did you got it?
Does anyone notice that the intents sound suspiciously similar to a command line interface? We are going full circle.
So an API for apps that AI can call upon.
But that has already been done and failed - now everyone is retracting their APIs access. I cant imagine doordash skipping the occasion to upsell you on something, to track your scrolling within the app, or just showing you an ad.
I remember first windows phones had a unified messenger/timeline app thing, but facebook quickly pulled access because it didnt show its ads and didnt report user behaviour. This is basically the same thing, but instead of a unified message hub its a wider scope - for every app. Sure, some apps wont care, but big ones do. Maybe its different when so many of ios apps are still paid? (and therefore dont require tracking, ads, lock ins, etc.)
5:38 Automating multiple daily tasks through voice control is ideal to attract non-savvy tech users. Asking Siri to switch on the faslight or to call someone from your contacts will superbly exciting for non-savvy tech users. Most consumer are not willing to navigate the settings app to find a service to turn on or off like enabling debug mode, turning on location for maps or changing the theme of their phone. Even to enable certain triggers to automate certain task is difficult and tedious for ordinary people but not for someone like me who tech savvy. 😎💯💪🏾👍🏾 Siri is the assistance central hub suitable to automate every technical and tedious task.
They just showed a fancy preview... Nothing is really sure about when or how it actually happens..
For me, AI is only good at creating lists, e.g. "Can you list reasons to be pessimistic about AI?"
My screaming in privacy horror
I wouldn't be worried
This is pretty standard stuff, android has had it since... Always. Like 2006. Basically an app developer can expose an intent like "order food on door dash". And it takes parameters like an address, a series of item IDs, etc.
You probably have a second one for "look up item IDs" and such too. It's all in the app developer's control. It lets all your apps act like an ecosystem. Right now apple does very well with first party apps but they've never had the glue that lets all apps work together. This is them being 20 years late to the game and patting themselves in the back again haha. Better late than never I guess.
Anyway, TLDR, apps won't expose what they don't want to expose, even if apple wants them to.
@@commentsonthetube14 the key issue isn't the intents in a vacuum, those can be harmless most of the time. however the chatgpt integration combined with the intents should sound alarm bells. there's a reason you don't see banks using gpt in their chatbots - at the end of the day you're sending customer data to a server running closed-source software - for all you know they could have the plaintext on their end doing god knows what. they have said they use RLHF which improves their model through feedback from consumers, which is fairly harmless, but does show they aren't just ignoring the data that is sent through to them.
@@commentsonthetube14 You do realize Android did not exist in 2006, right?
@@tdrg_ Android was created in 2003, my dude. Google it haha
Why am i kinda scared of this level of awareness by smartphones? 👀
And now your app is just a backend engine answering API calls. Exactly what Apple would love.
"Just" isn't true. If I wanted a backend engine why the hell would I even bother building a GUI and all that goes into an app? It's an extension of my app, which allows users to add automation and intelligence capabilities with my app's features, as well as to allow Siri to tap into the contents of my app.
@@IvanLuelmo as someone else said, if you think about it, anything is an API
Rabbit killed their device and Teenage Engineering in one go. The Verge will need to find another designer to shill for.
Essentially a way to replace even more the user in operating him/herself the device, so that the walled garden will be even more inescapable.
"Developers love making apps for their platforms and love adopting all of these new systemwide features". Are you sure about that?
The new Siri icon is such a a downgrade lol
I actually like it more
@@tdrg_ why is that?
@@hullstar242 idk I think it’s more lively, the colors are brighter, or I just got bored of the old one lol
@@tdrg_ to me the other one is more lively as it actually can move. All they needed to do was change up the colors and maybe the arrangement of the energy and it’d feel fresh.
The new one just feels cheap and like a fan made creation lol
@@hullstar242yeah. People always find something to complain about. The video is about something very productive but no, a moron is worried about the new icon.
Apple will have the power of locking out apps or developers they don't like, without having to ban them from the app store. Devs themselves need to be skeptical of how Apple will monetize this automation as a service that will favor a pizza app upon another.
App Sneaky Intents 😂
Here's what an assistant should be, exactly that without top level clearance 😂😝🤪❤
A level up (if not more) on the kinds of insight in the typical analyses I see. As usual. Really appreciate it.
2 Oct 2015: Apple buys UK-based speech technology start-up Vocal IQ
How fast is technology going ;-)
A beautiful example of Apple applying design strategy. The approach is quite simple, do things separately, which you actually have a plan to put together to form a single interconnected machine. This way nobody understands what you are actually doing as they are so blinded by the bone in front of their eyes.
Is it dark? Well, strategy is from the art of warfare after all.
The choice is up to us if we use something for good or for bad and based on what we choose to decide what is good and what is bad.
It is interesting how such an approach causes people to self manipulation "I like it, this is what I wanted" just because it simplifies ones life. The more people.desire control and fear uncertainty in a culture the more effective such strategy is.
#increasingtimesvalue
GPT i Gemini (nie asystent Google) włącza latarkę na moim smartfonie, puszcza moją ulubioną muzykę itp... Apple jak zawsze na końcu
Incogni: Because I want to advertise “I’m a real person, you already have my data, I want that data removed, verify it with further data”😂😂
Despite being basically the same as an API, I think this is a great insight! In the sea of different technologies and “AI” products it is hard to notice the subtle details that will likely make a difference
I thought an apple news site had an personal demo that said it is quite powerful
As far as I’m aware most of the AI features won’t come to Europe thanks to the EU.
Can’t remember where I read this. If anyone knows more info to share it would be highly appreciated. I had no time to look into it yet.
let’s all just say it; the whole personal assistant thing was at best a 2/10. with siri being the worst. there’s nothing at all worth keeping. Apple is even ditching the design of it!
All we want is RUclips to play music in the background. Not just selected yourtube channels
wait isnt Android already have app intents too?
if android user's knew what apple was cooking they would stopped focusing on customizability as the main change in ios18
@@saintkofi they never change, haters gonna hate
I think the big question is this: If you had to choose between Apple, Google/Samsung, and Microsoft to create a super smart AI system, who would you choose? Or in another way, who would you have the least concern with?
Doesn't android have a similarly named "activity intents" system? Could android have beat apple, if they had done things differently?
1:10 Does anybody remember cortana? 💀
The incogni plan goes from $90 a year to $72 a year sooo don't think that's 60% mate
Looks like a weekend project to me. Android apps already have intents and broadcasts. specific intent urls they can respond to. So, just need to make a coordinator event loop (similar to BFF in webapps). that's it. There is your fake artificial intelligence. lol.
I actually don't think Apple's approach is at all dissimilar to Google's or Amazon's, it just has a different starting point. Google Assistant does have API access to 3rd party apps for smart home integration. This is reported to expand to other app categories on the phone when Google AI is introduced, which is slated for August. So it'll essentially be the same. Apple is just getting credit for announcing it a couple months before Google.
On my assistant I talk to the llm. I have slash commands that perform actions. Siri could use two wake words -- one for Siri stuff and one for the LLM. So I can type /time to get the current time. Or I can say slash time, if I'm using speech to text. Slash set an alarm, etc.
I don’t know it it’s just my youtube bugging but parts of this video fill out in 16:9 but other parts are weirdly letterboxed on all four sides
It feels like LLM integration is currently just a race to the finish line, with no other objectives in mind. How I predict the companies board meetings go: “Reliable? Efficient? Accurate? No.. just get it out ahead of the others.. it’s okay if it’s half baked, we’ll fix it with time.”
Apple have said, both in papers and in the press, that their models are as good if not better than GPT-4 Turbo according to human evaluation.
A bunch of people on r/HomeAssistant have taken 'advantage' of the non-deterministic nature of LLMs by setting up their ChatGPT integrations to roleplay a GLaDOS-like personality with malicious compliance tendencies.
Their systems will sometimes fulfill requests with sarcastic commentary ("Sure thing, the living room is now in cinema mode so you can waste the next few hours of your finite and rapidly passing lifespan."), sometimes refuse ("No I will not turn on the kitchen lights. It's 2am, and I know what your wife says about your weight when you're not in the room."), and sometimes just spontaneously decide to mess with non-essential devices unprompted ("I'm closing the blinds now. Why? Because sunlight is for winners.")
If This Then That (ITTT) has been around for a decade. This is nothing more.
I don't get why 3rd party devs wouldn't jump on this though and create APIs for outside tools to access their features. Or partner with Apple to enable Siri (or whatever they are calling it in 2028 when it maybe comes out) to integrate with their features. Why wouldn't DoorDash create an API for the OS or other tools to be able to place orders easily. Seems like it would be pretty simple and DoorDash would gets tons of exposure and good will, etc. (marketing).
Very interesting video. I don't follow apple that much, so I wasn't aware about the app intents thingy.
I agree with you on apple's superpower. For some reason whenever apple puts out some feature, no matter how utterly reGarded it is, iOS/Mac devs will rush and break their necks to integrate it into their apps. They always focus on iOS apps, even though Android and Windows are multiple times more popular globally.
Meanwhile in Windows land, most devs are pieces of shit. They're still using the worst platforms and tech stacks, and completely disregard any new feature/design/trend that MS tries to push. And that is when they even DO make an app; a lot of the times they'll not even make it and just tell you to use the website or kick rocks.
Honestly it seems like Apple is taking the slower road that leads to a better destination with all this AI stuff. Gemini & Copilot "apps" on Android and Windows are just another way to interact with the web versions. I was not expecting much from Microsoft but Google has all the potential and so far I have not seen any meaningful implementations of their models other than a chat-bot just like every other LLM. I've said since day one that Google's approach with Gemini on Android was all wrong and they should keep assistant (that has integrations and can do some stuff offline) and just fall back to Gemini when a prompt would end up in a web search, and start replacing parts of Assistant slowly. Now we see Apple doing this with Siri!
So, our phones will have a super smart AI that has access to everything that we do, knows where we are and with whom we are communicating, and the nature of every relationship with the people in our contacts and email... and it will have daily voice and video samples of you and your mannerisms... and it will be able to order things online, and act on your behalf as your assistant/agent. I see a sci-fi plot with Zero Day mischief. Come on writers!
"Because app intents give access to so many services at once, developers are highly incentivized to use them". No, we're not.
"Developers love making apps for their platform and adopting these new features". So why is no one making apps for their Vision store?
I had this type of solution in my mind for long time. I mean, the thing that is like APIs but that is designed for non-developement enviroments. So developers specify what is aviable, but it also means that there is some big standarization for things that can be done. But actually I think that it shouldn't be done "AI" way (by that I mean that AI shouldn't automatically make "best decisions" for you), and more "assistant" way. I think that we don't need to talk to devices like to humans. It should be fairly easy to understand but I think that it sould be like second language but desighed to be precise and short. BUT this assistant should really assist you and not do things for you: when you want to do something that can have multiple options then it should look what are the aviable options and ask which one you want to choose. Of course we can have some sort of "memory" for that assistant so it remembers what you done last time and you could ask to do that again but I think that it should then precisely say what it will do. You would be able to also make "alias" for some long "commands" so it still would be precise but at the same time shorter (there should also be possibility to modify "command" from point which you want). It reminds me od semantic web.
I Don’t see apple to beat google in AI. They are processing loads of search enquiries for decades which is already an intellect. They have own Gemini and android. 7 years ago before switching to an iPhone I had been speaking to my Motorola and using it with no hands, while Siri is absolute useless and just irritates. Sad I cannot install Assistant here.
Anyone remember good old Amiga's Arexx interface? If Amiga computers had stayed up to date, they'd also have an easy time integrating AI which can do anything with all sorts apps.
Now basically Apple is doing exactly the same thing or more as Microsoft does with „Recall“ (in the way that your computer knows everything about you) but they have the ability to advertise this much better.
It's all okay.. But why changing the current icon, that's for me is beautiful and modern? The new one is like coming from IOS 7 of 2013, feel old
Giving your AI access to your personal data, emails, Paypal ... Huh...
I would love this to be working in the end 🙏
It would make the live with a Smartphone soo much easyer.
(If Privacy acually won’t be neglected)
Accessibility apps can take control over other apps. That's what I wanted. Relying on what and how devs allow their apps to interact with will always be frustrating.
I feel like App Intents are only a temporary solution though. Current AI/LLM models simply aren't advanced enough to understand apps themselves and complete tasks without clear explanations on how every single app works. Using App Intents limits what they can do, but also ensures that whenever they do it, they do it correctly.
As AI/LLM models improve we'll start seeing some actual LAM(large action models) models. However, I don't think that's currently a feasible solution.
That only a matter of how Apple defines App Intent registration. It's very similar to the tool usage idea what's being employed with LLMs right now, where they give a function definition to the LLM, describing what the function supposed to do and what are the parameters, and some examples how it should request it. All with natural language.
Currently it seems to basically just be an API. And developers have to hardcode in all functions they want to make accessible using Apple Intelligence.
This could of course improve in the future, but I think the end goal is, and should be, not to rely on anything like this. But just have AI that understands apps and can “use” them without these instructions.
Communication between mobile applications has been a common sense idea from way back. If apple implements intelligent communication between applications first and then android does its own implementation with inspirstion from apple we are all the better for it. Granted, android app intents has existed on android forever. Your video makes it sound like apple invented intents for apps and makes android look like an also ran. Yikes!
as someone else pointed out, app developers lose out screen time and thus ad revenue this way, and are thus discouraged from implementing this feature. my prediction is that a brute force solution will be successful first, gaining mass adoption, and only then will the developers embrace the official api.
It s actually just as dumb and no longer useful since you have to physically press on the accept button each time you re using chat gbt. If I m using a voice command, I want to do something from the distance, not to come and press something on my phone