I would *highly* recommend putting this training feature into the app instead of a website. On the web, with Android's default gesture navigation, it's far too easy to accidentally trigger the "back" gesture when trying to swipe words that start near the edges of the screen. This doesn't happen with the real keyboard app, but on web it constantly screws up your input and backs out of the tab before you can undo the input and try again.
Another thing that I have noticed is that "spelling hesitation" does have an effect on swiping. If you only have non hesitant examples in your swipe data that will not help you when people actually use the final product. Edit: And I suspect that you may run into this when you attempt to map the english dataset to non english, which likely has different "spelling hesitation" characteristics.
@@user-Aaron- Sure, lets say that you thinking of the sylibols (syllables) of a word SYL, now I am on the way to the letter "I" from "L", but on the way to "I" realize my mistake, maybe I pause, then head for the letter A. Then BLES. This effects the shape and timing. (This is a distinct issue from outright misspellings like SYLIBOLS) Now I know you might say, if I make a mistake, start over, yes, but ideally that is a last resort especially for longer words. If I find that I constantly need to start over every fifth word I will be tempted to switch back to tap typing with the word suggestions.
@@derstreber2 Okay, that is what I figured you'd meant. I think the problem though is that hesitation is an inherent feature of swiping. As I mentioned in another one of our conversations here, hesitation is used to specify things like doubled letters. I do think that you need to be fairly proficient in whatever language you are swiping in in order for it to be worthwhile, but it also gets easier the more you use it. I feel like spelling accuracy also goes up the more you swipe because you're effectively having to plan out your sentences more in advance since you're thinking words ahead instead of letters. I can only speak for myself of course but nearly all of the errors that occur while I swipe on a keyboard are from the keyboard not correctly identifying my swipe input for one reason or another, not from me misspelling/mis-swiping a word. Also, you don't necessarily need to even swipe a word out fully because of predictive text, and that's especially true for longer words because the word gets increasingly specific as you swipe more letters. For example, to swipe "incomprehensibly" I really only need to swipe out "incompreh" before it automatically predicts "incomprehensibly" and I just have to lift my finger for it to be automatically typed out. Also, if I instead mean to spell "incomprehensible" then I can kind of be careless with my swiping after swiping out "incompreh" because at that point there are really only two possible words: "incomprehensibly" and "incomprehensible". So as long as I don't somehow trick the keyboard into thinking I meant an entirely different word and make it switch to that instead, I really just have to make sure I end my swiping on the "e". EDIT: Hesitation may also be used as an indicator that we have made a mistake in our swiping and that if we continue swiping we may be making an attempt to correct it. For example, if I am trying to spell "blueberry" but accidentally swipe "ble" and then pause before continuing to swipe "uberry" it will sometimes still correctly output "blueberry". This is unreliable and inconsistent though and no doubt the likelihood of it happening is dependent on a number of variables, such as the particular word, its length, how unique the swiping pattern is, etc.
Nintype swiping fixes this by using spacebar as the end-of-word marker. this allows swiping with both hands alternately *on the same word*. i think it would fix hesitation too
No. Open source doesn't pay well for developers. And I really hate that good developers are forced to either live off on donations or work against their will for projects they are not passionate about. We want honest computing. FOSS is not a good business model, but things are changing.
What i have understood is that the swipe typing in the nightly version became far better than the normal version. I have swiped a lot in that website where we can contribute in building the swipe typing database
Problem is when users see a compound word, we want to swipe it in sections but the training model website doesn't let us compound-swipe (swipe, change keyboard, add dash, swipe rest).
When there is a bluetooth KB already attached to my phone the FUTO KB doesn't minimize automatically. Other input/swipe kbs do minimize when the usb keyboard is attached.
Just wondering: would the "Typing Heatmap" shown in Microsoft SwiftKey Keyboard be a useful data set for you guys? Also (depending on how exactly you're using the data you're collecting this may not be relevant), my only concern with the swiping exercise so far is that I can't tell if the word I just finished swiping would've actually came out spelled correctly if I'd done the same motion on Microsoft SwitKey Keyboard, Gboard, etc.
Part of the problem is that futo has no way of knowing if the word you swiped was correct, if they did this whole training exercise would not be necessary. I suppose they could try doing what Microsoft did when Bing first came out. Bing basically did a google search, compared it with their own search and tweaked their algorithm to be closer to google. But that would require futo to somehow get data from other keyboards, interactively. My understanding of what they are doing right now is like this. Say the word is "basket", what they end up with is hundreds of swipes that all claim to represent the word "basket", but all of these swipes will be different, (some will be completely wrong data, some of the data will be translated slightly to the right or the left, and this could be because of many reasons, different screen resolutions, right/left handed. etc) Then you shove all that data into a program that tries to find patterns in the data.
@@derstreber2 Yes, that is how I would assume they'd be doing it too but that's not necessarily the case. Or, they could be doing that but could also be using the data in other ways, like to train their LLM, which could potentially lead to decreased accuracy if not filtered and curated first. Using real examples, when I swipe the letters "antimony" on Microsoft SwiftKey Keyboard, "antimicrobial" is what comes out. Also, words like "decreased" and "deceased" are swiped very very similarly but presumably "decreased" is much more commonly used. And when I mean to spell "hello" but swipe too similarly to "help". All of the above could lead to a poorly trained LLM that gives poor results.
Im a fan of the keyboard, i keep changing back to my keepass keyboard for password manager access. Maybe give an option for customizing the emoji key to password manager ?
One question that I have, is swipe data coloring pixels or is it like nodes in a spline? If it is nodes, is the velocity of any given node in the swipe recorded or is it just the shape? Along with that question, can you tell when a finger rests on a key? (Either by multiple nodes being placed on top of eachother or timing data of some sort) If multiple nodes are not laid down (in the case of coloring pixels or in the case where nodes are not laid unless there is a large enough delta in motion), I would suggest figuring out how to capture that sort of information. Now that I think of it you could fill in pixels with a heat map of sorts, but that is less ideal with words that hit the same key multiple times because the line between frequency of a key press and duration of lingering is blurred .
Whether swipe speed is relevant or important may indeed be one of the things they're trying to figure out with this whole thing. Because if it is important, you can bet none of the corporations will share even that much information.
I'll try to hit 1500 words tonight. For this dataset I've caught myself slowing down to be more accurate. Particularly on words I don't know. That's not how I usually swipe type. I wonder what effect that will have on the model. I'm sure we all swipe the words we frequently use much faster. On commercial swipe keyboards I don't even think about spelling because it's usually right even when I'm wrong.
hey guy, nice work you're doing. i've been looking for such a keyboard since a long time. i support your open source approach very much. any plans to put out a japanese language pack any time soon?
A company saying "our thing isn't working too well" then laying out how they will fix it feels jarring. Also add a point system to encourage hard words.
Since there is FUTO Keyboard and FUTO Voice (both focused on eliminating cloud hosted privacy risks), I was thinking that OCR scanning of documents like receipts, etc., could benefit from a FUTO overhaul so that details of what should be a private transaction stays private.
OCR isn't much of a concern in that sense. It's the problem that was solved the longest time ago. It's already done fully locally in almost all implementations. Your only concern is where that information is stored before (picture) and after (text) the process. But that is completely outside the scope of an OCR app.
I have C1 fluency in French, German and English (and also native level luxembourgish, but let's be honest, that is NEVER going to happen) and am motivated to contribute. Once you get to the translation stage, I'd be happy to swipe about 2-4k identical sets of words in each of these languages if you want
I added some data, I hope you strongly bias towards short/simple words even beyond their frequency ranking since I assume most users switch to typing as soon as you have to think about spelling. I often accidentally swipe in the wrong direction and Apple will usually figure it out. They also make a lot of adjustments based on at least the 2-3 prior words. I guess my favorite words to swipe are the longer words but only when they are easy to spell like ‘through’ ‘direction’
One thing that I have noticed while submitting swipe data (about 1300 words so far) is that there are letters that are often blocked by my finger, this makes it hard to directly target a key. This results in arcing finger motions in order to visually see where to swipe to. I would suggest NOT centering the letters on the keys vertically, instead have the letter at the top of the key. This way I could target under the letter I want to select. This single change would make my swipes much more predictable. Edit: Also, especially with longer more complex words, the swipe line often fully obscures targets. This means that the longer the word is, the worse I get at targeting letters. If you could implement a fading transparency. (something like, directly under finger = 80% opacity, start of swipe = 10% opacity)
@@user-Aaron- No I don't. Is there something obvious that I am over looking? Edit: I mean, I hope I am not using it wrong. But now that you bring it up that is a potential issue with the website, because it is not an actual keyboard it does not give you feedback.
@@derstreber2 No, I was just wondering because as a longtime swiper I don't actually look at the keyboard at all while I'm swiping. It's become second-nature to me just like using a physical keyboard. I figured that was probably true for virtually all regular swipers otherwise they'd have just stuck to tap typing, but good on you for giving it a go!
@@user-Aaron- Yeah I did find that as I was getting more and more use to it, there were some keys that I could reliably accurately swipe without looking, keys near the edge of the screen (my phone case naturally stops my finger so there is no overshoot) But keys that are lower, in the middle, and often obscured, those keys I could not do without looking. And I know that the AI magic will sort of smooth out your fat fingers. Example, you type "Ravvit" you probably meant "Rabbit". But the less stuff you NEED to smooth out the better the keyboard will perform in the rare cases where your misspelling is closer to a valid word that you are not trying to type. Tell me, do most swipe keyboards have a way of going into a tap mode for cases where you just need to enter specific acronyms or symbols?
@@user-Aaron- Also, when you type double letters, (ll, ss, ee, etc) do you make a small motion to indicate that you want another of the same letter or do you treat it as one thing and trust the magic that it understands? 😀
AI is great for these sorts of problems. The issue isn't figuring out what letters you went over, the problem is figuring out the intention behind a swipe, sometimes your finger doesn't even go over most of the characters of the word, yet Gboard figures it out from the shape of the swipe and the context, sometimes you hesitate and you pass over letters that will mislead a simple algorithm
@@xymaryai8283 At least the keyboard shouldn't try to fix your perfectly fine typed swear word. A solution would be a list of words that should not be corrected neither corrected to.
Anuyone know how to get the entire post im typing in youtube spellchecked at once in word style easily on android? Keybaord spellchecks only do the current word and weirdly they dont highlight when something is sus. Weirdly enough Futo keyboard doesn't distinguish which of the identical suggestions is the one I need to add a new unrecognized word.
I've been unimpressed by the swipe on the FUTO keyboard, to be honest. It's not the worst, but it does feel less accurate than Gboard or Samsung keyboard.
In regards to the data set needing complex words. I think that common words are better, most people swipe common words and I dont think anyone would even attempt to swipe complex words. By adding all these weird complex words to the model all you are doing is making it unsure if i swiped the common word or some complex word. From using the Futo keyboard i am often really confused by the recommendations because I am typing a simple word but then a random complex word comes up as the first suggestion. No idea why those are suggested. Its a hard challenge to build a good predicitive text/swipe but its interesting to watch you guys do it.
@@user-Aaron- If anything, complex words are where I want swipe the most. Manually typing "if" isn't a huge deal, but if I try to swipe "supercalifragilisticexpialidocious" and it doesn't work, I _will_ be raging with every character I have to tap lol.
@@egarcia1360 Yes, exactly. Honestly with that word in particular I would just use a search engine, swipe _supercar,_ backspace the _r,_ and then copy and paste the result lol Boom: _Supercalifragilisticexpialidocious,_ none of which was tapped.
I would *highly* recommend putting this training feature into the app instead of a website. On the web, with Android's default gesture navigation, it's far too easy to accidentally trigger the "back" gesture when trying to swipe words that start near the edges of the screen. This doesn't happen with the real keyboard app, but on web it constantly screws up your input and backs out of the tab before you can undo the input and try again.
Another thing that I have noticed is that "spelling hesitation" does have an effect on swiping. If you only have non hesitant examples in your swipe data that will not help you when people actually use the final product. Edit: And I suspect that you may run into this when you attempt to map the english dataset to non english, which likely has different "spelling hesitation" characteristics.
Can you give an example of an effect that "spelling hesitation" has on swiping? And what would be an example of a "hesitation example"?
@@user-Aaron- Sure, lets say that you thinking of the sylibols (syllables) of a word SYL, now I am on the way to the letter "I" from "L", but on the way to "I" realize my mistake, maybe I pause, then head for the letter A. Then BLES. This effects the shape and timing.
(This is a distinct issue from outright misspellings like SYLIBOLS)
Now I know you might say, if I make a mistake, start over, yes, but ideally that is a last resort especially for longer words. If I find that I constantly need to start over every fifth word I will be tempted to switch back to tap typing with the word suggestions.
@@derstreber2 Okay, that is what I figured you'd meant. I think the problem though is that hesitation is an inherent feature of swiping. As I mentioned in another one of our conversations here, hesitation is used to specify things like doubled letters.
I do think that you need to be fairly proficient in whatever language you are swiping in in order for it to be worthwhile, but it also gets easier the more you use it. I feel like spelling accuracy also goes up the more you swipe because you're effectively having to plan out your sentences more in advance since you're thinking words ahead instead of letters.
I can only speak for myself of course but nearly all of the errors that occur while I swipe on a keyboard are from the keyboard not correctly identifying my swipe input for one reason or another, not from me misspelling/mis-swiping a word.
Also, you don't necessarily need to even swipe a word out fully because of predictive text, and that's especially true for longer words because the word gets increasingly specific as you swipe more letters.
For example, to swipe "incomprehensibly" I really only need to swipe out "incompreh" before it automatically predicts "incomprehensibly" and I just have to lift my finger for it to be automatically typed out. Also, if I instead mean to spell "incomprehensible" then I can kind of be careless with my swiping after swiping out "incompreh" because at that point there are really only two possible words: "incomprehensibly" and "incomprehensible". So as long as I don't somehow trick the keyboard into thinking I meant an entirely different word and make it switch to that instead, I really just have to make sure I end my swiping on the "e".
EDIT: Hesitation may also be used as an indicator that we have made a mistake in our swiping and that if we continue swiping we may be making an attempt to correct it.
For example, if I am trying to spell "blueberry" but accidentally swipe "ble" and then pause before continuing to swipe "uberry" it will sometimes still correctly output "blueberry". This is unreliable and inconsistent though and no doubt the likelihood of it happening is dependent on a number of variables, such as the particular word, its length, how unique the swiping pattern is, etc.
Nintype swiping fixes this by using spacebar as the end-of-word marker. this allows swiping with both hands alternately *on the same word*. i think it would fix hesitation too
Thanks so much.
I know so many proole complain and whine, but this is the biggest step forward to open source computing.
No. Open source doesn't pay well for developers. And I really hate that good developers are forced to either live off on donations or work against their will for projects they are not passionate about. We want honest computing. FOSS is not a good business model, but things are changing.
Writing from Futo Keyboard! Please keep working on the swipe algo! Love the project.
What i have understood is that the swipe typing in the nightly version became far better than the normal version.
I have swiped a lot in that website where we can contribute in building the swipe typing database
Problem is when users see a compound word, we want to swipe it in sections but the training model website doesn't let us compound-swipe (swipe, change keyboard, add dash, swipe rest).
That threw me off too the first time but maybe they're trying to train their LLM to account for hyphenated words?
When there is a bluetooth KB already attached to my phone the FUTO KB doesn't minimize automatically. Other input/swipe kbs do minimize when the usb keyboard is attached.
By the way, Gboard does support some hyphenated words in the same way that FUTO Keyboard wants to support them.
fuck gboard
Just wondering: would the "Typing Heatmap" shown in Microsoft SwiftKey Keyboard be a useful data set for you guys?
Also (depending on how exactly you're using the data you're collecting this may not be relevant), my only concern with the swiping exercise so far is that I can't tell if the word I just finished swiping would've actually came out spelled correctly if I'd done the same motion on Microsoft SwitKey Keyboard, Gboard, etc.
Part of the problem is that futo has no way of knowing if the word you swiped was correct, if they did this whole training exercise would not be necessary. I suppose they could try doing what Microsoft did when Bing first came out. Bing basically did a google search, compared it with their own search and tweaked their algorithm to be closer to google. But that would require futo to somehow get data from other keyboards, interactively.
My understanding of what they are doing right now is like this. Say the word is "basket", what they end up with is hundreds of swipes that all claim to represent the word "basket", but all of these swipes will be different, (some will be completely wrong data, some of the data will be translated slightly to the right or the left, and this could be because of many reasons, different screen resolutions, right/left handed. etc) Then you shove all that data into a program that tries to find patterns in the data.
@@derstreber2 Yes, that is how I would assume they'd be doing it too but that's not necessarily the case. Or, they could be doing that but could also be using the data in other ways, like to train their LLM, which could potentially lead to decreased accuracy if not filtered and curated first.
Using real examples, when I swipe the letters "antimony" on Microsoft SwiftKey Keyboard, "antimicrobial" is what comes out. Also, words like "decreased" and "deceased" are swiped very very similarly but presumably "decreased" is much more commonly used. And when I mean to spell "hello" but swipe too similarly to "help". All of the above could lead to a poorly trained LLM that gives poor results.
Im a fan of the keyboard, i keep changing back to my keepass keyboard for password manager access. Maybe give an option for customizing the emoji key to password manager ?
One question that I have, is swipe data coloring pixels or is it like nodes in a spline? If it is nodes, is the velocity of any given node in the swipe recorded or is it just the shape? Along with that question, can you tell when a finger rests on a key? (Either by multiple nodes being placed on top of eachother or timing data of some sort) If multiple nodes are not laid down (in the case of coloring pixels or in the case where nodes are not laid unless there is a large enough delta in motion), I would suggest figuring out how to capture that sort of information. Now that I think of it you could fill in pixels with a heat map of sorts, but that is less ideal with words that hit the same key multiple times because the line between frequency of a key press and duration of lingering is blurred .
Commenting for notification in case they respond. I'm wondering this too, it feels like swipe speed should play a role in word prediction.
Whether swipe speed is relevant or important may indeed be one of the things they're trying to figure out with this whole thing. Because if it is important, you can bet none of the corporations will share even that much information.
Do you have a num pad layout?
I'll try to hit 1500 words tonight.
For this dataset I've caught myself slowing down to be more accurate. Particularly on words I don't know. That's not how I usually swipe type. I wonder what effect that will have on the model. I'm sure we all swipe the words we frequently use much faster. On commercial swipe keyboards I don't even think about spelling because it's usually right even when I'm wrong.
Swipe like you would on Gboard and SwiftKey and others, that what the dataset should reflect
hey guy, nice work you're doing. i've been looking for such a keyboard since a long time. i support your open source approach very much. any plans to put out a japanese language pack any time soon?
A company saying "our thing isn't working too well" then laying out how they will fix it feels jarring. Also add a point system to encourage hard words.
Since there is FUTO Keyboard and FUTO Voice (both focused on eliminating cloud hosted privacy risks), I was thinking that OCR scanning of documents like receipts, etc., could benefit from a FUTO overhaul so that details of what should be a private transaction stays private.
OCR isn't much of a concern in that sense. It's the problem that was solved the longest time ago. It's already done fully locally in almost all implementations.
Your only concern is where that information is stored before (picture) and after (text) the process. But that is completely outside the scope of an OCR app.
I have C1 fluency in French, German and English (and also native level luxembourgish, but let's be honest, that is NEVER going to happen) and am motivated to contribute.
Once you get to the translation stage, I'd be happy to swipe about 2-4k identical sets of words in each of these languages if you want
I added some data, I hope you strongly bias towards short/simple words even beyond their frequency ranking since I assume most users switch to typing as soon as you have to think about spelling. I often accidentally swipe in the wrong direction and Apple will usually figure it out. They also make a lot of adjustments based on at least the 2-3 prior words. I guess my favorite words to swipe are the longer words but only when they are easy to spell like ‘through’ ‘direction’
One thing that I have noticed while submitting swipe data (about 1300 words so far) is that there are letters that are often blocked by my finger, this makes it hard to directly target a key. This results in arcing finger motions in order to visually see where to swipe to. I would suggest NOT centering the letters on the keys vertically, instead have the letter at the top of the key. This way I could target under the letter I want to select. This single change would make my swipes much more predictable.
Edit: Also, especially with longer more complex words, the swipe line often fully obscures targets. This means that the longer the word is, the worse I get at targeting letters. If you could implement a fading transparency. (something like, directly under finger = 80% opacity, start of swipe = 10% opacity)
Genuinely curious, do you not normally use a swipe keyboard?
@@user-Aaron- No I don't. Is there something obvious that I am over looking?
Edit: I mean, I hope I am not using it wrong. But now that you bring it up that is a potential issue with the website, because it is not an actual keyboard it does not give you feedback.
@@derstreber2 No, I was just wondering because as a longtime swiper I don't actually look at the keyboard at all while I'm swiping. It's become second-nature to me just like using a physical keyboard. I figured that was probably true for virtually all regular swipers otherwise they'd have just stuck to tap typing, but good on you for giving it a go!
@@user-Aaron- Yeah I did find that as I was getting more and more use to it, there were some keys that I could reliably accurately swipe without looking, keys near the edge of the screen (my phone case naturally stops my finger so there is no overshoot) But keys that are lower, in the middle, and often obscured, those keys I could not do without looking. And I know that the AI magic will sort of smooth out your fat fingers. Example, you type "Ravvit" you probably meant "Rabbit". But the less stuff you NEED to smooth out the better the keyboard will perform in the rare cases where your misspelling is closer to a valid word that you are not trying to type.
Tell me, do most swipe keyboards have a way of going into a tap mode for cases where you just need to enter specific acronyms or symbols?
@@user-Aaron- Also, when you type double letters, (ll, ss, ee, etc) do you make a small motion to indicate that you want another of the same letter or do you treat it as one thing and trust the magic that it understands? 😀
again, why not do an opt-in system on the keyboard itself? there is a lot more data to be had...
Futo is all about not gathering user data ^^
If it is a voluntary opt in, I don't see an issue with it.
i somehow feel like you should not need any neural networks or AI running in order to match a swiped curve to a word.
AI is great for these sorts of problems.
The issue isn't figuring out what letters you went over, the problem is figuring out the intention behind a swipe, sometimes your finger doesn't even go over most of the characters of the word, yet Gboard figures it out from the shape of the swipe and the context, sometimes you hesitate and you pass over letters that will mislead a simple algorithm
Please, include obscene words. People use them.
Agreed. I don't know many times a swipe keyboard filled my writing with unintended "duck" and "shot".
its always worse to accidentally swear than it is to accidentally quack
@@xymaryai8283 At least the keyboard shouldn't try to fix your perfectly fine typed swear word. A solution would be a list of words that should not be corrected neither corrected to.
Anuyone know how to get the entire post im typing in youtube spellchecked at once in word style easily on android? Keybaord spellchecks only do the current word and weirdly they dont highlight when something is sus.
Weirdly enough Futo keyboard doesn't distinguish which of the identical suggestions is the one I need to add a new unrecognized word.
I've been unimpressed by the swipe on the FUTO keyboard, to be honest. It's not the worst, but it does feel less accurate than Gboard or Samsung keyboard.
That's the point of allowing users to contribute swipes
In regards to the data set needing complex words. I think that common words are better, most people swipe common words and I dont think anyone would even attempt to swipe complex words. By adding all these weird complex words to the model all you are doing is making it unsure if i swiped the common word or some complex word. From using the Futo keyboard i am often really confused by the recommendations because I am typing a simple word but then a random complex word comes up as the first suggestion. No idea why those are suggested.
Its a hard challenge to build a good predicitive text/swipe but its interesting to watch you guys do it.
FWIW I absolutely do swipe complex words. I all but refuse to tap type anything if I can avoid it.
@@user-Aaron- If anything, complex words are where I want swipe the most. Manually typing "if" isn't a huge deal, but if I try to swipe "supercalifragilisticexpialidocious" and it doesn't work, I _will_ be raging with every character I have to tap lol.
@@egarcia1360 Yes, exactly. Honestly with that word in particular I would just use a search engine, swipe _supercar,_ backspace the _r,_ and then copy and paste the result lol
Boom: _Supercalifragilisticexpialidocious,_ none of which was tapped.
We have been specifically complaining that long words aren't swipable right now.
Ideally they should also weight it by likelihood of it being the next word in a sentence as well
Make the english one work good first before you start thinking about other languages. Atm swipe does not work at all even in english.
work _well_ 😉
Yup, that's what they said that they are doing and what they said they are addressing