Eye-tracking Mouse Using Convolutional Neural Networks and Webcam

An eye-tracking mouse that uses only a laptop's built-in webcam I built in Pytorch using convolutional neural networks and 2200 images of my left eye (sorry @ right eye). It's free, and implemented totally in software unlike the Tobii eye tracker.
Check out my website at andykong.org/
---- LINKS ----
Link to the code for this project: github.com/kon...
Link to face_recognition library: github.com/age...
Site for drawing my mouse location: processing.org...

[@amanraj2808]

why does it have so less views its amazing man

[@SOSBOY4EVER]

this feels like the beginning of something huge

[@pratikbharadwaj3315]

brilliant work, expect more of this in the future

[@nickz7396]

Really impressive. Eye like this.

[@HerezCheez]

Ugh!

[@kuzeysem1z]

that was my dream for every time of when i using computer. your been inspirationer for me

[@exxzxxe]

Good work, Andy!

[@ropori_piipo]

I would like to try this, check how accurate it is while eyes are tracking something on the screen. Eyes are better used for tracking and locking on to a moving object, I think.

[@SamZeloof]

Really nice but why are you using pyzo, kelly would be proud

[@simonstrandgaard5503]

Impressive. Nice

[@midosasuke5035]

Thank you, this will help a lot of people

[@chengjoseph9459]

Love your work Andy. Keep it up :)

[@minisvlog5440]

yea

[@81peas]

nice work

[@clrs8995]

💯

[@merveozdas3069]

Could you learn your code?I wonder it.Please.

[@mami_tomoe]

nice model

[@spoingus20]

Hi I have working with your model...Where did u train the y models?? couldn't find it in andyCNN

[@yan8054]

can you tell me how can I run the project please?? i really need your help

[@spoingus20]

@@yan8054 As far as I remember I used his Github only.

[@scart07]

excellent work!! could you please give us the tutorial or the site where you learned pytorch?thanks!

[@kong]

Sure! I learned most of my ML knowledge from Andrew Ng's Machine Learning course online, hosted at Coursera. The Pytorch just came along with wanting to do this project.

[@scart07]

@@kong thanks a lot!

[@scart07]

@Andy Kong hi again! Andy now i got trained a 2 output class, at the moment i can classifier between if, i look to the left or to the rigth, i am trying to do the same as you , that the CNN tell me the (x,y) where i looking at. i don’t know why but the CNN cant predict int number always i got numbers between (0,1) like a probability, but i see in your codes, we have a similar CNN structure but you have one CNN for x and other for y. but the output gives you exactly the coordenates. Also i see you implement some bloop detection and contours, but you don’t used it at all, don’t you. thanks for your advices about how can i get as a output the number for x and y.
my structure:
class Net(nn.Module):
def __init__(self):

super().__init__() # just run the init of parent class (nn.Module)
self.conv1 = nn.Conv2d(1, 32, 3) # input is 1 image, 32 output channels, 5x5 kernel / window
self.conv2 = nn.Conv2d(32, 64, 3) # input is 32, bc the first layer output 32. Then we say the output will be 64 channels, 5x5 kernel / window
self.conv3 = nn.Conv2d(64, 128, 3)
x = torch.randn(30,60).view(-1,1,30,60)
self._to_linear = None
self.convs(x)
self.fc1 = nn.Linear(self._to_linear, 1000) #flattening.
self.fc2 = nn.Linear(1000, 2) # 512 in, 2 out bc we're doing 2 classes (dog vs cat).
def convs(self, x):
# max pooling over 2x2
x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))
if self._to_linear is None:
self._to_linear = x[0].shape[0]*x[0].shape[1]*x[0].shape[2]
return x

def forward(self, x):
x = self.convs(x)
x = x.view(-1, self._to_linear) # .view is reshape ... this flattens X before
x = F.relu(self.fc1(x))
x = self.fc2(x) # bc this is our output layer. No activation here.
return F.softmax(x, dim=1)

[@kong]

@@scart07 The neural net outputs will always be between 0 and 1, try to normalize your y inputs before training in order to predict left and right. For instance, take the pixel value and divide by the max pixel value as the y, then multiply the outputs by the max pixel value to get the predicted pixel value.

[@scart07]

@@kong excelent !! i am going to try it! thanks a lot!!! =)

[@minisvlog5440]

how to run this project please tell me

[@kong]

Instructions in the github!

[@minisvlog5440]

@@kong it gives face detection error

[@minisvlog5440]

i have my final year project.. plz help me.. send me error free project
[e-mail]

[@prathikbafna8352]

@@minisvlog5440 hi bro did this project work for you?

[@minisvlog5440]

yes

[@nastyafom92]

does anyone have a working version of this program?

[@yan8054]

do you make it work? i need some help😭