Autonomous Self-Learning Robot (Q-Learning)
HTML-код
- Опубликовано: 25 май 2015
- This video illustrates the work performed in the
context of our bachelor's thesis.
The project was conducted in collaboration
with Oslo and Akershus University College of
Applied Sciences.
The purpose of the thesis has been to elucidate
the main methods of self-learning systems, and
develop a self-learning algorithm for an
appropriate demonstrator with the help of
a machine learning method called Q-learning.
The work is focused on the development of a
demonstrator in the form of an autonomous
car, known as an agent.
Codes and the final Thesis can be viewed and downloaded from the following sites.
Codes (MATLAB): github.com/kennydl/Reinforcme...
Thesis [Norwegian]: www.dropbox.com/s/mqkjhie1ahn...
Enjoy!
One of the very few robot with very professional presentation,thumbs up
Good work mate, really impressive
that was really beautiful... absolutly great
This is great! This is the sort of stuff I'd like to get doing eventually.
Hi! Please check out the new description for this video. You can now download the final report and MATLAB codes.
Great work bro good luck for the future
That is the most interesting video on the topics of ai..i ever seen..so crystal clear..so it basically gets.. certain amount of consciousness to do particular tasks
would be awesome to watch multiple robots going around the arena learning to avoid each other. $250 a piece would get a bit pricy though. Great work and thanks for sharing!
I agree! And with the correct speed and shape of the surroundings I'm sure we would've seen quite an setting of controlled chaos!
Well done! A very interesting project!
Thank you!
Seriously impressive. 😙
Hi, thank you for sharing nice paper and code. I have a question for ros simulation. I am doing the exactly same thing using UAV model but I do have a problem when running a simulation. When you run the simulation did you change the 'real time factor' which accelerates the simulation speed or else just run it as default speed. Also, I want to know the time for your training.
Awesome well done guys
Awesome project! Congrats!
Thank you!
Damn this was good
Amazing work guys!
Thank you!
This was very nice
This is awesome...... can you please provide us an English version of the thesis?
This is cool stuff
We did the same project
But didn't visualise it using matlab
good job an excellent work..
in hardware model what are the components do you used??
and what tool box you used in Matlab to simulate this code??
reply me a
sir asap
Отличное видео.
i know this video is old now but.. by the end of it, it seems the robot had learnt to follow things rather than avoid them all together. cool video too.
The agent is not programmed to understand if it's following or meating a object. If the object is close enought it will do an action in order to avoid it no matter what. I think following an object will give the agent to high rewards as the surroundings are allready "leaving". We should have implemented a way for the agent to understand this. We should also have sensors all the way around it. Thanks for the input!
Keep the good work !
Thanks!
awesome
What dc motors did you use and how do you control their speed precisely
great work!
Thanks!
that was beautiful... I am thinking of doing a project like that w/ Arduino or something .
Ain't Nothing To It But To Do It!
@@atlefjellangsther8073 can we do it by aurdino?
@@motibekuma530 Yes. He used an arduino to make this. But he also used matlab to code it so find out how to do that.
Nice work
Hi,
The Q-learning algorithm runs on the robot (on the Arduino) real-time or runs it on a PC and the PC sends control commands for the robot?
Hi! The Agent uses a Arduino UNO to continously read measurements from the sensors. The readings are then sent to a computer nearby using XBee. The algorithm it self is developed in Matlab and runs on the computer. The program returns the Agents next action trough the same channel. The Arduino-program is only controlling motors, reading measurements and sending/recieving data.
Thanks : )
Wich library?
is that code is for simulation alone ? to which function/file we will give ultrasonic value as input
Good job 👍
Do you have a git hub I can look at?
likewise
I Have a 4 port USB hub if that will help?
Best comment so far! ;)
Hi! Please check out the new description for this video. You can now download the final thesis and MATLAB codes.
@@LanDiEvil nice joke 😁😁😁
Amazing!
Nice work.
Would like to see more of this.
Thank you!
wow!
Hey Atle,
I am currently planning on my own thesis project an have a similiar idea (Arduino / Pi controlled drone with environmental mapping). is there any chance of getting your thesis to read? Are you on research gate?
@Norm T - Part of advancing the state-of-the-art is being intimately familiar with the current state-of-the-art.
Norm T, writing a thesis with no sources or references is a bad idea, probably impossible. Even the proposal needs references to current research in the area to prove it's something worth looking at.
Norm T - a person that will never need to worry about what they are speaking of.
Hi! Please check out the new description for this video. You can now download the final thesis and MATLAB codes.
Can u plz make a tutorial or give some instructions for assembly the parts (hardware, wiring) of this robo ?
Amazing
Thanks!
excelente nice
It's a great model. I really want to learn how to simulate a robot model in matlab. I programmed my proposed model to the robot directly without simulate it.
good job guys!!
Thanks!
Sir please upload english thesis.
This was great sir
Dear, wery nice, could be possible download the code? Could you tell me the link
tks
roberto
Great AI and Arduino so Cool
Thanks!
Just curious how to manage the second or third false echo in such confined space.
What boards do you use for learning?
I don't think that machine is learning, as the only sensors I can see is a bunch of a ultrasonic sensors. What does the machine actually 'learn'? The u/s sensors just output a distance, which could deemed as an arbitrary number at an arbitrary interval time period, can't see the correlation of these numbers in your context, if you could shed some light, it would be awesome!
Maybe he's registering the values that the ultrasonic sensors get and he can somehow calculate where the robot is with some accelerometer sensor, you can do that. It might also have some memory card. Therefore that could be "remembering", not sure if "learning"... Learning is also changing the way the robot thinks and I don't think he does that..
I mean since the robots are probably gettings images as inputs and using deep q-learning... The images are fed through a convolutional neural network and that CNN can definitely learn some highly abstract features that humans can't describe. But the machine can.
something to do is have it achive some goals, and comprehend, fail then wait then expressing
It's not fed with any instructions. It's learning on it's own to keep running.
Very good. I would be interested to read if you have any papers or documentation available
Hi Kansai! The papers are a part of a PhD our employer made after our project. I will look into if these papers are open for the public. Stay tuned!
Hi! Please check out the new description for this video. You can now download the final thesis and MATLAB codes.
Oh! so impressive. If you used q learning, I wonder how you decided on the next state. :)
Yes, we used Q-Learning. The next state was givven by the a change in distance between the agent and the surroundings.
Can you provide the C++ code for how to interact the agent(car) with MATLAB? I mean the code that's been uploaded to Arduino and how the communication has been setup.
Great!, What method did you use to do the Q-Learning ? Dynamic programming or neural network or what ?
Hi Mahmoud! We used both Neural Network and Q-Learning for then to deside witch is best. They both worked, but for the final version we used Q-learning, as we found it easier to see progression and changes in dessisions made by the agent.
- Atle
Thanks, is the source code available anywhere ?, as I am working on this github.com/harvitronix/reinforcement-learning-car but think that your code would be very helpful too because of the use of actual hardware ?
The source codes are at the moment not avaible for the public, I'm afraid. We are working on it.
The source codes are not available for downloading yet, I'm afraid. We'll make sure you are notified when source codes are ready!
Hi! Please check out the new description for this video. You can now download the final thesis and MATLAB codes.
Great Work sir would like to make myself one day
Thank you! And go for it!
wooow amazing. i am doing the same project for my MSc. how can i get document and simulation?
Thanks to post your thesis and the codes, maybe you project can help me in the future!
But i have a question what is the function of the bluetooth module? Sorry if you don´t undestand but English isn´t my fisrt languaje.
Hi Brando. The bluetooth module send all meauserements from the sensors to an computer running a MATLAB program. The program then runs the algorithm and returns a new action to the agent. This action is returned to the agent by using Bluetooth.
awesome! dude, tell me one thing how did you came to know how much is the range of those 4 sensors combined
you can set it
VIDEO STAR we can set the range of sensors in linear direction right? I want to know angular how wide it will go
Hi Shivam! We measured this in the beginning of the project. We had to now the accuracy and the range of all the sensors. We made a large map on the floor plotting stable readings while checking the distance from each sensor in real time.
Thanks a lot buddy!
It opensource on GitHub?
So Arduino doesn't think on its own?
Am I correct that you are communicating with more powerful PC which has the model and sends out the signals?
That's correct. The Arduino it self does nothing except sending data to the nearby computer and receiving/executing next action calculated by the computer which holds the algorithm.
Hi, very impressive !, I am working on robot motion planning project with deep Q-Learning, is it possible to share your code (with dynamic obstacles) ?, it would be very helpful for me. I can also share my project. Thanks !
Hi Mahmoud. The source codes are not available for downloading yet, I'm afraid. But the algorithm for the static obstacles and the dynamic obstacles are the same. Given a speed no higher than the reaction time of the agent, the codes work quite well also for dynamic obstacles. Good luck with your project!
Because you have multiple hcsr-04, How do you deal with ghost echo?
what is a ghost echo?
Are you familiar with Ultrasonic sensor? Then this might be useful. robotshop.com/letsmakerobots/reading-multiple-echoes-with-hc-sr04-ultrasonic-sensors
Yes I am aware but didn't knew about the ghost echo concept. Thanks!
you can, but he didnt ask you, or how it can, he asked how they did. you ready to inherit the earth?
Multiplexing them oughta do the job
Also, how long did the whole training process take?
The training, both trough simulation and phsycial, was done with 300 attempts. I think we used about three hours on real life training with the agent in order to make i work as shown in the last part of the video.
Where does the agent store the stuff it learns from the environment?
It sends it directly to a nearby computer that has a MATLAB program running. This is where all calculations are done.
Hi, I know It may has been too long after you making this video but I know C++ and arduino programming but new at Machine Learning could you please tell me to make this type project what I need to learn more??? this is so important for me thanks a lot in advance.
Hi! Please read some of my comments on the other posts. We used Matlab and Arduino in order to do this project. The Arduino "side" of the project was quite simple - reading values of the sensors and sending them to a computer trough an xBee. I recommend you to take a look on avaible source codes online and to read about machine learning. Then it's just to start programming!
I was wondering if you have interference between the 5 ULTRASONIC sensor. I stongly believe that its possible that they might mess up with each other...
Hi! It's all about how you devide the "responsibility" between each sensor. The Q-table discribed a givven distance for each sensor, meening that it was not problem that they where seeing some of the same areas. (All positions was unique). Thanks for good input!
Hey i'd be really intressted in more information too. Is the Thesis open to the public? or do you have a git? cheers
This is a part of a PhD paper. I will look into if this paper is public accessable at this point in time. We did not use GitHub - sorry!
Hi! Please check out the new description for this video. You can now download the final thesis and MATLAB codes.
how could you communicate with matlab and arduino ? like what did you use to implement what you did on matlab and put it on arduino ? great work btw , keep it up
Hi! Arduino reads from the sensors and sends every value to a computer running MATLAB. MATLAB calculates the next action and sends it back to the Arduino board. This makes us able to do a lot of operations without overloading the Arduino.
Hi! Please check out the new description for this video. You can now download the final report and MATLAB codes.
Hi my friend, I know the q-learnign algorithm and also i built a crawling robot which use q-table . However i am trying to find good simulation base. There are lots of options such as unity with C# , Matlab with ROS ans so on...My computer stuck for almost every options. Personally if i could use only one base such as matlab i could do this. I am new on matlab, therefore i only ask you that which tool did you used for this simulation in matlab?
Hi! Please check out the new description for this video. You can now download the final report and MATLAB codes.
Thanks , It will help to find correct way for matlab.
do you mind sharing your code? I think I would be ready to pay
did you get the code?
Hi! Please check out the new description for this video. You can now download the final report and MATLAB codes.
Hey dude want my 360 spinning electromagnetic based sensor ? Like a Lidar but cheaper!
Guy in the vid : Nah, I'll just put an ultrasonic sensor for every 15 degrees.
did u used continuous or discrete actions?
We used discrete actions where each sensor where given four sones . Delay in transfer, the size of the array for continous actions and the fact that you actually can read the array and understand how the robot should react when using discrete actions made us choose that solution.
can i see your code ? im building a neural networks for a boat maybe you have some good code that you can help me
The source codes are not available for downloading yet, I'm afraid.
Hi! Please check out the new description for this video. You can now download the final report and MATLAB codes.
But sir how to make this robot using arduino and please full video upload ple ple...
hi Atle Fjellang Sæther, what software did you use to simulate ?
Hi! We used MATLAB in order to make the simulation. It saved us a lot of time and also gave a good view of what we could expect from the agent it self.
Hi! Please check out the new description for this video. You can now download the final report and MATLAB codes.
Try removing 4 HC SR04 sensors and add a servo to the remaining one.
Wait, if you don't mind, could you tell me what your age is...??? Just asking, for even though I m a 15 yrs old, I didn't understand any material reqd for this experiment... And I actually donot possess any idea of what 4 HC SRO4 is...
Feeling quite ashamed of myself...
@@emamibhattacharjee1874 the HC SR04 is the ultra sonic sensor in the video
@@Pk-xx1my oo...Thank you so much for ur concern...
What program are you using at 1:37?
The simulator shown in this video is made using MATLAB.
the cause of delayed actions by the Bot. 1. too many sensors and Ardunio can not handle. 2. distance to take an escape should be more than 30 cm so that the processor gets the time to react. 3. One sonar with treversing platform in which the distance finding should be a function , and treversing should be a function. , the distance function should be called by the treversing function so that the distance reading is possible with each step of treversing. Bot will do fine in the very first attempt
Hi Mo! Actually - I think the Arduino is capable to scan the soroundings with even a few more sensors without any problems. The processor speed does not seem to be the problem - it's rather positioning of sensors, margins in the sensor readings and the size of the agent. As the agent has a diameter of around 25cm, it will take both space and time to move the agent in one or the other direction. I really like your ideas, but the project was about getting a completely "dumb" agent to learn from scratch and to improve it's driving over time. I believe that the problem with this version allways would be the size of the agent as explained earlier. I would've loved to repeat the project with a new and more agile agent to see if there would be any differens. Thanks for the input!
I am working with timer 1 to resolve this multiple process bottleneck issues , the processor stays Idle and unusable by other processes when engaged in first process. eg while measuring the distance (if distance < limit) the signal to change direction function does not initiate ,and the rover bumps into the obstacle. Either the (T)interrupts can help OR use of additional nano board just for driving directions with external interrupts from the main UNO(6 sonar unit) can be of help , but this speaks of inefficient programming , as One board has enough resources to handle such a small Logic , Please do post if you have a solution I am working on this for the Humanoid with 24 servos , and the cluster of wires is giving me headache ,
How did you fit an NN on an Arduino Uno???
It was not on the Arduino, it was in a computer running Matlab nearby!
Hii,, how can we make a robot movement autonomous ?
Can u inform about that,,, with codes
nice job!
but sr-04 has lots of fake data! how did you deal with it!
By measuring the range where the SR-04 is stable and gives good data. We saw that different shapes of obstacles gave different readings and experimented with size, shape, gloss and color in order to find the correct objects.
I'm a beginner learning arduino and this project is so nice,
can i get the program copy please so that i could learn how you managed.
So,please reply me with the program.
The source codes are not available for downloading yet, I'm afraid.
Hi! Please check out the new description for this video. You can now download the final thesis and MATLAB codes.
brother show me the code ? i want to learn
Hi! Please check out the new description for this video. You can now download the final report and MATLAB codes.
Most difficult work, but what about that if the device detects any obstacle and adds in its database that the location has some obstacle and has to be avoid and also it can make their own map?, Is it possible? Anyone?
can you send me Thesis in ENGLISH
I don't understand, why do you did a simulation? Do you use the table o r the weights of your neural network in the real platform? As I see, the real platform was trained from scratch.
Para cuando un sorteo del L298n h-bridge motor driver? Muy buen video, me ha gustado.
. y si me lio
Es muy barato ese driver, compratelo tu
Hard work won bro
Pls explain connection
God bless you brother, I wanna know, wich NN_uploadfile or function I have to use to send data to my robot in the way of ANN or Q-Table
Hi, could you do my thesis for me? Also, translate it into English? And, would you walk my dog? Also, floss my cat? Thank you.
can anyone tell how to do teaching or machine learning on Arduino...??
I think a quick Google or RUclips search for "Arduino how to" or "Learn Arduino" will get you started!
eso sería un avance mutuo al pensar que el robot y el humano trabajarán juntos peroooo como saben ses preocupante que puedan decidir ser "libre" deberían ponerle códigos de provisión o sea 0 daño a la raza humana o la otra opción instalar una IA en el celebró humano
In english, per favore?
this is so fukin cazy
Ossam
0:00 ; queue star wars music
Agreed!
Is this programmed in python
About how much would the parts cost?
I think we used about $250 on all parts including rotating obstacle, agent and surroundings. Most of the electronics where bought on eBay, while the rest where bought in our local hardware store.
Cool, thanks for responding!
Hiii..... I want to replicate this project
Do you have a paper documenting this project that I can view?
This is a part of a PhD paper. I will look into if this paper is public accessable at this point in time.
Hi! Please check out the new description for this video. You can now download the final thesis and MATLAB codes.
I think that the secret is in stopping recording in the best time =) just for a second before collision =))
(joking.. maybe =))
I think you might be right! But it's a proof of consept.
it is a very big conditionals of if and else but it is not artificial inteligence or machine learning. :(
Its great, anyone who used python ?
This Robo's gonna be robot's sherlock Holmes Someday maybe a billion decades later
still learning to avoid a miniature suitcase and file quite impressive strategy mmmhhh
aaand some trash cans ooo
Hello my friend
is te source code avaliabe now ?
The source codes are not available for downloading yet, I'm afraid!
Hi! Please check out the new description for this video. You can now download the final thesis and MATLAB codes.
please can you give it's circuit diagram
Hi Manoj. I probably could find this, but as this was not a electronic project, allmost all components where bought as modules on eBay and connected together. The task was to make a working algorithm. If you see the overview of the agent early in the video you will see the name of all components.
Hi! Please check out the new description for this video. You can now download the final thesis and MATLAB codes.