Shannon Entropy and Information Gain

Announcement: New Book by Luis Serrano! Grokking Machine Learning. bit.ly/grokkingML
40% discount code: serranoyt
Accompanying blog post: / 5810d35d54b4
0:00 Shannon Entropy and Information Gain
2:22 What ball will we pick?
4:33 Quiz
5:06 Question
5:14 Game
7:17 Probability of Winning
7:45 Products
11:00 What if there are more classes?
12:34 Sequence 2
13:44 Sequence 3
14:57 Naive Approach
15:34 Sequence 1
19:44 General Formula

[@sphengle]

This was exactly the baby step I needed to get me on my way with entropy. Far too many people try to explain it by going straight to the equation. There's no intuition in that. Brilliant explanation. I finally understand it.

[@jankinsics]

Sean Walsh feel the same way.

[@freemanguess8634]

With great knowledge comes low entropy

[@SerranoAcademy]

Hahaaa, love it!!!

[@fantomraja9137]

lol

[@hyperduality2838]

@@SerranoAcademy Repetition (redundancy) is dual to variation -- music.
Certainty is dual to uncertainty -- the Heisenberg certainty/uncertainty principle.
Syntropy (prediction) is dual to increasing entropy -- the 4th law of thermodynamics.
Randomness (entropy) is dual to order (predictability) -- "Always two there are" -- Yoda.

[@MrofficialC]

And low entropy is easier to rig

[@lani0]

You win

[@AlexMcClung97]

Excellent explanation, very clear and concise! I have always pondered the significance of the log in cross-entropy loss function. The explanation (particularly: "products are small and volatile, sums are good") completely clears this up.

[@carnivalwrestler]

Luis, you are such an incredibly gifted teacher and so meticulous in your explanations. Thank you for your hard work.

[@user-or7ji5hv8y]

how does one make something so complicated into something so intuitive that others can finally see the picture. your explanation itself is an amazing feat.

[@effemmkay]

I have been scared of delving into entropy in detail for so long because the first time I studied it, it wasn’t a good experience. All I want to say is THANK YOU!!!!!! I should have been supplementing the udacity ND lesson videos with these since the beginning.

[@mau_lopez]

What a great explanation ! I wish I had a teacher like you Luis, everything wold be way easier ! Thanks a lot

[@msctube45]

I needed this video to get me up to speed on entropy. Great job Luis!

[@dyutinrobin]

Thank you so much. This was the only video in youtube that clarified all my doubts regarding the topic of entropy.

[@MatheusSilva-dragon]

Wow, thank you, man.
I needed that information!
There are many ways to teach the same stuff!
That number of question stuff is great! It's good to have more than one way to measure something!

[@ketlebelninja]

This was one of the best explanations on entropy. Thanks

[@123liveo]

2nd time I found this video and loved it both times. Much better description than the prof at the uni I am at!!!

[@shekelboi]

Thanks a lot Luis, just had an exam about this Wednesday and your video helped me a lot to understand the whole concept.

[@therealsachin]

The best explanation about Shannon entropy that I have ever heard. Thanks!

[@eprabhat]

Luis, You have a great way of explaining. At times , I like your videos more than even some highly rated professors

[@sasthra3159]

Great clarity. Have never got this idea about the Shannon Entropy. Thank you. Great work!

[@Vuvuzella16]

This video is helping to keep me floating in my Data Science course; thank you so much for your time!

[@NoOne-uz4vs]

I'm studying Decision Tree (Machine Learning Algorithm) and it uses Entropy to efficiently build the tree. I finally understand the details. Thank you!!

[@sdsa007]

Wow! Awesome, so books and encyclopedias and biographies of Shannon to understand what you just clearly explained! Thank You!

[@Asli_Dexter]

i wish i had this lecture during college examination.....still it's nice to finally understand the intuition behind the formulas i already knew.

[@pixboi]

Teaching should be like this, from practice to theory - no the other way around!

[@patricklemaire225]

Great video! Now I understand what Claude Shannon discovered and how useful and essential maths are in Computer Science.

[@SenhorMsandiFelipe]

Gracias. Muito claro Senhor. I have been struggling to wrap my head around this and you just made it easy. Thank you.

[@drakZes]

Great work. Compared to my textbook you explained it 100 times better, Thank you.

[@elmoreglidingclub3030]

Excellent! Great explanation. Enjoyable video (except YT’s endless, annoying ads). Thank you for composing and posting.

[@SixStringTheory6]

Wow ..... I wish more people could teach like you this is so insightful

[@paulstevenconyngham7880]

this is a really great explanation, thanks so much for sharing mate!

[@mehmetzekeriyayangn3782]

You are the best.Such a great explanation.Better than lots of text books.

[@bismeetsingh352]

That was highly intuitive, thank you, sir, I appreciate the effort behind this.

[@jackallread]

Thanks for the relationship between knowledge and entropy, that was very helpful. Your explanation of statistics is also good! Though, I am only half way through the video at this point, I will finish it!
Thanks

[@haimmadmon3531]

Very good explanation - hope to hear more of your videos

[@poxyu_was_here]

Easy and Great explanation! Thank you very much, Luis

[@pkittali]

Lovely explanation...Superb

[@user-xo2vg9om4v]

It's very helpful for me to introduce the concept of entropy to students. Thank you for your clear presentation of entropy.

[@victorialeigh2726]

Hola Luis,
estupendo, espectacular, excelente!

[@karinasakurai9867]

Brilliant lecture! I learn so much with this explanation. Thanks from Brazil :)

[@RobertLugg]

I have learned so much from your teaching. Thank you.

[@scherwinn]

Very clever explanation of mighty ENTROPY.

[@moshad2002]

amazing explanation!! very clear and articulate. Thank you!

[@Kani132]

Very nice video. Insightful, inutuitive and very well explained. Thank you!

[@yikenicolezhang]

Thank you so much for explaining this concept!

[@rajudey1673]

Really, you have given us outstanding information.

[@amperro]

I watched it straight through. Very good.

[@rolfbecker4512]

Thank you very much for this beautiful and clear explanation!

[@MrMaipeople]

Great and Superb. Thank you so much for the most clearer explanation.

[@hanaizdihar4368]

What a great explanation! And so i subscribed😊

[@hanaelkhalifa2630]

Thank you for excellent explanation of entropy concept first... Then reach to final equation step-by-step it is really good and simple way

[@francismcguire6884]

Best instructor there is! Thanks

[@yhat314]

Lovely job Luis! Very very good!

[@krupalshah1487]

You explained it really very well. Thank you !

[@logosfabula]

Luis, you really are a great communicator. Looking forward to your other explanations.

[@symnshah]

Nice explanation. Thank you so much for this video.

[@tilugulilwa]

Superb step by step explanation

[@saadjabrani7781]

easy and thorough explanation, thank you !

[@user-or7ji5hv8y]

wow, another great and insightful presentation . really helps to build intuition

[@riderblack6401]

Thank you so much! Luis!

[@nijunicholas631]

Thanks..Got the intuition behind Entropy

[@subhashkonda5000]

Its always hard to understand the equations but u made it so simple :-)

[@andrashorvath2411]

Fantastic work.

[@scherwinn]

Mr. Luis Serrano III great job in Neural Network and Claude Shannon Entropy.

[@RenanCostaYT]

Great explanation, greetings from Brazil!

[@kasraamanat5453]

best, as always ❤️ thank you Luis❤️

[@kleberloayza7839]

hi Luis, nice to meet you, I am reading the book of Deep learning of Ian Godfellow, and I needed to view your video for understand the chapter, 3.13 information theory. thanks very much.

[@TheZilizopendwa]

Excellent presentation for an otherwise complex concept.

[@mulangonando2942]

I love the explanation of the negative sign in the Entropy Equation many people wonder

[@capuleto126]

Very well explained, thanks Luis

[@user-wi1rj4iw9y]

十分谢谢！ Thank you very much, Luis.

[@hcgaron]

wonderful video. Thanks!

[@clarakorfmacher7394]

Great Video! I really liked the intuitive approach. My professors was waaaay messier.

[@ishayauperelman2926]

Great video, simple and to the point.

[@abhishek_sengupta]

Awesome explanation!!

[@hoarong1993]

You are awesome!. I hope you will make more videos like this

[@JaroslawSchaller]

Beautiful explanation, thank you!

[@archerchian7761]

Very clear, thank you!

[@sirishachanumolu3723]

Best explanation of entropy. Thanks.

[@xThomas1995]

Thank you for the very good video. Easiest to understand so far.

[@mortezaaslanzadeh3735]

Awesome explanation, you rock

[@Dennis12869]

Best explanation I found so far

[@micahdelaurentis6551]

you killed it. Great video

[@Illumarnati]

Very nicely done.

[@AJK544]

your explain is perfect. Even though I am not good at listening english. I can understand everything :)

[@shakeelurrahman1846]

thanks a lot for such a beautiful explanation..!

[@markbordelon1601]

Perfect explanation of this

[@smitomborah467]

thanks a lot, Luis. This was helpful.

[@kingshukbanerjee748]

very lucid explanation - excellent, intuitive build-up to Shannon's theorem from scratch

[@sitharamubeen]

Fantastic explanation

[@LAChinthaka]

Thanks for the great explanation.

[@jahngad]

great , It helped me alot . Thank you Luis

[@ravivar1]

Thanks Luis!

[@kjaanishji2595]

Great presentation!!!

[@miguelfernandosilvacastron3279]

Thank you. Nice, concise explanation.

[@adonis100]

Fascinating. Excellent Job

[@jonathanfrancis]

Wow. Amazing video.

[@amitkumarmaiti5392]

Great Intuition Luis

[@Omsip123]

Very well explained, thank you

[@YouUndeground]

It was a great explanation! Thank you.

[@MostafaMASLOUHI]

Thank you very much. Very nice explanation.

[@zinahe]

Thank you very much for sharing. I found it very helpful.

[@pooltablemechanic]

Great, clear explanation! Thanks

[@carlitos5336]

Excelente explicación! Gracias por compartirla.

[@paulinagc6986]

Thank you so much. You are such a good teacher, really :D :D :D