Activation Functions - EXPLAINED!

We start with the whats/whys/hows. Then delve into details (math) with examples.
Follow me on M E D I U M: towardsdatasci...
REFERENCES
[1] Amazing discussion on the "dying relu problem": www.quora.com/...
[2] Saturating functions that "squeeze" inputs: stats.stackexc...
[3] Plot math functions beautifully with desmos: www.desmos.com/
[4] The paper on Exponential Linear units (ELU): arxiv.org/abs/...
[5] Relatively new activation function (swish): arxiv.org/pdf/...
[6] Used an Image of activation functions from this Pawan Jain's Blog: towardsdatasci...
[7] Why bias in Neural Networks? stackoverflow....

[@desalefentaw8658]

wow, one of the best highlights of activation functions on the internet. Thank you for doing this video

[@GauravSharma-ui4yd]

Awesome as always. Some points to ponder correct me if I am wrong
1. Relu is just not a activation but can also be thought as a self regularizer, as it offs all those neurones whose values are negative, so it's just a kind of automatic dropout.
2. A neutral net with just input and output layer, with softmax at the output layer is logistic regression, but when we add hidden layers in this network with no hidden activations then it's more Powerful than just vanilla logistic regression as it is now taking linear combination of linear combinations with different weight settings. But it still results in linear boundaries.
Lastly your contributions to the community is very valuable, clears a lot nitty-gritty details in short time. Keep going like this :)

[@generichuman_]

No, dropout is different. Random sets of neurons are turned off in order to cause the neurons to form redundancies which can make the model more robust. In the case of dying Relu, the same neurons are always dead, making them useless. Dropout is desirable and deliberate, dying Relu is not.

[@SkittlesWrap]

Straight to the point. Nice and super clean explanation for non-linear activation functions. Thanks!

[@UdemmyUdemmy]

the screetching noise is irrtitaing..else nice tutoial

[@prakharmishra-m5r]

I agree

[@malikkashifsaeed1938]

agreed. cringe and irritating

[@semon00]

I don't agree

[@PrymeOrigin]

One of the best explanations ive come across

[@jhondavidson2049]

I'm learning deep learning rn and using the deep learning book published by MIT press for the same. That's kinda complicated for me to understand especially these parts cause m still an undergrad and have 0 previous experience with this. Thank you for explaining this so well.

[@CodeEmporium]

Anytime :)

[@otabeknajimov9697]

best explanation of activation functions I ever seen

[@fahadmehfooz6970]

Amazing! Finally I am able to visualise vanishing gradient descent and dying relu.

[@CodeEmporium]

Glad!

[@adrianharo6586]

Great video!
The dissapointed gestures were a bit too much x'D
A question I did have as a beginner was.
What does it mean for a sigmoid gradient to "squeeze" values, as in they become smaller and smaller as they back propagate?

[@AnkityadavGrowConscious]

It means that sigmoid function will always output a value between 0 and 1 regardless of any real number input. Notice the mathematical formula and graph of a sigmoid function for better clarity. Any real number will be converted to a number between 0 and 1. Hence sigmoid is said to "squeeze" values.

[@deepakkota6672]

Wooo, Did I just noticed the complex explained simple. Thanks! Looking forward to more videos.

[@PritishMishra]

The most thing I love about your videos is the fun you add... Learning becomes a bit easier

[@linuxbrad]

7:48 "once it hits zero the neuron becomes useless and there is no learning" this explains so much, thank you!

[@oheldad]

Great video ! And what is more great - are the useful references you add at the description. ( For me (1)+(7) answer the questions I asked my self at the end of your video - so its was on point ) ! Thank you !

[@CodeEmporium]

Haha. Glad the references are useful! :)

[@RJYL]

Great explanation for activation function I like it so much

[@CodeEmporium]

Thanks so much for commenting

[@shivendunsahi]

I discovered your page just yesterday and might I say, YOU'RE AWESOME! Thanks for such good content bro.

[@CodeEmporium]

Thanks homie! Will dish out more soon!

[@rishabhmishra279]

Great explanation ! and the animations with maths formula and visualizing it is awesome !! Many thanks !

[@deepaksingh9318]

Wow... Perfect and easiest way to explain it..
Everyone talks about what activations do but nobody shows in how actually it looks like behind the algos..
And you explain things in the most easiest way which are so easy to understand and remember..
So a big like for. All your videos..
Could uh make more and more and DL.. 😄

[@CodeEmporium]

Thank you. I'm always thinking of more content :)

[@kanehooper00]

Excellent job. There is way too much "mysticism" around neural networks. This shows clearly that for a classification problem all the nerual net is doing is creating a boundary function. Of course it gets complicated in multiple dimensions. But your explanations and use of graphs is excellent

[@the-tankeur1982]

I hate you for making that noises, i want to learn, comedia is something i would pass on

[@nguyenngocly1484]

With ReLU f(x)=x is connect, f(x)=0 is disconnect. A ReLU net is a switched system of dot products, if that means anything to you.

[@mikewang8368]

better than most professors, thanks for great video

[@CodeEmporium]

Thanks!!

[@alonsomartinez9588]

Awesome vid! Small sug: I might check the volume levels, during the screaming in :56 it was a bit painful to my ear and possibly sounded like audio clipping.

[@rasikannanl3476]

great .. so many thanks ... need more explanation

[@shrikanthnc3664]

Great explanation! Had to switch to earphones though :P

[@alifia276]

Thank you for sharing! This video cleared my doubts and gave me a good introduction to learn
further

[@CodeEmporium]

Super glad :)

[@dazzykin]

Can you cover tanh activation? (Thanks for making this one so good!)

[@CodeEmporium]

I wonder if there is enough support that warrants a video on just tanh. Will look into it though! And thanks for the compliments :)

[@sgrimm7346]

Excellent explanation. Thank you.

[@phucphan4195]

thank you very much, this is really helpful

[@CodeEmporium]

Thanks:)

[@fazlayrabby3709]

Can you please explain this "No gradient means no learning"?

[@wagsman9999]

Beautiful explanation!

[@malekaburaddaha5910]

Thank you very much for the great, and smooth explanation. This was really perfect.

[@CodeEmporium]

Much appreciated Malek! Thanks for watching!

[@epiccabbage6530]

What are the axises on these graphs? Is it inputs, input*weights + bias for linear?

[@NITHIN-tu7qo]

did you get answer for it?

[@AmirhosseinKhademi-in6gs]

but we cannot use ReLU for the regression of functions with high degrees of derivatives!
In that case, we should still go with infinitely differentiable activation functions like "Tanh", right?

[@cheseremtitus1501]

Amazing presentation ,easy and captivating to grasp

[@CodeEmporium]

Glad you liked it! Thank you!

[@SeloniSinha]

wonderful explanation!!!

[@ShivamPanchbhai]

this guy is genius

[@AymaneArfaoui]

what does x and y represent in the graph you use to show the cats and dog points ?

[@simranjoharle4220]

This was really helpful! Thanks!

[@CodeEmporium]

Thanks for watching :)

[@Mohammed-rx6ok]

Amazing explanation and also funny 😅👏👏👏

[@tarkatirtha]

Lovely intro! I am learning at the age of 58!

[@DrparadoxDrparadox]

Great Video. Could you explain what U and V are equal to in this equation : o = Ux + V ? And How did you come up with the decision boundary equation and how did you determine the values of w1 and w2 ?
Thanks in advance

[@najinajari3531]

Great Video and great page :) Which softwares you use to make these videos ?

[@CodeEmporium]

Thanks! I use Camtasia Studio for the editing; Photoshop and draw.io for the images.

[@myrondcunha5670]

THIS HELPED SO MUCH! THANK YOU!

[@eeera-op8vw]

good explanation for a beginner

[@superghettoindian01]

Another great video
🎉🎉🎉!

[@CodeEmporium]

Thanks so much!

[@yachen6562]

Really awesome video!

[@kellaerictech]

That's great explanation

[@CodeEmporium]

Thanks so much for watching !

[@meghnasingh9941]

wow, that was really helpful, thanks a ton!!!!

[@CodeEmporium]

Glad to hear that. Thanks for watching!

[@younus6133]

oh man, amazing explanation.Thanks

[@ankitganeshpurkar]

Nicely explained

[@CodeEmporium]

Thanks for watching this too

[@youssofhammoud6335]

What I was looking for. Thanks!

[@linuxbrad]

9:03 what do you mean "most neurons are off during the forward step"?

[@MrGarg10may]

then why isn't leaky RELU ELU used everywhere in LSTM, GRU, Transformers ..? why is RELU used everywhere

[@mohammadsaqibshah9252]

This was an amazing video!!! Keep up the good work!

[@CodeEmporium]

Thanks so much!

[@prashantk3088]

really helpful..thanks

[@wucga9335]

so how do we know when to use relu or leacky relu? do we just use leacky relu all together in all cases?

[@aaryamansharma6805]

awesome video

[@vasudhatapriya6315]

How is softmax a linear function here? Shouldn't it be non linear?

[@pouyan74]

I've read at least three books on ANN's so far, but it's only now, after watching this video, that I have the intuition of what exactly is going on and how do activation functions break linearity!

[@fredrikt6980]

Great explanation. Just add more contrast to you color selection.

[@CodeEmporium]

My palette is rather bland i admit

[@kphk3428]

1:16 I couldn't see that there were different colors so I was confused.
Also I found the voicing of the training neural net annoying. But some people may like what other people dislike, so it's up to you to keep on voicing them.

[@gabe8168]

the dude is making these videos alone, if you don't like his voice that's on you, but he can't just change his voice

[@Nathouuuutheone]

What decides the shape of the boundary?

[@yukuchan]

nice video ♥

[@mangaenfrancais934]

Great video, keep going !

[@programmer4047]

So, we should always use leaky reLU

[@TheAscent_]

@6:24 How does passing what is a straight line into the softmax function also give us a straight line? Isn't the output, and consequently the decision boundary, a sigmoid?
Or is it the output before passing it into the activation function what counts as the decision boundary?

[@CodeEmporium]

6:45 - The line corresponds to those points in the feature space (the 2 feature values) where The sigmoid's height is 0.5.

[@prakharrai1090]

can we use linear activation with hinge loss for Linear svm for binary classification.

[@jigarshah1883]

Awesome video man !

[@jamesdunbar2386]

Quality video!

[@tahirali959]

good work bro keep it up
-

[@CodeEmporium]

Will do homie

[@ronin6158]

it should be possible to let (part of) the net optimize its own activation function no?

[@bartekdurczak4085]

good explanation but the noises are little bit annoying but thank you bro

[@LifeKiT-i]

With graphical calculator, your explanation is sanely clear!! thank you!!

[@CodeEmporium]

Thanks so much for the kind comment! Glad the strategy of explaining is useful :)

[@jaheerkalanthar816]

Thanks mate

[@patite3103]

Amazing!

[@uzairkhan7430]

awesome

[@masthanjinostra2981]

Benefited a lot

[@CodeEmporium]

Awesome! Glad!

[@x_ma_ryu_x]

Thanks for the tutorial. I found the noises very cringe.

[@undisclosedmusic4969]

Swish: activation function. Swift: programming language. More homework, less sound effects 😀

[@CodeEmporium]

Nice catch. I misspoke :)

[@ehsankhorasani_]

good job thank you

[@CodeEmporium]

Very welcome!

[@igorpostoev2077]

Thanks man)

[@ShinsekaiAcademy]

thanks my man.

[@CodeEmporium]

You are oh so welcome

[@VinVin21969]

plot twist: its not that the boundary no longer changes, the vanishing gradient cause the gradient to be very small , that we can assume it is negligible

[@CodeEmporium]

Danana nanana nanana nana

[@harishp6611]

yes! I liked it. Keep it up.

[@farhanfadhilah5247]

this is helpful, thanks :)

[@francycharuto]

gold, gold, gold.

[@jhondavidson2049]

Amazing!!!!!!!!!!!!!!!!!

[@CodeEmporium]

Thanks!!!!!!!!!!

[@Acampandoconfrikis]

thanks brah

[@abdussametturker]

thx. subscribed

[@zaidalattar2483]

Perfect explanation!... Thanks

[@CodeEmporium]

Much appreciated!

[@keanuhero303]

What's the +1 node on each layer?

[@avijain6277]

The bias term

[@splytrz]

I've been trying to make a convolutional autoencoder for mnist, and at first I used sigmoid activation on the convolutional part and it couldn't make anything better than just a black screen on the output but when I removed all activation functions it worked well. Does anyone have any idea why that happened?

[@fatgnome]

Are the outputs properly scaled back to pixel values after being squeezed by sigmoid?

[@splytrz]

@@fatgnome Yes. Otherwise the output wouldn't match with images. Also I checked model.summary() every time I made changes to the model.

[@Edu888777]

I still dont understand what a activation function is

[@abd0ulz942]

learn Activation Functions with Dora
but I honestly it is good

[@ExMuslimProphetMuhammad]

Bhai video shayad accha hoga but thumbnail pe Teri pic dekhke hi kafi log click na kare, I'm here just to let you know this:avoid putting your face on thumbnail or in video as no one is interested in seeing the educator while watching technical videos.

[@CodeEmporium]

You clicked. That's all i care about ;)

[@harshmankodiya9397]

gr8 exp

[@CodeEmporium]

Thank you. Really appreciate it

[@frankerz8339]

nice

[@t.lnnnnx]

followeeeed

[@CodeEmporium]

Replieeeeed. Thanks!

[@히안-p3j]

I don't understand..

[@wadyn95]

Wtf what's the sound of pictures...