(ML 14.10) Underflow and the log-sum-exp trick

How to avoid underflow (numbers too small to represent in a given computer language) in computations involving very small probabilities, by using the "log-sum-exp trick".

[@Ewerlopes]

Can you recommend any good material for us to ready more about this trick? Book, tutorial, etc? Thanks for the video! Very good!

[@samyu4473]

in the equation ...=b+log {sum {exp(ai-b) } }, what if you use b=min(ai) instead? since b is very negative, ai-b will be positive so underflow won't happen? for example log(exp(-20) + exp(-2)) = -20+log(1+exp(18)) and no underflow here

[@Simon0725]

But if a_i are very big, positive numbers then doing b = max(a_i) prevents the overflow from happening.

[@royzhou888]

what drawing software do you use? I would also like to use it for my online class , thanks

[@heheman3000]

Why wouldn't we shift so that 0 is at halfway between the min and the max a_i? Is it because we only care about the largest one?

[@apolorwang]

A very beautiful trick! Thanks, man.

[@JoshMilesVV]

Great Video!

[@dmit10]

Cool trick, I didn't know about it.

[@WilliamWangYang]

really smart trick. Now at least we get a 'b'. ^^