Term Frequency Inverse Document Frequency (TF-IDF) Explained

[@datamlistic]

There are several ways of computing TF-IDF (e.g. tf = log(1 + word_freq)). Have you experimented with other variations?

[@chiedozieonyearugbulem9363]

Amazing. This video explained exactly what I wanted from it. Thanks a bunch!

[@datamlistic]

Glad it helped! :)

[@noblemaster0173]

Best explanation ever, well detailed

[@datamlistic]

Thanks! Glad you think so! :)

[@TJ-hs1qm]

4:54 the motivation behind log is more likely this: assume only 1 document cat.txt (N=1) with the word "cat" in it. The idf("cat", cat.txt) would be 1 / 1 = 1. However we want the idf to equal 0 for super common terms (terms with no specific information about the document => noise) . The log will give us that idf("cat", cat.txt) = 0. So whenever the number of documents matches the number of occurrences the idf = 0... exactly like in 5:38

[@datamlistic]

Thanks for the clarification! :)

[@urjitpatil9712]

This was very helpful. Thank you!

[@datamlistic]

Thanks! Glad it was helpful!

[@annette4718]

Very well explained, thank you!

[@datamlistic]

Thanks! Glad it was helpful! :)

[@OmarAmil-n7u]

thank you for this video

[@datamlistic]

You're welcome! Glaf you liked it! :)

[@ABear999]

Well explained, thank you!

[@datamlistic]

Thanks! Glad it was helpful! :)

[@abhiramivenugopal9868]

excellent explanation

[@datamlistic]

Thanks! Glad you liked it! :)

[@serafeiml1041]

nice explanation thanks

[@datamlistic]

Glad you found it helpful! :)

[@IO279K4RMN]

thank you so much!!!

[@datamlistic]

You're welcome! I'm happy you enjoyed the explanation! :)

[@a7d2e5aff]

Merci!

[@datamlistic]

You're welcome! :)

[@TJ-hs1qm]

3:16 you didn't count the dot at the end of each line, so technically 2/7 etc. 😜
8:33 TF-IDF(doc2, "and") should be 0.022 not 0.22

[@datamlistic]

Weeell, yep, my bad :)