4:54 the motivation behind log is more likely this: assume only 1 document cat.txt (N=1) with the word "cat" in it. The idf("cat", cat.txt) would be 1 / 1 = 1. However we want the idf to equal 0 for super common terms (terms with no specific information about the document => noise) . The log will give us that idf("cat", cat.txt) = 0. So whenever the number of documents matches the number of occurrences the idf = 0... exactly like in 5:38
There are several ways of computing TF-IDF (e.g. tf = log(1 + word_freq)). Have you experimented with other variations?
Amazing. This video explained exactly what I wanted from it. Thanks a bunch!
Glad it helped! :)
Best explanation ever, well detailed
Thanks! Glad you think so! :)
4:54 the motivation behind log is more likely this: assume only 1 document cat.txt (N=1) with the word "cat" in it. The idf("cat", cat.txt) would be 1 / 1 = 1. However we want the idf to equal 0 for super common terms (terms with no specific information about the document => noise) . The log will give us that idf("cat", cat.txt) = 0. So whenever the number of documents matches the number of occurrences the idf = 0... exactly like in 5:38
Thanks for the clarification! :)
This was very helpful. Thank you!
Thanks! Glad it was helpful!
Very well explained, thank you!
Thanks! Glad it was helpful! :)
thank you for this video
You're welcome! Glaf you liked it! :)
Well explained, thank you!
Thanks! Glad it was helpful! :)
excellent explanation
Thanks! Glad you liked it! :)
nice explanation thanks
Glad you found it helpful! :)
thank you so much!!!
You're welcome! I'm happy you enjoyed the explanation! :)
Merci!
You're welcome! :)
3:16 you didn't count the dot at the end of each line, so technically 2/7 etc. 😜
8:33 TF-IDF(doc2, "and") should be 0.022 not 0.22
Weeell, yep, my bad :)