Rasa Algorithm Whiteboard - BytePair Embeddings

[@faangsde]

This channel is a gold mine! Thank you very much for sharing your insights!

[@distrologic2925]

How does the algorithm know when to stop merging tokens?

[@87456100]

Great video! I think the video description lacks a word in the sentence "They need way memory..."?

[@distrologic2925]

they were masking that word to test your listening comprehension

[@piyalikarmakar5979]

Thanks sir.. one query.. what's the difference between byte pair and wordpiece tokenization?

[@RasaHQ]

(Vincent here)
Great question! My impression is that they are very similar in practice but that the way for merging letters is slightly different. I could be wrong but I think workpiece uses a likelihood heuristic while bytepair uses counts.

[@alanliang9538]

thanks bro, best explaination i can find.

[@wibulord926]

thanks for usefull tutorial

[@shahzadmalik96]

First comment

[@gorgolyt]

congratulations, you win nothing.