Training and adding new tokens in a Pre-trained Tokenizer !!

[@tharunbhaskar6795]

Some quality content here. So the new tokens just get appended to the current tokenizer right?

[@Tensordroid]

Yes

[@CantBeSubh]

good shit bro

[@Tensordroid]

thanks

[@pranilpatil4109]

I did the same for 22 Indian languages. But when I searched a kannada language character in the tokens for a test purpose, it was not showing anything. Also, tokenizer separates punctuation as well. Your method of splitting is not optimal.