Great video, really helped me understand how this library can perform training. I do have a tip though, you'll get a more efficient filter if you just use `dataset = dataset.filter(lambda x: x['label'] == 0)`. Don't need to run an extra if statement on each row.
James, Great video! Curious if you need take measures to ensure that each anchor/positive pair in the training set is independent of the others, so that you don't inadvertently end up with a p3 that supports a1, for example... Similar to what might happen if you have duplicate data.
Very nice video - thanks a lot ! I was trying to do extractive text summarization based on BERT sentence embedding - taking the 768 dim for [CLS] token in each sentence - and putting a K-medoid clustering on the sentence embeddings and choosing K medoids as a summarization. I fine tuned BERT on MNR objective and used the mean pooled sentence embeddings now and apply the same K medoid process. The results are quite different and the later seems to be giving more salient points. I will try to see the ROUGE score difference on some benchmark dataset.
Great video! A question: suppose I want to add new words to the base model, because my domain contains specific jargon. What is the approach for fine tuning in this case? Thank you
Hi James, thanks for the video, well done! Regarding the scale factor, it could be important, specially when using the cosine similarity, because if there are small differences then the model could have a hard time learning good embeddings (specially if you are fine tuning for a particular domain). The best is to try different values and check what happens. Thanks again!
Hi James, nice video! I have a dataset on which I need to eventually perform text classification but I don't have labels. What I actually want to do is to discover the taxonomy semantically. What would be the definition of anchor, positive pair in this scenario?
Hello, i have one question : is it possible to fine tune sbert with unsupervised data ? I have 2 job catalogs that i need to match by textual similarity but i dont have labels ( for each job I only have it's title and description)
I had a similar question on the discord chat earlier, was that you? If so sorry I'm just repeating the same thing - but essentially all you need here are (anchor, positive) pairs, if you can extract *mostly* relevant sentence pairs from your data (relevant pairs just need to be similar, no labels required), you're good to go and you can use MNR loss with a few conditions. 1. I said *mostly* above, I mean if you had 99 sentence pairs that were genuinely related but then the odd 1 that were not, your model should be able to deal with that - just as long as there are not too many of these erroneous pairs 2. When you're randomly sampling the data into batches, you want to minimize the likelihood of introducing other pairs that could be viewed as similar (anchor, positives), so if 50% of your data is all talking about the same thing (aka high similarity), your model is going to find it hard (likely impossible) to differentiate between similar/dissimilar pairs and produce meaningful embeddings Hope that helps!
@@jamesbriggs hey, first of all, no it was not me that asked you that on discord. I actually saw only half of the video, so i still dont know what's MNR. I'll continue it later. I used to work with pre trained models from sentence-tramsformers, but to fine tune those, I need labelled data with similarity scores (which I dont have). My problem is kinda complex, is it possible to join this discord? I have a lot of questions and maybe i can find help there, thanks
I've put together a load of sentence transformer videos in this playlist: ruclips.net/p/PLIUOU7oqGTLgz-BI8bNMVGwQxIMuQddJO For that I'd recommend the 'how TSDAE works' video :)
Great video, really helped me understand how this library can perform training. I do have a tip though, you'll get a more efficient filter if you just use `dataset = dataset.filter(lambda x: x['label'] == 0)`. Don't need to run an extra if statement on each row.
James, Great video! Curious if you need take measures to ensure that each anchor/positive pair in the training set is independent of the others, so that you don't inadvertently end up with a p3 that supports a1, for example... Similar to what might happen if you have duplicate data.
Thanks for the video. If I have a question answer grade type of dataset, what would be the approach to fit on such a data?
Very nice video - thanks a lot ! I was trying to do extractive text summarization based on BERT sentence embedding - taking the 768 dim for [CLS] token in each sentence - and putting a K-medoid clustering on the sentence embeddings and choosing K medoids as a summarization. I fine tuned BERT on MNR objective and used the mean pooled sentence embeddings now and apply the same K medoid process. The results are quite different and the later seems to be giving more salient points. I will try to see the ROUGE score difference on some benchmark dataset.
Great video! A question: suppose I want to add new words to the base model, because my domain contains specific jargon. What is the approach for fine tuning in this case? Thank you
Hi James, thanks for the video, well done!
Regarding the scale factor, it could be important, specially when using the cosine similarity, because if there are small differences then the model could have a hard time learning good embeddings (specially if you are fine tuning for a particular domain). The best is to try different values and check what happens.
Thanks again!
Nice video. One thing I want to know is how to implement early stopping with sentence-transformer?
Hi James, nice video! I have a dataset on which I need to eventually perform text classification but I don't have labels. What I actually want to do is to discover the taxonomy semantically. What would be the definition of anchor, positive pair in this scenario?
How can we do this task with Tensorflow?
How to use also negative ?
Hello, i have one question : is it possible to fine tune sbert with unsupervised data ? I have 2 job catalogs that i need to match by textual similarity but i dont have labels ( for each job I only have it's title and description)
I had a similar question on the discord chat earlier, was that you? If so sorry I'm just repeating the same thing - but essentially all you need here are (anchor, positive) pairs, if you can extract *mostly* relevant sentence pairs from your data (relevant pairs just need to be similar, no labels required), you're good to go and you can use MNR loss with a few conditions.
1. I said *mostly* above, I mean if you had 99 sentence pairs that were genuinely related but then the odd 1 that were not, your model should be able to deal with that - just as long as there are not too many of these erroneous pairs
2. When you're randomly sampling the data into batches, you want to minimize the likelihood of introducing other pairs that could be viewed as similar (anchor, positives), so if 50% of your data is all talking about the same thing (aka high similarity), your model is going to find it hard (likely impossible) to differentiate between similar/dissimilar pairs and produce meaningful embeddings
Hope that helps!
@@jamesbriggs hey, first of all, no it was not me that asked you that on discord. I actually saw only half of the video, so i still dont know what's MNR. I'll continue it later. I used to work with pre trained models from sentence-tramsformers, but to fine tune those, I need labelled data with similarity scores (which I dont have).
My problem is kinda complex, is it possible to join this discord? I have a lot of questions and maybe i can find help there, thanks
Hi , is it possible to do semi-supervised text classification?
I've put together a load of sentence transformer videos in this playlist:
ruclips.net/p/PLIUOU7oqGTLgz-BI8bNMVGwQxIMuQddJO
For that I'd recommend the 'how TSDAE works' video :)
@@jamesbriggs thank you.
Can you please share the code? Thanks in advnce.
thank you for this video, helps a lot!
Next time please link the source code as well, otherwise a good video
Nice! Thank you once again! Someone bought an wacom.
haha you're right, it's super useful!