Видео

Is he from Asia

your voice_cloning python module have several issues

Please take a look at this code and tell me why it is not giving output for the sentence that is out of training data. import spacy from spacy.training import Example import random from spacy.util import minibatch nlp=spacy.load("en_core_web_md") from spacy.training import offsets_to_biluo_tags from sklearn.metrics import precision_recall_fscore_support from sklearn.metrics import accuracy_score nlp = spacy.load("en_core_web_md") def validate_alignment(text, entities): doc = nlp.make_doc(text) biluo_tags = offsets_to_biluo_tags(doc, entities) print(f"Text: '{text}'") print(f"Entities: {entities}") print(f"BILUO Tags: {biluo_tags}") TRAINING_DATA = [ ("My name is John", {"entities": [(11, 15, "PERSON")]}), ("I am John", {"entities": [(5, 9, "PERSON")]}), ("John this side", {"entities": [(0, 4, "PERSON")]}), ("Hello, my name is Jane Doe.", {"entities": [(17, 26, "PERSON")]}), ("Jane is my friend.", {"entities": [(0, 4, "PERSON")]}), ("I spoke to John yesterday.", {"entities": [(15, 19, "PERSON")]}), ("John and Jane are working together.", {"entities": [(0, 4, "PERSON"), (9, 13, "PERSON")]}), ("Delhi is a great city.", {"entities": [(0, 5, "PLACE")]}), ("Mumbai is a great city.", {"entities": [(0, 6, "PLACE")]}), ("Kolkata is a great city.", {"entities": [(0, 7, "PLACE")]}), ("Cheenai is a great city.", {"entities": [(0, 8, "PLACE")]}), ("Patna is a great city.", {"entities": [(0, 5, "PLACE")]}), ("Lucknow is a great city.", {"entities": [(0, 7, "PLACE")]}), ("The conference will be held in Delhi and Mumbai.", {"entities": [(32, 37, "PLACE"), (42, 48, "PLACE")]}), ("Add the pipeline name Pipeline1.", {"entities": [(22, 31, "PIPELINE")]}), ("Add the pipeline name PipelineXY.", {"entities": [(22, 32, "PIPELINE")]}), ("Add the pipeline name PipelineA.", {"entities": [(22, 31, "PIPELINE")]}), ("Add the pipeline name PipelineTest.", {"entities": [(22, 34, "PIPELINE")]}), ("My pipeline name is PipelineAB.", {"entities": [(22, 31, "PIPELINE")]}), ("The latest version is Pipeline2.", {"entities": [(21, 30, "PIPELINE")]}), ("We are using Pipeline3 for this project.", {"entities": [(19, 28, "PIPELINE")]}), ("The pipeline PipelineX was used for processing data from the SourceName MongoDb.", {"entities": [(4, 14, "PIPELINE"), (37, 44, "SOURCE")]}), ("PipelineY is the new update and it will be integrated with PipelineAB.", {"entities": [(0, 9, "PIPELINE"), (37, 45, "PIPELINE")]}), ("The SourceName will be MongoDb", {"entities": [(23, 30, "SOURCE")]}), ("The SourceName will be Postgres", {"entities": [(23, 31, "SOURCE")]}), ("The SourceName will be SQL", {"entities": [(23, 26, "SOURCE")]}), ("The SourceName will be SNOWFLAKE", {"entities": [(23, 32, "SOURCE")]}), ("Give Source as the Postgres", {"entities": [(19, 27, "SOURCE")]}), ("Give Source as the MongoDb", {"entities": [(19, 26, "SOURCE")]}), ("I want my SourceName to be SQL and DestinationName to be MongoDb.", {"entities": [(27, 30, "SOURCE"), (42, 49, "DESTINATION")]}), ("I can take source as SNOWFLAKE.",{"entities":[(21,30,"SOURCE")]}), ("The DestinationName is MongoDb", {"entities": [(23, 30, "DESTINATION")]}), ("The DestinationName is Postgres", {"entities": [(23, 31, "DESTINATION")]}), ("The DestinationName is SQL", {"entities": [(23, 26, "DESTINATION")]}), ("I want DestinationName as SNOWFLAKE", {"entities": [(26, 35, "DESTINATION")]}), ("I want my DestinationName to be SQL and SourceName to be Postgres.", {"entities": [(32, 35, "DESTINATION"), (50, 57, "SOURCE")]}), ("The Script will be a Spark.", {"entities": [(21, 26, "SCRIPT")]}), ("The Script will be a Python.", {"entities": [(21, 27, "SCRIPT")]}), ("I am using Python for my Script.", {"entities": [(15, 21, "SCRIPT")]}), ("The chosen Script is Spark and it will work with PipelineA.", {"entities": [(18, 23, "SCRIPT"), (42, 51, "PIPELINE")]}), ("I am choosing left-inner join", {"entities": [(14, 24, "JOINS")]}), ("I am choosing left-outer join", {"entities": [(14, 24, "JOINS")]}), ("The process includes a cross join", {"entities": [(18, 23, "JOINS")]}), ("Using a left join is preferred over an inner join", {"entities": [(9, 18, "JOINS"), (32, 41, "JOINS")]}), ("I want a cross join",{"entities":[(9,14,"JOINS")]}), ("I want a inner join",{"entities":[(9,14,"JOINS")]}) ] for text, annotations in TRAINING_DATA: validate_alignment(text, annotations['entities']) if "ner" not in nlp.pipe_names: ner = nlp.create_pipe("ner") nlp.add_pipe(ner, last=True) else: ner = nlp.get_pipe("ner") custom_labels = ["TECHNOLOGY", "PIPELINE", "PLACE", "SOURCE","DESTINATION","SCRIPT","JOINS"] for label in custom_labels: if label not in ner.labels: ner.add_label(label) train_data = [] for text, annotations in TRAINING_DATA: doc = nlp.make_doc(text) example = Example.from_dict(doc, annotations) train_data.append(example) optimizer = nlp.begin_training() for epoch in range(50): random.shuffle(train_data) losses = {} for batch in minibatch(train_data, size=8): for example in batch: nlp.update([example], drop=0.5, losses=losses) print(f"Epoch {epoch} - Losses: {losses}") nlp.to_disk("/home/datatroops/Session") nlp = spacy.load("/home/datatroops/Session") def extract_entities(text): doc = nlp(text) entities = {ent.label_: ent.text for ent in doc.ents} return entities texts = [ "PipelineXYZ is the new.", "The Pipeline name is PipelineA.", "The SourceName will be SNOWFLAKE", "The DestinationName is MongoDb", "The Script will be a Spark.", "PipelineXY is the new update.", "I am choosing lefti join", "I want my SourceName to be SNOWFLAKE", "SnowFlake", "I can take source as SNOWFLAKE.", "Give source as SNOWFLAKE.", "My script will be Spark.", # "My name is John.", # "My name is Shashwat.", "I want a cross join", "Using a left join is preferred over an inner join", "I want a inner join", "I want an outer join" ] for text in texts: print(f"Text: '{text}'") print("Extracted entities:", extract_entities(text))

if any model has for resume

great vid, I just have a quick question, How do I make a permanent link to share to others

is there any annotation tool for the preparing spacy ner data?

Thank you!

hello sir, does this model detect filler words ?

Its not working. ner_prediction(corpus=doc, compute='cpu') AttributeError: 'DataFrame' object has no attribute 'append'

instead of training an NER is there any way to pass a certain data into the spacy model i.e can we pass the custom data inside a spacy model?

Good video, so much tanks. You helped me a lot.

hey ,can you please provide the training notebook .thanks in advance

Hi, quick question: i had trained the model like you suggested. but when i loaded the best model and tested it on few docs, its returning the docs only instead of the entity. Can you suggest why this would be the case

Works great, but have a question. How can i calculate the metrics precision recall f1 accuracy scores

Good afternoon! Tell me, please, have you published a file with the training of the model? I really like your work and I want to develop in this field!

Hello, I ran the codes and trained the model on the entire dataset, but when I run the inference code, the predictions are empty. Any idea why? Could it have anything to do with the fact that I don't have pytorch_model.bin in my model folder, but model.safetensors instead?

Thx

Im just here to be nosy.. his man is intelligent and fine. Ok byee

Thanks Deepak for your guidance. I was struggling trying to get LocalDocs work with GPT4All and your method of using CSV files works better than LocalDocs. I have a question on further enhancing this. How do I get the model to do the following: 1) List more than one result? I tried k=4, but that did not give me any more than 1 result. 2) How do I get the model to summarize information? Example, I want it do simple things like 'How many novels were written by Tagore?' from a list of his works. Thanks in advance for your additional help.

the model you created is it based on the whole datasets?

Very useful work. But i am getting AttributeError: 'DataFrame' object has no attribute 'append'. can you pls recheck/update the code?

This is a good model that I've been using for my course project for some time. Your work is very much appreciated!

Hi Brother How to annotate automatically for large text data by this i have to do it mannually

Hey, great work dude! I am wondering where can i access this Named Entity Spacy Tagger @ 1:46 Thank you

sir, I am note able to import huggingsound, Please help

The issue with this is, if we want the openai to set I don't know the answer if some irrelevant question is asked, it still answers. How to implement that

❤❤❤

Does this work on intranet environment

I really like the second cluster. Its funny to see twitter at its worst.

"promosm" ❤️

Is it required valid openain credential?

GPT4ALLEmbedding() throwong error "GGML assert "

Thanks you for the amazing tuto. Can we annotat severals data in the same tIME ?

Excellent tutorial!!! It helped me to learn the custom NER, which otherwise looks difficult to follow in the spaCy documentation.

Such a lovely man

the real ones are here for a reason.

Good one,, i m trying to build the knowledge graph using this technique, but have got stuck into it. Would you please suggest me how to tackle it? 1- how to have the 2 edges from the same source node to destination node?I mean I have tried all possible ways best of my knowledge to build more than one transition edge from same source node to the same destination node in the same direction. 2- how to identify all the possible paths from the initial node to the final node, when there's a KG(knowledge graph) is available.

great video, I have a question tho let's say i trained a model with a TRAIN_DATA of 300 texts, now i have 200 more texts to train because the model was not accurate. is there a possibility to just train the same model with these 200 new texts or should i train a new model with all the 500 texts(it will take a long time)? if there is a way how pls ^^

Bring more of these amazing people loving these unscripted podcasts nowadays.

Great content sir . Thank you so much . Learnt a lot from this one . Keep growing 🎉

how can i access the code?

Very good one.. i have a special use case.. let me know how to connect with you do discuss the usecase

Great tutorial!!! Very concise.

Nice where we can see the code for above

ValueError("[E024] Could not find an optimal move to supervise the parser. Usually, this means that the model can't be updated in a way that's valid and satisfies the correct annotations specified in the GoldParse. For example, are all labels added to the model? If you're training a named entity recognizer, also make sure that none of your annotated entity spans have leading or trailing whitespace or punctuation. You can also use the `debug data` command to validate your JSON-formatted training data. For details, run: python -m spacy debug data --help") I am getting this error.......

I have this thing as my training data drive.google.com/file/d/1ssBswos2TAh8OTpcdTz7iDNqU2jCti7V/view?usp=drivesdk How to train now?

I am getting too many errors when I run the model_training code. I have tried running it on Google Colab, but I still cannot get any results. Can you please help me?

Salam Mr. Deepak John Reji, i've tried to follow your video step by step, but when i reach the step 5 - run the training code i had a error massage "TF-TRT Warning : could not find TensorRT" , I have tried so many ways on the internet but until now I still haven't found the right one, can you help me, oh yes, I used google colab to do this coding.

can this approach be useful to group google reviews into categories

Great work. Kindly provide the training notebook, codes on how to train the model. Thanks in advance