Multiple forms Processing using AI Builder (Different Layouts)
HTML-код
- Опубликовано: 2 ноя 2020
- This video will provide a step by step walkthrough of building a forms processing model using multiple layouts and utilizing the model in Power Apps and Power Automate.
Announcement - flow.microsoft.com/en-us/blog...
Use Case: Invoice Processing Solution
This process is ideal for automating your paper-based processes with AI & Power Platform.
AI Builder Playlist - • AI Builder for Power P...
#AIBuilder #FormsProcessing #MultipleLayouts #PowerApps #PowerAutomate #PowerPlatform
This is amazing ! Thank you for promoting such great features. This is so usefull.
Glad it was useful! Thanks for watching David.
I tried my first collection today but the array is not recognized properly. Some lines (2 of 5) are not detected. Is it possible to define manually the right size of the array ? Thanks.
@@djdhoy33 When you say array, do you mean table?
@@RezaDorrani yes sorry
@@djdhoy33 If lines are not detected, there is no way no highlight it.
Not sure if you can share your sample docs with me. I can check it and let you know the exact issue.
If you can share the files then send them to reza.dorrani@outlook.com
This is so cool, I had no idea it was this simple to use AI Builder. I hear AI and immediately think complex, the fact you can demo a use case in 20 minutes blows my mind!
Thank for this amazing feedback.
AI Builder truly brings the power of AI in a low code manner to the Power Platform.
You never cease to amaze me, Reza! Amazing work. Thank you.
Thanks again!
I can't say it enough. You are a life saver. An easy one. ;) Is just so easy ad fast to learn from you. The way you have to bring out the information and practice is just the best out there. Very greatfull :)
Thank You Tita.
That is exactly what I am looking for now. Thank you so much Reza
Great 👍
Thanks for watching
As always Reza..your videos will help me build something new and i will use it for my companies use case..thanks again
Thank you
Thanks Reza for the video on AI Builder.
Thank you for watching
Great stuff, I needed this today.
Thanks for watching
Brilliant and informative video. Thank you once again.
My pleasure!
Thanks for recommending this video.
You're welcome
simplesmente espetacular! Tudo que eu queria saber em um único vídeo. Muito obrigado, Reza.
Fico feliz em saber disso! Muito obrigado por assistir.
This so rocks - thank you!
Awesome! Thanks
Outstanding video Reza!!!
Thank You
so cool! thanks for sharing
Thanks for watching!
Thanks Reza great video
Thanks for watching Sundra
Awesome ... very welled explain ... Great work Reza.... keep sharing more ... :)
Thank You
thank you!
👍
This saved me!
👍
Reza, Thanks for great content! If we have to use this for the existing files in the sharepoint, which trigger should I use?
Run a scheduled flow, loop through your existing items and apply the AI.
After that, use the flow for new items.
An inspiring video, Reza. Would you share some more about the lookup column formatting of "Invoice Items" @18:15 that displaying a list of invoice items relating to own invoice ID? Does it refer to advance formatting using JSON?
I used column formatting for lookup info. It links to secondary list and pass the ID param to filter data in list to show the related items.
Great video. I also had a few questions. If I receive vendor invoices in a shared mailbox and need to route it for approval to department heads based on the cost centres where the po was raised, do I use power automate flows leveraging the ai model invoiceprocessing? Also if I would want to see the status of approval against each invoices in the document library as in adding an extra column , how can I link the approval flow to the status column for each invoice in the document library
Hello, for the vendor invoice routing from shared mailbox you would need to leverage flow (Power Automate) to route it for approval. I do have a nice approval videos playlist that might come in handy - ruclips.net/p/PLTyFh-qDKAiFUGTtwSwFT79Y7_jq_Aipe
The approval flow can update the status iof the column in your doc library. You can use the SharePoint update file properties action to do so.
This Video was extremely helpful! I have a question regarding the last part where you created a conditional format to link the invoices with the items and provided a quick link to see it. Do you have a video that covers how to do this?
Thanks Rishi.
I do not have a video on that. It’s been a long time since I did that. All I did was use column formatting in SharePoint to link to related list and filter items from url.
Hi Reza!
I wanted to know if it's possible to implement 2 to 3 models parallelly in a flow and filter out which model to use based on the domain of the email addresses from which I receive invoices since I have way too many invoices, all with different layouts and variety of information (200+ collections approx).
And appreciate the video! You make it super easy to understand.
You can create upto 100 or 200 per collection, I think.
You can also create multiple models, add a switch case in flow to check domain and accordingly call the respective model.
Great video Reza, is it possible to set some threshold for confidence in Automated Flow and documents with lower confidence send to AI Builder, for improvement or retrain the model?
Yes, possible. In flow, the form processor action output returns confidence scores for each of the data points captured. You can check those scores and accordingly send notifications to admins.
When you build the model, in the model screen there is a confidence score for the model as well. If low, then add more documents to train the model.
Hey Reza, Amazing!!
Would it also work if you uploaded word documents or pictures png or jpegs or scanned pdfs instead of pdf to your sharepoint?
Does the AI model identify itself which type of form (collection) the file contains? So file names dont matter..
Thanks
This video was done a while ago. AI Builder has had plenty of improvements since. I would recommend checking the documentation to see what formats are supported.
Hi Reza, another great video. One question, i have multiple files with multiple tables inside, is it possible to AI Builder read multiple tables in a file or just one table per file?
I believe the new version does multiple tables. I will recommend checking on the forums at powerusers.microsoft.com/ in case someone has done something similar.
Thanks for the video. I am going to start using this in the near future. When you started working with the form and you added the table, where did your database come from? Did you set up the "Invoice Items" table previously? I am not sure how to connect to my data source?
My data source is SharePoint. I had the lists setup and connected in the App. The idea of the video was to showcase the possibilities of Forms Processing.
@@RezaDorrani thanks for the info. I appreciate the quick response.
Great video Reza. I have a question: Why should I use AI builder from Power Apps and not Form recognizer from Azure. The difference in cost is huge. Thanks
Firstly, thank you for watching.
Great question on Azure Cognitive vs AI Builder.
This comes down to the low-code AI capabilities in Power Platform using AI Builder vs Azure cognitive services which requires more work to setup the form recognizer model, an azure subscription & then calling the API endpoint.
The entire video demo was done in 1 hr. since all I had to do was train my model, publish it, and then use it in Apps and flows using the AI builder controls/connectors.
Hi Reza. Thank you for a great video. Would this process apply to multi-page forms, like a CV where the experience table can expend across various pages?
AI Builder document processing model has multi-table capabilities
Great tutorial! I have one question though, how did you create that invoice items column link inside of invoices library? I'm really struggling with that.
Using JSON column formatting. Did this a long while ago and no longer have access to the code.
Hi Reza, thank you so much for these videos! I work for a SME and am looking to automate the AP process. I have created a flow for approval of a document and want to utilise the AI Builder to create a CSV file upload into our accounting software for supplier invoices. This works well for a one to one relationship E.g. One invoice = one cost category and job, but is there a way to work with more than one cost category or job per supplier invoice? E.g. multiple lines on the invoice for say labour and materials, or one invoice for works completed on multiple jobs? Thank you! P.s. I am a little late to the game with Power apps so apologies if this has already been answered / resolved and I haven't found it yet!
There are ways to related an invoice with multiple categories.
I do not have a video on that scenario though.
I am planning a live session on AI Builder next month with the product lead from Microsoft.
I will post about it on social media in a few weeks time - twitter.com/rezadorrani
@@RezaDorrani This would be amazing, thank you so much! Will follow you now for updates. Appreciate the response. C
Hello Reza, Great video as usual....my customer wants to process legal documents that do not have tables of data like the invoices. They want to look within the document for specific phrases (meta data is pre-defined). Does AI Builder Form processing still be relevant for this use case? Is this called Form Processing for a reason - i.e input should be a Form? Is there any other model available in AI Builder that is more apt for non-form like documents that run into several pages (anywhere between 4-100 pages). Thanks, Ravi
There are options with AI Builder. I will recommend to check with twitter.com/fdezjose
Reza! Thanks a lot for this informative video. I gave u thumb up. Nevertheless, I have one question: after trying to rebuild that model with my invoices, I faced some problems with the model to recognize tables. My model does not recognize the tables and I cannot do it manually. My guess is that the lack of clear separators in my tables are causing this problem. Is there any way to circumvent this problem? In your case you used digital invoices with clear tables; in reality not all companies create clear invoices unfortunately. Thanks a lot in advance!
Hello, table auto tagging is indeed an issue when clear seperators are not defined.
Good new is that the ability to manually tag not detected or wrongly detected tables is coming as part of 2021 Release Wave 1 release.
Great! In that case, I will wait for the Update :) thanks for the quick response Reza
@@Izzet1967 Most welcome. I got the response from the product team just for you :)
Hello, Thank you so much for this amazing videos, I really enjoyed and I have learned a lot, thank you thank you, with that said I do have one question my version does not le me modify the text table because there is no text option to change ThisItem.Description, you showed at 12:22 minute and just shows the table as object object, can you help me please :)
For the DataTable, the Items property must be FormProcessingControlName.Results.TableName
@@RezaDorrani Reza oh man, you are amazing thank you for replying - I did that but it doesn't show anything it just shows "Object Object" and there is no description property
@@eduardosacahui2695 post your issue here with screenshots
powerusers.microsoft.com/t5/AI-Builder/bd-p/AIBuilder1
Thanks Reza, when we're looking to see how similar the forms are, what is that based on? Is it just the layout? Meaning the position of where things are? What is there are similar positions, but there is additional text data in another form? Is that treated differently or no?
Also, anyway to train the AI model to extract data based on certain text it finds?
It is the layout (Position of elements).
Additonal text is fine. Data can change.
Once you retrieve the data, you can add your additional logic in apps or flows but not in the model while training.
@@RezaDorrani ok thank you! If I have hundreds and hundreds of varying categories, is there a way to use AI to train it to find the relevant info, without having to create one for each category? Maybe it combines keyword search + positioning to find the relevant info?
@@dustinek2973 Possible but depends on the use case at hand.
Awesome! brilliant Sir. I have Doubt can I use this for Price tag price checking , do this have adding people's similar in Google Form builder.
Not sure if it has something for price tag checking. Check in AI Builder documentation docs.microsoft.com/en-us/ai-builder/overview
@@RezaDorrani thank you 🙏🏻
Hi Reza, thank you for your video, I'm trying to build the same thing, but somehow my data table keeps showing [[error]] and [[object object]] in the quantity, item price and amount columns... Do you have any idea what is going on there? Thank you in advance.
I have not come across the mentioned issue and hence not sure what the reason for your issue is. I recommend posting your issue with screenshots on the forums at powerusers.microsoft.com
Thank you for this informative video. If the we want to extract from a different invoice do we need create a new collection and train it? and does it support other types like jpg, png ...
You can use a new collection for that.
I will recommend checking documentation for which file types are supported.
@@RezaDorrani ok, Thank you
Excellent video! I have a query. Power automate has a data extraction limitation of maximum 300 data, if I perform my flow in Power Apps I can extract more than 300 dataPower automate has a data extraction limitation of maximum 300 data, if I perform my flow in power apps can I extract more than 300 data?
Not sure about that limit and it’s workarounds to be honest
Hi Reza, This is a wonderful video. Can you help me by sharing the links of your Collection of Structured Sample Invoices?
Those were sample docs created. I dont think I have them anymore.
a bit of a question how would a document with the same structure (e.g. same inv number position) but different scale/layout (very slightly stretched) be stored? in the same collection or different collection
You could store it in same collection but then you need to train the model with data from both types of documents (scaled and unscaled)
Hi Reza, I Can't find 'Process and save information from from on my AI Builder on my Power Automate?
Action names have been updated since. Look for actions in AI Builder connector.
Hello Reza, Is it possible that this Model can fill out forms for you instead of extract data? Say forms that require name, and date and so on for example.
This model is purely for extracting data. Once extracted you could leverage that data - stored in variables or collection to fill a form.
How can we link the suppliers externally with our sharepoint portal that they will send the invoices?
Is it possible that they share the invoice and the AL builder will fetch the invoice data and save in to sharepoint portal?
Possible. Suppliers could send invoices in email, or store in a file share etc.
Hi Reza,
I m experiencing issue, created automated flow when new email received contains a request sheet in pdf format then added AI build model via form process. However, unable to update extracted value in Excel online into onedrive by adding New row, pls advise if you could create a video on this flow
Looks like your issue is more on adding data to excel and not with AI Builder retrieving values.
I can add this request to my backlog but I build videos when I get many requests thag have a similar pattern.
I will recommend you post your issue on the forums at powerusers.microsoft.com
Hello Reza, The licensing cost for this is too much :). We have closely 5000 forms per month to process and it costing around USD 2000-3000 per month. Do you know if this can be achieved with less price using azure cognitive services? By the way, we are uploading the documents on SharePoint itself but dont want to go with AI Builder Forms Processing. Any suggestions or references that you have?
Yes, it would be cheaper with Azure Cognitive Services. Compare it by plugging in your numbers in the Azure Calculator.
I have a query if you can answer. I have pdf files which has 10-12 single fields and two tables having multiple rows of data in the pdf.
I have trained the model and I want to save the information in an excel file. In power automate when I am creating the flow by using add row in a table option when I select all the fields what happens is the single row items gets duplicated till the numbers of rows are getting copied in the excel file. What I want is single field data is copied and then table data is copied. How to do it?
In short can you show me how to copy single field items and table items in an excel which is on one drive?
I would have to look at your flow in action to provide guidance. I will recommend posting your issue with screenshots on the forums at powerusers.microsoft.com
what are u getting the file methadata for ?
thank you!
I wanted to get the Media Type property of the document since AI builder requires the doc type.
However, you can also get it from the trigger using expressions. I wanted to keep video simple :)
Thanks Reza , Is it possible to read table rows content continuing from Page 1 to page 2. My use case is below
I need to extract information in tabular format from order confirmation pdfs received. Each pdf has multiple items and each item will have a Name, description, Vendor and delivery date.
So the table will have four columns: Name , description, Vendor, Delivery Date with each row representing an item.
The problem arises when some details for an item are present at the bottom of one page and the remaining details are on the next page. Example : Description in the table continuing in the page 2 from page 1 bottom , So unable to tag these rows which is continuing from page 1 to page 2.
For example: if this is the pdf
-----some text--------------------------------------------
-----some text---------------------------------------------
code: 1
description: this is first item
Vendor: XYZ1
delivery date: 12.01.2024
code: 102
description: this is second item
Vendor: XYZ2
delivery date: 13.01.2024
code: 103
description: this is third item
-------page 1 ends here---------
-------page 2 begins here--------
description(Continuing from Page): this is third item Continuing
Vendor: XYZ3
delivery date: 14.01.2024
code: 104
description: this is fourth item
Vendor: XYZ4
delivery date: 15.01.2024
code: 105
description: this is fifth item
Vendor: XYZ5
delivery date: 16.01.2024
---------some text here--------------------------------
------------------------------page 2 ends----------------------
------------------------------pdf ends----------------------------
The document cannot be tagged correctly using custom model when page 1 content - Description is continuing on Page 2 . For the above document, the tagged tables look like this
Code Description Vendor Delivery Date
101 this is first item XYZ1 11.01.2024
102 this is second item XYZ2 12.01.2024
103 this is third item XYZ3 13.01.2024
Code Description Vendor Delivery Date
Some text are
continuing
from page 1
104 this is fourth item XYZ4 14.01.2024
105 this is fifth item XYZ5 15.01.2024
When I did this video years ago it was not a feature. I believe now it is.
Il recommend checking documentation for AI Builder.
I need to build a solution similar to this, but we do not have Premium licenses. Is there ANY way I can utilize Power Automate actions alone to extract data from uploaded PDF forms and populate a SharePoint List? Thank you
Not without premium licensing.
thank you rezza but I wonder if I can use my model in an existant app
You can use models in existing Apps as well.
You can add the AI Builder controls and then connect to your model.
How is this "AI" different from diy code...if always the fields you look are in the same place/region you can just code a simple function?
AI is looking for fields within document. You cannot write a simple function in power apps to fetch data from within document.
Can it be used to create multiple IRS Form 1099 NEC?
Idea behind AI Builder forms processing is to read data from multiple form types. Create multiple forms would be a no but your could train the model to read information from multiple IRS forms.
Hi Reza,
Can this be achieved using power automate desktop? What do you think could be the challenges? This will be run by single person and not shared
Possible. I would have to try it out myself to know about any challenges :)
@@RezaDorrani I am trying it out :) . I was able to extract the data, struggling with form filling in powerapps.
@@KavitaSingh-oc5ku I will recommend you post your issue with screenshots on the forums at powerusers.microsoft.com
Hi, I tried a cool thing, thanks for the video.
But the question arose, if I have a large invoice, several pages and the table is split into several pages, how in this case to carry out the table tag
For videos split across pages, I believe you would have to tag them as seperate tables. This is something that I have not tested. But I can ask the team and let you know.
@@RezaDorrani Hi Reza, I am interested in processing several pages of invoices too. I see you guys discussed this a year ago. Just wondering if a new video was created since then?
@@alexandraafanasieva389 There have been advancements to AI Builder which allows tagging multiple tables. We have a session planned for AI Builder with Microsoft on the R&D show for March 2022.
Hi! Could you plz do video on how to trigger multiple collection with power automate?
I make videos when I get multiple requests on a particular topic.
Every quarter I ask subscribers for topic suggestions on the Community Tab of my RUclips channel.
Subscribers vote for topics and the highest voted topics get added to my backlog.
Make sure you post this in the next topic suggestion post. If it gets a lot of votes, I will add it to my backlog.
We have over 1,500 suppliers - do we need to create 15 models with 100 collections? How do we determine which model to use for each invoice?
There is a generic Invoice model available which you could use and works for all common invoice types.
For specific ones, you could use the model described in the video.
Cant see the action "Process and Save information from forms" in AI Builder. Any suggestions?
I believe the actions names have changed. Search for AI Builder in flow, it will list out the actions.
Good Day Reza,
I was wondering why don't you share your experience with multi Audit Check Lists. I can see lots of people interested in such application tip and trick with your brilliant ideas and final output.
I like your idea. So, how about bulk updating records - like bulk status updates, record deletions and copying items? Does that sound like a good video?
@@RezaDorrani to tell you the truth, yes, sound perfect like all your videos, I put reminders for all premium to be the first to add knowledge. You way of introducing ideas and your techniques are perfect.
@@ehabatme5324 Done - this video is on.
Bulk operations in Power Apps.
I have received some flak for always using SharePoint so for this video I will use CDS and SharePoint. Will show both flavors.
can AI Builder detect data in if the invoice has multiple pages?
I don't believe it can detect that.
I would recommend you post this question on the AI builder forum as well - powerusers.microsoft.com/t5/AI-Builder/bd-p/AIBuilder1
how to use a multiplication function in a power automaton flow knowing that we have used ai builder "extract data from doc" to extract the data we need and add it in an excel file
the multiplication function is a price field in a table extracted with ai builder * a Qty field in a sharepoint list
always appears an error message thank you for helping me
AI builder will read data as text.
You can use expressions in flow to convert it to int or float and then perform the multiplications.
I do not have a video reference on this scenario though. I will recommend checking on the forums at powerusers.microsoft.com/ in case someone has done something similar.
I want to extract the pdf data and tables without using AI Builder and desktop flow, Is there any other solution that i can go with?
Thanks in Advance!!
Im not sure if there are other ways.
I will recommend posting your query on forums at powerusers.microsoft.com
Hi Reza, i am really big fan of your videos and your tutorial helped me lot in the beginning of using power automate. I am test lead by profession in IT firm and music producer by interest. I need your help in some sort of tutorial if possible. I want to automate testing in one of our company legacy desktop application but the challenge which i am facing that when any new changes happens in the application it leads to fail my flow if mouse axis click is diffrent in my previous run. For instance, if i want to find a customer with customer number and i had created the flow with the help of windows recorder either with image recognition or through picking up UI element but if any build comes in and changes the UI element position or if someone else want to use my flow to test in different resolution monitor than it actually fails my flow. It would be really appreciated if you provide any solution for this through a tutorial or by messaging me.
I know you are god in power automate. I believe god will definetly help there child. :D
Your fan,
NNMusic
Firstly, thank you for following my videos.
UI flows now called desktop flows have progresses a lot in the last year.
MSFT now has a new tool called Power Automate Desktop.
Will highly recommend watching these videos - ruclips.net/p/PLi9EhCY4z99XC7jwJlMyHU0iYO8dR5cVc
Can you give me idea where this AI Invoice technique use? Which type of company use this
Form processing lets you read and save information from standard documents such as invoices or tax documents.
I have worked with customers who have automated their invoicing solutions using AI Builder.
docs.microsoft.com/en-us/ai-builder/form-processing-model-overview
For routing the forms for invoice approvals to department heads based on cost centres , what do you think is the best approach, would it be leveraging the ai model within flow , also how can I see the status of approval in the sharepoint document library as an extra column
@@reenuleenav Flow would be your best option. Flow has approvals action + it has the SharePoint action to update file properties.
From where i can get your invoice templates?
Those are ones I manually created. You can use any invoice of your choice and train model accordingly. Video was to share an idea on what can be achieved.
Hi Reza. I m try to get data from invoice having same table continuous in more than one page. I'm getting only first page table data only. Is there any solution for that please help on this.
I believe there was an update made since this video for multi page tables. Not sure if that is available yet. You will need to check AI Builder documentation for that.
@@RezaDorrani I checked but not found any proper solution for that
@@jallabalaram4331 Then may be its still a limitation. Try checking or posting your query on forums in case someone has done something similar
powerusers.microsoft.com
Is this limited for 4 variables?
Can you point to time in video where I have shown 4 variables?
Hi.. how we can fetch information from multiple pages in single PDF?
The AI Builder model also multiple page pdfs.
sir , i need to flow the data using a send button from power automate flow to dataverse....how i can do it ??
please help #URGENT
I do not have a video reference on this scenario and would have to try it out to provide guidance. I would recommend posting your issue/query with screenshots on the forums at powerusers.microsoft.com
I don't see forms processing i have document processing?
This video was done a while ago. Forms processing has been renamed to doc processing.
how to use a multiplication function in two fields model extract information from documents multiplied in a sharepoint field in a powerautomate
Model reads data in text format. You would need to convert it to numbers using Value function and then multiply.
@@RezaDorrani thank you for your answer, can you help me how to do that
@@hibaGh-km5ci I would have to recreate your scenario and try it out to know the steps. I do not have any references on it either. I would recommend posting your query on forums at powerusers.microsoft.com
Still does not capable to process complex document. If only microsoft provide custom layout where we can set data definiton manually. It is a long way, we need to wait for that feature.
I would recommend submitting an idea for this on the forum.
powerusers.microsoft.com/t5/Power-Apps-Ideas/idb-p/PowerAppsIdeas
What is the cost of the Add in?
500$ monthly to start with.
There is a calculator for AI Builder
powerapps.microsoft.com/en-us/ai-builder-calculator/
@@RezaDorrani rather expensive for a SMB. The AI features are incredible but is it possible to do this with OCR software running on an in-house server. Did you try that at all with PowerApps? If so I would love your feedback. I think Fujitsu has OCR software with a product like ScanSnap.
@@rashantha I have not tried other software. I have tried azure cognitive services.
but if is more that 100 types of invoices ?
There is a new Invoice processing model (productivity model) which does not require any training.
Its in preview - docs.microsoft.com/en-us/ai-builder/prebuilt-invoice-processing
@@RezaDorrani From the presentation, we have a collection and one invoice provider, so from start may problem is that I have over 500 different types of invoices (in my country the invoice format is not standardized. is ok to put different types of invoices in same collection?
@@adrianpetrescu8583 There is a new model called invoice model (preview) that does not require any training and works for most invoice types.