If you found this video helpful, please LIKE and share with friends/ colleagues - THANK YOU! Also you can download the notebooks used in this tutorial (3 in total): www.skool.com/microsoft-fabric/classroom/d154aad4?md=7157c442fb334f1ca5dd05d35c63d61c
Brilliant video. I liked how you used the "gx lite" approach at each medallion layer to build out the validation steps. This really makes the implementation a no brainer and removes some of the complexity.
Thanks Ramesh 🙌🏽 Yes, GX can sometimes be “over-engineered” I think which puts off a lot of teams. Sometimes just a simple validation is what is required 👍
It was insightful and provided a lot of valuable learning opportunities. Thank you for sharing your knowledge and expertise.Keep up the fantastic work!
hello, many thanks for this great clarification on validation. one small question: how do you validate data before loading them to warehouse? Notebooks are notebook supposed to reach warehouse. do you consider that everything is validated at silver layer ?
Thanks for the detailed training Will. Quick question: are validation notebooks also placed in a data pipeline? if yes, when organizing the data pipeline, is it recommended that the validation notebooks are executed before the actual cleansing notebooks?
Hi, good question, different strategies exist depending on your use case/ priorities. Sometimes DQ checks might be done before you write new data to a location (to prevent poor quality data being used), other times it is written to a location, then DQ checks are done
@@johnuzoma1823 Yes! Although you'll have to use the Checkpoint functionality to generate Data Docs for your datasets/ Expectations! That is not shown in this video
Excellent! William I would grateful if you expand a bit how treat great expectation results. I want struggling to create a database out of the JSON results
Good point! I'm sure Informatica will produce integrations with Fabric, but it's not something I've looked at currently. You can read more here: now.informatica.com/idmc-microsoft-fabric.html The approach I outlined is based on open-source technologies
Why spend more money on a product to do schema validation when you can create a pipeline that grabs the schema of our sources and then you can run an if condiition task that can evaluate the schema of your files\database table? I love throught process for data validation, but not pushing 3rd party products unnecessarily.
Hey, thanks for the comment! The approach I showed is based on a free, open-source Python library, Great Expectations (plus the other libraries I mentioned are also free). Most real-world use cases require a bit more than a binary if-statement 👍
@@LearnMicrosoftFabric Thanks for the quick response! I appreciate your feedback and will continue to watch your videos. I am quite curious of much fabric will be adopted. For now, I haven't seen much interest. doug
If you found this video helpful, please LIKE and share with friends/ colleagues - THANK YOU! Also you can download the notebooks used in this tutorial (3 in total): www.skool.com/microsoft-fabric/classroom/d154aad4?md=7157c442fb334f1ca5dd05d35c63d61c
Medallion architecture ✅
Data Validation ✅
Well explained ✅
You should be an MVP
your channel is the only source I use to learn Fabric. Thank you
Very kind… thanks for watching!!
One of the best videos on the subject. Great job
Thanks, very kind. I’m glad you enjoyed! ☺️
Brilliant video. I liked how you used the "gx lite" approach at each medallion layer to build out the validation steps. This really makes the implementation a no brainer and removes some of the complexity.
Thanks Ramesh 🙌🏽 Yes, GX can sometimes be “over-engineered” I think which puts off a lot of teams. Sometimes just a simple validation is what is required 👍
It was insightful and provided a lot of valuable learning opportunities. Thank you for sharing your knowledge and expertise.Keep up the fantastic work!
The topic has been covered in a great detail .. thanks for creating such an elaborative video
Amazing work again!! I was looking for DQ options for a project and this video summed it all so well!! Thank you sharing the Notebooks.
No problem, glad you found the video and the notebooks useful 😊
hello,
many thanks for this great clarification on validation. one small question: how do you validate data before loading them to warehouse? Notebooks are notebook supposed to reach warehouse. do you consider that everything is validated at silver layer ?
✅ a whole new level to enterprise data quality.
I made it that far, and many thanks again and again for this great contents.
Thanks for watching!
It was a great training. Thank you so much ✅
Very helpful channel for learning Microsoft Fabric 🥇👍😊
Amazing content, really helpful on my Microsoft Fabric journey! Thank you!
Glad you’re finding it helpful! ☺️ thanks for watching!
Your videos are so helpful .
Great to hear, thanks for watching!
Thanks Will , it was informative and simple .
Thanks for the detailed training Will. Quick question: are validation notebooks also placed in a data pipeline? if yes, when organizing the data pipeline, is it recommended that the validation notebooks are executed before the actual cleansing notebooks?
Hi, good question, different strategies exist depending on your use case/ priorities. Sometimes DQ checks might be done before you write new data to a location (to prevent poor quality data being used), other times it is written to a location, then DQ checks are done
@@LearnMicrosoftFabric Thanks Will. Also, is there a way to generate data docs from MS Fabric? I mean the data docs provided by great expectations.
@@johnuzoma1823 Yes! Although you'll have to use the Checkpoint functionality to generate Data Docs for your datasets/ Expectations! That is not shown in this video
@@LearnMicrosoftFabric yes I tried using the checkpoint functionality. But fabric doesn't have a cmd terminal to mount the html data docs
@@johnuzoma1823 Have you seen this video? ruclips.net/video/jrcr28HGtEA/видео.html
✅ great content on data validation!
Thanks for watching, glad you enjoyed!
Excellent video ❤
Thanks for watching!!
Will, thank you for share your knowledgement!
Excellent! William I would grateful if you expand a bit how treat great expectation results. I want struggling to create a database out of the JSON results
Have a search for how to work with python dictionaries 👍
Excelente contenido, he aprendido mucho el día de hoy ✅
Muchas gracias!
✅ super helpful, Will, thanks for this.
Glad you found it helpful! 🙌
Nice one.Interesting...
Thanks for watching, glad you found it interesting :)
Where does data quality solutions like informatica fit in pls?
Good point! I'm sure Informatica will produce integrations with Fabric, but it's not something I've looked at currently. You can read more here: now.informatica.com/idmc-microsoft-fabric.html
The approach I outlined is based on open-source technologies
Great content!!
Thanks a lot for watching!!
Are you on twitter as well?
Great video
Thanks for watching! I am not on Twitter nope, just LinkedIn
@@LearnMicrosoftFabric what's your name on linkedIn?
This was great stuff! Thanks!
Glad you enjoyed, thanks for watching 🙌🙌
Great content as usual!!!
Thank you!!
✅ Very useful!
glad you found it useful!
✅ - great stuff
Great Video! Even for a not so "deep techi" ;.-)
Thanks for watching! Hopefully not too many buzzwords 😂
✅ Great content!
Thanks a lot for watching and commenting!!
✅ Its already been said in a previous comment, but this really is the best resource I've found on this subject. 👍
Thanks for watching!! Glad you found it useful!
👍
thanks for watching!!
Why spend more money on a product to do schema validation when you can create a pipeline that grabs the schema of our sources and then you can run an if condiition task that can evaluate the schema of your files\database table? I love throught process for data validation, but not pushing 3rd party products unnecessarily.
Hey, thanks for the comment! The approach I showed is based on a free, open-source Python library, Great Expectations (plus the other libraries I mentioned are also free). Most real-world use cases require a bit more than a binary if-statement 👍
@@LearnMicrosoftFabric Thanks for the quick response! I appreciate your feedback and will continue to watch your videos. I am quite curious of much fabric will be adopted. For now, I haven't seen much interest. doug
✅
Thanks for watching!
✔
Thanks for watching!! Hope you enjoyed :)
Excellent content 👌
Thanks for watching! Glad you enjoyed 😊
✅
Thanks for watching!!
✅
Thanks for watching!!
✅
✅
Thanks a lot for watching!!
✅
✅
✅