I’d like to see a “data solution in 10 minutes” built in Synapse. Would be really powerful if that included SQL and Python to demonstrate the flexibility.
On the one hand this format could cover topics like Azure Synapse in conjunction with Power BI and on the other hand Synapse with SQL/Python/PySpark etc. I would be really interested in this. Also, do you recommend other youtube channels for Azure Synapse? I think it could be hard to keep up-to-date with the latest updates in Power BI AND Azure Synapse.
I dont think azure synapse is a magic wand (correct me). All I understand is that it is an ETL and orchestration tool and more . You still need to do that development work!
I JUST got access to our Azure instance Friday and was starting to look for resources to watch. Then this video comes up. You guys are the best and your timing is _impeccable_ .
Love it! Really happy to see you guys recording this series. Looks like it's going to be one of these "must see on Azure Synapse" materials we will share with the community!
Glad that you are going to do a detailed series on Azure Synapse. I just joined Microsoft and i have a opportunity to work on Azure Synapse and your detailed explanation will surely help me
Besides Guy in a Cube (love you guys!) you can check these resources to learn Azure Synapse: - Azure Synapse Analytics RUclips channel (see the Essentials playlist there): aka.ms/SynapseRUclips - The Get Started in 60 Minutes video: aka.ms/SynapseGetStarted - Azure Synapse Analytics Toolkit (a set of documents for quick start with different aspects of Azure Synapse): aka.ms/SynapseToolkit - RUclips channel of Data Toboggan community events (focused on Azure Synapse): ruclips.net/user/DataToboggan - Synapse Practitioner site (lots of free online learning paths): aka.ms/SynapsePractitioner - Learning paths on Microsoft Learn site: aka.ms/SynapseLearningPaths And, of course, stay tuned for more awesome Azure Synapse videos from Guy in a Cube ;-)
Subbed in the hopes of synapse analytics content! Keen to see it in action, the whole process from ingestion to transformation to power bi. Would love some demos, touching on the main functionalities.
Really helpful overview. For some reason I always assumed Synapse was just Azure's cloud database offering - didn't realize it was all of these other concepts too. Thank you!
In brief, as per my understanding: - Power BI provides strong ETL with Power Query and even data modeling. Moreover Power BI golden rule is to use a star schema. - ASA for sure provides the all the tools for datawarehose while lacks visulization features as far as I know. For DW, PBI star schema (e.g. 1:n dim to fact relashionship) is not striclty a must Moreover, because of the tons connector and modeling capabilities available in PBI, it can be used to pull and shape data from existing data marts and provide "enterprise" solutions just within PBI So what is/are the trigger(s) point(s) that suggest to move from pure PBI solution to ASA+PBI? I know, depends... ;-))))))))))
Doing ETL in Power BI is great if you only need to load to Power BI, if you need to get that data anywhere else it is pretty poor (Dataflows can be created to load to Dataverse tables but that also has its own limitations; Dataverse also has an Azure Synapse Link feature)
The dedicated sql pool in synapse can get very expensive. If your data volumes aren't in the terabyte range then you probably don't need a distributed architecture like synapse. I doubt the tooling or development experience in synapse would be a reason for any organisation to use it... azure sql + data factory gives you the same things The distributed nature of synapse is the whole reason it exists. This isn't a technology designed for 10GB databases :)
@@jbonomo have you tried serverless SQL pool in Azure Synapse? I think the combination of this engine and Power BI can meet the requirements for small business analytical use cases when data is not that large and should not drain the budget. Synapse is not a single engine (right now there are 4 of them - besides dedicated SQL pools also: serverless SQL pools, Spark and Data Explorer - all addressing different use cases and having different pricing models).
Amazing video guys! It would be nice to know the difference between the power bi in synapse and the regular one. what can I do in synapse and what I can't. and of course why would I use the one in synapse...
Great videos, as always. I have a couple of question regarding lake house. 1/ Our IT department is moving the analytical data to the Azure cloud. Upon this move they want to add a non-azure product for data virtualization. Does this make sense? What are the azure possibilities here? 2/ what about erroneous data in the data lake, how the identify the erroneous data, correct it and restart the process without the erroneous data 3/ what about showing data in reports containing enough data in order to avoid possibility to identify the source
1) Major drawbacks of Spark Pool in Synapse vs Data Bricks and vice versa? Which is better? When we are doing POC initially to evaluate Spark pool in Synapse, we have observed lot of drawbacks when compared to Data bricks. I would love hear from you with all the recent updates, can Spark pool perform much beyond than Data bricks? If yes on what basis? 2) Why not Azure Purview in Synpase? We can do governance directly there itself instead of a separate service? Any reason for making it separate? 3) One major drawback I observed is, sql pool can't support three part naming and we need to create separate schemas in the same sql pool to differentiate instead of creating different sql pool which is a costly operation. But it all together doesn't correlate with the traditional EDW modeling we do like we keep separate staging, logging, edw databases etc. 4) Why can't we delete duplicates direcly querying CTE pointing to Original table if the sharding pattern is other than Hash? We have alternatives to perform this task. But want to know the reason for why can't we directly delete using CTE? 5) Any concept of queue while submitting Spark jobs in Spark Pool? How much time will it take to spin up the cluster and run the jobs that are in queue? Comparison with data bricks make us learn better. Please help us in letting know the above queries. Will post further questions once they arise.
I come from the tradicional DW / BI and i'd love a session that would show the "state of the art", this is, ok, i have a DW on synapse, i need to pick these log times from machines to do some analysis, and some Excel files to do another job, and so kind of seeing, by example, all (or most) parts of synapse working and connecting together...
I like to see a use case of predictive analytics that gives me insights on how to get started on this tool and bring the product to finish line... from raw data to end result in power bi
Hellooooo... Love your videos... Always so filled with the right kind of Energy. You mentioned data orchestration and learning the clicks... What does that mean?...How do I start going from Power BI analyst to using Synapse and become more of a data engineer?
I know why you are doing this, but I hope you keep the message very clear, PowerBI is still an independent BI tool and works with any platform not only synapse
It'd be nice if there was a more affordable tier to explore Synapse. I was just looking on the Azure pricing calculator and looks like the cheapest monthly fee is a couple thousand dollars which makes it pretty tough to get much hands on exposure.
Does you guys have some sort of commercial relationship with Microsoft and Azure? If yes you probably should make a disclaimer at the beginning of the video.
Thanks for the valuable insights... can we have high level sample end to end solution done using Azure synapse with its integration with Power BI Thanks
I guess the problem (and oportunity) with Azure BI platform is that cost is often variable which discourage learning and experimentation from power users - something similar to Power BI Premium would be great which would lead to greater tool adoption
I did a POC attempting to connect a Power BI Dataflow to a partitioned ADLS Gen 2. Dataflow connector would not work, Then created a database and CETAS table in serverless SQL. Using Azure SQL connector works but the issue was that if you have partitions like puiYear=2022/puMonth=05/puDay=03 when you append the Gen 2 storage with another days worth of data it will never be seen in Power BI nor in SSMS. Finally if you use views instead of CETAS tables, when you append additional days to your partition all you have to do is refresh the DataFrame and DataSet and everything is properly refreshed. The appended data also shows up right away if you use SSMS to query the view. Problem is that this is NOT documented anywhere that I can find. Would really appreciate it if you could give us best practices for using partitioned Gen 2 storage with Power BI. I am looking for confirmation that this is the recommended method and that we will not have issues down the road,
I am having 2 projects on Synapse. The first one is extracting data from Dynamics AX 2012 on-premises SQL into synapse, trasnform the data into star-schema in Synapse, which being used on Power BI (BI report maintained by IT developers). The second one is extracting data from both Dynamics 365 Customer Engagement and Finance Operations thru the Azure Synapse Link (Data Lake connector), trasnform the data into star-schema in Synapse, then populate into Azure Analysis Service and into Power BI (maintained by >100 business users and self service thru ASA). I believe each component has a bit overlapping featurs however they have different target audience, different technology at the back, and different level of user friendiness. In fact, i can't find an official doc to differentiate them. It would be great if you could share you point of view. Thanks.
Synapse Analytics is definitely amazing. I was thinking about a good way to visualize Delta changes . B/c of all the fields and multiple versions, there's quite a bit of possibility.
How can I compare Synapse Analytics to AAS and SSAS. Can I use DAX? Does it has a visual diagram? It leverages column store? Does it has modern data sources with M code? How it performs?
When working with Azure Synapse Analytics, is it still best practice to work in different environments for DEV -> TEST -> PROD? Do DBA's still have a role in helping to promote code between those different environments? Also, if I use Azure Synapse, is there a need to use Azure Data Factory or Azure Data Lake independently, or is that all bundles into Synapse and does Synapse "replace" the independent use of those other products?
Hi Guys, I manage a IT Data systems and currently we use the traditional methods of ETL/ELT(ADF) and then use Azure SQL to process and store the data. How easy it is for me to move from this to AZURE synapse? Is it worth the transformation or this needs to be setup from the scratch. Thank you. VKP
I'm a Data Analyst with knowledge in SQL, Python and Excel. I want to get started with Azure Analytics. So, Azure Synapse would be enough for me or do I need something alongside that?
whats your vision on large datamodels that do not fit in power bi? direct query to synapse sometimes sends strange sql queries that don't perform either on large datasets +1 billion rows
Hi. What I have read is that Azure Synapse Analytics (the MPP bit) has big advantages once the data reaches ~ 1tb in size. Else it is not cost efficient. Is this true ?
Hi. On the SAAS platform like the Azure SQL MI / synapse, the debug mode is not available. this is making it very difficult to debug stored procedures which sometimes can get very complicated. We have been told by the MS Support team that the components required for debug are not available in the SAAS platform. I had to set up a VM, install SQL Developer Edition and do my work. Once the SP was ok, I deployed it to Azure SQL MI. Does the ASA overcome this limitation ?
Hi guys thanks for your videos, you are amazing... About Synapse, where can I get a free way for testing, with the a Azure account, you pay for Data Factory for example. Best wishes 😃
It's so frustrating to be subscribed to this channel. I only know some Power BI... and like what the fuck is spark? I really would like to understand the things you guys talk about on most of your videos but is way over my credentials.
Hey Ivan. I was in the same boat as you a few years back. Guy In a Cube is great because they have LOTS of content for beginners (look at their "Getting Started with Power BI" playlist). But I promise if you keep watching the new content and do some google searching on the side as you go, you will eventually see how powerful each one of their videos is. From one PBI guy to another
I’d like to see a “data solution in 10 minutes” built in Synapse. Would be really powerful if that included SQL and Python to demonstrate the flexibility.
Great idea! Please!!
Yes, please!
On the one hand this format could cover topics like Azure Synapse in conjunction with Power BI and on the other hand Synapse with SQL/Python/PySpark etc.
I would be really interested in this.
Also, do you recommend other youtube channels for Azure Synapse? I think it could be hard to keep up-to-date with the latest updates in Power BI AND Azure Synapse.
If u ever found it ... Please tag me
I dont think azure synapse is a magic wand (correct me). All I understand is that it is an ETL and orchestration tool and more . You still need to do that development work!
I JUST got access to our Azure instance Friday and was starting to look for resources to watch. Then this video comes up.
You guys are the best and your timing is _impeccable_ .
Love it! Really happy to see you guys recording this series. Looks like it's going to be one of these "must see on Azure Synapse" materials we will share with the community!
We are moving to ASA recently and couldn't be happier. I hope you guys will have more vids on it in the future.
Great to hear! The plan is to drop videos on Tuesdays going forward.
So happy you guys are branching out to Synapse! Very much looking forward to more videos with the GiaC production quality on this topic!
Glad that you are going to do a detailed series on Azure Synapse. I just joined Microsoft and i have a opportunity to work on Azure Synapse and your detailed explanation will surely help me
Yes Please, we need detailed videos about Synapse with databriks, power bi etc. all the Azure data services please
Just starting the journey. Looking to get the data journey to make sense and looking at the solution instead of the individual silos.
Is there a good YT channel for learning Synapse, like Guy in a Cube? Where do I start?
The plan is Guy in a Cube :) just getting started and will continue this journey. 👊
Besides Guy in a Cube (love you guys!) you can check these resources to learn Azure Synapse:
- Azure Synapse Analytics RUclips channel (see the Essentials playlist there): aka.ms/SynapseRUclips
- The Get Started in 60 Minutes video: aka.ms/SynapseGetStarted
- Azure Synapse Analytics Toolkit (a set of documents for quick start with different aspects of Azure Synapse): aka.ms/SynapseToolkit
- RUclips channel of Data Toboggan community events (focused on Azure Synapse): ruclips.net/user/DataToboggan
- Synapse Practitioner site (lots of free online learning paths): aka.ms/SynapsePractitioner
- Learning paths on Microsoft Learn site: aka.ms/SynapseLearningPaths
And, of course, stay tuned for more awesome Azure Synapse videos from Guy in a Cube ;-)
Love this. This is the content I'm looking for as I take the next step in my Power BI learning and my data journey.
Subbed in the hopes of synapse analytics content! Keen to see it in action, the whole process from ingestion to transformation to power bi. Would love some demos, touching on the main functionalities.
Really helpful overview. For some reason I always assumed Synapse was just Azure's cloud database offering - didn't realize it was all of these other concepts too. Thank you!
1) Migration. e.g. migrating on prem SQL data and SSIS packages (in bulk)
2) Homogenizing SQL & Power BI w/ ML libraries
Ayer vi un vídeo de ustedes, y hoy los busqué para que me expliquen Azure Synapse Analytics, y seguiré así porque explican bien. ¡Suscrito!
I would LOVE you explain spark! The way you explain the topics are great!!
In brief, as per my understanding:
- Power BI provides strong ETL with Power Query and even data modeling. Moreover Power BI golden rule is to use a star schema.
- ASA for sure provides the all the tools for datawarehose while lacks visulization features as far as I know. For DW, PBI star schema (e.g. 1:n dim to fact relashionship) is not striclty a must
Moreover, because of the tons connector and modeling capabilities available in PBI, it can be used to pull and shape data from existing data marts and provide "enterprise" solutions just within PBI
So what is/are the trigger(s) point(s) that suggest to move from pure PBI solution to ASA+PBI?
I know, depends... ;-))))))))))
Doing ETL in Power BI is great if you only need to load to Power BI, if you need to get that data anywhere else it is pretty poor (Dataflows can be created to load to Dataverse tables but that also has its own limitations; Dataverse also has an Azure Synapse Link feature)
Etl in power bi is very simple. If you want to be more powerful you should look at synapse
I was waiting for this video.👍 Great work
Woot! Appreciate that! 👊
The dedicated sql pool in synapse can get very expensive. If your data volumes aren't in the terabyte range then you probably don't need a distributed architecture like synapse. I doubt the tooling or development experience in synapse would be a reason for any organisation to use it... azure sql + data factory gives you the same things
The distributed nature of synapse is the whole reason it exists. This isn't a technology designed for 10GB databases :)
Glad to see someone say this. So much hype around Synapse but it is not cost effective for small businesses without TBs of data
@@jbonomo have you tried serverless SQL pool in Azure Synapse? I think the combination of this engine and Power BI can meet the requirements for small business analytical use cases when data is not that large and should not drain the budget. Synapse is not a single engine (right now there are 4 of them - besides dedicated SQL pools also: serverless SQL pools, Spark and Data Explorer - all addressing different use cases and having different pricing models).
Amazing video guys!
It would be nice to know the difference between the power bi in synapse and the regular one. what can I do in synapse and what I can't. and of course why would I use the one in synapse...
Great videos, as always. I have a couple of question regarding lake house. 1/ Our IT department is moving the analytical data to the Azure cloud. Upon this move they want to add a non-azure product for data virtualization. Does this make sense? What are the azure possibilities here? 2/ what about erroneous data in the data lake, how the identify the erroneous data, correct it and restart the process without the erroneous data 3/ what about showing data in reports containing enough data in order to avoid possibility to identify the source
I’d love to know about partitions on the large data tables.
let's start to do videos about Synapse!
1) Major drawbacks of Spark Pool in Synapse vs Data Bricks and vice versa? Which is better? When we are doing POC initially to evaluate Spark pool in Synapse, we have observed lot of drawbacks when compared to Data bricks. I would love hear from you with all the recent updates, can Spark pool perform much beyond than Data bricks? If yes on what basis?
2) Why not Azure Purview in Synpase? We can do governance directly there itself instead of a separate service? Any reason for making it separate?
3) One major drawback I observed is, sql pool can't support three part naming and we need to create separate schemas in the same sql pool to differentiate instead of creating different sql pool which is a costly operation. But it all together doesn't correlate with the traditional EDW modeling we do like we keep separate staging, logging, edw databases etc.
4) Why can't we delete duplicates direcly querying CTE pointing to Original table if the sharding pattern is other than Hash? We have alternatives to perform this task. But want to know the reason for why can't we directly delete using CTE?
5) Any concept of queue while submitting Spark jobs in Spark Pool? How much time will it take to spin up the cluster and run the jobs that are in queue? Comparison with data bricks make us learn better.
Please help us in letting know the above queries. Will post further questions once they arise.
Great video! Want to know more about Synapse!!
I want to know the difference between dedicate pool and serverless in Azure Synapse Analytics.
Stay tuned 👊
Looking forward to this series. Hope using Databricks vs Synapse is also covered in a way.
I come from the tradicional DW / BI and i'd love a session that would show the "state of the art", this is, ok, i have a DW on synapse, i need to pick these log times from machines to do some analysis, and some Excel files to do another job, and so kind of seeing, by example, all (or most) parts of synapse working and connecting together...
I like to see a use case of predictive analytics that gives me insights on how to get started on this tool and bring the product to finish line... from raw data to end result in power bi
Hellooooo... Love your videos... Always so filled with the right kind of Energy. You mentioned data orchestration and learning the clicks... What does that mean?...How do I start going from Power BI analyst to using Synapse and become more of a data engineer?
Guys, you are amazing! Or Insane Amazing I should say! Great intro and presentation of ASA! Keep up the good work 👍
Love your channel!
I'd like to know more about when to use Synapse Analytics vs DataBricks
You guys are amazing !! Crisp information.
I know why you are doing this, but I hope you keep the message very clear, PowerBI is still an independent BI tool and works with any platform not only synapse
It'd be nice if there was a more affordable tier to explore Synapse. I was just looking on the Azure pricing calculator and looks like the cheapest monthly fee is a couple thousand dollars which makes it pretty tough to get much hands on exposure.
Does you guys have some sort of commercial relationship with Microsoft and Azure? If yes you probably should make a disclaimer at the beginning of the video.
Loved the new intro and UI🔥🔥 #Bananas!
now that you've mentioned it.... Power BI with Azure Synapse, managed through DevOps and CI/CD would be a good start :P If possible
Thanks for the video, Adam your mic sounds like it's clipping a little 😅.
Thanks for the valuable insights... can we have high level sample end to end solution done using Azure synapse with its integration with Power BI
Thanks
I listened to this but I have no idea of whether or how I could use this as a normal mortal user.
I guess the problem (and oportunity) with Azure BI platform is that cost is often variable which discourage learning and experimentation from power users - something similar to Power BI Premium would be great which would lead to greater tool adoption
I did a POC attempting to connect a Power BI Dataflow to a partitioned ADLS Gen 2. Dataflow connector would not work, Then created a database and CETAS table in serverless SQL. Using Azure SQL connector works but the issue was that if you have partitions like puiYear=2022/puMonth=05/puDay=03 when you append the Gen 2 storage with another days worth of data it will never be seen in Power BI nor in SSMS. Finally if you use views instead of CETAS tables, when you append additional days to your partition all you have to do is refresh the DataFrame and DataSet and everything is properly refreshed. The appended data also shows up right away if you use SSMS to query the view. Problem is that this is NOT documented anywhere that I can find. Would really appreciate it if you could give us best practices for using partitioned Gen 2 storage with Power BI. I am looking for confirmation that this is the recommended method and that we will not have issues down the road,
Energy is awesome!
I am having 2 projects on Synapse. The first one is extracting data from Dynamics AX 2012 on-premises SQL into synapse, trasnform the data into star-schema in Synapse, which being used on Power BI (BI report maintained by IT developers).
The second one is extracting data from both Dynamics 365 Customer Engagement and Finance Operations thru the Azure Synapse Link (Data Lake connector), trasnform the data into star-schema in Synapse, then populate into Azure Analysis Service and into Power BI (maintained by >100 business users and self service thru ASA).
I believe each component has a bit overlapping featurs however they have different target audience, different technology at the back, and different level of user friendiness. In fact, i can't find an official doc to differentiate them. It would be great if you could share you point of view. Thanks.
So does it actually do what it says, analytics? Is it the same as SAS Viya where you can do exploratory and predictive analytics?
Synapse Analytics is definitely amazing. I was thinking about a good way to visualize Delta changes . B/c of all the fields and multiple versions, there's quite a bit of possibility.
I don't have Azure cloud experience can I directly jump in and learn ASA from scratch(including basics of Azure)
It will require a few clicks and an Azure Subscription. Our goal is to help you in that journey with getting started and moving beyond.
Hello Will you more video related to this topic and maybe DP-500 ?
In Azure Synapse Analytics, can we write back to the database using power apps from power bi dashboard.
How can I compare Synapse Analytics to AAS and SSAS. Can I use DAX? Does it has a visual diagram? It leverages column store? Does it has modern data sources with M code? How it performs?
I have the same doubt
I wanna know more! YES!!!!!
When working with Azure Synapse Analytics, is it still best practice to work in different environments for DEV -> TEST -> PROD? Do DBA's still have a role in helping to promote code between those different environments?
Also, if I use Azure Synapse, is there a need to use Azure Data Factory or Azure Data Lake independently, or is that all bundles into Synapse and does Synapse "replace" the independent use of those other products?
Is it possible to cover Azure Machine Learning please?
Hi Guys, I manage a IT Data systems and currently we use the traditional methods of ETL/ELT(ADF) and then use Azure SQL to process and store the data. How easy it is for me to move from this to AZURE synapse? Is it worth the transformation or this needs to be setup from the scratch. Thank you. VKP
CI/CD with synapse studio. Can we use synapse studio with one existing Data Factory project (Migration)
I'm a Data Analyst with knowledge in SQL, Python and Excel. I want to get started with Azure Analytics. So, Azure Synapse would be enough for me or do I need something alongside that?
Could you use Synapse for a asset management database in the Microsoft environment or is there something else?
whats your vision on large datamodels that do not fit in power bi? direct query to synapse sometimes sends strange sql queries that don't perform either on large datasets +1 billion rows
How does this differ from Azure Data Factory and can you have ingress or egress with it?
would you outline a use case to migrate from DB2 to Synapse vs MI
So Synapse integrates ADL, ADF, SQL WH/Spark & Power BI service. Can we use other data storage services like MySQL Managed instances and PostgreSQL?
Yup....You can use one of the Cosmos DB API's of MySQL, Postgres
Hi. What I have read is that Azure Synapse Analytics (the MPP bit) has big advantages once the data reaches ~ 1tb in size. Else it is not cost efficient. Is this true ?
Hi. On the SAAS platform like the Azure SQL MI / synapse, the debug mode is not available. this is making it very difficult to debug stored procedures which sometimes can get very complicated. We have been told by the MS Support team that the components required for debug are not available in the SAAS platform. I had to set up a VM, install SQL Developer Edition and do my work. Once the SP was ok, I deployed it to Azure SQL MI. Does the ASA overcome this limitation ?
Why costs are so high?
Keen to learn more 😀
more use cases on Azure Synapse Real-time Analytics
Hi guys thanks for your videos, you are amazing... About Synapse, where can I get a free way for testing, with the a Azure account, you pay for Data Factory for example. Best wishes 😃
Is Azure Synapse Analytics secure for GCC? My agency has been waiting for a while. 😉
How do you migrate from an on prem sql data warehouse to Synapse?
is azure synapse as good (comfortable to use) as sql server query analyser/management studio ? or not yet ?
No.
can somebody recommend me a resource to learn python oriented to ETL/ELT data manipulation in notebooks? (synapse studio)
I like the term "power house" . I would love to see a small usecase, so I can grasp the usefulness for my data
How it compares to Snowflake?
snowflake is better .. move cost effective.
why "yo..."?
Why not?
@@GuyInACube it is lame.
I think it is strawberries…
I do not recommend Azure synapse. When MS renames a product there is a reason why!
Enough of all this talking , you know what we like you to do … so why not doing it this time ? Huh 🤔
It's so frustrating to be subscribed to this channel. I only know some Power BI... and like what the fuck is spark? I really would like to understand the things you guys talk about on most of your videos but is way over my credentials.
Hey Ivan. I was in the same boat as you a few years back. Guy In a Cube is great because they have LOTS of content for beginners (look at their "Getting Started with Power BI" playlist). But I promise if you keep watching the new content and do some google searching on the side as you go, you will eventually see how powerful each one of their videos is. From one PBI guy to another