This is so cool! I can see this being a quick but powerful win for small businesses that want to harness their data without having a ton of overhead in getting services up & running!
Wow! This was the best explainer so far! I'm interested in tracking when other external sources will be available via shortcuts because my organization keeps data warehoused in other (non-Amazon) sources.
What's the performance impact of having data as shortcut compared to having these in one lake when we use it in different engines especially direct query on power bi
At a guess, copying the data into one lake is better for data manipulation. Transformations, and selects etc. Referencing the tables as a shortcut is akin to using a server less SQL instance with external tables. Physical data moving, indexing, and keys may not be possible. I'm guessing here though based on limited knowledge currently.
There's an on prem SQL Server option to copy data into a lakehouse in fabric. I'm not sure how that relates to one lake as I've just started this video and still getting used to Fabric terminology
Does anyone know if DirectLake is as performant as the import mode? As far as I know, each workspace is located in one region: do we pay cross regions transfer costs when using a shortcut between two workspaces located in 2 different regions?
Is my understanding correct that structured data is stored in Onelake in .Parquet files? If so how efficient is it as compared to storing data in RDBMS like SQL server?
It looks fantastic. I have a question. I use to store my data in Azure SQL. Is it better to migrate my data from Azure SQL to OneLake or it's better to stay like it is now.
Amazing vid. Very informative. Question: Is it possible to have a deeper dive into the inner workings? All the Marketing and Sales sessions/materials/etc. repeat the line (or something similar) that: "it's one OneLake and you can have one Fabric on top". Josh mentioned in this video that whoever owns (uploaded) the data, say to your store DB, continues to own it and the OneLake consumer gets a copy. I'm curious on how this is actually working. The last thing I want is our data and analytics team "connecting" to our application data stores and running crazy queries on our operational data. Given Josh said it's copied data, I'm curious to know more and what that means for debugging missing data, eventual consistency, atomicity, etc. Thanks!
Do you see anything on the horizon to allow an ISV to provision a onelake, setup domains and grant external guest users (the ISV's customers) access to different paritioned datasets? In otherwords is there anyway for an ISV to grant lakehouse to their SaaS customers?
All files are converted to parquet and then delta parquet. So basically you have to build an ELT framework for handling this... Really anoying it isn't done automatically and then generate scripts.. Like 10% extra developer time comñared to a storage blob into sql database - Sigh 😢
Hey Adam, does using one lake and direct lake storage mode in power bi semantic model allow to connect model to really big tables (hundreds of millions of records) while keeping visuals and dax fast and efficient? Thx in advance!
I like the shortcut table feature! Is there also a way to create shortcut tables to REST APIs or OData web APIs so that I can get the latest data from source systems like Sales Force or Business Central without copying the data on a schedule into OneLake where it will eventually get outdated or is missing columns I need at all?
When clients (say power bi) consuming a shortcut data source, data is transferred to one lake first, then forwarded to clients, right? If it's true, it sounds like copy the data eveytime consumes.
Thank you so much for the demo!! I have a question regarding a potential use case for this. I have a client that has multiple data locations, they're still moving into the cloud scene however they currently utilise an Access Db for a particular set of data. If this db was hosted on a vm within azure, could OneLake potentially reference data from Access Db?
If we use a different source other than S3 like ADLS for example can we still load the datasets directly from those external sources within our Power bI REPORT.
Slightly out of sync "Yo" at the beginning of the video. Literally unwatchable. p.s. brilliant content, thank you so much for providing so much insight and material into such a new product
not convinced about this yet, shortcuts are nice in theory, but performance will never be as good as if the data was imported.. secondly, look at the video at 9 min and 25-30 secs , the insert took about 5 seconds. i know we are nerds and we like new shiney things, but lets balance that with being practical.
Datalake and delta lake sucks imo. It's not drag and drop really. If you load incremental changes from a source like SAP or Oracle or API. You have to build an integrator in Data Factory, SQL scripts in Synapse to convert to Parquet files, do Transformations, Implement a Delta logic and Save in Delta Parquet, Then transform the data again, Build views, and then you can consume the data. I much prefer Data storage gen 2 or Storage Blobs and a regular SQL Database and do the transformation in Synapse.. Now you are reliant on Data Factory+Synapse+Parquet logic+OneLake+Synapse+Synapse or SQL serverless And It is super slow to read Parquet since It's basically thousands of KB size files to merge like heap pages in an SQL database.
This is so cool! I can see this being a quick but powerful win for small businesses that want to harness their data without having a ton of overhead in getting services up & running!
Wow! This was the best explainer so far! I'm interested in tracking when other external sources will be available via shortcuts because my organization keeps data warehoused in other (non-Amazon) sources.
Adam and Josh - thank you for the demo! As always...awesomeness has been extracted.
It has not been extracted, It was just linked. :D
What's the performance impact of having data as shortcut compared to having these in one lake when we use it in different engines especially direct query on power bi
Following.
I have the same question. Have you find a response?
Same question.
At a guess, copying the data into one lake is better for data manipulation. Transformations, and selects etc.
Referencing the tables as a shortcut is akin to using a server less SQL instance with external tables. Physical data moving, indexing, and keys may not be possible.
I'm guessing here though based on limited knowledge currently.
Hi Adam, it would be great if you can do an end to end demo on how to use Fabric based on a company scenario. 😉
We are working on putting some things together. stay tuned.
Can a shortcut be established to an on premise SQL server? Or, is it possible to sync OneLake with (on-prem) SQL server tables?
Looking forward to the response to this question.
There's an on prem SQL Server option to copy data into a lakehouse in fabric.
I'm not sure how that relates to one lake as I've just started this video and still getting used to Fabric terminology
I guess using a data gateway like you would from Power BI service.
Does anyone know if DirectLake is as performant as the import mode?
As far as I know, each workspace is located in one region: do we pay cross regions transfer costs when using a shortcut between two workspaces located in 2 different regions?
Can a on-prem SQL server be set up as a shortcut by using a remote gateway? This would ensure that our data at rest would still be on-prem.
Is my understanding correct that structured data is stored in Onelake in .Parquet files? If so how efficient is it as compared to storing data in RDBMS like SQL server?
Thanks a lot Adam and Josh !!!
It looks fantastic. I have a question. I use to store my data in Azure SQL. Is it better to migrate my data from Azure SQL to OneLake or it's better to stay like it is now.
Great stuff, thanks for sharing. Any chance to have a shortcut for Snowflake coming soon?
One lake to rule them all!
Amazing vid. Very informative.
Question:
Is it possible to have a deeper dive into the inner workings? All the Marketing and Sales sessions/materials/etc. repeat the line (or something similar) that: "it's one OneLake and you can have one Fabric on top". Josh mentioned in this video that whoever owns (uploaded) the data, say to your store DB, continues to own it and the OneLake consumer gets a copy.
I'm curious on how this is actually working. The last thing I want is our data and analytics team "connecting" to our application data stores and running crazy queries on our operational data.
Given Josh said it's copied data, I'm curious to know more and what that means for debugging missing data, eventual consistency, atomicity, etc.
Thanks!
What is the difference between data in onelake and data in adls gen2 datalake ?
Once you connected a data source in OneLake data hub, how do you disconnect and connect to data source not in OneLake?
Do you see anything on the horizon to allow an ISV to provision a onelake, setup domains and grant external guest users (the ISV's customers) access to different paritioned datasets? In otherwords is there anyway for an ISV to grant lakehouse to their SaaS customers?
So this is like synapse, whats the used case for One Lake rather than Synapse?
All files are converted to parquet and then delta parquet. So basically you have to build an ELT framework for handling this... Really anoying it isn't done automatically and then generate scripts.. Like 10% extra developer time comñared to a storage blob into sql database - Sigh 😢
what tool do you use to bluehighlight in your videos?
Hey Adam, does using one lake and direct lake storage mode in power bi semantic model allow to connect model to really big tables (hundreds of millions of records) while keeping visuals and dax fast and efficient? Thx in advance!
This is great
Glad you liked it! 👊
I like the shortcut table feature! Is there also a way to create shortcut tables to REST APIs or OData web APIs so that I can get the latest data from source systems like Sales Force or Business Central without copying the data on a schedule into OneLake where it will eventually get outdated or is missing columns I need at all?
Hi, I understand Premium gives you access to Fabric's features. But is OneLake priced separately?
When clients (say power bi) consuming a shortcut data source, data is transferred to one lake first, then forwarded to clients, right?
If it's true, it sounds like copy the data eveytime consumes.
Do we have an eta on the shortcuts for Dataverse? Will there be shortcuts for Business Central as well?
What if we have a DataHub, can we connect the MS Fabric One Lake to our DataHub ? Do we have to do data governance in 2 places ?
Thank you so much for the demo!! I have a question regarding a potential use case for this. I have a client that has multiple data locations, they're still moving into the cloud scene however they currently utilise an Access Db for a particular set of data. If this db was hosted on a vm within azure, could OneLake potentially reference data from Access Db?
can we create a dashboard of real-time streaming data without using network (internet)
If we use a different source other than S3 like ADLS for example can we still load the datasets directly from those external sources within our Power bI REPORT.
wait, this video showed there were GIAC socks for sale, but searching all over the store I see no socks. That's bananas.
Hi everyone! I really didn't get the idea of OneLake? Isn't it another Sharepoint?
So how much does this cost and can I get free datasets
How about cost and license structure?
what's the difference between onelake and onedrive ?
One Lake seems like Data Virtualization
Slightly out of sync "Yo" at the beginning of the video. Literally unwatchable.
p.s. brilliant content, thank you so much for providing so much insight and material into such a new product
not convinced about this yet, shortcuts are nice in theory, but performance will never be as good as if the data was imported.. secondly, look at the video at 9 min and 25-30 secs , the insert took about 5 seconds. i know we are nerds and we like new shiney things, but lets balance that with being practical.
Who died on Adam's shirt? I'm sad.
I realise now that Microsoft is just a factory for making names, azure, fabric, OneDrive, one lake. You really need to start talking sense
Nah… give Fabric a go. It’s a game -changer!
F this ran over my head. I get the dataflow but not data lake. Hmm..😢
Datalake and delta lake sucks imo.
It's not drag and drop really.
If you load incremental changes from a source like SAP or Oracle or API. You have to build an integrator in Data Factory, SQL scripts in Synapse to convert to Parquet files, do Transformations, Implement a Delta logic and Save in Delta Parquet, Then transform the data again, Build views, and then you can consume the data.
I much prefer Data storage gen 2 or Storage Blobs and a regular SQL Database and do the transformation in Synapse..
Now you are reliant on Data Factory+Synapse+Parquet logic+OneLake+Synapse+Synapse or SQL serverless
And It is super slow to read Parquet since It's basically thousands of KB size files to merge like heap pages in an SQL database.
great content, 👍bad sound quality and way to fast speaking for a dutch listner
OneLake just happened in 2023… Hmm… Better late than never anyway…