Видео

What is your favorite learning or feature for Fabric in 2024? Would love to hear!

Good to see you on YT and Happy Holidays.

Great video. Love to learn more about the Fabric feature of Multi-Cloud Integration; how this works with Azure, Microsoft 365, and supports data from AWS and Google Cloud, reducing vendor lock-in.

It would be great if you covered actual prod readiness of Fabric. There are still many features that are not ready and have bugs. Unfortunately, Fabric is currently lagging behind Snowflake and Databricks.

Let us know what you want to hear!

Viewer role allows user to access dataflow via power bi desktop.

Ted is awesome.

Let me know what you are most looking forward with on AI or CoPilots?

Great video. Thank you.

Thank you Kasper! Have watching Ted's stuff pretty hard-core for the last few months. Love how he's making me feel like a true dev!

Thanks for the session 🙏

@Kasper Ted on LinkedIn is wrong.

00:00:00 Bogdan: VP Lakehouse og Warehouse 00:00:30 Previously: MDX, Data mining add-ins, Vertipaq engine 00:01:13 Responsible for Synapse anlytics organisation; Synapse DWH, Synapse Engineering and Synapse Data science mv 00:01:57 "MS fra 1999, C++. 2001 SSAS department. MDX/DMX Data mining extensions = ML algoritms 2006 Project sandbox v Amir, Christian: Selfservice tool beginning. Power pivot. 2010 Left MS for ML i startups 2018 Arun Ulag, Christian, mfl: new Power BI version 2021: Synapse rolle - big data analytics" 00:04:00 Big hospital didnt want to use Data mining addin - allthough it predicted spot on 00:05:30 Improve core analytics by adding advanced analytichs technologis, like BI, AI og ML 00:06:17 "Big data definition has changes enormous. 2005: a Yahoo 1 terrabyte DMX database was the largest in the world, difficult to maintain. Today everbody expects 1 terrabyte to be processed. Pivot tables was the first element of advanced analytics in Excel, created from AS technology. Power pivot, bigger and bigger data volumes. More and more organically growing data solutions at departments and business units - and fewer curated cubes/data solutions. The idea was to democrotize data through Excel only. so biggest change last years: Everybody expects big data working for you, and with all kinds of tools" 00:09:23 "Fabric: Should have been ready a few years ago Synapse version was that Spark and DWH world didn't want to communicate. Individual technologies were developed without integration" 00:10:41 Some assumptions have changed from Synapse start in 2018 to 2021 00:11:01 "Openminded discussions. Amir, CAT team with Chris Webb. CW showed that customers retrieve data with Spark and 'serve' them with DWH" 00:11:57 Trident: core tech = Synapse, Data integration, Power BI 00:12:29 Focus: Data and customer centric not Engieen centric. Engieens, vectors to support data. 00:13:29 Kasper: B is responsible for the interaction layer of the lake 00:13:59 "Change and shape data. Spark/notebook = flexible and sexy DWH: relational constraints, but performance and scalability" 00:14:52 "Fabric gives max flexiblity to play with data - IN TABLES Data wants to be tables. Spark og DWH expresses data as tables" 00:15:50 Design tables: Some prefer Scala/Python, others SQL for transformating lake data to tables 00:16:01 "And tables can be loaded from OLTP systems, Operational databases Lakehouse and Warehouse can both store tables, but the way you shape are very different and they are being used by diff. Personas/tools" 00:16:42 But both Lakehouse and Warehouse stores in Delta parque now - other formats may be supported later 00:17:09 Lakehouse/Java job/notebook - SQL/SSMS, Azure data studie, new Fabric interface 00:17:35 Spark og SQL can read from each other, eg T-SQL can read Spark produced tables 00:18:20 SQL cant run ML, Spark can 00:19:10 "Different flavors of SQL: OLTP hyperscale: crazy performance. Find needle i haystack DWH SQL: Scan - how many needles Spark SQL: Also a good scanner, and at the same time integrating with Python dataframe mechanism in Spark OLTP SQL: good for transactions, operational databases" 00:20:56 "Warehouse/ Lakehouse: to what? B: technologies can be mixed Shortcut: Tables appear locale, so Spark, PBI etc can work with the tables" 00:21:54 If the data isn't in a table, like json/csv, the table is created, but aggregated tables are desired. This can be done in Spark (Lakehouse technology) or in DWH. Lakehouse and warehouse can see each other's data. Multi-table transactions in DWH 00:23:17 "Difference in relation to Azure. B: many differences Azure Synapse: customers only think of dedicated pools MS thinks: Collection of big data analytics technologies, 3 diff technologies Synapse spark, Dedicated pools, Synapse SQL serverless that operates on top of data in the lake. In Fabric, complexities are reduced; Dedicated and Serverless into one SQL engine" 00:24:58 Dedicated pool is a locked format, proprietary. Now stored in Delta parquet, Delta variant of parquet was chosen due to strong collaboration with Databricks. All Fabric engines can talk with native parquet 00:25:48 Dedicated pool requires a lot of tuning and trying compute nodes is difficult, and there is a hard limit. For instance, 10k users using the same Power BI report in direct query 00:26:49 Crucial: Storage and Compute are separated. Optional Compute can be added 00:27:36 What about migrating from Azure Synapse to Fabric? B: It's coming and is necessary due to the transition to parquet. 00:28:54 Azure Synapse will continue to run for many years. 00:29:24 Currently, experiences are being gathered about the needs for the new technology before establishing a migration technique. 00:29:58 Telemetry: many use it. Spark/event streaming to lake. Synapse link also initially stores in the lake. Meaning, a Lake-first approach. 00:31:17 Other customers don't have a lake; they fetch data directly in DWH. 00:31:40 Lake-first customers can get started immediately. Create a shortcut to the existing lake in 5 minutes. SQL will be slightly different, so existing stored procedures must be adjusted, and MS will help. 00:33:18 Migration, whether lake-first customers or not, Fabric will only go GA when migration techniques are in place. 00:33:41 K: new mindset, Fabric will be released continuously, like PBI. 00:35:26 Costs and sources of error are minimized by not having to store data in different ways, based on the end-user. Either PBI or ML. 00:36:57 SQL has two ways to store data. Tables row-wise (good for OLTP) or columns (good for AS). AS is only interested in a few columns. SQL has a CCI technology, columnar index, build on top of Vertipaq. In currenct dedicated warehouse, CCI makes analytics fast through efficient queries. 00:38:55 Parquet: a dictionary, columnar format with run-length encoding compression. A lot like Vertipaq, which was build 5 years before parquet. Vector of integers. Stores integers instead of string. Same conecpt as Vertipaq and CCI 00:40:02 The biggest pice of magic in Fabric regarding DWH storage: Transcoding, transforms parquet to to known high performance MS CCI memory structure, that allows SQL engines to run fast (a word here was impossible for me to hear :-)). Game changing move: from proprietary format to open parquet format. 00:41:09 Direct lake (own by Power BI team, not BCs team): A transcoding similar to CCI transcoding happens when parquet is transcoded to Vertipaq memory structure, which allows for fast Vertipaq scans. The core technology for transcoding is jointly developed by BC's team and the Power BI team. 00:42:07 K: How is this viewed in relation to import mode? 00:42:07 "Import: Vertipaq prepared data is served in ADLS Gen2. On recieving a DAX query, Power BI engine loads data to Vertipaq in memory structure and executes the queries. With import, Power BI does NOT transcode data - data is preencoded with Spark/DWH and stored as parquet. With Direct lake, transcoding changes parquet files to Vertipaq in memory structure, ready for queries with import-like speed. Difference: Import: PBI is in control of the data status, nothing can change the Vertipaq lake storage Direct lake: Data integration, Spark, SQL, etc. can change data status. In Delta format, there is a manifest describing file structure, which changes if data changes. Power BI must keep an eye on the lake, on the parquet manifest. Refresh must be 'agreed upon' with the user..." 00:45:07 "K: Direct lake is not Direct query BC: Be awary, that if SQL RLS/CLS is introduced, the SQL exclusive security mode - not onelake security, then PBI cannot respect RLS, and then it automatically switches to direct query, who respects the security contract with SQL secutity mode. Reason RLS/CLS use syntax, that cannot be share bw multiple engines. K: New podcast will come, with focus on direct lake" 00:47:19 Squirrel/Wardrobe analogy. SQL serverless, arbitrary number of nodes. Registers tables adhoc, but not fast, scurries around but not strong = squirrel. Dedicated: strong, but cumbersome to adjust 00:49:16 Fabric. Compute node must be good, good query plan, and good processing. 2 Distribution must be good 00:49:49 Query optimization: MS SQL has the best in the industry 00:50:10 "Unified query optimization approach Rowgroups have metadata statistics. Whether they are formed during ingestion or in a Spark job. Massive parallel processing" 00:50:53 Also inherited from serverless: Polaris engines, cloud-level resources, reassign task, scale up/down. 1 petabyte 00:52:46 Fabric: the best of both dedicated and serverless 00:54:09 Agile development: a big change for the Synapse groups, but there's a need for customer feedback. Complaints about the preview are good; it helps with focusing. 00:55:31 One security (universal security), operates at the storage level. Game changer. A big relief for users. 00:56:32 "The biggest challenge with Fabric: Adopting Parquet. Leaving proprietary format foundation for optimal SQL performance. An incredible number of people, products, and departments - much larger than Power BI. Shocked that it worked - the largest project he has worked on.""" 00:58:21 K: and this is just the beginning….. 00:58:33 B: Software as a service + project development have been brought together. 00:59:29 K: When he started, there were Box releases every 4 years. After the release, there was calm for 3 months. B has a box from 2005! 01:00:31 Best feature: Integration between Lakehouse and Warehouse - different engines can be used. 01:01:10 K: Different types of developers are now talking together, SQL and Spark worlds. This opens up a lot of new possibilities, SQL, Spark SQL. 01:02:15 Power BI: nothing changes if you don't want it to. BUT there are new opportunities, Notebooks, Visual Studio Code with Github Copilot. 01:02:52 B: Copilot will be everywhere in Fabric.

This was super informative. I will tune in to Jeffery Wang reading a phone book--he's that good!

This was an incredibly informative interview.

Really cant wait for SEMPAI (hope I'm spelling it right 😅). So far we've been using Azure Functions to query our models outside Power BI. We would then need to export results in files and then work with them This however is a game changer!

Justyna is one of my all time faves!....such an ambitious undertaking! #lfg

Interesting to think of the experiences collaborating together with delta in common. An amazing SaaS product for sure.

Interesting insights! Thanx Kasper and Bogdan.

That was great!!!

a live demo would be helpful especially about the more complex topics br

Is there something to use aggregations with distinct count in datasets with 300 millions rows or more?

2 of my faves! ..... such breakthrough technology!

What is the DevOps story in Fabric? Is it all seamlessly integrated or not yet?

Just a question, did the powerbi ms development os now the same team That is developing fabric?, I ask because I saw very familiar faces on ms build...

Waanzinnig. De techniek staat voor niks...

my question is this, do we still have to learn DAX and Excel FUNCTIONS ? I just purchase a Power BI course of $349, now I have fear that it might not be useful as reckoned. What is my faith as a data analyst?

Wonder what your take is on if Databricks still has a place in all this? Is it worth investing in setting up a new dataplatform with Databricks involved?

What in Fabric are you most excited about?

Hi Kasper, it was a nice video, thanks! If I understood well, you said that "for data scientists is better to connect to data lake directly compared to connect to a data warehouse". Why 😮?

Exciting! Bundling of services is the way moving forward! What's the pricing of SaaS?

Thanks Kasper, for this good overview!

Great introduction and summary of today's exciting news regarding Microsoft Fabric - well done!

Very Exciting! I was wondering what the big secret was. Looking forward to diving in!

Exciting! The journey for those of us starting with Excel leading to Power BI and now Fabric has been breathtaking! Looking forward to your rip with Josh! Sounds like PBI skills built are still relevant in a Fabric world!

great that you are back with podcast!

glad to see Kasper putting out content again!

PBI no demo? Hmm

Minute 5:30 - "Excel is transparent..." wow, thanks Bill !

hey, I really like the podcast and the questions that you asked. Can you add with next episodes under parts of video what is discussion about? like 1-3 min background , 4-10 min visualization types etc. It would be helpful :), cheers!

What is your favorite data visualization trick?

Would never think that a non-technical interview with Alberto would be so interesting. Awesome insights, especially about learning DAX and debugging measures. Increased my respect by 10 factors!

Thanks, but a demo would be nice.

Are you using PurView in combination with Power BI?

I’m glad to hear Jeffery himself share the challenges and opportunities of getting through the learning curve of DAX. Very relatable for many people :)

I stopped using dax and power bi on the first year when it was launch years ago. Because Microsoft dont allow personal email to create an account, they require a Company email. Thanks that now we have python which is free and of course more powerfull than DAX and Power BI. Yes it's more complicated because it's a true programming language, but learning python is worth it.

55:23 It Depends. The most common answer from a true data professional :)

What is your favorite memory of SQLBits and why do you like the data community? Let me know!

The best video about the Dataflows under the hood by far.

Multiple audiences for Apps is definitely up there. I had never been a huge fan of apps because of the restriction, but since the change I started to roll them out.