Видео

You may enjoy another talk of ours: endjin.com/what-we-think/talks/how-to-ensure-quality-and-avoid-inaccuracies-in-your-data-insights

Yes, we obviously very much agree!

Excellent! Yes, that's very much the point. Many of the new cloud native analytics platforms allow data folk to adopt and embrace the engineering practices that software folk have enjoyed for the past 25 years! 🎉

We're really glad you found it useful!

What aspects are you particularly interested in?

Hello. Thanks for your interest. We'll try to find time to do another demo to show how we could move the solution onto Microsoft Fabric and use OneLake. There are two other video series in our channel which step through data engineering features in Microsoft Fabric including how we make use of OneLake.

Thanks for the feedback! I worked with my colleague Paul on this project who is a visual design expert. My inclination is to throw more and more data onto the page. But Paul always challenges that. He makes sure we adhere to a user experience (UX) that is accessible, intuitive, delivers a user journey and visually pops. When we build reports like this for clients we also have a domain expert involved. So we do lots of small iterations to get the right balance between those three concerns: business goals, analytics and UX. This multi-disciplinary approach really helps to make sure you get a good product at the end.

Hi there. The Postcode data comes from: geoportal.statistics.gov.uk/datasets/a8a2d8d31db84ceea45b261bb7756771/about Ed

Part 3 is currently being worked on!

Hi there, You should be able to just change "blob" to "dfs" if your storage account is ADLS Gen2 enabled. If it's not ADLS Gen2 enabled, then sadly you can't create a shortcut to a Blob Storage account at the moment. Ed

@@endjin Thanks for your reply. I found out that my storage account is not ADLS Gen2 enabled.

Connect to both lakehouses on the left pane of the notebook. Use spark sql to select and manipulate the data from your bronzelake, then essentially do the merge similar to what is done in the video while selecting the silverlake.

Further to Ben's response, I would recommend using two-part naming (<lakehouse_name>.<table_name>) when transferring this data. However, if you're just wanting to expose the data from one lakehouse in another lakehouse, remember you can avoid copying data altogether by using Shortcuts. And if you need to do this programmatically, there's an API for this: learn.microsoft.com/en-us/rest/api/fabric/core/onelake-shortcuts/create-shortcut?tabs=HTTP#create-shortcut-one-lake-target-example Ed

Hi there, Thanks for the feedback! This video was recorded at a time when it wasn't possible to connect to ADLS Gen2 if it wasn't publicly accessible. However, now that's changed: if you'd like to understand how to access a partially restricted ADLS account, please see the documents here: learn.microsoft.com/en-us/fabric/security/security-trusted-workspace-access. If you've fully disabled public access, there's no easy way to create a Shortcut that I'm aware of, sadly. Hopefully that functionality will come. Ed

Not yet - but it will come! Thanks for your comment, Ed

Thanks for the comment! Ed

In this scenario, it would make more sense to run Spark locally. There are a few ways you can do that, but as you'd expect it's not entirely straightforward, and not something easily addressed in a comment.

Thanks for the kind words! Ed

There's a couple of ways to achieve this - neither are immediately obvious but definitely possible! There's no API for just running a notebook in Synapse, but you can submit a Spark batch job via the API. However, this requires a Python file as input, so it might mean pulling your tests out of a Notebook and writing and storing them separately in an associated ADO repo: learn.microsoft.com/en-us/rest/api/synapse/data-plane/spark-batch/create-spark-batch-job?view=rest-synapse-data-plane-2020-12-01&tabs=HTTP Possibly an easier route would be to create a separate Synapse Pipeline definition that runs your test notebook(s) and use the API to trigger that pipeline run from your ADO pipeline. This is a straightforward REST API but operates asynchronously, so you'd need to poll for completion as the pipeline/tests are running: learn.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/create-pipeline-run?view=rest-synapse-data-plane-2020-12-01&tabs=HTTP Hope that helps!

Hi - I don't believe there's a way in Synapse Studio to automatically script out the definitions like you can do in, say SQL Server Management Studio. But you can see the column definitions for you View if you find your database under the Data tab and expand the nodes in the explorer. Hope that helps!

Barry has ~9 parts planned!

Hi there, Thanks for your comment! It's a great question. Personally, unless there are strong requirements (e.g. high data sensitivity/unique security requirements) for splitting zones apart into separate workspaces, I would default to one workspace. In my experience, the same team is often involved in managing all three layers of the Lakehouse, and "end-users" mostly only get access to semantic models (or the "Gold" layer at a push), so giving every zone its own workspace isn't really justified. Deployment also becomes trickier when multiple workspaces are involved. All that being said, I can appreciate there are scenarios where multiple is more suitable. W.r.t. how to design for the future: my first comment would be "only do it if you need to". No need to change tack just for the sake of it. If a single workspace is working for you then just stick with it. But if you do need to switch for whatever reason, then sadly there'll always be a significant migration overhead. The best thing you can do is have well structured workspaces and notebooks like I've shown here. Highlight the artifacts that are relevant to each layer of the architecture so you can get a picture of the interdependencies. Deployment pipelines / REST APIs will be your friend if you have to migrate too - worth getting familiar with these if you haven't already. The beauty of the medallion architecture is that the core structure of your data pipelines stays the same, whatever workspace infrastructure architecture you opt for! Hope this helps, Ed

Thanks for the feedback. Glad to hear you are enjoying it. Yes - we are planning to release the code for this project on Git at some point soon.

Sadly you don't have that many options here without having to fall back to Python/Scala. You can modularize at a very basic level using notebooks as the "modules", containing a bunch of cells which contain Spark SQL commands. Then call these notebooks from the parent notebook. Otherwise, as you say, one step further would be defining UDFs using some Python and then using spark.udf.register to be able to invoke them from SQL. Ed

The HPA prefix stands for "House Price Analytics", although the architecture diagram on the second video has slightly old names, as you've probably noticed. The full version uses <medallion_layer>_Demo_LR, where LR stands for "Land Registry". Ed

@@endjin - Thanks for the clarification Ben.

Hi - thanks for the comment! Which Synapse Link are you using? Dataverse? If so, this uses the CDM model.json format which doesn't include header rows in the underlying CSV files. You would have to read the shortcut data, apply the schema manually, and then write the data out to another table (inside a Fabric notebook or something) if you wanted to use that existing data. However, if you're using Synapse Link for Dataverse, you should instead consider using the new "Link to Microsoft Fabric" feature available in Dataverse: learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-view-in-fabric. This will include the correct schema.