Advancing Spark - Lakehouse Observability with Unity Catalog System Tables

When Databricks SQL first came along, the governance nerds amongst us were very excited that we had a proper query history, but conversations with InfoSec still got very awkward when it came to table access, cluster costs, user behaviour and pretty much anything else we'd expect from modern platform telemetry. With Unity Catalog System Tables, we can now report on a huge amount of detail, as well as baking in various workflow triggers through Alerts!
In this video Simon walks through the setup of System Tables using dbdemos.ai, looks at the various artifacts created and shows some of the interesting insights we can now access!
For the official System Tables docs, take a look at: docs.databricks.com/administr...
For the demos - see: www.databricks.com/resources/...
If you're looking at overhauling your lake to give it the Lakehouse treatment, get in touch with Advancing Analytics

[@cav7]

Awesome vid. Can't wait to play around with this at work :)

[@krishnakoirala2088]

Awesome feature!!

[@chilltake]

Awesome stuff!

[@darkerdelirium]

Superb stuff :D

[@fb-gu2er]

This is great

[@MrMikereeve89]

Looks great! I presume this completely supersedes Overwatch? If so it’s yet another reason to switch to UC :)

[@MrMikereeve89]

Do you know how permissions will work on the system tables? Will it be up at the account admin level? It would be nice if it were at the workspace admin level so each workspace owner can see what is going on in their world. Or even further still each level of admin/ownership can see the stats on their objects.