Видео

radical

Nice! Any chance you could demo working with R notebooks on VS code in Databricks?

I'd love to see a video from yourself on how you automate schema deployments in databricks if possible! We have a process just now but I would say it is compute heavy as it checks the schema vs our ddl each time the pipeline runs and applies changes if a difference is found, which is maybe once a quarter, so quite a bit of wasted compute.

I learnt that databricks open sourced Unity but now I know what that means. Great stuff. Thanks for this.

Hi Dustin, thank you for the amazing explanation. I have few questions. 1. What is the industry practise - to have one job per repo or multiple jobs per repo? 2. In case I have around 5 to 6 jobs, is there a way I can have each job in a separate yaml file, have the cluster information in a common file, say commons/cluster-info and refer them in individual file just like the YAML anchor but across the files?

This is helpful Dustin, thank you. I have a question for you. This is kinda of related to your problem as I'm trying to figure out testing as well as the branching + deployment strategy (e.g. CI vs abbreviated Trunk based dev + PR + CI for automated testing) but I have perhaps 3 different types of "work" that are related but different possibly in the same repo. So setting up CI/CD pipelines and testing is downstream of the repo structure or well, if multiple repos. So that said, I'm trying to decide repo structure and I'm a bit torn on if there should be multiple repositories or separate repositories. I'm working as an embedded end-to-end ML/AI/Data engineer on a product team and we have 3 primary "Functions of work" as I like to call it: 1. Automated ETL Pipelines (e.g. Databricks workflows) 2. LLM Tools, Utilities, etc -- we use or plan to use a lot of LLMs 3. POCs for businesses using GenAI/ML/etc that may NEVER have an updated data set needed, so no needed DBX Workflow I don't think these should all be in the same repo. The main question is should the #3, which likely also does not have Unit Tests, Integration Tests should live in the same repo as #1 (DBX Workflows), perhaps in a separate top level directory called "Sandbox" or if that should be it's own Github repository. And if/when any one of these POCs gets the go ahead for regular data ingestion and/or "PROD usage" perhaps it is then ported over to "Workflows" (either the directory, if it lives in the same repo, or the Workflows Repo, if it's a separate repo). I feel this video applies very well to #1, perhaps to #2. And I should then keep #3 to itself. I have another question about CI/CD as it pertains to doing both a Medallion Architecture + DEV/TEST/PROD environments. There seems to be a ton of redundancy if I am using a CI pipeline for testing of my code in DEV on a Pull Request to then merge the code to MAIN in the DEV environment. At this point, the code is "tested" but environment wise, still lives in DEV and has not yet been pushed up through UAT or PROD.

Oh man, you have just saved me hours! I didn't know that I can copy yaml configure by switching UI to code version. Thanks a lot Dustin. 🙏🙏🙏🙏

is there a way to avoid authentication via PAT? when I use "databricks auth login --host x " and then I log in with my account, if I try to run a databricks command I get some permission errors, what could this mean? Could you make a video about Databricks Authentication?

Thank you for the awesome videos, those are very helpful. When i do bundle run it runs the workflow and after the run is done it deletes the workflow completely and I get message saying my workflow is deleted. I tried it removing the bundle run command it still deletes the workflow and there is no workflow, what is causing this ?

question, the notebooks are not defined in a yaml-fashion? if not, how are they considered to be deployed in the target workspace? just by define them in the jobs are they deploy?

Thank you for reply , the file is in the root , I will take the look to your suggestion . Thank you again and blessings !

Excellent. Thank you so much Dustin.

Thank you very much ! I have an issue in the step Validate bundle for dev enviroment , I am getting the error message Error: unable to locate bundle root: databricks.yml not found

Like deployed. thanks for sharing

Superb video, very detailed Thanks @dustin.... One Ques:: Do we need to create job in each databricks environment (QA,Prod) after the deployment or is it automatic created while deployment from dev to different environment

Couldn't we just use complex variables instead of using the YAML anchors?

Excelent material

At 2:35 you mention getting into some new databricks features like 'jump to code/definition' in a different video. Could you add a link to that video? The option to see where code is defined, especially 'intellisense-like-behaviour' is something I miss a lot, most of all when using the magic %run command to import functions from different notebooks.

Really good stuff! Thank you so much for these posts! It is very inspiring and I have some work to do to reach this level. We are very heavy on using SQL-code for transformations, using temporary views and cte:s. Is that a bad strategy in the sense that it makes it really hard to test? So for example instead of having a CASE-statement you would instead use a UDF that is more easily testable? How do you test big SQL-transformations?

Thanks for the video; I had been looking into how to set this up for a while and this video finally got me to a place of having a working process. I just about have all of this setup in my environment but the most recent issue I'm running into is if I develop locally and run a DLT pipeline from VSCode it sets everything up with my credentials. When I merge to my dev branch that triggers the CICD pipeline (running as the service principal), the step that runs the job throws an error that the tables defined in the DLT pipeline are managed by another pipeline (the one with my credentials). If I use DLT do I just never test from vscode, or do I need to go clean those up each time? Is there a better way to manage this?

Great content.

@DustinVannoy Will this work with a Databricks Personal Access Token instead of Service Connection/Service Principle?

Hi! Thanks for the content! Can you describe more in detail how you run automated tests. What do you test? Etc... Could be a topic for a future video? Real examples. Thanks again

Thanks, Dustin! You help me a lot for a production pipeline!

Still confused on how did I exact populate your Bundle Resources explorer ?

Thank you! I have a few processes that are not related to each other. Do I need to create a separate DAB for each one? How can I make the process more dynamic?

Great video!

Is there a way to define Policies as a resource and deploy . I have some 15 to 20 policies which my jobs can use any of them. If there is a way to manage these policies to apply policy changes, it will be very convenient

Is it possible to use DAB with a Standard Databricks cluster?

I tried to test this extension and Databricks Connect but when I run file *.py file with Databricks Connect spark session variable is not initialized. I got an error: pyspark.errors.exceptions.base.PySparkRuntimeError: [CONNECT_URL_NOT_SET] Cannot create a Spark Connect session because the Spark Connect remote URL has not been set. Please define the remote URL by setting either the 'spark.remote' option or the 'SPARK_REMOTE' environment variable. I didn't configure SPARK_REMOTE but I added explicit session creation: config = Config( host = dbs_host, token = access_token, cluster_id = cluster_id, ) spark = DatabricksSession.builder.sdkConfig(config).getOrCreate() I use Profile Auth. Type. Databricks Connect is enabled. Upload and run file works Databricks runtime is 15.4.x Databricks Connect 15.4.3

Great video, learned a lot! I do have a question; would it make sense to define a base environment for serverless notebooks and jobs, and in the bundle reference said default environment? Ideally it would be in one spot, so upgrading the package versions would be simple and easy to test. This way developers could be sure that any package they get used to, is available across the whole bundle.

Hi Dustin, nice video! Any plants to do the same but for Microsoft Fabric?

just great!!! I still using only the workspace UI and setups, but I really want to start using this VSCode extension.. I'm going to test some features and do some basic commands.. but anyway, just great! thank you very much

Custom Logs cannot be written via Diagnostic Settings? Reason for using spark-monitoring?

is the spill memory bad? @3:48

Thank you for the video 👌. I have question, how can I convert existing Databricks jobs(ML, python, sql etc) that were initiated manually in the past to start running through the pipeline? That is our Data engineering team were running these jobs manually but now we want to use DABs and Azure DevOps to run the jobs automatically. Thank you

Great video!! Much appreciate the effort put in to add join section, Dustin.

Hey Dustin, if we want to add more ressources like adls bronze silver and gold storage do we need to add it to the envi variables?

Very helpful. Thanks for sharing!!

my cluster is taking so much time to deploy , any ideas ?

Now do AWS 😂

Any way to see a plan like you would with terraform?

Still needs work. Issues I found so far: 1. The authentication can still clobber your CLI auth causing lots of confusion. 2. The file sync needs a full refresh option. Only way to currently do so is to delete the sync folder in the . databricks folder. 3. Sync needs to be 2 ways. Databricks/Spark connect is still not feature complete so you unfortunately have to use the notebook in some cases. 4. Overwrite job cluster feature installs your python whl onto your all purpose cluster but if you make any changes to the package l, it doesn't remove the old whl and update it with a new whl with your changes causing confusing errors.

We need PyCharm and DataGrip support!

Hey Dustin, We're currently implementing data bricks asset bundles using Azure DevOps to deploy workflows. The bundles we are using storing it in the GitHub. Can you please help me with the YAML script for it.

How to remove [dev my_user_name]. Please suggest

Hi Dustin, A basic question, how this method is different than configuring Azure portal -> databricks workspace home page -> Diagnostic Settings -> Exporting logs to Log Analytics

Thanks Dustin for the video. Is there a way where I can specify sub-set of resources (workflows, DLT pieplines) to run in specific env? For example, I would like to deploy only Unit test job in DEV and not in PROD env.

Hi Dustin, have you tried to configure and deploy a single node cluster by using Databricks Bundle?

Hi Dustin, is there any way to monitor compute event log from log analytics?