Видео

Hi Dustin, is there any way to monitor compute event log from log analytics?

Hey Dustin, We're currently exploring DABs, and it feels like this was made just for us!😅 Thanks a lot for sharing it!

Hey Dustin, Thanks for the amazing explanation! DABs are sure to be adopted by every dev team!

Thanks, one question I have some metadata files in the project folder, I need to copy this to a volume in Unity catlog. Is it possible through this deploy process ?

Wonderful tutorial, Thank you! This approach works effectively for running multiple tables in parallel when using spark.read and spark.write to a table. However, if the process involves reading with spark.read and then merging the data into a table based on a condition, one thread interferes with another, leading to thread failure. Is there any workaround for this?

Great content!! I am trying to deploy the same job into different environments DEV/QA/PRD. I want to override parameters passed to the job from variable-group defined on the Azure DevOps portal. Can you please suggest how to proceed on this?

I am learning DABs at this moment. So lucky that I found this video. Thank you, @DustinVannoy. Do you mind if I ask a couple of questions?

Now do it in Terraform ;) Great video as always!

Thanks a lot Dustin... Really appreciate it :)

Thanks for the video. It helped me a lot in my YT channel.

Great video Dustin! Especially on the advanced configuration of the databricks.yaml. I'd like to hear your opinion on the /src in the root of the folder. If you're team/organisation is used to work with a mono repo it would be great to have all common packages in the root, however, if you're more of a polyrepo kinda team/organisation, building and hosting the packages remotely (i.e. Nexus or something) could be a better approach in my opinion. Or am I missing something? How would you deal with a job where task 1 and task 2 have source code with conflicting dependencies?

Is there a way for python wheel tasks to combine the functionality we had without serverless to use: libraries: - whl../dist/*.whl so that the wheel gets deployed automatically with using serverless? As if I am trying to include environments for serverless I can't longer specify libraries for the wheel task (and therefore it is not deployed automatically) and I also need to hardcode my path for the wheel in the workspace. Could not find an example for that so far. All the best, Thomas

Hey, thanks for video! I was wondering if you know whether those audit logs are still exist even if audit log not configured (audit log/ log delivery)? I mean - are events will still be written in the back and once it is enabled (via system tables) could be consumed?

How do you enable intellisense?

how do you change the Catalog Name specific to an environment?

Thansk for the explanation! Is there any way to view exported DBC files in VS code?

Exciting stuff! Will definitely be trying to implement this in my future work!

Great video ! Is there a way to overide variables defined in the databricks.yml in each of the job yml definition so that the variable has a different value for that job only ?

Thanks a lot, @DustinVannoy for this great presentation! I have a question: which is the better approach for project structuration: one bundle yml config file for all my sub-projects or each sub-project have its own Databricks and bundle yml file? Thanks again :)

Once the code is deployed it gets uploaded in the shared folder can't we store that some where else like an artifact or storage account because there are chances that someone may deleted that bundle from shared folder. It is always like with databricks deployment before and after asset bundles.

Can we integrate Azure pipelines + DAB for ci cd implementation?

Loving bundles so far. Only issue so far I've had is the databricks vscode extension seems to be modifying my bundles yml file behind the scenes. For example when I attach to a cluster in the extension it will override my job cluster to use that attached cluster when I deploy to the dev target in development mode.

Hey, Great video! Small question, Why not just using the FAIR scheduler that doing that automatically?

Can delta sharing works with hive_metastore?

thanks so much for your help

Run driver program using multithreads using this as well. from threading import * # import threading from time import * # for demonstration we have added time module workerCount = 3 # number to control the program using threads def display(tablename): # function to read & load tables from X schema to Y Schema try: #spark.table(f'{tablename}').write.format('delta').mode('overwrite').saveAsTable(f'{tablename}'+'target') print(f'Data Copy from {tablename} -----To----- {tablename}_target is completed.') except : print("Data Copy Failed.") sleep(3) list = ['Table1','Table2','Table3','Table4','Table5', 'Table3', 'Table7', 'Table8'] # list of tables to process tablesPair = zip(list,list) # 1st list used for creating object & 2nd list used as table name & thread name counter = 0 for obj,value in tablesPair: obj = Thread(target=display, args=(value,), name=value) # creating Thread obj.start() # Starting Thread counter += 1 if counter % workerCount == 0: obj.join() # Hold untill 3rd Thread completes counter = 0

Hey Dustin, Thanks for the tutorial! I've successfully integrated the init script and have been receiving logs. However, I'm finding it challenging to identify the most useful logs and create meaningful dashboards. Could you create a video tutorial focusing on identifying the most valuable logs and demonstrating how to build dashboards from them? I think this would be incredibly helpful for myself and others navigating through the data. Looking forward to your insights!

Hello Dustin, Thank you for posting this video. This was very helpful!!! Pardon my ignorance but I have a question about initializing the Databricks bundle. The first step when you initialize the databricks bundle through CLI, does it create the required files in the databricks workspace folder. Additionally do we push the files from the databricks workspace to our git feature branch so that we can clone it to your local so that we can make the change in the configurations and push it back to git for deployment.

Hello, sir, Thank you for this tutorial. I successfully integrated with log analytics. Could you please show me what we can do with these logs and how to create dashboards? I am eagerly awaiting your response. Please guide me.

Hi Dustin, Thank you for sharing this approach I am going to use it for training spark ml models. I had a question on using daemon option. My understanding is that these threads will never terminate until a script ends. When do they in this example? Do they terminate at the end of the cell? Or after .join()? So when all items in the queue have completed. I really appreciate any explanation you provide.

Thank you for the session!

Hi thanks for the informative video! I have a question, instead of sending a list to the notebook, I send a single table to the notebook using a for each activity (synapse can do maximum 50 concurrent iterations). What would the difference be? Which would be more efficient? And what is best practice in this case? Thanks in advance!

For CPU intensive operations would this work since it uses threading? Also, can't we use multiprocessing if we want to achieve parallelism?

great content, thank you

It is creating duplicate jobs on re deployment of databricks.yml. How to avoid that?

is it possible to add approvers in asset bundle based code promotion ? Say one does not want the same dev to promote to prod, as prod could be maintained by other teams; or if the dev has to do cod promotion, it should go through an approval process. Also is it possible to add code scanning using something like sonarcube ?

Amazing content , thanks man. I'm learning a lot

Like @gardnmi, I also used the map method threadpool has. Didn't need a queue. I created a new cluster (tagged for the appropriate billing category) and set the max workers on both the cluster and threadpool: from concurrent.futures import ThreadPoolExecutor with ThreadPoolExecutor(max_workers=137) as threadpool: s3_bucket_path = 's3://mybucket/' threadpool.map(lambda table_name: create_bronze_tables(s3_bucket_path, table_name), tables_list)

Great video tutorial. Clear explanation. Thank you.

How long does it take to deploy the python wheel for you? For me it takes about 15 mins which makes me consider making wheel project separate from rest of the solution.

I was having so many issues using the other Threadpool library in a notebook, It cut my notebook runtime down by 70% but I couldn't get it to run in a databricks job. Your solution worked perfectly! Thank you so much!

Will this work if you read in a file, do some minor transformations and then save to ADLS? Would it work if we add in transformations basically?

Hi Dustin how can we send stderr logs into azure monitor

thanx alot for this video, but plz try to save the (almost dead) plant behind you :)

Hey Dustin, Thank you so much for the video, I still have one doubt, I've been running a streaming query in a notebook for over 10 hours. The streaming query statistics only show specific time intervals. How can I view input rate, process rate, and other stats for different timings or for the entire 10 hours to facilitate debugging?

I tried this. However, I noticed a issue when I have single notebook which creates multiple threads, where each thread is calling a function which creates the spark localtempviews, the views get overwritten by the second thread as it essentially is same spark session. How do I get around this?

Can we use it for AWs databricks as well?

Hi Dustin, great content, quick question, where can I find the library for Runtime v14 ?

Hi Dustin. How to implement the same using scala. I tried but the same solution is not working for me. Any advise?

How can i do this today on windows?