![Dustin Vannoy](/img/default-banner.jpg)
- Видео 54
- Просмотров 167 235
Dustin Vannoy
США
Добавлен 26 окт 2011
This channel is a resource for you to learn about modern data technologies and practices from a Data Engineer perspective, from kickstart tutorials to tips to go to the next level.
Dustin Vannoy is a consultant in data analytics and engineering. His specialties are modern data pipelines, data lakes, and data warehouses. He loves to share knowledge with the data engineering and science community.
Dustin Vannoy is a consultant in data analytics and engineering. His specialties are modern data pipelines, data lakes, and data warehouses. He loves to share knowledge with the data engineering and science community.
Databricks CI/CD: Azure DevOps Pipeline + DABs
Many organizations choose Azure DevOps for automated deployments on Azure. When deploying to Databricks you can take similar deploy pipeline code that you use for other projects but use it with Databricks Asset Bundles. This video shows most of the steps involved in setting this up by following along with a blog post that shares example code and steps.
* All thoughts and opinions are my own *
Blog post on DABs with Azure DevOps: medium.com/databricks-platform-sme/integrating-databricks-asset-bundles-into-a-ci-cd-pipeline-on-azure-7b181b26d9ae
Prior videos on DABs...
Intro: ruclips.net/video/uG0dTF5mmvc/видео.html
Advanced: ruclips.net/video/ZuQzIbRoFC4/видео.html
More from Dustin:
Website: dusti...
* All thoughts and opinions are my own *
Blog post on DABs with Azure DevOps: medium.com/databricks-platform-sme/integrating-databricks-asset-bundles-into-a-ci-cd-pipeline-on-azure-7b181b26d9ae
Prior videos on DABs...
Intro: ruclips.net/video/uG0dTF5mmvc/видео.html
Advanced: ruclips.net/video/ZuQzIbRoFC4/видео.html
More from Dustin:
Website: dusti...
Просмотров: 1 001
Видео
Databricks Asset Bundles: Advanced Examples
Просмотров 3,2 тыс.2 месяца назад
Databricks Asset Bundles is now GA (Generally Available). As more Databricks users start to rely on Databricks Asset Bundles (DABs) for their development and deployment workflows, let's look at some advanced patterns people have been asking for examples to help them get started. Blog post with these examples: dustinvannoy.com/2024/06/25/databricks-asset-bundles-advanced Intro post: dustinvannoy...
Introducing DBRX Open LLM - Data Engineering San Diego (May 2024)
Просмотров 2343 месяца назад
A special event presented by Data Engineering San Diego, Databricks User Group, and San Diego Software Engineers. Presentation: Introducing DBRX - Open LLM by Databricks By: Vitaliy Chiley, Head of LLM Pretraining for Mosaic at Databricks DBRX is an open-source LLM by Databricks which when recently released outperformed established open-source models on a set of standard benchmarks. Join us to ...
Monitoring Databricks with System Tables
Просмотров 2,4 тыс.6 месяцев назад
In this video I focus on a different side of monitoring: What do the Databricks system tables offer me for monitoring? How much does this overlap with the application logs and Spark metrics? Databricks System Tables are a public preview feature that can be enabled if you have Unity Catalog on your workspace. I introduce the concept in the first 3 minutes then summarize where this is most helpfu...
Databricks Monitoring with Log Analytics - Updated for DBR 11.3+
Просмотров 2,7 тыс.7 месяцев назад
In this video I show the latest way to setup and use Log Analytics for storing and querying you Databricks logs. My prior video covered the steps for earlier Databricks Runtime Versions (prior to 11.0). This video covers using the updated code for Databricks Runtime 11.3, 12.2, or 13.3. There are various options for monitoring Databricks, but since Log Analytics provides a way to easily query l...
Databricks CI/CD: Intro to Databricks Asset Bundles (DABs)
Просмотров 14 тыс.11 месяцев назад
Databricks Asset Bundles provide a way to use the command line to deploy and run a set of Databricks assets - like notebooks, Python code, Delta Live Tables pipelines, and workflows. This is useful both for running jobs that are being developed locally and for automating CI/CD processes that will deploy and test code changes. In this video I explain why Databricks Asset Bundles are a good optio...
Data + AI Summit 2023: Key Takeaways
Просмотров 611Год назад
Data AI Summit key takeaways from a Data Engineers perspective. Which features coming to Apache Spark and to Databricks are most exciting for data engineering? I cover that plus a decent amount of AI and LLM talk in this informal video. See the blog post for a bit more thought out summaries and links to many of the keynote demos related to the features I am excited about. Blog post: dustinvanno...
PySpark Kickstart - Read and Write Data with Apache Spark
Просмотров 782Год назад
Every Spark pipeline involves reading data from a data source or table and often ends with writing data. In this video we walk through some of the most common formats and cloud storage used for reading and writing with Spark. Includes some guidance on authenticating to ADLS, OneLake, S3, Google Cloud Storage, Azure SQL Database, and Snowflake. Once you have watched this tutorial, go find a free...
Spark SQL Kickstart: Your first Spark SQL application
Просмотров 800Год назад
Get hands on with Spark SQL to build your first data pipeline. In this video I walk you through how to read, transform, and write the NYC Taxi dataset which can be found on Databricks, Azure Synapse, or downloaded from the web to wherever you run Apache Spark. Once you have watched and followed along with this tutorial, go find a free dataset and try to write your own Spark application. * All t...
PySpark Kickstart - Your first Apache Spark data pipeline
Просмотров 3,6 тыс.Год назад
Get hands on with Python and PySpark to build your first data pipeline. In this video I walk you through how to read, transform, and write the NYC Taxi dataset which can be found on Databricks, Azure Synapse, or downloaded from the web to wherever you run Apache Spark. Once you have watched and followed along with this tutorial, go find a free dataset and try to write your own PySpark applicati...
Spark Environment - Azure Databricks Trial
Просмотров 380Год назад
In this video I cover how to setup a free Azure Trial and spin up a free Azure Databricks Trial. This is a great way to have an option for testing out Databricks and learning Apache Spark on Azure. Once setup you will see how to run a very simple test notebook. * All thoughts and opinions are my own * Additional links: Setup Databricks on AWS - ruclips.net/video/gEDS5DOUgY8/видео.html Setup Dat...
Spark Environment - Databricks Community Edition
Просмотров 937Год назад
In this video I cover how to setup a free Databricks community edition environment. This is a great way to have an option for testing out Databricks and learning Apache Spark, and it doesn’t expire after 14 days. It is limited functionality and scalability though, so you won’t be able to run a realistic proof of concept on this environment. Once setup you will see how to run a very simple test ...
Apache Spark DataKickstart - Introduction to Spark
Просмотров 1,1 тыс.Год назад
In this video I provide introduction to Apache Spark as part of my RUclips course Apache Spark DataKickstart. This video covers why Spark is popular, what it really is, and a bit about ways to run Apache Spark. Please check out other videos in this series by selecting the relevant playlist or subscribe and turn on notifications for new videos (coming soon). * All thoughts and opinions are my ow...
Unity Catalog setup for Azure Databricks
Просмотров 15 тыс.Год назад
Unity Catalog setup for Azure Databricks
Visual Studio Code Extension for Databricks
Просмотров 14 тыс.Год назад
Visual Studio Code Extension for Databricks
Parallel Load in Spark Notebook - Questions Answered
Просмотров 2,2 тыс.Год назад
Parallel Load in Spark Notebook - Questions Answered
Delta Change Feed and Delta Merge pipeline (extended demo)
Просмотров 2 тыс.Год назад
Delta Change Feed and Delta Merge pipeline (extended demo)
Data Engineering SD: Rise of Immediate Intelligence - Apache Druid
Просмотров 2412 года назад
Data Engineering SD: Rise of Immediate Intelligence - Apache Druid
Azure Synapse integration with Microsoft Purview data catalog
Просмотров 2,1 тыс.2 года назад
Azure Synapse integration with Microsoft Purview data catalog
Adi Polak - Chaos Engineering - Managing Stages in a Complex Data Flow - Data Engineering SD
Просмотров 1912 года назад
Adi Polak - Chaos Engineering - Managing Stages in a Complex Data Flow - Data Engineering SD
Azure Synapse Spark Monitoring with Log Analytics
Просмотров 4,4 тыс.2 года назад
Azure Synapse Spark Monitoring with Log Analytics
Parallel table ingestion with a Spark Notebook (PySpark + Threading)
Просмотров 13 тыс.2 года назад
Parallel table ingestion with a Spark Notebook (PySpark Threading)
SQL Server On Docker + deploy DB to Azure
Просмотров 4,3 тыс.2 года назад
SQL Server On Docker deploy DB to Azure
Michael Kennedy - 10 tips for developers and data scientists - Data Engineering SD
Просмотров 2122 года назад
Michael Kennedy - 10 tips for developers and data scientists - Data Engineering SD
Synapse Kickstart: Part 5 - Manage Hub
Просмотров 762 года назад
Synapse Kickstart: Part 5 - Manage Hub
Synapse Kickstart: Part 4 - Integrate and Monitor
Просмотров 2682 года назад
Synapse Kickstart: Part 4 - Integrate and Monitor
Synapse Kickstart: Part 3 - Develop Hub (Spark/SQL Scripts)
Просмотров 2892 года назад
Synapse Kickstart: Part 3 - Develop Hub (Spark/SQL Scripts)
Data Lifecycle Management with lakeFS - Data Engineering SD
Просмотров 3292 года назад
Data Lifecycle Management with lakeFS - Data Engineering SD
Synapse Kickstart: Part 2 - Data Hub and Querying
Просмотров 3352 года назад
Synapse Kickstart: Part 2 - Data Hub and Querying
Hi Dustin, is there any way to monitor compute event log from log analytics?
Hey Dustin, We're currently exploring DABs, and it feels like this was made just for us!😅 Thanks a lot for sharing it!
Hey Dustin, Thanks for the amazing explanation! DABs are sure to be adopted by every dev team!
Thanks, one question I have some metadata files in the project folder, I need to copy this to a volume in Unity catlog. Is it possible through this deploy process ?
Using Databricks CLI path, you can have command that copies data up to volume. Replace all the curly brace { } parts with your own values. databricks fs cp --overwrite {local_path} dbfs:/Volumes/{catalog}/{schema}/{volume_name}/{filename}
Wonderful tutorial, Thank you! This approach works effectively for running multiple tables in parallel when using spark.read and spark.write to a table. However, if the process involves reading with spark.read and then merging the data into a table based on a condition, one thread interferes with another, leading to thread failure. Is there any workaround for this?
Great content!! I am trying to deploy the same job into different environments DEV/QA/PRD. I want to override parameters passed to the job from variable-group defined on the Azure DevOps portal. Can you please suggest how to proceed on this?
The part that references variables group PrdVariables shows how you set different variables and values depending on target environment. - stage: toProduction variables: - group: PrdVariables condition: | eq(variables['Build.SourceBranch'], 'refs/heads/main') In the part where you deploy the bundle, you can pass in variable values. See the docs for how that can be set. docs.databricks.com/en/dev-tools/bundles/settings.html#set-a-variables-value
I am learning DABs at this moment. So lucky that I found this video. Thank you, @DustinVannoy. Do you mind if I ask a couple of questions?
Yes, ask away. I'll answer what I can.
Thank you, @@DustinVannoy. I wonder whether the following development progress does make sence. And if there any thing we could improve it. Background: (1) We have two Azure Databricks workspaces, one is for development, one is for production. (2) I am the only Data Engineer in our team, and we don't have dedicate QA. I am responsible to development and test. Who consume the data will do UAT. (3) We use Azure DevOps (repository and pipelines). Process: (1) Initialization (1.1) Create a new project by using `databricks bundle init` (1.2) Push the new project to Azure DevOps (1.3) On development DBR workspace, create a GIT Folder under `/Users/myname/` and link to the Azure DevOps repository (2) Development (2.1) Create a feature branch on DBR workspace (2.2) Do my development and hand test (2.3) Create a unit test job and the scheduled daily job (2.4) Create a pull request from the feature branch to the main branch on DBR workspace (3) CI (3.1) An Azure CI pipeline (build pipeline) will be trigerred after the pull request is created (3.2) The CI pipeline will check out the feature branch, and do `databricks bundle deploy` and `databricks bundle run --job the_unit_test_job` on the development DBR workspace by using Service Principal. (3.3) The test result will show on the pull request (4) CD (4.1) If everything looks good, the pull request will be approved (4.2) Manually trigger an Azure CD pipeline (release pipeline). Checkout the main branch, do `databricks bundle deploy` to the production DBR workspace by using Service Principal Explanation: (1) Because we are a small team and I am the only person who works on this, we do not have a `release` branch to simply the process (2) Due to the same reason, we also do not have a staging DBR workspace
Overall process is good. It’s typical not to have a separate QA person. I try to use yaml pipeline for the release step so code would look pretty similar to what you use to automate deploy to dev. I recommend having unit tests you can easily run as you build which is why I try to use Databricks-connect to run a few specific unit tests at a time. But, running workflows on all-purpose or serverless isn’t too bad an option for quick testing as you develop.
Now do it in Terraform ;) Great video as always!
🤣🤣 it may happen one day, but not today. I would probably need help from build5nines.com
Thanks a lot Dustin... Really appreciate it :)
Thanks for the video. It helped me a lot in my YT channel.
Great video Dustin! Especially on the advanced configuration of the databricks.yaml. I'd like to hear your opinion on the /src in the root of the folder. If you're team/organisation is used to work with a mono repo it would be great to have all common packages in the root, however, if you're more of a polyrepo kinda team/organisation, building and hosting the packages remotely (i.e. Nexus or something) could be a better approach in my opinion. Or am I missing something? How would you deal with a job where task 1 and task 2 have source code with conflicting dependencies?
Is there a way for python wheel tasks to combine the functionality we had without serverless to use: libraries: - whl../dist/*.whl so that the wheel gets deployed automatically with using serverless? As if I am trying to include environments for serverless I can't longer specify libraries for the wheel task (and therefore it is not deployed automatically) and I also need to hardcode my path for the wheel in the workspace. Could not find an example for that so far. All the best, Thomas
Are you trying to install the wheel in a notebook task, so you are required to install with %pip install? If you include the artifact section it should build and upload the wheel regardless of usage in a taks. You can predict the path within the .bundle deploy if you aren't setting mode: development, but I've been uploading it to a specific workspace or volume location. As environments for serverless evolve I may come back wtih more examples of how those should be used.
Hey, thanks for video! I was wondering if you know whether those audit logs are still exist even if audit log not configured (audit log/ log delivery)? I mean - are events will still be written in the back and once it is enabled (via system tables) could be consumed?
How do you enable intellisense?
how do you change the Catalog Name specific to an environment?
I would use a bundle variable and set it in the target overrides, then reference it anywhere you need it.
Thansk for the explanation! Is there any way to view exported DBC files in VS code?
You should export as source files instead of dbc files if you want to view and edit in VS Code.
Exciting stuff! Will definitely be trying to implement this in my future work!
Great video ! Is there a way to overide variables defined in the databricks.yml in each of the job yml definition so that the variable has a different value for that job only ?
If value is the same for a job across all targets you wouldn't use a variable. To override job values you would set those in the target section which I always include in databricks.yml.
Thanks a lot, @DustinVannoy for this great presentation! I have a question: which is the better approach for project structuration: one bundle yml config file for all my sub-projects or each sub-project have its own Databricks and bundle yml file? Thanks again :)
Once the code is deployed it gets uploaded in the shared folder can't we store that some where else like an artifact or storage account because there are chances that someone may deleted that bundle from shared folder. It is always like with databricks deployment before and after asset bundles.
You can set permissions on the workspace folder and I recommend also having it all checked into version control such as GitHub in case you ever need to recover an older version.
Can we integrate Azure pipelines + DAB for ci cd implementation?
Are you referring to Azure DevOps CI pipelines? You can do that and I am considering a video on that since it has been requested a few times.
@@DustinVannoy yes, thank you!
@@DustinVannoy Please, can you do that? hahaha
Video showing Azure DevOps Pipeline is published! ruclips.net/video/ZuQzIbRoFC4/видео.html
Loving bundles so far. Only issue so far I've had is the databricks vscode extension seems to be modifying my bundles yml file behind the scenes. For example when I attach to a cluster in the extension it will override my job cluster to use that attached cluster when I deploy to the dev target in development mode.
Which version of the extension are you on, 1.3.0?
@@DustinVannoyYup, I did have it on a pre release which I thought was the issue but switched back to 1.3.0 and the "feature" persisted.
Hey, Great video! Small question, Why not just using the FAIR scheduler that doing that automatically?
@@maoraharon3201 on Databricks you can now submit multiple tasks in parallel from a workflow/job which is my preferred approach in many cases.
Can delta sharing works with hive_metastore?
thanks so much for your help
Run driver program using multithreads using this as well. from threading import * # import threading from time import * # for demonstration we have added time module workerCount = 3 # number to control the program using threads def display(tablename): # function to read & load tables from X schema to Y Schema try: #spark.table(f'{tablename}').write.format('delta').mode('overwrite').saveAsTable(f'{tablename}'+'target') print(f'Data Copy from {tablename} -----To----- {tablename}_target is completed.') except : print("Data Copy Failed.") sleep(3) list = ['Table1','Table2','Table3','Table4','Table5', 'Table3', 'Table7', 'Table8'] # list of tables to process tablesPair = zip(list,list) # 1st list used for creating object & 2nd list used as table name & thread name counter = 0 for obj,value in tablesPair: obj = Thread(target=display, args=(value,), name=value) # creating Thread obj.start() # Starting Thread counter += 1 if counter % workerCount == 0: obj.join() # Hold untill 3rd Thread completes counter = 0
Hey Dustin, Thanks for the tutorial! I've successfully integrated the init script and have been receiving logs. However, I'm finding it challenging to identify the most useful logs and create meaningful dashboards. Could you create a video tutorial focusing on identifying the most valuable logs and demonstrating how to build dashboards from them? I think this would be incredibly helpful for myself and others navigating through the data. Looking forward to your insights!
This is what I have plus the related blog posts. ruclips.net/video/92oJ20XeQso/видео.htmlsi=OS-WZ_QrL-_kkwWu We mostly used out custom logs for driving dashboards but also evaluated some of the heap memory metrics regularly as well.
@@DustinVannoy Thank you. It means a lot :)
Hello Dustin, Thank you for posting this video. This was very helpful!!! Pardon my ignorance but I have a question about initializing the Databricks bundle. The first step when you initialize the databricks bundle through CLI, does it create the required files in the databricks workspace folder. Additionally do we push the files from the databricks workspace to our git feature branch so that we can clone it to your local so that we can make the change in the configurations and push it back to git for deployment.
Typically I am doing the bundle init and other bundle work locally and committing then pushing to version control. There are some ways to do this from workspace now but it's likely to get much easier in the future and I hope to share that out once publicly available.
Hello, sir, Thank you for this tutorial. I successfully integrated with log analytics. Could you please show me what we can do with these logs and how to create dashboards? I am eagerly awaiting your response. Please guide me.
please do this episode
Hi Dustin, Thank you for sharing this approach I am going to use it for training spark ml models. I had a question on using daemon option. My understanding is that these threads will never terminate until a script ends. When do they in this example? Do they terminate at the end of the cell? Or after .join()? So when all items in the queue have completed. I really appreciate any explanation you provide.
Thank you for the session!
Hi thanks for the informative video! I have a question, instead of sending a list to the notebook, I send a single table to the notebook using a for each activity (synapse can do maximum 50 concurrent iterations). What would the difference be? Which would be more efficient? And what is best practice in this case? Thanks in advance!
For CPU intensive operations would this work since it uses threading? Also, can't we use multiprocessing if we want to achieve parallelism?
great content, thank you
It is creating duplicate jobs on re deployment of databricks.yml. How to avoid that?
is it possible to add approvers in asset bundle based code promotion ? Say one does not want the same dev to promote to prod, as prod could be maintained by other teams; or if the dev has to do cod promotion, it should go through an approval process. Also is it possible to add code scanning using something like sonarcube ?
All that is done with your CICD tools that automate the deploy, not within Databricks Asset Bundle itself. So take a look at how to do that with Github Actions, Azure DevOps pipelines, or whatever you use to deploy.
Amazing content , thanks man. I'm learning a lot
Like @gardnmi, I also used the map method threadpool has. Didn't need a queue. I created a new cluster (tagged for the appropriate billing category) and set the max workers on both the cluster and threadpool: from concurrent.futures import ThreadPoolExecutor with ThreadPoolExecutor(max_workers=137) as threadpool: s3_bucket_path = 's3://mybucket/' threadpool.map(lambda table_name: create_bronze_tables(s3_bucket_path, table_name), tables_list)
Great video tutorial. Clear explanation. Thank you.
How long does it take to deploy the python wheel for you? For me it takes about 15 mins which makes me consider making wheel project separate from rest of the solution.
I am not currently working with Synapse but 15 minutes is too long if the wheel is already built and available to the spark pool for the install.
I was having so many issues using the other Threadpool library in a notebook, It cut my notebook runtime down by 70% but I couldn't get it to run in a databricks job. Your solution worked perfectly! Thank you so much!
Will this work if you read in a file, do some minor transformations and then save to ADLS? Would it work if we add in transformations basically?
Yes. If the transformations are different per source table you may want to provide the correct transformation function as an argument also. Or have something like a dictionary that maps source table to transformation logic.
Hi Dustin how can we send stderr logs into azure monitor
I'm not sure of a way to do this, but I haven't put too much time into it. I do not believe the library used in this video can do that, but if you figure out how to get it to write to log4j also then it will go to Azure Monitor / Log Analytics with the approach shown.
thanx alot for this video, but plz try to save the (almost dead) plant behind you :)
Great attention to detail! The plant has been taken care of😀
Hey Dustin, Thank you so much for the video, I still have one doubt, I've been running a streaming query in a notebook for over 10 hours. The streaming query statistics only show specific time intervals. How can I view input rate, process rate, and other stats for different timings or for the entire 10 hours to facilitate debugging?
Check out how to use Query Listener from this video and see if that covers what you are after. ruclips.net/video/iqIdmCvSwwU/видео.html
I tried this. However, I noticed a issue when I have single notebook which creates multiple threads, where each thread is calling a function which creates the spark localtempviews, the views get overwritten by the second thread as it essentially is same spark session. How do I get around this?
I would parameterize it so that each temp view has a unique name.
@@DustinVannoyyea i had that in mind, unfortunately i cannot as the existing jobs are stable in production. However, this is definitely useful for new implementation
I figured it. instead of calling the function i can use dbutils.notebook.run to invoke the notebook in seperate spark session. Thanks
Can we use it for AWs databricks as well?
Yes, it works with AWS.
Hi Dustin, great content, quick question, where can I find the library for Runtime v14 ?
Check out this video and the related blog for latest tested versions. It may work with 14 also but only tested with LTS runtimes. ruclips.net/video/CVzGWWSGWGg/видео.html
Hi Dustin. How to implement the same using scala. I tried but the same solution is not working for me. Any advise?
How can i do this today on windows?
I am close to finalizing a video on how to do this for newer runtimes and i build it on windows this time. I use WSL to build this on windows. For Databricks Runtimes 11.3 and above there is a branch named l4jv2 that works.