If your building packages then dbx is the obvious choice and without notebook support I'm not sure who this extension is intended for. Also, it's a bit surprising how featureless it is compared to the other databricks extension offered by paiqo.
Thanks for sharing your thoughts and calling out other options people can try. I see value in running PySpark code that are not notebooks in interactive mode with this extension. Packaging up the code and installing each time was a pain which is where databricks-connect was helpful. Once all the tests check out then committing and deploying would be a final step, ideally. And notebooks can be run they are just run as workflows. I am not sure if there is a good way to render the notebooks how VS Code renders .ipynb files since I usually do notebook development in the browser and Python/PySpark development in an IDE.
When you kick off from VS Code extension it runs against the interactive cluster you set, and you can run as a workflow so its easy to see it from the workspace. DBX is still an option for configuring jobs (with job compute) and deploying from local.
Databricks Connect v2 just became public preview and that is my preference for debugging code running on Databricks from local IDE. I have some other items to focus on but may come back with a better explanation of the role VS Code extension, DBX and Databricks Connect play in development. They compliment each other well.
@@DustinVannoy Hi, we are currently considering these options in my team atm. Specifically we are looking for the ability to degun on the DB cluster while inside VSCode. The extension is a limitation bc of its dependency on Unity Catalog. Did you end up releasing this video? Would be interesting to see your take on the 3 tools.
This is the best one ... I am fed up with running Note book from browser. Need to explore more on this
I'm unable to install that particular databricks ,says corruptziperror:central command signature not found,any help on this?
awesome content Dustin, was really helpfull for me that i'm a data eng jr, is good to know many ways to run or explore it downloading right now
How do you enable intellisense?
Thansk for the explanation!
Is there any way to view exported DBC files in VS code?
You should export as source files instead of dbc files if you want to view and edit in VS Code.
If your building packages then dbx is the obvious choice and without notebook support I'm not sure who this extension is intended for. Also, it's a bit surprising how featureless it is compared to the other databricks extension offered by paiqo.
Thanks for sharing your thoughts and calling out other options people can try. I see value in running PySpark code that are not notebooks in interactive mode with this extension. Packaging up the code and installing each time was a pain which is where databricks-connect was helpful. Once all the tests check out then committing and deploying would be a final step, ideally. And notebooks can be run they are just run as workflows. I am not sure if there is a good way to render the notebooks how VS Code renders .ipynb files since I usually do notebook development in the browser and Python/PySpark development in an IDE.
Hi, thank you so much . This helped me a lot! But can this code be deployed and run on cluster?
When you kick off from VS Code extension it runs against the interactive cluster you set, and you can run as a workflow so its easy to see it from the workspace. DBX is still an option for configuring jobs (with job compute) and deploying from local.
Great video! Is there any way we can run notebooks specific cell so that we don’t have to rerun something over and over? Thanks!
The extension doesn't support that in the versions I have tested.
Can we use it for AWs databricks as well?
Yes, it works with AWS.
Very nice explanation and Video
How does one debug when running from databricks using dbx?
Databricks Connect v2 just became public preview and that is my preference for debugging code running on Databricks from local IDE. I have some other items to focus on but may come back with a better explanation of the role VS Code extension, DBX and Databricks Connect play in development. They compliment each other well.
@@DustinVannoy Hi, we are currently considering these options in my team atm. Specifically we are looking for the ability to degun on the DB cluster while inside VSCode. The extension is a limitation bc of its dependency on Unity Catalog. Did you end up releasing this video? Would be interesting to see your take on the 3 tools.
Thank you !!!. So helpfull