There's a ton of videos ( one for every keyword) on RUclips on this subject. Most fails to deliver any useful knowledge, others are too narrow and/or incomprehensible. I'm so glad to have found this series. Your teaching style took me back to my college classes. Fairly detailed and well explained. So a big thanks to you for it Bryan 👍.
probably the first video that describes the difference between spark and databricks so cleanly and also the different components of spark with regards to where they are placed in the whole data engineering ecosystem .... Thanks a lot ...!!!
This is my 3 Rd teacher. You explained all well from an experienced person. I thought first what this old man gonna speak...now end up touching ur feet. Well done Bibin from India,Kerala
Man man I'm supporting this guy on patron.... What a class ...what a breakdown...you here about all these tools and 9 out of 10 times I'm drowning in them
I dont know how come i did not see your channel earlier, I am now hooked on to these, please make more and more, I like listening to your tutorials and making notes.
O M G I found what I was looking for. I've started working at Nestlé as a Data Science Analyst and I'm searching for a good playlist of Databricks and Spark to have a deeper understanding on this subject but you're the one that matched my way to learn and have lectures. A huge big thanks to you 🌟
Thanks Bryan for wonderful video, you kept me engaged and attentive through out of the video. Your explanation very crystal clear and one of the best on the internet. Thanks and god bless you healthy and energetic.
I have never seen such a straightforward, clear , concise explanation on this concept. till date, i have tried to understand Apache Spark and Databricks... but i've always had some convoluted understanding of them. thank you for much for this video.. it really helped me understand where things stand now.
Thank you, Thank you, Thank you for explaining it in the simplest way possible. At last I was able to understand what are Hadoop, Spark and Databricks, and what actually they do.
Hey Bryan, thank you for the excellent video. it put my mind at ease. I have seen that you have used Azure Databricks going forward. However my organization stores data on s3 and works predominantly in databricks platform itself. I was wondering if the knowledge you have shared will work good in direct databricks platform. I am a complete new beginner in this field, so apology for any silly questions
Hi Ishaq, Databricks is a complete self contained service available on AWS, Azure, and GCP. It should work the same on all three with the only differences being how it integrates with the cloud specific back end services like s3. Also, Azure integrates Databricks in a way that eliminates the need for the customer to have an agreement with Databricks and Microsoft. It appears as if it were an Azure service. I think AWS requires customers to license with Databricks and AWS when they set it up. So yes, overall, all the Databricks and Spark code and services should be the same on all 3 cloud platforms. Make sense?
Hi Bryan, I'm currently learning Data science on GCP as a beginner. I'm just scratching the surface about learning GCP tools/platform. I wanted to learn Spark and that is why I'm here. Would learning Spark and Databricks in a 'Microsoft Azure platform' be a right idea at this time given I'm focusing on GCP ? Thanks for making this course though, I see so much content here and I'm still on the first video!
Databricks is a service owned by the company Databricks that is available on AWS, Azure, and GCP. It should be the same on any of these platforms with the only differences being how cloud-specific resources are called or integrated, i.e Azure Synapse vs. Google's BigQuery. You should be fine using Databricks on GCP but let me know if you find significant differences. Make sense?
You can but you can define libraries for a cluster and Databricks will automatically re-install them ever time the cluster starts. You can even define libraries you want installed on every cluster if you like. Spark does not support cluster stop and start. You have too delete and re-create clusters if you want to stop paying for them. When you create a cluster, you have do do some work to install the libraries you want.
Its more than just coding. You need the background and concepts to be effective. It's a long video series and if you skip the foundation, you will never gain mastery.
@@BryanCafferky As someone who's been researching the Apache ecosystem for the last month, trying to make sense of what's what with so much overlapping functionality, I greatly appreciate this video. Thank you for the thorough explanation. I look forward to watching and learning from the rest of the videos in this playlist about Spark and Databricks. With that said, since some of these videos are a bit old, would you say any of the information in them is outdated?
@@stigmartinsen3359 The Databricks UI has changed a lot but the functionality has stayed. New functionality has been added such as Delta Lake, Unity Catalog, and Photon. See this video for an update on these: ruclips.net/video/9YJby_COOdc/видео.html
In general , what I notice is , compared to the past, they are over-complicating everything, especially that whole Azure thing is unnecessarily complex. At least on-premise was never this much work !
No. I disagree there. In fact, the point is that Cloud based Databricks is tons easier to use and provides much better tools than using open source Spark on prem. Not sure what you are looking at. Thanks for your comment.
@@BryanCafferky Eg: In Databricks , If I want to access dbfs files in another resource group - you have to create a "scope', get access to a vault secret, use the scope to mount that dbfs in your workspace hive metastore, write a script to mount, write a script to create a temp view and read the data from that delta table. In SQL Server: I can share connection string user/password with somebody else, they can connect to the database from SQL Management studio, enter the details and run as many queries as they want on that database, joining multiple tables etc etc.
There's a ton of videos ( one for every keyword) on RUclips on this subject. Most fails to deliver any useful knowledge, others are too narrow and/or incomprehensible. I'm so glad to have found this series. Your teaching style took me back to my college classes. Fairly detailed and well explained. So a big thanks to you for it Bryan 👍.
Thanks, Faisal. If you follow the entire series, you will get a solid foundation.
@@BryanCafferky I intend to go through this whole stuff.. Lesson 0 and 1 completed... onto lesson 2
This is one of the best and clearest explanations about Spark and Databricks on the internet.
The 1st class I did not sleep :). There is something special about how you explain and keep listeners hooked up. I am eager to go thru next sessions
Thanks! I appreciate the kind words and am glad the videos are helpful.
Finally, a Databricks RUclips series that makes sense! I'm using DB with AWS, but this all pertains. Thanks!
You're welcome!
Definetly the best serie for learning databricks
probably the first video that describes the difference between spark and databricks so cleanly and also the different components of spark with regards to where they are placed in the whole data engineering ecosystem .... Thanks a lot ...!!!
You're welcome!
yeah agree, so cleanly.
Spot On! I really liked how you transitioned from the broader umbrella of Hadoop> spark> Databricks.. Great job Bryan!...
Thank You!
This is my 3 Rd teacher. You explained all well from an experienced person. I thought first what this old man gonna speak...now end up touching ur feet. Well done
Bibin from India,Kerala
Love the accent, As a New Englander living abroad, it made me homesick! This intro video is wicked awesome.
Thanks. What accent? 🙂
Man man I'm supporting this guy on patron.... What a class ...what a breakdown...you here about all these tools and 9 out of 10 times I'm drowning in them
Thank You!
A very good class, especially for a Databricks virgin like me.
This video clearly explains the concepts around apache spark, databricks and the various offerings. Wonderful explanation thanks a ton 👏👍
Absolutely the best explanation of Databricks I've ever heard
I dont know how come i did not see your channel earlier, I am now hooked on to these, please make more and more, I like listening to your tutorials and making notes.
Wonderful presentation. Thank you so much, Bryan!🙏
You're welcome!
O M G
I found what I was looking for.
I've started working at Nestlé as a Data Science Analyst and I'm searching for a good playlist of Databricks and Spark to have a deeper understanding on this subject but you're the one that matched my way to learn and have lectures. A huge big thanks to you 🌟
Thanks so much! It is really great to hear feedback like that! Glad it helps you.
finally someone is talking clearly
Thanks Bryan for wonderful video, you kept me engaged and attentive through out of the video. Your explanation very crystal clear and one of the best on the internet. Thanks and god bless you healthy and energetic.
Thank you!
I have never seen such a straightforward, clear , concise explanation on this concept. till date, i have tried to understand Apache Spark and Databricks... but i've always had some convoluted understanding of them. thank you for much for this video.. it really helped me understand where things stand now.
Thanks. Glad the videos are helpful.
I tried to do data bricks academy and I got lost. Thanks to channel, I understand every nook and crannies. Thumbs up Brian!!
Thank you! Glad my videos are helping you.
Sir thank you so much! You match my learning style and you have a clear voice
Thanks. Glad the videos are helpful!
Thank you, Thank you, Thank you for explaining it in the simplest way possible. At last I was able to understand what are Hadoop, Spark and Databricks, and what actually they do.
So glad to hear that. It's why I do this channel. Thanks
Exactly what I was looking for !!! your best teacher is your last mistake! proved!
Great content, thank you for doing this general and historic view, sometimes it is necessary to understand the details.
Thanks for your videos Bryan - nice work. Really good amount of information clearly explained.
You're welcome! Thanks for watching.
Best DB tutorial out there!!!
You are awesome! There's hardly any other material which is as clear and condensed. Thank you for creating this video🙏
Fantastic video, thank you Bryan
You're Welcome!
your video has decluttered me a lot. Now am going to make a hdfs on my k8s cluster and spark operator
Love your explanation! It is so easy to understand
thanks for these videos i'm finishing up the data camp data engineer track and then jumping in on these
Great!
thanks really good summary of all these languages and how they came about .
Thank you Professor Bryan !
Guy has two left sides and still manages to make sense. Inspiring. :)
Thank you brother you are simply amazing
Incredible quality, thanks!
Great start to the series. Thank you!
YW!
such a clean explanation. Thank you!
YW
Hey Bryan, thank you for the excellent video. it put my mind at ease. I have seen that you have used Azure Databricks going forward. However my organization stores data on s3 and works predominantly in databricks platform itself. I was wondering if the knowledge you have shared will work good in direct databricks platform. I am a complete new beginner in this field, so apology for any silly questions
Hi Ishaq,
Databricks is a complete self contained service available on AWS, Azure, and GCP. It should work the same on all three with the only differences being how it integrates with the cloud specific back end services like s3. Also, Azure integrates Databricks in a way that eliminates the need for the customer to have an agreement with Databricks and Microsoft. It appears as if it were an Azure service. I think AWS requires customers to license with Databricks and AWS when they set it up. So yes, overall, all the Databricks and Spark code and services should be the same on all 3 cloud platforms. Make sense?
ohh man such a detailed and superbly structured content.... I wish I could take you out for beers sometime :)
Thanks. I appreciate the kind words. It's great to know my work is helpful.
The Best explanation, thank you
Fantastic video! Really well done, thank you
Thank you! Glad they help.
I would double that! So concise yet comprehensive overview! Thank you so much!
@@mehmetkaya4330 Thanks!
Thank you Brian so much for the wonderful contents!!!
YW. Glad it is helpful.
Thank you so much for your videos, i have been looking for this
You're welcome!
Hi Bryan, I'm currently learning Data science on GCP as a beginner. I'm just scratching the surface about learning GCP tools/platform. I wanted to learn Spark and that is why I'm here. Would learning Spark and Databricks in a 'Microsoft Azure platform' be a right idea at this time given I'm focusing on GCP ? Thanks for making this course though, I see so much content here and I'm still on the first video!
Databricks is a service owned by the company Databricks that is available on AWS, Azure, and GCP. It should be the same on any of these platforms with the only differences being how cloud-specific resources are called or integrated, i.e Azure Synapse vs. Google's BigQuery. You should be fine using Databricks on GCP but let me know if you find significant differences.
Make sense?
28:10 You couldn't install external libraries with open source spark?
You can but you can define libraries for a cluster and Databricks will automatically re-install them ever time the cluster starts. You can even define libraries you want installed on every cluster if you like. Spark does not support cluster stop and start. You have too delete and re-create clusters if you want to stop paying for them. When you create a cluster, you have do do some work to install the libraries you want.
thank you for your explanation about spark is ,Its confuse at firts sigh are these technologies for someone that never used .
You're welcome!
Thank you very much, it was very interesting and helpful
You're welcome!
Outstanding content!
This is an amazing video
Clingon !! That's when I subscribed 😅
Zank you sir for zis tutorial. It is most very velcome.
I wanted to know why we need to know this. I just felt like I was going through the motion while watching this.
Its more than just coding. You need the background and concepts to be effective. It's a long video series and if you skip the foundation, you will never gain mastery.
@@BryanCafferky As someone who's been researching the Apache ecosystem for the last month, trying to make sense of what's what with so much overlapping functionality, I greatly appreciate this video. Thank you for the thorough explanation. I look forward to watching and learning from the rest of the videos in this playlist about Spark and Databricks.
With that said, since some of these videos are a bit old, would you say any of the information in them is outdated?
@@stigmartinsen3359 The Databricks UI has changed a lot but the functionality has stayed. New functionality has been added such as Delta Lake, Unity Catalog, and Photon. See this video for an update on these: ruclips.net/video/9YJby_COOdc/видео.html
Brilliant analogy sir .. phone books example.. for distributed compute too good.
Thanks a lot!
You're welcome!
Great! Let's go!
Thanks for that
Is your book available in epub format?
How to drop cached data which was cached using delta cache into local storage ? I couldn't find a proper command.
That's a bit beyond the content of this video.
In general , what I notice is , compared to the past, they are over-complicating everything, especially that whole Azure thing is unnecessarily complex. At least on-premise was never this much work !
No. I disagree there. In fact, the point is that Cloud based Databricks is tons easier to use and provides much better tools than using open source Spark on prem. Not sure what you are looking at. Thanks for your comment.
@@BryanCafferky Eg: In Databricks , If I want to access dbfs files in another resource group - you have to create a "scope', get access to a vault secret, use the scope to mount that dbfs in your workspace hive metastore, write a script to mount, write a script to create a temp view and read the data from that delta table.
In SQL Server: I can share connection string user/password with somebody else, they can connect to the database from SQL Management studio, enter the details and run as many queries as they want on that database, joining multiple tables etc etc.
THANK YOuuuuuuuuu!!!!!!!
Where's Kubernetes as cluster manager?
thanks
Awesome
In real life this is something called "I stumbled and found a gold mine" !
Thank you, man! I was lost at first, I needed your Tutorial so so so so much!!
Glad it helped! I understand. It is a lot to learn.