What Is Data Engineering - Why Is Data Engineering Important?
HTML-код
- Опубликовано: 28 сен 2024
- What do data engineers do?
Do they build data pipelines?
Do they build data warehouses?
Sure, but why?
If you're team needs help setting up your data engineering processes, roadmap, or infrastructure , then set-up some time with me here - calendly.com/b...
If you enjoyed this video, check out some of my other top videos.
Top Courses To Become A Data Engineer
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
bit.ly/3NQVn7V
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy...
Or check out my blog
www.theseattle...
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.
If you like data engineering and you are looking to start you're own data engineering/analytics consulting company, then you should check out my new course here - courses.technicalfreelanceracademy.com/courses/starting-6-7-figure-consulting - and use the coupon code "deconsult" to get 50% off
Thanks for the quick rundown and the channel. I’m currently a mathematician who’s retraining in a Computer Science master’s program and have been looking to specialize in an area of CS. I think data/database systems/data engineering is the way forward for me. I like the idea of being able to provide utility for other people using computer systems.
You've definitely improved the video quality, I don't know if it is just the light but it looks really professional 🤩
Hahaha this...was a 2x thing. I have another video that looks like this. But I rented this studio. Really good deal though 40$ an hour.
Every upload makes me confident with the topics im learning of DE. I have a sense of what im doing. Thanks Ben!
Great to hear!
Hey Ben, hope you're good! I'm still learning like crazy 6-7 days a week. Every day gets easier in regards to the knowledge and coding practice. Every day, enjoying it more annd feeling pumped from conquering the tough challenges.
I am great! I hope you continue to stay pumped! You got this!!
@@SeattleDataGuy awesome, glad to hear it! Many thanks. I'll definitely make sure I continue to keep smashing it ;) 💪
would you summarize a bit how you learn data engineering? I'm kinda lost
@antoniusdaivap7759 are you already a software engineer? If yes, learn data engineering. If not, learn data analytics....check out: Microsoft Certified: Power BI Data Analyst Associate. Data engineering is not entry level, it requires minimum 3-5 years experience. Data Analytics is entry level and is the first step towards entering the data industry.
. . . in under 10[min]
u have reminded me
JUST HOW MUCH
i still have / want to learn
Whoa, loving the new lighting and background! Also great content, as always. Just fyi, your videos have helped me progress so much in my DE career. Thanks so much Ben!
thank you Ben
I usually hard to understand your content bcs the speaking pace is a lil bit too fast but seeing this video convince me, ur already corrected it.
great content!
I’m naïve to this field, but I have a question. Couldn’t someone in the IT department take on most of what you’re talking about? And with a trained AI, could half of a data engineer’s job be automated? What’s stopping a company from shifting a data engineer’s role to the IT department, with some AI assistance?
Hey, I got your reference form Luke Barousse's video and after watching one, I keep watching your videos for more information about the DE, I am really curious and eagur to learn and because of that I changed my line form Computer engineering to Network security to Data Science and Now I am just looking and learning new stuff from your channel. currently working on Pysparks and if you can suggest me better resources for it.
Keep it up, and appreciate your work.
Thank you.
I feel like writing a mail to you🤣🤣
Data engineering or DevOps needs some advice for the career.
How much time you spend on average doing k8s/docker/system administration?
what are the base fundamentals to expertise to become a good data engineer like tools and languages may vary and change but some base concepts haven't changed for years?
Good question! Have you checked out this video on DE skills?
ruclips.net/video/LgSHaOvNodA/видео.html
I tried to show how you should build a foundation of skills in it
Can you explain a very basic example of an ETL moving from SQL Server to S3 to Redshift? Or a different combination of technologies. I just want to see where SQL fits in with which tool, same for python. Everyone talks about needing these languages but doesn't explain how to actually use them. I have experience using the languages but not sure how to build a production ETL with them.
It depends if you want real time data transformation and loading into redshift, from what I understand you can use bash shell script automation to read the data from SQL server then transform it with bash or python, then load them to redshift, you can use something like Apache airflow to set up this ETL job, hope this is helpful
You can use something like Crontab to set up automating the execution of your scripts on your desired time interval
Should I learn Scala
Thanks man!
you're welcome!
Could you make a tutorial with a project with airflow??? NICE CONTENT! thank you!!!
Is that the same as being a statistician?
Guruji Pranam.
Could you please tell what are the certifications that will weigh on freshers resume to get hired as a data engineer? Or what are the things that hiring managers are looking for in a freshers profile to most likey get a call
If you do want a certificate, i'd only recommend you get 1 and then focus more of your time on data engineering projects. In terms of which one, any of the 3 big cloud providers will likely be good.
Ben, I really enjoy your videos, but the frequent quick pauses in your talks can be distracting and make it hard to follow your points. Speaking more smoothly to improve clarity would be nice. Keep up the good work!
0:33 DAG is represented as a cyclic figure ???
Because it’s a cyclic graph
Kinda preaching to the choir here
🤣
For someone who has worked in DE in top companies like Facebook, you tend to confuse your audience a lot. Please either support the case or reject it completely 😂. I am an aspiring Data engineering but some of your videos really confused whether to go for it or not.
could you give me an example when i have said not to go for data engineering? I am just not sure of when I have said not to become one. I have a video that titled "don't become a data engineer" but i believe even there, its more of a title and the actual content is still pretty supportive.
@@SeattleDataGuy Lol you took it seriously man I was kidding 😂. Love your videos by the way.
@@darshantawte7435 OOOOh! Hahaha, I misunderstood. I would hate to know I confused anyone. 🤣
@@SeattleDataGuy just one question, i know Data engineering is not a job for typical freshers so what can we do regarding that. Also how are DE opportunities in US as I had done some research earlier but haven't found a lot of jobs in this field(Atleast not by DE as the job description). Do companies provide internship for DE ?
man this was such a turn off
Why?
Thank you for the videos Ben. I’ve already started reading through The Fundamentals of Data Engineering and it’s a phenomenal book so far!
One unrelated question though, I’m personally very interested in networking more in this field and I’m wondering if there are any professional or technical “data conferences” you might recommend?
leveling up the cinematography!
sadly a 1x thing for now. I rented a great studio!
Hi Ben,
Thanks for Great Content!
Quick question how to create something like dynamic gif or video something like that you showed at @2:13 ruclips.net/video/wyGAYa2UMXQ/видео.html ?
woah!
ya know how u build an image of someone in ur head .. maybe it's a coworker ur S.O. talks about .. or a SubStack author
nothing bad - but u did not match at ALL the image i had concocted 😅
haha..are you saying I don't look like what you expected? I honestly feel like I have a few different looks...this one is shaggy hair but shaved...
Super great vid, thanks for the info
Glad it was helpful!
Good content. The part about having to deal with data curation as a Data Engineer cannot be understated. I never saw godd enough source data that did not require some kind of repair during the ingest phase
yaay new studio💥💥
Sadly this is not permanent
Hey, I want to pursue my career in data science (Forcefully) as I will be moving to another country and there is less chance of getting a job in my current domain (game-dev) as only a few companies exist. I do have a programming exp. of 5 years. What is your general perspective/thoughts, can anyone with programming skills adjust himself as a data-engineer? Is it future proof?
Why devops engineering get paid higher although data engineering harder? 🤔