If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k
I do security log management using Cribl and everything youve said is everythinf ive been thinking but in a different language 😂 Great to hear from someone else. We dont have data engineers where I work. Just me
Wow As an Informatica ETL developer myself planning to switch to Data Engineering I was just trying to understand this concept of Data pipeline, this is as much more simplified explaination although would request you a video on what kind of pipeline would you suggest based on different business scenario that you mentioned as we have some ETL or I say ELT tool build native to cloud such as Matillion ,IICS or manually coding the entire thing.
Good job on mentioning that there are too many tools. I’m a Business Systems Analyst right now who is trying to learn data engineering and there seems to be too much to learn.
I am here lookong to switch from analytics and digital lead to data engineer as i see myself doing it alongside software engineering (although they might sound different). So as i lean more towards SQL rather than Python. I got trained on SQL and SSIS and i never knew that was part of data engineering. In SSIS, you really learn a lot about the ETL process. This video really taught me to revisit the SSIS as there is a learn in the system and plus it is MS product.
Being someone who wants to migrate from a data analytics role into a data engineering role, what tools would you guys reccommend I start out with ? From the ETL/ELT operation to data warehousing and maybe post that some work on cloud tech. Any help would be appreciated.
If your SQL, Python and data warehousing is solid. Then you can start working on developing some basic pipelines and building core tables in something like snowflake, bigquery or databricks.
@@bryanneremita2860 Have you read kimball and inmon's books? Great for understanding warehousing as well as high level etl concepts. Oh and the fundamentals of data engineering too www.oreilly.com/library/view/fundamentals-of-data/9781098108298/
apart from being excellent in sql what else one should know if a datawarehouse guy wants to become a data guy, not the seattle one but an engineer one.
If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k
Great content, simple enough to understand and reflect on as to learn how the same knowledge can be interpreted by different people.
Glad you enjoyed it!
love the outro music! thanks man!
I do security log management using Cribl and everything youve said is everythinf ive been thinking but in a different language 😂 Great to hear from someone else. We dont have data engineers where I work. Just me
Sometimes that's how it goes. One person doing all of it. Good luck!
Wow As an Informatica ETL developer myself planning to switch to Data Engineering I was just trying to understand this concept of Data pipeline, this is as much more simplified explaination although would request you a video on what kind of pipeline would you suggest based on different business scenario that you mentioned as we have some ETL or I say ELT tool build native to cloud such as Matillion ,IICS or manually coding the entire thing.
Glad to hear this was easy to understand! That is my goal. I will put that video on the list
Good job on mentioning that there are too many tools. I’m a Business Systems Analyst right now who is trying to learn data engineering and there seems to be too much to learn.
For real...its never ending
Good to see Alexy🙂
yes I loved that he said yes to joining in!
I am here lookong to switch from analytics and digital lead to data engineer as i see myself doing it alongside software engineering (although they might sound different). So as i lean more towards SQL rather than Python. I got trained on SQL and SSIS and i never knew that was part of data engineering. In SSIS, you really learn a lot about the ETL process. This video really taught me to revisit the SSIS as there is a learn in the system and plus it is MS product.
Awesome! I would still say learning some python is a good idea!
@@SeattleDataGuy thank you very much mate
@@alphar85 You're welcome!
A lot of this content is geared for beginners and for folks trying to break into the role. What about advanced topics or in-depth technical debriefs?
What are some specific topics you are looking into?
Being someone who wants to migrate from a data analytics role into a data engineering role, what tools would you guys reccommend I start out with ? From the ETL/ELT operation to data warehousing and maybe post that some work on cloud tech. Any help would be appreciated.
If your SQL, Python and data warehousing is solid. Then you can start working on developing some basic pipelines and building core tables in something like snowflake, bigquery or databricks.
@@SeattleDataGuy thanks for the quick response. My sql and python is alright but i need to get Into warehousing and etl. Any tips on that?
@@bryanneremita2860 Have you read kimball and inmon's books? Great for understanding warehousing as well as high level etl concepts.
Oh and the fundamentals of data engineering too www.oreilly.com/library/view/fundamentals-of-data/9781098108298/
@@SeattleDataGuy thanks a lot. This will be helpful.
Hi,
Currently doing etl pipelines with airflow on gcp. Do I need to study about starflakes,snowflakes schema?
Thanks for the question! I think its always good for your foundation, yes.
@@SeattleDataGuy thanks!
apart from being excellent in sql what else one should know if a datawarehouse guy wants to become a data guy, not the seattle one but an engineer one.
Ya, I think I look like a data plumber than a data engineer
hahaha we really are
I didn’t understand 💩