For me, I has been forced by the project to be defacto data architect. I was Lead Data Engineer but we were implement our cloud data platform project and at the time I am the only senior DE, the rest of my team is all junior. So the job just force me to be an architect. Now my position still called DE but I rarely built pipeline myself. I am more like a team lead + architect role most of the time design the high lv solution, do some POC, give consultation to the juniors and a lot of meetings!
In my opinion that's how someone ought to become a data architect. You should be a senior (or lead) data engineer first. When hiring someone as a data architect who hasn't worked as a data engineer himself you might end up with the architect taking decisions that aren't aligned with the needs of the engineers which slows down the entire project. Software architects aren't data architects.
I don't have complete control of that. Googles new rules now state that they can show ads whenever they want and if you chose not to collect the youtube payout then they'll just take all of it.
Thanks, Ben! Would you make a video about data quality? I found myself recently using a library called Great Expectations to test the quality of data produced by my pipelines. Also a video about data catalogs would be cool. Would be great to hear your thoughts on these topics!
Are data architect role meant for someone who has vast experienced in building data pipelines? i've seen a company hired junior to do data architect work despite lacking experiences as data engineer which i think kinda weird.
Great video Ben! This is an important analysis that we must do consistently as Data professionals, to clearly understand what our responsibilities are! Yesterday I was reading Oreilly's book - Fundamentals of Data Engineering - and as you shared It seems that the Data Architect is closer to the business than the Data Engineer. And if we clearly understand the difference between the two roles it will be more productive for both. I'm currently working as a Data engineer, but I'm trying to thriller a path to becoming a Data Architect. Thanks for sharing your thoughts
Should I study IBM's data engineer certificate or study each cloud, SQL, python, ... certificate on cousera? Everyone give me advice. I learned basic C, Python and SQL in college.
Hey Ben, I'm currently a data analyst who is hoping to get into a data architect role. Would you say that the pathway to get there is to first become a data engineer to get more experience, then transition to a data architect?
Do data engineers always focus on pipelines for data analytics? Or is there a role more geared to improving/automating data business processes and the business’s overall data infrastructure?
Widely educated and experienced data architects and data-focused enterprise architects tend to focus more on improving and automating data business processes. Enterprise architect might collect information, draft blueprints, and manage changes to a) business architecture, b) data architecture, c) application architecture, d) technology architecture. Examples of blueprints might be a) business process and task workflow models, b) data asset to system mappings and concept models, c) data platform reference model and data warehouse implementation architecture, d) technology standards handbook and datawarehouse technology architectures. When I have done these, its typical that you have or it is useful to mix a few levels together. For example, bundle application architecture and technology architecture to a same blueprint or business and data architecture. In practice, the same Visio- or ppt-diagram includes application level items (Dynamics CRM, Azure data lake, PowerBI reports) and technologies (REST API between CRM and Azure, MS Stack as enterprise standard, DAX syntax in reports etC). Or combine business process items (sales, contracting, customer service, etc) to data layer (entities like consumer customer, sales product, service contract etc) to map their relations. Depending on the skills and background of the architect, they might work more on strategic level focusing on high level resources, policies and practices or more on operational and technical level like modeling logical data models and gathering detailed requirements for software projects. According to my experience, data engineers do not really work directly with business, draft business blueprints and take part of organizational implementation tasks - business layer. Unless organization quite a small and the same person has to be enterprise architect, data engineer and data analyst at the same time. In general, data engineers are most likely not expected to design business processes or strategies, for example.
Hey mang , love your channel , I been working as a Data analyst in financial services for the past year . I have no rush to become data engineer but if it happens happens , which Course would you recommend That covers most of the skills A data engender needs ? Thanks 🙏 .
Hi, I am thinking of transitioning into tech from a mechanical engineering career. Would you recommend any graduate degree to get into data engineering?
For me, I has been forced by the project to be defacto data architect. I was Lead Data Engineer but we were implement our cloud data platform project and at the time I am the only senior DE, the rest of my team is all junior. So the job just force me to be an architect. Now my position still called DE but I rarely built pipeline myself. I am more like a team lead + architect role most of the time design the high lv solution, do some POC, give consultation to the juniors and a lot of meetings!
Yeah as Lead Data Engineer I am trying to avoid this reclassification as I really love not only to design, but also to develop the stuff
In my opinion that's how someone ought to become a data architect. You should be a senior (or lead) data engineer first. When hiring someone as a data architect who hasn't worked as a data engineer himself you might end up with the architect taking decisions that aren't aligned with the needs of the engineers which slows down the entire project. Software architects aren't data architects.
Thanks for not showing ads. You’re the best
I don't have complete control of that. Googles new rules now state that they can show ads whenever they want and if you chose not to collect the youtube payout then they'll just take all of it.
Great video! Thanks a lot
Thanks, Ben! Would you make a video about data quality? I found myself recently using a library called Great Expectations to test the quality of data produced by my pipelines. Also a video about data catalogs would be cool. Would be great to hear your thoughts on these topics!
Great suggestion! I actually am working on an article on the topic so it might be a good idea to make a video part as well!
Are data architect role meant for someone who has vast experienced in building data pipelines?
i've seen a company hired junior to do data architect work despite lacking experiences as data engineer which i think kinda weird.
Great video Ben! This is an important analysis that we must do consistently as Data professionals, to clearly understand what our responsibilities are! Yesterday I was reading Oreilly's book - Fundamentals of Data Engineering - and as you shared It seems that the Data Architect is closer to the business than the Data Engineer. And if we clearly understand the difference between the two roles it will be more productive for both.
I'm currently working as a Data engineer, but I'm trying to thriller a path to becoming a Data Architect.
Thanks for sharing your thoughts
Should I study IBM's data engineer certificate or study each cloud, SQL, python, ... certificate on cousera? Everyone give me advice. I learned basic C, Python and SQL in college.
Hey Ben, I'm currently a data analyst who is hoping to get into a data architect role. Would you say that the pathway to get there is to first become a data engineer to get more experience, then transition to a data architect?
Do data engineers always focus on pipelines for data analytics? Or is there a role more geared to improving/automating data business processes and the business’s overall data infrastructure?
Widely educated and experienced data architects and data-focused enterprise architects tend to focus more on improving and automating data business processes.
Enterprise architect might collect information, draft blueprints, and manage changes to a) business architecture, b) data architecture, c) application architecture, d) technology architecture.
Examples of blueprints might be a) business process and task workflow models, b) data asset to system mappings and concept models, c) data platform reference model and data warehouse implementation architecture, d) technology standards handbook and datawarehouse technology architectures. When I have done these, its typical that you have or it is useful to mix a few levels together. For example, bundle application architecture and technology architecture to a same blueprint or business and data architecture. In practice, the same Visio- or ppt-diagram includes application level items (Dynamics CRM, Azure data lake, PowerBI reports) and technologies (REST API between CRM and Azure, MS Stack as enterprise standard, DAX syntax in reports etC). Or combine business process items (sales, contracting, customer service, etc) to data layer (entities like consumer customer, sales product, service contract etc) to map their relations.
Depending on the skills and background of the architect, they might work more on strategic level focusing on high level resources, policies and practices or more on operational and technical level like modeling logical data models and gathering detailed requirements for software projects.
According to my experience, data engineers do not really work directly with business, draft business blueprints and take part of organizational implementation tasks - business layer. Unless organization quite a small and the same person has to be enterprise architect, data engineer and data analyst at the same time. In general, data engineers are most likely not expected to design business processes or strategies, for example.
Hey mang , love your channel , I been working as a Data analyst in financial services for the past year . I have no rush to become data engineer but if it happens happens , which Course would you recommend That covers most of the skills A data engender needs ?
Thanks 🙏 .
Can you suggest where I can learn Data Engineering courses?
Yeah! I actually have a video on this, the courses are still relevant - ruclips.net/video/kW8_l57w74g/видео.html
By the way what is the song in the end of the video?
Hi, I am thinking of transitioning into tech from a mechanical engineering career. Would you recommend any graduate degree to get into data engineering?
Thanks a lot
Happy to help