data engineer interview questions
HTML-код
- Опубликовано: 1 окт 2023
- In this video I have talked about salting in spark
Directly connect with me on:- topmate.io/manish_kumar25
Discord channel:- / discord
Project details for resume :-
.Successfully led a data engineering project in a retail environment using technologies such as Apache Spark, Python, SQL, and Amazon S3 to optimize data processing.
.Implemented structured data models, including dimension and fact tables, to provide valuable context for point-of-sale data analysis.
Designed and executed an incentive program based on sales performance, enhancing motivation among sales teams by rewarding top performers.
Managed extensive daily data volumes of approximately 100GB, demonstrating the ability to handle large-scale data pipelines.
Employed Spark optimization techniques like caching and broadcast joins to improve data processing speed and efficiency.
Utilized Azure CI/CD pipelines for code deployment, and orchestrated workflows using Airflow and CRON jobs.
Detailed writeup to explain more during interview:-
As a Data Engineer on a project for a prominent offline grocery and kitchen supplies retailer, I applied my expertise in data engineering to drive critical improvements in their data processing and analysis operations.
The project primarily focused on processing and analyzing point-of-sale data, which was structured into dimension and fact tables to provide meaningful context for sales analysis. To further enhance employee motivation and performance, we designed and implemented an incentive program that rewarded salespeople with the highest sales volumes in each store.
Handling a substantial daily data volume of approximately 100GB, we leveraged Apache Spark and applied optimization techniques like data caching and broadcast joins to significantly accelerate data processing. This not only improved the speed of our data pipelines but also increased the efficiency of our data analysis.
We seamlessly integrated the code deployment process into the Azure CI/CD pipeline. As part of workflow automation, we orchestrated task scheduling using Airflow and CRON jobs.
One of the project's major achievements was the implementation of a customer engagement strategy that identified infrequent buyers and provided incentives in the form of coupons. This initiative not only boosted customer retention but also had a positive impact on the overall business growth.
For more queries reach out to me on my below social media handle.
Follow me on LinkedIn:- / manish-kumar-373b86176
Follow Me On Instagram:- / competitive_gyan1
Follow me on Facebook:- / manish12340
My Second Channel -- / @competitivegyan1
Interview series Playlist:- • Interview Questions an...
My Gear:-
Rode Mic:-- amzn.to/3RekC7a
Boya M1 Mic-- amzn.to/3uW0nnn
Wireless Mic:-- amzn.to/3TqLRhE
Tripod1 -- amzn.to/4avjyF4
Tripod2:-- amzn.to/46Y3QPu
camera1:-- amzn.to/3GIQlsE
camera2:-- amzn.to/46X190P
Pentab (Medium size):-- amzn.to/3RgMszQ (Recommended)
Pentab (Small size):-- amzn.to/3RpmIS0
Mobile:-- amzn.to/47Y8oa4 ( Aapko ye bilkul nahi lena hai)
Laptop -- amzn.to/3Ns5Okj
Mouse+keyboard combo -- amzn.to/3Ro6GYl
21 inch Monitor-- amzn.to/3TvCE7E
27 inch Monitor-- amzn.to/47QzXlA
iPad Pencil:-- amzn.to/4aiJxiG
iPad 9th Generation:-- amzn.to/470I11X
Boom Arm/Swing Arm:-- amzn.to/48eH2we
My PC Components:-
intel i7 Processor:-- amzn.to/47Svdfe
G.Skill RAM:-- amzn.to/47VFffI
Samsung SSD:-- amzn.to/3uVSE8W
WD blue HDD:-- amzn.to/47Y91QY
RTX 3060Ti Graphic card:- amzn.to/3tdLDjn
Gigabyte Motherboard:-- amzn.to/3RFUTGl
O11 Dynamic Cabinet:-- amzn.to/4avkgSK
Liquid cooler:-- amzn.to/472S8mS
Antec Prizm FAN:-- amzn.to/48ey4Pj
bhai iss video se mein fan hogya aapka. "logon ke pass experiance nahi hain, aur company ko experiance chahiye" 🔥 🔥
What a gem content sir 🥺 thankyou so much for in-depth video!
Hey Manish, I'm extremely thankful to you and all of your playlists. Especially this video is super problem solver one! No one teaches in so much depth as you do.
@Manish Kumar. All of your videos are more than a gem if anything exists like this. I am 4-5 YOE and never get to learn spark in such a depth , clarity , concise answers , questions. It is useful for 10 YOE as well I can vouch for it. I have ADHD issue, but your videos are too engaging that I can sit for long with it. I have got interested in learning. You must be an extra ordinary guy. Having knowledge is one thing , presenting it , putting it in so simple manner is what stands you apart. It is very difficult to be simple . Thanks once again
Sir your amazing. No one has created content till now on this.Wish to see more on this type of content .Being a fresher we need to have a clear idea about how the project works and we should know how to explain project to interviewer.
Thanks a lot manish bhaiya, u listen to even individual request. Big thabks to you. Loving your content.
Bhai apko salute, ekdam sidha, saaf or sach bolne ke liye
Wow... Manish bhai really loved this content. Please, I will encourage you to do more videos like this.
Thanks for amazing content . Spark playlist is amazing
This whole PlayList helped a lot.💡
Great Manish you don't say fake e,Xperince many times in your video your doing great job
bhai bhai 🙌...ultimate video ❤❤
Hey Manish! I am following all playlists and content, also I have given more than 30 interviews but have not been selected yet because of a scheduler if you can cover any of them or can cover a pipeline including airflow or any one schedular it will be very helpful. Without schedular knowledge, it's incomplete because each and every interview they are asking for it. You are explaining very well so I want to have an explanation in your depth knowledge. Thanks.
Waiting for 2nd part eagerly , related to last project. Please next time usi ko upload krna.
Thanks for the session!!
Thank You manish bhaiya
thank you so much bhaiya for amazing content 💝
most exciting video
Great video Manish .. What you have face problem while doing your projects and how to resolve it . Please answer this question as experience person.
Amazing...!!!