10 best practices for building Data lakes| Evolution of data lakes

This video talks about What is a data lake, evolution of data lakes and the 10 best practices to keep in mind while designing and developing data lakes
What are Data lake
• What are Data Lakes | ...
How to build healthy data pipelines
• How to Build healthy d...
Disaster Recovery
• Disaster Recovery in 1...
Chaos Engineering
• What is Chaos Engineer...

[@amriteshsingh2952]

Thanks for the great content, appreciate your work.

[@BigDataThoughts]

Thanks Amritesh

[@GauravSharmagvs]

Thanks for the video. It was very informative.

[@BigDataThoughts]

Thanks gaurav

[@arunalex1988]

Learnt few new dimensions related to Data Lake. Thank you so much for sharing your wealth of knowledge dear Shreya.

[@BigDataThoughts]

Thanks arun

[@TotuBabyBird]

❤

[@Learn2Share786]

Great content covered in right pace. Thank you!

[@BigDataThoughts]

Thanks Farhad

[@Praveen_Kumar_R_CBE]

Very useful content Shreya…

[@BigDataThoughts]

Thanks Praveen

[@ranjithpals]

Lots of great points you have highlighted and touched upon, as always very useful !! Thanks Shreya

[@BigDataThoughts]

Thanks Ranjith

[@kiranmudradi26]

Awesome video. Madam, can you please make video on how to tackle system design and what tools to use for building pipelines both in inhouse/cloud? It would be helpful.

[@BigDataThoughts]

Thanks Kiran sure

[@nikhilgupta110]

Very well described on concepts standpoint, One example bridging the concept would have been a value addition.
Please do suggest a book/articles for a deep drive as well.
I have few queries, it would be really helpful if you can provide some thoughts on that:
1. Let's say we are having a common batch pipeline for multiple customers from different data sources (s3, mysql etc. ) , One Identifying timestamp column as YYYYMMDD as it has frequency of 24hrs.Now if I want to convert it to real time ELT, what are the things to keep in mind?
2. Extending the above question, if we are running parallel ETL batches for let's say 30 customer each. What is the best optimization strategy, To run ELT jobs parallel for each customer (best performance) or Increase the nodes run the jobs? Which is the best way to scale keeping cost as a constraint?

[@BigDataThoughts]

1. From batch to Realtime it has to be a complete architecture and technology choice change. you need to also check if its real time ingestion and processing or real time consumption as well. This would determine how you are designing the system.
2. Orchestrating jobs in parallel , in sequence or a mix depends on many factors:
a. SLA
b. Is each flow independent or has dependency
c. Cost
d. Consumption pattern
you may not need to have all flows in parallel with higher nodes (as its extra cost) if the data isn't interdependent or doesn't has same SLAs