Видео

Is that hive table sub partioned on date string ? And then further bucketing.. would drastically reduce those 3 million files ... Without much info on that structure could not agree on performance issues in this example..

the audio is very very soft. even at 100%

the audio is very very soft. even at 100%

Can you do more of an intro ? Telling people what your tools does ? Tool looks amazing but it was only at the end I found out what it did.

Audio is too low, can't hear much even at max volume and it's crackled

Great content as always! Can you do some demo talking about the engines inside dremio config?

Thanks Alex! I have been watching most of your video and blog tutorials. You have really improved my knowledge in Datalakehouse, Apache Iceberg, and Dremio. However, I can't find any detailed enterprise BI cloud solution in Dremio Solar and Arctic. Can you please take us through an enterprise cloud BI solution using the Azure Data Lake Gen2 as the storage layer? Otherwise, could you point me to an existing blog on the Dremio website? Regarding this tutorial, do I really need Spark since I can accomplish all my tasks in Dremio? Cheer!

So to understand/use iceberg you also need 20 other components on top of running hadoop and all of the things that that requires? Seems like an... "improvement"? aka. Hadoop should have never been turned into a data warehouse to begin with.

Very poor sound quality!

First comment

Very nice and concise 😂

Great video ❤

Very nice tutorial, am beginning to see how branching is useful

This was super helpful!

Amazing ---

Instructor talks too fast that difficult to cop with specially for english as a second language speaker. I advice the instructor to slow down to be able to convey his message clearly , after all thank you for the architectural lecture about data lakehouse as it is a confusing term for many of us.

Pretty awesome talk! I would love to see geospatial formats be supported in Apache Iceberg, but I'm glad that this fork exists!

There are so many tools used here my head hurts. For "getting data into dwh" how can this possibly be seen as better than classic parque files in hive-partitioned storage with an external table on? In gcp this is 5 lines of sql code, in redshit you might have to add Glue to the mix. And then any sql-running tool can do the rest inside the database/dwh. In 99% of companies theres at best 3-4 data engineers, managing all this + data modelling would mean nothing ever gets done.

Alex is the man - explains these things and the context so well.

Excellent overview - thanks! - My question is about how best to get that first "Raw Copy" into Iceberg. Using the SQL Connector I can hook Dremio up to MS-SQL data sources, and they are then available for reflections - but I don't see any clear way to copy the data into an Iceberg table that would persist (in the case that the source data became unavailable) am I missing something?

How does Dremio support the Iceberg Rest Catalog........I cannot seem to find a way, other than Nessie to create an S3 resident Iceberg table in Dremio......

Why is the spark configuration with all of the lakehouse services hardcoded in a notebook? Shouldn’t these configurations be incorporated into the docker image you’re using for Spark?

This is so nice. Now I dont have to pay for databricks in order to learn Spark!

why did they create manifest list & manifest files as two separate layer? They could have just created metadata file & manifest file for each snapshot. What challenges we could have if we don't have manifest list in iceberg design

❤

Thanks for the tutorial. If I use CDC based ingestion as the data source, where the Spark Job for writing Iceberg Table ( steamWrite) runs on ? Is it Inside Airbyte?

Thanks!!

This was really helpful, thanks!

when we expire the snapshot fi our table created copy-on-write ot merge-on-read the what happen in that case.

The infrastructure is knowledgeable and able to address the new technology in a detailed pace. the challenge that I can see the instructor's pronunciation is not clear in some points. I advise that the instructor articulate the idea in clear pronunciation, as sometime he is too fast and some times too slow.

great presentation! just one question. In dremio, which would be the case scenario where a view WON'T need reflections activated? (and view has many columns and high volume)

Pregunta. ¿Necesitas realizar alguna conexión desde Dremio hacia Docker? A mi se me presenta el error Invalid staging location provided Al momento de cargar los archivos. ¡Por favor, ayuda!

⚠⚠Pregunta. ¿Necesitas realizar alguna conexión desde Dremio hacia Docker? A mi se me presenta el error Invalid staging location provided Al momento de cargar los archivos. ¡Por favor, ayuda!

⚠⚠Pregunta. ¿Necesitas realizar alguna conexión desde Dremio hacia Docker? A mi se me presenta el error -Invalid staging location provided- Al momento de cargar los archivos. ¡Por favor, ayuda!

How do you persist the data using volumes? When container is deleted I don't want to recreate everything. Please send me link to solution if there is one.

nice overview

🐬 fan already, next please show their corporate headquarters

🔥

Your voice is like an angel to fall asleep😇

Hi thanks this is really a great information to start with Apache Iceberg. But I have a question, when modern databases are already doing it with so much advance technology to prune and scan the data, why would we need to store the data in files format instead of directly loading them to a table ?

Thanks for the great video. Question: when we first the DELETE command in the lesson2 branch, does the data also appear in minio ? Like, does minio object storage shows both lesson2 branch and main branch separately ? I am curious this because on minio, there is only data and metadata partition, and there is not directory for main vs lesson2 branch.

Thank you so much! I have a question. I'm wondering if there might be any way to do these procedures automatically in Iceberg. Do I have to do these things in person every time?

it's really helpful for me!! Thank you so much

Hi Alex, Really tankful to you for such nice explanation and handson. I got stuck at 'CREATE BRANCH IF NOT EXISTS lesson2 IN nessie' . This keeps failing with error message "syntax error at or near 'BRANCH'". Am I missing something? Kindly assist.

Awsome Video !! At 3:18 when explaining different delete format I have question regards to the implementation : As the delete mode only accept MOR or COW , how exactly do I specify the delete operation to use Equality delete or Positional delete ??

very well explained...great job Dipankar

Great explanation and details

Great article Alex. Slight issue creating a view in Dremio, I get the following exception "Validation of view sql failed. Version context for table nessie.names must be specified using AT SQL syntax". Nothing obvious in the console output, any ideas?