Видео

So which rest catalog is the best choice for an Enterprise?

There are a whole lot of assumptions made here that, in my experience, just are not true. Eg. processing is almost always much more expensive than storage, so no cost savings there by saving on space at the expense of processing. Also, the warehouse is not a business model at all. Seems this is from a very narrow perspective of someone who focuses mainly on data science, rather than on data engineering. There is huge value in having a centralised location for data where the definitions are the same, the data is enriched from many sources into a cohesive and unified view of the data, much less duplication of data (so the marginal cost savings of space in this presentation are nullified anyway). The main reason that this approach is popular, at the moment, is that people don't have to wait so long to access the data and they can run off an do their own empire building in a silo. This is great for the individual, but not great at all for the business as a whole. Data scientists are used to mass duplication of data and massive processing costs in building their models, so this is normal for them. There is big trouble ahead for those who are designing everything around the very specific needs of machine learning. That is only a small part of the whole.

I have a pyspark glue job in aws, that is charge of compacting my iceberg table, it is using iceberg procedure to compact the table. It's job has been taking more than 4 hours running, i have been getting timeout, is there any efficient way to compact a table without spark ?

This conversation is totally biased because you guys are taking assumptions Iceberg is the de facto standard. And this is a narrative the Iceberg community is pushing in total disregard for what happening: - AWS support for Delta and Hudi was released in Redshift Spectrum before the support for Iceberg. When Iceberg support was added last year it was in read only. - Snowflake support Iceberg in read/Write and Delta in read only. - GCP BigQuery support both Iceberg and Delta - Oracle support Delta and Delta Sharing - Salesforce doesn’t only support Iceberg but also Delta and have Zero ETL integration with the Databricks platform. - Microsoft Fabric is built on top of Delta - OneHouse which is contributing a lot on Hudi released the Apache XTable to handle this interoperability between the 3 tables format and they’re working with Databricks on the Delta UniForm capability - and the list goes on… Databricks have more open track records than Snowflake, who only recently started to open up to the OSS community as a desperate move to stop the customer moving to Databricks. Delta with the Delta Kernel makes it easier to integrate with the Delta spec and support in a more uniform way Delta capabilities. In Iceberg one query engine might support version 1 and some capabilities of Version 2, or another only partially supporting version 2, which bring is set of challenges for customers! Iceberg won on the community side against the 2 other formats, and have used that to gain more momentum. But features wise, there are more innovations happening on the 2 other formats, than on Iceberg. Iceberg has some great features, but is that plus having a bigger community enough to claim it’s the de facto standard? Not sure about that. It might be time for the Iceberg community to stop playing politics and works with the 2 other communities to end this table format war, which will be the best way moving forward to have total interoperability. Do you guys remember the time when different cellular operator was using different technology, and peoples was having different phones to be able to contact friends, family and colleagues. Then when GSM was adopted, how easy it became moving from operator or from one state to another or traveling abroad? Now nobody thinks about it and we’re all using GSM world wide. Why can we have the same in data engineering?

Newbie here! Would a catalog be equivalent to the database schema?

Other than the Iceberg open data format I was unable to notice anything else I could reference as “open” from the suggested solution architecture.

Well, Iceberg is a spec, not a piece of software. It existed before Tabular and will exist after the acquisition. I'm really scratching my head about its long-term goal. I also wonder what Tabular customers have been told and if that product will continue to exist.

pls bro fix your mic, btw good video

Could you please show the reverse process, i.e., from athena to kafka cluster.

Great to hear these amazing announcements from Ori and Yoni about Upsolver's new features to support Apache Iceberg, announced at the Chill Data Summit NYC. What a brilliant first event 🎉

should we create real-time streaming data solutions?

How is this different than snowflake data warehouse and sql?

Hello can we have the datasets??

I was exploring this tool and this tools looks good. But, there is no sufficient materials online. I think more quality videos are needed. A good demo with various examples are mandatory.