Hello, I was trying to apply your suggestion about partition pruning on the existing table, by using the distinct values coming from the partitioning column of the incoming table to be merged, I wanted to do this dinamically but I found a corner case: in case I have an historization table that I'd like to update, if one new record has a new date (i.e. 2024), and the old version of that record was created in 2023, my merge condition "hist.BK_id == incoming.BK_id AND partition_year in (2024)" wouldn't allow me to update (flag is_current = False) the old record... I end up having duplicates and I cannot figure out an efficient way to include the partition without doing another expensive lookup. Do you have any suggestion for this use case?
The pace, tone and fairly detailed slides made this talk so pleasant to tune in. Thanks Justin! 👏👏👏
Great talk Justin! Learned a lot of new things
I don't understand how pruning on the left has any effect because the left is the source of the distinct dates in the first place
Very informative Justin
Gracias Justin , esta explicación está excelente
Great talk
Can you please share the deck or the KB articles shared in the slides?
Just because u get a separate bucket for atable doesn’t mean that the bucket will be in its own partition. S3 will scale
Is the slide deck available as a PDF somewhere?
Hello, I was trying to apply your suggestion about partition pruning on the existing table, by using the distinct values coming from the partitioning column of the incoming table to be merged, I wanted to do this dinamically but I found a corner case: in case I have an historization table that I'd like to update, if one new record has a new date (i.e. 2024), and the old version of that record was created in 2023, my merge condition "hist.BK_id == incoming.BK_id AND partition_year in (2024)" wouldn't allow me to update (flag is_current = False) the old record... I end up having duplicates and I cannot figure out an efficient way to include the partition without doing another expensive lookup. Do you have any suggestion for this use case?
2:32 I like that😍💋 💝💖❤️