not clear how this process would handle what happens if your source query, for silver in this context, though might be more relevant to gold, uses something like an aggregate, which dlt streaming doesn't like and you may have to fully materialize a table instead of streaming
We tried that just recently. Depending on how you approach this it may not work. In our case, we did not always call the DLT with the same tables to be processed. Any table that was processed previously, but not in a next run would be removed from unity (though the parquet files still exist - ie behavior like an external table). This is of course not acceptable, so we switched to meta data driven structured streaming. To put this a different way, if you call the pipeline with table a, then call it again with table b, table a is dropped. You'd have to always execute the pipeline with all tables relevant to the pipeline.
@@brads2041 you reload onboarding before each run to add or remove tables from group. So workflow might be: onboarding(can refresh each row addition removal for tables a,b) -> DLT Pipeline
not clear how this process would handle what happens if your source query, for silver in this context, though might be more relevant to gold, uses something like an aggregate, which dlt streaming doesn't like and you may have to fully materialize a table instead of streaming
Does this support Unity Catalog?
yes. the recent release support UC.
Can we have a video for loading multiple tables using single pipeline
I mean something of a for each activity
@@rishabhruwatia6201 you can check repo dlt-meta and check dlt-meta-demo or run integration tests
We tried that just recently. Depending on how you approach this it may not work. In our case, we did not always call the DLT with the same tables to be processed. Any table that was processed previously, but not in a next run would be removed from unity (though the parquet files still exist - ie behavior like an external table). This is of course not acceptable, so we switched to meta data driven structured streaming. To put this a different way, if you call the pipeline with table a, then call it again with table b, table a is dropped. You'd have to always execute the pipeline with all tables relevant to the pipeline.
@@brads2041 you reload onboarding before each run to add or remove tables from group. So workflow might be: onboarding(can refresh each row addition removal for tables a,b) -> DLT Pipeline
would appreciate if databricks comes with a proper explanation ...both the tutor's explanation aren't clear