Jarred from Dagster here. Thanks for the question - We are still prioritizing which platforms to support next, and we are taking input on which integrations and metrics are the most valuable. Databricks is on the list, but no ETA on when that will be released. Dagster Insights is customizable, so you can bring in data from your own Databricks instance.The biggest challenge for Databricks is mapping how it accounts for cost (which is focused on instances and run time, and not on a per query basis) to a Dagster run.
Is there a plan to provide out-of-the-box insights gathering for Databricks as well or will that be something that the community (i.e. us) will need to create?
Jarred from Dagster here. Thanks for the question - We are still prioritizing which platforms to support next, and we are taking input on which integrations and metrics are the most valuable. Databricks is on the list, but no ETA on when that will be released. Dagster Insights is customizable, so you can bring in data from your own Databricks instance.The biggest challenge for Databricks is mapping how it accounts for cost (which is focused on instances and run time, and not on a per query basis) to a Dagster run.
Is there a plan to provide out-of-the-box insights gathering for Databricks as well or will that be something that the community (i.e. us) will need to create?