Great explanations! One quick question, in dbt we can test/profile our data during 'T' (Transformation) and also run test for source data. How is 'Great Expectations' different than the features already built in dbt?
Frankly it's unnecessary to incorporate Great Expectations into the stack. You could indeed just replace it with dbt tests and be done. There are just a few things to consider when choosing between these two: - Great Expectation tests are written in Python while dbt tests are written (mostly) in SQL syntax. You might find a particular test to be easier to implement one way or another. Personally I think in 99% of cases dbt test cover everything you need. - Great Expectations provides more kinds of tests "out of the box". But to counter this point, there is the dbt-expectations package that provides out of the box tests as well. - [This one is huge!] Great Expectations requires you to load data from the database into your Python processing environment while dbt tests work within the database. So Great Expectations tests tend to be more expensive and also take longer to complete than equivalent dbt tests. - Great Expectations produces these data quality reports. Perhaps they are important to your organisation. I'm sure there are arguments for using Great Expectations as well. You certainly could use both as the talk suggests. I just couldn't think of a good use case. The talk didn't convince me but in its defense it's over been over 2 years. Perhaps dbt tests and the dbt-expectations package had less features at that time. Or perhaps they had a legacy codebase with Great Expectations test before they adopted dbt and didn't want to re-implement these tests. In any case my recommendation would be to ignore this advice and just go with dbt tests in most cases.
appreciate your simple explanations , thanks!
Great explanations! One quick question, in dbt we can test/profile our data during 'T' (Transformation) and also run test for source data. How is 'Great Expectations' different than the features already built in dbt?
Frankly it's unnecessary to incorporate Great Expectations into the stack. You could indeed just replace it with dbt tests and be done.
There are just a few things to consider when choosing between these two:
- Great Expectation tests are written in Python while dbt tests are written (mostly) in SQL syntax. You might find a particular test to be easier to implement one way or another. Personally I think in 99% of cases dbt test cover everything you need.
- Great Expectations provides more kinds of tests "out of the box". But to counter this point, there is the dbt-expectations package that provides out of the box tests as well.
- [This one is huge!] Great Expectations requires you to load data from the database into your Python processing environment while dbt tests work within the database. So Great Expectations tests tend to be more expensive and also take longer to complete than equivalent dbt tests.
- Great Expectations produces these data quality reports. Perhaps they are important to your organisation.
I'm sure there are arguments for using Great Expectations as well. You certainly could use both as the talk suggests. I just couldn't think of a good use case. The talk didn't convince me but in its defense it's over been over 2 years. Perhaps dbt tests and the dbt-expectations package had less features at that time. Or perhaps they had a legacy codebase with Great Expectations test before they adopted dbt and didn't want to re-implement these tests.
In any case my recommendation would be to ignore this advice and just go with dbt tests in most cases.
@@personalbranddata Thank you!
This is like making a small problem Very Big then creating a Giant Open Source to solve it!
Interesting challenge, at times, to test raw source data because you may have to apply some form of transform to get it into a testable state.
Would love to see a repo for this!
A bit late, but we got you: github.com/spbail/dag-stack
Nice intro.
Code? Repo ?
Is there a repo to refer ?
How to create dbt pipeline?
no repo?