I don't understand the claim to get rid of the combinatorial problem. In one approach the measure is path coverage, in the other it is equivalent classes. Comparing apples to oranges. And he doesn't even mention the downside of tightly coupling tests to implementation, instead of testing actual business logic specifications. In the end it always depends on multiple aspects whether integrated or isolated tests are more effective.
Go to his Code Whisperer site and have a look at his blog posts "Clearing Up the Integrated Tests Scam" (2015) and "Integrated Tests Are A Scam" (2021 Update)-you may find some clarification there. In fact the site has a series navigation and there are a number of articles under the "Integrated Tests Are A Scam" header that may address some of your concerns.
@@trignals It's a bad title and he ends up contradicting it. But because of talks (titles) like this I have to regularly argue with junior devs (and some senior devs) that they can't just rely on unit tests passing before they ship their code.
@@MarkArcher1 I agree the title doesn't reflect the nuance of the whole presentation. That being said your comment gave me the wrong impression when I read it. I can better see where you are coming from now though. If colleagues base their opinions on talk titles (ever) I would still say that's the problem, not the titles.
@@trignals fair enough. But I think the whole talk has problems. What he's saying might be true in the software domain he works in but it definitely isn't in the one I work in. Our integration tests are just as important as our unit tests in our CI pipeline.
@@MarkArcher1 Ok, you've tried to engage with it and found it doesn't work for you, that's definitely a perspective worth highlighting. If you think it might be domain related it would be relevant to mention either the domain or what seems particular about it. I don't know what domains JB or you have experience in so I really cannot comment other than to say it sounds like it could be interesting. Maybe others have had the same experience or found it worked in that kind of domain.
My hex/clean architecture tests are basically his def of integration tests. I only test input ports and only mock output ports. I much prefer it my way. My testing is low effort, high impact. My tests are super fast and refactoring is a breeze.
I have experienced with integration tests, bugs are hard to find where they fail exactl, ofc less preasure on how the code is designed for sure, less test coverage and take more time to write as well
Interesting talk, I wonder if you use pure values and no side effects inside your happy zone then you wouldn't even need these mocks/stubs and contract / expectation tests instead you will have simple unit based tests all the way through to the boundaries of your system. An imperative shell around a functional core.
I recommend this talk that has a succinct example of rewriting code to a functional core and moving I/O side effects to the top, ruclips.net/video/DJtef410XaM/видео.html. Now whenever I am tempted to add a test double, I think about whether I'm testing high level i/o or am avoiding isolating side effects.
Integration tests, not integrated. To test the integration of different components - that they are wired up correctly, not their logic, let alone design. Don't have to be end-to-end, just 2 components interacting is a get. The traversty he refers to has more to do with microservices than with tests.
Seems like the "Contract Tests" that he is referring to at 37:02 are just the "Isolated Tests" of the next layer at least I don't see any difference. Because the next layer obviously also implements some sort of an interface just like any other layer and probably has its own set of collaborators that it depends on hence would also require "Test Doubles", etc. Unless the next layer is the last layer which is talking to the outside world (a.k.a on the edge of the ring) which would require "Integration Tests" So I don't understand the notion of "Contract Tests" or the need to outline them differently as if they are something special.
it is impressive how this is the representation of the first conference in 2009 and it was 10 years ago, but still when I say this I just found a colleague that knows it, most of the colleagues in many different companies doesn't even know about it
look, if you need to unit-tests to clean up your design, then go for it. and if you insist upon hating code which cannot be unit-checked, then you have my blessing. ... BUT ... i don't need tests to write clean code. i need tests to perform checking. JB said it himself: mocks don't always behave like the things they stand in for and these are precisely the sorts of bugs that integration-tests catch. even code which is unit-checkable should be checked with integration-tests.
That sort of bugs is caught by contract tests. You could see contract tests as a kind of integration tests, but that is not the kind of integration tests JB is talking about.
@@am0e Actually the vast majority of code paths come from combining different paths through two or more modules, so how do you cover them with unit tests?
@@jonathanccast unit tests orchestrated together can become integration tests, but I don't see how they can as individual units. Mocks also don't become full implementations, but they can certainly reflect a good deal. Also I feel like we get so far off the mark by calling any of this "tests", these are just examples and will never cover the entire scope of possibilities (Gary Bernhardt has a great talk on this).
Honestly it’s both incredibly funny and extremely unsurprising that when faced with a legitimate code question, both this guy AND uncle Bob gave up and couldn’t answer the question. Because the longer I listen to both people talk, the more I’m convinced they’re not actually *saying* anything. They keep talking in abstracts and using these circular reasoning arguments to justify their stances. But they kinda don’t make sense. Like no matter how many layers of abstraction you have between one module and another in your code, within an application environment, they DO have to somehow communicate to one another. And it doesn’t matter how many isolated unit tests you have - if that communication breaks down, your application isn’t working. Testing those connections, in my experience, is the most valuable place to start testing.
I agree that using an interface to the items that are external to our program is needed; however, I don't understand the necessity of having interfaces between items that are within our code.
Defining APIs is basic to composing code, whether it's internal or external. If you have worked for decades and seen code that is super messy to work in, you'll start appreciating how import it is to make sure internal APIs are super clear and are respected. Also, internal APIs can changes much more easily than external, so there's even more value in internal contracts than external.
15 minutes into this video he argues as follows: 1. 100% tests passed (i.e. unit/isolated tests) but we found a mistake 2. that encourages us to write more integration tests 3. which allows us to design more sloppily 4. which provides more opportunities for mistakes point 3. above assumes the thing he wants to prove, which is the very definition of a circular argument.
Did you watch the whole talk? He's talking about the idea that instead of fixing the bug on the unit level and basically testing all possible outcomes with few simple tests, you write an integrations test on a higher level. An integration test has many dependencies and combinatorial explosion occurs, because you can't possibly test all the outcomes. That's when other bugs come from because the functionality has not been properly tested by the integration tests.
It's not circular. There's just a bit missing. Isolated tests shed light on your design problems, while integrated tests don't. THAT is what allows you to design more sloppily. He covers this in the two minutes at 8:14.
Only test through public interfaces, eng.libretexts.org/Bookshelves/Computer_Science/Programming_and_Computation_Fundamentals/Book%3A_Object-Oriented_Reengineering_Patterns_(Demeyer_Ducasse_and_Nierstrasz)/06%3A_Tests__Your_Life_Insurance/6.05%3A_Test_the_Interface_Not_the_Implementation
If your programming language doesn't provide a way to enforce contracts (interface), add that contract enforcing to your development process. At the end you can write your own contracts within the documentations and force yourself to respect them without actually needing some technology to enforce them. More or less on how an API works, just respect the contract
You can have RSpec tests for object contract. For example, create a shared_example that tests contracts (e.g expect(subject).to respond_to(:x)) and then include those contract, shared tests in the unit test file of subject. This means that whenever the contract changes, the unit tests would also fail.
So... you should write isolated tests. Because it encourages good design of systems. Which are systems that can be tested by isolated tests. Where have we established that isolated tests provide better design feedback? He keeps repeating "more integrated tests put less pressure on the design, so we design more sloppily" without any evidence for the proposition. This video is a scam.
I would definitely say that due to the fact unit tests (without testing implementation) require you to write more isolated code with the ability to mock dependencies leads to more modular code. Integration tests allow more flexibility. The quality of code under integration test is up to the discipline of the developer. This doesn't inherently mean integration tests are bad, but they allow for tightly coupled code. As for the argument about isolate tests providing better feedback, they by design identify the unit in which the failure occurred, this by design is isolated, this in my opinion provides a more precise identification of the problem area, but this isn't without exception, nothing in programing is.
There is an important argument that if you find it difficult to write the isolated test then you have bad design in the place which I agree to. This is the fast feedback loop which put pressure on you to rethink about your design which is not the case with end-to-end integration tests.
I kind of agree. Integrated tests is only a scam if it's the only base you rely on to validate your business... but they are necessary unless you have money to hire testers who will spend days manually testing everything... That said, a good base of isolated tests will highlight the weaknesses of your design early in the developing process therefore will force you to have a more modular design while respecting the business value of your application (and there is where TDD is powerful). You DO need integrated tests but only for validating the critical paths of your application in a integrated environment as much as you need the isolated tests. So yeah, the title of the video is controversial and by no means you should discard integrated tests because of that.
Disagree, here's what automated integration testing can do: ruclips.net/video/m5zrfTFKf_E/видео.htmlm32s This increases the quality of the end product, and confidence in each developer/designers ability to experiment (without worrying about whether a change introduced a bug) ...But maybe you'll classify the bot as "automated checking" which is arguing semantics. lets get the computers to do the (automatic) boring jobs and free up the humans to do more thoughtful exercises
It's simply an end-to-end test. It's nice to have one. It's not good to have thousands of them. In the case of the video you linked, they just have the happy path done to act as a kind of smoke test. You don't design your system with it tho, so it does not increase the quality. Just makes sure you didn't fuck up royally. Just follow the test pyramid.
This talk definitely goes to my Pantheon of the greatest talks I have ever experienced
This talk is very helpful! Watch it twice. Then come back six months later and watch it a third time. You will learn something new each time.
I watched it once, and saved it for next time too :)
I'm back after a few years and many books that put this into context
I don't understand the claim to get rid of the combinatorial problem. In one approach the measure is path coverage, in the other it is equivalent classes. Comparing apples to oranges.
And he doesn't even mention the downside of tightly coupling tests to implementation, instead of testing actual business logic specifications.
In the end it always depends on multiple aspects whether integrated or isolated tests are more effective.
Go to his Code Whisperer site and have a look at his blog posts "Clearing Up the Integrated Tests Scam" (2015) and "Integrated Tests Are A Scam" (2021 Update)-you may find some clarification there.
In fact the site has a series navigation and there are a number of articles under the "Integrated Tests Are A Scam" header that may address some of your concerns.
Spoiler alert, he eventually says you need a few integration tests.
*might need, in a specific location under certain circumstances
@@trignals It's a bad title and he ends up contradicting it. But because of talks (titles) like this I have to regularly argue with junior devs (and some senior devs) that they can't just rely on unit tests passing before they ship their code.
@@MarkArcher1 I agree the title doesn't reflect the nuance of the whole presentation. That being said your comment gave me the wrong impression when I read it. I can better see where you are coming from now though. If colleagues base their opinions on talk titles (ever) I would still say that's the problem, not the titles.
@@trignals fair enough. But I think the whole talk has problems. What he's saying might be true in the software domain he works in but it definitely isn't in the one I work in. Our integration tests are just as important as our unit tests in our CI pipeline.
@@MarkArcher1 Ok, you've tried to engage with it and found it doesn't work for you, that's definitely a perspective worth highlighting. If you think it might be domain related it would be relevant to mention either the domain or what seems particular about it. I don't know what domains JB or you have experience in so I really cannot comment other than to say it sounds like it could be interesting. Maybe others have had the same experience or found it worked in that kind of domain.
As soon as he was talking about interfaces the first thought in my head was clean (hexagonal, onion or whatever you wanna call it) architecture.
My hex/clean architecture tests are basically his def of integration tests. I only test input ports and only mock output ports. I much prefer it my way. My testing is low effort, high impact. My tests are super fast and refactoring is a breeze.
Thanks for the upload!
yes thanks for the upload because i was looking at it in Vimeo and is such a bad experience
I have experienced with integration tests, bugs are hard to find where they fail exactl, ofc less preasure on how the code is designed for sure, less test coverage and take more time to write as well
Wonderful talk. This video changed my concept of integrated tests.
Interesting talk, I wonder if you use pure values and no side effects inside your happy zone then you wouldn't even need these mocks/stubs and contract / expectation tests instead you will have simple unit based tests all the way through to the boundaries of your system. An imperative shell around a functional core.
I recommend this talk that has a succinct example of rewriting code to a functional core and moving I/O side effects to the top, ruclips.net/video/DJtef410XaM/видео.html. Now whenever I am tempted to add a test double, I think about whether I'm testing high level i/o or am avoiding isolating side effects.
Integration tests, not integrated. To test the integration of different components - that they are wired up correctly, not their logic, let alone design. Don't have to be end-to-end, just 2 components interacting is a get. The traversty he refers to has more to do with microservices than with tests.
Seems like the "Contract Tests" that he is referring to at 37:02 are just the "Isolated Tests" of the next layer at least I don't see any difference. Because the next layer obviously also implements some sort of an interface just like any other layer and probably has its own set of collaborators that it depends on hence would also require "Test Doubles", etc. Unless the next layer is the last layer which is talking to the outside world (a.k.a on the edge of the ring) which would require "Integration Tests" So I don't understand the notion of "Contract Tests" or the need to outline them differently as if they are something special.
it is impressive how this is the representation of the first conference in 2009 and it was 10 years ago, but still when I say this I just found a colleague that knows it, most of the colleagues in many different companies doesn't even know about it
look, if you need to unit-tests to clean up your design, then go for it. and if you insist upon hating code which cannot be unit-checked, then you have my blessing.
... BUT ...
i don't need tests to write clean code. i need tests to perform checking. JB said it himself: mocks don't always behave like the things they stand in for and these are precisely the sorts of bugs that integration-tests catch.
even code which is unit-checkable should be checked with integration-tests.
You still need to respond to his objection that use of integration tests means that you miss covering the vast majority of code paths.
That sort of bugs is caught by contract tests. You could see contract tests as a kind of integration tests, but that is not the kind of integration tests JB is talking about.
@@am0e Actually the vast majority of code paths come from combining different paths through two or more modules, so how do you cover them with unit tests?
@@harze By the time you're done getting it *right* your mocks have become full implementations and your unit tests have become integration tests
@@jonathanccast unit tests orchestrated together can become integration tests, but I don't see how they can as individual units. Mocks also don't become full implementations, but they can certainly reflect a good deal. Also I feel like we get so far off the mark by calling any of this "tests", these are just examples and will never cover the entire scope of possibilities (Gary Bernhardt has a great talk on this).
This beautiful bastard is a legend! Thank you, sir!
it seems pretty nice on the picture, unless J B interconnects nodes lying on same horizontal lines. Then we return back to n!
Honestly it’s both incredibly funny and extremely unsurprising that when faced with a legitimate code question, both this guy AND uncle Bob gave up and couldn’t answer the question. Because the longer I listen to both people talk, the more I’m convinced they’re not actually *saying* anything. They keep talking in abstracts and using these circular reasoning arguments to justify their stances. But they kinda don’t make sense.
Like no matter how many layers of abstraction you have between one module and another in your code, within an application environment, they DO have to somehow communicate to one another. And it doesn’t matter how many isolated unit tests you have - if that communication breaks down, your application isn’t working. Testing those connections, in my experience, is the most valuable place to start testing.
this guy is a savage, made me chuckle a couple of times
That's a common mistake. He is a sigma actually.
I agree that using an interface to the items that are external to our program is needed; however, I don't understand the necessity of having interfaces between items that are within our code.
That, my friend is the art.
Defining APIs is basic to composing code, whether it's internal or external. If you have worked for decades and seen code that is super messy to work in, you'll start appreciating how import it is to make sure internal APIs are super clear and are respected. Also, internal APIs can changes much more easily than external, so there's even more value in internal contracts than external.
15 minutes into this video he argues as follows:
1. 100% tests passed (i.e. unit/isolated tests) but we found a mistake
2. that encourages us to write more integration tests
3. which allows us to design more sloppily
4. which provides more opportunities for mistakes
point 3. above assumes the thing he wants to prove, which is the very definition of a circular argument.
I stopped when he said more integrated tests means less time for isolated tests... in which organisation do they do this?
Did you watch the whole talk? He's talking about the idea that instead of fixing the bug on the unit level and basically testing all possible outcomes with few simple tests, you write an integrations test on a higher level. An integration test has many dependencies and combinatorial explosion occurs, because you can't possibly test all the outcomes. That's when other bugs come from because the functionality has not been properly tested by the integration tests.
It's not circular. There's just a bit missing. Isolated tests shed light on your design problems, while integrated tests don't. THAT is what allows you to design more sloppily. He covers this in the two minutes at 8:14.
Curious how he deals with private and protected methods.
Only test through public interfaces, eng.libretexts.org/Bookshelves/Computer_Science/Programming_and_Computation_Fundamentals/Book%3A_Object-Oriented_Reengineering_Patterns_(Demeyer_Ducasse_and_Nierstrasz)/06%3A_Tests__Your_Life_Insurance/6.05%3A_Test_the_Interface_Not_the_Implementation
Also, okay for an interface to be only used by a test.
I'd call that a co-morbid spiral :D
So what do you do in a program like Ruby where there are no such things as interfaces?
You can use doubles
semaphoreci.com/community/tutorials/mocking-with-rspec-doubles-and-expectations
If your programming language doesn't provide a way to enforce contracts (interface), add that contract enforcing to your development process. At the end you can write your own contracts within the documentations and force yourself to respect them without actually needing some technology to enforce them. More or less on how an API works, just respect the contract
You can have RSpec tests for object contract. For example, create a shared_example that tests contracts (e.g expect(subject).to respond_to(:x)) and then include those contract, shared tests in the unit test file of subject. This means that whenever the contract changes, the unit tests would also fail.
So... you should write isolated tests. Because it encourages good design of systems. Which are systems that can be tested by isolated tests.
Where have we established that isolated tests provide better design feedback? He keeps repeating "more integrated tests put less pressure on the design, so we design more sloppily" without any evidence for the proposition.
This video is a scam.
I would definitely say that due to the fact unit tests (without testing implementation) require you to write more isolated code with the ability to mock dependencies leads to more modular code. Integration tests allow more flexibility. The quality of code under integration test is up to the discipline of the developer. This doesn't inherently mean integration tests are bad, but they allow for tightly coupled code.
As for the argument about isolate tests providing better feedback, they by design identify the unit in which the failure occurred, this by design is isolated, this in my opinion provides a more precise identification of the problem area, but this isn't without exception, nothing in programing is.
The moment you begin to write a few unit tests in the TDD fashion, the evidence will appear.
There is an important argument that if you find it difficult to write the isolated test then you have bad design in the place which I agree to. This is the fast feedback loop which put pressure on you to rethink about your design which is not the case with end-to-end integration tests.
I kind of agree. Integrated tests is only a scam if it's the only base you rely on to validate your business... but they are necessary unless you have money to hire testers who will spend days manually testing everything...
That said, a good base of isolated tests will highlight the weaknesses of your design early in the developing process therefore will force you to have a more modular design while respecting the business value of your application (and there is where TDD is powerful).
You DO need integrated tests but only for validating the critical paths of your application in a integrated environment as much as you need the isolated tests.
So yeah, the title of the video is controversial and by no means you should discard integrated tests because of that.
he's trying to hard to be funny & intelligent
He is the ultimate sigma male and you are being jelly.
Disagree,
here's what automated integration testing can do:
ruclips.net/video/m5zrfTFKf_E/видео.htmlm32s
This increases the quality of the end product, and confidence in each developer/designers ability to experiment
(without worrying about whether a change introduced a bug)
...But maybe you'll classify the bot as "automated checking" which is arguing semantics.
lets get the computers to do the (automatic) boring jobs
and free up the humans to do more thoughtful exercises
It's simply an end-to-end test. It's nice to have one. It's not good to have thousands of them. In the case of the video you linked, they just have the happy path done to act as a kind of smoke test. You don't design your system with it tho, so it does not increase the quality. Just makes sure you didn't fuck up royally. Just follow the test pyramid.