Software Testing Theory + A Few Less Obvious Testing Techniques
HTML-код
- Опубликовано: 1 авг 2024
- This video covers what software testing is and what it's not and how it relates to program correctness. I also talk about two less obvious testing techniques: mutation testing and property-based testing.
💡 Here's my FREE 7-step guide to help you consistently design great software: arjancodes.com/designguide.
If you want to play around with mutation testing or property-based testing, take a look at these frameworks:
- Mutmut (Python mutation testing library): mutmut.readthedocs.io/en/latest/
- Hypothesis (Python property-based testing library): hypothesis.readthedocs.io/en/...
🎓 Courses:
The Software Designer Mindset: www.arjancodes.com/mindset
The Software Designer Mindset Team Packages: www.arjancodes.com/sas
The Software Architect Mindset: Pre-register now! www.arjancodes.com/architect
Next Level Python: Become a Python Expert: www.arjancodes.com/next-level...
The 30-Day Design Challenge: www.arjancodes.com/30ddc
🛒 GEAR & RECOMMENDED BOOKS: kit.co/arjancodes.
💬 Join my Discord server here: discord.arjan.codes
🐦Twitter: / arjancodes
🌍LinkedIn: / arjancodes
🕵Facebook: / arjancodes
👀 Channel code reviewer board:
- Yoriz
- Ryan Laursen
- Sybren A. Stüvel
🔖 Chapters:
0:00 Intro
0:52 What tests are and what they aren't
1:43 Code example: tests cannot prove that a program is correct
4:44 Proving that a program is correct: Hoare logic
7:23 Different types of testing
8:53 Code example: mutation testing
12:03 More types of testing
13:07 About invariants
14:13 Code example: property-based testing
18:01 Types of property-based testing
19:29 Final thoughts
#arjancodes #softwaredesign #python
DISCLAIMER - The links in this description might be affiliate links. If you purchase a product or service through one of those links, I may receive a small commission. There is no additional charge to you. Thanks for supporting my channel so I can continue to provide you with free content each week!
I think I found out why nobody talks about Hoare logic on RUclips ;).
For anyone interested in property-based testing, i can highly recommend watching Scott Wlaschin's talk "The lazy programmer's guide to writing thousands of tests". The examples in this video is similar to the talk, but goes into more detail about it.
I discovered this youtube channel this week and has become my favorite channel since then. I'm finding a lot of value in yout videos. Thank you. Greentings from Costa Rica, Arjan.
Awesome, thank you!
You are very good at explaining these things. It's a pleasure watching. I use lots of automated testing when I code, but this is still most inspiring.
Thank you, glad you like the videos!
I never thought I'd hear about Hoare logic again after my Software Verification class. Great video!
Thanks Mark, glad you liked it!
What a great morning for me. Opening yt and finding a new Arjan video about the exact same thing I wanted to learn about
Good morning 😊
Thank you for these videos. I've learned most of what you put out myself, but I wish I had your videos on the process of programming a project. Whether it's testing, design paradigms etc. Fantastic work as always.
Thanks, glad you like them!
you're the best dude! thanks so much! great video!
Awesome! thanks for the video, no much people talks about testing in general, normally they stuck in TDD that's why video is refreshing. I'm interested in integration testing btw :)
Awesome video. Thanks!
Thank you for this video, looking forward to learn more about this topic 😀
As a self taught programmer, I've been doing property based testing for my code without knowing about it! Glad to see you explain it in a more general framework.
Thanks, Glad the content is useful.
Love it!
Hey Arjan. Would love to see a video on unit tests with examples. Or have you made one already? Thanks for the great content. P.S. Also very interested in the DETAP process.
Thanks for the great video!
I've gotten used to test my code lately on a unit test level, but usually I only find the time and motivation to test the lower level standalone testable components, when it comes to the higher level stuff that requires more work I kinda give up 😕
aaaayyyy new video, thank you for making those videos < 3
You’re welcome!
Really looking forward to more of your content.
Something you don't touch in your videos because I think you take if for granted at this point is what the interface between the program/programmer looks like.
In several of your previous videos you restructure code such that the way you create the relevant class and particular instances changes. Though I am still relatively new to it, I think about this a lot at my job... "how will use the utility I am making? what will they want to explicitly put into their program such that it makes sense not only to them, but to whoever comes later to add/modify it". This could be a great topic for you because, again, its much more of a intermediate-advanced level of software development that isn't often addressed but is a real-world issue encountered every time we write code with our future selves or others in mind.
Very good point! One of the reasons to set things up nicely is indeed to make the job easier for other developers who are going to work on the code later on, or for your future self, because you'll surely forget the details of what you did, and then having clean code just means it's a lot easier to get back into it. It's a nice topic for a video, though I'm not entirely sure how to do any entire video about just this point. If you have any specific suggestions of what you'd like to see in there, let me know.
@@ArjanCodes I believe it all boils down to the Object Oriented Design topic and that would be a very good place to start. Unfortunately very few people have posted videos on OOD.
The hardest part, for me at least, was to learn how to think about my code (and I'm still not there yet :-) ) and not learning the programming language itself. Learning the "when" and "why" it's just as important as "how". I don't believe one video would be enough but I would very much look forward to a series of videos on the OOD subject.
As aside note, it would be nice to see simple diagrams (UML diagrams might be too much) on how things (i.e. classes) interact with each other before and after you've done some refactoring. Sometimes it's better to have a picture than jumping up and down through the code to get an overview of the concept.
Thanks for all your hard work. I really enjoy your videos.
I work at a small company that doesn't do widescale testing. We do lots of verification with clients when setting up the programs and a few verifications/asserts within the program that we are getting results about what we expect, but I am very interested in a more robust framework that is also not insane to set up on day 1. I've dove into this topic several times but i do think that you have a great, slow and simple way of describing things so I very much looking forward to your videos!
You're already doing more than many other companies. I'll try to keep the practicalities of setting up testing in mind when I'm doing the followup videos, good point.
Thanks for the valuable content
You're welcome!
thank you for the video! Would be indeed interesting to hear your explanation of DTAP, and the differences between acceptance tests/e2e tests/integration tests. Thank you for your work!
Thanks Alessandro, will do!
3:14 VSCode has a nice feature where yo can edit multiple lines in one go (e.g. ctrl + alt + arrow keys on Windows) - perfect for your need here
Can you please make a video about Hypothesis? love your content!
thank you arjan
Thanks Brahim, happy you’re enjoying the content!
I have been writing unit tests at the company where I work for a while now. But when projects get older and new tests are added, by multiple people, I oftentimes find that the test code becomes very messy. I would love to see a video on how to structure tests for larger projects and how to think about writing maintainable and easy to understand tests.
My advice is, as someone who's written a lot of tests in my career as a software developer; ditch the unit tests.
Or at least most of them. You won't need 90% of them most likely. Think more about how the integration works between different parts of the app, rather than the individual units. You'll save much more time, won't need to keep maintaining tests that are already written-if you avoid testing implementation details, of course-and you can think much broader about edge-cases that a user might face by interacting with things.
Simplify your tests and don't over-think them. Tests are an extension of your code, not a separate piece.
And if you can help it and are able to, throw in some static typing as an extra bonus to ensure consistency of data usage across your codebase. Especially useful for data that you get or send to/from external sources.
I use property based testing and unit testing a lot. I find they complement each other very well.
I would like to use mutation testing some more. Are there any tools to help automate that ?
Really interesting. These are things that I do but didn't have the vocabulary to describe it so clearly. I'm working on integration testing currently as it is a gap at our company. I'm looking forward to your lesson on it.
Glad it was helpful!
I would love all you have to say about testing with Python! If you have some knowledge about practises about black box testing, please add that too :)
My favorite type of software testing is the "print that variable" approach xD
I have used the Python package faker successfully in the past. The nice thing about this package is that when it finds data that causes a test to fail, it keeps a copy of it, so it can replay the same test again
Thank you, Arjan for the very valuable topic. There is a question about testing.
You mention a few technics, and there are many more of them. So "how many tests we should create"? I mean not an exact number of course. But what criteria do you use to understand that you did enough?
Hey Arjan, when you do the video comparing test frameworks add Robot Framework to the mix. Great video love your content.
we love you Goncalo Moura
Thanks for the suggestion!
I am doing unit tests in my current work but it has been quite some work to setup because almost all our functions interacts with data from a database. It would be nice to have a video on that if you are interested in.
I have never heard of mutation tests but it seems really interesting to make your tests more robust !
Thanks, great suggestion!
Hey, love your videos. I was wondering if you were in the future ever going to post videos about languages other than python such as Java?
Hi, thanks! For the time being, I'm using Python as the main language on this channel as this is what the majority of my audience uses. It might change in the future though if I notice that there is a lot of demand for videos about a particular programming language.
Very informative, i've been using some of what you taught us at work. Still waiting for the python libraries video!
Glad to hear you liked it! To be sure: what do you mean by the "python libraries" video?
@@ArjanCodes Absolutely! I apologize for the ambiguity, i meant the python libraries for testing similar to Pytest
Got it. That one's in the pipeline!
How do you make sure you are not introducing bugs in your definition of properties? maybe writing test for your test and so forth ad infinitum.
i think i have done this property based testings, from how i get away with not testing property-tests is making them trivial and static reviewed.
I'm a faculty member teaching software testing. It's hard to explain a concept and introduce a tool but you just did that using Python! Looking forward to more of your videos! Please let me know if you know some good testing assignments that I can give my students to practice. Thanks! Also I just learned about 'typing' in Python!
Great to hear and glad you like the videos!
I use unit testing - refactoring and adding new features with no fear. What about testing UI ? Taking screenshots instead of snapshots and comparing them before and after website change.
Could you please show some examples using the pytest framework ?
I am really up to python frameworks comparison! I am currently using pytest as a software tester and I would be very appreciate to learn sth new related!
Interesting, is this theory a part of bigger topic which is called formal verification if i`m not mistaken?
Thanks for another quality video. I like the property test - would it be an idea to set the random seed? Otherwise you might end up with a test which fails some of the time, which is not much fun.
I totally agree and was going to comment the same thing. I think it's valuable to have reproducible test runs, and setting the seed seems to be necessary
Yes, great suggestion!
If a property-based test fails "some of the time," that means your code has a bug and it needs to be fixed. Perhaps you're worried that only some runs generate the random inputs that cause the failure? The fix to that is that every time the property-based test fails, you add a unit test with the offending inputs. Setting the random seed would be counter to the idea of randomized testing: setting a seed means that if your buggy code happens to work with e.g. the first 100 (deterministic) random inputs, you will never generate new inputs that catch the bugs.
@@leif_p Hmm, I suppose it depends on the context. I can see the value in true random testing as part of a specific test suite that's run manually. If it's part of a CI suite (which is what I was thinking about I guess) then I don't really want my tests failing suddenly when I am working on some other unrelated region of the code.
@@leif_p Yes I agree that it's less valuable in terms of completeness since you're testing a subset of random numbers. However, if this subset is big enough, and there is a bug in your code, you will likely discover it. The problem in my opinion with not setting the seed is that you won't be able to debug a failing test, and it could take forever to find what causes the bug. To conclude, I think it's good practice for unit tests to have deterministic inputs, but I really don't know for property-based tests.
"Beware of bugs in the above code; I have only proved it correct, not tried it." - Donald Knuth
Nice one!
Your videos just get more and more beautiful. Thank you.
Glad you like them!
Any good ways to test things that require timing and multiple processes/database connections to break e.g. simultaneous web requests incrementing a counter. Or would that be a case where you actually just have to argue to prove correctness? Or maybe also have something that periodically checks the number of increments in the logs matches the value of the counter (but then you have the same problem when retrieving the value of the counter to check against...)
Actually struggle more with the non-technical or decision-making side than finding technical solutions: e.g. in this case maybe the best solution is actually to decide you don't care about having an exact count and it's more important to spend time/effort on something else.
I was so frustrated by school, that they never taught ANY testing. Something they should have done from day 1, imo. By the time I finished I still had written 0 tests. I had taught myself about unit testing, testing frameworks, Property-Based testing, but had 0 practical skill, until I got my first job.
I wish software engineering schools focused on testing at all.
No, Arjan I'm not testing my code. I'm a junior developer at a small company we write some shit and deploy it, like true savages. A development environment is for the feint of heart. Jk, well not about my companies situation but jk about the last sentence. Anyways thank you for another amazing video, you've taught me a lot!
I know the feeling. That’s also how I started out with my company. In the beginning, robustness and tests don’t matter, because you’ll throw away everything in a few months and start over once you learn what your customer actually wants. The flip side of this is that when you do find something your customer wants, you have a lot of code to refactor and cleanup, and there’s little time. Life at a startup…
Hey Arjan, I know it's probably a bit basic for your channel. But would you be able to do a video on importing and file structures?
There seems to be a lot of mis-info on how to structure and import files/modules/packages and when/where to include init files. Especially when using something like pytest where the path is different (been having a lot of difficulty with this stuff at work, and can't seem to find a clear definitive guide on how to do it properly!!)
We had a contractor (developer) who asked our qa why our company tests so much? He was gone a month later.
Thank you for teaching us what we need to become real software engineers and not stay code monkeys forever. I think a full blown DevOps series would also be amazing.
BR
Great suggestion, thank you!
from the thumbnail, I thought that this was going to be you playing potion craft and I was confused
Next week, I’m going to turn myself into a python, so this thumbnail was foreshadowing.
This video has certainly opened my eyes to more types of testing than just unit-tests. But I can't shake the feeling that most tests are a waste of time because they test such trivial things. I suppose most complex problems are built out of trivial ones, but at the end of the day it may give a sense of false security if you spend a lot of time on trivial tests and never get around to the actual complexities your code is meant to solve.
What doesn't help is that examples like this always have to be generic, quick and easy to explain. I'd love to see an example of testing in a more complex code base.
I’m going to cover more complex testing examples in the future. The thing is that most tests are setup in a pretty simple way on purpose so they test only a single thing, making them easier to manage. Even though a particular test might seem trivial, from my own experience these things do break from time to time as you change the code. I was very happy that I had a test in place to point it out to me, because otherwise it would have been a customer.
Thanks for the great videos.
But i must add, even though unit testing is generally considered whitebox technique, in reality it is also blackbox technique.
It just depends how you test the "unit", the function.
Your function might call a series of other functions. You dont know their code and you dont care. You just care about the result.
Glad you like the videos! You’re right that to run unit tests you don’t know about other functions that are called. The way I look at it is that this doesn’t make your unit tests a black box testing technique, but that those functions actually fall outside of the scope of the what the box is. So the inside of the box is still white, but outside it’s pitch black ;).
Continue making great job! Greetings from Belarus!
Thank you, and will do!
I thought I was just using unit testing, but it seems Im doing different kind of tests too
Great video as usual. I really like how you explained property-based testing. Great job! Btw. you may also want to have a look at metamorphic testing: en.wikipedia.org/wiki/Metamorphic_testing
Thanks for the link, I will!
"Hoare Logic" seem very dangerous😄
when you use random numbers for property testing you may create a test suite that sometimes fails. A bit scary idea. You change something in a large codebase and then a test fails in some other part of that system... not sure I agree with such practice. I want my tests to be 100% repeatable.
Probably the best approach is to randomly generate lots of tests, but fix the seed value.
@@ArjanCodes yeah if you do simple manual tests. If you use hyphotesis it is all magic and I wasn't able to lock the randomised tests. Maybe I missed something.
When I look closer at hypothesis in one of my upcoming videos, I'll try to address this and hopefully find a way to do this.
If I interpret Hypothesis' documentation correctly, it seems they are using a fixed seed of 0, which you can override, see: hypothesis.readthedocs.io/en/latest/data.html#hypothesis.strategies.random_module.
@@ArjanCodes that's interesting. in the quick start documentation they mention that hypothesis builds a database of failing cases which suggests that the cases change. I understand that it is not a brute force algorithm with just random values but it smartly figures out what are the best ways to test the code. I didn't use it enough to know what they do and how it actually works.
"If we wanted to make sure this example was always checked we could add it in explicitly by using the @example decorator. This can be useful to show other developers (or your future self) what kinds of data are valid inputs, or to ensure that particular edge cases such as "" are tested every time. It’s also great for regression tests because although Hypothesis will remember failing examples, we don’t recommend distributing that database."
How about actually using the software through the UI like a real end user would? Is that really such a terrible idea?
No, not a terrible idea at all! A lot people and companies and this kind of end-to-end testing in addition to their unit tests. I don’t know if anyone just uses UI testing but maybe it’s feasible.
Absolutely, manual / end-to-end testing which I mention briefly at the end of the video is really important as there are always some aspects that are very challenging to cover for automatic testing techniques.
Geez... Hoare logic was proposed by Hoare in 1969. The universe does have a sense of humor, it seems.
You should never use random in tests as it means there is a chance, that logic won't work but test will still pass.
This might cause your code to be deployed to your server even though it has issues!!
Probably the best solution is to generate random tests (to get the benefit of having lots of tests), but fix the seed value so they're always the same random tests.
First!!!
Hey, Arjan! You're so fine! 🎶 You are so fine, you blow my mind! Hey, Arjan! 🎶 *clap 👏 clap 👏* Hey, Arjan! 🎶
*Request for Code Roast*
*Script* : ruclips.net/video/PiaUsymcP_M/видео.html
you need to update your background letterboard python, it's running python 2
Haha, good one! I don't have parenthesis letters, so actually my "keyboard" is broken :).
Found a goldmine
Thanks, glad you enjoyed the video!
This video made me hoareny.
Great name