Testing for Ghosts in the Machine: Assuring 'Good Enough' Software Quality in AI-based systems

Testing for Ghosts in the Machine: Assuring 'Good Enough' Software Quality in AI-based systems - Artur Patoka - PyCon Italia 2024
Elevator Pitch:
Join my talk on trying to master software quality in AI's unpredictable world. I'll share both challenges and solutions for developing reliable tests, with a focus on chatbots. Get ready to implement robust and automated testing strategies. Or at least get a bit closer to achieving it!
Description:
The presentation is the culmination of my experiences in two distinct projects - I have been professionally responsible for a quality of a chatbot and, despite the experiences, I've also created a hobby open-source project which utilizes the OpenAI API.
In this talk, I will explain the why - the state of current automated tests in the AI realm and why it's important. I plan to dissect various methodologies, illustrating their strengths and weaknesses through real-life examples, predominantly focusing on GPT-based chatbots.. The presentation will also cover dead ends in hope to spare the suffering for those embarking on their own ghostbusting adventures in AI. Finally, I'll share a few proved solutions which should greatly improve the quality and stability of an AI-based system on an example of a chat bot.
This presentation is designed to be accessible and engaging, requiring just basic knowledge about gen AI and software development cycle.
Learn more: 2024.pycon.it/...
#APIs #BestPractice #Testing #NaturalLanguageProcessing