BlueHat 2024: S18: Minting Silver Bullets is Challenging

BlueHat 2024: Session 18: Minting Silver Bullets is Challenging Presented by Josh Brown-White from Microsoft
Abstract: With the advent of AI coding assistance such as GitHub Copilot there has been the obvious interested in using AI as a silver bullet to automatically correct source code to fix security vulnerabilities. Unfortunately minting this silver bullet is far more complex than simply calling the Azure AI APIs, and the organizations who have been rushing AI generated automatic remediations have seen results ranging from poor to disastrous. Modifying existing code behavior automatically is a very different exercise than a developer guiding Copilot to generate net new code, but the failures to understand those differences have resulted in quite poor outcomes. Many AI fix suggestions today catastrophically alter code, while only a minority of suggestions even technically correct the issues. Only a tiny percentage can be safely merged unmodified in a Pull Request. Most of the current attempts are causing more harm than utility.
This talk will explain the challenges that need to be accounted for when using generative AI to modify existing source code to correct security vulnerabilities, and detail how combining generative AI with existing deterministic analysis techniques can yield far better results. The future where the system not only automatically detects security vulnerabilities in source, but automatically remediates many (though not all) of them is attainable. It turns out that the reason why silver bullets aren’t common is because they take a fair bit of work to make, but they can be made if you are willing to put in the work.

[@ndrixxx]

Cool tshirt, JBW. :) Love the talk.