Code More, Draw Less, and Debug Just a Little! - Yash Saboo

Code More, Draw Less, and Debug Just a Little! - Yash Saboo - PyCon Italia 2024
Elevator Pitch:
Understanding software architecture and data flow is vital when building any maintainable system and debugging the errors. While we are searching for life on Mars, our architectural diagrams and debugging tools remain manual and lifeless. Let's use GraphDB, pandas, and Python to add life to them.
Description:
Before I dive into how to draw less and debug just a little, let's recap your life as an engineer, which likely looks like one of the following:
- You often wonder before you go to sleep, "A short time ago in a galaxy not that far, far away, I updated my software's architecture diagram. So why is it stale… again?"
- Your team grew in size during the pandemic and so did your components and ownership. Has it become harder to onboard new engineers due to ever-changing software, dependencies, and owners?
- Your new release didn't go as planned because something broke in your new code. Wouldn’t it help if you had a holistic and interactive view to navigate between software components while understanding how they communicate with each other? When was the last time you used your architectural diagram as a debugging tool?
- You managed to quickly draw the data-flow diagram to help with debugging your system and you thought “this is majestic work right here!” But the feedback you received was, "it’s too detailed" or "it’s not detailed enough." While you’re trying to figure out the fine line between too many and too few details, your hairline is starting to show!
If you’ve fallen victim to any of these scenarios, behold! Your prayers have been answered!
In the first half of this talk, we’ll describe an approach that can automatically identify the software design and data flows within your systems. To achieve this, we use algorithmic scrapers and metadata profiling techniques that integrate with distributed trace, code structure, language dependencies, contribution ownerships, and other sources to avoid the toil of manually updating software architecture diagrams and debugging tools.
In the second part of this talk, we will dive deeper into the details of how to use Python data engineering libraries to enrich the collected data and process it for storage in a graph database. The graph structure not only signifies the relationships between components in a real-world manner, but also helps in generating multiple views of the software architecture in an easy and comprehensible fashion. Now, instead of the architecture and data flow diagram being a static JPEG, those auto-generative views can be compiled into interactive and immersive UIs for debugging.
My team built a debugging visualization tool for our organization, Bloomberg, but the approach we took is germane to all tech organizations. We want to share the challenges and lessons we learned during our journey to help you build a similar tool for your organization because why not “Code More, Draw Less, and Debug Just a Little”?!
Learn more: 2024.pycon.it/...
#Visualization #Pandas #Tooling #Architecture #GraphDatabases