How to use Data Contracts for Long-Term Schema Management

cnfl.io/podcast-episode-263 | Have you ever struggled with managing data long term, especially as the schema changes over time? In order to manage and leverage data across an organization, it’s essential to have well-defined guidelines and standards in place around data quality, enforcement, and data transfer. To get started, Abraham Leal (Customer Success Technical Architect, Confluent) suggests that organizations associate their Apache Kafka® data with a data contract (schema). A data contract is an agreement between a service provider and data consumers. It defines the management and intended usage of data within an organization. In this episode, Abraham talks to Kris about how to use data contracts and schema enforcement to ensure long-term data management.
When an organization sends and stores critical and valuable data in Kafka, more often than not it would like to leverage that data in various valuable ways for multiple business units. Kafka is particularly suited for this use case, but it can be problematic later on if the governance rules aren’t established up front.
With schema registry, evolution is easy due to its robust security guarantees. When managing data pipelines, you can also use GitOps automation features for an extra control layer. It allows you to be creative with topic versioning, upcasting/downcasting the data collected, and adding quality assurance steps at the end of each run to ensure your project remains reliable.
Abraham explains that Protobuf and Avro are the best formats to use rather than XML or JSON because they are built to handle schema evolution. In addition, they have a much lower overhead per-record, so you can save bandwidth and data storage costs by adopting them.
There’s so much more to consider, but if you are thinking about implementing or integrating with your data quality team, Abraham suggests that you use schema registry heavily from the beginning.
If you have more questions, Kris invites you to join the conversation. You can also watch the KOR Financial Current talk Abraham mentions or take Danica Fine’s free course on how to use schema registry on Confluent Developer.
EPISODE LINKS
► OS project: github.com/abraham-leal/cclou...
► KOR Financial Current Talk: cnfl.io/evolving-schemas-without-schema-evolution-episode-263
► The Key Concepts of Schema Registry: cnfl.io/schema-registry-101-e...
► Schema Evolution and Compatibility: cnfl.io/schema-evolution-and-...
► Schema Registry Made Simple by Confluent Cloud ft. Magesh Nandakumar: cnfl.io/schema-registry-confl...
► Kris Jenkins’ Twitter: / krisajenkins
► Streaming Audio Playlist: • Streaming Audio Podcas...
► Join the Confluent Community: cnfl.io/confluent-community-e...
► Learn more with Kafka tutorials, resources, and guides at Confluent Developer: cnfl.io/confluent-developer-e...
► Live demo: Intro to Event-Driven Microservices with Confluent: cnfl.io/demo-intro-to-event-driven-microservices-with-confluent-episode-263
► Use PODCAST100 to get an additional $100 of free Confluent Cloud usage: cnfl.io/try-cloud-episode-263
► Promo code details: cnfl.io/podcast100-details-ep...
TIMESTAMPS
0:00 - Intro
2:02 - What is a data contract?
3:27 - What are the problems with using JSON Blobs?
8:38 - What are the advantages of using Avro and Protobuf formats?
17:00 - What are schema references?
19:38 - What support is available for changing the data format?
22:33 - What are forwards, backwards, and full compatibility?
31:17 - What should you do if you have two different formats?
35:28 - What are the tradeoffs of doing topic versioning?
43:13 - What are upcasters and downcasters?
52:45 - Are there any recommended tools for making data discoverability easier?
56:02 - It's a wrap!
ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion - designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.
#streamprocessing #apachekafka #kafka #microservices #confluent

[@martykate]

Thank you a lot for such a cool guest and everything you are doing Kris!

[@astronemir]

Really awesome and great recommendations regarding using single source of truth via git, preferring extensibility, and open standards. I would have wanted a discussion of best practices when Kafka and topics are just one part of the data platform & pipeline, and how to have interoperability there when everything is not organized into streams and topics