Data Governance for Topics in Confluent Cloud Kafka

When it comes to managing topics in Confluent Cloud Kafka, one aspect stands out for its crucial role in ensuring the accuracy, availability, and security of your streaming data: data governance. This pillar of reliability is instrumental in making sound decisions and preventing issues like tracing flawed data back to its source in critical projects. Confluent Cloud offers Stream Governance tools, a suite of functionalities including Stream Lineage for tracing data origin and Schema Registry for maintaining data consistency, all of which are essential for implementing robust data governance practices.

https://www.confluent.io/blog/governing-data-with-confluent-stream-governance/

My team manages the Confluent Cloud Kafka platform, where Topics are grouped into Clusters, and Clusters are grouped into Environments. Our contract with Confluent includes a Stream Governance package.

In this blog post, I will describe how I developed a self-service capability to create Topics and apply Business metadata and Tags to them. Business metadata and Tags are crucial parts of the Topics’ Data Governance. I also tried to address the main aspects of Dynamic variables and Scalability.

The self-service capability is developed in Terraform. Sensitive details like API keys and secrets are stored in the Terraform workspace as “Sensitive” variables.

The code accepts these fields as the input.

cluster_id: <cluster ID>
topic_name: [multiple topics names can be provided here]
cleanup_policy: <delete or compact>
retention_ms: <based on user needs>
partitions_count: <based on user needs>
prevent_destroy: true or false (this controls deleting the Topics in Terraform code
tags: <provided by the requestor>
topic_requester_ID: <user ID of the Topics requestor>
data_owner_emailID: <email ID of the Topic owner>

Terraform code processes these inputs and does the following:

1. Create the Topics in the Cluster.
2. Add Business metadata (topic_requester_ID and data_owner_emailID) to the Topics.
3. Add Tags to the Topics.

Disclaimer: The posts here represent my personal views, not those of my employer or any specific vendor. Any technical advice or instructions are based on my knowledge and experience.