Kafka Ease: Simplifying Kafka Topic and ACL Management

Ricardo Domenzain — Fri, 22 Sep 2023 23:09:15 +0000

Kafka is a powerful distributed streaming platform, and managing topics and access control lists (ACLs) is a crucial part of Kafka administration. To make this task easier, I've created a Python library called Kafka Ease. Kafka Ease provides utilities to automate the creation of Kafka topics and ACLs, streamlining the administrative process.

Features

Create Kafka topics with a specified number of partitions and replication factor.
Define ACLs (Access Control Lists) for Kafka topics, controlling who can perform various operations on them.

Getting Started

To install Kafka Ease use the following command:

pip install kafka-ease

How to Use Kafka Ease

File Format

Kafka Ease uses YAML or JSON files for configuration. The configuration file must contain two main sections: topics and acls. Here's an example:

topics:
  - name: topic.name
    num_partitions: 1
    replication_factor: 1
    cleanup_policy: delete
    retention_days: 7

acls:
  - resource_type: TOPIC
    resource_name: topic.name
    principal: User:kafka-user
    host: "*"
    operation: READ
    permission_type: ALLOW
    pattern_type: LITERAL

Topics Configuration

name: Topic name
num_partitions: Number of partitions
replication_factor: Replication factor
cleanup_policy: Cleanup policy (delete, compact)
retention_days: Delete retention days

ACLs Configuration

resource_type: Resource type (UNKNOWN, ANY, CLUSTER, TOPIC, DELEGATION_TOKEN, GROUP, TRANSACTIONAL_ID)
resource_name: Resource name
principal: Principal (User:, Group:)
host: Host (*)
operation: Operation (ANY, ALL, READ, WRITE, CREATE, DELETE, ALTER, DESCRIBE, CLUSTER_ACTION, DESCRIBE_CONFIGS, ALTER_CONFIGS, IDEMPOTENT_WRITE)
permission_type: Permission type (ANY, DENY, ALLOW)
pattern_type: Pattern type (ANY, MATCH, LITERAL, PREFIXED)

Validating Configuration

To validate your configuration file, use the following command:

kafka-ease apply -f acl.yaml --only-validate

Expected output:

Validating file acl.yaml...
File: acl.yaml
YAML file detected
2 topics found.
2 ACLs found.
File acl.yaml is valid.

Applying Configuration

To apply your configuration to Kafka, use a command like this:

kafka-ease apply -f acl.yaml \
  --kafka-brokers kafka:9093 \
  --security-protocol SASL_PLAINTEXT \
  --sasl-mechanism SCRAM-SHA-256 \
  --sasl-username kafka-admin \
  --sasl-password SECRET

Expected output:

Applying file...
Kafka brokers: kafka:9093
Security protocol: SASL_PLAINTEXT
SASL mechanism: SCRAM-SHA-256
SASL username: kafka-admin
SASL password: ***************
File: acl.yaml
YAML file detected
1 topics found.
1 ACLs found.
Topic topic.name updated
Removing old ACLs 3
ACL User:kafka-user synced
File synced successfully.

More Information

For more details and options, check out the documentation.

Conclusions

In conclusion, Kafka Ease is a valuable Python library that simplifies Kafka topic and ACL management. With its features for creating and configuring topics and access control lists, it significantly streamlines administrative tasks for Kafka administrators. This tool can be seamlessly integrated into your CI/CD (Continuous Integration/Continuous Deployment) pipeline to enhance the Kafka configuration deployment process.

By incorporating Kafka Ease into your CI/CD pipeline, you can achieve the following benefits:

Automation: Kafka Ease automates the creation and management of Kafka topics and ACLs, reducing the manual effort required for these tasks.
Consistency: CI/CD pipelines ensure that Kafka configurations are consistently applied across different environments, reducing the risk of configuration errors.
Version Control: With Kafka Ease configuration files under version control, changes to Kafka configurations can be tracked and reviewed, enhancing collaboration among team members.
Testing: Before applying configurations to production Kafka clusters, you can thoroughly test them in staging or testing environments, ensuring the configurations work as expected.
Security: Kafka Ease supports security protocols like SASL, allowing you to securely manage Kafka configurations in your CI/CD pipeline.

By integrating Kafka Ease into your CI/CD workflow, you can achieve a more efficient and controlled process for managing Kafka topics and ACLs across your Kafka clusters, ultimately enhancing the reliability and security of your Kafka infrastructure.

GitHub Repository: Kafka Ease on GitHub

Happy Kafkaing! 🚀

Forem: Ricardo Domenzain