This is a continuation of my previous post on “Building Real-time Streaming Apps Using .NET Core and Kafka”. In this post, we are going to look at the security aspects of Kafka at a high level. We will also configure a .NET Core application to authenticate with a Kerberos-aware Kafka Cluster.
Kafka Security Overview:
Enabling security over Kafka is optional. However, if your organization is using Kafka to store critical data, enabling Kafka’s security features is crucial to protect your data from all kinds of cyber-attacks.
Four key security features that you can enable over Kafka:
Let’s assume that your organization’s KAFKA instance is configured to authenticate internal/external clients using the SASL GSSAPI (Kerberos) option, and assume your organization already has a Kerberos server internally (maybe using Active Directory).
So, any application/service that needs to communicate with Kafka should get authenticated using SASL(Kerberos).
What is SASL?
Simple Authentication Security Layer (SASL) is a framework that can be used with other protocols such as Kerberos, SMTP, etc. The basic idea is that an authentication mechanism is separated from the protocol.
GSSAPI is a SASL authentication mechanism for supporting Kerberos authentication.
What is Kerberos?
Kerberos is a network authentication protocol based on secret key cryptographies. In simple terms, instead of sharing passwords, communication partners share a cryptographic key, and they use knowledge of this key to verify one another’s identity. You can read more about Kerberos here.
SASL/GSSAPI is a great choice for enterprises as it allows the companies to manage security from within their Kerberos Server.
Let’s take our e-commerce use-case mentioned in our previous Kafka blog. Both “Order API” and “ProcessOrdersService” are .NET Core apps that produce/consume messages to/from Kafka topics.
With your organization using secured Kafka, these two apps need to use Kerberos tokens to get authenticated to the Kafka cluster.
Configure .NET Core Producer/Consumer for SASL/Kerberos Authentication:
Confluent’s .NET core libraries make configuration easy and there aren’t any extra steps. All that you need to do is generate a KeyTab file with your Kerberos principal (user alias in the network) and provide that as part of producer/consumer configuration settings.
Step 1: Create or update krb5.conf file in /etc/ path.
kdc = testkdcserver.myorg.com
Step 2: Generate a KeyTab file
It’s basically a file that contains a table of user accounts with an encrypted hash of the user’s password.
How do I generate a KeyTab file?
add_entry -password -p principal_name -k number -e encryption_type
How do I verify whether my KeyTab file really works or not?
Run the below command:
kinit username@MYDOMAIN.COM -k -t username.keytab
You should successfully authenticate without being prompted for a password. Success!
Step 3: Update producer/consumer config settings.
Step 4: Run your application and try producing/consuming messages to Kafka.
Note: If you are a windows user, setup WSL (Windows Subsystem for Linux) and try the above steps from ubuntu shell.
Enabling Kafka security features is very important and will protect your organization from cyber-attacks. To authenticate Kafka clients, configuring Kafka brokers with SASL/GSSAPI(Kerberos) is also a great choice for enterprises as it allows for security management within the Kerberos Server. Finally, Confluent’s .NET Core APIs have well-defined interfaces that neatly configure .NET Core apps so they can be authenticated to a SASL/GSSAPI(Kerberos)-enabled broker.
The Kafka community added a number of features that can be used, together or separately, to secure a Kafka cluster. Refer to the Confluent documentation for more details.
For detailed code, please refer my GitHub repository.
Srini is an Agile Transformation Engineer at TribalScale based out of Boston office, .NET web developer focused on micro-service-first architecture design.
TribalScale is a global innovation firm that helps enterprises adapt and thrive in the digital era. We transform teams and processes, build best-in-class digital products, and create disruptive startups. Learn more about us on our website. Connect with us on Twitter, LinkedIn & Facebook!