1. In this video, I created the topics using kakfa-topics.sh via the port 2181 (unsecured port). Moving forward, we need to create the kafka topics by using the parameter --bootstrap-server rather than --zookeeper for two reasons - 1) Security and 2) Removal of Zookeeper in the future from Kafka ecosystem. I have mentioned the steps to create kafka topics in a secured way in this link - github.com/vinclv/data-engineering-minds-kafka/blob/main/config/sasl_ssl/SecureTopicCreation.md. Please stop creating kafka topics via port 2181 anymore. You can also remove this port from zookeeper property file. 2. Regarding the user creation process via kafka-configs.sh for Windows users (Thanks to the comment from lord byron) => For windows users who are running it in the cmd, the command that you want is basically the same: kafka-configs.sh --zookeeper localhost:2182 --zk-tls-config-file zookeeper-client.properties --entity-type users --entity-name my-user --alter --add-config SCRAM-SHA-512=[password=DEM123], you just need to drop the ' from command
SASL can be standalone used for proving authentication between Client and Server application and Kerberos GSSAPI (one of the SASL mechanism) also provides the method to encrypt the data on source side which can be decrypted on destination side using algo that is agreed upon during context creation.
Hii sir, you content had been amazing and i was able to understand it very easily. If possible can you make viedo on how mirror maker 2.0 works and its implementation.
Your videos are amazing mate. Helped me a lot while working with Kafka on my workplace. Tbh your video was the best I found in regard to this topic. Keep up with good work ;)
Thank you for the video series, it has helped me with setting up secure kafka for a demo at my workplace. I implemented producer and consumer using KafkaJS and able to produce/consume without including the truststore. Does that mean its skipping the broker verification?
Hi Thanks for the video buddy ! Had just one query can we use the same user for kafka clients (producers and consumers property file) which you made superuser in kafka brokers ? or we need to create seperate users for producers and consumers ?
I have followed the tutorial and is able to configure the sasl ssl in my system. If i want to use it via public ip address so that other system can use it ehat do i have to do . Could you help regarding this
For windows users who are running it in the cmd, the command that you want is basically the same: kafka-configs.sh --zookeeper localhost:2182 --zk-tls-config-file zookeeper-client.properties --entity-type users --entity-name my-user --alter --add-config SCRAM-SHA-512=[password=DEM123], you just need to drop the ' from command :) cheers
I am using Amazon msk, my consumer consumes messages and suddenly in between I get error " not Authorization to access topic" or " not authorised to access join consumer group". This seems wierd, it works and suddenly gives these errors randomly.
Hi It was really helpful . I am facing one problem where I am getting error like :-No LoginModule found for org.apache.common.security.scram.ScramLoginModule . Are we missing some jass configuration or something in this video whihc is creating this issue ?
Thanks for your comments and suggestions. I updated the repo now. Could you please check here - github.com/vinclv/data-engineering-minds-kafka/blob/main/config/sasl_ssl/README.md
Hi, well explained .. thanks a ton. One doubt Kafka-acls are zk clients they are connected to zk using secured port and using zk-clients.properties as ACls are stored in zk ..in confluent documentation Kafka-acls command they are giving -bootstrap-server .. why ..plz let me know
Hi Chandan, u r right; it is indeed zookeeper client. If u look into this ticket - cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum, they are going to shutdown zookeeper completely very soon. This means all management activities done by zookeeper will be taken care by brokers itself in future and that's the reason they suggesting u to use --bootstrap-server instead. However, I checked the latest version of Kafka and this is not completely migrated aand hence u can continue using zookeeper way as I did in this video (u will just get a warning which u can ignore). Please remember this is applicable to any ZK client in the future. I explained it for kafka-configs.sh in this video (check from 17:50).
Excellent video !!! Working fine with SASL_SSL using console producer and console consumer , any sample links for code to create producer and consumer in node JS.? it will me much helpful. shortage of developers :)
Thanks a lot for your support. Unfortunately, I've never used JS so far for any of my use cases.I will check for sample code and let u know soon. Please send me a reminder in case i forget it :)
Great Explanations. Is it necessary to create a topic before enabling SASL? I was able to follow everything when creating a topic before enabling SASL but after enabling SASL I am not able to create any topic. I am getting an error.
@@DataEngineeringMinds Sorry for the late reply. I am getting the following error kafka-topics.sh --zookeeper localhost:2181 --create --topic test --replication-factor 1 --partitions 1 Error while executing topic command : KeeperErrorCode = NoAuth for /config/topics/test [2021-02-08 11:06:49,504] ERROR org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /config/topics/test at org.apache.zookeeper.KeeperException.create(KeeperException.java:120) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:564) at kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1644) at kafka.zk.KafkaZkClient.createOrSet$1(KafkaZkClient.scala:364) at kafka.zk.KafkaZkClient.setOrCreateEntityConfigs(KafkaZkClient.scala:374) at kafka.zk.AdminZkClient.createTopicWithAssignment(AdminZkClient.scala:93) at kafka.zk.AdminZkClient.createTopic(AdminZkClient.scala:57) at kafka.admin.TopicCommand$ZookeeperTopicService.createTopic(TopicCommand.scala:353) at kafka.admin.TopicCommand$TopicService.createTopic(TopicCommand.scala:196) at kafka.admin.TopicCommand$TopicService.createTopic$(TopicCommand.scala:191) at kafka.admin.TopicCommand$ZookeeperTopicService.createTopic(TopicCommand.scala:345) at kafka.admin.TopicCommand$.main(TopicCommand.scala:62) at kafka.admin.TopicCommand.main(TopicCommand.scala) (kafka.admin.TopicCommand$)
@@santoshsingh3815 Hi Santhosh, I think the issue is related to root user issue. Seems like ur System is not allowing non-root users to access ZK nodes. Can u activate zookeeper-shell and execute "getAcl /config/topics" and see what is the output? the "cdrwa" must be there for sasl type authenitcation. May be this blog helps you - community.cloudera.com/t5/Community-Articles/Zookeeper-Super-User-Authentication-and-Authorization/ta-p/246020
@@DataEngineeringMinds yes I tried here is what I am getting output of the "getAcl /config/topics" 'x509,'CN=localhost%2COU=ssl%2CO=demo%2CL=henrico%2CST=va%2CC=us : cdrwa 'world,'anyone : r
The property `zookeeper.set.acl=true` will force the first kafka broker that connect to zookeeper to modify zookeeper's nodes ACL and the rest of the Kafka brokers won't access then anymore, because it leaves the nodes with world:anyone:r and 509 auth for the first broker. This is the main cause of problems such as Noauth for /brokers/ids etc. I use to set manually the ACLs on each zookeeper node been used by kafka (list below) to avoid this issue and keeping the property `zookeeper.set.acl=false` /controller /brokers /kafka-acl /admin /isr_change_notification /controller_epoch /consumers /config Another Consideration to keep in mind is that zookeeper quorum cluster use to check the alias in the certificate present in the truststore against the machine hostname. Been needed to keep each public key in the truststore instead of a CA if you have hostname verification.
I guess that the idea is to first connect kafka to zookeeper through SSL then kafka creates the ACL in zookeeper. Later you can change the protocol to SASL
Hi bro...thanks for this video... I have struck with an while running the kafka broker..after the server configuration.. The error is Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) apache.zookeeper.KeeperExceptionsNoAuthException: KeeperErrorCode=NoAuth for /config/users/broker-admin-2 By the way im using RHEL os
Anothere great video..but this time I have an error: after modify server.properties and I try to create producer (min 37.17) trying to start kafka I receive the error: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /config/users/broker-admin @Data Engineering Minds Please can help me?
Hi Sunil, sorry for a delayed response. In Part-2, I've implemented PLAINTEXT. Just implement the same along with SASL (skipping the SSL part) in this video. It's very simple. However, never do this on Production.
Hi, Zookeeper security is available only from version 3.5; I mentioned about this in my last video while explaining SSL security. Let me know if you have any questions!!
Hi, To add something to my previous message - ZK supports mTLS from v3.5 and I authenticated my clients to ZK in this video via the same. I don't know how you are trying to authenticate your shell clients to ZK. Could you let me know about this and also Kafka version you are using so that I could provide you more details.
Interesting video, but at the SASL/SCRAM part, following along, all i get is: "[2021-04-18 10:20:18,576] INFO [SocketServer brokerId=0] Failed authentication with localhost/127.0.0.1 (Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512) (org.apache.kafka.common.network.Selector)"
This is definitely authentication issues and has nothing to do with the setup. Please check whether u used same SASL credential across all brokers and whether u made this user a super user.
@@DataEngineeringMinds I used only one broker. SSL section worked without much trouble. I spent maybe 2 hours trying different things without much result. But I'm just starting out on security, that may explain. That was pretty disheartening but I'll give it another shot
@@DataEngineeringMinds I started again, back to part 04, and now it seems to work. I am not sure what is the root cause, maybe the fact that i didn't enable ssl communication between zookeeper and kafka brokers. Anyway, thanks
It does not matter !! Kaka security is independent of zookeeper security. Pls post the config files of ur broker and zookeeper here and I will check to it !!
Thanks for your great video.During the SASL setup when i tried to execute this command sudo bin/kafka-configs --zookeeper localhost:5182 --zk-tls-config-file config/zookeeper-client.properties --alter --add-config 'SCRAM-SHA-512=[password=Dem123]' --entity-type users --entity-name kafka-admin i am getting this error. azureuser@Kafka-VM:/opt/kafka_2.13-3.2.1$ sudo bin/kafka-configs.sh --zookeeper localhost:5182 --zk-tls-config-file config/zookeeper-client.properties --alter --add-config 'SCRAM-SHA-512=[password=Dem123]' --entity-type users --entity-name kafka-admin Warning: --zookeeper is deprecated and will be removed in a future version of Kafka. Use --bootstrap-server instead to specify a broker to connect to. Error while executing config command with args '--zookeeper localhost:5182 --zk-tls-config-file config/zookeeper-client.properties --alter --add-config SCRAM-SHA-512=[password=Dem123] --entity-type users --entity-name kafka-admin' org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /config/users/kafka-admin Please help me on this
1. In this video, I created the topics using kakfa-topics.sh via the port 2181 (unsecured port). Moving forward, we need to create the kafka topics by using the parameter --bootstrap-server rather than --zookeeper for two reasons - 1) Security and 2) Removal of Zookeeper in the future from Kafka ecosystem. I have mentioned the steps to create kafka topics in a secured way in this link - github.com/vinclv/data-engineering-minds-kafka/blob/main/config/sasl_ssl/SecureTopicCreation.md. Please stop creating kafka topics via port 2181 anymore. You can also remove this port from zookeeper property file.
2. Regarding the user creation process via kafka-configs.sh for Windows users (Thanks to the comment from lord byron) => For windows users who are running it in the cmd, the command that you want is basically the same: kafka-configs.sh --zookeeper localhost:2182 --zk-tls-config-file zookeeper-client.properties --entity-type users --entity-name my-user --alter --add-config SCRAM-SHA-512=[password=DEM123], you just need to drop the ' from command
SASL can be standalone used for proving authentication between Client and Server application and Kerberos GSSAPI (one of the SASL mechanism) also provides the method to encrypt the data on source side which can be decrypted on destination side using algo that is agreed upon during context creation.
Best Tutorial I have seen for Kafka , In Depth Knowledge you shared.
Thanks for your feedback. Please help me spreading the channel 🙏
verymuch useful materials and contents for beginners who began learning Kafka.
Really good teacher explaining and detailing every step
Awesome tutorial. I was able to follow step -by -step till last. Thanks for uploading.
Very good videos man, well done. Are you ever going to do one with Keberos and/or OAuth2??
Amazing explanation man. Thanks a lot. When we can see videos regarding kafka-connect ?
Thank you for the tutorial. Clearly explained
SUPER video, thank you :-)
Hii sir, you content had been amazing and i was able to understand it very easily. If possible can you make viedo on how mirror maker 2.0 works and its implementation.
Thanks for your support. I will definitely include as part of the seriez
Your videos are amazing mate. Helped me a lot while working with Kafka on my workplace. Tbh your video was the best I found in regard to this topic. Keep up with good work ;)
Thanks for your support and motivating comments :) Much appreciated :)
Thank you for the video series, it has helped me with setting up secure kafka for a demo at my workplace. I implemented producer and consumer using KafkaJS and able to produce/consume without including the truststore. Does that mean its skipping the broker verification?
Hi Thanks for the video buddy ! Had just one query can we use the same user for kafka clients (producers and consumers property file) which you made superuser in kafka brokers ? or we need to create seperate users for producers and consumers ?
I have followed the tutorial and is able to configure the sasl ssl in my system. If i want to use it via public ip address so that other system can use it ehat do i have to do . Could you help regarding this
For windows users who are running it in the cmd, the command that you want is basically the same: kafka-configs.sh --zookeeper localhost:2182 --zk-tls-config-file zookeeper-client.properties --entity-type users --entity-name my-user --alter --add-config SCRAM-SHA-512=[password=DEM123], you just need to drop the ' from command :) cheers
thanks again. I will add it to the pinned comment :)
I am using Amazon msk, my consumer consumes messages and suddenly in between I get error " not Authorization to access topic" or " not authorised to access join consumer group". This seems wierd, it works and suddenly gives these errors randomly.
Million thanks bro :).
You’re welcome and thanks for your support. Please don’t forget to check the “pinned comment” for the latest update related to this video.
Really good 👏👏👏
Hi It was really helpful . I am facing one problem where I am getting error like :-No LoginModule found for org.apache.common.security.scram.ScramLoginModule . Are we missing some jass configuration or something in this video whihc is creating this issue ?
Excellent
How do I configure Mirror Maker 2 with SSL/SASL ??? as a broker client ???
I am not able to consume older msg from. Topic after follow your steps. Please suggest
Could you please create video on AWS MSK with SASL_SSL enabled with public access?
Great video, Thanks. Can you add the credential creating commands to the github repo, that will be very helpful .
Thanks for your comments and suggestions. I updated the repo now. Could you please check here - github.com/vinclv/data-engineering-minds-kafka/blob/main/config/sasl_ssl/README.md
can you extend this with a Spring Boot based Producer and consumer application with SASL
Hi Did you cover Kafka with Kerberos?
Hi, well explained .. thanks a ton. One doubt
Kafka-acls are zk clients they are connected to zk using secured port and using zk-clients.properties as ACls are stored in zk ..in confluent documentation Kafka-acls command they are giving -bootstrap-server .. why ..plz let me know
Hi Chandan, u r right; it is indeed zookeeper client. If u look into this ticket - cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum, they are going to shutdown zookeeper completely very soon. This means all management activities done by zookeeper will be taken care by brokers itself in future and that's the reason they suggesting u to use --bootstrap-server instead. However, I checked the latest version of Kafka and this is not completely migrated aand hence u can continue using zookeeper way as I did in this video (u will just get a warning which u can ignore). Please remember this is applicable to any ZK client in the future. I explained it for kafka-configs.sh in this video (check from 17:50).
@@DataEngineeringMinds thanks again ..
THank you very much for this!
Thanks for your support. Please help me spread this channel 😊🙏
Great video
Thanks for your support.
Excellent video !!! Working fine with SASL_SSL using console producer and console consumer ,
any sample links for code to create producer and consumer in node JS.? it will me much helpful. shortage of developers :)
Thanks a lot for your support. Unfortunately, I've never used JS so far for any of my use cases.I will check for sample code and let u know soon. Please send me a reminder in case i forget it :)
Great Explanations. Is it necessary to create a topic before enabling SASL? I was able to follow everything when creating a topic before enabling SASL but after enabling SASL I am not able to create any topic. I am getting an error.
Can u please post the error message here? Also did I restart the cluster (incremental restart - one broker at a time) after enabling SASL?
@@DataEngineeringMinds Sorry for the late reply. I am getting the following error
kafka-topics.sh --zookeeper localhost:2181 --create --topic test --replication-factor 1 --partitions 1
Error while executing topic command : KeeperErrorCode = NoAuth for /config/topics/test
[2021-02-08 11:06:49,504] ERROR org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /config/topics/test
at org.apache.zookeeper.KeeperException.create(KeeperException.java:120)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:564)
at kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1644)
at kafka.zk.KafkaZkClient.createOrSet$1(KafkaZkClient.scala:364)
at kafka.zk.KafkaZkClient.setOrCreateEntityConfigs(KafkaZkClient.scala:374)
at kafka.zk.AdminZkClient.createTopicWithAssignment(AdminZkClient.scala:93)
at kafka.zk.AdminZkClient.createTopic(AdminZkClient.scala:57)
at kafka.admin.TopicCommand$ZookeeperTopicService.createTopic(TopicCommand.scala:353)
at kafka.admin.TopicCommand$TopicService.createTopic(TopicCommand.scala:196)
at kafka.admin.TopicCommand$TopicService.createTopic$(TopicCommand.scala:191)
at kafka.admin.TopicCommand$ZookeeperTopicService.createTopic(TopicCommand.scala:345)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:62)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
(kafka.admin.TopicCommand$)
@@santoshsingh3815 Hi Santhosh, I think the issue is related to root user issue. Seems like ur System is not allowing non-root users to access ZK nodes. Can u activate zookeeper-shell and execute "getAcl /config/topics" and see what is the output? the "cdrwa" must be there for sasl type authenitcation. May be this blog helps you - community.cloudera.com/t5/Community-Articles/Zookeeper-Super-User-Authentication-and-Authorization/ta-p/246020
@@santoshsingh3815 Hi .. Did u have any luck?
@@DataEngineeringMinds yes I tried here is what I am getting output of the "getAcl /config/topics"
'x509,'CN=localhost%2COU=ssl%2CO=demo%2CL=henrico%2CST=va%2CC=us
: cdrwa
'world,'anyone
: r
The property `zookeeper.set.acl=true` will force the first kafka broker that connect to zookeeper to modify zookeeper's nodes ACL and the rest of the Kafka brokers won't access then anymore, because it leaves the nodes with world:anyone:r and 509 auth for the first broker.
This is the main cause of problems such as Noauth for /brokers/ids etc.
I use to set manually the ACLs on each zookeeper node been used by kafka (list below) to avoid this issue and keeping the property `zookeeper.set.acl=false`
/controller
/brokers
/kafka-acl
/admin
/isr_change_notification
/controller_epoch
/consumers
/config
Another Consideration to keep in mind is that zookeeper quorum cluster use to check the alias in the certificate present in the truststore against the machine hostname. Been needed to keep each public key in the truststore instead of a CA if you have hostname verification.
Thanks for inputs
I guess that the idea is to first connect kafka to zookeeper through SSL then kafka creates the ACL in zookeeper. Later you can change the protocol to SASL
Hi bro...thanks for this video... I have struck with an while running the kafka broker..after the server configuration..
The error is
Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) apache.zookeeper.KeeperExceptionsNoAuthException: KeeperErrorCode=NoAuth for /config/users/broker-admin-2
By the way im using RHEL os
pls anyone reply ...who knows the solution
Anothere great video..but this time I have an error: after modify server.properties and I try to create producer (min 37.17) trying to start kafka I receive the error: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /config/users/broker-admin
@Data Engineering Minds Please can help me?
is it possible kafka security without zookeeper security
hi,
can you explain how to implement SASL_PLAINTEXT for kafka security
Hi Sunil, sorry for a delayed response. In Part-2, I've implemented PLAINTEXT. Just implement the same along with SASL (skipping the SSL part) in this video. It's very simple. However, never do this on Production.
Hi. Could you also share how to authenticate kafka-acls.sh & other shell clients with zookeeper version 3.4.10
Hi, Zookeeper security is available only from version 3.5; I mentioned about this in my last video while explaining SSL security. Let me know if you have any questions!!
Hi, To add something to my previous message - ZK supports mTLS from v3.5 and I authenticated my clients to ZK in this video via the same. I don't know how you are trying to authenticate your shell clients to ZK. Could you let me know about this and also Kafka version you are using so that I could provide you more details.
is it possible kafka security without zookeeper sasl if possible please share the steps
Yes possible.
Interesting video, but at the SASL/SCRAM part, following along, all i get is:
"[2021-04-18 10:20:18,576] INFO [SocketServer brokerId=0] Failed authentication with localhost/127.0.0.1 (Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512) (org.apache.kafka.common.network.Selector)"
This is definitely authentication issues and has nothing to do with the setup. Please check whether u used same SASL credential across all brokers and whether u made this user a super user.
@@DataEngineeringMinds I used only one broker. SSL section worked without much trouble.
I spent maybe 2 hours trying different things without much result. But I'm just starting out on security, that may explain.
That was pretty disheartening but I'll give it another shot
@@josephmbimbi if u still did not find luck, post ur broker config here. I will check it out !!
@@DataEngineeringMinds I started again, back to part 04, and now it seems to work.
I am not sure what is the root cause, maybe the fact that i didn't enable ssl communication between zookeeper and kafka brokers.
Anyway, thanks
It does not matter !! Kaka security is independent of zookeeper security. Pls post the config files of ur broker and zookeeper here and I will check to it !!
Thanks for your great video.During the SASL setup when i tried to execute this command sudo bin/kafka-configs --zookeeper localhost:5182 --zk-tls-config-file config/zookeeper-client.properties --alter --add-config 'SCRAM-SHA-512=[password=Dem123]' --entity-type users --entity-name kafka-admin
i am getting this error.
azureuser@Kafka-VM:/opt/kafka_2.13-3.2.1$ sudo bin/kafka-configs.sh --zookeeper localhost:5182 --zk-tls-config-file config/zookeeper-client.properties --alter --add-config 'SCRAM-SHA-512=[password=Dem123]' --entity-type users --entity-name kafka-admin
Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Error while executing config command with args '--zookeeper localhost:5182 --zk-tls-config-file config/zookeeper-client.properties --alter --add-config SCRAM-SHA-512=[password=Dem123] --entity-type users --entity-name kafka-admin'
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /config/users/kafka-admin
Please help me on this