- Видео 9
- Просмотров 16 080
Aditya Sridhar
Добавлен 29 май 2014
I am Aditya, a software engineer, a curious techie, and a part-time tech blogger. I work extensively on big data technologies operating on massive volumes of data. I like teaching and create videos on various big data technologies like Apache Pinot, Apache Kafka, etc.
Kafka Connect Standalone Mode Example
This video shows how Kafka connect can be set up using standalone mode
After the Kafka connect cluster is set up, this video also shows how a file sink connector can be deployed in Kafka connect
Even if you are a beginner to Apache Kafka and Kafka connect, you can still follow the steps in this video easily to setup Kafka connect in standalone mode
Link to Kafka setup video: ruclips.net/video/_-Kuiv5ozdg/видео.html
After the Kafka connect cluster is set up, this video also shows how a file sink connector can be deployed in Kafka connect
Even if you are a beginner to Apache Kafka and Kafka connect, you can still follow the steps in this video easily to setup Kafka connect in standalone mode
Link to Kafka setup video: ruclips.net/video/_-Kuiv5ozdg/видео.html
Просмотров: 3 098
Видео
Kafka Connect Distributed Mode Example
Просмотров 3,2 тыс.3 года назад
This video shows how Kafka connect can be set up using distributed mode After the Kafka connect cluster is set up, this video also shows how a file sink connector can be deployed in Kafka connect Even if you are a beginner to Apache Kafka and Kafka connect, you can still follow the steps in this video easily Link to Kafka setup video: ruclips.net/video/_-Kuiv5ozdg/видео.html
5 Common Mistakes in a Kafka Setup
Просмотров 3403 года назад
This video talks about 5 common mistakes in a Kafka setup Apache Kafka is very powerful, but if the configurations are not set right, then it can cause major issues in production. In this video, we will see the 5 major mistakes which one does while installing Apache Kafka in their system. If you are new to Apache Kafka then this video will guide you in ensuring that you provide the right settin...
How to install Kafka in 2 minutes
Просмотров 3633 года назад
In this tutorial, we will install Kafka from scratch in 2 minutes If you are new to Kafka or you are a beginner to Kafka, this tutorial will explain how to set up Kafka within 2 minutes In this tutorial, we will set up a single broker Kafka cluster. We will also create a topic and produce and consume some messages from this topic #apachekafka #kafkatutorial #beginner
Kafka Partitions and Replication Explained
Просмотров 8133 года назад
This tutorial covers the concepts of Partitions and Replication in Apache Kafka. This video gives an introduction to how partitions and replication work in Apache Kafka. If you are a beginner to apache Kafka, then this video will be very helpful in understanding the core concept behind how Kafka works. This Apache Kafka video also has a demo to show how a Kafka topic can be created with multipl...
Create your first Kafka streams application | Kafka streams Tutorial
Просмотров 2,4 тыс.3 года назад
This Tutorial guides you in creating your first Kafka streams application. This tutorial will use Apache Kafka 2.8.0 and also use Kafka streams library 2.8.0 If you are a beginner to Kafka streams, then this video will give you a good idea of how to get started with Kafka streams Link to the GitHub repo for the code written as a part of this video: github.com/aditya-sridhar/kafka-streams-demo #...
How to Install Kafka without Zookeeper
Просмотров 5 тыс.3 года назад
This Kafka tutorial shows how Apache Kafka can be installed without Zookeeper As of the date of creation of this video, using Kafka without Zookeeper is still an experimental feature and not meant for production use By installing kafka without Zookeeper, kafka can support millions of partitions. KIP-500 shows how Kafka can be installed and used without Zookeeper by using kraft If you have never...
What is Pinot Data Explorer | Apache Pinot User Interface
Просмотров 3863 года назад
Apache Pinot is a real-time analytics system. This video explains about the Pinot data explorer. If you are interested in learning about Pinot, you can get my udemy course at a discount by using the following link www.udemy.com/course/apache-pinot-a-hands-on-course/?couponCode=PINOT_JULY_2021
Apache Pinot in less than 5 minutes | Introduction to Apache Pinot | What is Apache Pinot
Просмотров 2743 года назад
What is Apache Pinot? Apache Pinot is a real-time distributed analytics system. This tutorial will give a you quick Introduction to Apache Pinot in less than 5 minutes. Even if you are a beginner to real-time analytics, this video will give you a good idea of what Apache Pinot is If you are interested in learning more about Apache Pinot, you can get the course at a discounted price by using thi...
Good Morning, Aditya, I saw two your courses on Udemy for Apache Pinot, currently, i have a subscription for Udemy but your courses are not covered in it and I am not in a financial situation to afford them, can you please make an exception to gift it to me ? So, whenever, i grab a job in the coming 6 months, i re-pay you for that later, I would appreciate your response even if it is a "no".
Thank you sir
20 + enters...
Super
Hi Adiya, this is great to understand. currently latest version is 3.5.1, could please provide the video for production setup with karafka consumer
Glad it helped. Sure will try to create a video on the latest version
you should've ran atleast 2 kafka connect instance and run multiple connectors and shown how they are distributed
Will try creating a video on this as well
It is very helpful video Aditya. Tysm
Thanks Ganesh. Glad it is helpful
hey on 14:40 you wrote 00000000000000.log , what is that? when i write it it shows NoSuchFileException
Thank you for creating such an exceptional video! I thoroughly enjoyed it and learned a lot from it. Keep up the outstanding work, and I look forward to watching more of your content in the future.
Glad the video helped. Thanks
Video quality is bad - some of the configs are not readable. Host is best though.
My Observation : 1) when starting connect, follow the proper sequence of config files, the first one should be the worker config followed by the connector config. 2) In case you are getting error for connector jar not found. It is usually located in /libs folder, adding its path with the plugin.path property in the worker config should resolve the issue.
hey really nice video ..very helpful i was able to setup kafka_2.12-3.4.0 on my local machine now will deploy to my enterprise environment and will monitor how it performs with large data in tbs
Documentation claims that system can handle more kafka partitions by removing zookeeper
@@AdityaSridhar that's great that means i can increase no of consumers also for faster processing and high throughput
Yes. Consumers can be scaled up
A cluster should ideally be on different servers, right? will the same steps work if we use different servers (just by changing localhost to server IP in config files)
Yes it will work. It should be in different servers in a production setup
@AdityaSridhar, You ran all kafka instances on the same machine(server)? in an actual cluster it would be separate servers, correct?
Yes that is right
Tried on windows ?
curl: (7) Failed to connect to localhost port 8083 after 0 ms: Connection refused The port is not used on my machine Any solution
do you have any example with real database and how can we install that connector
UGH, blue on black and red on black should never be a thing. It is very hard to see what you are typing.
hi. thanks for the video in this same way possible to connect mongodb connector
thank you very much!
Nice explanation step by step
my kafka connect say Failed to find any class that implements Connector and which name matches FileStreamSink, how can I fix it?
Starting with version 6.2.1, the FileStream Sink and Source connector artifacts have been moved out of Kafka Connect
Can I used FileStreamSinkConnector instead @@marciovitormatos3997 ?
Yes, with kafka 3.6.1 you can used FileStreamSinkConnector but you need to add the following line to the first property file plugin.path=<install_dir>/kafka_2.12-3.6.1/libs/connect-file-3.6.1.jar
Hello, I need a script for Tls contact visa slot
nice explanation in simple words. You have given me kick start.
Glad it was useful
Excellent video! Its been to difficult to find a brief and concrete explanation on how to start Kafka Connect without the help of any other third party companies, like confluent. It seems that no one wants to find out how kafka it self works.
Thank you
Very well explained
Thank you
hey its a good demo, can you make a video with file source connector. When i tried it i was unable to do it.
Sure. Will create a video on the same.
liked the way you present it..you are to the point...keep it up............can you create a video on how Kafka is used in real-world...like how we implement it in production
Sure. Have created a video on some prod setup configurations here. ruclips.net/video/rNrf_LIjaIo/видео.html
hi there, I have a confusion please clear me out......Is it possible to implement Kafka connect on the java program if yes how can we ....if you have any documentation on that please share and help me out. thank you.
Kafka connect is already available out of box and you just need to set it up and run any connector you want as shown in the video. Only if you want to build a custom connector which is not available, you need to write a java program
Im get this error while running ./kafka-storage.sh random-uuid Error: Could not find or load main class kafka.tools.StorageTool Caused by: java.lang.ClassNotFoundException: kafka.tools.StorageTool
Try setting up Kafka 2.8 again and try running the command bin/kafka-storage.sh from the main kafka folder
Hello, I recently enrolled to your Apache Pinot Course on udemy. And I am working on that technology now. I need some help regarding ingesting data from ADLS. Can you please help me ?
hi, how about ACL and authentication ?
Authentication should still work with SSL. Regarding ACLs, We will need to wait for the production version of this to come out since this is still experimental.
Hello What are the other modes for running connectors? Like u did it in daemon mode
And if I need json converter not string then what are the more changes I need to do for that ? Plz help me
You can run it without Daemon mode. But that will directly output the connector logs in your console.
You can point the converter to org.apache.kafka.connect.json.JsonConverter
@@AdityaSridhar thank you
This is a great tutorial, thanks. I was able to configure a distributed cluster on my pi cluster using this.
Thanks. Kafka cluster on a pi cluster sounds really interesting.
Really nice video explaining how to set up Kafka without zookeeper 👍
Thank you
Your Content and Storytelling is azaming. Can you also give tutorial in Scala
Thank you for the feedback. Will create a Scala tutorial video in the future as well.