Kafka Spark Streaming Integration in java from scratch | Code walk through

Video explained: How to setup one data pipeline with help of Kafka - Spark Steaming integration, End to End.
All code and test.csv used in demo, can be download from github.com/bin...
Apache Kafka with spark streaming | Implementing Spark Steaming with Apache Kafka.
Spark Structured Streaming with Kafka using Java, Streaming Data Pipelines Demo, Read CSV from Kafka Topic using spark structured Streaming.
Kakfa Installation • How to install Apache ...
Kafka Introduction: • What is Apache Kafka |...
Kafka python • Kafka Tutorial | Pytho...
Kafka Java • Kafka Tutorial | Java ...
Spark installation on Windows | without Virtual box | Hands on
• Apache Spark Installat...
Spark Introduction • Apache Spark quick int...

[@prafullvernekar9217]

Very Nice tutorial Mr. Suman
You made it very simple
Thanks I appreciate

[@rezahamzeh3736]

@Binod Suman Academy any plan for creating a similar project in Python?! That would be great!

[@legohistory]

Update for 2022:
_For the KafkaProducer:_
Use following code
*"Stream fileStream = Files.lines(Paths.get(CsvFile));"*
instead of
*"URI uri = getClass().getClassLoader().getResource(CsvFile).toURI();*
*Stream FileStream = Files.lines(Paths.get(uri));"* and give the ABSOLUTE path to the CSVFILE variable
Also delete the _URIException_ exception in the header of the method.
_For the SparkKafkaConsumer:_
Add *"throws InterruptedException"* to main method header.

[@alladeabdou7539]

update for 2022 successfully working ! GG

[@emanuelebarbalaco720]

a very useful guide.
I was expecting the last step where the analysis with mllib is done. But very good video

[@binodsuman]

Thanks for motivating me. Good to know Kafka Spark Tutorial video you liked. Please visit ruclips.net/video/cfiXvwnCz3E/видео.html for all Spark related videos.

[@80sudha]

Excellent

[@erummehmood7219]

While running SparkKafkaConsumer.java, I got the following error:
Exception in thread "main" java.io.IOException: failure to login
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:700)
Caused by: javax.security.auth.login.LoginException: No LoginModule found for com.sun.security.auth.module.NTLoginModule
at java.base/javax.security.auth.login.LoginContext.invoke(Unknown Source)
Can u please help fixing this error?

[@sasikanthpvs]

Nice keep up the good work!

[@prafullvernekar9217]

error: no suitable method found for createDirectStream(JavaStreamingContext,Class,Class,Class,Class,Map,Set)
while compiling SparkKafkaConsumer.java

[@sheikhparvez6192]

same problem here, can someone please help??

[@souleymanebendaoudasow2571]

@@sheikhparvez6192 Any help ?

[@kainaazslittlemomentsbigme3410]

same error. can some one has working solution with updated dependencies?

[@shoaibd5006]

I was very good explanation. Thanks

[@shubhi2648]

Hi, I did go through the video What Is Apache Kafka (in the same playlist)
Are there any other prererequisite(s) to this video where you explain what actually is spark and spark streaming (feeling a little disconnect here)
Pls share with me the link if any video.
thanks sir

[@binodsuman]

Thanks Shubhi. Please follow this playlist ruclips.net/video/cfiXvwnCz3E/видео.html , it has all Spark related videos. Kindly let me know if it is useful for you. Keep learning !!

[@shubhi2648]

@@binodsuman Thanks a lot sir for your prompt reply.
Already excited to go through this new playlist.. !

[@AjithKumaraTube]

Thanks for the video.. Simple and nice..

[@hatberry]

Thank you it easy to understand. Could you please share code , I cannot find exact code in github.

[@abhishekkushare7878]

Hello , Sir I have put as CSV file in resources folder as you shown but still following error is occurring......
Exception in thread "main" java.lang.NullPointerException: Cannot invoke "java.net.URL.toURI()" because the return value of "java.lang.ClassLoader.getResource(String)" is null
at trial_kafka.kafka.Kafkaproducer.PublishMessages(Kafkaproducer.java:49)
at trial_kafka.kafka.Kafkaproducer.main(Kafkaproducer.java:38)
49 th line is this.....
--> URI uri = getClass().getClassLoader().getResource(CsvFile).toURI();
And 38 th is CSVKafkaProducer.publishmessgaes()
Please Guide me........

[@kumaraneesh1420]

Good one

[@NitishGondkar]

Please add referential documents in the description to follow along with video. Your copy paste is not helping in code walkthrough

[@namratamusale937]

How can I store the csv file generated in Hadoop ? Can you help with the code

[@kaushalkumar3017]

as usual easy to understand. please create python version also. thanks.

[@krishnamuduli6276]

Thanks for detailed information...

[@binodsuman]

Hello Krishna, Thank you for all nice words. Happy to know this Apache Kafka Spark integration Tutorial Video helped you to learn. Keep Learning !!

[@rezahamzeh3736]

@@binodsuman any plan for creating a similar project in Python?! That would be great!

[@udupi123456]

Good work Thanks a lot

[@binodsuman]

Thank you for all nice words. Glad to know this Apache Kafka Spark integration Tutorial Video helped you to learn. Keep Learning !!

[@parthvora8242]

Sir getting same error after java 1.8 in spark folder

[@theoscie16]

thanks for this vedio,please create this video in python .

[@amit22130]

Good Going :)

[@binodsuman]

Happy to know Amit, this SPARK Streaming Tutorial series helped you to learn something new. Keep Learning and thank you for your nice words !!

[@balasubramaniamjanakiraman7023]

Duration is 2 minutes and that caused delay in consuming message

[@legohistory]

20000 ms are 20 s