Kafka Streams 101: Getting Started (2023)

► TRY THIS YOURSELF: cnfl.io/kafka-streams-101-mod...
In this course, Sophie Blee-Goldman (Apache Kafka® Committer and Software Engineer) gets you started with Kafka Streams.
Kafka Streams is declarative, so you state what you want to do, rather than how to do it.
Imagine that you have a topic, from which you'd like to filter all records marked with the color "red." You could accomplish this with plain Kafka, but the equivalent Kafka Streams code would only take a third of the lines.
For a complete IMMERSIVE HANDS-ON EXPERIENCE: cnfl.io/kafka-streams-101-mod...
--
ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion - designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.
#kafka #kafkastreams #streamprocessing #apachekafka #confluent

[@johnkelty4001]

Thanks, this is very helpful info in a short amount of time!

[@sonnyps413]

4:50 Is when she starts explaining about Kafka Stream

[@elbowdashizzle]

The subscribe button in RUclips lit up when you said “subscribe to the topic” 😂

[@tomknud]

Good talk, thanks

[@mehrdadk.6816]

That was a great introduction

[@krishnasharma-nr3yg]

Thanks, very helpful

[@ankitlakum1]

Good info 😊

[@kartikprabakara3126]

Excellent

[@SzymekCRX]

Thank You!

[@jasper5016]

Fantastic tutorial. Thanks so much Sophie!

[@sapientum8]

TY

[@komiljonaliev3415]

Thank you for giving helpful information

[@karthickeayan]

okay my long pending doubt is cleared now 🙏🏽

[@javadahmadzadeh7129]

Thanks. it was really helpful but for me as a non-native English speaker, the speed of talking was so fast, and I had to use subtitles for better understanding

[@ighsight]

Excellent. Wondering can I use Kafka Streams with the KafkaJS library for Node JS?

[@adityanjsg99]

you may.. i am using with python

[@avimehenwal]

Does kafkajs in nodejs provide support for kafka streaming API ?

[@stream2learn]

Can we use Kafka Streams and Spark Streaming in the same project ?

[@ConfluentDevXTeam]

Hi, I'm Gilles Philippart from Confluent. I don't think there's anything that would prevent you from doing so, Kafka Streams is just another library that you pull into your application dependencies after all. I'm not sure there would be a good reason to do that though, as it might prove harder to tune the performance, scale, secure, troubleshoot, etc. Have you considered creating another microservice that would consume or produce to a topic to interact with the Spark Streaming app?

[@MrBist-yr4sy]

Kafka.
It’s no difference than philosophy names and definitions and describing how one thing or human can go from point A to point B.

[@shyammohan2007]

If each topic is a file, how is about a partition? Does it means the aforesaid file will be divided into a subset of file again where each sub-file is a partition?

[@ConfluentDevXTeam]

Hi Shyam, I'm Gilles Philippart from Confluent.
Sophie says things like "This log is a file and it's a special kind of file" then later on "the topic itself really is just a name that is given to the log. So on the broker, they store each log and directory names by that topic".
I agree that this part is a bit confusing as we normally associate a " Log" to a "Partition" not a topic. Sophie glossed over the details of how the data is stored physically on disk because that's not really the purpose of the course, but let me get this right:
1. A "topic" is a logical stream of records, and it is the primary unit of data organization in Kafka.
2. A "topic partition" is a logical unit that represents a subset of the data within a Kafka topic. Each partition is an ordered and immutable sequence of records.
3. Now, Kafka stores the data for each topic partition in a set of "log segments". Each log segment is a file on disk and represents a sequential chunk of data. The size of each log segment is configurable and typically ranges from a few kilobytes to several gigabytes. As new records are appended to a partition, they are written to the active log segment.
For example, the directory structure of the first partition of the "account-deposits" topic would look like this:
/var/lib/kafka/data/account-deposits-1
00000000000047926734.log
00000000000047926734.index
...
00000000000052497535.log
00000000000052497535.index
If you want to learn about Kafka internals, check out developer.confluent.io/learn-kafka/architecture/broker/
Hope this helps!

[@sghsghdk]

So kafka streams is just a convinience client api? will the network traffic be equivalent?

[@gillesphilippart5986]

Kafka Streams is a streaming processing library to join streams (topics) and simplify the building of stateful applications (windowing, aggregations). If might consume more network traffic due to the fact that most of the local state is also stored in Kafka for backup, but that's usually a good thing as it make your application more reliable!

[@dzilen]

🔥🔥🔥

[@NoorquackerInd]

Kafka Streams, aka reducing boilerplate by transitioning to functional programming

[@AndriiKovalov]

Very confident

[@tachikomakusanagi3744]

Very strange choice of name, definitely not going to lead to concerns of big tech overreach :)

[@billukancha8078]

Give some real world example not the virtual one please 🙄

[@casiojapi]

estoy enamorado

[@Wyvernnnn]

Don't be that guy

[@denisdominguez7287]

Oye concentrate que esto es técnico 😁😁😁