As always if you liked the video, I would be happy if you leave a thumbs up and subscribe for more DevOps content 🙂 ► This video was sponsored by CNCF - www.cncf.io 🙌🏼 ▬▬▬▬▬▬ T I M E S T A M P S ⏰ ▬▬▬▬▬▬ 0:00 - Intro 0:06 - Why we need log data? 0:29 - Challenges of Logging 1:21 - Challenges of Logging in Kubernetes 3:21 - How does Fluent Bit work? 6:47 - Fluent Bit in Kubernetes 7:21 - Advantages of Fluent Bit 8:26 - Fluent Bit vs. Fluentd
It wasn't clear to me near the end the difference between fluent bit and fluentd. From the initial comparison it makes it seem like fluent bit is superior in terms of resource usage and overall footprint, but then we can combine them, but why? Is it that fluent bit sacrifices feature set to enable this better perf and therefore we should prefer it until we need something more advanced?
They do overlap in functionality quite a bit, and you can use fluent-bit by itself quite a bit of the time. The difference is that fluent-bit is designed as a log forwarder whereas fluentd is designed as a log aggregator. fluent-bit is meant to sit with the services producing the logs, preprocessing them before forwarding them on, whereas fluentd is intended to receive log data over a network.
@@talideon hmm, I think I see what you're saying. From what I'm understanding, when I use fluentd, I have td-agent which is by the service which sends the actual logs to fluentd, but it sounds like fluent-bit can essentially replace td-agent in that it sits on the machine where the service is to get,process, then forward the logs. so in my scenario, I have services+(td-agent for each machine that the services are on) -> fluentd -> outputs like elasticsearch for fluentbit, it sounds like you can do either A) services+(fluentbit for each machine that the services are on) -> outputs like elasticsearch? or B) services+(fluentbit for each machine that the services are on) -> fluentd (to collect the logs all in one spot) -> outputs like elasticsearch It would seem like B would be better if you have multiple outputs you are sending to, as then you just have 1 place to change that. I.e. if you wanted to also send info to DataDog, just change it in fluentd, where if you had 5 different machines each with their own fluentbit, you'd have to change the configuration of all of them if you wanted them all to start sending to a new service. Which may or may not be a big effort depending on how your architecture is setup, but either way it's messier since you have to make the change and ensure it's consistent on more machines. The extra step though to aggregate/collect all the logs in one place at fluentd is extra work/processing though. Fluentbit sounds a lot lower cost in terms of memory use though. I'd be curious how fluentbit compares to td-agent edit, I read that "td-agent is a stable distribution package of Fluentd", which I think means that td-agent is actually fluentd or a portion of it. That said, it sounds like fluent-bit is preferred over td-agent for the forwarding portion at least on each machine that has the service you wish to get logs from. I'm curious as to the reliability, as I heard fluentd can store logs on hard drive, and use that as a means to hold/send the logs later if something goes down. I wonder if fluent-bit has that feature
@@timothyn4699 I know that this is a year late but fluentbit is the way to go now with Open Telemetry becoming the industry standard observability collector and protocol. The best practice would be to have fluentbit running on your servers or in your k8s clusters and have them forward logs and metrics to an otel collector cluster over HTTP. The otel collector can then send data to any backend tool that supports it (which is basically all of the big ones) so you can visualize them with Grafana, New Relic, etc. The method of running a log scraper and then sending them to an aggregator like Fluentd works fine but it is much easier to maintain an observability stack across thousands of apps with the fluentbit -> otel collector -> observability backend pattern
Hi Nana, would like to Learn more on to troubleshooting pod errors such as crashloppback off, evicted pods, probe failures. Please make a video on that
Dear Nana, good afternoon can you please , bring up a video to explain how i can use fluent bit and store data in an s3 bucket, then to be visualized in grafana?
I will add something to it. As u mentioned it is implemented as a demonset. But I feel it has more use case when implemented as a side car. This I have implemented in few projects
please kindly make us a video on how to install and configure fluent Bit in k8s cluster, and link the collected logs and deposit it in an s3 bucket ...
You are the true DevOps guru !! Like the way you explain. 🙌 The DevOps tool of the month series is awesome ❤️ Btw, I have a query here, if you can help to answer. Considering a large amount of data to be ingested and analyzed for metrics purpose, do you suggest Fluentbit over Loki (coming from Grafana Labs). To me Fluentbit is much appealing in terms of low footprint and ability to serve a variety of different landscape.
Great video as always! Thanks, Nana. Can you please make a video where you give a quick demo of how fluent bit works, how to create config files, etc. :)
It is very usefull for me I have updated fluent-bit-splunk helmchart version from version 0.12.3 to 0.16.2 but container is not up , here I am getting error message like"backoff"", how to resolve this issue and can u please help me out this
![[DOKKAN BATTLE] Worldwide Campaign Announcement Video Part 2!](http://i.ytimg.com/vi/JpT5Voak6WA/mqdefault.jpg)
As always if you liked the video, I would be happy if you leave a thumbs up and subscribe for more DevOps content 🙂
► This video was sponsored by CNCF - www.cncf.io 🙌🏼
▬▬▬▬▬▬ T I M E S T A M P S ⏰ ▬▬▬▬▬▬
0:00 - Intro
0:06 - Why we need log data?
0:29 - Challenges of Logging
1:21 - Challenges of Logging in Kubernetes
3:21 - How does Fluent Bit work?
6:47 - Fluent Bit in Kubernetes
7:21 - Advantages of Fluent Bit
8:26 - Fluent Bit vs. Fluentd
Wow, you are one of the best teachers on RUclips in the IT space. Thank you Nana :)
Thanks Parth, appreciate your nice feedback! :)
It wasn't clear to me near the end the difference between fluent bit and fluentd. From the initial comparison it makes it seem like fluent bit is superior in terms of resource usage and overall footprint, but then we can combine them, but why? Is it that fluent bit sacrifices feature set to enable this better perf and therefore we should prefer it until we need something more advanced?
They do overlap in functionality quite a bit, and you can use fluent-bit by itself quite a bit of the time. The difference is that fluent-bit is designed as a log forwarder whereas fluentd is designed as a log aggregator. fluent-bit is meant to sit with the services producing the logs, preprocessing them before forwarding them on, whereas fluentd is intended to receive log data over a network.
@@talideon hmm, I think I see what you're saying. From what I'm understanding, when I use fluentd, I have td-agent which is by the service which sends the actual logs to fluentd, but it sounds like fluent-bit can essentially replace td-agent in that it sits on the machine where the service is to get,process, then forward the logs.
so in my scenario, I have services+(td-agent for each machine that the services are on) -> fluentd -> outputs like elasticsearch
for fluentbit, it sounds like you can do either
A) services+(fluentbit for each machine that the services are on) -> outputs like elasticsearch?
or
B) services+(fluentbit for each machine that the services are on) -> fluentd (to collect the logs all in one spot) -> outputs like elasticsearch
It would seem like B would be better if you have multiple outputs you are sending to, as then you just have 1 place to change that. I.e. if you wanted to also send info to DataDog, just change it in fluentd, where if you had 5 different machines each with their own fluentbit, you'd have to change the configuration of all of them if you wanted them all to start sending to a new service. Which may or may not be a big effort depending on how your architecture is setup, but either way it's messier since you have to make the change and ensure it's consistent on more machines.
The extra step though to aggregate/collect all the logs in one place at fluentd is extra work/processing though. Fluentbit sounds a lot lower cost in terms of memory use though. I'd be curious how fluentbit compares to td-agent
edit, I read that "td-agent is a stable distribution package of Fluentd", which I think means that td-agent is actually fluentd or a portion of it. That said, it sounds like fluent-bit is preferred over td-agent for the forwarding portion at least on each machine that has the service you wish to get logs from. I'm curious as to the reliability, as I heard fluentd can store logs on hard drive, and use that as a means to hold/send the logs later if something goes down. I wonder if fluent-bit has that feature
@@timothyn4699 I know that this is a year late but fluentbit is the way to go now with Open Telemetry becoming the industry standard observability collector and protocol. The best practice would be to have fluentbit running on your servers or in your k8s clusters and have them forward logs and metrics to an otel collector cluster over HTTP. The otel collector can then send data to any backend tool that supports it (which is basically all of the big ones) so you can visualize them with Grafana, New Relic, etc.
The method of running a log scraper and then sending them to an aggregator like Fluentd works fine but it is much easier to maintain an observability stack across thousands of apps with the fluentbit -> otel collector -> observability backend pattern
Perfect timing: I'm currently working on using fluent bit with elasticsearch
Incredible simple explanations. Really thank you for sharing your time.
Thank you, glad you liked it Gerardo :)
Hi Nana, thanks for the Awesome content. Waiting for Jaeger with gRPC +Kafka.
Thanks Nana with many concise information. A ton of information include in 10 minutes video.
Thanks for explaining this tool so precisely, covering all the aspects. Loved it!
I love your vids! Must have watched k8s at least 10 times haha
Happy to hear Anya 😀
As usual great content. Clearly explained and with the perfect detail level for this media platform
Thanks José, happy to hear you find it useful :)
Hi Nana, would like to Learn more on to troubleshooting pod errors such as crashloppback off, evicted pods, probe failures.
Please make a video on that
As always, gr8 tech video under 10min nugget. Would be nice to see that same in action
Thank you Nana! You are the best :) I love how you make all complex topics seem simple :)
Very helpful video, thanks Nana!
Thanks nana for amazing fluentbit video. Please make a next video about detailed installation video of fluentbit
Thank you, Nana! Summarized and to the point! :)
i bet nobody understood what are the differences between fluent bit and fluentd from this video
thank you though for making these courses/videos
Dear Nana, good afternoon can you please , bring up a video to explain how i can use fluent bit and store data in an s3 bucket, then to be visualized in grafana?
I will add something to it. As u mentioned it is implemented as a demonset. But I feel it has more use case when implemented as a side car. This I have implemented in few projects
please kindly make us a video on how to install and configure fluent Bit in k8s cluster, and link the collected logs and deposit it in an s3 bucket ...
Thanks for explation. I didn't find a way to filter logs entry base on severity ( info, warn, etc )
Can you provide more videos about logging with ELK stack plz
You are the true DevOps guru !! Like the way you explain. 🙌
The DevOps tool of the month series is awesome ❤️
Btw, I have a query here, if you can help to answer. Considering a large amount of data to be ingested and analyzed for metrics purpose, do you suggest Fluentbit over Loki (coming from Grafana Labs). To me Fluentbit is much appealing in terms of low footprint and ability to serve a variety of different landscape.
Great video as always! Thanks, Nana.
Can you please make a video where you give a quick demo of how fluent bit works, how to create config files, etc. :)
you are the best here , thank u for keep doing best tutorial ever , we love you ✌❤
Very clear... 😎
👍🏻 for animations
I would like to know more about the Kubernetes Descheduler.
hi Nana please help do a detailed course on MicroServices, your experience and teaching style would help us learning the best practices in detail.
How is Logstash in elk different from Fluentbit? Could you please explain
Thank you for great content!
what about fluent bit reliability? can it lose logs under any circumstances? Can I use it if data is extremely important?
very clear explanation, thanks!
Whats the difference between this and logstash
Superb explanation
Would love to see content on Grafana Loki
Thanks! How about doing a video on GitOps with k8s, ArgoCD, & Jenkins X.
Sounds interesting, thanks for your suggestion!
Thanks for making valuable video.
Could you cover opentelemetry and compare fluentd?
Very good application mam. can you please do add some real time working demo. thank you for your help
Can you make vector and Vector remap language?
Thanks for another great video! Do you have any plans to create video or udemy course in more details how to configure fluentd or fluent bit?
How much time do you spend on editing? Are you using after effects or any slideshow maker?
Please make course on Kafka Prometheus Grafana
Please Make video on 12 factor app brief intro with example
I need a Video to configure fluentbit in K8S which collects the application logs from Each pod and present it in grafana
I love your videos!!
You could talk about Bazel :)
Could you please create videos on open shift ?
very nice content and understandable 👍
Awesome Job! thanks!
What is the difference between Fluent Bit and Prometheus?
Thanks Nana, I can not tell you what I would do without your channel!
ps: I am in love with you, please, don't break my heart :)
It is very usefull for me
I have updated fluent-bit-splunk helmchart version from version 0.12.3 to 0.16.2 but container is not up , here I am getting error message like"backoff"", how to resolve this issue and can u please help me out this
Excellent.
What's the tool that you use for animation?
Why the plug for Splunk, but otherwise good content
next episode may be golang for devops?
Nana you're suck a legend, your videos are great!
Can you do a comparo between logstash and fluent bit
Or promtail from grafana labs
hey im curious, what tools do you use to create your videos?
Hi, we are using a simple editing tool Screenflow
New Relic maybe
🙏 and ❤
Can you help me to understand Adaptor pattern with an simple yaml file and how I can learn yaml
🙏🏼
i love u nana
Second as well
First comment
Lol, devops is like an ocean.
Merci !
Thanks for your support Sebastien! 🙏
First
congrats
Funny how this problem has been solved dozens, if not hundreds, of times and new "solutions" just keep coming.
Stop talkin in wired assent ..
Thanks nana for amazing fluentbit video. Please make a next video about detailed installation video of fluentbit.