► Follow me on IG for behind the scenes content 😊👉🏼 bit.ly/2F3LXYJ ► This video was sponsored by the Cloud Native Computing Foundation www.cncf.io/ 🙌🏼 ► More infos on Fluentd: www.fluentd.org/ ▬▬▬▬▬▬ T I M E S T A M P S ⏰ ▬▬▬▬▬▬ 0:00 - Intro 0:26 - Why we actually need log data? 1:40 - Challenges of collecting and consuming logs 3:40 - Solution - a unified logging layer 4:17 - How does Fluentd work 6:29 - How to configure Fluentd? 9:03 - Built-In Reliability 10:09 - Other Use Cases
Have had fluentd daemonset set up in my kubernetes cluster, pushing pod, service, and other logs to elasticsearch for a month or so now. Really like the simplicity and flexibility it offers.
Very interesting. I heard about fluentd in another place and came here to find some info. Glad I found your video, easy to understand and digest. Thanks!
Nice, my attention was drawn to fluentd by your Udemy course and I was looking for more information about it a few days ago and now you made a video about it 👌
first of all, you are my favourite in my internet world, I learn a lot from you, thank you mam. sorry for my english. in my language which is my favourite आपको बहुत बहुत धन्यवाद।
I will actually make a follow-up video on Fluentbit as well. One of the common setups, especially in bigger clusters is, fluentbit pods as collectors and fluentd as an aggregator. Because fluentbit is more lightweight, so such setup would give you good performance while saving resources.
Great Video Nana! I use FluentD for quite some time already collecting logs and sending them to specific indexes outisde of the cluster so they can easially be observed using query's within observability platforms :)
Can fluentd collect logs using push-model like logstash? It is convenient to just add handler/appender to e.g. logback configuration and forget about log format.
That's a common question. Fluentd is more fit for k8s than Logstash, it integrates better and was created with k8s in mind. Plus it has some advantages, like routing the logs is much easier in Fluentd as well as Logstash needs additional service for data persistence. These are some of the main diffs.
Hey Nana, the content of your video is always very good and easily understandable. Except for your audio level, it seems low to me. I have tried many ways to increase your audio and try different devices as well but it didnt increase the sound volume. please make sure your videos are having little high audio quality so that some one can easily listen to it without earphones or headphones. I hope it wouldn`t appear offensive to you.
Yes Fluentbit is more lightweight. Needs much less resources. However Fluentd has more features, plugins. So depends on what you need to configure. If Fluentbit functionality is enough, then you can go with it. I will actually make a video about Fluentbit next month :)
Dear Nana, I like your videos they are very good with good content. But in this case it was not the quality I was expecting :) your videos are watched by IT pros you need to give more in dept knowledge instead of just high level overview. Please :) But overall I like your vides, keep up the good work :)
Thanks for this video. is there a way to limit nested parsing in json parser plugin @t ? as current it's parsing json in all levels to lots of field getting created. to wanted to limit josn parsing on first level.
Thanks for the video...Currently I am using Fluentd but it's can't go beyond 2k/EventPerSec/PerIntance. Where I am looking for 20k/EPS/PerInstance with each average event size of 800 Byte. Do you have any suggestion for the same?
Can you search/parse collect a specific string in fluentD like sumo logic. Can you set up dashboards on FluentD itself without pushing it to a destination system
Hello.. thanks for taking the time to put together this video. It’s excellent, it really does help with conceptualizing how to utilize fluentd.. I got it up and running and I am capturing logs from several of my back up processes and syslogs from few servers. Something I’ve not been able to figure out is how to dynamically assign labels to my Loki match blocks. I’ve used the rewrite_tag_filter plugin to push log lines to different match blocks where I have different labels but that feels like I’m hard coding the same thing over and over just to get one label to be different. I’d like to store the program or process name in a variable based on the log line and use it as a label. Is that possible? I’ve tried several different ways but I have not been successful doing that as yet. Any info would be greatly appreciated. Thanks again *edit when I say labels I’m referring to labels used in Loki and visible in Grafana
I would say that I don't consume logs, I don't eat them, I don't inhale or inject them into my bloodstream ;-) I normally just view them. I prefer consuming food.
Dear Nana, when i try to deploy fluentd using helm chart by default it deploys as root user. i changed to non root user by using security context. But the problem is it errors out "[error]: #0 unexpected error error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/fluentd-containers.log.pos". Can you please advise in case if you had this issue
I am having a bad experience for the last 3 days even though I have paid for the course. Please provide me a refund if you cannot help me open the second video of this course....
Hi Ash, can you please send an email to techworld-with-nana@nnsoftware.at and provide email with which you have purchased the course? So we can check. On RUclips comments there is no support Is this your email: goashtech@gmail.com ?
I don't think the explanation is good. it is so abstracted and I wonder if you really have configured it by your self. Is Fluentd flies along the applications like the animation you put on 4:26?
► Follow me on IG for behind the scenes content 😊👉🏼 bit.ly/2F3LXYJ
► This video was sponsored by the Cloud Native Computing Foundation www.cncf.io/ 🙌🏼
► More infos on Fluentd: www.fluentd.org/
▬▬▬▬▬▬ T I M E S T A M P S ⏰ ▬▬▬▬▬▬
0:00 - Intro
0:26 - Why we actually need log data?
1:40 - Challenges of collecting and consuming logs
3:40 - Solution - a unified logging layer
4:17 - How does Fluentd work
6:29 - How to configure Fluentd?
9:03 - Built-In Reliability
10:09 - Other Use Cases
Have had fluentd daemonset set up in my kubernetes cluster, pushing pod, service, and other logs to elasticsearch for a month or so now. Really like the simplicity and flexibility it offers.
Never tried fluentd before, but I definitively will, thanks to this video. I love this channel. Please keep 'em coming.
Glad to hear, thank you Farid :)
That's amazing 🔥
We need more tutorials on EFK stack 🙏
Very interesting. I heard about fluentd in another place and came here to find some info. Glad I found your video, easy to understand and digest. Thanks!
Nice intro. I was expecting more of a how to actually install and configure fluentd
Nice, my attention was drawn to fluentd by your Udemy course and I was looking for more information about it a few days ago and now you made a video about it 👌
😊👍
first of all, you are my favourite in my internet world, I learn a lot from you, thank you mam. sorry for my english. in my language which is my favourite आपको बहुत बहुत धन्यवाद।
Dear Nana. Very interesting Your opinion. Fluentd vs Fluent-bit? And which product is better to use in 2021? Thank you 😊
I will actually make a follow-up video on Fluentbit as well. One of the common setups, especially in bigger clusters is, fluentbit pods as collectors and fluentd as an aggregator. Because fluentbit is more lightweight, so such setup would give you good performance while saving resources.
@@TechWorldwithNana Thank you Nana!
Great Video Nana! I use FluentD for quite some time already collecting logs and sending them to specific indexes outisde of the cluster so they can easially be observed using query's within observability platforms :)
what about logstash !! thanks for the video .
Fluentd is actually more fit for K8s, it was actually created with k8s in mind.
@@TechWorldwithNana Fluentd was created when such things as Kubernetes does not exist at all
Thanks for your videos, I'm a pretty bad english listener and I understood all you said and I learned a lot with !!!!
If I want anyone collecting my data... It's Fluentd!
wow, super good quality video, thanks for the great effort!
Thank you. The contents are same as your Fluentbit lecture. Could you please give bit precise use case info when to use which one or how to use both.
Great. Nana. You always enlightened my knowledge. EFK becomes my target K8s logging framework. Thanks.
Another bang on video thank you for your help please add some slides to take notes.
I strugle with this a few days ago. I wish i had found this video earlier! thanks
Hi Anna, your video is always perfect and helpful. I wonder how you can make like this video, what are tools to do this
Nice explaination. Good job. Can you also do a video on OpenTelemetry?
This content is so good it should be paid
Hi @TechWorldwothNana when the the DevSecOps course going to be available
Can fluentd collect logs using push-model like logstash? It is convenient to just add handler/appender to e.g. logback configuration and forget about log format.
Nice - short but informative. Thank you.
Thanks Vasily :)
Could you share your views on how Fluentd is different then splunk for logging apart from the fact that splunk is commercial
The DevOps hero we didn't know we needed :)
It's worth watching and learning too. How it differs from Jaeger ? Please throw some light on that. Thanks in advance.
Dear Nana, could you please provide the video on design , how to get the health and pod level metrics for the applications deployed in openshift?
Filebit or logstash also does same thing then why we should go with fluentd..
And also plz explain fluentd config file in more details..
That's a common question. Fluentd is more fit for k8s than Logstash, it integrates better and was created with k8s in mind. Plus it has some advantages, like routing the logs is much easier in Fluentd as well as Logstash needs additional service for data persistence. These are some of the main diffs.
Spectacular explanation!!! 👏🏽👏🏽👏🏽👏🏽👏🏽
You are the best woman in the world , I love ur info ❤
I do really like the way you express and deliver your thoughts, much, much appreciated!
Please go on with your videos! :)
+
BTW - how can we support you, donating anyhow? Thanks!
very well broken down, the config snippets and animations help to visualize it a as well, thanks !
Thank you for the nice presentation. What tool would you recommend using when visualizing the log data?
I've used EFK stack for that so I think Kibana is really great for visualising logs!
Hey Nana, the content of your video is always very good and easily understandable.
Except for your audio level, it seems low to me. I have tried many ways to increase your audio and try different devices as well but it didnt increase the sound volume. please make sure your videos are having little high audio quality so that some one can easily listen to it without earphones or headphones.
I hope it wouldn`t appear offensive to you.
should I choose Fluentd over logstash/filebeat? @nana
Thanks for sharing..
Though AWS documents suggesting to use FluentBit instead of FluentD..
Yes Fluentbit is more lightweight. Needs much less resources. However Fluentd has more features, plugins. So depends on what you need to configure. If Fluentbit functionality is enough, then you can go with it. I will actually make a video about Fluentbit next month :)
@@TechWorldwithNana You care for us so much :)
Dear Nana, I like your videos they are very good with good content. But in this case it was not the quality I was expecting :) your videos are watched by IT pros you need to give more in dept knowledge instead of just high level overview. Please :)
But overall I like your vides, keep up the good work :)
Very well explained.
As usual the best!!
A great way of explanation indeed.
As usual Absolutely fantastic content thanks a lot for your wonderful explanation.
great explanation as always Nana, what is the logging solution we can integrate for Prometheus to collect application logs?
You don’t have patreon or smth like that? You are doing an amazing job helping the rest of the community and I would like to give back smth :)
Nice video, but I can't get the difference between this tool and a broker tool as for example CAMEL
Thank you ! very useful content
Thank you Mark!
great video 🎉
Great content (same comment again but this content is really good)
Happy to hear, thank you Daniel 😊
Thanks for your videos
Thank you so much! Is very interisting
Great content as always. Thank you!
Is it worth using fluentd together with Prometheus? Or is it more like an alternative?
Do you have any plans on creating videos about podman, buildah, skopeo?
I didn't plan it for now, but will definitely consider them! 👍
Thanks for this video. is there a way to limit nested parsing in json parser plugin @t ? as current it's parsing json in all levels to lots of field getting created. to wanted to limit josn parsing on first level.
What's the difference with ELK stack?
Need to know this too.
Can Fluentd collect pod/containers metrics ? if not, what do so in Docker & Kubernetes environments ?
Amazing video, thanks a lot🔥.Could you also please make one lesson on using ArgoCD for Kubernetes as the GitOps continuous delivery tool?
Can this be used for logs from iOS application?
Do you have lessons on linked in as well ? Couldn’t find ?
Can u please let me know how to push specific namespace logs only to elasticsearch from fluentd in k8s any specific parameters we need to use ?
Thanks for the video...Currently I am using Fluentd but it's can't go beyond 2k/EventPerSec/PerIntance. Where I am looking for 20k/EPS/PerInstance with each average event size of 800 Byte. Do you have any suggestion for the same?
Hi Nana, Can Fluentd replace #autodiscover feature of filebeat? Which helps to get the logs of #AUTOSCALING environment
Can you search/parse collect a specific string in fluentD like sumo logic. Can you set up dashboards on FluentD itself without pushing it to a destination system
I'm new to all this. Is nlog just another logger like fluentd?
Thank you !!!
It should be somehow marked that this is K8s related instruction. Im looking for plain docker instructions
Hey how does it compare to cribl ? I cant rely spot wheres the diffences right now.
very useful
Hello.. thanks for taking the time to put together this video. It’s excellent, it really does help with conceptualizing how to utilize fluentd.. I got it up and running and I am capturing logs from several of my back up processes and syslogs from few servers. Something I’ve not been able to figure out is how to dynamically assign labels to my Loki match blocks. I’ve used the rewrite_tag_filter plugin to push log lines to different match blocks where I have different labels but that feels like I’m hard coding the same thing over and over just to get one label to be different. I’d like to store the program or process name in a variable based on the log line and use it as a label. Is that possible? I’ve tried several different ways but I have not been successful doing that as yet. Any info would be greatly appreciated. Thanks again
*edit when I say labels I’m referring to labels used in Loki and visible in Grafana
which one is most fit for gcp cloud run logs collecting?
Nana , sound volume is a bit low , :)
Thanks for the feedback!
Thanks for the video. is it possible fluent-logger-java integrate in android? if possible please provide details?
Thank you - Crystal clear - ***** Stars
Thanks Fayaz 🙏
I would say that I don't consume logs, I don't eat them, I don't inhale or inject them into my bloodstream ;-) I normally just view them. I prefer consuming food.
Dear Nana, when i try to deploy fluentd using helm chart by default it deploys as root user. i changed to non root user by using security context. But the problem is it errors out "[error]: #0 unexpected error error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/fluentd-containers.log.pos". Can you please advise in case if you had this issue
why not looking at a more capable log management solution such as syslog-ng?
start at 4:22 :)
How to create Fluentd index with application name , Like Fluend-containername , deployed with helm
Subbed👍🏻
Hello.. how is it different from Splunk?
You could compare FluentD to Splunk forwarders. Preparsing, transforms etc. It's not a full product as Splunk, just a part of a stack.
thanx ❤❤
Suberb mam
still wonder why logstash is left behind and fluentd becomes more common
დიდი მადლობა მსგავსი კარგი ტუტორიალები არსად არაა)
Nana, I need your help,...!!!!
I am having a bad experience for the last 3 days even though I have paid for the course. Please provide me a refund if you cannot help me open the second video of this course....
Hi Ash, can you please send an email to techworld-with-nana@nnsoftware.at and provide email with which you have purchased the course? So we can check. On RUclips comments there is no support
Is this your email: goashtech@gmail.com ?
it would be much better if you explained how to install and manage it than listening of 10 minuts of bla bla bla fluentd is good fluentd is the best
First
why pointing the finger in thumbnail ? '-[
all logs to splunk, period !
many Splunk users use Fluentd to perform the data ingestion to Splunk, there is a good value having the option to control "which" data is ingested.
I don't think the explanation is good. it is so abstracted and I wonder if you really have configured it by your self. Is Fluentd flies along the applications like the animation you put on 4:26?