I keep referencing this video again and again. This is a great beginner level tutorial to basics of getting logs into Graylog, separating them into streams, and searching through logs with Graylog.
Thank you, I spent lots of time configuring ELK from scratch, but the work greylog has done is awesome, its simple and does the job well thank you for showing this
I was hung up on how to identify and separate out logs for a project/application once I send the logs from FluentD to Graylog. Your explanation on streams/indices/rules helped cleared out that confusion. Thank you so much
I’ve played around a bit now and I’ve found you can really set a single “syslog” input for multiple servers. Then you create the index and streams. But when you create the stream “rule”, you can use the “gl2_remote_ip” field to only filter by certain syslog sources. So for pfSense, it would be the router. And for any unifi devices, it’s the IP of the device itself (AP, switch, etc). You can set the stream to be a so for device, or a group. This way you don’t have to have a separate input with a unique port number for EVERY remote server :)
I've been dragging my feet for about a year now on making a decision with respect to log aggregation from a handful of proxies I manage all over the world. I checked this video out and decided to give Graylog a try and I absolutely love it! The install is super, super easy and I had a Graylog instance running and ingesting data from several of my proxies within 2 hours. Now its just a matter of a tweaking queries and dashboards to let me see precisely what I need. Awesome video, as always, Tom - I for sure would have spun my wheels on the streams / indices / extractors /etc!
I've been using Graylog at many of my customers for a few years now. Excellent product. I've been able to setup some really informative dashboards and alerts. It works well after you make a few tweaks. One thing I found is to make sure to adjust the heap size to get good performance. Other than that, it works great. We are ingesting Windows logs, NAS storage logs, WIFI AP logs, Firewall and Switch logs and VMware logs. The difficult part is narrowing down the scope of the data to the things you really need, but once you have it you can build dashboards that provide concise information. I have been using the grok patterns to categorize data from firewalls and it helps to build more informative dashboards and allow greater flexibility in presenting the data. Excellent tutorial.
Thank You for showing this piece of software. I was working on setting up an ELK stack for just syslogs and is has been a few days utter failure and making me question my chosen profession and my proficiency at it. I have chosen to take a different route for logs because of the sheer admin cost. It's just two of use for 4500+ Customers and 100 Employee's.
Thanks for the great video! I have been wanting to get into graylog for a while, this video finally got me to get off my butt. Still trying to figure it all out, but this was a great start. I was able to very easily set up the free enterprise license since it seems highly unlikely I will be ingesting over 5gb/day in my homelab.
Ran a Graylog VM and couldn't figure out why it wouldn't ingest my ESXi, TrueNAS and NetScaler logs. I imagine it was the extractor, stream, index architecture that I didn't understand. Great job of addressing that upfront and not just going thru a procedural next, next, next configuration
Thanks for this video Tom! I was just starting to work on this. I’d love to see a video that is specifically about getting Suricata logs into Graylog if that’s something you’re interested in!
Hi, I'm trying to send the suricata logs on pfsense to graylog and then show into grafana. but not luck yet. Only can show filterlog logs but not from suricata. if someone have this done I appreciate some kind of help. Thanks a lot
Nice tutorial. Thank you for not spending the first 30 minutes explaining your life history, begging to be excused for not posting on RUclips, a tour of your house, with 10 minutes of please like and subscribe
I had to use RAW udp (in order to see pfsense logs) and not syslog for some readon for the logs to actually popup, it was a rough start.... I'm at pfsense 2.4.4-p3 and i can see you have other options in yours as BSDlogging in your fw, so updating it might be good/fix things for me ... :)
Very good explaining video. I have one question: I have multiple servers / raspberry pis where i want to get the syslogs, however with 100 raspis, i dont like to create 100 different inputs with different ports. Can different hosts ude the same port and can the graylog distinct between the indiviual sources? In the end, all the data of those devices can land in the same location, with some filter to separate out some specific messages to be saved in another location, however i would need to know where the logentry came from, when using the same port. Is this possible?
Graylog tweeted your "almost done my 2023 Graylog update, need some help with an issue" tweet. Docker seems to be the way these days. I saw that you were able to fix the issue you were having with the config file. Are you going to release a tutorial soon?
How do i send log from different subnet? I created a pfsense vip for graylog server. I can ssh and ping graylog server by this ip but not able to open web gui and send log.
You could compare for yourself... install Kibana and visualize the same data for comparison or install two instances (one Graylog and one ElasticStack) to evaluate the two. GrayLog is a bit more intuitive than setting up an ElasticStack instance and is a matter of preference. Here's a ElasticStack alternative to try and compare for yourself: github.com/pfelk/pfelk A video comparison would be great!
@Leeroy - Syslog was created in the 1980s and by default does not encrypt, transmitting everything in the clear. However, Syslog-NG is capable of transmitting over TLS via TCP. Additionally, Syslog-NG was developed to add additional security and filtering options. This same setup is feasible (replacement to syslog) and capable of leveraging TLS (e.g. encryption).
Question to Master : how can i secure graylog to only receive secured message ? i do not want every one can use my graylog server to send message , please HELP !!!!
I am completely newbie in logging analysis and this domain, can you please tell me if Graylog is SAAS based solution in any way? I mean when we ingest log data for analysis do we need to ingest it in their SAAS platform?
I don't know if UniFi would report switch MAC/port changes over syslog?. Try getting a Cisco switch to report MAC table 😅 to graylog, omg. I spent 2 last days trying to.. syslog, nope.. Then SNMP should be possible, but SNMP graylog plugin refuses to play nice, least with cisco MIBS... OK, i get it thru Telegraf then it has options for that. Switch sends snmp notifications to Telegraf, and Telegraf reports back to graylog (it has nice and simple output for that), but nope, at most i can see something changed, but not the MAC... so far that i can see. It's possible however to request the mac table over snmp, but needs some serious parsing to understand which port its on.. but/and then youd haft to do that every minute, instead of just getting a notification.
I subscribed to your channel recently and I am very glad now because of videos like this. Q: I have services sending emails. Can Graylog receive or check email? Also, can I set it up so that it alerts me if an email for a task that is scheduled was not received? Thank you!
@@LAWRENCESYSTEMS I have services and routers sending email messages when an event appears. I was wondering if Graylog can extract those messages via IMAP/POP3 and analyse them?
Great video.. but on a default Ubuntu 21 Graylog install I've found that using port 514 results in "permission denied" as the lower ports are restricted to all but root users (which the Graylog server apparently is not running as) My pfsense will not send logs on any other port than 514, despite what may be entered in the System Logs settings. Sio I've configured Graylog's input to port 1514 and set the server input like this: iptables -t nat -A PREROUTING -p tcp --dport 514 -j REDIRECT --to 1514 iptables -t nat -A PREROUTING -p udp --dport 514 -j REDIRECT --to 1514 Oh goody. Now I'm flooded with logs in Graylog :/
@@MiguelCruzer I live in the Netherlands and my timezone is thus gmt+1 , at the time the docker image only supported utc so I had to modify the dockerfile and re-compile it.
@@ItsQuintFX That sounds like a config issue, I'm not sure if this is new or not but docs say any config option can be passed as an ENV_VAR preceded with the GRAYLOG_ name. docs.graylog.org/en/4.0/pages/installation/docker.html#configuration Hope this helpsl
@@LAWRENCESYSTEMS thank you for your response. If you happen to change things up and start working with grafana. You can send graylog logs to grafana and have beautiful siem graphs.
The whole idea here is A.I for cyber security. With the graphs and the alerting system. Have scripts programmed in python or an other language to react to it. That's my project I'm working on.
Yes. Elastisearch is a big problem. One server is not enough even for relative small logs docs.graylog.org/en/4.0/pages/configuration/elasticsearch.html We strongly recommend to use a dedicated Elasticsearch cluster for your Graylog setup
have it running in proxmox as CT container, with 4gb ram and data on zfs mirror with two 6-10 year old HDD. syslog from various vm, physical linux machines, raspberries and opnsense (but without every firewallevent). use only default index set, after about one year there are 6.8GiB of data. query works just fine, nothing to complain. so I would say, give it a try!
Very neat product, I actually work with Splunk. This is super cool but doesn't have all the features that Splunk does. You should totally take a look at it, I know it is closed source but it is a damn good product.
![ROSÉ(로제) - APT. (Band VER.) (With. 이영지) [더 시즌즈-이영지의 레인보우] | KBS 241129 방송](http://i.ytimg.com/vi/IW8qaemx74g/mqdefault.jpg)
Related Forum Post
forums.lawrencesystems.com/t/open-source-logging-getting-started-with-graylog/8797
Hey, we are doing some RUclips clean up and just came across the video! What a great tutorial! Thanks for taking the time to make it :)
Thanks!
ok
I keep referencing this video again and again. This is a great beginner level tutorial to basics of getting logs into Graylog, separating them into streams, and searching through logs with Graylog.
Glad you enjoyed it!
Thank you,
I spent lots of time configuring ELK from scratch, but the work greylog has done is awesome, its simple and does the job well
thank you for showing this
I was hung up on how to identify and separate out logs for a project/application once I send the logs from FluentD to Graylog. Your explanation on streams/indices/rules helped cleared out that confusion. Thank you so much
Fantastic!
I’ve played around a bit now and I’ve found you can really set a single “syslog” input for multiple servers. Then you create the index and streams. But when you create the stream “rule”, you can use the “gl2_remote_ip” field to only filter by certain syslog sources. So for pfSense, it would be the router. And for any unifi devices, it’s the IP of the device itself (AP, switch, etc). You can set the stream to be a so for device, or a group. This way you don’t have to have a separate input with a unique port number for EVERY remote server :)
Thanks, I was just wondering how to mitigate this problem. Your explanation was perfect.
I've been dragging my feet for about a year now on making a decision with respect to log aggregation from a handful of proxies I manage all over the world. I checked this video out and decided to give Graylog a try and I absolutely love it! The install is super, super easy and I had a Graylog instance running and ingesting data from several of my proxies within 2 hours. Now its just a matter of a tweaking queries and dashboards to let me see precisely what I need. Awesome video, as always, Tom - I for sure would have spun my wheels on the streams / indices / extractors /etc!
could you share what queries you've used for your dashboards or any free resources available. Thanks
I've been using Graylog at many of my customers for a few years now. Excellent product. I've been able to setup some really informative dashboards and alerts. It works well after you make a few tweaks. One thing I found is to make sure to adjust the heap size to get good performance. Other than that, it works great. We are ingesting Windows logs, NAS storage logs, WIFI AP logs, Firewall and Switch logs and VMware logs. The difficult part is narrowing down the scope of the data to the things you really need, but once you have it you can build dashboards that provide concise information. I have been using the grok patterns to categorize data from firewalls and it helps to build more informative dashboards and allow greater flexibility in presenting the data. Excellent tutorial.
Thank You Tom. I am looking at implementing Graylog in my home network and your video content was very helpful!
Thank You for showing this piece of software. I was working on setting up an ELK stack for just syslogs and is has been a few days utter failure and making me question my chosen profession and my proficiency at it. I have chosen to take a different route for logs because of the sheer admin cost. It's just two of use for 4500+ Customers and 100 Employee's.
I’ve just been thinking about how there must be something like this out there. Thank you! I’ll play with this!!
I set this up in 2016, we had 3 customers all sending logs to centralized Graylog server; it was fun!
This is Great Tom. I have been looking for this video on this topic. Thanks.
We use this at my work. It’s dope.
So helpful! Great tuto! New sub in here.
Greets from Uruguay.
Thanks! This video helped me to get graylog to start seeing incoming data.
Been wanting to move away from Splunk for a while, thanks for hitting the high points!!
Thanks again for another really good breakdown using real world and human understandable examples.
Thanks for the great video! I have been wanting to get into graylog for a while, this video finally got me to get off my butt. Still trying to figure it all out, but this was a great start. I was able to very easily set up the free enterprise license since it seems highly unlikely I will be ingesting over 5gb/day in my homelab.
Ran a Graylog VM and couldn't figure out why it wouldn't ingest my ESXi, TrueNAS and NetScaler logs. I imagine it was the extractor, stream, index architecture that I didn't understand. Great job of addressing that upfront and not just going thru a procedural next, next, next configuration
Graylog FTW
Been using it since 2013.
Pair it with fluentd and you are done.
Add your manual :)
Ea! que tu haces por estos lares? jajaja Saludos mi pana
Thanks for this video Tom! I was just starting to work on this. I’d love to see a video that is specifically about getting Suricata logs into Graylog if that’s something you’re interested in!
I am also hoping to export Suricata events to Grafana for visualization if that's something you're interested in exploring.
Hi, I'm trying to send the suricata logs on pfsense to graylog and then show into grafana. but not luck yet. Only can show filterlog logs but not from suricata. if someone have this done I appreciate some kind of help. Thanks a lot
can't wait to use this !!
Nice tutorial. Thank you for not spending the first 30 minutes explaining your life history, begging to be excused for not posting on RUclips, a tour of your house, with 10 minutes of please like and subscribe
This is awesome!! Great video!
Hi beb
Good Timing considering the ElasticSearch license melt down
ElasticSearch licence meltdown? What happened?
It’s no longer open source and they blamed Amazon for it.
GrayLog utilizes ElasticSearch - what do you exactly mean?
I had to use RAW udp (in order to see pfsense logs) and not syslog for some readon for the logs to actually popup, it was a rough start.... I'm at pfsense 2.4.4-p3 and i can see you have other options in yours as BSDlogging in your fw, so updating it might be good/fix things for me ... :)
I see you also have Elastic Search running. would be interesting to hear what your pro's cons are vs using Greylog.
Graylog is much easier to setup and maintain.
Very good explaining video.
I have one question:
I have multiple servers / raspberry pis where i want to get the syslogs, however with 100 raspis, i dont like to create 100 different inputs with different ports. Can different hosts ude the same port and can the graylog distinct between the indiviual sources?
In the end, all the data of those devices can land in the same location, with some filter to separate out some specific messages to be saved in another location, however i would need to know where the logentry came from, when using the same port. Is this possible?
You could use one port and then parse the data by host name.
Graylog tweeted your "almost done my 2023 Graylog update, need some help with an issue" tweet. Docker seems to be the way these days. I saw that you were able to fix the issue you were having with the config file. Are you going to release a tutorial soon?
Yeah, hoping to get it done by early next week.
@@LAWRENCESYSTEMS awesome, thank you!
Have you ever covered SIEMs? I would love to get pointed to the best ones out there, OSS if possible. Thanks!
@S K We use Qradar. It's a great product only very expensive.
Check out wazuh
Thanks for the brilliant video. I'm planning to integrate unifi controller to graylog . do you have any idea where I get the extractors for inputs
github.com/lawrencesystems/graylog_extractors
How does this compare with splunk?
SYSLOG appears to be a no-brainer. But what about Windows server logs?
Yes docs.graylog.org/en/4.0/pages/sending/windows.html
thanks, Thanos, glad you're getting into soft instead of...well...
Nice video. I'll try this on my network and see what it can do.
very interesting! is there a log4j adapter? work?
will there be a AI Threat analysis followup video?
Hey Lawnrence any recommendations on user authentication and navigation logs?
Graylog
@@LAWRENCESYSTEMS thanks i'll try that
Awe-some, Tom!
How do i send log from different subnet? I created a pfsense vip for graylog server. I can ssh and ping graylog server by this ip but not able to open web gui and send log.
Thanks for this! I'm also interested in how this compare to ELK and even Splunk.
You could compare for yourself... install Kibana and visualize the same data for comparison or install two instances (one Graylog and one ElasticStack) to evaluate the two. GrayLog is a bit more intuitive than setting up an ElasticStack instance and is a matter of preference. Here's a ElasticStack alternative to try and compare for yourself: github.com/pfelk/pfelk
A video comparison would be great!
Great & Useful vid
So it just accepts entries on specific port or is there any auth?
How do I know that gathered data is legit?
Syslog does not use auth, but some of the other support input types do.
@@LAWRENCESYSTEMS Tell me that it is not plain text at least..
@@Mr.Leeroy You should probably read up a bit more on how syslog works.
@@LAWRENCESYSTEMS it has been on 2do list for far too long.
@Leeroy - Syslog was created in the 1980s and by default does not encrypt, transmitting everything in the clear. However, Syslog-NG is capable of transmitting over TLS via TCP. Additionally, Syslog-NG was developed to add additional security and filtering options. This same setup is feasible (replacement to syslog) and capable of leveraging TLS (e.g. encryption).
How is it you always post a video for a solution right when I'm looking for a solution?
Same here!
Interesting video, I decided to create my own laboratory! What about alerts? How do I make an alert for a specific log?
docs.graylog.org/en/4.1/pages/alerts.html#alerts
Hi, have you tried integrating syslog-ng to graylog ? Have tried it but the format was bad. If you have tried it, do you have any suggestions ?
I have not, but their forums can be helpful for parsing formats.
Great video.
Thank you Tom.
vAppliance no longer available/supported from Graylog: 4:00 "no time commitment loading a VM." ☹
Question to Master : how can i secure graylog to only receive secured message ? i do not want every one can use my graylog server to send message , please HELP !!!!
I am completely newbie in logging analysis and this domain, can you please tell me if Graylog is SAAS based solution in any way? I mean when we ingest log data for analysis do we need to ingest it in their SAAS platform?
As he already told us, it is not, because you can run it on your own server.
Unfortunately Graylog has removed the pre-built Virtual Machine Appliance downloads from the website.
Yup. but they do have docker images
I don't know if UniFi would report switch MAC/port changes over syslog?.
Try getting a Cisco switch to report MAC table 😅 to graylog, omg. I spent 2 last days trying to.. syslog, nope.. Then SNMP should be possible, but SNMP graylog plugin refuses to play nice, least with cisco MIBS... OK, i get it thru Telegraf then it has options for that. Switch sends snmp notifications to Telegraf, and Telegraf reports back to graylog (it has nice and simple output for that), but nope, at most i can see something changed, but not the MAC... so far that i can see.
It's possible however to request the mac table over snmp, but needs some serious parsing to understand which port its on.. but/and then youd haft to do that every minute, instead of just getting a notification.
I subscribed to your channel recently and I am very glad now because of videos like this.
Q: I have services sending emails. Can Graylog receive or check email?
Also, can I set it up so that it alerts me if an email for a task that is scheduled was not received?
Thank you!
It can send an email based on parameters that you define. www.graylog.org/features/alerting
@@LAWRENCESYSTEMS I have services and routers sending email messages when an event appears. I was wondering if Graylog can extract those messages via IMAP/POP3 and analyse them?
I am listening to someone speak about graylog and said the words star 410 or start up 410 . something 410. Do anyone know what that is?
¯\_(ツ)_/¯
Many-many-many-many thanks! Very good tutorial! Still curious, what is in "Memez" bookmark folder xD
My meme stash
Ever made a comparison between ELK-Stack and Greylog?
Nope, don't really plan to
Great video.. but on a default Ubuntu 21 Graylog install I've found that using port 514 results in "permission denied" as the lower ports are restricted to all but root users (which the Graylog server apparently is not running as)
My pfsense will not send logs on any other port than 514, despite what may be entered in the System Logs settings.
Sio I've configured Graylog's input to port 1514 and set the server input like this:
iptables -t nat -A PREROUTING -p tcp --dport 514 -j REDIRECT --to 1514
iptables -t nat -A PREROUTING -p udp --dport 514 -j REDIRECT --to 1514
Oh goody. Now I'm flooded with logs in Graylog :/
very cool.. Thank you!
Last time I used elastics, I cried every 3 months having to manually rotate the DB.
wonder how Graylog compares with solutions such as Kafka
Does this integrate with Active Directory?
Yes, via LDAP.
Looks like they've changed the offering around enterprise and free 5GB. Looks like it might only be 2GB now.
tried it, it's nice, but they need to resolve the timezone issue if that hasn't been solved already
Not an issue that I had so maybe it was resolved.
What timezone issue?
@@MiguelCruzer I live in the Netherlands and my timezone is thus gmt+1 , at the time the docker image only supported utc so I had to modify the dockerfile and re-compile it.
@@ItsQuintFX That sounds like a config issue, I'm not sure if this is new or not but docs say any config option can be passed as an ENV_VAR preceded with the GRAYLOG_ name. docs.graylog.org/en/4.0/pages/installation/docker.html#configuration
Hope this helpsl
Can you do an video on putting graylog logs into grafana. That would be much appreciated.
I don't use grafana so not likely.
@@LAWRENCESYSTEMS thank you for your response. If you happen to change things up and start working with grafana. You can send graylog logs to grafana and have beautiful siem graphs.
Then you can integrate zabbix with grafana and pretty much have a one stop shop.
The whole idea here is A.I for cyber security. With the graphs and the alerting system. Have scripts programmed in python or an other language to react to it. That's my project I'm working on.
What version on graylog are you using?
GRAYLOG 4.0
Good instroduction, even for GL v5 THANKS
This all use free or enterprise...?
because all view different with free mode
Make more vids bout graylog. Keep it up . We will help you out with the subs 👊🙏
Make a demo bout geo iplocation dashboard. Thsnks!
Can this be run on windows machine?
Don't think so
Does it need a lot of ram?
Yes. Elastisearch is a big problem. One server is not enough even for relative small logs
docs.graylog.org/en/4.0/pages/configuration/elasticsearch.html
We strongly recommend to use a dedicated Elasticsearch cluster for your Graylog setup
@@emanuelmilani7976 ah Thanks
have it running in proxmox as CT container, with 4gb ram and data on zfs mirror with two 6-10 year old HDD. syslog from various vm, physical linux machines, raspberries and opnsense (but without every firewallevent). use only default index set, after about one year there are 6.8GiB of data. query works just fine, nothing to complain. so I would say, give it a try!
@@sku2007 will do, thanks
not finding an ova for use
I might be able to help you with some regex stuff, so hit me up if you still have questions.
i don't know why, but you always trigger siri for me. i can't for the life of me trigger it myself though...
Very neat product, I actually work with Splunk. This is super cool but doesn't have all the features that Splunk does. You should totally take a look at it, I know it is closed source but it is a damn good product.
Expensive as frig
@@foobarturkey You are telling me lol. They have some trial license floating around that lets you do some stuff at home.
Now if it had SNMP too.
Tom, regex101.com is my go to for testing out new expressions.
Thanks I will check that out, I have been using regexr.com/
thanks
t-shirt made me laugh.
👍😁 since 2018
"1566 i like that number... " We know... ¬¬
"I'm code. I regex."
Seriously, if you [still] need help, let me know.
Recently configured myself in production. Works perfectly.
github.com/pfelk/pfelk
Suggest you check out wazuh :)
I've used it, I'm just not competent enough at it to do tutorials
31:24 ^,
Thanks
better, but is also more professional
too complex artich
This all use free or enterprise...?
because all view different with free mode