Thank you, I spent lots of time configuring ELK from scratch, but the work greylog has done is awesome, its simple and does the job well thank you for showing this
I keep referencing this video again and again. This is a great beginner level tutorial to basics of getting logs into Graylog, separating them into streams, and searching through logs with Graylog.
I was hung up on how to identify and separate out logs for a project/application once I send the logs from FluentD to Graylog. Your explanation on streams/indices/rules helped cleared out that confusion. Thank you so much
I've been dragging my feet for about a year now on making a decision with respect to log aggregation from a handful of proxies I manage all over the world. I checked this video out and decided to give Graylog a try and I absolutely love it! The install is super, super easy and I had a Graylog instance running and ingesting data from several of my proxies within 2 hours. Now its just a matter of a tweaking queries and dashboards to let me see precisely what I need. Awesome video, as always, Tom - I for sure would have spun my wheels on the streams / indices / extractors /etc!
I've been using Graylog at many of my customers for a few years now. Excellent product. I've been able to setup some really informative dashboards and alerts. It works well after you make a few tweaks. One thing I found is to make sure to adjust the heap size to get good performance. Other than that, it works great. We are ingesting Windows logs, NAS storage logs, WIFI AP logs, Firewall and Switch logs and VMware logs. The difficult part is narrowing down the scope of the data to the things you really need, but once you have it you can build dashboards that provide concise information. I have been using the grok patterns to categorize data from firewalls and it helps to build more informative dashboards and allow greater flexibility in presenting the data. Excellent tutorial.
I’ve played around a bit now and I’ve found you can really set a single “syslog” input for multiple servers. Then you create the index and streams. But when you create the stream “rule”, you can use the “gl2_remote_ip” field to only filter by certain syslog sources. So for pfSense, it would be the router. And for any unifi devices, it’s the IP of the device itself (AP, switch, etc). You can set the stream to be a so for device, or a group. This way you don’t have to have a separate input with a unique port number for EVERY remote server :)
Thank You for showing this piece of software. I was working on setting up an ELK stack for just syslogs and is has been a few days utter failure and making me question my chosen profession and my proficiency at it. I have chosen to take a different route for logs because of the sheer admin cost. It's just two of use for 4500+ Customers and 100 Employee's.
Thanks for the great video! I have been wanting to get into graylog for a while, this video finally got me to get off my butt. Still trying to figure it all out, but this was a great start. I was able to very easily set up the free enterprise license since it seems highly unlikely I will be ingesting over 5gb/day in my homelab.
Ran a Graylog VM and couldn't figure out why it wouldn't ingest my ESXi, TrueNAS and NetScaler logs. I imagine it was the extractor, stream, index architecture that I didn't understand. Great job of addressing that upfront and not just going thru a procedural next, next, next configuration
Thanks for this video Tom! I was just starting to work on this. I’d love to see a video that is specifically about getting Suricata logs into Graylog if that’s something you’re interested in!
Hi, I'm trying to send the suricata logs on pfsense to graylog and then show into grafana. but not luck yet. Only can show filterlog logs but not from suricata. if someone have this done I appreciate some kind of help. Thanks a lot
Very good explaining video. I have one question: I have multiple servers / raspberry pis where i want to get the syslogs, however with 100 raspis, i dont like to create 100 different inputs with different ports. Can different hosts ude the same port and can the graylog distinct between the indiviual sources? In the end, all the data of those devices can land in the same location, with some filter to separate out some specific messages to be saved in another location, however i would need to know where the logentry came from, when using the same port. Is this possible?
I had to use RAW udp (in order to see pfsense logs) and not syslog for some readon for the logs to actually popup, it was a rough start.... I'm at pfsense 2.4.4-p3 and i can see you have other options in yours as BSDlogging in your fw, so updating it might be good/fix things for me ... :)
Question to Master : how can i secure graylog to only receive secured message ? i do not want every one can use my graylog server to send message , please HELP !!!!
How do i send log from different subnet? I created a pfsense vip for graylog server. I can ssh and ping graylog server by this ip but not able to open web gui and send log.
Nice tutorial. Thank you for not spending the first 30 minutes explaining your life history, begging to be excused for not posting on RUclips, a tour of your house, with 10 minutes of please like and subscribe
I don't know if UniFi would report switch MAC/port changes over syslog?. Try getting a Cisco switch to report MAC table 😅 to graylog, omg. I spent 2 last days trying to.. syslog, nope.. Then SNMP should be possible, but SNMP graylog plugin refuses to play nice, least with cisco MIBS... OK, i get it thru Telegraf then it has options for that. Switch sends snmp notifications to Telegraf, and Telegraf reports back to graylog (it has nice and simple output for that), but nope, at most i can see something changed, but not the MAC... so far that i can see. It's possible however to request the mac table over snmp, but needs some serious parsing to understand which port its on.. but/and then youd haft to do that every minute, instead of just getting a notification.
Graylog tweeted your "almost done my 2023 Graylog update, need some help with an issue" tweet. Docker seems to be the way these days. I saw that you were able to fix the issue you were having with the config file. Are you going to release a tutorial soon?
@Leeroy - Syslog was created in the 1980s and by default does not encrypt, transmitting everything in the clear. However, Syslog-NG is capable of transmitting over TLS via TCP. Additionally, Syslog-NG was developed to add additional security and filtering options. This same setup is feasible (replacement to syslog) and capable of leveraging TLS (e.g. encryption).
Great video.. but on a default Ubuntu 21 Graylog install I've found that using port 514 results in "permission denied" as the lower ports are restricted to all but root users (which the Graylog server apparently is not running as) My pfsense will not send logs on any other port than 514, despite what may be entered in the System Logs settings. Sio I've configured Graylog's input to port 1514 and set the server input like this: iptables -t nat -A PREROUTING -p tcp --dport 514 -j REDIRECT --to 1514 iptables -t nat -A PREROUTING -p udp --dport 514 -j REDIRECT --to 1514 Oh goody. Now I'm flooded with logs in Graylog :/
I am completely newbie in logging analysis and this domain, can you please tell me if Graylog is SAAS based solution in any way? I mean when we ingest log data for analysis do we need to ingest it in their SAAS platform?
I subscribed to your channel recently and I am very glad now because of videos like this. Q: I have services sending emails. Can Graylog receive or check email? Also, can I set it up so that it alerts me if an email for a task that is scheduled was not received? Thank you!
@@LAWRENCESYSTEMS I have services and routers sending email messages when an event appears. I was wondering if Graylog can extract those messages via IMAP/POP3 and analyse them?
You could compare for yourself... install Kibana and visualize the same data for comparison or install two instances (one Graylog and one ElasticStack) to evaluate the two. GrayLog is a bit more intuitive than setting up an ElasticStack instance and is a matter of preference. Here's a ElasticStack alternative to try and compare for yourself: github.com/pfelk/pfelk A video comparison would be great!
Yes. Elastisearch is a big problem. One server is not enough even for relative small logs docs.graylog.org/en/4.0/pages/configuration/elasticsearch.html We strongly recommend to use a dedicated Elasticsearch cluster for your Graylog setup
have it running in proxmox as CT container, with 4gb ram and data on zfs mirror with two 6-10 year old HDD. syslog from various vm, physical linux machines, raspberries and opnsense (but without every firewallevent). use only default index set, after about one year there are 6.8GiB of data. query works just fine, nothing to complain. so I would say, give it a try!
Very neat product, I actually work with Splunk. This is super cool but doesn't have all the features that Splunk does. You should totally take a look at it, I know it is closed source but it is a damn good product.
@@MiguelCruzer I live in the Netherlands and my timezone is thus gmt+1 , at the time the docker image only supported utc so I had to modify the dockerfile and re-compile it.
@@ItsQuintFX That sounds like a config issue, I'm not sure if this is new or not but docs say any config option can be passed as an ENV_VAR preceded with the GRAYLOG_ name. docs.graylog.org/en/4.0/pages/installation/docker.html#configuration Hope this helpsl
@@LAWRENCESYSTEMS thank you for your response. If you happen to change things up and start working with grafana. You can send graylog logs to grafana and have beautiful siem graphs.
The whole idea here is A.I for cyber security. With the graphs and the alerting system. Have scripts programmed in python or an other language to react to it. That's my project I'm working on.
i don't know what it is about your voice, but it is attracting my cat and she is trying to smash her face under my laptop, like trying to burrow under it. she is obsessed with my laptop. she has never done this before. nor when i pause.
Related Forum Post
forums.lawrencesystems.com/t/open-source-logging-getting-started-with-graylog/8797
Hey, we are doing some RUclips clean up and just came across the video! What a great tutorial! Thanks for taking the time to make it :)
Thanks!
ok
Thank you,
I spent lots of time configuring ELK from scratch, but the work greylog has done is awesome, its simple and does the job well
thank you for showing this
I keep referencing this video again and again. This is a great beginner level tutorial to basics of getting logs into Graylog, separating them into streams, and searching through logs with Graylog.
Glad you enjoyed it!
I was hung up on how to identify and separate out logs for a project/application once I send the logs from FluentD to Graylog. Your explanation on streams/indices/rules helped cleared out that confusion. Thank you so much
Fantastic!
I've been dragging my feet for about a year now on making a decision with respect to log aggregation from a handful of proxies I manage all over the world. I checked this video out and decided to give Graylog a try and I absolutely love it! The install is super, super easy and I had a Graylog instance running and ingesting data from several of my proxies within 2 hours. Now its just a matter of a tweaking queries and dashboards to let me see precisely what I need. Awesome video, as always, Tom - I for sure would have spun my wheels on the streams / indices / extractors /etc!
could you share what queries you've used for your dashboards or any free resources available. Thanks
I've been using Graylog at many of my customers for a few years now. Excellent product. I've been able to setup some really informative dashboards and alerts. It works well after you make a few tweaks. One thing I found is to make sure to adjust the heap size to get good performance. Other than that, it works great. We are ingesting Windows logs, NAS storage logs, WIFI AP logs, Firewall and Switch logs and VMware logs. The difficult part is narrowing down the scope of the data to the things you really need, but once you have it you can build dashboards that provide concise information. I have been using the grok patterns to categorize data from firewalls and it helps to build more informative dashboards and allow greater flexibility in presenting the data. Excellent tutorial.
I’ve played around a bit now and I’ve found you can really set a single “syslog” input for multiple servers. Then you create the index and streams. But when you create the stream “rule”, you can use the “gl2_remote_ip” field to only filter by certain syslog sources. So for pfSense, it would be the router. And for any unifi devices, it’s the IP of the device itself (AP, switch, etc). You can set the stream to be a so for device, or a group. This way you don’t have to have a separate input with a unique port number for EVERY remote server :)
Thanks, I was just wondering how to mitigate this problem. Your explanation was perfect.
Thank You for showing this piece of software. I was working on setting up an ELK stack for just syslogs and is has been a few days utter failure and making me question my chosen profession and my proficiency at it. I have chosen to take a different route for logs because of the sheer admin cost. It's just two of use for 4500+ Customers and 100 Employee's.
Thank You Tom. I am looking at implementing Graylog in my home network and your video content was very helpful!
I’ve just been thinking about how there must be something like this out there. Thank you! I’ll play with this!!
I set this up in 2016, we had 3 customers all sending logs to centralized Graylog server; it was fun!
Been wanting to move away from Splunk for a while, thanks for hitting the high points!!
Thanks for the great video! I have been wanting to get into graylog for a while, this video finally got me to get off my butt. Still trying to figure it all out, but this was a great start. I was able to very easily set up the free enterprise license since it seems highly unlikely I will be ingesting over 5gb/day in my homelab.
Ran a Graylog VM and couldn't figure out why it wouldn't ingest my ESXi, TrueNAS and NetScaler logs. I imagine it was the extractor, stream, index architecture that I didn't understand. Great job of addressing that upfront and not just going thru a procedural next, next, next configuration
This is Great Tom. I have been looking for this video on this topic. Thanks.
So helpful! Great tuto! New sub in here.
Greets from Uruguay.
We use this at my work. It’s dope.
Thanks! This video helped me to get graylog to start seeing incoming data.
Thanks for this video Tom! I was just starting to work on this. I’d love to see a video that is specifically about getting Suricata logs into Graylog if that’s something you’re interested in!
I am also hoping to export Suricata events to Grafana for visualization if that's something you're interested in exploring.
Hi, I'm trying to send the suricata logs on pfsense to graylog and then show into grafana. but not luck yet. Only can show filterlog logs but not from suricata. if someone have this done I appreciate some kind of help. Thanks a lot
Thanks again for another really good breakdown using real world and human understandable examples.
vAppliance no longer available/supported from Graylog: 4:00 "no time commitment loading a VM." ☹
Very good explaining video.
I have one question:
I have multiple servers / raspberry pis where i want to get the syslogs, however with 100 raspis, i dont like to create 100 different inputs with different ports. Can different hosts ude the same port and can the graylog distinct between the indiviual sources?
In the end, all the data of those devices can land in the same location, with some filter to separate out some specific messages to be saved in another location, however i would need to know where the logentry came from, when using the same port. Is this possible?
You could use one port and then parse the data by host name.
This is awesome!! Great video!
Hi beb
can't wait to use this !!
I had to use RAW udp (in order to see pfsense logs) and not syslog for some readon for the logs to actually popup, it was a rough start.... I'm at pfsense 2.4.4-p3 and i can see you have other options in yours as BSDlogging in your fw, so updating it might be good/fix things for me ... :)
How does this compare with splunk?
Question to Master : how can i secure graylog to only receive secured message ? i do not want every one can use my graylog server to send message , please HELP !!!!
How do i send log from different subnet? I created a pfsense vip for graylog server. I can ssh and ping graylog server by this ip but not able to open web gui and send log.
Thanks for the brilliant video. I'm planning to integrate unifi controller to graylog . do you have any idea where I get the extractors for inputs
github.com/lawrencesystems/graylog_extractors
Have you ever covered SIEMs? I would love to get pointed to the best ones out there, OSS if possible. Thanks!
@S K We use Qradar. It's a great product only very expensive.
Check out wazuh
Nice tutorial. Thank you for not spending the first 30 minutes explaining your life history, begging to be excused for not posting on RUclips, a tour of your house, with 10 minutes of please like and subscribe
I see you also have Elastic Search running. would be interesting to hear what your pro's cons are vs using Greylog.
Graylog is much easier to setup and maintain.
Graylog FTW
Been using it since 2013.
Pair it with fluentd and you are done.
Add your manual :)
Ea! que tu haces por estos lares? jajaja Saludos mi pana
Good Timing considering the ElasticSearch license melt down
ElasticSearch licence meltdown? What happened?
It’s no longer open source and they blamed Amazon for it.
GrayLog utilizes ElasticSearch - what do you exactly mean?
Nice video. I'll try this on my network and see what it can do.
very interesting! is there a log4j adapter? work?
will there be a AI Threat analysis followup video?
Hey Lawnrence any recommendations on user authentication and navigation logs?
Graylog
@@LAWRENCESYSTEMS thanks i'll try that
SYSLOG appears to be a no-brainer. But what about Windows server logs?
Yes docs.graylog.org/en/4.0/pages/sending/windows.html
I don't know if UniFi would report switch MAC/port changes over syslog?.
Try getting a Cisco switch to report MAC table 😅 to graylog, omg. I spent 2 last days trying to.. syslog, nope.. Then SNMP should be possible, but SNMP graylog plugin refuses to play nice, least with cisco MIBS... OK, i get it thru Telegraf then it has options for that. Switch sends snmp notifications to Telegraf, and Telegraf reports back to graylog (it has nice and simple output for that), but nope, at most i can see something changed, but not the MAC... so far that i can see.
It's possible however to request the mac table over snmp, but needs some serious parsing to understand which port its on.. but/and then youd haft to do that every minute, instead of just getting a notification.
I am listening to someone speak about graylog and said the words star 410 or start up 410 . something 410. Do anyone know what that is?
¯\_(ツ)_/¯
Graylog tweeted your "almost done my 2023 Graylog update, need some help with an issue" tweet. Docker seems to be the way these days. I saw that you were able to fix the issue you were having with the config file. Are you going to release a tutorial soon?
Yeah, hoping to get it done by early next week.
@@LAWRENCESYSTEMS awesome, thank you!
So it just accepts entries on specific port or is there any auth?
How do I know that gathered data is legit?
Syslog does not use auth, but some of the other support input types do.
@@LAWRENCESYSTEMS Tell me that it is not plain text at least..
@@Mr.Leeroy You should probably read up a bit more on how syslog works.
@@LAWRENCESYSTEMS it has been on 2do list for far too long.
@Leeroy - Syslog was created in the 1980s and by default does not encrypt, transmitting everything in the clear. However, Syslog-NG is capable of transmitting over TLS via TCP. Additionally, Syslog-NG was developed to add additional security and filtering options. This same setup is feasible (replacement to syslog) and capable of leveraging TLS (e.g. encryption).
Great video.. but on a default Ubuntu 21 Graylog install I've found that using port 514 results in "permission denied" as the lower ports are restricted to all but root users (which the Graylog server apparently is not running as)
My pfsense will not send logs on any other port than 514, despite what may be entered in the System Logs settings.
Sio I've configured Graylog's input to port 1514 and set the server input like this:
iptables -t nat -A PREROUTING -p tcp --dport 514 -j REDIRECT --to 1514
iptables -t nat -A PREROUTING -p udp --dport 514 -j REDIRECT --to 1514
Oh goody. Now I'm flooded with logs in Graylog :/
I am completely newbie in logging analysis and this domain, can you please tell me if Graylog is SAAS based solution in any way? I mean when we ingest log data for analysis do we need to ingest it in their SAAS platform?
As he already told us, it is not, because you can run it on your own server.
Great & Useful vid
wonder how Graylog compares with solutions such as Kafka
Many-many-many-many thanks! Very good tutorial! Still curious, what is in "Memez" bookmark folder xD
My meme stash
Hi, have you tried integrating syslog-ng to graylog ? Have tried it but the format was bad. If you have tried it, do you have any suggestions ?
I have not, but their forums can be helpful for parsing formats.
Great video.
I subscribed to your channel recently and I am very glad now because of videos like this.
Q: I have services sending emails. Can Graylog receive or check email?
Also, can I set it up so that it alerts me if an email for a task that is scheduled was not received?
Thank you!
It can send an email based on parameters that you define. www.graylog.org/features/alerting
@@LAWRENCESYSTEMS I have services and routers sending email messages when an event appears. I was wondering if Graylog can extract those messages via IMAP/POP3 and analyse them?
Thanks for this! I'm also interested in how this compare to ELK and even Splunk.
You could compare for yourself... install Kibana and visualize the same data for comparison or install two instances (one Graylog and one ElasticStack) to evaluate the two. GrayLog is a bit more intuitive than setting up an ElasticStack instance and is a matter of preference. Here's a ElasticStack alternative to try and compare for yourself: github.com/pfelk/pfelk
A video comparison would be great!
Ever made a comparison between ELK-Stack and Greylog?
Nope, don't really plan to
Does this integrate with Active Directory?
Yes, via LDAP.
Thank you Tom.
Awe-some, Tom!
Interesting video, I decided to create my own laboratory! What about alerts? How do I make an alert for a specific log?
docs.graylog.org/en/4.1/pages/alerts.html#alerts
very cool.. Thank you!
How is it you always post a video for a solution right when I'm looking for a solution?
Same here!
Good instroduction, even for GL v5 THANKS
thanks, Thanos, glad you're getting into soft instead of...well...
Unfortunately Graylog has removed the pre-built Virtual Machine Appliance downloads from the website.
Yup. but they do have docker images
What version on graylog are you using?
GRAYLOG 4.0
Make more vids bout graylog. Keep it up . We will help you out with the subs 👊🙏
Can this be run on windows machine?
Don't think so
Does it need a lot of ram?
Yes. Elastisearch is a big problem. One server is not enough even for relative small logs
docs.graylog.org/en/4.0/pages/configuration/elasticsearch.html
We strongly recommend to use a dedicated Elasticsearch cluster for your Graylog setup
@@emanuelmilani7976 ah Thanks
have it running in proxmox as CT container, with 4gb ram and data on zfs mirror with two 6-10 year old HDD. syslog from various vm, physical linux machines, raspberries and opnsense (but without every firewallevent). use only default index set, after about one year there are 6.8GiB of data. query works just fine, nothing to complain. so I would say, give it a try!
@@sku2007 will do, thanks
not finding an ova for use
Very neat product, I actually work with Splunk. This is super cool but doesn't have all the features that Splunk does. You should totally take a look at it, I know it is closed source but it is a damn good product.
Expensive as frig
@@foobarturkey You are telling me lol. They have some trial license floating around that lets you do some stuff at home.
Last time I used elastics, I cried every 3 months having to manually rotate the DB.
tried it, it's nice, but they need to resolve the timezone issue if that hasn't been solved already
Not an issue that I had so maybe it was resolved.
What timezone issue?
@@MiguelCruzer I live in the Netherlands and my timezone is thus gmt+1 , at the time the docker image only supported utc so I had to modify the dockerfile and re-compile it.
@@ItsQuintFX That sounds like a config issue, I'm not sure if this is new or not but docs say any config option can be passed as an ENV_VAR preceded with the GRAYLOG_ name. docs.graylog.org/en/4.0/pages/installation/docker.html#configuration
Hope this helpsl
This all use free or enterprise...?
because all view different with free mode
Looks like they've changed the offering around enterprise and free 5GB. Looks like it might only be 2GB now.
Can you do an video on putting graylog logs into grafana. That would be much appreciated.
I don't use grafana so not likely.
@@LAWRENCESYSTEMS thank you for your response. If you happen to change things up and start working with grafana. You can send graylog logs to grafana and have beautiful siem graphs.
Then you can integrate zabbix with grafana and pretty much have a one stop shop.
The whole idea here is A.I for cyber security. With the graphs and the alerting system. Have scripts programmed in python or an other language to react to it. That's my project I'm working on.
I might be able to help you with some regex stuff, so hit me up if you still have questions.
Tom, regex101.com is my go to for testing out new expressions.
Thanks I will check that out, I have been using regexr.com/
Make a demo bout geo iplocation dashboard. Thsnks!
thanks
👍😁 since 2018
31:24 ^,
Thanks
i don't know why, but you always trigger siri for me. i can't for the life of me trigger it myself though...
Now if it had SNMP too.
i don't know what it is about your voice, but it is attracting my cat and she is trying to smash her face under my laptop, like trying to burrow under it. she is obsessed with my laptop. she has never done this before. nor when i pause.
🐈
t-shirt made me laugh.
Recently configured myself in production. Works perfectly.
github.com/pfelk/pfelk
Suggest you check out wazuh :)
I've used it, I'm just not competent enough at it to do tutorials
"1566 i like that number... " We know... ¬¬
"I'm code. I regex."
Seriously, if you [still] need help, let me know.
better, but is also more professional
too complex artich
This all use free or enterprise...?
because all view different with free mode