Consolidated Logging / Syslog Solution (27001)

[DHR]

I've come to the point where I need to begin looking at logging for ISO 27001 compliance.

Budgets are tight (aren't they always!) but I'm thinking the easiest way is to use some form of syslog server in our Windows domain based environment and regularly review the most frequent or critical events logged?

Has anyone done this recently? Ideally looking at open source, free solutions to make our compliance lives just that little bit easier?

 

[LizardKing]

Graylog, if your not huge its pretty straightforward out of the box with a little learning investment, a bit more complex once you have the need to spread out the components!

its linux but its web based and unless you go down the rabbit hole of its capability you will never see know otherwise.

Otherwise theres free splunk if your not expecting massive amounts of data

 

[Throrik]

I’m a big advocate of the ELK stack.

 

[DHR]

Not sure what put me of ELK, greylog looks like an ideal one to get up and running today. @LizardKing you collecting windows logs with it?

 

[Ev0]

ELK comes to mind as mentioned when looking at 'free' solutions, Graylog looks like it could be useful too.

Splunk and QRadar have free tiers if data volumes are low enough, not sure how good these are if you then need to expand into paid product land.

When looking at paid solutions Alienvault is usualy a cheap one, but if you just want centralised log storage might be overkill.

 

[BongoHunter]

ELK everytime for me, it's cross platform, widely used, the free version is great and there's a paid for version available at a reasonable price if needed, and it can deal with most common log formats straight out of the box.

 

[LizardKing]

Not sure what put me of ELK, greylog looks like an ideal one to get up and running today. @LizardKing you collecting windows logs with it?

Yeah we log all and everything with it and use it for various alerts (ie security group changes, firewall logins etc), then use Grafana to produce pointless fancy outputs that execs love.

It handles the 5-6million messages a day we throw at it with ease on a single vm. (firewalls can be very chatty!)

 

[Duke]

Another vote for Graylog, very good tool, we use it at work.

 

[Ev0]

It handles the 5-6million messages a day we throw at it with ease on a single vm. (firewalls can be very chatty!)

At those event rates I’d imagine/hope most systems could handle that on a single vm.

How much resources you need to give the vm though, that’s another story

 

[SupraWez]

I've used Nagios Log Server a few times and does the trick.

 

[LizardKing]

At those event rates I’d imagine/hope most systems could handle that on a single vm.

How much resources you need to give the vm though, that’s another story

lol I know what your getting at its all about scaling out these days. however for ref I cant remember the specs but its not a lot, something along the lines of 4gb ram, 2 ancient xeon cores and raid10 local storage.

Works out at about 200-300mb an hour i think so no not huge amounts of data. The elastic search works pretty well considering the specs!

Id have gone for splunk given its the industry standard, but the beancounters wouldn't have gone for it as its primarily a box ticking project.

 

[Ev0]

Yeah it’s not too bad spec wise, pretty light really which is great when you want something at low/no cost that’ll also run on relatively inexpensive hardware.

I work in this area so know more about the commercial systems of this ilk than the ‘free’ stuff, but it is impressive what can be done on this side of it all.

Got a few clients who do some cool stuff taking data out of their systems and using Kibana to create some great visualisations.

 

[DHR]

Go my teeth stuck into graylog, bit of a learning curve, windows logs were a nightmare to get right but i'm there now, well with the logging, need to come up with some fancy dashboards.

Currently starting to pipe in ASA logs which is a joy

 

[droyden]

Have you gone for a single server deployment?
I am also trying this at the moment, currently single server but will look at splitting it all out if we go for it

 

[DHR]

Have you gone for a single server deployment?
I am also trying this at the moment, currently single server but will look at splitting it all out if we go for it

Yes single server, I'm hoping it'll be enough for our environment, I'm selectively logging only critical systems at the moment. Firewall logs are going to be a challenge.

One thing I've not gotten my head around yet is retention and disk space management, anyone have any input on that when you're not using the enterprise version?

 

[LizardKing]

the rotations depends on your requirements. Ie do you need them for 30 days or are you space limited, or do you need to archive them off.

though to keep it simple its probably easier to just set the index size to 2gb, rotation strategy to delete, then set the max number so it works out to ~10gb below your HD space. then you can figure out your burn rate. Running out of space is not an easy fix

so a 50gb hard drive set the max number to ~20.

You can be clever and pipe different log sources to different indices sets and have separate retention policy's on those.

Theres some good dashboard templates in the market place for AD stuff that will get you going pretty quickly!

 

[DHR]

Have I made a mistake not going for version 3? I've been working with 2.5 so far?

Having a nightmare with it trying to bring in Jira security logs too

 

[droyden]

Don't think v3 is out yet, just rc and beta?

 

[DHR]

It was RC until today I think, I'm fuming!! Got notified of it by 2.5

Being handed one over a grok pattern and extractor too.

I’m using the following pattern which works on the following log entry...

2018-03-19 15:11:46,181 https-jsse-nio-443-exec-246 JiraServiceAccount 911x8252778x1 - 192.168.0.1 /rest/api/2/search The user ‘ServiceAccount’ has PASSED authentication.

%{TIMESTAMP_ISO8601} %{DATA:UNWANTED} %{USERNAME:UNWANTED} %{DATA:UNWANTED} %{DATA:UNWANTED} %{IPV4} %{PATH}

I need to extract the username between quotes from “The user ‘ServiceAccount’ has PASSED authentication.” To do this i’ve set a grok pattern up called JIRASECUSER which is the following regex

(?<=The user ‘).*?(?=’) DeleteEdit

Again this works in isolation as does the %{JIRASECUSER} on it’s own in the extractor.

As soon as I try to use the following the filter refuses to run:

%{TIMESTAMP_ISO8601} %{DATA:UNWANTED} %{USERNAME:UNWANTED} %{DATA:UNWANTED} %{DATA:UNWANTED} %{IPV4} %{PATH} %{JIRASECUSER}

Totally clueless as to where to go from there now. Believe any errors are logged somewhere but I can't find the server.log anywhere on the OVA appliance

 

[Ev0]

Does Graylog come with any ready made parsers, or ready made connectors to get the data in from various sources?