Automatic log file analysis using NoSQL?

[Amos Shapira]

Hello,

As part of PCI-DSS compliance I'm working on (ref:
http://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard),
we need to implement automatic log file analysis and alerting. (It's also a
Good Thing(TM) to have such a thing in place in general).

LogWatch is not enough since it can't handle the amount of logs generated by
our system (we generate ~6Gb of compressed HTTP daemon access log files
every 24 hours alone, not to mention many other log files and more to come
as we progress with PCI compliance) and still requires someone to manually
go through its reports.

Instead, I see many ads for commercial systems which can analyse log files
in near real time and generate custom alerts about suspicious activity
outside a learned activity pattern. These systems cost a fortune.

On the other hand - I saw mentions of open-source system which dump log
files onto a NoSQL database and achieve the same functionality with free
tools.

Alas - I lost the references for the later.

Closest thing I found is Flume (https://github.com/cloudera/flume). Someone
tells me that it also does the actual analysis but I don't see this
mentioned on its web site.

Does anyone else here have an idea about such systems?

Thanks,

--Amos
Does anyone
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20110216/98e57e25/attachment.html>