Automatic log file analysis using NoSQL?

[Elazar Leibovich]

I never used it, but I saw ads about splunk for log management.

http://www.splunk.com/

2011/2/16 Amos Shapira <amos.shapira at gmail.com>

> Hello,
>
> As part of PCI-DSS compliance I'm working on (ref:
> http://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard),
> we need to implement automatic log file analysis and alerting. (It's also a
> Good Thing(TM) to have such a thing in place in general).
>
> LogWatch is not enough since it can't handle the amount of logs generated
> by our system (we generate ~6Gb of compressed HTTP daemon access log files
> every 24 hours alone, not to mention many other log files and more to come
> as we progress with PCI compliance) and still requires someone to manually
> go through its reports.
>
> Instead, I see many ads for commercial systems which can analyse log files
> in near real time and generate custom alerts about suspicious activity
> outside a learned activity pattern. These systems cost a fortune.
>
> On the other hand - I saw mentions of open-source system which dump log
> files onto a NoSQL database and achieve the same functionality with free
> tools.
>
> Alas - I lost the references for the later.
>
> Closest thing I found is Flume (https://github.com/cloudera/flume).
> Someone tells me that it also does the actual analysis but I don't see this
> mentioned on its web site.
>
> Does anyone else here have an idea about such systems?
>
> Thanks,
>
> --Amos
> Does anyone
>
> _______________________________________________
> Linux-il mailing list
> Linux-il at cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20110216/549152aa/attachment.html>