Automatic log file analysis using NoSQL?

[Amos Shapira]

On 16 February 2011 21:52, Elazar Leibovich <elazarl at gmail.com> wrote:

> I never used it, but I saw ads about splunk for log management.
>
> http://www.splunk.com/
>

Thanks.

I'm aware of Splunk, our Customer Engineering people use it.

However:
1. It's bloody expensive (they license by size of uncompressed data fed).
2. It doesn't seem to scale (it uses a single instance of an RDBMS database,
can be very slow even before we start feeding the interesting log files into
it. We currently limit its use to feeding of very specific CSV-formatted log
files)
3. I'm not aware of automatic log analysis capability in it - it's nice to
graph, draw statistics and maybe generate reports (when it finally comes up
with a result), but not for alert generation.

Cheers,

--Amos


> 2011/2/16 Amos Shapira <amos.shapira at gmail.com>
>
>> Hello,
>>
>>
>> As part of PCI-DSS compliance I'm working on (ref:
>> http://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard),
>> we need to implement automatic log file analysis and alerting. (It's also a
>> Good Thing(TM) to have such a thing in place in general).
>>
>> LogWatch is not enough since it can't handle the amount of logs generated
>> by our system (we generate ~6Gb of compressed HTTP daemon access log files
>> every 24 hours alone, not to mention many other log files and more to come
>> as we progress with PCI compliance) and still requires someone to manually
>> go through its reports.
>>
>> Instead, I see many ads for commercial systems which can analyse log files
>> in near real time and generate custom alerts about suspicious activity
>> outside a learned activity pattern. These systems cost a fortune.
>>
>> On the other hand - I saw mentions of open-source system which dump log
>> files onto a NoSQL database and achieve the same functionality with free
>> tools.
>>
>> Alas - I lost the references for the later.
>>
>> Closest thing I found is Flume (https://github.com/cloudera/flume).
>> Someone tells me that it also does the actual analysis but I don't see this
>> mentioned on its web site.
>>
>> Does anyone else here have an idea about such systems?
>>
>> Thanks,
>>
>> --Amos
>> Does anyone
>>
>> _______________________________________________
>> Linux-il mailing list
>> Linux-il at cs.huji.ac.il
>> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20110216/9b2d5b64/attachment-0001.html>