<div dir="ltr"><div class="gmail_quote">(Replying to a private message on the list since Tom said he meant this to go to the list in another private reply)<br><br>On 16 February 2011 21:39, Tom Goren <span dir="ltr"><<a href="mailto:tom@tomgoren.com">tom@tomgoren.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr">Using Flume (part of the hadoop stack) is a viable option for this task, however it introduces its own significant levels of complication - set ahead resources to tackle the learning curve - and also the related technologies (lots of java and map reduce buzzwords).<div>
<br></div><div>There is a reason these commercial implementations cost a fortune - real time analysis of large amounts of data requires lots of CPU and disk space...</div><div><br></div><div>Just my two cents,</div></div>
</blockquote><div><br>Thanks.<br><br>I think I can appreciate the large investment in learning NoSQL, MapReduce and friends. We have a few people in the company who have some experience with NoSQL clusters in general and with Hadoop in particular (they used such stuff elsewhere, some of them are looking into Hadoop specifically as part of our advanced research), I'm also a bit familiar with Java (been earning my bread for 6 years coding Java in the past).<br>
<br>Anyway - as it usually happens - the next google after I sent this question in half-despair, brought up Chukwa (<a href="http://wiki.apache.org/hadoop/Chukwa">http://wiki.apache.org/hadoop/Chukwa</a>), which also mentions Salsa (<a href="http://www.usenix.org/event/wasl08/tech/full_papers/tan/tan_html/">http://www.usenix.org/event/wasl08/tech/full_papers/tan/tan_html/</a>). I think this is what I heard about before. Now I have to see how mature this is.<br>
<br>Cheers,<br><br>--Amos<br> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr"><div>
<br>
</div><div>Tom.<br><br><div class="gmail_quote">2011/2/16 Amos Shapira <span dir="ltr"><<a href="mailto:amos.shapira@gmail.com" target="_blank">amos.shapira@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div><div></div><div class="h5">
<div dir="ltr">Hello,<br><br>As part of PCI-DSS compliance I'm working on (ref: <a href="http://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard" target="_blank">http://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard</a>), we need to implement automatic log file analysis and alerting. (It's also a Good Thing(TM) to have such a thing in place in general).<br>
<br>LogWatch is not enough since it can't handle the amount of logs generated by our system (we generate ~6Gb of compressed HTTP daemon access log files every 24 hours alone, not to mention many other log files and more to come as we progress with PCI compliance) and still requires someone to manually go through its reports.<br>
<br>Instead, I see many ads for commercial systems which can analyse log files in near real time and generate custom alerts about suspicious activity outside a learned activity pattern. These systems cost a fortune.<br><br>
On the other hand - I saw mentions of open-source system which dump log files onto a NoSQL database and achieve the same functionality with free tools.<br><br>Alas - I lost the references for the later.<br><br>Closest thing I found is Flume (<a href="https://github.com/cloudera/flume" target="_blank">https://github.com/cloudera/flume</a>). Someone tells me that it also does the actual analysis but I don't see this mentioned on its web site.<br>
<br>Does anyone else here have an idea about such systems?<br><br>Thanks,<br><br>--Amos<br>Does anyone <br></div>
<br></div></div>_______________________________________________<br>
Linux-il mailing list<br>
<a href="mailto:Linux-il@cs.huji.ac.il" target="_blank">Linux-il@cs.huji.ac.il</a><br>
<a href="http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il" target="_blank">http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il</a><br>
<br></blockquote></div><br></div></div>
</blockquote></div><br></div>