Cat on (RAM) steroids

[Eli Billauer]

Thanks for that one. "yum install buffer". How simple.


So I tried it out:


$ dd if=/dev/zero bs=1M count=256  > /dev/null
256+0 records in
256+0 records out
268435456 bytes (268 MB) copied, 0.0258608 s, 10.4 GB/s


Original throughput is 10.4 GB/sec. Looks not so bad.


$ dd if=/dev/zero bs=1M count=256  | cat > /dev/null

256+0 records in
256+0 records out

268435456 bytes (268 MB) copied, 0.199799 s, 1.3 GB/s


The regular cat is somewhat slower, then.


Now let's try buffer:


$ dd if=/dev/zero bs=1M count=256 | buffer -m 256m > /dev/null
Cannot handle that many blocks, aborting!


Hmmm... The default buffer size is too small, so asking for 256MB didn't 
work well. Let's ask for 128kByte buffers:


$ dd if=/dev/zero bs=1M count=256 | buffer -s 128k -m 256m > /dev/null

256+0 records in
256+0 records out
268435456 bytes (268 MB) copied, 0.379354 s, 708 MB/s

On one hand, it's not all that impressive that the rate went down. But 
it's still way above the 100 MB/sec needed. On the other hand, going


$ dd if=/dev/zero bs=1M count=256 | buffer -s 128k -m 256m | hexdump -C 
 > /dev/null


gives exactly the same result, except that command prompt returns after 
a second or so, which is the time it took for hexdump to consume the data.


So I suppose we have a winner. Thanks again.


   Eli



Baruch Even wrote:

> On Sat, Feb 4, 2012 at 4:27 PM, Eli Billauer <eli at billauer.co.il 
> <mailto:eli at billauer.co.il>> wrote:
>
>     Hi all,
>
>
>     I need a simple command-line program, which works as a plain FIFO
>     stream buffer with a huge RAM. Something I can do:
>
>
>     $ fatcat -b 256M /dev/datasource | ./my_shaky_data_sink
>
>
>     The idea is that "fatcat" reads data whenever available and stores
>     it to non-swappable RAM. It then pushes the data to stdout. So
>     it's just like good old "cat", only with a potentially large tummy.
>
>
> I've used pv or buffer for such purposes.
>
> Baruch


-- 
Web: http://www.billauer.co.il


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20120204/5c0019f2/attachment-0001.html>