<div dir="ltr">On Wed, May 8, 2013 at 10:47 PM, Oleg Goldshmidt <span dir="ltr"><<a href="mailto:pub@goldshmidt.org" target="_blank">pub@goldshmidt.org</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im"><br>
</div>Disclaimer: I am definitely not an expert on the subject matter and I<br>
hardly know what I am talking about (in this case?). Creativity is no<br>
substitute for knowing what you are doing.<br>
<br>
Now let me try and get creative.<br>
<br>
What is your purpose? Just doing something fancy to impress your boss</blockquote><div><br></div><div style>My real purpose, or the official stated purpose ;-)</div><div style>The thing is, we build 400MB artifact multiple times a day. Say 5<span style="background-color:rgb(249,249,249);color:rgb(51,51,51);font-family:'Open Sans','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:14px;line-height:20px">×200</span><span style="background-color:rgb(249,249,249);color:rgb(51,51,51);font-family:'Open Sans','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:14px;line-height:20px">×400</span><span style="background-color:rgb(249,249,249);color:rgb(51,51,51);font-family:'Open Sans','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:14px;line-height:20px">=400Gb a year, not sure I have the space. This means I have to maintain the repository (delete old build results, etc).</span></div>
<div style><span style="background-color:rgb(249,249,249);color:rgb(51,51,51);font-family:'Open Sans','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:14px;line-height:20px">OTOH, if I use dedupe technique, I can keep all build artifacts and forget about it altogether. I'll never ever fill a modern 250Gb disk.</span></div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> or<br>
truly save space, e.g., if this stuff - everything that gets built - is<br>
backed up? I'll assume the latter.<br>
<br>
[Aside: if it is not backed up, how many versions do you really need to<br>
keep and why is it an issue?]<br>
<br>
1. I would probably look into using a version control system rather than<br>
a filesystem.<br>
<br>
a) Modern version control systems are often/usually capable of<br>
storing binary diffs between revisions. Frankly, I've never looked<br>
at how git or mercurial do that (probably quite well), but even,<br>
say, SVN should be able to store a binary diff on commit. IIRC SVN<br>
diffs using xdelta or similar.<br></blockquote><div><br></div><div style>I suspect they don't work well on gzipped content:</div><div style><br></div><div style>Binary file with diff:</div><div style><br></div>
<div style><div>(fabenv_mac)❯ du -h .git/objects</div><div>4.0K<span class="" style="white-space:pre"> </span>.git/objects/08</div><div>232K<span class="" style="white-space:pre"> </span>.git/objects/3d</div><div>4.0K<span class="" style="white-space:pre"> </span>.git/objects/44</div>
<div>4.0K<span class="" style="white-space:pre"> </span>.git/objects/84</div><div>232K<span class="" style="white-space:pre"> </span>.git/objects/d7</div><div>4.0K<span class="" style="white-space:pre"> </span>.git/objects/ee</div>
<div> 0B<span class="" style="white-space:pre"> </span>.git/objects/info</div><div> 0B<span class="" style="white-space:pre"> </span>.git/objects/pack</div><div>480K<span class="" style="white-space:pre"> </span>.git/objects</div>
<div><div>(fabenv_mac)❯ git gc</div><div>Counting objects: 6, done.</div><div>Delta compression using up to 8 threads.</div><div>Compressing objects: 100% (6/6), done.</div><div>Writing objects: 100% (6/6), done.</div><div>
Total 6 (delta 1), reused 0 (delta 0)</div><div>(fabenv_mac)❯ find .git/objects/ -type f|xargs du -h</div><div>4.0K<span class="" style="white-space:pre"> </span>.git/objects//info/packs</div><div>4.0K<span class="" style="white-space:pre"> </span>.git/objects//pack/pack-bd546ad638a3a27e16e57298469558cdd5018879.idx</div>
<div>216K<span class="" style="white-space:pre"> </span>.git/objects//pack/pack-bd546ad638a3a27e16e57298469558cdd5018879.pack</div></div><div><br></div><div style>However when it's gzipped:</div><div style><br></div><div style>
<div>(fabenv_mac)❯ find .git/objects/ -type f|xargs du -h</div><div>4.0K<span class="" style="white-space:pre"> </span>.git/objects//2a/8fc1caff222272cb043bbf18d240c54315f9d0</div><div>4.0K<span class="" style="white-space:pre"> </span>.git/objects//4e/71017582e4f46b3641d27084e5cae0c3303974</div>
<div>216K<span class="" style="white-space:pre"> </span>.git/objects//70/81d2b08bc00dff607aea60e9c6fecbc6950b16</div><div>216K<span class="" style="white-space:pre"> </span>.git/objects//8e/71116f4a7f89af36051b8b431427c0e88ab741</div>
<div>4.0K<span class="" style="white-space:pre"> </span>.git/objects//92/00e8eaf6093e6cfd07735bc9fe30da4e86db33</div><div>4.0K<span class="" style="white-space:pre"> </span>.git/objects//9d/e5e4af60673998992579be40960d65a5b498a3</div>
<div><div>(fabenv_mac)❯ git gc</div><div>Counting objects: 6, done.</div><div>Delta compression using up to 8 threads.</div><div>Compressing objects: 100% (6/6), done.</div><div>Writing objects: 100% (6/6), done.</div><div>
Total 6 (delta 0), reused 0 (delta 0)</div><div>(fabenv_mac)❯ du -sh .git/objects</div><div>440K<span class="" style="white-space:pre"> </span>.git/objects</div><div>(fabenv_mac)❯ find .git/objects/ -type f|xargs du -h</div>
<div>4.0K<span class="" style="white-space:pre"> </span>.git/objects//info/packs</div><div>4.0K<span class="" style="white-space:pre"> </span>.git/objects//pack/pack-5253e59d6e6950fbbf8455310bb32e3004ded6b2.idx</div><div>
432K<span class="" style="white-space:pre"> </span>.git/objects//pack/pack-5253e59d6e6950fbbf8455310bb32e3004ded6b2.pack</div></div></div></div><div> </div><div style>Note the total size didn't change when the same two versions of the file (gcc binary with the first byte changed) were gzipp'd.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
b) I suppose one can write commit/get (I use this terminology only<br>
because I mentioned SVN, consider it generic) hooks for most<br>
version control systems to tar/untar (and possibly zip/unzip jars)<br>
if you really need something close to what you described.<br></blockquote><div><br></div><div style>All your suggestions are basically good, but they mean I have to change the work style of all the team.</div><div style>
The main benefit in my suggestion is, that it's completely transparent. I add a single mount command to the directory I already keep my binary files, and that's it. Everything still works as usual, except I never need to worry about deleting anything.</div>
<div style>BTW Java artifacts have a very easy to set-up and known deployment mechanism (binary repository with a known protocol to keep binary build products, known API for how to get a build product, etc). It's good to keep your work environment as standard as you reasonably can.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
3. I *heard* of lessfs but I have absolutely no idea if it is relevant<br>
(search and check?).<br></blockquote><div><br></div><div style>I need to check how it supports gzip.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
4. MVFS (Multi-Version FileSystem - the underlying technology of<br>
Rational's ClearCase) comes to mind. It's not open source (or cheap).<br>
It is not userspace. It is probably only available as a part of<br>
ClearCase. Just mentioning for completeness.<br>
<br>
If none of the above is even remotely relevant, sorry for the noise.<br>
<span class=""><font color="#888888"><br>
--<br>
Oleg Goldshmidt | <a href="mailto:pub@goldshmidt.org">pub@goldshmidt.org</a><br>
</font></span></blockquote></div><br></div></div>