On Thu, Dec 1, 2016 at 8:03 AM, Steven Watanabe <watanabesj@gmail.com> wrote:
AMDG
On 11/30/2016 08:05 PM, Rene Rivera wrote:
OK. Some cleanup of really old files. And removed the incoming results
so
that they get redone on the next uploads. For curiosity.. And so that people get an idea of resources.. The results for develop+master take up 14G *compressed*.
If you're measuring the size of the zip file, it isn't compressed. (unless someone changed the implementation.) The reason for this is that I found that rsync handles the uncompressed zip much better. (zip compresses files individually and there are a lot of very similar small files).
Re rsync.. Really? When I first implemented this it seemed rsync did fine with the compressed files. It correctly sent small chunks over on the spots that changed only. Now the chunks contained more than just the small file segments.. But it seemed that it was still sending over a small overall change. What I do remember is that using uncompressed made the server use less CPU. At one point it became a problem though. As the server was bogged down other operations the server was doing. So I can understand wanting to reduce that. At the time, the archive was less than 1 GB, so
I prioritized network usage over disk space. To change this you should be able to replace the nocompression_sink with deflate_sink in html_writer.
Perhaps we can use minimal compression, so that decompression is still low CPU, but we still see some disk space savings? After all it's all text which compressed well even at low compression levels. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail