war story: parallel(1) command

[David Thornton]

FWIW I hashed a 2GB file 100 times with each of the digests available via
openssl, for a total of 1000 runs.

Elapse time , seconds:

md5  Average =  0.7401 <------THIS is the fastest
mdc2  Average =  46.8681
rmd160  Average =  2.2449
sha  Average =  4.06665
sha1  Average =  1.3751 <- close second.
sha224  Average =  3.6005
sha256  Average =  3.6019
sha384  Average =  6.7991
sha512  Average =  6.8885

I shuffled the runs hoping to "even out" caching.

crappy little low power Intel(R) Atom(TM) CPU  230   @ 1.60GHz

David Thornton




On Sun, Jul 28, 2013 at 3:01 PM, D. Hugh Redelmeier <hugh-pmF8o41NoarQT0dZR+AlfA at public.gmane.org> wrote:

> | From: William Muriithi <william.muriithi-Re5JQEeQqe8AvxtiuMwx3w at public.gmane.org>
>
> | For your information,  git handle this the same way, all files are hashed
> | and only one copy is kept if any share a hash. They use SHA-1, apparently
> | because its more collision resistant without being too CPU intensive
>
> SHA-1 is probably more resistant to adversaries creating collisions.
> If you aren't in an adversarial situation, that doesn't matter.
>
> | > When I did this, I soon found that there was plenty of CPU left over,
> | > so apparently md5sum is disk-bound on my machine (Core Quad Duo 6600,
> | > 2.5" external drive connected via USB 3.0).
>
> Sorry, the CPU is a Core 2 Quad Q6600.  I accidentally garbled the
> name.
>
> | Interesting, I would have guessed its CPU bound too. Goes a long way to
> | show it don't help buying cutting edge CPU now unless its for energy
> | efficient.
>
> I still foolishly think of this as cutting edge.  But of course it was
> released six years ago!  Things aren't getting faster very fast.
>
> | What filesystem is on the USB drive? NTFS by any chance? I have found
> that
> | git seem more responsive on Linux than windows. Either the windows port
> | sucks or ntfs is just too slow compared to ext4.
>
> My impression (not measured) is that NTFS on Linux is sluggish.  I
> think that it goes through a userland process (using FUSE), perhaps
> for patent reasons.  I try to avoid it because I don't trust it to be
> problem-free on Linux.  That is likely a no-longer-justified fear.
>
> This filesystem was ext3 on 2.5" external USB3.0 drive.  2.5" drives
> are slow, but ext3 and USB3.0 should be fast (untested opinions).
> --
> The Toronto Linux Users Group.      Meetings: http://gtalug.org/
> TLUG requests: Linux topics, No HTML, wrap text below 80 columns
> How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gtalug.org/pipermail/legacy/attachments/20130728/0895a73b/attachment.html>