poor Raid5 write speed

[William Park]

On Tue, Jul 27, 2010 at 03:45:18PM -0400, Lennart Sorensen wrote:
> On Tue, Jul 27, 2010 at 12:36:44PM -0400, William Park wrote:
> > My Software Raid5 has lower write speed than a single harddisk.
> > I tried copying whole bunch of *.iso
> > 
> >     - from Raid5 to a separate harddisk		--> 130 MB/s
> >     - from the separate harddisk to Raid5	--> 110 MB/s
> > 
> > Is this normal?  
> 
> Depends on your setup.
> 
> What raid block size?

chunksize = 64kB

> How many disks in the raid?

4 x 1TB

> What stride size is your filesystem using?
> Which filesystem?

mkfs.ext4 -m0.1 -E stride=16,stripe-width=48
where
    stride = chunksize / blocksize = 64kB/4kB = 16
    stripe-width = stride * (4 - 1) = 16 * 3 = 48

> Which controller and on which bus?

nVidia MP55 (nForce 570 Ultra, Asus M2N-E).

> 
> Which CPU?

Single-core Athlon64 LE-1620 (2.4GHz, 1MB, 45W).
DDR2-667 (AM2 special, 8GB)

> 
> Here is what I get:
> 
> mythtv64:~# hdparm -tT /dev/sd[abcd] /dev/md[012]
> 
> mythtv64:~# hdparm -tT /dev/sd[abcd] /dev/md[012]
> 
> /dev/sda:
>  Timing cached reads:   6248 MB in  2.00 seconds = 3126.86 MB/sec
>  Timing buffered disk reads:  296 MB in  3.01 seconds =  98.46 MB/sec
> 
> /dev/sdb:
>  Timing cached reads:   6012 MB in  2.00 seconds = 3007.89 MB/sec
>  Timing buffered disk reads:  286 MB in  3.01 seconds =  95.00 MB/sec
> 
> /dev/sdc:
>  Timing cached reads:   6394 MB in  2.00 seconds = 3199.19 MB/sec
>  Timing buffered disk reads:  276 MB in  3.17 seconds =  87.09 MB/sec
> 
> /dev/sdd:
>  Timing cached reads:   5274 MB in  2.00 seconds = 2638.96 MB/sec
>  Timing buffered disk reads:  298 MB in  3.01 seconds =  99.06 MB/sec
> 
> /dev/md0:
>  Timing cached reads:   5830 MB in  2.00 seconds = 2916.87 MB/sec
>  Timing buffered disk reads:  272 MB in  3.00 seconds =  90.54 MB/sec
> 
> /dev/md1:
>  Timing cached reads:   5156 MB in  2.00 seconds = 2580.49 MB/sec
>  Timing buffered disk reads:  302 MB in  3.01 seconds = 100.21 MB/sec
> 
> /dev/md2:
>  Timing cached reads:   6388 MB in  2.00 seconds = 3195.83 MB/sec
>  Timing buffered disk reads:  700 MB in  3.04 seconds = 230.24 MB/sec
> 
> md0 and md1 are raid1 on sda/b and sdc/d respectively, while md2 is
> raid5 across all 4 disks.  So I get about 100MB/s on one disk, and about
> 230MB/s on raid5 on 4 disks.  I am certainly satisfied with that.

# hdparm -tT /dev/sd[bcdef] /dev/md5

/dev/sdb:
 Timing cached reads:   1684 MB in  2.00 seconds = 842.19 MB/sec
 Timing buffered disk reads:  392 MB in  3.00 seconds = 130.63 MB/sec

/dev/sdc:
 Timing cached reads:   1756 MB in  2.00 seconds = 877.90 MB/sec
 Timing buffered disk reads:  342 MB in  3.01 seconds = 113.56 MB/sec

/dev/sdd:
 Timing cached reads:   1738 MB in  2.00 seconds = 868.45 MB/sec
 Timing buffered disk reads:  288 MB in  3.00 seconds =  95.98 MB/sec

/dev/sde:
 Timing cached reads:   1842 MB in  2.00 seconds = 919.92 MB/sec
 Timing buffered disk reads:  300 MB in  3.02 seconds =  99.23 MB/sec

/dev/sdf:
 Timing cached reads:   1790 MB in  2.00 seconds = 895.27 MB/sec
 Timing buffered disk reads:  358 MB in  3.00 seconds = 119.19 MB/sec

/dev/md5:
 Timing cached reads:   1706 MB in  2.00 seconds = 852.76 MB/sec
 Timing buffered disk reads:  786 MB in  3.00 seconds = 261.62 MB/sec

/dev/sdb is my root disk, and it does 130MB/s for both sequential read
and write.  That's good.  /dev/md5 is Raid5 over /dev/sd[cdef].  Its
sequential read is 260MB/s, but sequential write is 110MB/s.  Now,
that's bad.

-- 
William

--
The Toronto Linux Users Group.      Meetings: http://gtalug.org/
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists