Well, I got my hands on one of the fusion-io ioDrives a couple of weeks ago. unfortunately they do not work in the version of VMware ESX that we are using, though they are working on drivers for the 64bit ESX4. I did not have time to set up a physical machine to test on our application running SQL server 2005 so I have just quickly done some IO benchmarks in Linux at home. I was going to test btrfs and its SSD mode at the same time it but hit too many problems trying to get the drivers and the btrfs module in the kernel together.
First setup a 4 drive raid0 array for comparison
root@George:/home/tom# fdisk -l | grep 500
Disk /dev/sdb doesn’t contain a valid partition table
Disk /dev/sdc doesn’t contain a valid partition table
Disk /dev/sdd doesn’t contain a valid partition table
Disk /dev/sde doesn’t contain a valid partition table
Disk /dev/sdb: 500.1 GB, 500107862016 bytes
Disk /dev/sdc: 500.1 GB, 500107862016 bytes
Disk /dev/sdd: 500.1 GB, 500107862016 bytes
Disk /dev/sde: 500.1 GB, 500107862016 bytes
root@George:/home/tom# mdadm --create /dev/md0 -l 0 -n 4 /dev/sd[bcde]
mdadm: array /dev/md0 started.
root@George:/home/tom# mkfs.ext2 /dev/md0
root@George:/home/tom# mkfs.ext2 /dev/fioa
fio:
I was looking for an iometer-a-like for linux to quickly get some semi-meaningful results (bonnie++ was returning results saying it was too quick to measure or something)
Fio lets you create a text file description of a workload, with a choice of IO libs and loads of options, you can set concurrent jobs also, see this excellent linux.com article for more info
random-read-test-aio-32thread-20G.fio
[random-read]
rw=randread
size=20G
ioengine=libaio
iodepth=32
direct=1
invalidate=1
root@George:/fusionio# fio /fio/random-read-test-aio-32thread-20G.fio
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
Starting 1 process
random-read: Laying out IO file(s) (1 file(s) / 20480MiB)
Jobs: 1 (f=1): [r] [100.0% done] [139M/0K /s] [35K/0 iops] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=25705
read : io=20480MiB, bw=157614KiB/s, iops=39403, runt=133056msec
slat (usec): min=4096, max=4096, avg=4096.00, stdev= 0.00
clat (usec): min=324, max=325739, avg=788.05, stdev=1238.41
bw (KiB/s) : min=45800, max=196536, per=100.24%, avg=157996.42, stdev=22439.38
cpu : usr=9.23%, sys=71.81%, ctx=2180058, majf=1, minf=698
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued r/w: total=5242880/0, short=0/0
lat (usec): 500=0.01%, 750=78.27%, 1000=10.86%
lat (msec): 2=9.31%, 4=1.11%, 10=0.30%, 20=0.10%, 50=0.04%
lat (msec): 100=0.01%, 500=0.01%
Run status group 0 (all jobs):
READ: io=20480MiB, aggrb=157614KiB/s, minb=157614KiB/s, maxb=157614KiB/s, mint=133056msec, maxt=133056msec
Disk stats (read/write):
fioa: ios=5244775/2, merge=0/0, ticks=303464/0, in_queue=0, util=0.00%
random-write-test-aio-32thread-20G.fio
[random-write]
rw=randwrite
size=20G
ioengine=libaio
iodepth=32
direct=1
invalidate=1
root@George:/fusionio# fio /fio/random-write-test-aio-32thread-20G.fio
random-write: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
Starting 1 process
random-write: Laying out IO file(s) (1 file(s) / 20480MiB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/18210K /s] [0/4446 iops] [eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=7105
write: io=20480MiB, bw=23406KiB/s, iops=5851, runt=895978msec
slat (usec): min=4096, max=4096, avg=4096.00, stdev= 0.00
clat (msec): min=1, max=322, avg= 5.30, stdev= 6.29
bw (KiB/s) : min= 0, max=94080, per=99.99%, avg=23404.82, stdev=13749.60
cpu : usr=3.19%, sys=27.08%, ctx=5303118, majf=0, minf=4369
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued r/w: total=0/5242880, short=0/0
lat (msec): 2=14.96%, 4=46.44%, 10=17.79%, 20=20.45%, 50=0.23%
lat (msec): 100=0.06%, 250=0.06%, 500=0.01%
Run status group 0 (all jobs):
WRITE: io=20480MiB, aggrb=23406KiB/s, minb=23406KiB/s, maxb=23406KiB/s, mint=895978msec, maxt=895978msec
Disk stats (read/write):
fioa: ios=164/5339709, merge=0/0, ticks=28/961368, in_queue=0, util=0.00%
Now the RAID0 array,took all night to complete the same tests.
root@George:/raid0# fio /fio/random-read-test-aio-32thread-20G.fio ;
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [4136K/0K /s] [1010/0 iops] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=20079
read : io=20480MiB, bw=3475KiB/s, iops=868, runt=6033707msec
slat (usec): min=4096, max=4096, avg=4096.00, stdev= 0.00
clat (usec): min=3, max=1048K, avg=36802.65, stdev=37338.47
bw (KiB/s) : min= 982, max= 4367, per=100.10%, avg=3478.65, stdev=279.09
cpu : usr=0.55%, sys=1.84%, ctx=4607245, majf=0, minf=30634
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued r/w: total=5242880/0, short=0/0
lat (usec): 4=0.01%, 50=0.01%, 100=0.01%, 250=0.03%, 500=0.09%
lat (usec): 750=0.01%, 1000=0.01%
lat (msec): 2=0.02%, 4=0.93%, 10=16.14%, 20=26.65%, 50=32.57%
lat (msec): 100=16.79%, 250=6.56%, 500=0.19%, 750=0.01%, 1000=0.01%
lat (msec): 2000=0.01%
Run status group 0 (all jobs):
READ: io=20480MiB, aggrb=3475KiB/s, minb=3475KiB/s, maxb=3475KiB/s, mint=6033707msec, maxt=6033707msec
Disk stats (read/write):
md0: ios=5242880/4900, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
sdb: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
sdc: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
sdd: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
sde: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
root@George:/raid0# fio /fio/random-write-test-aio-32thread-20G.fio
random-write: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
Starting 1 process
random-write: Laying out IO file(s) (1 file(s) / 20480MiB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/9427K /s] [0/2301 iops] [eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=2789
write: io=20480MiB, bw=10075KiB/s, iops=2518, runt=2081336msec
slat (usec): min=4096, max=4096, avg=4096.00, stdev= 0.00
clat (msec): min=1, max=773, avg=12.31, stdev= 7.27
bw (KiB/s) : min= 0, max=20552, per=100.09%, avg=10083.67, stdev=2143.10
cpu : usr=1.62%, sys=11.61%, ctx=5332417, majf=0, minf=10120
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued r/w: total=0/5242880, short=0/0
lat (msec): 2=0.01%, 4=0.01%, 10=23.86%, 20=75.38%, 50=0.70%
lat (msec): 100=0.04%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
Run status group 0 (all jobs):
WRITE: io=20480MiB, aggrb=10075KiB/s, minb=10075KiB/s, maxb=10075KiB/s, mint=2081336msec, maxt=2081336msec
Disk stats (read/write):
md0: ios=159/5353967, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
sdb: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
sdc: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
sdd: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
sde: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
Key figures are IOPS, bw (bandwith) and completion time.
IOPS:
read – 39403/868 ~45x
write – 5851/2518 ~2x
bw
read – 157614/3475 ~45x
write – 23406/10075 ~2x
Time:
read – 133/6033 ~45x
write – 895/2081 ~2x
I am surprised the array wrote so fast ~800 IOPS for read and write was my expectation, I did not have time to repeat with a different IO lib and cant explain why random writes would be 3x faster than random reads on a RAID0 array so I think its to do with libaio.
In short my home PC temporarily had read IOPS equivalent to 200 hard disks spinning at 15’000 RPM, ace.
Prices are as follows:
80GB ioDrive (SLC) US$3,600
160GB ioDrive (SLC) US$7,200
320GB ioDrive (MLC) US$14,400
320GB ioDrive Duo (SLC) US$11,900
640GB ioDrive Duo (MLC) US$9,795
See http://www.fusionio.com/Products.aspx for more information.
Share on Facebook
Posted by tom under Uncategorized | 2 Comments »
