FusionIO ioDrive

July 6th 2009 09:11 pm

Well, I got my hands on one of the fusion-io ioDrives a couple of weeks ago. unfortunately they do not work in the version of VMware ESX that we are using, though they are working on drivers for the 64bit ESX4. I did not have time to set up a physical machine to test on our application running SQL server 2005 so I have just quickly done some IO benchmarks in Linux at home. I was going to test btrfs and its SSD mode at the same time it but hit too many problems trying to get the drivers and the btrfs module in the kernel together.

First setup a 4 drive raid0 array for comparison

root@George:/home/tom# fdisk -l | grep 500
Disk /dev/sdb doesn’t contain a valid partition table
Disk /dev/sdc doesn’t contain a valid partition table
Disk /dev/sdd doesn’t contain a valid partition table
Disk /dev/sde doesn’t contain a valid partition table
Disk /dev/sdb: 500.1 GB, 500107862016 bytes
Disk /dev/sdc: 500.1 GB, 500107862016 bytes
Disk /dev/sdd: 500.1 GB, 500107862016 bytes
Disk /dev/sde: 500.1 GB, 500107862016 bytes

root@George:/home/tom# mdadm --create /dev/md0 -l 0 -n 4 /dev/sd[bcde]
mdadm: array /dev/md0 started.

root@George:/home/tom# mkfs.ext2 /dev/md0
root@George:/home/tom# mkfs.ext2 /dev/fioa

fio:
I was looking for an iometer-a-like for linux to quickly get some semi-meaningful results (bonnie++ was returning results saying it was too quick to measure or something)
Fio lets you create a text file description of a workload, with a choice of IO libs and loads of options, you can set concurrent jobs also, see this excellent linux.com article for more info

random-read-test-aio-32thread-20G.fio 

[random-read]
rw=randread
size=20G
ioengine=libaio
iodepth=32
direct=1
invalidate=1

root@George:/fusionio# fio /fio/random-read-test-aio-32thread-20G.fio
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
Starting 1 process
random-read: Laying out IO file(s) (1 file(s) / 20480MiB)
Jobs: 1 (f=1): [r] [100.0% done] [139M/0K /s] [35K/0 iops] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=25705
  read : io=20480MiB, bw=157614KiB/s, iops=39403, runt=133056msec
    slat (usec): min=4096, max=4096, avg=4096.00, stdev= 0.00
    clat (usec): min=324, max=325739, avg=788.05, stdev=1238.41
    bw (KiB/s) : min=45800, max=196536, per=100.24%, avg=157996.42, stdev=22439.38
  cpu          : usr=9.23%, sys=71.81%, ctx=2180058, majf=1, minf=698
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=5242880/0, short=0/0
     lat (usec): 500=0.01%, 750=78.27%, 1000=10.86%
     lat (msec): 2=9.31%, 4=1.11%, 10=0.30%, 20=0.10%, 50=0.04%
     lat (msec): 100=0.01%, 500=0.01%

Run status group 0 (all jobs):
READ: io=20480MiB, aggrb=157614KiB/s, minb=157614KiB/s, maxb=157614KiB/s, mint=133056msec, maxt=133056msec

Disk stats (read/write):
  fioa: ios=5244775/2, merge=0/0, ticks=303464/0, in_queue=0, util=0.00%

random-write-test-aio-32thread-20G.fio 

[random-write]
rw=randwrite
size=20G
ioengine=libaio
iodepth=32
direct=1
invalidate=1
root@George:/fusionio# fio /fio/random-write-test-aio-32thread-20G.fio
random-write: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
Starting 1 process
random-write: Laying out IO file(s) (1 file(s) / 20480MiB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/18210K /s] [0/4446 iops] [eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=7105
  write: io=20480MiB, bw=23406KiB/s, iops=5851, runt=895978msec
    slat (usec): min=4096, max=4096, avg=4096.00, stdev= 0.00
    clat (msec): min=1, max=322, avg= 5.30, stdev= 6.29
    bw (KiB/s) : min=    0, max=94080, per=99.99%, avg=23404.82, stdev=13749.60
  cpu          : usr=3.19%, sys=27.08%, ctx=5303118, majf=0, minf=4369
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/5242880, short=0/0
     lat (msec): 2=14.96%, 4=46.44%, 10=17.79%, 20=20.45%, 50=0.23%
     lat (msec): 100=0.06%, 250=0.06%, 500=0.01%
Run status group 0 (all jobs):
  WRITE: io=20480MiB, aggrb=23406KiB/s, minb=23406KiB/s, maxb=23406KiB/s, mint=895978msec, maxt=895978msec
Disk stats (read/write):
  fioa: ios=164/5339709, merge=0/0, ticks=28/961368, in_queue=0, util=0.00%

Now the RAID0 array,took all night to complete the same tests.

root@George:/raid0# fio /fio/random-read-test-aio-32thread-20G.fio ;
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [4136K/0K /s] [1010/0 iops] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=20079
  read : io=20480MiB, bw=3475KiB/s, iops=868, runt=6033707msec
    slat (usec): min=4096, max=4096, avg=4096.00, stdev= 0.00
    clat (usec): min=3, max=1048K, avg=36802.65, stdev=37338.47
    bw (KiB/s) : min=  982, max= 4367, per=100.10%, avg=3478.65, stdev=279.09
  cpu          : usr=0.55%, sys=1.84%, ctx=4607245, majf=0, minf=30634
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=5242880/0, short=0/0
     lat (usec): 4=0.01%, 50=0.01%, 100=0.01%, 250=0.03%, 500=0.09%
     lat (usec): 750=0.01%, 1000=0.01%
     lat (msec): 2=0.02%, 4=0.93%, 10=16.14%, 20=26.65%, 50=32.57%
     lat (msec): 100=16.79%, 250=6.56%, 500=0.19%, 750=0.01%, 1000=0.01%
     lat (msec): 2000=0.01%
Run status group 0 (all jobs):
   READ: io=20480MiB, aggrb=3475KiB/s, minb=3475KiB/s, maxb=3475KiB/s, mint=6033707msec, maxt=6033707msec
Disk stats (read/write):
  md0: ios=5242880/4900, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sdb: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sdc: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sdd: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sde: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%

root@George:/raid0# fio /fio/random-write-test-aio-32thread-20G.fio

random-write: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
Starting 1 process
random-write: Laying out IO file(s) (1 file(s) / 20480MiB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/9427K /s] [0/2301 iops] [eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=2789
  write: io=20480MiB, bw=10075KiB/s, iops=2518, runt=2081336msec
    slat (usec): min=4096, max=4096, avg=4096.00, stdev= 0.00
    clat (msec): min=1, max=773, avg=12.31, stdev= 7.27
    bw (KiB/s) : min=    0, max=20552, per=100.09%, avg=10083.67, stdev=2143.10
  cpu          : usr=1.62%, sys=11.61%, ctx=5332417, majf=0, minf=10120
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/5242880, short=0/0
     lat (msec): 2=0.01%, 4=0.01%, 10=23.86%, 20=75.38%, 50=0.70%
     lat (msec): 100=0.04%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
Run status group 0 (all jobs):
  WRITE: io=20480MiB, aggrb=10075KiB/s, minb=10075KiB/s, maxb=10075KiB/s, mint=2081336msec, maxt=2081336msec
Disk stats (read/write):
  md0: ios=159/5353967, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sdb: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sdc: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sdd: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sde: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%

Key figures are IOPS, bw (bandwith) and completion time.

IOPS:
read – 39403/868 ~45x
write – 5851/2518 ~2x

bw
read – 157614/3475 ~45x
write – 23406/10075 ~2x

Time:
read – 133/6033 ~45x
write – 895/2081 ~2x

I am surprised the array wrote so fast ~800 IOPS for read and write was my expectation, I did not have time to repeat with a different IO lib and cant explain why random writes would be 3x faster than random reads on a RAID0 array so I think its to do with libaio.

In short my home PC temporarily had read IOPS equivalent to 200 hard disks spinning at 15′000 RPM, ace.

Prices are as follows:
80GB ioDrive (SLC) US$3,600
160GB ioDrive (SLC) US$7,200
320GB ioDrive (MLC) US$14,400
320GB ioDrive Duo (SLC) US$11,900
640GB ioDrive Duo (MLC) US$9,795

See http://www.fusionio.com/Products.aspx for more information.

Posted by tom under Uncategorized | 2 Comments »

2 Responses to “FusionIO ioDrive”

  1. blackburried responded on 07 Jul 2009 at 3:18 pm #

    Note: “Starting 1 process”

    … increase your thread count.

  2. John Cagle responded on 07 Jul 2009 at 3:47 pm #

    Tom,

    According to the results, it doesn’t look like you’re using multiple threads for your fio test. You have named your fio control files using “32thread”, but the file doesn’t have “numjobs=” parameter. I suggest using numjobs=N, where N is the number of CPU cores (or hyperthreaded cores) in your system. For example, if you have a single Nehalem processor, uses “numjobs=8″ (4 cores x 2 hyperthreads).

    Without using multiple threads (jobs) you won’t see the maximum performance from the ioDrive.

    You should also try adding the “softrandommap=1″ parameter — without this parameter, the random map used may take a lot of RAM/CPU to maintain (to have a “perfect” random access pattern that doesn’t repeat access to any specific block).

    Also, before you do any read tests, make sure you write the same amount of data to the ioDrive. If the ioDrive has a fresh “fio-format”, it will have no data, and if you read from it, it will just return zero’s without actually accessing the NAND flash. Without writing first, you could get inaccurate results.

    Also, this technology is available for blade servers now from HP:
    http://en.wikipedia.org/wiki/IO_Accelerator

    Cheers,
    John

Trackback URI | Comments RSS

Leave a Reply