Cultural Adventures

January 22nd 2010

The last two weekends I have traveled across to London from Faversham to see friends and visit a few museums.

During the weekend of 9/10th I met up with my sister in the British Museum and showed her around (one of my 101 goals ticked off in the process). I was very pleased to finally see the flood tablets I had read about a lot (in brief – a pre-biblical flood narrative dating back to Sumeria that is part of the epic of Gilgamesh that the Genesis writers seem to have borrowed from). Also mid last year a metal-detectorist found the largest hoard of Anglo-Saxon gold ever found, it has been named “The Staffordshire Horde” and is on display there while they catalog it.

On the Sunday we went to the National Gallery and saw The Sacred Made Real a wonderful display of Spanish polychromatic wooden sculpture. It is the best exhibit I have seen in ages and finishes this weekend, go and see it if you can. It is interesting to see sculpture in colour as we are so used to classical and neo-classical sculpture being the very austere white. Interestingly greco-roman sculpture would originally have been painted anyway so the neo-classicists were harking back to something that never really existed.

The next weekend I stayed in Richmond for a friends birthday and we popped to the V&A on saturday and well just wow. I only spent an hour there but it is amazing that London has (at least) 3 really world class museums, it is on par with the British Museum and the Metropolitan Museum Of Art – really fabulous, I will return soon.

On the Sunday I went and looked at the Beatles to Bowie exhibition of 60’s photography, which was pretty good, then returned to Faversham.

All in all good fun in the greatest city in the world.

Posted by tom under 101 & jolly & museum | No Comments »

Building A Backblaze Storage Pod

January 5th 2010

A while ago I saw the Backblaze storage pod and was impressed.

Like many others I thought:

  • I want one
  • Wouldn’t it work great with ZFS
  • The hardware sucks

Building one

I used to make and sell storage servers using Linux a while ago with media streaming software and easy setup (before all the ready-rolled ones came out) so the software side is not a major challenge. Backblaze have released a template for the cases and a list of other components and coincidentally a good friend is a Mechanical Design Engineer and can work on it for me. The cost for the cases drops precipitously if you buy in bulk and he is looking at making it able to be stored flat and easily assembled by folding edges together so I will take the plunge and buy a load, if anyone wants to get hold of some, please get in touch.

ZFS

As soon as you see so many disks in a case like that, it’s hard not to think of Sun’s Thumper and ZFS.

I’ve blogged about ZFS before and given talks on it. With so many disks to fail (either noisily or silently)  data loss is inevitable (and worse – you may not even be alerted), ZFS would solve this (or at least ensure you know about it). BackBlaze use custom application logic to work around this, using TomCat and HTTPS.

It’s Not Highly Available

A chap at Sun has a critique here that is totally spot on and he makes a few great points about subtle changes to Sun’s design to accommodate vibration, noise and electromagnetic radiation. In so many ways the hardware is inadequate and does not have the uptime characteristics of a device in Suns range. That said though an individual device from Sun is not as HA as, say, an EMC SAN (with mirrored write cache, dual SPs etc) as it too relies mostly on commodity hardware. For FiveNines availability you need to decide what you are doing to protect against device failure anyway, the BackBlaze devices just fail faster – that’s your trade-off.

It wont be fast

That is largely a feature of the disks and the controllers; you could get a better motherboard, disks and faster controllers, perhaps eschew the port multipliers too, if performance is a problem. A very cool new feature of ZFS (L2ARC / Hybrid Storage Pools) allows for using SSD as a second level cache, that would help. In linux dm-cache (or here)  could probably achieve something similar.

How can you make it HA?

This is really another blog (and a few weeks work hacking out the ideas), but I can think of several ways of doing what BackBlaze do in their software stack to export files (via NFS, SMB or other protocol) or block devices (ATAoE, iSCSI, NBD etc) in a robust manner.

I have ordered some of the port multipliers, got my friend working on the case and will buy the sundry bits over the next few days.

This is one of my101 goals

Posted by tom under 101 & BackBlazePod & ZFS | 1 Comment »

101 Goals in 1001 Days

January 3rd 2010

I have decided on a kind of hyper-resolution for the new year, 101 goals in 1001 days.
I have a mindmap of the ones I have settled on so far and have started working on some of them already.

To follow (allowing me to tick off some of the meta-goals):

Success criteria for each
Divide them up into “Sustained Effort”, “1-off” and “Requires Break” to make sure I don’t overextend myself. I plan on taking roughly quarterly 1 or 2 week breaks to try and do these tasks. For the sustained ones I need progress metrics, am I on track to complete in the time given?

The actual 101 things!
Need to settle on the actual 101 things, most are on there but it’s still a bit fluid.

As one of my goals is to blog more, you should be hearing more from me in 2010.

Posted by tom under 101 & Life | 1 Comment »

Can Google create community?

September 30th 2009

I have been meaning to blog for ages about a site I use called Goodreads. It is really excellent, I signed up ages ago and cant remember exactly how I found it but I think it might have been a Rails app. Anyway, it is a fairly simple idea, log what books you read (or want to), rate them and share reviews with others, I am on there as thattommyhall. What convinced me to use it was the really stellar monthly emails with interviews with authors and picks of books with a certain theme.

I have been playing with my new HTC Hero and there is a Google Android app that can scan books and add them to the (easilly overlooked) My Library section of Google Books. While the app is good, there is no feeling of community there really. Orkut does not seem to have taken off and they released Wave today but I dont know if they can really get a feeling of community in their apps. I love the stuff they put out, use gmail and docs in particular pretty much constantly but think facebook, flickr and lots of other sites somehow seem to get more of a communtity vibe, perhaps it’s even the google branding – it all looks the same.

Anyway, I have things to do today, if anyone gets an invite to Wave and wants to send me one, please do.

Posted by tom under Life | No Comments »

Getting VMware Certified Professional (VCP) on vSphere 4

September 28th 2009

On Saturday I took and passed the VCP410 exam to get VCP4.

It was not really that hard, though I have been reading about vSphere since before it shipped, follow loads of blogs on VMware, installed it as soon as it came into beta and migrated my companies clusters to it relatively early. I would say if you have VCP3 and have used vSphere you should be OK.

The frustrating thing about the exam was the questions on the config max document, in my view if you are approaching the maximums you could just look it up and memorisation is a pointless exercise. A lot of the maximums are just decisions someone in vmware made, how many NFS stores by default ? (8), max? (64). What it the tree-depth per resource pool? (12… unless you use DRS, then it’s 10). This kind of memorisation is stupid, pointless, hoop-jumping and will be the difference between passing and failing for lots of people.

The exam (like most IT certs) is multiple choice so the questions are fairly mundane and of course there is only 1 correct answer. When interviewing candidates, I always prefer questions that start “what is your” rather than “what is” as anything that is so unsubtle as to only have one answer is probably too uninteresting to spend time discussing.

I did do a nights worth of revision however, using:

Also worth considering are

You may like to see the things I have added to delicious on vmware over the last few years.

Good luck if you take it too!

Posted by tom under VMware | 2 Comments »

Hiatus, Departure, Return

September 22nd 2009

It’s been ages since I blogged as I have been mad busy in work though a lot has happened recently.

I have:

  • Left thebigword
  • Packed up my house
  • Sold/Gave away most of my possessions (keeping only books and my PC, as my friend Ben said “proves you are principally concerned with knowledge”)
  • Left Leeds
  • Gone to India
  • Returned (earlier than planned but refreshed and excited about the future)
  • Set up a limited company to go contracting
  • Started plotting a move to London
  • Begun making big lifestyle changes – drinking less, eating better and training for the Paris Marathon, a triathlon and a return to India for some Hardcore Mountaineering next year

All is well in TomLand, expect more posts now I’m not so sillybusy!

Posted by tom under Life | No Comments »

FusionIO ioDrive

July 6th 2009

Well, I got my hands on one of the fusion-io ioDrives a couple of weeks ago. unfortunately they do not work in the version of VMware ESX that we are using, though they are working on drivers for the 64bit ESX4. I did not have time to set up a physical machine to test on our application running SQL server 2005 so I have just quickly done some IO benchmarks in Linux at home. I was going to test btrfs and its SSD mode at the same time it but hit too many problems trying to get the drivers and the btrfs module in the kernel together.

First setup a 4 drive raid0 array for comparison

root@George:/home/tom# fdisk -l | grep 500
Disk /dev/sdb doesn’t contain a valid partition table
Disk /dev/sdc doesn’t contain a valid partition table
Disk /dev/sdd doesn’t contain a valid partition table
Disk /dev/sde doesn’t contain a valid partition table
Disk /dev/sdb: 500.1 GB, 500107862016 bytes
Disk /dev/sdc: 500.1 GB, 500107862016 bytes
Disk /dev/sdd: 500.1 GB, 500107862016 bytes
Disk /dev/sde: 500.1 GB, 500107862016 bytes

root@George:/home/tom# mdadm --create /dev/md0 -l 0 -n 4 /dev/sd[bcde]
mdadm: array /dev/md0 started.

root@George:/home/tom# mkfs.ext2 /dev/md0
root@George:/home/tom# mkfs.ext2 /dev/fioa

fio:
I was looking for an iometer-a-like for linux to quickly get some semi-meaningful results (bonnie++ was returning results saying it was too quick to measure or something)
Fio lets you create a text file description of a workload, with a choice of IO libs and loads of options, you can set concurrent jobs also, see this excellent linux.com article for more info

random-read-test-aio-32thread-20G.fio 

[random-read]
rw=randread
size=20G
ioengine=libaio
iodepth=32
direct=1
invalidate=1

root@George:/fusionio# fio /fio/random-read-test-aio-32thread-20G.fio
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
Starting 1 process
random-read: Laying out IO file(s) (1 file(s) / 20480MiB)
Jobs: 1 (f=1): [r] [100.0% done] [139M/0K /s] [35K/0 iops] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=25705
  read : io=20480MiB, bw=157614KiB/s, iops=39403, runt=133056msec
    slat (usec): min=4096, max=4096, avg=4096.00, stdev= 0.00
    clat (usec): min=324, max=325739, avg=788.05, stdev=1238.41
    bw (KiB/s) : min=45800, max=196536, per=100.24%, avg=157996.42, stdev=22439.38
  cpu          : usr=9.23%, sys=71.81%, ctx=2180058, majf=1, minf=698
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=5242880/0, short=0/0
     lat (usec): 500=0.01%, 750=78.27%, 1000=10.86%
     lat (msec): 2=9.31%, 4=1.11%, 10=0.30%, 20=0.10%, 50=0.04%
     lat (msec): 100=0.01%, 500=0.01%

Run status group 0 (all jobs):
READ: io=20480MiB, aggrb=157614KiB/s, minb=157614KiB/s, maxb=157614KiB/s, mint=133056msec, maxt=133056msec

Disk stats (read/write):
  fioa: ios=5244775/2, merge=0/0, ticks=303464/0, in_queue=0, util=0.00%

random-write-test-aio-32thread-20G.fio 

[random-write]
rw=randwrite
size=20G
ioengine=libaio
iodepth=32
direct=1
invalidate=1
root@George:/fusionio# fio /fio/random-write-test-aio-32thread-20G.fio
random-write: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
Starting 1 process
random-write: Laying out IO file(s) (1 file(s) / 20480MiB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/18210K /s] [0/4446 iops] [eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=7105
  write: io=20480MiB, bw=23406KiB/s, iops=5851, runt=895978msec
    slat (usec): min=4096, max=4096, avg=4096.00, stdev= 0.00
    clat (msec): min=1, max=322, avg= 5.30, stdev= 6.29
    bw (KiB/s) : min=    0, max=94080, per=99.99%, avg=23404.82, stdev=13749.60
  cpu          : usr=3.19%, sys=27.08%, ctx=5303118, majf=0, minf=4369
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/5242880, short=0/0
     lat (msec): 2=14.96%, 4=46.44%, 10=17.79%, 20=20.45%, 50=0.23%
     lat (msec): 100=0.06%, 250=0.06%, 500=0.01%
Run status group 0 (all jobs):
  WRITE: io=20480MiB, aggrb=23406KiB/s, minb=23406KiB/s, maxb=23406KiB/s, mint=895978msec, maxt=895978msec
Disk stats (read/write):
  fioa: ios=164/5339709, merge=0/0, ticks=28/961368, in_queue=0, util=0.00%

Now the RAID0 array,took all night to complete the same tests.

root@George:/raid0# fio /fio/random-read-test-aio-32thread-20G.fio ;
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [4136K/0K /s] [1010/0 iops] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=20079
  read : io=20480MiB, bw=3475KiB/s, iops=868, runt=6033707msec
    slat (usec): min=4096, max=4096, avg=4096.00, stdev= 0.00
    clat (usec): min=3, max=1048K, avg=36802.65, stdev=37338.47
    bw (KiB/s) : min=  982, max= 4367, per=100.10%, avg=3478.65, stdev=279.09
  cpu          : usr=0.55%, sys=1.84%, ctx=4607245, majf=0, minf=30634
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=5242880/0, short=0/0
     lat (usec): 4=0.01%, 50=0.01%, 100=0.01%, 250=0.03%, 500=0.09%
     lat (usec): 750=0.01%, 1000=0.01%
     lat (msec): 2=0.02%, 4=0.93%, 10=16.14%, 20=26.65%, 50=32.57%
     lat (msec): 100=16.79%, 250=6.56%, 500=0.19%, 750=0.01%, 1000=0.01%
     lat (msec): 2000=0.01%
Run status group 0 (all jobs):
   READ: io=20480MiB, aggrb=3475KiB/s, minb=3475KiB/s, maxb=3475KiB/s, mint=6033707msec, maxt=6033707msec
Disk stats (read/write):
  md0: ios=5242880/4900, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sdb: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sdc: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sdd: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sde: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%

root@George:/raid0# fio /fio/random-write-test-aio-32thread-20G.fio

random-write: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
Starting 1 process
random-write: Laying out IO file(s) (1 file(s) / 20480MiB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/9427K /s] [0/2301 iops] [eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=2789
  write: io=20480MiB, bw=10075KiB/s, iops=2518, runt=2081336msec
    slat (usec): min=4096, max=4096, avg=4096.00, stdev= 0.00
    clat (msec): min=1, max=773, avg=12.31, stdev= 7.27
    bw (KiB/s) : min=    0, max=20552, per=100.09%, avg=10083.67, stdev=2143.10
  cpu          : usr=1.62%, sys=11.61%, ctx=5332417, majf=0, minf=10120
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/5242880, short=0/0
     lat (msec): 2=0.01%, 4=0.01%, 10=23.86%, 20=75.38%, 50=0.70%
     lat (msec): 100=0.04%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
Run status group 0 (all jobs):
  WRITE: io=20480MiB, aggrb=10075KiB/s, minb=10075KiB/s, maxb=10075KiB/s, mint=2081336msec, maxt=2081336msec
Disk stats (read/write):
  md0: ios=159/5353967, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sdb: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sdc: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sdd: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sde: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%

Key figures are IOPS, bw (bandwith) and completion time.

IOPS:
read – 39403/868 ~45x
write – 5851/2518 ~2x

bw
read – 157614/3475 ~45x
write – 23406/10075 ~2x

Time:
read – 133/6033 ~45x
write – 895/2081 ~2x

I am surprised the array wrote so fast ~800 IOPS for read and write was my expectation, I did not have time to repeat with a different IO lib and cant explain why random writes would be 3x faster than random reads on a RAID0 array so I think its to do with libaio.

In short my home PC temporarily had read IOPS equivalent to 200 hard disks spinning at 15′000 RPM, ace.

Prices are as follows:
80GB ioDrive (SLC) US$3,600
160GB ioDrive (SLC) US$7,200
320GB ioDrive (MLC) US$14,400
320GB ioDrive Duo (SLC) US$11,900
640GB ioDrive Duo (MLC) US$9,795

See http://www.fusionio.com/Products.aspx for more information.

Posted by tom under Uncategorized | 2 Comments »

ZFS, it’s sometimes good to know how screwed you are.

February 11th 2009

I have just had a disk fail on my NAS, actually it happened ages ago but I was too broke to replace it. At the same time as one being faulted, another was degraded through having too many errors. Below is my interaction with ZFS to discover the extent of the problem and “fix” it.

Continue Reading »

Posted by tom under OpenSolaris & ZFS | No Comments »

Christmassy Shizzle

December 22nd 2008

Last weekend I went down to London again for a Christmas jolly with some great old uni friends, it was a lovely weekend, good wholesome fun.

On the Saturday I went to see the Wallace Collection while some girlies shopped on Kensington Highstreet (where the sales are apparently excellent but I’m not sure I could have tolerated a moment). There was an Osbert Lancaster exhibit there that was great, alongside the permanent collection which is excellent. I had never heard of him before but hes is definitely worth investigating, the blurb on the wall described him as a dandy aesthete, something I have always considered myself.

In the nighttime we watched the 1951 Scrooge with Alastair Sim in an incredible private cinema, ate nice food, got a little squiffy (port was involved you will be surprised to learn)

On the Sunday we went to see the Babylon exhibit in the British Museum and I returned to Leeds feeling tired but ace. Babylon has appeared in art of all forms, not least a few Jazz numbers.

Posted by tom under Life & archaeology | No Comments »

Byzantium at the Royal Academy of Arts

December 6th 2008

I was in London a few weeks ago and saw the wonderful exhibit at the Royal Academy of Arts.

I have been interested in Byzantine history since a historian friend at university described how they were essentially the eastern Roman empire, called themselves Romans, spoke Greek, were Christian and survived well into the Middle Ages. As ever, wikipedia has a good introduction to the topic.

Off to London again next weekend, going to see the Babylon exhibit at the British Museum and have a lovely Christmas celebration with some good old friends.

Posted by tom under Life & archaeology | No Comments »

Next »