thattommyhall.com » ZFS

Building A Backblaze Storage Pod

tom — Tue, 05 Jan 2010 12:41:38 +0000

[edit]You can buy BackBlaze pods now at OpenStorageSystems [/edit]

A while ago I saw the Backblaze storage pod and was impressed.

Like many others I thought:

I want one
Wouldn’t it work great with ZFS
The hardware sucks

Building one

I used to make and sell storage servers using Linux a while ago with media streaming software and easy setup (before all the ready-rolled ones came out) so the software side is not a major challenge. Backblaze have released a template for the cases and a list of other components and coincidentally a good friend is a Mechanical Design Engineer and can work on it for me. The cost for the cases drops precipitously if you buy in bulk and he is looking at making it able to be stored flat and easily assembled by folding edges together so I will take the plunge and buy a load, if anyone wants to get hold of some, please get in touch.

ZFS

As soon as you see so many disks in a case like that, it’s hard not to think of Sun’s Thumper and ZFS.

I’ve blogged about ZFS before and given talks on it. With so many disks to fail (either noisily or silently) data loss is inevitable (and worse – you may not even be alerted), ZFS would solve this (or at least ensure you know about it). BackBlaze use custom application logic to work around this, using TomCat and HTTPS.

It’s Not Highly Available

A chap at Sun has a critique here that is totally spot on and he makes a few great points about subtle changes to Sun’s design to accommodate vibration, noise and electromagnetic radiation. In so many ways the hardware is inadequate and does not have the uptime characteristics of a device in Suns range. That said though an individual device from Sun is not as HA as, say, an EMC SAN (with mirrored write cache, dual SPs etc) as it too relies mostly on commodity hardware. For FiveNines availability you need to decide what you are doing to protect against device failure anyway, the BackBlaze devices just fail faster – that’s your trade-off.

It wont be fast

That is largely a feature of the disks and the controllers; you could get a better motherboard, disks and faster controllers, perhaps eschew the port multipliers too, if performance is a problem. A very cool new feature of ZFS (L2ARC / Hybrid Storage Pools) allows for using SSD as a second level cache, that would help. In linux dm-cache (or here) could probably achieve something similar.

How can you make it HA?

This is really another blog (and a few weeks work hacking out the ideas), but I can think of several ways of doing what BackBlaze do in their software stack to export files (via NFS, SMB or other protocol) or block devices (ATAoE, iSCSI, NBD etc) in a robust manner.

I have ordered some of the port multipliers, got my friend working on the case and will buy the sundry bits over the next few days.

This is one of my101 goals

Share on Facebook

ZFS, it’s sometimes good to know how screwed you are.

tom — Wed, 11 Feb 2009 21:30:02 +0000

I have just had a disk fail on my NAS, actually it happened ages ago but I was too broke to replace it. At the same time as one being faulted, another was degraded through having too many errors. Below is my interaction with ZFS to discover the extent of the problem and “fix” it.

(this is after I had swapped out c2t1d0, only then did I know I had data corruption, should have periodically scrubed the pool!)

Lets see how bad it is
root@nas:~# zpool status pool: Toms state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: resilver completed after 4h57m with 163802 errors on Sun Feb 8 16:47:46 2009 config


NAME        STATE     READ WRITE CKSUM

Toms        DEGRADED    35     0  320K

raidz1    DEGRADED    35     0  320K

c2t0d0  DEGRADED     0     0     0  too many errors

c2t2d0  DEGRADED    89     0     3  too many errors

c2t1d0  ONLINE       0     0     0

c2t4d0  DEGRADED    12     0   243  too many errors

raidz1    ONLINE       0     0     0

c2t5d0  ONLINE       0     0     0

c2t7d0  ONLINE       0     0     0

c2t3d0  ONLINE       0     0     0

c2t6d0  ONLINE       0     0     0
errors: 140338 data errors, use '-v' for a list
pool: syspool

state: ONLINE

scrub: none requested

config:
NAME        STATE     READ WRITE CKSUM

syspool     ONLINE       0     0     0

c1t1d0s0  ONLINE       0     0     0

errors: No known data errors

Ok, so what files are affected
root@nas:~# zpool status -v ..................(cut) /share/TalkingBooks/TTC/Concert Masterworks/CD10-Track07.mp3 /share/TalkingBooks/TTC/Science in the Twentieth Century A Social-Intellectual Survey/09-10 - Subterranean Fury.mp3 /share/TalkingBooks/TTC/Foundations of Western Civilization/45-2 - The Protestant Reformation - John Calvin.mp3 /share/TalkingBooks/TTC/The Long 19th Century, European History from 1789 to 1917/05.05 The Napoleonic Era, 1799-1815.mp3 /share/TalkingBooks/TTC/Great World Religions - Islam/Lecture 6d Paths To God -- Islamic Law And Mysticism.mp3 /share/TalkingBooks/TTC/History of Ancient Egypt/Lecture 23 The Murder of Tutankhamen.mp3 /share/TalkingBooks/TTC/The Symphony/00-02 Page 20-21.jpg /share/TalkingBooks/TTC/The United States And The Middle East 1914- 9-11/19-3 The First Palestinian Intifada.mp3 /share/TalkingBooks/TTC/World of Byzantium/TTC - World of Byzantium - Guidebook Part 2/[World.of.Byzantium.part.II]16-17.jpg /share/TalkingBooks/TTC/How to Read and Understand Poetry/13 Free Verse Track 2.mp3 /share/Tunes/Tom Jones - Reload/04 All Mine with Divine Comedy.mp3 /share/Tunes/Tom Jones - Reload/10 Sometimes We Cry with Van Morrison.mp3 /share/Tunes/Tom Jones - Reload/08 You Need Love Like I Do with Heather Small.mp3 /share/Tunes/Tom Jones - Reload/07 Sexbomb with Mousse T.mp3 /share/Tunes/Tom Jones - Reload/06 I'm Left, You're Right, She's Gone with James Dean Bradfield.mp3 ..................(cut)

Lets take a snapshot before we delete anything
root@nas:~# zfs list NAME USED AVAIL REFER MOUNTPOINT Toms 1.78T 421G 40.4K none Toms@one 0 - 40.4K - Toms/share 1.43T 421G 31.3G /share Toms/share/500G 336G 421G 336G /share/500G Toms/share/Notes 38.9G 421G 38.9G /share/Notes Toms/share/SetupFiles 72.1G 421G 72.1G /share/SetupFiles Toms/share/TalkingBooks 144G 421G 144G /share/TalkingBooks Toms/share/Tunes 61.5G 421G 61.5G /share/Tunes Toms/share/Videos 337G 421G 337G /share/Videos Toms/share/apps 28.4K 421G 28.4K /share/apps Toms/share/downloads 359G 421G 359G /share/downloads Toms/share/files 34.6G 421G 34.6G /share/files Toms/share/xbox 52.8G 421G 52.8G /share/xbox Toms/tom 7.59G 421G 7.59G /tom syspool 1.17G 225G 23K none syspool/rootfs-nmu-000 1.17G 225G 1.12G legacy syspool/rootfs-nmu-000@initial 49.8M - 631M

root@nas:~# zfs snapshot Toms@b4delete
root@nas:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
Toms 1.78T 421G 40.4K none
Toms@one 0 – 40.4K -
Toms@b4delete 0 – 40.4K -
Toms/share 1.43T 421G 31.3G /share
Toms/share/500G 336G 421G 336G /share/500G
Toms/share/Notes 38.9G 421G 38.9G /share/Notes
Toms/share/SetupFiles 72.1G 421G 72.1G /share/SetupFiles
Toms/share/TalkingBooks 144G 421G 144G /share/TalkingBooks
Toms/share/Tunes 61.5G 421G 61.5G /share/Tunes
Toms/share/Videos 337G 421G 337G /share/Videos
Toms/share/apps 28.4K 421G 28.4K /share/apps
Toms/share/downloads 359G 421G 359G /share/downloads
Toms/share/files 34.6G 421G 34.6G /share/files
Toms/share/xbox 52.8G 421G 52.8G /share/xbox
Toms/tom 7.59G 421G 7.59G /tom
Toms/vffs 349G 421G 349G /vffs
syspool 1.17G 225G 23K none
syspool/rootfs-nmu-000 1.17G 225G 1.12G legacy
syspool/rootfs-nmu-000@initial 49.8M – 631M -
Just how many file are effected?
root@nas:~# zpool status -v | grep " /share/" | wc -l 2618

Lets get rid (probably can be done neater, I had to google for the sed syntax to strip whitespace.)
root@nas:~# zpool status -v | grep " /share" | sed 's/^[ \t]*//;s/[ \t]*$//' | xargs -d "\n" rm root@nas:~# zpool status -v | grep " /share/" | wc -l 0
No bad files now

Clear the errors on the disks (hopefully they are not really ALL bad)
root@nas:~# zpool clear Toms root@nas:~# zpool status pool: Toms state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: resilver completed after 4h57m with 163802 errors on Sun Feb 8 16:47:46 2009 config:

NAME STATE READ WRITE CKSUM
Toms ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c2t2d0 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0
c2t4d0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c2t5d0 ONLINE 0 0 0
c2t7d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0
c2t6d0 ONLINE 0 0 0

errors: 140338 data errors, use ‘-v’ for a list

pool: syspool
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
syspool ONLINE 0 0 0
c1t1d0s0 ONLINE 0 0 0

errors: No known data errors
Still some errors, need to figure out what this means
root@nas:/share# zpool status -v ..................(cut) Toms/share/TalkingBooks:<0x61ff> Toms/share/TalkingBooks:<0x62ff> Toms/share/TalkingBooks:<0x64ff> Toms/share/TalkingBooks:<0x65ff> Toms/share/Tunes:<0x239> Toms/share/Tunes:<0x247> Toms/share/Tunes:<0x34e> Toms/share/Tunes:<0x47f> ..................(cut)

Share on Facebook

Databases and Lustre on ZFS’s DMU, New CIFS Stuff

tom — Thu, 03 Apr 2008 12:12:03 +0000

When I first heard about ZFS and its features, I was intrigued by a comment by Bill More about the possibility of having a database or other app directly consume the DMU that ZFS uses for filesystems or volumes. After I did a spot of research when editing the ZFS page on Wikipedia I noticed the “Last Word In Filesystems” pdf has been updated since I last looked, here are the 2 pages that excited me. With Suns recent acquisition of MySQL and lustre we seem to have arrived there now. Lustre support is excellent as it solves the only failing of ZFS, that it is not clustered.

The in-kernel CIFS stuff gets a mention too

Great work

See the full presentation and Bill Moore’s slightly outdated video

Share on Facebook

Two Thumpers For Tommy?

tom — Thu, 03 Apr 2008 12:01:14 +0000

After me banging on for a year about how cool ZFS is, my boss is finally convinced and wants to get a Thumper (or 2). Depending on budgetary constraints we may be getting a couple of these to implement a warm backup solution for our current data and about 5 years worth from now (they are 24Tb each)

48 Drives in a 4U rackmount for 20k, great value and unbeatable storage density.

Share on Facebook

OpenSolaris Gets In-Kernel CIFS Server

tom — Sun, 11 Nov 2007 22:08:26 +0000

Exporting a ZFS filesystem via NFS is easy, you just use the command

zfs set sharesmb=on POOLNAME/FSNAME

In order to share via CIFS (sometimes called SMB) to windows clients, you used to have to use Samba. This is not a problem for simple setups that dont use Active Directory, but can be a pain if you do as the ACL model in Windows is completely different to Unix (One of the few areas Windows is better in my opinion). Samba get around this by adding hooking up to OpenLDAP, adding a VFS layer (pdf) and reimplementing Kerboros/LDAP to match the Microsoft version, all of which are far from trivial to set up. I love Samba and think it is one of the best pieces of free software there is.

Sun have just released a CIFS implementation in the SunOS kernel, based on a stack that they got when they bought Procom. As well as serving up files via CIFS, they have modified the ACL system to be a sort of hybrid or old style Posix and Windows style ones. I think they have been planning this for a while as NFSv4 ACLs are similar to NTFS ones but still map on to Posix style eventually. So the end result is you will be able to write

zfs set sharecifs=on POOLNAME/FSNAME

and have the filesystem exported. The fact that it is in-kernel and integrated with ZFS is cool, but this on-disk storing of the ACL information is the best feature. Check out Mike Shapiro and Alan Wright‘s blogs on the topic, the full spec and the project page for more details.

That is not to say Samba is useless, I was very excited by the examples here, particularly the patch to implement shadow copies using ZFS, but maybe people will move away from it in favour of a more tightly integrated in-kernel implementation. The CIFS client in Linux is now in-kernel (still written by the Samba guys though) as it is in SunOS and Darwin (Sun ported the Darwin version). Untill NFSv4, Samba was definitely the best way to share files between Unixes, due to some nifty Posix Extensions and the throughput from the efficient implementation (beating in-kernel NFS from Linux in my testing, it may still be the best way). Samba beats Windows servers for throughput and number of clients served, the SMBTorture suite they developed even got mentioned (search for Evil Empire) by a Microsoft guy as they now use it to test their implementation.

When I get it on my NAS, I’ll post benchmarks.

Share on Facebook

Why I love ZFS

tom — Sun, 04 Nov 2007 20:59:08 +0000

No silent data corruption because of a checksum on each block, copy on write means no fsck EVER (it does not even have one), no RAID5 write hole, add disks to the pool on the fly, put volumes there as well as filesystems and export them via iSCSI or NFS, quotas, compression and it is easy to use. Check it out here, or watch this video

It is so easy to use, here is my command line session when adding 4 disks to my array. Took me perhaps 5 minutes and its worth noting that there was no pause between issuing each command as its all pretty much instant. I had to import the pool at the beginning as I had just installed OpenSolaris developer edition to see if it was any better than Nexenta (it was not). The spacing of the output is a little off, but hopefully its still understandable.

# zfs list no datasets available # zpool import Toms cannot import 'Toms': pool may be in use from other system use '-f' to import anyway # zpool import -f Toms # zfs list NAME USED AVAIL REFER MOUNTPOINT Toms 1.33T 9.84G 40.4K none Toms/500G 723G 9.84G 723G /share/500G Toms/Notes 25.9G 9.84G 25.9G /share/Notes Toms/files 25.6G 9.84G 25.6G /share/files Toms/share 532G 9.84G 46.4K /share Toms/share/SetupFiles 72.2G 9.84G 72.2G /share/SetupFiles Toms/share/TalkingBooks 139G 9.84G 139G /share/TalkingBooks Toms/share/Tunes 57.5G 9.84G 57.5G /share/Tunes Toms/share/Videos 263G 9.84G 263G /share/Videos Toms/share/apps 159M 9.84G 159M /share/apps Toms/share/downloads 26.9K 9.84G 26.9K /share/downloads Toms/xbox 50.7G 9.84G 50.7G /share/xbox # zpool status pool: Toms state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on older software versions. scrub: none requested config:

NAME STATE READ WRITE CKSUM
Toms ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c2t2d0 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0

errors: No known data errors

# zpool add Toms raidz c2t4d0 c2t5d0 c2t6d0 c2t7d0
invalid vdev specification
use ‘-f’ to override the following errors:
/dev/dsk/c2t4d0s0 is part of exported or potentially active ZFS pool
mypool. Please see zpool(1M).
# zpool add -f Toms raidz c2t4d0 c2t5d0 c2t6d0 c2t7d0
# zpool status
pool: Toms
state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using ‘zpool upgrade’. Once this is done, the
pool will no longer be accessible on older software versions.
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
Toms ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c2t2d0 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c2t4d0 ONLINE 0 0 0
c2t5d0 ONLINE 0 0 0
c2t6d0 ONLINE 0 0 0
c2t7d0 ONLINE 0 0 0

errors: No known data errors
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
Toms 1.33T 888G 40.4K none
Toms/500G 723G 888G 723G /share/500G
Toms/Notes 25.9G 888G 25.9G /share/Notes
Toms/files 25.6G 888G 25.6G /share/files
Toms/share 532G 888G 46.4K /share
Toms/share/SetupFiles 72.2G 888G 72.2G /share/SetupFiles
Toms/share/TalkingBooks 139G 888G 139G /share/TalkingBooks
Toms/share/Tunes 57.5G 888G 57.5G /share/Tunes
Toms/share/Videos 263G 888G 263G /share/Videos
Toms/share/apps 159M 888G 159M /share/apps
Toms/share/downloads 26.9K 888G 26.9K /share/downloads
Toms/xbox 50.7G 888G 50.7G /share/xbox
# zpool upgrade Toms
This system is currently running ZFS pool version 8.

Successfully upgraded ‘Toms’ from version 7 to version 8

Share on Facebook

How to get a solaris kernel and GNU userland

tom — Sat, 03 Nov 2007 17:43:43 +0000

I love the SunOS kernel. ZFS, Dtrace, Zones all make it desirable. But OpenSolaris is a code repository, not a distribution and you have to go elsewhere if you want a free distro. Nexenta looked the most promising, but then went quiet for a while (at least on their webpage) after shifting their focus to a smaller package set they called Nexenta Core Platform. I was very pleased to see an offer of collaboration from the Debian project leader in their forums and now Nexenta.com has released a new NAS package (not free though) and at the same time Suns “Project Nevada” seems to be bearing fruit after hiring Ian Murdock (the chap who started Debian)

OpenSolaris Developer/Community editions pale in comparision to Ubuntu for range of packages and usability, perhaps Nevada can sort it out. I am hopeful that Nexenta get it together and be the GNU/Solaris distro I really want (see here for some screencasts by Martin Man showing how cool it already is).

Share on Facebook