See the paper here

He says ‘Atomic blocks as we have introduced them so far are utterly inadequate to coordinate concurrent programs. They lack two key facilities: blocking and choice’ so I guess the implication is Clojures STM is inferior, any thoughts?

I had to use a constraint on a ref and try/catch to get the same effect (though I hate using exceptions for control flow it does seem to work)

I think a better solution might be had using watchers, how would you do it?
Any good links explaining differences in the STMs?

(ns santa.core)

(declare join-group)

(defn start-thread
  [fn]
  (.start
   (Thread. fn)))

(defn random-delay []
  (Thread/sleep (rand-int 1000)))

(def printer (agent 0))

(defn threadsafe-print
  [somestring]
  (send printer println somestring))

(defn make-gate
  [max-size]
  (ref {:max-size max-size
        :remaining 0}
       :validator #(>= (:remaining %) 0)))

(defn pass-gate
  [gate]
  (try 
    (dosync (alter gate #(update-in % [:remaining] dec)))
    (catch IllegalStateException e
      (do ;(Thread/sleep 20)
          (pass-gate gate)))))

(defn helper
  [group task]
  (let [[in-gate out-gate] (join-group group)]
    (do (pass-gate in-gate)
        (task)
        (pass-gate out-gate)
        )))

(defn thread [group id work-fn]
  (start-thread
   (fn []
     (random-delay)
     (helper group #(work-fn id))
     (recur))))

(defn deliver-toys [id]
  (threadsafe-print
   (str "Reindeer " id " delivering toys")))

(defn meet-in-study [id]
  (threadsafe-print
   (str "Elf " id " meeting in the study")))

(defn reindeer-thread [group id]
  (thread group id deliver-toys))

(defn elf-thread [group id]
  (thread group id meet-in-study))

(defn operate-gate
  [gate]
  (dosync (alter gate assoc :remaining (:max-size @gate)))
  (while (> (:remaining @gate) 0)
    ;(Thread/sleep 20)
    ))

(defn new-group
  [max-size]
  (ref {:max-size max-size
        :remaining max-size
        :in-gate (make-gate max-size)
        :out-gate (make-gate max-size)
        }
       :validator #(>= (:remaining %) 0)))

(defn join-group                  
  [group]
  (try 
    (dosync (let [{:keys [remaining max-size in-gate out-gate]} @group]
              (alter group #(update-in % [:remaining] dec))
              [in-gate out-gate]))
    (catch IllegalStateException e
      (do ;(Thread/sleep 20)
          (join-group group)))))


(def elf-group (new-group 3))
(def reindeer-group (new-group 9))

(defn handle-group [group group-name]
  (let [current-group @group
        {:keys [max-size remaining in-gate out-gate]} current-group]
    (if (= 0 remaining)
      (do (dosync
           (ref-set group {:max-size max-size
                           :remaining max-size
                           :in-gate (make-gate max-size)
                           :out-gate (make-gate max-size)}))
          (threadsafe-print (str "***** With" group-name "*****"))
          (operate-gate in-gate)
          (operate-gate out-gate)
          true))))

(defn santa []
  (while (handle-group elf-group "Elves"))
  (handle-group reindeer-group "Reindeer")
  (recur))

(defn -main
  [& args]
  (dotimes [i 9]
    (do (println "Launching reindeer thread:" i)
        (reindeer-thread reindeer-group i)))
  (dotimes [i 10]
    (do (println "Launching elf thread" i)
        (elf-thread elf-group i)))
  (santa))

Share on Facebook

Qualification Round
For the qualifier I was in Leeds so didn’t have much time to look at it but solved the first in a few mins as I had most of the ‘plumbing’ (ie taking their input and spitting out a file in the correct format) from last year.

Given a string s, little Johnny defined the beauty of the string as the sum of the beauty of the letters in it.
The beauty of each letter is an integer between 1 and 26, inclusive, and no two letters have the same beauty. Johnny doesn’t care about whether letters are uppercase or lowercase, so that doesn’t affect the beauty of a letter. (Uppercase ‘F’ is exactly as beautiful as lowercase ‘f’, for example.)
You’re a student writing a report on the youth of this famous hacker. You found the string that Johnny considered most beautiful. What is the maximum possible beauty of this string?

(def letters
  (set "abcdefghijklmnopqrstuvwxyz"))

;; If you don't know clojure it might be interesting that sets are functions of their elements
;; and return either nil or the element, so can be used as predicates 

(defn counts [s]
  (frequencies (filter letters (str/lower-case s))))

;; Here we filter the string for letters we care about (after lowercasing it)
;; and create a map of the characters to there count in the string

;; eg (counts "Good luck in the Facebook Hacker Cup this year!")
;; {\a 3, \b 1, \c 4, \d 1, \e 4, \f 1, \g 1, \h 3, \i 2, \k 3, \l 1, \n 1, \o 4, \p 1, \r 2, \s 1, \t 2, \u 2, \y 1}

(defn beauty [s]
  (reduce + (map *
                 (reverse (sort (vals (counts s))))
                 (iterate dec 26))))

;; To make the score as big as possible you want the most frequent letter to be worth 26
;; then each one descending by frequency is work 25, 24 etc
;; (reverse (sort (vals (counts s)))) it the frequencies in descending order
;; (iterate dec 26) is 26, 25, 24 etc
;; we multiply each and then add them

Round 1
I had the evening cleared from about 8pm (it launched 6PM GMT, so I probably lost before I started anyway after seeing how rapidly others did them!)

You are given an array a with N ≤ 10 000 different integer numbers and a number, K, where 1 ≤ K ≤ N. For all possible subsets of a of size K find the sum of their maximal elements modulo 1 000 000 007.

Easy! 1-liner in clojure.

(defn sum-max-of-subsets [cards k]
  (reduce + (map #(apply max %) (comb/combinations cards k))))

My basic idea to make it faster was using a bit of combinatorics. If the deck is [1,5,7,11...] (it’s not a coincidence I ordered them now) with a hand size (k) of 3 then 1 and 5 will never be the max, 7 will be in the hand [1,5,7] and 11 will be the max of [1,5,11] [1,7,11] and [5,7,11], in general the nth card will be maximum “n-1 choose k” times, so the problem becomes easier, loop thorough the elements finding out how many times it would be the maximum. See here if that does not make sense.

(def factorial
  (memoize (fn [n]
             (reduce *' (range 1 (inc n))))))

(defn nCks [k]
  (let [fac (factorial k)]
    (map #(/ % fac)
         (reductions (fn [acc n]
                       (* acc (/ n (- n k))))
                     fac
                     (iterate inc (inc k))))))

(defn sum-max-of-subsets [cards k]
  (let [sorted (drop (dec k) (sort cards))
        nck (nCks (dec k))]
    (mod (reduce +'
                 (map * sorted nck))
         1000000007)))

(defn bigparse [s]
  (bigint (Integer. s)))

(defn parse-input [line1 line2]
  (let [k (bigparse (second (str/split line1 #"\s+")))
        cards (map bigparse (str/split line2 #"\s+"))
        ]
    (sum-max-of-subsets cards k)))

For the other problem I groped around the next day trying to solve it, thought ‘this looks perfect for core.logic’ then proceeded to procrastinate and watch The Shield and eat bacon. I am OK with this.

Code is of course on github

Share on Facebook

$ poop
Sorry, command-not-found has crashed! Please file a bug report at 

https://bugs.launchpad.net/command-not-found/+filebug

Please include the following information with the report 
command-not-found version: 0.2.44

It turns out that it was to do with locale settings, I fixed it by

$ sudo locale-gen en_GB.UTF-8
$ sudo update-locale LANG=en_GB.UTF-8

Share on Facebook

As an example, below is my soln to #73 using it, I can run it with
# lein run -m offline-4clojure.p73

; Analyze a Tic-Tac-Toe Board - Hard
; A <a href="http://en.wikipedia.org/wiki/Tic-tac-toe">tic-tac-toe</a> 
; board is represented by a two dimensional vector. 
; X is represented by :x, O is represented by :o, and empty is represented by :e.  
; A player wins by placing three Xs or three Os in a horizontal, vertical, or diagonal row.  
; Write a function which analyzes a tic-tac-toe board and returns :x if X has won, :o if O has won, and nil if neither player has won.
; tags - game
; restricted - 
(ns offline-4clojure.p73
  (:use clojure.test))

(def __
(fn [board]
  (let [triples
        (fn [board]
          (map #(map (partial get-in board) %)
               (concat
                (for [x [0 1 2]]
                  [[x 0] [x 1] [x 2]])
                (for [y [0 1 2]]
                  [[0 y] [1 y] [2 y]]) 
                [[[0 0] [1 1] [2 2]]
                 [[0 2] [1 1] [2 0]]])))]
    (ffirst (filter #(or (= [:o :o :o] %)
                         (= [:x :x :x] %))
                    (triples board)))))

)

(defn -main []
  (are [x] x
(= nil (__ [[:e :e :e]
            [:e :e :e]
            [:e :e :e]]))
(= :x (__ [[:x :e :o]
           [:x :e :e]
           [:x :e :o]]))
(= :o (__ [[:e :x :e]
           [:o :o :o]
           [:x :e :x]]))
(= nil (__ [[:x :e :o]
            [:x :x :e]
            [:o :x :o]]))
(= :x (__ [[:x :e :e]
           [:o :x :e]
           [:o :e :x]]))
(= :o (__ [[:x :e :o]
           [:x :o :e]
           [:o :e :x]]))
(= nil (__ [[:x :o :x]
            [:x :o :x]
            [:o :x :o]]))))

Share on Facebook

Firstly, about the 101 goals in 1001 days. I am so glad I did it, just the act of thinking about what you want to achieve is worthwhile, putting a time limit on it so it’s not a bucket list for someday but a list of things you are seriously going to try achieve in a few years. The 1001 days is interesting, short enough to feel some pressure but long enough to have a few of each season for anything that is time sensitive.
I think I was a bit ambitious with my list, though I always knew it was a reach I probably bit off more than I could chew. That said the list was formed when I was single and freelance and I would not swap either the last few years at my job (still at Forward) or with Petra for a few more goals ticked.

Before I go through them all I just want to say the most transformative thing of the whole 1001 days has been this year, in particular running and the impact it has had on my fitness.
When I talked about the 101 goals lots of people got excited but everyone kind of did a double take when they saw I was doing a marathon, I think they did not believe I could (and I can’t blame them, I’ve never been super-fit and was in my worst shape ever). I started in Jan with the NHS Couch to 5k and then an excellent training schedule provided by Oxfam as I was entered into Edinburgh for them in May. I trained really hard for it as you can tell from the runkeeper stats, 88,365 calories – 592.2 miles and active for nearly 6 days in total!

The purple in May is when I started biking after not having a bike for 15 years or so and unfortunately I came off it a few days before the marathon and broke my arm and dislocated my shoulder. I was so ready and had raised so much for Oxfam that I thought I had to do it so (after being dissuaded from duct taping my arm still) I decided to walk it, and finished just in time for medals!

So that went well but I was unable to train for quite a while and missed the chance of doing a triathlon this year.

I used my new fitness level to try and climb every 3000ft mountain in Wales in a continuous walk on a weekend when a months worth of rain fell in a day and we had 40mph winds, managed 11 of them.

After going on a safari in Kenya with Petra, my sister Carrie and my Mum (which Carrie captures perfectly here I met my friend Colin (who I have known since the first day of primary school, actually not the only friend I can say that of I am lucky to have a few) in tanzania and we climbed Kilimanjaro. It was the hardest thing I have ever done, but not in a satisfying way, the typical kili trip does not allow enough time to acclimatize and your resistance to altitude is largely genetic and it’s quite a dirty and busy mountain. Glad I did it? Yeah, proved I have some serious resolve. Would I recommend? No, there are nicer challenges that reward preperation more and are cleaner and more beautiful. Maybe I should say more but a portered trip walking only 5h a day and going too high too quickly does not compare to the Alps or even UK hill days. Maybe it’s still too soon and retrospective pleasure has not kicked in yet but I am sure I’ll stand by it.

The week after I did the Liverpool marathon too, basically I thought it was a toss-up between the red blood cells from kili and a twisted ankle how well I would do but in the end I had just not got enough long runs in and my quads packed in at 22 miles and I needed to walk a bit.
Me and Carrie, who apparantly I inspired to enter the marathon (though she beat me by 15 mins!)



The founding members of the Bread Wine and Cheese Athletics association crossing the line together then finishing the goodies we took along with us.

I’m still raising money at justgiving.com/tommys-final-push for WarChild. All money donated is doubled by Forward next week so every penny counts twice!

So, smashing it this year but how did I do overall? Dayzero has me on 42 complete. I am quite pleased but have decided to extend to the end of the year and try and tick a few more off, not least so I can have a finishing party. My goals, my rules after all

I will be doing another 101 goals, building up to swimming the Helespont and doing an ironman (quite a reach as I can’t swim at the moment!)

What would I change? Blog more about it, involve more people, have as much drive as I had this year for the whole period.

I am really please to have encouraged half a dozen others to start lists.

Done (42)
1, Teetotalitarianism for 3 months
2, Cheeseless for 3 months
9, Read GEB
11, Reread all Dennett books
15, Proofread for Project Gutenberg
48, Create a Backblaze storage pod
53, Make Jam
66, Via Ferrata in Italy
78, Learn to use Emacs
82, Visit Egypt
83, Re-visit Louvre
85, Visit Pergamon Museum
86, Give Carrie a British Museum Tour
92, Read “An Ode Less Travelled“, do the exercises (but not share them!)
97, Be 1/3 through in 2010
100, Set success criteria / progression metrics for each goal
5, Lose 2 stone
10, Write book reviews for each book I read\
76, Do on average 1 Project Euler problem per week
88, Go to the theatre on average once a month
101, Do 100 day updates
44, Visit The Uffizi in Florence (was Get CCEE)
61, Do a UK long distance path
3, Do a marathon
21, Read GTD
24, Swim with sharks
73, Make a Dots and Boxes program
79, Raise £5005 for charity
39, Take Mum, Dad and Carrie to the Welsh Mountain Zoo
56, Make beer
51, Investigate Visa situation for Australia
52, Investigate Visa situation for US
54, Grow mushrooms
32, Bungee Jump
34, Vineyard tour
96, Let loans run course and don’t get any more
64, Climb a continental highest mountain
33, Safari
20, Organise a big bash for my 30th
84, Revisit Met Museum
28, Drive Off Road
31, Hire the whole of Salvos Salumeria for an evening

Attempting before Christmas (23)
13, Release 303 books on bookcrossing.com
68, Complete Pimsleur German
72, Read “Winning Ways”
74, Read AI: A Modern Approach
75, Watch SICP, do exercises from book
77, Complete “Real World Haskell”
91, Memorise 10 poems
46, Listen to Radio 4 / British Museum – A History of the World in 100 Objects and view each of them (was Get VCAP)
89, Return to the Theatre by the lake
7, Write an article for Plus new writers
14, Read a short story for librivox
16, Send Dennett a letter
17, Send Dawkins a letter
18, Read Joyce
23, Organise all my DVDs
38, Take dad to an opera
58, Cook a 4 course meal for 20 friends
55, Paint a watercolor
57, Make wine
71, Learn 10 magic tricks
80, Talk about Free Software at a school
87, Go on wine tasting course
99, Have a completion party

Abandoned / Failed (36)
67, Do another alpine 4000m peak
19, Blog on average once a week
50, Move 10 people to FreeAgent
95, Pay off all credit cards
8, Read all the VSIs
12, Read all PG Wodehouse
81, Watch all TTC Art history DVDs
90, See all world heritage sites in the UK
43, Visit the rijksmuseum (was Get CCNP)
45, Give blood every 20 weeks (was Get MCITP – Enterprise Admin)
47, Make a Munro bagging site in Rails (was Say to a recruiter “I dont work ” and turn down work)
60, Hike on average once a month
62, Do a big hike in Europe
35, Visit 5 Michelin 3* restaurants
37, Visit porto
94, Go to Edinburgh festival
4, Do a triathlon
6, Attend martial arts classes for 3 months
22, Spend 3 months in another country
25, Paraglide
26, Learn to play bongos
27, Skydive
29, Do a banger rally
30, Have a track day
36, See Northern Lights
40, Do 1000 things in London
41, Do a standup comedy course
42, Visit Japan
49, Work only 100 days in a year
59, Do a photography course
63, Attend NIM
69, Learn to dance
70, Learn to play golf
93, Go to Melbourne Comedy Festival
65, Volunteer for the mountain bothies association
98, Have done 2/3 by day 666

Share on Facebook

We stayed at the Wynn again.

Beautiful view from my room

On the first day we went carting.

The final day I went to do a skyjump off the Stratosphere (have a DVD I’ll upload when I find it)

Between that some drinking and gambling…

Random head coming out of the lake in the Wynn

Then home!

Share on Facebook

84 – Revisit Met Museum
Went while I was at Hadoop World (pic is actually the Natural History museum but what the hell)

During my recent Africa Trip:
24 – Swim with sharks

33 – Safari
Visited Addo Elephant Park during my recent South Africa Trip

34 – Vinyard tour
I visited the boekenhoutskloof vinyard in Franschhoek South Africa

I’ll do a full write-up of the Africa Trip soon, it was amazing.

So now out of the 101 I have done 24 with 18 on track and only 300 days to go! Need to get a shift on and tick off as much as I can.

The dayzeroproject site is back up, I am on there as thattommyhall

Share on Facebook

The archive is available and I thought I would share my highlights:

• Vilayanur Ramachandran: The Emerging Mind was the first one I ever listened to, great peek into how the brain works
• Edward Said: The representation of the Intellectual
• John Searle: Minds Brains And Science. I dont agree with Searle but find him very engaging and thought provoking.
• Bertrand Russell: Authority and the Individual. The first Reith lecture was by a personal hero.

Share on Facebook

(picture from Wikipedia)

We did it ultralight, using kit I have blogged about before. My mate Ben got well into expedition planning mode and prepared an optimal food mix for the trip and introduced us to SCROGIN (Sultanas Chocolate Raisons Orange Ginger Imagination Nuts) and ANZAC biscuits (his lovely other half is a kiwi).

I was pleased to fit it all in a 30L sack, made the walking much easier than it might have been.

As I had just been in Lisbon for a stag do the weekend before I was not feeling 100% when we got the sleeper to Inverness on the Monday night but we arrived somewhat fresh and started walking immediatly. The sleeper is really nice and I would deffinatly recommend it over flying if you need an early start in Scotland, see ScotRail. By the end of Tuesday we had got most of the way to Invermoriston (nearly 30 miles) but were all exhausted. We wildcamped with some stunning views.

The Wednesday we walked to Fort Augustus and decided to take a B&B for the night as non of us had slept well and our legs and feet were killing. We were fortunate enough to stay at Old Pier House which was lovely and we got moving again on the Thursday with much more enthusiasm than we ended the day before.

Thursday night we got past laggan and camped at a campsite on the north of Loch Lochy.

Friday was an epic day, taking in the 2 munros ( Meall na Teanga and Sròn a’ Choire Ghairbh and walking about 25 miles then (we thought) finishing the walk.

We had actually just reached Neptune’s Staircase and we wound up bivvying at the start line of Maggies Monster Bike and Hike. We must have looked quite odd…

We spent the first few hours of the Saturday finishing it off and arriving at Fort William where we ate the biggest amount of food we could.

A great hike with 2 great guys and as it is a UK long distance path it is another of my 101 goals in 1001 days days ticked off

Share on Facebook

As all the talks will be online soon I will just mention a few things that I enjoyed.

The two keynotes were excellent, Doug Cutting on the history of Hadoop and Ted Dunning on the future. Both were very interesting and had a great feel for the community aspect of Open Source software. Ted works for MapR technologies but the talk was not a sales pitch. Ted spoke about how Hadoop fails currently to get the most out of the components and what we might get if we could. MapR are used by EMC for their new Hadoop distro, among other things I think they have reimplemented HDFS. An interesting number of companies had got some pretty big amounts of funding to build front-ends to Hadoop, DataMeer have an excel-like web frontend that looks interesting.

Talks I enjoyed were:

NODE.JS FOR HEAVY I/O
A superb intro to Node.js, with an example small enough to fit on a slide but not completely trivial.

TIME SERIES OR CAUSAL ANALYSIS WITHOUT LIMITS!
Shivek was awesome, engaging and enthusiastic. The topic itself was fascinating, using
Pi Calculus to reason about and design map/reduce algorithms. He made the point that most Hadoop jobs are datacentric but showed how to do some more mathscentric algorithms like FFTs

OH LEONHARD, WHERE ART THOU?
Jim Webber on graph databases in general and Neo4J in particular. Quite a nice reference to Euler in the title. If your data is a graph, why not have a database that is too?

REALTIME BIG DATA AT FACEBOOK WITH HADOOP AND HBASE
From Jonathan Gray, this talk was really interesting – amazing the throughput they are getting from HBase. I think Forward are more like Facebook than Google (more freedom within teams, choice of tech/roll your own vs Google wanting everything on BigTable. I cringed a bit at the thought of loads of servers running random C++ apps all over the place though…)

NEWER DEVELOPMENTS IN LARGE DATA TECHNIQUES
Joseph Turian from MetaOptimise gave a great overview of recent academic work on Machine Learning and Natural LAnguage Processing, buzzwords to look out for are: Deep Learning, Semantic Hashing and Semantic Parsing. Also look at GraphLab, Machine Learning on graph databases

DIGITISED DUTCH CULTURAL HERITAGE, MAHOUT & HADOOP
COMPOSING MAHOUT CLUSTERING JOBS
Two good talks on using Mahout, the first is on a Dutch Gov project, Images for the future to archive and categorise AV heritage resources. The second had a nice demo of categorising stack-overflow.

Lightning Talks:
The Lustre filesystem from Eric Barton of Whamcloud talked about how his company are developing Lustre outside Sun/Oracle and he was trying to see where it could fit in with Hadoop. Luster is the other end of the spectrum from HDFS/Hadoop, really quick but assuming fast, highly available storage behind it. I would love to see some integration with Lustre or Ceph in a Hadoop-like system.

I gave a talk on the Flume Firehose Abs and I made at Forward last week, it was OK (though I still think no-one has done a good job of selling ZeroMQ in 10 minutes!). Slides are here (I’ll do another post about it as well, quite an entertaining fallout from it over twitter.)

Share on Facebook

The wiki page HiveCompressedStorage lists the possibilities.

Basically you have 3 decisions:
TextFile or SequenceFile tables
TextFile

• Can be compressed in place.
• Can gzip/bzip before you LOAD DATA into your table
• Only gzip/bzip are supported
• Gzip is not splitable

SequenceFile

• Need to create a SequenceFile table and do a SELECT/INSERT into it
• Can use any supported compression codec
• All compression codecs are splitable. All the cool kids use LZO or Snappy
• Does not work- At least for me (help appreciated!)

Which compression algorithm

• gzip – Quite slow, good compression, not splitable, supported in TextFile table
• bzip – Slowest, best compression, splitable, supported in TextFile table
• LZO – Not in standard distro (licensing issues), fast, splitable
• Snappy – New from google, Not in standard distro (but licence compatable), Very fast

Block or Record compression (for SequenceFile tables)
The docs say

The value for io.seqfile.compression.type determines how the compression is performed. If you set it to RECORD you will get as many output files as the number of map/reduce jobs. If you set it to BLOCK, you will get as many output files as there were input files. There is a tradeoff involved here — large number of output files => more parellel map jobs => lower compression ratio.

But I got the same number of files regardless of what I selected and the total size suggested they were not even compressed so I dont know what is going on.

For simplicity I chose gziped TextFile tables because

• It worked (always criteria zero)
• Most of our files were not huge anyway and the technique described below keeps some of the parallelism
• Can be done on the table in place
• Each partition can be compressed separately
• The space is saved incrementally and realised immediately
• Testing showed for our load it was not much of a performance hit
• We are feeling more pain on space than query performance at the moment, our hourly runs complete in ~20mins)

require 'rubygems'
require 'date'
require 'rbhive'

countrys = %w[at au br de dk es fr in it jp mx nl no pl pt ru se uk us za]
dates = (Date.parse('2011-01-01')..Date.parse('2011-04-30'))

RBHive.connect('hiveserver') do |con|
  dates.each do |date|
    countrys.each do |country|
      query = "insert overwrite table keywords partition (dated='#{date}', country = '#{country}')
              select account,campaign,ad_group,keyword_id,keyword,match_type,status,
              first_page_bid,quality_score,distribution,max_cpc,destination_url,ad_group_status,
              campaign_status,currency_code,impressions,clicks,ctr,cpc,
              cost,avg_position,account_id,campaign_id,adgroup_id 
              from keywords where dated='#{date}' and country='#{country}'"
      begin
        con.set('mapred.output.compression.codec','org.apache.hadoop.io.compress.GzipCodec')
        con.set('hive.exec.compress.output','true')
        con.set('mapred.output.compress','true')
        con.set('mapred.compress.map.output','true')
        con.set('hive.merge.mapredfiles','true')
        con.set('hive.merge.mapfiles','true')
        con.execute(query)
      rescue => e
        puts "#########################"
        puts e.message
        puts "#########################"
      end
    end
  end 
end 

You can actually download and compress the data directly to HDFS as Hive does not know what data is inside the folders on HDFS, just their layout but I thought better to do it via hive and let Hadoop parallelise it. I would have carried on doing it this way but with other tables it was too slow (too many partitions, difficult to parallelise hive server). I stopped using rbhive, dropped to using hive -e to execute the querys and used the lovely autopartitioning in later hive versions. Notice you can SELECT * now and it automatically does what it needs to to insert results into the correct partitions.

require 'rubygems'
require 'date'

countrys = %w[at au br de dk es fr in int it jp kr mx nl no pl pt ru se uk us za]

dates = (Date.parse('2010-12-02')..Date.parse('2011-05-01'))

dates.each do |date|
  query = ""
  query += "SET hive.exec.compress.output=true;"
  query += "SET mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;"
  query += "set mapred.job.priority=VERY_LOW;" 
  query += "set hive.exec.dynamic.partition=true;"
  query += "set mapred.output.compress=true;"
  query += "set mapred.compress.map.output=true;"
  query += "set hive.merge.mapredfiles=true;"
  query += "set hive.merge.mapfiles=true;"
  query += "insert overwrite table hourly_clicks 
            partition (dated='#{date}', country, hour) 
            select * from hourly_clicks where dated='#{date}'"
  query = "hive -e \"#{query}\""
  puts "running #{query}"
  `#{query}`
end

The reason I am not (very) worried about losing parallelism is that some of our partition contained big .csv’s and the output of INSERT OVERWRITE was multiple .gz files (looked to me like as many as there were mappers, for example a 700M text file became ~10 .gz files) so they will still be read in parallel by mappers as the original CSV was.

Open to suggestions about better ways to achieve this, this does not preclude doing something better later.

Share on Facebook

current = ''
file_count = 0
total_size = 0

output = File.open('output.csv','w')

IO.popen('hadoop fs -lsr /user/hive/warehouse').each_line do |line|
  split = line.split(/\s+/)
  #permissions,replication,user,group,size,mod_date,mod_time,path
  next unless split.size == 8
  path = split[7]
  size = split[4]
  permissions = split[0]
  tablename=path.split('/')[4]
  if tablename != current
    average_size = file_count == 0 ? 0 : total_size/file_count
    result = "#{current},#{file_count},#{total_size},#{average_size}"
    unless current==''
      puts result
      output.puts result
    end
    total_size = 0
    current = tablename
    file_count = 0
  end
  file_count += 1 unless permissions[0] == 'd'
  total_size += size.to_i
end

Lots of our files were small so I am going to experiment with different partitioning and compression schemes.

Share on Facebook

old lock file: /var/lib/mongodb/mongod.lock. probably means unclean shutdown
recommend removing file and running –repair
see: http://dochub.mongodb.org/core/repair for more information

As the docs do not seem to have much of an alternative to running –repair I looked for a way to automate it from upstart. Mongo creates a mongod.lock file in the data directory with the processes PID in and on a safe shutdown removes the PID, leaving the file there.

This upstart scripts includes a pre-start script that checks if the lock file exists, reads it, makes sure there is a PID there, makes sure no mongod processes exist with that PID then performs the repair as the mongodb user.

limit nofile 20000 20000

kill timeout 300

env MONGO_DATA=/var/lib/mongodb/
env MONGO_LOGS=/var/log/mongodb/
env MONGO_EXE=/usr/bin/mongod
env MONGO_CONF=/etc/mongodb.conf

pre-start script
  mkdir -p $MONGO_DATA
  mkdir -p $MONGO_LOGS
  if [ -f $MONGO_DATA/mongod.lock ]; then
    mongo_pid=`cat $MONGO_DATA/mongod.lock`
    if [ ! -z $mongo_pid ]; then
      if [ ! `pgrep mongo | grep "$mongo_pid" | wc -l` -gt 0 ]; then
        rm $MONGO_DATA/mongod.lock
        sudo -u mongodb /usr/bin/mongod --config /etc/mongodb.conf --repair
        touch $MONGO_DATA/repaired-`date "+%Y%m%d-%H%M%S"`
      fi
    fi
  fi  
end script

start on runlevel [2345]
stop on runlevel [06]

script
  if [ -f /etc/default/mongodb ]; then . /etc/default/mongodb; fi
  exec start-stop-daemon --start --quiet --chuid mongodb --exec  $MONGO_EXE -- --config $MONGO_CONF
end script

Share on Facebook

I borrowed ideas from John Leach’s excellent Ruby: Reinventing the Wheel talk, this DepOps: The War Is Over presentation and rambled incoherently about a talk I just saw at the UKUUG Spring Conference from the author of cfengine, see here a nice description of the project (you can see how it has influenced Puppet)

Here are the slides (first time I have used Scribd, it is excellent. Much better than slideshare)
DevOps

I like the James White Manifesto , it chimes really strongly with me.

In particular

On Infrastructure
—————–
There is one system, not a collection of systems.
The desired state of the system should be a known quantity.
The “known quantity” must be machine parseable.
The actual state of the system must self-correct to the desired state.
The only authoritative source for the actual state of the system is the system.
The entire system must be deployable using source media and text files.

Soon they will post videos and I will get to see myself give a talk for the first time.

Share on Facebook

Completed – 16
1, Teetotalitarianism for 3 months
2, Cheeseless for 3 months
9, Read GEB
11, Reread all Dennett books
15, Proofread for Project Guttenburg
48, Create a Backblaze storage pod
53, Make Jam
66, Via Feratta in Italy
78, Learn to use Emacs
I suppose you can never fully learn it but I do use it for my development now
82, Visit Egypt
83, Re-visit Louvre
85, Visit Pergamon Museum
86, Give Carrie a British Museum Tour
92, Read “An Ode Less Travelled“, do the exercises (but not share them!)
Read it while in Egypt.
97, Be 1/3 through in 2010
100, Set success criteria / progression metrics for each goal

On Track – 16
5, Lose 2 stone
10, Write book reviews for each book I read
Where I havent yet I have added a task to rememberthemilk to do so
13, Release 303 books on bookcrossing.com
88 available here, let me know if you like any and I will post them to you.
19,Blog on average once a week
50, Move 10 people to FreeAgent
68, Complete Pimsleur German
Changed from Spanish as I now live with a lovely German lady.
72, Read “Winning Ways”
read 1/2 of part 1 (of 4)
74, Read AI: A Modern Approach
75, Watch SICP, do exercises from book
Started a book club in work, seems to have stalled but I’ll start banging the drum again now I’ve settled in my new house.
76, Do on average 1 Project Euler problem per week
77, Complete “Real World Haskell”
88, Go to the theatre on average once a month
Way ahead on this, started a monthly theatre club but we managed to schedule a dozen things for the first few months of 2011
91, Memorise 10 poems
Not quite settled on the 10 but between listening to Jorge Louis Borges, This Craft Of Verse and The Ode Less Travelled I have quite a list to choose from.
95, Pay off all credit cards
96, Let loans run course and dont get any more
101, Do 100 day updates
This is one right

Behind – 4
8, Read all the VSIs
12, Read all PG Wodehouse
81, Watch all TTC Art history DVDs
90, See all world heritage sites in the UK

Changing – 5
Lots of the work related ones dont make sense any more now that I have gone full time and moved into development so I am making the following changes.
43, Visit the rijksmuseum (was Get CCNP)
44, Visit The Uffizi in Florence (was Get CCEE)
45, Give blood every 20 weeks (was Get MCITP – Enterprise Admin)
46, Listen to Radio 4 / British Museum – A History of the World in 100 Objects and view each of them (was Get VCAP)
The above are all taken from a mate who just did his own 101 list.
47, Make a Munro bagging site in Rails (was Say to a recruiter “I dont work ” and turn down work)

Planning – 12
60, Hike on average once a month
61, Do a UK long distance path
67, Do another alpine 4000m peak
62, Do a big hike in Europe
64, Climb a continental highest mountain
33, Safari
20, Organise a big bash for my 30th
The fitness aspect of these goals is where I am behind the most (though I am still a stone lighter than when I started) so I am concentrating the next six months on these goals, ending with summiting kilimanjaro for my 30th then returning to a big party.
35, Visit 5 Michelin 3* restaurants
37, Visit porto
Will go with Petra in the spring
84, Revisit Met Museum
A good mate has just moved to NYC so this should happen as soon as he is settled.
89, Return to the Theatre by the lake
My first trip with Petra was to here and we loved it. Will be going in the spring.
94, Go to Edinburgh festival
Will go at the beginning of August.

Not Started – 48
3, Do a marathon
4, Do a triathlon
6, Attend martial arts classes for 3 months
7, Write an artice for Plus new writers
14, Read a short story for librivox
16, Send Dennett a letter
17, Send Dawkins a letter
18, Read Joyce
21, Read GTD
22, Spend 3 months in another country
23, Organise all my DVDs
24, Swim with sharks
25, Paraglide
26, Learn to play bongos
27, Skydive
28, Drive Offroad
29, Do a banger rally
30, Have a track day
31, Hire the whole of Salvos Salumeria for an evening
32, Bungee Jump
34, Vinyard tour
36, See Northern Lights
38, Take dad to an opera
39, Take Mum, Dad and Carrie to the Welsh Mountain Zoo
40, Do 1000 things in London
41, Do a standup comedy course
42, Visit Japan
49, Work only 100 days in a year
51, Investigate Visa situation for Australia
52, Investigate Visa situation for US
54, Grow mushrooms
55, Paint a water colour
56, Make beer
57, Make wine
58, Cook a 4 course meal for 20 friends
59, Do a photography course
63, Attend NIM
69, Learn to dance
70, Learn to play golf
71, Learn 10 magic tricks
73, Make a Dots and Boxes program
79, Raise £5005 for charity
80, Talk about Free Software at a school
87, Go on wine tasting course
93, Go to Melbourne Comedy Festival
99, Have a completion party
65, Volunteer for the mountain bothys association
98, Have done 2/3 by day 666

I am quite heartened by the progress to be honest, considering that I spent half of last year working outside the UK, now things are settling down I should be able to churn through them faster.

If you want to join in on some, let me know!

Share on Facebook

Original MRI on Linux

  def build
    pid = fork { run_command }
    Process.detach(pid)
  end

  def run_command
    `sudo /opt/script/deployserver/setupnewserver.sh -p #{poolserver} -i #{ip} -s #{@size} -v #{@vlan} -a "#{@owner}" -n #{@name} -e "#{@email}"`
  end

IronRuby
We tried IronRuby and the same bit of the script broke as on win32 MRI (though I was pleased and surprised that Sinatra worked)

  def build
    WindowsProcess.start "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe", 
"-PSConsoleFile \"C:\\Program Files (x86)\\VMware\\Infrastructure\\vSphere PowerCLI\\vim.psc1\" \"& C:\\script\\DataStoreUsage.ps1\""
  end

Using the following DotNet code

class WindowsProcess
  def self.start(file, arguments)
    process = System::Diagnostics::Process.new
    process.StartInfo.FileName = file
    process.StartInfo.CreateNoWindow = true
    process.StartInfo.Arguments = arguments
    process.Start
  end
end

Workaround using Windows “start” command
I had hoped the module at win32utils would let me just use the original script but fork did not work properly still.

  
def build
  commandstr = "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe -PSConsoleFile \"C:\\Program Files (x86)\\VMware\\Infrastructure\\vSphere PowerCLI\\vim.psc1\" \"& C:\\Sites\\vmdeploy\\PrepNewMachine.ps1 -type #{@type} -machinename #{@name} -size #{@size} -vlan #{@vlan} -creator #{@owner} -creatoremail #{@email} -ipaddress #{ip}"

  system ("start #{commandstr} > ./log/#{@name}.log 2>&1")
end

This uses the windows “start” command and works pretty well.

Share on Facebook

Measuring IOPS
In linux I like the iostat command, though there are a few ways to get at the info. In debian/ubuntu it is in the sysstat package (ie: sudo apt-get install sysstat)

root@MACHINENAME:/home/deploy# iostat 1
Linux 2.6.24-28-server (MACHINENAME.forward.co.uk) 	18/02/11
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
    45.51    0.00    1.85    0.62       0.00       52.03

Device:        tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
cciss/c0d0     4.00       0.00       40.00          0       40
cciss/c0d1     4.00       0.00       64.00          0       64
cciss/c0d2    12.00       0.00      248.00          0       248
cciss/c0d3     0.00       0.00        0.00          0       0
cciss/c0d4    25.00       0.00      320.00          0       320
cciss/c0d5     0.00       0.00        0.00          0       0
cciss/c0d6    30.00       0.00      344.00          0       344
cciss/c0d7    42.00    3144.00        0.00         3144     0

iostat 1 refreshes everysecond, if you do it over a longer period it will average the results. tps is what you are interested in, Transactions Per Second (ie IOPS). -x will give a more detailed output and separate out reads and writes and let you know how much data is going in and out per second.

What is a good or bad number though?
As with most metrics, if the first time you look at it is when you are in trouble then it’s less helpful. You should have an idea of how much IO you typically do, then if you experience issues and are doing 10x that or only getting 1/10 from the disks then you have a good candidate explanation for them.

How much can I expect from my storage?
It depends how fast the disks are spinning, and how many there is.
As a rule of thumb I assume for a single disk:
7.2k RPM -> ~100 IOPS
10k RPM -> ~150 IOPS
15k RPM -> ~200 IOPS
Our hadoop servers were pushing about 70 IOPS to each disk at peak and they are 7.2k ones so that is in line with this estimate.

See here for a breakdown of why these are good estimates for random IOs from a single disk. Interestingly a large amount of it comes from the latency of the platter spinning, which is why SSDs do so well for random IO (Compared to a 15k disk, ~50x for writes, ~200x reads)
See also:
A concrete example of faster CPU causing higher %iowait while actually doing more transactions here

Extreme Linux Performance Monitoring and Tuning: Part 1 (pdf) and Part 2 (pdf) from ufsdump.org/

Share on Facebook

You only need two things – the built-in “sc” command and an executable from Windows Server 2003 Resource Kit Tools called srvany (works with 2008 too). Get just that exe here (if you trust me of course )

Creating the service

Check it exists

Set Parameters In The Registry
Configure it at HKLM/SYSTEM/CurrentControlSet/Services/APPNAME/Parameters

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VMdeploy\Parameters]
"Application"="C:\\Ruby192\\bin\\ruby"
"AppParameters"="C:\\Sites\\vmdeploy\\server.rb -p 80"
"AppDirectory"="C:\\Sites\\vmdeploy"
"AppEnvironment"=hex(7):65,00,78,00,61,00,6d,00,70,00,6c,00,65,00,3d,00,32,00,\
  37,00,00,00,62,00,6c,00,61,00,68,00,3d,00,63,00,3a,00,5c,00,74,00,65,00,6d,\
  00,70,00,66,00,69,00,6c,00,65,00,73,00,00,00,00,00

Note the AppEnvironment is a multiline string, the rest are strings

This lets you run any executable file, change the directory you run it from and pass any arguments or environment variables so should cover most use cases.

I will be sharing the code for both the Sinatra app and the PowerShell deploy script in later posts.

Share on Facebook

Just a brief outline:

Work: Started the year down in Kent working on a XenApp deployment on VMware at the beginning of the year, then over in Dubai building a VMware View solution, went to Libya and built the corporate infrastructure for their biggest telco, spent 2 months working on INGs next gen datacentre in the Hague and ended in London for Forward working on merging USwitch’s infrastructure after the acquisition.
While this was fun, I got fatigued with traveling and living out of a suitcase and was very happy to settle down a bit in London (which I think is the greatest city in the world) and work for a great company which I am very pleased and proud to say I have joined permanently. It is exciting to work somewhere using so much great tech and with so many sound people (with huge brains).

Jollys: I managed to go to Russia, visiting Moscow and St Petersburg (OK, that was Dec 2009 but what the hell), the Czech Republic, Italy, go to Oklahoma for a friends wedding followed by a visit to two ancestral puebloan sites, Berlin, Paris, Vegas and Egypt. I feel privileged to have the opportunity to have done all this and am still a bit amazed at just how much happened.

Life: I have always thought their was an embarrassment of riches when I look at the great people I get to call my friends, I don’t get to see any of them enough and no amount is too much. Thanks to you all, I always say you are what makes the universe great for me, you conspire to make it so even though you don’t all know one another. The biggest change this year was finding someone patient enough to pair up with me full time, thanks Petra – I love you. We are moving in together soon and I look forward to having a permanent home in the greatest city in the world, with a spare room – please come visit!. Thanks to my family too, we had some bad news in 2010 but you all dealt with it with the usual aplomb, you are all ace.

Resolutions There is no new resolutions for 2010, I will be reporting on the 101 goals in 1001 days soon.

2011: Looks set to be the best year ever, thanks in advance for helping make it so

Share on Facebook

I had previously moaned about seeing Jimmy Wales’s face every time I logged in, like this

and laughed my head off at this piss take from The Daily What:

I found today at Information Is Beautiful the following demonstration of just how effective the campaign has been though.

Wikimedia have done some nice analysis of the campaign on the Meta Wiki if you are interested.

Give to Wikipedia here

Share on Facebook

Will #VMware Load-Based Teaming remove the need for #Cisco EtherChannel? Discuss….

I long ago investigated NIC Teaming algorithms and settled on IP Hash with Cisco Etherchannels for most environments, only really using something else if the client happened not have stacked switches. Thanks to Scott Lowe for this superb article on the matter.

When vSphere 4.1 came out with Load Based Teaming, I was pleased that at last we had an algorithm that would have a go at proper load balancing and not just load distribution but had not got round to investigating much more.

At Forward we have just updated to 4.1, Enterprise Plus and have bought some shiny new Extreme Summit X650 Series 10G switches; so Simon’s comment was particularly apropos.

I had decided I wanted to try and use LBT but was unsure if I should port-channel the uplink ports. It turns out you can’t. I thought maybe you should to be honest, it does not mention in the dvSwitch guide as far as I can see but the ESX host requirements for link aggregation KB (updated today) is very clear

• The switch must be set to perform 802.3ad link aggregation in static mode ON and the virtual switch must have its load balancing method set to Route based on IP hash.
• Enabling either Route based on IP hash without 802.3ad aggregation or vice-versa disrupts networking

ie you need both IP Hash and EtherChannel and neither will work without the other.

In answer to Simon’s question, my feeling is you may still get better performance from EtherChannel and IP based hash for some workloads but would guess “usually” LBT wins. I think the case where you may get better utilisation is when certain VMs have very high bandwidth requirements to different IPs. As described here IP Hash is the only way to allow traffic from one vNIC to leave over different pNICs at the same time.

It is interesting that even with LBT bandwidth is still limited to the maximum bandwidth a single pNIC can provide for individual VMs / vmkernels, also IP hash will not get higher than a single pNIC for a vMotion or other point to point connections. So 10G is going to perform better for these operations than 10x1G, however you team them.

Share on Facebook

In the 3par storage I could see that IO was unusually high of late.

Then I looked at the top-n datastores by IOPS and graphed them

A huge jump for sharedstorage8, so I looked at its VMs

and found the culprit VM.

Here it is against our big “Superhero” database and the vCenter server with the DB.

A lot of IO from a machine the owner thought was doing nothing!

Hyper9 is a pretty good tool for reporting, alerting and troubleshooting your VMware infrastructure, the query language is lucene based and this gives you lots of options in creating custom views and alerts.

Share on Facebook

The second is my del.icio.us tags, when is later going to happen?

With delicious closing I am looking at alternatives, send recommendations if you have any.

Share on Facebook

We flew out last Thursday and stayed for 3 nights at the Wynn, which is a great hotel.

Thursday: Arrive and sleep (forgive me I only just got back from Egypt after flying to Manchester instead of London and having to sit all night on a freezing cold coach!).

Friday: Flew in a helicopter into the Grand Canyon which was rather awesome.

Sat: Went to see Zumanity by Cirque du Soleil, it was amazing, I must see them again.

Sunday: A few of us went shooting,
An M16

An H&K MP5

A Mac-10

not firing it gangster style unfortunately.

A Tommygun

A Desert Eagle

And I did a Shotgun

And I’ve got the T-shirt to prove it

Peace Through Superior Firepower indeed

Pics On Flickr

Vids On Youtube

Share on Facebook