Archive for the ‘linux’ Category

Running –repair on MongoDB via Upstart

May 13th 2011

One of our servers running MongoDB crashed today and we encountered the typical

old lock file: /var/lib/mongodb/mongod.lock. probably means unclean shutdown
recommend removing file and running –repair
see: http://dochub.mongodb.org/core/repair for more information

As the docs do not seem to have much of an alternative to running –repair I looked for a way to automate it from upstart. Mongo creates a mongod.lock file in the data directory with the processes PID in and on a safe shutdown removes the PID, leaving the file there.

This upstart scripts includes a pre-start script that checks if the lock file exists, reads it, makes sure there is a PID there, makes sure no mongod processes exist with that PID then performs the repair as the mongodb user.

limit nofile 20000 20000

kill timeout 300

env MONGO_DATA=/var/lib/mongodb/
env MONGO_LOGS=/var/log/mongodb/
env MONGO_EXE=/usr/bin/mongod
env MONGO_CONF=/etc/mongodb.conf

pre-start script
  mkdir -p $MONGO_DATA
  mkdir -p $MONGO_LOGS
  if [ -f $MONGO_DATA/mongod.lock ]; then
    mongo_pid=`cat $MONGO_DATA/mongod.lock`
    if [ ! -z $mongo_pid ]; then
      if [ ! `pgrep mongo | grep "$mongo_pid" | wc -l` -gt 0 ]; then
        rm $MONGO_DATA/mongod.lock
        sudo -u mongodb /usr/bin/mongod --config /etc/mongodb.conf --repair
        touch $MONGO_DATA/repaired-`date "+%Y%m%d-%H%M%S"`
      fi
    fi
  fi
end script

start on runlevel [2345]
stop on runlevel [06]

script
  if [ -f /etc/default/mongodb ]; then . /etc/default/mongodb; fi
  exec start-stop-daemon --start --quiet --chuid mongodb --exec $MONGO_EXE -- --config $MONGO_CONF
end script
view raw mongodb.conf This Gist brought to you by GitHub.

Posted by tom under devops & linux & mongodb | 1 Comment »

Measuring Disk Usage In Linux (%iowait vs IOPS)

February 18th 2011

This occurred to me when looking at our Hadoop servers today, lots of our devs use IOWait as an indicator of IO performance but there are better measures. IOWait is a CPU metric, measuring the percent of time the CPU is idle, but waiting for an I/O to complete. Strangely – It is possible to have healthy system with nearly 100% iowait, or have a disk bottleneck with 0% iowait. A much better metric is to look at disk IO directly and you want to find the IOPS (IO Operations Per Second).

Measuring IOPS
In linux I like the iostat command, though there are a few ways to get at the info. In debian/ubuntu it is in the sysstat package (ie: sudo apt-get install sysstat)

root@MACHINENAME:/home/deploy# iostat 1
Linux 2.6.24-28-server (MACHINENAME.forward.co.uk) 	18/02/11
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
    45.51    0.00    1.85    0.62       0.00       52.03

Device:        tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
cciss/c0d0     4.00       0.00       40.00          0       40
cciss/c0d1     4.00       0.00       64.00          0       64
cciss/c0d2    12.00       0.00      248.00          0       248
cciss/c0d3     0.00       0.00        0.00          0       0
cciss/c0d4    25.00       0.00      320.00          0       320
cciss/c0d5     0.00       0.00        0.00          0       0
cciss/c0d6    30.00       0.00      344.00          0       344
cciss/c0d7    42.00    3144.00        0.00         3144     0

iostat 1 refreshes everysecond, if you do it over a longer period it will average the results. tps is what you are interested in, Transactions Per Second (ie IOPS). -x will give a more detailed output and separate out reads and writes and let you know how much data is going in and out per second.

What is a good or bad number though?
As with most metrics, if the first time you look at it is when you are in trouble then it’s less helpful. You should have an idea of how much IO you typically do, then if you experience issues and are doing 10x that or only getting 1/10 from the disks then you have a good candidate explanation for them.

How much can I expect from my storage?
It depends how fast the disks are spinning, and how many there is.
As a rule of thumb I assume for a single disk:
7.2k RPM -> ~100 IOPS
10k RPM -> ~150 IOPS
15k RPM -> ~200 IOPS
Our hadoop servers were pushing about 70 IOPS to each disk at peak and they are 7.2k ones so that is in line with this estimate.

See here for a breakdown of why these are good estimates for random IOs from a single disk. Interestingly a large amount of it comes from the latency of the platter spinning, which is why SSDs do so well for random IO (Compared to a 15k disk, ~50x for writes, ~200x reads)
See also:
A concrete example of faster CPU causing higher %iowait while actually doing more transactions here

Extreme Linux Performance Monitoring and Tuning: Part 1 (pdf) and Part 2 (pdf) from ufsdump.org/

Posted by tom under linux | No Comments »

Ubuntu is great. Ubuntu is Debian?

February 19th 2008

I have run Linux systems for about a decade, but not always as my main OS though. Whereas I used to enjoy the learning curve linux forced on you, at some point it was both more interesting and easier to use than windows. The selection of packages in Ubuntu is superb and there is no show-stopping apps that I need to run windows for. With all the compiz niceness we are winning the eye candy wars, and of course it increases my productivity… marginally. The installer is unbeatable, people who think windows is easier are mad or deluded by the fact someone else did it for them. Hardware support is brilliant in Linux in general and particularly in Ubuntu; even if you dont like binary drivers, having stuff work is what you want.

I have been amazed at how well Ubuntu works on my laptop (a Dell D430), every device works, I have Compiz on my Intel graphics card, the USB DVD-RW works, the docking station works, all the function keys work, everything works. The only slight niggle was attaching a projector the other day, I had never done dual monitors and Fn-F8 seemed to work but I had to change my screen resolution to match the projectors, I could not get it to work in dual monitor mode (I only tried a few mins as the pressure was on to get an image up)

Ubuntu is Debian. I have said this to people before, possibly to be controversial but I think there is a point to be made. Ubuntu is Debian at least in the way Debian is GNU + Linux + X.org + …, it would not exist without the work of the Debian developers. It is distinct enough that I will never state that identity again, I was over egging the pudding. I like the idea of them selecting a core of packages from Debian testing/unstable, QAing them, adding apps not found in there and tweaking the settings. I spent an awful long time trying to do that single handedly in the past, using apt-pining and loads of 3rd party repos, and it was a pain and certainly not as stable as Ubuntu is. I like that they recompile Debian packages and make them available in Universe and Multiverse, but think they are a bit coy about saying where they come from, “a snapshot of the free world” or something I think they say. I think people perhaps need to be a bit more thoughtful when giving credit, hard work by Debian, the kernel guys, Gnome and pick_your_favorite_app will make Ubuntu look better, as of course will all the good work the Ubuntu guys do. I think I will continue to use Debian stable for servers for some time though, Ubuntu LTS does not seem as safe a bet to me.

Posted by tom under linux & Ubuntu | No Comments »

GNU PDF

December 3rd 2007

In the great tradition of minimising the barrior to only using free software the FSF have announced GNU PDF. This is a fully feartured PDF library for apps to view or create PDFs so you should see an Acrobat killer in your favourite distro sometime.

We use Acrobat in work and don’t get anything like our moneys worth, we basically use it to annotate PDFs that come back from translators and print to PDF. I have used PDFCreator for one of those tasks, but the other is more of a hurdle.

PDF is a pretty good open standard. I would be very glad to see a GPL3 library that fully implements all its features.

Posted by tom under linux | No Comments »

Font Goodness

November 15th 2007

While I am not all that particular about most things, poor fonts annoy me; particularly if most of the desktop is OK. The Linux (strictly speaking XFree86) font display was always either ugly or hard to get working. Ubuntu has sorted this out in the main, but emacs-gtk was ugly. Emacs wizards will not be bothered about the GTK version as they are so productive from the command line but I like having the menus there. Fortunately I found this package.

While I am on the topic of fonts, I must recommend learning LaTeX. If you have ever read a maths or science paper you will recognise the output; beautiful, well structured documents. LaTeX is an extension to TeX, which was created by Donald Knuth as he was unhappy with the quality of maths journals. Checkout AUCTeX and Flyspell (a syntax-aware spellchecker) to more easily create the source files in emacs. For a WYSIWYG tex editor, see TeXmacs. For a WYSIWYM (What you see is what you MEAN) editor, see LyX.

I was looking for a good template to use for a CV, as Jane needs one and I would like to keep mine up to date with things I am currently doing, when I found this advice. I will probably create a LaTeX style using some of the hints there as my current CV was made using some crappy windows software. The site points to quite an interesting article on using type heirachys to show the relative importance of parts of a document, graphic designer types should check out that page as it is pretty interesting.

Posted by tom under Emacs & LaTeX & linux | No Comments »

Pointless Testing

November 13th 2007

Usually I am in favour of testing, when people say “weighing a cow does not make it any heavier” I reply that if you measure nothing, you will never see any improvements. Usually people can come up with cogent arguments against the particular things examined but it never seems to be the case that measuring nothing is the answer, some better metric needs to be created.

I stumbled across a site that reminded me of when I used to read tomshardware to see how fast new PC parts really were (or look at nice graphs and read a conclusion telling me). Their benchmarks of Ubuntu vs Fedora and The Last 12 Linux Kernels strike me as pretty pointless. Pages of graphs saying things are the same bar noise, as you would probably expect. The conclusion of the distro shootout at least says you should (could?) not base your decision on the results. The Linux one is interesting:

The only benchmark where there was a definitive improvement was with the network performance. Granted, these 12 Linux kernels were only tested on one system and in eight different benchmarks. We will continue benchmarking the Linux kernel in different environments and report back once we have any new findings.

Of all the benchmarks, I would have predicted in advance that the network one would be noisiest. I would suggest that you would get different results on different machines, perhaps they should use different clocks too.

Do these people creating the graphs know about repeating measurements, averaging and statistical significance tests? I think not, so what is the point?

Posted by tom under linux & Mathematics | No Comments »

Bittorrent In Linux

November 9th 2007

Azureus is crap in Linux.

That is to say neither the Ubuntu package or the download from the webpage worked for me straight away.

Deluge on the other hand is working a treat though. It’s written for Gnome in C++ and Python so I may be able to get my head around the code too.

Now I can continue downloading more stuff than there is hours in the day to watch it.

Posted by tom under linux & Ubuntu | No Comments »