thattommyhall.com

One of our servers running MongoDB crashed today and we encountered the typical

old lock file: /var/lib/mongodb/mongod.lock. probably means unclean shutdown
recommend removing file and running –repair
see: http://dochub.mongodb.org/core/repair for more information

As the docs do not seem to have much of an alternative to running –repair I looked for a way to automate it from upstart. Mongo creates a mongod.lock file in the data directory with the processes PID in and on a safe shutdown removes the PID, leaving the file there.

This upstart scripts includes a pre-start script that checks if the lock file exists, reads it, makes sure there is a PID there, makes sure no mongod processes exist with that PID then performs the repair as the mongodb user.

limit nofile 20000 20000

kill timeout 300

env MONGO_DATA=/var/lib/mongodb/
env MONGO_LOGS=/var/log/mongodb/
env MONGO_EXE=/usr/bin/mongod
env MONGO_CONF=/etc/mongodb.conf

pre-start script
  mkdir -p $MONGO_DATA
  mkdir -p $MONGO_LOGS
  if [ -f $MONGO_DATA/mongod.lock ]; then
    mongo_pid=`cat $MONGO_DATA/mongod.lock`
    if [ ! -z $mongo_pid ]; then
      if [ ! `pgrep mongo | grep "$mongo_pid" | wc -l` -gt 0 ]; then
        rm $MONGO_DATA/mongod.lock
        sudo -u mongodb /usr/bin/mongod --config /etc/mongodb.conf --repair
        touch $MONGO_DATA/repaired-`date "+%Y%m%d-%H%M%S"`
      fi
    fi
  fi  
end script

start on runlevel [2345]
stop on runlevel [06]

script
  if [ -f /etc/default/mongodb ]; then . /etc/default/mongodb; fi
  exec start-stop-daemon --start --quiet --chuid mongodb --exec  $MONGO_EXE -- --config $MONGO_CONF
end script

Share on Facebook

Posted by tom under devops & linux & mongodb | 1 Comment »

I gave a talk recently at the Forward Tech away day entitled We Are All DevOps and it went down quite well. Forward is an unusual environment, the devs are trusted to do lots of the typical sysadmin role and the boundary between Dev and Ops is very blurred. During my first few months in the search team I kept mindmapping stuff I wanted to talk about but only got round to making the slides the day before so it was a bit underprepared but I hope useful for people.

I borrowed ideas from John Leach’s excellent Ruby: Reinventing the Wheel talk, this DepOps: The War Is Over presentation and rambled incoherently about a talk I just saw at the UKUUG Spring Conference from the author of cfengine, see here a nice description of the project (you can see how it has influenced Puppet)

Here are the slides (first time I have used Scribd, it is excellent. Much better than slideshare)
DevOps

I like the James White Manifesto , it chimes really strongly with me.

In particular

On Infrastructure
—————–
There is one system, not a collection of systems.
The desired state of the system should be a known quantity.
The “known quantity” must be machine parseable.
The actual state of the system must self-correct to the desired state.
The only authoritative source for the actual state of the system is the system.
The entire system must be deployable using source media and text files.

Soon they will post videos and I will get to see myself give a talk for the first time.

Share on Facebook

Posted by tom under devops | No Comments »