Friday, July 29, 2011

System Administration Nirvana

I don't consider myself a true sysadmin, but inevitably you need to dabble a bit as an admin if you want to build anything fun. Part of my current project is a custom python daemon. My web application posts jobs to a database (if you will allow me to call mongo a db...), which the python daemon monitors. When a job is posted to the db, the daemon picks it up and does the processing.

Of course, I have somewhere between a few and several bugs in my monitoring process. So, from time to time, I need to restart the process. I just invited my first alpha user to test out the site, so while my audience is minuscule, I'm still very worried about the site being dysfunctional.

Enter RightScale. Today, in about 4 hours, I learned all about monitoring at rightscale (they use collectd) and I enabled it for my job monitoring servers. It was easy to add a plugin to monitor my custom application -- I just configured the standard processes plugin to track my daemon. Immediately, I was able to see count, cpu usage, mem usage, and disk io for my process. Very useful. I added an escalation to email me when the process crashed*. That was neat... but then I had this vision of myself fishing with my son, getting an urgent email, making him quit fishing early (tears), and then speeding home, all just to type "kill -9 ". So, I made a custom alert escalation on rightscale to restart the deamon if it crashes. Pretty simple, but something that would have taken days in the past. I would have spent a week just comparing all the options for monitoring systems, and figuring out how to install one on all my servers. 

Another nifty trick - when I invited my testers to the site, I wanted to have separate staging and production environments. So, I clicked the "clone" button, and presto, my whole environment was replicated. heroku_san made it even easier for the web application.

Anyway, wish me luck as the first user tries out my new project!


* Yeah, some of my bugs are still crashing bugs. Sorry Joel Spolsky, I don't have a QA team for this either. I do have 200+ unit tests though!

Friday, July 15, 2011

Dunkin' Donuts and Kettleers

The Dunkin' Donuts in our little town is always full of interesting people. Munchkins and mild coffee attract fans from across all social strata. Everyone from LandRover driving I-Bankers to LandRover driving local cops to fisherman to pasty software engineers turn up for the same fix. This year, the guy manning the counter is unusually chatty and well-spoken. He's downright loquacious and eloquent. This morning I asked him if he was going to get out to enjoy the weather (top ten day today: warm, breezy, and no humidity). "No, but I go to a baseball game almost every night, so I can't really complain. I write the game summaries for the Cotuit Kettleers and whatever I feel like writing on their blog."  His blog is titled "Living the Dream" and he told me that the pay is so paltry he has to work a full week at Dunkies in order to be able to stay capeside for the summer. Despite staying inside on days like today, he seems pretty thrilled. I'm not sure if he did it intentionally, but he couldn't have chosen a better side job. He will likely meet half the county of Barnstable at that Quags.* For a guy looking to make it in media (dream job is writing for ESPN), meeting people and making an impression is a big deal.  Writing every single day for the summer doesn't hurt either.

*Quagonut == Donut in family slang. Hence Dunkin' Donuts == Dunkin' Quagonuts, or Quags for short. Like Rock 'n Roll, Quagonut might once have been quite vulgar, but now it is as pure as a powered jelly donut.

Thursday, July 14, 2011

pymongo and index creation

For my current project, I'm working with a pretty big document db - 100M documents so far, and I am considering scaling it to 25Billion over the next few weeks.

The indexes on the document collections take several minutes to create or ensure. By default, mongo locks the collection while the index is "ensured". For smaller collections, it is no big deal. But for a rebuild that takes up to an hour, the blocking operation is a big problem. Turns out mongodb supports background rebuilds, but it wasn't clear that the PyMongo driver exposed it. PyMongo's documentation on collection.ensure_index didn't mention backgrounding. However, the driver has a passthrough for keyword args, so based on mongodb's documentation for backgrounding, I just tried it. Seems to work!

Centerville Library

I discovered yet another reason to love the public libraries out there. We're down capeside, so I had been working from the house this morning. That kind of blew up my whole transition from play to work, since my office is one corner of the kids' play area in the basement. Most of this week I was coding with one hand and wrestling one of the boys with the other.

Today I drove over to the Centerville Public Library. Quaint little place, and just jammed with people pursuing all kinds of interests. I see a guy writing for a magazine, a bunch of kids doing puzzles (sunny here, but the wind's blowing too hard to do anything even near the water), and there are four guys playing chess. Play isn't the right word - they are doing battle. A grown man, maybe in his 40s just resigned a game and stormed outside to cool off - he was too upset by his play to continue. He was midway through a trouncing, having his hat handed to him by a gentleman twice his age. Nothing like a beating from an octogenarian to make you angry. 

If I get my next milestone coded up soon, I'm going to ask for a game...

Wednesday, July 13, 2011

Sabbatical == Coding 10hrs/day == pure joy

After a nine year sprint, I decided to take a restful break from Tamale. What's more relaxing than building a new piece of software?  I'm a few weeks away from a preview release, but I'm having a dandy time building. I had forgotten the sheer pleasure of focusing on one idea for hours on end. The daily routine is pretty terrific - wake up with the kids, a family breakfast, and then I head to the public library. In addition to the free wifi, and the surprisingly robust internet speeds, the library is the perfect level of quiet but not boring. I can fit in plenty of quality people watching (mostly moms corralling kids, but there are a lot of retirees too) while my unit tests run. In the past, I relied on starbucks for working on these pet projects. But I have to suggest using your public library. Not only is it free, but I don't overdose on coffee and baked goods (though the library does have a keurig for $1/cup). Best of all, I can walk to the library from my house. I think this is key for productivity - leaving the kids and homelife at home for a few hours lets me really immerse myself in my project. Of course, I do walk home for lunch and sneak in some playtime with the kids, but again, the walk back and forth is a great way to mentally transition from play to work and back again. Likewise, walking home makes me leave all the coding problems at the library, so I can really enjoy my time at home.

If you are thinking of a web application, before you get started, here is one page you have to read. My friend and Tamale co-founder Nader started Kapost a year and half ago or thereabouts. He took the time to carefully document the best technologies he found during the setup of his new product development team. For my current project, I found his notes invaluable. My personal favorites are:

  • Heroku. After a decade of wrestling with deployment, I bow in awe of Heroku's git push deployment. Rails ain't too shabby either, but I think Heroku is a bigger deal than Rails itself.
  • MongoHQ. Despite the ribbing I'll receive for saying this, I'm a fan of mongodb, which is webscale. MongoHQ  does a great job with provisioning new instances on AWS.
  • AWS and RightScale. I've done hobby work on the google app engine, but this is the first time I've used amazon. In addition to the infrastructure I'm using that runs on top of AWS, (web application via heroku, and the mongodb instances via mongohq), I needed to run some custom python applications as well as a third-party windows application (I know, I know). RightScale makes the most sense if you want to run infrastructure on multiple clouds (e.g. Rackspace and AWS), which I am not doing. But, the organization of all your scripts and templates in RightScale is so nice I'm using it for a pure AWS deployment. My favorite concept from RightScale is the "roll forward" approach to deployment and the inherent versioning of all your system config and data. I also love Elastic Block Storage. It is amazingly useful to be able to move a hard drive between machines, or use it to hold state as you update a machine config.  

I actually wrote a slim version of my new application on google app engine. I was pretty happy with the experience, except that I wanted to be able to access the data store from a web application environment and a standalone application environment. GAE was excellent for the web app, putting aside whatever you think of django vs rails of course, but it wasn't possible to share the data storage with non-GAE applications. My biggest hesitation was learning Ruby and Rails. I was excited to be coding, and I feel pretty comfortable in python. However, I didn't know the first thing about Ruby or the Rails framework. Precursory searches left me thinking that I was missing out on something, but that it would be difficult to catch up with the RoR community. In my experience, it is much more effective to keep current on a technology from its inception, than it is to ramp up and stay current with a more mature ecosystem. Rails is roaring along, growing in many directions at once, so it is a bit of a bear to learn on your own. Luckily, I found a tremendous book - Ruby On Rails Tutorial. One thing about this book set it apart from every other technology tutorial I've read - the author urges you to read all the chapters, in order, and to really code as you read. I found it was the perfect way to catch up on RoR - the book explains how the major parts of the community work, what the current best practices are (e.g. he explains how to use github), and how to do more research. Best of all, I was able to use the tutorial as a baseline for my own application. By the final chapters of the book, he has you working out many to many and self-referential data models. Rather than implement the twitter clone that is the basis of the tutorial, I just mapped the ideas to my own problem. For seasoned developers, I think this is the ideal approach. You have example code and explanation of the concepts in the tutorial, and you can be productive and minimize the tedium by translating the examples to your own domain.