Peterbe.com

After about two months of evening hacking I'm finally ready to release my latest project: DoneCal.com

It's a simple calendar that doesn't get in your way. You just click on a day and type what you did that day. DoneCal can be an ideal replacement to boring spreadsheet-like timesheets. And unlike regular timesheets/timetrackers with tags you immediately get statistics about how you've spent your time.

I'm personally excited about the Bookmarklet because I practically live in my webbrowser and now I can quickly type what I've just done (could be a piece of support work for a client) with one single click.

If you're a project manager trying to track what your developers are working on, ask them to start tracking time on DoneCal and then ask them to share their calendar with you. They can set up their share so that it only shares on relevant tags.

I'm going to improving it more and more as feedback comes in. Hopefully later this week I'm going to be writing about the technical side of this since this is my first web app built with the uber-fast Tornado framework

This isn't rocket science but it might help someone else.

I often do testing of my various restful HTTP APIs on the command line with curl but often the format the server spits out is very compact and not easy to read. So I pipe it to a little script I've written. Used like this:

$ curl http://worklog/api/events.json?u=1234 | jsonpprint
{'events': [{'allDay': True,
            'end': 1290211200.0,
            'id': '4ce6a2096da6814e5b000000',
            'start': 1290211200.0,
            'title': '@DoneCal test sample'},
           {'allDay': True,
            'end': 1290729600.0,
            'id': '4ce6a22b6da6814e5b000001',
...

MongoKit is a Python wrapper on top of pymongo that adds structure and validation and some other bits and pieces. It's like an ORM but not for an SQL database but for a document store like MongoDB. It's a great piece of code because it's thin. It's very careful not to molly cuddle you and your access to the source. What I discovered was that I was doing an advanced query and with the results they we instantiated as class instances and later turned into JSON for the HTTP response. Bad idea. I don't need them to be objects really so with MongoKit it's possible to go straight to the source and that's what I did.

With few very simple changes I managed to make my restful API app almost 10 times faster!!

Read the whole story here

I've been doing more and more web development with Tornado recently. It's got an awesome class for running client HTTP calls in your integration tests. To run a normal GET it looks something like this:

from tornado.testing import AsyncHTTPTestCase
class ApplicationTestCase(AsyncHTTPTestCase):
   def get_app(self):
       return app.Application(database_name='test', xsrf_cookies=False)

   def test_homepage(self):
       url = '/'
       self.http_client.fetch(self.get_url(url), self.stop)
       response = self.wait()
       self.assertTrue('Click here to login' in response.body)

Now, to run a POST request you can use the same client. It looks something like this:

   def test_post_entry(self):
       url = '/entries'
       data = dict(comment='Test comment')
       from urllib import urlencode
       self.http_client.fetch(self.get_url(url), self.stop, 
                              method="POST",
                              data=urlencode(data))
       response = self.wait()
       self.assertEqual(response.code, 302)

Dawned on me that the Django auth user model that ships with Django is like the string built-in of a high level programming language. With the string built-in it's oh so tempting to add custom functionality to it like a fancy captialization method or some other function that automatically strips whitespace or what not. Yes, I'm looking at you Prototype for example.

By NOT doing that, and leaving it as it is, you automatically manage to Keep It Simple Stupid and your application code makes sense to the next developer who joins your project.

I'm not a smart programmer but I'm a smart developer in that I'm good at keeping things pure and simple. It means I can't show off any fancy generators, monads or metaclasses but it does mean that fellow coders who follow my steps can more quickly hit the ground running.

My colleagues and I now have more than ten Django projects that rely on, without overriding, the django.contrib.auth.models.User class and there has been many times where I've been tempted to use it as a base class or something instead but in retrospect I'm wholeheartedly happy I didn't. The benefit isn't technical; it's a matter of teamwork and holistic productivity.

django-mongokit is the project you want to use if you want to connect your Django project to your MongoDB database via the pymongo Python wrapper. An alternative (dare I say competing alternative) is MongoEngine which is bridge between Django and straight to pymongo. The immediate difference you notice is the syntax. django-mongokit looks like MongoKit syntax and MongoEngine looks like Django ORM. They both accomplish pretty much the same thing. So, which one is fastest?

First of all, remember this? where I showed how django-mongokit sped past the SQL ORM like a lightning bullet. Well appears MongoEngine is even faster.

That's an average of 23% faster for all three operations!

About 5 years ago Martijn Faasen wrote the wonderful blog article What is Pythonic. One thing that I feel is extremely Pythonic is to not compare certain thing to other things when Python has built-in understanding of what false or true means.

Having reviewed/read a lot of beginner code or senior code but of people coming from lower-level languages I often see this:

if variable == False:
...
if variable == 0:
...
if variable == None:
...
if len(variable) == 0:
...
if variable == []:
...
if variable == {}:
...
if ORM.filter(user=variable).count == 0:
...
if not bool(variable):
... 

To be Pythonic is to understand that Python evaluates all of these to false. All built in types have a perfectly sensible boolean operator which is automatically used in an if statement or an embedded if statement in a list comprehension. Keep it clean a pure just like this to check for true:

if not variable:
...
if not ORM.filter(user=variable):
...

And if you have your custom class such as the example just above with the pseudo "ORM" it's easy to extend it by writing your own custom __bool__ like this:

class MyCustomType(somebuiltintype):
   ...
   def __bool__(self):
       return self.somedate and self.somecondition

By playing along with Python just the way Guido indented it you can abstract yourself from being overly dependent of types. By doing the shorthand notation a variable that is otherwise a list can be None if it's not set and your code will continue to work.

All the above might not be true for more explicit lower-level languages like C++ but it sure is Pythonic in Python and that's a good thing.

Ludvig Ericson ("Sweden-based backend-centric super-programmer.") yesterday released the most important Python package you'll ever see this year. Sort of. It animates a little fish on your terminal that goes back and forth across the screen.

Maybe I'm exaggerating a bit. This is the kind of superficial hype that made Rails successful at least. What the package is really useful for is a great start for those who want to do those fancy writes to the terminal without linebreaks. Spoiler alert:

sys.stderr.write("\x1b[2K\r" + fish + "\r")

I've learned a couple of things this week on deploying my first site to use a user friendly OpenID.

My first revelation was when I realized that Google and Yahoo! have solved the usability stumbling block that you can use them as providers without having to know a personally unique URL. For example, for Yahoo! it's just http://yahoo.com which means that you don't need to offer a cryptic URL form and you can just show it as a logo image.

The second thing is that Google's hybrid OpenID + OAuth isn't as complicated as it sounds. It's basically a light extension to the OpenID "protocol" whereby you say, "while you're at it, also give me a OAuth token please so that I can connect back into Google's services later". What's important to understand though is that if you use this you need to know the "scope". scope is a URL to a service. Google Docs is a service for example and you need to search the web to figure out what the scope URL is for that service.

The third revelation was when I understood the difference between Simple Registration Extension (SREG) and Attribute Exchange (AX). Basically, AX is a newer more modern alternative and SREG was the first one. AX is better but some OpenID providers don't yet support it. Google for example, only supports AX. Key to be able to support not just Google's OpenID but any OpenID is that you can request both AX and SREG and whichever one works will be returned.

The fourth thing that helped a lot to understand was the Google's OpenID has a bug in its implementation of Attribute Exchange. Actually, perhaps it's a deliberate design choice they've made but in my opinion a bad one. Unless you say you require email, firstname, lastname, country etc. it won't return it. If you use the if_available directive you won't get it. Another bug/bad design choice is that Google seems to not forward the country attribute. It can happily do first- and last name but not country even if the documentation claims so.

The fifth thing is that python-openid is a lot easier to work with than you think. You don't need to do any crazy network checks or callbacks. For initiating the challenge all you're effectively doing is creating a long URL. If you don't like the API methods python openid offers, just add your own with:

redirect_url += '&openid.ax.mode=fetch_request' # etc.

After so many years since OpenID arrived, I'm only now excited about it. It's tonnes easier to implement than OAuth and now it's actually really pleasant to use as an end user.

uwsgi is the latest and greatest WSGI server and promising to be the fastest possible way to run Nginx + Django. Proof here But! Is it that simple? Especially if you're involving Django herself.

So I set out to benchmark good old threaded fcgi and gunicorn and then with a source compiled nginx with the uwsgi module baked in I also benchmarked uwsgi. The first mistake I did was testing a Django view that was using sessions and other crap. I profiled the view to make sure it wouldn't be the bottleneck as it appeared to take only 0.02 seconds each. However, with fcgi, gunicorn and uwsgi I kept being stuck on about 50 requests per second. Why? 1/0.02 = 50.0!!! Clearly the slowness of the Django view was thee bottleneck (for the curious, what took all of 0.02 was the need to create new session keys and putting them into the database).

So I wrote a really dumb Django view with no sessions middleware enabled. Now we're getting some interesting numbers:

fcgi (threaded)              640 r/s
fcgi (prefork 4 processors)  240 r/s (*)
gunicorn (2 workers)         1100 r/s
gunicorn (5 workers)         1300 r/s
gunicorn (10 workers)        1200 r/s (?!?)
uwsgi (2 workers)            1800 r/s
uwsgi (5 workers)            2100 r/s
uwsgi (10 workers)           2300 r/s

(* this made my computer exceptionally sluggish as CPU when through the roof)