Peterbe.com

I saw this today on the Zope mailing list:

TypeError: open() takes at least 1946389116 arguments (4 given)

For anybody who's ever used Python you'll find this funny.

To set the Expires header in my web application I used to use the Zope DateTime function rfc822() which doesn't return the date in GMT format. Here's how I used it:

>>> from DateTime import DateTime
>>> hours = 5
>>> then = DateTime() + hours/24.0
>>> then.rfc822()
'Thu, 16 Aug 2007 20:43:59 +0100'

Then I found out (from using YSlow) that it's better to use the GMT format (RFC 1123), and here's how to do that in Zope:

>>> from App.Common import rfc1123_date
>>> from time import time
>>> rfc1123_date(time() + 3600*hours)
'Thu, 16 Aug 2007 19:45:12 GMT'

(notice that even though my locale is here in London, because of the summer time an hour is added)

Well, I thought I'd do a quick benchmark to compare the two approaches because I suspected that rfc1123_date() was faster because you don't have to create a DateTime() object. Here's what the two methods looked like to benchmark it:

   def foo1(self, hours):
       t0 = time()
       now = DateTime()
       then = now+float(hours/24.0)
       x=then.rfc822()
       return time()-t0

   def foo2(self, hours):
       t0 = time()
       x = rfc1123_date(time() + 3600*hours)
       return time()-t0

The result was as I expected, rfc1123_date() was much faster. Here are the results for 10,000 iterations:

Times1: 1.25969386101
Times2: 0.16867017746

round(1.25969386101/0.16867017746)=7.0

But a benchmark on something like this is a bit non-sense. Why? Because even if there's a 7 times difference, you'll only ever need one of these iterations per page. Not 10,000. The first function foo1() takes 0.00013 seconds.

Conclusion, worry more about getting the right output rather than speed at this kind of level.

There are several ways of serving static files from Zope. The simplest way is to just do something like this:

size = os.stat(file_path)[stat.ST_SIZE]
REQUEST.RESPONSE.setHeader('Content-Type','image/jpeg')
REQUEST.RESPONSE.setHeader('Content-length',int(size))
return open(file_path, 'rb').read()

The disadvantage with this is if the file is say >1Mb, the whole 1Mb needs to be loaded into RAM before it can be sent. This is surely garbage collected afterwards but if you serve many of these at the same time, there's obviously the risk that the RAM gets too full.

The alternative is to use filestream_iterator from the ZPublisher which is an iterator that sends out chunks of the file at a time. Note that to use the filestream_iterator you must set the Content-Length header first. Here's how to use it:

from ZPublisher.Iterators import filestream_iterator
...
size = os.stat(file_path)[stat.ST_SIZE]
REQUEST.RESPONSE.setHeader('Content-Type','image/jpeg')
REQUEST.RESPONSE.setHeader('Content-length',int(size))
return filestream_iterator(file_path, 'rb')

I did a little unofficial benchmark to test if there's any speed difference between the two approaches. I'm currently only interested in serving photographs (jpg) in this app and I'm sure other results would be found if you did bigger filesystem files. These jpgs are in average about 200 Kb. During the course of writing this I tought I'd also try using sendfile as well to see if that makes it any faster:

Serving 526 jpgs with...
function             average total time (5 tests)
  open()               5.07750701904
  filestream_iterator  4.32704296112
  sendfile             4.431672620774

Interesting. Using filestream_iterator shaves off a few milliseconds but it's over many many servings. Conclusion, the benefit of filestream_iterator is not speed, it's RAM (un)usage.

Note. From reading the LocalFS source code I noticed that they only bother with the filestream_iterator if the filesize is greater than 128Kb. I copied that too into my app now but don't understand why. Is there a disadvantage of serving tiny files with filestream_iterator() Anybody?

Worth noting that sendfile wasn't any faster but I suppose you only see a speed boost with sendfile on much larger files.

This week we improved MOBi with a neat little feature that makes it possible to decide what the SMS response should be to inbound SMS on your own server.

To test this you need to be UK resident and willing to part with the 25p it costs to receive the response. Send peterbe is NN (where NN is a number, how old you think I am) to 83211 and await a result. I've set up the sub word "is *" to forward all inbound SMS to an address on www.peterbe.com and here on this server I run the following Python Script:

##parameters=sender, message
##
guess = message.split()[-1]
try:
    age = int(guess)
except ValueError:
    return "That's not a number :-("

if age > 27:
    return "That's not it! It's less than that. Try again"
elif age < 27:
    return "That's not it! It's more than that. Try again"
return "Wow! You know my age!!"

Today I bumped into a nasty challenge when trying to copy a Zope image from ZODB to the filesystem. The reason why it wasn't easy was that Zope splits the file up into chunks if it's upload size is more than 65Kb (2^16 bytes) which I guess it does for being able to "stream" the file back into the browser when viewing the image instead of sending one large blob.

So, this didn't work:

>>> imgobj.meta_type
Image
>>> open(temp_file_path,'wb').write(imgobj.data)

...if the data is big because sometimes imgobj.data is a string (the binary data) and sometimes is a ImplicitAcquirerWrapper that you have to iterate over.

That's when Sascha Welter stepped up on the #zope channel on IRC with a snippet of code that does it properly:

def write_zodb_file_to_tempfile( source, temp_file ):
   """
   Write data from a file from the ZODB / Zope 
   to a tempfile on the filesystem.
   input:
   - source (a zope object, not just a string with the object id!!)
   - temp_file (a writable file object)
   """
   data = source.data
   if isinstance(data, StringType): #StringType
       temp_file.write( str(data) )
   else:
       (start,end) = (0,source.getSize())
       pos = 0
       while source.data is not None:
           l =  len(data.data)
           pos = pos + l
           if pos > start:
               # We are within the range
               lstart = l - (pos - start)
               if lstart < 0: lstart = 0
               # find the endpoint
               if end <= pos:
                   lend = l - (pos - end)
                   temp_file.write(data[lstart:lend])
                   break
               # Not yet at the end, transmit what we have.
               temp_file.write(data[lstart:])
           data = data.next
   temp_file.flush()
   temp_file.seek(0, 0)

I've wanted to Gzip the static content (CSS & Javascript) on my sites for a long time but never found a good enough solution. mod_gzip is out of the question because as far as I've understood it it does a compression on the fly, every time you request the file.

Other solutions have disappointed me because enabling gzip compression has been for all content. I don't want my HTML files gzipped because they're rendered on the fly based on business logic plus by compressing the HTML files. Long story short my yet-to-be-released app now serves the following files from Zope but are only compressed and whitespace slimmed once per server restart:

FILE               ORIG SIZE  NEW SIZE   REDUCTION
screen.css             15224      2738   556%
print.css               2633       885   298%
jquery-latest.js*      57712     18130   318%
jquery-latest.pack.js  20461     10513   195%
common.js               3803      1131   336%
complete-expense.js    18184      2847   639%
Total                 118017     36244   326%

* only used in debug mode

Yes that's right. The static files are now 326% smaller in file size and since the complexity is O(1) the CPU overhead is virtually none. What's cool about this is as an application developer I don't have to worry about it once the files have been set on the class. I never actually see the slimmed or compressed files. The code is also clever enough to serve the uncompressed version of the file if the server doesn't accept gzip using HTTP_ACCEPT_ENCODING.

To make things even more optimized; add the fact that I'm running this Zope behind Squid and with this command:

# nice -n 20 ab2 -n1000 -c10 http://XXX/misc_/MExpenses/screen.css

I get this lovely result:

Concurrency Level:      10
Time taken for tests:   0.372630 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      3168000 bytes
HTML transferred:       2738000 bytes
Requests per second:    2683.63 [#/sec] (mean)
Time per request:       3.726 [ms] (mean)
Time per request:       0.373 [ms] (mean, across all concurrent requests)
Transfer rate:          8300.46 [Kbytes/sec] received

Bare in mind that this is also proxied by Apache2 to get the extra comfort that Apache2 gives us.

Doing the same benchmark on the screen.css file WITH Squid but without gzipping and slimming it I get this result:

Concurrency Level:      10
Time taken for tests:   1.471274 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      15613552 bytes
HTML transferred:       15224000 bytes
Requests per second:    679.68 [#/sec] (mean)
Time per request:       14.713 [ms] (mean)
Time per request:       1.471 [ms] (mean, across all concurrent requests)
Transfer rate:          10363.13 [Kbytes/sec] received

(Without Squid caching, my Zope in this instance does 242.4 [#/sec] for the same file)

Apart from making Zope does less hard work for stupid static files, I've managed to go from 242 requests per second up to 2684 requests per second. Not bad :)