A blog and website by Peter Bengtsson
In 2001 I started my first and perhaps most successful Open Source project I've ever made: IssueTrackerProduct. After nearly a decade of maintaining it I have now officially abandoned it.
It all started when I needed a way to track feedback on my personal website. That's why it was originally called "SiteTrackerProduct". I needed something where I could collect bug reports and any other pieces of feedback and then process it in some structured fashion. It was therefore very important that it would be possible to run the application open for anonymous access. People should be able to submit bugs and issues without having to create an account. You see, kids, back in that day it was actually very common that sites would force users to register and create accounts even just because the content owner wanted it. These days, it's common knowledge that to get people to open up and share anything for others to benefit you make it absolutely trivial to jump straight in without having to see a registration page that looks like a tax return form.
Now, since I long ago abandoned the Zope2 application server technology stack and I no longer use IssueTrackerProduct for anything real it's no longer feasible to maintain this project. In the last five years or so we were actually using it actively to track all projects at Fry-IT where I used to work. I have to say, even though we did grow out of it, it was actually successful. It handled the load (after some much needed patches towards optimization) and it was easy for people to actually use since unlike many other bug trackers, it focused on the non-technical end user first and foremost. As much as possible was done to make it trivial to type in your bug or issue and it automatically took care of all notifications and access rights.
Being a personal Open Source project, over the years, it became a melting pot for experimenting and perfecting various new ideas. Many of them we take for granted today but back then it was quite novel if I may say so. This includes:
Writing all of this, I can not resist to get a bit nostalgic. I did sink A LOT of time into this project. Today when I look back at the code and almost feel sick seeing all the mistakes that I made. Much of the ugliness of the code can be attributed partially to the fact that I often used and abused the code to add new features. Also, because we often needed some features (since it was used to manage all of our projects) "yesterday" and then it was hard to justify doing things "properly". For example, the main .py file is over 14,000 lines of code!
I did called it "perhaps most successful Open Source project I've ever made" in the first sentence. The reason for that is that over the years many many people have downloaded it and installed and let it be used by thousands of users. That's something to be proud of.
Anyway! It's time to move on. So long and thank you for all the fish!
The code is still available at github.com/peterbe/IssueTrackerProduct
We were already aware of Riak before we started using CouchDB, but we werent sure about trusting a new product at this point, so we decided, after some benchmark, to go for CouchDB.
After the first couple of months, it was obvious that this was a bad choice.
Our main problems with CouchDB is scalability, versionning and stability.
Once we store a document in CouchDB, we modify it at least twice after the original write. Each modification generates a new version of the document. This feature is nice for some use-cases, but we dont need it, and theres no way to disable it, so the size of our databases started to become really important. Youll probably tell me hey, you know you can compact your database ?, and Ill answer sure. The trouble is that we never managed to get it to compact an entire database without crashing (well, to be honest, with the last version of CouchDB we finally managed to compact one database).
The second issue is that one database == one file. When you have multiple small databases, this is fine. When you a have only a few databases, and some grow to more than 1TB, the problems keep growing too (and its a real pain to backup).
We also had a lot of random crashes with CouchDB, even if the last version was quite stable."
Does that sound familiar, fellow Zope developer? I know a lot about ZODB but little about CouchDB. One thing that a lot of people don't know about ZODB is that it's very fast and I think this is true about CouchDB too. Speed isn't the same as a raw speed of inserts/queries because with the concurrency variable added the story gets a lot more complex.
It's the exact same perspectives I've always had on ZODB:
1) It's really convenient and powerful
2) It being a single HUGE file makes it hard to scale
3) Versioning can be nifty but it's often not needed and causes headache with the packing
4) It works great but when it cracks it cracks hard and cryptically
This cost me a good hour of debugging so I thought I'd share it in case anybody else stumbles across the same problem. In the end, to solve my problem I had to add debug statements to
StringIO.py to be able to find out where in my Zope Page Template a non-unicode string with non-ascii characters appeared and messed things up.
The error I was getting was this, which I suspect several Zope developers have encountered before:
UnicodeDecodeError: \ 'ascii' codec can't decode byte 0xc3 in position 46: ordinal not in range(128)
The traceback only mentions files in the innards of ZPT of which none you can really do anything about. We all know that the key to avoid Unicode error is to be consistent. You can do this:
>>> '\xc3' + 'string' '\xc3string' >>> u'\xc3' + u'string' u'\xc3string'
But you can't do this:
>>> '\xc3' + u'string' Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)
So, how did I get these non-unicode strings into my application in first place. Simple, I have a search parameter
q and end up with a URL like this:
And the template had this little innocent piece of code:
<input tal:attributes="value request/q"/>
That's what f'ed everything up. So, I ended up having to add this:
<input tal:attributes="value python:context.unicodify(request['q'])"/>
With this little helper function in the base class:
def unicodify(self, s): if isinstance(s, str): return unicode(s, 'utf8') return s
So, hopefully by writing this it will help someone else making the same trivial mistake and not wasting their evening with sporadic print statements all over their frameworks code.
Doing some on-site consulting on an old Zope CMS that has been developed by many different developers over many years. It's pretty good and has lots of powerful features but over the years certain things have been allowed to slip. One problem was that you couldn't click the "Properties" tab. The reason was that it was trying to fetch properties that didn't exist anymore. What had happened was that the class attribute
_properties (which is used by the "Properties" tab in the ZMI) had been stored as a persistent attribute. Here's how to solve that:
def manage_fixPropertiesProblem(self): """ fix so _properties becomes a class attribute instead """ if '_properties' in self.__dict__.keys(): del self._properties return "Awesome!"
I've just released a new little project in Python for tracking memory usage in Zope2 applications with the added benefit that you can hopefully see what URL causes which memory usage "jumps". Hopefully this can help Zope2 developers find out what causes RAM bloat but can also help in helping you optimize your application by early in the development process find out what uses too much RAM. I wouldn't be surprised that there is already a program that does something like this. I've just never seen one. Also by putting this out as an Open Source project and blogging about it hopefully more clever people than me will come forward and point out the right way to do things.
I've also used Google Code this time to manage the project. I've used it before but only for hosting a public SVN for the IssueTrackerProduct SVN. I have to say that I was quite impressed with Google Code this second time. I think it's still fundamentally wrong to confuse people with by offering both download and SVN checkout. I did both this time but I think I might give up on the downloads because who out there, who understands that he/she needs to debug RAM usage, doesn't know how to use SVN?
Finally a little disclaimer: By writing about this here, preparing it on Google Code and writing a README.txt file I've now spent more time "managing" the project than I have on coding it. It's an early test release which hopefully will stir up some ideas for genuine important improvements. I had fun coding it as well since this is my first attempt with Flot which has been great to work with. You get very quick and powerful results. Lastly, I haven't tested this in anything but 32-bit Ubuntu Linux and Firefox.
Here is a sample report: 2008-05-30_16.47.32__3.8_minutes
Martijn Faasen is my hero. Not only is an absolutely brilliant coder he's also able talk so that mortals understand.
What he's replying about is mainly the question "What does Grok give me that, say, django does not?"
Yes, you clever people. It's the same link. For some reason all the great documentation goes into replies on the mailing list rather than into a concise web page with cookbook, book and styled and funny tutorials. Why is that? They've actually made it quite easy now to enter documentation on grok.zope.org with the new Plone site.
An equally important question is: Why don't I do something about it rather than to complain? Well, I've written one how-to at least. My other "excuse" is that I'm not yet an expert enough and hence writing good documentation takes a very long time.
I think there's an important philosophical and political issue at hand. The Grok community is filled with really clever people who are very senior in the web development industry who like using mailing lists and perhaps more importantly, don't need documentation since they can study source code and unit tests to answer their questions. I know this is a sensitive statement but I'll take my chances since it implies that these guys are smarter (or perhaps just more time on their hands).
My internal battle of which new web framework to put my energy into continues. Today (thanks to Martijn's post) Grok earned one more point.