I'm the proud creator of django-static which is a Django app that takes care of how you serve your static media the best way possible. Although some of these things are subjective generally this is the ideal checklist of servicing your static media:
- Cache headers must be set to infinity
- URLs must be unique so that browsers never have to depend on refreshing
- The developer (who decided which media to include) should not have to worry himself with deployment
- The developer/artist (who makes the media) should not have to worry himself with deployment
- All Javascript and CSS must be whitespace optimized in a safe way and served with Gzip
- All images referenced inside CSS should be taken care of too
- It must be possible to combine multiple resources of Javascript or CSS into one
- It must be possible to easily test production deployment in development environment without too much effort
- A sysadmin shouldn't have to understand a developers Django application
- A development environment must be unhindered by this optimization
- Processing overhead of must be kept to a minimum
- Must be possible to easily say which resources can be whitespace optimized and which can not
So let's get started setting all of this up in your Django + Nginx environment. Let's start with the Django side of things.
Download and install django-static by first running something like easy_install django-static then add django_static to INSTALLED_APPS in your settings.py. and add this to enable 'django-static':
DJANGO_STATIC = TrueThen edit your base.html template from this:
<html>
<link rel="stylesheet" href="/css/screen.css">
<link rel="stylesheet" href="/css/typo.css">
<link rel="stylesheet" href="/css/print.css" media="print">
<body>
<img src="/img/logo.png" alt="Logo">
{% block body %}{% endblock %}
<script type="text/javascript" src="/js/site.js"></script>
<script type="text/javascript">
window.onload = function() {
dostuff();
};
</script>
</body>To this new optimized version:
{% load django_static %}
<html>
{% slimall %}
<link rel="stylesheet" href="/css/screen.css">
<link rel="stylesheet" href="/css/typo.css">
<link rel="stylesheet" href="/css/print.css" media="print">
{% endslimall %}
<body>
<img src="{% staticfile "/img/logo.png" %}" alt="Logo">
{% block body %}{% endblock %}
<script type="text/javascript" src="{% slimfile "/js/site.js" %}"></script>
<script type="text/javascript">
{% slimcontent %}
window.onload = function() {
dostuff();
};
{% endslimcontent %}
</script>
</body>
</html>django_static when loaded offers you the following tags:
-
staticfile<filename> -
slimfile<filename> -
slimcontent ... endslimcontent -
staticall ... endstaticall -
slimall ... endslimall
All the tags with the word slim are copies of the equivalent without; but on its way to publication it attempts to whitespace optimize the content. Now, rendering this, what do you get? It will look something like this if you view the rendered source:
<html>
<link rel="stylesheet" href="/css/screen_typo.1269174558.css">
<link rel="stylesheet" href="/css/print.1269178381.css" media="print">
<body>
<img src="/img/logo.1269170122.png" alt="Logo">
[[[ MAIN CONTENT SNIPPED ]]]
<script type="text/javascript" src="/js/site.1269198161.js"></script>
<script type="text/javascript">
indow.onload=function(){dostuff()};
</script>
</body>As you can see timestamps are put into the URLs. These timestamps are the modification time of the files which means that you never run the risk of serving an old file by an already used name.
The next step is to wire this up in your Nginx. Here is the relevant rewrite rule:
location ^~ /css/ {
root /var/mydjangosite/media;
expires max;
access_log off;
}
location ^~ /js/ {
root /var/mydjangosite/media;
expires max;
access_log off;
}
location ^~ /img/ {
root /var/mydjangosite/media;
expires max;
access_log off;
}That wasn't particularly pretty. Besides as we haven't done any configuration yet this means that files like print.1269178381.css has been created inside your media files directory. Since these files are sooner or later going to be obsolete and they're never going to get included in your source control we probably want to put them somewhere else. Add this setting to your 'settings.py':
DJANGO_STATIC_SAVE_PREFIX = '/tmp/cache-forever'That means that all the whitespace optimized files are put in this place instead. And the files that aren't whitespace optimized have symlinks into this directory.
The next problem with the Nginx config lines is that we're repeating ourselves for each prefix. Let's instead set a general prefix with this config:
DJANGO_STATIC_NAME_PREFIX = '/cache-forever'And with that in place you can change your Nginx config to this:
location ^~ /cache-forever/ {
root /tmp;
expires max;
access_log off;
}django-static is wired up to depend on slimmer if available but you can use different ones, namely Yahoo! YUI Compressor and Google Closure Tools. So, let's use YUI Compressor for whitespace optimizing the CSS and Google Closure for the whitespace optimize the Javascript. Add this to your 'settings.py':
DJANGO_STATIC_CLOSURE_COMPILER = '/var/lib/stuff/compiler.jar'
DJANGO_STATIC_YUI_COMPRESSOR = '/var/lib/stuff/yuicompressor-2.4.2.jar'Now we get the best possible whitespace optimization and a really neat Nginx configuration. Lastly (and this is optional) we might want to serve the static media from a different domain name as the browser won't download more than two resources at a time from the same domain. Or even better you might have a domain name dedicated that never accepts or sends any cookie headers (for example, Yahoo! uses yimg.com). This is accomplished by setting this setting:
DJANGO_STATIC_MEDIA_URL = 'http://static.peterbe.com' # no trailing slashNow you're ready to go! Every single item on the big list above is checked. With a few easy steps and some modifications to your templates you can get the simplest yet best performing setup for your static media. As an example, study the static media URLs and headers of crosstips.org.
Some people might prefer to use a remote CDN to host the static media. This is something django-static is currently not able to do but I'm more than happy to accept patches and ideas from people who want to use it in production and who are eager to help. Everything else still applies. We would just need a callback function that can handle the network copy.
UPDATE
At the time of writing, version 1.3.7 has 93% test coverage. The number of lines of tests is double to the actual code itself. Code: 661 lines. Tests: 1358
Comments
Post your own commentAre there any tests before using django-static and after?
Nginx has a certain max speed for serving static content in a stress test (e.g. ab). That speed is the same even if the URL is poorly chosen. What django-static provides is a guarantee that the names are never poorly chosen.
Also, when the expires headers are set to expire in 2037 the browser will certainly never attempt to try to download it again unless the person running the browser hits the Shift+Refresh which tells the browser to ignore its own cache memory and go back to the server.
Looks pretty cool! I'm currently using django-compressor, which operates in a similar way (using templatetags). The downside of this approach is that a fair bit of processing (at the very least, parsing of HTML and some filesystem mtime checks) has to be done on each and every request to a Django page, even in the best-case scenario when no static files have been modified.
Because there's no comprehensive list of "files to be served" in any one location, there's no way to provide a "build step" that you run once when you deploy changes, and thereafter let the code trust that nothing has changed.
I used to use django-compress which forces you to define all your media assets in the settings file, thus allowing it to provide a run-once build step.
On the whole, I think this is a relatively fair tradeoff for the super-convenient templatetag system; I much prefer not having to change settings whenever I add a media file.
django-static works by maintaining a global dictionary variable that whenever you render the template it maps a filename (e.g. "/js/foo.js") to an "optimized" filename (e.g. "/js/foo.123513563.js")
Only the first time does it need to do the necessary symlinking and whitespace compression so there is some extremely tiny overhead for each rendering.
django-compressor, last time I checked, doesn't support sorting out the images inside the CSS files automatically. That's where most images are and those need to get the best filename rewrites to get the best setup.
Right; it doesn't need to slim or write the files unless something has changed, but it does need to parse the HTML inside the template tags and check whether anything needs updated. This is the same as django-compressor. I agree that that overhead isn't a big concern.
django-compressor does already find and rewrite image links in CSS to make them absolute; would be easy to have it also tag them with a querystring for cache-busting at the same step. Guess I've never needed that because we use the Compass CSS framework, so image links in our CSS are already tagged that way before they ever get to the deploy step.
I'm using django-staticfiles at the moment. The big advantage is that you can have a yourapp/media/yourapp/ directory per application, just like with templates. django-staticfiles will find them all. And there's an option to collect them all into one directory, ready for serving with apache/nginx.
It does not do minimization. And it does not add a timestamp to a filename, so aggressive caching is out. Those two are real nice features of django-static.
So I'm now breaking my head trying to come up with a a tactic for combining the best of both :-)
That idea of having static media per app is a great one because sometimes I find myself creating something invoicing.js or user_admin.css which are clearly not "global".
Then again, templates are ideally placed in the root /templates/myapp/ not inside /myapp/templates/. I guess we could and should do the same with static stuff.
If an a app is going to be reused in quite a number of sites (like we're planning at our company), placing it inside de app is the best way. Otherwise you're duplicating the same template in every site's root/templates/* directory...
Very interesting post. I've been looking for an automated way to handle the file renames, compression, etc. in my setup. Currently I have an nginx load balancer sitting in front of django/apache app servers. nginx is also serving the static content so that apache does not have to worry about it. How would this solution work inside of that infrastructure since the nginx balancer doesn't even have Django running on it? Would it be a simple matter of adding another step to the build to transfer the stamped files to the nginx servers?
That would indeed require some upgrades to django-static as it's currently not able to write across the network.
It's not just that but some people might want to have the files written to a CDN which would be different every time so the best approach would be to write something that makes it possible to define a function or something that takes care of the network write or network copy. In a future version maybe.
1st off, this is a very useful django app! But, I faced some problem dealing with slimfile. I've got slimmer up and running. When I put something like this {% slimfile "css/style.css" %}, there is one image (ref inside style.css) didn't get staticfile applied to it.
When I check, style.1277132930.css the image is still pointing to the "old" one, i.e. background-image:url(../img/hello.png).
Am I doing something wrong here? Any idea?
Thanks!
More info. I've the following media file structure: <project-dir>/media/{css/img/js}. I've also a /cache-forever/ dir under the /media/ dir.
I noticed that django-static created a img dir under the project dir but with no file.
Hi Matthew, If you're experiencing bugs or problems contact me on github or my normal email address. http://github.com/peterbe
Peter
Hey Pete,
So, what do you do if you're using it with Sorl thumbnails? Eg. I'm loading cropped images on the page using SORL like this:
<img src="{% thumbnail object.photo 200x200 crop %}"/>
...and I want it to be a static file served using this django-static magic above.
For that you don't use django-static. Just make sure the files are served from Nginx and you've achieved a lot.
Where's the gzip configuration - I don't see it in the Nginx configuration?
A lot of the whole Nginx config has been left out. It's quite generic and doesn't really affect that way you make it work with Django. Check out the Nginx documentation for the latest tips.
any view on django-static vs django-assets for compressiong css and js ?
I prefer django_compressor. You can use YUI and all that stuff, but you also have the ability to use precompressors.
For example, you can use CoffeeScript, and have it compile to JavaScript, then minify it.
For another example, you can use Less, or Sass/SCSS or CleverCSS, compile it, then minify it.
All automatic of course. I really love it.