Filtered by JavaScript, Python

Page 27

Reset

premailer 2.9.0 and new rules for `base_url`

May 11, 2015
0 comments Python

I just pushed out a new release of premailer which comes with a pretty big change.

What it means is that the way the base_url and any href= or src= gets combined. For example, you used to be able to set Premailer(html, base_url='http://example.com/subfolder') and combined with <img src="/images/foo.png"> it would become <img src="http://example.com/subfolder/images/foo.png">.

Not any more. The joining works exactly like the Python built-in urljoin() works. E.g.


>>> from urllib.parse import urljoin  # python 3
>>> urljoin('https://example.com', '/image.png')
'https://example.com/image.png'
>>> urljoin('https://example.com/subfolder', '/image.png')
'https://example.com/image.png'
>>> urljoin('https://example.com/subfolder/', '/image.png')
'https://example.com/image.png'
>>> urljoin('https://example.com/subfolder/', '//image.png')
'https://image.png'
>>> urljoin('https://example.com/subfolder/', '//mycdn.com/image.png')
'https://mycdn.com/image.png'
>>> urljoin('http://example.com/subfolder/', '//mycdn.com/image.png')
'http://mycdn.com/image.png'
>>> urljoin('https://example.com/subfolder', 'image.png')
'https://example.com/image.png'
>>> urljoin('https://example.com/subfolder/', 'image.png')
'https://example.com/subfolder/image.png'

So basically, if you think you tried to do something odd with your base_url check it over carefully when you upgrade to version 2.9.0.

Thank you @ewjoachim and @graingert for your help!

Use closure for your Django context processors

May 9, 2015
11 comments Python, Django

The idea with template context processors in Django is to inject some defaults thing to be available when rendering a template that is rendered with a request.

I.e. instead of...:


def view1(request):
    context = {
        'name': 'View 1', 
        'on_dev_server': request.get_host() in settings.DEV_HOSTNAMES
    }
    return render(request, 'view1.html', context)

def view2(request):
    context = {
        'name': 'View 2', 
        'other': 'things', 
        'on_dev_server': request.get_host() in settings.DEV_HOSTNAMES
    }
    return render(request, 'view2.html', context)

And in your nominal templates/base.html you might have something like this:


  ...
  <footer>
  <p>&copy; You 2015</p>
  {% if on_dev_server %}
    <p color="red">Note! We're currently on a dev server!</p>
  {% endif %}
  </footer>
  ...

Instead you do this trick; in your settings.py you write down the list of defaults plus the one you want to always have available:


TEMPLATE_CONTEXT_PROCESSORS = (
    "django.contrib.auth.context_processors.auth",
    "django.template.context_processors.static",
    "myproject.myapp.context_processors.debug_info",
)

And to accompany that you define your myprojects/myapp/context_processors.py like so:


def debug_info(request):
    return {
        'on_dev_server': request.get_host() in settings.DEV_HOSTNAMES,
    }

So far so good.

However, there's a problem with this. Two problems in fact.

First problem is that when all the templates in your big complicated website renders, it's quite possible that some pages don't need everything you set up in your context processors. That might mean a heck of a lot of extra computation when it won't ever be displayed.

For example, I have a project where most pages have a sidebar where I show "Trending Events" which is something I compute in a context_processors.py function called def sidebar_events(request):. But the sidebar is not always shown and on the pages where it's not shown it's a waste to compute the stuff that sidebar_events computes. Also, I have management pages which uses a totally different base.html template. So there's a big chance you're wasting precious CPU.

Another problem is that of code-readability (aka. how frustrating is this to debug for someone else or yourself after months of idle activity). If you're skimming through your base.html and you see this "random" variable called on_dev_server it's very very hard to tell where the heck that's defined. Hopefully grepping the whole source code is a way to go. A much better way to solve that problem would be sensible namespace naming.

And also, by being too liberal with globally scoped variables there's a chance you might clash from a different piece of functionality that uses the same variable names. That chance is smaller when you use namespaces.

So, to remedy this, let your template context processor functions return closures. It wraps the request automagically.

Let's rewrite our trivial example from above, the context_processors.py should now look like this:


def debug_info(request):
    def inner():
        return {
            'on_dev_server': request.get_host() in settings.DEV_HOSTNAMES,
        }
    return {'debug_info': inner}

Now executing that becomes more optional and more deliberate in the template instead. E.g.


  ...
  <footer>
  <p>&copy; You 2015</p>
  {% set debug_info = debug_info() %}
  {% if debug_info['on_dev_server'] %}
    <p color="red">Note! We're currently on a dev server!</p>
  {% endif %}
  </footer>
  ...

This makes it more explicity which is a good thing. It also has the potential to be avoided if the stuff in there isn't needed in some templates.

Using lazy loading images on Air Mozilla

April 23, 2015
0 comments Mozilla, JavaScript

Starting today, (almost) all the thumbnails below the fold on Air Mozilla are not loaded.

The way it works, is that I use a library called Lazyr.js which notices when you scroll down and when certain pictures are going to be in view, it changes the <img> tag's src.

So it basically looks like this:


<article>
  <h3>Event 1</h3>
  <img src="event1.png">
</article>

<article>
  <h3>Event 2</h3>
  <img src="event2.png">
</article>

<article>
  <h3>Event 3</h3>
  <img src="event3.png">
</article>

<article>
  <h3>Event 4</h3>
  <img src="placeholder.png" data-lazyr="event4.png">
</article>

<article>
  <h3>Event 5</h3>
  <img src="placeholder.png" data-lazyr="event5.png">
</article>

<article>
  <h3>Event 6</h3>
  <img src="placeholder.png" data-lazyr="event6.png">
</article>

That means that to load this page it needs to download, only:

event1.png
event2.png
event3.png
placeholder.png

Only 4 images instead of the otherwise 6 (in this example).

When you scroll down to see the rest of the list, it then also downloads:

event4.png
event5.png
event6.png

The actual numbers on Air Mozilla is that there are 10 events page page and I lazy load 6 of them.

You can see the results when comparing this WebPageTest with this one.

There is more work to do though. At the moment, the thumbnails in the sidebar (Trending and Upcoming events) are above the fold when you're browsing but below the fold when you're viewing an individual event. That's something I have yet to implement.

AngularJS $q notify and resolve as a local GET proxy

April 18, 2015
1 comment AngularJS, JavaScript

Something tells me there are already solutions like this out there that are written by much smarter people who have tests and package.json etc. Perhaps my Friday-brain failed at googling them up.

So, the issue I'm having is an angular app that uses a ui-router to switch between controllers.

In almost every controller it looks something like this:


app.controller('Ctrl', function($scope, $http) {
  /* The form that needs this looks something like this:
      <input name="first_name" ng-model="stuff.first_name">
   */
  $scope.stuff = {};
  $http.get('/stuff/')
  .success(function(response) {
    $scope.stuff = response.stuff;
  })
  .error(function() {
    console.error.apply(console, arguments);
  });
})

(note; ideally you push this stuff into a service, but doing it here in the controller illustrates what matters in this point)

So far so good. But so far so slow.

Every time the controller is activated, the AJAX GET is fired and it might be slow because of network latency.
And I might switch to this controller repeatedly within one request/response session of loading the app.

So I wrote this:


app.service('localProxy',
    ['$q', '$http', '$timeout',
    function($q, $http, $timeout) {
        var service = {};
        var memory = {};

        service.get = function(url, store, once) {
            var deferred = $q.defer();
            var already = memory[url] || null;
            if (already !== null) {
                $timeout(function() {
                    if (once) {
                        deferred.resolve(already);
                    } else {
                        deferred.notify(already);
                    }
                });
            } else if (store) {
                already = sessionStorage.getItem(url);
                if (already !== null) {
                    already = JSON.parse(already);
                    $timeout(function() {
                        if (once) {
                            deferred.resolve(already);
                        } else {
                            deferred.notify(already);
                        }
                    });
                }
            }

            $http.get(url)
            .success(function(r) {
                memory[url] = r;
                deferred.resolve(r);
                if (store) {
                    sessionStorage.setItem(url, JSON.stringify(r));
                }
            })
            .error(function() {
                deferred.reject(arguments);
            });
            return deferred.promise;
        };

        service.remember = function(url, data, store) {
            memory[url] = data;
            if (store) {
                sessionStorage.setItem(url, JSON.stringify(data));
            }
        };

        return service;
    }]
)

And the way you use it is that it basically returns twice. First from the "cache", then from the network request response.

So, after you've used it at least once, when you request data from it, you first get the cached stuff (from memory or from the browser's sessionStorage) then a little bit later you get the latest and greatest response from the server. For example:


app.controller('Ctrl', function($scope, $http, localProxy) {
  $scope.stuff = {};
  localProxy('/stuff/')
  .then(function(response) {
    // network response
    $scope.stuff = response.stuff;
  }, function() {
    // reject/error
    console.error.apply(console, arguments);
  }, function(response) {
    // update
    $scope.stuff = response.stuff;
  });
})

Note how it sets $scope.stuff = response.stuff twice. That means that the page can load first with the cached data and shortly after the latest and greatest from the server.
You get to look at something whilst waiting for the server but you don't have to worry too much about cache invalidation.

Sure, there is a risk. If your server response is multiple seconds slow, your user might for example, start typing something into a form (once it's loaded from cache) and when the network request finally resolves, what xhe typed in is overwritten or conflicting.

The solution to that problem is that you perhaps put the form in a read-only mode until the network request resolves. At least you get something to look at sooner rather than later.

The default implementation above doesn't store things in sessionStorage. It just stores it in memory as you're flipping between controllers. Alternatively, you might want to use a more persistent approach so then you instead use:


controller( // same as above
  localProxy('/stuff/', true)
  // same as above
)

Sometimes there's data that is very unlikely to change. Perhaps you just need the payload for a big drop-down widget or something. In that case, it's fine if it exists in the cache and you don't need a server response. Then set the third parameter to true, like this:


controller( // same as above
  localProxy('/stuff/', true, true)
  // same as above
)

This way, it won't fire twice. Just once.

Another interesting expansion on this is, if you change the data after it comes back. A good example is if you request data to fill in a form that user updates. After the user has changed some of it, you might want to pre-emptivly cache that too. Here's an example:


app.controller('Ctrl', function($scope, $http, localProxy) {
  $scope.stuff = {};
  var url = '/stuff/';
  localProxy(url)
  .then(function(response) {
    // network response
    $scope.stuff = response.stuff;
  }, function() {
    // reject/error
    console.error.apply(console, arguments);
  }, function(response) {
    // update
    $scope.stuff = response.stuff;
  });

  $scope.save = function() {
      // update the cache
      localProxy.remember(url, $scope.stuff); 
      $http.post(url, $scope.stuff);
  };
})

What do you think? Is it useful? Is it "bonkers"?

I can think of one possible beautification, but I'm not entirely sure how to accomplish it.
Thing is, I like the API of $http.get that it returns a promise with a function called success, error and finally. The ideal API would look something like this:


app.controller('Ctrl', function($scope, $http) {
  $scope.stuff = {};
  // angular's $http service expanded
  $http.getLocalProxy('/stuff/')
  .success(function(cached, response) {
    /* Imagine something like:
        <p class="warning" ng-if="from_cache">Results you see come from caching</p>
     */
    $scope.from_cache = cached;
    $scope.stuff = response.stuff;
  })
  .error(function() {
    console.error.apply(console, arguments);
  });
})

That API looks and feels just like the regular $http.get function but with an additional first argument to the success promise callback.

Match the whole word in auto complete maybe

April 10, 2015
2 comments Web development, JavaScript

(For context, I released Autocompeter.com last week and now I'm thinking about improvements)

I posted a question on Twitter about which highlighting formatting people prefer and got some interesting feedback. More about that later.

The piece of feedback that really got my attention came from my friend Honza Král.
He wondered if not the whole word should be highlighted instead of just the beginning of the word.

I've actually been thinking about that too but never got around to trying it out. Until now.

Before

Before

After

After

What do you think?

I have the code in a branch and I'm still mulling it over. There's sort of a convention to just highlight based on what you've typed so far. I don't want to be too weird because when people don't feel familiar they don't like what they see even if the new actually is better.

gulp-header is the best!

April 9, 2015
1 comment JavaScript

For Autocompeter I develop with gulp. It's like Grunt but better.

One thing I wanted was that when it makes the src/autocompeter.js --> minify() --> dist/autocompeter.min.js step I also wanted to put in a little preample header into the minified file.

First I thought, since UglifyJS supports a --preamble option that that'd be the route to go. I didn't get very far.

Then I thought I had to write my own plugin. So I started reading the documentation about how to write a plugin and partially thinking "Oh I don't have time to do this" and also "Oh finally a chance to sit down and really understand how gulp plugins work". I was wrong. The documentation for writing plugins say:

"Your plugin shouldn't do things that other plugins are responsible for... ...It should not add headers, gulp-header does that"

Oh! So there is already a great plugin for this! Long story short; here's how I used it. The output is that the version number is now on the first line of autocompeter.min.js.

I'm starting to like gulp more and more. There's even a dedicate nice index of all available plugins.

Autocompeter 1.1.8 and smooth typing

April 6, 2015
0 comments Go, JavaScript

One of the most constructive pieces of feedback I got when Autocompeter was on Hacker News was that when you type something with lots of predictable results the results overlay would "flicker".

E.g. you type "javascript" in a nice and steady pace and the overlay would shrink and grow and shrink and grow very rapidly.

The reason it happened was due to a bug in the javascript code that filtered results whilst waiting for the next AJAX request from the latest typed character. E.g. you type "ash" and the results comes back with "ashley", "ashes", "Ashford". Then you add a "l" so now we start a new AJAX query for "ashl" and whilst waiting for that output from the server we can start filtering out "ashes" and "Ashford" because we can pre-emptively know that that won't be in the new result set.

The bug was a bad function that filtered the existing results on a second rendering whilst waiting for the next AJAX. It was easy to fix and this is included in version 1.1.8.

The reason I failed to notice this was because I had inserted some necessary optimizations when the network latency was very very slow but hadn't tested it in a realistic network latency environment. E.g. a decent DSL connection but nevertheless something more advanced that just connecting to localhost.

Autocompeter.com

April 2, 2015
8 comments Go, JavaScript

About a year ago I found a great article about how to use Redis to index every prefix of every word as an index and thus a super fast way of building an autocomplete service. The idea is that you take all your titles and index them like this; if the title is "My Title" you store a key for m, my, t, ti, tit, titl and title. That means you can do very fast lookups as someone is typing unfinished words.

Anyway. I was running this merrily here on my personal blog but I liked it so much and I wanted to use it on aother site for work that I thought it'd be time to extract it into its own little microservice. All I needed was a name and my friend and colleague jezdez suggested I call it "autocompeter". So that it became.

The original implementation was written in Python and at the time I was learning Go and was eager to have something to build in Go. So I built this microservice in Go using the Negroni web framework.

The idea is that you own and run a website. You have a search feature on your website but you don't have a nifty autocomplete (aka. live search) thing on it. So, you send me all your titles, URLs and optionally their "popularity ranking" (basically a score). I'll index them on autocompeter.com under your domain. You have to sign in with GitHub to set up an API Auth Key.

Then, you put this into your HTML:


<script src="//cdn.jsdelivr.net/autocompeter/1/autocompeter.min.js"></script>
<script>
Autocompeter(document.querySelector('input[name="q"]');
</script>

Also, you'll need to download the CSS and put into your site. I don't recommend pointing to a CDN for CSS.

And that's all you have to do. The REST API, options for the Javascript integration and CSS integration in the documentation.

The Javascript is framework-free meaning it's just pure DOM manipulation and works in IE and modern browsers. The minified file is only 4.2Kb minified (2Kb gzipped).

All code is Open Source under a BSD license. Everything is free but there's no SLA as of yet.

I'm going to be blogging more and more about feature development, benchmarks and other curious things I learn developing this further.

Bye bye httpretty. Welcome back good old mock!

March 19, 2015
4 comments Python

After a day of pushing 9 commits to a PR to finally get Travis to build a simple python package on python 2.6, 2.7, 3.3 and 3.4 I finally gave up and ripped out all of httpretty and replaced it with good old mock.patch()

I was getting all sorts of strange warnings in py3.3 and 3.4 got stuck all the time.
This is not the first time httpretty has been causing confusion so from now on I'm giving up on httpretty. Ithink it was too good to be true to work reliably. Honestly, it might be python's fault for not being better made available to cool libs like httpretty.

By the way, here's one of those errors where Python 3.4 just hangs which stopped being the case once I took out httpretty. And here you can see the clear failure to deactivate the monkeypatch even after the test is complete in Python 3.3.

Median size of Javascript libs on jsDelivr

February 24, 2015
0 comments JavaScript

If you haven't heard of jsDelivr then you've missed out on a great project! It's basically a free CDN to use for Open Source projects. I added my own yesterday and it was as easy as making a pull request with the initial file, some metadata and a file that tells them where to pick up new versions from (e.g. GitHub).

Anyway, they now host A LOT of files. 8,941 to be exact (1,927 unique file names), at the time of writing. So I thought I'd check out what the median size is.

The median size is: 7.4Kb

The average is 28.6Kb and the standard deviation is 73Kb! so we can basically ignore that and just focus on the median.

Check out the histogram

That's pretty big! If you exclude those bigger than 100Kb the median shrinks to 6.5Kb. Still pretty big.

I'm proud to say that my own is only 50% the size of the median size.