Peterbe.com

My personal blog was a regular Django website with jQuery (later switched to Cash) for dynamic bits. In December 2021 I rewrote it in NextJS. It was a fun journey and NextJS is great but it's really not without some regrets.

Some flashpoints for note and comparison:

React SSR is awesome

The way infinitely nested comments are rendered is isomorphic now. Before I had to code it once as a Jinja2 template thing and once as a Cash (a fork of jQuery) thing. That's the nice and the promise of JavaScript React and server-side rendering.

JS bloat

The total JS payload is now ~111KB in 16 files. It used to be ~36KB in 7 files. :(

Before

After

Data still comes from Django

Like any website, the web pages are made up from A) getting the raw data from a database, B) rendering that data in HTML.
I didn't want to rewrite all the database queries in Node (inside getServerSideProps).

What I did was I moved all the data gathering Django code and put them under a /api/v1/ prefix publishing simple JSON blobs. Then this is exposed on 127.0.0.1:3000 which the Node server fetches. And I wired up that that API endpoint so I can debug it via the web too. E.g. /api/v1/plog/sort-a-javascript-array-by-some-boolean-operation

Now, all I have to do is write some TypeScript interfaces that hopefully match the JSON that comes from Django. For example, here's the getServerSideProps code for getting the data to this page:

const url = `${API_BASE}/api/v1/plog/`;
const response = await fetch(url);
if (!response.ok) {
  throw new Error(`${response.status} on ${url}`);
}
const data: ServerData = await response.json();
const { groups } = data;

return {
  props: {
    groups,
  },
};

I like this pattern! Yes, there are overheads and Node could talk directly to PostgreSQL but the upside is decoupling. And with good outside caching, performance never matters.

Server + CDN > static site generation

I considered full-blown static generation, but it's not an option. My little blog only has about 1,400 blog posts but you can also filter by tags and combinations of tags and pagination of combinations of tags. E.g. /oc-JavaScript/oc-Python/p3 So the total number of pages is probably in the tens of thousands.

So, server-side rendering it is. To accomplish that I set up a very simple Express server. It proxies some stuff over to the Django server (e.g. /rss.xml) and then lets NextJS handle the rest.

import next from "next";
import express from "express";

const app = next();
const handle = app.getRequestHandler();

app
  .prepare()
  .then(() => {
    const server = express();

    server.use(handle);

    server.listen(port, (err) => {
      if (err) throw err;
      console.log(`> Ready on http://localhost:${port}`);
    });
  })

Now, my site is behind a CDN. And technically, it's behind Nginx too where I do some proxy_pass in-memory caching as a second line of defense.
Requests come in like this:

1. from user to CDN
2. from CDN to Nginx
3. from Nginx to Express (proxy_pass)
4. from Express to next().getRequestHandler()

And I set Cache-Control in res.setHeader("Cache-Control", "public,max-age=86400") from within the getServerSideProps functions in the src/pages/**/*.tsx files. And once that's set, the response will be cached both in Nginx and in the CDN.

Any caching is tricky when you need to do revalidation. Especially when you roll out a new central feature in the core bundle. But I quite like this pattern of a slow-rolling upgrade as individual pages eventually expire throughout the day.

This is a nasty bug with this and I don't yet know how to solve it. Client-side navigation is dependent of hashing. So loading this page, when done with client-side navigation, becomes /_next/data/2ps5rE-K6E39AoF4G6G-0/en/plog.json (no, I don't know how that hashed URL is determined). But if a new deployment happens, the new URL becomes /_next/data/UhK9ANa6t5p5oFg3LZ5dy/en/plog.json so you end up with a 404 because you started on a page based on an old JavaScript bundle, that is now invalid.

Thankfully, NextJS handles it quite gracefully by throwing an error on the 404 so it proceeds with a regular link redirect which takes you away from the old page.

Client-side navigation still sucks. Kinda.

Next has a built-in <Link> component that you use like this:

import Link from "next/link";

...

<Link href={"/plog/" + post.oid}>
  {post.title}
</Link>

Now, clicking any of those links will automatically enable client-side routing. Thankfully, it takes care of preloading the necessary JavaScript (and CSS) simply by hovering over the link, so that when you eventually click it just needs to do an XHR request to get the JSON necessary to be able to render the page within the loaded app (and then do the pushState stuff to change the URL accordingly).

It sounds good in theory but it kinda sucks because unless you have a really good Internet connection (or could be you hit upon a CDN-cold URL), nothing happens when you click. This isn't NextJS's fault, but I wonder if it's actually horribly for users.

Yes, it sucks that a user clicks something but nothing happens. (I think it would be better if it was a button-press and not a link because buttons feel more like an app whereas links have deeply ingrained UX expectations). But most of the time, it's honestly very fast and when it works it's a nice experience. It's a great piece of functionality for more app'y sites, but less good for websites whose most of the traffic comes from direct links or Google searches.

NextJS has built-in critical CSS optimization

Critical inline CSS is critical (pun intended) for web performance. Especially on my poor site where I depend on a bloated (and now ancient) CSS framework called Semantic-UI. Without inline CSS, the minified CSS file would become over 200KB.

In NextJS, to enable inline critical CSS loading you just need to add this to your next.config.js:

    experimental: { optimizeCss: true },

and you have to add critters to your package.json. I've found some bugs with it but nothing I can't work around.

Conclusion and what's next

I'm very familiar and experienced with React but NextJS is new to me. I've managed to miss it all these years. Until now. So there's still a lot to learn. With other frameworks, I've always been comfortable that I don't actually understand how Webpack and Babel work (internally) but at least I understood when and how I was calling/depending on it. Now, with NextJS there's a lot of abstracted magic that I don't quite understand. It's hard to let go of that. It's hard to get powerful tools that are complex and created by large groups of people and understand it all too. If you're desperate to understand exactly how something works, you inevitably have to scale back the amount of stuff you're leveraging. (Note, it might be different if it's absolute core to what you do for work and hack on for 8 hours a day)

The JavaScript bundles in NextJS lazy-load quite decently but it's definitely more bloat than it needs to be. It's up to me to fix it, partially, because much of the JS code on my site is for things that technically can wait such as the interactive commenting form and the auto-complete search.

But here's the rub; my site is not an app. Most traffic comes from people doing a Google search, clicking on my page, and then bugger off. It's quite static that way and who am I to assume that they'll stay and click around and reuse all that loaded JavaScript code.

With that said; I'm going to start an experiment to rewrite the site again in Remix.

UPDATE Feb 4, 2022 Yes, as commenter Matthias pointed out, you can let JavaScript implicitly make the conversion of boolean minus boolean to end number a number of -1, 0, or 1.

Original but edited blog post below...

---

Imagine you have an array like this:

const items = [
  { num: 'one', labels: [] },
  { num: 'two', labels: ['foo'] },
  { num: 'three', labels: ['bar'] },
  { num: 'four', labels: ['foo'] },
  { num: 'five', labels: [] },
];

What you want, is to sort them in a way that all those entries that have a label foo come first, but you don't want to "disturb" the existing order. Essentially you want this to be the end result:

{ num: 'two', labels: ['foo'] },
{ num: 'four', labels: ['foo'] },

{ num: 'one', labels: [] },
{ num: 'three', labels: ['bar'] },
{ num: 'five', labels: [] },

Here's a way to do that:

items.sort(
  (itemA, itemB) =>
    itemB.labels.includes('foo') - itemA.labels.includes('foo')
);
console.log(items);

And the outcome is:

[
  { num: 'two', labels: [ 'foo' ] },
  { num: 'four', labels: [ 'foo' ] },
  { num: 'one', labels: [] },
  { num: 'three', labels: [ 'bar' ] },
  { num: 'five', labels: [] }
]

The simple trick is to turn then test operation into a number (0 or 1) and you can do that with Number.

Boolean minus boolean is operator overloaded to become an integer. From the Node repl...

> true - false
1
> false - true
-1
> false - false
0
> true - true
0

At work, we use Brotli (using the Node builtin zlib) to compress these large .json files to .json.br files. When using zlib.brotliCompress you can set options to override the quality number. Here's an example of it at quality 6:

import { promisify } from 'util'
import zlib from 'zlib'
const brotliCompress = promisify(zlib.brotliCompress)

const options = {
  params: {
    [zlib.constants.BROTLI_PARAM_MODE]: zlib.constants.BROTLI_MODE_TEXT,
    [zlib.constants.BROTLI_PARAM_QUALITY]: 6,
  },
}

export async function compress(data) {
  return brotliCompress(data, options)
}

But what if you mess with that number. Surely, the files will become smaller, but at what cost? Well, I wrote a Node script that measured how long it would take to compress 6 large (~25MB each) .json file synchronously. Then, I put them into a Google spreadsheet and voila:

Size

Time

Miles away from rocket science but I thought it was cute to visualize as a way of understanding the quality option.

I just have to write this down because that's the rule; if I find myself googling something basic like this more than once, it's worth blogging about.

Suppose you have a string and you want to pad with empty spaces. You have 2 options:

>>> s = "peter"
>>> s.ljust(10)
'peter     '
>>> f"{s:<10}"
'peter     '

The f-string notation is often more convenient because it can be combined with other formatting directives.
But, suppose the number 10 isn't hardcoded like that. Suppose it's a variable:

>>> s = "peter"
>>> width = 11
>>> s.ljust(width)
'peter      '
>>> f"{s:<width}"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Invalid format specifier

Well, the way you need to do it with f-string formatting, when it's a variable like that is this syntax:

>>> f"{s:<{width}}"
'peter      '

You can't batch-add/bulk-insert documents in the Firebase Web SDK. But you can with the Firebase Admin Node SDK. Like, in a Firebase Cloud Function. Here's an example of how to do that:

const firestore = admin.firestore();
let batch = firestore.batch();
let counter = 0;
let totalCounter = 0;
const promises = [];
for (const thing of MANY_MANY_THINGS) {
  counter++;
  const docRef = firestore.collection("MY_COLLECTION").doc();
  batch.set(docRef, {
    foo: thing.foo,
    bar: thing.bar,
    favNumber: 0,
  });
  counter++;
  if (counter >= 500) {
    console.log(`Committing batch of ${counter}`);
    promises.push(batch.commit());
    totalCounter += counter;
    counter = 0;
    batch = firestore.batch();
  }
}
if (counter) {
  console.log(`Committing batch of ${counter}`);
  promises.push(batch.commit());
  totalCounter += counter;
}
await Promise.all(promises);
console.log(`Committed total of ${totalCounter}`);

I'm using this in a Cloud HTTP function where I can submit a large amount of data and have each one fill up a collection.

Starting today, I'm a Hubber. Meaning, I work for GitHub. I'll be joining the GitHub Docs team to help technical writers document all the various products that GitHub have. Since I haven't actually started coding anything yet, I don't want to claim I know how it works or exactly what I'll be working but on, but I do know that the site at hand is docs.github.com and I've previously taken a lot of inspiration from this site when building the MDN rewrite.

If you are a Hubber, too, and reading this; Hi! Let's be friends! I'm a friendly guy. Please ping me and say hi.
I'll be working from home here in Mount Pleasant, South Carolina. I have 2 young kids; Tucker and Charlotte, who mean the world to me. And my backbone-of-life wife Ashley. My hobbies are coding, swimming, golf, and cooking.

I find this so fiddly! I love TypeScript and will continue to use it if there's a choice. But I just wanted to write a simple async function wrapper and I had to Google for it and nothing was quite right. Here's my simple solution, as an example:

function wrappedAsyncFunction<T>(
    fn: (...args: any[]) => Promise<T>
  ): (...args: any[]) => Promise<T> {
    return async function(...args: any[]) {
      console.time("Took");
      try {
        return await fn(...args);
      } catch(error) {
        console.warn("FYI, an error happened:", error);
        throw error;
      } finally {
        console.timeEnd("Took");
      }

    };
  }

Here's a Playground demo

What I use it for is to wrap my Firebase Cloud Functions so that if any error happens, I can send that error to Rollbar. In particular, here's an example of it in use:

diff --git a/functions/src/cleanup-thumbnails.ts b/functions/src/cleanup-thumbnails.ts
index 46bdb34..a3e8d54 100644
--- a/functions/src/cleanup-thumbnails.ts
+++ b/functions/src/cleanup-thumbnails.ts
@@ -2,6 +2,8 @@ import * as admin from "firebase-admin";
 import * as functions from "firebase-functions";
 import { logger } from "firebase-functions";

+import { wrappedLogError } from "./rollbar-logger";
+
 const OLD_DAYS = 30 * 6; // 6 months
 // const ADDITIONAL_DAYS_BACK = 5;
 // const ADDITIONAL_DAYS_BACK = 15;
@@ -9,7 +11,7 @@ const PREFIX = "thumbnails";

 export const scheduledCleanupThumbnails = functions.pubsub
   .schedule("every 24 hours")
-  .onRun(async () => {
+  .onRun(wrappedLogError(async () => {
     logger.debug("Running scheduledCleanupThumbnails");

...

And my wrappedLogError looks like this:

export function wrappedLogError<T>(
  fn: (...args: any[]) => Promise<T>
): (...args: any[]) => Promise<T> {
  return async function(...args: any[]) {
    try {
      return await fn(...args);
    } catch (error) {
      logError(error);
      throw error;
    }
  };
}

I'm not sure it's the best or correct way to do it, but it seems to work. Perhaps there's a more correct solution but for now I'll ship this because it seems to work fine.

Today I launched a really cool new feature to That's Groce!: Ability to upload photos of ingredients and have the food words automatically suggested to be added to your shopping list.

It's best explained with this 26-second video.

The general idea

The idea is that you know you're going to cook that Vegetarian Curry Lasagna on page 123 in Jamie's Summer Cookbook. Either you read the ingredients and type in each ingredient you're going to need to buy (because you know what's in your pantry and fridge) or just take a photo of the whole ingredient listing. Now, when you're in the store and wonder: "I remember I need red peppers, but how many was it again?!".
But not only that, once you've taken a photo of the list of ingredients, it can help you populate your shopping list.

How it works in That's Groce! is that your photo is turned into a block of text, and from that text certain "food words" are extracted and all else is ignored. The food words are based on a database of 3,600+ English food words that I've gathered and also manually curated. There are lots of caveats. The 3,600+ food words are not perfect and there are surprisingly many combinations and plural vs. singular that can make the list incomplete. But, that's where you can help! If you find that there's a word it correctly scanned but didn't suggest, you can type in your own suggestions for everyone to benefit from. If you type in something it didn't manage to spot, I'll review that and add it to the global database.

Food-word recognition

The word recognition is done using Google Cloud Vision API which is a powerful machine-learning-based service from Google Cloud that is stunningly accurate. Tidy camera photos of cookbooks with good light is nearly perfect, but it can also do an impressive job with photos of handwritten recipes, like this:

One thing you will find is that it's often hard to only take a photo of the actual list of ingredients. Often, a chunk of the cooking instructions or the recipe "back story" gets into the photo frame. These words aren't actually ingredients and can lead to surprising food word suggestions that you definitely don't need to buy. It's not perfect but at least you'll have a visual memory of what you're cooking as you're standing there in the grocery store.

Options

Understandably, it's nearly impossible for the app to know what you have in your pantry/fridge/freezer/spice rack. But a lot of recipes spell out exactly everything you need, not for buying, but for cooking. E.g. 1 teaspoon salt. But salt (or pepper or sugar or butter) is the kind of foodstuff you probably always have at home. So then it doesn't make sense to suggest that you put "salt" on the shopping list. To solve for that, you can override your own preferred options of special keywords. For example, I've added "salt", "pepper", "sugar, "table salt", "black pepper" as words that can always be ignored. You can also train the system to set up aliases. For example, a lot of recipes call for "lemon zest" but what you actually purchase is a lemon and then grate it yourself on the grater. So you can add special keywords that act as aliases for other words. Here's one example:

Help out!

Over the last couple of days, I've been snapping lots of photos from lots of different cookbooks, and every time I begin to think the database is getting good, I stumble on some new food word. For example, I recently tested a photo of a recipe that called for "Jackfruit". What even is that?! Anyway, it is inevitable that certain words are missing. But that's where we can help each other. If you test it out and you notice that it correctly scanned the text but the word wasn't suggested, click the "Suggest" button and type in your suggestions. Together we can, one food word at a time, eradicate misses.

There's still some work to be done to make the database even stronger by setting up clever aliases for everyone. For example, a lot of recipes call for "(some number) cloves garlic" but it can easily get confused for the spice "cloves" and the root vegetable "garlic". So, perhaps we can train it to recognize "cloves garlic" to actually just mean "garlic".

Also, the database is currently only in (primarily American) English. The platform would support other languages but I would definitely need a hand to seed it with more words in other languages.

Try it out

If you haven't set up an account yet (it's still free!), to test it, you can go to https://thatsgroce.web.app, click "Get started without signing in", go into your newly created shopping list, and press the "Photos" button. Try uploading some snapped photos from your cookbooks. Please please let me know what you think!

To do this in Python:

def print_person(name="peter", dob=1979):
    print(f"name={name}\tdob={dob}")


print_person() 
# prints: name=peter dob=1979

print_person(name="Tucker")
# prints: name=Tucker    dob=1979

print_person(dob=2013)
# prints: name=peter dob=2013

print_person(sex="boy")
# TypeError: print_person() got an unexpected keyword argument 'sex'

...in TypeScript:

function printPerson({
  name = "peter",
  dob = 1979
}: { name?: string; dob?: number } = {}) {
  console.log(`name=${name}\tdob=${dob}`);
}

printPerson();
// prints: name=peter    dob=1979

printPerson({});
// prints: name=peter    dob=1979

printPerson({ name: "Tucker" });
// prints: name=Tucker   dob=1979

printPerson({ dob: 2013 });
// prints: name=peter    dob=2013


printPerson({ gender: "boy" })
// Error: Object literal may only specify known properties, and 'gender' 

Here's a Playground copy of it.

It's not a perfect "transpose" across the two languages but it's sufficiently similar.
The trick is that last = {} at the end of the function signature in TypeScript which makes it possible to omit keys in the passed-in object.

By the way, the pure JavaScript version of this is:

function printPerson({ name = "peter", dob = 1979 } = {}) {
  console.log(`name=${name}\tdob=${dob}`);
}

But, unlike Python and TypeScript, you get no warnings or errors if you'd do printPerson({ gender: "boy" }); with the JavaScript version.

Because I always forget, if you're using certbot to create certs for your Nginx server, you'll need to it up so it works on HTTP as well as HTTPS. But once you're done, you're going to want all HTTP traffic to redirect to HTTPS. The correct syntax is:

server {
    server_name mydomain.example.com;
    include /etc/nginx/snippets/letsencrypt-acme-challenge.conf;
    location / {
      return 301 https://mydomain.example.com$request_uri;
    }
}

And that letsencrypt-acme-challenge.conf looks like this (code comments stripped):

location ^~ /.well-known/acme-challenge/ {
    default_type "text/plain";
    root         /var/www/html;
    break;
}
location = /.well-known/acme-challenge/ {
    return 404;
}

This way, a GET request for http://mydomain.example.com/.well-known/acme-challenge/test.html will be 200 OK if there's a file called /var/www/html/.well-known/acme-challenge/test.html. And http://mydomain.example.com/.well-known/acme-challenge/does-not-exist.html will 404 Not Found.

But all and any other GET request will redirect. E.g. http://mydomain.example.com/whatever -- 301 Moved Permanently --> https://mydomain.example.com/whatever.