A blog and website by Peter Bengtsson

Filtered home page!
Currently only showing blog entries under thecategory: That's Groce!. Clear filter

Preemptive grocery shopping list item suggestions

That's Groce!

tl;dr; Preemptive suggestions are grocery shopping list item suggestions that appear before you have typed anything. It's like it can read your mind (based on AI).

In the next version of That's Groce! it will include a powerful feature I've long wanted to add but a lot of work and testing had to go into it: Preemptive suggestions.

Basically, That's Groce! knows which shopping list items you add often. It also knows when you last added it. It knows that you haven't added it to the list right now. And it knows which are more common than others. That's called the "Popularity contest".

The suggestions now appear underneath the add-new-item form. They appear where search suggestions usually appear. Before it used to be blank there until you typed something. For example, typing "b" would, in my household's case, show: "Blueberries", "Bananas", "Spreadable butter", etc.

Suggest-as-you-type suggestions

The way preemptive suggestion work is ultimately quite simple, but a mouthful:

  1. Exclude all items that are currently on your active list (to-shop and done).
  2. Exclude those that haven't been added at least 2 times in the last 90 days.
  3. Look at when it was last added. Look up its (rolling median) frequency. Exclude those that have been added more recently than their frequency.
  4. Sort by most popular (a.k.a. most frequently added)
  5. Show only the top 4 (for space on mobile devices)

As an example; we often buy "Bananas". Let's assume its history of adding it to the list is:

  1. Sunday, Jan 30
  2. Tuesday, Feb 8
  3. Monday, Feb 14
  4. Sunday, Feb 20

The difference between those days, on average is about 6 days (for the sake of example). Now, if today's date is Monday, Feb 28, that means it was 8 days since it was last added (Sunday, Feb 20). Therefore, you add to the list this more frequently than it was last added! Therefore, include it as a preemptive suggestion.

The idea, and hope, of this, is to remove the labor of having to type "co" to make the "Coffee ☕️" suggestion appear. How about just making it appear simply by thinking about coffee? Or better, even before you think of coffee, so you don't forget to buy some more!

The new feature has already been released but it might take a while to appear on your account. And if you're a new user don't expect this to show up at all until you've used the app for a while. It needs to learn from your shopping habits. But have patience! You'll love it!

Preview of that it can look like


Note that "Milk" was appearing because it's a frequent item in our household, but it's no longer one of the suggestions because it's already been added to the list. And "Eggs" is also a very common item, but it's not appearing either because it was added (and checked off) in the last shopping trip.

Please post a comment if you have thoughts or questions.

From photo of ingredients, to your shopping list

Web development, That's Groce!, Firebase

Today I launched a really cool new feature to That's Groce!: Ability to upload photos of ingredients and have the food words automatically suggested to be added to your shopping list.

It's best explained with this 26-second video.

The general idea

Food words found
The idea is that you know you're going to cook that Vegetarian Curry Lasagna on page 123 in Jamie's Summer Cookbook. Either you read the ingredients and type in each ingredient you're going to need to buy (because you know what's in your pantry and fridge) or just take a photo of the whole ingredient listing. Now, when you're in the store and wonder: "I remember I need red peppers, but how many was it again?!".
But not only that, once you've taken a photo of the list of ingredients, it can help you populate your shopping list.

How it works in That's Groce! is that your photo is turned into a block of text, and from that text certain "food words" are extracted and all else is ignored. The food words are based on a database of 3,600+ English food words that I've gathered and also manually curated. There are lots of caveats. The 3,600+ food words are not perfect and there are surprisingly many combinations and plural vs. singular that can make the list incomplete. But, that's where you can help! If you find that there's a word it correctly scanned but didn't suggest, you can type in your own suggestions for everyone to benefit from. If you type in something it didn't manage to spot, I'll review that and add it to the global database.

Food-word recognition

The word recognition is done using Google Cloud Vision API which is a powerful machine-learning-based service from Google Cloud that is stunningly accurate. Tidy camera photos of cookbooks with good light is nearly perfect, but it can also do an impressive job with photos of handwritten recipes, like this:

Handwritten recipe photo

One thing you will find is that it's often hard to only take a photo of the actual list of ingredients. Often, a chunk of the cooking instructions or the recipe "back story" gets into the photo frame. These words aren't actually ingredients and can lead to surprising food word suggestions that you definitely don't need to buy. It's not perfect but at least you'll have a visual memory of what you're cooking as you're standing there in the grocery store.


Understandably, it's nearly impossible for the app to know what you have in your pantry/fridge/freezer/spice rack. But a lot of recipes spell out exactly everything you need, not for buying, but for cooking. E.g. 1 teaspoon salt. But salt (or pepper or sugar or butter) is the kind of foodstuff you probably always have at home. So then it doesn't make sense to suggest that you put "salt" on the shopping list. To solve for that, you can override your own preferred options of special keywords. For example, I've added "salt", "pepper", "sugar, "table salt", "black pepper" as words that can always be ignored. You can also train the system to set up aliases. For example, a lot of recipes call for "lemon zest" but what you actually purchase is a lemon and then grate it yourself on the grater. So you can add special keywords that act as aliases for other words. Here's one example:


Help out!

Foodwords database sample
Over the last couple of days, I've been snapping lots of photos from lots of different cookbooks, and every time I begin to think the database is getting good, I stumble on some new food word. For example, I recently tested a photo of a recipe that called for "Jackfruit". What even is that?! Anyway, it is inevitable that certain words are missing. But that's where we can help each other. If you test it out and you notice that it correctly scanned the text but the word wasn't suggested, click the "Suggest" button and type in your suggestions. Together we can, one food word at a time, eradicate misses.

There's still some work to be done to make the database even stronger by setting up clever aliases for everyone. For example, a lot of recipes call for "(some number) cloves garlic" but it can easily get confused for the spice "cloves" and the root vegetable "garlic". So, perhaps we can train it to recognize "cloves garlic" to actually just mean "garlic".

Also, the database is currently only in (primarily American) English. The platform would support other languages but I would definitely need a hand to seed it with more words in other languages.

Try it out

If you haven't set up an account yet (it's still free!), to test it, you can go to, click "Get started without signing in", go into your newly created shopping list, and press the "Photos" button. Try uploading some snapped photos from your cookbooks. Please please let me know what you think!

Please post a comment if you have thoughts or questions.

downloadAndResize - Firebase Cloud Function to serve thumbnails

Web development, That's Groce!, Node, JavaScript

UPDATE 2020-12-30

With sharp after you've loaded the image (sharp(contents)) make sure to add .rotate() so it automatically rotates the image correctly based on EXIF data.

UPDATE 2020-12-13

I discovered that sharp is much better than jimp. It's order of maginitude faster. And it's actually what the Firebase Resize Images extension uses. Code updated below.

I have a Firebase app that uses the Firebase Cloud Storage to upload images. But now I need thumbnails. So I wrote a cloud function that can generate thumbnails on-the-fly.

There's a Firebase Extension called Resize Images which is nicely done but I just don't like that strategy. At least not for my app. Firstly, I'm forced to pick the right size(s) for thumbnails and I can't really go back on that. If I pick 50x50, 1000x1000 as my sizes, and depend on that in the app, and then realize that I actually want it to be 150x150, 500x500 then I'm quite stuck.

Instead, I want to pick any thumbnail sizes dynamically. One option would be a third-party service like imgix, CloudImage, or Cloudinary but these are not free and besides, I'll need to figure out how to upload the images there. There are other Open Source options like picfit which you install yourself but that's not an attractive option with its implicit complexity for a side-project. I want to stay in the Google Cloud. Another option would be this AppEngine function by Albert Chen which looks nice but then I need to figure out the access control between that and my Firebase Cloud Storage. Also, added complexity.

As part of your app initialization in Firebase, it automatically has access to the appropriate storage bucket. If I do:

const storageRef = storage.ref();
uploadTask = storageRef.child('images/photo.jpg').put(file, metadata);
... the Firebase app, it means I can do:

      .then((downloadData) => {
        const contents = downloadData[0]; my cloud function and it just works!

And to do the resizing I use Jimp which is TypeScript aware and easy to use. Now, remember this isn't perfect or mature but it works. It solves my needs and perhaps it will solve your needs too. Or, at least it might be a good start for your application that you can build on. Here's the function (in functions/src/index.ts):

interface StorageErrorType extends Error {
  code: number;

const codeToErrorMap: Map<number, string> = new Map();
codeToErrorMap.set(404, "not found");
codeToErrorMap.set(403, "forbidden");
codeToErrorMap.set(401, "unauthenticated");

export const downloadAndResize = functions
  .runWith({ memory: "1GB" })
  .https.onRequest(async (req, res) => {
    const imagePath = req.query.image || "";
    if (!imagePath) {
      res.status(400).send("missing 'image'");
    if (typeof imagePath !== "string") {
      res.status(400).send("can only be one 'image'");
    const widthString = req.query.width || "";
    if (!widthString || typeof widthString !== "string") {
      res.status(400).send("missing 'width' or not a single string");
    const extension = imagePath.toLowerCase().split(".").slice(-1)[0];
    if (!["jpg", "png", "jpeg"].includes(extension)) {
      res.status(400).send(`invalid extension (${extension})`);
    let width = 0;
    try {
      width = parseInt(widthString);
      if (width < 0) {
        throw new Error("too small");
      if (width > 1000) {
        throw new Error("too big");
    } catch (error) {
      res.status(400).send(`width invalid (${error.toString()}`);

      .then((downloadData) => {
        const contents = downloadData[0];
          `downloadAndResize (${JSON.stringify({
          })}) downloadData.length=${humanFileSize(contents.length)}\n`

        const contentType = extension === "png" ? "image/png" : "image/jpeg";
          .then((buffer) => {
            res.setHeader("content-type", contentType);
            // TODO increase some day
            res.setHeader("cache-control", `public,max-age=${60 * 60 * 24}`);
          .catch((error: Error) => {
            console.error(`Error reading in with sharp: ${error.toString()}`);
              .send(`Unable to read in image: ${error.toString()}`);
      .catch((error: StorageErrorType) => {
        if (error.code && codeToErrorMap.has(error.code)) {
        } else {

function humanFileSize(size: number): string {
  if (size < 1024) return `${size} B`;
  const i = Math.floor(Math.log(size) / Math.log(1024));
  const num = size / Math.pow(1024, i);
  const round = Math.round(num);
  const numStr: string | number =
    round < 10 ? num.toFixed(2) : round < 100 ? num.toFixed(1) : round;
  return `${numStr} ${"KMGTPEZY"[i - 1]}B`;

Here's what a sample URL looks like.

I hope it helps!

I think the next thing for me to consider is to extend this so it uploads the thumbnail back and uses the getDownloadURL() of the created thumbnail as a redirect instead. It would be transparent to the app but saves on repeated views. That'd be a good optimization.

Please post a comment if you have thoughts or questions.

Default food shopping items isn't for everyone

That's Groce!

An old friend of mine, from The Netherlands, contacted me about That's Groce! because the default food word suggestions simple don't make sense to him since they're all in English. American English too, I suspect.

Let's back up a bit. Here's a picture of some of the ~100 default food words that are meant to help you when you start out:

Some of the default food word suggestions

The idea is that until you've started using That's Groce! it'll take a while to get your patterns settled. The assumption is that for most people it makes sense to have some sensible default suggestions. This way, as you start typing ch it can suggest: "Cherries 🍒", "Chilis 🌶", "Chicken 🍗", etc.

Now, you can turn this functionality off. The option looks like this:

Disable default suggestions

What's cool about this new feature is that the feature was borne from feedback. My friend used the "Feedback" form and is actually the first one to ever do so. Thanks Ivo!

Feedback option

Please post a comment if you have thoughts or questions.

Popularity contest for your grocery list

Web development, Mobile, That's Groce!

tl;dr; Up until recently, when you started to type a new entry in your That's Groce shopping list, the suggestions that would appear weren't sorted intelligently. Now they are. They're sorted by popularity.

The whole point with the suggestions that appear is to make it easier for you to not have to type the rest. The first factor that decides which should appear is simply based on what you've typed so far. If you started typing ch we can suggest:

  • Cherry tomatoes
  • Chocolate chips
  • Mini chocolate chips
  • Rainbow chard
  • Goat cheese
  • Chickpeas
  • etc.

They all contain ch in some form (starting of words only). But space is limited and you can't show every suggestion. So, if you're going to cap it to only show, say, 4 suggestions; which ones should you show first?
I think the solution is to do it by frequency. I.e. items you often put onto the list.

How to calculate the frequency

The way That's Groce now does it is that it knows the exact times a certain item was added to the list. It then takes that list and applies the following algorithm:

For each item...

  1. Discard the dates older than 3 months
  2. Discard any duplicates from clusters (e.g. you accidentally added it and removed it and added it again within minutes)
  3. Calculate the distance (in seconds) between each add
  4. From the last 4 times it was added, take the median value of the distance between

So the frequency becomes a number of seconds. It should feel somewhat realistic. In my family, it actually checks out. We buy bananas every week but sometimes slightly more often than that and in our case, the number comes to ~6 days.

The results

Before sorting by popularity
Before sorting by popularity

After sorting by popularity
After sorting by popularity

Great! The chances of appreciating and picking one of the suggestions is greater if it's more likely to be what you were looking for. And things that have been added frequently in the past are more likely to be added again.

How to debug this

There's now a new page called the "Popularity contest". You get to it from the "List options" button in the upper right-hand corner. On its own, it's fairly "useless" because it just lists them. But it's nice to get a feeling for what your family most frequently add to the list. A lot more can probably be done to this page but for now, it really helps to back up the understanding of how the suggestions are sorted when you're adding new entries.

Popularity contest

If you look carefully at my screenshot here you'll notice two little bugs. There are actually two different entries for "Lemon 🍋" and that was from the early days when that could happen.
Also, another bug is that there's one entry called "Bananas" and one called "Bananas 🍌" which is also something that's being fixed in the latest release. My own family's list predates those fixes.

Hope it helps!

Please post a comment if you have thoughts or questions.

That's Groce!

Web development, Mobile, Preact, That's Groce!, Firebase

tl;dr That's Groce! is: A mobile web app to help families do grocery shopping and meal planning. Developed out of necessity by a family (Peter and Ashley) and used daily in their home.

Hopefully, the About page explains what it does.

Sample list
Screenshot of a sample list

The backstory

We used to use Wunderlist, but that stopped working. Next, we tried Cozi and that worked for a while but it was buggy and annoying in so many ways. Finally, we gave up and decided to build our own. Exactly how we need it to be, as efficient as possible.

We also tried a couple of regular to-do list apps where you can have shared accounts but we wanted something perfectly tailored towards the specific needs of family grocery shopping (and meal planning). That's how That's Groce! was born.

The killer features

The about page does a good job of listing the killer features but let's emphasize it one more time.

  • Free and simple.
  • Is really smart about suggestions and auto-complete for speedy entry.
  • Works offline (our usual grocery store has no reception unless you brave onto their WiFi).
  • Is real-time, so every other device updates immediately as you update or add something on your device.
  • Shared list; you can have your own list but you can invite co-owners.
  • You can sort your grocery list in the same order you usually walk through your grocery store.

It's not an app store app

Saved as Home screen app

You won't find it on the Apple App store. It's a web app that's been tailored to work well in mobile web browsers (iOS Safari) and you can use the "Add to Home screen" so it looks and acts like a regular app.
It would be nice to try to make it a regular native mobile app but that takes significant time which is hard to find but certainly something to aspire to if it can be done in a nice way.

"Really smart about suggestions"

What does that killer feature mean? (At the time of writing (Oct 2020), it isn't launched yet but the pieces are coming together.) Are there certain stables you buy recurringly? Like milk or bananas or Cheerios. If the app can start to see a pattern of commonly added items, it can suggest it immediately so when you're making your list on Monday morning, you just need to tap to add those.

Another important thing is that as you type, it can suggest many things based on the first or couple of characters you type in, but you can't suggest every single possible word so which one should you suggest first?
The way That's Groce! works is that it learns based on the number of times and how recently you add something to your list. As of today, look what happens when I type a on my list:

Suggestions based on typing 'a'
When I type a it suggests things that start with "A" but based on frequency.

The more you use it, the better the suggestions get.

Also, to get you started, over 100 items are preloaded as good suggestions but that's just to get you up and running. Once your family starts to use it, your own suggestions get better and better over time.

"Same order you usually walk through your grocery store"

This was important to us because we found we walk through the aisles in pretty much the same way. Every time. When you walk in you have your produce (veggies first, then fruit, then salad stuff) on the right. Then baked goods and deli. Then meats and alcohol. Etc. So if you can group your items based on these descriptions you can be really efficient with your list and it becomes a lot easier to cross off sections of the store and not have to scroll up and down or having to walk back to pick up that pizza dough all the way back at the deli section.

For this to work, you need to type in groups for your items. But you can call them whatever you like. If you want to type "Aisle 1", "Aisle 2", "Dairy stuff" you can. It's all up to you. Keep in mind that it might feel like a bit of up-front work at first, and it is, but your list is learning so you essentially only have to do it once.

Don't be a slave to your list!

If you do decide to try it, keep one thing in mind: You're in control. You don't need to type in perfect descriptions, amounts, groups, and quantities. If you don't know how to spell "bee-ar-naise sauce", don't worry about it. It's your list. You can type whatever you want or need. A lot of to-do lists invite you with complex options to organize the hell out of your list items. Don't do that. Think of That's Groce! as a fridge post-it note that you and your partner keep in their pocket that automatically synchronizes.

You can help

We built this for ourselves but it's built in a way that any family can use it and hopefully also be better organized. But once you sign in you can submit feedback for suggestions. And if you're into coding, the whole app is Open Source so it's fairly easy to modify the code or even host it yourself if you wanted to:

Also, if you do try it and like it, please consider going to the Share the ❤️ page and, you know, share it with friends. Much appreciated!

Please post a comment if you have thoughts or questions.