NodeJS fs walk() or glob or fast-glob

31 August 2019   3 comments   JavaScript

It started with this:

function walk(directory, filepaths = []) {
    const files = fs.readdirSync(directory);
    for (let filename of files) {
        const filepath = path.join(directory, filename);
        if (fs.statSync(filepath).isDirectory()) {
            walk(filepath, filepaths);
        } else if (path.extname(filename) === '.md') {
            filepaths.push(filepath);
        }
    }
    return filepaths;
}

And you use it like this:

const foundFiles = walk(someDirectoryOfMine);
console.log(foundFiles.length);

I thought, perhaps it's faster or better to use glob. So I installed that.
Then I found, fast-glob which sounds faster. You use both in a synchronous way.

I have a directory with about 450 files, of which 320 of them are .md files. Let's compare:

walk: 10.212ms
glob: 37.492ms
fg: 14.200ms

I measured it using console.time like this:

console.time('walk');
const foundFiles = walk(someDirectoryOfMine);
console.timeEnd('walk');
console.log(foundFiles.length);

I suppose those packages have other fancier features but, I guess this just goes to show, keep it simple.

UPDATE June 2021

The origins of this blog post were that I need a simple function to find files on disk. Later, the requirements became a bit more complex so I needed something a bit more advanced. In shopping around I found fdir which, from testing, performed excellently and has a great API (and documentation). I would handsdown use that again.

Comments

Elliot

Nice article

Ruben

Your thesis is ambiguous. Simple is an overloaded term that means different things for different people and projects. If I'm writing a one-off tool that will be used and thrown away, for me "simple" means whatever is faster to write (i.e. glob or flast-glob). If I'm going to write that code many times for different types of file patterns, at some point I will be generalizing it so much that I will end up creating yet another version of glob, so using the original glob is simpler too.

In general, I believe that each dependency has to support its own weight in any project that will be maintained, so there are indeed cases in which writing the code yourself is strictly better than using a dependency, but I suspect it's not as common as you make it seem in this post.

Peter Bengtsson

I can definitely see how "keep it simple" can be interpreted as ambiguous.
What I *meant* to say is that; if you don't need the bells and whistles a third-party package offers, it's sometimes best, if applicable, to write your own.
Thanks for your comment!

Your email will never ever be published.

Related posts