Implement cache

[gajus]

A simple cache using LRU strategy (e.g. https://www.npmjs.com/package/lru-cache).

[gajus]

#81

[morganpackard]

I've been wondering about caching in gulp-babel myself. Is the fancy cache and the hash stuff really necessary? My much more naive cache I just whipped up seems to work great

        // var cache = {}  defined at the top of the file

        var contents = file.contents.toString();

        cache[file.path] = cache[file.path] || {
            lastUncompiled: "",
            lastCompiled: ""
        }

        if(cache[file.path].lastUncompiled !== contents){
            cache[file.path].lastUncompiled = contents;
            cache[file.path].lastCompiled = babel.transform(contents, fileOpts)
        }

        var res = cache[file.path].lastCompiled;

[gajus]

Your solution is effectively leaking memory.

On Mar 4, 2016, at 04:13, Morgan Packard [email protected] wrote:

I've been wondering about caching in gulp-babel myself. Is the fancy cache and the hash stuff really necessary? My much more naive cache I just whipped up seems to work great

    // var cache = {}  defined at the top of the file

    var contents = file.contents.toString();

    cache[file.path] = cache[file.path] || {
        lastUncompiled: "",
        lastCompiled: ""
    }

    if(cache[file.path].lastUncompiled !== contents){
        cache[file.path].lastUncompiled = contents;
        cache[file.path].lastCompiled = babel.transform(contents, fileOpts)
    }

    var res = cache[file.path].lastCompiled;

—
Reply to this email directly or view it on GitHub.

[gajus]

ps I have released gulp-babel2 to test LRU. Using it until something about cache is done in gulb-babel. It saves a lot of time on large project (difference between 7 seconds and few ms)

On Mar 4, 2016, at 04:13, Morgan Packard [email protected] wrote:

I've been wondering about caching in gulp-babel myself. Is the fancy cache and the hash stuff really necessary? My much more naive cache I just whipped up seems to work great

    // var cache = {}  defined at the top of the file

    var contents = file.contents.toString();

    cache[file.path] = cache[file.path] || {
        lastUncompiled: "",
        lastCompiled: ""
    }

    if(cache[file.path].lastUncompiled !== contents){
        cache[file.path].lastUncompiled = contents;
        cache[file.path].lastCompiled = babel.transform(contents, fileOpts)
    }

    var res = cache[file.path].lastCompiled;

—
Reply to this email directly or view it on GitHub.

[morganpackard]

@gajus where is this gulp-babel2 located? Github link?

• My naive solution has an unlimited cache size, but it only grows as long as you pass new files. The cache will be roughly twice as big as the project source code.
• What's the benefit of using the sha1 hash for comparison, rather than just storing the complete file and comparing that? Presumably hashing is going to slow the algorithm. Sure, your cache will use roughly half as much memory if you use a hash, rather than the complete source code to see if recompiling is necessary, but is memory footprint really an issue?

[gajus]

My naive solution has an unlimited cache size, but it only grows as long as you pass new files. The cache will be roughly twice as big as the project source code.

https://github.com/gajus/gulp-babel/blob/issue-80/index.js

What's the benefit of using the sha1 hash for comparison, rather than just storing the complete file and comparing that? Presumably hashing is going to slow the algorithm. Sure, your cache will use roughly half as much memory if you use a hash, rather than the complete source code to see if recompiling is necessary, but is memory footprint really an issue?

With codebases as large as thousands of files, memory footprint is a very big issue.

[morganpackard]

OK! Thanks for your responses. I hope an official caching gulp-babel gets
released soon.

On Fri, Mar 4, 2016 at 10:48 AM, Gajus Kuizinas [email protected]
wrote:

My naive solution has an unlimited cache size, but it only grows as long
as you pass new files. The cache will be roughly twice as big as the
project source code.

https://github.com/gajus/gulp-babel

What's the benefit of using the sha1 hash for comparison, rather than just
storing the complete file and comparing that? Presumably hashing is going
to slow the algorithm. Sure, your cache will use roughly half as much
memory if you use a hash, rather than the complete source code to see if
recompiling is necessary, but is memory footprint really an issue?

With codebases as large as thousands of files, memory footprint is a very
big issue.

—
Reply to this email directly or view it on GitHub
#80 (comment).

Morgan Packard
web: www.morganpackard.com
twitter: @morganpackard
phone: (720) 891-0122

[loganfsmyth]

I'm curious, how does this compare to using something like gulp-cached to simply skip passing the file through to gulp-babel at all? It's not immediately obvious to me what integrating this into the plugin provides that couldn't be accomplished with composable gulp tasks rather than building it into the plugin.

[morganpackard]

Based on my reading of gulp-cached, it will not pass a file downstream if it's cached. Useful for linting or other validation, but not so useful for my build system, where I transpile and then concatenate all files in to one big one. To use a separate gulp plugin to cache babel results you'd need something that can still deliver the cached results downstream. Maybe a twopart usage pattern similar to gulp-sourcemaps would be idea. Something like this:

                gulp.src([
                source_folder + '/**/*.js'
                .pipe(cache.init())
                .pipe(babel)
                .pipe(cache.write())
                .pipe(concat)
                ;

The call to cache.init() would create an object that would intercept previously compiled files with no changes. Cache.write() would output those previously compiled files. If a version of a file doesn't exist in the cache, it passes it on to babel. The tricky thing is that it would have to then cache the results, so would have to remember what the input to babel was and successfully update the cache key, and then wait for the output. I'm too green to gulp to know how difficult this would be.

[morganpackard]

Looks like gulp-remember paired with gulp-cached may do more or less what I described above.

[michaelBenin]

I've achieved this with gulp-changed.

[hzoo]

Closing this issue as it's been inactive for several months. Please feel free to open a new issue or leave a comment asking to reopen this issue. Thanks! ✌️

If you just have an usage issue or want help, I would recommend checking out our slack community or StackOverflow.