Include zstd

[kristojorg]

Hi there,

I am attempting to use these during a parallel job execution. I would like to use actions/cache@v2 instead of upload-artifact for sharing my build and cypress dependencies between jobs. This is because it is both faster, and a simpler DX (I don't have to tar and then extract node_modules, etc). However, using the cache action in the container results in a different version being applied, which means my cache is always missed. According to this issue, it's because zstd is not available in the container. I've confirmed that that is true.

It would be very helpful to add it to the containers! Let me know what you think, thanks!

PS. This is my workflow:

name: Open Ebooks Integration with Cypress
# https://docs.github.com/en/actions/reference/events-that-trigger-workflows
on: [push]

env:
  GH_PACKAGE_AUTH_TOKEN: ${{ secrets.GH_PACKAGE_AUTH_TOKEN }}

jobs:
  install:
    name: Install and Build
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Use Node.js v14
        uses: actions/setup-node@v2
        with:
          node-version-file: ".nvmrc"
          cache: "npm"

      - name: Use NPM v8
        run: npm i -g npm@8

      - name: Install dependencies
        run: npm ci --audit=false

      - name: Is Axisnow package available?
        run: ls node_modules/@nypl-simplified-packages

      - name: Use Next.js build cache from prev workflows
        uses: actions/cache@v2
        with:
          path: ${{ github.workspace }}/.next/cache
          # Generate a new cache whenever packages or source files change.
          key: ${{ runner.os }}-nextjs-${{ hashFiles('**/package-lock.json') }}-${{ hashFiles('**.[jt]s', '**.[jt]sx') }}
          # If source files changed but packages didn't, rebuild from a prior cache.
          restore-keys: |
            ${{ runner.os }}-nextjs-${{ hashFiles('**/package-lock.json') }}-

      - name: Save full Next.js build to cache
        uses: actions/cache@v2
        with:
          path: .next
          # Only valid for this commit
          key: ${{ github.sha }}

      - name: Build app
        run: npm run cypress:build:ci

  prepare-cypress-deps:
    name: Prepare Cypress deps
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Use Node.js v14
        uses: actions/setup-node@v2
        with:
          node-version: '14'

      - name: Cache Cypress binary and node_modules
        id: cache-cypress
        uses: actions/cache@v2
        with:
          path: |
            ~/.cache/Cypress
            node_modules
          key: cypress-cache-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
      
      - name: Install cypress
        run: npx cypress install

      - run: npx cypress cache list

  chrome:
    name: Chrome Tests
    runs-on: ubuntu-latest
    needs: [install, prepare-cypress-deps]
    container:
      image: cypress/browsers:node14.17.0-chrome88-ff89
      # options: --user 1001 # https://github.com/cypress-io/github-action#firefox
    strategy:
      fail-fast: false
      matrix:
        # run copies of the current job in parallel
        containers: [1, 2, 3, 4, 5, 6]
    steps:

      - name: Checkout
        uses: actions/checkout@v2

      - name: Use Node.js v14
        uses: actions/setup-node@v2
        with:
          node-version-file: ".nvmrc"
          cache: "npm"

      - name: Restore cached build
        uses: actions/cache@v2
        with:
          path: .next
          key: ${{ github.sha }}

      - name: Restore cached cypress deps
        id: cache-cypress
        uses: actions/cache@v2
        with:
          path: |
            ~/.cache/Cypress
            node_modules
          key: cypress-cache-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}

      - name: Start server and run Cypress tests (chrome)
        uses: cypress-io/github-action@v2
        with:
          install: false
          start: npm run cypress:start:ci
          browser: chrome
          record: true
          parallel: true
          group: "Integration - Chrome"
          spec: cypress/tests/open-ebooks/integration/**/*
        env:
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
          # Recommended: pass the GitHub token lets this action correctly
          # determine the unique run id necessary to re-run the checks
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

[sgronblo]

Is anyone awake here?

The official GH Cypress action uses @actions/cache which prefers to use zstd. Since it's not included in the image you will see a large amount of warnings in your debug output and I'm guessing it falls back to gzip.

[MikeMcC399]

@sgronblo

https://github.com/actions/toolkit/tree/main/packages/cache#save-cache says:

"Saves a cache containing the files in paths using the key provided. The files would be compressed using zstandard compression algorithm if zstd is installed, otherwise gzip is used."

You are right that debug level logs, for instance from running https://github.com/cypress-io/github-action/blob/master/.github/workflows/example-docker.yml show @actions/cache looking for zstd first, however this is not surfaced as an error and caching works.

Nevertheless if zstd can be added to the Cypress Docker images, it would make the use of @actions/cache smoother.

Note that even if zstd is available @actions/cache still logs several debug lines when zstd is found, so it does not make the debug output quiet.

[amakhrov]

It affects us, too. We don't even want to share artifacts. Just regular yarn install before running Cypress would benefit from yarn cache - which is always missed due to a different cache version when inside the cypress/browsers:xxx container.

UPDATE: Actually, I take it back. The cache version mismatch is caused by the different mount point inside the container. The path of the cache folder is used to determine the cache version. In the container it's /__w/<repo>/<repo>/.yarn/cache, while directly in the GHA runner the path is /runner/_work/<repo>/<repo>/.yarn/cache

[mohammedhammoud]

Any plans to include zstd?

[MikeMcC399]

• PR feat(deps): add zstd to factory image #1196 submitted to add zstd to images.

[MikeMcC399]

cypress/factory:4.1.0 now includes zstd in addition to gzip

$ docker run -it --rm cypress/factory:4.1.0
root@c2517276f175:/# whereis gzip
gzip: /usr/bin/gzip
root@c2517276f175:/# whereis zstd
zstd: /usr/bin/zstd
root@c2517276f175:/# zstd --version
*** Zstandard CLI (64-bit) v1.5.4, by Yann Collet ***

Any other Cypress Docker images published from now on will also include zstd.