Skip to content

Commit

Permalink
ci: check space left before importing mongodb files (#9959)
Browse files Browse the repository at this point in the history
  • Loading branch information
alexgarel authored Mar 23, 2024
1 parent 4edbfd0 commit 58afefa
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,12 @@ import_prod_data:
@echo "🥫 Removing old archive in case you have one"
( rm -f ./html/data/openfoodfacts-mongodbdump.gz || true ) && ( rm -f ./html/data/gz-sha256sum || true )
@echo "🥫 Downloading full MongoDB dump from production …"
# verify we got sufficient space, NEEDED is in octet, LEFT in ko, we normalize to MB and NEEDED is multiplied by two (because it also will be imported)
NEEDED=$$(curl -s --head https://static.openfoodfacts.org/data/openfoodfacts-mongodbdump.gz|grep -i content-length: |cut -d ":" -f 2|tr -d " \r\n\t"); \
LEFT=$$(df . -k --output=avail |tail -n 1); \
NEEDED=$$(($$NEEDED/1048576 * 2)); \
LEFT=$$(($$LEFT/1024)); \
if [[ $$LEFT -lt $$NEEDED ]]; then >&2 echo "NOT ENOUGH SPACE LEFT ON DEVICE: $$NEEDED MB > $$LEFT MB"; exit 1; fi
wget --no-verbose https://static.openfoodfacts.org/data/openfoodfacts-mongodbdump.gz -P ./html/data
wget --no-verbose https://static.openfoodfacts.org/data/gz-sha256sum -P ./html/data
cd ./html/data && sha256sum --check gz-sha256sum
Expand Down

0 comments on commit 58afefa

Please sign in to comment.