From adf7444bd76c460cbfbc0864234456c50c3b3361 Mon Sep 17 00:00:00 2001 From: Johnson Sun Date: Sun, 15 Dec 2024 11:48:23 +0800 Subject: [PATCH] feat: Use docker build cache when running `docker compose build` References * https://docs.docker.com/build/builders/drivers/ * https://docs.docker.com/reference/cli/docker/buildx/create/#use * https://docs.docker.com/reference/compose-file/build/#cache_from * https://github.com/docker/buildx#set-buildx-as-the-default-builder * https://github.com/docker/buildx/issues/107 --- README.md | 30 ++++++++++++++++++++++++++++- aloha_ws/docker/compose.yaml | 3 +++ cartographer_ws/docker/compose.yaml | 3 +++ docs/index.md | 30 ++++++++++++++++++++++++++++- gazebo_world_ws/docker/compose.yaml | 3 +++ husky_ws/docker/compose.yaml | 3 +++ kobuki_ws/docker/compose.yaml | 3 +++ orbslam3_ws/docker/compose.yaml | 3 +++ rtabmap_ws/docker/compose.yaml | 3 +++ template_ws/docker/compose.yaml | 3 +++ tests/diff_base/docker/compose.yaml | 3 +++ tests/lint_comp_template.py | 2 +- vlp_ws/docker/compose.yaml | 3 +++ 13 files changed, 89 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index ec965a53..76fe510e 100644 --- a/README.md +++ b/README.md @@ -61,12 +61,40 @@ mkdocs serve # Go to https://127.0.0.1:8000 to view the site. ``` -## Link Documentation to ROS2 Workspaces +### Link Documentation to ROS2 Workspaces ```sh scripts/setup_link.sh ``` +## Reusing Docker Build Cache + +The default (`docker`) build driver does not support pulling pre-built build cache from Docker Hub. Changing to the `docker-container` build driver allow reusing build cache from Docker Hub, however may introduce a few minute overhead after building any image (for sending tarballs). This is a tradeoff you should consider. + +The default `docker` build driver cannot pull pre-built caches from Docker Hub. Using the `docker-container` driver [enables cache reuse](https://docs.docker.com/build/builders/drivers/) but may add [a few minutes of overhead](https://github.com/docker/buildx/issues/107) for sending tarballs after the build. You should consider this tradeoff if you choose to switch the build driver. + +### Switching to `docker-container` Build Driver + +```sh +# Install buildx as `docker build` alias +docker buildx install +# Create and use new builder +docker buildx create --name docker-container --driver docker-container --driver-opt default-load=true --use +``` + +After setting this, using `docker compose build` will use the build cache from Docker Hub. If you want to switch back to the default `docker` build driver, follow the instructions below. + +### Switching back to `docker` Build Driver + +```sh +# Uninstall buildx as `docker build` alias +docker buildx uninstall +# Unuse the created builder and remove it +docker buildx use default +docker buildx stop docker-container +docker buildx rm -f --all-inactive +``` + ## Acknowledgement The code is mainly contributed by [Johnson](https://github.com/j3soon), [Yu-Zhong Chen](https://github.com/YuZhong-Chen), [Assume Zhan](https://github.com/Assume-Zhan), [Lam Chon Hang](https://github.com/ClassLongJoe1112), and others. For a full list of contributors, please refer to the [contribution list](https://github.com/j3soon/ros2-essentials/graphs/contributors). diff --git a/aloha_ws/docker/compose.yaml b/aloha_ws/docker/compose.yaml index f1591b15..df8599b0 100644 --- a/aloha_ws/docker/compose.yaml +++ b/aloha_ws/docker/compose.yaml @@ -21,6 +21,9 @@ services: # Reference: https://docs.docker.com/compose/compose-file/build/#platforms # platforms: # - "linux/arm64" + cache_from: + - j3soon/ros2-aloha-ws:buildcache-amd64 + - j3soon/ros2-aloha-ws:buildcache-arm64 image: j3soon/ros2-aloha-ws container_name: ros2-aloha-ws stdin_open: true diff --git a/cartographer_ws/docker/compose.yaml b/cartographer_ws/docker/compose.yaml index f9d88e9b..e7760f8c 100644 --- a/cartographer_ws/docker/compose.yaml +++ b/cartographer_ws/docker/compose.yaml @@ -21,6 +21,9 @@ services: # Reference: https://docs.docker.com/compose/compose-file/build/#platforms # platforms: # - "linux/arm64" + cache_from: + - j3soon/ros2-cartographer-ws:buildcache-amd64 + - j3soon/ros2-cartographer-ws:buildcache-arm64 image: j3soon/ros2-cartographer-ws container_name: ros2-cartographer-ws stdin_open: true diff --git a/docs/index.md b/docs/index.md index ec965a53..76fe510e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -61,12 +61,40 @@ mkdocs serve # Go to https://127.0.0.1:8000 to view the site. ``` -## Link Documentation to ROS2 Workspaces +### Link Documentation to ROS2 Workspaces ```sh scripts/setup_link.sh ``` +## Reusing Docker Build Cache + +The default (`docker`) build driver does not support pulling pre-built build cache from Docker Hub. Changing to the `docker-container` build driver allow reusing build cache from Docker Hub, however may introduce a few minute overhead after building any image (for sending tarballs). This is a tradeoff you should consider. + +The default `docker` build driver cannot pull pre-built caches from Docker Hub. Using the `docker-container` driver [enables cache reuse](https://docs.docker.com/build/builders/drivers/) but may add [a few minutes of overhead](https://github.com/docker/buildx/issues/107) for sending tarballs after the build. You should consider this tradeoff if you choose to switch the build driver. + +### Switching to `docker-container` Build Driver + +```sh +# Install buildx as `docker build` alias +docker buildx install +# Create and use new builder +docker buildx create --name docker-container --driver docker-container --driver-opt default-load=true --use +``` + +After setting this, using `docker compose build` will use the build cache from Docker Hub. If you want to switch back to the default `docker` build driver, follow the instructions below. + +### Switching back to `docker` Build Driver + +```sh +# Uninstall buildx as `docker build` alias +docker buildx uninstall +# Unuse the created builder and remove it +docker buildx use default +docker buildx stop docker-container +docker buildx rm -f --all-inactive +``` + ## Acknowledgement The code is mainly contributed by [Johnson](https://github.com/j3soon), [Yu-Zhong Chen](https://github.com/YuZhong-Chen), [Assume Zhan](https://github.com/Assume-Zhan), [Lam Chon Hang](https://github.com/ClassLongJoe1112), and others. For a full list of contributors, please refer to the [contribution list](https://github.com/j3soon/ros2-essentials/graphs/contributors). diff --git a/gazebo_world_ws/docker/compose.yaml b/gazebo_world_ws/docker/compose.yaml index 7a035e59..f204cad1 100644 --- a/gazebo_world_ws/docker/compose.yaml +++ b/gazebo_world_ws/docker/compose.yaml @@ -21,6 +21,9 @@ services: # Reference: https://docs.docker.com/compose/compose-file/build/#platforms # platforms: # - "linux/arm64" + cache_from: + - j3soon/ros2-gazebo-world-ws:buildcache-amd64 + - j3soon/ros2-gazebo-world-ws:buildcache-arm64 image: j3soon/ros2-gazebo-world-ws container_name: ros2-gazebo-world-ws stdin_open: true diff --git a/husky_ws/docker/compose.yaml b/husky_ws/docker/compose.yaml index 099cf41d..c89a640b 100644 --- a/husky_ws/docker/compose.yaml +++ b/husky_ws/docker/compose.yaml @@ -21,6 +21,9 @@ services: # Reference: https://docs.docker.com/compose/compose-file/build/#platforms # platforms: # - "linux/arm64" + cache_from: + - j3soon/ros2-husky-ws:buildcache-amd64 + - j3soon/ros2-husky-ws:buildcache-arm64 image: j3soon/ros2-husky-ws container_name: ros2-husky-ws stdin_open: true diff --git a/kobuki_ws/docker/compose.yaml b/kobuki_ws/docker/compose.yaml index 915b393d..a5ba5f89 100644 --- a/kobuki_ws/docker/compose.yaml +++ b/kobuki_ws/docker/compose.yaml @@ -21,6 +21,9 @@ services: # Reference: https://docs.docker.com/compose/compose-file/build/#platforms # platforms: # - "linux/arm64" + cache_from: + - j3soon/ros2-kobuki-ws:buildcache-amd64 + - j3soon/ros2-kobuki-ws:buildcache-arm64 image: j3soon/ros2-kobuki-ws container_name: ros2-kobuki-ws stdin_open: true diff --git a/orbslam3_ws/docker/compose.yaml b/orbslam3_ws/docker/compose.yaml index cbd65bff..3dd48c8a 100644 --- a/orbslam3_ws/docker/compose.yaml +++ b/orbslam3_ws/docker/compose.yaml @@ -21,6 +21,9 @@ services: # Reference: https://docs.docker.com/compose/compose-file/build/#platforms # platforms: # - "linux/arm64" + cache_from: + - j3soon/ros2-orbslam3-ws:buildcache-amd64 + - j3soon/ros2-orbslam3-ws:buildcache-arm64 image: j3soon/ros2-orbslam3-ws container_name: ros2-orbslam3-ws stdin_open: true diff --git a/rtabmap_ws/docker/compose.yaml b/rtabmap_ws/docker/compose.yaml index 822f5245..33e4e051 100644 --- a/rtabmap_ws/docker/compose.yaml +++ b/rtabmap_ws/docker/compose.yaml @@ -21,6 +21,9 @@ services: # Reference: https://docs.docker.com/compose/compose-file/build/#platforms # platforms: # - "linux/arm64" + cache_from: + - j3soon/ros2-rtabmap-ws:buildcache-amd64 + - j3soon/ros2-rtabmap-ws:buildcache-arm64 image: j3soon/ros2-rtabmap-ws container_name: ros2-rtabmap-ws stdin_open: true diff --git a/template_ws/docker/compose.yaml b/template_ws/docker/compose.yaml index f6b6ea55..739a2cca 100644 --- a/template_ws/docker/compose.yaml +++ b/template_ws/docker/compose.yaml @@ -21,6 +21,9 @@ services: # Reference: https://docs.docker.com/compose/compose-file/build/#platforms # platforms: # - "linux/arm64" + cache_from: + - j3soon/ros2-template-ws:buildcache-amd64 + - j3soon/ros2-template-ws:buildcache-arm64 image: j3soon/ros2-template-ws container_name: ros2-template-ws stdin_open: true diff --git a/tests/diff_base/docker/compose.yaml b/tests/diff_base/docker/compose.yaml index 8dafaf83..cd1a870a 100644 --- a/tests/diff_base/docker/compose.yaml +++ b/tests/diff_base/docker/compose.yaml @@ -21,6 +21,9 @@ services: # Reference: https://docs.docker.com/compose/compose-file/build/#platforms # platforms: # - "linux/arm64" + cache_from: + - j3soon/ros2-{PLACEHOLDER}-ws:buildcache-amd64 + - j3soon/ros2-{PLACEHOLDER}-ws:buildcache-arm64 image: j3soon/ros2-{PLACEHOLDER}-ws container_name: ros2-{PLACEHOLDER}-ws stdin_open: true diff --git a/tests/lint_comp_template.py b/tests/lint_comp_template.py index ac28e381..16553e04 100644 --- a/tests/lint_comp_template.py +++ b/tests/lint_comp_template.py @@ -88,6 +88,6 @@ def error(msg, i): compare_file_with_template(".gitignore") compare_file_with_template("docker/.bashrc") compare_file_with_template("docker/.dockerignore", ignored_workspaces=["ros1_bridge_ws"]) -compare_file_with_template("docker/compose.yaml", ignored_workspaces=["ros1_bridge_ws"]) +compare_file_with_template("docker/compose.yaml", ignored_workspaces=["ros1_bridge_ws", "jazzy_template_ws"]) compare_file_with_template("docker/Dockerfile", ignored_workspaces=["ros1_bridge_ws", "jazzy_template_ws"]) compare_file_with_template("README.md") diff --git a/vlp_ws/docker/compose.yaml b/vlp_ws/docker/compose.yaml index e776fca9..f9a0a62a 100644 --- a/vlp_ws/docker/compose.yaml +++ b/vlp_ws/docker/compose.yaml @@ -21,6 +21,9 @@ services: # Reference: https://docs.docker.com/compose/compose-file/build/#platforms # platforms: # - "linux/arm64" + cache_from: + - j3soon/ros2-vlp-ws:buildcache-amd64 + - j3soon/ros2-vlp-ws:buildcache-arm64 image: j3soon/ros2-vlp-ws container_name: ros2-vlp-ws stdin_open: true