Moving or copying a lxd storage volume to a different project fails #14762

neofright · 2025-01-08T21:05:53Z

Required information

Distribution: Ubuntu
Distribution version: 22.04
The output of "snap list --all lxd core20 core22 core24 snapd":

Name    Version         Rev    Tracking       Publisher   Notes
core20  20240705        2379   latest/stable  canonical✓  base,disabled
core20  20240911        2434   latest/stable  canonical✓  base
core22  20241001        1663   latest/stable  canonical✓  base,disabled
core22  20241119        1722   latest/stable  canonical✓  base
lxd     5.21.2-2f4ba6b  30131  5.21/stable    canonical✓  disabled,in-cohort
lxd     5.21.2-084c8c8  31214  5.21/stable    canonical✓  in-cohort
snapd   2.63            21759  latest/stable  canonical✓  snapd,disabled
snapd   2.66.1          23258  latest/stable  canonical✓  snapd

The output of "lxc info" or if that fails:
- Kernel version: 5.15.0-126-generic
- LXC version: 5.21.2 LTS
- LXD version: 5.21.2 LTS
- Storage backend in use: zfs

Issue description

It appears to be impossible to change the project of a storage volume when moving or copying it. Am I doing something wrong? The documentation clearly shows --target-project is a supported flag:

--target-project       Move to a project different from the source

Steps to reproduce

# Moving the volume does not change the project
$ lxc storage volume move local/custom_volume local/custom_volume_renamed --target-project=non-default-project --target=$(hostname -s) --destination-target=$(hostname -s)
$ lxc storage volume show local custom_volume_renamed | grep project
project: default

# Copying the volume does not change the project
$ lxc storage volume copy local/custom_volume local/custom_volume_renamed --target-project=non-default-project --target=$(hostname -s) --destination-target=$(hostname -s)
$ lxc storage volume show local custom_volume_renamed | grep project
project: default

Information to attach

Any relevant kernel output (dmesg)
Container log (lxc info NAME --show-log)
Container configuration (lxc config show NAME --expanded)
Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
Output of the client with --debug
Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

The text was updated successfully, but these errors were encountered:

neofright · 2025-01-08T21:25:38Z

Creating a custom storage volume and specifying --project also does not change the project:

$ lxc storage volume create local test --project non-default-project
$ lxc storage volume show local test | grep project
project: default

tomponline · 2025-01-09T08:57:48Z

@kadinsayani want to take this one?

kadinsayani · 2025-01-10T16:16:53Z

I'm not able to reproduce this issue.

Moving a storage volume to a target project:

❯ lxc storage volume create default custom_volume
Storage volume custom_volume created

❯ lxc storage volume move default/custom_volume default/custom_volume_renamed --target-project=non-default
Storage volume moved successfully!

❯ lxc project switch non-default

❯ lxc storage volume show default custom_volume_renamed | grep project
project: non-default

Creating a new storage volume in a specified project:

❯ lxc project create non-default-project
Project non-default-project created

❯ lxc storage volume create default test --project non-default-project
Storage volume test created

❯ lxc project switch non-default-project

❯ lxc storage volume show default test | grep project
project: non-default-project

I'll continue investigating further.

kadinsayani · 2025-01-10T16:40:06Z

Can you please share which LXC version you're using: lxc --version? Thanks.

neofright · 2025-01-10T17:03:41Z

Thanks for your replies.

Can you please share which LXC version you're using: lxc --version? Thanks.

$ lxd --version
5.21.2 LTS
$ lxc --version
5.21.2 LTS

neofright · 2025-01-10T17:10:46Z

I've tried using lxc project switch as this command is new to me.

After switching project if I lxc list then the expected containers are returned to indicate the project has changed.

However I sadly get the same results:

$ lxc project switch non-default-project
$ lxc storage volume create local test
Storage volume test created
$ lxc storage volume show local test | grep project
project: default

Ideally, an attempt to reproduce it would be made on the same lxd version and using zfs as a backend. I don't know what kind of testing suites you have available. If you need any more information from me about the system/cluster then please let me know.

It's somewhat encouraging that you can't reproduce it, albeit only if you are using a newer version of lxd! However, as this is a production system I am hesitant to update the cluster if it won't resolve this particular issue.

kadinsayani · 2025-01-10T17:17:56Z

Thanks for sharing your results. I wasn't able to reproduce the issue under the same LXD version with a ZFS backend.

root@v1:~# lxd --version
5.21.2 LTS
root@v1:~# lxc --version
5.21.2 LTS
root@v1:~# lxc storage ls
+---------+--------+--------------------------------------------+-------------+---------+---------+
|  NAME   | DRIVER |                   SOURCE                   | DESCRIPTION | USED BY |  STATE  |
+---------+--------+--------------------------------------------+-------------+---------+---------+
| default | zfs    | /var/snap/lxd/common/lxd/disks/default.img |             | 1       | CREATED |
+---------+--------+--------------------------------------------+-------------+---------+---------+
| local   | zfs    | /var/snap/lxd/common/lxd/disks/local.img   |             | 1       | CREATED |
+---------+--------+--------------------------------------------+-------------+---------+---------+
root@v1:~# lxc project switch default
root@v1:~# lxc storage volume create local test2 --project non-default-project
Storage volume test2 created
root@v1:~# lxc project switch non-default-project
root@v1:~# lxc storage volume ls
+-------+--------+-------+-------------+--------------+---------+
| POOL  |  TYPE  | NAME  | DESCRIPTION | CONTENT-TYPE | USED BY |
+-------+--------+-------+-------------+--------------+---------+
| local | custom | test  |             | filesystem   | 0       |
+-------+--------+-------+-------------+--------------+---------+
| local | custom | test2 |             | filesystem   | 0       |
+-------+--------+-------+-------------+--------------+---------+
root@v1:~# lxc storage volume show local test2 | grep project
project: non-default-project

I'll continue investigating and trying to identify potential causes for your issue and let you know if I require further information.

kadinsayani · 2025-01-10T17:22:44Z

Can you please share the full output of lxc storage list and lxc storage show local? Thanks.

neofright · 2025-01-10T18:22:23Z

I've been using the pool name local in all of my examples above just for simplicity but the pool is actually called localzfs as shown below:

$ lxc storage list
+----------------+--------+-------------+---------+---------+
|      NAME      | DRIVER | DESCRIPTION | USED BY |  STATE  |
+----------------+--------+-------------+---------+---------+
| localzfs       | zfs    |             | 1020    | CREATED |
+----------------+--------+-------------+---------+---------+
| migration-temp | dir    |             | 1       | CREATED |
+----------------+--------+-------------+---------+---------+

The non-default-project in my examples is also really decom-containers and the project we have been using to remove containers from service. We noticed recently that for systems that are moved to this project, their custom storage volumes have not been moved to the project and that is the issue I am trying to get to the bottom of.

Attached is the output of lxc storage list (over 1000 lines):
lxc_storage_list.txt

kadinsayani · 2025-01-10T18:30:22Z

Thanks for the added context.

Can you please share the output of both lxc storage show localzfs and lxc storage info localzfs?

neofright · 2025-01-10T18:45:31Z

Thanks for the added context.

Can you please share the output of both lxc storage show localzfs and lxc storage info localzfs?

storage_info_localzfs.txt
storage_show_localzfs.txt

kadinsayani · 2025-01-10T20:35:24Z

Can you please also attach the main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log) after attempting to create a storage volume with a target project? Thanks.

neofright · 2025-01-13T12:38:45Z

Can you please also attach the main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log) after attempting to create a storage volume with a target project? Thanks.

Running $ lxc storage volume create localzfs test --project=decom-containers doesn't cause anything to be printed to /var/snap/lxd/common/lxd/logs/lxd.log.

e.g.

$ sudo tail -n1 /var/snap/lxd/common/lxd/logs/lxd.log
time="2025-01-12T00:28:12Z" level=warning msg="Dqlite: attempt 1: server infra-lxdh-dev-03.mustard:8443: dial: Failed connecting to HTTP endpoint \"infra-lxdh-dev-03.mustard:8443\": context deadline exceeded"
$ lxc storage volume create localzfs test --project=decom-containers
Storage volume test created
$ sudo tail -n1 /var/snap/lxd/common/lxd/logs/lxd.log
time="2025-01-12T00:28:12Z" level=warning msg="Dqlite: attempt 1: server infra-lxdh-dev-03.mustard:8443: dial: Failed connecting to HTTP endpoint \"infra-lxdh-dev-03.mustard:8443\": context deadline exceeded"

kadinsayani · 2025-01-15T18:16:28Z

The error message Dqlite: attempt 1: server infra-lxdh-dev-03.mustard:8443: dial: Failed connecting to HTTP endpoint \"infra-lxdh-dev-03.mustard:8443\": context deadline exceeded indicates that LXD is unable to connect to the infra-lxdh-dev-03.mustard:8443 server. Can you please share the output of lxc cluster list? In your reproducer, what is the output hostname -s?

neofright · 2025-01-20T10:54:28Z

The error message Dqlite: attempt 1: server infra-lxdh-dev-03.mustard:8443: dial: Failed connecting to HTTP endpoint \"infra-lxdh-dev-03.mustard:8443\": context deadline exceeded indicates that LXD is unable to connect to the infra-lxdh-dev-03.mustard:8443 server. Can you please share the output of lxc cluster list? In your reproducer, what is the output hostname -s?

$ hostname -s
infra-lxdh-dev-04
$ lxc cluster list
+-------------------+----------------------------------------+------------------+--------------+----------------+--------------------------------------+-----------+--------------------------------+
|       NAME        |                  URL                   |      ROLES       | ARCHITECTURE | FAILURE DOMAIN |             DESCRIPTION              |   STATE   |            MESSAGE             |
+-------------------+----------------------------------------+------------------+--------------+----------------+--------------------------------------+-----------+--------------------------------+
| infra-lxdh-dev-03 | https://infra-lxdh-dev-03.mustard:8443 | database-standby | x86_64       | default        | Ubuntu 16.04 only. Hosted on pm-neon | ONLINE    | Fully operational              |
+-------------------+----------------------------------------+------------------+--------------+----------------+--------------------------------------+-----------+--------------------------------+
| infra-lxdh-dev-04 | https://10.104.240.123:8443            | database-standby | x86_64       | default        | Ubuntu 18.04+. Hosted on pm-neon     | ONLINE    | Fully operational              |
+-------------------+----------------------------------------+------------------+--------------+----------------+--------------------------------------+-----------+--------------------------------+
| infra-lxdh-dev-08 | https://infra-lxdh-dev-08.mustard:8443 | database         | x86_64       | default        | Retired - do not use me              | EVACUATED | Unavailable due to maintenance |
+-------------------+----------------------------------------+------------------+--------------+----------------+--------------------------------------+-----------+--------------------------------+
| infra-lxdh-dev-09 | https://infra-lxdh-dev-09.mustard:8443 | database         | x86_64       | default        | Retired - do not use me              | EVACUATED | Unavailable due to maintenance |
+-------------------+----------------------------------------+------------------+--------------+----------------+--------------------------------------+-----------+--------------------------------+
| pm-argon          | https://pm-argon.mustard:8443          | database-leader  | x86_64       | default        | Mixed testdev - 18.04+ only          | ONLINE    | Fully operational              |
|                   |                                        | database         |              |                |                                      |           |                                |
+-------------------+----------------------------------------+------------------+--------------+----------------+--------------------------------------+-----------+--------------------------------+

kadinsayani self-assigned this Jan 10, 2025

kadinsayani added the Incomplete Waiting on more information from reporter label Jan 10, 2025

kadinsayani removed the Incomplete Waiting on more information from reporter label Jan 10, 2025

kadinsayani added the Incomplete Waiting on more information from reporter label Jan 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Moving or copying a lxd storage volume to a different project fails #14762

Moving or copying a lxd storage volume to a different project fails #14762

neofright commented Jan 8, 2025 •

edited

Loading

neofright commented Jan 8, 2025

tomponline commented Jan 9, 2025

kadinsayani commented Jan 10, 2025 •

edited

Loading

kadinsayani commented Jan 10, 2025

neofright commented Jan 10, 2025

neofright commented Jan 10, 2025

kadinsayani commented Jan 10, 2025 •

edited

Loading

kadinsayani commented Jan 10, 2025 •

edited

Loading

neofright commented Jan 10, 2025 •

edited

Loading

kadinsayani commented Jan 10, 2025

neofright commented Jan 10, 2025 •

edited

Loading

kadinsayani commented Jan 10, 2025 •

edited

Loading

neofright commented Jan 13, 2025 •

edited

Loading

kadinsayani commented Jan 15, 2025 •

edited

Loading

neofright commented Jan 20, 2025

Moving or copying a lxd storage volume to a different project fails #14762

Moving or copying a lxd storage volume to a different project fails #14762

Comments

neofright commented Jan 8, 2025 • edited Loading

Required information

Issue description

Steps to reproduce

Information to attach

neofright commented Jan 8, 2025

tomponline commented Jan 9, 2025

kadinsayani commented Jan 10, 2025 • edited Loading

kadinsayani commented Jan 10, 2025

neofright commented Jan 10, 2025

neofright commented Jan 10, 2025

kadinsayani commented Jan 10, 2025 • edited Loading

kadinsayani commented Jan 10, 2025 • edited Loading

neofright commented Jan 10, 2025 • edited Loading

kadinsayani commented Jan 10, 2025

neofright commented Jan 10, 2025 • edited Loading

kadinsayani commented Jan 10, 2025 • edited Loading

neofright commented Jan 13, 2025 • edited Loading

kadinsayani commented Jan 15, 2025 • edited Loading

neofright commented Jan 20, 2025

neofright commented Jan 8, 2025 •

edited

Loading

kadinsayani commented Jan 10, 2025 •

edited

Loading

kadinsayani commented Jan 10, 2025 •

edited

Loading

kadinsayani commented Jan 10, 2025 •

edited

Loading

neofright commented Jan 10, 2025 •

edited

Loading

neofright commented Jan 10, 2025 •

edited

Loading

kadinsayani commented Jan 10, 2025 •

edited

Loading

neofright commented Jan 13, 2025 •

edited

Loading

kadinsayani commented Jan 15, 2025 •

edited

Loading