Rook Ceph with local partition possible?

[TimoGoetze]

Hi,

are there any already known reasons, why it should not work to use the post_create_commands and:

shrink the existing partition to some useful value that leaves enough space for rook ceph and provision it on the local empty space?

I personally need readwritemany storage and a soulution for this as high available as possible.

Thoughts welcome. I could try after vacation and share a working config afterward.

Wanted feature?

Cheers

Timo

[vitobotta]

The post create commands are executed during Cloud Init. Perhaps changing the partition layout during that process isn't supported by Cloud Init.

[TimoGoetze]

Parts of Perplexity.ai's answer to that question might me wrong, but the source of https://askubuntu.com/questions/1019772/how-to-setup-partitions-on-my-boot-disk-with-cloud-config describes a way that seems to have worked already. Lets try.. The rest of AIs answer(citate):

To change the partition layout during the cloud-init process on Ubuntu or Debian, you need to carefully configure the cloud-config file to specify the desired partitioning and filesystem setup. Here’s a step-by-step guide based on the provided sources:
Cloud-Config File Structure
Here is an example of a cloud-config file that partitions the disk into three parts: / (root fs), /home, and /var, each with the specified percentages.

text

#cloud-config

resize_rootfs: false

disk_setup:
  /dev/vda:
    table_type: 'mbr'
    layout:
      - 25  # Root FS (25% of the disk)
      - 50  # Home (50% of the disk)
      - 25  # Var (25% of the disk)
    overwrite: true

fs_setup:
  - label: root_fs
    filesystem: 'ext4'
    device: /dev/vda
    partition: vda1
    overwrite: true```

  - label: home_disk
    filesystem: 'xfs'
    device: /dev/vda
    partition: vda2
    overwrite: true
  - label: var_disk
    filesystem: 'xfs'
    device: /dev/vda
    partition: vda3
    overwrite: true

runcmd:
  - [ partx, --update, /dev/vda ]
  - [ mkfs.xfs, /dev/vda2 ]
  - [ mkfs.xfs, /dev/vda3 ]
  - [ partprobe ]
  - parted /dev/vda set 1 boot on

mounts:
  - ["/dev/vda1", "/", "ext4", "defaults", "0", "0"]
  - ["/dev/vda2", "/home", "xfs", "defaults", "0", "0"]
  - ["/dev/vda3", "/var", "xfs", "defaults", "0", "0"]

Key Points

resize_rootfs: false: This ensures that the root filesystem is not automatically resized by cloud-init.
disk_setup: Defines the partition table and layout. Here, it uses MBR and specifies the percentages for each partition.
fs_setup: Configures the filesystems for each partition.
runcmd: Runs commands to update the partition table, create filesystems, and set the boot flag.
mounts: Specifies the mount points for each partition[1](https://forum.proxmox.com/threads/disk-partition-using-cloud-init-config.147349/)
[3](https://askubuntu.com/questions/1019772/how-to-setup-partitions-on-my-boot-disk-with-cloud-config)
[6](https://learn.microsoft.com/de-de/Azure/virtual-machines/linux/cloudinit-configure-swapfile)
.

Important Considerations

Ensure that the overwrite option is set to true if you want to overwrite any existing partitions.
The partx, mkfs.xfs, and partprobe commands are used to update the partition table and create the filesystems.
If you are using GPT instead of MBR, you need to adjust the table_type accordingly[3](https://askubuntu.com/questions/1019772/how-to-setup-partitions-on-my-boot-disk-with-cloud-config)
.

By using this configuration, you can achieve the desired partition layout during the initial boot process managed by cloud-init.

[TimoGoetze]

try or is the above useless?

[vitobotta]

try or is the above useless?

I have no idea. I haven't tested with custom partition layouts so far.

[TimoGoetze]

It is working. Just add some things to the templates/cloud-init.yaml and recompile with --static. After these Changes, only 50% of master and worker SSDs are formatted and used for the cluster. The other 50% can be used for e.g. rook-ceph or other storage providers. I would want to make it configurable and offer a PR, but I am not sure, whats the best way AND I dont know yet, how autoscaler behaves. So, ideas welcome.

#cloud-config
preserve_hostname: true
resize_rootfs: false

write_files:
{{ eth1_str }}

{{ growpart_str }}

- path: /etc/systemd/system/ssh.socket.d/listen.conf
  content: |
    [Socket]
    ListenStream=
    ListenStream={{ ssh_port}}

- path: /etc/configure-ssh.sh
  permissions: '0755'
  content: |
    if systemctl is-active ssh.socket > /dev/null 2>&1
    then
      # OpenSSH is using socket activation
      systemctl disable ssh
      systemctl daemon-reload
      systemctl restart ssh.socket
      systemctl stop ssh
    else
      # OpenSSH is not using socket activation
      sed -i 's/^#*Port .*/Port {{ ssh_port }}/' /etc/ssh/sshd_config
    fi
    systemctl restart ssh
- path: /etc/growroot-disabled
  content: |
  # Any text


packages: [{{ packages_str }}]

runcmd:
{{ post_create_commands_str }}
- [ sgdisk, -e, /dev/sda ]
- [ partprobe ]
- [ parted, -s, /dev/sda, mkpart, primary, ext4, "50%", "100%" ]
- [ growpart, /dev/sda, "1" ]
- [ resize2fs, /dev/sda1 ]