Skip to content

KVM: create an encrypted RBD volume with room for its LUKS2 header - #14725

Draft
weizhouapache wants to merge 2 commits into
apache:mainfrom
weizhouapache:24-rbd-encrypted-volume-luks2-header
Draft

weizhouapache wants to merge 2 commits into
apache:mainfrom
weizhouapache:24-rbd-encrypted-volume-luks2-header

Conversation

@weizhouapache

Copy link
Copy Markdown
Member

Description

KVM: create an encrypted RBD volume with room for its LUKS2 header

librbd keeps the LUKS2 header in the RBD image, in front of the data, so an
encrypted RBD volume created with the size of its volume showed the Instance
16 MiB less, while a qemu LUKS volume on NFS, SharedMountPoint or local
storage shows the Instance the size of its volume. Create the image of an
encrypted RBD volume with room for the header instead, so that the Instance
sees the size of the volume on every pool type, and the size of the volume
in the database is what the Instance sees.

The size of the header is the larger of 16 MiB and the object size of the
image (measured: 16 MiB with objects of 1 to 16 MiB, 32 MiB with objects of
32 MiB), so the image is grown when its objects are larger.

- LibvirtStorageAdaptor.createPhysicalDisk() creates an encrypted RBD
  image with the header on top of the size it is given.
- LibvirtResizeVolumeCommandWrapper answers with the size the Instance
  sees for an encrypted RBD volume, so the database no longer gets the
  size of the image (the size of the volume + 16 MiB) after a resize.
- The CoW clone of a template for an encrypted ROOT volume is made from a
  snapshot of the template grown by the header, and the template is shrunk
  back right after the snapshot. The template used to stay 16 MiB larger, so
  the next unencrypted clone of it, if its base snapshot did not exist yet,
  was 16 MiB larger than its template. The clone is grown to the requested
  size from its own size, and the full copy of a template makes room for a
  header larger than 16 MiB as well.

Tests:
- unit test for the size of the header
- the TestVolumeEncryption smoke tests already expect the Instance to see
  the size of the volume

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

When qemu-img convert into an RBD pool failed or timed out, the half-written
image was left behind. Every later copy of the same template then failed with
"rbd create: File exists", so deployments from that template broke on the pool
until the image was removed by hand.

- check whether the destination RBD image exists before converting, and refuse
  to overwrite it, as it may be a complete image used by clones
- remove the image again if the copy created it and then failed
- do not print the Ceph key in the "Failed to convert" error
librbd keeps the LUKS2 header in the RBD image, in front of the data, so an
encrypted RBD volume created with the size of its volume showed the Instance
16 MiB less, while a qemu LUKS volume on NFS, SharedMountPoint or local
storage shows the Instance the size of its volume. Create the image of an
encrypted RBD volume with room for the header instead, so that the Instance
sees the size of the volume on every pool type, and the size of the volume
in the database is what the Instance sees.

The size of the header is the larger of 16 MiB and the object size of the
image (measured: 16 MiB with objects of 1 to 16 MiB, 32 MiB with objects of
32 MiB), so the image is grown when its objects are larger.

- LibvirtStorageAdaptor.createPhysicalDisk() creates an encrypted RBD
  image with the header on top of the size it is given.
- LibvirtResizeVolumeCommandWrapper answers with the size the Instance
  sees for an encrypted RBD volume, so the database no longer gets the
  size of the image (the size of the volume + 16 MiB) after a resize.
- The CoW clone of a template for an encrypted ROOT volume is made from a
  snapshot of the template grown by the header, and the template is shrunk
  back right after the snapshot. The template used to stay 16 MiB larger, so
  the next unencrypted clone of it, if its base snapshot did not exist yet,
  was 16 MiB larger than its template. The clone is grown to the requested
  size from its own size, and the full copy of a template makes room for a
  header larger than 16 MiB as well.

Tests:
- unit test for the size of the header
- the TestVolumeEncryption smoke tests already expect the Instance to see
  the size of the volume
@weizhouapache

Copy link
Copy Markdown
Member Author

@calvix

Thanks for your work on encrypted RBD volumes. Great work!

I have created this PR to allocate an additional 16 MiB or 32 MiB for the LUKS2 header of encrypted volumes on Ceph RBD. This ensures that the guest VM sees the exact volume size reported by CloudStack, keeping the behaviour consistent with encrypted volumes on NFS, Local and StorPool.

With the current implementation, the size visible inside the guest OS is the RBD volume size minus the 16 MiB LUKS2 header.

Please note that this change might affect your PR #14656.

@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 4.80000% with 119 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.78%. Comparing base (c5bd11a) to head (848d8d8).
⚠️ Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
.../hypervisor/kvm/storage/LibvirtStorageAdaptor.java 4.87% 117 Missing ⚠️
...rce/wrapper/LibvirtResizeVolumeCommandWrapper.java 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##               main   #14725    +/-   ##
==========================================
  Coverage     20.77%   20.78%            
- Complexity    20855    20873    +18     
==========================================
  Files          6053     6053            
  Lines        548230   548381   +151     
  Branches      73237    73257    +20     
==========================================
+ Hits         113903   113981    +78     
- Misses       420988   421046    +58     
- Partials      13339    13354    +15     
Flag Coverage Δ
uitests 3.86% <ø> (ø)
unittests 22.22% <4.80%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant