Major corruption while moving LXC storage to MooseFS. Errors not cancelling 'move storage' tasks.

[JoaGamo]

Hello, I'm facing major corruption, luckily I had backups for most things. The actual scenario is moving every single mountpoint from all LXCs to Moosefs. At this exact point when this problem happened I was rebooting the moosefs master safely (systemctl restart moosefs-master). I'm on MooseFS Community Edition.

I mounted moosefs with Use Block Device feature enabled

First problem that I'm facing, the plugin isn't cancelling operations on failures, there's an example where I did a move + delete source and this happened

started block device: (/dev/mfs/mfsmaster.main_9421__images_178_vm-178-disk-0->/dev/nbd2 : MFS://images/178/vm-178-disk-0 : 120.000GiB)
Creating filesystem with 31457280 4k blocks and 7864320 inodes
Filesystem UUID: cebb78ff-31d1-4478-899c-213db950b394
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424, 20480000, 23887872
/dev/rbd1

Number of files: 123,836 (reg: 110,154, dir: 13,680, link: 2)
Number of created files: 123,834 (reg: 110,154, dir: 13,678, link: 2)
Number of deleted files: 0
Number of regular files transferred: 110,154
Total file size: 36,692,106,197 bytes
Total transferred file size: 36,692,106,080 bytes
Literal data: 36,692,106,080 bytes
Matched data: 0 bytes
File list size: 3,997,413
File list generation time: 0.052 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 36,709,001,584
Total bytes received: 2,206,455

sent 36,709,001,584 bytes  received 2,206,455 bytes  17,296,211.09 bytes/sec
total size is 36,692,106,197  speedup is 1.00
can't unmap MooseFS device '/images/178/vm-178-disk-0': error receiving data from '/dev/mfs/nbdsock': Connection timed out
volume deactivation failed: mfsBlock:178/vm-178-disk-0
Removing image: 1% complete...
Removing image: 2% complete...
Removing image: 3% complete...
Removing image: 4% complete...
Removing image: 5% complete...
Removing image: 6% complete...
Removing image: 7% complete...
// The source image got completely deleted, im truncating the log.

Why is this important?
Not only because it completely nuked the source image (!), but after moving a secondary mountpoint from this same LXC with ID 178, 15 minutes later, it overwrote the 'vm-178-disk-0' image. To put it in better words:

Both move operations failed with the same log shown above. It could be due to a moosefs master restart in the middle or simply that the server lost connection.

Before migration, my rootfs: was vm-178-disk-0, my mp0: was vm-178-disk-1. First I started moving the rootfs from my source storage to MooseFS. After some time, this error appears Connection timed out, but the source image is pruned anyway. As I did not see any error, I started moving mp0:, which overwrote the previous 'vm-178-disk-0' that was in /mnt/pve/Block/images/178/. I will never know if the rootfs image did copy successfully and moosefs overwrote it

[JoaGamo]

Also I'm not sure I can trust the block devices implementation for now, as any volumes, valid or not, don't show up in the Proxmox GUI. Not sure if this is a bug on my end though

[Zorlin]

Hello, I'm facing major corruption, luckily I had backups for most things. The actual scenario is moving every single mountpoint from all LXCs to Moosefs. At this exact point when this problem happened I was rebooting the moosefs master safely (systemctl restart moosefs-master). I'm on MooseFS Community Edition.

Hi JoaGamo - thank you so much for trying the plugin and sorry you ran into issues. Thank you for having backups and for such a detailed bug report! We're still ironing out the kinks on this plugin, but it's odd that this happened.

At this exact point when this problem happened I was rebooting the moosefs master safely (systemctl restart moosefs-master). I'm on MooseFS Community Edition.

What version of the plugin was installed when/while you were doing this? We had a major corruption bug we fixed a while back, the only one I'm aware of so far - #32 - here - I will try to reproduce your issues, I'm not yet sure how we'll do so but we can definitely look into adding some safeties for the failure modes you ran into.

[Zorlin]

Also I'm not sure I can trust the block devices implementation for now, as any volumes, valid or not, don't show up in the Proxmox GUI. Not sure if this is a bug on my end though

Not so much a bug - same happens with Ceph RBD I believe; they will appear as .raw files on disk on MooseFS in the mountpoint you chose (within the images/VMID/blahblah.raw structure Proxmox sets up) which should be visible in the Storage section underneath each host within a PVE cluster.

What version of Proxmox are you running?

Thanks!

[JoaGamo]

Hello! I re-read my issue and I apologise if it did sound insulting, I had a bad moment, backups do saved that day

I'm running Proxmox 9.0.10 (no-subscription repository)
I experienced this issue with the pve-moosefs plugin version 0.1.10 (beta).

same happens with Ceph RBD I believe

It may have been an issue of the past that isn't there now :), I can see all my ceph volumes showing up in each pool under any host's storage section. I do not see anything under any Proxmox host for MooseFS block storage.

---

Adding more information, I am migrating from a Ceph cluster to MooseFS. To do this, I added both a MooseFS file storage and a MooseFS block device storage under the Datacenter section.
My intention was:

• MooseFS block device storage <-> Ceph replacement
• MooseFS File storage <-> CephFS replacement

Networking side, my MooseFS master listens on the 192.168.0.0/24 subnet, which is connectable by all nodes moosefs clients. The chunkservers have an extra 10gbit interface on a separate 10.0.1.0/24 network, which is connectable by almost all mfs-clients except one (This one proxmox-node does not run a chunkserver, it uses the moosefs feature mfsipmap.cfg to connect to the chunkservers as a mfs-client)

even if the network setup is "weird", I could reproduce this issue with all nodes but not consistently, I'm looking for a clear pattern. Storage size is not, as I can reproduce it with containers ranging between 8-300gb rootfs volumes. I'm thinking it could lose connection if the transfer was too slow and held for too long? Notice in this log, even if we are talking about 2 all-flash pools (one pool is on ceph, the other is on MooseFS), the speed was around 14,460,639.52 bytes/sec (14MB/s). This special node runs exclusively on a 1gbit interface, it may have been overloaded at this point in time and lost connection mid-transfer. I recently did another move and got 75MB/s which is more acceptable.

/var/log/pve/tasks/E# grep -e "timed out" * -H -A3 -B30
:move_volume:182:root@pam:-started block device: (/dev/mfs/mfsmaster.gamo_9421__images_182_vm-182-disk-0->/dev/nbd1 : MFS://images/182/vm-182-disk-0 : 8.000GiB)
:move_volume:182:root@pam:-Creating filesystem with 2097152 4k blocks and 524288 inodes
:move_volume:182:root@pam:-Filesystem UUID: 30bdf8fa-92aa-477b-becc-1c8e0bea3734
:move_volume:182:root@pam:-Superblock backups stored on blocks: 
:move_volume:182:root@pam:-     32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
:move_volume:182:root@pam:-/dev/rbd16
:move_volume:182:root@pam:-
:move_volume:182:root@pam:-Number of files: 248,445 (reg: 213,632, dir: 22,754, link: 12,029, special: 30)
:move_volume:182:root@pam:-Number of created files: 248,443 (reg: 213,632, dir: 22,752, link: 12,029, special: 30)
:move_volume:182:root@pam:-Number of deleted files: 0
:move_volume:182:root@pam:-Number of regular files transferred: 115,189
:move_volume:182:root@pam:-Total file size: 5,961,525,664 bytes
:move_volume:182:root@pam:-Total transferred file size: 4,626,911,866 bytes
:move_volume:182:root@pam:-Literal data: 4,626,911,866 bytes
:move_volume:182:root@pam:-Matched data: 0 bytes
:move_volume:182:root@pam:-File list size: 17,422,453
:move_volume:182:root@pam:-File list generation time: 0.033 seconds
:move_volume:182:root@pam:-File list transfer time: 0.000 seconds
:move_volume:182:root@pam:-Total bytes sent: 4,662,841,038
:move_volume:182:root@pam:-Total bytes received: 15,175,846
:move_volume:182:root@pam:-
:move_volume:182:root@pam:-sent 4,662,841,038 bytes  received 15,175,846 bytes  14,460,639.52 bytes/sec
:move_volume:182:root@pam:-total size is 5,961,525,664  speedup is 1.27
:move_volume:182:root@pam::can't unmap MooseFS device '/images/182/vm-182-disk-0': error receiving data from '/dev/mfs/nbdsock': Connection timed out
:move_volume:182:root@pam:-volume deactivation failed: GamoBlock:182/vm-182-disk-0
:move_volume:182:root@pam:-TASK OK

To me, the log looks like if it did sent the stuff correctly, but starting the container yields this error

sync_wait: 34 An error occurred in another process (expected sequence number 7)
__lxc_start: 2119 Failed to spawn container "182"
TASK ERROR: startup for container '182' failed

Exploring further this issue out of curiosity led me to this error during the container startup

$ lxc-start -n 182 -F -l DEBUG -o /tmp/lxc-182.log
start - ../src/lxc/start.c:start:2209 - No such file or directory - Failed to exec "/sbin/init"

I mounted this container with pct mount. I found some of my own files, but /bin, /etc and many directories were missing, making the container not bootable and not worth repairing.

other stuff that happens was already mentioned, if you moved another mountpoint (mp0:, mp1:, etc) of the same container to mfs-block storage, it will overwrite the rootfs volume permanently.

Some containers which were running fine, after moving their storage to mfs-block, sometimes the connection timed out issue happened during the transfer, causing Proxmox to continue their move storage script.

I think it's preferable, as you mentioned, to add safe-guards in case of lost connections. Proxmox shouldn't continue if an issue like this happens.

[Zorlin]

Don't worry, I definitely didn't take it as insulting.

I'll look into this over the weekend or next time I have some bandwidth for PVE work.

Thank you, sincerely, for reporting the issue and for being patient with us!

It may take a bit of time for me to replicate your environment properly. I use nested virtualisation and ~3PB of storage (blend of HDD, NVMe and SSD around the world) for testing. Thanks again for the detail, it will make it much easier to test your scenario (albeit virtualised)

[Zorlin]

other stuff that happens was already mentioned, if you moved another mountpoint (mp0:, mp1:, etc) of the same container to mfs-block storage, it will overwrite the rootfs volume permanently.

Oof x 💯

We definitely can add some safeties to that. That's bad.

[Zorlin]

Also, if you're wanting to run in production, my advice is run without bdev for now - I'm very unconfident that LXC + bdev is fully prod-ready to the extent you might be feeling it is - VMs are pretty solid with bdev, but in 99% of cases you don't need bdev for decent performance via pve-moosefs anyways :)

[Zorlin]

Will be creating scenarios to test this tonight.

[Zorlin]

Hi @JoaGamo

I believe this is fixed in an upcoming PVE MooseFS patch.

  ---
  Issue #50 Fix Report: LXC Migration Corruption Prevention

  Problem

  LXC storage migrations to MooseFS could result in silent data corruption when the MooseFS master restarted or
  network issues occurred during transfer. The migration would report success but leave the container with
  incomplete/corrupted data.

  Root Cause

  Proxmox's storage_migrate() flow has no post-migration verification step. When volume_import() completes,
  Proxmox assumes success and deletes the source volume, even if the imported volume is incomplete or corrupted.

  From DeepWiki analysis of proxmox/pve-storage:
  "There are no explicit verification steps for the integrity or completeness of the migrated volume on the target
   storage before the source snapshot (and implicitly, the source volume) is deleted"

  Solution Implemented

  1. Volume Import Verification (MooseFSPlugin.pm:1084-1165)

  Added volume_import() override with three-stage integrity verification for mfsbdev volumes:

  sub volume_import {
      # Call parent import
      my $result = $class->SUPER::volume_import(...);

      # Verify for mfsbdev volumes
      if ($scfg->{mfsbdev} && $vtype eq 'images' && $format eq 'raw') {
          # Verification 1: .size file exists and has valid numeric content
          # Verification 2: Image file exists
          # Verification 3: Actual file size matches expected size

          # On failure: cleanup imported volume and die with clear error
      }

      return $result;
  }

  Benefits:
  - Detects incomplete transfers before source deletion
  - Cleans up corrupted imports automatically
  - Provides clear error messages for troubleshooting
  - Prevents silent data corruption

  2. Multiple Cluster Support (MooseFSPlugin.pm:253-270)

  Added mfsnbdlink advanced property to support multiple MooseFS clusters:

  properties => {
      mfsnbdlink => {
          description => "Socket path for mfsbdev daemon (default: /dev/mfs/nbdsock)",
          type => 'string',
      }
  }

  options => {
      mfsnbdlink => { optional => 1, advanced => 1 },
  }

  Updated all mfsbdev operations to use custom socket when configured:
  - alloc_image: map with -l socket flag
  - free_image: unmap with -l socket flag
  - filesystem_path: map with -l socket flag
  - with_nbd_unmapped: list/unmap/remap with -l socket flag

  Usage:
  # Start separate mfsbdev daemon for unstable cluster
  mfsbdev start -H 10.7.1.37 -P 9521 -l /dev/mfs/nbdsock-unstable

  # Add storage with custom socket
  pvesm add moosefs unstable-storage \
    --path /mnt/mfs-unstable \
    --mfsmaster 10.7.1.37 \
    --mfsport 9521 \
    --mfsbdev 1 \
    --mfsnbdlink /dev/mfs/nbdsock-unstable

  3. Comprehensive Test Suite (pve-moosefs-tests)

  Created test_migration_master_restart.rs to validate the fix:

  Test Scenario:
  1. Creates LXC container on local-lvm
  2. Writes test data for integrity verification
  3. Starts migration to MooseFS in background thread
  4. Restarts MooseFS master during migration (simulates issue #50)
  5. Verifies migration behavior

  Test Infrastructure:
  - Dedicated unstable MooseFS cluster (ports 9520/9521/9522)
  - Separate mfsbdev daemon (/dev/mfs/nbdsock-unstable)
  - Isolated from production for safe testing

  Test Results: ✅ PASS
  - Migration fails cleanly when master restarts
  - Error: "mfsbdev map failed: link exists"
  - No silent corruption - safer failure mode achieved

  Files Modified

  - MooseFSPlugin.pm: Added volume_import verification + mfsnbdlink property
  - test_migration_master_restart.rs: Test for issue #50 scenario

  Testing

  All tests pass:
  cargo test test_lxc_migration_with_master_restart
  # Test result: ok. 1 passed; 0 failed

  Impact

  - Before: Silent data corruption when master restarts during migration
  - After: Migration fails cleanly with clear error - no data loss

  Commits

  - 9c4c8d8 - Add mfsnbdlink property for multiple MooseFS cluster support
  - f5a1ccb - Add test for issue #50: LXC migration with master restart
  - (Previous session) - Add volume_import verification

  ---