This repository was archived by the owner on Mar 9, 2026. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 181
This repository was archived by the owner on Mar 9, 2026. It is now read-only.
cuDSS OOM in global_positioning on RTX 3090 (24GB): CUDSS_STATUS_ALLOC_FAILED #239
Copy link
Copy link
Open
colmap/colmap
#4217Description
Reproducible cuDSS allocation failure during global_positioning on 4262-image fisheye dataset
I am seeing a reproducible cuDSS allocation failure during
global_positioning on a 4262-image fisheye dataset.
This looks similar to #215 / #191, but I’m posting full reproducible details from my setup.
Environment
- OS: Ubuntu 24.04.3 LTS
- CPU: AMD Ryzen 9 5900XT (32 logical CPUs)
- RAM: 64 GB
- GPU: NVIDIA GeForce RTX 3090 (24 GB VRAM)
- NVIDIA driver: 570.211.01
Build / Versions
- GLOMAP binary:
/home/ubu/dev/src/glomap_cuda120/install/bin/glomap glomap --helpshows:This version was compiled with CUDA!-
Local glomap repo revision used for this build:
14e9104c2e60bc3f68fbc8b645b0d79a62f97f7a -
COLMAP in the same pipeline:
COLMAP 3.13.0 (Commit d69fddb2 on 2026-02-15 with CUDA)
Dataset
- 4262 images (2-camera rig), total image size approximately 2.4–2.6 GB
- Image dimensions:
3840x3840
Feature extraction settings
SiftExtraction.max_image_size=3200SiftExtraction.max_num_features=16384SiftExtraction.peak_threshold=0.0033- GPU feature extraction enabled
Extracted features (from logs)
- min: 10426
- max: 21180
- mean: 15638.68 per image
Command used
/home/ubu/dev/src/glomap_cuda120/run_glomap_120.sh mapper \
--database_path /.../database/database.db \
--image_path /.../distorted/images \
--output_path /.../distorted/sparse \
--GlobalPositioning.use_gpu 1 \
--GlobalPositioning.gpu_index -1 \
--BundleAdjustment.use_gpu 1 \
--BundleAdjustment.gpu_index -1 \
--BundleAdjustment.optimize_intrinsics 1 \
--BundleAdjustment.optimize_principal_point 1 \
--BundleAdjustment.optimize_rig_poses 1 \
--BundleAdjustment.max_num_iterations 300 \
--skip_pruning 0
I also reproduced the same failure with --skip_pruning 1.
Error output
During Running global positioning ...:
Linear solver fatal error: cudssExecute with CUDSS_PHASE_FACTORIZATION failed Got error: CUDSS_STATUS_ALLOC_FAILED
Terminating: Linear solver failed due to unrecoverable non-numeric causes.
Ceres Solver Report ... Termination: FAILURE
Important behavior
Even after this solver failure, mapper continues and exports a COLMAP model.
In that case, the model quality can be invalid or corrupted (for example, extremely large reprojection error in analyzer output).
If I force CPU solvers
(GlobalPositioning.use_gpu=0, BundleAdjustment.use_gpu=0),
the reconstruction completes with normal stats.
Questions
-
Is this expected behavior when cuDSS runs out of memory
(i.e. continue execution and still export a model),
or should
mapperfail hard in this case? -
Are there any recommended flags or settings to reduce cuDSS memory pressure
in
global_positioningfor datasets of this scale?
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels