Skip to content
This repository was archived by the owner on Mar 9, 2026. It is now read-only.
This repository was archived by the owner on Mar 9, 2026. It is now read-only.

cuDSS OOM in global_positioning on RTX 3090 (24GB): CUDSS_STATUS_ALLOC_FAILED #239

@DuVogel87

Description

@DuVogel87

Reproducible cuDSS allocation failure during global_positioning on 4262-image fisheye dataset

I am seeing a reproducible cuDSS allocation failure during global_positioning on a 4262-image fisheye dataset.

This looks similar to #215 / #191, but I’m posting full reproducible details from my setup.


Environment

  • OS: Ubuntu 24.04.3 LTS
  • CPU: AMD Ryzen 9 5900XT (32 logical CPUs)
  • RAM: 64 GB
  • GPU: NVIDIA GeForce RTX 3090 (24 GB VRAM)
  • NVIDIA driver: 570.211.01

Build / Versions

  • GLOMAP binary: /home/ubu/dev/src/glomap_cuda120/install/bin/glomap
  • glomap --help shows: This version was compiled with CUDA!
  • Local glomap repo revision used for this build:
    14e9104c2e60bc3f68fbc8b645b0d79a62f97f7a
  • COLMAP in the same pipeline:
    COLMAP 3.13.0 (Commit d69fddb2 on 2026-02-15 with CUDA)

Dataset

  • 4262 images (2-camera rig), total image size approximately 2.4–2.6 GB
  • Image dimensions: 3840x3840

Feature extraction settings

  • SiftExtraction.max_image_size=3200
  • SiftExtraction.max_num_features=16384
  • SiftExtraction.peak_threshold=0.0033
  • GPU feature extraction enabled

Extracted features (from logs)

  • min: 10426
  • max: 21180
  • mean: 15638.68 per image

Command used

/home/ubu/dev/src/glomap_cuda120/run_glomap_120.sh mapper \
  --database_path /.../database/database.db \
  --image_path /.../distorted/images \
  --output_path /.../distorted/sparse \
  --GlobalPositioning.use_gpu 1 \
  --GlobalPositioning.gpu_index -1 \
  --BundleAdjustment.use_gpu 1 \
  --BundleAdjustment.gpu_index -1 \
  --BundleAdjustment.optimize_intrinsics 1 \
  --BundleAdjustment.optimize_principal_point 1 \
  --BundleAdjustment.optimize_rig_poses 1 \
  --BundleAdjustment.max_num_iterations 300 \
  --skip_pruning 0

I also reproduced the same failure with --skip_pruning 1.


Error output

During Running global positioning ...:

Linear solver fatal error: cudssExecute with CUDSS_PHASE_FACTORIZATION failed Got error: CUDSS_STATUS_ALLOC_FAILED
Terminating: Linear solver failed due to unrecoverable non-numeric causes.
Ceres Solver Report ... Termination: FAILURE

Important behavior

Even after this solver failure, mapper continues and exports a COLMAP model.

In that case, the model quality can be invalid or corrupted (for example, extremely large reprojection error in analyzer output).

If I force CPU solvers (GlobalPositioning.use_gpu=0, BundleAdjustment.use_gpu=0), the reconstruction completes with normal stats.


Questions

  1. Is this expected behavior when cuDSS runs out of memory (i.e. continue execution and still export a model), or should mapper fail hard in this case?
  2. Are there any recommended flags or settings to reduce cuDSS memory pressure in global_positioning for datasets of this scale?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions