Detect non-determinism for some cases on `maisi_ct_generative` bundle

[yiheng-wang-nv]

I tested the following 4 cases:

1. {'body_region': ['chest'], 'anatomy_list': ['aorta'], 'output_size': [256, 256, 128], 'spacing': [1, 1, 1], 'num_output_samples': 1}
2. {'body_region': ['chest'], 'anatomy_list': ['left lung upper lobe'], 'output_size': [256, 256, 128], 'spacing': [1, 1, 1], 'num_output_samples': 1}
3. {'body_region': ['abdomen'], 'anatomy_list': ['liver'], 'controllable_anatomy_size': [['hepatic tumor', 0.3], ['liver', 0.5]], 'output_size': [256, 256, 128], 'spacing': [1, 1, 1], 'num_output_samples': 1}
4. {'body_region': ['abdomen'], 'anatomy_list': ['spleen'], 'output_size': [256, 256, 128], 'spacing': [1, 1, 1], 'num_output_samples': 1}

For each case, I run inference twice with random_seed = 0, and only case 3 produce same outputs.

Hi @dongyang0122 @guopengf @Can-Zhao , could you help to confirm this issue? Thanks!

[yiheng-wang-nv]

I will add some tests to check the determinism after fixing the issue

[yiheng-wang-nv]

Did some quick check with @dongyang0122 , it seems remove controllable_anatomy_size will have the non-determinism issue

[Can-Zhao]

I checked the code. It might be the reason that set_deterministic(seed) is defined inside LDMSampler. But function find_mask() is defined outside LDMSampler, so it may not follow set_deterministic(seed). Tutorial does not observe this because we set set_determinism(seed) globally in the inference.py.

A quick fix is to add
set_determinism(seed) to
"run": [
"$@ldm_sampler.sample_multiple_images(@num_output_samples)"
],

[yiheng-wang-nv]

Hi @Can-Zhao , thanks for the explain. I added set_determinism in the sample_multiple_images function, and can see that the mask file found is the same for multiple runs. However, the generated images are still different. The log is attached
maisi_determinism_check.log

[Can-Zhao]

output name is defined to has output_postfix = datetime.now().strftime("%Y%m%d_%H%M%S_%f")
So their filenames cannot be the same

[yiheng-wang-nv]

Hi @Can-Zhao , thanks, actually the name does not matter. If the content of two files are the same, will have same md5sum. We can get same md5sum if having controllable_anatomy_size

[yiheng-wang-nv]

maisi_with_controllable_anatomy_size.log

attached a new log that have same md5sum cc @Can-Zhao

[Can-Zhao]

I see. After the mask selection, we augmented the mask. The augmentation() is also outside LDMSampler.

[yiheng-wang-nv]

Thanks @Can-Zhao , but after multiple rounds of tests, I still meet the non-determinism issue. Could you help to double confirm if it works?