Overhead with variable-based `bp5` series

[aeriforme]

Describe the bug
Performance kill when opening/closing a variable-based bp5 series with many iterations.

To Reproduce
Standalone reproducer in Python:

import openpmd_api as io
from tqdm import tqdm
adios2_cfg = r'''
{
  "iteration_encoding": "variable_based",
  "adios2": {
    "engine": {
      "type": "bp5",
      "parameters": {
        "StatsLevel": "0"
      }
    }
  }
}
'''
OUTPUT_PATH = "myseries.bp5"
series = io.Series(OUTPUT_PATH, io.Access_Type.create_linear, adios2_cfg)
N = 1000000
for i in tqdm(range(N)):
    it = series.snapshots()[i]
    it.close()

Expected behavior
I expect much less overhead just to open/close the iterations, and not to change in time.

Software Environment

• version of openPMD-api: 0.17.0
• installed openPMD-api via: conda-forge
• operating system:
• machine: Perlmutter at NERSC - login nodes
• name and version of Python implementation: CPython 3.11.14
• version of ADIOS2: 2.11.0
• name and version of MPI: N/A

Additional context
The overhead builds up as the loop moves through the iterations.
Starts around ~1k iterations/s and goes down to ~200 iterations/s after ~100k iterations.

[ax3l]

Thank you for posting the reproducer, @aeriforme!

ccing @guj and @franzpoeschel : let us benchmark this deeper. I am surprised this is so slow:

• 1ms per open when we start seems excessive
• linear slowdown over time as we add more snapshots is probably a data container issue that we can solve, too

What do you think?

[franzpoeschel]

I am off-work this week, but will have a look next week. Achieving linear performance with the number of Iterations should be a goal for this release cycle, but we are not there yet.
I think, with the current release, we have this goal down inside the ADIOS2 backend. For our frontend, we will have to move to a workflow where old Iterations are fully erased from our datastructures upon opening a new one. This has not been done yet, since the details of handle invalidation can get tricky.

One suspicion: indexOf() inside the Iteration flushing logic is linear in the number of Iterations. We should be able to optimize this; after 100K Iterations, the cost for this operation may become dominant.

[guj]

At the mean time, can you see how long it takes to load the file with ADIOS python library?
e.g.
f = adios2.FileReader(str(fname))
n = f.num_steps()
If you point the file location, I can try it.

[aeriforme]

@guj I am not reading any file, I'm creating one from scratch.

[guj]

I thought you have the file already.
nevermind, let me try out on my laptop

[guj]

ok with fix, it takes 90 secs to store 1million iterations on my Mac.
100%|██████████████████████████████████| 1000000/1000000 [01:29<00:00, 11215.49it/s]