idalib: `open_database` fails to handle existing database files and exits prematurely

[Young-Lord]

Description

When using ida as backend, analyzing any file with an existing database (packed or unpacked) will cause capa to exit with return code 1, without any logs or cleanup.

I tried ida -R <file>, and it says Switch '-R' can be used only when loading a new file, which I believe is the reason behind failed open_database call.
I can confirm the error is introduced in 0686305 , which passes switch -R to idapro.open_database. idapro.open_database will directly cause exit_group(1) instead of returning an error code.

Steps to Reproduce

1. Open tests/data/009c2377b67997b0da1579f4bbc822c1.exe_ in IDA
2. Save database (produce .i64 file in same directory) and exit IDA
3. Run capa -b ida tests/data/009c2377b67997b0da1579f4bbc822c1.exe_ -d

Note, in current version of capa (commit 6579e01), because of the lack of idapro.close_database, running capa -b <file> will cause idapro to leave behind unpacked database files (*.id0 *.id1 *.id2 *.nam *.til); When trying to open_database against the file again, it will pick up the unpacked databases and cause the same issue.

Expected behavior:

capa displays analyze report correctly.

Actual behavior:

capa exits with return code 1 after capa.loader: idalib: opening database... | loader.py:396. Any code after idapro.open_database is not executed.

Versions

capa: commit 6579e01
IDA Pro: Version 9.3.260213.91fc47de Linux x86_64 (64-bit address size)
Python: 3.13.5
OS: Arch Linux

Additional Information

Related issue: #2808

---

I'd be happy to implement the fix once we decide on a solution.

[williballenthin]

good catch, thanks for calling this out.

we'll have to first check to see if an existing database is present. if so, then we should use that and not provide -R (but probably still provide the Lumina flags). if not, then we should create the .i64 with the -R flag.