Manually close deleted Comments and Downtimes

[yhabteab]

tests: Simulate vanishing Comments and Downtimes

Ensure that removed Comment and Downtime objects attached to Hosts,
vanishing after a config pack redeployment, are correctly closed by
Icinga DB.

---

icingadb: Manually close deleted Comments and Downtimes

It is possible for objects to vanish in Icinga 2, e.g., when a new
config pack is deployed as the Director does. Icinga DB already handles
this within its synchronization logic, explicitly deleting removed
objects.

However, these deleted objects might have open logical dependencies,
such as entries in the history tables. For example, only deleting
a Downtime object is not enough, as both its downtime_history and
sla_history_downtime entries needs to be updated as well.

In this example, these updates are required to not show the Downtimes
anymore, have a valid SLA calculation, and ensure a working downtime
history retention.

As Icinga DB does not receive the change from Icinga 2 over the usual
"icinga:history:stream:downtime" Redis stream, it must be reproduced or
mocked by Icinga DB itself. Now, Icinga DB is able to close the history
for both vanished Comment and Downtime objects itself.

Co-Authored-By: Yonas Habteab yonas.habteab@icinga.com

---

tests: Bump icinga-testing for Config Pack support

Icinga/icinga-testing#34

---

Fixes #910.

[lippserd]

@oxzi Would you like to do the first round of reviews here?

[oxzi]

In general and when excluding the open TODOs: LGTM, thanks!

I have spent a bit of a time testing this PR in different situations and it worked as expected. Btw, I have pushed a rebased version to the fake-downtime-end-event-review-oxzi branch, also including an initial draft for an integration test, 0414c2a.

I would say: Let's continue with this PR by also populating the other tables.

@yhabteab: If you currently do not have the time and want to hand this PR over, I am free to continue here.

Thanks again, looks good so far!

[yhabteab]

@yhabteab: If you currently do not have the time and want to hand this PR over, I am free to continue here.

I'm working on a completely different thing right now, so you can of course take over this :).

[oxzi]

@lippserd: If you want to, feel free to start reviewing this PR now.

[oxzi]

Added a check against deleted objects which aren't even in the database. This happened during multiple bigger redeployments in my test setup.

[Al2Klimov]

Indeed. If I add and remove a comment/downtime, I get two additional events in the history tab.

Except if I stop Redis between addition and removal and restart Icinga 2 afterwards to clear eventually buffered history events. Or if I rm /var/lib/icinga2/api/packages/_api/4b444b3c-7063-4318-93f2-8b41cb6ba1ec/conf.d/*/*.conf and restart Icinga. Or if I simply use curl -fvku iw2:iw2 -X DELETE -H 'Accept: application/json' https://127.0.0.1:5665/v1/objects/comments. Then I only get the addition events, even with snapshot packages.

👎The runtime behavior doesn't even change with this PR. See curl command above.

👍In my other scenarios I indeed get removal events for both types.

[oxzi]

Or if I simply use curl -fvku iw2:iw2 -X DELETE -H 'Accept: application/json' https://127.0.0.1:5665/v1/objects/comments. Then I only get the addition events, even with snapshot packages.

This is actually a loophole in Icinga 2. When directly deleting comment objects and not using the remove-comment API action, the object will be deleted, but Icinga 2 does not generate history.

https://github.com/Icinga/icinga2/blob/9bffe0616911d8ab0a8c589a6fc17eee6564bdf8/lib/icingadb/icingadb-objects.cpp#L2354-L2358

However, the change is being propagated as a runtime update, where Icinga DB acts upon.

icinga2   | [2026-01-27 16:03:40 +0000] information/HttpServerConnection: Request DELETE /v1/objects/comments (from [::ffff:172.18.0.1]:41148, user: root, agent: curl/8.17.0, status: OK) took total 0ms.
[...]
icingadb  | 2026-01-27T16:03:58.150Z    INFO    runtime-updates Deleted 2 Comment items
icingadb  | 2026-01-27T16:03:58.150Z    DEBUG   database        Executed "DELETE FROM \"comment\" WHERE id IN (?)" with 2 rows

While this is definitely something to look into, this is not really what this PR tries to address. The core are disappearing or vanishing objects, such as after a Director deployment.

this is not really what this PR tries to address. The core are disappearing or vanishing objects, such as after a Director deployment.