System time regression causes heartbeat failure and app termination

[owny990312]

Environment

iceoryx 2.95.4

Problem

Our system has time synchronization, which has a small probability of causing time regression. Through manual time regression testing, we found this causes iceoryx to crash (possibly due to the heartbeat mechanism causing iceoryx to consider nodes dead and forcibly release resources). Can we avoid this problem by using steady_clk?

[elfenpiff]

In general, you can avoid this by using the monotonic clock, but some constructs like semaphores do not support this by default, only with non-POSIX-compliant like sem_clockwait instead of sem_wait. Our central daemon is based on some of those constructs and therefore this cannot be realized easily.

With iceoryx2 we have solved the problem. There is even an example called "health monitoring" which addresses the individual aspects.
I would recommend to switch to iceoryx2 if possible.

[owny990312]

In general, you can avoid this by using the monotonic clock, but some constructs like semaphores do not support this by default, only with non-POSIX-compliant like sem_clockwait instead of sem_wait. Our central daemon is based on some of those constructs and therefore this cannot be realized easily.通常可以通过使用单调时钟来避免这个问题，但像信号量这样的某些构造默认不支持这一点，只能通过非 POSIX 兼容的 sem_clockwait 而不是 sem_wait 来实现。我们的中央守护进程基于这些构造，因此这不容易实现。

With iceoryx2 we have solved the problem. There is even an example called "health monitoring" which addresses the individual aspects.使用 iceoryx2 我们已经解决了这个问题。甚至有一个名为“健康监控”的示例，该示例涵盖了各个方面的内容。 I would recommend to switch to iceoryx2 if possible.如果可能的话，我建议切换到 iceoryx2。

Could you please let us know when the official version of iceoryx2 will be released? We are very much looking forward to it.

[elfenpiff]

@owny990312 iceoryx2 is already production ready and the version v0.6.1 was released in May and is already used for medical devices, high frequency trading and automotive.

The reason why we decided not to go for v1 + is that not all the features we envision are yet in iceoryx2. Some language bindings like go or advanced messaging patterns like blackboard or not yet available.

But the existing features like publish subscribe, request response, events are already much more advanced then their classic iceoryx counterpart.

[owny990312]

@owny990312 iceoryx2 is already production ready and the version v0.6.1 was released in May and is already used for medical devices, high frequency trading and automotive.iceoryx2 已经具备生产环境使用条件，v0.6.1 版本于五月发布，并已被用于医疗设备、高频交易和汽车领域。

The reason why we decided not to go for v1 + is that not all the features we envision are yet in iceoryx2. Some language bindings like go or advanced messaging patterns like blackboard or not yet available.我们决定不采用 v1+ 的原因是，我们设想的一些功能在 iceoryx2 中尚未实现。例如，一些语言绑定如 go 或高级消息模式如 blackboard 还未提供。

But the existing features like publish subscribe, request response, events are already much more advanced then their classic iceoryx counterpart.但现有的功能如发布/订阅、请求/响应、事件等，已经比经典 iceoryx 的对应功能先进得多。

We sincerely appreciate your commitment and contributions. Does iceoryx2 still have introspection tools to monitor subscriptions and memory usage?"

[elfenpiff]

@owny990312 Yes it has. The tool is called iox2 and can be called inside the main repo with

cargo build
cargo --bin iox2 -- --list

and the list command shows you all available commands.

In iceoryx2 there is no more need to monitor the memory usage since.

1. iceoryx2 guarantees you that you never run out of memory
2. acquires only the resources you need to handle the created communication setup
3. and has therefore no central memory configuration

The pain with classic iceoryx was that you needed to monitor the memory so that you were roughly able to configure the central payload memory block in roudi. But iceoryx2 is completely decentral, no longer requires a central payload memory block or central daemon and all the services acquire exactly the memory they need and not more - which makes iceoryx2 incredibly memory efficient.

[owny990312]

@owny990312 Yes it has. The tool is called iox2 and can be called inside the main repo with是的，这个工具叫做 iox2 ，可以在主仓库中调用它。

cargo build
cargo --bin iox2 -- --list
and the list command shows you all available commands.

In iceoryx2 there is no more need to monitor the memory usage since.在 iceoryx2 中，您不再需要监控内存使用情况。

1. iceoryx2 guarantees you that you never run out of memoryiceoryx2 保证您不会内存不足
2. acquires only the resources you need to handle the created communication setupiceoryx2 只会获取处理创建的通信设置所需的资源
3. and has therefore no central memory configuration并且因此没有中央内存配置

The pain with classic iceoryx was that you needed to monitor the memory so that you were roughly able to configure the central payload memory block in roudi.经典 iceoryx 的痛点在于你需要监控内存，以便大致能够配置 roudi 中的中央负载内存块。 But iceoryx2 is completely decentral, no longer requires a central payload memory block or central daemon and all the services acquire exactly the memory they need and not more - which makes iceoryx2 incredibly memory efficient.但 iceoryx2 完全去中心化，不再需要中央负载内存块或中央守护进程，所有的服务只会获取它们所需的精确内存，而不会更多 - 这使得 iceoryx2 极其内存高效。

As far as I know, iox2 does not have a 'listen' mode, only a 'waitset' mode. What examples should I look at?

[elfenpiff]

@owny990312 iceoryx2 does not have a Listener as you understand it from classic iceoryx.

With classic iceoryx listener I mean a construct that starts a background thread and works similar to a waitset. With iceoryx2 you have to build this yourself, but it should be easy. Just create a thread on your own, pack the waitset in there and handle all incoming events.

A good starting point is the event_multiplexing example.

iceoryx2 has also a Listener but it is something completely different. It is one communication endpoint of the event messaging pattern and allows the user to wait for an incoming event which was sent with the Notifier the other communication endpoint of the event messaging pattern. The usage is illustrated in the event example.

[elBoberido]

@owny990312 I guess you are experiencing a variation of #2193 and/or #1361

That was one of the reasons for the big refactoring which ended up becoming iceoryx2

[owny990312]

@owny990312 I guess you are experiencing a variation of #2193 and/or #1361我想你遇到的问题是 #2193 和/or #1361 的变体

That was one of the reasons for the big refactoring which ended up becoming iceoryx2这是一次大规模重构的原因之一，最终演变成了 iceoryx2

Yes, I think it's very relevant.

I have a question: since it will take time for us to integrate iox2, can we directly modify the PROCESS_KEEP_ALIVE_TIMEOUT in iox1 to make it a bit larger to ensure better timeout tolerance? However, timeout monitoring should still be enabled because there might be cases where certain nodes coredump, causing the shared memory they occupy to not be released, which could lead to memory shortage for other normal nodes.

[elfenpiff]

can we directly modify the PROCESS_KEEP_ALIVE_TIMEOUT in iox1 to make it a bit larger to ensure better timeout tolerance?

You should be able to do this with the -t parameter in iox-roudi. So with iox-roudi -t 10 the cleanup is delayed by additional 10 seconds.

causing the shared memory they occupy to not be released, which could lead to memory shortage for other normal nodes.

With iceoryx2 this issue is completely fixed.

[owny990312]

can we directly modify the PROCESS_KEEP_ALIVE_TIMEOUT in iox1 to make it a bit larger to ensure better timeout tolerance?我们是否可以直接在 iox1 中修改 PROCESS_KEEP_ALIVE_TIMEOUT 使其稍微大一些，以确保更好的超时容错？

You should be able to do this with the -t parameter in iox-roudi. So with iox-roudi -t 10 the cleanup is delayed by additional 10 seconds.你可以在 -t 中使用 iox-roudi 参数做到这一点。因此，通过 iox-roudi -t 10 ，清理操作会被额外延迟 10 秒。

causing the shared memory they occupy to not be released, which could lead to memory shortage for other normal nodes.这会导致它们占用的共享内存不被释放，从而可能导致其他正常节点出现内存不足的情况。

With iceoryx2 this issue is completely fixed.在 iceoryx2 中，这个问题已经被完全修复。

But from looking at the code, these parameters (-t/-k) might not directly take effect， iox still kills my program , because the core timeout logic seems to depend on PROCESS_KEEP_ALIVE_TIMEOUT. Is this accurate?

void ProcessManager::monitorProcesses() noexcept
{
static_assert(runtime::PROCESS_KEEP_ALIVE_TIMEOUT > runtime::PROCESS_KEEP_ALIVE_INTERVAL,
"keep alive timeout too small");
auto timeout = runtime::PROCESS_KEEP_ALIVE_TIMEOUT.toMilliseconds();
auto heartbeatIterator = m_heartbeatPool->begin();
while (heartbeatIterator != m_heartbeatPool->end())
{
auto currentHeartbeatIterator = heartbeatIterator++;
auto elapsedMilliseconds = currentHeartbeatIterator->elapsed_milliseconds_since_last_beat();
if (elapsedMilliseconds > timeout)
{
bool removed{false};
auto processIterator = m_processList.begin();
while (processIterator != m_processList.end())
{
if (processIterator->getHeartbeatPoolIndex() != currentHeartbeatIterator.to_index())
{
++processIterator;
continue;
}

            IOX_LOG(Warn,
                    "Application " << processIterator->getName() << " not responding (last response "
                                   << elapsedMilliseconds << " milliseconds ago) --> removing it");

            removed = removeProcessAndDeleteRespectiveSharedMemoryObjects(
                processIterator, TerminationFeedback::DO_NOT_SEND_ACK_TO_PROCESS);
            break;
        }
        if (!removed)
        {
            IOX_LOG(Warn,
                    "Could not find application for corresponding heartbeat! HeartbeatPoolIndex: "
                        << currentHeartbeatIterator.to_index());
            m_heartbeatPool->erase(currentHeartbeatIterator);
        }
    }
}

}

[elBoberido]

@owny990312 there is currently no option to configure the PROCESS_KEEP_ALIVE_TIMEOUT. See also #2193 (comment)