Iceoryx2 in Kubernetes, FailedToDeliverSignal.

[DrissiReda]

Hello, did anyone ever use iceoryx2 in Kubernetes?

I want to run a pod with multiple containers that communicate with each other via iceoryx2. I used the event based communication rust example.

With docker compose it works well simply by mounting the same /dev/shm to all services.

I thought it would be the same with Kubernetes, simply creating an emptyDir and mounting it to /dev/shm and adding shareProcessNamespace: true but that is clearly not enough.

I don't think the issue is with the code because it still works well and communication indeed does take place, I just keep getting this warning.

I'm getting this warning by the sender when it is supposed to notify the next container.

Does anybody have any idea?

Here is the warning:

45 [W] Notifier { listener_connections: SingleThreaded { data: ListenerConnections { connections: [UnsafeCell { .. }, UnsafeCell { .. }, 
UnsafeCell { .. }, UnsafeCell { .. }, UnsafeCell { .. }, UnsafeCell { .. }, UnsafeCell { .. }, UnsafeCell { .. }, UnsafeCell { .. }, UnsafeCell { .. }, 
UnsafeCell { .. }, UnsafeCell { .. }, UnsafeCell { .. }, UnsafeCell { .. }, UnsafeCell { .. }, UnsafeCell { .. }], service_state: ServiceState { 
dynamic_storage: Storage { shm: SharedMemory { name: FileName { value: FixedSizeByteString<255> { len: 94, data: 
"iox2_305ad9523c6b202364d581359ec3d2c5743e42e7_626772a4b6690f9fccde77d45ea5cec2a3f01513.dynamic" } }, size: 2739, 
base_address: 0x7f1ed0413000, has_ownership: false, file_descriptor: FileDescriptor { value: 11, is_owned: true }, memory_lock: 
None, mapping_offset: 0 }, name: FileName { value: FixedSizeByteString<255> { len: 40, data: 
"626772a4b6690f9fccde77d45ea5cec2a3f01513" } }, _phantom_data: 
PhantomData<iceoryx2::service::dynamic_config::DynamicConfig> }, additional_resource: NoResource, static_config: StaticConfig { 
service_id: ServiceId(RestrictedFileName { value: FixedSizeByteString<64> { len: 40, data: 
"626772a4b6690f9fccde77d45ea5cec2a3f01513" } }), service_name: ServiceName { value: FixedSizeByteString<255> { len: 29, data: 
"udp_source_to_processor_event" } }, attributes: AttributeSet(FixedSizeVec<iceoryx2::service::attribute::Attribute, 8> { 
MetaVec<iceoryx2::service::attribute::Attribute, iceoryx2_bb_elementary::relocatable_ptr::GenericRelocatablePointer> { capacity: 8, 
len: 0, is_initialized: true, content: [  ] } }), messaging_pattern: Event(StaticConfig { max_notifiers: 16, max_listeners: 16, max_nodes: 
36, event_id_max_value: 255, deadline: None, notifier_created_event: None, notifier_dropped_event: None, notifier_dead_event: 
None }) }, shared_node: SharedNode { id: NodeId(UniqueSystemId { value: 15161197936293908601288785935, pid: 15, 
creation_time: Time { clock_type: Monotonic, seconds: 2706293, nanoseconds: 821890187 } }), details: NodeDetails { executable: 
FileName { value: FixedSizeByteString<255> { len: 10, data: "udp-source" } }, name: NodeName { value: FixedSizeByteString<128> { 
len: 0, data: "" } }, config: Config { global: Global { root_path: Path { value: FixedSizeByteString<255> { len: 14, data: "/dev/shm/iceoryx2/" } 
}, prefix: FileName { value: FixedSizeByteString<255> { len: 5, data: "iox2_" } }, service: Service { directory: Path { value: 
FixedSizeByteString<255> { len: 8, data: "services" } }, data_segment_suffix: FileName { value: FixedSizeByteString<255> { len: 5, data: 
".data" } }, static_config_storage_suffix: FileName { value: FixedSizeByteString<255> { len: 8, data: ".service" } }, 
dynamic_config_storage_suffix: FileName { value: FixedSizeByteString<255> { len: 8, data: ".dynamic" } }, creation_timeout: 500ms, 
connection_suffix: FileName { value: FixedSizeByteString<255> { len: 11, data: ".connection" } }, event_connection_suffix: FileName { 
value: FixedSizeByteString<255> { len: 6, data: ".event" } }, blackboard_mgmt_suffix: FileName { value: FixedSizeByteString<255> { 
len: 16, data: ".blackboard_mgmt" } }, blackboard_data_suffix: FileName { value: FixedSizeByteString<255> { len: 16, data: 
".blackboard_data" } } }, node: Node { directory: Path { value: FixedSizeByteString<255> { len: 5, data: "nodes" } }, monitor_suffix: 
FileName { value: FixedSizeByteString<255> { len: 13, data: ".node_monitor" } }, static_config_suffix: FileName { value: 
FixedSizeByteString<255> { len: 8, data: ".details" } }, service_tag_suffix: FileName { value: FixedSizeByteString<255> { len: 12, data: 
".service_tag" } }, cleanup_dead_nodes_on_creation: true, cleanup_dead_nodes_on_destruction: true } }, defaults: Defaults { 
publish_subscribe: PublishSubscribe { max_subscribers: 8, max_publishers: 2, max_nodes: 20, subscriber_max_buffer_size: 2, 
subscriber_max_borrowed_samples: 2, publisher_max_loaned_samples: 2, publisher_history_size: 0, enable_safe_overflow: true, 
unable_to_deliver_strategy: Block, subscriber_expired_connection_buffer: 128 }, event: Event { max_listeners: 16, max_notifiers: 16, 
max_nodes: 36, event_id_max_value: 255, deadline: None, notifier_created_event: None, notifier_dropped_event: None, 
notifier_dead_event: None }, request_response: RequestResonse { enable_safe_overflow_for_requests: true, 
enable_safe_overflow_for_responses: true, max_active_requests_per_client: 4, max_response_buffer_size: 2, max_servers: 2, 
max_clients: 8, max_nodes: 20, max_borrowed_responses_per_pending_response: 2, max_loaned_requests: 2, 
server_max_loaned_responses_per_request: 2, client_unable_to_deliver_strategy: Block, server_unable_to_deliver_strategy: Block, 
client_expired_connection_buffer: 128, enable_fire_and_forget_requests: true, server_expired_connection_buffer: 128 }, blackboard: 
Blackboard { max_readers: 8, max_nodes: 20 } } } }, monitoring_token: UnsafeCell { .. }, registered_services: RegisteredServices { 
data: Mutex { data: {ServiceId(RestrictedFileName { value: FixedSizeByteString<64> { len: 40, data: 
"626772a4b6690f9fccde77d45ea5cec2a3f01513" } }): (ContainerHandle { index: 0, container_id: 5 }, 1), ServiceId(RestrictedFileName 
{ value: FixedSizeByteString<64> { len: 40, data: "f3a12830905c19d9a0f610df1e08b64dd0bea1ed" } }): (ContainerHandle { index: 0, 
container_id: 2 }, 1)}, poisoned: false, .. } }, signal_handling_mode: HandleTerminationRequests, _details_storage: Storage { name: 
FileName { value: FixedSizeByteString<255> { len: 4, data: "node" } }, config: Configuration { path: Path { value: 
FixedSizeByteString<255> { len: 49, data: "/dev/shm/iceoryx2/nodes/15161197936293908601288785935" } }, suffix: FileName { value: 
FixedSizeByteString<255> { len: 8, data: ".details" } }, prefix: FileName { value: FixedSizeByteString<255> { len: 5, data: "iox2_" } } }, 
has_ownership: false, file: File { path: Some(FilePath { value: FixedSizeByteString<255> { len: 67, data: "/dev/shm/iceoryx2/
nodes/15161197936293908601288785935/iox2_node.details" } }), file_descriptor: FileDescriptor { value: 3, is_owned: true }, 
has_ownership: false }, len: 1753 } }, static_storage: Storage { name: FileName { value: FixedSizeByteString<255> { len: 40, data: 
"626772a4b6690f9fccde77d45ea5cec2a3f01513" } }, config: Configuration { path: Path { value: FixedSizeByteString<255> { len: 22, 
data: "/dev/shm/iceoryx2/services" } }, suffix: FileName { value: FixedSizeByteString<255> { len: 8, data: ".service" } }, prefix: FileName { 
value: FixedSizeByteString<255> { len: 5, data: "iox2_" } } }, has_ownership: false, file: File { path: Some(FilePath { value: 
FixedSizeByteString<255> { len: 76, data: "/dev/shm/iceoryx2/services/iox2_626772a4b6690f9fccde77d45ea5cec2a3f01513.service" } }), 
file_descriptor: FileDescriptor { value: 10, is_owned: true }, has_ownership: false }, len: 224 } }, list_state: UnsafeCell { .. } } }, 
default_event_id: TriggerId(0), event_id_max_value: 255, dynamic_notifier_handle: Some(ContainerHandle { index: 0, container_id: 4 
}), notifier_id: UniqueNotifierId(UniqueSystemId { value: 173687987221341145061855854607, pid: 15, creation_time: Time { 
clock_type: Monotonic, seconds: 2706293, nanoseconds: 825710062 } }), on_drop_notification: None, node_id: 
NodeId(UniqueSystemId { value: 15161197936293908601288785935, pid: 15, creation_time: Time { clock_type: Monotonic, seconds: 
2706293, nanoseconds: 821890187 } }) } 

| Unable to send notification via connection Connection { notifier: Notifier { sender: UnixDatagramSender { socket: 
UnixDatagramSocket { name: FilePath { value: FixedSizeByteString<255> { len: 55, data: "/dev/shm/iceoryx2/
iox2_252489222219920036535883268117.event" } }, is_non_blocking: true, file_descriptor: FileDescriptor { value: 12, is_owned: true } 
} }, name: FileName { value: FixedSizeByteString<255> { len: 30, data: "252489222219920036535883268117" } } }, listener_id: 
UniqueListenerId(UniqueSystemId { value: 252489222219920036535883268117, pid: 21, creation_time: Time { clock_type: 
Monotonic, seconds: 2706303, nanoseconds: 802566275 } }), node_id: NodeId(UniqueSystemId { value: 
14687546625650675411064979477, pid: 21, creation_time: Time { clock_type: Monotonic, seconds: 2706303, nanoseconds: 
796213498 } }) } due to FailedToDeliverSignal. 

[elBoberido]

@DrissiReda we did not yet try to run iceoryx2 within a Kubernets pod. In theory this should work. Only inter-pod communication is a bit challenging. From my limited understanding of Kubernets, it should be possible to have a shared memory communication between container in the same pod.

Are you able to run the basic pub-sub example in multiple container?
Can you try to run the basic event example with log level set to tracing? You just need to set the IOX2_LOG_LEVEL environment variable to trace.

This will give us more information.

By default, iceoryx2 uses UnixDomainSockets for Events on Linux. They are created in /tmp/iceoryx2 by default. You could check if they are visible between the container.

If the events with the unix domain sockets do not work, there is also an alternative implementation with semaphores which are placed in shared memory. So if the basic pub-sub example works, it would be worth a try to check if the semaphore based event implementation works across container boundaries.

[DrissiReda]

I might have misspoke, it still doesn't work and I have failures in notifications.

I would've liked a definitive way of exclusively waiting for events even if it is slower, simply so I can benchmark correctly the tradeoff of latency/cpu usage between busy polling and event based communication.

Intra pod communication would certainly be more challenging which is why I'm limiting myself to inter container communication inside the same pod.

I don't think this is a sweet spot setup issue since docker handles 20000Hz without a problem and kubernetes struggles with 2200Hz

[elBoberido]

Oh, there might still be a warning that an event cannot be delivered if there is a fast producer and a slow consumer, e. g. when the OS does not schedule the consumer for a long time. Losing events is not that critical, since you will be woken up anyway and can consume the samples. What might be an issue is when you lose samples. With a fast producer and slow consumer, you have basically three options to handle backpressure

• increase the queue size between producer and consumer
• discard the samples (oldest or newest)
• block producer once the queue is full

[elBoberido]

Regarding the frequency. We once had a user where SELinux or AppArmor was active in a Docker setup, which massivly slowed down the events, since it intercepted all UDS messages to analyze them. Maybe Kubernets does something similar. In this case, semaphores might be helpful. I can give you instructions on how to use them later today. Unfortunately, you need a small patch if you also want to use them with C++

[DrissiReda]

I'm very interested, I am content with rust for now.

[DrissiReda]

Oh, there might still be a warning that an event cannot be delivered if there is a fast producer and a slow consumer, e. g. when the OS does not schedule the consumer for a long time. Losing events is not that critical, since you will be woken up anyway and can consume the samples. What might be an issue is when you lose samples. With a fast producer and slow consumer, you have basically three options to handle backpressure

* increase the queue size between producer and consumer

* discard the samples (oldest or newest)

* block producer once the queue is full

I don't have selinux enabled, I tried following your ideas

Here is how I create my producers:

.service_builder(&producer_service_name)
.publish_subscribe::<CustomStruct>()
.history_size(128000)
.subscriber_max_buffer_size(512000)
.subscriber_max_borrowed_samples(256000)
.enable_safe_overflow(true)
.open_or_create()?;

And consumers:

.service_builder(&consumer_service_name)
.publish_subscribe::<CustomStruct>()
.history_size(128000)
.subscriber_max_buffer_size(512000)
.subscriber_max_borrowed_samples(512000)
.open()

Parameter choice is arbitrary, but it helped me raise the frequency.

Good thing is I raised frequency of generated messages to 6000 instead of 2200 thanks to changing these variables. At 6000 I lose around 3 messages at once every 15 seconds. And processing speed is around 4500 msgs/s

However the more I raise the generated messages frequency, the more the processing frequency is raised and messages loss.

For example at 100 000 generated msgs/s. I can see 12 000 processed msgs/s and a loss of around 15msgs at once every 15 or so seconds.

Increasing to 1M generated msgs/s shows me that the 12 000 processed msgs/s is the true limit of my system since it doesn't increase beyond that.
Memory used is around 100Mb, and the size of each message is a struct of 10 u64 ints.

There doesn't seem to be a hard limit on throughput, but more an overhead that reduces the processing speed but I don't know what it is.

EDIT: going back to busy polling. I have no loss in messages, but the throughput is still limited to 14K messages/s