What does this library for?

[gfxcc]

Could you explain the features of your library and what problems does it try to solve?

[yuanzhubi]

For non-blocking I/O multiplexing network programming, it is difficult to deal with several combination of context in one thread. For programming convenient, we have creating one thread per request models to splitting the context. However, it leads to large cost in thread context switch, and stack cost of the threads when the count of threads is over 1000. So we also have micro-thread, or stackful-coroutine to manually control the context switch and low down the stack cost of each coroutines. See
https://github.com/boostorg/context
https://github.com/boostorg/coroutine
https://github.com/yyzybb537/libgo
https://github.com/Tencent/libco
https://github.com/owt5008137/libcopp

However, when the count of coroutines is over 1M, we notice their stack cost is unbearable. For safety, in most of the coroutines library, the stack size is initialize to a big count(128K for example, which will use mmap to malloc in ptmalloc library). Some of them use an automatic growing stack trick, with a write protected memory page at the end to trigger further growth. However, it is also wast memory address, and waste a memory page at least(4K or 8K). All the solutions above are not affordable in 8G memory server for large number connections.

The stack allocated for each coroutines is to save the context. However the size of data need to be save/restore in stack(For std::string object, the size is not object.length() but sizeof(std::string) ) during each coroutines context switching is usually very small. Most stack is prepared only for some function that cost large number of stack, such as snprintf(useful in string formatting or logging, which cost about 3KB stack in my x64 glibc), quick sort(in the worst case, the size of stack cost is linear to the input array size!)

call_in_stack is to solve the problems above.

char *buffer = new char[4096]
call_in_stack(buffer, 4096, &printf, "Hello world!\n");

Now printf run in buffer as its stack but not cost the stack of the current coroutines. What's more, if the buffer is a (thread local) global variable, other coroutines (in the same thread ) can call_in_stack(buffer, 4096, xxxx) as well. They can share the buffer, which leads to no conflict if the function xxx does not call (neither direct nor not direct) the coroutines switching function(which means only one coroutines is visiting the buffer in one time)!

So all the function, that not does not call the coroutines switching function, can run in the same (thread local) global buffer. This reduces the need of stack cost of coroutines or thread if you need.

Using call_in_stack with sigaltstack (see http://man7.org/linux/man-pages/man2/sigaltstack.2.html) to avoid the stack cost of signal handler in current coroutine, my project creates coroutines with only 512+24B stack(24B is for n-way associative cache optimization ) for network programming. Thats's enough!

[yuanzhubi]

So generally speaking, this library is to help to reduce the stack cost of the stackful-coroutines library. We can create coroutines with much smaller stack!

In another view, just think the function entry of the stackful-corouines is a "kernel interrupt handler", and call_in_stack is as to "return to the user space codes from kernel codes"(with interrupt mask) ! When you return from call_in_stack, it is like you are doing a "system call to kernel codes and may lead to coroutines switching"!

This is a new programming pattern, adding a coroutines states between traditional kernel states and user states to splitting the context！Yes, we are facing several non-blocking I/O requests like interrupt that kernel need to deal with(previous request not finished, new one comes.), so we manually create a "coroutine kernel", leaving the codes that can run in call_in_stack as the "real user space codes. "