extmod/zephyr_kernel: NSI investigation - preserved for reference.

This commit contains an investigation into using Zephyr's Native Simulator
Infrastructure (NSI) for the POSIX/Unix threading layer. This work is being
preserved for future reference, but NSI is not suitable for MicroPython's
Unix port due to fundamental architectural incompatibility.

## What is NSI?

NSI (Native Simulator Infrastructure) is Zephyr's official POSIX threading
implementation, using cooperative threading to accurately simulate single-
core microcontroller behavior on native (Unix/Linux) systems.

## How NSI Works

NSI's nct.c (Native CPU Threading) module:
- Creates pthreads that immediately block on semaphores (nct_wait_until_allowed)
- Threads only execute when explicitly swapped in via nct_swap_threads()
- Only one thread runs at a time (enforced single-core MCU simulation)
- Requires explicit scheduler calls for any thread to make progress

## Why NSI Doesn't Work for MicroPython Unix Port

**Architectural Mismatch:**
- NSI: Cooperative threading (explicit scheduler swaps required)
- MicroPython: Expects preemptive threading (OS scheduler manages threads)

**Result:**
- Threads created successfully via nct_new_thread()
- Threads immediately blocked on semaphore waiting for nct_swap_threads()
- MicroPython never calls swap functions (expects automatic scheduling)
- All created threads hung indefinitely, never executed

**Unix Port Benefits:**
The Unix port should leverage true concurrent multi-threading:
- Multi-core CPUs can run threads in parallel
- C library calls can execute concurrently (coordinated by GIL)
- Better utilization of native system resources

NSI artificially restricts to single-threaded execution (MCU simulation),
which defeats these advantages.

## Files in This Commit

- extmod/zephyr_kernel/posix_nsi_board.c: NSI-based board layer
- extmod/zephyr_kernel/zephyr_kernel.mk: Build config for NSI
- ZEPHYR_POC_FINDINGS.md: Research findings document
- ZEPHYR_THREADING_RESEARCH.md: Investigation notes

## Decision

Continue with minimal pthread adapter (posix_minimal_board.c) that provides
true preemptive threading while adapting Zephyr primitives to pthreads.
This preserves Unix port benefits while enabling Zephyr threading API.

Signed-off-by: Andrew Leech <[email protected]>

Author: pi-anl

ZEPHYR_POC_FINDINGS.md

@@ -0,0 +1,225 @@
+# Zephyr Threading POC - Findings and Blockers
+
+## Summary
+
+Initial Proof of Concept attempt to integrate Zephyr kernel for threading in Unix port revealed significant integration challenges. While the concept is sound, the implementation complexity is higher than anticipated.
+
+## Work Completed
+
+### Infrastructure Created
+1. ✅ `extmod/zephyr_kernel/` directory structure
+2. ✅ `extmod/zephyr_kernel/zephyr_config.h` - Fixed CONFIG_ definitions (161 lines)
+3. ✅ `extmod/zephyr_kernel/zephyr_kernel.h` - Integration API
+4. ✅ `extmod/zephyr_kernel/kernel/mpthread_zephyr.c` - MicroPython threading API implementation (320 lines)
+5. ✅ `ports/unix/zephyr_arch_unix.c` - Unix architecture layer (150 lines)
+6. ✅ Modified `ports/unix/Makefile` - Added MICROPY_ZEPHYR_THREADING build option
+7. ✅ Modified `ports/unix/mpthreadport.h` - Conditional compilation for Zephyr
+8. ✅ Modified `ports/unix/main.c` - Handle different mp_thread_init signatures
+
+### Code Statistics
+- Total new code: ~700 lines
+- Modified existing code: ~50 lines
+- Identified 18 Zephyr kernel files needed for minimal threading
+
+## Build Issues Encountered
+
+### Issue 1: Architecture Detection
+**Problem**: Zephyr's `toolchain/gcc.h` didn't recognize x86_64 architecture.
+
+**Solution Attempted**: Added CONFIG_X86_64 and CONFIG_64BIT definitions to zephyr_config.h
+
+**Status**: ✅ Partially resolved (still some edge cases)
+
+### Issue 2: Missing Generated Headers
+**Problem**: Zephyr build system generates several header files that don't exist in source:
+- `zephyr/syscall_list.h`
+- `zephyr/syscalls/time_units.h`
+- `zephyr/syscalls/*.h` (various syscall headers)
+
+**Impact**: These are core to Zephyr's header structure. Stubbing them out causes cascading issues.
+
+**Root Cause**: Zephyr's header files are tightly coupled to the build system's code generation.
+
+**Status**: ⚠️ Partial workaround attempted, not fully resolved
+
+### Issue 3: Syscall System Integration
+**Problem**: Zephyr uses a complex syscall system with:
+- Macro-based syscall definitions
+- Generated wrapper code
+- Architecture-specific trampolines
+- Userspace/kernel space separation
+
+**Attempted Solutions**:
+1. Define `__syscall` as empty macro → Conflicts with Zephyr's definitions
+2. Stub out syscall_list.h → Missing dependent generated files
+3. Use `-D` to redefine includes → ISO C99 macro naming issues
+
+**Status**: ❌ Blocked - Syscall system deeply integrated into headers
+
+### Issue 4: Header Include Order Dependencies
+**Problem**: Zephyr headers have complex include dependencies:
+```
+kernel.h → kernel_includes.h → syscall.h → syscall_list.h (generated)
+                             → toolchain.h → arch-specific headers
+                             → sys/atomic.h → time_units.h → syscalls/time_units.h (generated)
+```
+
+**Impact**: Can't include `<zephyr/kernel.h>` without full build system
+
+**Status**: ❌ Fundamental architectural issue
+
+## Technical Analysis
+
+### Why Integration is Difficult
+
+1. **Generated Code Dependency**
+   - Zephyr build generates ~20+ header files
+   - These files contain syscall wrappers, device trees, Kconfig output
+   - Not feasible to manually stub all of them
+
+2. **Tight Coupling to Build System**
+   - CMake introspects source files to generate syscall lists
+   - Kconfig generates configuration headers
+   - Device tree compiler generates hardware definitions
+   - Build system is not just configuration - it's code generation
+
+3. **Userspace/Kernel Separation**
+   - Zephyr designed for userspace/kernel separation
+   - Syscall mechanism is core architecture
+   - Can't easily bypass for "kernel-only" use
+
+4. **Architecture-Specific Code**
+   - Each arch has different header requirements
+   - Context switching depends on arch-specific types
+   - Atomic operations vary by architecture
+
+### What Would Be Needed for Success
+
+To make this work, would need:
+
+1. **Minimal Zephyr Build**
+   - Run CMake to generate required headers
+   - Extract only kernel .c files and generated headers
+   - Package as pre-built component
+
+2. **Alternative: Header-Only Wrapper**
+   - Don't use Zephyr headers at all
+   - Manually reimplement minimal thread/mutex/sem types
+   - Link against pre-compiled Zephyr kernel library
+
+3. **Alternative: Fork and Simplify**
+   - Fork Zephyr kernel sources
+   - Remove syscall system completely
+   - Create standalone kernel library
+   - High maintenance burden
+
+## Recommendations
+
+### Short Term: Alternative Approaches
+
+#### Option A: Pre-Generated Zephyr Build
+1. Use Zephyr build system to compile kernel for each architecture
+2. Package as static library + minimal headers
+3. MicroPython ports link against pre-built library
+4. **Pros**: Gets Zephyr functionality, avoids header issues
+5. **Cons**: Adds binary dependency, complicates build
+
+#### Option B: Minimal Kernel Reimplementation
+1. Study Zephyr's thread.c, sched.c, mutex.c implementations
+2. Extract core algorithms
+3. Reimplement in standalone files without Zephyr dependencies
+4. **Pros**: Clean integration, no Zephyr coupling
+5. **Cons**: Significant development effort, loses Zephyr updates
+
+#### Option C: Use Different RTOS
+1. Consider lighter-weight RTOS (FreeRTOS, ThreadX, RT-Thread)
+2. These have simpler header structures
+3. **Pros**: Might integrate easier
+4. **Cons**: Different feature set, less mature than Zephyr
+
+### Long Term: Unified Threading API
+
+Regardless of backend choice, create abstraction layer:
+
+```c
+// extmod/mpthread_rtos.h - Unified RTOS threading API
+typedef struct mp_rtos_thread mp_rtos_thread_t;
+typedef struct mp_rtos_mutex mp_rtos_mutex_t;
+
+mp_rtos_thread_t* mp_rtos_thread_create(void *(*entry)(void*), void *arg);
+void mp_rtos_mutex_init(mp_rtos_mutex_t *mutex);
+// ... etc
+```
+
+Then implement for:
+- Zephyr (if feasible)
+- FreeRTOS
+- ThreadX
+- Pthreads (Unix/POSIX)
+- Custom (STM32, RP2)
+
+## Lessons Learned
+
+1. **Header-Only Integration is Insufficient**
+   - Modern RTOS frameworks rely on code generation
+   - Can't just "include headers" without build system
+
+2. **Build System Complexity**
+   - Zephyr's build system is integral, not optional
+   - This was underestimated in initial research
+
+3. **Syscall System is Core**
+   - Can't be easily disabled or stubbed
+   - Permeates all Zephyr APIs
+
+4. **POC Validated Concerns**
+   - Research document correctly identified this as high-risk
+   - Build system integration was flagged as key challenge
+   - Findings confirm need for different approach
+
+## Next Steps
+
+### Immediate Actions
+
+1. **Document Findings** ✅ (this document)
+2. **Update Research Document** with lessons learned
+3. **Propose Alternative Strategy** to user
+
+### Options for User
+
+**Option 1**: Accept limited scope
+- Proceed with Unix POC using pthread-backed Zephyr API
+- Don't use actual Zephyr kernel
+- Just standardize API shape
+- ~2 hours additional work
+
+**Option 2**: Use pre-built Zephyr libraries
+- Set up proper Zephyr builds for each port
+- Export as static libraries
+- Higher complexity but gets real Zephyr
+- ~1-2 days additional work
+
+**Option 3**: Pivot to different approach
+- Abandon Zephyr integration
+- Either keep port-specific threading OR
+- Use simpler RTOS (FreeRTOS already used in ESP32)
+- Design unified API over existing implementations
+
+**Option 4**: Minimal Zephyr fork
+- Fork just kernel/ directory from Zephyr
+- Strip out syscall dependencies
+- Maintain as separate library
+- ~2-3 days initial, ongoing maintenance
+
+## Conclusion
+
+The POC successfully demonstrated:
+- ✅ API design is sound
+- ✅ Integration points identified correctly
+- ✅ Code structure is reasonable
+- ⚠️ Build integration is more complex than anticipated
+- ❌ Cannot use Zephyr headers without build system
+
+**Recommendation**: Either commit to full Zephyr build integration (Option 2) or pivot to alternative approach (Options 3 or 4).
+
+The "minimal extraction" approach (original plan) is not viable due to generated header dependencies.