|
| 1 | +# Copilot Instructions for esptool |
| 2 | + |
| 3 | +This document provides essential information for coding agents working on the esptool repository to minimize exploration time and avoid common pitfalls. |
| 4 | + |
| 5 | +## Repository Overview |
| 6 | + |
| 7 | +**What esptool does:** esptool is a Python-based, open-source, platform-independent serial utility for flashing, provisioning, and interacting with Espressif SoCs (ESP32, ESP8266, and variants). It provides four main command-line tools: |
| 8 | +- `esptool.py` - Main flashing and chip interaction tool |
| 9 | +- `espefuse.py` - eFuse (one-time programmable memory) management |
| 10 | +- `espsecure.py` - Security-related operations (signing, encryption) |
| 11 | +- `esp_rfc2217_server.py` - RFC2217 serial-over-TCP server |
| 12 | + |
| 13 | +**Project type:** Python 3.10+ package with console entry points, using setuptools build system and Click/rich_click for CLI interfaces. Supports ESP32, ESP32-S2, ESP32-S3, ESP32-C2, ESP32-C3, ESP32-C5, ESP32-C6, ESP32-H2, ESP32-P4, ESP8266, and other Espressif chips. |
| 14 | + |
| 15 | +**Repository size:** ~200 files, primarily Python code with some binary stub flasher files and YAML configuration files. |
| 16 | + |
| 17 | +## Build and Development Setup |
| 18 | + |
| 19 | +### Essential Commands (always run in this order) |
| 20 | + |
| 21 | +1. **Install dependencies:** |
| 22 | + ```bash |
| 23 | + python -m pip install --upgrade pip |
| 24 | + pip install 'setuptools>=64' |
| 25 | + pip install --extra-index-url https://dl.espressif.com/pypi -e .[dev] |
| 26 | + ``` |
| 27 | + **Critical:** Always use the extra index URL for Espressif-specific packages. Installation may take 2-3 minutes due to cryptography compilation. |
| 28 | + |
| 29 | +2. **Build the project:** |
| 30 | + ```bash |
| 31 | + python setup.py build |
| 32 | + ``` |
| 33 | + This creates build/ directory with compiled Python packages. |
| 34 | + |
| 35 | +3. **Verify installation works:** |
| 36 | + ```bash |
| 37 | + esptool.py --help |
| 38 | + espefuse.py --help |
| 39 | + espsecure.py --help |
| 40 | + esp_rfc2217_server.py --help |
| 41 | + ``` |
| 42 | + |
| 43 | +### Testing |
| 44 | + |
| 45 | +**Quick host-based tests (no hardware required, ~2-3 minutes):** |
| 46 | +```bash |
| 47 | +pytest -m host_test |
| 48 | +``` |
| 49 | +This runs: test_imagegen.py, test_image_info.py, test_mergebin.py, test_modules.py, test_espsecure.py |
| 50 | + |
| 51 | +**HSM tests (requires SoftHSM2 setup):** |
| 52 | +```bash |
| 53 | +pytest test/test_espsecure_hsm.py |
| 54 | +``` |
| 55 | + |
| 56 | +**Virtual eFuse tests (safe, no real hardware affected):** |
| 57 | +```bash |
| 58 | +pytest test_espefuse.py --chip esp32 |
| 59 | +``` |
| 60 | + |
| 61 | +**Hardware tests (requires real ESP devices, NOT safe for CI):** |
| 62 | +```bash |
| 63 | +pytest test_esptool.py --port /dev/ttyUSB0 --chip esp32 --baud 230400 |
| 64 | +``` |
| 65 | + |
| 66 | +### Code Quality and Pre-commit |
| 67 | + |
| 68 | +**Install pre-commit hooks:** |
| 69 | +```bash |
| 70 | +pip install pre-commit |
| 71 | +pre-commit install -t pre-commit -t commit-msg |
| 72 | +``` |
| 73 | + |
| 74 | +**Run all checks:** |
| 75 | +```bash |
| 76 | +pre-commit run --all-files |
| 77 | +``` |
| 78 | + |
| 79 | +**Individual checks:** |
| 80 | +- `python -m ruff check .` - Linting (replaces flake8) |
| 81 | +- `python -m ruff format .` - Code formatting (replaces black) |
| 82 | +- `python -m mypy esptool/ espefuse/ espsecure/` - Type checking |
| 83 | + |
| 84 | +## Project Architecture and Layout |
| 85 | + |
| 86 | +### Key Directories |
| 87 | +- **Root scripts** (`esptool.py`, `espefuse.py`, etc.) - Thin wrapper scripts for backward compatibility |
| 88 | +- **`esptool/`** - Main flashing tool package |
| 89 | + - `targets/` - Chip-specific implementations (ESP32, ESP8266, etc.) |
| 90 | + - `targets/stub_flasher/1/` and `targets/stub_flasher/2/` - Binary flasher stubs for each chip (JSON format) |
| 91 | + - `cmds.py` - Command implementations |
| 92 | + - `loader.py` - Core chip communication logic |
| 93 | +- **`espefuse/`** - eFuse management tool |
| 94 | + - `efuse/` - Chip-specific eFuse definitions |
| 95 | + - `efuse_defs/` - YAML eFuse definition files |
| 96 | +- **`espsecure/`** - Security operations tool |
| 97 | +- **`test/`** - Comprehensive test suite |
| 98 | + - `images/` - Test firmware images and binaries |
| 99 | + - `elf2image/` - ELF to image conversion test cases |
| 100 | +- **`ci/`** - CI/CD scripts and helpers |
| 101 | +- **`docs/`** - Sphinx documentation |
| 102 | + |
| 103 | +### Configuration Files |
| 104 | +- **`pyproject.toml`** - Main project configuration (dependencies, build system, tool settings) |
| 105 | +- **`.pre-commit-config.yaml`** - Pre-commit hook configuration |
| 106 | +- **`.github/workflows/`** - GitHub Actions CI workflows |
| 107 | +- **`.gitlab-ci.yml`** - GitLab CI configuration |
| 108 | + |
| 109 | +### Dependencies |
| 110 | +**Runtime:** bitstring, cryptography>=43.0.0, pyserial>=3.3, reedsolo, PyYAML, intelhex, rich_click, click<9 |
| 111 | +**Development:** pyelftools, coverage, pre-commit, pytest, pytest-rerunfailures, requests, czespressif |
| 112 | + |
| 113 | +## Validation and CI/CD |
| 114 | + |
| 115 | +### GitHub Actions Workflows |
| 116 | +- **`test_esptool.yml`** - Main test suite (Python 3.10-3.13 matrix, host tests, pre-commit) |
| 117 | +- **`build_esptool.yml`** - Build binaries for multiple platforms |
| 118 | +- **`dangerjs.yml`** - PR review automation |
| 119 | + |
| 120 | +### Pre-commit Checks (must pass for PR acceptance) |
| 121 | +1. **ruff** - Python linting and formatting |
| 122 | +2. **mypy** - Type checking (excludes wrapper scripts) |
| 123 | +3. **sphinx-lint** - Documentation linting |
| 124 | +4. **codespell** - Spell checking |
| 125 | +5. **conventional-precommit-linter** - Commit message format validation |
| 126 | + |
| 127 | +### Common Build Issues and Solutions |
| 128 | +- **Network timeouts during pip install:** Use `--timeout 300` flag or retry. The Espressif PyPI index (https://dl.espressif.com/pypi) may be slow. |
| 129 | +- **Missing cryptography:** Ensure you have build tools installed (`apt-get install build-essential` on Ubuntu) |
| 130 | +- **Import errors:** Always install with `-e .[dev]` for development work |
| 131 | +- **Test failures on first run:** Some tests download flasher stubs; retry if network issues occur |
| 132 | +- **ModuleNotFoundError for rich_click:** Means dependencies aren't installed; run pip install command above |
| 133 | +- **SoftHSM2 errors:** For HSM tests, run `./ci/setup_softhsm2.sh` after installing softhsm2 package |
| 134 | + |
| 135 | +## Code Style and Standards |
| 136 | + |
| 137 | +- **Line length:** 88 characters (Black style) |
| 138 | +- **Linting:** ruff (configured in pyproject.toml) |
| 139 | +- **Formatting:** ruff format (replaces black) |
| 140 | +- **Type hints:** mypy checking enabled (partial coverage) |
| 141 | +- **Commit messages:** Conventional Commits format required |
| 142 | + |
| 143 | +## Hardware Testing Notes |
| 144 | + |
| 145 | +**NEVER run hardware tests in CI or on unknown devices.** Hardware tests can: |
| 146 | +- Erase flash memory permanently |
| 147 | +- Modify eFuses (one-time programmable, irreversible) |
| 148 | +- Brick devices if interrupted |
| 149 | + |
| 150 | +Only use `test_esptool.py` and `test_esptool_sdm.py` on dedicated test hardware with explicit `--port`, `--chip`, and `--baud` parameters. |
| 151 | + |
| 152 | +## Performance Considerations |
| 153 | + |
| 154 | +- **esptool operations:** Can take 10-60 seconds for large flash operations |
| 155 | +- **cryptography compilation:** 2-3 minutes during initial pip install |
| 156 | +- **Host tests:** ~2-3 minutes total runtime |
| 157 | +- **Build process:** ~30 seconds for clean build |
| 158 | + |
| 159 | +## Environment Variables |
| 160 | + |
| 161 | +esptool recognizes these environment variables for default behavior: |
| 162 | +- **`ESPTOOL_CHIP`** - Default chip type (auto, esp32, esp8266, etc.) |
| 163 | +- **`ESPTOOL_PORT`** - Default serial port |
| 164 | +- **`ESPTOOL_BAUD`** - Default baud rate |
| 165 | +- **`ESPTOOL_FF`** - Default flash frequency |
| 166 | +- **`ESPTOOL_FM`** - Default flash mode |
| 167 | +- **`ESPTOOL_FS`** - Default flash size |
| 168 | +- **`ESPTOOL_BEFORE`** - Default reset mode before operation |
| 169 | +- **`ESPTOOL_AFTER`** - Default reset mode after operation |
| 170 | + |
| 171 | +For testing: |
| 172 | +- **`ESPTOOL_TEST_USB_OTG`** - Enable USB OTG testing mode |
| 173 | +- **`ESPTOOL_TEST_FLASH_SIZE`** - Minimum flash size for large flash tests |
| 174 | + |
| 175 | +## Trust These Instructions |
| 176 | + |
| 177 | +These instructions are current as of the repository state and have been validated against the actual build system, CI/CD workflows, and documentation. Only search for additional information if: |
| 178 | +1. Commands fail with errors not covered here |
| 179 | +2. You need chip-specific information not in the targets/ directory |
| 180 | +3. You encounter new hardware or features not documented |
| 181 | + |
| 182 | +For chip-specific behavior, always check `esptool/targets/` directory first before searching elsewhere. |
0 commit comments