Skip to content

feat(ci): parallelize jobs #141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Arqu merged 7 commits into from
Oct 23, 2025
Merged

feat(ci): parallelize jobs #141

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
dc56627
feat(ci): parallelize jobs
Arqu Oct 22, 2025
ceedde0
fix wasm
Arqu Oct 22, 2025
fa8d397
centralize lists
Arqu Oct 23, 2025
80a7ba1
fix browser-chat wasm build
Arqu Oct 23, 2025
0a6f20d
uhh
Arqu Oct 23, 2025
464abf0
fix
Arqu Oct 23, 2025
05338a2
fix
Arqu Oct 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
227 changes: 151 additions & 76 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,89 +13,164 @@ env:
RUSTFLAGS: -Dwarnings
RUSTDOCFLAGS: -Dwarnings
MSRV: "1.85"
RS_EXAMPLES_LIST: "dumbpipe-web,extism/host,extism/iroh-extism-host-functions,extism/plugin,iroh-automerge,iroh-automerge-repo,iroh-gateway,frosty,browser-echo,framed-messages,custom-router"
WASM_EXAMPLES_LIST: "browser-echo"
RS_EXAMPLES_LIST: "browser-chat browser-chat/cli browser-echo custom-router dumbpipe-web framed-messages frosty iroh-automerge iroh-automerge-repo iroh-gateway extism/host extism/plugin extism/iroh-extism-host-functions"
WASM_EXAMPLES_LIST: "browser-chat browser-echo"
IROH_FORCE_STAGING_RELAYS: "1"

jobs:
build_and_test_nix:
timeout-minutes: 30
name: Build and test (Nix)
runs-on: ${{ matrix.runner }}
# Prepare dependencies and cache them for all other jobs
prepare:
name: Prepare dependencies
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- name: Checkout
uses: actions/checkout@v4
with:
submodules: recursive

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
components: clippy,rustfmt

- name: Add wasm target
run: rustup target add wasm32-unknown-unknown

- name: Setup Rust cache
uses: Swatinem/rust-cache@v2
with:
shared-key: "iroh-examples-shared"
cache-all-crates: true

- name: Fetch dependencies
run: |
for example in $RS_EXAMPLES_LIST; do
if [ -f "$example/Cargo.toml" ]; then
echo "Fetching dependencies for $example"
cargo fetch --manifest-path "$example/Cargo.toml" --locked 2>/dev/null || cargo fetch --manifest-path "$example/Cargo.toml"
fi
done

# Check formatting for all examples in a single job (fast)
fmt:
name: Format check
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout
uses: actions/checkout@v4
with:
submodules: recursive

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
components: rustfmt

- name: Check formatting
run: |
for example in $RS_EXAMPLES_LIST; do
if [ -f "$example/Cargo.toml" ]; then
echo "Checking format: $example"
cargo fmt --manifest-path "$example/Cargo.toml" --all -- --check
fi
done

# Build WASM examples
wasm:
name: WASM build
runs-on: ubuntu-latest
timeout-minutes: 20
needs: prepare
steps:
- name: Checkout
uses: actions/checkout@v4
with:
submodules: recursive

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable

- name: Add wasm target
run: rustup target add wasm32-unknown-unknown

- name: Setup Rust cache
uses: Swatinem/rust-cache@v2
with:
shared-key: "iroh-examples-shared"

- name: Build WASM examples
run: |
# browser-echo is a single package
Copy link
Member

@Frando Frando Oct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use the WASM_EXAMPLES_LIST from above here instead of manually invoking the build for each wasm crate?

Copy link
Collaborator Author

@Arqu Arqu Oct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

browser-chat is a special case but I've amended it now

cd browser-echo
RUSTFLAGS='--cfg getrandom_backend="wasm_js"' cargo build --target wasm32-unknown-unknown --lib
cd ..

# browser-chat is a workspace, build only the browser-wasm package
cd browser-chat
RUSTFLAGS='--cfg getrandom_backend="wasm_js"' cargo build --target wasm32-unknown-unknown -p chat-browser
cd ..
env:
RUST_LOG: ${{ runner.debug && 'DEBUG' || 'INFO'}}

# Parallel testing matrix for all examples
test:
name: Test ${{ matrix.example }}
runs-on: ubuntu-latest
timeout-minutes: 20
needs: prepare
strategy:
fail-fast: false
matrix:
name: [ubuntu-latest]
rust: [stable]
include:
- name: ubuntu-latest
os: ubuntu-latest
release-os: linux
release-arch: amd64
runner: [self-hosted, linux, X64]
env:
SCCACHE_GHA_ENABLED: "true"
RUSTC_WRAPPER: "sccache"
example:
Copy link
Member

@Frando Frando Oct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to hold this up, but if you happen to have an idea how to easily again use the RS_EXAMPLES_LIST from above instead of the manual list here, we'd be down to a single place where examples are listed. However if this is not straightforward, let's leave it.

Maybe we can add a note to the readme or to a CONTRIBUTING.md with instructions where you have to add the paths to new examples when contributing an example?

Copy link
Collaborator Author

@Arqu Arqu Oct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sadly we cannot create a matrix dynamically so we have to maintain both or I have to do a much more complex setup which Im still not sure works.

- browser-chat
- browser-chat/cli
- browser-echo
- custom-router
- dumbpipe-web
- framed-messages
- frosty
- iroh-automerge
- iroh-automerge-repo
- iroh-gateway
- extism/host
- extism/plugin
- extism/iroh-extism-host-functions
steps:
- name: Checkout
uses: actions/checkout@v4
with:
submodules: recursive

- name: Install ${{ matrix.rust }}
uses: dtolnay/rust-toolchain@master
with:
toolchain: ${{ matrix.rust }}
components: clippy,rustfmt

- name: Add wasm target
run: rustup target add wasm32-unknown-unknown

- name: Run sccache-cache
uses: mozilla-actions/[email protected]

- name: check
run: |
for i in ${RS_EXAMPLES_LIST//,/ }
do
echo "Checking $i"
cargo check --manifest-path $i/Cargo.toml --all-features
done
env:
RUST_LOG: ${{ runner.debug && 'DEBUG' || 'INFO'}}

- name: wasm
run: |
for i in ${WASM_EXAMPLES_LIST//,/ }
do
echo "Checking wasm $i"
cd $i
# the rust flag should get picked up from the `config.toml` file
# but cargo build seems to not be picking it up
# this dependency exists for both `browser-chat` and `browser-echo`
# so I'm adding the flag to the command
RUSTFLAGS='--cfg getrandom_backend="wasm_js"' cargo build --target wasm32-unknown-unknown
cd ..
done
env:
RUST_LOG: ${{ runner.debug && 'DEBUG' || 'INFO'}}

- name: fmt
run: |
for i in ${RS_EXAMPLES_LIST//,/ }
do
echo "Checking $i"
cargo fmt --all --manifest-path $i/Cargo.toml -- --check
done
env:
- name: Checkout
uses: actions/checkout@v4
with:
submodules: recursive

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
components: clippy

- name: Setup Rust cache
uses: Swatinem/rust-cache@v2
with:
shared-key: "iroh-examples-shared"
workspaces: ${{ matrix.example }}

- name: Check
run: |
echo "Checking ${{ matrix.example }}"
cargo check --manifest-path "${{ matrix.example }}/Cargo.toml" --all-features
env:
RUST_LOG: ${{ runner.debug && 'DEBUG' || 'INFO'}}

- name: clippy
run: |
for i in ${RS_EXAMPLES_LIST//,/ }
do
echo "Checking $i"
cargo clippy --manifest-path $i/Cargo.toml
done
env:
- name: Clippy
run: |
echo "Running clippy on ${{ matrix.example }}"
cargo clippy --manifest-path "${{ matrix.example }}/Cargo.toml" --all-features
env:
RUST_LOG: ${{ runner.debug && 'DEBUG' || 'INFO'}}

- name: Test
run: |
echo "Testing ${{ matrix.example }}"
cargo test --manifest-path "${{ matrix.example }}/Cargo.toml" --all-features
env:
RUST_LOG: ${{ runner.debug && 'DEBUG' || 'INFO'}}
1 change: 1 addition & 0 deletions browser-chat/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[workspace]
resolver = "2"
members = ["shared", "cli", "browser-wasm"]
default-members = ["shared", "browser-wasm"]

[workspace.dependencies]
# we define iroh dependencies here to make upgrading easier.
Expand Down
2 changes: 1 addition & 1 deletion browser-chat/cli/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,6 @@ iroh = { workspace = true, default-features = false }
n0-future = "0.3"
rand = "0.9.2"
chat-shared = { version = "0.1.0", path = "../shared" }
tokio = { version = "1.43.0", features = ["rt", "macros"] }
tracing-subscriber = "0.3.19"
data-encoding = "2.9"
tokio = { version = "1.43.0", features = ["rt-multi-thread", "macros", "io-std"] }
Loading