Skip to content

Commit 5c89efc

Browse files
torfjeldeyebaigithub-actions[bot]shravanngoswamiimhauru
authored
Improvements to DynamicPPLBenchmarks (#346)
* bigboy update to benchmarks * make models return random variables as NamedTuple as it can be useful for downstream tasks * add benchmarking of evaluation with SimpleVarInfo with NamedTuple * added some information about the execution environment * added judgementtable_single * added benchmarking of SimpleVarInfo, if present * added ComponentArrays benchmarking for SimpleVarInfo * formatting * Apply suggestions from code review Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Update benchmarks/benchmarks.jmd Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Benchmarking CI * Julia script for benchmarking on top of current setup * keep old results for reference * updated benchmarking setup * applied suggested changes * updated benchmarks/README.md * setup benchmarking CI * Update benchmark models (#826) * Update models to benchmark plus small style changes * Make benchmark times relative. Add benchmark documentation. * Choose whether to show linked or unlinked benchmark times * Make table header more concise * Make benchmarks not depend on TuringBenchmarking.jl, and run `]dev ..` (#834) * Make benchmarks not depend on TuringBenchmarking.jl * Make benchmarks.jl dev the local DDPPL version * Add benchmarks compat bounds * Use ForwardDiff with dynamic benchmark model * Benchmarking.yml: now comments raw markdown table enclosed in triple backticks * Benchmarking.yml: now includes the SHA of the DynamicPPL commit in Benchmark Results comment * Benchmark more with Mooncake * Add model dimension to benchmark table * Add info print * Fix type instability in benchmark model * Remove done TODO note * Apply suggestions from code review Co-authored-by: Penelope Yong <[email protected]> * Fix table formatting bug * Simplify benchmark suite code * Use StableRNG --------- Co-authored-by: Hong Ge <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Shravan Goswami <[email protected]> Co-authored-by: Markus Hauru <[email protected]> Co-authored-by: Markus Hauru <[email protected]> Co-authored-by: Penelope Yong <[email protected]> Co-authored-by: Shravan Goswami <[email protected]>
1 parent 0b13164 commit 5c89efc

File tree

8 files changed

+430
-353
lines changed

8 files changed

+430
-353
lines changed

‎.github/workflows/Benchmarking.yml‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
name: Benchmarking
2+
3+
on:
4+
pull_request:
5+
6+
jobs:
7+
benchmarks:
8+
runs-on: ubuntu-latest
9+
10+
steps:
11+
- name: Checkout Repository
12+
uses: actions/checkout@v4
13+
with:
14+
ref: ${{ github.event.pull_request.head.sha }}
15+
16+
- name: Set up Julia
17+
uses: julia-actions/setup-julia@v2
18+
with:
19+
version: '1'
20+
21+
- name: Install Dependencies
22+
run: julia --project=benchmarks/ -e 'using Pkg; Pkg.instantiate()'
23+
24+
- name: Run Benchmarks
25+
id: run_benchmarks
26+
run: |
27+
# Capture version info into a variable, print it, and set it as an env var for later steps
28+
version_info=$(julia -e 'using InteractiveUtils; versioninfo()')
29+
echo "$version_info"
30+
echo "VERSION_INFO<<EOF" >> $GITHUB_ENV
31+
echo "$version_info" >> $GITHUB_ENV
32+
echo "EOF" >> $GITHUB_ENV
33+
34+
# Capture benchmark output into a variable
35+
echo "Running Benchmarks..."
36+
benchmark_output=$(julia --project=benchmarks benchmarks/benchmarks.jl)
37+
38+
# Print benchmark results directly to the workflow log
39+
echo "Benchmark Results:"
40+
echo "$benchmark_output"
41+
42+
# Set the benchmark output as an env var for later steps
43+
echo "BENCHMARK_OUTPUT<<EOF" >> $GITHUB_ENV
44+
echo "$benchmark_output" >> $GITHUB_ENV
45+
echo "EOF" >> $GITHUB_ENV
46+
47+
# Get the current commit SHA of DynamicPPL
48+
DPPL_COMMIT_SHA=$(git rev-parse HEAD)
49+
echo "DPPL_COMMIT_SHA=$DPPL_COMMIT_SHA" >> $GITHUB_ENV
50+
51+
COMMIT_URL="https://github.com/${{ github.repository }}/commit/$DPPL_COMMIT_SHA"
52+
echo "DPPL_COMMIT_URL=$COMMIT_URL" >> $GITHUB_ENV
53+
54+
- name: Find Existing Comment
55+
uses: peter-evans/find-comment@v3
56+
id: find_comment
57+
with:
58+
issue-number: ${{ github.event.pull_request.number }}
59+
comment-author: github-actions[bot]
60+
61+
- name: Post Benchmark Results as PR Comment
62+
uses: peter-evans/create-or-update-comment@v4
63+
with:
64+
issue-number: ${{ github.event.pull_request.number }}
65+
body: |
66+
## Benchmark Report for Commit [`${{ env.DPPL_COMMIT_SHA }}`](${{ env.DPPL_COMMIT_URL }})
67+
### Computer Information
68+
```
69+
${{ env.VERSION_INFO }}
70+
```
71+
### Benchmark Results
72+
```
73+
${{ env.BENCHMARK_OUTPUT }}
74+
```
75+
comment-id: ${{ steps.find_comment.outputs.comment-id }}
76+
edit-mode: replace

‎benchmarks/Project.toml‎

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,23 @@ uuid = "d94a1522-c11e-44a7-981a-42bf5dc1a001"
33
version = "0.1.0"
44

55
[deps]
6+
ADTypes = "47edcb42-4c32-4615-8424-f2b9edc5f35b"
67
BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
7-
DiffUtils = "8294860b-85a6-42f8-8c35-d911f667b5f6"
88
Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
99
DynamicPPL = "366bfd00-2699-11ea-058f-f148b4cae6d8"
10-
LibGit2 = "76f85450-5226-5b5a-8eaa-529ad045b433"
11-
Markdown = "d6f4376e-aef5-505a-96c1-9c027394607a"
12-
Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
13-
Weave = "44d3d7a6-8a23-5bf8-98c5-b353f8df5ec9"
10+
ForwardDiff = "f6369f11-7733-5829-9624-2563aa707210"
11+
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
12+
LogDensityProblems = "6fdf6af0-433a-55f7-b3ed-c6c6e0b8df7c"
13+
Mooncake = "da2b9cff-9c12-43a0-ae48-6db2b0edb7d6"
14+
PrettyTables = "08abe8d2-0d0c-5749-adfa-8a2ac140af0d"
15+
ReverseDiff = "37e2e3b7-166d-5795-8a7a-e32c996b4267"
16+
StableRNGs = "860ef19b-820b-49d6-a774-d7a799459cd3"
17+
18+
[compat]
19+
ADTypes = "1.14.0"
20+
BenchmarkTools = "1.6.0"
21+
Distributions = "0.25.117"
22+
ForwardDiff = "0.10.38"
23+
LogDensityProblems = "2.1.2"
24+
PrettyTables = "2.4.0"
25+
ReverseDiff = "1.15.3"

‎benchmarks/README.md‎

Lines changed: 3 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,5 @@
1-
To run the benchmarks, simply do:
1+
To run the benchmarks, run this from the root directory of the repository:
22

33
```sh
4-
julia --project -e 'using DynamicPPLBenchmarks; weave_benchmarks();'
5-
```
6-
7-
```julia
8-
julia> @doc weave_benchmarks
9-
weave_benchmarks(input="benchmarks.jmd"; kwargs...)
10-
11-
Weave benchmarks present in benchmarks.jmd into a single file.
12-
13-
Keyword arguments
14-
≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡
15-
16-
• benchmarkbody: JMD-file to be rendered for each model.
17-
18-
• include_commit_id=false: specify whether to include commit-id in the default name.
19-
20-
• name: the name of directory in results/ to use as output directory.
21-
22-
• name_old=nothing: if specified, comparisons of current run vs. the run pinted to by name_old will be included in the generated document.
23-
24-
• include_typed_code=false: if true, output of code_typed for the evaluator of the model will be included in the weaved document.
25-
26-
• Rest of the passed kwargs will be passed on to Weave.weave.
27-
```
4+
julia --project=benchmarks benchmarks/benchmarks.jl
5+
```

‎benchmarks/benchmark_body.jmd‎

Lines changed: 0 additions & 49 deletions
This file was deleted.

‎benchmarks/benchmarks.jl‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
using Pkg
2+
# To ensure we benchmark the local version of DynamicPPL, dev the folder above.
3+
Pkg.develop(; path=joinpath(@__DIR__, ".."))
4+
5+
using DynamicPPLBenchmarks: Models, make_suite, model_dimension
6+
using BenchmarkTools: @benchmark, median, run
7+
using PrettyTables: PrettyTables, ft_printf
8+
using StableRNGs: StableRNG
9+
10+
rng = StableRNG(23)
11+
12+
# Create DynamicPPL.Model instances to run benchmarks on.
13+
smorgasbord_instance = Models.smorgasbord(randn(rng, 100), randn(rng, 100))
14+
loop_univariate1k, multivariate1k = begin
15+
data_1k = randn(rng, 1_000)
16+
loop = Models.loop_univariate(length(data_1k)) | (; o=data_1k)
17+
multi = Models.multivariate(length(data_1k)) | (; o=data_1k)
18+
loop, multi
19+
end
20+
loop_univariate10k, multivariate10k = begin
21+
data_10k = randn(rng, 10_000)
22+
loop = Models.loop_univariate(length(data_10k)) | (; o=data_10k)
23+
multi = Models.multivariate(length(data_10k)) | (; o=data_10k)
24+
loop, multi
25+
end
26+
lda_instance = begin
27+
w = [1, 2, 3, 2, 1, 1]
28+
d = [1, 1, 1, 2, 2, 2]
29+
Models.lda(2, d, w)
30+
end
31+
32+
# Specify the combinations to test:
33+
# (Model Name, model instance, VarInfo choice, AD backend, linked)
34+
chosen_combinations = [
35+
(
36+
"Simple assume observe",
37+
Models.simple_assume_observe(randn(rng)),
38+
:typed,
39+
:forwarddiff,
40+
false,
41+
),
42+
("Smorgasbord", smorgasbord_instance, :typed, :forwarddiff, false),
43+
("Smorgasbord", smorgasbord_instance, :simple_namedtuple, :forwarddiff, true),
44+
("Smorgasbord", smorgasbord_instance, :untyped, :forwarddiff, true),
45+
("Smorgasbord", smorgasbord_instance, :simple_dict, :forwarddiff, true),
46+
("Smorgasbord", smorgasbord_instance, :typed, :reversediff, true),
47+
("Smorgasbord", smorgasbord_instance, :typed, :mooncake, true),
48+
("Loop univariate 1k", loop_univariate1k, :typed, :mooncake, true),
49+
("Multivariate 1k", multivariate1k, :typed, :mooncake, true),
50+
("Loop univariate 10k", loop_univariate10k, :typed, :mooncake, true),
51+
("Multivariate 10k", multivariate10k, :typed, :mooncake, true),
52+
("Dynamic", Models.dynamic(), :typed, :mooncake, true),
53+
("Submodel", Models.parent(randn(rng)), :typed, :mooncake, true),
54+
("LDA", lda_instance, :typed, :reversediff, true),
55+
]
56+
57+
# Time running a model-like function that does not use DynamicPPL, as a reference point.
58+
# Eval timings will be relative to this.
59+
reference_time = begin
60+
obs = randn(rng)
61+
median(@benchmark Models.simple_assume_observe_non_model(obs)).time
62+
end
63+
64+
results_table = Tuple{String,Int,String,String,Bool,Float64,Float64}[]
65+
66+
for (model_name, model, varinfo_choice, adbackend, islinked) in chosen_combinations
67+
@info "Running benchmark for $model_name"
68+
suite = make_suite(model, varinfo_choice, adbackend, islinked)
69+
results = run(suite)
70+
eval_time = median(results["evaluation"]).time
71+
relative_eval_time = eval_time / reference_time
72+
ad_eval_time = median(results["gradient"]).time
73+
relative_ad_eval_time = ad_eval_time / eval_time
74+
push!(
75+
results_table,
76+
(
77+
model_name,
78+
model_dimension(model, islinked),
79+
string(adbackend),
80+
string(varinfo_choice),
81+
islinked,
82+
relative_eval_time,
83+
relative_ad_eval_time,
84+
),
85+
)
86+
end
87+
88+
table_matrix = hcat(Iterators.map(collect, zip(results_table...))...)
89+
header = [
90+
"Model",
91+
"Dimension",
92+
"AD Backend",
93+
"VarInfo Type",
94+
"Linked",
95+
"Eval Time / Ref Time",
96+
"AD Time / Eval Time",
97+
]
98+
PrettyTables.pretty_table(
99+
table_matrix;
100+
header=header,
101+
tf=PrettyTables.tf_markdown,
102+
formatters=ft_printf("%.1f", [6, 7]),
103+
)

0 commit comments

Comments
 (0)