Skip to content

Commit 027cac0

Browse files
rossarmstrongrossarmstrong
authored
Release v0.3.0: New Features and Fixes (#2)
* 📦chore(release): Version bump: v0.2.0 → v0.3.0 * chore: remove outdated comments from Cargo.toml * Add: Minimum version numbers to project.optional-dependencies in pyproject.toml * chore: rename benchmark file for clarity * feat: add wer_analysis module for detailed WER analytics * Add type annotations to utils.py and update docstrings for clarity * 📄 docs(readme): updated the content information * 📄 docs(readme): updated the content information * 📄 docs(readme): updated the content information * 📄 docs(readme): updated the content information * 📄 docs(readme): updated the content information * 📄 docs(readme): updated the content information * 📄 docs(readme): updated the content information * 📄 docs(readme): updated the content information * 📄 docs(readme): updated the content information * chore: update project status to Production/Stable in pyproject.toml * 📄 docs(changelog): update for v0.3.0 release * 📄 docs(changelog): update for v0.3.0 release * Add: latest uv.lock file for reproducible Python dependency management
1 parent 77ed452 commit 027cac0

15 files changed

+638
-25
lines changed

‎CHANGELOG.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
2323

2424
---
2525

26+
## [0.3.0] - 2025-05-16
27+
28+
### Added
29+
- `wer_analysis` module for detailed WER analytics and word-level error breakdown
30+
- New utilities: `to_pandas()` and `to_polars()` for converting analysis results into Pandas and Polars DataFrames.
31+
32+
### Changed
33+
- Added minimum version requirements for all packages in [project.optional-dependencies] in pyproject.toml. This improves dependency management and reduces risk of incompatibility with older package versions.
34+
- Updated `__init__.py` to expose analysis, to_pandas, and to_polars at the top level for easier access.
35+
- Updated README.md to include detailed user guide and instructions for using the analysis() function.
36+
- Documented optional dependency installation steps for Pandas and Polars.
37+
- Included instructions for converting analysis results to Pandas and Polars DataFrames using the `to_pandas()` and `to_polars()` utilities.
38+
39+
### Fixed
40+
- Added type annotations to all public functions in `utils.py`, resolving Pylance warnings about unknown or missing types.
41+
- Improved docstrings and code comments for better clarity and maintainability.
42+
43+
---
44+
2645
## [0.2.0] - 2025-05-14
2746

2847
### Added

‎README.md‎

Lines changed: 94 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@
4141

4242
🧩 **Robust:** Designed to handle edge cases gracefully, including empty strings and mismatched sequences<br>
4343

44-
📐 **Accurate:** Carefully tested to ensure consistent and reliable results<br>
44+
📐 **Insightful:** Provides rich word-level error breakdowns, including substitutions, insertions, deletions, and weighted error rates<br>
4545

4646
🛡️ **Production-Ready:** Minimal dependencies, memory-efficient, and engineered for stability<br>
4747

@@ -73,7 +73,7 @@ import werx
7373

7474
### Examples:
7575

76-
#### 1. Single sentence comparison
76+
### 1. Single sentence comparison
7777

7878
*Python Code:*
7979
```python
@@ -86,7 +86,9 @@ print(wer)
8686
0.25
8787
```
8888

89-
#### 2. Corpus level Word Error Rate Calculation
89+
<br/>
90+
91+
### 2. Corpus level Word Error Rate Calculation
9092

9193
*Python Code:*
9294
```python
@@ -101,7 +103,9 @@ print(wer)
101103
0.2
102104
```
103105

104-
#### 3. Weighted Word Error Rate Calculation (Custom Weights)
106+
<br/>
107+
108+
### 3. Weighted Word Error Rate Calculation
105109

106110
*Python Code:*
107111
```python
@@ -124,6 +128,92 @@ print(wer)
124128
0.15
125129
```
126130

131+
<br/>
132+
133+
### 4. Complete Word Error Rate Breakdown
134+
135+
The `analysis()` function provides a complete breakdown of word error rates, supporting both standard WER and weighted WER calculations.
136+
137+
It delivers detailed, per-sentence metrics—including insertions, deletions, substitutions, and word-level error tracking, with the flexibility to customize error weights.
138+
139+
Results are easily accessible through standard Python objects or can be conveniently converted into Pandas and Polars DataFrames for further analysis and reporting.
140+
141+
142+
#### 4a. Getting Started
143+
144+
*Python Code:*
145+
```python
146+
ref = ["the quick brown fox"]
147+
hyp = ["the quick brown dog"]
148+
149+
results = werx.analysis(ref, hyp)
150+
151+
print("Inserted:", results[0].inserted_words)
152+
print("Deleted:", results[0].deleted_words)
153+
print("Substituted:", results[0].substituted_words)
154+
155+
```
156+
157+
*Results Output:*
158+
```
159+
Inserted Words : []
160+
Deleted Words : []
161+
Substituted Words: [('fox', 'dog')]
162+
```
163+
164+
<br/>
165+
166+
#### 4b. Converting Analysis Results to a DataFrame
167+
168+
*Note:* To use this module, you must have either `pandas` or `polars` (or both) installed.
169+
170+
*Install Pandas / Polars for DataFrame Conversion*
171+
```python
172+
uv pip install pandas
173+
uv pip install polars
174+
```
175+
176+
*Python Code:*
177+
```python
178+
ref = ["i love cold pizza", "the sugar bear character was popular"]
179+
hyp = ["i love pizza", "the sugar bare character was popular"]
180+
results = werx.analysis(
181+
ref, hyp,
182+
insertion_weight=2,
183+
deletion_weight=2,
184+
substitution_weight=1
185+
)
186+
```
187+
We’ve created a special utility to make working with DataFrames seamless.
188+
Just import the following helper:
189+
190+
```python
191+
import werx
192+
from werx.utils import to_polars, to_pandas
193+
```
194+
195+
You can then easily convert analysis results to get output using **Polars**:
196+
```python
197+
# Convert to Polars DataFrame
198+
df_polars = to_polars(results)
199+
print(df_polars)
200+
```
201+
202+
Alternatively, you can also use **Pandas** depending on your preference:
203+
```python
204+
# Convert to Pandas DataFrame
205+
df_pandas = to_pandas(results)
206+
print(df_pandas)
207+
```
208+
209+
*Results Output:*
210+
211+
| wer | wwer | ld | n_ref | insertions | deletions | substitutions | inserted_words | deleted_words | substituted_words |
212+
|--------|--------|-----|-------|------------|-----------|---------------|----------------|----------------|---------------------|
213+
| 0.25 | 0.50 | 1 | 4 | 0 | 1 | 0 | [] | ['cold'] | [] |
214+
| 0.1667 | 0.1667 | 1 | 6 | 0 | 0 | 1 | [] | [] | [('bear', 'bare')] |
215+
216+
127217
<br/>
128218

129219
## 📄 License

‎benchmarks/memory_comparison_synthetic.py‎ renamed to ‎benchmarks/memory_comparison_synthetic_data.py‎

File renamed without changes.

‎benchmarks/speed_comparison_librispeech.py‎ renamed to ‎benchmarks/speed_comparison_librispeech_data.py‎

File renamed without changes.

‎benchmarks/speed_comparison_synthetic.py‎ renamed to ‎benchmarks/speed_comparison_synthetic_data.py‎

File renamed without changes.

‎benchmarks/speed_comparison_werx_dataframes.py‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
from datasets import load_dataset
2+
import werx
3+
import werpy
4+
import timeit
5+
from werx.utils import to_pandas, to_polars
6+
7+
# Load the consolidated CSV from the Hugging Face Hub
8+
dataset = load_dataset(
9+
"analyticsinmotion/librispeech-eval",
10+
data_files="all_splits.csv",
11+
split="train"
12+
)
13+
14+
# Specify which split and model/version to evaluate
15+
split = "test-clean"
16+
model_name = "whisper-base"
17+
model_version = "v20240930"
18+
19+
# Filter references and hypotheses for the chosen split/model/version
20+
filtered = dataset.filter(
21+
lambda x: x["split"] == split and
22+
x["model_name"] == model_name and
23+
x["model_version"] == model_version
24+
)
25+
26+
filtered = list(filtered)
27+
references = [str(werpy.normalize(row["reference"])) for row in filtered]
28+
hypotheses = [str(werpy.normalize(row["hypothesis"])) for row in filtered]
29+
30+
# --- Run werx.analysis once for Standard and Weighted ---
31+
results_standard = werx.analysis(references, hypotheses)
32+
results_weighted = werx.analysis(references, hypotheses, insertion_weight=2, deletion_weight=2, substitution_weight=1)
33+
34+
# --- DataFrame conversion tools ---
35+
df_tools = {
36+
"Pandas (Standard)": lambda: to_pandas(results_standard),
37+
"Polars (Standard)": lambda: to_polars(results_standard),
38+
"Pandas (Weighted)": lambda: to_pandas(results_weighted),
39+
"Polars (Weighted)": lambda: to_polars(results_weighted),
40+
}
41+
42+
# --- Run + time each DataFrame conversion using timeit ---
43+
df_results = []
44+
n_repeats = 10
45+
46+
for name, func in df_tools.items():
47+
total_time = timeit.timeit(func, number=n_repeats)
48+
avg_time = total_time / n_repeats
49+
# Actually create the DataFrame once for display
50+
df = func()
51+
df_results.append((name, df, avg_time))
52+
53+
# --- Sort by fastest execution time ---
54+
df_results.sort(key=lambda x: x[2])
55+
56+
# --- Print CLI-friendly table ---
57+
print("\nWERX Analysis: DataFrame Conversion Benchmark (Ordered by Speed)\n")
58+
print(f"{'Method':<20} {'Rows':<8} {'Cols':<6} {'Time (s)':<12}")
59+
print("-" * 50)
60+
for name, df, t in df_results:
61+
n_rows = len(df)
62+
n_cols = len(df.columns)
63+
print(f"{name:<20} {n_rows:<8} {n_cols:<6} {t:.6f}")
64+
65+
# --- Optionally, show a preview of each DataFrame ---
66+
for name, df, _ in df_results:
67+
print(f"\n{name} DataFrame preview:")
68+
print(df.head())

‎benchmarks/wer_analysis_results.py‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
import werx
2+
from werx.utils import to_polars, to_pandas
3+
4+
# ****************************
5+
# Test 1 - Sentence-Level WER Analysis with Word-Level Errors
6+
# ****************************
7+
8+
ref = [
9+
'it is consumed domestically and exported to other countries',
10+
'rufino street in makati right inside the makati central business district',
11+
'its estuary is considered to have abnormally low rates of dissolved oxygen',
12+
'he later cited his first wife anita as the inspiration for the song',
13+
'no one else could claim that'
14+
]
15+
16+
hyp = [
17+
'it is consumed domestically and exported to other countries',
18+
'rofino street in mccauti right inside the macasi central business district',
19+
'its estiary is considered to have a normally low rates of dissolved oxygen',
20+
'he later sighted his first wife anita as the inspiration for the song',
21+
'no one else could claim that'
22+
]
23+
24+
results = werx.analysis(ref, hyp)
25+
print("===== Test 1: Full Analysis Results =====")
26+
print(results)
27+
28+
# Inspect detailed word-level analysis for the second sentence (index 1)
29+
first_result = results[1]
30+
31+
print("\n===== Test 1: Detailed Word-Level Errors for Sentence 2 =====")
32+
print("Inserted Words :", first_result.inserted_words)
33+
print("Deleted Words :", first_result.deleted_words)
34+
print("Substituted Words:", first_result.substituted_words)
35+
36+
37+
# ****************************
38+
# Test 2 - Simple WER Calculation Example
39+
# ****************************
40+
41+
ref2 = ["this is a test"]
42+
hyp2 = ["this was a test"]
43+
results = werx.analysis(ref2, hyp2)
44+
45+
print("\n===== Test 2: Simple WER Calculation =====")
46+
print(f"WER: {results[0].wer}")
47+
48+
# ****************************
49+
# Test 3 - Weighted WER Calculation Example
50+
# ****************************
51+
52+
ref3 = ["i love cold pizza", "the sugar bear character was popular"]
53+
hyp3 = ["i love pizza", "the sugar bare character was popular"]
54+
results = werx.analysis(ref3, hyp3, insertion_weight=2, deletion_weight=2, substitution_weight=1)
55+
56+
print("\n===== Test 3: Weighted WER Calculation =====")
57+
print(f"Weighted WER (wwer): {results[0].wwer}")
58+
59+
# ****************************
60+
# Test 4 - Results with Polars Example
61+
# ****************************
62+
63+
ref4 = ["i love cold pizza", "the sugar bear character was popular"]
64+
hyp4 = ["i love pizza", "the sugar bare character was popular"]
65+
66+
results = werx.analysis(ref4, hyp4)
67+
68+
# Convert results to Polars DataFrame
69+
df_polars = to_polars(results)
70+
71+
print("\n===== Test 4: Polars DataFrame Output =====")
72+
print(df_polars)
73+
74+
# ****************************
75+
# Test 5 - Weghted WER with Polars Example
76+
# ****************************
77+
78+
results = werx.analysis(ref, hyp, insertion_weight=2, deletion_weight=2, substitution_weight=1)
79+
80+
# Convert results to Polars DataFrame
81+
df_polars = to_polars(results)
82+
83+
print("\n===== Test 5: Polars DataFrame Output =====")
84+
print(df_polars)
85+
86+
# ****************************
87+
# Test 6 - Results with Pandas Example
88+
# ****************************
89+
90+
ref6 = ["i love cold pizza", "the sugar bear character was popular"]
91+
hyp6 = ["i love pizza", "the sugar bare character was popular"]
92+
93+
results = werx.analysis(ref6, hyp6)
94+
95+
# Convert results to Polars DataFrame
96+
df_pandas = to_pandas(results)
97+
98+
print("\n===== Test 4: Pandas DataFrame Output =====")
99+
print(df_pandas)
100+
101+
# ****************************
102+
# Test 7 - Weghted WER with Pandas Example
103+
# ****************************
104+
105+
results = werx.analysis(ref, hyp, insertion_weight=2, deletion_weight=2, substitution_weight=1)
106+
107+
# Convert results to Polars DataFrame
108+
df_pandas = to_pandas(results)
109+
110+
print("\n===== Test 5: Pandas DataFrame Output =====")
111+
print(df_pandas)

‎pyproject.toml‎

Lines changed: 13 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[project]
22
name = "werx"
3-
version = "0.2.0"
3+
version = "0.3.0"
44
description = "A high-performance Python package for calculating Word Error Rate (WER), powered by Rust."
55
readme = "README.md"
66
authors = [
@@ -9,7 +9,7 @@ authors = [
99
requires-python = ">=3.10"
1010
license = {file = 'LICENSE'}
1111
classifiers = [
12-
"Development Status :: 4 - Beta",
12+
"Development Status :: 5 - Production/Stable",
1313
"License :: OSI Approved :: Apache Software License",
1414
"Programming Language :: Python :: 3",
1515
"Programming Language :: Python :: 3 :: Only",
@@ -60,14 +60,19 @@ dev = [
6060
"ruff >=0.11.8",
6161
]
6262

63+
dataframes = [
64+
"polars >=1.29.0",
65+
"pandas >=2.2.3"
66+
]
67+
6368
benchmarks = [
64-
"werpy",
65-
"jiwer",
66-
"pywer",
69+
"werpy >=3.1.0",
70+
"jiwer >=3.1.0",
71+
"pywer >=0.1.1",
6772
"torchmetrics",
68-
"evaluate",
69-
"memory-profiler",
70-
"datasets",
73+
"evaluate >=0.4.3",
74+
"memory-profiler >=0.61.0",
75+
"datasets >=3.6.0",
7176
]
7277

7378
[project.urls]

‎src/werx/__init__.py‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
1-
__version__ = "0.2.0"
1+
__version__ = "0.3.0"
22
from .wer import wer
33
from .weighted_wer import weighted_wer, wwer
4+
from .wer_analysis import analysis
5+
from .utils import to_polars, to_pandas
46

5-
__all__ = ["wer", "weighted_wer", "wwer"]
7+
__all__ = ["wer", "weighted_wer", "wwer", "analysis", "to_polars", "to_pandas"]

0 commit comments

Comments
 (0)