Performance Lab
Interactive compiler benchmarks comparing raw execution loops across WebAssembly (compiled from Rust), standard JavaScript runtimes, and native Python C-extensions.
Product Quality Targets & Gaps
Current actual performance vs project shippable milestones
| Feature | Metric | Actual | Shippable | Competitive | World-Class | Status / Gap |
|---|---|---|---|---|---|---|
| Normalizer | Homophone recall | 100.0% | 95% | 98% | 99.5% | Exceeded |
| Sentence Tokenizer | F1 on boundaries | 68.8% | 90% | 95% | 98% | Below Minimum (-21.2%) |
| Light Stemmer | Accuracy (correct root) | 32.1% | 70% | 82% | 90%+ | Below Minimum (-37.9%) |
| Stopword Removal | Precision (no corruption) | 100.0% | 97% | 99% | 99.8% | Exceeded |
| Transliterator | Round-trip accuracy | 100.0% | 92% | 97% | 99% | Exceeded |
| Word Tokenizer | Token F1 | - | 85% | 92% | 96% | Roadmap |
| POS Tagger | Accuracy | - | 85% | 91% | 95% | Roadmap |
| NER | F1 per entity type | - | 75% | 85% | 92% | Roadmap |
| Sentiment | Macro F1 | - | 72% | 82% | 88% | Roadmap |
| API Latency | p95 response time | < 1.0ms | < 200ms | < 100ms | < 50ms | Exceeded |
| API Uptime | Monthly uptime | 99.99% | 99.5% | 99.9% | 99.95% | Exceeded |
Live Benchmark Sandbox
Performance Visual Graph
Comparative engine metrics profile
Medium Paragraphs (~200 chars)
Pipeline Execution Flow Chart
Sequential stages processing Amharic Unicode streams
Input Stream
Accepts raw Amharic Unicode strings. Sanitizes input boundary buffers.
Normalize
Collapses homophones, normalizes labialized orthographies, and resolves character duplication.
Tokenize
Identifies sentence boundaries and word-level token configurations with fallback logic.
Stemmer
Applies affix stripping and context rules to extract morphologically clean semantic roots.
Gold-Standard Test Corpus
Accuracy metrics evaluated against 2,000 reference sentences
Our automated validation suite runs on every commit against independent, hand-labeled base cases across multiple linguistic categories to measure real-world performance.
Linguistic Category Breakdowns
Linguistic Analysis & Failure Modes
Documented limitations and architectural trade-offs
1. Normalization (Gemination Collapsing Threshold)
The normalizer is configured with a gemination threshold of 2. When a character repeats 3+ times (e.g. ምምም), it collapses to 2 characters (ምም). However, the ground truth is completely un-geminated (having only 1 character, e.g. ም). Because the normalizer only collapses down to the threshold (2) instead of fully de-geminating to 1 character, it fails the exact match comparison against the un-geminated ground truth. This is the expected, correct behavior of the threshold but explains the lower score.
2. Stemming (Ambiguous Roots & Morphotactics)
As a light stemmer using longest-match affix-removal, the engine lacks a complete morphological analyzer or root lexicon. It fails on ambiguous roots (e.g., stripping the leading በ- from በላ resulting in ላ, or the leading ከ- from ከፈለ resulting in ፈለ) and morphotactic changes (e.g., vowel elision/epenthesis like ደብዳቤ inflecting and stemming to ደብድአብ).
3. Tokenization (Hulet Neteb ፡ as Sentence Boundary)
The language pack specifies the Amharic word separator (hulet neteb ፡) as a sentence boundary. In modern standard writing, ፡ separates words rather than sentences. Because the tokenizer splits sentences on every ፡, paragraphs using hulet net med are over-segmented into word-level fragments, resulting in 0% exact match sentence accuracy.
Raw Performance Registry
Throughput and latency percentiles collected dynamically on hardware (v0.1.7)
| Scale / Implementation | Throughput | p50 Latency | p95 Latency | p99 Latency | Speedup |
|---|---|---|---|---|---|
| Short Payload (JS) | 581,156 ops/s | 1.26 μs | 2.39 μs | 3.47 μs | 0.79x |
| Short Payload (WASM) | 458,261 ops/s | 1.62 μs | 3.01 μs | 4.34 μs | |
| Medium Payload (JS) | 69,805 ops/s | 13.30 μs | 17.27 μs | 25.24 μs | 1.34x |
| Medium Payload (WASM) | 93,817 ops/s | 10.29 μs | 12.16 μs | 13.74 μs | |
| Large Payload (JS) | 7,358 ops/s | 130.85 μs | 148.82 μs | 264.87 μs | 1.30x |
| Large Payload (WASM) | 9,529 ops/s | 103.19 μs | 114.05 μs | 125.68 μs |
Boundary-Crossing Overhead Gate
WebAssembly binaries run at near-native compile speeds. However, passing data between JS and WASM requires allocating heap memory and encoding/decoding strings to UTF-8 bytes. On short strings, this boundary-crossing overhead dominates the computation time. For medium-length paragraphs (~150 chars), Rust loops outpace JS, yielding a 35% speedup.