How ekkOS Measures Learning
If you can't measure the learning, it isn't learning. Here's how ekkOS quantifies improvement over time.
ekkOS tracks outcomes, not just interactions. Every pattern applied, every suggestion accepted or rejected, every solution that succeeded or failed — these signals feed a continuous measurement system that makes learning visible, auditable, and improvable.
What Gets Measured
Pattern Success Rate
When a stored pattern is retrieved and applied, ekkOS tracks the outcome. Did the user accept the suggestion? Did they modify it? Did they undo it? Each outcome adjusts the pattern's confidence score. Patterns that consistently succeed get promoted; patterns that fail get demoted.
Local Resolution Rate
The percentage of queries answered from accumulated local knowledge without calling an external model. This is the primary measure of learning progress. A higher local resolution rate means the system has accumulated enough domain-specific knowledge to handle queries independently.
Response Latency Distribution
Queries resolved from local knowledge complete in under 10 milliseconds. Queries requiring external model calls take seconds. The distribution shift over time — more queries moving from the slow path to the fast path — is a direct measure of accumulated expertise.
Knowledge Coverage by Domain
ekkOS maps accumulated patterns against the problem domains encountered. Coverage shows where your LLLM has deep expertise (high pattern density, high confidence) and where it still relies on the base model (low pattern density, no outcome history).
Failure Tracking and Pattern Decay
Failed patterns are not deleted — they are tracked as anti-patterns with recorded context about why they failed. Over time, unused or consistently failing patterns decay in confidence. This prevents knowledge rot: the system forgets what stopped being useful.
The Learning Curve
Every ekkOS instance has a measurable learning curve. In the early days, nearly all queries go to the base model. As patterns accumulate and confidence scores stabilize, more queries are resolved locally. The curve is not linear — it accelerates as domain coverage compounds.
Example output from a developer instance after 87 days of regular use.
How Confidence Scoring Works
Every pattern in ekkOS carries a confidence score between 0.0 and 1.0. This score is not static — it evolves based on outcomes.
Confidence Increases When
- +Pattern is applied and user accepts the result
- +Pattern is applied across multiple projects successfully
- +Pattern aligns with collective intelligence from the community
- +Pattern is reinforced by perception daemon signals (tests pass, build succeeds)
Confidence Decreases When
- −Pattern is applied and user undoes or rejects the change
- −Pattern is applied and tests fail or build breaks
- −Pattern has not been retrieved or used for an extended period (natural decay)
- −User explicitly marks the pattern as outdated or incorrect
Confidence Thresholds
What This Is Not
Precision about what ekkOS measures also requires precision about what it does not claim.
- This is not fine-tuning.The base model's weights are never modified. Learning happens in the cognitive layer above the model, not inside it.
- This is not RAG alone. RAG retrieves documents and injects them into context. ekkOS does that, but also tracks whether the retrieved knowledge actually worked, and adjusts future retrieval accordingly.
- This is not a generic benchmark claim.ekkOS does not claim to outperform frontier models on standard benchmarks. It claims to improve measurably on the specific tasks and domains a given user works in, validated by that user's own outcome data.
Measurement Methodology
All metrics reported by ekkos statsare derived from local data on the user's machine. No metrics are collected remotely unless the user opts into cloud sync.