feat(map): add --compact tiered mode

.ontos-internal/strategy/proposals/v3.4/v3.4_spec.md

@@ -0,0 +1,301 @@
+---
+id: v3_4_compact_tiered_spec
+type: strategy
+status: draft
+depends_on: [tiered_context_loading_assessment]
+concepts: [context-map, tiered-loading, token-budget, compact-mode]
+date: 2026-02-28
+---
+
+# v3.4 Spec: `--compact tiered` Mode
+
+## Scope
+
+**Ships:** A single new `--compact tiered` mode for `ontos map`.
+
+**Does NOT ship:**
+- No changes to `--compact basic` or `--compact rich` behavior
+- No log archival (`--archive-logs`)
+- No MCP server or dynamic loading
+- No changes to defaults — `--compact` with no argument still means `basic`
+- No changes to AGENTS.md or generated map defaults
+- No `.ontos.toml` configuration changes
+- No new dependencies
+
+**Size estimate:** ~50-80 lines production code, ~80-100 lines tests.
+
+---
+
+## Changes by File
+
+### 1. `ontos/commands/map.py`
+
+#### Change A: Extend `CompactMode` enum (line 27)
+
+```python
+class CompactMode(Enum):
+    """Compact output mode for context map."""
+    OFF = "off"
+    BASIC = "basic"
+    RICH = "rich"
+    TIERED = "tiered"    # NEW
+```
+
+#### Change B: Branch in `generate_context_map()` (lines 115-117)
+
+The current code does an early return for ALL non-OFF compact modes. Tiered mode needs access to `config` (for `_generate_tier1_summary`), so it must NOT use the same early return. Add the tiered branch **before** the existing compact check:
+
+```python
+# Current:
+if options.compact != CompactMode.OFF:
+    return _generate_compact_output(docs, options.compact), result
+
+# New:
+if options.compact == CompactMode.TIERED:
+    return _generate_tiered_compact_output(docs, config, options), result
+elif options.compact != CompactMode.OFF:
+    return _generate_compact_output(docs, options.compact), result
+```
+
+BASIC/RICH path remains identical.
+
+#### Change C: New function `_generate_tiered_compact_output()` (insert after `_generate_compact_output`, ~line 604)
+
+```python
+def _generate_tiered_compact_output(
+    docs: Dict[str, DocumentData],
+    config: Dict[str, Any],
+    options: GenerateMapOptions,
+) -> str:
+    """Generate tiered compact context map.
+
+    Combines Tier 1 prose summary with type-ranked compact lines:
+    - Kernel + Strategy (rank 0-1): RICH format (with summaries)
+    - Product + Atom (rank 2-3): BASIC format (id:type:status)
+    - Logs (rank 4): count + latest ID only
+    """
+    from ontos.core.ontology import TYPE_DEFINITIONS
+
+    # Partition by type rank
+    top_level: Dict[str, DocumentData] = {}
+    detail_level: Dict[str, DocumentData] = {}
+    other_level: Dict[str, DocumentData] = {}
+    log_docs: list = []
+
+    for doc_id, doc in sorted(docs.items()):
+        doc_type = _val(doc.type)
+        type_def = TYPE_DEFINITIONS.get(doc_type)
+        rank = type_def.rank if type_def else None
+
+        if rank in (0, 1):
+            top_level[doc_id] = doc
+        elif rank in (2, 3):
+            detail_level[doc_id] = doc
+        elif rank == 4:
+            log_docs.append(doc)
+        else:
+            other_level[doc_id] = doc
+
+    # Log summary: count + latest
+    # Sort uses shared _log_date_sort_key() helper (dated entries first, undated by ID)
+    log_lines = [f"logs:{len(log_docs)}"]
+    if log_docs:
+        log_docs_sorted = sorted(log_docs, key=_log_date_sort_key, reverse=True)
+        latest = log_docs_sorted[0]
+        log_lines.append(f"latest:{latest.id}")
+
+    # Assemble sections — reuse existing rendering functions
+    sections = [
+        _generate_tier1_summary(docs, config, options),
+        "",
+        "### Kernel + Strategy",
+        _generate_compact_output(top_level, CompactMode.RICH) if top_level else "(none)",
+        "",
+        "### Product + Atom",
+        _generate_compact_output(detail_level, CompactMode.BASIC) if detail_level else "(none)",
+        "",
+        "### Other",
+        _generate_compact_output(other_level, CompactMode.BASIC) if other_level else "(none)",
+        "",
+        "### Logs",
+        "\n".join(log_lines),
+    ]
+    return "\n".join(sections)
+```
+
+**Design notes:**
+- Reuses `_generate_tier1_summary()` and `_generate_compact_output()` — no new rendering logic.
+- Log sorting reuses the exact `_generate_tier1_summary()` pattern (map.py:219-223): falsy check on `date`, fallback to `doc.id`.
+- "Other" section catches types present in `DocumentType` enum but absent from `TYPE_DEFINITIONS` (currently: `reference`, `concept`, `unknown` — the live repo has 3 such docs).
+- `TYPE_DEFINITIONS` import can be moved to module level if preferred.
+
+### 2. `ontos/cli.py`
+
+#### Change D: Add `"tiered"` to CLI choices (line 165)
+
+```python
+# Current:
+p.add_argument("--compact", nargs="?", const="basic", default="off",
+               choices=["basic", "rich"],
+               help="Compact output: 'basic' (default) or 'rich' (with summaries)")
+
+# New:
+p.add_argument("--compact", nargs="?", const="basic", default="off",
+               choices=["basic", "rich", "tiered"],
+               help="Compact output: 'basic' (default), 'rich' (with summaries), or 'tiered' (prose + ranked compact)")
+```
+
+The deprecated `tree` command alias also gets `"tiered"` added to its `--compact` choices to maintain backward compatibility. The existing `CompactMode(args.compact)` conversion (line 658) handles the new enum value automatically.
+
+### 3. `tests/test_map_compact.py`
+
+#### Change E: Add tiered mode coverage tests
+
+Tests call `_generate_tiered_compact_output()` directly. Unlike existing BASIC/RICH tests (which use a simple namedtuple), tiered tests need richer mocks because `_generate_tier1_summary()` accesses more fields. Use `DocumentData` from `ontos.core.types`:
+
+```python
+from ontos.core.types import DocumentData, DocumentType, DocumentStatus
+from pathlib import Path
+
+def _make_doc(doc_id, doc_type, status="active", summary="", date=""):
+    """Helper to create minimal DocumentData for tiered tests."""
+    fm = {}
+    if summary:
+        fm["summary"] = summary
+    if date:
+        fm["date"] = date
+    return DocumentData(
+        id=doc_id,
+        type=DocumentType(doc_type) if isinstance(doc_type, str) else doc_type,
+        status=DocumentStatus(status) if isinstance(status, str) else status,
+        filepath=Path(f"docs/{doc_id}.md"),
+        frontmatter=fm,
+        content="",
+        depends_on=[],
+        impacts=[],
+        tags=[],
+        aliases=[],
+        describes=[],
+    )
+```
+
+**Tests:**
+
+1. **`test_tiered_output_partitions_by_rank`** — Create kernel, strategy, product, atom, log docs. Assert:
+   - `"### Kernel + Strategy"` section contains kernel/strategy IDs
+   - `"### Product + Atom"` section contains product/atom IDs
+   - `"### Logs"` section contains `logs:1` and `latest:` line
+
+2. **`test_tiered_output_empty_partitions`** — Pass only kernel docs. Assert `"### Product + Atom"` shows `(none)`, `"### Logs"` shows `logs:0`.
+
+3. **`test_tiered_log_ordering`** — Create 3 log docs with different dates. Assert `latest:` references the most recent.
+
+4. **`test_tiered_unknown_type_goes_to_other`** — Create a doc with `DocumentType.REFERENCE` (exists in the enum but has no entry in `TYPE_DEFINITIONS`, so rank is `None`). Assert it appears under `"### Other"`. Note: only valid `DocumentType` enum values can be passed — `DocumentType("mystery")` raises `ValueError`.
+
+5. **`test_compact_mode_enum_tiered`** — Assert `CompactMode("tiered") == CompactMode.TIERED`.
+
+6. **`test_generate_context_map_dispatches_tiered`** — Integration test. Call `generate_context_map(docs, config, GenerateMapOptions(compact=CompactMode.TIERED))` with a minimal doc set. Assert output contains `### Kernel + Strategy` and `### Logs`, and does NOT contain the full Tier 2 document index table. This covers the dispatch branch (Change B) end-to-end — if the branch is missing or misordered, this test fails.
+
+### 4. `examples/context_map_benchmark.py` (optional)
+
+#### Change F: Replace simulation with real call
+
+Once `--compact tiered` ships, `_render_tiered_compact_simulation()` (lines 142-184) can be replaced with a call to the real `_generate_tiered_compact_output()`. Nice-to-have, not required for v3.4.
+
+---
+
+## Expected Output Format
+
+Structure is illustrative. Exact content depends on current repo state (doc count, types, dates). `### Other` will contain docs with types not in `TYPE_DEFINITIONS` (e.g., `reference`, `unknown` — the live repo currently has 3 such docs).
+
+```
+## Tier 1: Essential Context
+
+### Project Summary
+- **Name:** ontos
+- **Doc Count:** <current count>
+- **Last Updated:** <date>
+
+### Recent Activity
+| Log | Status | Summary |
+|-----|--------|---------|
+| <latest-log> | active | ... |
+
+### Key Documents
+- `ontos_manual` (<N> dependents)
+
+### Critical Paths
+- **Docs Root:** docs/
+- **Logs:** docs/logs/
+
+---
+
+### Kernel + Strategy
+ontos_manual:kernel:active:"<summary if present>"
+ontos_agent_instructions:kernel:active
+<other kernel/strategy docs...>
+
+### Product + Atom
+ontology_spec:atom:active
+<other product/atom docs...>
+
+### Other
+migration_v2_to_v3:reference:active
+v301:unknown:scaffold
+v323:unknown:scaffold
+
+### Logs
+logs:<count>
+latest:<most-recent-log-id>
+```
+
+Benchmark reference: `tiered_compact_sim` measured 275 tokens at 49 docs (95% reduction from full map). Actual output will vary as the repo evolves.
+
+---
+
+## Acceptance Criteria
+
+1. `ontos map --compact tiered` produces output with all required sections: Tier 1 prose, `### Kernel + Strategy`, `### Product + Atom`, `### Other`, `### Logs`
+2. Output token count is at least 90% smaller than full map output (verify with benchmark script or `estimate_tokens()`)
+3. `ontos map --compact` (no arg) still defaults to BASIC — unchanged
+4. `ontos map --compact basic` and `--compact rich` — unchanged
+5. All existing tests pass (`pytest tests/ -v --tb=short`)
+6. Tiered coverage tests pass (including integration coverage for the dispatch branch and shared Tier 1 regressions)
+7. Benchmark (`examples/context_map_benchmark.py`) remains runnable in the project dev environment
+
+---
+
+## Anti-scope
+
+- Do NOT change the default compact mode
+- Do NOT modify `_generate_compact_output()`
+- `_generate_tier1_summary()` may be updated **only** for narrowly-scoped changes that keep tiered output internally consistent (e.g., shared log ordering) or prevent crashes in the shared Tier 1 log rendering path (e.g., non-string summary coercion). No behavioral changes beyond these are permitted.
+- Do NOT add `.ontos.toml` configuration
+- Do NOT change AGENTS.md or generated artifact defaults
+- Do NOT add log archival, MCP, or any feature beyond `--compact tiered`
+- Do NOT make `tiered` the default for anything
+
+---
+
+## Implementation Sequence
+
+1. Add `TIERED` to `CompactMode` enum
+2. Add `"tiered"` to CLI choices
+3. Add dispatch branch in `generate_context_map()` (before existing compact check)
+4. Write `_generate_tiered_compact_output()` function
+5. Write 6 tests (5 unit + 1 integration)
+6. Run full test suite + benchmark to verify
+7. Commit as `feat(map): add --compact tiered mode`
+
+---
+
+## Key Files
+
+| File | Role |
+|------|------|
+| `ontos/commands/map.py` | Core: enum, dispatch, new function |
+| `ontos/cli.py` | CLI registration |
+| `ontos/core/ontology.py` | `TYPE_DEFINITIONS` with rank values |
+| `ontos/core/types.py` | `DocumentData` dataclass |
+| `tests/test_map_compact.py` | Test file to extend |
+| `examples/context_map_benchmark.py` | Reference simulation |

docs/logs/2026-03-01_tiered-final-cleanup-and-merge.md

@@ -0,0 +1,59 @@
+---
+id: log_20260301_tiered-final-cleanup-and-merge
+type: log
+status: active
+event_type: fix
+source: cli
+branch: v3.4_Tiered_context
+created: 2026-03-01
+---
+
+# tiered-final-cleanup-and-merge
+
+## Summary
+
+Completed final cleanup for the `--compact tiered` PR before merge. The changes remove the last behavior drift in shared Tier 1 rendering, align full-map log ordering with the tiered renderer, tighten regression coverage, and update the v3.4 proposal text so the documented test scope matches the implementation.
+
+## Root Cause
+
+The branch was functionally merge-safe after the earlier review fixes, but two follow-up issues remained:
+
+- the Tier 1 summary fallback treated every falsy summary as missing instead of only `None`/missing values
+- the full-map timeline still used raw ID ordering while Tier 1 and tiered mode had moved to date-aware log ordering
+
+That left small but avoidable consistency debt in the renderer and tests.
+
+## Fix Applied
+
+The cleanup made three targeted changes:
+
+- narrowed the Tier 1 summary fallback so only missing or explicit `None` values become `No summary`
+- updated the Tier 3 timeline to reuse `_log_date_sort_key()` so dated/undated logs are ordered consistently across the full map and tiered output
+- added regressions for empty-string summary preservation and full-map date-aware timeline ordering, and updated the v3.4 proposal wording to reflect tiered coverage tests instead of the stale “6 tests” phrasing
+
+## Testing
+
+- `/Users/jonathanoh/Dev/Ontos-dev/.venv/bin/pytest -q -s tests/test_map_compact.py` (`20 passed`)
+- `/Users/jonathanoh/Dev/Ontos-dev/.venv/bin/pytest -q -s` (`939 passed, 2 skipped`)
+
+## Goal
+
+Leave PR #79 with no known review debt before merge.
+
+## Key Decisions
+
+- Finish the work in a separate git worktree so unrelated untracked files in the main worktree remain untouched.
+- Keep the fixes minimal and scoped to behavior already discussed in review rather than broadening the PR.
+
+## Alternatives Considered
+
+- Merge as-is and accept the small behavior drift.
+- Defer the cleanup to a follow-up PR.
+
+Both were rejected because the remaining issues were easy to fix safely in the same branch.
+
+## Impacts
+
+- `--compact tiered` and full-map log sections now agree on recent-log ordering.
+- Explicit empty summaries are preserved instead of being rewritten.
+- The PR history now includes a matching Ontos log entry for the final cleanup and merge step.

ontos/cli.py

@@ -162,8 +162,8 @@ def _register_map(subparsers, parent):
     p.add_argument("--obsidian", action="store_true",
                    help="Enable Obsidian-compatible output (wikilinks only)")
     p.add_argument("--compact", nargs="?", const="basic", default="off",
-                   choices=["basic", "rich"],
-                   help="Compact output: 'basic' (default) or 'rich' (with summaries)")
+                   choices=["basic", "rich", "tiered"],
+                   help="Compact output: 'basic' (default), 'rich' (with summaries), or 'tiered' (prose + ranked compact)")
     p.add_argument("--filter", "-F", "-f", metavar="EXPR",
                    help="Filter documents by expression (e.g., 'type:strategy'). Use -F; -f is deprecated.")
     p.add_argument("--no-cache", action="store_true",
@@ -602,7 +602,7 @@ def _register_tree_alias(subparsers, parent):
     p.add_argument("--output", "-o", type=Path, help="Output path")
     p.add_argument("--obsidian", action="store_true", help="Enable Obsidian output")
     p.add_argument("--compact", nargs="?", const="basic", default="off",
-                   choices=["basic", "rich"], help="Compact output")
+                   choices=["basic", "rich", "tiered"], help="Compact output")
     p.add_argument("--filter", "-f", metavar="EXPR", help="Filter documents")
     p.add_argument("--no-cache", action="store_true", help="Bypass cache")
     _add_scope_argument(p)