Skip to content
Crosswalker
🚧 Early alpha — building the foundation. See the roadmap →

v0.1.6 — Bases query layer

Updated

Move the v0.1 output query layer from Dataview to Bases (Obsidian’s official structured-data query system). Update the templates that the plugin emits — concept-note query examples, evidence-link aggregations, coverage-matrix views — so they work natively in Bases without requiring users to install Dataview. Per the project memory commitment (saved 2026-05-04).

Status

Section titled “Status”

🚧 In progress — Phases 1, 1.5, 2, 3, 3.5a, 3.5b, 3.5c, 3.6, 4, 4.5, 4.6, 4.7, 5, 6, 6.1, 6.2, 6.3 ✅ done; v0.1.7 next (recipe-runtime composer + more shapes + exporters)

PhaseWhat shipsBuildManual test
Phase 1 — Recipe query: block schemaBoth A/B discriminator styles + 5 reference recipes + 23 unit tests✅ 2026-05-09✅ Signed off 2026-05-15 (scenarios 1-5 + edge cases)
Phase 1.5 — Test infrastructureDeterministic fixtures + drift CI gate + Phase 2-5 test scaffolds✅ 2026-05-09✅ Covered by CI
Phase 2 — SSSOM TSV import + materialized closureParser + importer + modal UX + eager closure precompute + 25 new tests✅ 2026-05-10⏸ E2E only — manual scenarios in TEST_PHASE2_SSSOM_IMPORT.md not yet walked
Phase 3 — crosswalkerPivot registered Bases viewOne custom view (Component subclass) + pure pivot-grid helper + reference .base file + 8 view options + heatmap mode + 37 new tests✅ 2026-05-10⏸ E2E only — manual scenarios in TEST_PHASE3_PIVOT_VIEW.md not yet walked
Phase 3.5a — Wide-event NDJSON loggerSeverity methods + span helper + trace correlation + size rotation + 18 unit tests✅ 2026-05-11⏸ Inferred working (other phases rely on it). New commands/settings unexercised.
Phase 3.5b — Logger settings UI + commandsCategory filters + verbose toggle + Open/Export/Clear commands✅ 2026-05-11⏸ Commands not yet invoked from palette; category filters unexercised
Phase 3.5c — Call-site sweepMigrated 30+ call sites from legacy shim to info/warn/error(category, op, msg) + threaded trace_id through wizard / SSSOM importer / Tier 2 projector entry points; backward-compat shim removed✅ 2026-05-15⏸ Inferred working via 241/241 unit tests; manual smoke pending
Phase 3.6 — Wizard draft sessionsDraftStore module + auto-save on column-config changes + Step 1 always-visible drafts section + 3 commands + 3 settings + 17 unit tests✅ 2026-05-15✅ Signed off 2026-05-15 (full end-to-end flow verified: save, abandon, resume, complete, auto-delete)
Phase 4 — Recipe-picker UX (codeblock-only — superseded by 4.5)Command palette → modal → inline editor. Original syntax: inline \“basecodeblock at cursor. **Superseded by Phase 4.5** (correct![[file.base]]` syntax). Codeblocks from Phase 4 still work; no auto-migration.⚠️ Superseded 2026-05-15
Phase 4.5 — Frontmatter-driven query notes + .base file generation + ![[embed]]Architecturally-correct flow per Obsidian Bases docs: canonical query in note frontmatter, plugin-generated .base file at _crosswalker/views/q-ID.base, Bases-native ![[VIEW_FILE]] embed at cursor. AJV-validated frontmatter schema. UPDATE-mode auto-detected when frontmatter already present. New Crosswalker: Refresh query views command + onLayoutReady auto-refresh. 49 new tests. Re-homed by Phase 4.6 — canonical state moves from host-note frontmatter to per-query folder; not reverted, relocated.✅ 2026-05-15 (re-homed by 4.6)⏸ Manual smoke test superseded by Phase 4.6
Phase 4.6 — Query-state-location refactor (Layout B+)Per-query folders at _crosswalker/queries/SLUG/ with index.md as canonical state + view.base sibling + reserved derivative subfolders (materialized/, exports/, snapshots/ for Phase 5+/v0.1.7/v0.1.8). Explicit ![[SLUG/view.base]] embed format (no folder-note magic — works in vanilla Obsidian + Mobile). Slug-collision: refuse-and-prompt (picker) + -HEX4 (programmatic). query_id is durable identity; slug is rename-safe. One-shot Crosswalker: Migrate queries to folder layout command. ~20-case edge-case policy table in synthesis log. ~+40 tests. Schema bump 1 → 2.✅ 2026-05-18⏸ Manual smoke + migration test pending
Phase 4.7 — 3-command UX split: Embed existing + Browse queriesNew Crosswalker: Embed existing query into note (lightweight picker over _crosswalker/queries/, inserts ![[SLUG/view.base]] at cursor — zero scan) + Crosswalker: Browse my queries (discovery surface with per-row Open / Embed here / Delete actions). Completes the synthesis-log §3 “create / embed / browse” command split. Pure read via new query-scanner.ts module. +12 tests.✅ 2026-05-18⏸ Manual smoke pending
Phase 5 — Join primitive substrate + materialization + sparse-pivot HARD guardEstablishes the join primitive layer that powers all view shapes (pivot today; table/list/hierarchy/timeline downstream). New src/views/join-primitives.ts with 5 pure-function primitives: innerJoin (default), leftOuterJoin, rightOuterJoin, fullOuterJoin, antiJoin (Layer A primitive #6). Recipe schema’s Join.kind enum extended with "anti". Pivot view: HARD guard at 250K cells; explicit empty-state diagnostic explaining likely causes (missing SSSOM imports, filter scope, confidence threshold) instead of silent empty grid. Materialization writer (materialize.ts) writes SLUG/materialized/result.json per Layout B+. Opt-in Crosswalker: Materialize this query (snapshot) command. Shape-agnostic — reusable for v0.1.7 (table/list) + v0.1.8 (audit snapshots). +28 tests (20 join primitives + 8 materialize).✅ 2026-05-18⏸ Manual smoke pending
Phase 6 — Layer A primitive expansion (bind / set-op / diff)Closes the Ch 29 8-primitive set. Three new pure-function primitives: bind(rows, name, fn) adds derived columns (computed-dimension queries — “evidence > 1 year old”); setOp(left, right, (keyOf, mode, conflictStrategy)) for union/intersection/difference (“controls in BOTH NIST and CIS” — inexpressible from anti-join alone); diff(before, after, (keyOf, ignoreFields, equalsFn)) returns (added, removed, changed) with per-field deltas (load-bearing for v0.1.8 audit-trail). Concept page query-primitives.mdx rewritten to lock the 8-primitive set + 3 worked examples. Engine-neutral, no Obsidian dep. +47 tests (12 bind + 15 set-op + 20 diff).✅ 2026-05-18n/a (pure logic)
Phase 6.1 — Integration tests over realistic fixturesNew tests/helpers/fixture-loader.ts loads 9 realistic fixtures (NIST CSF, 800-53 AC, ISO 27001, SOC 2, CIS v8, MITRE ATT&CK + 3 crosswalks). New tests/integration/primitives-on-realistic-data.test.ts exercises every Layer A primitive against real-shape data: filter / bind / aggregate / anti-join / join modes / set-op / diff / cross-fixture composition. Closes the testing-infrastructure gap surfaced by user audit. +30 integration tests; 31 suites / 509 tests / all pass.✅ 2026-05-19✅ Real data
Phase 6.2 — Streaming Layer A primitives (iterable-first)User direction: optimize join logic from beginning; streaming approach (ChunkyCSV pattern). Refactor primitives to iterable-first BEFORE wiring recipe runtime. Hash-build right side; stream left. New filter-primitive.ts explicit module. New *Stream variants: filterStream, bindStream/bindManyStream, innerJoinStream/leftOuterJoinStream/antiJoinStream/executeJoinStream, intersectionStream/differenceStream/unionStream/setOpStream. Array overloads preserved (backward compat). Streamability matrix documented per primitive. Ch 34 deliverable queued for spill-to-disk + chunked-merge research. +15 tests (array/stream parity + lazy-evaluation probe + hash-build-shape probe + pipelined composition); 32 suites / 524 tests / all pass.✅ 2026-05-19✅ Streaming verified
Phase 6.3 — Benchmark + bundled-fixture import (testable surface)User direction: testable surface + perf logging. New Crosswalker: Run primitives benchmark (perf) command synthesizes data at 100/1k/10k scales, times every primitive (array + stream) via performance.now(), emits NDJSON perf events into debug log + copies formatted summary to clipboard. New Crosswalker: Import bundled test fixture (dev) command — modal lists 2 bundled SSSOM crosswalks (ISO→SOC2 10 mappings; CSF→ATT&CK 13 mappings), one-click import via existing importSssom(). Lets user populate vault with realistic data without manual file copying — pivot views finally render with real data. +6 tests; 33 suites / 530 tests / all pass.✅ 2026-05-19✅ User-testable

Known issues captured but not yet fixed (filed in .workspace/)

Section titled “Known issues captured but not yet fixed (filed in .workspace/)”
IssueSeverityWhere surfaced
curie resolves to unknown:AC-1 instead of FRAMEWORK_ID:AC-1 — frameworkId not threaded to CURIE builderLow (queryable frontmatter still works)Phase 1 manual test 2026-05-15
related_controls renders comma-separated string as ONE wikilink ([[AC-2, AC-3, PM-9]]) instead of N wikilinksMedium (links don’t resolve)Phase 1 manual test 2026-05-15

Dependencies

Section titled “Dependencies”
  • v0.1.5 — Tier 2 sidecar (transitive queries route through the sidecar; simple frontmatter queries route through Bases natively) ✅
  • Ch 27 research challenge — Bases query layer architecture — PRE-REQUISITE. Three deliverables landed 2026-05-07 (A, B, C). Convergent verdict: Hybrid (Bases default + SQL escape hatch) with Pattern B+D from deliverable C as the leading recommendation.
  • Ch 28 research challenge — Bases query layer follow-on stress test — PRE-REQUISITE. Three deliverables landed 2026-05-07 (A, B, C). A and C agree on registerBasesView as primary mechanism, single crosswalkerCoverageMatrix view, _crosswalker/ underscore folder convention, manifest-first OpenTimestamps audit model, junctions-as-audit-truth, mobile/Publish parity as binding constraint. They diverge on materialization-in-v0.1.6 (A: opt-in command; C: defer entirely to v0.1.8). Synthesis log will resolve.

Blocks: v0.1.7 (some exporters reuse the same query layer)

Scope

Section titled “Scope”

This milestone is the v0.1.6 instance of Crosswalker’s general ontology-web query engine. Per the synthesis log, the engine separates concerns into three layers: query primitives, view shapes, and recipes. v0.1.6 ships the Bases mechanism + the pivot view shape + the launch-market Coverage Matrix recipe. v0.1.7 adds codeblock processor; v0.1.8 adds materialized snapshot output.

In:

  • Bases-shaped query templates the plugin emits in concept-note bodies (e.g., “Controls in this family”, “Evidence covering this control”)
  • Single custom Bases view: crosswalkerPivot registered via registerBasesView — pivot shape parameterized by recipe; “Coverage Matrix” is the launch-market recipe instance (renamed from crosswalkerCoverageMatrix for ontology-web alignment)
  • Reference .base file shipped to _crosswalker/views/coverage-matrix.base on first plugin run (idempotent; never overwrites user edits)
  • Recipe loader with JSON Schema validation (.crosswalker/recipes/*.yaml); recipes drive .base emission
  • Recipe schema query: block — additive bump to recipe.schema.json per Ch 31 deliverables. Ship BOTH schema styles behind a dev/advanced setting (locked 2026-05-09): default = A (oneOf+const discriminator); advanced setting flips to B (if/then/else). Both are semantically equivalent; users can A/B-test.
  • 5 reference recipes (one per v0.1 view shape) ship with the plugin — coverage matrix, crosswalk density, orphan controls, hierarchy view, list view
  • crosswalker-bases SKILL.md (per kepano/obsidian-skills) — LLM-assisted recipe authoring guidance shipped in plugin
  • Recipe-picker UX (per Ch 32): command palette → modal recipe picker → inline parameter editor → embedded \“base` block in note
  • Embedded base block default — query results render via embedded base blocks in canonical query notes; NO separate file emission for browse experience (per Ch 32)
  • Opt-in materialization command — Crosswalker: Materialize this recipe (manual, not auto-refresh) for audit/share/Publish use case (per D1 lock)
  • SSSOM TSV import — read .sssom.tsv files (open-standard mapping format used across BioPortal/OxO/OBO Foundry/Biomappings); generate one junction note per row in _crosswalker/mappings/SOURCE-to-TARGET/. Locked 2026-05-09 (per Ch 35 + user direction “might as well do it earlier”)
  • Materialized crosswalk_index table in sqlite cache — pre-computed (subject_id, object_id, predicate_id, confidence) joined table per ontology pair, with (subject_id, object_id) index. Built on import; refreshed incrementally on SSSOM file change. Locked 2026-05-09 (per Ch 35)
  • Materialized concept_closure table — pre-computed (ancestor, descendant, depth) per concept scheme the user opts in. The standard closure-table pattern; far faster than repeated recursive CTE at query time. Locked 2026-05-09 (per Ch 35)
  • Sparse-pivot rendering helper — suppresses empty rows/columns; warns the user at >100K cells (forces pair-selection / triple-selection upfront per BioPortal/OxO/OLIR UI precedent). Locked 2026-05-09 (per Ch 35)
  • _crosswalker/ folder convention with first-run prompt: “Add _crosswalker/views/ to Excluded Files? [Yes/No]”
  • Bases-disabled fallback Notice — detect registerBasesView returning false and show a Notice with one-click enable for the core Bases plugin
  • Updated docs / starter recipes referencing Bases (not Dataview)
  • Compatibility check: Tier 1 frontmatter shapes (concept_note, junction_note, crosswalk_edge) are queryable in Bases without restructure

Out (deferred to v0.1.7):

  • crosswalker-query codeblock processor (deferred indefinitely per D2 — steps on plugin-ecosystem toes)
  • sqlite-wasm user-facing surface (kept internal in v0.1.6 — crosswalkerPivot calls plugin.queryClosure etc. under the hood; no user-facing SQL per D5)
  • Full recipe wizard UI (multi-step form for complex parameter sets — recipes hand-authored YAML or via picker in v0.1.6)
  • Second custom Bases view: crosswalkerHierarchy (per Ch 30)
  • Per-ontology partitioning + tripwire estimation + bounded LRU closure cache + mobile-mode constraints (per Ch 37)

Out (deferred to v0.1.8 — audit-trail milestone):

  • Materialization (Reports/_generated/ or _crosswalker/audit/) — MOVED INTO v0.1.6 (opt-in command + ontology-data materialization both ship in v0.1.6 per D1 lock)
  • OpenTimestamps integration (cryptographic timestamping of audit snapshots)
  • Audit-snapshot manifests
  • Deterministic Markdown writer + golden-file tests
  • Datalog-shaped derivations: sub-DSL inside recipe YAML (for SSSOM provenance, per Ch 36)
  • Tier 3 SPARQL referral mechanism (Oxigraph + Fuseki; per Settled #16)

Out (out of scope, may never ship):

  • Building a Bases plugin / extension — Bases is upstream Obsidian; we just emit queries that Bases can run
  • Datacore as backup option
  • Cross-vault federation (anti-pattern per Ch 27/28)
  • Reintroducing Dataview

Concrete tasks

Section titled “Concrete tasks”

Bases query templates:

  • Audit existing query templates in src/generation/ and recipes/starter/ for Dataview-style notation; list every occurrence
  • Translate each query template to Bases-shape (covered in .claude/skills/obsidian-bases/)
  • Verify Tier 1 frontmatter keys (tags.contains(), file.hasTag()) work as expected with Bases — note Obsidian’s Bases querying nested-tag-prefix via file.hasTag() per Ch 22 §1.3
  • Update generation-engine templates that get inserted into concept-note bodies on import

Custom Bases view (crosswalkerPivot) — Phase 3 ✅ Done 2026-05-10:

  • Implement CrosswalkerPivotView extends Component (Obsidian 1.10+ public API). Read TaskNotes v4 source code as reference — src/views/crosswalker-pivot-view.ts
  • Register via plugin.registerBasesView('crosswalker-pivot', ...) wrapped through src/views/bases-api.ts — gates on Obsidian 1.10.0+; idempotent re-register
  • View renders pivot shape via src/views/pivot-grid.ts pure helper (rows × cols × cells); reads controller.entries (filtered BasesEntry[]); plugin.queryCrosswalk/queryClosure available for view-side enrichment via factory closure
  • View options panel: 8 controls — rowsBy (property), colsBy (property), cellOp (dropdown), cellOf (property), empty (dropdown), heatmap (toggle), rowSort (dropdown), colSort (dropdown)
  • Ship reference .base file to _crosswalker/views/coverage-matrix.base on first plugin run (idempotent; user-edit-safe) — src/views/reference-base-files.ts + templates/coverage-matrix.base
  • Bases-disabled fallback: structured RegistrationResult with reason discriminant; meaningful Notices for no-public-api / bases-disabled / error paths
  • CSS styling — .crosswalker-pivot-grid + heatmap CSS custom property (theme-aware)
  • 37 unit tests (31 pivot-grid + 6 reference-base-files)

Recipe loader + schema bump:

  • Implement BOTH schema styles in spec/recipe.schema.json (per Ch 31): style A (oneOf+const discriminator — default) + style B (if/then/else discriminator — advanced setting)
  • Add plugin setting “Recipe schema style” with A (default) / B (advanced) toggle; both validators ship; users A/B test which feels better
  • Implement recipe loader in src/import/recipe-loader.ts with AJV validation against the active style
  • Recipe → .base emission pipeline (recipes drive .base file generation)
  • Ship 5 reference recipes (one per v0.1 view shape: pivot/table/list/cards/hierarchy)
  • Ship crosswalker-bases SKILL.md per kepano/obsidian-skills (LLM-authoring guide)

Recipe-picker UX (per Ch 32):

  • Command palette: Crosswalker: Insert query into note
  • Modal recipe-picker (lists shipped + user recipes; “Raw YAML escape” option for power users)
  • Inline parameter editor (only the recipe’s exposed parameters; not full schema editing)
  • On confirm: insert embedded \“base` block at cursor position in note

SSSOM TSV import + materialization (locked 2026-05-09 per Ch 35) — Phase 2 ✅ Done 2026-05-10:

  • Implement .sssom.tsv parser per SSSOM 0.15+ spec (subject_id, predicate_id, object_id, mapping_justification, confidence, mapping_set_id, etc.) — src/import/sssom-parser.ts + 19 unit tests
  • Generate one junction note per TSV row in _crosswalker/mappings/SOURCE-to-TARGET/ with frontmatter mirroring SSSOM columns — src/import/sssom-importer.ts builds synthetic crosswalk-edge recipe + delegates to generateFromRecipe; SKOS→STRM predicate normalization preserves original SSSOM predicate as sssom_predicate
  • mappings table populated via existing v0.1.5 P3 projector path (mappings table is already SSSOM-shaped from P3; importer triggers projection after junction notes land)
  • Eager concept_closure materialization on import — precomputeClosureForOntologyPair(db, source, target) in src/tier2/queries.ts; exposed as plugin.precomputeClosure
  • User-facing import UX: file picker → SSSOM validation → preview row count + ontology pair detected → confirm — src/import/sssom-import-modal.ts + Crosswalker: Import SSSOM mapping file command
  • Incremental refresh on SSSOM file change (rebuild only affected pair) — DEFERRED: idempotent re-imports work via overwriteMode: 'replace'; per-file watcher deferred to v0.1.7
  • Sparse-pivot rendering helper (warn at >100K cells; force pair-selection) — DEFERRED to Phase 5 (lives at the rendering layer)

Materialization command (per D1 lock):

  • Command palette: Crosswalker: Materialize this recipe (manual; no auto-refresh)
  • Output to _crosswalker/audit/RECIPE_ID/TIMESTAMP.md
  • Frontmatter flags: crosswalker.materialized: true, do_not_edit: true
  • Default .gitignore entry for _crosswalker/audit/ (user opts in to commit)
  • Timestamped new files; never overwrite (.latest.md symlink-style alias pointer)

_crosswalker/ folder convention:

  • First-run flow: prompt “Add _crosswalker/views/ to Excluded Files? [Yes/No]”
  • Document the namespace convention in user-facing docs

Documentation updates:

Success criteria

Section titled “Success criteria”
  • Every emitted concept-note body that contains a query renders as a working Bases view in Obsidian
  • No Dataview-specific syntax (dv.pages(), inline:: value, WHERE, FROM) appears in plugin-emitted output
  • User can answer the canonical questions (“What evidence covers AC-2?”, “Which ISO controls map to NIST AC family?”) via Bases without installing extra plugins
  • Coverage matrix view (one row per control, columns showing crosswalks to other frameworks) renders in Bases

Files to touch

Section titled “Files to touch”
  • src/generation/templates/ — Bases-shaped query templates
  • src/generation/generation-engine.ts — body-template rendering
  • recipes/starter/*.json — updated emitted-body templates (if applicable)
  • docs/src/content/docs/concepts/metadata-ecosystem.mdx — Bases focus
  • docs/src/content/docs/features/import-wizard.mdx — Bases examples
  • docs/src/content/docs/getting-started/grc-teams.mdx — example workflows

Open questions

Section titled “Open questions”
  • Bases’s nested-tag-prefix support — file.hasTag('framework/nist/ac') matches framework/nist/ac/ac-2? Confirm at implementation; if not, may need to emit fully-enumerated tag arrays
  • Does Bases handle wikilink-array frontmatter (parent: [[A]]) cleanly, or do we need to use the modern links: property for crosswalks?
Section titled “Related”

Concept pages:

Agent context:

Design decisions (synthesis logs):

Research deliverables:

Spec & schema files:

Skills:

Other milestones: