Skip to content
Crosswalker
🚧 Early alpha — building the foundation. See the roadmap →

ChunkyCSV and transformation engines

Created Updated

Context

Section titled “Context”

This log entry captures the initial idea. It has since been expanded by transform engine research (14 engines evaluated, custom build decided) and documented as the Transform stage of the pipeline.

The idea

Section titled “The idea”

For manual migrations between framework versions (or any structured data transformation), users need the ability to:

  1. Define transformations declaratively (config-as-code)
  2. OR run them manually with their own code (via CLI)
  3. OR use an engine that handles common patterns automatically (the custom transform engine)

ChunkyCSV

Section titled “ChunkyCSV”

The user created ChunkyCSV — a structured-to-tabular conversion engine. This could be a key component of Crosswalker’s transformation pipeline.

Research needed:

  • What does ChunkyCSV do exactly? How does it handle hierarchy flattening, nested data, array expansion?
  • How does it compare to PapaParse (currently used for CSV parsing)?
  • Could it replace or augment the transform system defined in src/types/config.ts?
  • What about xlsx-based transformation tools?

Connection to roadmap

Section titled “Connection to roadmap”

This fits into multiple phases:

  • Foundation: The as-code data format for representing transformations
  • Formats: The transform system implementation (20+ types defined, 0% implemented)
  • Evolution: Manual migration paths when auto-migration isn’t possible

The broader pattern

Section titled “The broader pattern”

The transformation engine is a key architectural component:

  • Import transforms: Clean, normalize, restructure incoming data (current focus)
  • Migration transforms: Convert between framework versions (Evolution phase)
  • Export transforms: Convert to OSCAL, reports, dashboards (Community phase)

A general-purpose transformation engine that handles all three would be more valuable than three separate systems.

Section titled “Related”