MAP Holon Data Loader Design Specification (v1.4)¶

📘 Summary¶

The Holon Data Loader converts holon data presented in JSON files into Holons and HolonRelationships that are staged and committed to a single MAP Space using existing MAP APIs.

Because MAP type definitions are themselves holons, the loader supports importing descriptor holons just like any other data. In canonical JSON imports, the type field is shorthand for a DescribedBy relationship, so runtime holons and schema descriptors share the same loader surface.

Input files are syntactically validated against a JSON Schema to ensure they represent well-formed holons, properties, and relationships.

Before persistence, public Commit invokes the shared, Holochain-independent semantic validator over every live candidate in the staged Nursery. Semantic findings reject the complete Commit attempt without writing nodes or relationships. Holochain validation callbacks remain responsible for persistence-level integrity enforcement.

🔄 What’s Changed in v1.4¶

Descriptor attachment attempts defaults; the final default-population and enum-materialization pass remains authoritative after assembly.
Saved write sources are promoted to staged replacements with net-change lifecycle accounting: replayed edges are no-ops, and new edges retain definitional or graph-only effects.
Commit can target unchanged replacements accepted as NoAction using their saved identities; see relationship anchoring.

🔄 What’s Changed in v1.3¶

This revision replaces inverse normalization with declared-only authoring and common Commit validation. The loader assembles raw input while descriptor contracts may still be incomplete.

Imports author declared relationships; the loader does not reverse inverse endpoints or rewrite them into declared occurrences.
Pass 2 attaches DescribedBy, then Extends, then remaining relationships using ungoverned assembly writes. No loader-specific inverse-orientation classifier authorizes those writes.
Commit rejects nonempty relationship collections absent from the completed source's effective declared contract before candidate persistence. Both inverse and unknown names produce RuleViolation { code: "UndeclaredRelationship" } and LoadCommitStatus = Rejected.
Rejected staged sources remain available with projected actionable findings; independent operational loader errors remain load errors and may prevent Commit from being invoked.
Commit materializes required inverses from accepted declared occurrences.

🔄 What’s Changed in v1.2¶

Unified $ref Model
Replaced mixed and ambiguous reference semantics with a clean, key-based model
Removed distinction between staged vs saved references in syntax
Eliminated temp_key
All references now use stable keys or IDs
Simplifies authoring and loader implementation
Introduced Staged-First Resolution
Key-based references now resolve:
1. Against holons in the current import
2. Then against persisted holons
Enables order-independent and circular references
Simplified $ref Syntax
Default form is now just key
# prefix retained for backward compatibility but made semantically inert
Opaque Key Handling
Keys are treated as opaque strings by the loader
The loader does not parse type information out of key text
Any type-derived key structure is the responsibility of key rules, not loader syntax
Moved Authoring Details to Authoring Guide
JSON structure, $ref usage examples, and formatting rules relocated
Design spec now focuses on architecture and semantics
Explicit $ref Resolution Semantics
Defined deterministic resolution order
Established staged precedence over persisted holons

🧠 Design Principles¶

Principle	Description
Holonic Uniformity	Everything — including types — is a holon
Two-Pass Resolution	Prevents ordering constraints and supports circular references
Identity-Based Referencing	Loader references resolve by logical identity using opaque keys
Descriptor Integrity	Descriptor holons must satisfy their `DescribedBy` meta-type and inherited structural anchors
Import Scope	One import targets one HolonSpace
Staged-First Resolution	References prefer holons in the current import
Minimal Syntax	Concise reference model with limited, consistent prefixes

🔗 `$ref` Model (Design-Level Specification)¶

All holon-to-holon references are expressed using a $ref string. For the loader, $ref is interpreted as an opaque key reference, not as a lifecycle marker.

🧠 Conceptual Model¶

A $ref identifies a holon by logical identity using its key.

The loader resolves references using a staged-first strategy, making imports deterministic and order-independent.

✅ Supported `$ref` Forms¶

1. Local Reference by Key¶

Allowed variants:

future-primal
#future-primal

Design Semantics:

Identifies a holon by logical key
The key is treated as an opaque string by the loader
# prefix is retained for backward compatibility but is semantically inert

🔍 Resolution Semantics¶

Key-Based References¶

For any reference of the form:

key
#key

Target resolution proceeds as follows:

Check staged holons (current import)
If not found, check saved holons (DHT)
If not found, fail resolution

For a relationship write source, use the staged source if present; otherwise promote the saved holon to a staged replacement. Fail if neither exists. A base-key lookup reporting not found means no staged match and permits the saved lookup; other lookup errors propagate.

⚠️ Critical Design Guarantees¶

$ref syntax does not encode lifecycle state (staged vs saved)
# prefix MUST NOT alter resolution behavior
There is no separate namespace for staged holons
Key-based references are deterministic due to staged-first resolution
If a key exists in both staged and saved holons:
Staged holon takes precedence
The loader treats keys as opaque strings and does not parse type information from key text

🧩 Design Implications of the `$ref` Model¶

1. Elimination of `temp_key`¶

No transient identifier namespace is required
Keys serve as stable identifiers across both staged and saved contexts
Simplifies authoring and reduces cognitive overhead

2. Order Independence¶

Because references resolve against staged holons:

Holons may reference others defined later in the file
Circular references are naturally supported
Import ordering is no longer significant

3. Unified Identity Model¶

There is no distinction in syntax between:

referencing a holon being created
referencing an existing holon

This enables:

seamless merging of imports with existing data
consistent mental model for authors and tooling

4. Opaque Keys¶

Loader references remain concise and portable because they use keys
Any type-derived key prefix is part of the key itself, not a loader qualifier
The loader does not infer type from key text

This keeps loader behavior simple and aligned with key-rule ownership of key structure.

🧩 Process Overview¶

🧭 Step-by-Step Flow¶

Define holons (e.g., Airtable)
Export CSV
Convert to JSON
Validate against JSON Schema
Parse into HolonImportSpec
Invoke Holon Data Loader
Stage holons (Pass 1)
Resolve relationships (Pass 2)
Run the final default-population and enum-materialization pass
Invoke Commit
Commit validates the actual explicit state of every live Nursery candidate
Commit persists only when validation succeeds

💾 Staging and Commit Process¶

Pass 1: Stage Holons¶

Create in-memory holon representations
Populate properties only
Register keys for resolution
Queue relationships for Pass 2

Pass 2: Resolve and Stage Relationships¶

Resolve all $ref targets
Resolve descriptor DescribedBy links first through with_descriptor(), which may already populate some defaults against the contract currently available
Resolve Extends links next so descriptor ancestry is queryable
Preserve authored relationship input for common Commit validation; do not infer inverse orientation from targets
Inline embedded keyless holons
Populate remaining relationship links against the now-queryable descriptor graph

When a write source resolves to a saved holon not yet staged, assembly writes to its staged replacement, retaining cloned baseline relationships. Skip targets already present; after Pass 2c, account for net additions against the assembled contract using effective IsDefinitional: true produces a new version; false receives graph-only accounting. Undeclared names preserve lifecycle and are rejected by Commit. Other classification failures are diagnosed and conservatively produce a new version. This accounting does not authorize relationships. Reattaching the same descriptor preserves its edge; replacing it is definitional. See relationship mutation.

Final Default-Population and Enum-Materialization Pass¶

After relationship assembly:

Normalize local enum defaults across staged property descriptors.
Attempt populate_defaults() on each staged holon, then materialize its enum properties, including string tokens copied by early attachment attempts.
Collect operational errors with loader provenance across holons; skip Commit if any occur.

An undescribed holon remains untouched for Commit's missing-descriptor finding. Success does not guarantee completeness or validity; see the shared population contract. Diagnostic error holons author DescribedBy directly without attempting defaults, so reporting does not depend on the possibly malformed descriptor graph being reported.

Commit¶

Invoke the shared Holon Validator over the complete staged Nursery
Persist holons and relationships only when blocking validation failures are absent
Write nodes and SmartLinks through normal Commit processing
Use the shared target-identity rules, including saved source anchors for targets accepted as NoAction in this invocation

🔁 Declared vs Inverse Relationships¶

Declared relationships are the authoritative authored facts
Local inverse relationships are derived and materialized by Commit
Cross-space inverse relationships are deferred to the receiving Space's pull-driven processing
Inverse relationships are not directly authored by the loader

Import Authoring Rule¶

Imports must author every relationship in its declared orientation. The loader does not normalize inverse-oriented input: it neither reverses endpoints nor deduplicates an inverse-authored occurrence against its declared counterpart.

Imported inverse or unknown relationship names are preserved during assembly. Once the graph is complete, Commit requires each nonempty authored collection to be licensed by its source's effective declared contract. Violations return LoadCommitStatus = Rejected, zero committed holons, and RuleViolation { code: "UndeclaredRelationship" } findings on the offending staged sources. Their relationship subjects identify source, name, and target through guest → wire → client projection. The transaction remains available for correction and retry. No node or relationship is written by the rejected invocation.

This semantic rejection is not an operational HolonLoadError. Unresolved import references and other operational loader failures remain distinct and may yield Skipped before Commit.

Materializing the local inverse occurrence remains Commit's responsibility, derived from the declared occurrence the import authored.

📌 Keyed vs Keyless Holons¶

Feature	Keyed Holons	Keyless Holons
Has `key`	Yes	No
Can be referenced via `$ref`	Yes	No
Must be embedded	Optional	Required
Can be relationship target	Yes	No

Keyless holons: - exist only within parent context - must not be referenced independently

🔍 Validation Lifecycle¶

1. Pre-Load (Schema Validation)¶

JSON Schema ensures structural correctness
Cascading validation across Meta, Core, and Domain schemas

2. Loader Resolution Diagnostics¶

$ref targets must resolve
No references to keyless holons
Key uniqueness enforced
Relationship input is preserved for source-contract assessment by Commit
Loader-specific relationship/reference diagnostics reported

3. Commit Validation¶

Commit evaluates the active Capability 1 semantic rules over every live staged candidate
Blocking semantic findings return a rejected response and prevent all node and relationship persistence
Commit also checks populated authored names against effective declared relationship contracts
Broader relationship endpoint, collection, and cardinality validation remain future capabilities

🔮 Future Enhancements¶

Schema validation for $ref expressions
Streaming imports
Reference diagnostics
Preview (dry-run) mode
Cross-space resolution via relationship navigation or Dance requests if later required

📎 Summary¶

The Holon Data Loader provides:

A unified import mechanism for types and instances
Deterministic, order-independent loading via two-pass resolution
A simplified, key-based $ref model
Commit-driven validation across schema and runtime layers
Seamless integration with MAP’s holonic and agent-centric architecture

The $ref model is central to this design, enabling deterministic staged-first, saved-second resolution while keeping loader responsibilities narrow: resolve opaque keys, not interpret key structure.