Project_Velocity/.Agent Context/Sprint 1/Biomimetic Agentic Orchestration Layer/Implementation Blueprint_ Repository and Runtime Mapping.md

Implementation Blueprint_ Repository and Runtime Mapping

Date: 2026-04-14
Status: Draft implementation blueprint
Purpose: Convert the biomimetic orchestration design into a repository-level build plan tied directly to the current Project Velocity codebase.

1. Purpose

This document defines the exact implementation skeleton for the colony orchestration layer inside Project Velocity.

2. Current State Mapping

Project Velocity already has enough root infrastructure to support a colony orchestration layer without introducing a second backend center.

2.1 Root FastAPI and Router Truth

The current root backend in backend/main.py already mounts the relevant operational surfaces:

• backend/api/routes_catalyst.py
• backend/api/routes_crm.py
• backend/api/routes_oracle.py
• backend/oracle/router_v1.py
• backend/routers/sentinel.py
• backend/routers/cctv.py
• backend/routers/scenes.py
• backend/routers/videos.py
• backend/routers/vault.py

This means the colony layer must integrate into the current root, not compete with it.

2.2 Oracle Truth

Oracle already contains a meaningful planning spine:

• backend/oracle/prompt_orchestrator.py
• backend/oracle/persona_service.py
• backend/oracle/policy_service.py
• backend/oracle/data_access_gateway.py
• backend/oracle/action_service.py
• backend/oracle/router_v1.py

This is critical. The colony layer must not replace Oracle v1. It must become the next orchestration layer beneath or beside the current prompt orchestration concepts.

2.3 CRM Truth

CRM is now real enough to serve as an internal mission substrate:

• backend/api/routes_crm.py exposes leads, chat logs, kanban, demographics, seed flow, and analytics scatter
• /ws/crm already exists in backend/main.py
• frontend CRM bootstrap already exists under app/src/hooks/useCrmBootstrap.ts

The colony therefore has a real internal state store to reason over. It should use CRM as one of its first mission domains.

2.4 Nemoclaw and MCP Truth

Project Velocity already includes root-owned Python append layers:

• backend/services/nemoclaw_runtime.py
• backend/services/mcp_registry.py
• backend/services/nemoclaw_client.py

These are not yet a full orchestration platform, but they provide three essential things:

• workflow dispatch preview semantics
• external and internal tool abstraction
• model-facing reasoning utilities

2.5 Frontend Truth

The frontend already exposes product surfaces that will eventually call the colony:

• Oracle page shell
• CRM and pipeline surfaces
• Catalyst and marketing tab
• Sentinel live session and perception surfaces

The correct implementation path is therefore back-to-front:

• build colony runtime and contracts first
• expose root API entry points second
• consume them from Oracle and CRM first

3. Target Runtime Shape

The target runtime must use a split-brain architecture with one root authority and one specialized orchestration service.

3.1 Root Authority

The current Python root remains authoritative for:

• authentication
• PostgreSQL access
• domain data models
• Oracle page and canvas persistence
• CRM persistence
• Sentinel ingestion and response
• Catalyst persistence and route ownership

3.2 Colony Runtime

The new TypeScript colony service becomes authoritative for:

• mission normalization
• task graph planning
• prompt package generation
• worker provisioning
• inter-agent execution
• aggregation and review
• orchestration traces

3.3 Guard Plane

Nemoclaw-derived governance remains external to worker prompts and should mediate:

• tool scope
• network egress class
• model routing class
• writeback validation
• audit events

3.4 Why This Split Is Correct

This split is the least disruptive implementation path because:

• Project Velocity already has a mature Python root
• Open Multi Agent is TypeScript-native and should remain where it is strongest
• domain systems should not be reimplemented in the colony runtime
• orchestration should not directly own CRM, Oracle, or Sentinel tables

4. File-by-File Repository Blueprint

The following structure should be created.

4.1 New Top-Level Service

Create:

services/
  colony-orchestrator/

This service is the fork-and-extend boundary for Open Multi Agent concepts.

4.2 TypeScript Service Tree

Create:

services/colony-orchestrator/
  package.json
  tsconfig.json
  README.md
  src/
    index.ts
    config/
      env.ts
      models.ts
    core/
      colony-runtime.ts
      mission-runner.ts
      task-graph.ts
      scheduler.ts
      worker-factory.ts
    contracts/
      mission.ts
      task.ts
      prompt-package.ts
      research-artifact.ts
      librarian-pass.ts
      worker-result.ts
      aggregation-packet.ts
      review-packet.ts
      policy-decision.ts
    planner/
      mission-normalizer.ts
      planner-agent.ts
      decomposition-policy.ts
    prompt-master/
      prompt-master-agent.ts
      template-registry.ts
      exemplar-registry.ts
      prompt-package-builder.ts
    librarian/
      librarian-agent.ts
      source-catalog.ts
      route-card-builder.ts
      cache-index.ts
    researcher/
      researcher-agent.ts
      search-provider.ts
      browser-provider.ts
      citation-normalizer.ts
    workers/
      worker-agent.ts
      worker-context-builder.ts
      result-normalizer.ts
    aggregation/
      aggregator-agent.ts
      evidence-matrix.ts
      contradiction-detector.ts
    review/
      reviewer-agent.ts
      review-policy.ts
      release-gate.ts
    policy/
      tool-policy.ts
      model-routing-policy.ts
      writeback-policy.ts
      risk-classifier.ts
    adapters/
      oracle-adapter.ts
      crm-adapter.ts
      catalyst-adapter.ts
      sentinel-adapter.ts
      python-root-client.ts
    persistence/
      mission-store.ts
      artifact-store.ts
      pheromone-store.ts
    telemetry/
      audit-log.ts
      mission-trace.ts
      metrics.ts
  tests/
    contracts/
    planner/
    prompt-master/
    librarian/
    researcher/
    integration/

4.3 New Python Root Files

Create:

backend/
  api/
    routes_colony.py
  services/
    colony_gateway.py
    colony_repository.py
    colony_policy_bridge.py
  db/
    schema_colony.sql

4.4 Why These Files Exist

routes_colony.py is the root public API surface for colony mission submission and inspection.

colony_gateway.py is the Python integration client that talks to the TypeScript colony service.

colony_repository.py is the root-owned persistence adapter for colony mission records that must live near current PostgreSQL patterns.

colony_policy_bridge.py is the place where root policy, Nemoclaw-derived guard logic, and domain-specific action constraints are translated into colony-consumable decisions.

schema_colony.sql is the root schema source for mission and artifact persistence.

5. Python Root Integration Plan

The Python root must integrate the colony in a way that preserves current ownership.

5.1 New Root Routes

Mount a new router in backend/main.py:

from backend.api.routes_colony import router as colony_router
app.include_router(colony_router, prefix="/api/colony", tags=["Colony"])

5.2 Required Root Endpoints

Phase 1 root endpoints should be:

• POST /api/colony/missions
• GET /api/colony/missions/{mission_id}
• GET /api/colony/missions/{mission_id}/artifacts
• POST /api/colony/missions/{mission_id}/approve
• POST /api/colony/missions/{mission_id}/reject
• GET /api/colony/health

5.3 Root Responsibilities

The Python root should:

• authenticate and authorize caller context
• attach tenant and actor context
• classify requested mission type
• persist mission metadata
• call colony runtime
• validate requested writebacks
• surface mission state to existing product surfaces

5.4 Explicit Non-Responsibilities

The Python root should not:

• perform colony DAG decomposition itself
• own worker prompt package generation
• run direct multi-agent loops inside FastAPI request handlers

Those belong in the colony service.

6. TypeScript Colony Service Plan

The colony service should be implemented as a narrowly scoped internal service, not as a second public product shell.

6.1 Source of Truth for Execution

The service should own:

• mission runtime state
• task graph state
• prompt package generation
• agent execution
• review packets
• orchestration traces

6.2 Upstream Fork Strategy

Do not copy the entire Open Multi Agent repository directly into the root app. Instead:

1. create services/colony-orchestrator/
2. import or adapt upstream concepts into src/core/
3. keep Velocity-specific roles outside the upstream-like core folders

The forked pieces should roughly correspond to:

• OpenMultiAgent -> colony-runtime.ts
• TaskQueue -> task-graph.ts
• Scheduler -> scheduler.ts
• AgentPool -> worker-factory.ts plus internal pool manager
• team/messaging and memory/shared -> mission-local context bus and artifact retrieval

6.3 Colony Role Mapping

Map the biomimetic metaphor to concrete runtime classes:

• queen -> root product surface plus mission ID ownership
• planner ant -> planner/planner-agent.ts
• prompt master ant -> prompt-master/prompt-master-agent.ts
• researcher ant -> researcher/researcher-agent.ts
• librarian ant -> librarian/librarian-agent.ts
• worker ants -> workers/worker-agent.ts
• aggregator ant -> aggregation/aggregator-agent.ts
• reviewer ant -> review/reviewer-agent.ts

6.4 Execution Model

Per mission:

1. mission-normalizer.ts creates mission envelope
2. planner-agent.ts creates task graph
3. prompt-package-builder.ts creates prompt package per task
4. librarian-agent.ts creates route cards and scoped passes
5. researcher-agent.ts creates evidence artifacts where required
6. worker-agent.ts executes task-specific work
7. aggregator-agent.ts synthesizes outputs
8. reviewer-agent.ts performs one-pass review
9. final packet is returned to root

6.5 Mission Classes for Phase 1

Implement only three mission classes first:

• oracle_advisory
• crm_lead_intelligence
• catalyst_strategy_brief

Do not start with generalized free-form autonomous missions. Use typed mission classes only.

7. Mission, Artifact, and Persistence Plan

The current Open Multi Agent model is intentionally short-lived and mostly in-memory. Project Velocity needs persistence from day one.

7.1 Root Database Objects

Create the following tables in backend/db/schema_colony.sql:

• colony_missions
• colony_tasks
• colony_prompt_packages
• colony_research_artifacts
• colony_librarian_passes
• colony_worker_results
• colony_aggregation_packets
• colony_review_packets
• colony_policy_decisions
• colony_pheromone_signals

7.2 Mandatory Columns

Every table should include:

• primary ID
• mission ID foreign key where applicable
• tenant_id
• actor_id
• status
• payload jsonb
• created_at
• updated_at

7.3 Mission Envelope Contract

Mission payload should minimally contain:

• mission_id
• mission_type
• origin_surface
• user_goal
• normalized_goal
• tenant_id
• actor_id
• risk_level
• sensitivity_class
• time_budget_ms
• token_budget
• requested_outputs

7.4 Why Persistence Is Mandatory

Persistence is not optional because the product needs:

• sales-demo reliability
• auditability
• mission replay and debugging
• operator review visibility
• future approvals and resumability

8. Adapter Plan for Oracle, CRM, Catalyst, and Sentinel

8.1 Oracle Adapter

The Oracle adapter should be the first-class Phase 1 adapter.

It should:

• accept a prompt from Oracle UI or Oracle helper routes
• translate it into oracle_advisory mission type
• enrich with tenant, actor, page, branch, and target lead context
• return structured renderable packet plus optional writeback proposal

Do not bypass backend/oracle/router_v1.py. Instead add a colony-backed path under Oracle helper or Oracle v1 integration.

8.2 CRM Adapter

The CRM adapter should:

• accept a lead or kanban stage context
• create crm_lead_intelligence mission type
• pull relevant lead, chat log, and stage data through root APIs or repository methods
• return structured insights, next-step suggestions, and optional message drafts

8.3 Catalyst Adapter

Catalyst should be Phase 1.5 or Phase 2, but the file boundary should exist now.

It should eventually create catalyst_strategy_brief missions using:

• campaign state
• analytics
• audience context
• ad network context

8.4 Sentinel Adapter

Sentinel should initially be an evidence provider, not an autonomous mission origin.

That means Sentinel data can be attached to missions as:

• perception evidence
• emotional interest signals
• room-level events

but Sprint 1 should not make Sentinel the lead actor in colony execution.

9. Governance and Nemoclaw Guard Plan

Governance is where this system either becomes production-ready or stays a demo.

9.1 Policy Layers

There should be four policy layers:

1. mission admission policy
2. tool and data scope policy
3. model routing policy
4. writeback release policy

9.2 Mission Admission Policy

Before the colony starts, classify:

• mission type
• sensitivity
• autonomy tier
• required review level

9.3 Tool and Data Scope Policy

Every worker must receive an explicit allowed tool list and allowed data scope list. No worker should receive generic open access to all MCP tools or all root data.

9.4 Model Routing Policy

At minimum define:

• tier_local_private
• tier_cloud_standard
• tier_high_reasoning

This routing decision must be computed outside the worker prompt and carried into the prompt package.

9.5 Writeback Release Policy

No worker should directly mutate CRM or Oracle state.

All writebacks should go through:

• review-packet
• root policy bridge
• root writeback validator

9.6 Current Code to Reuse

Directly leverage:

• backend/oracle/action_service.py
• backend/oracle/policy_service.py
• backend/services/nemoclaw_runtime.py
• backend/services/mcp_registry.py

These should not be discarded. They should be bridged into the colony runtime.

10. Testing and Operationalization Plan

Testing must be mission-centered, not only framework-centered.

10.1 Required Test Layers

Create tests for:

• schema validation
• planner outputs
• prompt package generation
• librarian routing passes
• research artifact normalization
• worker result normalization
• aggregation packets
• review decisions
• policy rejection
• root adapter integration

10.2 Root Integration Tests

Add Python integration tests for:

• POST /api/colony/missions
• Oracle adapter mission submission
• CRM adapter mission submission
• writeback approval and rejection flow

10.3 Operational Readiness

The service is only considered production-candidate when it has:

• health endpoint
• structured logs
• trace correlation IDs
• mission replay view
• failure reason visibility
• environment-driven provider config

11. Build Order

The implementation order should be fixed.

Step 1

Create services/colony-orchestrator/ with:

• package setup
• base runtime shell
• contract files

Step 2

Create backend/db/schema_colony.sql and repository helpers.

Step 3

Create backend/services/colony_gateway.py and backend/api/routes_colony.py.

Step 4

Implement:

• mission normalizer
• planner
• prompt package builder

Step 5

Implement:

• librarian
• researcher provider interfaces

Step 6

Implement:

• worker execution
• aggregator
• reviewer

Step 7

Implement:

• Oracle adapter
• CRM adapter

Step 8

Implement governance bridge and writeback approval flow.

Step 9

Run end-to-end acceptance on Oracle and CRM mission classes.

12. Non-Negotiable Decisions

These decisions should be frozen unless strong evidence proves them wrong:

• no second Python backend center
• no replacement of Oracle v1
• no uncontrolled context sharing
• no prompt-only governance
• no direct worker writebacks to root state
• no generic mission types before typed domain missions succeed

13. Bottom Line

The colony layer should be added as a new orchestration service plus a thin root integration layer.

The build should start where Project Velocity is already strong:

• Oracle
• CRM
• root PostgreSQL
• existing policy and Nemoclaw helper surfaces

That is the shortest path from architecture to a sellable working product.