Project_Velocity/.Agent Context/Sprint 1/Biomimetic Agentic Orchestration Layer/Colony Database Schema and Root API Spec.md

Colony Database Schema and Root API Spec

Date: 2026-04-14
Status: Draft implementation artifact
Purpose: Define the PostgreSQL schema, root repository behavior, and FastAPI contract required to support the colony orchestration layer inside the current Project Velocity backend.

1. Purpose

This document specifies the database and root API layer for the colony system.

The colony runtime itself should not become the canonical persistence owner. The Project Velocity Python root must remain the system of record for mission persistence, policy decisions, and domain writeback approvals.

2. Design Principles

2.1 Root Owns Persistence

The TypeScript colony service may keep ephemeral runtime state in memory, but canonical mission and artifact records must be persisted through the Python root.

2.2 JSONB Plus Indexed Columns

Every artifact should be stored in canonical jsonb form, but high-value lookup fields must also be denormalized into indexed scalar columns.

2.3 Mission-Centric Linking

All records must be queryable by mission_id first. This is the primary unit of orchestration, debugging, audit, and approval.

2.4 No Hidden Writebacks

All proposed mutations to CRM or Oracle state must be stored as explicit proposals before approval or rejection.

3. Root Schema Files

Add:

• backend/db/schema_colony.sql

Optionally add:

• backend/db/schema_colony_indexes.sql
• backend/db/schema_colony_seed.sql

For Sprint 1, a single schema_colony.sql is sufficient if it remains readable.

4. Required Tables

Phase 1 requires the following root-owned tables:

• colony_missions
• colony_tasks
• colony_prompt_packages
• colony_research_artifacts
• colony_librarian_passes
• colony_worker_results
• colony_aggregation_packets
• colony_review_packets
• colony_policy_decisions
• colony_writeback_proposals
• colony_event_log
• colony_pheromone_signals

5. Detailed Table Plan

5.1 colony_missions

Purpose:

Stores the root mission envelope and lifecycle state.

Recommended columns:

CREATE TABLE IF NOT EXISTS colony_missions (
  mission_id           UUID PRIMARY KEY,
  mission_type         TEXT NOT NULL,
  origin_surface       TEXT NOT NULL,
  tenant_id            TEXT NOT NULL,
  actor_id             TEXT NOT NULL,
  actor_role           TEXT,
  risk_level           TEXT NOT NULL,
  sensitivity_class    TEXT NOT NULL,
  status               TEXT NOT NULL,
  review_status        TEXT,
  time_budget_ms       INTEGER,
  token_budget         INTEGER,
  user_goal            TEXT NOT NULL,
  normalized_goal      TEXT NOT NULL,
  context_refs         JSONB NOT NULL DEFAULT '{}'::jsonb,
  requested_outputs    JSONB NOT NULL DEFAULT '[]'::jsonb,
  payload              JSONB NOT NULL DEFAULT '{}'::jsonb,
  created_at           TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updated_at           TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  completed_at         TIMESTAMPTZ
);

Indexes:

• (tenant_id, created_at DESC)
• (mission_type, created_at DESC)
• (origin_surface, created_at DESC)
• (status, created_at DESC)

5.2 colony_tasks

Purpose:

Stores the task graph produced by the planner and its runtime state.

Recommended columns:

CREATE TABLE IF NOT EXISTS colony_tasks (
  task_id              TEXT PRIMARY KEY,
  mission_id           UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
  role_type            TEXT NOT NULL,
  status               TEXT NOT NULL,
  objective            TEXT NOT NULL,
  depends_on           JSONB NOT NULL DEFAULT '[]'::jsonb,
  required_capabilities JSONB NOT NULL DEFAULT '[]'::jsonb,
  allowed_tools        JSONB NOT NULL DEFAULT '[]'::jsonb,
  required_data_scopes JSONB NOT NULL DEFAULT '[]'::jsonb,
  success_criteria     JSONB NOT NULL DEFAULT '[]'::jsonb,
  payload              JSONB NOT NULL DEFAULT '{}'::jsonb,
  created_at           TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updated_at           TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  completed_at         TIMESTAMPTZ
);

Indexes:

• (mission_id, created_at ASC)
• (mission_id, status, updated_at DESC)
• (mission_id, role_type, updated_at DESC)

5.3 colony_prompt_packages

Purpose:

Stores prompt packages produced by the prompt master.

Recommended columns:

CREATE TABLE IF NOT EXISTS colony_prompt_packages (
  package_id           TEXT PRIMARY KEY,
  mission_id           UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
  task_id              TEXT NOT NULL REFERENCES colony_tasks(task_id) ON DELETE CASCADE,
  role_type            TEXT NOT NULL,
  template_id          TEXT,
  template_version     TEXT,
  model_route          TEXT,
  tool_scope           JSONB NOT NULL DEFAULT '[]'::jsonb,
  constraints          JSONB NOT NULL DEFAULT '{}'::jsonb,
  payload              JSONB NOT NULL DEFAULT '{}'::jsonb,
  created_at           TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updated_at           TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

5.4 colony_research_artifacts

Purpose:

Stores external research artifacts and normalized citations.

Recommended columns:

CREATE TABLE IF NOT EXISTS colony_research_artifacts (
  artifact_id          TEXT PRIMARY KEY,
  mission_id           UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
  task_id              TEXT REFERENCES colony_tasks(task_id) ON DELETE CASCADE,
  provider             TEXT NOT NULL,
  query                TEXT NOT NULL,
  result_count         INTEGER NOT NULL DEFAULT 0,
  payload              JSONB NOT NULL DEFAULT '{}'::jsonb,
  created_at           TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

5.5 colony_librarian_passes

Purpose:

Stores scoped route cards and internal access passes issued to workers.

Recommended columns:

CREATE TABLE IF NOT EXISTS colony_librarian_passes (
  pass_id              TEXT PRIMARY KEY,
  mission_id           UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
  task_id              TEXT NOT NULL REFERENCES colony_tasks(task_id) ON DELETE CASCADE,
  allowed_resource_families JSONB NOT NULL DEFAULT '[]'::jsonb,
  expires_at           TIMESTAMPTZ,
  payload              JSONB NOT NULL DEFAULT '{}'::jsonb,
  created_at           TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

5.6 colony_worker_results

Purpose:

Stores normalized worker outputs.

Recommended columns:

CREATE TABLE IF NOT EXISTS colony_worker_results (
  result_id            TEXT PRIMARY KEY,
  mission_id           UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
  task_id              TEXT NOT NULL REFERENCES colony_tasks(task_id) ON DELETE CASCADE,
  confidence           DOUBLE PRECISION,
  output_text          TEXT,
  citations            JSONB NOT NULL DEFAULT '[]'::jsonb,
  tool_trace_refs      JSONB NOT NULL DEFAULT '[]'::jsonb,
  payload              JSONB NOT NULL DEFAULT '{}'::jsonb,
  created_at           TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

5.7 colony_aggregation_packets

Purpose:

Stores synthesized pre-review outputs.

Recommended columns:

CREATE TABLE IF NOT EXISTS colony_aggregation_packets (
  packet_id            TEXT PRIMARY KEY,
  mission_id           UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
  summary              TEXT,
  contradiction_count  INTEGER NOT NULL DEFAULT 0,
  coverage_gap_count   INTEGER NOT NULL DEFAULT 0,
  payload              JSONB NOT NULL DEFAULT '{}'::jsonb,
  created_at           TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

5.8 colony_review_packets

Purpose:

Stores review results and final approved outputs.

Recommended columns:

CREATE TABLE IF NOT EXISTS colony_review_packets (
  packet_id            TEXT PRIMARY KEY,
  mission_id           UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
  review_status        TEXT NOT NULL,
  issue_count          INTEGER NOT NULL DEFAULT 0,
  payload              JSONB NOT NULL DEFAULT '{}'::jsonb,
  created_at           TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

5.9 colony_policy_decisions

Purpose:

Stores explicit allow or deny decisions.

Recommended columns:

CREATE TABLE IF NOT EXISTS colony_policy_decisions (
  decision_id          TEXT PRIMARY KEY,
  mission_id           UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
  task_id              TEXT,
  decision_type        TEXT NOT NULL,
  subject              TEXT NOT NULL,
  decision             TEXT NOT NULL,
  reason               TEXT,
  payload              JSONB NOT NULL DEFAULT '{}'::jsonb,
  created_at           TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

5.10 colony_writeback_proposals

Purpose:

Stores proposed domain mutations before approval.

Recommended columns:

CREATE TABLE IF NOT EXISTS colony_writeback_proposals (
  proposal_id          TEXT PRIMARY KEY,
  mission_id           UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
  target_entity_type   TEXT NOT NULL,
  target_entity_id     TEXT NOT NULL,
  action_type          TEXT NOT NULL,
  approval_status      TEXT NOT NULL DEFAULT 'pending',
  payload              JSONB NOT NULL DEFAULT '{}'::jsonb,
  created_at           TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updated_at           TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  resolved_at          TIMESTAMPTZ
);

5.11 colony_event_log

Purpose:

Stores lifecycle events for audit and replay.

Recommended columns:

CREATE TABLE IF NOT EXISTS colony_event_log (
  event_id             BIGSERIAL PRIMARY KEY,
  mission_id           UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
  task_id              TEXT,
  event_type           TEXT NOT NULL,
  event_source         TEXT NOT NULL,
  payload              JSONB NOT NULL DEFAULT '{}'::jsonb,
  created_at           TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

5.12 colony_pheromone_signals

Purpose:

Stores compact reinforcement signals for future routing improvements.

Recommended columns:

CREATE TABLE IF NOT EXISTS colony_pheromone_signals (
  signal_id            BIGSERIAL PRIMARY KEY,
  mission_type         TEXT NOT NULL,
  role_type            TEXT NOT NULL,
  signal_type          TEXT NOT NULL,
  subject              TEXT NOT NULL,
  signal_strength      DOUBLE PRECISION NOT NULL,
  evidence_count       INTEGER NOT NULL DEFAULT 1,
  payload              JSONB NOT NULL DEFAULT '{}'::jsonb,
  created_at           TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

6. Root Python Modules

Create:

• backend/services/colony_repository.py
• backend/services/colony_gateway.py
• backend/services/colony_policy_bridge.py
• backend/api/routes_colony.py

6.1 colony_repository.py

Responsibilities:

• ensure schema readiness
• insert and update mission rows
• insert artifact rows
• retrieve mission detail view
• retrieve mission artifact collection
• resolve writeback proposals

6.2 colony_gateway.py

Responsibilities:

• build HTTP client to colony service
• submit mission envelopes
• fetch mission state
• fetch artifact snapshots
• handle retries and timeout classification

6.3 colony_policy_bridge.py

Responsibilities:

• classify mission risk
• derive tool scope
• derive model routing tier
• validate writeback proposals
• produce policy_decision artifacts

6.4 routes_colony.py

Responsibilities:

• expose root-owned public endpoints
• authenticate callers
• authorize by origin surface and mission type
• persist mission creation
• call colony runtime
• expose approval and rejection endpoints

7. Root API Surface

7.1 POST /api/colony/missions

Purpose:

Create a new mission.

Request body:

{
  "mission_type": "oracle_advisory",
  "origin_surface": "oracle",
  "user_goal": "Summarize high-intent waterfront leads and recommend actions.",
  "context_refs": {
    "lead_ids": ["lead_123"],
    "page_id": "page_abc",
    "branch_id": "main"
  },
  "options": {
    "include_external_research": true
  }
}

Response:

{
  "status": "ok",
  "data": {
    "mission_id": "uuid",
    "mission_type": "oracle_advisory",
    "status": "accepted"
  }
}

7.2 GET /api/colony/missions/{mission_id}

Purpose:

Fetch mission summary state.

Response should include:

• mission status
• review status
• current stage
• trace summary
• final output if completed

7.3 GET /api/colony/missions/{mission_id}/artifacts

Purpose:

Fetch mission artifact collection for operator inspection.

Response should include:

• task graph
• prompt packages
• research artifacts
• worker results
• aggregation packet
• review packet
• policy decisions
• writeback proposals

7.4 POST /api/colony/missions/{mission_id}/approve

Purpose:

Approve pending writeback proposal or gated finalization step.

7.5 POST /api/colony/missions/{mission_id}/reject

Purpose:

Reject pending writeback proposal or gated finalization step.

7.6 GET /api/colony/health

Purpose:

Expose root-to-colony connectivity and root persistence readiness.

Response should include:

• route health
• db readiness
• colony service reachability
• last connectivity check timestamp

8. Mission Lifecycle in Root

Mission lifecycle states should be:

• accepted
• planning
• packaging
• retrieving
• executing
• aggregating
• reviewing
• awaiting_approval
• completed
• failed
• rejected

These states should be stored in colony_missions.status.

9. Writeback Release Flow

The writeback release sequence should be:

1. reviewer produces writeback_proposal
2. proposal stored in colony_writeback_proposals
3. root policy bridge evaluates proposal
4. if mission requires approval, proposal waits in pending
5. approval endpoint resolves proposal
6. only then does root call existing CRM or Oracle write path

No colony worker should bypass this flow.

10. Root Repository Contract

The Python root needs one repository module that hides SQL and exposes a stable service contract to routes_colony.py, routes_oracle.py, and routes_crm.py.

Recommended file:

• backend/services/colony_repository.py

Required methods:

• create_mission(mission_envelope: dict) -> dict
• update_mission_status(mission_id: str, status: str, review_status: str | None = None) -> dict
• store_task_graph(mission_id: str, tasks: list[dict]) -> int
• store_prompt_packages(mission_id: str, packages: list[dict]) -> int
• store_research_artifacts(mission_id: str, artifacts: list[dict]) -> int
• store_librarian_passes(mission_id: str, passes: list[dict]) -> int
• store_worker_results(mission_id: str, results: list[dict]) -> int
• store_aggregation_packet(mission_id: str, packet: dict) -> dict
• store_review_packet(mission_id: str, packet: dict) -> dict
• store_policy_decision(mission_id: str, decision: dict) -> dict
• store_writeback_proposal(mission_id: str, proposal: dict) -> dict
• append_event(mission_id: str, event_type: str, payload: dict) -> dict
• append_pheromone_signal(mission_id: str, signal: dict) -> dict
• get_mission_summary(mission_id: str) -> dict | None
• get_mission_artifacts(mission_id: str) -> dict
• list_missions(filters: dict) -> list[dict]
• resolve_writeback_proposal(mission_id: str, proposal_id: str, resolution: str, actor_id: str) -> dict

Repository rules:

• repository methods must be idempotent where retries are expected
• artifact inserts should use unique IDs controlled by the caller, not generated ad hoc in SQL
• repository methods should raise typed exceptions for not_found, conflict, and validation_error
• root services should never handcraft SQL in route modules

11. Root API Contract

The root FastAPI layer should remain the only externally consumed colony API in Sprint 1.

Recommended files:

• backend/api/routes_colony.py
• backend/services/colony_gateway.py

11.1 POST /api/colony/missions

Purpose:

Create a mission, forward it to the TypeScript colony service, and persist the accepted mission envelope immediately.

Request body shape:

{
  "mission_type": "oracle_advisory",
  "origin_surface": "oracle",
  "tenant_id": "tenant-desineuron",
  "actor_id": "user-123",
  "actor_role": "advisor",
  "user_goal": "Prepare a project-aware response for this lead.",
  "context_refs": {
    "lead_id": "lead-001",
    "page_id": "oracle-main"
  },
  "requested_outputs": [
    "final_answer",
    "writeback_proposal"
  ],
  "payload": {
    "prompt": "What is the best next action for this lead?"
  }
}

Response shape:

{
  "mission_id": "uuid",
  "status": "accepted",
  "review_status": "not_required",
  "current_stage": "planning",
  "accepted_at": "timestamp",
  "links": {
    "self": "/api/colony/missions/uuid",
    "artifacts": "/api/colony/missions/uuid/artifacts"
  }
}

11.2 GET /api/colony/missions/{mission_id}

Purpose:

Fetch mission summary state.

Response should include:

• mission status
• review status
• current stage
• trace summary
• final output if completed
• pending approval summary if applicable

11.3 GET /api/colony/missions/{mission_id}/artifacts

Purpose:

Fetch mission artifact collection for operator inspection.

Response should include:

• task graph
• prompt packages
• research artifacts
• worker results
• aggregation packet
• review packet
• policy decisions
• writeback proposals

This endpoint should support:

• ?include_payloads=true|false
• ?artifact_type=worker_results
• ?task_id=...

11.4 POST /api/colony/missions/{mission_id}/approve

Purpose:

Approve pending writeback proposal or gated finalization step.

Request shape:

{
  "proposal_id": "proposal-001",
  "resolution_note": "approved after operator review"
}

11.5 POST /api/colony/missions/{mission_id}/reject

Purpose:

Reject pending writeback proposal or gated finalization step.

Request shape:

{
  "proposal_id": "proposal-001",
  "resolution_note": "insufficient evidence for automated writeback"
}

11.6 GET /api/colony/health

Purpose:

Expose root-to-colony connectivity and root persistence readiness.

Response should include:

• route health
• db readiness
• colony service reachability
• last connectivity check timestamp
• schema version loaded by root

12. Mission Lifecycle in Root

Mission lifecycle states should be:

• accepted
• planning
• packaging
• retrieving
• executing
• aggregating
• reviewing
• awaiting_approval
• completed
• failed
• rejected

These states should be stored in colony_missions.status.

Review states should be:

• not_required
• pending
• approved
• rejected

Writeback proposal states should be:

• pending
• approved
• rejected
• expired
• superseded

State transition rules:

• accepted may move only to planning or failed
• planning may move only to packaging, failed, or rejected
• packaging may move only to retrieving, executing, failed, or rejected
• retrieving may move only to executing, failed, or rejected
• executing may move only to aggregating, failed, or rejected
• aggregating may move only to reviewing, failed, or rejected
• reviewing may move only to completed, awaiting_approval, failed, or rejected
• awaiting_approval may move only to completed, rejected, or failed
• terminal states are completed, failed, and rejected

13. Writeback Release Flow

The writeback release sequence should be:

1. reviewer produces writeback_proposal
2. proposal stored in colony_writeback_proposals
3. root policy bridge evaluates proposal
4. if mission requires approval, proposal waits in pending
5. approval endpoint resolves proposal
6. only then does root call existing CRM or Oracle write path

No colony worker should bypass this flow.

Additional rules:

• every executed writeback must store resolved_by, resolved_at, and execution_result
• root must store the target route or service name that performed the approved mutation
• a rejected proposal must remain queryable for audit and future training

14. Root Query Shapes

14.1 Mission Summary Query

Need one summary query that joins:

• colony_missions
• latest colony_review_packets
• latest colony_writeback_proposals

This should return one operator-facing mission summary object.

14.2 Mission Artifact Query

Need one artifact query bundle that returns all mission-linked rows by table.

This should be grouped in a stable response shape so the frontend and operator tools do not need to know the physical table order.

14.3 Mission Event Replay Query

Need one query returning ordered colony_event_log rows by created_at ASC.

14.4 Mission Inbox Query

Need one query to list:

• missions currently awaiting approval
• missions failed in the last 24 hours
• missions completed but unresolved for operator follow-up

This powers internal operations and demo readiness.

15. Concurrency, Idempotency, and Retry Rules

Root API and repository behavior must be explicit here because the colony runtime will retry on network and model failures.

Required rules:

• POST /api/colony/missions should accept a caller-supplied request_id header for idempotent creation
• duplicate mission creation with the same request_id should return the existing mission record
• artifact writes should use deterministic IDs where practical
• approval endpoints must reject double approval of the same proposal
• mission status updates should use optimistic concurrency with updated_at or revision number checks
• failed root-to-colony dispatch attempts should still preserve the accepted mission record with a clear failure event

16. Security and Access Control

Root API enforcement must happen before requests reach the TypeScript service.

Required controls:

• mission creation requires authenticated actor context
• Oracle-origin missions can only reference Oracle-allowed data scopes
• CRM-origin missions can only reference CRM-allowed data scopes unless explicitly elevated
• artifact inspection endpoint requires privileged operator or internal service role
• approval and rejection endpoints require writeback authority, not just read authority
• root must redact sensitive fields before returning artifacts to standard UI consumers

17. Migration and Rollout Plan

Database rollout order:

1. apply schema_colony.sql
2. verify tables and indexes
3. deploy colony_repository.py
4. deploy colony_gateway.py
5. deploy routes_colony.py
6. mount routes in backend/main.py
7. enable mission create and health only
8. enable mission status
9. enable artifacts
10. enable approval routes

Rollout rule:

Do not expose approval endpoints in production before repository writes and artifact replay are proven stable.

18. Acceptance Criteria

This layer is complete when:

• schema file exists and applies cleanly
• root endpoints exist and validate payloads
• root persists mission and artifacts
• root can fetch mission and artifact state
• writeback proposals are stored and require root-mediated approval
• mission status transitions are enforced centrally
• duplicate mission create requests are handled safely
• no existing root routes regress

19. Ticket Breakdown

Minimum ticket set:

1. create schema_colony.sql
2. implement colony_repository.py
3. implement colony_gateway.py
4. implement routes_colony.py
5. mount colony router in backend/main.py
6. add idempotency and approval tests
7. add operator artifact inspection tests

20. Bottom Line

The database and root API layer are what turn the colony from a framework experiment into a product subsystem. Without this layer, the orchestration runtime remains difficult to govern and difficult to sell.