sagnik/Project_Velocity
2
1
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
acfc602157dadcb44d73995c15311970cae396e3
Project_Velocity/.Agent Context/Sprint 1/Biomimetic Agentic Orchestration Layer/Colony Database Schema and Root API Spec.md

868 lines
23 KiB
Markdown
Raw Blame History

# Colony Database Schema and Root API Spec
**Date:** 2026-04-14
**Status:** Draft implementation artifact
**Purpose:** Define the PostgreSQL schema, root repository behavior, and FastAPI contract required to support the colony orchestration layer inside the current Project Velocity backend.
## 1. Purpose
This document specifies the database and root API layer for the colony system.
The colony runtime itself should not become the canonical persistence owner. The Project Velocity Python root must remain the system of record for mission persistence, policy decisions, and domain writeback approvals.
## 2. Design Principles
### 2.1 Root Owns Persistence
The TypeScript colony service may keep ephemeral runtime state in memory, but canonical mission and artifact records must be persisted through the Python root.
### 2.2 JSONB Plus Indexed Columns
Every artifact should be stored in canonical `jsonb` form, but high-value lookup fields must also be denormalized into indexed scalar columns.
### 2.3 Mission-Centric Linking
All records must be queryable by `mission_id` first. This is the primary unit of orchestration, debugging, audit, and approval.
### 2.4 No Hidden Writebacks
All proposed mutations to CRM or Oracle state must be stored as explicit proposals before approval or rejection.
## 3. Root Schema Files
Add:
- `backend/db/schema_colony.sql`
Optionally add:
- `backend/db/schema_colony_indexes.sql`
- `backend/db/schema_colony_seed.sql`
For Sprint 1, a single `schema_colony.sql` is sufficient if it remains readable.
## 4. Required Tables
Phase 1 requires the following root-owned tables:
- `colony_missions`
- `colony_tasks`
- `colony_prompt_packages`
- `colony_research_artifacts`
- `colony_librarian_passes`
- `colony_worker_results`
- `colony_aggregation_packets`
- `colony_review_packets`
- `colony_policy_decisions`
- `colony_writeback_proposals`
- `colony_event_log`
- `colony_pheromone_signals`
## 5. Detailed Table Plan
### 5.1 `colony_missions`
Purpose:
Stores the root mission envelope and lifecycle state.
Recommended columns:
```sql
CREATE TABLE IF NOT EXISTS colony_missions (
mission_id UUID PRIMARY KEY,
mission_type TEXT NOT NULL,
origin_surface TEXT NOT NULL,
tenant_id TEXT NOT NULL,
actor_id TEXT NOT NULL,
actor_role TEXT,
risk_level TEXT NOT NULL,
sensitivity_class TEXT NOT NULL,
status TEXT NOT NULL,
review_status TEXT,
time_budget_ms INTEGER,
token_budget INTEGER,
user_goal TEXT NOT NULL,
normalized_goal TEXT NOT NULL,
context_refs JSONB NOT NULL DEFAULT '{}'::jsonb,
requested_outputs JSONB NOT NULL DEFAULT '[]'::jsonb,
payload JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
completed_at TIMESTAMPTZ
);
```
Indexes:
- `(tenant_id, created_at DESC)`
- `(mission_type, created_at DESC)`
- `(origin_surface, created_at DESC)`
- `(status, created_at DESC)`
### 5.2 `colony_tasks`
Purpose:
Stores the task graph produced by the planner and its runtime state.
Recommended columns:
```sql
CREATE TABLE IF NOT EXISTS colony_tasks (
task_id TEXT PRIMARY KEY,
mission_id UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
role_type TEXT NOT NULL,
status TEXT NOT NULL,
objective TEXT NOT NULL,
depends_on JSONB NOT NULL DEFAULT '[]'::jsonb,
required_capabilities JSONB NOT NULL DEFAULT '[]'::jsonb,
allowed_tools JSONB NOT NULL DEFAULT '[]'::jsonb,
required_data_scopes JSONB NOT NULL DEFAULT '[]'::jsonb,
success_criteria JSONB NOT NULL DEFAULT '[]'::jsonb,
payload JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
completed_at TIMESTAMPTZ
);
```
Indexes:
- `(mission_id, created_at ASC)`
- `(mission_id, status, updated_at DESC)`
- `(mission_id, role_type, updated_at DESC)`
### 5.3 `colony_prompt_packages`
Purpose:
Stores prompt packages produced by the prompt master.
Recommended columns:
```sql
CREATE TABLE IF NOT EXISTS colony_prompt_packages (
package_id TEXT PRIMARY KEY,
mission_id UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
task_id TEXT NOT NULL REFERENCES colony_tasks(task_id) ON DELETE CASCADE,
role_type TEXT NOT NULL,
template_id TEXT,
template_version TEXT,
model_route TEXT,
tool_scope JSONB NOT NULL DEFAULT '[]'::jsonb,
constraints JSONB NOT NULL DEFAULT '{}'::jsonb,
payload JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
### 5.4 `colony_research_artifacts`
Purpose:
Stores external research artifacts and normalized citations.
Recommended columns:
```sql
CREATE TABLE IF NOT EXISTS colony_research_artifacts (
artifact_id TEXT PRIMARY KEY,
mission_id UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
task_id TEXT REFERENCES colony_tasks(task_id) ON DELETE CASCADE,
provider TEXT NOT NULL,
query TEXT NOT NULL,
result_count INTEGER NOT NULL DEFAULT 0,
payload JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
### 5.5 `colony_librarian_passes`
Purpose:
Stores scoped route cards and internal access passes issued to workers.
Recommended columns:
```sql
CREATE TABLE IF NOT EXISTS colony_librarian_passes (
pass_id TEXT PRIMARY KEY,
mission_id UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
task_id TEXT NOT NULL REFERENCES colony_tasks(task_id) ON DELETE CASCADE,
allowed_resource_families JSONB NOT NULL DEFAULT '[]'::jsonb,
expires_at TIMESTAMPTZ,
payload JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
### 5.6 `colony_worker_results`
Purpose:
Stores normalized worker outputs.
Recommended columns:
```sql
CREATE TABLE IF NOT EXISTS colony_worker_results (
result_id TEXT PRIMARY KEY,
mission_id UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
task_id TEXT NOT NULL REFERENCES colony_tasks(task_id) ON DELETE CASCADE,
confidence DOUBLE PRECISION,
output_text TEXT,
citations JSONB NOT NULL DEFAULT '[]'::jsonb,
tool_trace_refs JSONB NOT NULL DEFAULT '[]'::jsonb,
payload JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
### 5.7 `colony_aggregation_packets`
Purpose:
Stores synthesized pre-review outputs.
Recommended columns:
```sql
CREATE TABLE IF NOT EXISTS colony_aggregation_packets (
packet_id TEXT PRIMARY KEY,
mission_id UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
summary TEXT,
contradiction_count INTEGER NOT NULL DEFAULT 0,
coverage_gap_count INTEGER NOT NULL DEFAULT 0,
payload JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
### 5.8 `colony_review_packets`
Purpose:
Stores review results and final approved outputs.
Recommended columns:
```sql
CREATE TABLE IF NOT EXISTS colony_review_packets (
packet_id TEXT PRIMARY KEY,
mission_id UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
review_status TEXT NOT NULL,
issue_count INTEGER NOT NULL DEFAULT 0,
payload JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
### 5.9 `colony_policy_decisions`
Purpose:
Stores explicit allow or deny decisions.
Recommended columns:
```sql
CREATE TABLE IF NOT EXISTS colony_policy_decisions (
decision_id TEXT PRIMARY KEY,
mission_id UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
task_id TEXT,
decision_type TEXT NOT NULL,
subject TEXT NOT NULL,
decision TEXT NOT NULL,
reason TEXT,
payload JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
### 5.10 `colony_writeback_proposals`
Purpose:
Stores proposed domain mutations before approval.
Recommended columns:
```sql
CREATE TABLE IF NOT EXISTS colony_writeback_proposals (
proposal_id TEXT PRIMARY KEY,
mission_id UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
target_entity_type TEXT NOT NULL,
target_entity_id TEXT NOT NULL,
action_type TEXT NOT NULL,
approval_status TEXT NOT NULL DEFAULT 'pending',
payload JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
resolved_at TIMESTAMPTZ
);
```
### 5.11 `colony_event_log`
Purpose:
Stores lifecycle events for audit and replay.
Recommended columns:
```sql
CREATE TABLE IF NOT EXISTS colony_event_log (
event_id BIGSERIAL PRIMARY KEY,
mission_id UUID NOT NULL REFERENCES colony_missions(mission_id) ON DELETE CASCADE,
task_id TEXT,
event_type TEXT NOT NULL,
event_source TEXT NOT NULL,
payload JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
### 5.12 `colony_pheromone_signals`
Purpose:
Stores compact reinforcement signals for future routing improvements.
Recommended columns:
```sql
CREATE TABLE IF NOT EXISTS colony_pheromone_signals (
signal_id BIGSERIAL PRIMARY KEY,
mission_type TEXT NOT NULL,
role_type TEXT NOT NULL,
signal_type TEXT NOT NULL,
subject TEXT NOT NULL,
signal_strength DOUBLE PRECISION NOT NULL,
evidence_count INTEGER NOT NULL DEFAULT 1,
payload JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
## 6. Root Python Modules
Create:
- `backend/services/colony_repository.py`
- `backend/services/colony_gateway.py`
- `backend/services/colony_policy_bridge.py`
- `backend/api/routes_colony.py`
### 6.1 `colony_repository.py`
Responsibilities:
- ensure schema readiness
- insert and update mission rows
- insert artifact rows
- retrieve mission detail view
- retrieve mission artifact collection
- resolve writeback proposals
### 6.2 `colony_gateway.py`
Responsibilities:
- build HTTP client to colony service
- submit mission envelopes
- fetch mission state
- fetch artifact snapshots
- handle retries and timeout classification
### 6.3 `colony_policy_bridge.py`
Responsibilities:
- classify mission risk
- derive tool scope
- derive model routing tier
- validate writeback proposals
- produce `policy_decision` artifacts
### 6.4 `routes_colony.py`
Responsibilities:
- expose root-owned public endpoints
- authenticate callers
- authorize by origin surface and mission type
- persist mission creation
- call colony runtime
- expose approval and rejection endpoints
## 7. Root API Surface
### 7.1 `POST /api/colony/missions`
Purpose:
Create a new mission.
Request body:
```json
{
"mission_type": "oracle_advisory",
"origin_surface": "oracle",
"user_goal": "Summarize high-intent waterfront leads and recommend actions.",
"context_refs": {
"lead_ids": ["lead_123"],
"page_id": "page_abc",
"branch_id": "main"
},
"options": {
"include_external_research": true
}
}
```
Response:
```json
{
"status": "ok",
"data": {
"mission_id": "uuid",
"mission_type": "oracle_advisory",
"status": "accepted"
}
}
```
### 7.2 `GET /api/colony/missions/{mission_id}`
Purpose:
Fetch mission summary state.
Response should include:
- mission status
- review status
- current stage
- trace summary
- final output if completed
### 7.3 `GET /api/colony/missions/{mission_id}/artifacts`
Purpose:
Fetch mission artifact collection for operator inspection.
Response should include:
- task graph
- prompt packages
- research artifacts
- worker results
- aggregation packet
- review packet
- policy decisions
- writeback proposals
### 7.4 `POST /api/colony/missions/{mission_id}/approve`
Purpose:
Approve pending writeback proposal or gated finalization step.
### 7.5 `POST /api/colony/missions/{mission_id}/reject`
Purpose:
Reject pending writeback proposal or gated finalization step.
### 7.6 `GET /api/colony/health`
Purpose:
Expose root-to-colony connectivity and root persistence readiness.
Response should include:
- route health
- db readiness
- colony service reachability
- last connectivity check timestamp
## 8. Mission Lifecycle in Root
Mission lifecycle states should be:
- `accepted`
- `planning`
- `packaging`
- `retrieving`
- `executing`
- `aggregating`
- `reviewing`
- `awaiting_approval`
- `completed`
- `failed`
- `rejected`
These states should be stored in `colony_missions.status`.
## 9. Writeback Release Flow
The writeback release sequence should be:
1. reviewer produces `writeback_proposal`
2. proposal stored in `colony_writeback_proposals`
3. root policy bridge evaluates proposal
4. if mission requires approval, proposal waits in `pending`
5. approval endpoint resolves proposal
6. only then does root call existing CRM or Oracle write path
No colony worker should bypass this flow.
## 10. Root Repository Contract
The Python root needs one repository module that hides SQL and exposes a stable service contract to `routes_colony.py`, `routes_oracle.py`, and `routes_crm.py`.
Recommended file:
- `backend/services/colony_repository.py`
Required methods:
- `create_mission(mission_envelope: dict) -> dict`
- `update_mission_status(mission_id: str, status: str, review_status: str | None = None) -> dict`
- `store_task_graph(mission_id: str, tasks: list[dict]) -> int`
- `store_prompt_packages(mission_id: str, packages: list[dict]) -> int`
- `store_research_artifacts(mission_id: str, artifacts: list[dict]) -> int`
- `store_librarian_passes(mission_id: str, passes: list[dict]) -> int`
- `store_worker_results(mission_id: str, results: list[dict]) -> int`
- `store_aggregation_packet(mission_id: str, packet: dict) -> dict`
- `store_review_packet(mission_id: str, packet: dict) -> dict`
- `store_policy_decision(mission_id: str, decision: dict) -> dict`
- `store_writeback_proposal(mission_id: str, proposal: dict) -> dict`
- `append_event(mission_id: str, event_type: str, payload: dict) -> dict`
- `append_pheromone_signal(mission_id: str, signal: dict) -> dict`
- `get_mission_summary(mission_id: str) -> dict | None`
- `get_mission_artifacts(mission_id: str) -> dict`
- `list_missions(filters: dict) -> list[dict]`
- `resolve_writeback_proposal(mission_id: str, proposal_id: str, resolution: str, actor_id: str) -> dict`
Repository rules:
- repository methods must be idempotent where retries are expected
- artifact inserts should use unique IDs controlled by the caller, not generated ad hoc in SQL
- repository methods should raise typed exceptions for `not_found`, `conflict`, and `validation_error`
- root services should never handcraft SQL in route modules
## 11. Root API Contract
The root FastAPI layer should remain the only externally consumed colony API in Sprint 1.
Recommended files:
- `backend/api/routes_colony.py`
- `backend/services/colony_gateway.py`
### 11.1 `POST /api/colony/missions`
Purpose:
Create a mission, forward it to the TypeScript colony service, and persist the accepted mission envelope immediately.
Request body shape:
```json
{
"mission_type": "oracle_advisory",
"origin_surface": "oracle",
"tenant_id": "tenant-desineuron",
"actor_id": "user-123",
"actor_role": "advisor",
"user_goal": "Prepare a project-aware response for this lead.",
"context_refs": {
"lead_id": "lead-001",
"page_id": "oracle-main"
},
"requested_outputs": [
"final_answer",
"writeback_proposal"
],
"payload": {
"prompt": "What is the best next action for this lead?"
}
}
```
Response shape:
```json
{
"mission_id": "uuid",
"status": "accepted",
"review_status": "not_required",
"current_stage": "planning",
"accepted_at": "timestamp",
"links": {
"self": "/api/colony/missions/uuid",
"artifacts": "/api/colony/missions/uuid/artifacts"
}
}
```
### 11.2 `GET /api/colony/missions/{mission_id}`
Purpose:
Fetch mission summary state.
Response should include:
- mission status
- review status
- current stage
- trace summary
- final output if completed
- pending approval summary if applicable
### 11.3 `GET /api/colony/missions/{mission_id}/artifacts`
Purpose:
Fetch mission artifact collection for operator inspection.
Response should include:
- task graph
- prompt packages
- research artifacts
- worker results
- aggregation packet
- review packet
- policy decisions
- writeback proposals
This endpoint should support:
- `?include_payloads=true|false`
- `?artifact_type=worker_results`
- `?task_id=...`
### 11.4 `POST /api/colony/missions/{mission_id}/approve`
Purpose:
Approve pending writeback proposal or gated finalization step.
Request shape:
```json
{
"proposal_id": "proposal-001",
"resolution_note": "approved after operator review"
}
```
### 11.5 `POST /api/colony/missions/{mission_id}/reject`
Purpose:
Reject pending writeback proposal or gated finalization step.
Request shape:
```json
{
"proposal_id": "proposal-001",
"resolution_note": "insufficient evidence for automated writeback"
}
```
### 11.6 `GET /api/colony/health`
Purpose:
Expose root-to-colony connectivity and root persistence readiness.
Response should include:
- route health
- db readiness
- colony service reachability
- last connectivity check timestamp
- schema version loaded by root
## 12. Mission Lifecycle in Root
Mission lifecycle states should be:
- `accepted`
- `planning`
- `packaging`
- `retrieving`
- `executing`
- `aggregating`
- `reviewing`
- `awaiting_approval`
- `completed`
- `failed`
- `rejected`
These states should be stored in `colony_missions.status`.
Review states should be:
- `not_required`
- `pending`
- `approved`
- `rejected`
Writeback proposal states should be:
- `pending`
- `approved`
- `rejected`
- `expired`
- `superseded`
State transition rules:
- `accepted` may move only to `planning` or `failed`
- `planning` may move only to `packaging`, `failed`, or `rejected`
- `packaging` may move only to `retrieving`, `executing`, `failed`, or `rejected`
- `retrieving` may move only to `executing`, `failed`, or `rejected`
- `executing` may move only to `aggregating`, `failed`, or `rejected`
- `aggregating` may move only to `reviewing`, `failed`, or `rejected`
- `reviewing` may move only to `completed`, `awaiting_approval`, `failed`, or `rejected`
- `awaiting_approval` may move only to `completed`, `rejected`, or `failed`
- terminal states are `completed`, `failed`, and `rejected`
## 13. Writeback Release Flow
The writeback release sequence should be:
1. reviewer produces `writeback_proposal`
2. proposal stored in `colony_writeback_proposals`
3. root policy bridge evaluates proposal
4. if mission requires approval, proposal waits in `pending`
5. approval endpoint resolves proposal
6. only then does root call existing CRM or Oracle write path
No colony worker should bypass this flow.
Additional rules:
- every executed writeback must store `resolved_by`, `resolved_at`, and `execution_result`
- root must store the target route or service name that performed the approved mutation
- a rejected proposal must remain queryable for audit and future training
## 14. Root Query Shapes
### 14.1 Mission Summary Query
Need one summary query that joins:
- `colony_missions`
- latest `colony_review_packets`
- latest `colony_writeback_proposals`
This should return one operator-facing mission summary object.
### 14.2 Mission Artifact Query
Need one artifact query bundle that returns all mission-linked rows by table.
This should be grouped in a stable response shape so the frontend and operator tools do not need to know the physical table order.
### 14.3 Mission Event Replay Query
Need one query returning ordered `colony_event_log` rows by `created_at ASC`.
### 14.4 Mission Inbox Query
Need one query to list:
- missions currently awaiting approval
- missions failed in the last 24 hours
- missions completed but unresolved for operator follow-up
This powers internal operations and demo readiness.
## 15. Concurrency, Idempotency, and Retry Rules
Root API and repository behavior must be explicit here because the colony runtime will retry on network and model failures.
Required rules:
- `POST /api/colony/missions` should accept a caller-supplied `request_id` header for idempotent creation
- duplicate mission creation with the same `request_id` should return the existing mission record
- artifact writes should use deterministic IDs where practical
- approval endpoints must reject double approval of the same proposal
- mission status updates should use optimistic concurrency with `updated_at` or revision number checks
- failed root-to-colony dispatch attempts should still preserve the accepted mission record with a clear failure event
## 16. Security and Access Control
Root API enforcement must happen before requests reach the TypeScript service.
Required controls:
- mission creation requires authenticated actor context
- Oracle-origin missions can only reference Oracle-allowed data scopes
- CRM-origin missions can only reference CRM-allowed data scopes unless explicitly elevated
- artifact inspection endpoint requires privileged operator or internal service role
- approval and rejection endpoints require writeback authority, not just read authority
- root must redact sensitive fields before returning artifacts to standard UI consumers
## 17. Migration and Rollout Plan
Database rollout order:
1. apply `schema_colony.sql`
2. verify tables and indexes
3. deploy `colony_repository.py`
4. deploy `colony_gateway.py`
5. deploy `routes_colony.py`
6. mount routes in `backend/main.py`
7. enable mission create and health only
8. enable mission status
9. enable artifacts
10. enable approval routes
Rollout rule:
Do not expose approval endpoints in production before repository writes and artifact replay are proven stable.
## 18. Acceptance Criteria
This layer is complete when:
- schema file exists and applies cleanly
- root endpoints exist and validate payloads
- root persists mission and artifacts
- root can fetch mission and artifact state
- writeback proposals are stored and require root-mediated approval
- mission status transitions are enforced centrally
- duplicate mission create requests are handled safely
- no existing root routes regress
## 19. Ticket Breakdown
Minimum ticket set:
1. create `schema_colony.sql`
2. implement `colony_repository.py`
3. implement `colony_gateway.py`
4. implement `routes_colony.py`
5. mount colony router in `backend/main.py`
6. add idempotency and approval tests
7. add operator artifact inspection tests
## 20. Bottom Line
The database and root API layer are what turn the colony from a framework experiment into a product subsystem. Without this layer, the orchestration runtime remains difficult to govern and difficult to sell.
Reference in New Issue View Git Blame Copy Permalink