Built the Sentinel Tab
This commit is contained in:
Sagnik
277
.Agent Context/Sprint 1/The Sentinel Bibel.md
Normal file
277
.Agent Context/Sprint 1/The Sentinel Bibel.md
Normal file
@@ -0,0 +1,277 @@
|
||||
# The Sentinel Bibel
|
||||
|
||||
Updated: April 2, 2026
|
||||
|
||||
## 1. What The Sentinel Is
|
||||
|
||||
The Sentinel is the biometric, session-intelligence, and attention-scoring engine inside Project Velocity. It exists to help brokers understand how a prospect reacts to a guided property walkthrough in real time, without streaming raw webcam video to a remote backend.
|
||||
|
||||
It fuses:
|
||||
|
||||
- browser-side facial blend-shape capture
|
||||
- scene-aware timestamp correlation
|
||||
- CRM context
|
||||
- asset-open notifications
|
||||
- CCTV-assisted auto-mode enrichment
|
||||
- lead tagging and session finalization
|
||||
|
||||
The output is not just a number. The output is operational intelligence.
|
||||
|
||||
## 2. Core Principles
|
||||
|
||||
### Local-first perception
|
||||
|
||||
The webcam is processed in the browser through MediaPipe. The backend receives only compact biometric packets, not raw video.
|
||||
|
||||
### NVMe-first deployment
|
||||
|
||||
Runtime data, prompts, PostgreSQL, certificates, and backend release assets belong on `/opt/dlami/nvme`, not the root volume.
|
||||
|
||||
### PostgreSQL is the source of truth
|
||||
|
||||
No Supabase database runtime is used for the Sentinel workflow. Session, consent, vault, CCTV, and sentiment history all land in PostgreSQL.
|
||||
|
||||
### Broker-facing intelligence must be real time
|
||||
|
||||
The system pushes QD updates and vault-open signals back through WebSockets so the broker can act during or immediately after the session.
|
||||
|
||||
### Architecture follows truth, not plan nostalgia
|
||||
|
||||
The original plan centered more heavily on local Ollama/OpenShell inference. The current deployed truth is NVIDIA-primary reasoning with OpenShell/Ollama still present as neighboring infrastructure.
|
||||
|
||||
## 3. How The Sentinel Works
|
||||
|
||||
### Assigned Mode
|
||||
|
||||
1. Broker selects a marketing video
|
||||
2. Broker selects `Assigned Mode`
|
||||
3. Broker selects an existing lead
|
||||
4. Prospect watches the video while the browser emits biometric packets
|
||||
5. Backend scores QD using scene context plus CRM context
|
||||
6. Lead score updates and notifications are visible in real time
|
||||
|
||||
### Auto Mode
|
||||
|
||||
1. Broker selects a marketing video
|
||||
2. Broker selects `Auto Mode`
|
||||
3. Session runs without a pre-bound lead
|
||||
4. Browser emits biometric packets tied to a session UUID
|
||||
5. CCTV events enrich the same session through `/api/cctv/event`
|
||||
6. Session closes through `/api/sentinel/session/complete`
|
||||
7. Auto-mode matcher links an existing lead or creates a new one
|
||||
|
||||
## 4. The Files That Matter
|
||||
|
||||
### Frontend
|
||||
|
||||
`app/src/components/modules/Sentinel.tsx`
|
||||
Sentinel shell and tab structure.
|
||||
|
||||
`app/src/components/modules/sentinel/SentinelLiveSession.tsx`
|
||||
The broker-facing session wizard. This is the orchestration UI.
|
||||
|
||||
`app/src/components/modules/sentinel/PerceptionPlayer.tsx`
|
||||
Video playback, webcam capture, packet emission, and session closeout.
|
||||
|
||||
`app/src/hooks/useMediapipeFaceLandmarker.ts`
|
||||
MediaPipe bootstrap and frame detection.
|
||||
|
||||
`app/src/hooks/useVelocitySocket.ts`
|
||||
Notification and perception WebSocket connection logic.
|
||||
|
||||
`app/src/components/layout/NotificationCenter.tsx`
|
||||
Top-bar operational notification panel.
|
||||
|
||||
### Backend
|
||||
|
||||
`backend/main.py`
|
||||
FastAPI entrypoint and router registration.
|
||||
|
||||
`backend/routers/sentinel.py`
|
||||
Perception WebSocket, notifications, consent, session completion, tag-lead route, QD score endpoint.
|
||||
|
||||
`backend/routers/vault.py`
|
||||
Trackable links and vault-open broadcast flow.
|
||||
|
||||
`backend/routers/scenes.py`
|
||||
Scene CSV upload and scene retrieval.
|
||||
|
||||
`backend/routers/videos.py`
|
||||
Marketing-video catalog loading and fallback filesystem discovery.
|
||||
|
||||
`backend/routers/cctv.py`
|
||||
CCTV event ingestion and auto-mode finalization.
|
||||
|
||||
`backend/services/nemoclaw_client.py`
|
||||
NVIDIA-primary prompt client and response parsing.
|
||||
|
||||
`backend/services/auto_mode_matcher.py`
|
||||
Auto-mode session-to-lead attribution logic.
|
||||
|
||||
`backend/db/schema.sql`
|
||||
Core Sentinel relational model.
|
||||
|
||||
`backend/db/schema_addendum.sql`
|
||||
Scene maps, perception sessions, CCTV events.
|
||||
|
||||
## 5. The Tables That Matter
|
||||
|
||||
`users_and_roles`
|
||||
RBAC and broker identity.
|
||||
|
||||
`leads_intelligence`
|
||||
Lead record, tags, QD score, broker assignment.
|
||||
|
||||
`velocity_vault_assets`
|
||||
Trackable asset share records.
|
||||
|
||||
`omnichannel_logs`
|
||||
Event history and sentiment spikes.
|
||||
|
||||
`consent_log`
|
||||
Biometric consent actions.
|
||||
|
||||
`video_scene_maps`
|
||||
Scene labels and timestamp ranges for each marketing video.
|
||||
|
||||
`perception_sessions`
|
||||
Assigned or auto-mode session records.
|
||||
|
||||
`cctv_events`
|
||||
OCR/vehicle/visitor-derived session evidence.
|
||||
|
||||
## 6. Ports and Paths
|
||||
|
||||
Public/backend-facing truth:
|
||||
|
||||
- `22` SSH
|
||||
- `443` nginx TLS
|
||||
- `127.0.0.1:8001` Uvicorn
|
||||
- `127.0.0.1:5432` PostgreSQL
|
||||
- `8080` OpenShell gateway
|
||||
- `11434` Ollama
|
||||
|
||||
Paths:
|
||||
|
||||
- `/opt/dlami/nvme/velocity/current`
|
||||
- `/opt/dlami/nvme/velocity/env`
|
||||
- `/opt/dlami/nvme/velocity/venv`
|
||||
- `/opt/dlami/nvme/velocity/tls`
|
||||
- `/opt/dlami/nvme/nemoclaw/prompts`
|
||||
- `/opt/dlami/nvme/pgdata/14/velocity`
|
||||
- `/opt/dlami/nvme/assets`
|
||||
- `/opt/dlami/nvme/assets/videos`
|
||||
- `/opt/dlami/nvme/assets/videos/catalog.json`
|
||||
|
||||
## 7. What Is True About NemoClaw
|
||||
|
||||
The Sentinel still conceptually uses NemoClaw as its reasoning layer, but the active truth is:
|
||||
|
||||
- The backend uses NVIDIA-hosted completions as primary inference
|
||||
- The active primary model is `nvidia/nemotron-3-super-120b-a12b`
|
||||
- The backend prompt layer is implemented in repo code, not hidden inside a separate agent black box
|
||||
- OpenShell and Ollama still exist on the node, but they are not the active primary scoring path
|
||||
|
||||
This distinction matters. If someone assumes the live QD score is coming from local Ollama, they are wrong.
|
||||
|
||||
## 8. Event Flows
|
||||
|
||||
### QD update flow
|
||||
|
||||
Frontend packet
|
||||
-> `/api/sentinel/ws/perception`
|
||||
-> scene lookup in `video_scene_maps`
|
||||
-> QD scoring in `nemoclaw_client.py`
|
||||
-> `perception_sessions` and `omnichannel_logs`
|
||||
-> QD broadcast
|
||||
-> frontend QD ring and notifications
|
||||
|
||||
### Marketing video flow
|
||||
|
||||
Frontend live-session picker
|
||||
-> `GET /api/videos/marketing`
|
||||
-> catalog read from `/opt/dlami/nvme/assets/videos/catalog.json`
|
||||
-> fallback recursive scan of `/opt/dlami/nvme/assets/videos`
|
||||
-> MP4 served through `/assets/videos/...`
|
||||
-> hover-preview card and full `PerceptionPlayer` playback
|
||||
|
||||
### Vault flow
|
||||
|
||||
Broker generates velocity link
|
||||
-> prospect opens `/vault/{tracking_hash}`
|
||||
-> backend logs `WS_ASSET_OPENED`
|
||||
-> notification broadcast
|
||||
-> NotificationCenter shows the event
|
||||
|
||||
### Auto-mode CCTV flow
|
||||
|
||||
OCR bridge posts to `/api/cctv/event`
|
||||
-> backend profiles plate/vehicle via `cctv_profiler`
|
||||
-> `cctv_events` row is written
|
||||
-> session evidence is updated
|
||||
-> session closes
|
||||
-> `auto_mode_matcher.py` links or creates lead
|
||||
|
||||
## 9. What Can Break
|
||||
|
||||
### Structured output from the model
|
||||
|
||||
The NVIDIA path is reachable, but model output is not always perfect JSON under the full prompt. This is the main runtime fragility.
|
||||
|
||||
### Wrong WebSocket path
|
||||
|
||||
The correct Sentinel socket namespace is `/api/sentinel/ws/...`. Any client using `/ws/...` directly will miss the router prefix and fail.
|
||||
|
||||
### Root-volume drift
|
||||
|
||||
If deployment scripts start writing runtime state back to root instead of NVMe, disk-pressure regressions will return.
|
||||
|
||||
### Video catalog drift
|
||||
|
||||
If new walkthrough files are copied to NVMe without catalog updates, the system still loads them, but the presentation order and labels can become unreliable.
|
||||
|
||||
### Dynamic public IP assumptions
|
||||
|
||||
If external scripts hardcode an old public IP, they will fail. Current truth must be rechecked whenever AWS reassigns the address.
|
||||
|
||||
## 10. What “Done” Actually Means
|
||||
|
||||
The Sentinel is done at the engineering level when:
|
||||
|
||||
- all milestone code exists
|
||||
- backend services start cleanly on NVMe-backed runtime
|
||||
- PostgreSQL schema is live
|
||||
- frontend build succeeds
|
||||
- backend tests pass
|
||||
- Playwright tests pass
|
||||
- health endpoint is live over TLS
|
||||
|
||||
The Sentinel is done at the operational level only when:
|
||||
|
||||
- a stable DNS or Elastic IP is assigned
|
||||
- trusted TLS is installed
|
||||
- production CCTV/OCR source is wired in
|
||||
- a live broker walkthrough has been accepted on the deployed system
|
||||
|
||||
## 11. Commands Worth Remembering
|
||||
|
||||
```bash
|
||||
curl -k https://54.152.236.10/health
|
||||
sudo systemctl status velocity-backend.service
|
||||
sudo systemctl status postgresql@14-velocity.service
|
||||
sudo systemctl status nginx
|
||||
sudo systemctl status nemoclaw-velocity.service
|
||||
sudo -u postgres psql -d velocity -c '\dt'
|
||||
```
|
||||
|
||||
## 12. Final Rule
|
||||
|
||||
Do not maintain two truths.
|
||||
|
||||
If deployment reality changes, update:
|
||||
|
||||
- the SRS
|
||||
- `nemoclaw_setup_truth.md`
|
||||
- this Bibel
|
||||
|
||||
before anyone else builds on stale assumptions.
|
||||
Reference in New Issue
Block a user