278 lines
8.1 KiB
Markdown
278 lines
8.1 KiB
Markdown
# The Sentinel Bibel
|
|
|
|
Updated: April 2, 2026
|
|
|
|
## 1. What The Sentinel Is
|
|
|
|
The Sentinel is the biometric, session-intelligence, and attention-scoring engine inside Project Velocity. It exists to help brokers understand how a prospect reacts to a guided property walkthrough in real time, without streaming raw webcam video to a remote backend.
|
|
|
|
It fuses:
|
|
|
|
- browser-side facial blend-shape capture
|
|
- scene-aware timestamp correlation
|
|
- CRM context
|
|
- asset-open notifications
|
|
- CCTV-assisted auto-mode enrichment
|
|
- lead tagging and session finalization
|
|
|
|
The output is not just a number. The output is operational intelligence.
|
|
|
|
## 2. Core Principles
|
|
|
|
### Local-first perception
|
|
|
|
The webcam is processed in the browser through MediaPipe. The backend receives only compact biometric packets, not raw video.
|
|
|
|
### NVMe-first deployment
|
|
|
|
Runtime data, prompts, PostgreSQL, certificates, and backend release assets belong on `/opt/dlami/nvme`, not the root volume.
|
|
|
|
### PostgreSQL is the source of truth
|
|
|
|
No Supabase database runtime is used for the Sentinel workflow. Session, consent, vault, CCTV, and sentiment history all land in PostgreSQL.
|
|
|
|
### Broker-facing intelligence must be real time
|
|
|
|
The system pushes QD updates and vault-open signals back through WebSockets so the broker can act during or immediately after the session.
|
|
|
|
### Architecture follows truth, not plan nostalgia
|
|
|
|
The original plan centered more heavily on local Ollama/OpenShell inference. The current deployed truth is NVIDIA-primary reasoning with OpenShell/Ollama still present as neighboring infrastructure.
|
|
|
|
## 3. How The Sentinel Works
|
|
|
|
### Assigned Mode
|
|
|
|
1. Broker selects a marketing video
|
|
2. Broker selects `Assigned Mode`
|
|
3. Broker selects an existing lead
|
|
4. Prospect watches the video while the browser emits biometric packets
|
|
5. Backend scores QD using scene context plus CRM context
|
|
6. Lead score updates and notifications are visible in real time
|
|
|
|
### Auto Mode
|
|
|
|
1. Broker selects a marketing video
|
|
2. Broker selects `Auto Mode`
|
|
3. Session runs without a pre-bound lead
|
|
4. Browser emits biometric packets tied to a session UUID
|
|
5. CCTV events enrich the same session through `/api/cctv/event`
|
|
6. Session closes through `/api/sentinel/session/complete`
|
|
7. Auto-mode matcher links an existing lead or creates a new one
|
|
|
|
## 4. The Files That Matter
|
|
|
|
### Frontend
|
|
|
|
`app/src/components/modules/Sentinel.tsx`
|
|
Sentinel shell and tab structure.
|
|
|
|
`app/src/components/modules/sentinel/SentinelLiveSession.tsx`
|
|
The broker-facing session wizard. This is the orchestration UI.
|
|
|
|
`app/src/components/modules/sentinel/PerceptionPlayer.tsx`
|
|
Video playback, webcam capture, packet emission, and session closeout.
|
|
|
|
`app/src/hooks/useMediapipeFaceLandmarker.ts`
|
|
MediaPipe bootstrap and frame detection.
|
|
|
|
`app/src/hooks/useVelocitySocket.ts`
|
|
Notification and perception WebSocket connection logic.
|
|
|
|
`app/src/components/layout/NotificationCenter.tsx`
|
|
Top-bar operational notification panel.
|
|
|
|
### Backend
|
|
|
|
`backend/main.py`
|
|
FastAPI entrypoint and router registration.
|
|
|
|
`backend/routers/sentinel.py`
|
|
Perception WebSocket, notifications, consent, session completion, tag-lead route, QD score endpoint.
|
|
|
|
`backend/routers/vault.py`
|
|
Trackable links and vault-open broadcast flow.
|
|
|
|
`backend/routers/scenes.py`
|
|
Scene CSV upload and scene retrieval.
|
|
|
|
`backend/routers/videos.py`
|
|
Marketing-video catalog loading and fallback filesystem discovery.
|
|
|
|
`backend/routers/cctv.py`
|
|
CCTV event ingestion and auto-mode finalization.
|
|
|
|
`backend/services/nemoclaw_client.py`
|
|
NVIDIA-primary prompt client and response parsing.
|
|
|
|
`backend/services/auto_mode_matcher.py`
|
|
Auto-mode session-to-lead attribution logic.
|
|
|
|
`backend/db/schema.sql`
|
|
Core Sentinel relational model.
|
|
|
|
`backend/db/schema_addendum.sql`
|
|
Scene maps, perception sessions, CCTV events.
|
|
|
|
## 5. The Tables That Matter
|
|
|
|
`users_and_roles`
|
|
RBAC and broker identity.
|
|
|
|
`leads_intelligence`
|
|
Lead record, tags, QD score, broker assignment.
|
|
|
|
`velocity_vault_assets`
|
|
Trackable asset share records.
|
|
|
|
`omnichannel_logs`
|
|
Event history and sentiment spikes.
|
|
|
|
`consent_log`
|
|
Biometric consent actions.
|
|
|
|
`video_scene_maps`
|
|
Scene labels and timestamp ranges for each marketing video.
|
|
|
|
`perception_sessions`
|
|
Assigned or auto-mode session records.
|
|
|
|
`cctv_events`
|
|
OCR/vehicle/visitor-derived session evidence.
|
|
|
|
## 6. Ports and Paths
|
|
|
|
Public/backend-facing truth:
|
|
|
|
- `22` SSH
|
|
- `443` nginx TLS
|
|
- `127.0.0.1:8001` Uvicorn
|
|
- `127.0.0.1:5432` PostgreSQL
|
|
- `8080` OpenShell gateway
|
|
- `11434` Ollama
|
|
|
|
Paths:
|
|
|
|
- `/opt/dlami/nvme/velocity/current`
|
|
- `/opt/dlami/nvme/velocity/env`
|
|
- `/opt/dlami/nvme/velocity/venv`
|
|
- `/opt/dlami/nvme/velocity/tls`
|
|
- `/opt/dlami/nvme/nemoclaw/prompts`
|
|
- `/opt/dlami/nvme/pgdata/14/velocity`
|
|
- `/opt/dlami/nvme/assets`
|
|
- `/opt/dlami/nvme/assets/videos`
|
|
- `/opt/dlami/nvme/assets/videos/catalog.json`
|
|
|
|
## 7. What Is True About NemoClaw
|
|
|
|
The Sentinel still conceptually uses NemoClaw as its reasoning layer, but the active truth is:
|
|
|
|
- The backend uses NVIDIA-hosted completions as primary inference
|
|
- The active primary model is `nvidia/nemotron-3-super-120b-a12b`
|
|
- The backend prompt layer is implemented in repo code, not hidden inside a separate agent black box
|
|
- OpenShell and Ollama still exist on the node, but they are not the active primary scoring path
|
|
|
|
This distinction matters. If someone assumes the live QD score is coming from local Ollama, they are wrong.
|
|
|
|
## 8. Event Flows
|
|
|
|
### QD update flow
|
|
|
|
Frontend packet
|
|
-> `/api/sentinel/ws/perception`
|
|
-> scene lookup in `video_scene_maps`
|
|
-> QD scoring in `nemoclaw_client.py`
|
|
-> `perception_sessions` and `omnichannel_logs`
|
|
-> QD broadcast
|
|
-> frontend QD ring and notifications
|
|
|
|
### Marketing video flow
|
|
|
|
Frontend live-session picker
|
|
-> `GET /api/videos/marketing`
|
|
-> catalog read from `/opt/dlami/nvme/assets/videos/catalog.json`
|
|
-> fallback recursive scan of `/opt/dlami/nvme/assets/videos`
|
|
-> MP4 served through `/assets/videos/...`
|
|
-> hover-preview card and full `PerceptionPlayer` playback
|
|
|
|
### Vault flow
|
|
|
|
Broker generates velocity link
|
|
-> prospect opens `/vault/{tracking_hash}`
|
|
-> backend logs `WS_ASSET_OPENED`
|
|
-> notification broadcast
|
|
-> NotificationCenter shows the event
|
|
|
|
### Auto-mode CCTV flow
|
|
|
|
OCR bridge posts to `/api/cctv/event`
|
|
-> backend profiles plate/vehicle via `cctv_profiler`
|
|
-> `cctv_events` row is written
|
|
-> session evidence is updated
|
|
-> session closes
|
|
-> `auto_mode_matcher.py` links or creates lead
|
|
|
|
## 9. What Can Break
|
|
|
|
### Structured output from the model
|
|
|
|
The NVIDIA path is reachable, but model output is not always perfect JSON under the full prompt. This is the main runtime fragility.
|
|
|
|
### Wrong WebSocket path
|
|
|
|
The correct Sentinel socket namespace is `/api/sentinel/ws/...`. Any client using `/ws/...` directly will miss the router prefix and fail.
|
|
|
|
### Root-volume drift
|
|
|
|
If deployment scripts start writing runtime state back to root instead of NVMe, disk-pressure regressions will return.
|
|
|
|
### Video catalog drift
|
|
|
|
If new walkthrough files are copied to NVMe without catalog updates, the system still loads them, but the presentation order and labels can become unreliable.
|
|
|
|
### Dynamic public IP assumptions
|
|
|
|
If external scripts hardcode an old public IP, they will fail. Current truth must be rechecked whenever AWS reassigns the address.
|
|
|
|
## 10. What “Done” Actually Means
|
|
|
|
The Sentinel is done at the engineering level when:
|
|
|
|
- all milestone code exists
|
|
- backend services start cleanly on NVMe-backed runtime
|
|
- PostgreSQL schema is live
|
|
- frontend build succeeds
|
|
- backend tests pass
|
|
- Playwright tests pass
|
|
- health endpoint is live over TLS
|
|
|
|
The Sentinel is done at the operational level only when:
|
|
|
|
- a stable DNS or Elastic IP is assigned
|
|
- trusted TLS is installed
|
|
- production CCTV/OCR source is wired in
|
|
- a live broker walkthrough has been accepted on the deployed system
|
|
|
|
## 11. Commands Worth Remembering
|
|
|
|
```bash
|
|
curl -k https://54.152.236.10/health
|
|
sudo systemctl status velocity-backend.service
|
|
sudo systemctl status postgresql@14-velocity.service
|
|
sudo systemctl status nginx
|
|
sudo systemctl status nemoclaw-velocity.service
|
|
sudo -u postgres psql -d velocity -c '\dt'
|
|
```
|
|
|
|
## 12. Final Rule
|
|
|
|
Do not maintain two truths.
|
|
|
|
If deployment reality changes, update:
|
|
|
|
- the SRS
|
|
- `nemoclaw_setup_truth.md`
|
|
- this Bibel
|
|
|
|
before anyone else builds on stale assumptions.
|