forked from sagnik/Project_Velocity
feat: Overlay the mathematical Sun Path over the live camera feed or 3D model view (#8)
#7 Task completed. Co-authored-by: Sayan Datta <sayan@Sayans-MacBook-Air.local> Reviewed-on: sagnik/Project_Velocity#8
This commit is contained in:
@@ -1,512 +1,197 @@
|
||||
# Dream Weaver — iOS ↔ Backend Integration Guide
|
||||
**Version:** 2.0 | **Updated:** 2026-03-09 | **Server:** `54.172.172.2` | **Port:** `8080`
|
||||
|
||||
> This document is for **Sayan** (iOS / Swift) and **Sourik** (backend review).
|
||||
> It describes exactly how the iPad app should talk to the Dream Weaver AI and how keywords from a user tap become a full ComfyUI generation.
|
||||
|
||||
---
|
||||
|
||||
## 1. Architecture Overview
|
||||
|
||||
```
|
||||
┌────────────────────┐ HTTP/S ┌──────────────────────────────┐
|
||||
│ │ ─── POST image ───► │ Dream Weaver Gateway │
|
||||
│ iPad App (Swift) │ │ FastAPI port 8080 │
|
||||
│ │ ◄── PNG result ─── │ dw_gateway.py │
|
||||
└────────────────────┘ └─────────────┬────────────────┘
|
||||
│ internal HTTP
|
||||
▼
|
||||
┌─────────────────────────┐
|
||||
│ ComfyUI Engine │
|
||||
│ port 8188 │
|
||||
│ RealVisXL V5.0 Ltng │
|
||||
│ 4× NVIDIA L4 (96 GB) │
|
||||
└─────────────────────────┘
|
||||
```
|
||||
|
||||
**Key rule:** The iPad app **never** talks to ComfyUI directly. It only talks to the Gateway on `:8080`.
|
||||
|
||||
---
|
||||
|
||||
## 2. How Keywords Become Prompts
|
||||
|
||||
### 2.1 The Prompt Expansion System
|
||||
|
||||
Each interior style in the app is backed by a **prompt template** in `comfy_engine/prompts/`. When the user taps a style card (or types keywords), those keywords get **merged into the template** to build the final ComfyUI prompt injected into node `9` (positive CLIPTextEncode).
|
||||
|
||||
**Prompt template structure** (from [scandinavian_minimalist.txt](file:///F:/Workin%20In%20Progress/DESINEURON/GITLAB/Project_Velocity/comfy_engine/prompts/scandinavian_minimalist.txt)):
|
||||
```
|
||||
POSITIVE PROMPT:
|
||||
scandinavian minimalist interior design, light oak wood flooring, neutral beige textiles,
|
||||
abundant natural light streaming through large windows, clean white walls, ...
|
||||
Style Weight: <lora:Interior_Style_Scandi:0.8>
|
||||
|
||||
NEGATIVE PROMPT:
|
||||
(worst quality, low quality, illustration, 3d render...), heavy ornamentation,...
|
||||
|
||||
TECHNICAL PARAMETERS:
|
||||
- Denoising Strength: 0.70
|
||||
- CFG Scale: 7.0
|
||||
- Recommended Sampler: dpmpp_2m_karras
|
||||
- Steps: 30-40
|
||||
```
|
||||
|
||||
### 2.2 Keyword Expansion Flow
|
||||
|
||||
```
|
||||
User taps: ["marble", "gold", "luxury"]
|
||||
+
|
||||
Style selected: "art_deco"
|
||||
│
|
||||
▼
|
||||
Backend expands:
|
||||
base_prompt = art_deco_luxe.txt (POSITIVE PROMPT section)
|
||||
user_keywords_str = "marble, gold, luxury"
|
||||
final_prompt = base_prompt + ", " + user_keywords_str
|
||||
│
|
||||
▼
|
||||
Injected into ComfyUI workflow:
|
||||
node "9" → CLIPTextEncode → text: [final_prompt]
|
||||
node "10" → CLIPTextEncode → text: [negative_prompt from template]
|
||||
node "1" → LoadImage → image: [uploaded filename]
|
||||
node "13" → KSampler → denoise: 0.72, cfg: 7.5, steps: 35
|
||||
```
|
||||
|
||||
### 2.3 Available Styles and Their Keywords (for the Style Picker UI)
|
||||
|
||||
| Style ID | Display Name | Suggested Keywords Palette |
|
||||
|---|---|---|
|
||||
| `scandinavian` | Scandinavian Minimalist | oak, linen, white, hygge, cozy, birch, natural |
|
||||
| `art_deco` | Art Deco Luxe | gold, marble, velvet, geometric, 1920s, brass, crystal |
|
||||
| `biophilic` | Biophilic Organic | green wall, stone, rattan, terracotta, botanical, moss |
|
||||
| `cyberpunk` | Cyberpunk Neon | neon, chrome, holographic, dark, LED, futuristic, blade runner |
|
||||
| `japandi` | Japandi Fusion | wabi-sabi, ash wood, ceramic, zen, minimal, shoji, serene |
|
||||
|
||||
---
|
||||
|
||||
## 3. API Reference — What Sayan Needs to Call
|
||||
|
||||
### BASE URL
|
||||
```
|
||||
http://54.172.172.2:8080
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> Once we attach an Elastic IP or domain, swap this in `AppConfig.swift`.
|
||||
|
||||
---
|
||||
|
||||
### 3.1 `GET /health` — Liveness Check
|
||||
|
||||
Call this on app launch to confirm the server is up before showing the Generate button.
|
||||
|
||||
**Request:**
|
||||
```http
|
||||
GET http://54.172.172.2:8080/health
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"comfyui": true,
|
||||
"gpu": "4x NVIDIA L4 (96GB VRAM)",
|
||||
"model": "RealVisXL V5.0 Lightning"
|
||||
}
|
||||
```
|
||||
|
||||
**Swift:**
|
||||
```swift
|
||||
func checkServerHealth() async throws -> Bool {
|
||||
let url = URL(string: "\(AppConfig.baseURL)/health")!
|
||||
let (data, _) = try await URLSession.shared.data(from: url)
|
||||
let json = try JSONDecoder().decode(HealthResponse.self, from: data)
|
||||
return json.status == "ok"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3.2 `POST /dream-weaver` — Submit Generation Job (Async)
|
||||
|
||||
Use this for the main generation flow. Returns a `job_id` immediately; poll for result.
|
||||
|
||||
**Request:** `multipart/form-data`
|
||||
| Field | Type | Required | Description |
|
||||
|---|---|---|---|
|
||||
| [image](file:///F:/Workin%20In%20Progress/DESINEURON/GITLAB/Project_Velocity/comfy_engine/scripts/a100_deployment_executor.py#306-322) | File (JPEG/PNG) | ✅ | The room photo from camera or library |
|
||||
| [style](file:///F:/Workin%20In%20Progress/DESINEURON/GITLAB/Project_Velocity/comfy_engine/scripts/dreamweaver_batch_processor.py#394-414) | String | ✅ | One of: `scandinavian`, `art_deco`, `biophilic`, `cyberpunk`, `japandi` |
|
||||
| `keywords` | String | ➕ | Comma-separated user keywords e.g. `"gold, marble, luxury"` |
|
||||
| `denoise` | Float | ➕ | 0.5–0.85 (default `0.72`). Higher = more creative |
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"job_id": "a1b2c3d4-...",
|
||||
"status": "processing",
|
||||
"poll_url": "/dream-weaver/status/a1b2c3d4-...",
|
||||
"result_url": "/dream-weaver/result/a1b2c3d4-..."
|
||||
}
|
||||
```
|
||||
|
||||
**Swift example:**
|
||||
```swift
|
||||
func submitGeneration(image: UIImage, style: String, keywords: [String]) async throws -> GenerationJob {
|
||||
let url = URL(string: "\(AppConfig.baseURL)/dream-weaver")!
|
||||
var request = URLRequest(url: url)
|
||||
request.httpMethod = "POST"
|
||||
|
||||
let boundary = UUID().uuidString
|
||||
request.setValue("multipart/form-data; boundary=\(boundary)", forHTTPHeaderField: "Content-Type")
|
||||
|
||||
var body = Data()
|
||||
// Image field
|
||||
let imageData = image.jpegData(compressionQuality: 0.85)!
|
||||
body.append("--\(boundary)\r\n".data(using: .utf8)!)
|
||||
body.append("Content-Disposition: form-data; name=\"image\"; filename=\"room.jpg\"\r\n".data(using: .utf8)!)
|
||||
body.append("Content-Type: image/jpeg\r\n\r\n".data(using: .utf8)!)
|
||||
body.append(imageData)
|
||||
body.append("\r\n".data(using: .utf8)!)
|
||||
|
||||
// Style field
|
||||
body.append("--\(boundary)\r\n".data(using: .utf8)!)
|
||||
body.append("Content-Disposition: form-data; name=\"style\"\r\n\r\n".data(using: .utf8)!)
|
||||
body.append("\(style)\r\n".data(using: .utf8)!)
|
||||
|
||||
// Keywords field (user tapped keywords)
|
||||
if !keywords.isEmpty {
|
||||
let kwString = keywords.joined(separator: ", ")
|
||||
body.append("--\(boundary)\r\n".data(using: .utf8)!)
|
||||
body.append("Content-Disposition: form-data; name=\"keywords\"\r\n\r\n".data(using: .utf8)!)
|
||||
body.append("\(kwString)\r\n".data(using: .utf8)!)
|
||||
}
|
||||
|
||||
body.append("--\(boundary)--\r\n".data(using: .utf8)!)
|
||||
request.httpBody = body
|
||||
|
||||
let (data, _) = try await URLSession.shared.data(for: request)
|
||||
return try JSONDecoder().decode(GenerationJob.self, from: data)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3.3 `GET /dream-weaver/status/{job_id}` — Poll Job Status
|
||||
|
||||
Poll every **2 seconds** until `ready == true`.
|
||||
|
||||
**Response while processing:**
|
||||
```json
|
||||
{ "status": "processing", "ready": false, "style": "art_deco" }
|
||||
```
|
||||
|
||||
**Response when done:**
|
||||
```json
|
||||
{
|
||||
"status": "done",
|
||||
"ready": true,
|
||||
"result_url": "/dream-weaver/result/a1b2c3d4-...",
|
||||
"style": "art_deco"
|
||||
}
|
||||
```
|
||||
|
||||
**Swift polling loop:**
|
||||
```swift
|
||||
func pollForResult(jobId: String) async throws -> URL {
|
||||
let statusURL = URL(string: "\(AppConfig.baseURL)/dream-weaver/status/\(jobId)")!
|
||||
|
||||
for _ in 0..<150 { // max 5 min (150 × 2s)
|
||||
try await Task.sleep(nanoseconds: 2_000_000_000) // 2 seconds
|
||||
let (data, _) = try await URLSession.shared.data(from: statusURL)
|
||||
let status = try JSONDecoder().decode(JobStatus.self, from: data)
|
||||
|
||||
if status.ready {
|
||||
return URL(string: "\(AppConfig.baseURL)/dream-weaver/result/\(jobId)")!
|
||||
}
|
||||
if status.status == "error" {
|
||||
throw DreamWeaverError.generationFailed(status.error ?? "Unknown")
|
||||
}
|
||||
}
|
||||
throw DreamWeaverError.timeout
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3.4 `GET /dream-weaver/result/{job_id}` — Download Result Image
|
||||
|
||||
Returns a PNG image stream directly. Download and display to user.
|
||||
|
||||
```swift
|
||||
func downloadResult(resultURL: URL) async throws -> UIImage {
|
||||
let (data, _) = try await URLSession.shared.data(from: resultURL)
|
||||
guard let image = UIImage(data: data) else {
|
||||
throw DreamWeaverError.invalidImageData
|
||||
}
|
||||
return image
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3.5 `POST /dream-weaver/sync` — One-Shot Blocking Call
|
||||
|
||||
For **testing only** or fast network connections. Waits up to 120 seconds and returns the image directly.
|
||||
|
||||
```swift
|
||||
// Same multipart form as /dream-weaver but returns PNG bytes directly
|
||||
// Not recommended for production — use async flow above
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Keyword → Prompt Expansion (Backend Change Required)
|
||||
|
||||
> [!IMPORTANT]
|
||||
> The current [dw_gateway.py](file:///C:/Windows/Temp/dw_gateway.py) does NOT yet accept `keywords` from the app.
|
||||
> Sagnik needs to add keyword expansion to the gateway. Here is the exact code change:
|
||||
|
||||
**In [dw_gateway.py](file:///C:/Windows/Temp/dw_gateway.py), update [build_workflow()](file:///C:/Windows/Temp/dw_gateway.py#33-65):**
|
||||
|
||||
```python
|
||||
# Prompt library — maps style ID to (positive_base, negative, cfg, denoise, steps)
|
||||
STYLE_LIBRARY = {
|
||||
"scandinavian": {
|
||||
"pos": "scandinavian minimalist interior design, light oak wood flooring, neutral beige textiles, abundant natural light, clean white walls, simple functional furniture, cozy hygge atmosphere, architectural photography, 8k resolution, photorealistic",
|
||||
"neg": "(worst quality, low quality, illustration, 3d render, painting, cartoon, sketch), blurry, distorted, extra windows, unrealistic lighting, structural changes",
|
||||
"cfg": 7.0, "denoise": 0.70, "steps": 30,
|
||||
},
|
||||
"art_deco": {
|
||||
"pos": "art deco luxury interior design, geometric chevron patterns, gold brass accents, rich velvet upholstery emerald and sapphire, sunburst mirrors, polished marble flooring, crystal chandeliers, 1920s glamour, 8k resolution, photorealistic",
|
||||
"neg": "(worst quality, low quality, illustration, 3d render, painting, cartoon, sketch), blurry, distorted, structural changes, rustic, minimalism, cheap materials",
|
||||
"cfg": 7.5, "denoise": 0.72, "steps": 30,
|
||||
},
|
||||
"biophilic": {
|
||||
"pos": "biophilic organic interior design, living green walls with ferns and moss, natural stone accent walls, rattan and bamboo furniture, abundant houseplants, earth tone sage green and terracotta, 8k resolution, photorealistic, dappled sunlight",
|
||||
"neg": "(worst quality, low quality, illustration, 3d render, painting, cartoon, sketch), blurry, distorted, structural changes, synthetic materials, plastic plants",
|
||||
"cfg": 7.0, "denoise": 0.68, "steps": 30,
|
||||
},
|
||||
"cyberpunk": {
|
||||
"pos": "cyberpunk neon interior design, high contrast LED strip lighting electric blue and hot pink, reflective chrome surfaces, dark matte walls, futuristic furniture, glowing circuit patterns, tech-noir blade runner aesthetic, 8k resolution, photorealistic, volumetric fog",
|
||||
"neg": "(worst quality, low quality, illustration, 3d render, painting, cartoon, sketch), blurry, distorted, structural changes, natural daylight, rustic elements",
|
||||
"cfg": 8.0, "denoise": 0.75, "steps": 30,
|
||||
},
|
||||
"japandi": {
|
||||
"pos": "japandi fusion interior design, wabi-sabi textures, low-profile furniture, muted earth tones warm grays soft browns, handmade ceramic accents, light ash wood, shoji screen elements, minimal decoration, zen garden elements, 8k resolution, photorealistic, serene",
|
||||
"neg": "(worst quality, low quality, illustration, 3d render, painting, cartoon, sketch), blurry, distorted, structural changes, bright colors, ornate decoration, cluttered",
|
||||
"cfg": 6.5, "denoise": 0.70, "steps": 30,
|
||||
},
|
||||
}
|
||||
|
||||
def build_workflow(img_name: str, style: str = "scandinavian",
|
||||
keywords: str = "", denoise_override: float = None) -> dict:
|
||||
s = STYLE_LIBRARY.get(style, STYLE_LIBRARY["scandinavian"])
|
||||
|
||||
# Merge user keywords into base positive prompt
|
||||
pos = s["pos"]
|
||||
if keywords.strip():
|
||||
pos = pos + ", " + keywords.strip()
|
||||
|
||||
cfg = s["cfg"]
|
||||
denoise = denoise_override if denoise_override else s["denoise"]
|
||||
steps = s["steps"]
|
||||
neg = s["neg"]
|
||||
|
||||
return {
|
||||
"1": {"class_type": "CheckpointLoaderSimple",
|
||||
"inputs": {"ckpt_name": "realvisxlV50_v50LightningBakedvae.safetensors"}},
|
||||
"2": {"class_type": "LoadImage",
|
||||
"inputs": {"image": img_name, "upload": "image"}},
|
||||
"3": {"class_type": "CLIPTextEncode",
|
||||
"inputs": {"text": pos, "clip": ["1", 1]}}, # ← POSITIVE PROMPT
|
||||
"4": {"class_type": "CLIPTextEncode",
|
||||
"inputs": {"text": neg, "clip": ["1", 1]}}, # ← NEGATIVE PROMPT
|
||||
"5": {"class_type": "VAEEncode",
|
||||
"inputs": {"pixels": ["2", 0], "vae": ["1", 2]}},
|
||||
"6": {"class_type": "KSampler",
|
||||
"inputs": {"model": ["1", 0], "positive": ["3", 0], "negative": ["4", 0],
|
||||
"latent_image": ["5", 0],
|
||||
"seed": int(time.time()) % 999983,
|
||||
"steps": steps, "cfg": cfg,
|
||||
"sampler_name": "dpmpp_2m", "scheduler": "karras",
|
||||
"denoise": denoise}},
|
||||
"7": {"class_type": "VAEDecode",
|
||||
"inputs": {"samples": ["6", 0], "vae": ["1", 2]}},
|
||||
"8": {"class_type": "SaveImage",
|
||||
"inputs": {"images": ["7", 0], "filename_prefix": f"dreamweaver_{style}"}}
|
||||
}
|
||||
```
|
||||
|
||||
**Also update the endpoint signatures** to accept `keywords`:
|
||||
```python
|
||||
@app.post("/dream-weaver")
|
||||
async def dream_weaver(
|
||||
image: UploadFile = File(...),
|
||||
style: str = Form(default="scandinavian"),
|
||||
keywords: str = Form(default=""), # ← ADD THIS
|
||||
denoise: float = Form(default=0.0), # 0.0 = use style default
|
||||
):
|
||||
...
|
||||
wf = build_workflow(comfy_name, style=style, keywords=keywords,
|
||||
denoise_override=denoise if denoise > 0 else None)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Swift Data Models
|
||||
|
||||
```swift
|
||||
// AppConfig.swift
|
||||
struct AppConfig {
|
||||
static let baseURL = "http://54.172.172.2:8080"
|
||||
// Change this to HTTPS domain once SSL is set up
|
||||
}
|
||||
|
||||
// Models
|
||||
struct GenerationJob: Codable {
|
||||
let jobId: String
|
||||
let status: String
|
||||
let pollUrl: String
|
||||
let resultUrl: String
|
||||
|
||||
enum CodingKeys: String, CodingKey {
|
||||
case jobId = "job_id"
|
||||
case status, pollUrl = "poll_url", resultUrl = "result_url"
|
||||
}
|
||||
}
|
||||
|
||||
struct JobStatus: Codable {
|
||||
let status: String
|
||||
let ready: Bool
|
||||
let resultUrl: String?
|
||||
let error: String?
|
||||
|
||||
enum CodingKeys: String, CodingKey {
|
||||
case status, ready, resultUrl = "result_url", error
|
||||
}
|
||||
}
|
||||
|
||||
struct HealthResponse: Codable {
|
||||
let status: String
|
||||
let comfyui: Bool
|
||||
}
|
||||
|
||||
enum DreamWeaverError: Error {
|
||||
case generationFailed(String)
|
||||
case timeout
|
||||
case invalidImageData
|
||||
}
|
||||
|
||||
// Style model for the style picker
|
||||
struct InteriorStyle: Identifiable {
|
||||
let id: String // used as the `style` form field value
|
||||
let displayName: String
|
||||
let keywords: [String] // shown as tappable chips in UI
|
||||
let previewImage: String // local asset name
|
||||
}
|
||||
|
||||
let availableStyles: [InteriorStyle] = [
|
||||
InteriorStyle(id: "scandinavian", displayName: "Scandinavian", keywords: ["oak","linen","white","hygge","cozy","birch"], previewImage: "style_scandi"),
|
||||
InteriorStyle(id: "art_deco", displayName: "Art Deco Luxe", keywords: ["gold","marble","velvet","geometric","brass","crystal"], previewImage: "style_artdeco"),
|
||||
InteriorStyle(id: "biophilic", displayName: "Biophilic", keywords: ["green wall","stone","rattan","terracotta","botanical"], previewImage: "style_biophilic"),
|
||||
InteriorStyle(id: "cyberpunk", displayName: "Cyberpunk Neon", keywords: ["neon","chrome","LED","futuristic","dark","holographic"], previewImage: "style_cyberpunk"),
|
||||
InteriorStyle(id: "japandi", displayName: "Japandi Fusion", keywords: ["wabi-sabi","ceramic","ash wood","zen","minimal"], previewImage: "style_japandi"),
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Complete Generation Flow (UI → Server → Result)
|
||||
|
||||
```
|
||||
1. User opens camera / library → picks a room photo
|
||||
2. User selects a style card → style ID captured
|
||||
3. User optionally taps keyword chips → keywords[] array built
|
||||
4. User taps "Generate" →
|
||||
|
||||
POST /dream-weaver (multipart)
|
||||
image: <jpeg data>
|
||||
style: "art_deco"
|
||||
keywords: "gold, marble, luxury hotel"
|
||||
denoise: 0.72
|
||||
|
||||
5. Server returns { job_id: "abc123", status: "processing" }
|
||||
6. App shows loading/progress UI
|
||||
7. App polls GET /dream-weaver/status/abc123 every 2 seconds
|
||||
8. When ready == true →
|
||||
GET /dream-weaver/result/abc123 → returns PNG bytes
|
||||
9. App displays result full-screen with save/share options
|
||||
```
|
||||
|
||||
**Expected latency on 4× L4 GPU server:** `~15–20 seconds` end-to-end.
|
||||
|
||||
---
|
||||
|
||||
## 7. WebSocket Progress (Optional Advanced Feature)
|
||||
|
||||
If Sayan wants a real-time progress bar (e.g. "Step 12/30"), connect directly to ComfyUI's WebSocket **only if port 8188 is opened**. Otherwise, polling `/status` is sufficient.
|
||||
|
||||
```swift
|
||||
// WebSocket — only if 8188 is exposed externally
|
||||
class ComfyProgressWebSocket: NSObject, URLSessionWebSocketDelegate {
|
||||
var onProgress: ((Int, Int) -> Void)?
|
||||
var task: URLSessionWebSocketTask?
|
||||
|
||||
func connect(clientId: String) {
|
||||
let url = URL(string: "ws://54.172.172.2:8188/ws?clientId=\(clientId)")!
|
||||
task = URLSession.shared.webSocketTask(with: url)
|
||||
task?.resume()
|
||||
listen()
|
||||
}
|
||||
|
||||
private func listen() {
|
||||
task?.receive { [weak self] result in
|
||||
if case .success(let message) = result,
|
||||
case .string(let text) = message,
|
||||
let data = text.data(using: .utf8),
|
||||
let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
|
||||
let type = json["type"] as? String, type == "progress",
|
||||
let inner = json["data"] as? [String: Int] {
|
||||
self?.onProgress?(inner["value"] ?? 0, inner["max"] ?? 30)
|
||||
}
|
||||
self?.listen() // recurse
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> Port 8188 is currently **not open externally** in the security group. Only port 8080 is. To use WebSocket progress, Sagnik needs to add an inbound rule for 8188. Until then, using `/status` polling every 2s gives good enough UX.
|
||||
|
||||
---
|
||||
|
||||
## 8. Error Handling
|
||||
|
||||
| HTTP Status | Meaning | UI Action |
|
||||
|---|---|---|
|
||||
| `200` | Success | Show result or job_id |
|
||||
| `404` on `/status` | Job expired (> 30 min) | "Session expired. Please retry." |
|
||||
| `500` | Generation failed (OOM, model error) | "Generation failed. Try a simpler image." |
|
||||
| Connection error | Server down or no internet | "Checking server…" + retry logic |
|
||||
|
||||
The job [status](file:///C:/Windows/Temp/dw_gateway.py#154-164) field can also be `"error"` with an `error` field explaining what failed.
|
||||
|
||||
---
|
||||
|
||||
## 9. Quick Checklist for Sayan
|
||||
|
||||
- [ ] Update `AppConfig.swift` with `baseURL = "http://54.172.172.2:8080"`
|
||||
- [ ] Implement `POST /dream-weaver` multipart with `image + style + keywords`
|
||||
- [ ] Implement polling loop on `GET /dream-weaver/status/{job_id}`
|
||||
- [ ] Implement image download from `GET /dream-weaver/result/{job_id}`
|
||||
- [ ] Add `GET /health` check on app launch
|
||||
- [ ] Build keyword chips UI with the 5 style palettes from Section 2.3
|
||||
- [ ] Test with the 20 sample images in `comfy_engine/test_inputs/`
|
||||
|
||||
## 10. Quick Checklist for Sagnik (backend)
|
||||
|
||||
- [ ] Update [dw_gateway.py](file:///C:/Windows/Temp/dw_gateway.py) with the full `STYLE_LIBRARY` dict (Section 4)
|
||||
- [ ] Add `keywords: str = Form(default="")` to both POST endpoints
|
||||
- [ ] Pass keywords into [build_workflow()](file:///C:/Windows/Temp/dw_gateway.py#33-65) for prompt expansion
|
||||
- [ ] Redeploy gateway on port 8080 (`nohup python3 dw_gateway.py &`)
|
||||
- [ ] (Optional) Open port 8188 in security group for WebSocket progress
|
||||
# Dream Weaver API v2 — iOS Integration Guide (Dynamic Keywords)
|
||||
**Version:** 2.0-FINAL | **Updated:** 2026-03-09 | **Server:** `54.172.172.2` | **Port:** `8082`
|
||||
|
||||
> This document is for **Sayan** (iOS / Swift).
|
||||
> Dream Weaver API v2 introduces a **Dynamic Keyword to Local LLM Prompt Expansion** system.
|
||||
> The app no longer relies on 5 hardcoded styles. Users can pick ANY keywords, and a local LLM (Qwen 3.5 27B via Ollama) will generate a photorealistic interior design prompt based on the room type without sending data to the cloud.
|
||||
|
||||
> [!CAUTION]
|
||||
> **PORT 8080 IS DEAD.** Do not use port 8080 anymore. The old gateway process has been completely killed. If you try to send `POST /dream-weaver` or `/docs` to port 8080 you will get a 404. You MUST change your `AppConfig.baseURL` parameter to use port **`8082`**.
|
||||
|
||||
---
|
||||
|
||||
## 1. Architecture Overview (API v2)
|
||||
|
||||
```
|
||||
┌────────────────────┐ HTTP/S ┌──────────────────────────────┐
|
||||
│ │ ── keywords ────► │ Dream Weaver Gateway v2 │
|
||||
│ iPad App (Swift) │ │ FastAPI port 8082 │
|
||||
│ │ ◄── PNG result ── │ dw_gateway_v2.py │
|
||||
└────────────────────┘ └─────────────┬────────────────┘
|
||||
│ LLM Prompt Expansion
|
||||
│ (Local Ollama: Qwen 3.5 27B)
|
||||
▼
|
||||
┌─────────────────────────┐
|
||||
│ ComfyUI Engine │
|
||||
│ port 8188 │
|
||||
│ RealVisXL V5.0 Ltng │
|
||||
└─────────────────────────┘
|
||||
```
|
||||
|
||||
**Key changes in v2:**
|
||||
1. The API now runs on port **`8082`** to avoid conflicts.
|
||||
2. The [style](file:///F:/Workin%20In%20Progress/DESINEURON/GITLAB/Project_Velocity/comfy_engine/scripts/dreamweaver_batch_processor.py#394-414) parameter is deprecated in favor of `keywords` (array of strings) and [room_type](file:///F:/Workin%20In%20Progress/DESINEURON/GITLAB/Project_Velocity/comfy_engine/scripts/dw_gateway_v2.py#171-186).
|
||||
|
||||
---
|
||||
|
||||
## 2. Dynamic Keyword Expansion Flow
|
||||
|
||||
Instead of injecting keywords into a rigid template, the new backend reads the `keywords` and [room_type](file:///F:/Workin%20In%20Progress/DESINEURON/GITLAB/Project_Velocity/comfy_engine/scripts/dw_gateway_v2.py#171-186), and asks a local LLM (Qwen 3.5 27B) to act as an interior designer:
|
||||
|
||||
1. **User input:** `keywords: ["blue marble", "gold veins", "renaissance"]`, `room_type: "bathroom"`
|
||||
2. **Backend LLM Expansion:** The LLM knows that a "bathroom" cannot have beds and needs wet-area materials. It creates a rich positive prompt: *"renaissance revival luxury interior design, blue veined marble flooring, gold brass fixtures..."*
|
||||
3. **ComfyUI Generation:** The expanded prompt is sent to ComfyUI for generation.
|
||||
|
||||
**Supported Room Types:**
|
||||
`bedroom`, `living_room`, `bathroom`, `kitchen`, `dining_room`, `home_office`, `hallway`, `balcony`.
|
||||
|
||||
---
|
||||
|
||||
## 3. API Reference — New v2 Endpoints
|
||||
|
||||
### BASE URL
|
||||
```
|
||||
http://54.172.172.2:8082
|
||||
```
|
||||
|
||||
### 3.1 `GET /health` — Liveness Check
|
||||
Call this on app launch to confirm the v2 server is up.
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"comfyui": true,
|
||||
"gpu": "4x NVIDIA L4 (96GB VRAM)",
|
||||
"model": "RealVisXL V5.0 Lightning",
|
||||
"llm_expansion": true,
|
||||
"version": "2.0.0"
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 `GET /room-types`
|
||||
Returns all supported room types and their required design context (useful if you want to build UI tooltips).
|
||||
```json
|
||||
{
|
||||
"room_types": {
|
||||
"bedroom": {
|
||||
"description": "a private sleeping space",
|
||||
"key_elements": ["bed", "bedside tables", "wardrobe", "soft lighting", "textiles", "headboard"]
|
||||
},
|
||||
...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3.3 `POST /dream-weaver/expand` (Preview Prompt)
|
||||
Use this if you want the user to **preview** the LLM's generated prompt before committing to a generation.
|
||||
**Request (JSON):**
|
||||
```json
|
||||
{
|
||||
"keywords": ["blue marble", "gold veins", "renaissance"],
|
||||
"room_type": "bathroom"
|
||||
}
|
||||
```
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"style_name": "Renaissance Luxury",
|
||||
"positive_prompt": "renaissance revival luxury interior design, blue veined marble flooring...",
|
||||
"negative_prompt": "(worst quality, low quality...), extra windows...",
|
||||
"cfg": 7.5,
|
||||
"denoise": 0.72,
|
||||
"steps": 30,
|
||||
"source": "ollama_local"
|
||||
}
|
||||
```
|
||||
|
||||
### 3.4 `POST /dream-weaver` (Submit Generation)
|
||||
Use this for the main generation flow.
|
||||
**Request:** `multipart/form-data`
|
||||
| Field | Type | Required | Description |
|
||||
|---|---|---|---|
|
||||
| [image](file:///F:/Workin%20In%20Progress/DESINEURON/GITLAB/Project_Velocity/comfy_engine/scripts/a100_deployment_executor.py#306-322) | File | ✅ | The room photo (JPEG/PNG) |
|
||||
| `keywords` | String | ✅ | Comma-separated user keywords e.g. `"gold, marble, luxury"` |
|
||||
| [room_type](file:///F:/Workin%20In%20Progress/DESINEURON/GITLAB/Project_Velocity/comfy_engine/scripts/dw_gateway_v2.py#171-186) | String | ✅ | e.g. `"living_room"`, `"bedroom"` |
|
||||
| `additional_notes` | String | ➕ | (Optional) e.g. `"make it feel like a luxury hotel"` |
|
||||
| `denoise` | Float | ➕ | (Optional) 0.5–0.85. If omitted, LLM decides. |
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"job_id": "a1b2c3d4-...",
|
||||
"status": "processing",
|
||||
"prompt_preview": "renaissance revival luxury interior design...",
|
||||
"poll_url": "/dream-weaver/status/a1b2c3d4-...",
|
||||
"result_url": "/dream-weaver/result/a1b2c3d4-..."
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Polling & Downloading (Unchanged from v1)
|
||||
|
||||
**Poll Job Status:**
|
||||
`GET /dream-weaver/status/{job_id}` every 2 seconds until `ready == true`.
|
||||
|
||||
**Download Result:**
|
||||
`GET /dream-weaver/result/{job_id}` returns the raw PNG stream.
|
||||
|
||||
---
|
||||
|
||||
## 5. Updated Swift Example (v2)
|
||||
|
||||
```swift
|
||||
func submitGenerationV2(image: UIImage, roomType: String, keywords: [String]) async throws -> GenerationJob {
|
||||
let url = URL(string: "\(AppConfig.baseURL)/dream-weaver")!
|
||||
var request = URLRequest(url: url)
|
||||
request.httpMethod = "POST"
|
||||
|
||||
let boundary = UUID().uuidString
|
||||
request.setValue("multipart/form-data; boundary=\(boundary)", forHTTPHeaderField: "Content-Type")
|
||||
|
||||
var body = Data()
|
||||
|
||||
// 1. Image
|
||||
let imageData = image.jpegData(compressionQuality: 0.85)!
|
||||
body.appendMultipartForm(boundary: boundary, name: "image", filename: "room.jpg", contentType: "image/jpeg", data: imageData)
|
||||
|
||||
// 2. Room Type
|
||||
body.appendMultipartForm(boundary: boundary, name: "room_type", value: roomType)
|
||||
|
||||
// 3. Keywords
|
||||
let kwString = keywords.joined(separator: ", ")
|
||||
body.appendMultipartForm(boundary: boundary, name: "keywords", value: kwString)
|
||||
|
||||
body.append("--\(boundary)--\r\n".data(using: .utf8)!)
|
||||
request.httpBody = body
|
||||
|
||||
let (data, _) = try await URLSession.shared.data(for: request)
|
||||
return try JSONDecoder().decode(GenerationJob.self, from: data)
|
||||
}
|
||||
|
||||
// Helper extension for building multipart forms cleanly
|
||||
extension Data {
|
||||
mutating func appendMultipartForm(boundary: String, name: String, value: String) {
|
||||
self.append("--\(boundary)\r\n".data(using: .utf8)!)
|
||||
self.append("Content-Disposition: form-data; name=\"\(name)\"\r\n\r\n".data(using: .utf8)!)
|
||||
self.append("\(value)\r\n".data(using: .utf8)!)
|
||||
}
|
||||
|
||||
mutating func appendMultipartForm(boundary: String, name: String, filename: String, contentType: String, data: Data) {
|
||||
self.append("--\(boundary)\r\n".data(using: .utf8)!)
|
||||
self.append("Content-Disposition: form-data; name=\"\(name)\"; filename=\"\(filename)\"\r\n".data(using: .utf8)!)
|
||||
self.append("Content-Type: \(contentType)\r\n\r\n".data(using: .utf8)!)
|
||||
self.append(data)
|
||||
self.append("\r\n".data(using: .utf8)!)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Sayan's Action Checklist (v2)
|
||||
|
||||
- [ ] Change `AppConfig.baseURL` port to `8082` (e.g., `http://54.172.172.2:8082`).
|
||||
- [ ] Add a UI element for the user to select the **Room Type** (`bedroom`, `living_room`, `bathroom`, etc.).
|
||||
- [ ] Change the `POST /dream-weaver` payload from `{style}` to `{keywords, room_type}`.
|
||||
- [ ] (Optional) Use the new `GET /dream-weaver/expand` endpoint to let the user preview and edit the AI-generated prompt before generating.
|
||||
|
||||
72
.Agent Context/Bibels/infrastructure_deployment_manifest.md
Normal file
72
.Agent Context/Bibels/infrastructure_deployment_manifest.md
Normal file
@@ -0,0 +1,72 @@
|
||||
# Dream Weaver — Infrastructure & Connectivity Manifest
|
||||
|
||||
**Environment:** Production AWS Node
|
||||
**Last Verified:** 2026-03-14
|
||||
**Status:** ✅ HEALTHY
|
||||
|
||||
---
|
||||
|
||||
## 🖥️ Server Instance
|
||||
> [!IMPORTANT]
|
||||
> **Active Public IP: `54.91.19.60`**
|
||||
> The previous Elastic IP (`54.172.172.2`) is currently detached and will time out. Ensure all your connection strings use the active IP.
|
||||
|
||||
| Component | Value |
|
||||
|---|---|
|
||||
| **Instance ID** | `i-0e4eab5fe67cf9abe` |
|
||||
| **Instance Name** | Desineuron AWS Node 4x L4 (96GB VRAM) Spot |
|
||||
| **Public IP (Active)** | **`54.91.19.60`** |
|
||||
| **Private IP** | `172.31.46.190` |
|
||||
| **VPC ID** | `vpc-081d2397920aad268` |
|
||||
|
||||
---
|
||||
|
||||
## 🛡️ Security Group Settings
|
||||
**Security Group ID:** `sg-0b144c17b1b89f4c6` (Synapse-Ops)
|
||||
The following Inbound rules are explicitly confirmed and open:
|
||||
|
||||
| Protocol | Port | Source | Description |
|
||||
|---|---|---|---|
|
||||
| TCP | **22** | `0.0.0.0/0` | SSH Access |
|
||||
| TCP | **8082** | `0.0.0.0/0` | **Dream Weaver API v2 (Current)** |
|
||||
| TCP | **8188** | `0.0.0.0/0` | ComfyUI Internal API |
|
||||
| TCP | **8000** | `0.0.0.0/0` | ComfyUI Web UI (Alternate) |
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Services & Endpoints
|
||||
|
||||
### 1. Dream Weaver Gateway v2
|
||||
* **Port:** `8082`
|
||||
* **Status:** ✅ Active
|
||||
* **Health Check:** `http://54.91.19.60:8082/health`
|
||||
* **Main Endpoint:** `POST http://54.91.19.60:8082/dream-weaver`
|
||||
|
||||
### 2. ComfyUI Engine
|
||||
* **Port:** `8188`
|
||||
* **Status:** ✅ Active
|
||||
* **Prompt Endpoint:** `POST http://54.91.19.60:8188/prompt`
|
||||
|
||||
---
|
||||
|
||||
## 🔑 SSH Configuration
|
||||
|
||||
**Local Key File Path:** `f:\Workin In Progress\DESINEURON\GITLAB\Project_Velocity\desineuron-l4-node.pem`
|
||||
|
||||
### Quick Connect Command
|
||||
```bash
|
||||
ssh -i "path/to/desineuron-l4-node.pem" ubuntu@54.91.19.60
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Operator Checklist (Troubleshooting)
|
||||
|
||||
1. **Verify API Process:**
|
||||
`ps aux | grep dw_gateway_v2`
|
||||
2. **Check Logs:**
|
||||
`tail -f /home/ubuntu/gw_v2.log`
|
||||
3. **Check Port Listeners:**
|
||||
`sudo netstat -tulpn | grep 8082`
|
||||
4. **No Zombie Processes:**
|
||||
Port 8080 has been cleared. Only 8082 is serving the gateway.
|
||||
Reference in New Issue
Block a user