sayan/Project_Velocity
1
1
Fork 0
forked from sagnik/Project_Velocity
Code Issues Pull Requests Activity
Files
b87f1f6f5a64d140c9245a03f9b4437e160f7c93
Project_Velocity/.Agent Context/Bibels/dreamweaver_ios_integration_guide.md

20 KiB
Raw Blame History

Dream Weaver — iOS ↔ Backend Integration Guide

Version: 2.0 | Updated: 2026-03-09 | Server: 54.172.172.2 | Port: 8080

This document is for Sayan (iOS / Swift) and Sourik (backend review).
It describes exactly how the iPad app should talk to the Dream Weaver AI and how keywords from a user tap become a full ComfyUI generation.

1. Architecture Overview

┌────────────────────┐       HTTP/S        ┌──────────────────────────────┐
│                    │ ─── POST image ───► │  Dream Weaver Gateway        │
│  iPad App (Swift)  │                     │  FastAPI  port 8080          │
│                    │ ◄── PNG result ───  │  dw_gateway.py               │
└────────────────────┘                     └─────────────┬────────────────┘
                                                         │ internal HTTP
                                                         ▼
                                            ┌─────────────────────────┐
                                            │  ComfyUI Engine         │
                                            │  port 8188              │
                                            │  RealVisXL V5.0 Ltng    │
                                            │  4× NVIDIA L4 (96 GB)   │
                                            └─────────────────────────┘

Key rule: The iPad app never talks to ComfyUI directly. It only talks to the Gateway on :8080.

2. How Keywords Become Prompts

2.1 The Prompt Expansion System

Each interior style in the app is backed by a prompt template in comfy_engine/prompts/. When the user taps a style card (or types keywords), those keywords get merged into the template to build the final ComfyUI prompt injected into node 9 (positive CLIPTextEncode).

Prompt template structure (from scandinavian_minimalist.txt):

POSITIVE PROMPT:
scandinavian minimalist interior design, light oak wood flooring, neutral beige textiles,
abundant natural light streaming through large windows, clean white walls, ...
Style Weight: <lora:Interior_Style_Scandi:0.8>

NEGATIVE PROMPT:
(worst quality, low quality, illustration, 3d render...), heavy ornamentation,...

TECHNICAL PARAMETERS:
- Denoising Strength: 0.70
- CFG Scale: 7.0
- Recommended Sampler: dpmpp_2m_karras
- Steps: 30-40

2.2 Keyword Expansion Flow

User taps: ["marble", "gold", "luxury"]
           +
Style selected: "art_deco"
       │
       ▼
Backend expands:
  base_prompt = art_deco_luxe.txt (POSITIVE PROMPT section)
  user_keywords_str = "marble, gold, luxury"
  final_prompt = base_prompt + ", " + user_keywords_str
       │
       ▼
Injected into ComfyUI workflow:
  node "9" → CLIPTextEncode → text: [final_prompt]
  node "10" → CLIPTextEncode → text: [negative_prompt from template]
  node "1"  → LoadImage      → image: [uploaded filename]
  node "13" → KSampler       →  denoise: 0.72, cfg: 7.5, steps: 35

2.3 Available Styles and Their Keywords (for the Style Picker UI)

Style ID Display Name Suggested Keywords Palette
scandinavian Scandinavian Minimalist oak, linen, white, hygge, cozy, birch, natural
art_deco Art Deco Luxe gold, marble, velvet, geometric, 1920s, brass, crystal
biophilic Biophilic Organic green wall, stone, rattan, terracotta, botanical, moss
cyberpunk Cyberpunk Neon neon, chrome, holographic, dark, LED, futuristic, blade runner
japandi Japandi Fusion wabi-sabi, ash wood, ceramic, zen, minimal, shoji, serene

3. API Reference — What Sayan Needs to Call

BASE URL

http://54.172.172.2:8080

Note

Once we attach an Elastic IP or domain, swap this in AppConfig.swift.

3.1 GET /health — Liveness Check

Call this on app launch to confirm the server is up before showing the Generate button.

Request:

GET http://54.172.172.2:8080/health

Response:

{
  "status": "ok",
  "comfyui": true,
  "gpu": "4x NVIDIA L4 (96GB VRAM)",
  "model": "RealVisXL V5.0 Lightning"
}

Swift:

func checkServerHealth() async throws -> Bool {
    let url = URL(string: "\(AppConfig.baseURL)/health")!
    let (data, _) = try await URLSession.shared.data(from: url)
    let json = try JSONDecoder().decode(HealthResponse.self, from: data)
    return json.status == "ok"
}

3.2 POST /dream-weaver — Submit Generation Job (Async)

Use this for the main generation flow. Returns a job_id immediately; poll for result.

Request: multipart/form-data

Field Type Required Description
image File (JPEG/PNG) ✅ The room photo from camera or library
style String ✅ One of: scandinavian, art_deco, biophilic, cyberpunk, japandi
keywords String ➕ Comma-separated user keywords e.g. "gold, marble, luxury"
denoise Float ➕ 0.5–0.85 (default 0.72). Higher = more creative

Response:

{
  "job_id": "a1b2c3d4-...",
  "status": "processing",
  "poll_url": "/dream-weaver/status/a1b2c3d4-...",
  "result_url": "/dream-weaver/result/a1b2c3d4-..."
}

Swift example:

func submitGeneration(image: UIImage, style: String, keywords: [String]) async throws -> GenerationJob {
    let url = URL(string: "\(AppConfig.baseURL)/dream-weaver")!
    var request = URLRequest(url: url)
    request.httpMethod = "POST"

    let boundary = UUID().uuidString
    request.setValue("multipart/form-data; boundary=\(boundary)", forHTTPHeaderField: "Content-Type")

    var body = Data()
    // Image field
    let imageData = image.jpegData(compressionQuality: 0.85)!
    body.append("--\(boundary)\r\n".data(using: .utf8)!)
    body.append("Content-Disposition: form-data; name=\"image\"; filename=\"room.jpg\"\r\n".data(using: .utf8)!)
    body.append("Content-Type: image/jpeg\r\n\r\n".data(using: .utf8)!)
    body.append(imageData)
    body.append("\r\n".data(using: .utf8)!)

    // Style field
    body.append("--\(boundary)\r\n".data(using: .utf8)!)
    body.append("Content-Disposition: form-data; name=\"style\"\r\n\r\n".data(using: .utf8)!)
    body.append("\(style)\r\n".data(using: .utf8)!)

    // Keywords field (user tapped keywords)
    if !keywords.isEmpty {
        let kwString = keywords.joined(separator: ", ")
        body.append("--\(boundary)\r\n".data(using: .utf8)!)
        body.append("Content-Disposition: form-data; name=\"keywords\"\r\n\r\n".data(using: .utf8)!)
        body.append("\(kwString)\r\n".data(using: .utf8)!)
    }

    body.append("--\(boundary)--\r\n".data(using: .utf8)!)
    request.httpBody = body

    let (data, _) = try await URLSession.shared.data(for: request)
    return try JSONDecoder().decode(GenerationJob.self, from: data)
}

3.3 GET /dream-weaver/status/{job_id} — Poll Job Status

Poll every 2 seconds until ready == true.

Response while processing:

{ "status": "processing", "ready": false, "style": "art_deco" }

Response when done:

{
  "status": "done",
  "ready": true,
  "result_url": "/dream-weaver/result/a1b2c3d4-...",
  "style": "art_deco"
}

Swift polling loop:

func pollForResult(jobId: String) async throws -> URL {
    let statusURL = URL(string: "\(AppConfig.baseURL)/dream-weaver/status/\(jobId)")!
    
    for _ in 0..<150 {  // max 5 min (150 × 2s)
        try await Task.sleep(nanoseconds: 2_000_000_000)  // 2 seconds
        let (data, _) = try await URLSession.shared.data(from: statusURL)
        let status = try JSONDecoder().decode(JobStatus.self, from: data)
        
        if status.ready {
            return URL(string: "\(AppConfig.baseURL)/dream-weaver/result/\(jobId)")!
        }
        if status.status == "error" {
            throw DreamWeaverError.generationFailed(status.error ?? "Unknown")
        }
    }
    throw DreamWeaverError.timeout
}

3.4 GET /dream-weaver/result/{job_id} — Download Result Image

Returns a PNG image stream directly. Download and display to user.

func downloadResult(resultURL: URL) async throws -> UIImage {
    let (data, _) = try await URLSession.shared.data(from: resultURL)
    guard let image = UIImage(data: data) else {
        throw DreamWeaverError.invalidImageData
    }
    return image
}

3.5 POST /dream-weaver/sync — One-Shot Blocking Call

For testing only or fast network connections. Waits up to 120 seconds and returns the image directly.

// Same multipart form as /dream-weaver but returns PNG bytes directly
// Not recommended for production — use async flow above

4. Keyword → Prompt Expansion (Backend Change Required)

Important

The current dw_gateway.py does NOT yet accept keywords from the app.
Sagnik needs to add keyword expansion to the gateway. Here is the exact code change:

In dw_gateway.py, update build_workflow():

# Prompt library — maps style ID to (positive_base, negative, cfg, denoise, steps)
STYLE_LIBRARY = {
    "scandinavian": {
        "pos": "scandinavian minimalist interior design, light oak wood flooring, neutral beige textiles, abundant natural light, clean white walls, simple functional furniture, cozy hygge atmosphere, architectural photography, 8k resolution, photorealistic",
        "neg": "(worst quality, low quality, illustration, 3d render, painting, cartoon, sketch), blurry, distorted, extra windows, unrealistic lighting, structural changes",
        "cfg": 7.0, "denoise": 0.70, "steps": 30,
    },
    "art_deco": {
        "pos": "art deco luxury interior design, geometric chevron patterns, gold brass accents, rich velvet upholstery emerald and sapphire, sunburst mirrors, polished marble flooring, crystal chandeliers, 1920s glamour, 8k resolution, photorealistic",
        "neg": "(worst quality, low quality, illustration, 3d render, painting, cartoon, sketch), blurry, distorted, structural changes, rustic, minimalism, cheap materials",
        "cfg": 7.5, "denoise": 0.72, "steps": 30,
    },
    "biophilic": {
        "pos": "biophilic organic interior design, living green walls with ferns and moss, natural stone accent walls, rattan and bamboo furniture, abundant houseplants, earth tone sage green and terracotta, 8k resolution, photorealistic, dappled sunlight",
        "neg": "(worst quality, low quality, illustration, 3d render, painting, cartoon, sketch), blurry, distorted, structural changes, synthetic materials, plastic plants",
        "cfg": 7.0, "denoise": 0.68, "steps": 30,
    },
    "cyberpunk": {
        "pos": "cyberpunk neon interior design, high contrast LED strip lighting electric blue and hot pink, reflective chrome surfaces, dark matte walls, futuristic furniture, glowing circuit patterns, tech-noir blade runner aesthetic, 8k resolution, photorealistic, volumetric fog",
        "neg": "(worst quality, low quality, illustration, 3d render, painting, cartoon, sketch), blurry, distorted, structural changes, natural daylight, rustic elements",
        "cfg": 8.0, "denoise": 0.75, "steps": 30,
    },
    "japandi": {
        "pos": "japandi fusion interior design, wabi-sabi textures, low-profile furniture, muted earth tones warm grays soft browns, handmade ceramic accents, light ash wood, shoji screen elements, minimal decoration, zen garden elements, 8k resolution, photorealistic, serene",
        "neg": "(worst quality, low quality, illustration, 3d render, painting, cartoon, sketch), blurry, distorted, structural changes, bright colors, ornate decoration, cluttered",
        "cfg": 6.5, "denoise": 0.70, "steps": 30,
    },
}

def build_workflow(img_name: str, style: str = "scandinavian",
                   keywords: str = "", denoise_override: float = None) -> dict:
    s = STYLE_LIBRARY.get(style, STYLE_LIBRARY["scandinavian"])
    
    # Merge user keywords into base positive prompt
    pos = s["pos"]
    if keywords.strip():
        pos = pos + ", " + keywords.strip()
    
    cfg     = s["cfg"]
    denoise = denoise_override if denoise_override else s["denoise"]
    steps   = s["steps"]
    neg     = s["neg"]
    
    return {
        "1": {"class_type": "CheckpointLoaderSimple",
              "inputs": {"ckpt_name": "realvisxlV50_v50LightningBakedvae.safetensors"}},
        "2": {"class_type": "LoadImage",
              "inputs": {"image": img_name, "upload": "image"}},
        "3": {"class_type": "CLIPTextEncode",
              "inputs": {"text": pos, "clip": ["1", 1]}},    # ← POSITIVE PROMPT
        "4": {"class_type": "CLIPTextEncode",
              "inputs": {"text": neg, "clip": ["1", 1]}},    # ← NEGATIVE PROMPT
        "5": {"class_type": "VAEEncode",
              "inputs": {"pixels": ["2", 0], "vae": ["1", 2]}},
        "6": {"class_type": "KSampler",
              "inputs": {"model": ["1", 0], "positive": ["3", 0], "negative": ["4", 0],
                         "latent_image": ["5", 0],
                         "seed": int(time.time()) % 999983,
                         "steps": steps, "cfg": cfg,
                         "sampler_name": "dpmpp_2m", "scheduler": "karras",
                         "denoise": denoise}},
        "7": {"class_type": "VAEDecode",
              "inputs": {"samples": ["6", 0], "vae": ["1", 2]}},
        "8": {"class_type": "SaveImage",
              "inputs": {"images": ["7", 0], "filename_prefix": f"dreamweaver_{style}"}}
    }

Also update the endpoint signatures to accept keywords:

@app.post("/dream-weaver")
async def dream_weaver(
    image: UploadFile = File(...),
    style: str = Form(default="scandinavian"),
    keywords: str = Form(default=""),        # ← ADD THIS
    denoise: float = Form(default=0.0),      # 0.0 = use style default
):
    ...
    wf = build_workflow(comfy_name, style=style, keywords=keywords,
                        denoise_override=denoise if denoise > 0 else None)

5. Swift Data Models

// AppConfig.swift
struct AppConfig {
    static let baseURL = "http://54.172.172.2:8080"
    // Change this to HTTPS domain once SSL is set up
}

// Models
struct GenerationJob: Codable {
    let jobId: String
    let status: String
    let pollUrl: String
    let resultUrl: String
    
    enum CodingKeys: String, CodingKey {
        case jobId = "job_id"
        case status, pollUrl = "poll_url", resultUrl = "result_url"
    }
}

struct JobStatus: Codable {
    let status: String
    let ready: Bool
    let resultUrl: String?
    let error: String?
    
    enum CodingKeys: String, CodingKey {
        case status, ready, resultUrl = "result_url", error
    }
}

struct HealthResponse: Codable {
    let status: String
    let comfyui: Bool
}

enum DreamWeaverError: Error {
    case generationFailed(String)
    case timeout
    case invalidImageData
}

// Style model for the style picker
struct InteriorStyle: Identifiable {
    let id: String       // used as the `style` form field value
    let displayName: String
    let keywords: [String]   // shown as tappable chips in UI
    let previewImage: String // local asset name
}

let availableStyles: [InteriorStyle] = [
    InteriorStyle(id: "scandinavian",  displayName: "Scandinavian",  keywords: ["oak","linen","white","hygge","cozy","birch"], previewImage: "style_scandi"),
    InteriorStyle(id: "art_deco",      displayName: "Art Deco Luxe",  keywords: ["gold","marble","velvet","geometric","brass","crystal"], previewImage: "style_artdeco"),
    InteriorStyle(id: "biophilic",     displayName: "Biophilic",      keywords: ["green wall","stone","rattan","terracotta","botanical"], previewImage: "style_biophilic"),
    InteriorStyle(id: "cyberpunk",     displayName: "Cyberpunk Neon", keywords: ["neon","chrome","LED","futuristic","dark","holographic"], previewImage: "style_cyberpunk"),
    InteriorStyle(id: "japandi",       displayName: "Japandi Fusion",  keywords: ["wabi-sabi","ceramic","ash wood","zen","minimal"], previewImage: "style_japandi"),
]

6. Complete Generation Flow (UI → Server → Result)

1. User opens camera / library → picks a room photo
2. User selects a style card  → style ID captured
3. User optionally taps keyword chips → keywords[] array built
4. User taps "Generate" →

   POST /dream-weaver (multipart)
     image: <jpeg data>
     style: "art_deco"
     keywords: "gold, marble, luxury hotel"
     denoise: 0.72

5. Server returns { job_id: "abc123", status: "processing" }
6. App shows loading/progress UI
7. App polls GET /dream-weaver/status/abc123  every 2 seconds
8. When ready == true  →
   GET /dream-weaver/result/abc123  →  returns PNG bytes
9. App displays result full-screen with save/share options

Expected latency on 4× L4 GPU server: ~15–20 seconds end-to-end.

7. WebSocket Progress (Optional Advanced Feature)

If Sayan wants a real-time progress bar (e.g. "Step 12/30"), connect directly to ComfyUI's WebSocket only if port 8188 is opened. Otherwise, polling /status is sufficient.

// WebSocket — only if 8188 is exposed externally
class ComfyProgressWebSocket: NSObject, URLSessionWebSocketDelegate {
    var onProgress: ((Int, Int) -> Void)?
    var task: URLSessionWebSocketTask?
    
    func connect(clientId: String) {
        let url = URL(string: "ws://54.172.172.2:8188/ws?clientId=\(clientId)")!
        task = URLSession.shared.webSocketTask(with: url)
        task?.resume()
        listen()
    }
    
    private func listen() {
        task?.receive { [weak self] result in
            if case .success(let message) = result,
               case .string(let text) = message,
               let data = text.data(using: .utf8),
               let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
               let type = json["type"] as? String, type == "progress",
               let inner = json["data"] as? [String: Int] {
                self?.onProgress?(inner["value"] ?? 0, inner["max"] ?? 30)
            }
            self?.listen()  // recurse
        }
    }
}

Note

Port 8188 is currently not open externally in the security group. Only port 8080 is. To use WebSocket progress, Sagnik needs to add an inbound rule for 8188. Until then, using /status polling every 2s gives good enough UX.

8. Error Handling

HTTP Status Meaning UI Action
200 Success Show result or job_id
404 on /status Job expired (> 30 min) "Session expired. Please retry."
500 Generation failed (OOM, model error) "Generation failed. Try a simpler image."
Connection error Server down or no internet "Checking server…" + retry logic

The job status field can also be "error" with an error field explaining what failed.

9. Quick Checklist for Sayan

  • Update AppConfig.swift with baseURL = "http://54.172.172.2:8080"
  • Implement POST /dream-weaver multipart with image + style + keywords
  • Implement polling loop on GET /dream-weaver/status/{job_id}
  • Implement image download from GET /dream-weaver/result/{job_id}
  • Add GET /health check on app launch
  • Build keyword chips UI with the 5 style palettes from Section 2.3
  • Test with the 20 sample images in comfy_engine/test_inputs/

10. Quick Checklist for Sagnik (backend)

  • Update dw_gateway.py with the full STYLE_LIBRARY dict (Section 4)
  • Add keywords: str = Form(default="") to both POST endpoints
  • Pass keywords into build_workflow() for prompt expansion
  • Redeploy gateway on port 8080 (nohup python3 dw_gateway.py &)
  • (Optional) Open port 8188 in security group for WebSocket progress
Reference in New Issue View Git Blame Copy Permalink