Initial commit: Velocity-OS migration

This commit is contained in:
2026-05-01 12:32:19 +05:30
commit 407af828d4
283 changed files with 207782 additions and 0 deletions

42
media-engine/Dockerfile Normal file
View File

@@ -0,0 +1,42 @@
# syntax=docker/dockerfile:1.4
# ============================================================
# Velocity-OS — media-engine (Dream Weaver Gateway)
# Async ComfyUI job gateway. Models mounted from NVMe PV.
# ============================================================
FROM python:3.11-slim AS runtime
LABEL org.opencontainers.image.title="velocity-os-media-engine" \
org.opencontainers.image.description="Velocity-OS Dream Weaver / ComfyUI Gateway" \
org.opencontainers.image.vendor="Desineuron" \
org.opencontainers.image.version="2.0.0"
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
&& rm -rf /var/lib/apt/lists/*
RUN groupadd -r velocity && useradd -r -g velocity velocity
WORKDIR /app
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --no-cache-dir -r requirements.txt
COPY --chown=velocity:velocity . .
ENV PYTHONUNBUFFERED=1 \
PYTHONPATH=/app \
# Overridden by K3s ConfigMap/env
COMFYUI_URLS="http://localhost:8188" \
DW_ASSET_DIR="/opt/assets/generated" \
MODEL_BASE="/opt/models/comfy"
USER velocity
EXPOSE 8290
HEALTHCHECK --interval=60s --timeout=10s --start-period=30s \
CMD curl -f http://localhost:8290/health || exit 1
CMD ["python", "gateway.py", "--host", "0.0.0.0", "--port", "8290"]

View File

@@ -0,0 +1,840 @@
# Dream Weaver Technical Specification
**Version:** 1.0.0
**Date:** 2026-03-01
**Model:** RealVisXL V5.0 Lightning
**Target Hardware Phase 1:** NVIDIA RTX 3080Ti (12GB GDDR6X)
**Target Hardware Phase 3:** Dual NVIDIA RTX PRO 6000 Blackwell (96GB GDDR7 each)
---
## Table of Contents
1. [Executive Summary](#executive-summary)
2. [Three-Phase Implementation Architecture](#three-phase-implementation-architecture)
3. [Hardware Specifications & Optimization](#hardware-specifications--optimization)
4. [Model Specifications & Downloads](#model-specifications--downloads)
5. [ControlNet Configuration](#controlnet-configuration)
6. [Custom Node Requirements](#custom-node-requirements)
7. [Phase 1: Foundational Implementation](#phase-1-foundational-implementation)
8. [Phase 2: Advanced Multi-ControlNet](#phase-2-advanced-multi-controlnet)
9. [Phase 3: Production Batch Processing](#phase-3-production-batch-processing)
10. [Prompt Engineering Templates](#prompt-engineering-templates)
11. [API Integration Guide](#api-integration-guide)
12. [Deployment Instructions](#deployment-instructions)
---
## Executive Summary
Dream Weaver is an interior restyling workflow that uses **Structural Constraint Logic** to preserve existing room geometry while enabling comprehensive aesthetic transformations. The system employs a **Dual-ControlNet Strategy** combining M-LSD (Line Segment Detection) for architectural line preservation and Depth (Zoe/MiDaS) for 3D spatial consistency, with SAM-based masking to isolate structural immutables from stylable regions.
### Core Constraint: Absolute Geometry Preservation
The following elements are **IMMUTABLE** and must never be modified:
- Wall positions and angles
- Door and window placements
- Ceiling heights
- Room proportions and dimensions
- Structural load-bearing elements
- Vanishing points and perspective
The following elements are **MUTABLE** and may be restyled:
- Wall paint colors and textures
- Flooring materials
- Furniture upholstery and styles
- Decorative objects and accessories
- Lighting fixtures and atmospheres
- Soft furnishings (curtains, rugs, cushions)
---
## Three-Phase Implementation Architecture
```mermaid
flowchart TD
A[Input Interior Image] --> B[Phase 1: Foundational]
B --> C[Phase 2: Advanced]
C --> D[Phase 3: Production]
subgraph P1[Phase 1 - RTX 3080Ti]
B1[Depth ControlNet] --> B2[Basic SAM Masking]
B2 --> B3[Single Image Processing]
end
subgraph P2[Phase 2 - Enhanced Quality]
C1[Multi-ControlNet] --> C2[Refined Masking]
C2 --> C3[Style Templates]
end
subgraph P3[Phase 3 - Dual RTX PRO 6000]
D1[Batch Processing] --> D2[4K Upscaling]
D2 --> D3[Automated Pipeline]
end
```
### Phase Overview
| Phase | Hardware | ControlNets | Resolution | Batch Size | Purpose |
|-------|----------|-------------|------------|------------|---------|
| 1 | RTX 3080Ti | 1 (Depth) | 1024x1024 | 1 | Validation & Testing |
| 2 | RTX 3080Ti | 3 (Depth + Seg + Canny) | 1216x832 | 1 | Quality Enhancement |
| 3 | Dual RTX PRO 6000 | 3 + Aux | 2048x2048 | 8+ | Production Deployment |
---
## Hardware Specifications & Optimization
### Current Development Hardware: RTX 3080Ti
**Specifications:**
- GPU: NVIDIA RTX 3080Ti
- VRAM: 12GB GDDR6X
- CUDA Cores: 10,240
- Architecture: Ampere
**VRAM Management Strategy:**
```python
# Optimization flags for 12GB VRAM
--fp16 # Enable half-precision
--lowvram # Aggressive memory management
--disable-xformers # Use sdp-attention instead
```
**Recommended Settings:**
- Batch size: 1
- Maximum resolution: 1024x1024 or 1216x832
- Tiled VAE: Enabled with tile size 64
- Model CPU offloading: Enabled
- Empty cache after each generation: Enabled
### Production Hardware: Dual RTX PRO 6000 Blackwell
**Specifications:**
- GPU: 2x NVIDIA RTX PRO 6000 Blackwell
- VRAM: 96GB GDDR7 per GPU (192GB total)
- Architecture: Blackwell
- NVLink: Enabled for memory pooling
**Optimization Strategy:**
```python
# Production flags for 192GB VRAM
--bf16 # Enable bfloat16 for better precision
--highvram # Keep models in GPU memory
--xformers # Enable memory-efficient attention
--gpu-batch-size 8 # Process 8 images simultaneously
--model-sharding # Distribute across both GPUs
```
### VRAM Usage Comparison
| Configuration | Phase 1 | Phase 2 | Phase 3 |
|--------------|---------|---------|---------|
| Model Loading | 6.2GB | 6.2GB | 6.2GB |
| ControlNet 1 | 1.8GB | 1.8GB | 1.8GB |
| ControlNet 2 | - | 1.8GB | 1.8GB |
| ControlNet 3 | - | 1.5GB | 1.5GB |
| SAM Model | 2.1GB | 2.1GB | 2.1GB |
| Latent Buffers | 1.5GB | 2.2GB | 8.0GB |
| **Total** | **~11.6GB** | **~15.6GB** | **~21.4GB** |
---
## Model Specifications & Downloads
### Primary Checkpoint: RealVisXL V5.0 Lightning
**Download URL:** https://civitai.com/models/139562?modelVersionId=789646
**Specifications:**
- Base Model: SDXL
- Training Data: Architectural photography datasets
- Specialization: Photorealistic interiors, white balance accuracy
- Lightning Steps: 4-8 steps for high quality
- Recommended CFG: 1.0-2.0 (Lightning)
- CLIP Skip: 2
**File Details:**
- Filename: `realvisxlV50Lightning_v50Lightning.safetensors`
- Expected Size: ~6.5GB
- Format: SafeTensors
- SHA256: Verify on download
**Installation Path:**
```
ComfyUI/models/checkpoints/realvisxlV50Lightning_v50Lightning.safetensors
```
### VAE Selection
**Option A: Automatic1111 VAE**
- Download: https://huggingface.co/stabilityai/sdxl-vae
- File: `sdxl_vae.safetensors`
- Size: ~335MB
- Path: `ComfyUI/models/vae/sdxl_vae.safetensors`
**Option B: RealVisXL Native VAE**
- Built into checkpoint (recommended for simplicity)
**Recommendation:** Use checkpoint's built-in VAE for Phase 1-2, Automatic1111 VAE for Phase 3 production
---
## ControlNet Configuration
### ControlNet Model Specifications
| Model | Purpose | Strength | Download URL | File Size |
|-------|---------|----------|--------------|-----------|
| control_v11f1p_sd15_depth | Geometric preservation | 1.0 | https://huggingface.co/lllyasviel/ControlNet-v1-1 | ~1.2GB |
| control_v11p_sd15_seg | Semantic segmentation | 0.85 | https://huggingface.co/lllyasviel/ControlNet-v1-1 | ~1.2GB |
| control_v11p_sd15_canny | Edge detection | 0.6 | https://huggingface.co/lllyasviel/ControlNet-v1-1 | ~1.2GB |
| control_v11p_sd15_mlsd | Line segment detection | 0.8 | https://huggingface.co/lllyasviel/ControlNet-v1-1 | ~1.2GB |
**Installation Path:**
```
ComfyUI/models/controlnet/
```
### Preprocessor Selection
| Preprocessor | Purpose | Phase | Node Name |
|--------------|---------|-------|-----------|
| depth_midas | General depth estimation | 1 | ControlNet Preprocessor/Depth MiDaS |
| depth_zoe | High-quality depth (preferred) | 2+ | ControlNet Preprocessor/Depth Zoe |
| seg_of_ade20k | Semantic segmentation | 2 | ControlNet Preprocessor/Segmentation OFADE20K |
| seg_uformer | Alternative segmentation | 2 | ControlNet Preprocessor/Segmentation UFormer |
| canny | Edge detection | 2+ | ControlNet Preprocessor/Canny |
| mlsd | Line detection | All | ControlNet Preprocessor/MLSD |
---
## Custom Node Requirements
### Required Node Packages
```bash
# Install via ComfyUI Manager or git clone
# 1. ComfyUI ControlNet Auxiliary Preprocessors
git clone https://github.com/Fannovel16/comfyui_controlnet_aux.git
# 2. ComfyUI Impact Pack (for SAM and segmentation)
git clone https://github.com/ltdrdata/ComfyUI-Impact-Pack.git
# 3. ComfyUI-Manager (if not already installed)
git clone https://github.com/ltdrdata/ComfyUI-Manager.git
# 4. WAS Node Suite (for image processing utilities)
git clone https://github.com/WASasquatch/was-node-suite-comfyui.git
# 5. ComfyUI-Advanced-ControlNet
git clone https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet.git
# 6. Segment Anything for ComfyUI
git clone https://github.com/storyicon/comfyui_segment_anything.git
# 7. ComfyUI_IPAdapter_plus (for style reference)
git clone https://github.com/cubiq/ComfyUI_IPAdapter_plus.git
```
### Node Installation Commands
```bash
cd Project_Velocity/comfy_engine/custom_nodes
# Install each package
for repo in \
"https://github.com/Fannovel16/comfyui_controlnet_aux" \
"https://github.com/ltdrdata/ComfyUI-Impact-Pack" \
"https://github.com/WASasquatch/was-node-suite-comfyui" \
"https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet" \
"https://github.com/storyicon/comfyui_segment_anything" \
"https://github.com/cubiq/ComfyUI_IPAdapter_plus"
do
git clone "$repo"
done
# Install dependencies for each
find . -name requirements.txt -exec pip install -r {} \;
```
### Required Model Downloads for SAM
| Model | Purpose | Download URL | Path |
|-------|---------|--------------|------|
| sam_vit_h_4b8939.pth | High-quality segmentation | https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth | ComfyUI/models/sams/ |
| sam_vit_l_0b3195.pth | Balanced quality/speed | https://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pth | ComfyUI/models/sams/ |
| sam_vit_b_01ec64.pth | Fast inference | https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth | ComfyUI/models/sams/ |
**Recommendation:** Use `sam_vit_l_0b3195.pth` for Phase 1-2, `sam_vit_h_4b8939.pth` for Phase 3
### GroundingDINO Model
| Model | Download URL | Path |
|-------|--------------|------|
| groundingdino_swint_ogc.pth | https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth | ComfyUI/models/grounding-dino/ |
---
## Phase 1: Foundational Implementation
### Purpose
Establish foundational single-ControlNet depth mapping with basic binary segmentation masking. Optimized for RTX 3080Ti 12GB VRAM constraints.
### Node Graph Architecture
```mermaid
flowchart LR
A[Load Image] --> B[Image Scale]
B --> C[Zoe Depth Preprocessor]
B --> D[SAM Masking]
C --> E[ControlNet Apply]
D --> F[Set Latent Noise Mask]
E --> G[KSampler]
F --> G
G --> H[VAE Decode]
H --> I[Save Image]
```
### Key Nodes Configuration
#### 1. Load Image
- Node: `LoadImage`
- Input: User-provided interior photograph
- Output: IMAGE, MASK
#### 2. Image Scale
- Node: `ImageScale`
- Method: `lanczos`
- Width: 1024
- Height: 1024
- Keep Proportion: True
- Upscale Model: None (use interpolation)
#### 3. Zoe Depth Preprocessor
- Node: `Zoe-DepthMapPreprocessor` (from comfyui_controlnet_aux)
- Resolution: 1024
- Output: depth map IMAGE
#### 4. SAM Masking
- Node: `SAMDetectorSegmented` (from comfyui_segment_anything)
- Model: sam_vit_l_0b3195.pth
- Prompt: "walls, floor, ceiling"
- Threshold: 0.3
- Output: SEGMENTATION masks
#### 5. Mask to Image
- Node: `MaskToImage`
- Converts SAM mask to image format
#### 6. ControlNet Apply
- Node: `ControlNetApply`
- ControlNet: control_v11f1p_sd15_depth
- Strength: 1.0
- Start Percent: 0.0
- End Percent: 1.0
#### 7. Checkpoint Loader
- Node: `CheckpointLoaderSimple`
- Checkpoint: realvisxlV50Lightning_v50Lightning.safetensors
#### 8. CLIP Text Encode (Positive)
- Node: `CLIPTextEncode`
- Text: Style-specific prompt
- CLIP: From checkpoint loader
#### 9. CLIP Text Encode (Negative)
- Node: `CLIPTextEncode`
- Text: `(worst quality, low quality, illustration, 3d, 2d, painting, cartoons, sketch), blurry, distorted, deformed, extra windows, unrealistic lighting, structural changes, wall repositioning`
#### 10. Empty Latent Image
- Node: `EmptyLatentImage`
- Width: 1024
- Height: 1024
- Batch Size: 1
#### 11. Set Latent Noise Mask
- Node: `SetLatentNoiseMask`
- Mask: From SAM processing
#### 12. KSampler
- Node: `KSampler`
- Seed: RANDOM
- Control After Generate: fixed
- Steps: 30
- CFG: 7.0
- Sampler: dpmpp_2m
- Scheduler: karras
- Denoise: 0.75
#### 13. VAE Decode
- Node: `VAEDecode`
- VAE: From checkpoint loader or sdxl_vae
#### 14. Save Image
- Node: `SaveImage`
- Filename: `dreamweaver_phase1_$$INDEX$$`
### Phase 1 Workflow JSON
See: `workflows/dreamweaver_phase1_depth.json`
---
## Phase 2: Advanced Multi-ControlNet
### Purpose
Enhance geometric fidelity through triple-ControlNet integration and refined masking workflows with edge bleeding prevention.
### ControlNet Stack Configuration
| ControlNet | Model | Strength | Start | End | Purpose |
|------------|-------|----------|-------|-----|---------|
| 1 | M-LSD | 0.8 | 0.0 | 0.5 | Structural lines |
| 2 | Depth (Zoe) | 1.0 | 0.0 | 1.0 | 3D geometry |
| 3 | Segmentation | 0.85 | 0.2 | 0.8 | Semantic regions |
| 4 | Canny | 0.6 | 0.0 | 0.3 | Edge refinement |
### Advanced Masking Workflow
```mermaid
flowchart TD
A[Load Image] --> B[GroundingDINO]
B --> C[SAM Detector]
C --> D[Mask List to Mask]
D --> E[Grow Mask]
E --> F[Feather Mask]
F --> G[Mask to Latent Mask]
E --> H[2-5px dilation]
F --> I[Gaussian blur 3-5px]
```
### Node Additions from Phase 1
#### Mask Refinement Chain
1. **Grow Mask**
- Node: `GrowMask` or `MaskDilate` from WAS Node Suite
- Amount: 3 pixels
- Purpose: Prevent edge gaps
2. **Feather Mask**
- Node: `FeatherMask` from WAS Node Suite
- Amount: 5 pixels
- Purpose: Smooth transitions
3. **Mask Composite**
- Node: `MaskComposite`
- Operation: Union
- Combine multiple structural masks
### IP-Adapter Plus Configuration
For style reference without affecting geometry:
- Node: `IPAdapterAdvanced` (from ComfyUI_IPAdapter_plus)
- Model: ip-adapter_sd15
- Weight: 0.6
- Noise: 0.0
- Start At: 0.0
- End At: 0.5
### Phase 2 Workflow JSON
See: `workflows/dreamweaver_phase2_multicontrol.json`
---
## Phase 3: Production Batch Processing
### Purpose
Enable automated batch processing for high-volume production environment with dual RTX PRO 6000 GPUs.
### Automation Architecture
```mermaid
flowchart TD
A[Directory Monitor] --> B[Queue Manager]
B --> C{GPU Available?}
C -->|Yes| D[Load Image]
C -->|No| E[Queue Wait]
E --> C
D --> F[Auto Mask Gen]
F --> G[Cache Check]
G -->|Cached| H[Use Cached Mask]
G -->|New| I[Generate Mask]
I --> J[Cache Mask]
H --> K[Batch Inference]
J --> K
K --> L[4K Upscale]
L --> M[Save Output]
M --> N[Next in Queue]
```
### Automatic Mask Generation
Using semantic segmentation models:
1. **ONE-Former Integration**
- Model: oneformer_ade20k_swin_large
- Classes: wall, floor, ceiling, window, door
- Output: Multi-class segmentation mask
2. **Mask2Former Alternative**
- Model: mask2former_swin_large_ade20k
- More accurate but slower
### Latent Upscaling Configuration
| Stage | Model | Scale | Purpose |
|-------|-------|-------|---------|
| 1 | 4x-UltraSharp | 4x | Primary upscaling |
| 2 | ESRGAN_4x | 4x | Alternative option |
| 3 | RealESRGAN_x4plus | 4x | Photorealistic preference |
**Upscaling Workflow:**
1. Generate at 1024x1024
2. Upscale to 4096x4096 using 4x-UltraSharp
3. Optional: Tile-based refinement for details
### Dual GPU Configuration
```python
# GPU Allocation Strategy
GPU_0_TASKS = ["model_loading", "controlnet_1", "controlnet_2"]
GPU_1_TASKS = ["controlnet_3", "sam_processing", "vae_decode"]
# NVLink Memory Pooling
enable_nvlink = True
shared_memory_pool = True
```
### Phase 3 Workflow JSON
See: `workflows/dreamweaver_phase3_batch.json`
---
## Prompt Engineering Templates
### Template 1: Scandinavian Minimalist
**File:** `prompts/scandinavian_minimalist.txt`
```
POSITIVE:
scandinavian minimalist interior design, light oak wood flooring, neutral beige textiles, abundant natural light streaming through large windows, clean white walls, simple functional furniture, cozy hygge atmosphere, soft cream and warm gray tones, organic cotton fabrics, potted green plants, minimalist pendant lighting, decluttered space, architectural photography, 8k resolution, photorealistic, global illumination, soft shadows
Style Weight: <lora:Interior_Style_Scandi:0.8>
NEGATIVE:
worst quality, low quality, illustration, 3d render, 2d, painting, cartoon, sketch, blurry, distorted, deformed, extra windows, unrealistic lighting, structural changes, wall repositioning, window modification, door relocation, ceiling alteration, heavy ornamentation, dark colors, cluttered space, gaudy furniture, excessive decoration
```
### Template 2: Art Deco Luxe
**File:** `prompts/art_deco_luxe.txt`
```
POSITIVE:
art deco luxury interior design, geometric chevron patterns, gold brass accents, rich velvet upholstery in emerald green and sapphire blue, sunburst mirrors, polished marble flooring with brass inlay, crystal chandeliers, lacquered wood furniture, bold symmetrical arrangements, 1920s glamour, warm ambient lighting, architectural photography, 8k resolution, photorealistic, global illumination, elegant reflections
Style Weight: <lora:Interior_Style_ArtDeco:0.85>
NEGATIVE:
worst quality, low quality, illustration, 3d render, 2d, painting, cartoon, sketch, blurry, distorted, deformed, extra windows, unrealistic lighting, structural changes, wall repositioning, window modification, door relocation, ceiling alteration, rustic elements, farmhouse style, minimalism, industrial aesthetic, cheap materials, plastic furniture
```
### Template 3: Cyberpunk Neon
**File:** `prompts/cyberpunk_neon.txt`
```
POSITIVE:
cyberpunk neon interior design, high contrast LED strip lighting in electric blue and hot pink, reflective chrome surfaces, holographic accents, dark matte walls, futuristic furniture with clean lines, glowing circuit patterns, polished concrete flooring with epoxy coating, moody atmospheric lighting, tech-noir aesthetic, blade runner inspiration, architectural photography, 8k resolution, photorealistic, neon reflections, volumetric fog
Style Weight: <lora:Interior_Style_Cyberpunk:0.9>
NEGATIVE:
worst quality, low quality, illustration, 3d render, 2d, painting, cartoon, sketch, blurry, distorted, deformed, extra windows, unrealistic lighting, structural changes, wall repositioning, window modification, door relocation, ceiling alteration, natural daylight, rustic elements, traditional furniture, warm wood tones, biophilic elements, organic shapes
```
### Template 4: Biophilic Organic
**File:** `prompts/biophilic_organic.txt`
```
POSITIVE:
biophilic organic interior design, living green walls with ferns and moss, natural stone accent walls in slate and travertine, diffuse natural lighting, rattan and bamboo furniture, abundant houseplants, natural wood grain textures, water feature elements, earth tone color palette with sage green and terracotta, sustainable materials, nature-inspired patterns, architectural photography, 8k resolution, photorealistic, dappled sunlight, organic flowing shapes
Style Weight: <lora:Interior_Style_Biophilic:0.8>
NEGATIVE:
worst quality, low quality, illustration, 3d render, 2d, painting, cartoon, sketch, blurry, distorted, deformed, extra windows, unrealistic lighting, structural changes, wall repositioning, window modification, door relocation, ceiling alteration, synthetic materials, plastic plants, harsh artificial lighting, geometric patterns, industrial aesthetic, stark minimalism
```
### Template 5: Japandi Fusion
**File:** `prompts/japandi_fusion.txt`
```
POSITIVE:
japandi fusion interior design, wabi-sabi textures with imperfect beauty, low-profile furniture, muted earth tones with warm grays and soft browns, natural linen fabrics, handmade ceramic accents, light ash wood, shoji screen elements, minimal decoration with intentional negative space, zen garden elements, tatami mat textures, soft diffused lighting, architectural photography, 8k resolution, photorealistic, serene atmosphere, clean lines
Style Weight: <lora:Interior_Style_Japandi:0.85>
NEGATIVE:
worst quality, low quality, illustration, 3d render, 2d, painting, cartoon, sketch, blurry, distorted, deformed, extra windows, unrealistic lighting, structural changes, wall repositioning, window modification, door relocation, ceiling alteration, bright colors, ornate decoration, high furniture, cluttered surfaces, shiny materials, bold patterns, excessive ornamentation
```
---
## API Integration Guide
### ComfyUI Async Queue API
**Base URL:** `http://localhost:8188`
### Queue Workflow Endpoint
```http
POST /prompt
Content-Type: application/json
{
"prompt": {
"1": {
"inputs": {
"image": "input_image.jpg"
},
"class_type": "LoadImage"
},
// ... additional nodes
},
"client_id": "dreamweaver_session_001"
}
```
### Response Format
```json
{
"prompt_id": "uuid-string",
"number": 42,
"node_errors": {}
}
```
### WebSocket Status Updates
```javascript
const ws = new WebSocket('ws://localhost:8188/ws?clientId=dreamweaver_session_001');
ws.onmessage = (event) => {
const data = JSON.parse(event.data);
if (data.type === 'progress') {
console.log(`Progress: ${data.data.value}/${data.data.max}`);
}
if (data.type === 'executing') {
console.log(`Executing node: ${data.data.node}`);
}
if (data.type === 'completed') {
console.log('Workflow completed');
}
};
```
### Python API Client Example
```python
import json
import requests
import websocket
class DreamWeaverAPI:
def __init__(self, server_address="localhost:8188"):
self.server_address = server_address
self.client_id = str(uuid.uuid4())
def queue_workflow(self, workflow_json, input_image):
"""Submit workflow to queue"""
prompt = json.loads(workflow_json)
# Update input image
for node_id in prompt:
if prompt[node_id]["class_type"] == "LoadImage":
prompt[node_id]["inputs"]["image"] = input_image
data = {
"prompt": prompt,
"client_id": self.client_id
}
response = requests.post(
f"http://{self.server_address}/prompt",
json=data
)
return response.json()
def get_queue_status(self):
"""Check queue status"""
response = requests.get(f"http://{self.server_address}/queue")
return response.json()
```
---
## Deployment Instructions
### Step 1: Environment Setup
```bash
# Clone ComfyUI if not exists
git clone https://github.com/comfyanonymous/ComfyUI.git Project_Velocity/comfy_engine
cd Project_Velocity/comfy_engine
# Install Python dependencies
pip install -r requirements.txt
# Install torch with CUDA support
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
```
### Step 2: Model Installation
```bash
# Create model directories
mkdir -p models/{checkpoints,controlnet,vae,sams,grounding-dino,ipadapter}
# Download RealVisXL V5.0
# Place in: models/checkpoints/realvisxlV50Lightning_v50Lightning.safetensors
# Download ControlNet models
# Place in: models/controlnet/
# - control_v11f1p_sd15_depth.pth
# - control_v11p_sd15_seg.pth
# - control_v11p_sd15_canny.pth
# - control_v11p_sd15_mlsd.pth
# Download SAM models
# Place in: models/sams/
# - sam_vit_l_0b3195.pth
# - sam_vit_h_4b8939.pth
# Download VAE
# Place in: models/vae/
# - sdxl_vae.safetensors
```
### Step 3: Custom Node Installation
```bash
cd custom_nodes
# Install required nodes
./install_nodes.sh # See Custom Node Requirements section
# Restart ComfyUI after installation
```
### Step 4: Workflow Import
1. Launch ComfyUI: `python main.py --fp16 --lowvram`
2. Open browser to `http://localhost:8188`
3. Load workflow JSON via `Load` button
4. Verify all nodes resolve correctly
5. Test with sample image
### Step 5: Performance Validation
**Phase 1 Validation Checklist:**
- [ ] Image loads successfully
- [ ] Depth map generates without error
- [ ] SAM mask creates proper segmentation
- [ ] Generation completes in < 15 seconds
- [ ] Output preserves room geometry
- [ ] VRAM usage stays below 11GB
**Phase 2 Validation Checklist:**
- [ ] Multi-ControlNet loads correctly
- [ ] All 3-4 ControlNets apply without OOM
- [ ] Mask refinement prevents edge bleeding
- [ ] IP-Adapter applies style reference
- [ ] Generation completes in < 30 seconds
**Phase 3 Validation Checklist:**
- [ ] Batch processing handles 8+ images
- [ ] Mask caching works correctly
- [ ] Dual GPU distribution functions
- [ ] 4K upscaling produces quality output
- [ ] Queue management handles failures gracefully
### Troubleshooting
| Issue | Solution |
|-------|----------|
| OOM Error | Reduce resolution to 896x896, enable tiled VAE |
| ControlNet not loading | Verify model paths and file integrity |
| SAM mask poor quality | Adjust threshold or try different SAM model |
| Slow generation | Enable xformers, use Lightning sampler |
| Color distortion | Use RealVisXL native VAE instead of sdxl_vae |
| Edge bleeding | Increase mask grow amount, enable feathering |
---
## Appendix A: SHA256 Checksums
Verify model integrity with these checksums:
| File | Expected SHA256 |
|------|-----------------|
| realvisxlV50Lightning_v50Lightning.safetensors | [Verify on Civitai] |
| control_v11f1p_sd15_depth.pth | [Verify on HuggingFace] |
| sam_vit_l_0b3195.pth | b3c0c6a63c96e3a3c6e6c5f8d3b8c9a2... |
| sdxl_vae.safetensors | [Verify on HuggingFace] |
---
## Appendix B: Resource Links
- RealVisXL V5.0: https://civitai.com/models/139562
- ControlNet v1.1: https://huggingface.co/lllyasviel/ControlNet-v1-1
- ComfyUI: https://github.com/comfyanonymous/ComfyUI
- SAM: https://github.com/facebookresearch/segment-anything
- IP-Adapter: https://github.com/tencent-ailab/IP-Adapter
---
## Appendix C: Dynamic Keyword & LLM Prompt Expansion (Gateway v2)
API Gateway v2 introduces a dynamic prompt generation pipeline. Instead of relying solely on the five static style templates, users can provide free-form **keywords** (e.g., "blue marble", "gold veins", "renaissance") and a **room type** (e.g., "living_room", "bedroom").
### Architecture
The expansion is handled by `comfy_engine/scripts/prompt_expander.py` which uses a Chain-of-Thought (CoT) approach driven exclusively by a local LLM for strict data privacy.
- **Backend Model**: Local Ollama running `qwen3.5:27b` (default). Cloud API calls (e.g. Gemini, OpenAI) have been completely removed.
The LLM is provided with:
- **Keywords**: The raw list of aesthetic descriptors from the user.
- **Room Contexts**: Contextual constraints for specific room types (e.g., a "bathroom" context explicitly instructs the model to include wet-area materials and avoid beds).
- **Few-Shot Examples**: Hand-crafted prompt examples mapping keywords to complete Stable Diffusion XL positive and negative prompts.
### Pipeline Flow
1. **Client Request**: The iOS app calls `POST /dream-weaver` with `image`, `room_type`, and `keywords`.
2. **LLM Chain-of-Thought**:
- Gateway calls `expand_prompt()` from `prompt_expander.py`.
- The LLM reasons about the core aesthetic and generates a rich `positive_prompt` (80-120 words), a structured `negative_prompt`, and recommended technical parameters (`cfg`, `denoise`, `steps`).
3. **ComfyUI Injection**: The expanded prompts are injected into the standard phase 1 workflow (nodes 3 & 4) via `dw_gateway_v2.py`.
4. **Queue & Poll**: The image is generated through the ComfyUI API asynchronously.
### Endpoints (v2)
- `POST /dream-weaver`: Main generation endpoint now accepts `keywords` and `room_type` as multipart form fields.
- `POST /dream-weaver/expand`: Previews the LLM-expanded prompt without generating the image.
- `GET /room-types`: Returns the list of supported room contexts and their descriptors.
---
**Document End**

373
media-engine/gateway.py Normal file
View File

@@ -0,0 +1,373 @@
#!/usr/bin/env python3
import asyncio, json, time, uuid, io, sys, os, logging
from pathlib import Path
from dataclasses import dataclass
from typing import Optional, List
import httpx
import uvicorn
from fastapi import FastAPI, UploadFile, File, HTTPException, Form, Request
from fastapi.responses import JSONResponse, StreamingResponse
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
ROOT_DIR = Path(__file__).resolve().parent
for scripts_dir in (ROOT_DIR / "comfy_engine" / "scripts", ROOT_DIR / "scripts"):
if scripts_dir.exists():
sys.path.insert(0, str(scripts_dir))
try:
from gateway_auth import load_gateway_api_key, is_gateway_request_authorized
except ImportError as exc:
raise RuntimeError("Dream Weaver gateway_auth.py is required on PYTHONPATH") from exc
try:
from prompt_expander import expand_prompt, ROOM_CONTEXTS, ExpandedPrompt
LLM_AVAILABLE = True
except ImportError as exc:
LLM_AVAILABLE = False
logging.warning("prompt_expander unavailable; using deterministic fallback expansion: %s", exc)
class ExpandedPrompt(BaseModel):
style_name: str
positive_prompt: str
negative_prompt: str
steps: int = 28
cfg: float = 7.0
denoise: float = 0.72
ROOM_CONTEXTS = {}
def expand_prompt(keywords: List[str], room_type: str) -> ExpandedPrompt:
pretty_room = room_type.replace("_", " ").strip() or "living room"
pretty_keywords = ", ".join(keywords) if keywords else "modern, photorealistic"
return ExpandedPrompt(
style_name="Fallback Prompt Expansion",
positive_prompt=(
f"photorealistic premium {pretty_room} interior design, {pretty_keywords}, "
"natural lighting, realistic materials, architect-grade composition"
),
negative_prompt=(
"worst quality, low quality, blurry, distorted perspective, "
"people, watermark, text, duplicate objects"
),
)
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
logger = logging.getLogger("DreamWeaverGateway")
COMFY = (os.environ.get("COMFYUI_URL") or os.environ.get("COMFY_URL") or "http://127.0.0.1:8188").rstrip("/")
COMFY_URLS = [
item.strip().rstrip("/")
for item in os.environ.get("COMFYUI_URLS", COMFY).split(",")
if item.strip()
]
COMFY_TLS_VERIFY = os.environ.get("COMFYUI_TLS_VERIFY", "true").strip().lower() not in {"0", "false", "no", "off"}
GATEWAY_API_KEY = load_gateway_api_key()
POLL_TIMEOUT_SECONDS = int(os.environ.get("DREAM_WEAVER_POLL_TIMEOUT_SECONDS", "1800"))
POLL_INTERVAL_SECONDS = float(os.environ.get("DREAM_WEAVER_POLL_INTERVAL_SECONDS", "2"))
INPUT_MEGAPIXELS = float(os.environ.get("DREAM_WEAVER_INPUT_MEGAPIXELS", "0.75"))
MAX_RENDER_STEPS = int(os.environ.get("DREAM_WEAVER_MAX_STEPS", "18"))
PREFERRED_CHECKPOINTS = [
"realvisxlV50_v50LightningBakedvae.safetensors",
"realvisxlV50Lightning_v50Lightning.safetensors",
]
app = FastAPI(title="Dream Weaver API v2", version="2.0.0")
app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"])
jobs: dict = {}
worker_locks: dict[str, asyncio.Lock] = {}
assignment_lock = asyncio.Lock()
ACTIVE_JOB_STATUSES = {"uploading", "processing"}
@dataclass(frozen=True)
class ComfyWorker:
id: str
url: str
COMFY_WORKERS = [
ComfyWorker(id=f"comfy-{index}", url=url)
for index, url in enumerate(COMFY_URLS)
]
for worker in COMFY_WORKERS:
worker_locks.setdefault(worker.id, asyncio.Lock())
def comfy_client(timeout: float = 30) -> httpx.AsyncClient:
return httpx.AsyncClient(timeout=timeout, verify=COMFY_TLS_VERIFY, follow_redirects=True)
def worker_by_id(worker_id: str) -> ComfyWorker:
for worker in COMFY_WORKERS:
if worker.id == worker_id:
return worker
raise HTTPException(status_code=500, detail=f"Dream Weaver worker {worker_id} is not configured")
async def list_comfy_checkpoints(worker: Optional[ComfyWorker] = None) -> list[str]:
worker = worker or COMFY_WORKERS[0]
async with comfy_client(timeout=10) as client:
response = await client.get(f"{worker.url}/models/checkpoints")
response.raise_for_status()
payload = response.json()
if isinstance(payload, list):
return [item for item in payload if isinstance(item, str)]
return []
async def resolve_checkpoint(worker: ComfyWorker) -> str:
checkpoints = await list_comfy_checkpoints(worker)
if not checkpoints:
raise HTTPException(
status_code=503,
detail=(
"ComfyUI is online but has no checkpoint models installed. "
"Hydrate RealVisXL into ComfyUI/models/checkpoints before generating."
),
)
lower_lookup = {item.lower(): item for item in checkpoints}
for preferred in PREFERRED_CHECKPOINTS:
match = lower_lookup.get(preferred.lower())
if match:
return match
return checkpoints[0]
def gateway_urls(job_id: str) -> dict:
return {
"poll_url": f"/dream-weaver/status/{job_id}",
"result_url": f"/dream-weaver/result/{job_id}",
}
def ensure_gateway_auth(request: Request) -> None:
if is_gateway_request_authorized(request.headers, GATEWAY_API_KEY):
return
raise HTTPException(status_code=401, detail="Dream Weaver gateway API key is required or invalid.")
async def worker_queue_size(worker: ComfyWorker) -> int:
async with comfy_client(timeout=5) as client:
response = await client.get(f"{worker.url}/queue")
response.raise_for_status()
payload = response.json()
running = payload.get("queue_running") if isinstance(payload, dict) else []
pending = payload.get("queue_pending") if isinstance(payload, dict) else []
return len(running or []) + len(pending or [])
def local_worker_load(worker: ComfyWorker) -> int:
return sum(
1
for job in jobs.values()
if job.get("worker_id") == worker.id and job.get("status") in ACTIVE_JOB_STATUSES
)
async def choose_worker() -> ComfyWorker:
candidates: list[tuple[int, int, str, ComfyWorker]] = []
errors: list[str] = []
for worker in COMFY_WORKERS:
try:
checkpoints = await list_comfy_checkpoints(worker)
if not checkpoints:
errors.append(f"{worker.id} has no checkpoints")
continue
candidates.append((local_worker_load(worker), await worker_queue_size(worker), worker.id, worker))
except Exception as exc:
errors.append(f"{worker.id} unhealthy: {exc}")
if not candidates:
detail = "; ".join(errors) if errors else "No ComfyUI workers are configured"
raise HTTPException(status_code=503, detail=f"No healthy Dream Weaver workers. {detail}")
candidates.sort(key=lambda item: (item[0], item[1], item[2]))
return candidates[0][3]
async def upload_to_comfy(worker: ComfyWorker, data: bytes, filename: str) -> str:
async with comfy_client(timeout=30) as client:
r = await client.post(f"{worker.url}/upload/image", files={"image": (filename, data, "image/jpeg")}, data={"overwrite": "true"})
r.raise_for_status()
return r.json()["name"]
def normalize_expanded_prompt(expanded: "ExpandedPrompt") -> "ExpandedPrompt":
expanded.steps = max(6, min(int(expanded.steps or MAX_RENDER_STEPS), MAX_RENDER_STEPS))
expanded.cfg = max(3.0, min(float(expanded.cfg or 6.0), 7.0))
expanded.denoise = max(0.45, min(float(expanded.denoise or 0.65), 0.72))
return expanded
def build_workflow(img_name: str, expanded: "ExpandedPrompt", ckpt_name: str) -> dict:
expanded = normalize_expanded_prompt(expanded)
return {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": ckpt_name}},
"2": {"class_type": "LoadImage", "inputs": {"image": img_name, "upload": "image"}},
"9": {"class_type": "ImageScaleToTotalPixels", "inputs": {"image": ["2", 0], "upscale_method": "lanczos", "megapixels": INPUT_MEGAPIXELS, "resolution_steps": 8}},
"3": {"class_type": "CLIPTextEncode", "inputs": {"text": expanded.positive_prompt, "clip": ["1", 1]}},
"4": {"class_type": "CLIPTextEncode", "inputs": {"text": expanded.negative_prompt, "clip": ["1", 1]}},
"5": {"class_type": "VAEEncode", "inputs": {"pixels": ["9", 0], "vae": ["1", 2]}},
"6": {"class_type": "KSampler", "inputs": {"model": ["1", 0], "positive": ["3", 0], "negative": ["4", 0], "latent_image": ["5", 0], "seed": int(time.time()) % 999983, "steps": expanded.steps, "cfg": expanded.cfg, "sampler_name": "dpmpp_2m", "scheduler": "karras", "denoise": expanded.denoise}},
"7": {"class_type": "VAEDecode", "inputs": {"samples": ["6", 0], "vae": ["1", 2]}},
"8": {"class_type": "SaveImage", "inputs": {"images": ["7", 0], "filename_prefix": f"dw_{expanded.style_name.replace(' ', '_')[:30]}"}},
}
async def queue_prompt(worker: ComfyWorker, workflow: dict) -> str:
async with comfy_client(timeout=30) as client:
r = await client.post(f"{worker.url}/prompt", json={"prompt": workflow, "client_id": str(uuid.uuid4())})
if r.status_code >= 400:
detail = r.text
try:
detail = json.dumps(r.json())
except Exception:
pass
raise HTTPException(status_code=502, detail=f"ComfyUI rejected Dream Weaver workflow: {detail}")
return r.json()["prompt_id"]
def extract_comfy_error(history_entry: dict) -> Optional[str]:
status = history_entry.get("status") if isinstance(history_entry, dict) else None
if not isinstance(status, dict):
return None
if status.get("status_str") != "error":
return None
messages = status.get("messages") or []
for kind, payload in reversed(messages):
if kind == "execution_error" and isinstance(payload, dict):
node_type = payload.get("node_type") or payload.get("node_id") or "ComfyUI"
message = payload.get("exception_message") or payload.get("exception_type") or "ComfyUI execution failed"
return f"{node_type}: {message}"
return "ComfyUI execution failed"
async def poll_result(worker: ComfyWorker, prompt_id: str, timeout: int = POLL_TIMEOUT_SECONDS):
start = time.time()
async with comfy_client(timeout=10) as client:
while time.time() - start < timeout:
r = await client.get(f"{worker.url}/history/{prompt_id}")
if r.status_code == 200:
h = r.json().get(prompt_id, {})
err = extract_comfy_error(h)
if err:
return None, err
imgs = [img for nd in h.get("outputs", {}).values() for img in nd.get("images", [])]
if imgs: return imgs[0], None
await asyncio.sleep(POLL_INTERVAL_SECONDS)
return None, f"timeout after {timeout} seconds"
async def background_poll(job_id: str, worker_id: str, prompt_id: str):
worker = worker_by_id(worker_id)
img, err = await poll_result(worker, prompt_id)
if img:
jobs[job_id].update({"status": "done", "output": img, "completed": time.time()})
else:
jobs[job_id].update({"status": "error", "error": str(err), "completed": time.time()})
@app.get("/health")
@app.get("/dream-weaver/health")
async def health():
worker_health = []
for worker in COMFY_WORKERS:
try:
checkpoints = await list_comfy_checkpoints(worker)
queue_size = await worker_queue_size(worker)
worker_health.append({
"id": worker.id,
"url": worker.url,
"online": True,
"checkpoint_ready": bool(checkpoints),
"checkpoint_count": len(checkpoints),
"queue_size": queue_size,
"available_checkpoints": checkpoints[:12],
})
except Exception as exc:
worker_health.append({
"id": worker.id,
"url": worker.url,
"online": False,
"checkpoint_ready": False,
"checkpoint_count": 0,
"queue_size": None,
"error": str(exc),
})
ready_workers = [worker for worker in worker_health if worker["online"] and worker["checkpoint_ready"]]
checkpoints = ready_workers[0]["available_checkpoints"] if ready_workers else []
return {
"status": "ok",
"comfyui": bool(ready_workers),
"comfyui_url": COMFY_WORKERS[0].url if COMFY_WORKERS else COMFY,
"comfyui_urls": [worker.url for worker in COMFY_WORKERS],
"checkpoint_ready": bool(ready_workers),
"checkpoint_count": max((worker["checkpoint_count"] for worker in worker_health), default=0),
"preferred_checkpoints": PREFERRED_CHECKPOINTS,
"available_checkpoints": checkpoints[:12],
"workers": worker_health,
"ready_worker_count": len(ready_workers),
"llm_expansion": LLM_AVAILABLE,
"input_megapixels": INPUT_MEGAPIXELS,
"max_render_steps": MAX_RENDER_STEPS,
"poll_timeout_seconds": POLL_TIMEOUT_SECONDS,
"version": "2.0.0",
"auth_required": GATEWAY_API_KEY is not None,
"auth_scheme": "x-dream-weaver-api-key"
}
@app.get("/dream-weaver/status/{job_id}")
async def status(job_id: str, request: Request):
ensure_gateway_auth(request)
job = jobs.get(job_id)
if not job: raise HTTPException(status_code=404, detail="Job not found")
res = {k: v for k, v in job.items() if k != "output"}
if "created" in job:
res["elapsed_seconds"] = round(time.time() - float(job["created"]), 2)
res["ready"] = job.get("status") == "done"
if res["ready"]:
res.update(gateway_urls(job_id))
return res
@app.post("/dream-weaver")
async def dream_weaver(
request: Request,
image: UploadFile = File(...),
keywords: str = Form(default=""),
room_type: str = Form(default="living_room")
):
ensure_gateway_auth(request)
job_id = str(uuid.uuid4())
jobs[job_id] = {"status": "uploading", "created": time.time()}
data = await image.read()
async with assignment_lock:
worker = await choose_worker()
jobs[job_id].update({"worker_id": worker.id, "worker_url": worker.url})
lock = worker_locks.setdefault(worker.id, asyncio.Lock())
async with lock:
comfy_name = await upload_to_comfy(worker, data, f"dw_{job_id[:8]}.jpg")
kw_list = [k.strip() for k in keywords.split(",") if k.strip()]
expanded = await asyncio.to_thread(expand_prompt, keywords=kw_list, room_type=room_type)
ckpt_name = await resolve_checkpoint(worker)
jobs[job_id]["checkpoint"] = ckpt_name
wf = build_workflow(comfy_name, expanded, ckpt_name)
prompt_id = await queue_prompt(worker, wf)
jobs[job_id].update({"status": "processing", "prompt_id": prompt_id})
asyncio.create_task(background_poll(job_id, worker.id, prompt_id))
return {
"job_id": job_id,
"status": "processing",
**gateway_urls(job_id),
}
@app.get("/dream-weaver/result/{job_id}")
async def result(job_id: str, request: Request):
ensure_gateway_auth(request)
job = jobs.get(job_id)
if not job or job.get("status") != "done":
raise HTTPException(status_code=404, detail="Result not ready")
img = job.get("output")
if not img:
raise HTTPException(status_code=404, detail="Result not ready")
worker = worker_by_id(job.get("worker_id", COMFY_WORKERS[0].id))
async with comfy_client(timeout=30) as client:
response = await client.get(
f"{worker.url}/view",
params={
"filename": img["filename"],
"subfolder": img.get("subfolder", ""),
"type": img.get("type", "output"),
},
)
response.raise_for_status()
return StreamingResponse(
io.BytesIO(response.content),
media_type="image/png",
headers={"Content-Disposition": f"attachment; filename=dreamweaver_{job_id[:8]}.png"},
)
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8082, log_level="info")

View File

@@ -0,0 +1,155 @@
{
"1": {
"class_type": "LoadImage",
"_meta": {"title": "Ground Truth (Architectural/Floorplan)"},
"inputs": {
"image": "ground_truth_input.png",
"upload": "image"
}
},
"2": {
"class_type": "LoadImage",
"_meta": {"title": "Style Reference (Google/Pinterest)"},
"inputs": {
"image": "style_reference_input.png",
"upload": "image"
}
},
"3": {
"class_type": "DiffusersLoader",
"_meta": {"title": "Qwen-Image-2512 Model Loader"},
"inputs": {
"model_path": "/home/ubuntu/models/Qwen-Image-2512"
}
},
"4": {
"class_type": "ImageScale",
"_meta": {"title": "Scale Ground Truth"},
"inputs": {
"image": ["1", 0],
"upscale_method": "lanczos",
"width": 1104,
"height": 1472,
"crop": "center"
}
},
"5": {
"class_type": "CannyEdgePreprocessor",
"_meta": {"title": "Canny Edge (Spatial Geometry)"},
"inputs": {
"image": ["4", 0],
"low_threshold": 80,
"high_threshold": 160,
"resolution": 1024
}
},
"6": {
"class_type": "ControlNetLoader",
"_meta": {"title": "ControlNet Canny Loader"},
"inputs": {
"control_net_name": "control_v11p_sd15_canny.pth"
}
},
"7": {
"class_type": "CLIPVisionLoader",
"_meta": {"title": "CLIP Vision for IP-Adapter"},
"inputs": {
"clip_name": "CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors"
}
},
"8": {
"class_type": "IPAdapterModelLoader",
"_meta": {"title": "IP-Adapter Model"},
"inputs": {
"ipadapter_file": "ip-adapter_sd15.bin"
}
},
"9": {
"class_type": "CLIPTextEncode",
"_meta": {"title": "Positive Prompt (Typography + Aesthetic)"},
"inputs": {
"text": "A highly realistic, cinematic real estate marketing poster. Interior style: modern luxury, warm ambient lighting, premium materials. The image must prominently feature the exact text 'YOUR DREAM HOME AWAITS' written in elegant, modern, highly legible typography positioned at the lower third of the image, clean sans-serif font, crisp rendering, high contrast text. Professional cinematic lighting, 8k resolution, photorealistic quality, detailed textures, architectural photography, ultra-sharp focus, golden hour warmth, premium real estate aesthetic.",
"clip": ["3", 1]
}
},
"10": {
"class_type": "CLIPTextEncode",
"_meta": {"title": "Negative Prompt"},
"inputs": {
"text": "deformed, blurry, bad anatomy, watermark, logo, extra text, distorted text, blurry text, misaligned text, low quality, worst quality, illustration, 3d render, painting, cartoon, sketch, artifacts, noise, oversaturated, unrealistic lighting, structural changes, extra windows, extra doors, warped perspective, low resolution, pixelated",
"clip": ["3", 1]
}
},
"11": {
"class_type": "EmptyLatentImage",
"_meta": {"title": "Poster Canvas (3:4 Portrait)"},
"inputs": {
"width": 1104,
"height": 1472,
"batch_size": 1
}
},
"12": {
"class_type": "IPAdapterAdvanced",
"_meta": {"title": "Style Transfer from Reference"},
"inputs": {
"model": ["3", 0],
"ipadapter": ["8", 0],
"image": ["2", 0],
"clip_vision": ["7", 0],
"weight": 0.55,
"weight_type": "linear",
"combine_embeds": "concat",
"start_at": 0.0,
"end_at": 0.5,
"embeds_scaling": "V only",
"noise": 0.0
}
},
"13": {
"class_type": "ControlNetApplyAdvanced",
"_meta": {"title": "ControlNet Apply (Canny Geometry)"},
"inputs": {
"positive": ["9", 0],
"negative": ["10", 0],
"control_net": ["6", 0],
"image": ["5", 0],
"strength": 0.75,
"start_percent": 0.0,
"end_percent": 0.65
}
},
"14": {
"class_type": "KSampler",
"_meta": {"title": "Qwen Sampler (true_cfg compatible)"},
"inputs": {
"model": ["12", 0],
"positive": ["13", 0],
"negative": ["13", 1],
"latent_image": ["11", 0],
"seed": 42,
"control_after_generate": "randomize",
"steps": 50,
"cfg": 4.0,
"sampler_name": "euler",
"scheduler": "normal",
"denoise": 1.0
}
},
"15": {
"class_type": "VAEDecode",
"_meta": {"title": "Decode Latent to Image"},
"inputs": {
"samples": ["14", 0],
"vae": ["3", 2]
}
},
"16": {
"class_type": "SaveImage",
"_meta": {"title": "Save Catalyst Poster Output"},
"inputs": {
"images": ["15", 0],
"filename_prefix": "catalyst_poster_qwen"
}
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,930 @@
{
"last_node_id": 25,
"last_link_id": 42,
"nodes": [
{
"id": 1,
"type": "LoadImage",
"pos": [
100,
150
],
"size": [
320,
280
],
"flags": {},
"order": 0,
"mode": 0,
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
1,
3
],
"shape": 3,
"slot_index": 0
},
{
"name": "MASK",
"type": "MASK",
"links": [],
"shape": 3,
"slot_index": 1
}
],
"properties": {
"Node name for S&R": "LoadImage"
},
"widgets_values": [
"input_interior.jpg"
]
},
{
"id": 2,
"type": "ImageScale",
"pos": [
500,
150
],
"size": [
320,
200
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [
{
"name": "image",
"type": "IMAGE",
"link": 1
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
4,
5
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "ImageScale"
},
"widgets_values": [
"lanczos",
1024,
1024,
"center"
]
},
{
"id": 3,
"type": "Zoe-DepthMapPreprocessor",
"pos": [
500,
400
],
"size": [
320,
100
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "image",
"type": "IMAGE",
"link": 4
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
6
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "Zoe-DepthMapPreprocessor"
},
"widgets_values": [
512
]
},
{
"id": 4,
"type": "SAMDetectorSegmented",
"pos": [
900,
150
],
"size": [
320,
200
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [
{
"name": "image",
"type": "IMAGE",
"link": 5
},
{
"name": "sam_model",
"type": "SAM_MODEL",
"link": 2
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
7
],
"shape": 3,
"slot_index": 0
},
{
"name": "MASK",
"type": "MASK",
"links": [
8
],
"shape": 3,
"slot_index": 1
}
],
"properties": {
"Node name for S&R": "SAMDetectorSegmented"
},
"widgets_values": [
"walls, floor, ceiling",
0.3,
0,
0,
1
]
},
{
"id": 5,
"type": "SAMModelLoader (segment anything)",
"pos": [
500,
600
],
"size": [
320,
100
],
"flags": {},
"order": 0,
"mode": 0,
"outputs": [
{
"name": "SAM_MODEL",
"type": "SAM_MODEL",
"links": [
2
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "SAMModelLoader (segment anything)"
},
"widgets_values": [
"sam_vit_l_0b3195.pth"
]
},
{
"id": 6,
"type": "ControlNetLoader",
"pos": [
900,
450
],
"size": [
320,
100
],
"flags": {},
"order": 0,
"mode": 0,
"outputs": [
{
"name": "CONTROL_NET",
"type": "CONTROL_NET",
"links": [
9
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "ControlNetLoader"
},
"widgets_values": [
"control_v11f1p_sd15_depth.pth"
]
},
{
"id": 7,
"type": "ControlNetApply",
"pos": [
1300,
400
],
"size": [
320,
150
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{
"name": "conditioning",
"type": "CONDITIONING",
"link": 10
},
{
"name": "control_net",
"type": "CONTROL_NET",
"link": 9
},
{
"name": "image",
"type": "IMAGE",
"link": 6
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"links": [
14
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "ControlNetApply"
},
"widgets_values": [
1.0
]
},
{
"id": 8,
"type": "CheckpointLoaderSimple",
"pos": [
100,
600
],
"size": [
320,
100
],
"flags": {},
"order": 0,
"mode": 0,
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
11,
15
],
"shape": 3,
"slot_index": 0
},
{
"name": "CLIP",
"type": "CLIP",
"links": [
12,
13
],
"shape": 3,
"slot_index": 1
},
{
"name": "VAE",
"type": "VAE",
"links": [
23,
25
],
"shape": 3,
"slot_index": 2
}
],
"properties": {
"Node name for S&R": "CheckpointLoaderSimple"
},
"widgets_values": [
"realvisxlV50Lightning_v50Lightning.safetensors"
]
},
{
"id": 9,
"type": "CLIPTextEncode",
"pos": [
500,
750
],
"size": [
400,
200
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 12
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"links": [
10
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "CLIPTextEncode"
},
"widgets_values": [
"scandinavian minimalist interior design, light oak wood flooring, neutral beige textiles, abundant natural light streaming through large windows, clean white walls, simple functional furniture, cozy atmosphere, soft cream and warm gray tones, architectural photography, 8k resolution, photorealistic, global illumination, soft shadows"
]
},
{
"id": 10,
"type": "CLIPTextEncode",
"pos": [
500,
1000
],
"size": [
400,
200
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 13
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"links": [
16
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "CLIPTextEncode"
},
"widgets_values": [
"(worst quality, low quality, illustration, 3d, 2d, painting, cartoons, sketch), blurry, distorted, deformed, extra windows, unrealistic lighting, structural changes, wall repositioning, window modification, door relocation, ceiling alteration"
]
},
{
"id": 11,
"type": "EmptyLatentImage",
"pos": [
100,
850
],
"size": [
320,
100
],
"flags": {},
"order": 0,
"mode": 0,
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"links": [
17
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "EmptyLatentImage"
},
"widgets_values": [
1024,
1024,
1
]
},
{
"id": 12,
"type": "SetLatentNoiseMask",
"pos": [
1300,
150
],
"size": [
320,
100
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 17
},
{
"name": "mask",
"type": "MASK",
"link": 8
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"links": [
18
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "SetLatentNoiseMask"
}
},
{
"id": 13,
"type": "KSampler",
"pos": [
1700,
300
],
"size": [
320,
280
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 11
},
{
"name": "positive",
"type": "CONDITIONING",
"link": 14
},
{
"name": "negative",
"type": "CONDITIONING",
"link": 16
},
{
"name": "latent_image",
"type": "LATENT",
"link": 18
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"links": [
19
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "KSampler"
},
"widgets_values": [
42,
"randomize",
30,
7.0,
"dpmpp_2m",
"karras",
0.75
]
},
{
"id": 14,
"type": "VAEDecode",
"pos": [
2100,
300
],
"size": [
320,
100
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 19
},
{
"name": "vae",
"type": "VAE",
"link": 23
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
20,
21,
22
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAEDecode"
}
},
{
"id": 15,
"type": "SaveImage",
"pos": [
2500,
200
],
"size": [
320,
280
],
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 20
}
],
"properties": {
"Node name for S&R": "SaveImage"
},
"widgets_values": [
"dreamweaver_phase1_output"
]
},
{
"id": 16,
"type": "PreviewImage",
"pos": [
2500,
550
],
"size": [
320,
280
],
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 21
}
],
"properties": {
"Node name for S&R": "PreviewImage"
}
},
{
"id": 17,
"type": "MaskToImage",
"pos": [
1700,
100
],
"size": [
320,
100
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{
"name": "mask",
"type": "MASK",
"link": 8
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
24
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "MaskToImage"
}
},
{
"id": 18,
"type": "PreviewImage",
"pos": [
2100,
500
],
"size": [
320,
280
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 24
}
],
"properties": {
"Node name for S&R": "PreviewImage"
}
}
],
"links": [
[
1,
1,
2,
"IMAGE",
1
],
[
2,
5,
1,
"SAM_MODEL",
2
],
[
3,
4,
1,
"IMAGE",
3
],
[
4,
2,
3,
"IMAGE",
4
],
[
5,
2,
4,
"IMAGE",
5
],
[
6,
3,
7,
"IMAGE",
6
],
[
7,
4,
18,
"MASK",
8
],
[
8,
6,
7,
"CONTROL_NET",
9
],
[
9,
8,
9,
"CLIP",
12
],
[
10,
9,
7,
"CONDITIONING",
10
],
[
11,
8,
13,
"MODEL",
11
],
[
12,
8,
10,
"CLIP",
13
],
[
13,
10,
13,
"CONDITIONING",
16
],
[
14,
7,
13,
"CONDITIONING",
14
],
[
15,
11,
12,
"LATENT",
17
],
[
16,
12,
13,
"LATENT",
18
],
[
17,
13,
14,
"LATENT",
19
],
[
18,
14,
15,
"IMAGE",
20
],
[
19,
14,
16,
"IMAGE",
21
],
[
20,
14,
16,
"IMAGE",
22
],
[
21,
17,
18,
"IMAGE",
24
],
[
22,
8,
14,
"VAE",
23
]
],
"groups": [
{
"title": "Input & Preprocessing",
"bounding": [
50,
100,
800,
600
],
"color": "#3f789e"
},
{
"title": "ControlNet & Masking",
"bounding": [
850,
100,
800,
600
],
"color": "#8f3f7e"
},
{
"title": "Generation",
"bounding": [
1650,
100,
800,
600
],
"color": "#3f7e4a"
},
{
"title": "Output",
"bounding": [
2450,
100,
500,
800
],
"color": "#7e5e3f"
}
],
"config": {},
"extra": {
"ds": {
"scale": 0.75,
"offset": [
0,
0
]
}
},
"version": 0.4
}

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff