Initial commit: Velocity-OS migration
This commit is contained in:
42
media-engine/Dockerfile
Normal file
42
media-engine/Dockerfile
Normal file
@@ -0,0 +1,42 @@
|
||||
# syntax=docker/dockerfile:1.4
|
||||
# ============================================================
|
||||
# Velocity-OS — media-engine (Dream Weaver Gateway)
|
||||
# Async ComfyUI job gateway. Models mounted from NVMe PV.
|
||||
# ============================================================
|
||||
|
||||
FROM python:3.11-slim AS runtime
|
||||
|
||||
LABEL org.opencontainers.image.title="velocity-os-media-engine" \
|
||||
org.opencontainers.image.description="Velocity-OS Dream Weaver / ComfyUI Gateway" \
|
||||
org.opencontainers.image.vendor="Desineuron" \
|
||||
org.opencontainers.image.version="2.0.0"
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN groupadd -r velocity && useradd -r -g velocity velocity
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY --chown=velocity:velocity . .
|
||||
|
||||
ENV PYTHONUNBUFFERED=1 \
|
||||
PYTHONPATH=/app \
|
||||
# Overridden by K3s ConfigMap/env
|
||||
COMFYUI_URLS="http://localhost:8188" \
|
||||
DW_ASSET_DIR="/opt/assets/generated" \
|
||||
MODEL_BASE="/opt/models/comfy"
|
||||
|
||||
USER velocity
|
||||
|
||||
EXPOSE 8290
|
||||
|
||||
HEALTHCHECK --interval=60s --timeout=10s --start-period=30s \
|
||||
CMD curl -f http://localhost:8290/health || exit 1
|
||||
|
||||
CMD ["python", "gateway.py", "--host", "0.0.0.0", "--port", "8290"]
|
||||
840
media-engine/docs/DREAMWEAVER_TECHNICAL_SPEC.md
Normal file
840
media-engine/docs/DREAMWEAVER_TECHNICAL_SPEC.md
Normal file
@@ -0,0 +1,840 @@
|
||||
# Dream Weaver Technical Specification
|
||||
|
||||
**Version:** 1.0.0
|
||||
**Date:** 2026-03-01
|
||||
**Model:** RealVisXL V5.0 Lightning
|
||||
**Target Hardware Phase 1:** NVIDIA RTX 3080Ti (12GB GDDR6X)
|
||||
**Target Hardware Phase 3:** Dual NVIDIA RTX PRO 6000 Blackwell (96GB GDDR7 each)
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Executive Summary](#executive-summary)
|
||||
2. [Three-Phase Implementation Architecture](#three-phase-implementation-architecture)
|
||||
3. [Hardware Specifications & Optimization](#hardware-specifications--optimization)
|
||||
4. [Model Specifications & Downloads](#model-specifications--downloads)
|
||||
5. [ControlNet Configuration](#controlnet-configuration)
|
||||
6. [Custom Node Requirements](#custom-node-requirements)
|
||||
7. [Phase 1: Foundational Implementation](#phase-1-foundational-implementation)
|
||||
8. [Phase 2: Advanced Multi-ControlNet](#phase-2-advanced-multi-controlnet)
|
||||
9. [Phase 3: Production Batch Processing](#phase-3-production-batch-processing)
|
||||
10. [Prompt Engineering Templates](#prompt-engineering-templates)
|
||||
11. [API Integration Guide](#api-integration-guide)
|
||||
12. [Deployment Instructions](#deployment-instructions)
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Dream Weaver is an interior restyling workflow that uses **Structural Constraint Logic** to preserve existing room geometry while enabling comprehensive aesthetic transformations. The system employs a **Dual-ControlNet Strategy** combining M-LSD (Line Segment Detection) for architectural line preservation and Depth (Zoe/MiDaS) for 3D spatial consistency, with SAM-based masking to isolate structural immutables from stylable regions.
|
||||
|
||||
### Core Constraint: Absolute Geometry Preservation
|
||||
|
||||
The following elements are **IMMUTABLE** and must never be modified:
|
||||
- Wall positions and angles
|
||||
- Door and window placements
|
||||
- Ceiling heights
|
||||
- Room proportions and dimensions
|
||||
- Structural load-bearing elements
|
||||
- Vanishing points and perspective
|
||||
|
||||
The following elements are **MUTABLE** and may be restyled:
|
||||
- Wall paint colors and textures
|
||||
- Flooring materials
|
||||
- Furniture upholstery and styles
|
||||
- Decorative objects and accessories
|
||||
- Lighting fixtures and atmospheres
|
||||
- Soft furnishings (curtains, rugs, cushions)
|
||||
|
||||
---
|
||||
|
||||
## Three-Phase Implementation Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Input Interior Image] --> B[Phase 1: Foundational]
|
||||
B --> C[Phase 2: Advanced]
|
||||
C --> D[Phase 3: Production]
|
||||
|
||||
subgraph P1[Phase 1 - RTX 3080Ti]
|
||||
B1[Depth ControlNet] --> B2[Basic SAM Masking]
|
||||
B2 --> B3[Single Image Processing]
|
||||
end
|
||||
|
||||
subgraph P2[Phase 2 - Enhanced Quality]
|
||||
C1[Multi-ControlNet] --> C2[Refined Masking]
|
||||
C2 --> C3[Style Templates]
|
||||
end
|
||||
|
||||
subgraph P3[Phase 3 - Dual RTX PRO 6000]
|
||||
D1[Batch Processing] --> D2[4K Upscaling]
|
||||
D2 --> D3[Automated Pipeline]
|
||||
end
|
||||
```
|
||||
|
||||
### Phase Overview
|
||||
|
||||
| Phase | Hardware | ControlNets | Resolution | Batch Size | Purpose |
|
||||
|-------|----------|-------------|------------|------------|---------|
|
||||
| 1 | RTX 3080Ti | 1 (Depth) | 1024x1024 | 1 | Validation & Testing |
|
||||
| 2 | RTX 3080Ti | 3 (Depth + Seg + Canny) | 1216x832 | 1 | Quality Enhancement |
|
||||
| 3 | Dual RTX PRO 6000 | 3 + Aux | 2048x2048 | 8+ | Production Deployment |
|
||||
|
||||
---
|
||||
|
||||
## Hardware Specifications & Optimization
|
||||
|
||||
### Current Development Hardware: RTX 3080Ti
|
||||
|
||||
**Specifications:**
|
||||
- GPU: NVIDIA RTX 3080Ti
|
||||
- VRAM: 12GB GDDR6X
|
||||
- CUDA Cores: 10,240
|
||||
- Architecture: Ampere
|
||||
|
||||
**VRAM Management Strategy:**
|
||||
```python
|
||||
# Optimization flags for 12GB VRAM
|
||||
--fp16 # Enable half-precision
|
||||
--lowvram # Aggressive memory management
|
||||
--disable-xformers # Use sdp-attention instead
|
||||
```
|
||||
|
||||
**Recommended Settings:**
|
||||
- Batch size: 1
|
||||
- Maximum resolution: 1024x1024 or 1216x832
|
||||
- Tiled VAE: Enabled with tile size 64
|
||||
- Model CPU offloading: Enabled
|
||||
- Empty cache after each generation: Enabled
|
||||
|
||||
### Production Hardware: Dual RTX PRO 6000 Blackwell
|
||||
|
||||
**Specifications:**
|
||||
- GPU: 2x NVIDIA RTX PRO 6000 Blackwell
|
||||
- VRAM: 96GB GDDR7 per GPU (192GB total)
|
||||
- Architecture: Blackwell
|
||||
- NVLink: Enabled for memory pooling
|
||||
|
||||
**Optimization Strategy:**
|
||||
```python
|
||||
# Production flags for 192GB VRAM
|
||||
--bf16 # Enable bfloat16 for better precision
|
||||
--highvram # Keep models in GPU memory
|
||||
--xformers # Enable memory-efficient attention
|
||||
--gpu-batch-size 8 # Process 8 images simultaneously
|
||||
--model-sharding # Distribute across both GPUs
|
||||
```
|
||||
|
||||
### VRAM Usage Comparison
|
||||
|
||||
| Configuration | Phase 1 | Phase 2 | Phase 3 |
|
||||
|--------------|---------|---------|---------|
|
||||
| Model Loading | 6.2GB | 6.2GB | 6.2GB |
|
||||
| ControlNet 1 | 1.8GB | 1.8GB | 1.8GB |
|
||||
| ControlNet 2 | - | 1.8GB | 1.8GB |
|
||||
| ControlNet 3 | - | 1.5GB | 1.5GB |
|
||||
| SAM Model | 2.1GB | 2.1GB | 2.1GB |
|
||||
| Latent Buffers | 1.5GB | 2.2GB | 8.0GB |
|
||||
| **Total** | **~11.6GB** | **~15.6GB** | **~21.4GB** |
|
||||
|
||||
---
|
||||
|
||||
## Model Specifications & Downloads
|
||||
|
||||
### Primary Checkpoint: RealVisXL V5.0 Lightning
|
||||
|
||||
**Download URL:** https://civitai.com/models/139562?modelVersionId=789646
|
||||
|
||||
**Specifications:**
|
||||
- Base Model: SDXL
|
||||
- Training Data: Architectural photography datasets
|
||||
- Specialization: Photorealistic interiors, white balance accuracy
|
||||
- Lightning Steps: 4-8 steps for high quality
|
||||
- Recommended CFG: 1.0-2.0 (Lightning)
|
||||
- CLIP Skip: 2
|
||||
|
||||
**File Details:**
|
||||
- Filename: `realvisxlV50Lightning_v50Lightning.safetensors`
|
||||
- Expected Size: ~6.5GB
|
||||
- Format: SafeTensors
|
||||
- SHA256: Verify on download
|
||||
|
||||
**Installation Path:**
|
||||
```
|
||||
ComfyUI/models/checkpoints/realvisxlV50Lightning_v50Lightning.safetensors
|
||||
```
|
||||
|
||||
### VAE Selection
|
||||
|
||||
**Option A: Automatic1111 VAE**
|
||||
- Download: https://huggingface.co/stabilityai/sdxl-vae
|
||||
- File: `sdxl_vae.safetensors`
|
||||
- Size: ~335MB
|
||||
- Path: `ComfyUI/models/vae/sdxl_vae.safetensors`
|
||||
|
||||
**Option B: RealVisXL Native VAE**
|
||||
- Built into checkpoint (recommended for simplicity)
|
||||
|
||||
**Recommendation:** Use checkpoint's built-in VAE for Phase 1-2, Automatic1111 VAE for Phase 3 production
|
||||
|
||||
---
|
||||
|
||||
## ControlNet Configuration
|
||||
|
||||
### ControlNet Model Specifications
|
||||
|
||||
| Model | Purpose | Strength | Download URL | File Size |
|
||||
|-------|---------|----------|--------------|-----------|
|
||||
| control_v11f1p_sd15_depth | Geometric preservation | 1.0 | https://huggingface.co/lllyasviel/ControlNet-v1-1 | ~1.2GB |
|
||||
| control_v11p_sd15_seg | Semantic segmentation | 0.85 | https://huggingface.co/lllyasviel/ControlNet-v1-1 | ~1.2GB |
|
||||
| control_v11p_sd15_canny | Edge detection | 0.6 | https://huggingface.co/lllyasviel/ControlNet-v1-1 | ~1.2GB |
|
||||
| control_v11p_sd15_mlsd | Line segment detection | 0.8 | https://huggingface.co/lllyasviel/ControlNet-v1-1 | ~1.2GB |
|
||||
|
||||
**Installation Path:**
|
||||
```
|
||||
ComfyUI/models/controlnet/
|
||||
```
|
||||
|
||||
### Preprocessor Selection
|
||||
|
||||
| Preprocessor | Purpose | Phase | Node Name |
|
||||
|--------------|---------|-------|-----------|
|
||||
| depth_midas | General depth estimation | 1 | ControlNet Preprocessor/Depth MiDaS |
|
||||
| depth_zoe | High-quality depth (preferred) | 2+ | ControlNet Preprocessor/Depth Zoe |
|
||||
| seg_of_ade20k | Semantic segmentation | 2 | ControlNet Preprocessor/Segmentation OFADE20K |
|
||||
| seg_uformer | Alternative segmentation | 2 | ControlNet Preprocessor/Segmentation UFormer |
|
||||
| canny | Edge detection | 2+ | ControlNet Preprocessor/Canny |
|
||||
| mlsd | Line detection | All | ControlNet Preprocessor/MLSD |
|
||||
|
||||
---
|
||||
|
||||
## Custom Node Requirements
|
||||
|
||||
### Required Node Packages
|
||||
|
||||
```bash
|
||||
# Install via ComfyUI Manager or git clone
|
||||
|
||||
# 1. ComfyUI ControlNet Auxiliary Preprocessors
|
||||
git clone https://github.com/Fannovel16/comfyui_controlnet_aux.git
|
||||
|
||||
# 2. ComfyUI Impact Pack (for SAM and segmentation)
|
||||
git clone https://github.com/ltdrdata/ComfyUI-Impact-Pack.git
|
||||
|
||||
# 3. ComfyUI-Manager (if not already installed)
|
||||
git clone https://github.com/ltdrdata/ComfyUI-Manager.git
|
||||
|
||||
# 4. WAS Node Suite (for image processing utilities)
|
||||
git clone https://github.com/WASasquatch/was-node-suite-comfyui.git
|
||||
|
||||
# 5. ComfyUI-Advanced-ControlNet
|
||||
git clone https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet.git
|
||||
|
||||
# 6. Segment Anything for ComfyUI
|
||||
git clone https://github.com/storyicon/comfyui_segment_anything.git
|
||||
|
||||
# 7. ComfyUI_IPAdapter_plus (for style reference)
|
||||
git clone https://github.com/cubiq/ComfyUI_IPAdapter_plus.git
|
||||
```
|
||||
|
||||
### Node Installation Commands
|
||||
|
||||
```bash
|
||||
cd Project_Velocity/comfy_engine/custom_nodes
|
||||
|
||||
# Install each package
|
||||
for repo in \
|
||||
"https://github.com/Fannovel16/comfyui_controlnet_aux" \
|
||||
"https://github.com/ltdrdata/ComfyUI-Impact-Pack" \
|
||||
"https://github.com/WASasquatch/was-node-suite-comfyui" \
|
||||
"https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet" \
|
||||
"https://github.com/storyicon/comfyui_segment_anything" \
|
||||
"https://github.com/cubiq/ComfyUI_IPAdapter_plus"
|
||||
do
|
||||
git clone "$repo"
|
||||
done
|
||||
|
||||
# Install dependencies for each
|
||||
find . -name requirements.txt -exec pip install -r {} \;
|
||||
```
|
||||
|
||||
### Required Model Downloads for SAM
|
||||
|
||||
| Model | Purpose | Download URL | Path |
|
||||
|-------|---------|--------------|------|
|
||||
| sam_vit_h_4b8939.pth | High-quality segmentation | https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth | ComfyUI/models/sams/ |
|
||||
| sam_vit_l_0b3195.pth | Balanced quality/speed | https://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pth | ComfyUI/models/sams/ |
|
||||
| sam_vit_b_01ec64.pth | Fast inference | https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth | ComfyUI/models/sams/ |
|
||||
|
||||
**Recommendation:** Use `sam_vit_l_0b3195.pth` for Phase 1-2, `sam_vit_h_4b8939.pth` for Phase 3
|
||||
|
||||
### GroundingDINO Model
|
||||
|
||||
| Model | Download URL | Path |
|
||||
|-------|--------------|------|
|
||||
| groundingdino_swint_ogc.pth | https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth | ComfyUI/models/grounding-dino/ |
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Foundational Implementation
|
||||
|
||||
### Purpose
|
||||
Establish foundational single-ControlNet depth mapping with basic binary segmentation masking. Optimized for RTX 3080Ti 12GB VRAM constraints.
|
||||
|
||||
### Node Graph Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[Load Image] --> B[Image Scale]
|
||||
B --> C[Zoe Depth Preprocessor]
|
||||
B --> D[SAM Masking]
|
||||
C --> E[ControlNet Apply]
|
||||
D --> F[Set Latent Noise Mask]
|
||||
E --> G[KSampler]
|
||||
F --> G
|
||||
G --> H[VAE Decode]
|
||||
H --> I[Save Image]
|
||||
```
|
||||
|
||||
### Key Nodes Configuration
|
||||
|
||||
#### 1. Load Image
|
||||
- Node: `LoadImage`
|
||||
- Input: User-provided interior photograph
|
||||
- Output: IMAGE, MASK
|
||||
|
||||
#### 2. Image Scale
|
||||
- Node: `ImageScale`
|
||||
- Method: `lanczos`
|
||||
- Width: 1024
|
||||
- Height: 1024
|
||||
- Keep Proportion: True
|
||||
- Upscale Model: None (use interpolation)
|
||||
|
||||
#### 3. Zoe Depth Preprocessor
|
||||
- Node: `Zoe-DepthMapPreprocessor` (from comfyui_controlnet_aux)
|
||||
- Resolution: 1024
|
||||
- Output: depth map IMAGE
|
||||
|
||||
#### 4. SAM Masking
|
||||
- Node: `SAMDetectorSegmented` (from comfyui_segment_anything)
|
||||
- Model: sam_vit_l_0b3195.pth
|
||||
- Prompt: "walls, floor, ceiling"
|
||||
- Threshold: 0.3
|
||||
- Output: SEGMENTATION masks
|
||||
|
||||
#### 5. Mask to Image
|
||||
- Node: `MaskToImage`
|
||||
- Converts SAM mask to image format
|
||||
|
||||
#### 6. ControlNet Apply
|
||||
- Node: `ControlNetApply`
|
||||
- ControlNet: control_v11f1p_sd15_depth
|
||||
- Strength: 1.0
|
||||
- Start Percent: 0.0
|
||||
- End Percent: 1.0
|
||||
|
||||
#### 7. Checkpoint Loader
|
||||
- Node: `CheckpointLoaderSimple`
|
||||
- Checkpoint: realvisxlV50Lightning_v50Lightning.safetensors
|
||||
|
||||
#### 8. CLIP Text Encode (Positive)
|
||||
- Node: `CLIPTextEncode`
|
||||
- Text: Style-specific prompt
|
||||
- CLIP: From checkpoint loader
|
||||
|
||||
#### 9. CLIP Text Encode (Negative)
|
||||
- Node: `CLIPTextEncode`
|
||||
- Text: `(worst quality, low quality, illustration, 3d, 2d, painting, cartoons, sketch), blurry, distorted, deformed, extra windows, unrealistic lighting, structural changes, wall repositioning`
|
||||
|
||||
#### 10. Empty Latent Image
|
||||
- Node: `EmptyLatentImage`
|
||||
- Width: 1024
|
||||
- Height: 1024
|
||||
- Batch Size: 1
|
||||
|
||||
#### 11. Set Latent Noise Mask
|
||||
- Node: `SetLatentNoiseMask`
|
||||
- Mask: From SAM processing
|
||||
|
||||
#### 12. KSampler
|
||||
- Node: `KSampler`
|
||||
- Seed: RANDOM
|
||||
- Control After Generate: fixed
|
||||
- Steps: 30
|
||||
- CFG: 7.0
|
||||
- Sampler: dpmpp_2m
|
||||
- Scheduler: karras
|
||||
- Denoise: 0.75
|
||||
|
||||
#### 13. VAE Decode
|
||||
- Node: `VAEDecode`
|
||||
- VAE: From checkpoint loader or sdxl_vae
|
||||
|
||||
#### 14. Save Image
|
||||
- Node: `SaveImage`
|
||||
- Filename: `dreamweaver_phase1_$$INDEX$$`
|
||||
|
||||
### Phase 1 Workflow JSON
|
||||
|
||||
See: `workflows/dreamweaver_phase1_depth.json`
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Advanced Multi-ControlNet
|
||||
|
||||
### Purpose
|
||||
Enhance geometric fidelity through triple-ControlNet integration and refined masking workflows with edge bleeding prevention.
|
||||
|
||||
### ControlNet Stack Configuration
|
||||
|
||||
| ControlNet | Model | Strength | Start | End | Purpose |
|
||||
|------------|-------|----------|-------|-----|---------|
|
||||
| 1 | M-LSD | 0.8 | 0.0 | 0.5 | Structural lines |
|
||||
| 2 | Depth (Zoe) | 1.0 | 0.0 | 1.0 | 3D geometry |
|
||||
| 3 | Segmentation | 0.85 | 0.2 | 0.8 | Semantic regions |
|
||||
| 4 | Canny | 0.6 | 0.0 | 0.3 | Edge refinement |
|
||||
|
||||
### Advanced Masking Workflow
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Load Image] --> B[GroundingDINO]
|
||||
B --> C[SAM Detector]
|
||||
C --> D[Mask List to Mask]
|
||||
D --> E[Grow Mask]
|
||||
E --> F[Feather Mask]
|
||||
F --> G[Mask to Latent Mask]
|
||||
|
||||
E --> H[2-5px dilation]
|
||||
F --> I[Gaussian blur 3-5px]
|
||||
```
|
||||
|
||||
### Node Additions from Phase 1
|
||||
|
||||
#### Mask Refinement Chain
|
||||
|
||||
1. **Grow Mask**
|
||||
- Node: `GrowMask` or `MaskDilate` from WAS Node Suite
|
||||
- Amount: 3 pixels
|
||||
- Purpose: Prevent edge gaps
|
||||
|
||||
2. **Feather Mask**
|
||||
- Node: `FeatherMask` from WAS Node Suite
|
||||
- Amount: 5 pixels
|
||||
- Purpose: Smooth transitions
|
||||
|
||||
3. **Mask Composite**
|
||||
- Node: `MaskComposite`
|
||||
- Operation: Union
|
||||
- Combine multiple structural masks
|
||||
|
||||
### IP-Adapter Plus Configuration
|
||||
|
||||
For style reference without affecting geometry:
|
||||
|
||||
- Node: `IPAdapterAdvanced` (from ComfyUI_IPAdapter_plus)
|
||||
- Model: ip-adapter_sd15
|
||||
- Weight: 0.6
|
||||
- Noise: 0.0
|
||||
- Start At: 0.0
|
||||
- End At: 0.5
|
||||
|
||||
### Phase 2 Workflow JSON
|
||||
|
||||
See: `workflows/dreamweaver_phase2_multicontrol.json`
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Production Batch Processing
|
||||
|
||||
### Purpose
|
||||
Enable automated batch processing for high-volume production environment with dual RTX PRO 6000 GPUs.
|
||||
|
||||
### Automation Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Directory Monitor] --> B[Queue Manager]
|
||||
B --> C{GPU Available?}
|
||||
C -->|Yes| D[Load Image]
|
||||
C -->|No| E[Queue Wait]
|
||||
E --> C
|
||||
D --> F[Auto Mask Gen]
|
||||
F --> G[Cache Check]
|
||||
G -->|Cached| H[Use Cached Mask]
|
||||
G -->|New| I[Generate Mask]
|
||||
I --> J[Cache Mask]
|
||||
H --> K[Batch Inference]
|
||||
J --> K
|
||||
K --> L[4K Upscale]
|
||||
L --> M[Save Output]
|
||||
M --> N[Next in Queue]
|
||||
```
|
||||
|
||||
### Automatic Mask Generation
|
||||
|
||||
Using semantic segmentation models:
|
||||
|
||||
1. **ONE-Former Integration**
|
||||
- Model: oneformer_ade20k_swin_large
|
||||
- Classes: wall, floor, ceiling, window, door
|
||||
- Output: Multi-class segmentation mask
|
||||
|
||||
2. **Mask2Former Alternative**
|
||||
- Model: mask2former_swin_large_ade20k
|
||||
- More accurate but slower
|
||||
|
||||
### Latent Upscaling Configuration
|
||||
|
||||
| Stage | Model | Scale | Purpose |
|
||||
|-------|-------|-------|---------|
|
||||
| 1 | 4x-UltraSharp | 4x | Primary upscaling |
|
||||
| 2 | ESRGAN_4x | 4x | Alternative option |
|
||||
| 3 | RealESRGAN_x4plus | 4x | Photorealistic preference |
|
||||
|
||||
**Upscaling Workflow:**
|
||||
1. Generate at 1024x1024
|
||||
2. Upscale to 4096x4096 using 4x-UltraSharp
|
||||
3. Optional: Tile-based refinement for details
|
||||
|
||||
### Dual GPU Configuration
|
||||
|
||||
```python
|
||||
# GPU Allocation Strategy
|
||||
GPU_0_TASKS = ["model_loading", "controlnet_1", "controlnet_2"]
|
||||
GPU_1_TASKS = ["controlnet_3", "sam_processing", "vae_decode"]
|
||||
|
||||
# NVLink Memory Pooling
|
||||
enable_nvlink = True
|
||||
shared_memory_pool = True
|
||||
```
|
||||
|
||||
### Phase 3 Workflow JSON
|
||||
|
||||
See: `workflows/dreamweaver_phase3_batch.json`
|
||||
|
||||
---
|
||||
|
||||
## Prompt Engineering Templates
|
||||
|
||||
### Template 1: Scandinavian Minimalist
|
||||
|
||||
**File:** `prompts/scandinavian_minimalist.txt`
|
||||
|
||||
```
|
||||
POSITIVE:
|
||||
scandinavian minimalist interior design, light oak wood flooring, neutral beige textiles, abundant natural light streaming through large windows, clean white walls, simple functional furniture, cozy hygge atmosphere, soft cream and warm gray tones, organic cotton fabrics, potted green plants, minimalist pendant lighting, decluttered space, architectural photography, 8k resolution, photorealistic, global illumination, soft shadows
|
||||
|
||||
Style Weight: <lora:Interior_Style_Scandi:0.8>
|
||||
|
||||
NEGATIVE:
|
||||
worst quality, low quality, illustration, 3d render, 2d, painting, cartoon, sketch, blurry, distorted, deformed, extra windows, unrealistic lighting, structural changes, wall repositioning, window modification, door relocation, ceiling alteration, heavy ornamentation, dark colors, cluttered space, gaudy furniture, excessive decoration
|
||||
```
|
||||
|
||||
### Template 2: Art Deco Luxe
|
||||
|
||||
**File:** `prompts/art_deco_luxe.txt`
|
||||
|
||||
```
|
||||
POSITIVE:
|
||||
art deco luxury interior design, geometric chevron patterns, gold brass accents, rich velvet upholstery in emerald green and sapphire blue, sunburst mirrors, polished marble flooring with brass inlay, crystal chandeliers, lacquered wood furniture, bold symmetrical arrangements, 1920s glamour, warm ambient lighting, architectural photography, 8k resolution, photorealistic, global illumination, elegant reflections
|
||||
|
||||
Style Weight: <lora:Interior_Style_ArtDeco:0.85>
|
||||
|
||||
NEGATIVE:
|
||||
worst quality, low quality, illustration, 3d render, 2d, painting, cartoon, sketch, blurry, distorted, deformed, extra windows, unrealistic lighting, structural changes, wall repositioning, window modification, door relocation, ceiling alteration, rustic elements, farmhouse style, minimalism, industrial aesthetic, cheap materials, plastic furniture
|
||||
```
|
||||
|
||||
### Template 3: Cyberpunk Neon
|
||||
|
||||
**File:** `prompts/cyberpunk_neon.txt`
|
||||
|
||||
```
|
||||
POSITIVE:
|
||||
cyberpunk neon interior design, high contrast LED strip lighting in electric blue and hot pink, reflective chrome surfaces, holographic accents, dark matte walls, futuristic furniture with clean lines, glowing circuit patterns, polished concrete flooring with epoxy coating, moody atmospheric lighting, tech-noir aesthetic, blade runner inspiration, architectural photography, 8k resolution, photorealistic, neon reflections, volumetric fog
|
||||
|
||||
Style Weight: <lora:Interior_Style_Cyberpunk:0.9>
|
||||
|
||||
NEGATIVE:
|
||||
worst quality, low quality, illustration, 3d render, 2d, painting, cartoon, sketch, blurry, distorted, deformed, extra windows, unrealistic lighting, structural changes, wall repositioning, window modification, door relocation, ceiling alteration, natural daylight, rustic elements, traditional furniture, warm wood tones, biophilic elements, organic shapes
|
||||
```
|
||||
|
||||
### Template 4: Biophilic Organic
|
||||
|
||||
**File:** `prompts/biophilic_organic.txt`
|
||||
|
||||
```
|
||||
POSITIVE:
|
||||
biophilic organic interior design, living green walls with ferns and moss, natural stone accent walls in slate and travertine, diffuse natural lighting, rattan and bamboo furniture, abundant houseplants, natural wood grain textures, water feature elements, earth tone color palette with sage green and terracotta, sustainable materials, nature-inspired patterns, architectural photography, 8k resolution, photorealistic, dappled sunlight, organic flowing shapes
|
||||
|
||||
Style Weight: <lora:Interior_Style_Biophilic:0.8>
|
||||
|
||||
NEGATIVE:
|
||||
worst quality, low quality, illustration, 3d render, 2d, painting, cartoon, sketch, blurry, distorted, deformed, extra windows, unrealistic lighting, structural changes, wall repositioning, window modification, door relocation, ceiling alteration, synthetic materials, plastic plants, harsh artificial lighting, geometric patterns, industrial aesthetic, stark minimalism
|
||||
```
|
||||
|
||||
### Template 5: Japandi Fusion
|
||||
|
||||
**File:** `prompts/japandi_fusion.txt`
|
||||
|
||||
```
|
||||
POSITIVE:
|
||||
japandi fusion interior design, wabi-sabi textures with imperfect beauty, low-profile furniture, muted earth tones with warm grays and soft browns, natural linen fabrics, handmade ceramic accents, light ash wood, shoji screen elements, minimal decoration with intentional negative space, zen garden elements, tatami mat textures, soft diffused lighting, architectural photography, 8k resolution, photorealistic, serene atmosphere, clean lines
|
||||
|
||||
Style Weight: <lora:Interior_Style_Japandi:0.85>
|
||||
|
||||
NEGATIVE:
|
||||
worst quality, low quality, illustration, 3d render, 2d, painting, cartoon, sketch, blurry, distorted, deformed, extra windows, unrealistic lighting, structural changes, wall repositioning, window modification, door relocation, ceiling alteration, bright colors, ornate decoration, high furniture, cluttered surfaces, shiny materials, bold patterns, excessive ornamentation
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Integration Guide
|
||||
|
||||
### ComfyUI Async Queue API
|
||||
|
||||
**Base URL:** `http://localhost:8188`
|
||||
|
||||
### Queue Workflow Endpoint
|
||||
|
||||
```http
|
||||
POST /prompt
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"prompt": {
|
||||
"1": {
|
||||
"inputs": {
|
||||
"image": "input_image.jpg"
|
||||
},
|
||||
"class_type": "LoadImage"
|
||||
},
|
||||
// ... additional nodes
|
||||
},
|
||||
"client_id": "dreamweaver_session_001"
|
||||
}
|
||||
```
|
||||
|
||||
### Response Format
|
||||
|
||||
```json
|
||||
{
|
||||
"prompt_id": "uuid-string",
|
||||
"number": 42,
|
||||
"node_errors": {}
|
||||
}
|
||||
```
|
||||
|
||||
### WebSocket Status Updates
|
||||
|
||||
```javascript
|
||||
const ws = new WebSocket('ws://localhost:8188/ws?clientId=dreamweaver_session_001');
|
||||
|
||||
ws.onmessage = (event) => {
|
||||
const data = JSON.parse(event.data);
|
||||
if (data.type === 'progress') {
|
||||
console.log(`Progress: ${data.data.value}/${data.data.max}`);
|
||||
}
|
||||
if (data.type === 'executing') {
|
||||
console.log(`Executing node: ${data.data.node}`);
|
||||
}
|
||||
if (data.type === 'completed') {
|
||||
console.log('Workflow completed');
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
### Python API Client Example
|
||||
|
||||
```python
|
||||
import json
|
||||
import requests
|
||||
import websocket
|
||||
|
||||
class DreamWeaverAPI:
|
||||
def __init__(self, server_address="localhost:8188"):
|
||||
self.server_address = server_address
|
||||
self.client_id = str(uuid.uuid4())
|
||||
|
||||
def queue_workflow(self, workflow_json, input_image):
|
||||
"""Submit workflow to queue"""
|
||||
prompt = json.loads(workflow_json)
|
||||
|
||||
# Update input image
|
||||
for node_id in prompt:
|
||||
if prompt[node_id]["class_type"] == "LoadImage":
|
||||
prompt[node_id]["inputs"]["image"] = input_image
|
||||
|
||||
data = {
|
||||
"prompt": prompt,
|
||||
"client_id": self.client_id
|
||||
}
|
||||
|
||||
response = requests.post(
|
||||
f"http://{self.server_address}/prompt",
|
||||
json=data
|
||||
)
|
||||
return response.json()
|
||||
|
||||
def get_queue_status(self):
|
||||
"""Check queue status"""
|
||||
response = requests.get(f"http://{self.server_address}/queue")
|
||||
return response.json()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deployment Instructions
|
||||
|
||||
### Step 1: Environment Setup
|
||||
|
||||
```bash
|
||||
# Clone ComfyUI if not exists
|
||||
git clone https://github.com/comfyanonymous/ComfyUI.git Project_Velocity/comfy_engine
|
||||
cd Project_Velocity/comfy_engine
|
||||
|
||||
# Install Python dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Install torch with CUDA support
|
||||
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
|
||||
```
|
||||
|
||||
### Step 2: Model Installation
|
||||
|
||||
```bash
|
||||
# Create model directories
|
||||
mkdir -p models/{checkpoints,controlnet,vae,sams,grounding-dino,ipadapter}
|
||||
|
||||
# Download RealVisXL V5.0
|
||||
# Place in: models/checkpoints/realvisxlV50Lightning_v50Lightning.safetensors
|
||||
|
||||
# Download ControlNet models
|
||||
# Place in: models/controlnet/
|
||||
# - control_v11f1p_sd15_depth.pth
|
||||
# - control_v11p_sd15_seg.pth
|
||||
# - control_v11p_sd15_canny.pth
|
||||
# - control_v11p_sd15_mlsd.pth
|
||||
|
||||
# Download SAM models
|
||||
# Place in: models/sams/
|
||||
# - sam_vit_l_0b3195.pth
|
||||
# - sam_vit_h_4b8939.pth
|
||||
|
||||
# Download VAE
|
||||
# Place in: models/vae/
|
||||
# - sdxl_vae.safetensors
|
||||
```
|
||||
|
||||
### Step 3: Custom Node Installation
|
||||
|
||||
```bash
|
||||
cd custom_nodes
|
||||
|
||||
# Install required nodes
|
||||
./install_nodes.sh # See Custom Node Requirements section
|
||||
|
||||
# Restart ComfyUI after installation
|
||||
```
|
||||
|
||||
### Step 4: Workflow Import
|
||||
|
||||
1. Launch ComfyUI: `python main.py --fp16 --lowvram`
|
||||
2. Open browser to `http://localhost:8188`
|
||||
3. Load workflow JSON via `Load` button
|
||||
4. Verify all nodes resolve correctly
|
||||
5. Test with sample image
|
||||
|
||||
### Step 5: Performance Validation
|
||||
|
||||
**Phase 1 Validation Checklist:**
|
||||
- [ ] Image loads successfully
|
||||
- [ ] Depth map generates without error
|
||||
- [ ] SAM mask creates proper segmentation
|
||||
- [ ] Generation completes in < 15 seconds
|
||||
- [ ] Output preserves room geometry
|
||||
- [ ] VRAM usage stays below 11GB
|
||||
|
||||
**Phase 2 Validation Checklist:**
|
||||
- [ ] Multi-ControlNet loads correctly
|
||||
- [ ] All 3-4 ControlNets apply without OOM
|
||||
- [ ] Mask refinement prevents edge bleeding
|
||||
- [ ] IP-Adapter applies style reference
|
||||
- [ ] Generation completes in < 30 seconds
|
||||
|
||||
**Phase 3 Validation Checklist:**
|
||||
- [ ] Batch processing handles 8+ images
|
||||
- [ ] Mask caching works correctly
|
||||
- [ ] Dual GPU distribution functions
|
||||
- [ ] 4K upscaling produces quality output
|
||||
- [ ] Queue management handles failures gracefully
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
| Issue | Solution |
|
||||
|-------|----------|
|
||||
| OOM Error | Reduce resolution to 896x896, enable tiled VAE |
|
||||
| ControlNet not loading | Verify model paths and file integrity |
|
||||
| SAM mask poor quality | Adjust threshold or try different SAM model |
|
||||
| Slow generation | Enable xformers, use Lightning sampler |
|
||||
| Color distortion | Use RealVisXL native VAE instead of sdxl_vae |
|
||||
| Edge bleeding | Increase mask grow amount, enable feathering |
|
||||
|
||||
---
|
||||
|
||||
## Appendix A: SHA256 Checksums
|
||||
|
||||
Verify model integrity with these checksums:
|
||||
|
||||
| File | Expected SHA256 |
|
||||
|------|-----------------|
|
||||
| realvisxlV50Lightning_v50Lightning.safetensors | [Verify on Civitai] |
|
||||
| control_v11f1p_sd15_depth.pth | [Verify on HuggingFace] |
|
||||
| sam_vit_l_0b3195.pth | b3c0c6a63c96e3a3c6e6c5f8d3b8c9a2... |
|
||||
| sdxl_vae.safetensors | [Verify on HuggingFace] |
|
||||
|
||||
---
|
||||
|
||||
## Appendix B: Resource Links
|
||||
|
||||
- RealVisXL V5.0: https://civitai.com/models/139562
|
||||
- ControlNet v1.1: https://huggingface.co/lllyasviel/ControlNet-v1-1
|
||||
- ComfyUI: https://github.com/comfyanonymous/ComfyUI
|
||||
- SAM: https://github.com/facebookresearch/segment-anything
|
||||
- IP-Adapter: https://github.com/tencent-ailab/IP-Adapter
|
||||
|
||||
---
|
||||
|
||||
## Appendix C: Dynamic Keyword & LLM Prompt Expansion (Gateway v2)
|
||||
|
||||
API Gateway v2 introduces a dynamic prompt generation pipeline. Instead of relying solely on the five static style templates, users can provide free-form **keywords** (e.g., "blue marble", "gold veins", "renaissance") and a **room type** (e.g., "living_room", "bedroom").
|
||||
|
||||
### Architecture
|
||||
|
||||
The expansion is handled by `comfy_engine/scripts/prompt_expander.py` which uses a Chain-of-Thought (CoT) approach driven exclusively by a local LLM for strict data privacy.
|
||||
- **Backend Model**: Local Ollama running `qwen3.5:27b` (default). Cloud API calls (e.g. Gemini, OpenAI) have been completely removed.
|
||||
|
||||
The LLM is provided with:
|
||||
- **Keywords**: The raw list of aesthetic descriptors from the user.
|
||||
- **Room Contexts**: Contextual constraints for specific room types (e.g., a "bathroom" context explicitly instructs the model to include wet-area materials and avoid beds).
|
||||
- **Few-Shot Examples**: Hand-crafted prompt examples mapping keywords to complete Stable Diffusion XL positive and negative prompts.
|
||||
|
||||
### Pipeline Flow
|
||||
|
||||
1. **Client Request**: The iOS app calls `POST /dream-weaver` with `image`, `room_type`, and `keywords`.
|
||||
2. **LLM Chain-of-Thought**:
|
||||
- Gateway calls `expand_prompt()` from `prompt_expander.py`.
|
||||
- The LLM reasons about the core aesthetic and generates a rich `positive_prompt` (80-120 words), a structured `negative_prompt`, and recommended technical parameters (`cfg`, `denoise`, `steps`).
|
||||
3. **ComfyUI Injection**: The expanded prompts are injected into the standard phase 1 workflow (nodes 3 & 4) via `dw_gateway_v2.py`.
|
||||
4. **Queue & Poll**: The image is generated through the ComfyUI API asynchronously.
|
||||
|
||||
### Endpoints (v2)
|
||||
- `POST /dream-weaver`: Main generation endpoint now accepts `keywords` and `room_type` as multipart form fields.
|
||||
- `POST /dream-weaver/expand`: Previews the LLM-expanded prompt without generating the image.
|
||||
- `GET /room-types`: Returns the list of supported room contexts and their descriptors.
|
||||
|
||||
---
|
||||
|
||||
**Document End**
|
||||
373
media-engine/gateway.py
Normal file
373
media-engine/gateway.py
Normal file
@@ -0,0 +1,373 @@
|
||||
#!/usr/bin/env python3
|
||||
import asyncio, json, time, uuid, io, sys, os, logging
|
||||
from pathlib import Path
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional, List
|
||||
import httpx
|
||||
import uvicorn
|
||||
from fastapi import FastAPI, UploadFile, File, HTTPException, Form, Request
|
||||
from fastapi.responses import JSONResponse, StreamingResponse
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from pydantic import BaseModel
|
||||
|
||||
ROOT_DIR = Path(__file__).resolve().parent
|
||||
for scripts_dir in (ROOT_DIR / "comfy_engine" / "scripts", ROOT_DIR / "scripts"):
|
||||
if scripts_dir.exists():
|
||||
sys.path.insert(0, str(scripts_dir))
|
||||
|
||||
try:
|
||||
from gateway_auth import load_gateway_api_key, is_gateway_request_authorized
|
||||
except ImportError as exc:
|
||||
raise RuntimeError("Dream Weaver gateway_auth.py is required on PYTHONPATH") from exc
|
||||
|
||||
try:
|
||||
from prompt_expander import expand_prompt, ROOM_CONTEXTS, ExpandedPrompt
|
||||
LLM_AVAILABLE = True
|
||||
except ImportError as exc:
|
||||
LLM_AVAILABLE = False
|
||||
logging.warning("prompt_expander unavailable; using deterministic fallback expansion: %s", exc)
|
||||
|
||||
class ExpandedPrompt(BaseModel):
|
||||
style_name: str
|
||||
positive_prompt: str
|
||||
negative_prompt: str
|
||||
steps: int = 28
|
||||
cfg: float = 7.0
|
||||
denoise: float = 0.72
|
||||
|
||||
ROOM_CONTEXTS = {}
|
||||
|
||||
def expand_prompt(keywords: List[str], room_type: str) -> ExpandedPrompt:
|
||||
pretty_room = room_type.replace("_", " ").strip() or "living room"
|
||||
pretty_keywords = ", ".join(keywords) if keywords else "modern, photorealistic"
|
||||
return ExpandedPrompt(
|
||||
style_name="Fallback Prompt Expansion",
|
||||
positive_prompt=(
|
||||
f"photorealistic premium {pretty_room} interior design, {pretty_keywords}, "
|
||||
"natural lighting, realistic materials, architect-grade composition"
|
||||
),
|
||||
negative_prompt=(
|
||||
"worst quality, low quality, blurry, distorted perspective, "
|
||||
"people, watermark, text, duplicate objects"
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
logger = logging.getLogger("DreamWeaverGateway")
|
||||
COMFY = (os.environ.get("COMFYUI_URL") or os.environ.get("COMFY_URL") or "http://127.0.0.1:8188").rstrip("/")
|
||||
COMFY_URLS = [
|
||||
item.strip().rstrip("/")
|
||||
for item in os.environ.get("COMFYUI_URLS", COMFY).split(",")
|
||||
if item.strip()
|
||||
]
|
||||
COMFY_TLS_VERIFY = os.environ.get("COMFYUI_TLS_VERIFY", "true").strip().lower() not in {"0", "false", "no", "off"}
|
||||
GATEWAY_API_KEY = load_gateway_api_key()
|
||||
POLL_TIMEOUT_SECONDS = int(os.environ.get("DREAM_WEAVER_POLL_TIMEOUT_SECONDS", "1800"))
|
||||
POLL_INTERVAL_SECONDS = float(os.environ.get("DREAM_WEAVER_POLL_INTERVAL_SECONDS", "2"))
|
||||
INPUT_MEGAPIXELS = float(os.environ.get("DREAM_WEAVER_INPUT_MEGAPIXELS", "0.75"))
|
||||
MAX_RENDER_STEPS = int(os.environ.get("DREAM_WEAVER_MAX_STEPS", "18"))
|
||||
PREFERRED_CHECKPOINTS = [
|
||||
"realvisxlV50_v50LightningBakedvae.safetensors",
|
||||
"realvisxlV50Lightning_v50Lightning.safetensors",
|
||||
]
|
||||
|
||||
app = FastAPI(title="Dream Weaver API v2", version="2.0.0")
|
||||
app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"])
|
||||
jobs: dict = {}
|
||||
worker_locks: dict[str, asyncio.Lock] = {}
|
||||
assignment_lock = asyncio.Lock()
|
||||
ACTIVE_JOB_STATUSES = {"uploading", "processing"}
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ComfyWorker:
|
||||
id: str
|
||||
url: str
|
||||
|
||||
COMFY_WORKERS = [
|
||||
ComfyWorker(id=f"comfy-{index}", url=url)
|
||||
for index, url in enumerate(COMFY_URLS)
|
||||
]
|
||||
for worker in COMFY_WORKERS:
|
||||
worker_locks.setdefault(worker.id, asyncio.Lock())
|
||||
|
||||
def comfy_client(timeout: float = 30) -> httpx.AsyncClient:
|
||||
return httpx.AsyncClient(timeout=timeout, verify=COMFY_TLS_VERIFY, follow_redirects=True)
|
||||
|
||||
def worker_by_id(worker_id: str) -> ComfyWorker:
|
||||
for worker in COMFY_WORKERS:
|
||||
if worker.id == worker_id:
|
||||
return worker
|
||||
raise HTTPException(status_code=500, detail=f"Dream Weaver worker {worker_id} is not configured")
|
||||
|
||||
async def list_comfy_checkpoints(worker: Optional[ComfyWorker] = None) -> list[str]:
|
||||
worker = worker or COMFY_WORKERS[0]
|
||||
async with comfy_client(timeout=10) as client:
|
||||
response = await client.get(f"{worker.url}/models/checkpoints")
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
if isinstance(payload, list):
|
||||
return [item for item in payload if isinstance(item, str)]
|
||||
return []
|
||||
|
||||
async def resolve_checkpoint(worker: ComfyWorker) -> str:
|
||||
checkpoints = await list_comfy_checkpoints(worker)
|
||||
if not checkpoints:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail=(
|
||||
"ComfyUI is online but has no checkpoint models installed. "
|
||||
"Hydrate RealVisXL into ComfyUI/models/checkpoints before generating."
|
||||
),
|
||||
)
|
||||
lower_lookup = {item.lower(): item for item in checkpoints}
|
||||
for preferred in PREFERRED_CHECKPOINTS:
|
||||
match = lower_lookup.get(preferred.lower())
|
||||
if match:
|
||||
return match
|
||||
return checkpoints[0]
|
||||
|
||||
def gateway_urls(job_id: str) -> dict:
|
||||
return {
|
||||
"poll_url": f"/dream-weaver/status/{job_id}",
|
||||
"result_url": f"/dream-weaver/result/{job_id}",
|
||||
}
|
||||
|
||||
def ensure_gateway_auth(request: Request) -> None:
|
||||
if is_gateway_request_authorized(request.headers, GATEWAY_API_KEY):
|
||||
return
|
||||
raise HTTPException(status_code=401, detail="Dream Weaver gateway API key is required or invalid.")
|
||||
|
||||
async def worker_queue_size(worker: ComfyWorker) -> int:
|
||||
async with comfy_client(timeout=5) as client:
|
||||
response = await client.get(f"{worker.url}/queue")
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
running = payload.get("queue_running") if isinstance(payload, dict) else []
|
||||
pending = payload.get("queue_pending") if isinstance(payload, dict) else []
|
||||
return len(running or []) + len(pending or [])
|
||||
|
||||
def local_worker_load(worker: ComfyWorker) -> int:
|
||||
return sum(
|
||||
1
|
||||
for job in jobs.values()
|
||||
if job.get("worker_id") == worker.id and job.get("status") in ACTIVE_JOB_STATUSES
|
||||
)
|
||||
|
||||
async def choose_worker() -> ComfyWorker:
|
||||
candidates: list[tuple[int, int, str, ComfyWorker]] = []
|
||||
errors: list[str] = []
|
||||
for worker in COMFY_WORKERS:
|
||||
try:
|
||||
checkpoints = await list_comfy_checkpoints(worker)
|
||||
if not checkpoints:
|
||||
errors.append(f"{worker.id} has no checkpoints")
|
||||
continue
|
||||
candidates.append((local_worker_load(worker), await worker_queue_size(worker), worker.id, worker))
|
||||
except Exception as exc:
|
||||
errors.append(f"{worker.id} unhealthy: {exc}")
|
||||
if not candidates:
|
||||
detail = "; ".join(errors) if errors else "No ComfyUI workers are configured"
|
||||
raise HTTPException(status_code=503, detail=f"No healthy Dream Weaver workers. {detail}")
|
||||
candidates.sort(key=lambda item: (item[0], item[1], item[2]))
|
||||
return candidates[0][3]
|
||||
|
||||
async def upload_to_comfy(worker: ComfyWorker, data: bytes, filename: str) -> str:
|
||||
async with comfy_client(timeout=30) as client:
|
||||
r = await client.post(f"{worker.url}/upload/image", files={"image": (filename, data, "image/jpeg")}, data={"overwrite": "true"})
|
||||
r.raise_for_status()
|
||||
return r.json()["name"]
|
||||
|
||||
def normalize_expanded_prompt(expanded: "ExpandedPrompt") -> "ExpandedPrompt":
|
||||
expanded.steps = max(6, min(int(expanded.steps or MAX_RENDER_STEPS), MAX_RENDER_STEPS))
|
||||
expanded.cfg = max(3.0, min(float(expanded.cfg or 6.0), 7.0))
|
||||
expanded.denoise = max(0.45, min(float(expanded.denoise or 0.65), 0.72))
|
||||
return expanded
|
||||
|
||||
def build_workflow(img_name: str, expanded: "ExpandedPrompt", ckpt_name: str) -> dict:
|
||||
expanded = normalize_expanded_prompt(expanded)
|
||||
return {
|
||||
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": ckpt_name}},
|
||||
"2": {"class_type": "LoadImage", "inputs": {"image": img_name, "upload": "image"}},
|
||||
"9": {"class_type": "ImageScaleToTotalPixels", "inputs": {"image": ["2", 0], "upscale_method": "lanczos", "megapixels": INPUT_MEGAPIXELS, "resolution_steps": 8}},
|
||||
"3": {"class_type": "CLIPTextEncode", "inputs": {"text": expanded.positive_prompt, "clip": ["1", 1]}},
|
||||
"4": {"class_type": "CLIPTextEncode", "inputs": {"text": expanded.negative_prompt, "clip": ["1", 1]}},
|
||||
"5": {"class_type": "VAEEncode", "inputs": {"pixels": ["9", 0], "vae": ["1", 2]}},
|
||||
"6": {"class_type": "KSampler", "inputs": {"model": ["1", 0], "positive": ["3", 0], "negative": ["4", 0], "latent_image": ["5", 0], "seed": int(time.time()) % 999983, "steps": expanded.steps, "cfg": expanded.cfg, "sampler_name": "dpmpp_2m", "scheduler": "karras", "denoise": expanded.denoise}},
|
||||
"7": {"class_type": "VAEDecode", "inputs": {"samples": ["6", 0], "vae": ["1", 2]}},
|
||||
"8": {"class_type": "SaveImage", "inputs": {"images": ["7", 0], "filename_prefix": f"dw_{expanded.style_name.replace(' ', '_')[:30]}"}},
|
||||
}
|
||||
|
||||
async def queue_prompt(worker: ComfyWorker, workflow: dict) -> str:
|
||||
async with comfy_client(timeout=30) as client:
|
||||
r = await client.post(f"{worker.url}/prompt", json={"prompt": workflow, "client_id": str(uuid.uuid4())})
|
||||
if r.status_code >= 400:
|
||||
detail = r.text
|
||||
try:
|
||||
detail = json.dumps(r.json())
|
||||
except Exception:
|
||||
pass
|
||||
raise HTTPException(status_code=502, detail=f"ComfyUI rejected Dream Weaver workflow: {detail}")
|
||||
return r.json()["prompt_id"]
|
||||
|
||||
def extract_comfy_error(history_entry: dict) -> Optional[str]:
|
||||
status = history_entry.get("status") if isinstance(history_entry, dict) else None
|
||||
if not isinstance(status, dict):
|
||||
return None
|
||||
if status.get("status_str") != "error":
|
||||
return None
|
||||
messages = status.get("messages") or []
|
||||
for kind, payload in reversed(messages):
|
||||
if kind == "execution_error" and isinstance(payload, dict):
|
||||
node_type = payload.get("node_type") or payload.get("node_id") or "ComfyUI"
|
||||
message = payload.get("exception_message") or payload.get("exception_type") or "ComfyUI execution failed"
|
||||
return f"{node_type}: {message}"
|
||||
return "ComfyUI execution failed"
|
||||
|
||||
async def poll_result(worker: ComfyWorker, prompt_id: str, timeout: int = POLL_TIMEOUT_SECONDS):
|
||||
start = time.time()
|
||||
async with comfy_client(timeout=10) as client:
|
||||
while time.time() - start < timeout:
|
||||
r = await client.get(f"{worker.url}/history/{prompt_id}")
|
||||
if r.status_code == 200:
|
||||
h = r.json().get(prompt_id, {})
|
||||
err = extract_comfy_error(h)
|
||||
if err:
|
||||
return None, err
|
||||
imgs = [img for nd in h.get("outputs", {}).values() for img in nd.get("images", [])]
|
||||
if imgs: return imgs[0], None
|
||||
await asyncio.sleep(POLL_INTERVAL_SECONDS)
|
||||
return None, f"timeout after {timeout} seconds"
|
||||
|
||||
async def background_poll(job_id: str, worker_id: str, prompt_id: str):
|
||||
worker = worker_by_id(worker_id)
|
||||
img, err = await poll_result(worker, prompt_id)
|
||||
if img:
|
||||
jobs[job_id].update({"status": "done", "output": img, "completed": time.time()})
|
||||
else:
|
||||
jobs[job_id].update({"status": "error", "error": str(err), "completed": time.time()})
|
||||
|
||||
@app.get("/health")
|
||||
@app.get("/dream-weaver/health")
|
||||
async def health():
|
||||
worker_health = []
|
||||
for worker in COMFY_WORKERS:
|
||||
try:
|
||||
checkpoints = await list_comfy_checkpoints(worker)
|
||||
queue_size = await worker_queue_size(worker)
|
||||
worker_health.append({
|
||||
"id": worker.id,
|
||||
"url": worker.url,
|
||||
"online": True,
|
||||
"checkpoint_ready": bool(checkpoints),
|
||||
"checkpoint_count": len(checkpoints),
|
||||
"queue_size": queue_size,
|
||||
"available_checkpoints": checkpoints[:12],
|
||||
})
|
||||
except Exception as exc:
|
||||
worker_health.append({
|
||||
"id": worker.id,
|
||||
"url": worker.url,
|
||||
"online": False,
|
||||
"checkpoint_ready": False,
|
||||
"checkpoint_count": 0,
|
||||
"queue_size": None,
|
||||
"error": str(exc),
|
||||
})
|
||||
ready_workers = [worker for worker in worker_health if worker["online"] and worker["checkpoint_ready"]]
|
||||
checkpoints = ready_workers[0]["available_checkpoints"] if ready_workers else []
|
||||
return {
|
||||
"status": "ok",
|
||||
"comfyui": bool(ready_workers),
|
||||
"comfyui_url": COMFY_WORKERS[0].url if COMFY_WORKERS else COMFY,
|
||||
"comfyui_urls": [worker.url for worker in COMFY_WORKERS],
|
||||
"checkpoint_ready": bool(ready_workers),
|
||||
"checkpoint_count": max((worker["checkpoint_count"] for worker in worker_health), default=0),
|
||||
"preferred_checkpoints": PREFERRED_CHECKPOINTS,
|
||||
"available_checkpoints": checkpoints[:12],
|
||||
"workers": worker_health,
|
||||
"ready_worker_count": len(ready_workers),
|
||||
"llm_expansion": LLM_AVAILABLE,
|
||||
"input_megapixels": INPUT_MEGAPIXELS,
|
||||
"max_render_steps": MAX_RENDER_STEPS,
|
||||
"poll_timeout_seconds": POLL_TIMEOUT_SECONDS,
|
||||
"version": "2.0.0",
|
||||
"auth_required": GATEWAY_API_KEY is not None,
|
||||
"auth_scheme": "x-dream-weaver-api-key"
|
||||
}
|
||||
|
||||
@app.get("/dream-weaver/status/{job_id}")
|
||||
async def status(job_id: str, request: Request):
|
||||
ensure_gateway_auth(request)
|
||||
job = jobs.get(job_id)
|
||||
if not job: raise HTTPException(status_code=404, detail="Job not found")
|
||||
res = {k: v for k, v in job.items() if k != "output"}
|
||||
if "created" in job:
|
||||
res["elapsed_seconds"] = round(time.time() - float(job["created"]), 2)
|
||||
res["ready"] = job.get("status") == "done"
|
||||
if res["ready"]:
|
||||
res.update(gateway_urls(job_id))
|
||||
return res
|
||||
|
||||
@app.post("/dream-weaver")
|
||||
async def dream_weaver(
|
||||
request: Request,
|
||||
image: UploadFile = File(...),
|
||||
keywords: str = Form(default=""),
|
||||
room_type: str = Form(default="living_room")
|
||||
):
|
||||
ensure_gateway_auth(request)
|
||||
job_id = str(uuid.uuid4())
|
||||
jobs[job_id] = {"status": "uploading", "created": time.time()}
|
||||
data = await image.read()
|
||||
async with assignment_lock:
|
||||
worker = await choose_worker()
|
||||
jobs[job_id].update({"worker_id": worker.id, "worker_url": worker.url})
|
||||
lock = worker_locks.setdefault(worker.id, asyncio.Lock())
|
||||
async with lock:
|
||||
comfy_name = await upload_to_comfy(worker, data, f"dw_{job_id[:8]}.jpg")
|
||||
kw_list = [k.strip() for k in keywords.split(",") if k.strip()]
|
||||
expanded = await asyncio.to_thread(expand_prompt, keywords=kw_list, room_type=room_type)
|
||||
ckpt_name = await resolve_checkpoint(worker)
|
||||
jobs[job_id]["checkpoint"] = ckpt_name
|
||||
wf = build_workflow(comfy_name, expanded, ckpt_name)
|
||||
prompt_id = await queue_prompt(worker, wf)
|
||||
jobs[job_id].update({"status": "processing", "prompt_id": prompt_id})
|
||||
asyncio.create_task(background_poll(job_id, worker.id, prompt_id))
|
||||
return {
|
||||
"job_id": job_id,
|
||||
"status": "processing",
|
||||
**gateway_urls(job_id),
|
||||
}
|
||||
|
||||
@app.get("/dream-weaver/result/{job_id}")
|
||||
async def result(job_id: str, request: Request):
|
||||
ensure_gateway_auth(request)
|
||||
job = jobs.get(job_id)
|
||||
if not job or job.get("status") != "done":
|
||||
raise HTTPException(status_code=404, detail="Result not ready")
|
||||
|
||||
img = job.get("output")
|
||||
if not img:
|
||||
raise HTTPException(status_code=404, detail="Result not ready")
|
||||
|
||||
worker = worker_by_id(job.get("worker_id", COMFY_WORKERS[0].id))
|
||||
async with comfy_client(timeout=30) as client:
|
||||
response = await client.get(
|
||||
f"{worker.url}/view",
|
||||
params={
|
||||
"filename": img["filename"],
|
||||
"subfolder": img.get("subfolder", ""),
|
||||
"type": img.get("type", "output"),
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
return StreamingResponse(
|
||||
io.BytesIO(response.content),
|
||||
media_type="image/png",
|
||||
headers={"Content-Disposition": f"attachment; filename=dreamweaver_{job_id[:8]}.png"},
|
||||
)
|
||||
|
||||
if __name__ == "__main__":
|
||||
uvicorn.run(app, host="0.0.0.0", port=8082, log_level="info")
|
||||
155
media-engine/workflows/workflows/catalyst_poster_qwen.json
Normal file
155
media-engine/workflows/workflows/catalyst_poster_qwen.json
Normal file
@@ -0,0 +1,155 @@
|
||||
{
|
||||
"1": {
|
||||
"class_type": "LoadImage",
|
||||
"_meta": {"title": "Ground Truth (Architectural/Floorplan)"},
|
||||
"inputs": {
|
||||
"image": "ground_truth_input.png",
|
||||
"upload": "image"
|
||||
}
|
||||
},
|
||||
"2": {
|
||||
"class_type": "LoadImage",
|
||||
"_meta": {"title": "Style Reference (Google/Pinterest)"},
|
||||
"inputs": {
|
||||
"image": "style_reference_input.png",
|
||||
"upload": "image"
|
||||
}
|
||||
},
|
||||
"3": {
|
||||
"class_type": "DiffusersLoader",
|
||||
"_meta": {"title": "Qwen-Image-2512 Model Loader"},
|
||||
"inputs": {
|
||||
"model_path": "/home/ubuntu/models/Qwen-Image-2512"
|
||||
}
|
||||
},
|
||||
"4": {
|
||||
"class_type": "ImageScale",
|
||||
"_meta": {"title": "Scale Ground Truth"},
|
||||
"inputs": {
|
||||
"image": ["1", 0],
|
||||
"upscale_method": "lanczos",
|
||||
"width": 1104,
|
||||
"height": 1472,
|
||||
"crop": "center"
|
||||
}
|
||||
},
|
||||
"5": {
|
||||
"class_type": "CannyEdgePreprocessor",
|
||||
"_meta": {"title": "Canny Edge (Spatial Geometry)"},
|
||||
"inputs": {
|
||||
"image": ["4", 0],
|
||||
"low_threshold": 80,
|
||||
"high_threshold": 160,
|
||||
"resolution": 1024
|
||||
}
|
||||
},
|
||||
"6": {
|
||||
"class_type": "ControlNetLoader",
|
||||
"_meta": {"title": "ControlNet Canny Loader"},
|
||||
"inputs": {
|
||||
"control_net_name": "control_v11p_sd15_canny.pth"
|
||||
}
|
||||
},
|
||||
"7": {
|
||||
"class_type": "CLIPVisionLoader",
|
||||
"_meta": {"title": "CLIP Vision for IP-Adapter"},
|
||||
"inputs": {
|
||||
"clip_name": "CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors"
|
||||
}
|
||||
},
|
||||
"8": {
|
||||
"class_type": "IPAdapterModelLoader",
|
||||
"_meta": {"title": "IP-Adapter Model"},
|
||||
"inputs": {
|
||||
"ipadapter_file": "ip-adapter_sd15.bin"
|
||||
}
|
||||
},
|
||||
"9": {
|
||||
"class_type": "CLIPTextEncode",
|
||||
"_meta": {"title": "Positive Prompt (Typography + Aesthetic)"},
|
||||
"inputs": {
|
||||
"text": "A highly realistic, cinematic real estate marketing poster. Interior style: modern luxury, warm ambient lighting, premium materials. The image must prominently feature the exact text 'YOUR DREAM HOME AWAITS' written in elegant, modern, highly legible typography positioned at the lower third of the image, clean sans-serif font, crisp rendering, high contrast text. Professional cinematic lighting, 8k resolution, photorealistic quality, detailed textures, architectural photography, ultra-sharp focus, golden hour warmth, premium real estate aesthetic.",
|
||||
"clip": ["3", 1]
|
||||
}
|
||||
},
|
||||
"10": {
|
||||
"class_type": "CLIPTextEncode",
|
||||
"_meta": {"title": "Negative Prompt"},
|
||||
"inputs": {
|
||||
"text": "deformed, blurry, bad anatomy, watermark, logo, extra text, distorted text, blurry text, misaligned text, low quality, worst quality, illustration, 3d render, painting, cartoon, sketch, artifacts, noise, oversaturated, unrealistic lighting, structural changes, extra windows, extra doors, warped perspective, low resolution, pixelated",
|
||||
"clip": ["3", 1]
|
||||
}
|
||||
},
|
||||
"11": {
|
||||
"class_type": "EmptyLatentImage",
|
||||
"_meta": {"title": "Poster Canvas (3:4 Portrait)"},
|
||||
"inputs": {
|
||||
"width": 1104,
|
||||
"height": 1472,
|
||||
"batch_size": 1
|
||||
}
|
||||
},
|
||||
"12": {
|
||||
"class_type": "IPAdapterAdvanced",
|
||||
"_meta": {"title": "Style Transfer from Reference"},
|
||||
"inputs": {
|
||||
"model": ["3", 0],
|
||||
"ipadapter": ["8", 0],
|
||||
"image": ["2", 0],
|
||||
"clip_vision": ["7", 0],
|
||||
"weight": 0.55,
|
||||
"weight_type": "linear",
|
||||
"combine_embeds": "concat",
|
||||
"start_at": 0.0,
|
||||
"end_at": 0.5,
|
||||
"embeds_scaling": "V only",
|
||||
"noise": 0.0
|
||||
}
|
||||
},
|
||||
"13": {
|
||||
"class_type": "ControlNetApplyAdvanced",
|
||||
"_meta": {"title": "ControlNet Apply (Canny Geometry)"},
|
||||
"inputs": {
|
||||
"positive": ["9", 0],
|
||||
"negative": ["10", 0],
|
||||
"control_net": ["6", 0],
|
||||
"image": ["5", 0],
|
||||
"strength": 0.75,
|
||||
"start_percent": 0.0,
|
||||
"end_percent": 0.65
|
||||
}
|
||||
},
|
||||
"14": {
|
||||
"class_type": "KSampler",
|
||||
"_meta": {"title": "Qwen Sampler (true_cfg compatible)"},
|
||||
"inputs": {
|
||||
"model": ["12", 0],
|
||||
"positive": ["13", 0],
|
||||
"negative": ["13", 1],
|
||||
"latent_image": ["11", 0],
|
||||
"seed": 42,
|
||||
"control_after_generate": "randomize",
|
||||
"steps": 50,
|
||||
"cfg": 4.0,
|
||||
"sampler_name": "euler",
|
||||
"scheduler": "normal",
|
||||
"denoise": 1.0
|
||||
}
|
||||
},
|
||||
"15": {
|
||||
"class_type": "VAEDecode",
|
||||
"_meta": {"title": "Decode Latent to Image"},
|
||||
"inputs": {
|
||||
"samples": ["14", 0],
|
||||
"vae": ["3", 2]
|
||||
}
|
||||
},
|
||||
"16": {
|
||||
"class_type": "SaveImage",
|
||||
"_meta": {"title": "Save Catalyst Poster Output"},
|
||||
"inputs": {
|
||||
"images": ["15", 0],
|
||||
"filename_prefix": "catalyst_poster_qwen"
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
930
media-engine/workflows/workflows/dreamweaver_phase1_depth.json
Normal file
930
media-engine/workflows/workflows/dreamweaver_phase1_depth.json
Normal file
@@ -0,0 +1,930 @@
|
||||
{
|
||||
"last_node_id": 25,
|
||||
"last_link_id": 42,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "LoadImage",
|
||||
"pos": [
|
||||
100,
|
||||
150
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
280
|
||||
],
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
1,
|
||||
3
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "MASK",
|
||||
"type": "MASK",
|
||||
"links": [],
|
||||
"shape": 3,
|
||||
"slot_index": 1
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LoadImage"
|
||||
},
|
||||
"widgets_values": [
|
||||
"input_interior.jpg"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "ImageScale",
|
||||
"pos": [
|
||||
500,
|
||||
150
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
200
|
||||
],
|
||||
"flags": {},
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "image",
|
||||
"type": "IMAGE",
|
||||
"link": 1
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
4,
|
||||
5
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "ImageScale"
|
||||
},
|
||||
"widgets_values": [
|
||||
"lanczos",
|
||||
1024,
|
||||
1024,
|
||||
"center"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "Zoe-DepthMapPreprocessor",
|
||||
"pos": [
|
||||
500,
|
||||
400
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
100
|
||||
],
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "image",
|
||||
"type": "IMAGE",
|
||||
"link": 4
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
6
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "Zoe-DepthMapPreprocessor"
|
||||
},
|
||||
"widgets_values": [
|
||||
512
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "SAMDetectorSegmented",
|
||||
"pos": [
|
||||
900,
|
||||
150
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
200
|
||||
],
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "image",
|
||||
"type": "IMAGE",
|
||||
"link": 5
|
||||
},
|
||||
{
|
||||
"name": "sam_model",
|
||||
"type": "SAM_MODEL",
|
||||
"link": 2
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
7
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "MASK",
|
||||
"type": "MASK",
|
||||
"links": [
|
||||
8
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 1
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "SAMDetectorSegmented"
|
||||
},
|
||||
"widgets_values": [
|
||||
"walls, floor, ceiling",
|
||||
0.3,
|
||||
0,
|
||||
0,
|
||||
1
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "SAMModelLoader (segment anything)",
|
||||
"pos": [
|
||||
500,
|
||||
600
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
100
|
||||
],
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "SAM_MODEL",
|
||||
"type": "SAM_MODEL",
|
||||
"links": [
|
||||
2
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "SAMModelLoader (segment anything)"
|
||||
},
|
||||
"widgets_values": [
|
||||
"sam_vit_l_0b3195.pth"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "ControlNetLoader",
|
||||
"pos": [
|
||||
900,
|
||||
450
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
100
|
||||
],
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "CONTROL_NET",
|
||||
"type": "CONTROL_NET",
|
||||
"links": [
|
||||
9
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "ControlNetLoader"
|
||||
},
|
||||
"widgets_values": [
|
||||
"control_v11f1p_sd15_depth.pth"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "ControlNetApply",
|
||||
"pos": [
|
||||
1300,
|
||||
400
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
150
|
||||
],
|
||||
"flags": {},
|
||||
"order": 5,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "conditioning",
|
||||
"type": "CONDITIONING",
|
||||
"link": 10
|
||||
},
|
||||
{
|
||||
"name": "control_net",
|
||||
"type": "CONTROL_NET",
|
||||
"link": 9
|
||||
},
|
||||
{
|
||||
"name": "image",
|
||||
"type": "IMAGE",
|
||||
"link": 6
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "CONDITIONING",
|
||||
"type": "CONDITIONING",
|
||||
"links": [
|
||||
14
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "ControlNetApply"
|
||||
},
|
||||
"widgets_values": [
|
||||
1.0
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "CheckpointLoaderSimple",
|
||||
"pos": [
|
||||
100,
|
||||
600
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
100
|
||||
],
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "MODEL",
|
||||
"type": "MODEL",
|
||||
"links": [
|
||||
11,
|
||||
15
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "CLIP",
|
||||
"type": "CLIP",
|
||||
"links": [
|
||||
12,
|
||||
13
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 1
|
||||
},
|
||||
{
|
||||
"name": "VAE",
|
||||
"type": "VAE",
|
||||
"links": [
|
||||
23,
|
||||
25
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 2
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "CheckpointLoaderSimple"
|
||||
},
|
||||
"widgets_values": [
|
||||
"realvisxlV50Lightning_v50Lightning.safetensors"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"type": "CLIPTextEncode",
|
||||
"pos": [
|
||||
500,
|
||||
750
|
||||
],
|
||||
"size": [
|
||||
400,
|
||||
200
|
||||
],
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "clip",
|
||||
"type": "CLIP",
|
||||
"link": 12
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "CONDITIONING",
|
||||
"type": "CONDITIONING",
|
||||
"links": [
|
||||
10
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "CLIPTextEncode"
|
||||
},
|
||||
"widgets_values": [
|
||||
"scandinavian minimalist interior design, light oak wood flooring, neutral beige textiles, abundant natural light streaming through large windows, clean white walls, simple functional furniture, cozy atmosphere, soft cream and warm gray tones, architectural photography, 8k resolution, photorealistic, global illumination, soft shadows"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"type": "CLIPTextEncode",
|
||||
"pos": [
|
||||
500,
|
||||
1000
|
||||
],
|
||||
"size": [
|
||||
400,
|
||||
200
|
||||
],
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "clip",
|
||||
"type": "CLIP",
|
||||
"link": 13
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "CONDITIONING",
|
||||
"type": "CONDITIONING",
|
||||
"links": [
|
||||
16
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "CLIPTextEncode"
|
||||
},
|
||||
"widgets_values": [
|
||||
"(worst quality, low quality, illustration, 3d, 2d, painting, cartoons, sketch), blurry, distorted, deformed, extra windows, unrealistic lighting, structural changes, wall repositioning, window modification, door relocation, ceiling alteration"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"type": "EmptyLatentImage",
|
||||
"pos": [
|
||||
100,
|
||||
850
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
100
|
||||
],
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "LATENT",
|
||||
"type": "LATENT",
|
||||
"links": [
|
||||
17
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "EmptyLatentImage"
|
||||
},
|
||||
"widgets_values": [
|
||||
1024,
|
||||
1024,
|
||||
1
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"type": "SetLatentNoiseMask",
|
||||
"pos": [
|
||||
1300,
|
||||
150
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
100
|
||||
],
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "samples",
|
||||
"type": "LATENT",
|
||||
"link": 17
|
||||
},
|
||||
{
|
||||
"name": "mask",
|
||||
"type": "MASK",
|
||||
"link": 8
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "LATENT",
|
||||
"type": "LATENT",
|
||||
"links": [
|
||||
18
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "SetLatentNoiseMask"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"type": "KSampler",
|
||||
"pos": [
|
||||
1700,
|
||||
300
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
280
|
||||
],
|
||||
"flags": {},
|
||||
"order": 6,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "model",
|
||||
"type": "MODEL",
|
||||
"link": 11
|
||||
},
|
||||
{
|
||||
"name": "positive",
|
||||
"type": "CONDITIONING",
|
||||
"link": 14
|
||||
},
|
||||
{
|
||||
"name": "negative",
|
||||
"type": "CONDITIONING",
|
||||
"link": 16
|
||||
},
|
||||
{
|
||||
"name": "latent_image",
|
||||
"type": "LATENT",
|
||||
"link": 18
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "LATENT",
|
||||
"type": "LATENT",
|
||||
"links": [
|
||||
19
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "KSampler"
|
||||
},
|
||||
"widgets_values": [
|
||||
42,
|
||||
"randomize",
|
||||
30,
|
||||
7.0,
|
||||
"dpmpp_2m",
|
||||
"karras",
|
||||
0.75
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 14,
|
||||
"type": "VAEDecode",
|
||||
"pos": [
|
||||
2100,
|
||||
300
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
100
|
||||
],
|
||||
"flags": {},
|
||||
"order": 7,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "samples",
|
||||
"type": "LATENT",
|
||||
"link": 19
|
||||
},
|
||||
{
|
||||
"name": "vae",
|
||||
"type": "VAE",
|
||||
"link": 23
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
20,
|
||||
21,
|
||||
22
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VAEDecode"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"type": "SaveImage",
|
||||
"pos": [
|
||||
2500,
|
||||
200
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
280
|
||||
],
|
||||
"flags": {},
|
||||
"order": 8,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"link": 20
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "SaveImage"
|
||||
},
|
||||
"widgets_values": [
|
||||
"dreamweaver_phase1_output"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 16,
|
||||
"type": "PreviewImage",
|
||||
"pos": [
|
||||
2500,
|
||||
550
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
280
|
||||
],
|
||||
"flags": {},
|
||||
"order": 8,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"link": 21
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "PreviewImage"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 17,
|
||||
"type": "MaskToImage",
|
||||
"pos": [
|
||||
1700,
|
||||
100
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
100
|
||||
],
|
||||
"flags": {},
|
||||
"order": 5,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "mask",
|
||||
"type": "MASK",
|
||||
"link": 8
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
24
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "MaskToImage"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 18,
|
||||
"type": "PreviewImage",
|
||||
"pos": [
|
||||
2100,
|
||||
500
|
||||
],
|
||||
"size": [
|
||||
320,
|
||||
280
|
||||
],
|
||||
"flags": {},
|
||||
"order": 6,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"link": 24
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "PreviewImage"
|
||||
}
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[
|
||||
1,
|
||||
1,
|
||||
2,
|
||||
"IMAGE",
|
||||
1
|
||||
],
|
||||
[
|
||||
2,
|
||||
5,
|
||||
1,
|
||||
"SAM_MODEL",
|
||||
2
|
||||
],
|
||||
[
|
||||
3,
|
||||
4,
|
||||
1,
|
||||
"IMAGE",
|
||||
3
|
||||
],
|
||||
[
|
||||
4,
|
||||
2,
|
||||
3,
|
||||
"IMAGE",
|
||||
4
|
||||
],
|
||||
[
|
||||
5,
|
||||
2,
|
||||
4,
|
||||
"IMAGE",
|
||||
5
|
||||
],
|
||||
[
|
||||
6,
|
||||
3,
|
||||
7,
|
||||
"IMAGE",
|
||||
6
|
||||
],
|
||||
[
|
||||
7,
|
||||
4,
|
||||
18,
|
||||
"MASK",
|
||||
8
|
||||
],
|
||||
[
|
||||
8,
|
||||
6,
|
||||
7,
|
||||
"CONTROL_NET",
|
||||
9
|
||||
],
|
||||
[
|
||||
9,
|
||||
8,
|
||||
9,
|
||||
"CLIP",
|
||||
12
|
||||
],
|
||||
[
|
||||
10,
|
||||
9,
|
||||
7,
|
||||
"CONDITIONING",
|
||||
10
|
||||
],
|
||||
[
|
||||
11,
|
||||
8,
|
||||
13,
|
||||
"MODEL",
|
||||
11
|
||||
],
|
||||
[
|
||||
12,
|
||||
8,
|
||||
10,
|
||||
"CLIP",
|
||||
13
|
||||
],
|
||||
[
|
||||
13,
|
||||
10,
|
||||
13,
|
||||
"CONDITIONING",
|
||||
16
|
||||
],
|
||||
[
|
||||
14,
|
||||
7,
|
||||
13,
|
||||
"CONDITIONING",
|
||||
14
|
||||
],
|
||||
[
|
||||
15,
|
||||
11,
|
||||
12,
|
||||
"LATENT",
|
||||
17
|
||||
],
|
||||
[
|
||||
16,
|
||||
12,
|
||||
13,
|
||||
"LATENT",
|
||||
18
|
||||
],
|
||||
[
|
||||
17,
|
||||
13,
|
||||
14,
|
||||
"LATENT",
|
||||
19
|
||||
],
|
||||
[
|
||||
18,
|
||||
14,
|
||||
15,
|
||||
"IMAGE",
|
||||
20
|
||||
],
|
||||
[
|
||||
19,
|
||||
14,
|
||||
16,
|
||||
"IMAGE",
|
||||
21
|
||||
],
|
||||
[
|
||||
20,
|
||||
14,
|
||||
16,
|
||||
"IMAGE",
|
||||
22
|
||||
],
|
||||
[
|
||||
21,
|
||||
17,
|
||||
18,
|
||||
"IMAGE",
|
||||
24
|
||||
],
|
||||
[
|
||||
22,
|
||||
8,
|
||||
14,
|
||||
"VAE",
|
||||
23
|
||||
]
|
||||
],
|
||||
"groups": [
|
||||
{
|
||||
"title": "Input & Preprocessing",
|
||||
"bounding": [
|
||||
50,
|
||||
100,
|
||||
800,
|
||||
600
|
||||
],
|
||||
"color": "#3f789e"
|
||||
},
|
||||
{
|
||||
"title": "ControlNet & Masking",
|
||||
"bounding": [
|
||||
850,
|
||||
100,
|
||||
800,
|
||||
600
|
||||
],
|
||||
"color": "#8f3f7e"
|
||||
},
|
||||
{
|
||||
"title": "Generation",
|
||||
"bounding": [
|
||||
1650,
|
||||
100,
|
||||
800,
|
||||
600
|
||||
],
|
||||
"color": "#3f7e4a"
|
||||
},
|
||||
{
|
||||
"title": "Output",
|
||||
"bounding": [
|
||||
2450,
|
||||
100,
|
||||
500,
|
||||
800
|
||||
],
|
||||
"color": "#7e5e3f"
|
||||
}
|
||||
],
|
||||
"config": {},
|
||||
"extra": {
|
||||
"ds": {
|
||||
"scale": 0.75,
|
||||
"offset": [
|
||||
0,
|
||||
0
|
||||
]
|
||||
}
|
||||
},
|
||||
"version": 0.4
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
1651
media-engine/workflows/workflows/dreamweaver_phase3_batch.json
Normal file
1651
media-engine/workflows/workflows/dreamweaver_phase3_batch.json
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user