# ControlNet

[ControlNet](https://huggingface.co/papers/2302.05543) mengarahkan model difusi yang telah dilatih sebelumnya menggunakan gambar kontrol struktural (garis tepi, kedalaman, atau pose) sambil Anda tetap menggunakan prompt teks. Model dasar dibekukan dan jaringan paralel ditambahkan, di mana residual-nya memandu denoiser, sehingga Anda mendapatkan kontrol tata letak tanpa perlu fine-tuning.

Muat ControlNet untuk kontrol yang Anda butuhkan (misalnya canny), lalu lewati sebagai `controlnet=` ke [from_pretrained()](/docs/diffusers/v0.43.0/id/api/pipelines/overview#diffusers.DiffusionPipeline.from_pretrained). Gunakan `controlnet_conditioning_scale` untuk mengatur seberapa kuat kontrol mengarahkan generasi.

> [!TIP]
> ControlNet tersedia untuk banyak model seperti [Flux](../api/pipelines/controlnet_flux), [Hunyuan-DiT](../api/pipelines/controlnet_hunyuandit), [Stable Diffusion 3](../api/pipelines/controlnet_sd3), dan lainnya. Contoh dalam panduan ini menggunakan Flux dan Stable Diffusion XL. Untuk T2I-Adapter pada checkpoint keluarga Stable Diffusion yang lebih lama, lihat [Legacy adapters](./legacy_adapters#t2i-adapter). Gunakan ControlNet untuk pekerjaan generasi yang dapat dikendalikan yang baru.

Tab di bawah ini menampilkan ControlNet untuk teks-ke-gambar, gambar-ke-gambar, dan inpainting.

Buat gambar canny dengan [opencv-python](https://github.com/opencv/opencv-python).

```py
import cv2
import numpy as np
from PIL import Image
from diffusers.utils import load_image

original_image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/non-enhanced-prompt.png"
)

image = np.array(original_image)

low_threshold = 100
high_threshold = 200

image = cv2.Canny(image, low_threshold, high_threshold)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
canny_image = Image.fromarray(image)
```

Lewatkan gambar canny ke pipeline. Gunakan parameter `controlnet_conditioning_scale` untuk menentukan seberapa besar bobot yang diberikan pada kontrol.

```py
import torch
from diffusers.utils import load_image
from diffusers import FluxControlNetPipeline, FluxControlNetModel

controlnet = FluxControlNetModel.from_pretrained(
    "InstantX/FLUX.1-dev-Controlnet-Canny", dtype=torch.bfloat16
)
pipeline = FluxControlNetPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev", controlnet=controlnet, dtype=torch.bfloat16
).to("cuda")  # or "mps", "xpu", "cpu"

prompt = """
A photorealistic overhead image of a cat reclining sideways in a flamingo pool floatie holding a margarita. 
The cat is floating leisurely in the pool and completely relaxed and happy.
"""

pipeline(
    prompt, 
    control_image=canny_image,
    controlnet_conditioning_scale=0.5,
    num_inference_steps=50, 
    guidance_scale=3.5,
).images[0]
```

  
    
    gambar asli
  
  
    
    canny image
  
  
    
    gambar yang dihasilkan
  

Buat peta kedalaman dengan pipeline estimasi kedalaman dari Transformers.

```py
import torch
import numpy as np
from PIL import Image
from transformers import DPTImageProcessor, DPTForDepthEstimation
from diffusers import ControlNetModel, StableDiffusionXLControlNetImg2ImgPipeline, AutoencoderKL
from diffusers.utils import load_image

device = "cuda"  # or "mps", "xpu", "cpu"
depth_estimator = DPTForDepthEstimation.from_pretrained("Intel/dpt-hybrid-midas").to(device)
feature_extractor = DPTImageProcessor.from_pretrained("Intel/dpt-hybrid-midas")

def get_depth_map(image):
    image = feature_extractor(images=image, return_tensors="pt").pixel_values.to(device)
    with torch.no_grad(), torch.autocast(device):
        depth_map = depth_estimator(image).predicted_depth

    depth_map = torch.nn.functional.interpolate(
        depth_map.unsqueeze(1),
        size=(1024, 1024),
        mode="bicubic",
        align_corners=False,
    )
    depth_min = torch.amin(depth_map, dim=[1, 2, 3], keepdim=True)
    depth_max = torch.amax(depth_map, dim=[1, 2, 3], keepdim=True)
    depth_map = (depth_map - depth_min) / (depth_max - depth_min)
    image = torch.cat([depth_map] * 3, dim=1)
    image = image.permute(0, 2, 3, 1).cpu().numpy()[0]
    image = Image.fromarray((image * 255.0).clip(0, 255).astype(np.uint8))
    return image

image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/non-enhanced-prompt.png"
).resize((1024, 1024))
depth_image = get_depth_map(image)
```

Lewatkan peta kedalaman ke pipeline. Gunakan parameter `controlnet_conditioning_scale` untuk menentukan seberapa besar bobot yang diberikan pada kontrol.

```py
controlnet = ControlNetModel.from_pretrained(
    "diffusers/controlnet-depth-sdxl-1.0-small",
    dtype=torch.float16,
)
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", dtype=torch.float16)
pipeline = StableDiffusionXLControlNetImg2ImgPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet,
    vae=vae,
    dtype=torch.float16,
).to(device)

prompt = """
A photorealistic overhead image of a cat reclining sideways in a flamingo pool floatie holding a margarita. 
The cat is floating leisurely in the pool and completely relaxed and happy.
"""
controlnet_conditioning_scale = 0.5
pipeline(
    prompt,
    image=image,
    control_image=depth_image,
    controlnet_conditioning_scale=controlnet_conditioning_scale,
    strength=0.99,
    num_inference_steps=100,
).images[0]
```

  
    
    gambar asli
  
  
    
    peta kedalaman
  
   
    
    gambar yang dihasilkan
  

Buat gambar mask yang menandai piksel mana yang akan diganti, lalu lewati sebagai `mask_image` bersama gambar kontrol.

```py
import cv2
import torch
import numpy as np
from PIL import Image
from diffusers.utils import load_image
from diffusers import StableDiffusionXLControlNetInpaintPipeline, ControlNetModel

init_image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/non-enhanced-prompt.png"
)
init_image = init_image.resize((1024, 1024))
mask_image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat_mask.png"
)
mask_image = mask_image.resize((1024, 1024))

def make_canny_condition(image):
    image = np.array(image)
    image = cv2.Canny(image, 100, 200)
    image = image[:, :, None]
    image = np.concatenate([image, image, image], axis=2)
    image = Image.fromarray(image)
    return image

control_image = make_canny_condition(init_image)
```

Lewatkan mask dan gambar kontrol ke pipeline. Gunakan parameter `controlnet_conditioning_scale` untuk menentukan seberapa besar bobot yang diberikan pada kontrol.

```py
controlnet = ControlNetModel.from_pretrained(
    "diffusers/controlnet-canny-sdxl-1.0", dtype=torch.float16
)
pipeline = StableDiffusionXLControlNetInpaintPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", controlnet=controlnet, dtype=torch.float16
).to("cuda")  # or "mps", "xpu", "cpu"
pipeline(
    "a cute and fluffy bunny rabbit",
    num_inference_steps=100,
    strength=0.99,
    controlnet_conditioning_scale=0.5,
    image=init_image,
    mask_image=mask_image,
    control_image=control_image,
).images[0]
```

  
    
    gambar asli
  
  
    
    mask gambar
  
   
    
    gambar yang dihasilkan
  

## Multi-ControlNet

Komposisikan beberapa ControlNet (misalnya canny + depth) dengan meneruskannya sebagai daftar. Topang (mask) wilayah yang tumpang tindih jika memungkinkan, dan atur setiap `controlnet_conditioning_scale`. Bangun `canny_image` dengan OpenCV Canny (seperti pada text-to-image) dan `depth_image` dengan DPT (seperti pada image-to-image). Ubah ukuran setiap gambar kontrol ke ukuran pipeline sebelum Anda memanggil pipeline.

```py
import cv2
import torch
import numpy as np
from PIL import Image
from transformers import DPTImageProcessor, DPTForDepthEstimation
from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel, AutoencoderKL
from diffusers.utils import load_image

original_image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/non-enhanced-prompt.png"
)
image = np.array(original_image)
canny_image = Image.fromarray(
    np.concatenate([cv2.Canny(image, 100, 200)[:, :, None]] * 3, axis=2)
)

device = "cuda"  # or "mps", "xpu", "cpu"
depth_estimator = DPTForDepthEstimation.from_pretrained("Intel/dpt-hybrid-midas").to(device)
feature_extractor = DPTImageProcessor.from_pretrained("Intel/dpt-hybrid-midas")

def get_depth_map(image):
    image = feature_extractor(images=image, return_tensors="pt").pixel_values.to(device)
    with torch.no_grad(), torch.autocast(device):
        depth_map = depth_estimator(image).predicted_depth

    depth_map = torch.nn.functional.interpolate(
        depth_map.unsqueeze(1),
        size=(1024, 1024),
        mode="bicubic",
        align_corners=False,
    )
    depth_min = torch.amin(depth_map, dim=[1, 2, 3], keepdim=True)
    depth_max = torch.amax(depth_map, dim=[1, 2, 3], keepdim=True)
    depth_map = (depth_map - depth_min) / (depth_max - depth_min)
    image = torch.cat([depth_map] * 3, dim=1)
    image = image.permute(0, 2, 3, 1).cpu().numpy()[0]
    image = Image.fromarray((image * 255.0).clip(0, 255).astype(np.uint8))
    return image

depth_image = get_depth_map(original_image.resize((1024, 1024)))

controlnets = [
    ControlNetModel.from_pretrained(
        "diffusers/controlnet-canny-sdxl-1.0", dtype=torch.float16
    ),
    ControlNetModel.from_pretrained(
        "diffusers/controlnet-depth-sdxl-1.0-small", dtype=torch.float16
    ),
]

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", dtype=torch.float16)
pipeline = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", controlnet=controlnets, vae=vae, dtype=torch.float16
).to(device)

prompt = """
a relaxed rabbit sitting on a striped towel next to a pool with a tropical drink nearby, 
bright sunny day, vacation scene, 35mm photograph, film, professional, 4k, highly detailed
"""
negative_prompt = "lowres, bad anatomy, worst quality, low quality, deformed, ugly"

images = [canny_image.resize((1024, 1024)), depth_image.resize((1024, 1024))]

pipeline(
    prompt,
    negative_prompt=negative_prompt,
    image=images,
    num_inference_steps=100,
    controlnet_conditioning_scale=[0.5, 0.5],
).images[0]
```

  
    
    canny image
  
  
    
    peta kedalaman
  
   
    
    gambar yang dihasilkan
  

## Mode tebak

[Mode tebak](https://github.com/lllyasviel/ControlNet/discussions/188) menghasilkan gambar dari kontrol saja ketika Anda tidak memiliki (atau tidak ingin menggunakan) prompt teks. Mode ini menskalakan residual ControlNet berdasarkan kedalaman blok, dari sekitar `0.1` di blok down awal hingga `1.0` di blok tengah. Hal ini mencegah lapisan awal terlalu membatasi, sementara blok yang lebih dalam tetap membawa struktur.

```py
import torch
from diffusers.utils import load_image
from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel

controlnet = ControlNetModel.from_pretrained(
  "diffusers/controlnet-canny-sdxl-1.0", dtype=torch.float16
)
pipeline = StableDiffusionXLControlNetPipeline.from_pretrained(
  "stabilityai/stable-diffusion-xl-base-1.0",
  controlnet=controlnet,
  dtype=torch.float16
).to("cuda")  # or "mps", "xpu", "cpu"

canny_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/canny-cat.png")
pipeline(
  "",
  image=canny_image,
  guess_mode=True
).images[0]
```

  
    
    canny image
  
  
    
    gambar yang dihasilkan