# HunyuanDiT2DControlNetModel

HunyuanDiT2DControlNetModel 是 [Hunyuan-DiT](https://huggingface.co/papers/2405.08748) 的 ControlNet 实现。

ControlNet 由 Lvmin Zhang、Anyi Rao 和 Maneesh Agrawala 在 [Adding Conditional Control to Text-to-Image Diffusion Models](https://huggingface.co/papers/2302.05543) 中提出。

使用 ControlNet 模型，您可以提供额外的控制图像来调节并控制 Hunyuan-DiT 的生成过程。例如，如果您提供一张深度图，ControlNet 模型将生成一张保留深度图中空间信息的图像。这是一种更灵活、更精确地控制图像生成过程的方式。

该论文的摘要如下：

*我们提出了 ControlNet，一种用于为大型预训练文本到图像扩散模型添加空间条件控制的神经网络架构。ControlNet 锁定生产就绪的大型扩散模型，并复用其使用数十亿张图像预训练得到的深层且鲁棒的编码层作为强大的骨干网络，以学习多样化的条件控制。该神经架构通过“零卷积”（零初始化的卷积层）进行连接，这些层使参数从零开始逐步增长，并确保在微调过程中不会产生有害噪声。我们使用 Stable Diffusion 测试了多种条件控制，例如边缘、深度、分割、人体姿态等，支持使用单一或多个条件，以及有或无提示词的情况。我们展示了 ControlNet 的训练在小规模（<50k）和大规模（>1m）数据集上均具有鲁棒性。大量结果表明，ControlNet 有助于将图像扩散模型的应用扩展到更广泛的领域。*

此代码由腾讯混元团队实现。您可以在 [腾讯混元](https://huggingface.co/Tencent-Hunyuan) 上找到 Hunyuan-DiT ControlNet 的预训练检查点。

## 加载 HunyuanDiT2DControlNetModel 的示例

```py
from diffusers import HunyuanDiT2DControlNetModel
import torch
controlnet = HunyuanDiT2DControlNetModel.from_pretrained("Tencent-Hunyuan/HunyuanDiT-v1.1-ControlNet-Diffusers-Pose", dtype=torch.float16)
```

## HunyuanDiT2DControlNetModel[[diffusers.HunyuanDiT2DControlNetModel]]

#### diffusers.HunyuanDiT2DControlNetModel[[diffusers.HunyuanDiT2DControlNetModel]]

```python
diffusers.HunyuanDiT2DControlNetModel(conditioning_channels: int = 3, num_attention_heads: int = 16, attention_head_dim: int = 88, in_channels: int | None = None, patch_size: int | None = None, activation_fn: str = 'gelu-approximate', sample_size = 32, hidden_size = 1152, transformer_num_layers: int = 40, mlp_ratio: float = 4.0, cross_attention_dim: int = 1024, cross_attention_dim_t5: int = 2048, pooled_projection_dim: int = 1024, text_len: int = 77, text_len_t5: int = 256, use_style_cond_and_image_meta_size: bool = True)
```

[源代码](https://github.com/huggingface/diffusers/blob/v0.43.0/src/diffusers/models/controlnets/controlnet_hunyuan.py#L40)

#### forward[[diffusers.HunyuanDiT2DControlNetModel.forward]]

```python
forward(hidden_states, timestep, controlnet_cond: Tensor, conditioning_scale: float = 1.0, encoder_hidden_states = None, text_embedding_mask = None, encoder_hidden_states_t5 = None, text_embedding_mask_t5 = None, image_meta_size = None, style = None, image_rotary_emb = None, return_dict = True)
```

[源代码](https://github.com/huggingface/diffusers/blob/v0.43.0/src/diffusers/models/controlnets/controlnet_hunyuan.py#L215)

**参数：**

hidden_states (`torch.Tensor`，形状为 `(batch size, dim, height, width)`)：输入张量。

timestep ( `torch.LongTensor`, *可选*) : 用于指示去噪步骤。

controlnet_cond ( `torch.Tensor` ) ：ControlNet 的条件输入。

conditioning_scale ( `float` ) : 指示条件缩放比例。

encoder_hidden_states ( `torch.Tensor`，形状为 `(batch size, sequence len, embed dims)`，*可选*) : 用于交叉注意力层的条件嵌入。这是 `BertModel` 的输出。

text_embedding_mask : torch.Tensor 形状为 `(batch, key_tokens)` 的注意力掩码会应用于 `encoder_hidden_states`。这是 `BertModel` 的输出。

encoder_hidden_states_t5 ( `torch.Tensor`，形状为 `(batch size, sequence len, embed dims)`，*可选*)：用于交叉注意力层的条件嵌入。这是 T5 文本编码器的输出。

text_embedding_mask_t5 : torch.Tensor 形状为 `(batch, key_tokens)` 的注意力掩码会应用于 `encoder_hidden_states`。这是 T5 文本编码器的输出。

image_meta_size (torch.Tensor) ：条件嵌入，指示图像尺寸

style : torch.Tensor: 指示样式的条件嵌入

image_rotary_emb (`torch.Tensor`) : 在注意力计算过程中应用于查询和键张量的图像旋转嵌入。

return_dict : bool 是否返回字典。

**返回值：**

如果 `return_dict` 为 True，则返回 `HunyuanControlNetOutput`，否则返回一个 `tuple`，其中第一个元素是 ControlNet 块样本列表。

[HunyuanDiT2DControlNetModel](/docs/diffusers/v0.43.0/zh/api/models/controlnet_hunyuandit#diffusers.HunyuanDiT2DControlNetModel) 的前向方法。

#### set_attn_processor[[diffusers.HunyuanDiT2DControlNetModel.set_attn_processor]]

```python
set_attn_processor(processor: diffusers.models.attention_processor.AttnProcessor | diffusers.models.attention_processor.CustomDiffusionAttnProcessor | diffusers.models.attention_processor.AttnAddedKVProcessor | diffusers.models.attention_processor.AttnAddedKVProcessor2_0 | diffusers.models.attention_processor.JointAttnProcessor2_0 | diffusers.models.attention_processor.PAGJointAttnProcessor2_0 | diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0 | diffusers.models.attention_processor.FusedJointAttnProcessor2_0 | diffusers.models.attention_processor.AllegroAttnProcessor2_0 | diffusers.models.attention_processor.AuraFlowAttnProcessor2_0 | diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0 | diffusers.models.attention_processor.FluxAttnProcessor2_0 | diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU | diffusers.models.attention_processor.FusedFluxAttnProcessor2_0 | diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU | diffusers.models.attention_processor.CogVideoXAttnProcessor2_0 | diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0 | diffusers.models.attention_processor.XFormersAttnAddedKVProcessor | diffusers.models.attention_processor.XFormersAttnProcessor | diffusers.models.attention_processor.XLAFlashAttnProcessor2_0 | diffusers.models.attention_processor.AttnProcessorNPU | diffusers.models.attention_processor.AttnProcessor2_0 | diffusers.models.attention_processor.MochiVaeAttnProcessor2_0 | diffusers.models.attention_processor.MochiAttnProcessor2_0 | diffusers.models.attention_processor.StableAudioAttnProcessor2_0 | diffusers.models.attention_processor.HunyuanAttnProcessor2_0 | diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0 | diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0 | diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0 | diffusers.models.attention_processor.LuminaAttnProcessor2_0 | diffusers.models.attention_processor.FusedAttnProcessor2_0 | diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor | diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0 | diffusers.models.attention_processor.SlicedAttnProcessor | diffusers.models.attention_processor.SlicedAttnAddedKVProcessor | diffusers.models.attention_processor.SanaLinearAttnProcessor2_0 | diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0 | diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0 | diffusers.models.attention_processor.SanaMultiscaleLinearAttention | diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0 | diffusers.models.attention_processor.SanaMultiscaleAttentionProjection | diffusers.models.attention_processor.IPAdapterAttnProcessor | diffusers.models.attention_processor.IPAdapterAttnProcessor2_0 | diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor | diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0 | diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0 | diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0 | diffusers.models.attention_processor.LoRAAttnProcessor | diffusers.models.attention_processor.LoRAAttnProcessor2_0 | diffusers.models.attention_processor.LoRAXFormersAttnProcessor | diffusers.models.attention_processor.LoRAAttnAddedKVProcessor | dict[str, diffusers.models.attention_processor.AttnProcessor | diffusers.models.attention_processor.CustomDiffusionAttnProcessor | diffusers.models.attention_processor.AttnAddedKVProcessor | diffusers.models.attention_processor.AttnAddedKVProcessor2_0 | diffusers.models.attention_processor.JointAttnProcessor2_0 | diffusers.models.attention_processor.PAGJointAttnProcessor2_0 | diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0 | diffusers.models.attention_processor.FusedJointAttnProcessor2_0 | diffusers.models.attention_processor.AllegroAttnProcessor2_0 | diffusers.models.attention_processor.AuraFlowAttnProcessor2_0 | diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0 | diffusers.models.attention_processor.FluxAttnProcessor2_0 | diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU | diffusers.models.attention_processor.FusedFluxAttnProcessor2_0 | diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU | diffusers.models.attention_processor.CogVideoXAttnProcessor2_0 | diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0 | diffusers.models.attention_processor.XFormersAttnAddedKVProcessor | diffusers.models.attention_processor.XFormersAttnProcessor | diffusers.models.attention_processor.XLAFlashAttnProcessor2_0 | diffusers.models.attention_processor.AttnProcessorNPU | diffusers.models.attention_processor.AttnProcessor2_0 | diffusers.models.attention_processor.MochiVaeAttnProcessor2_0 | diffusers.models.attention_processor.MochiAttnProcessor2_0 | diffusers.models.attention_processor.StableAudioAttnProcessor2_0 | diffusers.models.attention_processor.HunyuanAttnProcessor2_0 | diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0 | diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0 | diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0 | diffusers.models.attention_processor.LuminaAttnProcessor2_0 | diffusers.models.attention_processor.FusedAttnProcessor2_0 | diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor | diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0 | diffusers.models.attention_processor.SlicedAttnProcessor | diffusers.models.attention_processor.SlicedAttnAddedKVProcessor | diffusers.models.attention_processor.SanaLinearAttnProcessor2_0 | diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0 | diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0 | diffusers.models.attention_processor.SanaMultiscaleLinearAttention | diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0 | diffusers.models.attention_processor.SanaMultiscaleAttentionProjection | diffusers.models.attention_processor.IPAdapterAttnProcessor | diffusers.models.attention_processor.IPAdapterAttnProcessor2_0 | diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor | diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0 | diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0 | diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0 | diffusers.models.attention_processor.LoRAAttnProcessor | diffusers.models.attention_processor.LoRAAttnProcessor2_0 | diffusers.models.attention_processor.LoRAXFormersAttnProcessor | diffusers.models.attention_processor.LoRAAttnAddedKVProcessor])
```

[源代码](https://github.com/huggingface/diffusers/blob/v0.43.0/src/diffusers/models/controlnets/controlnet_hunyuan.py#L141)

**参数：**

processor (`dict` 类型的 `AttentionProcessor` 或单个 `AttentionProcessor`)：将作为所有 **所有** `Attention` 层的处理器设置的已实例化的处理器类或处理器类字典。如果 `processor` 是字典，其键需要定义到对应交叉注意力处理器的路径。在设置可训练注意力处理器时，强烈建议采用此方式。

设置用于计算注意力的注意力处理器。