# PANODIFFUSION: 360-DEGREE PANORAMA OUT-PAINTING VIA DIFFUSION

Tianhao Wu1, Chuanxia Zheng2 & Tat-Jen Cham1

1Nanyang Technological University tianhao001@e.ntu.edu.sg, astjcham@ntu.edu.sg 2University of Oxford cxzheng@robots.ox.ac.uk

# ABSTRACT

Generating complete 360° panoramas from narrow field of view images is ongoing research as omnidirectional RGB data is not readily available. Existing GAN-based approaches face some barriers to achieving higher quality output, and have poor generalization performance over different mask types. In this paper, we present our 360° indoor RGB-D panorama outpainting model using latent diffusion models (LDM), called PanoDiffusion. We introduce a new bi-modal latent diffusion structure that utilizes both RGB and depth panoramic data during training, which works surprisingly well to outpaint depth-free RGB images during inference. We further propose a novel technique of introducing progressive camera rotations during each diffusion denoising step, which leads to substantial improvement in achieving panorama wraparound consistency. Results show that our PanoDiffusion not only significantly outperforms state-of-the-art methods on RGB-D panorama outpainting by producing diverse well-structured results for different types of masks, but can also synthesize high-quality depth panoramas to provide realistic 3D indoor models.

# 1 INTRODUCTION

Omnidirectional 360° panoramas serve as invaluable assets in various applications, such as lighting estimation (Gardner et al., 2017; 2019; Song & Funkhouser, 2019) and new scene synthesis (Somanath & Kurz, 2021) in the Augmented and Virtual Reality (AR & VR). However, an obvious limitation is that capturing, collecting, and restoring a dataset with 360° images is a high-effort and high-cost undertaking (Akimoto et al., 2019; 2022), while manually creating a 3D space from scratch can be a demanding task (Lee et al., 2017; Choi et al., 2015; Newcombe et al., 2011).

![](images/3d83ba30c671eb3e3c48b8317dcd338484ef61989c68dab59fe4e1101f165fb8.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern room with blue lighting and interior furniture (no visible text or symbols)
</details>

(a) Masked input

![](images/d5a1eb9344e5bc4db7a31fcbc8e3206733f31ec64a5a38750688939cf39a68c3.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior views of a modern living room with curved furniture and ambient lighting (no visible text or symbols)
</details>

(b) Ours PanoDiffusion

![](images/69976f163a4cc2572bb3c86feddb5e86f9c584e178dde8f4514c462ebe59c59e.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a modern minimalist space with curved white walls, circular design, and minimal furniture (no visible text or symbols)
</details>

(c) BIPS

![](images/4848592a0805815666b94a243b9c24f615381dadd4d808def3ee46bb57649bf4.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a modern living room with curved ceiling design, featuring minimalist furniture and digital screens (no visible text or symbols)
</details>

(d) OmniDreamer   
Figure 1: Example results of 360◦ Panorama Outpainting on various masks. Compared to BIPS (Oh et al., 2022) and OmniDreamer (Akimoto et al., 2022), our model not only effectively generates semantically meaningful content and plausible appearances with many objects, such as beds, sofas and TV’s, but also provides multiple and diverse solutions for this ill-posed problem. (Masked regions are shown in blue for better visualization. Zoom in to see the details.)

To mitigate this dataset issue, the latest learning methods (Akimoto et al., 2019; Somanath & Kurz, 2021; Akimoto et al., 2022; Oh et al., 2022) have been proposed, with a specific focus on generating omnidirectional RGB panoramas from narrow field of view (NFoV) images. These methods are typically built upon Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), which have shown remarkable success in creating new content. However, GAN architectures face some notable problems, including 1) mode collapse (seen in Fig. 1(c)), 2) unstable training (Salimans et al., 2016), and 3) difficulty in generating multiple structurally reasonable objects (Epstein et al., 2022). These limitations lead to obvious artifacts in synthesizing complex scenes (Fig. 1).

The recent endeavors of (Lugmayr et al., 2022; Li et al., 2022; Xie et al., 2023; Wang et al., 2023) directly adopt the latest latent diffusion models (LDMs) (Rombach et al., 2022) in image inpainting tasks, which achieve a stable training of generative models and spatially consistent images. However, specifically for a 360° panorama outpainting scenario, these inpainting works usually lead to grossly distorted results. This is because: 1) the missing (masked) regions in 360° panorama outpainting is generally much larger than masks in normal inpainting and 2) it necessitates generating semantically reasonable objects within a given scene, as opposed to merely filling in generic background textures in an empty room (as shown in Fig. 1 (c)). To achieve this, we propose an alternative method for 360° indoor panorama outpainting via the latest latent diffusion models (LDMs) (Rombach et al., 2022), termed as PanoDiffusion. Unlike existing diffusion-based inpainting methods, we introduce depth information through a novel bi-modal latent diffusion structure during the training, which is also significantly different from the latest concurrent works (Tang et al., 2023; Lu et al., 2023) that aims for text-guided 360° panorama image generation. Our key motivation for doing so is that the depth information is crucial for helping the network understand the physical structure of objects and the layout of the scene (Ren et al., 2012). It is worth noting that our model only uses partially visible RGB images as input during inference, without requirement for any depth information, yet achieving significant improvement on both RGB and depth synthesis (Tables 1 and 2).

Another distinctive challenge in this task stems from the unique characteristic of panorama images: 3) both ends of the image must be aligned to ensure the integrity and wraparound consistency of the entire space, given that the indoor space lacks a definitive starting and ending point. To enhance this property in the generated results, we introduce two strategies: First, during the training process, a camera-rotation approach is employed to randomly crop and stitch the images for data augmentation. It encourages the networks to capture information from different views in a 360° panorama. Second, a two-end alignment mechanism is applied at each step of the inference denoising process (Fig. 4), which explicitly enforces the two ends of an image to be wraparound-consistent.

We evaluate the proposed method on the Structured3D dataset (Zheng et al., 2020). Experimental results demonstrate that our PanoDiffusion not only significantly outperforms previous state-of-theart methods, but is also able to provide multiple and diverse well-structured results for different types of masks (Fig. 1). In summary, our main contributions are as follows:

• A new bi-modal latent diffusion structure that utilizes both RGB and depth panoramic data to better learn spatial layouts and patterns during training, but works surprisingly well to outpaint normal RGB-D panoramas during inference, even without depth input;   
• A novel technique of introducing progressive camera rotations during each diffusion denoising step, which leads to substantial improvement in achieving panorama wraparound consistency;   
• With either partially or fully visible RGB inputs, our PanoDiffusion can synthesize highquality indoor RGB-D panoramas simultaneously to provide realistic 3D indoor models.

# 2 RELATED WORK

# 2.1 IMAGE INPAINTING/OUTPAINTING

Driven by advances in various generative models, such as VAEs (Kingma & Welling, 2014) and GANs (Goodfellow et al., 2014), a series of learning-based approaches (Pathak et al., 2016; Iizuka et al., 2017; Yu et al., 2018; Zheng et al., 2019; Zhao et al., 2020; Wan et al., 2021; Zheng et al., 2022) have been proposed to generate semantically meaningful content from a partially visible image. More recently, state-of-the-art methods (Lugmayr et al., 2022; Li et al., 2022; Xie et al., 2023;

![](images/14cb532fb50f59a9bf155c30c04291f30105210b0440922ff2045fb5ab2df734.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    A["Training Stage: x殼b, ε1, Z殼b"] --> B["Denoising Process"]
    B --> C["Refine Stage: z̃_rgb, D₁"]
    C --> D["SR"]
    D --> E["Output"]
    
    subgraph Training Stage
        F["x_d, ε2, Z_d"]
        G["z_T, ε_T"]
    end
    
    subgraph Inference Stage
        H["x_γgb, ε1, Z_γgb"]
        I["z_γgb, ε1, Z_γgb"]
        J["z_γgbd, ε2, Z_γgbd"]
        K["z_γgbd, ε2, Z_γgbd"]
        L["z_T, ε_T"]
    end
    
    style Training Stage fill:#f9f9f9,stroke:#333
    style Inference Stage fill:#f9f9f9,stroke:#333
```
</details>

Figure 2: The overall pipeline of our proposed PanoDiffusion method. (a) During training, the model is optimized for RGB-D panorama synthesis, without the mask. (b) During inference, however, the depth information is no longer needed for masked panorama outpainting. (c) Finally, a super-resolution model is implemented to further enhance the high-resolution outpainting. We only show the input/output of each stage and omit the details of circular shift and adding noise. Note that the VQ-based encoder-decoders are pre-trained in advance, and fixed in the rest of our framework.

Wang et al., 2023) directly adopt the popular diffusion models (Rombach et al., 2022) for image inpainting, achieving high-quality completed images with consistent structure and diverse content. However, these diffusion-based models either focus on background inpainting, or require input text as guidance to produce plausible objects within the missing regions. This points to an existing gap in achieving comprehensive and contextually rich inpainting/outpainting results across a wider spectrum of scenarios, especially in the large scale 360° Panorama scenes.

# 2.2 360° PANORAMA OUTPAINTING

Unlike NFoV images, 360° panorama images are subjected to nonlinear perspective distortion, such as equirectangular projection. Consequently, objects and layouts within these images appear substantially distorted, particularly those closer to the top and bottom poles. The image completion has to not only preserve the distorted structure but also ensure visual plausibility, with the additional requirement of wraparound-consistency at both ends. Previous endeavors (Akimoto et al., 2019; Somanath & Kurz, 2021) mainly focused on deterministic completion of 360° RGB images, with BIPS (Oh et al., 2022) further extending this to RGB-D panorama synthesis. In order to generate diverse results (Zheng et al., 2019; 2021), various strategies have been employed. For instance, SIG-SS (Hara et al., 2021) leverages a symmetry-informed CVAE, while OmniDreamer (Akimoto et al., 2022) employs transformer-based sampling. In contrast, our PanoDiffusion is built upon DDPM, wherein each reverse diffusion step inherently introduces stochastic, naturally resulting in multiple and diverse results. Concurrently with our work, MVDiffusion (Tang et al., 2023) generates panorama images by sampling consistent multi-view images, and AOGNet (Lu et al., 2023) does 360° outpainting through an autoregressive process. Compared to the concurrent models, our PanoDiffusion excels in generating semantically multi-objects for large masked regions, without the need of text prompts. More importantly, PanoDiffusion is capable of simultaneously generating the corresponding RGB-D output, using only partially visible RGB images as input during the inference.

# 3 METHODS

Given a $3 6 0 ^ { \circ }$ image $x \in \mathbb { R } ^ { H \times W \times C }$ , degraded by a number of missing pixels to become a masked image $x _ { m }$ , our main goal is to infer semantically meaningful content with reasonable geometry for the missing regions, while simultaneously generating visually realistic appearances. While this task is conceptually similar to conventional learning-based image inpainting, it presents greater challenges due to the following differences: 1) our output is a 360° RGB-D panorama that requires wraparound consistency; 2) the masked/missing areas are generally much larger than the masks in traditional inpainting; 3) our goal is to generate multiple appropriate objects within a scene, instead of simply filling in with the generic background; 4) the completed results should be structurally plausible, which can be reflected by a reasonable depth map.

![](images/0181a49d2f9f73a72b2c76cf20dd712ec25d43db715e77eac06e4eb57cd4f49c.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    A["Data Loader"] --> B["Z0"]
    B --> C["Random"]
    C --> D["Z0 rotated"]
    D --> E["Diffusion"]
    E --> F["ZT"]
    F --> G["(a) Training Stage"]
    G --> H["Mask"]
    H --> I["Zt visible"]
    I --> J["Zt mixed"]
    J --> K["(b) Inference Stage"]
    K --> L["denoise"]
    L --> M["Zt-1"]
    M --> N["Next Iteration"]
    
    subgraph Training Stage
        O["Z"] --> P["noise"]
        P --> Q["Mask Inv."]
        Q --> R["Zt invisible"]
    end
    
    subgraph Inference Stage
        S["RGB channel"] & T["Depth channel"] & U["Camera rotation"] & V["Next Iteration"]
    end
    
    style Training Stage fill:#f9f,stroke:#333
    style Inference Stage fill:#bbf,stroke:#333
```
</details>

Figure 3: Our LDM outpainting structure with camera rotation mechanism. During training (a), we randomly select a rotation angle to generate a new panorama for data augmentation. During inference (b), we sample the visible region from the encoded features (above) and the invisible part from the denoising output (below). The depth map is not needed, and is set to random noise. At each denoising step, we crop a ${ \bf 9 0 ^ { \circ } }$ -equivalent area of the intermediate result from the right and stitch it to the left, denoted by the circle following $z _ { t } ^ { m i x e d }$ — this strongly improves wraparound consistency.

To tackle these challenges, we propose a novel diffusion-based framework for $3 6 0 ^ { \circ }$ panoramic outpainting, called PanoDiffusion. The training process, as illustrated in Fig. 2(a), starts with two branches dedicated to RGB x and depth $d _ { x }$ information. Within each branch, following (Rombach et al., 2022), the input data is first embedded into the latent space using the corresponding pretrained VQ model. These representations are then concatenated to yield $z _ { r g b d } ,$ which subsequently undergoes the forward diffusion step to obtain $z _ { T }$ . The resulting $z _ { T }$ is then subjected to inverse denoising, facilitated by a trained UNet, ultimately returning to the original latent domain. Finally, the pre-trained decoder is employed to rebuild the completed RGB-D results.

During inference, our system takes a masked RGB image as input and conducts panoramic outpainting. It is noteworthy that our proposed model does not inherently require harder-to-acquire depth maps as input, relying solely on a partial RGB image (Fig. 2(b)). The output is further super-resolved into the final image in a refinement stage (Fig. 2(c)).

# 3.1 PRELIMINARIES

Latent Diffusion. Our PanoDiffusion builds upon the latest Latent Diffusion Model (LDM) (Rombach et al., 2022), which executes the denoising process in the latent space of an autoencoder. This design choice yields a twofold advantage: it reduces computational costs while maintaining a high level of visual quality by storing the domain information in the encoder $\mathcal { E } ( \cdot )$ and decoder $\mathcal { D } ( \cdot )$ . During the training, the given image $x _ { 0 }$ is initially embedded to yield the latent representation $z _ { 0 } = \mathcal { E } ( x _ { 0 } )$ , which is then perturbed by adding the noise in a Markovian manner:

$$
q (z _ {t} | z _ {t - 1}) = \mathcal {N} (z _ {t}; \sqrt {1 - \beta_ {t}} z _ {t - 1}, \beta_ {t} I), \tag {1}
$$

where $t = [ 1 , \cdots , T ]$ is the number of steps in the forward process. The hyperparameters $\beta _ { t }$ denote the noise level at each step t. For the denoising process, the network in LDM is trained to predict the noise as proposed in DDPM (Ho et al., 2020), where the training objective can be expressed as:

$$
\mathcal {L} = \mathbb {E} _ {\mathcal {E} (x _ {0}), \epsilon \sim \mathcal {N} (0, I), t} [ | | \epsilon_ {\theta} (z _ {t}, t) - \epsilon | | _ {2} ^ {2} ] \tag {2}
$$

Diffusion Outpainting. The existing pixel-level diffusion inpainting methods (Lugmayr et al., 2022; Horita et al., 2022) are conceptually similar to that used for image generation, except xt incorporates partially visible information, rather than purely sampling from a Gaussian distribution during the inference. In particular, let $x _ { 0 }$ denote the original image in step 0, while $x _ { 0 } ^ { v i s i b l e } = m \odot x _ { 0 }$ and $x _ { 0 } ^ { \dot { \bar { \iota } } n v i s i b l e } = ( 1 - \dot { m _ { ) } } \odot x _ { 0 }$ 0  contain the visible and missing pixels, respectively. Then, as shown in Fig. 3, the reverse denoising sampling process unfolds as follows:

$$
x _ {t} ^ {\text { visible }} \sim q (x _ {t} | x _ {t - 1}), \tag {3}
$$

$$
x _ {t - 1} ^ {\text { invisible }} \sim p _ {\theta} (x _ {t - 1} | x _ {t}), \tag {4}
$$

$$
x _ {t - 1} = m \odot x _ {t - 1} ^ {\text { visible }} + (1 - m) \odot x _ {t - 1} ^ {\text { invisible }}. \tag {5}
$$

![](images/afe625f11c811efcb0729f7d7b2a934a18e133277e77055b4163e94d3fbc826f.jpg)

<details>
<summary>text_image</summary>

Rotate 90°
Rotate 90°
Rotate 90°
Denoising Process
</details>

Figure 4: An example of our two-end alignment mechanism. During inference, we rotate the scene for 90° in each denoising step. Within a total of 200 sampling steps, our PanoDiffusion will effectively achieve wraparound consistency.

Here, q is the forward distribution in the diffusion process and $p _ { \theta }$ is the inverse distribution. After T iterations, x0 is restored to the original image space.

Relation to Prior Work. In contrast to these inpainting methods at pixel-level, our PanoDiffusion builds upon the LDM. Despite the fact that the original LDM provided the ability to inpainting images, such inpainting focuses on removing objects from the image, rather than generating a variety of meaningful objects in panoramic outpainting. In short, the $x _ { 0 }$ is embedded into the latent space, yielding $z _ { 0 } = \mathcal { E } ( x _ { 0 } )$ , while the subsequent sampling process follows the equations (3)-(5). The key motivation behind this is to perform our task on higher resolution 512×1024 panoramas. More importantly, we opt to ${ \bf g 0 }$ beyond RGB outpainting, and to deal with RGB-D synthesis (Sec. 3.3), which is useful for downstream tasks in 3D reconstruction. Additionally, existing approaches can not ensure the wraparound consistency during completion, while our proposed rotational outpainting mechanism in Sec. 3.2 significantly improves such a wraparound consistency.

# 3.2 WRAPAROUND CONSISTENCY MECHANISM

Camera Rotated Data Augmentation. It is expected that the two ends of any $3 6 0 ^ { \circ }$ panorama should be seamlessly aligned, creating a consistent transition from one end to the other. This is especially crucial in applications where a smooth visual experience is required, such as 3D reconstruction and rendering. To promote this property, we implement a circular shift data augmentation, termed camera-rotation, to train our PanoDiffusion. As shown in Fig. 3(a), we randomly select a rotation angle, which is subsequently employed to crop and reassemble image patches, generating a new panorama for training purposes.

Two-Ends Alignment Sampling. While the above camera-rotation technique can improve the model’s implicit grasp of the wraparound consistency using the augmentation of data examples, it may not impose strong enough constraints on wraparound alignment of the results. Therefore, in the inference process, we introduce a novel two-end alignment mechanism that can be seamlessly integrated into our latent diffusion outpainting process. In particular, the reverse denoising process within the DDPM is characterized by multiple iterations, rather than a single step. During each iteration, we apply the camera-rotation operation, entailing 90° rotation of both the latent vectors and mask, before performing a denoising outpainting step. This procedure more effectively connects the two ends of the panorama from the previous step, resulting in significant improvement in visual results (as shown in Fig. 8). Without changing the size of the images, generating overlapping content, or introducing extra loss functions, we provide ‘hints’ to the model by rotating the panorama horizontally, thus enhancing the effect of alignment at both ends (examples shown in Fig. 4).

# 3.3 BI-MODAL LATENT DIFFUSION MODEL

In order to deal with RGB-D synthesis, one straightforward idea could be to use Depth as an explicit condition during training and inference, where the depth information may be compressed into latent space and then introduced into the denoising process of the RGB images via concatenation or cross-attention. However, we found that such an approach often leads to blurry results in our experiments. Alternatively, using two parallel LDMs to reconstruct Depth and RGB images separately, together with a joint loss, may also appear to be an intuitive solution. Nonetheless, this idea presents significant implementation challenges due to the computational resources required for multiple LDMs.

Therefore, we devised a bi-modal latent diffusion structure to introduce depth information while generating high-quality RGB output. It is important to note that this depth information is solely necessary during the training phase. Specifically, we trained two VAE models independently for RGB and depth images, and then concatenate $\bar { z _ { r g b } } \in \mathbb { R } ^ { h \times w \times 3 }$ with $z _ { d e p t h } \in \mathbb { R } ^ { h \times w \times \hat { 1 } }$ at the latent level to get $z _ { r g b d } \in \mathbb { R } ^ { h \times w \times 4 }$ . The training of VAEs is exactly the same as in (Rombach et al., 2022) with downsampling factor $f { = } 4$ . Then we follow the standard process to train an unconditional DDPM with $z _ { r g b d }$ via a variant of the original LDM loss:

![](images/42b76e4152582958122b6d6205707507de255aeb624481eee2bc0e4356fb9610.jpg)

<details>
<summary>natural_image</summary>

Exterior view of a modern office building (no signage)
</details>

(a) NFoV

![](images/4a0fbfd609fbe03f9b6a0ff15719fec8f016db86640d267d9a51a0daa7e7c8c5.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern living room with sofa, coffee table, and large windows (no visible text or symbols)
</details>

(b) Camera

![](images/d49ee4be6b292010146a1c0110b4ec080b698375f8afa9031619bfc6f975964a.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a modern luxury lounge with ambient lighting and seating areas (no visible text or symbols)
</details>

(c) Random

![](images/98c21c7db88ea2185648a8aecf45b60bde7568104f97d3d4f852740242a61a06.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern lounge or reception area with curved seating and a large black circular object (no visible text or symbols)
</details>

(d) Layout   
Figure 5: Examples of various mask types. See text for details.

$$
\mathcal {L} _ {R G B - D} = \mathbb {E} _ {z _ {r g b d}, \epsilon \sim \mathcal {N} (0, 1), t} [ \| \epsilon_ {\theta} (z _ {t}, t) - \epsilon \| _ {2} ^ {2} ], z _ {r g b d} = \mathcal {E} _ {1} (x) \oplus \mathcal {E} _ {2} (d _ {x}) \tag {6}
$$

Reconstructed RGB-D images can be obtained by decoupling $z _ { r g b d }$ and decoding. It is important to note that during training, we use the full RGB-D image as input, without masks. Conversely, during the inference stage, the model can perform outpainting of the masked RGB image directly without any depth input, with the fourth channel of $z _ { r g b d }$ replaced by random noise.

# 3.4 REFINENET

Although mapping images to a smaller latent space via an autoencoder prior to diffusion can save training space and thus allow larger size inputs, the panorama size of 512×1024 is still a heavy burden for LDM (Rombach et al., 2022). Therefore, we adopt a two-stage approach to complete the outpainting task. Initially, the original input is downscaled to 256×512 as the input to the LDM. Correspondingly, the image size of the LDM output is also 256×512. Therefore, an additional module is needed to upscale the output image size to 512×1024. Since panorama images are distorted and the objects and layouts do not follow the regular image patterns, we trained a super-resolution GAN model for panoramas to produce visually plausible results at a higher resolution.

# 4 EXPERIMENTS

# 4.1 EXPERIMENTAL DETAILS

Dataset. We estimated our model on the Structured3D dataset (Zheng et al., 2020), which provides 360° indoor RGB-D data following equirectangular projection with a 512×1024 resolution. We split the dataset into 16930 train, 2116 validation, and 2117 test instances.

Metrics. For RGB outpainting, due to large masks, we should not require the completed image to be exactly the same as the original image, since there are many plausible solutions (e.g. new furniture and ornaments, and their placement). Therefore, we mainly report the following datasetlevel metrics: 1) Fre´chet Inception Distance (FID) (Heusel et al., 2017), 2) Spatial FID (sFID) (Nash et al., 2021), 3) density and coverage (Naeem et al., 2020). FID compares the distance between distributions of generated and original images in a deep feature domain, while sFID is a variant of FID that uses spatial features rather than the standard pooled features. Additionally, density reflects how accurate the generated data is to the real data stream, while coverage reflects how well the generated data generalizes the real data stream. For depth synthesis, we use RMSE, MAE, AbsREL, and Delta1.25 as implemented in (Cheng et al., 2018; Zheng et al., 2019), which are commonly used to measure the accuracy of depth estimates.

Mask Types. Most works focused on generating omnidirectional images from NFoV images (Fig. 5(a)). However, partial observability may also occur due to sensor damage in 360° cameras. Such masks can be roughly simulated by randomly sampling a number of NFoV camera views within the panorama (Fig. 5(b)). We also experimented with other types of masks, such as randomly generated regular masks (Fig. 5(c)). Finally, the regions with floors and ceilings in panoramic images are often less interesting than the central regions. Hence, we also generated layout masks that muffle all areas except floors and ceilings, to more incisively test the model’s generative power (Fig. 5(d)).

![](images/82bef2cccd504b785379309af7945b77135f97d3d147034fdb311e552a58d282.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a modern bedroom with large wall art, featuring a bed, lamp, and TV (no visible text or symbols)
</details>

(a) Ground Truth

![](images/0cc4fb67ad2956e20e81543aaf91200ba07186a246e5ca12a008c8060c0415e9.jpg)  
(b) Masked Input

![](images/589a007ae43f289edadfd5e163f4c86b9900581f4ad59da4166b131d54d5dd88.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a modern bedroom with large wall art, built-in beds, and minimal furniture (no visible text or symbols)
</details>

(c) PanoDiffusion (RGB)

![](images/c5f09d7dcbbf896d40662c32525b7729972e0ee9188a8b57dab5c9ebbc6bf758.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a modern minimalist living room with curved white walls, green furniture, and wall-mounted screens (no visible text or symbols)
</details>

(d) PanoDiffusion (RGB-D)

![](images/fd0034cd1d998d3c1f767669e210099c4ad3122d625a8536a7ba30409e9a8e40.jpg)  
(e) BIPSECCV’2022

![](images/3327c7593eeb5ca68c5c2eb3b5c940daf5800a97138aa2250fb929d671fd4578.jpg)  
(f) OmniDreamerCVPR’2022

![](images/cf93008221934209b8cbc60d449cdb2b86e78cfee884b34d9a4b734edca38afb.jpg)  
(g) TFillCVPR’2022

![](images/32d218b91c2bf25c86d5ee58788d07b220395ff5461063bcd692c2eed70369b8.jpg)  
(h) Inpaint AnythingarXiv’2023   
Figure 6: Qualitative comparison for RGB panorama outpainting. Our PanoDiffusion generated more objects with appropriate layout, and with better visual quality. For BIPS and OmniDreamer, despite the seemingly reasonable results, the outpainted areas tend to fill the walls and lack diverse items. As for TFill, it generates blurry results for large invisible areas. For Inpaint anything, it generates multiple objects but they appear to be structurally and semantically implausible. Compared to them, PanoDiffusion generates more reasonable details in the masked region, such as pillows, paintings on the wall, windows, and views outside. More comparisons are provided in Appendix.

Table 1: Quantitative results for RGB outpainting. All models were re-trained and evaluated using the same standardized dataset. Note that, we tested all models without the depth input. 

<table><tr><td rowspan="2">Methods</td><td colspan="4">Camera Mask</td><td colspan="4">NFoV Mask</td><td colspan="4">Layout Mask</td><td colspan="4">Random Box Mask</td></tr><tr><td>FID ↓</td><td>sFID ↓</td><td>D ↑</td><td>C ↑</td><td>FID ↓</td><td>sFID ↓</td><td>D ↑</td><td>C ↑</td><td>FID ↓</td><td>sFID ↓</td><td>D ↑</td><td>C ↑</td><td>FID ↓</td><td>sFID ↓</td><td>D ↑</td><td>C ↑</td></tr><tr><td>BIPS</td><td>31.70</td><td>28.89</td><td>0.769</td><td>0.660</td><td>57.69</td><td>44.68</td><td>0.205</td><td>0.277</td><td>32.25</td><td>24.66</td><td>0.645</td><td>0.579</td><td>25.35</td><td>22.60</td><td>0.676</td><td>0.798</td></tr><tr><td>OmniDreamer</td><td>65.47</td><td>37.04</td><td>0.143</td><td>0.175</td><td>62.56</td><td>36.24</td><td>0.125</td><td>0.184</td><td>82.71</td><td>28.40</td><td>0.103</td><td>0.120</td><td>45.10</td><td>24.12</td><td>0.329</td><td>0.576</td></tr><tr><td>LaMa</td><td>115.92</td><td>107.69</td><td>0.034</td><td>0.082</td><td>125.77</td><td>136.32</td><td>0.002</td><td>0.006</td><td>129.77</td><td>35.23</td><td>0.018</td><td>0.043</td><td>45.25</td><td>24.21</td><td>0.429</td><td>0.701</td></tr><tr><td>TFill</td><td>83.84</td><td>61.40</td><td>0.075</td><td>0.086</td><td>93.62</td><td>76.13</td><td>0.037</td><td>0.027</td><td>97.99</td><td>43.40</td><td>0.046</td><td>0.052</td><td>46.84</td><td>30.72</td><td>0.368</td><td>0.574</td></tr><tr><td>Inpainting Anything</td><td>97.38</td><td>54.73</td><td>0.076</td><td>0.133</td><td>105.77</td><td>59.70</td><td>0.054</td><td>0.035</td><td>92.18</td><td>32.00</td><td>0.116</td><td>0.085</td><td>46.30</td><td>26.71</td><td>0.372</td><td>0.632</td></tr><tr><td>RePaint</td><td>82.84</td><td>84.39</td><td>0.096</td><td>0.105</td><td>95.38</td><td>82.35</td><td>0.0639</td><td>0.078</td><td>69.14</td><td>31.63</td><td>0.294</td><td>0.263</td><td>55.47</td><td>38.78</td><td>0.433</td><td>0.581</td></tr><tr><td>PanoDiffusion</td><td>21.55</td><td>26.95</td><td>0.867</td><td>0.708</td><td>21.41</td><td>27.80</td><td>0.790</td><td>0.669</td><td>23.06</td><td>22.39</td><td>1.000</td><td>0.737</td><td>16.13</td><td>20.39</td><td>1.000</td><td>0.883</td></tr></table>

Baseline Models. For RGB panorama outpainting, we mainly compared with the following stateof-the-art methods: including image inpainting models LaMa (Suvorov et al., 2022)WACV’2022 and TFill (Zheng et al., 2022)CVPR’2022, panorama outpainting models BIPS (Oh et al., 2022)ECCV’2022 and OmniDreamer (Akimoto et al., 2022)CVPR’2022, diffusion-based image inpainting models Repaint (Lugmayr et al., 2022)CVPR’2022 and Inpaint Anything (Yu et al., 2023)arXiv’2023. To evaluate the quality of depth panorama, we compare our method with three image-guided depth synthesis methods including BIPS (Oh et al., 2022), NLSPN (Park et al., 2020), and CSPN (Cheng et al., 2018). All models are retrained on the Structured3D dataset using their publicly available codes.

# 4.2 MAIN RESULTS

Following prior works, we report the quantitative results for RGB panorama outpainting with camera masks in Table 1. All instantiations of our model significantly outperform all state-of-the-art models. Specifically, the FID score is substantially better (relative 67.0% improvement).

It is imperative to note that our model is trained unconditionally, with masks only employed during the inference phase. Therefore, it is expected to handle a broader spectrum of mask types. To validate this assertion, we further evaluated our model with the baseline models across all four different mask types (displayed in Fig. 5). The results in Table 1 show that PanoDiffusion consistently outperforms the baseline models on all types of masks. Conversely, baseline models’ performance displays significant variability in the type of mask used. Although the visible regions of the layout masks are always larger than the camera masks, the performances of baseline models on camera masks are significantly better. This is likely because the masks in the training process are closer to the NFoV distribution. In contrast, PanoDiffusion has a more robust performance, producing high-quality and diverse output images for all mask distributions.

The qualitative results are visualized in Fig. 6. Here we show an example of outpainting on a layout mask (more comparisons in Appendix). Besides the fact that PanoDiffusion generates more visually realistic results than baseline models, comparing the RGB (trained without depth) and RGB-D versions of our PanoDiffusion, in Fig. 6(c), some unrealistic structures are generated on the center wall, and when we look closely at the curtains generated by the RGB model, the physical structure of the edges is not quite real. In contrast, the same region of RGB-D result (Fig. 6(d)) appears more structurally appropriate. Such improvement proves the advantages of jointly learning to synthesize depth data along with RGB images, even when depth is not used during test time, suggesting the depth information is significant for assisting the RGB completion.

![](images/21c7e82f133f1956b4d767b5b3f540ee21f5f7a7da9ece3f10e3685be02597cb.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a modern bedroom with curved walls, a bed, and a window (no visible text or symbols)
</details>

(a) RGB Input

![](images/7728f6f11ea48b6804a9b0b7d1664363fce6b7b54deb70fe1f5722ed80879938.jpg)

<details>
<summary>natural_image</summary>

Thermal or heat map visualization of a 3D object with color gradient from blue to red (no text or symbols)
</details>

(b) Depth GT

![](images/d2ee004f39936546cd48670a66d488286406eef3519ebbf0ab40cf7bd5f6c90c.jpg)

<details>
<summary>natural_image</summary>

Thermal or heat map visualization of a rounded object with red-orange core and yellow-yellow sides (no text or symbols)
</details>

(c) Ours PanoDiffusion

![](images/4e62f9ae6ffa2cdd817e03bd69147bc1582932b84e9c8b4b200f00bbec04f22f.jpg)

<details>
<summary>natural_image</summary>

Thermal or heat map visualization with red-orange gradient and circular shapes (no text or symbols)
</details>

(d) CSPN

![](images/9440bb926eadc50efc23cf9fae1fefd6145ebddebe38acd82123bfe53af63ff2.jpg)

<details>
<summary>natural_image</summary>

Thermal imaging view of a room with heat signatures indicating temperature distribution (no text or symbols)
</details>

(e) BIPS

![](images/7fa479e38a729d153e6a2e2ac6f0e85475a3e02cff0206d98924650d5f092f03.jpg)

<details>
<summary>natural_image</summary>

Thermal or heat map image showing a circular object with red-orange intensity, surrounded by yellow and blue gradients (no text or symbols)
</details>

(f) NLSPN   
Figure 7: Qualitative comparison for depth panorama synthesis.

Table 2: Depth map ablations. All models are trained and evaluated on the Structured3D dataset. 

<table><tr><td rowspan="2">Noise Level</td><td colspan="4">Camera Mask</td><td colspan="4">NFoV Mask</td><td colspan="4">Layout Mask</td><td colspan="4">Random Box Mask</td></tr><tr><td>FID ↓</td><td>sFID ↓</td><td>D ↑</td><td>C ↑</td><td>FID ↓</td><td>sFID ↓</td><td>D ↑</td><td>C ↑</td><td>FID ↓</td><td>sFID ↓</td><td>D ↑</td><td>C ↑</td><td>FID ↓</td><td>sFID ↓</td><td>D ↑</td><td>C ↑</td></tr><tr><td>no depth</td><td>24.33</td><td>29.00</td><td>0.667</td><td>0.635</td><td>24.01</td><td>30.00</td><td>0.639</td><td>0.617</td><td>25.37</td><td>22.92</td><td>0.785</td><td>0.677</td><td>17.88</td><td>21.21</td><td>0.913</td><td>0.857</td></tr><tr><td>50%</td><td>21.65</td><td>28.12</td><td>0.678</td><td>0.660</td><td>21.99</td><td>29.37</td><td>0.678</td><td>0.561</td><td>24.24</td><td>23.05</td><td>0.855</td><td>0.724</td><td>17.02</td><td>21.22</td><td>0.919</td><td>0.837</td></tr><tr><td>30%</td><td>21.78</td><td>27.96</td><td>0.714</td><td>0.674</td><td>21.78</td><td>29.39</td><td>0.643</td><td>0.658</td><td>24.11</td><td>23.00</td><td>0.919</td><td>0.724</td><td>16.87</td><td>21.25</td><td>0.937</td><td>0.855</td></tr><tr><td>10%</td><td>21.68</td><td>27.79</td><td>0.721</td><td>0.658</td><td>21.49</td><td>29.74</td><td>0.558</td><td>0.620</td><td>24.02</td><td>22.68</td><td>0.938</td><td>0.741</td><td>16.60</td><td>21.02</td><td>0.932</td><td>0.853</td></tr><tr><td>full depth</td><td>21.55</td><td>26.95</td><td>0.867</td><td>0.708</td><td>21.41</td><td>27.80</td><td>0.790</td><td>0.669</td><td>23.06</td><td>22.39</td><td>1.000</td><td>0.737</td><td>16.13</td><td>20.39</td><td>1.000</td><td>0.883</td></tr></table>

(a) Usage of depth maps (training). We use different sparsity levels of depth for training and the results (more intense color means better performance) verify the effectiveness of depth for RGB outpainting. It also proves that the model can accept sparse depth as input.

<table><tr><td>Methods</td><td>Input Depth</td><td>FID ↓</td><td>sFID ↓</td><td>D ↑</td><td>C ↑</td></tr><tr><td rowspan="2">BIPSPanoDiffusion</td><td rowspan="2">fully visible</td><td>29.74</td><td>30.59</td><td>0.931</td><td>0.721</td></tr><tr><td>21.90</td><td>26.78</td><td>0.829</td><td>0.693</td></tr><tr><td rowspan="2">BIPSPanoDiffusion</td><td rowspan="2">partial visible</td><td>31.70</td><td>28.89</td><td>0.769</td><td>0.660</td></tr><tr><td>22.34</td><td>26.74</td><td>0.856</td><td>0.686</td></tr><tr><td rowspan="2">BIPSPanoDiffusion</td><td rowspan="2">fully masked</td><td>68.79</td><td>42.62</td><td>0.306</td><td>0.412</td></tr><tr><td>21.55</td><td>26.95</td><td>0.867</td><td>0.708</td></tr></table>

(b) Usage of depth maps (inference). BIPS heavily relies on the availability of input depth during inference, while our model is minimally affected.

<table><tr><td>Methods</td><td>Input Depth</td><td>RMSE ↓</td><td>MAE ↓</td><td>AbsREL ↓</td><td>Delta1.25 ↑</td></tr><tr><td>BIPS</td><td rowspan="4">fully masked</td><td>323</td><td>207</td><td>0.1842</td><td>0.8436</td></tr><tr><td>CSPN</td><td>374</td><td>282</td><td>0.2273</td><td>0.6618</td></tr><tr><td>NLSPN</td><td>284</td><td>183</td><td>0.1692</td><td>0.8544</td></tr><tr><td>PanoDiffusion</td><td>276</td><td>193</td><td>0.1355</td><td>0.9060</td></tr><tr><td>BIPS</td><td rowspan="4">partial visible</td><td>247</td><td>136</td><td>0.1098</td><td>0.9032</td></tr><tr><td>CSPN</td><td>291</td><td>195</td><td>0.1547</td><td>0.8182</td></tr><tr><td>NLSPN</td><td>221</td><td>124</td><td>0.1058</td><td>0.9143</td></tr><tr><td>PanoDiffusion</td><td>219</td><td>123</td><td>0.1127</td><td>0.9278</td></tr></table>

(c) Depth panorama synthesis. Our model outperforms baseline models in most of the metrics.

# 4.3 ABLATION EXPERIMENTS

We ran a number of ablations to analyze the effectiveness of each core component in our PanoDiffusion. Results are shown in tables 2 and 3 and figs. 7 and 8 and discussed in detail next.

Depth Maps. In practice applications, depth data may exhibit sparsity due to the hardware limitations (Park et al., 2020). To ascertain the model’s proficiency in accommodating sparse depth maps as input, we undertook a training process using depth maps with different degrees of sparsity (i.e., randomized depth value will be set to 0). The result is reported in Table 2(a). The denser colors in the table represent better performance. As the sparsity of the depth input decreases, the performance of RGB outpainting constantly improves. Even if we use 50% sparse depth for training, the result is overall better than the original LDM.

We then evaluated the importance of depth maps during inference, and compared it with the state-ofthe-art BIPS (Oh et al., 2022), which is also trained with RGB-D images. The quantitative results are reported in Table 2(b). As can be seen, BIPS’s performance appears to deteriorate significantly when the input depth visual area is reduced. Conversely, our PanoDiffusion is not sensitive to these depth maps, indicating that the generic model has successfully handled the modality. Interestingly, we noticed that having fully visible depth at test time did not improve the performance of PanoDiffusion, and in fact, the result deteriorated slightly. A reasonable explanation is that during the training process, the signal-to-noise ratios (SNR) of RGB and depth pixels are roughly the same within each iteration since no masks were used. However, during outpainting, the SNR balance will be disrupted when RGB input is masked and depth input is fully visible. Therefore, the results are degraded, but only slightly because PanoDiffusion has effectively learned the distribution of spatial visual patterns across all modalities, without being overly reliant on depth. This also explains why our model is more robust to depth inputs with different degrees of visibility.

![](images/d5dbd6268ba6315d69e41a9d411bf3959b801064e5fe4f4b36881d2ca91abe04.jpg)

<details>
<summary>natural_image</summary>

Three-panel interior view showing a wooden-framed cabinet, a transparent cylindrical chamber, and a tiled wall with decorative elements (no visible text or symbols)
</details>

(a) PanoDiffusion w/ TA

![](images/8a23566290df4f56a8fff4e2f738d1179b8097054778409c5ba3982b904d0685.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern minimalist room with curved glass display and transparent interior (no visible text or symbols)
</details>

(b) PanoDiffusion w/o TA

![](images/8c6163657a1df9f54170ab2705fe97e089c70cc55bd0f2433de63c436221ff50.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern living room with furniture and lighting fixtures (no visible text or symbols)
</details>

(c) OmniDreamer

![](images/0a7c33e7b51990a57ff9e013252ae7d20faee0ce97e36eb9c3ff9d78328f9a26.jpg)

<details>
<summary>natural_image</summary>

Three-panel interior view of a modern building interior: front, side, and interior (no visible text or symbols)
</details>

(d) BIPS   
Figure 8: Examples of stitched ends of the outpainted images. For each image, the left half was unmasked (i.e. ground truth), while the right half was masked and synthesized. The results generated with rotation are more naturally connected at both ends (a).

Table 3: Two-end alignment ablations. Using rotational outpainting, we achieve optimal consistency at both ends of the PanoDiffusion output. 

<table><tr><td>Methods \ Mask Type</td><td>Camera</td><td>NfoV</td><td>Layout</td><td>Random Box</td><td>Methods \ Mask Type</td><td>Camera</td><td>NfoV</td><td>Layout</td><td>Random Box</td></tr><tr><td>PanoDiffusion (w/ rotation)</td><td>90.41</td><td>89.74</td><td>88.01</td><td>85.04</td><td>PanoDiffusion (w/o rotation)</td><td>125.82</td><td>128.33</td><td>128.10</td><td>128.19</td></tr><tr><td>BIPS</td><td>117.59</td><td>96.82</td><td>132.15</td><td>148.78</td><td>OmniDreamer</td><td>115.6</td><td>109.00</td><td>146.37</td><td>136.68</td></tr><tr><td>LaMa</td><td>119.51</td><td>119.39</td><td>133.54</td><td>136.35</td><td>TFill</td><td>155.16</td><td>157.60</td><td>136.94</td><td>122.96</td></tr></table>

Finally, we evaluated the depth synthesis ability of PanoDiffusion, seen in Table 2(c) and Fig. 7. The results show that our model achieves the best performance on most of the metrics and the qualitative results also show that PanoDiffusion is able to accurately estimate the depth map. This not only indicates that PanoDiffusion can be used for depth synthesis and estimation but also proves that it has learned the spatial patterns of panorama images.

Two-end Alignment. Currently, there is no metric to evaluate the performance of aligning the two ends of an image. To make a reasonable comparison, we make one side of the input image fully visible, and the other side fully masked and then compare the two ends of output. Based on the Left-Right Consistency Error (LRCE) (Shen et al., 2022) which is used to evaluate the consistency of two ends of the depth maps, we designed a new RGB-LRCE to calculate the difference between the two ends of the image: $\begin{array} { r } { L R C E = \frac { 1 } { N } \sum _ { i = 1 } ^ { N } \left| P _ { f i r s t } ^ { c o l } - P _ { l a s t } ^ { c o l } \right| } \end{array}$ − , and reported results in table 3.

The qualitative results are shown in fig. 8. To compare as many results, we only show the end regions that are stitched together to highlight the contrast. They show that the consistency of the two ends of the results is improved after the use of rotational outpainting, especially the texture of the walls and the alignment of the layout. Still, differences can be found with rotated outpainting. We believe it is mainly due to the fact that rotational denoising is based on the latent level, which may introduce extra errors during decoding.

# 5 CONCLUSION

In this paper, we show that our proposed method, the two-stage RGB-D PanoDiffusion, achieves state-of-the-art performance for indoor RGB-D panorama outpainting. The introduction of depth information via our bi-modal LDM structure significantly improves the performance of the model. Such improvement illustrates the effectiveness of using depth during training as an aid to guide RGB panorama generation. In addition, we show that the alignment mechanism we employ at each step of the denoising process of the diffusion model enhances the wraparound consistency of the results. With the use of these novel mechanisms, our two-stage structure is capable of generating high-quality RGB-D panoramas at 512×1024 resolution.

# REFERENCES

Naofumi Akimoto, Seito Kasai, Masaki Hayashi, and Yoshimitsu Aoki. 360-degree image completion by two-stage conditional gans. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 4704–4708. IEEE, 2019.   
Naofumi Akimoto, Yuhi Matsuo, and Yoshimitsu Aoki. Diverse plausible 360-degree image outpainting for efficient 3dcg background creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11441–11450, 2022.   
Xinjing Cheng, Peng Wang, and Ruigang Yang. Depth estimation via affinity learned with convolutional spatial propagation network. In Proceedings of the European conference on computer vision (ECCV), pp. 103–119, 2018.   
Sungjoon Choi, Qian-Yi Zhou, and Vladlen Koltun. Robust reconstruction of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5556– 5565, 2015.   
Dave Epstein, Taesung Park, Richard Zhang, Eli Shechtman, and Alexei A Efros. Blobgan: Spatially disentangled scene representations. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV, pp. 616–635. Springer, 2022.   
Marc-Andre Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Chris- ´ tian Gagne, and Jean-Franc¸ois Lalonde. Learning to predict indoor illumination from a single´ image. ACM Transactions on Graphics (TOG), 36(6):1–14, 2017.   
Marc-Andre Gardner, Yannick Hold-Geoffroy, Kalyan Sunkavalli, Christian Gagn ´ e, and Jean- ´ Franc¸ois Lalonde. Deep parametric indoor lighting estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7175–7183, 2019.   
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.   
Takayuki Hara, Yusuke Mukuta, and Tatsuya Harada. Spherical image generation from a single image by considering scene symmetry. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1513–1521, 2021.   
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.   
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.   
Daichi Horita, Jiaolong Yang, Dong Chen, Yuki Koyama, and Kiyoharu Aizawa. A structure-guided diffusion model for large-hole diverse image completion. arXiv preprint arXiv:2211.10437, 2022.   
Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM Transactions on Graphics (ToG), 36(4):1–14, 2017.   
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In editor (ed.), Proceedings of the International Conference on Learning Representations (ICLR), 2014.   
Jeong-Kyun Lee, Jaewon Yea, Min-Gyu Park, and Kuk-Jin Yoon. Joint layout estimation and global multi-view registration for indoor reconstruction. In Proceedings of the IEEE international conference on computer vision, pp. 162–171, 2017.   
Wenbo Li, Xin Yu, Kun Zhou, Yibing Song, Zhe Lin, and Jiaya Jia. Sdm: Spatial diffusion model for large hole image inpainting. arXiv preprint arXiv:2212.02963, 2022.   
Zhuqiang Lu, Kun Hu, Chaoyue Wang, Lei Bai, and Zhiyong Wang. Autoregressive omni-aware outpainting for open-vocabulary 360-degree image generation. arXiv preprint arXiv:2309.03467, 2023.

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11461–11471, 2022.   
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pp. 7176–7185. PMLR, 2020.   
Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint arXiv:2103.03841, 2021.   
Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE international symposium on mixed and augmented reality, pp. 127–136. Ieee, 2011.   
Changgyoon Oh, Wonjune Cho, Yujeong Chae, Daehee Park, Lin Wang, and Kuk-Jin Yoon. Bips: Bi-modal indoor panorama synthesis via residual depth-aided adversarial learning. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVI, pp. 352–371. Springer, 2022.   
Jinsun Park, Kyungdon Joo, Zhe Hu, Chi-Kuei Liu, and In So Kweon. Non-local spatial propagation network for depth completion. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16, pp. 120–136. Springer, 2020.   
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536–2544, 2016.   
Xiaofeng Ren, Liefeng Bo, and Dieter Fox. Rgb-(d) scene labeling: Features and algorithms. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2759–2766. IEEE, 2012.   
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High- ¨ resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695, 2022.   
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.   
Zhijie Shen, Chunyu Lin, Kang Liao, Lang Nie, Zishuo Zheng, and Yao Zhao. Panoformer: Panorama transformer for indoor 360-degree depth estimation. In European Conference on Computer Vision, pp. 195–211. Springer, 2022.   
Gowri Somanath and Daniel Kurz. Hdr environment map estimation for real-time augmented reality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11298–11306, 2021.   
Shuran Song and Thomas Funkhouser. Neural illumination: Lighting prediction for indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6918–6926, 2019.   
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2149–2159, 2022.   
Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mvdiffusion: Enabling holistic multi-view image generation with correspondence-aware diffusion. arXiv preprint 2307.01097, 2023.   
Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao. High-fidelity pluralistic image completion with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4692–4701, October 2021.

Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18359–18369, 2023.   
Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22428–22437, 2023.   
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5505–5514, 2018.   
Tao Yu, Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, and Zhibo Chen. Inpaint anything: Segment anything meets image inpainting. arXiv preprint arXiv:2304.06790, 2023.   
Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, I Eric, Chao Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. In International Conference on Learning Representations, 2020.   
Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Pluralistic image completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1438–1447, 2019.   
Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Pluralistic free-form image completion. In International Journal of Computer Vision, pp. 2786–2805, 2021.   
Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. T2net: Synthetic-to-realistic translation for solving single-image depth estimation tasks. In Proceedings of the European conference on computer vision (ECCV), pp. 767–783, 2018.   
Chuanxia Zheng, Tat-Jen Cham, Jianfei Cai, and Dinh Phung. Bridging global context interactions for high-fidelity image completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11512–11522, 2022.   
Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16, pp. 519–535. Springer, 2020.