# Spherical-Nested Diffusion Model for Panoramic Image Outpainting

Xiancheng Sun 1 Senmao Ma 1 Shengxi Li 1 2 Mai Xu 1 Jingyuan Xia 3 Lai Jiang 1 2 Xin Deng 1 2 Jiali Wang 4

# Abstract

Panoramic image outpainting acts as a pivotal role in immersive content generation, allowing for seamless restoration and completion of panoramic content. Given the fact that the majority of generative outpainting solutions operates on planar images, existing methods for panoramic images address the sphere nature by soft regularisation during the end-to-end learning, which still fails to fully exploit the spherical content. In this paper, we set out the first attempt to impose the sphere nature in the design of diffusion model, such that the panoramic format is intrinsically ensured during the learning procedure, named as sphericalnested diffusion (SpND) model. This is achieved by employing spherical noise in the diffusion process to address the structural prior, together with a newly proposed spherical deformable convolution (SDC) module to intrinsically learn the panoramic knowledge. Upon this, the proposed method is effectively integrated into a pre-trained diffusion model, outperforming existing state-ofthe-art methods for panoramic image outpainting. In particular, our SpND method reduces the FID values by more than 50% against the state-of-the-art PanoDiffusion method. Codes are publicly available at https://github.com/ chronos123/SpND.

# 1. Introduction

Most recently, panoramic images have received increasing research efforts due to their critical roles in applications such as virtual reality (VR) (Wan Abd Arif et al., 2009; Wang,

![](images/49685e8c6d9937e8a5a9562cd6e95a8914461d654e971a69c77b6c1a0d7734b0.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    subgraph (a)
        A["Planar Grid"] --> B["Spherical loss"]
        B --> C["Depth Cues"]
        C --> D["Conv"]
        D --> E["Conv"]
        E --> F["..."]
        F --> G["Conv"]
        G --> H["∅"]
        H --> I["Completed Panoramic Image"]
    end

    subgraph (b)
        J["Partial Image"] --> K["Spherical Grid"]
        K --> L["Structural Prior"]
        L --> M["SDC"]
        L --> N["SDC"]
        L --> O["..."]
        O --> P["SDC"]
        P --> Q["∅"]
        Q --> R["Completed Panoramic Image"]
    end
```
</details>

Figure 1. Comparison between the proposed and existing methods for panoramic image outpainting. (a) Existing methods learn panoramic content by soft regularisations, in a macro way, thus outpainting irregular 3D arrangement and artefacts. (b) The proposed method incorporates the sphere nature within the design, thus in a micro way, which is able to outpaint realistic and high-quality panoramic images. Note that E denotes for the encoder and D denotes for the decoder.

2019; Zhang & Kou, 2022), augmented reality (AR) (Lee & Tsai, 2015; Eiris et al., 2018), and autonomous driving (Huang et al., 2018; Yurtsever et al., 2020; Gao et al., 2022). Although existing methods (Rombach et al., 2022) are capable of generating high-quality content, they are designed for planar images and fail to generalise well to panoramic images, due to the spherical distortion introduced by the equirectangular projection (ERP). Consequently, the generation of panoramic images has emerged as a pivotal area of research, drawing extensive research interest seeking to produce high-quality panoramic content. We also notice several text-to-panorama generation methods, including PanFusion (Zhang et al., 2024) and MVDiffusion (Tang et al., 2023) by incorporating attention mechanism to partially address panoramic cues. They still fail to address spatial continuity and contextual coherence for panoramic image outpainting.

As one of the main-stream tasks of panoramic image generation, panoramic image outpainting has been pioneered by works (Akimoto et al., 2019; Sumantri & Park, 2020), which are able to complete scenes with the 360◦ × 180◦ field-ofview (FOV). To ensure spherical continuity of the generated edges, one of the core characteristics of panoramic images, existing solutions include the usage of circular padding technique (Hara et al., 2021; Akimoto et al., 2022) and rotation within the latent space (Wang et al., 2023a). A further crucial problem is the accurate capturing against the panoramic content, which is distorted by ERP during the training. Existing methods relieve this by employing the panoramicguided loss (Wang et al., 2024), extra depth modality cues (Wu et al., 2024b), or unconstrained deformable convolution operations (Wu et al., 2024a), which essentially impose soft regularisations in an macro way to guide the overall training procedure. Generative models trained in an end-to-end style by the soft regularisations, however, may not be able to operate as expected, thus failing to fully exploit the spherical geometry and panoramic content as illustrated in Figure 1.

In this paper, we set out the first attempt to outpaint panoramic images by incorporating their intrinsic sphere nature in an micro way, i.e., by the sophisticated design of generative models. More specifically, we first propose to sample identically and independently distributed (i.i.d.) spherical noise on the 3D surface in the diffusion model, corresponding to the sphere nature of panoramic images. To the best of our knowledge, all the existing methods sample noise on the planar latent space. Though, the i.i.d. spherical noise essentially results into non-i.i.d. planar noise after the ERP projection; this ensures that our method suits well in the panoramic image outpainting task. More importantly, given the fact that an irregular grid is required to maintain the regular grid on the sphere, we propose to adaptively perceive the receptive field on the sphere, instead of on the plane, in which the spherical deformable convolution (SDC) layer is developed in particular for the panoramic content. Our SDC layer introduces a fundamentally novel convolution operation to mitigate panoramic deformation, which is distinct from existing strategies such as feature re-organisation (Shen et al., 2022) or sphere-guided loss (de La Garanderie et al., 2018; Jaus et al., 2023) that are employed for panoramic depth estimation and segmentation tasks. Built upon the SDC layers, we propose our spherical-nested diffusion (SpND) model, based on a newly developed circular mask encoder (CME) for the consistency across edges. The experimental results have verified the significantly superior performances of our SpND model. Our contributions are three-fold:

• We seamlessly incorporate the spherical noise by investigating ERP distortion when training diffusion models, such that the structural prior of panoramic images can be well accommodated.   
• We propose the SDC layer that is the first successful attempt to satisfy the intrinsic sphere nature within generative model architectures, with adaptive and consistent receptive fields.   
• We develop the SpND model by incorporating the spherical noise and SDC layer as fundamental modules,

accomplished by the CME to ensure the high-quality panoramic image outpainting.

# 2. Related Works

Image Outpainting. The image outpainting task aims to fill unknown regions beyond the boundaries of given content. Based on advancements of generative models, such as variational autoencoders (VAEs) (Kingma & Welling, 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014), a series of image outpainting methods has been developed (Pathak et al., 2016; Iizuka et al., 2017; Yu et al., 2018; Zheng et al., 2019; Wang et al., 2019; K. & Ali, 2020; Zhao et al., 2021). The generative models allow for outpainting semantically meaningful content when completing images based on a provided portion from an overall image. The most recent advancements in diffusion model (Rombach et al., 2022) with the transformer architecture (Vaswani et al., 2017) have further improved completed images with consistent structure and semantics (Dosovitskiy et al., 2021; Esser et al., 2021; Wan et al., 2021; Zheng et al., 2022; Yao et al., 2022; Lugmayr et al., 2022; Xie et al., 2023; Wang et al., 2023b; Li et al., 2024). However, regarding panoramic images, these methods overlook the unique characteristics of panoramic images, including omnidirectional continuity across edges and spherical distortion.

Panoramic Image Outpainting. Different from planar images with a narrow FOV, panoramic images with a $3 6 0 ^ { \circ } \times 1 8 0 ^ { \circ }$ FOV exhibit spherical distortion due to the ERP projection. Consequently, objects in the ERP format undergo significant distortion, particularly for those located near the top and bottom regions of the image. Image completion methods for panoramic images are thus required to address the spherical distortion inherent in their structure and ensure the preservation of omnidirectional continuity. Early studies (Sumantri & Park, 2020; Akimoto et al., 2019; Lu et al., 2021) primarily focus on ensuring omnidirectional continuity by adopting convolution operation with the circular padding strategy. To address the spherical distortion, there are strategies such as using extra depth information (Oh et al., 2022), adding projection loss (Somanath & Kurz, 2021), and leveraging scene-symmetry properties (Hara et al., 2021). In order to generate visually pleasant content, Omnidreamer (Akimoto et al., 2022) employs transformerbased sampling strategy to implicitly learn the spherical structure. To achieve a smooth transition between the real and generated content, Cylin-Painting (Liao et al., 2024) incoporates learnable positional encoding cues with existing features that are still processed by standard 2D convolution. Leveraging the powerful diffusion model, PanoDiff (Wang et al., 2023a) employs the ControlNet (Zhang et al., 2023) with the positions as an additional input and also uses spherical loss to regularise panoramic content. To further accomodate intricate visual details, AOG-Net (Lu et al., 2024)

proposes an autoregressive pipeline for panoramic image outpainting, utilising feature remapping for panoramic content. On the other hand, PanoDiffusion (Wu et al., 2024b) introduces the depth modality during training to enhance panoramic awareness. Unfortunately, the above methods dominantly address the distortion by soft regularisations, from the perspective of guiding the overall end-to-end training, i.e., in a macro way. This may result in suboptimal performance since the macro constraints may not fully contribute to learning the panoramic content as expected.

# 3. Methodology

# 3.1. Spherical Noise for Structural Prior

The noise plays an essential role that relates to the randomness of outpainting, which benefits the overall performance by considering the panoramic structural prior. More specifically, since panoramic images are commonly projected into the ERP planar format (Ye et al., 2018) for processing, we aim to build an explicit structural prior knowledge for the ERP format to guide the panoramic image outpainting process. Given the fact that images in the ERP format exhibit higher pixel density near the equator and lower pixel density near the poles, modelling the coherence of panoramic images using i.i.d. Gaussian noise, the de facto choice for planar images, is mismatched with the intrinsic characteristics of panoramic ERP format. Instead, by modelling panoramic images as pixels uniformly distributed over the surface of a sphere, we are able to capture pixel coherence in the ERP format by sampling i.i.d. noise uniformly distributed in 3D space and projecting it into the ERP format at given resolutions.

Firstly, we take a analysis on the spherical property of the ERP format. We denote u, v as the coordinates in the ERP format and θ, ϕ are the spherical coordinates on the surface of the sphere, two widely used coordinates for panoramic images. Correspondingly, the transformation from spherical format to the ERP planar can be formulated by $\begin{array} { r } { u = \theta w \big / 2 \pi , v = ( \phi + \frac { \pi } { 2 } ) h \big / \pi } \end{array}$ , where w and h are the width and height of the image, respectively. Also, the spherical coordinates $( \theta , \phi )$ are defined by x = cos ϕ cos $\theta , y =$ cos ϕ sin θ, z = sin ϕ where x, y, z are the Cartesian coordinates in 3D space. Then, we assume that there are $m _ { w }$ sampling points per row in the ERP format. The actual geodesic distance $d _ { v }$ between two adjacent sampling points on the spherical surface, at height v in the ERP format, is given by

$$
d _ {v} = \frac {2 \pi}{m _ {w}} \cos \left(\frac {\pi v}{h} - \frac {\pi}{2}\right). \tag {1}
$$

Therefore, given the points at poles (v = 0 and $v = h )$ , $d _ { v } = 0$ means that all points essentially correspond to a single point in 3D space. We summarise this property as the sphere nature of the ERP format.

![](images/57cbcf9734630922214c552f9be77da995d7e202c08f304e2388037101a99c58.jpg)  
5=5

![](images/c5fb85eccf1c583aba80efe69db6a8e67f672404ee51adb994dbf1c3c9420156.jpg)  
5=30

![](images/3103db74e598fe33086e23ef448e37459954df259d0fea968df7735e50fee52e.jpg)  
\$ =1000   
Figure 2. Illustration of the i.i.d. spherical noise which exhibits non-i.i.d. nature after the ERP projection. We illustrate examples with different sampling densities ζ on the 3D spherical surface at a resolution of 128 × 64.

Based on the above analysis, we propose to sample noise in 3D space and transform it into the ERP format to capture the intrinsic structural prior. The uniform spherical noise is sampled by adopting sampling density ζ per unit length in 3D space. Accordingly, the number of sampling points $m _ { \phi }$ at latitude ϕ on the spherical surface can be formulated by

$$
m _ {\phi} = \lceil \zeta \cdot 2 \pi \cos \phi \rceil \tag {2}
$$

where 2π cos ϕ denotes the geodesic length and ⌈·⌉ is the ceiling function. Note that the number of sampling point at poles is 1. ζ essentially controls the sampling density of the i.i.d. Gaussian noise on 3D spherical surface. Therefore, given $m _ { w }$ as the fixed number of points for a row in the ERP format, for latitudes where $m _ { \phi } < m _ { w }$ , one sampling point on the spherical surface corresponds to multiple points within a single row in the ERP format. This one-to-many mapping inherently introduces spatial coherence in the ERP domain. As a result, i.i.d. noise in 3D space is no longer i.i.d. when projected into the ERP format. In contrast, for latitudes where $m _ { \phi } \ \ge \ m _ { w }$ , the i.i.d. property within a row in the ERP representation can be preserved. Both the sampling density ζ and the resolution, which determines the number of sampling points $m _ { w } ,$ , play an important role in modelling the sphere nature of the ERP format. Since the resolution of feature is set as 128 × 64 in our model, we further illustrate the non-i.i.d. noise (aka. structural prior) with different ζ in Figure 2. We also illustrate detailed analysis in Appendix A, by choosing ζ = 30 for our SpND model.

# 3.2. Spherical Deformable Convolution

Generally speaking, the deformable convolution (Dai et al., 2017) decomposes the standard convolution operation into two steps: 1) Sampling certain points from the input feature $\mathbf { F } _ { \mathrm { i n } }$ using a regular grid R; 2) Summing the sampled points weighted by W. In the standard convolution, the grid

$$
\mathcal {R} = \{(- 1, - 1), (- 1, 0), (- 1, 1), (0, - 1), \dots , (0, 1), (1, 1) \} \tag {3}
$$

defines a $3 \times 3$ kernel with stride 1 and determines the receptive field. For each location $\mathbf { p } _ { 0 }$ on the output feature $\mathbf { F _ { \mathrm { o u t } } }$ , the deformable convolution can be formulated by

$$
\mathbf {F} _ {\text { out }} (\mathbf {p} _ {0}) = \sum_ {\mathbf {p} _ {n} \in \mathcal {R}} \mathbf {W} (\mathbf {p} _ {n}) \cdot \mathbf {F} _ {\text { in }} (\mathbf {p} _ {0} + \mathbf {p} _ {n} + \Delta \mathbf {p} _ {n}), \tag {4}
$$

where $\mathbf { p } _ { n }$ enumerates the locations in R and $\{ \Delta \mathbf { p } _ { n } | n =$ $1 , \cdots , N \}$ denote the learnable offsets, where $N = | \mathcal { R } |$ .

Although deformable convolution improves the flexibility of the convolution operation, it still operates as the planar convolution, thus falling short in highly complex geometric scenes, such as spherical data of panoramic images. This issue stems from the fact that directly applying deformable convolution to panoramic images results in suboptimal initialization, as it relies on the regular grid R as the initial grid (Dai et al., 2017); this misaligns with the spherical structure inherited from panoramic images.

To address the mismatch, we propose the novel SDC layer that leverages the spherical gird, while integrating dynamic adaptability of deformable convolution for precise local feature aggregation. More specifically, the spherical neighbourhood of a kernel at position p0 can be defined as a spherical grid $\mathcal { R } ^ { s }$ as introduced in (Coors et al., 2018), using gnomonic projection (Frederick Pearson, 1990) and a tangent plane centred at p0 on sphere. The coordinate for $\mathbf { p _ { 0 } } = \left( \theta _ { 0 } , \phi _ { 0 } \right)$ is defined by its latitude $\phi _ { 0 } \in [ - \frac { \pi } { 2 } , \frac { \pi } { 2 } ]$ and longitude $\theta _ { 0 } \in [ - \pi , \pi ]$ . Recall that w and h are the width and height of the image in ERP format (Ye et al., 2018). Then, the $3 \times 3$ grid $\mathcal { R } ^ { t }$ on the tangent plane (Coors et al., 2018) is reformulated from (3) as

$$
\begin{array}{l} \mathcal {R} ^ {t} = \left\{\left(- \tan \Delta_ {\theta}, - \sec \Delta_ {\theta} \tan \Delta_ {\phi}\right), \left(- \tan \Delta_ {\theta}, 0\right), \right. \\ \left(- \tan \Delta_ {\theta}, \sec \Delta_ {\theta} \tan \Delta_ {\phi}\right), (0, - \tan \Delta_ {\phi}), \tag {5} \\ \dots , (0, \tan \Delta_ {\phi}), (\tan \Delta_ {\theta}, \sec \Delta_ {\theta} \tan \Delta_ {\phi}) \}, \\ \end{array}
$$

where $\begin{array} { l } { \Delta _ { \phi } \ = \ \frac { \pi } { h } } \end{array}$ and $\begin{array} { r } { \Delta _ { \theta } \ = \ \frac { 2 \pi } { w } } \end{array}$ denote unit steps on the tangent plane. By representing the elements within $\mathcal { R } ^ { t }$ as $\mathcal R ^ { t } = \{ ( \Delta i _ { n } , \Delta j _ { n } ) | n = 1 , \cdots , N \}$ , we can obtain the spherical gird $\mathcal R ^ { s } = \{ ( \theta ^ { n } , \phi ^ { n } ) | n = 1 , \cdot \cdot \cdot , N \}$ centred at position p0 through

$$
\theta^ {n} = \theta_ {0} + \tan^ {- 1} \left(\frac {\Delta i _ {n}}{\cos \theta_ {0} - \Delta j _ {n} \sin \theta_ {0}}\right), \tag {6}
$$

$$
\phi^ {n} = \sin^ {- 1} \left(\frac {1}{1 + \Delta d} (\sin \phi_ {0} + \Delta j _ {n} \cos \phi_ {0})\right),
$$

where $\Delta d = \sqrt { ( \Delta i _ { n } ) ^ { 2 } + ( \Delta j _ { n } ) ^ { 2 } }$ (Coors et al., 2018).

Therefore, the spherical grid Rs defines the $3 \times 3$ neighbourhood around position p0 on the sphere in our SDC layers. While a kernel size of 3 is used here as an example, the kernel size can be arbitrarily chosen in practice.

Furthermore, since the spherical grid serves as the initial grid for the SDC layer, the usage of unconstrained learnable offsets $\Delta { \bf p } _ { n }$ may deteriorate the deformable convolution to neglect the spherical prior knowledge. To address this issue, we impose constraints on the offsets to ensure that the SDC layer consistently integrates the spherical prior knowledge. For each location p0 on the output feature $\mathbf { F _ { \mathrm { o u t } } }$ and the input feature $\mathbf { F } _ { \mathrm { i n } }$ , the SDC layer can be formulated as

$$
\mathbf {F} _ {\text { out }} (\mathbf {p} _ {0}) = \sum_ {\tilde {\mathbf {p}} _ {n} \in \mathcal {R} ^ {s}} \mathbf {W} (\tilde {\mathbf {p}} _ {n}) \cdot \mathbf {F} _ {\text { in }} (\mathbf {p} _ {0} + \tilde {\mathbf {p}} _ {n} + \Delta \mathbf {p} _ {n}), \tag {7}
$$

where $\tilde { { \bf p } } _ { n }$ enumerates the locations in the spherical grid $\mathcal { R } ^ { s }$ , W denotes the learnable weights of the kernel, and the constraint for the learnable offsets is formulated by $| \Delta \mathbf { p } _ { n } | \leq$ $\sigma \cdot h$ where h denotes the height and σ is a hyperparameter which defines the ratio against the overall height.

# 3.3. Spherical-Nested Diffusion Model

Considering the sphere nature of panoramic images, our SpND model achieves panoramic image outpainting by integrating the structural prior and the newly proposed SDC layer. The overall architecture of our SpND model is illustrated in Figure 3. We further develop the CME module to preserve omnidirectional continuity and the spherical structure, thereby significantly enhancing the quality of the generated content. More specifically, our CME is developed to encode the masked input image, which contains the known portion within panoramic images. Then, the trainable CME module employs $3 \times 3$ circular convolution layers with the sigmoid linear unit (SiLU) activation function (Elfwing et al., 2018), to transform the 3-channel masked image into the 256-channel feature, with an overall downsampling ratio of 8. Subsequently, the 256-channel latent feature is fed into the prior injection layer, which aggregates the structural prior for the ERP format. This process produces an intermediate output containing both ERP structural information and image completion details. We denote $\mathbf { X _ { m } }$ as the masked input and $\mathbf { P } _ { \mathrm { E R P } }$ as the ERP structural prior. The prior injection layer operats as follows,

$$
\mathbf {F} _ {\mathrm{pri}} = \mathbf {m} \odot \operatorname{CME} [ \mathbf {X} _ {\mathbf {m}} ] + (1 - \mathbf {m}) \odot \mathbf {P} _ {\mathrm{ERP}} \tag {8}
$$

where $\mathbf { F } _ { \mathrm { p r i } }$ denotes the aggregated feature, m is the mask, ⊙ represents the element-wise production, and $\mathrm { C M E } [ \cdot ]$ means the encode process of the CME module. Note that the mask m is provided by the input.

The spherical net is then established based on the sphericalnested blocks (SpNBs), which is introduced to extract panoramic features that guide the pre-trained diffusion model for panorama outpainting. More specifically, each SpNB incorporates an SDC layer to effectively encode spherical information, catering for the sphere nature with panoramic images. Furthermore, a vanilla deformable convolution layer is employed to bridge the spherical feature and the pre-trained diffusion model. This additional layer enhances compatibility with the pre-trained diffusion model, allowing for effective integration and improved performances.

Aligned with the architecture of ControlNet (Zhang et al., 2023), we adopt the SiLU activation function (Elfwing et al.,

![](images/cae9717b3c74814ecb3057b21233241a3496cd14a47026fdba238e4a109f10d9.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    A["Masked Input"] --> B["Circular Conv 3x3"]
    B --> C["Circular Conv 3x3"]
    C --> D["Circular Conv 3x3"]
    D --> E["Spherical Net"]
    E --> F["Prior Injection"]
    F --> G["SpNB"]
    G --> H["SpNB"]
    H --> I["..."]
    I --> J["SpNB"]
    J --> K["..."]
    K --> L["Spherical Net"]
    L --> M["Zt-1"]
    M --> N["Denoising U-Net"]
    N --> O["Zt-1"]
    O --> P["Spherical Net"]
    P --> Q["Z0"]
    Q --> R["Image Decoder"]
    R --> S["Output Panorama"]
    
    T["Spherical Deformable Convolution (SDC)"] --> U["Conv Kernel"] --> V["S-Grid"] --> W["Deform"] --> X["C"] --> Y["Spherical Kernel"] --> Z["Offset Kernel"] --> AA["Output"]
    
    AB["Spherical-Nested Block (SpNB)"] --> AC["SiLU"] --> AD["SDC"] --> AE["TimeEmbed"] --> AF["SiLU"] --> AG["Deformable Conv"] --> AH["CrossAttn"] --> AI["Cross-Attn"]
    
    subgraph Legend
        J
        K
        L
        M
        N
        O
        P
        Q
        R
        S
        T
        U
        V
        W
        X
        Y
        Z
        AA
        AB
        AC
        AD
        AE
        AF
        AG
        AH
        AI
        AJ
        AK
        AL
        AM
        AN
        AO
        AP
        AQ
        AR
        AS
        AT
        AU
        AV
        AW
        AX
        AY
        AZ
        BA
        BB
        BC
        BD
        BE
        BF
        BG
        BH
        BI
        BJ
        BK
        BL
        BM
        BN
        BO
        BP
        BPB
        BPB
        BPB
    end
    
    style Legend fill:#f9f9f9,stroke-dasharray: 5 5
```
</details>

Figure 3. The overall architecture of our proposed SpND model. The input of our SpND model is the masked images, which are first encoded by our circular mask encoder (CME) module. Subsequently, the prior injection layer integrates structural prior knowledge tailored for the ERP format into the encoded masked features. These features are subsequently refined by the spherical net, composed of SpNBs, which incorporate spherical prior knowledge inherent in the SDC layers of the SpNBs. The refined features then guide a pre-trained diffusion model to perform panoramic image outpainting using the repaint technique.

2018) to process features encapsulating the spherical property, combined with information from the original image, and sequentially integrate into the middle and output layers of the diffusion model. Assume that $f _ { \mathrm { S D } }$ denotes the pretrained diffusion model and fPDN denotes the spherical net. The process can be formulated as

$$
\epsilon_ {\boldsymbol {\psi}} = f _ {\mathrm{SD}} \left[ \mathbf {Z} _ {t}, \mathbf {T}, f _ {\mathrm{PDN}} \left[ \mathbf {F} _ {\text { pri }}, \mathbf {T} \right] \right], \tag {9}
$$

where T is the embedding of the input prompt and $\epsilon _ { \psi }$ is the output of our SpND model. Note that ψ denotes the learnable parameters. Furthermore, the final output $\epsilon _ { \psi }$ is decoded by the image decoder to get the completed panoramic image. Note that the image decoder is a pre-trained VAE model (Rombach et al., 2022). Moreover, the text embedding T is derived by encoding the text prompt using the widely adopted CLIP model (Radford et al., 2021).

During the training process, the pre-trained diffusion model and the VAE for encoding panoramic images are frozen, whilst the CME and the spherical net are trainable, as illustrated in Figure 3. We follow the standard process (Rombach et al., 2022) to train the conditional diffusion model via the default diffusion loss, which can be formulated by

$$
\mathcal {L} = \mathbb {E} _ {t, \mathbf {z} _ {0}, \epsilon} [ | | \epsilon - \epsilon_ {\psi} [ \mathbf {Z} _ {t}, t, \mathbf {T} ] | | _ {2} ^ {2} ], \tag {10}
$$

where ϵ is the Gaussian noise, t denotes the time, and $\mathbf { Z } _ { t }$ is the latent feature in the pre-trained diffusion model.

During the inference stage, the repaint technique (Lugmayr et al., 2022) is employed in our SpND model, with the mask aligned to the input, so as to perform the image outpainting task. We provide more details for the repainting technique in Appendix B. Utilizing the repaint technique for the pre-trained diffusion model ensures that the masked area remains consistent with the input, thereby enabling seamless image outpainting without introducing distortion to the provided regions.

# 4. Experimental Results

# 4.1. Experimental Settings

Datasets. To evaluate the performance of our SpND model, we employed the widely applied Matterport3D (Chang et al., 2017) and Structured3D (Zheng et al., 2020) dataset for comparison. Similar to (Lin et al., 2019), we obtained 10912 panoramic images with size 1024 × 512 for the Matterport3D dataset. A total of 9, 820 images were selected for the training, and all 1, 0912 images were used for evaluation to compute the sufficient statistics. For the Structured3D dataset, we followed the methodology outlined in (Wu et al., 2024b) to obtain 21,133 images, of which 19,019 images were used for training and all 21,133 images were used for evaluation to compute the sufficient statistics. Prompts were generated by the BLIP-2 (Li et al., 2023) model for both datasets.

Implementation Details. Our SpND model was trained based on the pre-trained weights of (Zhang et al., 2023). The hyperparameter ζ was set to 30 to obtain the structural prior for the ERP format. During training, we used the AdamW optimizer (Loshchilov & Hutter, 2019) with a learning rate of $1 0 ^ { - 5 }$ and a batch size of 4. During inference, the classifier-free guidance scale (Ho & Salimans, 2021) was set to 3.0 with a DDIM (Song et al., 2021) step of 50. Additionally, the hyperparameter σ in SDC layers was set to 0.04 to constrain the learnable offset. To rigorously evaluate the efficacy of integrating prior knowledge, we utilised the general prompt ”Outpainting” to validate our approach. This choice was motivated for fair comparisons, by the fact that several baseline methods do not incorporate prompts as part of their input. Furthermore, to demonstrate that performance could potentially be further improved with application-specific prompts, we trained an additional model incorporating varying text prompts, denoted as $\mathrm { S p N D _ { p r o m p t } }$ .

Table 1. Quantitative comparisons with state-of-the-art methods on Matterport3D and Structured3D. The best results and the second-best results are highlighted in bold, underline. 

<table><tr><td>-</td><td colspan="5">Matterport3D</td><td colspan="5">Structured3D</td></tr><tr><td>Methods</td><td>FID ↓</td><td> $FID_{hori}$  ↓</td><td>Avg ↑</td><td>Pre ↑</td><td>Recall ↑</td><td>FID ↓</td><td> $FID_{hori}$  ↓</td><td>Avg ↑</td><td>Pre ↑</td><td>Recall ↑</td></tr><tr><td>SIG-SS (Hara et al., 2021)</td><td>64.81</td><td>50.53</td><td>0.22</td><td>0.42</td><td>0.01</td><td>117.84</td><td>72.51</td><td>0.15</td><td>0.29</td><td>0.01</td></tr><tr><td>TFill (Zheng et al., 2022)</td><td>77.69</td><td>85.28</td><td>0.08</td><td>0.15</td><td>0.01</td><td>85.00</td><td>84.49</td><td>0.04</td><td>0.06</td><td>0.01</td></tr><tr><td>OmniDreamer (Akimoto et al., 2022)</td><td>51.69</td><td>44.44</td><td>0.31</td><td>0.45</td><td>0.16</td><td>20.63</td><td>24.94</td><td>0.35</td><td>0.51</td><td>0.19</td></tr><tr><td>PanoDiff (Wang et al., 2023a)</td><td>13.45</td><td>13.11</td><td>0.50</td><td>0.24</td><td>0.75</td><td>11.11</td><td>9.75</td><td>0.53</td><td>0.36</td><td>0.69</td></tr><tr><td>AOG-Net (Lu et al., 2024)</td><td>38.56</td><td>44.27</td><td>0.10</td><td>0.16</td><td>0.04</td><td>85.39</td><td>98.11</td><td>0.10</td><td>0.19</td><td>0.01</td></tr><tr><td>PanoDiffusion (Wu et al., 2024b)</td><td>39.43</td><td>17.34</td><td>0.49</td><td>0.55</td><td>0.42</td><td>9.80</td><td>10.84</td><td>0.49</td><td>0.55</td><td>0.43</td></tr><tr><td>SpND (Ours)</td><td>9.08</td><td>10.64</td><td>0.62</td><td>0.52</td><td>0.72</td><td>7.37</td><td>6.05</td><td>0.62</td><td>0.62</td><td>0.62</td></tr><tr><td> $SpND_{prompt}$  (Ours)</td><td>6.67</td><td>6.58</td><td>0.64</td><td>0.52</td><td>0.76</td><td>4.60</td><td>4.75</td><td>0.64</td><td>0.66</td><td>0.62</td></tr></table>

![](images/8a169bea3d39f22d14c8d5394ca610241bac20ea4e1f167213718596c64c8639.jpg)  
(a) Ground Truth

![](images/0f468ac971bbef1cb06bd8d261d28d257c1fef55f85f2e94f6947fc4b31ac8d4.jpg)  
(b) Input Masked Image

![](images/f45f382dd6cdf3bc58b62442c00fcea6d0c49071d61a8ce2107820d2e4a523d5.jpg)  
(c) OmniDreamer

![](images/940b6e19e94a94cd66578b6759878d504b6e64a179df2fabffa05855c9fa5492.jpg)  
(d) PanoDiffusion

![](images/87b1f6498fa931f6d06db7144e377a427e396608e4e59b0ed6f614ab55c6fbe8.jpg)  
(e) SpND (ours)

![](images/35b96eb2d2ef8263adbff2cb7523c5de806e1b5e2844f098d714cfabce56a112.jpg)  
(f) AOG-Net

![](images/cdac49a99da3eb253bcdb77159b5a765f7bee4c8076d3745a2bd15abb915bc53.jpg)  
(g)TFill

![](images/30e48c4d50ce5051219f87159c7c6d78e1317116e0b3d4d413a8af0e67025f90.jpg)  
(h) SIG-SS

![](images/1190d0dd5388b6e723b3d12f1e108813db5837f0a916fcaa68f49c16b1f6cf96.jpg)  
(i)PanoDiff

![](images/b31102e559104ea79f09ee1c9807048dcbcb19848abe4660882cb2e47c748b9b.jpg)  
(i) SpNDprompt (ours)

![](images/e260901142d9b2859efaaa8c48c85af18a3df5561dc16c71361e9ee562049170.jpg)  
(a) Ground Truth

![](images/bb9c138a6113d7000eed1c8426ba65102ff795ddb2ead39b4f1f6319bb467c03.jpg)  
(b) Input Masked Image

![](images/5632b27bf8a84edee830269f3fea35888df0b45fd304f4fa7fd17fb104e37190.jpg)  
(c) OmniDreamer

![](images/5864284bce6f3bf493dcb7844b70663c9a52ed03c243bef633d79ba0932811d0.jpg)  
(d) PanoDiffusion

![](images/b7cba3ff46a64e92bd3eec970ec85f2db16d100ab47d284b2b894afa6e0fb0b1.jpg)  
(e) SpND (ours)

![](images/af74d84dd45dbb6d841e69d34746fb3a3f311387e9bf437264cfa5e5c233b697.jpg)  
(f) AOG-Net

![](images/5503369eabfc3da627978e6c6ecbc90d58087cf9b22473f2e8308163c75db7e8.jpg)  
(g)TFill

![](images/9a3de346f82bec218d716a0a738904814d212cb01f4add1616ee6f443561a6f9.jpg)  
(h) SIG-SS

![](images/25ffaeb9f4695630257a4a533caa4ed7da031b5d2873aea2399d6e553b0d3e64.jpg)  
(i) PanoDiff

![](images/a71119ba1f5cd18b60abb43e268aa6bc3aab8f4591efab0428ed323d60c04e20.jpg)  
(j) SpNDprompt (ours)   
Figure 4. Qualitative comparisons of panoramic image outpainting on the Matterport3D dataset. The centre mask is employed to generate the masked input image, and the results are the corresponding completed images obtained by different methods.

Baselines. For panoramic image outpainting, we mainly compared with the existing state-of-the-art panorama outpainting methods including SIG-SS (Hara et al., 2021), OmniDreamer (Akimoto et al., 2022), PanoDiff (Wang et al., 2023a), AOG-Net (Lu et al., 2024), and PanoDiffusion (Wu et al., 2024b). Additionally, we compared with the image inpaingting method TFill (Zheng et al., 2022) following PanoDiffusion (Wu et al., 2024b). We fine-tuned all the baseline methods on the Matterport3D dataset, whilst fine-tuning almost all the baseline methods on the Structured3D datasets (except for the PanoDiffusion model where the pre-trained weights exist on Structured3D) for fair comparison.

Metrics. We utilised the commonly adopted Frechet in- ´ ception distance (FID) metric to assess the quality of the completed panoramic images. More specifically, we followed the method in (Wu et al., 2024b) to directly compute the FID score. To further refine the evaluation of image quality for panoramic images, we adopted the approach proposed in (Tang et al., 2023) and computed $\mathrm { F I D _ { h o r i } }$ . This was achieved by calculating the $\mathrm { F I D _ { h o r i } }$ scores using 8 equally spaced $5 1 2 \times 5 1 2$ cropped images, extracted from a panoramic image with size $1 0 2 4 \times 5 1 2$ . Additionally, the precision (denoted as Pre) and recall metrics (Kynka¨anniemi ¨ et al., 2019) were also calculated to comprehensively assess outpainting performance. A higher precision value indicates better image quality, while a higher recall value reflects improved diversity in the outpainted images. We average the precision and recall metric (denoted as Avg) to comprehensively evaluate the performance for panoramic image outpainting.

![](images/9c1ebf3ee695d07812d0789ffb07794a904751a80745d6221532ea3bf3faf936.jpg)  
(a) Ground Truth

![](images/57fb2f39f7d22cc9e1b82cffef4674e05303423c9c917deb2c7188c48b6b22f3.jpg)  
(b) Input Masked Image

![](images/a911fdc736ba703cfffb2be55bf771b2762fec6d1ce5b72b612127726bea25e4.jpg)  
(c) OmniDreamer

![](images/b35df0ae1c9bfd07beecfe6c694f02b79d6ff9959a5e71a5d9001c9f162a0222.jpg)  
(d) PanoDiffusion

![](images/311f7f7f135ac7ebfb046cf8714c41dc4782843db8c23642a9b74122cd2be111.jpg)  
(e) SpND(ours)

![](images/b6a17e5ee73e14a80541742acb8f272787f3acd3be377d4cede107393139d8c9.jpg)  
(f) AOG-Net

![](images/2d7fe088a7bcc9a92dae3a7429380051ff2de21fbb5bf57bf61cbc779d25580a.jpg)  
(g) TFill

![](images/d22e482b5378c70e1cd95f2e37c7c6e759b44985b872a269fe022576fe20ee7e.jpg)  
(h) SIG-SS

![](images/a0ba4de7e749a0a24431d0a53ce080f66307900d2de798cba5d5eda9fdcdb581.jpg)  
(i) PanoDiff

![](images/ea151121c2acb721eae38de6a4b4cc6f8ad35dce503879895e303ad872c09a0d.jpg)  
(j) SpNDprompt (ours)

![](images/26a388f6662baec3888b12d47450f1b4ded5bc0e7a2dafbd9f2b5f731b77eed4.jpg)  
(a) Ground Truth

![](images/ecc1401782fbbb7a6d2c621ef6196deea9752d0c85f7a148f70441b8719771c2.jpg)  
(b) Input Masked Image

![](images/b1b9c7b7e4685f78250202c6f34a0b1e1e5e03bf785c3641cf4609b340ae6fff.jpg)  
(c) OmniDreamer

![](images/9b667ec40f95afa21645ab26bba43c88ee4b4fee527e66190b8d1aa49b378207.jpg)  
(d) PanoDiffusion

![](images/a3ad70ec9859093789368fedb103d3aca96e53ed7e6c72b4d227e60080099e91.jpg)  
(e) SpND (ours)

![](images/7904d6aa9b79fa8c7489a2c49bad26667a66ff20fd9c6536994b51ae282a3e7d.jpg)  
(f) AOG-Net

![](images/c8fd2c85b8f619f459289c910e35a1a100b8fc796674c2a74562d0fe473e2cb3.jpg)  
(g) TFill

![](images/05e7af0927abb8958321ab2de20b4c4ef6fa9b4a90eff49e0a6596e56ca5aa78.jpg)  
(h) SIG-SS

![](images/00c4be4ea721316be507d63be1e54c05d1f595b9497ab1befbc1057c9de0c55f.jpg)  
(i)PanoDiff

![](images/5d0d0c60eeef8593a059471985cee9ca6682921f0574d5821460a5c7712312b8.jpg)  
(j) SpNDprompt (ours)   
Figure 5. Qualitative comparison of panoramic image outpainting on the Structured3D dataset. The centre mask is employed to generate the masked input image, and the results are the corresponding completed images obtained by different methods.

# 4.2. Comparison for Panoramic Image Outpainting

Quantitative Results. The widely adopted centre mask was employed in performing the panoramic image outpainting task for all comparing methods. As reported in Table 1, the proposed SpND model consistently achieves the best performance by taking into account both the quality and diversity of the outpainted images. More importantly, with specific prompts as input, the overall outpainting performance can be even better across all metrics, as shown in the last two rows of Table 1. While PanoDiff achieves superior recall for the Structured3D dataset, its FID and the precision are quite limited. In addition, the GAN-based methods including SIG-SS and TFill exhibit extreme low recall scores, indicating that the panoramic outpainting diversity is poor.

Qualitative Results. We randomly chose 2 scenes from both dataset and reported the subjective results in Figures 4 and 5. As can be seen from those figures, panoramic images outpainted by our SpND model demonstrate distinct superior quality and optimal alignment with the provided input sections, also achieving the smoothest transitions at the boundary between the real and generated components compared to other methods. Furthermore, with the newly proposed structural prior and SDC layers, our SpND model preserves the spherical structure well and outpaints visually compelling panoramic content. In addition, image quality and semantic continuity can be further improved through the usage of specific prompts in our SpND model.

Outpainting from View Images. To further evaluate the performance and the robustness of the proposed SpND model, we conducted panoramic image outpainting experiments using view images. For comparison, we selected the best two existing methods based on their performance in prior experiments and performed the comparison on the Structured3D dataset1. As illustrated in Figure 6 and Table 2, our SpND model demonstrates superior performances, achieving the best results both subjectively and objectively.

Outpainting with Mask near the Pole. To further evaluate the performance of our method on different masks, we conducted additional experiments on masks located near the poles, whereby the spherical distortion of masks becomes apparent. More specifically, we projected a $2 5 6 \times 2 5 6$ image onto the ERP format at the viewpoint $\theta = 9 0 ^ { \circ } , \phi = 6 0 ^ { \circ }$ , to obtain a highly-distorted mask near the pole. We then retrained our SpND method, along with the best two baseline methods (i.e., PanoDiff and PanoDiffusion) on the Matterport3D dataset for comparison. We report the quantitative results in Table 3 and the qualitative results in Figure 7, which verifies the superior performance of our SpND method. This also verifies the advantages of our SpND method that arises from the spherical noise and SDC operation to accommodate panoramic deformation.

![](images/8510625d4c414e7253257516edc756440455a361aa8609d9a7e188351ba63b3f.jpg)  
(a) Ground Truth

![](images/c6954490d94f80a1b018ec0c691804e501e0809e0538e300188e7ddf1fbee716.jpg)  
(b) Masked Input

![](images/19b9690bc1f7fe317fd5372c1e0b261d23bb58994fc70fe45e84765606b264f0.jpg)  
(c) PanoDiffusion

![](images/80a01661de6204785668d186c36997ef44450f79f1136cd7cca9231b212bc83b.jpg)  
(d) PanoDiff

![](images/a3ede3d49888bba539e80a33fc1f5974db06a994e57ee411b496bff0970219a6.jpg)  
(e) SpNDprompt

Figure 6. Qualitative comparison of panoramic image outpainting from view images on the Structured3D dataset. The centre view mask is employed to generate the masked input panoramic view, and the results are the completed images obtained by different methods.   
![](images/5f37e881869ead59070a6000cc3381edd01c02894a273e995a838f632f55a872.jpg)  
(a) Ground Truth

![](images/a9aa65219757bdfff576560420f2d950bd889ffa615578524d30002fe867dcec.jpg)  
(b) Masked Input

![](images/61c478c5a26971ad87a5fad452490a39805cb0c275b1d2f51e7585e7b70c5dc9.jpg)  
(c) PanoDiffusion

![](images/e4a3783bfa03e7353408d12cda5c4639c2629e65c0af257ca48347b1366b35ea.jpg)  
(d) PanoDiff

![](images/5c683cd07ad0f18dd460be9e03f775a61e66d4b33603538aaeea5bce34a83194.jpg)  
(e) SpNDprompt   
Figure 7. Qualitative comparison of panoramic image outpainting with mask near the pole on the Matterport3d dataset. The distorted mask near the pole is employed to generate the masked input image, and the results are the completed images obtained by different methods.

Table 2. Comparisons for panoramic image outpainting from view images on the Structured3D dataset. The best results are in bold. 

<table><tr><td>Methods</td><td>FID ↓</td><td> $FID_{hori}$  ↓</td><td>Avg ↑</td><td>Pre ↑</td><td>Recall ↑</td></tr><tr><td>PanoDiffusion</td><td>18.06</td><td>17.64</td><td>0.36</td><td>0.38</td><td>0.34</td></tr><tr><td>PanoDiff</td><td>9.20</td><td>8.19</td><td>0.54</td><td>0.40</td><td>0.68</td></tr><tr><td> $SpND_{prompt}$  (ours)</td><td>4.23</td><td>3.98</td><td>0.64</td><td>0.66</td><td>0.61</td></tr></table>

Table 3. Comparisons for panoramic image outpainting with mask near the pole on Matterport3D. The best results are in bold. 

<table><tr><td>Methods</td><td>FID ↓</td><td> $FID_{hori}$  ↓</td><td>Avg ↑</td><td>Pre ↑</td><td>Recall ↑</td></tr><tr><td>PanoDiffusion</td><td>15.92</td><td>17.76</td><td>0.48</td><td>0.54</td><td>0.42</td></tr><tr><td>PanoDiff</td><td>13.97</td><td>14.91</td><td>0.48</td><td>0.30</td><td>0.65</td></tr><tr><td> $SpND_{prompt}$  (ours)</td><td>8.49</td><td>8.84</td><td>0.57</td><td>0.44</td><td>0.69</td></tr></table>

# 4.3. Ablation Study

Effectiveness of Core Components. We first conducted an ablation study on the Matterport3D dataset to evaluate the impact of integrating the structure prior $\mathbf { P } _ { \mathrm { E R P } }$ , the spherical grid $\mathcal { R } ^ { s }$ , and the learnable offsets $\Delta { \bf p } _ { n }$ for the SDC layer, all of which consist of the core components in our SpND model. For simplicity, all ablation studies were conducted using the general prompt ”Outpainting” and report the results in Table 4 and Figure 8. More specifically, we verified the effectiveness of incorporating the structural prior knowledge tailored for the ERP format, as well as the spherical

Table 4. Ablation study of core components in SpND model. The comparison includes three variants of our SpND model: without the structural prior (denoted as w./o. $\mathbf { P } _ { \mathrm { E R P } } ) ;$ ; without the SDC layer (denoted as w./o. Rs) and the one without the learnable offsets (denoted as w./o. $\Delta { \bf p } _ { n } )$ . The best results are in bold. 

<table><tr><td>Methods</td><td>FID ↓</td><td> $FID_{hori} \downarrow$ </td><td>Avg ↑</td><td>Pre ↑</td><td>Recall ↑</td></tr><tr><td>w/o.  $P_{ERP}$ </td><td>9.93</td><td>12.05</td><td>0.53</td><td>0.42</td><td>0.63</td></tr><tr><td>w/o.  $\mathcal{R}^s$ </td><td>11.64</td><td>15.65</td><td>0.52</td><td>0.41</td><td>0.62</td></tr><tr><td>w/o.  $\Delta p_n$ </td><td>12.76</td><td>11.17</td><td>0.56</td><td>0.46</td><td>0.65</td></tr><tr><td>SpND (ours)</td><td>9.08</td><td>10.64</td><td>0.62</td><td>0.52</td><td>0.72</td></tr></table>

prior knowledge embedded in the grid $\mathcal { R } ^ { s }$ of the SDC layer. The integration of structural prior knowledge enhances the quality and diversity of completed images, as evidenced by improvements in FID, precision, and recall metrics in Table 4. The enhancements in image quality are shown in Figure 8. Furthermore, leveraging the spherical grid $\mathcal { R } ^ { s }$ within the SDC layer as the spherical prior knowledge yields additional improvements in image quality and diversity. These experimental results underscore the critical components in substantially enhancing the performance of panoramic image outpainting. Please note that, the learnable offsets, in conjunction with the spherical grid $\mathcal { R } ^ { s }$ , play a crucial role in generating high-quality panoramic content, whereas the spherical prior $\mathcal { R } ^ { s }$ alone does not yield comparable results. As reported in Table 4, using the spherical grid $\mathcal { R } ^ { s }$ without the learnable offsets results in a significant degradation in both outpainting quality and diversity.

Effectiveness of Packed Modules. We further evaluated the effectiveness of the architecture employed in our spherical net and the prior injection layer on Matterport3D dataset. Note that without the spherical noise and the proposed SpNB, our SpND model reduces to a variant of Control-Net (Zhang et al., 2023) with the CME module (denoted as Repaint). Moreover, since our SDC layer uses the spherical grid similar to SphereNet (Coors et al., 2018), we further compared a variant model which replaces the deformable convolution and the SDC layer in the SpNB with the spherical convolution proposed in (Coors et al., 2018) (denoted as SphereNet). We report the results in Table 5 and Figure 8. From the table and figure, the proposed SpND model, which integrates the sphere nature in a micro way, significantly enhances the quality of the generated content. However, this improvement comes at the cost of a slight reduction in outpainting diversity. Overall, the outpainting performance is notably improved, as evidenced by the FID metric in Table 5. More importantly, our model consistently achieves the best performance, demonstrating that directly employing spherical convolution fails to effectively integrate with the pre-trained diffusion model.

![](images/5b2d625e0ed553d2d935f8c2eb88bd6a5e5061ea90e198fad9a1e15328dd467b.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern hallway with curved walls and carpeted flooring (no visible text or symbols)
</details>

(a) Ground Truth

![](images/21e73b5010ebdecc953838cf4bb2c5d10df412c2f2e880094a7146b8a42c38df.jpg)

<details>
<summary>natural_image</summary>

Interior hallway view with a single light fixture on the wall (no text or symbols visible)
</details>

(b) Input Masked Image

![](images/48dee8760992ca512fb81c34b83f350d9d7db78418ac54cf81536cd84b771cfd.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern hallway with curved walls and decorative wall art (no visible text or symbols)
</details>

(c) w./0. △pn

![](images/dc521ee3eef6f83366de4eca8d7973cdec0cde3c4c47f4f3f4937170bfe43040.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern, minimalist corridor with curved walls and ceiling fixtures (no visible text or symbols)
</details>

![](images/bcf5102da0d1b8f1044df24d89a9f5a55c2982c3af1bb071a5dc274a647513f2.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern architectural corridor with curved concrete walls and glass windows (no visible text or symbols)
</details>

(e) w./0. R

![](images/b58f4f23dcc19f9f09e090a695d3861bf4c24cc2e984905f37d6cccd1d4376a9.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior hallway with curved walls and stone flooring, no visible text or symbols
</details>

(f)RePaint

![](images/00f415b10c2ad912a37fe9f77f8edeb1a4e2e6fd317be901e56412556b0d19cb.jpg)

<details>
<summary>natural_image</summary>

Interior hallway with two large cylindrical tanks, no visible text or symbols
</details>

(g) SphereNet

![](images/1bc91f08f3704ee81cc46a26f13f6093f8fb62c2b9d06be6525b53c27fc20c5d.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior hallway with curved walls, a large red-and-white abstract painting on the left wall, and framed artworks on the right wall (no visible text or symbols)
</details>

(h) SpND (ours)   
Figure 8. Qualitative comparison result of the ablation study on Matterport3D dataset including most our ablation settings and the ground truth. Specifically, this comparison includes 3 variants of our SpND model described in Table 4, including SpND without the structural prior (denoted as w./o. PERP); SpND without the SDC layer (denoted as w./o. Rs) and SpND without the learnable offsets (denoted as w./o. $\Delta { \bf p } _ { n } ) $ , and two variants RePaint and SphereNet described in Table 5.

Table 5. Ablation study of packed modules in the SpND model. The comparison includes two variants: Repaint that utilises circular convolution without integrating the prior knowledge and SphereNet that replaces the deformable convolution and the SDC layer with spherical convolution proposed in (Coors et al., 2018). The best results are in bold. 

<table><tr><td>Methods</td><td>FID ↓</td><td> $FID_{hori}$  ↓</td><td>Avg ↑</td><td>Pre ↑</td><td>Recall ↑</td></tr><tr><td>Repaint</td><td>10.10</td><td>12.89</td><td>0.58</td><td>0.42</td><td>0.74</td></tr><tr><td>SphereNet</td><td>11.78</td><td>15.74</td><td>0.54</td><td>0.41</td><td>0.67</td></tr><tr><td>SpND (ours)</td><td>9.08</td><td>10.64</td><td>0.62</td><td>0.52</td><td>0.72</td></tr></table>

Table 6. Ablation study evaluating the impact of varying sampling densities ζ quantitatively on the Matterport3D dataset. The best results are in bold. 

<table><tr><td>Sampling Density</td><td>FID ↓</td><td> $FID_{hori}$  ↓</td><td>Avg ↑</td><td>Pre ↑</td><td>Recall ↑</td></tr><tr><td> $\zeta = 15$ </td><td>12.74</td><td>15.52</td><td>0.52</td><td>0.38</td><td>0.65</td></tr><tr><td> $\zeta = 30$ </td><td>9.08</td><td>10.64</td><td>0.62</td><td>0.52</td><td>0.72</td></tr><tr><td> $\zeta = 45$ </td><td>8.84</td><td>10.27</td><td>0.62</td><td>0.51</td><td>0.72</td></tr><tr><td> $\zeta = \infty (w./o. P_{ERP})$ </td><td>9.93</td><td>12.05</td><td>0.53</td><td>0.42</td><td>0.63</td></tr></table>

Quantitative Evaluations for Spherical Noise. The noni.i.d. noise, which serves as explicit structural prior knowledge tailored for the ERP format, constitutes a key com-

ponent of our SpND model, with the parameter ζ playing a pivotal role as mentioned in Section 3.1. To conduct a quantitative analysis of how sampling density influences the performance, we further performed an ablation study by varying ζ and report the results in Table 6. Here, ζ = ∞ corresponds to the w./o. PERP configuration in Table 4, which employs the widely applied i.i.d. noise. As shown in Table 4, our SpND method achieves optimal performance at ζ = 45 and ζ = 30, consistent with the analysis in Appendix A. In particular, ζ = 30 achieves performance comparable to ζ = 45 while offering improved computational efficiency. A reduced sampling density (ζ = 15) marginally degrades performance but still surpasses existing baseline methods. These results confirm the rationale and robustness of our proposed spherical noise.

# 5. Conclusion

In this paper, we have proposed the novel spherical-nested diffusion (SpND) model, that for the first time integrated panoramic characteristics within the design of network architecture, namely, in a micro way. More specifically, we first developed the spherical noise catering for the structural prior of panoramic images. To capture correct content within panoramic format, we have adopted the spherical grid within deformable convolution and have proposed the spherical deformable convolution (SDC) layers. Leveraging the structural prior and the SDC layer, we have introduced the SpND model which integrated the spherical net with a pre-trained diffusion model to achieve superior outpainting for panoramic images. Extensive experiments have verified the superior performances for outpainting high-quality and diverse panoramic images, surpassing all the baselines. We may also need to point out our limitations by using Repaint technique in the latent space, which leads to slight distortion between the input and output against masks. Future works are expected to further refine the mask areas during the diffusion process.

# Acknowledgement

This work was supported by Natural Science Foundation of China (NSFC) under Grants 62450131, 62206011, 62231002 and 62401027, and Beijing Natural Science Foundation under Grant L223021.

# Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

# References

Akimoto, N., Kasai, S., Hayashi, M., and Aoki, Y. 360- degree image completion by two-stage conditional gans. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 4704–4708, 2019. doi: 10.1109/ICIP. 2019.8803435.   
Akimoto, N., Matsuo, Y., and Aoki, Y. Diverse plausible 360-degree image outpainting for efficient 3dcg background creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11441–11450, June 2022.   
Chang, A., Dai, A., Funkhouser, T., Halber, M., Niessner, M., Savva, M., Song, S., Zeng, A., and Zhang, Y. Matterport3d: Learning from rgb-d data in indoor environments. In International Conference on 3D Vision (3DV), 2017.   
Coors, B., Condurache, A. P., and Geiger, A. Spherenet: Learning spherical representations for detection and classification in omnidirectional images. In Proceedings of the European conference on computer vision (ECCV), pp. 518–533, 2018.   
Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., and Wei, Y. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 764–773, 2017.   
de La Garanderie, G. P., Abarghouei, A. A., and Breckon, T. P. Eliminating the blind spot: Adapting 3d object detection and monocular depth estimation to 360 panoramic imagery. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 789–807, 2018.   
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.

Eiris, R., Gheisari, M., and Esmaeili, B. Using panoramic augmented reality to develop a virtual safety training environment. 04 2018. doi: 10.1061/9780784481288. 004.   
Elfwing, S., Uchibe, E., and Doya, K. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3–11, 2018.   
Esser, P., Rombach, R., and Ommer, B. Taming Transformers for High-Resolution Image Synthesis . In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12868–12878, Los Alamitos, CA, USA, June 2021. IEEE Computer Society. doi: 10.1109/CVPR46437.2021.01268.   
Frederick Pearson, I. Map ProjectionsTheory and Applications. CRC Press, 1990.   
Gao, S., Yang, K., Shi, H., Wang, K., and Bai, J. Review on panoramic imaging and its applications in scene understanding. IEEE Transactions on Instrumentation and Measurement, 71:1–34, 2022. doi: 10.1109/TIM.2022. 3216675.   
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.   
Hara, T., Mukuta, Y., and Harada, T. Spherical image generation from a single image by considering scene symmetry. Proceedings of the AAAI Conference on Artificial Intelligence, 35(2):1513–1521, May 2021. doi: 10.1609/aaai.v35i2.16242.   
Ho, J. and Salimans, T. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.   
Huang, X., Cheng, X., Geng, Q., Cao, B., Zhou, D., Wang, P., Lin, Y., and Yang, R. The apolloscape dataset for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.   
Iizuka, S., Simo-Serra, E., and Ishikawa, H. Globally and locally consistent image completion. ACM Trans. Graph., 36(4), July 2017. ISSN 0730-0301. doi: 10.1145/3072959.3073659.   
Jaus, A., Yang, K., and Stiefelhagen, R. Panoramic panoptic segmentation: Insights into surrounding parsing for mobile agents via unsupervised contrastive learning. IEEE Transactions on Intelligent Transportation Systems, 24 (4):4438–4453, 2023.

K., V. and Ali, R. Image outpainting and harmonization using gans. International Journal of Scientific Research in Computer Science, Engineering and Information Technology, pp. 294–297, 05 2020. doi: 10.32628/CSEIT206370.   
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In Bengio, Y. and LeCun, Y. (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.   
Kynka¨anniemi, T., Karras, T., Laine, S., Lehtinen, J., and ¨ Aila, T. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.   
Lee, I.-C. and Tsai, F. Applications of panoramic images: From 720° panorama to interior 3d models of augmented reality. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XL-4/W5:189–192, 2015.   
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730–19742. PMLR, 2023.   
Li, W., Yu, X., Zhou, K., Song, Y., and Lin, Z. Image inpainting via iteratively decoupled probabilistic modeling. In The Twelfth International Conference on Learning Representations, 2024.   
Liao, K., Xu, X., Lin, C., Ren, W., Wei, Y., and Zhao, Y. Cylin-painting: Seamless 360° panoramic image outpainting and beyond. IEEE Transactions on Image Processing, 33:382–394, 2024.   
Lin, C. H., Chang, C., Chen, Y., Juan, D., Wei, W., and Chen, H. COCO-GAN: generation by parts via conditional coordinating. In IEEE International Conference on Computer Vision (ICCV), 2019.   
Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.   
Lu, C.-N., Chang, Y.-C., and Chiu, W.-C. Bridging the visual gap: Wide-range image blending. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 843–851, June 2021.   
Lu, Z., Hu, K., Wang, C., Bai, L., and Wang, Z. Autoregressive omni-aware outpainting for open-vocabulary 360- degree image generation. Proceedings of the AAAI Conference on Artificial Intelligence, 38(13):14211–14219, Mar. 2024. doi: 10.1609/aaai.v38i13.29332.

Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., and Van Gool, L. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11461–11471, June 2022.   
Oh, C., Cho, W., Chae, Y., Park, D., Wang, L., and Yoon, K.- J. Bips: Bi-modal indoor panorama synthesis via residual depth-aided adversarial learning. In Avidan, S., Brostow, G., Cisse, M., Farinella, G. M., and Hassner, T. (eds.), ´ Computer Vision – ECCV 2022, pp. 352–371, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031- 19787-1.   
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., and Efros, A. A. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.   
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PMLR, 2021.   
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684–10695, June 2022.   
Shen, Z., Lin, C., Liao, K., Nie, L., Zheng, Z., and Zhao, Y. Panoformer: panorama transformer for indoor 360 depth estimation. In European Conference on Computer Vision, pp. 195–211. Springer, 2022.   
Somanath, G. and Kurz, D. Hdr environment map estimation for real-time augmented reality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11298–11306, June 2021.   
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.   
Sumantri, J. S. and Park, I. K. 360 panorama synthesis from a sparse set of images with unknown field of view. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2386–2395, 2020.   
Tang, S., Zhang, F., Chen, J., Wang, P., and Furukawa, Y. Mvdiffusion: enabling holistic multi-view image generation with correspondence-aware diffusion. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, 2023.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998–6008, 2017.   
Wan, Z., Zhang, J., Chen, D., and Liao, J. High-fidelity pluralistic image completion with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4692–4701, October 2021.   
Wan Abd Arif, W. N., Wan Ahmad, W. F., Nordin, S. M., Abdullah, A., and Sivapalan, S. Designing 3 dimensional virtual reality using panoramic image. In Badioze Zaman, H., Robinson, P., Petrou, M., Olivier, P., Schroder, H., and ¨ Shih, T. K. (eds.), Visual Informatics: Bridging Research and Practice, pp. 404–414, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg. ISBN 978-3-642-05036-7.   
Wang, C. Research on panoramic image processing technology based on virtual reality technology. In 2019 International Conference on Virtual Reality and Intelligent Systems (ICVRIS), pp. 55–58, 2019. doi: 10.1109/ICVRIS. 2019.00022.   
Wang, H., Xiang, X., Fan, Y., and Xue, J.-H. Customizing 360-degree panoramas through text-to-image diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4933–4943, 2024.   
Wang, J., Chen, Z., Ling, J., Xie, R., and Song, L. 360- degree panorama generation from few unregistered nfov images. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 6811–6821, 2023a.   
Wang, S., Saharia, C., Montgomery, C., Pont-Tuset, J., Noy, S., Pellegrini, S., Onoe, Y., Laszlo, S., Fleet, D. J., Soricut, R., Baldridge, J., Norouzi, M., Anderson, P., and Chan, W. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18359–18369, June 2023b.   
Wang, Y., Tao, X., Shen, X., and Jia, J. Wide-context semantic image extrapolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.   
Wu, T., Li, X., Qi, Z., Hu, D., Wang, X., Shan, Y., and Li, X. Spherediffusion: Spherical geometry-aware distortion resilient diffusion model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 6126–6134, 2024a.

Wu, T., Zheng, C., and Cham, T.-J. Panodiffusion: 360- degree panorama outpainting via diffusion. In The Twelfth International Conference on Learning Representations, 2024b.   
Xie, S., Zhang, Z., Lin, Z., Hinz, T., and Zhang, K. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22428–22437, June 2023.   
Yao, K., Gao, P., Yang, X., Sun, J., Zhang, R., and Huang, K. Outpainting by queries. In Avidan, S., Brostow, G., Cisse,´ M., Farinella, G. M., and Hassner, T. (eds.), Computer Vision – ECCV 2022, pp. 153–169, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-20050-2.   
Ye, Y., Alshina, E., and Boyce, J. Algorithm descriptions of projection format conversion and video quality metrics in 360lib. In Joint Video Exploration Team of ITU-T SG, 07 2018.   
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T. S. Generative image inpainting with contextual attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.   
Yurtsever, E., Lambert, J., Carballo, A., and Takeda, K. A survey of autonomous driving: Common practices and emerging technologies. IEEE Access, 8:58443–58469, 2020. doi: 10.1109/ACCESS.2020.2983149.   
Zhang, C., Wu, Q., Cruz Gambardella, C., Huang, X., Phung, D., Ouyang, W., and Cai, J. Taming stable diffusion for text to 360◦ panorama image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.   
Zhang, G. and Kou, X. Research and implementation of digital 3d panoramic visual communication technology based on virtual reality. International Journal of Communication Systems, 35(5):e4802, 2022.   
Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In IEEE International Conference on Computer Vision (ICCV), 2023.   
Zhao, S., Cui, J., Sheng, Y., Dong, Y., Liang, X., Chang, E. I., and Xu, Y. Large scale image completion via comodulated generative adversarial networks. In International Conference on Learning Representations (ICLR), 2021.   
Zheng, C., Cham, T.-J., and Cai, J. Pluralistic image completion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1438–1447, 2019.

Zheng, C., Cham, T.-J., Cai, J., and Phung, D. Bridging global context interactions for high-fidelity image completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11512–11522, June 2022.   
Zheng, J., Zhang, J., Li, J., Tang, R., Gao, S., and Zhou, Z. Structured3d: A large photo-realistic dataset for structured 3d modeling. In Proceedings of The European Conference on Computer Vision (ECCV), 2020.

# A. Analysis on the Structural Prior for the ERP Format

Compared to the equatorial region, rows closer to the poles correspond to a smaller number of points on the sphere being mapped onto the fixed $m _ { w }$ points in the ERP format. This discrepancy results in a one-to-many mapping, where multiple pixels in the ERP representation correspond to the same point in the 3D space, leading to redundancy and distortion in these regions. Since the number of points per row $( m _ { w } )$ in ERP format is given by the width w, further analysis on (2) reveals that as the resolution of the ERP format decreases or the sampling density ζ in 3D space increases, the distribution of the uniform spherical noise more closely approximates i.i.d. Gaussian noise within the ERP representation.

To comprehensively analyse the properties and the influence of the hyperparameter ζ and the width w on the structural prior, we conducted an experiment and present the results in Figure A.1. Note that images with a resolution of $1 2 8 \times 6 4$ are consistent with the resolution of latent features in our SpND model, while images with a resolution of $1 0 2 4 \times 5 1 2$ match the resolution of the images in datasets. By comparing the columns in Figure A.1, it can be concluded that structural prior with higher resolution produced using the same ζ in the ERP format exhibit fewer rows that satisfy the i.i.d. property, consistent with our analysis. Additionally, by comparing rows with the same resolution in Figure A.1, it can be observed that noise in 3D space with smaller ζ tends to produce a structural prior with fewer rows satisfying the i.i.d. property at the same resolution. For ζ = 1000 and a resolution of 128 × 64, the structural prior closely approximates i.i.d. Gaussian noise, with only the top and bottom rows significantly violating the i.i.d. property. Since the noise is adopted in the latent space in our SpND model, we adopt the noise distribution in the ERP format with a resolution of $1 2 8 \times 6 4$ as our structural prior.

To determine the optimal value of the hyperparameter ζ, we conducted a comparison between the visualised noise and the latent features, as illustrated in Figure A.2. This analysis led us to select ζ = 30, as it most accurately preserves the structural characteristics of the feature map with a resolution of $1 2 8 \times 6 4$ while maintaining the highest efficiency in panorama outpainting.

![](images/cec30940a2b78d81f9ebc9c3d749cb7407272f1aa0d98730f0a5515d36af83db.jpg)

<details>
<summary>text_image</summary>

128×64
ζ = 5
ζ = 15
ζ = 30
ζ = 45
ζ = 100
ζ = 1000
512×1024
ζ = 5
ζ = 15
ζ = 30
ζ = 45
ζ = 100
ζ = 1000
</details>

Figure A.1. Analysis on the impact of resolution (width w, height h) and sampling density ζ in 3D space to the structural prior for the ERP format. The structural prior is obtained by projecting the i.i.d. Gaussian noise in 3D space with varying density ζ, into the ERP format. The first row shows projections with a resolution of $1 2 8 \times 6 4$ , while the second row presents those with a resolution of $1 0 2 4 \times 5 1 2$ .

![](images/fd14672ab96fa5272f4f2d0b2135f481bb74231d36eeda6115bcd2d50b96d85f.jpg)

<details>
<summary>natural_image</summary>

Grainy black-and-white image with no discernible text, symbols, or structured content.
</details>

(a) Noise with ζ = 15

![](images/c94432ecbd26d8a9705f65d9f8404030ddfda2a2ce09ef24a7df03d8bbb72e3d.jpg)

<details>
<summary>natural_image</summary>

Close-up of a textured gray surface with two red horizontal lines (no text or symbols)
</details>

(b) Noise with ζ = 30

![](images/146bdf37b7453c7db867df946eb1527b0a5c9669e8eb28c7ce77fc088509f07f.jpg)

<details>
<summary>natural_image</summary>

Abstract color gradient background with no discernible text, symbols, or structured elements
</details>

(c) Real feature in latent space

![](images/8d21a3bad87287182732362c70961a6073afd7342e48750a4535e9ec89d3dd21.jpg)

<details>
<summary>natural_image</summary>

Close-up of a textured gray surface with red horizontal lines (no text or symbols)
</details>

(d) Noise with ζ = 45   
Figure A.2. Comparison of noise with different values of ζ and the real feature in the latent space encoded by the VAE. The red line is used to distinguish rows that approximate the i.i.d. property from those that do not, thereby highlighting regions where the noise significantly deviates from the characteristics of i.i.d. Gaussian noise.

Since a specially designed noise pattern was employed as structural prior knowledge for the ERP format, this study aimed to analyse its impact on the resulting outputs. As demonstrated in (a) and (c) of Figure A.3, changing the noise used to realize the structural prior $\mathbf { P } _ { \mathrm { E R P } }$ does not affect the generated content. In contrast, changing the noise Z used for the pretrained diffusion model primarily impacts the generated content, as illustrated in (b) and (d) in Figure. A.3. This indicates that the noise pattern associated with $\mathbf { P } _ { \mathrm { E R P } }$ primarily captures the spherical curvature, instead of controlling the specific details of the generated content. Note that the masked input is identical in (a) and (b), as well as in (c) and (d) in Figure A.3.

![](images/47840f2023d5a95af8ae519b5ec2f96fcd7d85cddb1c9a4e6363be2db02a58c0.jpg)  
Figure A.3. The impact of the structural prior $\mathbf { P } _ { \mathrm { E R P } }$ on the outputs is examined by sampling different noise patterns in 3D space as the structural prior $\mathbf { P } _ { \mathrm { E R P } }$ , while keeping the feature Z consistent across rows (a) and (c). In rows (b) and (d), the same 3D noise is used for the structural prior, but the feature Z is varied. Rows (a) and (b) present samples from the Structured3D dataset, while rows (c) and (d) display samples from the Matterport3D dataset.

# B. Repaint Operation with Mask Control

The repaint operation (Lugmayr et al., 2022) involves adopting a mask m to regulate specific regions within the generated sample while preserving the remaining areas unchanged. Specifically, the mask m determines the regions that should remain unchanged by assigning a value of 1, while the remaining areas, represented by a value of 0, are updated during the diffusion process. At each timestep t, our SpND model predicts the denoised sample $\dot { \bf Z } _ { t - 1 } ^ { \mathrm { p r e d i c t e d } }$ , which can be formulated as

$$
\hat {\mathbf {Z}} _ {t - 1} ^ {\text { predicted }} = \frac {1}{\sqrt {\alpha_ {t}}} \left[ \mathbf {Z} _ {t} ^ {\text { predicted }} - \frac {1 - \alpha_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \epsilon_ {\psi} \left[ \mathbf {Z} _ {t} ^ {\text { predicted }}, t \right] \right], \tag {B.1}
$$

where $\begin{array} { r } { \bar { \alpha } _ { t } = \prod _ { i = 1 } ^ { t } \alpha _ { i } } \end{array}$ and The parameter α serves as a hyperparameter that controls the variance of the noise. Assume that η denotes for the $i . i . d .$ Gaussian noise and $\mathbf { Z } _ { 0 }$ corresponds to the ground-truth feature. Subsequently, to achieve mask control, we update the denoised sample $\hat { \mathbf { Z } } _ { t - 1 } ^ { \mathrm { p r e d i c t e d } }$ given $\mathbf { Z } _ { t - t } ^ { \mathrm { r e a l } }$ according to the formula as

$$
\mathbf {Z} _ {t - 1} ^ {\text { predicted }} = \mathbf {m} \odot \hat {\mathbf {Z}} _ {t - 1} ^ {\text { predicted }} + (1 - \mathbf {m}) \odot \mathbf {Z} _ {t - t} ^ {\text { real }}, \tag {B.2}
$$

$$
\mathbf {Z} _ {t - t} ^ {\text { real }} = \sqrt {\bar {\alpha} _ {t - 1}} \cdot \mathbf {Z} _ {0} + \sqrt {1 - \bar {\alpha} _ {t - 1}} \cdot \eta \tag {B.3}
$$

where $\mathbf { Z } _ { t - 1 } ^ { \mathrm { p r e d i c t e d } }$ represents the outpainted feature and ⊙ denotes element-wise multiplication. Note that (B.2) formulates the repaint process and (B.3) formulates the forward process of the diffusion model. The repaint operation, as defined in (B.1)-(B.3), ensures that the masked regions retain their original values, while the unmasked regions are selectively updated. Note that this effect of the repaint operation aligns seamlessly with the objective of the image outpainting task. By leveraging the repaint technique, our SpND model effectively performs outpainting for panoramic images while preserving the integrity of the original part of the image.

# C. Experiment on Martain Image Outpainting

We further conducted experiments on a small dataset consisting of Martian panoramic images to evaluate the generalizability and the robustness of our SpND model. Specifically, we collected 145 Martian panoramic images captured by the

Perseverance rover, with a resolution of $5 1 2 \times 1 7 2$ , from publicly available sources. These images were padded with black borders to a final size of $5 1 2 \times 2 5 6$ . As shown in Figure C.1, the proposed $\mathrm { S p N D _ { p r o m p t } }$ model effectively addresses the black borders at the top and bottom of the image, generating high-quality panoramic content for Martian images with a limited number of training samples. Consistent with our other experiments, we employed the BLIP-2 model (Li et al., 2023) to generate the prompts for the Martian images. Note that the masked input can be obtained by projecting view images into the ERP format.

![](images/0fb70fd254a5a7c07d03b7810bac71dde546b1256437de9335fafa3ddc8e95a0.jpg)  
Figure C.1. Qualitative results of panoramic image outpainting on the Martian dataset based on the given perspective view.

# D. Experiment on Input Versatility

In addition, we have demonstrated that our model can accommodate multiple partial-view and overlapping inputs by adjusting the training masks accordingly on the Matterport3D dataset. More specifically, for the multi-input scenario, we evaluated our SpND method based on two viewpoints centred at $( \theta _ { 1 } = - 9 0 ^ { \circ } , \phi _ { 1 } = - 1 5 ^ { \circ } )$ and $( \theta _ { 2 } = 9 0 ^ { \circ } , \phi _ { 2 } = 1 5 ^ { \circ } )$ within the ERP format, thus denoted as Dual. Moreover, for the overlapping scenario, we evaluated our SpND method on two viewpoints centered at $( \theta _ { 1 } = 2 0 ^ { \circ } , \phi _ { 1 } = 0 ^ { \circ } )$ and $( \theta _ { 2 } = 9 0 ^ { \circ } , \phi _ { 2 } = 1 5 ^ { \circ } )$ with overlapped regions, also using the ERP format, thus denoted as Overlapping. We then report the subjective results in Figure D.1 and Figure D.2. As illustrated in the figures, the proposed $\mathrm { S p N D _ { p r o m p t } }$ model is capable of achieving high-quality panoramic image completion under varying masks and multiple input conditions. We can conclude that our method can generalize to multiple overlapping and partial-view inputs.

![](images/0c1bf1e3b8c3178c26354daf354de10d281d219a48eeee0d8946ed449c90f86d.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern bathroom with toilet and window (no text or symbols visible)
</details>

Ground Truth

![](images/a0cdd231bfd6fa9fc5a1889f1ee9dd3aba5f6890980fa87aec0c797395b62764.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern hallway with a large window and curved ceiling light (no text or symbols visible)
</details>

(a) Input Masked Image

![](images/41bb8414e8a9cc0b8cf2046a3403905e2b790bd898ca7b76e96e4b18ed4b4d15.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a bathroom with bathtub, toilet, and window (no text or symbols visible)
</details>

(b) Dual

![](images/2543689136ad394fff40824ea2f92ab1ac3389a11d368263a0236267a5fdbeb5.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a modern, empty living room with curved furniture and ceiling lighting (no visible text or symbols)
</details>

Ground Truth

![](images/8c790d2d77c9dc0af2b4e22007670eca9bc6414f0fda438dfb57eff36cc9dad2.jpg)

<details>
<summary>natural_image</summary>

Two interior views of a modern, minimalist room with furniture and lighting (no visible text or symbols)
</details>

(a) Input Masked Image

![](images/796085af7f660548e28956468ab4667648e7d3d3513e398afa5d4dce5badc5e9.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a modern entertainment or massage room with beige sofas and ceiling beams (no visible text or symbols)
</details>

(b) Dual   
Figure D.1. Qualitative results of $\mathrm { S p N D _ { p r o m p t } }$ with multiple partial-view inputs, including ground truth, masked input, and images completed by our $\mathrm { S p N D } _ { \mathrm { p r o m p t } } .$

![](images/f46c48b932c545aef4b87bc09d5c8ce4fbe4b22380bca25d18877c65081adbb0.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a spacious hallway with curved walls and railings, no visible text or symbols
</details>

Ground Truth

![](images/99e74c6167815a0fd9780ec497ba0c186cf917ec6d678c5077e3f6a2da6798c1.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern building with curved ceiling and staircase (no visible text or symbols)
</details>

(a) Input Masked Image

![](images/525594a0694c8b2f4f657eda8d27ab012393e6607693f52e328962f055ef894c.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern hallway with curved white walls and large windows (no visible text or symbols)
</details>

(b) Overlapping

![](images/d8385d3b0eb36ebc783fdafb84aff2b5d5e201a206ffe9139fc12b28e275ca29.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern bathroom with built-in door, counter, and decorative wall art (no visible text or symbols)
</details>

Ground Truth

![](images/614c98cb3fd6d9890f4a5feba40595e9f644e94e07b9dca99cf48d666d49afdf.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern bathroom with glass doors and ceiling lights (no visible text or symbols)
</details>

(a) Input Masked Image

![](images/2453e161e723984dba54a7763a790b425f780244cbffc56d8044f45a5849faf4.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a modern bathroom with large doors, built-in shower, and wall-mounted furniture (no visible text or symbols)
</details>

(b) Overlapping   
Figure D.2. Qualitative results of $\mathrm { S p N D _ { p r o m p t } }$ with multiple overlapping inputs, including ground truth images, masked input images, and images completed by our $\mathrm { S p N D _ { p r o m p t } }$ .