# 360-Degree Panorama Generation from Few Unregistered NFoV Images

Jionghao Wang∗ Shanghai Jiao Tong University shanemankiw@sjtu.edu.cn

Ziyu Chen∗ Shanghai Jiao Tong University 1252060456@sjtu.edu.cn

Jun Ling Shanghai Jiao Tong University lingjun@sjtu.edu.cn

![](images/aed60375477a58cd6d47ffd6f6edbe986588017d8d849894ab0ea82c70cd73be.jpg)

<details>
<summary>text_image</summary>

Rong Xie
Shanghai Jiao Tong University
xierong@sjtu.edu.cn
Li Song†
Shanghai Jiao Tong University
song_li@sjtu.edu.cn
I₀ →
"bar on
an island"
I₀, I₁ →
"Christmas
night"
{Iᵢ} →
"fireworks
show"
Input NFOVs
Partial Panoramas
Visibility Masks
Text Prompts
Generated Panoramas
as Lighting
as Background
</details>

Figure 1: Illustration of our PanoDiff, a novel approach that is capable of synthesizing fine-grained and diverse 360-degree panorama from a(few) unregistered NFoV (Narrow Field of View) image(s) and text prompts.

# ABSTRACT

360◦ panoramas are extensively utilized as environmental light sources in computer graphics. However, capturing a $3 6 0 ^ { \circ } \times 1 8 0 ^ { \circ }$ panorama poses challenges due to the necessity of specialized and costly equipment, and additional human resources. Prior studies develop various learning-based generative methods to synthesize panoramas from a single Narrow Field-of-View (NFoV) image, but they are limited in alterable input patterns, generation quality, and controllability. To address these issues, we propose a novel pipeline called PanoDiff, which efficiently generates complete 360◦ panoramas using one or more unregistered NFoV images captured from arbitrary angles. Our approach has two primary components to

overcome the limitations. Firstly, a two-stage angle prediction module to handle various numbers of NFoV inputs. Secondly, a novel latent diffusion-based panorama generation model uses incomplete panorama and text prompts as control signals and utilizes several geometric augmentation schemes to ensure geometric properties in generated panoramas. Experiments show that PanoDiff achieves state-of-the-art panoramic generation quality and high controllability, making it suitable for applications such as content editing.

# CCS CONCEPTS

• Computing methodologies → Computer vision; Computer graphics.

# KEYWORDS

360-degree panorama, generative models, multimodal models, latent diffusion, image pose estimation

# ACM Reference Format:

Jionghao Wang, Ziyu Chen, Jun Ling, Rong Xie, and Li Song. 2023. 360- Degree Panorama Generation from Few Unregistered NFoV Images. In Proceedings of the 31st ACM International Conference on Multimedia (MM ’23), October 29-November 3, 2023, Ottawa, ON, Canada. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3581783.3612508

# 1 INTRODUCTION

Panoramic images, which capture an extensive field of view encompassing a full 360◦ horizontal by 180◦ vertical FoV scene, have become increasingly significant across various applications, such as environment lighting, VR/AR, and autonomous driving system. However, obtaining high-quality panoramic images can be both time-consuming and costly, as they typically necessitate the use of specialized panoramic cameras or stitching software to combine images from multiple perspectives. Our method addresses two main limitations regarding previous generating methods, namely input pattern and generation quality & controllability.

For Input pattern. Most previous works [1, 5, 37] only support a single FoV region in the center of an incomplete panorama as input. However, relying solely on a single input region restricts flexibility when controlling specific contents of the generated scene. Such flexibility is particularly important for applications that require precise control over the visual elements and ensure accurate representations of the intended scene.

In terms of generation quality & controllability. Generating a complete 360-degree panorama from NFoV images could be viewed as a large-hole inpainting problem [35]. Previous methods [1, 11, 37] typically all rely on GAN(Generative Adversarial Networks) [10] based methods. Besides, previous methods approach this problem as an image-conditioned generation task, leaving their pipeline with little control and flexibility over the generation results. A more recent work [5] proposes to use additional text guidance for a GAN-based image inpainting method [41]. However, diffusion-based image generation methods [14] have shown impressive results on various generative tasks, and models trained on large datasets [29, 33] have shown better performance and robustness against GAN-based models [7, 15]. Moreover, GANs have limited mode coverage and are difficult to scale for modeling complex multimodal distributions. Unlike GANs, likelihood-based models such as diffusion models are capable of learning the complex distribution of natural images, resulting in the generation of high-quality images, as mentioned in [7, 29].

To overcome these limitations, a pipeline capable of accepting a flexible number of NFoV image(s) as input and generating highfidelity panoramas is crucial. Nevertheless, this endeavor presents two main challenges: 1) estimating relative camera poses and accurately warping the input NFoV image(s) on the panorama, and 2) using a latent-diffusion-model-based method to generate the entire panorama from input partial panorama of various shapes.

This paper introduces PanoDiff, a novel pipeline that efficiently generates complete 360° panoramas using one or more unregistered NFoV images captured from arbitrary angles. The proposed method overcomes the limitations of existing methods, it enables the generation of high-quality panoramas from incomplete 360- degree panoramas that warped from any number of NFoV inputs. This is achieved by addressing two key challenges. First, we propose a robust two-stage angle prediction pipeline that classifies image pairs based on their overlap before regressing specific angle values. Second, we train a hypernetwork that controls a pre-trained large latent-diffusion model, and utilizes geometric augmentation schemes during both training and inference sampling phases to ensure the geometric properties of the generated panoramas.

We summarize our contributions as follows. Firstly, the first flexible framework for generating panoramic images from single or multiple NFoV inputs. A two-stage pose estimation module is specially designed for relative pose estimation. Secondly, as far as we know, PanoDiff is the first latent-diffusion-based panorama outpainting model that can not only handle incomplete panoramas of various shapes as input but also supports text prompts. Finally, PanoDiff outperforms existing methods in terms of quantitative and qualitative results on different input types, as demonstrated through abundant experiments.

# 2 RELATED WORKS

# 2.1 Rotation Estimation

Camera pose estimation has been a long-standing task in computer vision [19]. Most prior works focus on identifying various visual cues from pairs of images. For instance, the classic SIFT [22] extracts scale and direction invariant features from input images and matches them to find correspondences. Some approaches utilize neural networks for more robust feature extraction or graph neural networks for enhanced matching [6, 28, 32]. However, these methods do not account for extreme rotations where the input images have little to no overlap, known as wide baseline scenarios. Some earlier works focus on mining specially handcrafted cues [24] to address this problem, but they are not generalizable for scenes without a specific clue. The work that inspired us the most is [2], which estimates pair-wise rotation angles in both overlap and wide baseline scenarios. However, the estimation network occasionally misidentifies an overlapping pair of images as non-overlapping, which severely impacts the generation quality later on.

# 2.2 Diffusion Models

Diffusion models have shown impressive ability in generative tasks [7, 14]. Generally, diffusion models represent the image generative process as a denoising process, where a noise map sampled from white Gaussian noise is iteratively denoised using a learned prior distribution ???? (???? −1|???? ). The objective function for training diffusion models is simplified as

$$
L _ {\text { simple }} = E _ {t, x _ {0}, \epsilon} \left[ | \epsilon - \epsilon_ {\theta} (x _ {t}, t) | ^ {2} \right] \tag {1}
$$

which has been shown to be approximately equivalent to optimizing the prior probability distribution’s variational lower bound [14].

To further improve the efficiency and effectiveness of the diffusion model, recent work has proposed to conduct the denoising process in latent space instead of pixel space [29]. In addition, they integrate multi-head cross-attention mechanism [3, 36] in denoising U-Net [25, 30] blocks. This approach enables features of different modalities to guide the generation process[16], e.g. CLIP-ViT [27]. Working on latent space along with some other sampling schedule [34] could also potentially make the sampling more efficient. Controlling the pre-trained model parameters by training hypernetworks[12, 40] can improve its performance on a more specific task while preserving its original generative capability.

# 2.3 Diffusion-based Inpainting

The task of inpainting was predominantly been done by GANbased models [21, 35, 42]. Recently, diffusion-based methods have been proposed and have already achieved promising results [23, 38]. However, [23, 38] operate the diffusion and sampling process solely on image space rather than latent space, which limits their generation flexibility. Recently, [31] offered an image inpainting model finetuned from Stable Diffusion [29] and demonstrates strong text-controlled inpainting ability.

![](images/b9b22358467639c75bffde80ddbf9e6cf015c90bed14cd1e6af4947a573d91f1.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph LR
    I --> Classification["Classification"]
    Classification --> Rθ[Rθ]
    Rθ --> Regression["Regression"]
    Regression --> TextPrompt["Text Prompt"]
    TextPrompt --> TextPrompt1["τθ"]
    TextPrompt1 --> TextPrompt2["c_text"]
    TextPrompt2 --> Sampling["Sampling"]
    Sampling --> FLDM["FLDM"]
    FLDM --> Panorama["Panorama"]

    subgraph Classification
        Ecc["εc"] --> DCM
        Erc["εc"] --> DCM
        DCM --> Dcc["Dc"]
    end

    subgraph Regression
        Esr["εr"] --> DCM
        Esr --> DCM
        DCM --> Dc
        Dc --> Dx["Dx"]
        Dx --> RegressionHead["Regression Head"]
        Dcm --> DCM
        Dcm --> Dc
        Dc --> Dx
        Dx --> DenseCorrelationModule["Dense Correlation Module"]
        Dcm --> DenseCorrelationModule
        Dc --> DenseCorrelationModule
        Dm --> DenseCorrelationModule
        Dm --> DenseCorrelationModule
        Dm --> DenseCorrelationModule
        Dm --> DenseCorrelationModule
    end

    subgraph TextPrompt
        Fproj["Fproj"] --> TextPrompt1["Text Prompt"]
        TextPrompt1 --> TextPrompt2["[P,M"]]
        TextPrompt2 --> TextPrompt3["c_text"]
        TextPrompt3 --> TextPrompt4["c_P"]
        TextPrompt4 --> TextPrompt5["z_T"]
        TextPrompt5 --> Sampling["Sampling"]
        Sampling --> Sampling1["z0"]
        Sampling1 --> Sampling2["zT"]
        Sampling2 --> Sampling3["zP"]
        Sampling3 --> Sampling4["zP"]
        Sampling4 --> Sampling5["zP"]
        Sampling5 --> Sampling6["zP"]
        Sampling6 --> Sampling7["zP"]
        Sampling7 --> Sampling8["zP"]
        Sampling8 --> Sampling9["zP"]
        Sampling9 --> Sampling10["zP"]
        Sampling10 --> Sampling11["zP"]
        Sampling11 --> Sampling12["zP"]
        Sampling12 --> Sampling13["zP"]
        Sampling13 --> Sampling14["zP"]
        Sampling14 --> Sampling15["zP"]
        Sampling15 --> Sampling16["zP"]
        Sampling16 --> Sampling17["zP"]
        Sampling17 --> Sampling18["zP"]
        Sampling18 --> Sampling19["zP"]
        Sampling19 --> Sampling20["zP"]
        Sampling20 --> Sampling21["zP"]
        Sampling21 --> Sampling22["zP"]
        Sampling22 --> Sampling23["zP"]
        Sampling23 --> Sampling24["zP"]
        Sampling24 --> Sampling25["zP"]
        Sampling25 --> Sampling26["zP"]
        Sampling26 --> Sampling27["zP"]
        Sampling27 --> Sampling28["zP"]
        Sampling28 --> Sampling29["zP"]
        Sampling29 --> Sampling30["zP"]
        Sampling30 --> Sampling31["zP"]
        Sampling31 --> Sampling32["zP"]
        Sampling32 --> Sampling33["zP"]
        Sampling33 --> Sampling34["zP"]
        Sampling34 --> Sampling35["zP"]
        Sampling35 --> Sampling36["zP"]
        Sampling36 --> Sampling37["zP"]
        Sampling37 --> Sampling38["zP"]
        Sampling38 --> Sampling39["zP"]
        Sampling39 --> Sampling40["zP"]
        Sampling40 --> Sampling41["zP"]
        Sampling41 --> Sampling42["zP"]
        Sampling42 --> Sampling43["zP"]
        Sampling43 --> Sampling44["zP"]
        Sampling44 --> Sampling45["zP"]
        Sampling45 --> Sampling46["zP"]
        Sampling46 --> Sampling47["zP"]
        Sampling47 --> Sampling48["zP"]
        Sampling48 --> Sampling49["zP"]
        Sampling49 --> Sampling50["zP"]
        Sampling50 --> Sampling51["zP"]
        Sampling51 --> Sampling52["zP"]
        Sampling52 --> Sampling53["zP"]
        Sampling53 --> Sampling54["zP"]
        Sampling54 --> Sampling55["zP"]
        Sampling55 --> Sampling56["zP"]
        Sampling56 --> Sampling57["zP"]
        Sampling57 --> Sampling58["zP"]
        Sampling58 --> Sampling59["zP"]
        Sampling59 --> Sampling60["zP"]
        Sampling60 --> Sampling61["zP"]
        Sampling61 --> Sampling62["zP"]
        Sampling62 --> Sampling63["zP"]
        Sampling63 --> Sampling64["zP"]
        Sampling64 --> Sampling65["zP"]
        Sampling65 --> Sampling66["zP"]
        Sampling66 --> Sampling67["zP"]
        Sampling67 --> Sampling68["zP"]
        Sampling68 --> Sampling69["zP"]
        Sampling69 --> Sampling70["zP"]
        Sampling70 --> Salmon6z["Sampling"]
    end

    subgraph Training
        Ecc1["εc"] --> DCM
        Erc1["εc"] --> DCM
    end

    classDef label fill:#f9f,stroke:#333,stroke-width:2px;
    class I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,Z,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,E,F,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,E,F,G,H,I,J,k,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,E,F,G,H,I,J,k,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,E,F,G,H,I,J,k,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,E,F,G,H,I,J,k,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,C,D,E,F,G,H,I,J,k,L,M,N,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,I,j,k,l,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,a,b,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,t,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k(l,a,b,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o-p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,jk,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z;c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,s,t,u,v,w,x,y,z,c,d,e,f,g,h,i,j,k,l,m,m,m,m,n,o,p,q,r,\( \text{Sampling} \) 
    end
```
</details>

Figure 2: An overview of generating panorama from a few NFoV images. We first calculate their relative rotations based on a two-stage angle prediction network $\mathcal { R } _ { \theta }$ (Sec. 3), then project them using backward equirectangular projection $\mathcal { F } _ { \mathrm { p r o j } }$ (Sec. 4) to obtain partial panorama $P$ and visibility mask ??. Finally, we feed [??, ??] along with text prompts to our control-based latent diffusion model $\mathcal { F } _ { \mathrm { L D M } }$ (Sec. 4) and sample iteratively in our rotating schedule (Sec. 4.4.2) to get the final generated panorama.

# 2.4 Panorama Generation

With recent advancements in deep generative neural networks, several works adopt different forms of GAN-based methods to generate full panoramas [1, 5, 20, 37], such as VQGAN [8] and CoModGAN [41]. However, these methods are limited to singleview input, in contrast to our alterable input patterns. One work has utilized a few NFoV images as input [11] with accurate camera positions already given, thus cannot deal with unregistered camera inputs.

Recently, some researchers have utilized diffusion models to generate panoramas from text prompts alone [4]. However, this work does not support partial FoV as input, and thus users have limited control over the outcome.

# 3 RELATIVE POSE ESTIMATION

# 3.1 Problem Formulation

Unlike previous panorama generation pipelines, our method takes either single or multiple NFoV images as input. In the case of multiple NFoV (Narrow Field-of-View) images captured at the same location but in different poses, our objective is to estimate the relative poses between them. In the context of omnidirectional images, where no camera translation is involved, we formulate the relative camera poses as rotation angles. Specifically, we view the $3 6 0 ^ { \circ }$ panorama as a spherical map and express the rotation angles as the 1) lookat direction, which includes the longitude ??, latitude $\beta ,$ and 2) roll angle ??. For a single image, our pipeline estimates only $\beta$ and places it in lateral center, i.e., $\alpha = \gamma = 0$ . Formally, for a set of ?? NFoV images $I = I _ { 0 } , I _ { 1 } , . . . , I _ { N - 1 }$ , we aim to learn a model $\mathcal { R } _ { \theta }$ that predicts their relative angles:

$$
\mathcal {R} _ {\theta} (\mathrm{I}) \rightarrow [ \alpha , \beta , \gamma ] \tag {2}
$$

where $\alpha = \{ \alpha _ { i } \} _ { i = 0 , . . . , N - 1 } , \beta = \{ \beta _ { i } \} _ { i = 0 , . . . , N - 1 } \mathrm { a n d } \gamma = \{ \gamma _ { i } \} _ { i = 0 , . . . , N - 1 } ,$ respectively. We decompose this task into estimating pair-wise camera relative rotation, which is scalable and can be applied to pair input and sometimes more than two images. We design a pairwise angle prediction network that estimates the relative angles between images. For a set of ?? NFoV images $\left( I _ { 0 } , I _ { 1 } , . . . , I _ { N - 1 } \right)$ , we select one image $I _ { 0 }$ as an anchor and estimate the relative poses $[ \Delta \alpha , \Delta \beta ,$ , Δ??] of the remaining images with respect to the anchor. Following the assumptions made in [2], we consider that cameras are typically upright, allowing us to estimate the absolute pitch angle (vertical/latitude) instead of the relative angle. We also assume that the camera roll angles are static and remain unchanged at $^ { 0 , }$ i.e., $\forall i \in 0 , . . . , N - 1 , \gamma _ { i } = 0$ . Consequently, our network ${ \mathcal { R } } _ { \theta }$ can be expressed as:

$$
\mathcal {R} _ {\theta} (I _ {0}, I _ {i}) \rightarrow [ \Delta \alpha_ {i \rightarrow 0}, \beta_ {0}, \beta_ {i} ] \tag {3}
$$

where $\Delta \alpha _ { i \to 0 }$ is the relative longitude angle between the images, and $\beta _ { 0 } , \beta _ { i }$ are the absolute lattitude angle for ??0 and $I _ { i } ,$ respectively.

# 3.2 Two-stage Angle Prediction

In the context of FoV overlapping, we categorize image-pair relationships into two distinct types: overlap and wide baseline. In overlap scenarios (green pair in Fig. 3), two NFoV images share a portion of their FoV, resulting in an overlap on the panorama. Conversely, in wide baseline situations (red pair in Fig. 3), the NFoV image pair does not have overlapping FoV, leading to two separate regions on the panorama. The ultimate goal of our method is to create a plausible panorama for further applications. However, a single regression network [2] is not robust enough to differentiate between these two cases. Consequently, images without any overlap might sometimes be predicted to have an overlap, creating severe artifacts in the input for our controlled LDM, which would result in unsatisfactory results. Furthermore, the precision of a single regression network is insufficient.

To address these problems, we propose a two-stage angle prediction pipeline that initially performs classification to determine whether the two images have an overlap, and subsequently regresses the precise angles $[ \Delta \alpha _ { i  0 } , \beta _ { 0 } , \beta _ { i } ]$ . This approach significantly reduces the issue of wide baseline images being mistakenly predicted to overlap. The overview of our rotation prediction module can be seen in Fig. 3. We adopt the same backbone structure as [2], which consists of a feature encoder and a 4D dense correlation module. Within the two-stage pipeline, a total of three backbones are utilized, each with its own prediction head. To estimate the relative camera rotation between a pair of images, we first pass the image pair to our classifier to determine if the images are overlapping or wide apart. Subsequently, we feed the image pair to its corresponding regressor based on the classification result.

![](images/6a8c120ee6136dbf745d884d09f00852278ce699e263f96e59386e4e50a221e5.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    A["Pair 1"] --> B["Classification Network"]
    C["Pair 2"] --> B
    B --> D["Overlap Backbone"]
    D --> E["Dense Correlation Module"]
    E --> F["Regression Head"]
    F --> G["Partial Panorama"]
    F --> H["Visibility Mask"]
    I["Pair 1"] --> J["Overlap Backbone"]
    K["Pair 2"] --> J
    J --> L["Wide Backbone"]
    L --> M["Dense Correlation Module"]
    M --> N["Regression Head"]
    N --> O["Partial Panorama"]
    N --> P["Visibility Mask"]
```
</details>

Figure 3: Illustration of our two-stage angle prediction network. $\delta _ { o } , \delta _ { n }$ represent feature encoders, and $\mathcal { D } _ { \Delta \alpha _ { i } } , \mathcal { D } _ { \beta _ { 0 } } , \mathcal { D } _ { \beta _ { i } }$ serve as prediction heads for overlap and wide-baseline angle regression, respectively. The pipeline first estimates whether the input image pair has an overlap, then directs the image pair to the corresponding regression network.

In terms of training, our classifier is trained on our full set of training image pairs. A specialized penalty loss function is employed for misclassified cases, which effectively mitigates failure instances. As for our regressors, we separate our training image pairs into two splits based on their overlapping status and train the overlap branch and wide branch separately on their respective data splits. That is, for the overlap regressor, we train it using primarily overlap data pairs, and vice versa for the wide baseline regressor.

# 4 PANORAMA GENERATION

# 4.1 Formulation

With Sec. 3, we can formulate a set of image-angle pairs, i.e., $\{ \mathrm { I } _ { 0 } , 0 , \beta _ { 0 } \}$ and $\{ \mathrm { I } _ { \mathbf { i } } , \Delta \alpha _ { i  0 } , \beta _ { i } \} _ { i = 1 , \dots , N - 1 }$ . In this part, our goal is to take the images and their relative angles to produce a complete $3 6 0 ^ { \circ } \times 1 8 0 ^ { \circ }$ panorama. We divide this problem into two parts: 1) project the image set $I = \{ \mathrm { I } _ { \mathrm { i } } \}$ onto the omnidirectional map based on their estimated angles $\pmb { \alpha } = \{ \alpha _ { i } \} _ { i = 0 , \ldots , N - 1 }$ and $\pmb { \beta } = \{ \beta _ { i } \} _ { i = 0 , \dots , N - 1 } ;$ and 2) generate the full panorama using the incomplete input as a control signal. Specifically, we formulate this problem as:

$$
\mathcal {F} _ {\mathrm{proj}} (\boldsymbol {I}, \boldsymbol {\alpha}, \boldsymbol {\beta}) \rightarrow P, M,
$$

$$
\mathcal {F} _ {\mathrm{LDM}} (P, M, \text { text }) \rightarrow \text { Pano }, \tag {4}
$$

where $\mathcal { F } _ { \mathrm { p r o j } }$ and $\mathcal { F } _ { \mathrm { L D M } }$ represent the projection and diffusion generation operations, respectively. The projection operation $\mathcal { F } _ { \mathrm { p r o j } }$ takes the NFoV image set and their angles as input and produces a partial panorama image ?? and a visibility mask ??, indicating which parts of the FoV are missing. Subsequently, the LDM (Latent Diffusion Model) operation $\mathcal { F } _ { \mathrm { L D M } }$ accepts the incomplete ??, visibility mask ??, and a text prompt as input, iteratively generating the full panorama Pano.

# 4.2 Recap: Controlling Stable Diffusion

We propose a generative process based on training a hypernetwork over a pre-trained Stable Diffusion (SD) model[29], a large text-toimage generative model utilizing latent diffusion. Latent diffusion models iteratively perform denoise sampling in latent space. At time step ?? , given input latent $z _ { t }$ and condition $y ,$ the denoiser network is formulated as $\epsilon _ { \theta } ( z _ { t } , t , y )$ . The denoiser in SD employs a U-Net bottleneck architecture.

[40] proposed to use combine trainable copies from pre-trained SD model parameters, and zero-convolution blocks, whose parameters are initialized as all zero. Concretely, suppose there is a neural network block $\mathcal { F } ( \cdot ; \Theta )$ from the pre-trained SD denoiser U-Net, which takes an input feature map $\mathbf { x } \in \mathrm { R } ^ { h \times w \times c }$ and outputs a feature y, i.e. $\mathbf { y } = \mathcal { F } ( \mathbf { x } ; \boldsymbol { \Theta } )$ . To add more conditions as control signals, two zero-convolution layers $z _ { 1 } , z _ { 2 }$ and their parameters $\Theta _ { z 1 } , \Theta _ { z 2 }$ , and a trainable copy of the original neural block and its parameters as $\mathcal { F } _ { c }$ and $\Theta _ { c }$ are introduced. Now given a new condition signal c, the new controlled output feature ${ \bf y } _ { c }$ can be written as:

$$
\mathbf {y} _ {c} = \mathcal {F} (\mathbf {x}; \Theta) + \mathcal {Z} _ {2} \left(\mathcal {F} _ {c} \left(\mathbf {x} + \mathcal {Z} _ {1} (\mathbf {c}; \Theta_ {z 1}); \Theta_ {c}\right); \Theta_ {z 2}\right) \tag {5}
$$

It is worth noting that the original operation $\mathcal { F } ( \cdot ; \Theta )$ is locked and does not produce any gradient for backpropagation. In this setting, the new set of trainable parameters are $\Theta _ { c } , \Theta _ { z 1 } , \Theta _ { z 2 }$ , where the initial $\Theta _ { c }$ is copied from Θ, and both $\Theta _ { z 1 }$ and $\Theta _ { z 2 }$ are initialized as zeros. This allows the controlled input at the first step to be identical to the original output, i.e., $\mathbf { y } _ { c } = \mathbf { y }$ . This property is favorable as it maintains the generation ability of the pre-trained model and makes the updating process of $\Theta _ { c } , \Theta _ { z 1 } , \Theta _ { z 2 }$ more stable. An intuitive illustration of this scheme can be seen in Fig. 4.

![](images/74ed616654b67bd856d77e419368e332cbfa77061931695bd89968df7d29ba58.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    C --> Z1["z₁(·; θz₁)"]
    Z1 --> Add["+"]
    Add --> Fc["F_c(·; θ_c)"]
    Fc --> Z2["z₂(·; θz₂)"]
    Z2 --> Add
    Add --> Yc["y_c"]
    X["x"] --> Ff["F(·; Θ)"]
    Ff --> Add
    style X fill:#d4edda,stroke:#333
    style Yc fill:#d4edda,stroke:#333
    style Ff fill:#d4edda,stroke:#333
    style Z1 fill:#d4edda,stroke:#333
    style Z2 fill:#d4edda,stroke:#333
    style Add fill:#d4edda,stroke:#333
    style Yc fill:#d4edda,stroke:#333
    style Fc fill:#d4edda,stroke:#333
    style Z1 fill:#d4edda,stroke:#333
    style Z2 fill:#d4edda,stroke:#333
    style Add fill:#d4edda,stroke:#333
    style Fc fill:#d4edda,stroke:#333
    style Z1 fill:#d4edda,stroke:#333
    style Z2 fill:#d4edda,stroke:#333
```
</details>

Figure 4: An intuitive explanation of integrating a new control signal into an existing model block.

# 4.3 Controlling LDM with Partial FoV

Given a noisy latent space feature $z _ { t }$ at time step ?? , we want to learn a denoiser that could predict the noise $\epsilon _ { t }$ as in:

$$
\epsilon_ {t} = \epsilon_ {\theta} (z _ {t}, t, \mathbf {c} _ {\mathrm{text}}, \mathbf {c} _ {\mathrm{P}}) \tag {6}
$$

Here, $\mathbf { c } _ { \mathrm { t e x t } }$ and $\mathbf { c } _ { \mathrm { P } } = [ M , P ]$ represent the text conditions and input panorama conditions, respectively. We introduce two components for our partial-FoV-controlled LDM, namely Pano-Denoiser $\epsilon _ { \theta }$ and Pano-Encoder $\varepsilon _ { P }$ .

Our Pano-Denoiser is designed as discussed in Sec. 4.2, where we employ control units to introduce control signals to the pre-trained Stable Diffusion model. In practice, we follow the approach of [40] and add control units to the four encoder blocks and one middle block of the denoising U-Net model used in Stable Diffusion, while utilizing zero-convolutions for the other four decoder blocks.

In addition to the Pano-Denoiser, we incorporate a shallow Pano-Encoder $\varepsilon _ { P }$ to construct our controlling condition. In order to feed image-space signals, such as ?? and ??, into the latent space as control signals, we use $\varepsilon _ { p }$ to encode them into latent features that are compatible with the original SD U-Net and consequently our control unit. The latent features are then passed to our learnable encoders and zero convolutions connected to the middle and decoder parts of the original SD U-Net.

![](images/7dc9f2df9e3b4a0a534055857b97a826f5d354ac3b3bb0582b3be58163b136b2.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    A["+ Gaussian noise"] --> B["Diffusion Process"]
    B --> C["Latent Space Diffusion"]
    C --> D["Latent Space Denoising"]
    D --> E["Image Space"]
    
    subgraph Latent Space Diffusion
        F["z_T"] --> G["Pano-Denoiser ε_θ"]
        H["z_0"] --> I["Sampling"]
        J["z_T-1"] --> K["Sampling"]
        L["z_0"] --> M["Sampling"]
    end
    
    subgraph Image Space
        N["ε_A"] --> O["Output"]
        P["ρ_A"] --> Q["Output"]
        R["ε_P"] --> S["Output"]
    end
    
    F --> T["C_P"]
    H --> U["C_P"]
    I --> V["C_P"]
    K --> W["C_P"]
    M --> X["C_P"]
    T --> Y["Original SD Blocks (Locked)"]
    U --> Z["Trainable Copy Blocks"]
    V --> AA["Zero Convolution Blocks"]
    W --> AB["Pano-Denoiser"]
    X --> AC["Pano-Encoder"]
    Y --> AD["+ Gaussian noise"]
    Z --> AE["+ Gaussian noise"]
    AA --> AF["+ Gaussian noise"]
    AB --> AG["+ Gaussian noise"]
    AC --> AH["+ Gaussian noise"]
    AD --> AI["+ Gaussian noise"]
    AE --> AJ["+ Gaussian noise"]
    AF --> AK["+ Gaussian noise"]
    AG --> AL["+ Gaussian noise"]
    AH --> AM["+ Gaussian noise"]
    AI --> AN["+ Gaussian noise"]
    AJ --> AO["+ Gaussian noise"]
    AK --> AP["+ Gaussian noise"]
    AL --> AQ["+ Gaussian noise"]
    AM --> AR["+ Gaussian noise"]
    AN --> AS["+ Gaussian noise"]
    AO --> AT["+ Gaussian noise"]
    AP --> AU["+ Gaussian noise"]
    AQ --> AV["+ Gaussian noise"]
    AR --> AW["+ Gaussian noise"]
    AS --> AX["+ Gaussian noise"]
    AT --> AY["+ Gaussian noise"]
    AU --> AZ["+ Gaussian noise"]
    AV --> BA["+ Gaussian noise"]
    AW --> BB["+ Gaussian noise"]
    AX --> BC["+ Gaussian noise"]
    AY --> BD["+ Gaussian noise"]
    AZ --> BE["+ Gaussian noise"]
    BA --> BF["+ Gaussian noise"]
    BB --> BG["+ Gaussian noise"]
    BC --> BH["+ Gaussian noise"]
    BE --> BI["+ Gaussian noise"]
    BF --> BJ["+ Gaussian noise"]
    BG --> BK["+ Gaussian noise"]
    BH --> BL["+ Gaussian noise"]
    BI --> BM["+ Gaussian noise"]
```
</details>

Figure 5: An overview of our partial-FoV-controlled latent diffusion model. The upper part and lower part are the diffusion process and sampling process of the model, respectively. Note that the text conditions and classifier-free guidance process are omitted from the figure for simplicity.

As depicted in Fig. 5, during the diffusion process (upper part of the figure), the encoder $\varepsilon _ { A }$ of the AutoEncoder transforms the image into feature space, and then Gaussian noises are added iteratively to the latent feature, as described in [14]. For the denoising process, our Pano-Denoiser utilizes the partial panorama ?? and visibility mask ?? as control signals and denoises the input noisy latent ???? iteratively in our rotating schedule(Sec. 4.4.2) for ?? steps until $z _ { 0 }$ is generated. Finally, $z _ { 0 }$ is decoded by the decoder $\mathcal { E } _ { D }$ of the AutoEncoder to produce the final panorama.

# 4.4 Denoising in 360-Degree

In this subsection, we focus on denoising 360-degree panoramas while preserving their unique geometric characteristics. During the training stage, we introduce a rotation equivariance loss to enforce rotation consistency in the latent space. During the inference stage, we employ a customized rotating schedule to enhance the robustness and maintain the geometric integrity of the generated panoramas. Additionally, we implement a circular padding technique during inference to mitigate edge effects and prevent geometric discontinuity.

4.4.1 Rotation Equivariance Loss. Panoramas are captured to depict a completely spherical environment, which means they exhibit rotation equivariance. This property implies that if we apply a transformation based on an ???? (3) rotation (limited to 2 degrees of freedom corresponding to longitude and latitude rotations) to a panorama image, it should still be able to represent the same scene.

As our method functions within the latent space, the constraint on our denoiser can be expressed as:

$$
\epsilon_ {\theta} (\mathcal {A} (R) z _ {t}, t, \mathbf {c} _ {\text { text }}, \mathcal {A} (R) \mathbf {c} _ {\mathrm{P}}) = \mathcal {A} (R) \epsilon_ {\theta} (z _ {t}, t, \mathbf {c} _ {\text { text }}, \mathbf {c} _ {\mathrm{P}}), \tag {7}
$$

![](images/5c585c2728f0cbd55c33b850bd0cb724adca0bfecd49dcc535c943c9d2454947.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph LR
    A["z_T"] -->|−ε| B["z_{T-1}"]
    B -->|−ε| C["z_1"]
    C -->|−ε| D["z_0"]
    E["z_T"] -->|−ε| F["z_{T-1}"]
    F -->|−ε| G["z_1"]
    G -->|−ε| H["z_0"]
    style A fill:#fff,stroke:#000
    style B fill:#fff,stroke:#000
    style C fill:#fff,stroke:#000
    style D fill:#fff,stroke:#000
    style E fill:#fff,stroke:#000
    style F fill:#fff,stroke:#000
    style G fill:#fff,stroke:#000
    style H fill:#fff,stroke:#000
    note right of F "Ours"
    note right of G "Target origin"
    note right of H "Image center"
```
</details>

Figure 6: An illustration of our rolling schedule. The target origin represents the position of the original pixel corresponding to longitude and latitude coordinate $( \theta = 0 , \phi = 0 )$ on the panorama.

where A (??) represents an image space projecting operation, similar to $\mathcal { F } _ { \mathrm { p r o j } }$ . This equation implies that when the latent feature $z _ { t }$ and all input image-space conditions undergo a transformation by ${ \mathcal { R } } ( R )$ , the predicted noise should also transform accordingly. To enforce this behavior, we introduce a rotation perturbation for latent features during the training phase. Consequently, the training objective evolves from Eq. 1 into:

$$
\mathcal {L} =
$$

$$
\mathbb {E} _ {\boldsymbol {z} _ {0}, t, \boldsymbol {c} _ {\mathrm{text}}, \boldsymbol {c} _ {\mathrm{P}}, \epsilon} [ \| \mathcal {A} (\Delta R) \epsilon - \epsilon_ {\theta} (\mathcal {A} (\Delta R) \boldsymbol {z} _ {t}, t, \boldsymbol {c} _ {\mathrm{text}}, \mathcal {A} (\Delta R) \boldsymbol {c} _ {\mathrm{P}}) \| _ {2} ^ {2} ]
$$

4.4.2 Rotating Schedule. During the inference process, we employ a customized schedule as depicted in Fig. 6.

As illustrated in Figure 6, the denoising steps are executed while the latent feature ???? undergoes a scheduled transformation A (Δ???? ) in a step-by-step manner. It is important to note that this schedule is consistent with the rotation constraint incorporated during the training phase, as both involve the same type of transformation in the latent space. The rotating schedule enhances the robustness of our method and facilitates the generation of panoramas with improved geometric integrity.

4.4.3 Circular Padding. As discussed in [11], generative methods operating in the latent space may lead to geometric discontinuity due to border effects of convolution operations. To address this issue, during inference time, we implement circular padding to mitigate edge effects. Specifically, the right portion of the latent feature is concatenated to the left side, while the left part of the original latent feature is concatenated to the right side. This process is illustrated in Fig. 7.

![](images/ada6f1a86aee1a373410461f087c65b7dfb40ce89183aee7c92a7e6146da5f46.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    A["Original Latent"] --> B["Copy & Concat"]
    B --> A'[A']
    A' --> B'[B']
    B' --> A'[A']
    style A fill:#f9f,stroke:#333
    style B fill:#ccf,stroke:#333
    style A' fill:#cfc,stroke:#333
    style B' fill:#fcc,stroke:#333
```
</details>

Figure 7: An illustration of our circular padding technique. Thin slices of the feature, denoted as ?? and ??, from both the left and right ends of the latent feature, are copied and concatenated to the right and left sides of the latent feature as ??′, ??′, respectively.

Table 1: FID↓ results compared with other generation methods for quantitative evaluation. 

<table><tr><td rowspan="2">Methods</td><td colspan="3">SUN360 [39]</td><td colspan="3">Laval [9]</td></tr><tr><td>Single</td><td>Pair (GT rots)</td><td>Pair (Pred rots)</td><td>Single</td><td>Pair (GT rots)</td><td>Pair (Pred rots)</td></tr><tr><td>SIG-SS [11]</td><td>13.06</td><td>15.94</td><td>16.50</td><td>-</td><td>-</td><td>-</td></tr><tr><td>StyleLight [37]</td><td>-</td><td>-</td><td>-</td><td>22.53</td><td>34.10</td><td>33.88</td></tr><tr><td>Omni-Comp [1]</td><td>14.69</td><td>12.38</td><td>12.63</td><td>18.91</td><td>14.62</td><td>14.73</td></tr><tr><td>Omni-Adjust [1]</td><td>11.27</td><td>9.93</td><td>10.63</td><td>9.60</td><td>10.47</td><td>10.92</td></tr><tr><td>ImmerseGAN [5]</td><td>9.26</td><td>10.46</td><td>10.57</td><td>12.69</td><td>24.23</td><td>24.51</td></tr><tr><td>PanoDiff (Ours)</td><td>8.61</td><td>6.94</td><td>7.04</td><td>7.72</td><td>6.96</td><td>7.18</td></tr></table>

After the sampling process, the decoder generates an image of shape $( w + 2 w _ { p } ) \times h ,$ where $w _ { p }$ is the extra width from the padded feature. To produce a standard panorama, the extra width is removed from the final image.

# 5 EXPERIMENTS

# 5.1 Implementation Details

5.1.1 Data Preparation. We conducted experiments using realworld 360-degree panoramic image datasets SUN360 [39] and Laval Indoor [9]. SUN360 comprises both indoor and outdoor scenes, while Laval Indoor comprises solely indoor scenes. For SUN360, we randomly selected 2000/500 panoramas for training/testing, respectively. For Laval Indoor, we followed the approach in [11] and chose 289 images for testing. Notably, we do not train our model using the Laval Indoor dataset as there are already indoor scenes within our selected training data from the SUN360 dataset.

We used two input types in our experiments: a single input, which is a single NFoV image with a $9 0 ^ { \circ }$ FoV placed at the center of the panorama, and paired input, which is a pair of NFoVs with relative rotation, allowing us to validate our method’s capability to generate a panorama from multiple input images. We generated five pairs for each of our training/testing panoramas, resulting in 10,000/2,500 pairs of training/testing inputs on SUN360.

5.1.2 Training. We train our angle prediction network and latent diffusion model separately. The angle prediction network is trained with the strategies mentioned in 3.2. Our Pano-denoiser and Pano-Encoder are trained for 7 epochs on pair inputs. For both single and pair input, we take in NFoV images of shape 256 × 256 and produce 1024 × 512 panoramas. During training, we generate input text prompts using BLIP [18].

5.1.3 Metrics. We evaluate our method using two kinds of metrics.

Panorama Generation. We use Fréchet Inception Distance (FID) [13, 17] as our quantitative metric since FID can report the visual quality of generated panorama images to some extent. Besides, it has also been adopted by prior studies [1, 11, 37].

Rotation Estimation. Following Cai et al. [2], we evaluate the geodesic error of the estimated rotation matrix (denoted by Rˆ ) and the ground truth matrix (denoted by R) using arccos ( ???? (R?? Rˆ ) −1 ). $\frac { t r ( \mathbf { R } ^ { T } \hat { \mathbf { R } } ) - 1 } { 2 } )$ 2

5.1.4 Baselines. We compared our method to three previous SOTA methods. Omni-Dreamer [1] is trained on SUN360 with 47938 images. For the Laval dataset, Omni-Dreamer is finetuned on the model trained on SUN360 and with 1837 training samples. SIG-SS [11] is only trained on SUN360 with 50000 training images. Stylelight [37]

Table 2: Evaluation of relative pose estimation on SUN360 [39] and Laval Indoor [9]. We present the Average geodesic error $\mathbf { A v g } ( ^ { \circ } \downarrow )$ and the percentage of pairs with errors under 10 degrees 10◦(%↑). 

<table><tr><td rowspan="2">Pair Type</td><td rowspan="2">Method</td><td colspan="2">SUN360 [39]</td><td colspan="2">Laval [9]</td></tr><tr><td>Avg(°↓)</td><td>10°(%↑)</td><td>Avg(°↓)</td><td>10°(%↑)</td></tr><tr><td rowspan="2">Overlap</td><td>1-stage</td><td>7.19</td><td>90.27</td><td>4.21</td><td>96.74</td></tr><tr><td>2-stage</td><td>3.58</td><td>95.61</td><td>3.66</td><td>96.88</td></tr><tr><td rowspan="2">Wide Baseline</td><td>1-stage</td><td>41.52</td><td>40.64</td><td>32.31</td><td>62.34</td></tr><tr><td>2-stage</td><td>27.12</td><td>49.77</td><td>29.46</td><td>62.62</td></tr><tr><td rowspan="2">All</td><td>1-stage</td><td>24.29</td><td>65.54</td><td>18.00</td><td>79.86</td></tr><tr><td>2-stage</td><td>15.31</td><td>72.77</td><td>16.32</td><td>80.07</td></tr></table>

is trained on the Laval dataset, the same training set as used in Omni-Dreamer. We inferred these models with their officially released trained models on our test split. Since their training split is unknown to us, images in our test set could be in their training set. We also reached out to the authors of ImmerseGAN [5] and acquired their model’s results on our test set.

# 5.2 Quantitative Evaluation

We examine the performance of our approach from two perspectives, namely the accuracy of rotation estimation and the panorama generation quality.

Rotation Estimation. We evaluate the performance of relative rotation estimation on SUN360 [39] and Laval Indoor [9] datasets. For clarity, we separate the input pairs into two categories based on whether they belong to the ’overlap’ or ’wide baseline’. We compare two relative rotation estimation models: our proposed two-stage model, and a single-stage model which is designed the same as [2]. Table. 2 reports the evaluation results, which demonstrate that the two-stage relative rotation estimation method significantly outperforms the single-stage approach in terms of average relative pose estimation error for both ‘overlap’ and ‘wide baseline’. It is noted that both methods trained only on the SUN360 dataset.

Panorama Generation. The primary results are summarized in Table. 1. As demonstrated in the table, our method attains the most favorable FID metrics with all three input types on both datasets. The terms GT rots and Pred rots denote the utilization of ground truth relative angles and predicted angles from our network ${ \mathcal { R } } _ { \theta } ,$ , respectively. Despite the imperfect nature of the FID metric [17, 26], the significant margin of our method’s superiority substantiates its overall effectiveness. It is noteworthy that our model is NOT trained on the Laval Indoor dataset, yet it surpasses the performance of previous methods that were specifically trained on this dataset.

![](images/272a0e260d9a8a7c5ca4fb7a705b0721b073bdc09912faae85bc292923c23dc7.jpg)

<details>
<summary>text_image</summary>

Input
SIG-SS
Omni-Comp
Omni-Adjust
ImmerseGAN
PanoDiff
Ground Truth
</details>

Figure 8: Out-painting results on SUN360 Dataset [39]. Omni-Comp and Omni-Adjust denote the CompletionNet and AdjustmentNet outputs of Omni-Dreamer [1], respectively.   
![](images/952bece5eb4daab5d1d3082adb4af66b3f1248346e0dc91428940cb6b6bc3a75.jpg)

<details>
<summary>text_image</summary>

Input
StyleLight
Omni-Comp
Omni-Adjust
ImmerseGAN
PanoDiff
Ground Truth
</details>

Figure 9: Out-painting results on Laval Indoor Dataset [9]. We evaluate the models of StyleLight, Omni-Comp, and Omni-Adjust, by utilizing their officially released models trained on Laval Indoor Dataset. The results obtained from ImmerseGAN were graciously provided by the authors who ran their model for our evaluation.

# 5.3 Qualitative Comparison

The visual results are shown in Figure 8. To illustrate the generation quality of our method, we compare our method to previous works under three kinds of inputs: single NFoV input(top two rows), the paired NFoV input with GT rots(middle two rows), and the paired NFoV input with Pred rots(bottom two rows). As can be observed from the results, both SIG-SS and Omni-Adjust exhibit inconsistencies in generating boundaries between the input patches and the out-painting content. While the generated images of Omni-Comp show a marginal improvement in boundary issues, they compromise the quality and realism of the generated content to achieve smooth output boundaries. Omni-Adjust produces relatively more realistic content, but it occasionally out-paints the wrong content. For instance, in the second row, Omni-Adjust wrongly out-paints a ‘beach’ while the input region indicates a grassland scene. ImmerseGAN excels at generating satisfactory results with a single FoV input, but struggles with paired FoV input due to the lack of specific training for handling multiple FoVs. PanoDiff outperforms previous approaches in content consistency, realism, and texture continuity, maintaining high-quality and realistic content generation.

We proceed to look at the generalizing performance of our approach on Laval Indoor Dataset[9] and present the results in Figure 9 with both single and paired inputs. Note that our model was NOT trained on the Laval Indoor dataset but solely on SUN360. Nonetheless, our method still achieves high-quality panorama generation with consistent visual properties, e.g., lighting, temperature, and geometrical consistency.

# 5.4 Ablation Study

We validate the efficacy of key components in our approach, i.e., the denoising strategies, and the circular padding strategy.

Denoising Strategies in 360-Degree. We conduct quantitative comparisons in Table. 3 regarding the usage of our rotation equivariance loss, and rotating schedule. These denoising strategies assist our model to understand the geometric characteristics of panoramas in latent space. As could be seen from the table, both our strategies contribute significantly to the quality of our panorama generation.

![](images/e0bfd0edfd1667ac11b2532361bdde80a83a656ceaa903a2f2f79c3a80731ec5.jpg)

Figure 10: Control panorama generation with text prompts. This demonstrates the capability of our method to generate diverse high-quality results with various text prompts, while still able to maintain the geometric characteristics of panoramas.   
![](images/633d26b9de43867639c5bf092b0870fd7676a351d8471ffc6cda6d6683e86cc3.jpg)

<details>
<summary>natural_image</summary>

Four-panel image comparing a circular parking system with and without rotating 180°, showing different interior views (no text or symbols on the images themselves)
</details>

Figure 11: Circular padding solves the problem of discontinuity between the left and right sides in panorama generation.

Table 3: FID↓ results of PanoDiff with different strategies. ‘Equi.’ denotes the usage of our rotation equivariance loss in Sec. 4.4.1, and ‘Schedule’ denotes our rotating denoising schedule in Sec. 4.4.2. 

<table><tr><td>Strategy</td><td>Equi.</td><td>Schedule</td><td>FID ↓</td></tr><tr><td rowspan="3">Choices</td><td>-</td><td>-</td><td>7.88</td></tr><tr><td>√</td><td>-</td><td>6.73</td></tr><tr><td>√</td><td>√</td><td>6.56</td></tr></table>

Circular Padding For Continuity. When decoding the latent feature ??0, the inherited border effect caused by the domain gap in latent space and image space frequently occurs. To alleviate this issue, we implement a circular padding strategy as described in 4.4.3. In Figure 11, we validate the necessity of circular padding with visual result comparisons by rotating 180◦ horizontally from the output image. As observed, circular padding seamlessly eliminates discontinuity between both ends.

# 5.5 Applications

Text Editing. We extend the editing capability of our method. As depicted in Figure 10, our method not only inherits the powerful text-to-image generation ability from Stable Diffusion but also preserves the sound geometric properties of panoramas.

Environment Texture. Panoramic images can serve as environment textures in 3DCG software, which can provide background

lighting for 3D assets. Figure 12 shows some examples of our generated panoramas as environmental textures. As can be found, our method produces diverse panoramas that not only serve as plausible rendering backgrounds (first two) but also provide environmental lighting (last two).

![](images/e5ce75614ed5046f4fad241f86b047d1a74d6dd67a0232dec6b747d005783700.jpg)  
Figure 12: Examples of panoramic images used as environment textures.

Multiple NFoV As Input. Our framework includes a two-stage relative pose estimation module, allowing it to handle multiple NFoV images (e.g., >2) of the same scene as input. As shown in Figure 1, our pipeline can robustly generate high-quality panoramic images using multiple NFoV images. Please refer to the supplementary for more generated 360-degree panoramas.

# 6 CONCLUSION

In this paper, we present PanoDiff, a novel framework that generates 360◦ panoramas from one or more NFoV inputs. The pipeline consists of two main modules, namely rotation estimation and panorama generation. Our two-stage rotation estimation network first classifies the input image pairs into overlap and wide baseline scenarios and then performs precise angle prediction. In panorama generation, we use incomplete partial panoramas along with text prompts as signals to generate diverse panoramas. We hope that our work inspires further research in panorama generation for advanced applications, including style control, direct HDRI panorama generation, and related areas.

Acknowledgement. This work was supported by the Fundamental Research Funds for the Central Universities, STCSM under Grant 22DZ2229005, 111 project BP0719010.

# REFERENCES

[1] Naofumi Akimoto, Yuhi Matsuo, and Yoshimitsu Aoki. 2022. Diverse plausible 360- degree image outpainting for efficient 3dcg background creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11441– 11450.   
[2] Ruojin Cai, Bharath Hariharan, Noah Snavely, and Hadar Averbuch-Elor. 2021. Extreme rotation estimation using dense correlation volumes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14566–14575.   
[3] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. 2021. Crossvit: Crossattention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF international conference on computer vision. 357–366.   
[4] Zhaoxi Chen, Guangcong Wang, and Ziwei Liu. 2022. Text2light: Zero-shot text-driven HDR panorama generation. ACM Transactions on Graphics (TOG) 41, 6 (2022), 1–16.   
[5] Mohammad Reza Karimi Dastjerdi, Yannick Hold-Geoffroy, Jonathan Eisenmann, Siavash Khodadadeh, and Jean-François Lalonde. 2022. Guided Co-Modulated GAN for 360° Field of View Extrapolation. In 2022 International Conference on 3D Vision (3DV). IEEE, 475–485.   
[6] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. 2018. Superpoint: Self-supervised interest point detection and description. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 224–236.   
[7] Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems 34 (2021), 8780–8794.   
[8] Patrick Esser, Robin Rombach, and Bjorn Ommer. 2021. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12873–12883.   
[9] Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde. 2017. Learning to predict indoor illumination from a single image. arXiv preprint arXiv:1704.00090 (2017).   
[10] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Commun. ACM 63, 11 (2020), 139–144.   
[11] Takayuki Hara, Yusuke Mukuta, and Tatsuya Harada. 2022. Spherical Image Generation From a Few Normal-Field-of-View Images by Considering Scene Symmetry. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).   
[12] Heathen. [n. d.]. Discussion on Stable Diffusion WebUI. https://github.com/ automatic1111/stable-diffusion-webui/discussions/2670   
[13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017).   
[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33 (2020), 6840–6851.   
[15] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. 2022. Cascaded Diffusion Models for High Fidelity Image Generation. J. Mach. Learn. Res. 23, 47 (2022), 1–33.   
[16] Jonathan Ho and Tim Salimans. 2022. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022).   
[17] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2022. The Role of ImageNet Classes in Fr\’echet Inception Distance. arXiv preprint arXiv:2203.06026 (2022).   
[18] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In ICML.   
[19] Kang Liao, Lang Nie, Shujuan Huang, Chunyu Lin, Jing Zhang, Yao Zhao, Moncef Gabbouj, and Dacheng Tao. 2023. Deep Learning for Camera Calibration and Beyond: A Survey. arXiv preprint arXiv:2303.10559 (2023).   
[20] Kang Liao, Xiangyu Xu, Chunyu Lin, Wenqi Ren, Yunchao Wei, and Yao Zhao. 2022. Cylin-Painting: Seamless 360 {\deg} Panoramic Image Outpainting and Beyond with Cylinder-Style Convolutions. arXiv preprint arXiv:2204.08563 (2022).   
[21] Guilin Liu, Aysegul Dundar, Kevin J Shih, Ting-Chun Wang, Fitsum A Reda, Karan Sapra, Zhiding Yu, Xiaodong Yang, Andrew Tao, and Bryan Catanzaro. 2022. Partial convolution for padding, inpainting, and image synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).

[22] David G Lowe. 2004. Distinctive image features from scale-invariant keypoints. International journal of computer vision 60 (2004), 91–110.   
[23] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. 2022. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11461–11471.   
[24] Wei-Chiu Ma, Anqi Joyce Yang, Shenlong Wang, Raquel Urtasun, and Antonio Torralba. 2022. Virtual correspondence: Humans as a cue for extreme-view geometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15924–15934.   
[25] Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. 2018. Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv:1804.03999 (2018).   
[26] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. 2022. On Aliased Resizing and Surprising Subtleties in GAN Evaluation. In CVPR.   
[27] Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In ICML.   
[28] Chris Rockwell, Justin Johnson, and David F. Fouhey. 2022. The 8-Point Algorithm as an Inductive Bias for Relative Pose Prediction by ViTs. In 3DV.   
[29] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2021. High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2112.10752 [cs.CV]   
[30] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer, 234–241.   
[31] RunwayML. 2021. Stable Diffusion. https://github.com/runwayml/stablediffusion.   
[32] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. 2020. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 4938–4947.   
[33] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 (2022).   
[34] Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020).   
[35] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. 2022. Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2149–2159.   
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).   
[37] Guangcong Wang, Yinuo Yang, Chen Change Loy, and Ziwei Liu. 2022. Stylelight: Hdr panorama generation for lighting estimation and editing. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV. Springer, 477–492.   
[38] Yinhuai Wang, Jiwen Yu, and Jian Zhang. 2022. Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model. arXiv preprint arXiv:2212.00490 (2022).   
[39] Jianxiong Xiao, Krista A Ehinger, Aude Oliva, and Antonio Torralba. 2012. Recognizing scene viewpoint using panoramic place representation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2695–2702.   
[40] Lvmin Zhang and Maneesh Agrawala. 2023. Adding Conditional Control to Text-to-Image Diffusion Models. arXiv:2302.05543 [cs.CV]   
[41] Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu. 2021. Large Scale Image Completion via Co-Modulated Generative Adversarial Networks. In International Conference on Learning Representations (ICLR).   
[42] Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu. 2021. Large scale image completion via co-modulated generative adversarial networks. arXiv preprint arXiv:2103.10428 (2021).