# DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion

Weicai Ye1,3,∗ Chenhao Ji2,∗ Zheng Chen4 Junyao Gao2 Xiaoshui Huang3

Song-Hai Zhang4 Wanli Ouyang3 Tong He3,B Cairong Zhao2,B Guofeng Zhang1,B

1State Key Lab of CAD&CG, Zhejiang University 2Tongji University

3Shanghai AI Laboratory 4Tsinghua Univerisity

maikeyeweicai@gmail.com jichenhao@tongji.edu.cn zhangguofeng@zju.edu.cn

![](images/0dea0e45872ef5e4cdf5f901c033e2a560fd46dac0e9cd6a3b5cf21f3c7d5f0d.jpg)

<details>
<summary>natural_image</summary>

Four-panel interior view of a modern minimalist room with curved wooden flooring and white walls, showing interior furniture and lighting (no text or symbols visible)
</details>

“A well-lit living space with wooden floors and a white ceiling, featuring an open doorway leading to a hallway. The room is furnished with a dining table and chairs. ”

![](images/6c7d73650abda857a4296a7efbef1f37ab96501ddf879245133de8635694dbe1.jpg)

<details>
<summary>natural_image</summary>

Four-panel interior view of a modern kitchen and dining area, showing white walls, dark kitchen areas, and curved ceiling structures (no visible text or symbols)
</details>

“A kitchen with white cabinets and black countertops. A well-lit room with white walls, wood floors, and a couch in the corner.”

![](images/58d920bb4fbc35df7592eb337f85b8bf90e7b7083d750a70b4d729495f95ce8d.jpg)

<details>
<summary>natural_image</summary>

Four-panel interior view of a modern room with wooden flooring, featuring minimalist furniture and curved white panels (no text or symbols visible)
</details>

“A well-lit room with wooden floors, and a door that leads to a dining area. The room is decorated with a vase of flowers and features a ceiling with a light fixture.”

![](images/fd3b4c7a1ea63347b5f8e0c8424cf8506004d27cf76710cf01075a26041bc381.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern bedroom with curved wooden furniture and white walls (no text or symbols visible)
</details>

“A well-maintained room with white walls and wooden floors. The room features a brown carpet, a bed, and a wooden dresser. A bedroom with an open door leading to the hallway.”   
Figure 1: DiffPano allows scalable and consistent panorama generation (i.e. room switching) with given unseen text descriptions and camera poses. Each column represents the generated multi-view panoramas, switching from one room to another.

# Abstract

Diffusion-based methods have achieved remarkable achievements in 2D image or 3D object generation, however, the generation of 3D scenes and even 360◦ images remains constrained, due to the limited number of scene datasets, the complexity of 3D scenes themselves, and the difficulty of generating consistent multi-view images. To address these issues, we first establish a large-scale panoramic video-text dataset containing millions of consecutive panoramic keyframes with corresponding panoramic depths, camera poses, and text descriptions. Then, we propose a novel text-driven panoramic generation framework, termed DiffPano, to achieve scalable, consistent, and diverse panoramic scene generation. Specifically, benefiting from the powerful generative capabilities of stable diffusion, we fine-tune a single-view text-to-panorama diffusion model with LoRA on the established panoramic videotext dataset. We further design a spherical epipolar-aware multi-view diffusion model to ensure the multi-view consistency of the generated panoramic images.

Extensive experiments demonstrate that DiffPano can generate scalable, consistent, and diverse panoramic images with given unseen text descriptions and camera poses. Code and dataset are available at https://zju3dv.github.io/DiffPano.

# 1 Introduction

Generating scenarios from the text descriptions that meet one’s expectations is an imaginative and marvelous journey, which has many potential applications, such as VR roaming [62] for metaverse [68, 70, 71, 74], physical world simulation [2, 6, 14], embodied agents in scene navigation and manipulation [77], etc. With the advent of the AIGC era, several works [8, 14, 20, 29, 42, 43, 48, 51, 52] on object generation even scene generation has emerged. However, they generally only generate a series of perspective images, making it impossible to comprehensively simulate the entire environment for scene understanding [27, 19, 67] and reconstruction [3, 5, 66, 64, 30, 35, 34, 26, 21, 65, 7, 12, 47, 49, 58, 69]. Given this, some methods [72, 9] try to generate panoramas to solve these problems by taking advantage of the inherent characteristics of panoramic images, which can capture the surrounding environment with a single shot.

These methods can be roughly divided into four categories: 1) Directly single-view equirectangular projection (ERP) panorama generation [72]. However, its camera is immovable, making it incapable of scene exploration; 2) Multiple perspective views generation methods [20, 43, 48, 40] without considering multi-view panorama generation. 3) The inpainting solutions are based on the infinite expansion of a single perspective view, which lacks 3D awareness. Single scene optimization methods [53] with inpainting have no generalization ability and cost too much time for optimization. 4) Directly extending the multiple perspective views generation methods [43] to the ERP panorama, which is difficult to converge and results in poor multi-view consistency (see Fig. 5).

This paper aims to generate scalable and multi-view consistent panoramic images from text descriptions and camera poses (see Fig. 1) with many potential applications such as immersive VR roaming with unlimited scapes and preview for interior home design. However, achieving this goal is not trivial. To the best of our knowledge, there is currently a lack of rich and diverse panoramic datasets to meet the task of text-to-multi-view ERP panorama generation. To this end, we propose a novel panoramic video-text dataset and a generation framework suitable for the text-to-multi-view panorama generation task, advancing the development of this field. Specifically, we first establish a large-scale panoramic video-text dataset using Habitat Simulator [41] (see Sec. 3), which contains millions of panoramic keyframes and corresponding panoramic depths, camera poses, and text descriptions. Next, built upon the proposed dataset, we propose a generation framework for consistent multi-view ERP panorama generation (see Sec. 4), termed DiffPano. The DiffPano framework consists of a single-view text-to-panorama diffusion model (see Sec. 4.1) and a spherical epipolar-aware multiview diffusion model (see Sec. 4.2). The single-view text-to-panorama diffusion model is obtained by fine-tuning the stable diffusion model [38] of perspective images using LoRA [18]. Considering that the single-view pano-based diffusion model cannot guarantee the consistency of generated multiview panoramas with different camera poses, we derive a spherical epipolar constraint applicable to panoramic images, inspired by the perspective epipolar constraint. We then incorporated it as a spherical epipolar-aware attention module (see Sec. 4.2) into the multi-view panoramic diffusion model to ensure the multi-view consistency of the generated ERP panoramic images.

Since there are no related methods for comparison, we try to extend the MVDream method [43] to generate multi-view ERP panoramas. Trained and tested on the proposed panoramic video-text dataset, extensive experiments demonstrated that compared to the modified MVDream, our proposed multi-view panorama generation based on spherical epipolar-aware attention can generate more scalable and consistent panoramic images. Our method also demonstrates the generalization ability of the original diffusion model to generate satisfactory multi-view ERP panoramas with given unseen text descriptions and camera poses.

Our contributions can be summarized as follows: 1) To the best of our knowledge, we are the first to propose a scalable and consistent multi-view panorama generation task from text descriptions and camera poses. 2) We established a large-scale diverse and rich panoramic video-text dataset, which fosters the research of text-to-panoramic video generation. 3) We propose a novel text-driven panoramic generation framework with a spherical epipolar attention module, allowing scalable and consistent panorama generation with unseen text descriptions and camera poses.

# 2 Related work

# 2.1 Single-View Panorama Generation

Recently, latent diffusion model (LDM) methods have attracted widespread attention, and many single-view panorama generation works [72, 48, 76, 24, 60, 56, 33, 54, 62] have emerged, achieving remarkably impressive results. Among them, MVDiffusion [48] simultaneously generates eight fixed-viewpoint perspective images through a multi-view Correspondence-Aware diffusion model and stitches them together to produce a panorama. However, it cannot support the generation of top and bottom views, and the generated panorama resembles wide-angle images with an extensive field of view rather than true 360◦ images. Some methods [55, 13] solve this problem using equirectangular projection (ERP) and try to facilitate the interaction between the left and right sides during the panorama generation process to enhance the left-right continuity property inherent in ERP images. To address the domain gap between panorama and perspective images, PanFusion [72] proposed a novel dual-branch diffusion model that mitigates the distortion of perspective images projected on panoramas while providing global layout guidance. However, its more complex model architecture incurs longer inference times for panorama generation. In addition, PanFusion cannot be expanded as an effective pre-trained model to the multi-view panorama generation task due to its excessive network parameters. To strike a balance between computational complexity and ensuring left-right continuity of panoramas, our proposed single-view panorama-based stable diffusion model only requires fine-tuning with LoRA [18] to learn panoramic styles and achieve good edge continuity while maintaining higher generation speed and simpler architecture.

The existing single-view panorama generation methods cannot achieve scalable panorama generation. The core of our paper lies in the generation of multi-view consistent panoramic images, which we will introduce in Section 2.2. More importantly, the single-view panoramic image generated by previous methods mainly supports 3DoF roaming, while our method can generate multi-view panoramic images for 6DoF roaming, which can serve as the inputs for 360◦ Gaussian Splatting [23] or 360◦ NeRF [10, 11]. Our method also has a great potential value in 360◦ relightable novel view synthesis with the combination of 360◦ multi-view inverse rendering method [25].

# 2.2 Multi-View Image Generation

To the best of our knowledge, there is no work focusing on multi-view panorama generation. We review the existing works about multi-view generation for perspective images in this part.

Zero123 [29] laid the foundation for 3D object generation based on multi-view generation, while the pose-guided diffusion model [51, 40] explored consistent view synthesis of scenes. However, iteratively applying the diffusion model to generate individual views in the multi-view generation task may lead to poor multi-view consistency of generated images due to accumulated errors. To generate high-quality multi-view images simultaneously, some methods [31, 43, 57, 32] modify the UNets in the diffusion model into a multi-branch form and achieve the effect of generating consistent multi-view images through the interaction between different branch.

Currently, most multi-view generation tasks focus on generating multi-view perspective images of single objects [43, 61, 63, 28, 46, 45] or scenes [51, 15], while minimal research has been conducted on multi-view panorama generation. Narrow FoV (field-of-view) drawbacks of perspective images lead to the fact that the existing generation methods can only generate a very local region of the scene at a time. Our work focuses on the task of exploring the generation of 360◦ images from multiple different viewpoints. Due to the camera projection difference between panoramic and perspective images, achieving consistency in multi-view panoramas is challenging. It is impossible to directly apply the existing epipolar attention module [51, 20] to multi-view panoramas. We strive to derive the spherical epipolar line formula for panoramic images and propose a spherical epipolar attention module to ensure the multi-view consistency of the generated panoramas.

# 2.3 Panoramic Dataset

Great progress in text to single-view panorama generation has been witnessed. However, text-to-multiview panorama generation is still a blank slate. One of the main limitations of this task is the lack of suitable datasets. The common panoramic datasets used in single-view panorama generation consist of indoor HDR dataset [16], outdoor HDR dataset [73], HDR360-UHD dataset [9], Structured3D [75],

![](images/03662fbaf2806b7dfd54edd4036df3aa387ac3bc4580fdbb88a21ce53f952a40.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    A["3D Scene"] --> B["Habitat Simulator"]
    B --> C["Camera Trajectory"]
    C --> D["RGB and Depth CubeMap"]
    D --> E["BLIP2"]
    E --> F["BLIP2"]
    G["Stitched Panorama and Depth"] --> H["Prompt: Given six text prompts that describe the appearance of a room ... Your task is to summarize these six given text prompts into a single caption that describe what the room looks like."]
    H --> I["LLM"]
    I --> J["A well-furnished bedroom with beige walls, a white bed, and a white wardrobe. The floor is carpeted. The room also features a painting on the wall and a white door. The ceiling is tan, and the carpet is soft and comfortable."]
    J --> K["A well-furnished bedroom with beige walls, a white bed, and a white wardrobe. The floor is carpeted. The room also features a painting on the wall and a white door. The ceiling is tan, and the carpet is soft and comfortable."]
    L["CubeMap Caption"] --> M["View 1: The ceiling in a bathroom. View 2: An empty room with a painting on the wall. View 3: A bedroom with beige walls and a white bed. View 4: A bedroom with tan walls and a carpeted floor. View 5: An empty room with white doors. View 6: The carpet in a bedroom."]
    M --> N["..."]
    N --> O["BLIP2"]
```
</details>

Figure 2: Panoramic Video Construction and Caption Pipeline.

Standford 2D-3D-S [1], and Matterport3D dataset [4], etc. Most of these datasets are relatively small in scale and only have single-view panoramas, which cannot support multi-view panorama generation, except Matterport3D [4]. In addition, the sky box images in Matterport3D [4] contain only sparse views. Although HM3D [37] provides the textured mesh of 1000 scenes, it lacks the corresponding text description for each view. To generate multi-view panoramas, we render cube maps at each viewpoint in the 3D meshes of HM3D, using the Habitat Simulator [41], and stitch them into panoramas. We generate complete text descriptions corresponding to the panoramas by using Blip2 [22] to create text descriptions for each face of the cube map separately, and then summarizing them using Llama2 [50]. In this way, we obtain a panoramic video-text dataset that includes camera poses, corresponding panoramas, and text descriptions of the panoramas, which facilitates subsequent multi-view panorama generation tasks.

# 3 Panoramic Video-Text Dataset

Due to the lack of high-quality panorama-text datasets, most text-to-panorama generation tasks require researchers to construct their own datasets. The dataset constructed in PanFusion [72] suffers from blurriness at the top and bottom of the panoramic images, and the corresponding text descriptions are not precise enough. To address these issues, we utilize the Habitat Simulator [41] to randomly select positions within the scenes of the Habitat Matterport 3D (HM3D) [37] dataset and render the six-face cube maps. These cube maps are then interpolated and stitched together to form panoramas so we can obtain panoramas with clear tops and bottoms. To generate more precise text descriptions for the panoramas, we first use BLIP2 [22] to generate corresponding text descriptions for each obtained cube map, and then employ Llama2 [50] to summarize and receive accurate and complete text descriptions. Furthermore, the Habitat Simulator allows us to render images based on camera trajectories within the HM3D scenes, enabling the creation of a dataset that simultaneously includes camera poses, panoramas, and corresponding text descriptions. This dataset will be utilized in the multi-view panorama generation (see Sec. 4.2).

# 4 Proposed Method: DiffPano

DiffPano is capable of generating multi-view consistent panoramas conditioned on camera viewpoints and textual descriptions, as illustrated in Fig. 1. In this section, we first introduce our single-view panorama stable diffusion in Sec. 4.1. We then elaborate on how to extend single-view panorama generation to multi-view consistent panorama generation by leveraging the spherical epipolar attention module in Sec. 4.2.

![](images/c8fb6caec28b5971e20b6ef5387726c3ded497c058e8f6a95f92335fa83c0972.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    A["Noise"] --> B["Text Prompt"]
    B --> C["Stable Diffusion LoRA"]
    C --> D["Generated Panorama"]
    D --> E["Spherical Epipolar-Aware Multi-View Diffusion"]
    E --> F["Target View"]
    E --> G["Epipolar Line Computation"]
    E --> H["Source View"]
    E --> I["Spherical Epipolar-Aware Module"]
    I --> J["Camera Poses R,T1:N"]
    J --> K["X1:N"]
    K --> L["Denoising UNet"]
    L --> M["Image with Text Prompts P1:N"]
    L --> N["Image with Noise X1:N"]
    L --> O["Image with Camera Poses R,T1:N"]
    style A fill:#f9f,stroke:#333
    style B fill:#bbf,stroke:#333
    style C fill:#bfb,stroke:#333
    style D fill:#ffb,stroke:#333
    style E fill:#ffb,stroke:#333
    style F fill:#cfc,stroke:#333
    style G fill:#cfc,stroke:#333
    style H fill:#cfc,stroke:#333
    style I fill:#fcc,stroke:#333
    style J fill:#ffc,stroke:#333
    style K fill:#ffc,stroke:#333
    style L fill:#fff,stroke:#333
    style M fill:#fff,stroke:#333
    style N fill:#fff,stroke:#333
    style O fill:#fff,stroke:#333
```
</details>

Figure 3: DiffPano Framework. The DiffPano framework consists of a single-view text-to-panorama diffusion model and a spherical epipolar-aware multi-view diffusion model. It can support text to single-view panorama or multi-view panorama generation.

# 4.1 Single-View Panorama-Based Stable Diffusion

A straightforward way to generate a single-view panorama from text is to train a text-to-panorama diffusion model from scratch with a large number of text-panorama pairs, which is both timeconsuming and computationally expensive. However, stable diffusion [38] leverages a vast amount of perspective images and their corresponding textual descriptions as training data, endowing it with excellent prior knowledge of perspective images and strong text understanding capabilities. An economical and effective way for panorama generation is to fine-tune the trained perspective diffusion model with a few text-panorama pairs. To this end, panorama generation from text can be regarded as a style transfer of images generated by stable diffusion, converting them from perspective style to panoramic style, and requiring them to satisfy the left-right continuity property of panoramas.

LoRA-based fine-tuning Diffusion models for text-to-image generations possess excellent prior knowledge of 2D images and strong text comprehension capabilities. We aim to preserve these abilities of the model while fine-tuning it to generate images in the style of panoramas. We employ the Low-Rank Adaptation (LoRA) [18] fine-tuning method, which has been previously used in large language models. Compared to full fine-tuning, LoRA is faster and requires fewer computational resources. In our approach, we freeze all the parameters of the original Stable Diffusion model and add trainable layers to the UNet component using the LoRA fine-tuning method. We then train the model using our custom-created panorama-text dataset. To improve the left-right continuity of the generated images, we perform data augmentation on the panorama training dataset by randomly concatenating a portion of the right side of the panorama to the left side. Experiments demonstrate that the panoramas generated using this method exhibit satisfactory left-right continuity.

# 4.2 Spherical Epipolar-Aware Multi-View Diffusion

Built upon our proposed single-view panorama stable diffusion in Sec. 4.1, we extend the single-view diffusion model to a multi-view diffusion model with a spherical epipolar-aware attention module to generate multi-view scalable and consistent panoramas.

Spherical Epipolar-Aware Attention Module Epipolar attention was proposed in [20, 51] to ensure consistency between generated multi-view perspective images. However, due to the differences in imaging methods between perspective and panoramic views, existing epipolar attention cannot be directly used for panoramic views. To overcome this challenge, we derived the epipolar line for panoramic images in the equirectangular projection (ERP), and the specific proof process is provided in Appendix A. Equation 14 shows the mathematical form of the spherical epipolar line in ERP images. The spherical epipolar line is visualized in the spherical epipolar-aware attention module of Fig. 3. We extend the principle of epipolar attention to panoramic images to implement the spherical epipolar-aware attention module. Given a pixel p in the target view, we calculate its corresponding spherical coordinates $\mathbf { p } _ { s p h e r e }$ based on the spherical projection process:

$$
\theta = \left(0. 5 - \frac {x _ {p i x}}{W}\right) \cdot 2 \pi \tag {1}
$$

$$
\phi = (0. 5 - \frac {y _ {p i x}}{H}) \cdot \pi ,
$$

where $x _ { p i x }$ and $y _ { p i x }$ are the pixel coordinates of p, θ and $\phi$ are its corresponding spherical coordinates and W and H are the resolutions of panorama. We then transform the spherical coordinate system to the Cartesian coordinate system to obtain the camera coordinates $p _ { c a m e r a }$ corresponding to p :

$$
x _ {c a m} = \cos (\phi) \cdot \sin (\theta)
$$

$$
y _ {c a m} = \sin (\phi) \tag {2}
$$

$$
z _ {c a m} = \cos (\phi) \cdot \cos (\theta).
$$

The camera coordinates of $\mathbf { p } _ { c a m }$ are converted to world coordinates $\mathbf { p } _ { w o r l d }$ through the camera’s pose matrix. This allows us to compute the ray from the camera center to $\mathbf { p } _ { w o r l d }$ in the world coordinate system and construct the spherical epipolar attention module.

Given N feature maps $F = F _ { i } | 1 \leq i \leq N$ corresponding to $N$ panoramic images and their respective camera pose matrices, we implement cross-attention between different views through the spherical epipolar-aware module. During the generation process, each feature map in $F$ can be considered as the target view, and the K nearest views are selected from the remaining features as reference views.

For each feature point in the target view feature map, we uniformly sample $S$ points on the ray between the feature point and the camera. All sampled points are reprojected onto the feature maps of the K reference views, and the corresponding feature values are obtained through interpolation. We denote the features in the target view as $q ,$ and the features of all sampled points in the reference views as k and $v .$ The cross-attention is then constructed using these features.

Let $p _ { i }$ be a feature point in the target view feature map $F _ { t }$ , and $p _ { i , j } | 1 \leq j \leq S$ be the S uniformly sampled points on the ray between $p _ { i }$ and the camera center. We reproject these points onto the K reference view feature maps $\mathbf { \tilde { { F } } } _ { r _ { k } } | 1 \leq k \leq K$ to obtain the corresponding feature values $f _ { i , j , k } | 1 \le j \le S , 1 \le k \le K$ . The query $q _ { i }$ , key $k _ { i }$ , and value $v _ { i }$ for the cross-attention mechanism are defined as follows:

$$
q _ {i} = F _ {t} (p _ {i}), \quad k _ {i} = f _ {i, j, k} | 1 \leq j \leq S, 1 \leq k \leq K, \quad v _ {i} = f _ {i, j, k} | 1 \leq j \leq S, 1 \leq k \leq K. \tag {3}
$$

The cross-attention output $o _ { i }$ for the feature point $p _ { i }$ is computed as:

$$
o _ {i} = \text { Attention } (q _ {i}, k _ {i}, v _ {i}) = \text { softmax } (\frac {q _ {i} k _ {i} ^ {T}}{\sqrt {d}}) v _ {i}, \tag {4}
$$

where d is the dimension of the query and key vectors.

Positional Encoding To enhance the model’s understanding of 3D spatial relationships between different views, we follow EpiDiff [20] to employ the positional encoding method from Light Field Networks (LFN) [44]. In the world coordinate system, let $p _ { i }$ be a pixel in the target view, and $o _ { i }$ and $d _ { i }$ be the origin and direction of the ray between $p _ { i }$ and the camera center, respectively. The Plücker coordinates $r _ { i }$ of the ray are computed as:

$$
r _ {i} = \left(o _ {i} \times d _ {i}, d _ {i}\right). \tag {5}
$$

For each sampled point $p _ { i , j }$ on the ray, its corresponding spherical depth $z _ { i , j }$ is transformed using a harmonic transformation to get $\gamma _ { z } ( z _ { i , j } )$ . Similarly, the Plücker coordinates $r _ { i }$ are transformed as $\gamma _ { r } ( r _ { i } )$ .The positionally encoded features $\gamma _ { r } ( r _ { i } )$ and $\gamma _ { z } ( z _ { i , j } )$ are then concatenated to obtain the combined positional encoding $\gamma ( r _ { i } , z _ { i , j } )$ :

$$
\gamma (r _ {i}, z _ {i, j}) = [ \gamma_ {r} (r _ {i}), \gamma_ {z} (z _ {i, j}) ]. \tag {6}
$$

The combined positional encoding $\gamma ( r _ { i } , z _ { i , j } )$ is then concatenated with the feature maps $F _ { t }$ and $F _ { r _ { k } } | 1 \leq k \leq K$ to obtain the enhanced feature representations $\hat { F } t$ and ${ \hat { F } r _ { k } } | 1 \leq k \leq K$ :

$$
\hat {F} t (p _ {i}) = [ F _ {t} (p _ {i}), \gamma (r _ {i}, z _ {i, j}) ], \quad \hat {F} r _ {k} (p _ {i, j}) = [ F _ {r _ {k}} (p _ {i, j}), \gamma (r _ {i}, z _ {i, j}) ], \tag {7}
$$

where [·, ·] denotes concatenation. These enhanced feature representations are then used in the cross-attention mechanism to improve the model’s understanding of 3D spatial relationships between different views.

Two-Stage Training The main difference between panoramic images and perspective images is that panoramic images contain 360◦ content of the surroundings, while perspective images only contain content from a given viewpoint. Therefore, when the camera only rotates or translates by a small amount, the corresponding content in the panoramic image hardly changes. To make the generated multi-view panoramic images better match each corresponding text, we divide the training into two stages. In the first stage, we use the selected dataset with almost no change in image content (small camera movement) for training, which enhances the effect of the spherical epipolar-aware attention module. In the second stage, we increase the camera movement distance between each viewpoint and train with images that generate new content, improving the model’s ability to understand text based on changes in perceived spatial location while ensuring multi-view consistency and enhancing scalability.

# 5 Experiment

Dataset We leverage the Habitat Simulator [41] to render a panoramic video dataset based on the Habitat Matterport 3D (HM3D) dataset [37]. The pipeline of dataset rendering and captioning is shown in Sec. 3. After post-processing such as dataset filtering, we constructed 8,508 panorama-text pairs as training sets for single-view panorama generation. For multi-view panorama generation, we constructed 19,739 multi-view panorama-text pairs with nearly identical image content and 18,704 multi-view panorama-text pairs with different image contents as training sets. For specific details regarding the dataset, please refer to Appendix B.

Implementation Details In the multi-view panorama generation, we simultaneously generate N = 4 panoramas from different viewpoints. Within the spherical epipolar-aware attention module, we consider the two nearest views to the target view as reference views, i.e., K = 3, and sample S = 10 points along each ray. We conducted separate training for 100 epochs on datasets with almost identical image content and datasets with varying image content. Please refer to Appendix C for further implementation details.

Evaluation Metrics To evaluate the performance of our proposed single-view panorama-based stable diffusion model, we employ three commonly used metrics: Fréchet Inception Distance (FID) [17], Inception Score (IS) [39], and CLIP Score (CS) [36]. FID measures the similarity between the distribution of generated panoramas and the distribution of real images. IS assesses the quality and diversity of the generated panoramas. CS is utilized to evaluate the consistency between the input text and the generated panoramas. To further evaluate the consistency of multi-view panorama generation, we leverage the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) [59] metrics. PSNR and SSIM quantify the pixel-level differences and structural similarity between the generated views respectively.

# 5.1 Single-View Panorama Generation

Baselines We evaluate the performance of our proposed method by comparing it with the following baseline approaches for text-to-panorama generation:

• Text2Light [9]is a two-stage approach that first generates a low-resolution panorama based on the input text, and then expands it to ultra-high resolution.   
• PanFusion [72]is a dual-branch text-to-panorama model that aims to mitigate the distortion caused by projecting perspective images onto a panoramic canvas while providing global layout guidance.

Since the panoramic images generated by MVDiffusion [48] do not include top and bottom viewpoints, they are not complete panoramas. Therefore, we do not compare our method with MVDiffusion.

![](images/17d87fd6ec720187d56c68541d6ddbaa17dce9f78c03afae786f9c2a190134aa.jpg)

<details>
<summary>text_image</summary>

Text2Light
PanFusion
Pano-SD (Ours)
"A living room with hardwood floors, a fireplace, and large windows."
"A cozy study with built-in bookshelves and a leather armchair."
"A bedroom with a ceiling fan, gray walls, hardwood floors, a bed, and a TV on the wall."
"A cozy built-in daybed with throw pillows and blankets."
"A spacious corridor with hardwood floors."
</details>

Figure 4: Text to Panorama Comparison between TextLight, PanFusion, and Ours.

Table 1: Quantitative Panorama Comparisons with Baseline Methods 

<table><tr><td>Method</td><td>FID↓</td><td>IS↑</td><td>CS↑</td><td>Inference time(s)↓</td></tr><tr><td>Text2Light</td><td>76.50</td><td>3.60</td><td>27.48</td><td>50.4</td></tr><tr><td>PanFusion</td><td>47.62</td><td>4.49</td><td>28.73</td><td>27.6</td></tr><tr><td>Pano-SD(Ours)</td><td>48.52</td><td>3.30</td><td>28.98</td><td>5.1</td></tr></table>

Quantitative Results Table 1 presents the quantitative results. The FID, IS, and CS values of Text2Light are from the PanFusion [72]. We trained our proposed Pano-SD using the same dataset as PanFusion [72] and re-evaluated the performance of PanFusion. From the results, it can be observed that our method slightly outperforms others in terms of CS value and achieves a significantly lower FID value compared to Text2Light, which is close to PanFusion [72]. Moreover, due to the simplicity and efficiency of our model architecture, our method has a substantial advantage in inference time, which can significantly improve the efficiency of subsequent multi-view generation tasks.

Qualitative Results The qualitative comparison results are shown in Figure 4. We compare our method with the two models trained on their respective datasets. The panoramas generated by Text2Light exhibit poor left-right consistency. On the other hand, the panoramas generated by PanFusion suffer from blurriness at the bottom and top regions, which affects the overall integrity of the panoramas. In contrast, our model is capable of generating panoramas with clear bottom and top regions and better left-right continuity. However, due to the quality of the dataset, the image quality may be slightly inferior.

# 5.2 Multi-View Panorama Generation

To the best of our knowledge, there is no method for multi-view panorama generation, the existing SOTA method MVDream for perspective images cannot be directly applied to multi-view panorama generation. To verify the validity of our proposed spherical epipolar-aware attention module, we adapted MVDream to the panorama generation task as a comparative baseline.

Specifically, we remove the spherical epipolar-aware attention module from our method and load the pre-trained LoRA layers from Pano-SD. We then convert the 3D self-attention in the UNet to the form used in MVDream [43]. Additionally, we transform the camera pose matrix into camera embeddings through a 2-layer MLP and add it as a residual to the time embeddings. The qualitative comparison of the experiment is shown in Fig 5. Under the same training iteration, DiffPano significantly outperforms MVDream [43] in terms of multi-view consistency. Our method can maintain consistency in the details of multi-view images, while MVDream can only achieve a certain level of similarity in the overall images. Even compared to MVDream with twice the number of training iterations, our method still performs better in terms of consistency.

Table 2: User Study of Text to Multi-view Panoramas 

<table><tr><td>Method</td><td>Image quality↑</td><td>Image-text consistency↑</td><td>Multi-view consistency↑</td></tr><tr><td>MVDream</td><td>3.6</td><td>3.7</td><td>3.8</td></tr><tr><td>PanFusion</td><td>3.8</td><td>4.0</td><td>3.0</td></tr><tr><td>DiffPano(Ours)</td><td>4.0</td><td>4.2</td><td>4.3</td></tr></table>

Table 3: Ablation Study on the Number of Sampling Points and Reference Frames 

<table><tr><td></td><td>FID↓</td><td>IS↑</td><td>CS↑</td><td>PSNR↑</td><td>SSIM↑</td><td>Inference time(s)↓</td></tr><tr><td>K=1, S=10</td><td>73.30</td><td>3.40</td><td>22.14</td><td>29.99</td><td>0.69</td><td>30.06</td></tr><tr><td>K=2, S=10</td><td>66.02</td><td>3.34</td><td>22.92</td><td>32.04</td><td>0.79</td><td>33.12</td></tr><tr><td>K=3, S=10</td><td>69.89</td><td>3.58</td><td>22.76</td><td>32.29</td><td>0.81</td><td>35.79</td></tr><tr><td>K=4, S=6</td><td>68.39</td><td>3.57</td><td>22.74</td><td>33.32</td><td>0.86</td><td>26.61</td></tr><tr><td>K=4, S=8</td><td>67.30</td><td>3.54</td><td>22.66</td><td>33.00</td><td>0.84</td><td>32.23</td></tr><tr><td>K=4, S=10</td><td>65.98</td><td>3.37</td><td>22.59</td><td>33.39</td><td>0.87</td><td>37.91</td></tr><tr><td>K=4, S=12</td><td>62.79</td><td>3.26</td><td>22.65</td><td>32.89</td><td>0.83</td><td>43.72</td></tr></table>

User Study We collected 20 text prompts and recruited nearly 50 volunteers to evaluate text-tomulti-view panoramic image generation. Evaluation metrics of multi-view ERP panorama generation include the quality of multi-view panoramic images, multi-view consistency, and the consistency between the text and multi-view panoramas. Experimental results on Tab. 2 show that our method can generate multi-view panoramic images with better quality, higher text and image similarity, and more consistent multi-view images, compared with MVDream [43] and PanFusion [72]. See more qualitative results in Fig. 6 to Fig. 10.

# 5.3 Ablation Study

![](images/1d7071c693d27635875ce2fda161ca094e9ab5f29b3d74448da32cb194594e49.jpg)  
Figure 5: Comparisons with MVDream. DiffPano can generate more consistent multi-view panoramas. "MVDream×2" denotes MVDream is trained with twice iteration number relative to our method.

Table 4: Ablation Study of One-Stage vs Two-Stage Training 

<table><tr><td></td><td>FID↓</td><td>IS↑</td><td>LPIPS↓</td><td>PSNR↑</td><td>SSIM↑</td></tr><tr><td>One-stage</td><td>82.92</td><td>4.52</td><td>0.0454</td><td>31.64</td><td>0.74</td></tr><tr><td>Two-stage</td><td>74.08</td><td>3.13</td><td>0.0610</td><td>31.54</td><td>0.73</td></tr></table>

Number of Reference Views and Sample Points To explore the influence of varying numbers of reference frames and sampling points on model performance, we generate 5-frame multi-view panoramas simultaneously, and compute the FID, IS, and CS metrics for the first frame of each group to assess the quality of the generated panorama under different quantities of reference frames and sampling points. Furthermore, we set the same camera pose for the first and last frames of each generated panorama group, and calculate the PSNR and SSIM values between these two frames to evaluate the model’s multi-view consistency. As shown in Table. 3, increasing the number of reference frames and sampling points can improve the quality of generated panoramas to a certain extent, but the changes in image diversity and consistency with text remain marginal. With an increasing number of reference frames and sampling points, the model’s consistency exhibits improvement, however, when the number of sampling points becomes excessive (S =12), the multi-view consistency of the model diminishes.

One-Stage vs Two-Stage We conduct ablation experiments on one-stage and two-stage training. Table. 4 shows that the two-stage method will obtain the higher FID values. The IS of the two-stage training method is lower, and the diversity is reduced to a certain extent, which is slightly worse than the one-stage training. However, it should be noted that the images after one-stage training will have ghosting, but the two-stage will not.

# 6 Conclusion

We have proposed the panoramic video-text dataset and panorama generation framework with spherical epipolar-aware attention for text-to-single-view or multi-view panorama generation. Extensive experiments demonstrate that our method can achieve scalable, consistent, and diverse multi-view panoramas.

Limitation and Future Work Although our method demonstrates the ability to generate consistent multi-view panoramas under the same setting as the training phase, it is important to note that as the number of frames increases during inference, the model tends to hallucinate content.

Exploring the use of video diffusion models to improve the consistency of generated multi-view panoramas is a promising direction. Longer panoramic videos are expected to be realized based on the generated panoramas as conditions.

# 7 Acknowledgements

This work was partially supported by the National Key Research and Development Program of China (No. 2023YFF0905104) and National Natural Science Foundation of China (Nos. 61932003, 62076184 and 62473286).

# References

[1] Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint arXiv:1702.01105, 2017.   
[2] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. https://openai.com/research/video-generation-modelsas-world-simulators, 2024.   
[3] Weiwei Cai, Weicai Ye, Peng Ye, Tong He, and Tao Chen. Dynasurfgs: Dynamic surface reconstruction with planar-based gaussian splatting. arXiv preprint arXiv:2408.13972, 2024.

[4] Angel X. Chang, Angela Dai, Thomas A. Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from RGB-D data in indoor environments. In 2017 International Conference on 3D Vision, 3DV 2017, Qingdao, China, October 10-12, 2017, pages 667–676. IEEE Computer Society, 2017.   
[5] Danpeng Chen, Hai Li, Weicai Ye, Yifan Wang, Weijian Xie, Shangjin Zhai, Nan Wang, Haomin Liu, Hujun Bao, and Guofeng Zhang. PGSR: Planar-based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction. arXiv preprint arXiv:2406.06521, 2024.   
[6] Junyi Chen, Di Huang, Weicai Ye, Wanli Ouyang, and Tong He. Where am i and what will i see: An auto-regressive model for spatial localization and view prediction. arXiv preprint arXiv:2410.18962, 2024.   
[7] Junyi Chen, Weicai Ye, Yifan Wang, Danpeng Chen, Di Huang, Wanli Ouyang, Guofeng Zhang, Yu Qiao, and Tong He. Gigags: Scaling up planar-based 3d gaussians for large scene surface reconstruction. arXiv preprint arXiv:2409.06685, 2024.   
[8] Yiwen Chen, Tong He, Di Huang, Weicai Ye, Sijin Chen, Jiaxiang Tang, Xin Chen, Zhongang Cai, Lei Yang, Gang Yu, et al. Meshanything: Artist-created mesh generation with autoregressive transformers. arXiv preprint arXiv:2406.10163, 2024.   
[9] Zhaoxi Chen, Guangcong Wang, and Ziwei Liu. Text2light: Zero-shot text-driven HDR panorama generation. ACM Trans. Graph., 41(6):195:1–195:16, 2022.   
[10] Zheng Chen, Yan-Pei Cao, Yuan-Chen Guo, Chen Wang, Ying Shan, and Song-Hai Zhang. Panogrf: Generalizable spherical radiance fields for wide-baseline panoramas. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.   
[11] Zheng Chen, Chen Wang, Yuan-Chen Guo, and Song-Hai Zhang. Structnerf: Neural radiance fields for indoor scenes with structural hints. IEEE Trans. Pattern Anal. Mach. Intell., 45(12):15694–15705, 2023.   
[12] Xiao Cui, Weicai Ye, Yifan Wang, Guofeng Zhang, Wengang Zhou, Tong He, and Houqiang Li. Streetsurfgs: Scalable urban street surface reconstruction with planar-based gaussian splatting. arXiv preprint arXiv:2410.04354, 2024.   
[13] Mengyang Feng, Jinlin Liu, Miaomiao Cui, and Xuansong Xie. Diffusion360: Seamless 360 degree panoramic image generation based on diffusion models. CoRR, abs/2311.13141, 2023.   
[14] Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-t2x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.05945, 2024.   
[15] Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul Srinivasan, Jonathan T Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. arXiv preprint arXiv:2405.10314, 2024.   
[16] Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde. Learning to predict indoor illumination from a single image. arXiv preprint arXiv:1704.00090, 2017.   
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.   
[18] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.   
[19] Chenxi Huang, Yuenan Hou, Weicai Ye, Di Huang, Xiaoshui Huang, Binbin Lin, Deng Cai, and Wanli Ouyang. Nerf-det++: Incorporating semantic cues and perspective-aware depth supervision for indoor multi-view 3d detection. arXiv preprint arXiv:2402.14464, 2024.   
[20] Zehuan Huang, Hao Wen, Junting Dong, Yaohui Wang, Yangguang Li, Xinyuan Chen, Yan-Pei Cao, Ding Liang, Yu Qiao, Bo Dai, and Lu Sheng. Epidiff: Enhancing multi-view synthesis via localized epipolar-constrained diffusion. CoRR, abs/2312.06725, 2023.

[21] Hai Li, Weicai Ye, Guofeng Zhang, Sanyuan Zhang, and Hujun Bao. Saliency guided subdivision for single-view mesh reconstruction. In 2020 International Conference on 3D Vision (3DV), pages 1098–1107. IEEE, 2020.   
[22] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping languageimage pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 19730–19742. PMLR, 2023.   
[23] Longwei Li, Huajian Huang, Sai-Kit Yeung, and Hui Cheng. Omnigs: Omnidirectional gaussian splatting for fast radiance field reconstruction using omnidirectional images. CoRR, abs/2404.03202, 2024.   
[24] Renjie Li, Panwang Pan, Bangbang Yang, Dejia Xu, Shijie Zhou, Xuanyang Zhang, Zeming Li, Achuta Kadambi, Zhangyang Wang, and Zhiwen Fan. 4k4dgen: Panoramic 4d generation at 4k resolution. arXiv preprint arXiv:2406.13527, 2024.   
[25] Zhen Li, Lingli Wang, Mofang Cheng, Cihui Pan, and Jiaqi Yang. Multi-view inverse rendering for large-scale real-world indoor scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 12499– 12509. IEEE, 2023.   
[26] Jiuming Liu, Guangming Wang, Weicai Ye, Chaokang Jiang, Jinru Han, Zhe Liu, Guofeng Zhang, Dalong Du, and Hesheng Wang. Difflow3d: Toward robust uncertainty-aware scene flow estimation with iterative diffusion-based refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15109–15119, 2024.   
[27] Jiuming Liu, Ruiji Yu, Yian Wang, Yu Zheng, Tianchen Deng, Weicai Ye, and Hesheng Wang. Point mamba: A novel point cloud backbone based on state space model with octree-based ordering strategy. arXiv preprint arXiv:2403.06467, 2024.   
[28] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10072–10083, 2024.   
[29] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object, 2023.   
[30] Xiangyu Liu, Weicai Ye, Chaoran Tian, Zhaopeng Cui, Hujun Bao, and Guofeng Zhang. Coxgraph: multi-robot collaborative, globally consistent, online dense reconstruction system. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8722–8728. IEEE, 2021.   
[31] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from a single-view image. CoRR, abs/2309.03453, 2023.   
[32] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, and Wenping Wang. Wonder3d: Single image to 3d using cross-domain diffusion. CoRR, abs/2310.15008, 2023.   
[33] Zhuqiang Lu, Kun Hu, Chaoyue Wang, Lei Bai, and Zhiyong Wang. Autoregressive omni-aware outpainting for open-vocabulary 360-degree image generation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 14211–14219, 2024.   
[34] Yuhang Ming, Minyang Xu, Xingrui Yang, Weicai Ye, Weihan Wang, Yong Peng, Weichen Dai, and Wanzeng Kong. Viper: Visual incremental place recognition with adaptive mining and lifelong learning. arXiv preprint arXiv:2407.21416, 2024.   
[35] Yuhang Ming, Weicai Ye, and Andrew Calway. idf-slam: End-to-end rgb-d slam with neural implicit mapping and deep feature tracking. arXiv preprint arXiv:2209.07919, 2022.   
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.

[37] Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.   
[38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In CVPR, pages 10674–10685. IEEE, 2022.   
[39] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.   
[40] Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, and Jiajun Wu. Zeronvs: Zero-shot 360-degree view synthesis from a single real image. CoRR, abs/2310.17994, 2023.   
[41] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.   
[42] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view diffusion base model. CoRR, abs/2310.15110, 2023.   
[43] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multiview diffusion for 3d generation. CoRR, abs/2308.16512, 2023.   
[44] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Frédo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In NeurIPS, pages 19313–19325, 2021.   
[45] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision, pages 1–18. Springer, 2025.   
[46] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653, 2023.   
[47] Shengji Tang, Weicai Ye, Peng Ye, Weihao Lin, Yang Zhou, Tao Chen, and Wanli Ouyang. Hisplat: Hierarchical 3d gaussian splatting for generalizable sparse-view reconstruction. arXiv preprint arXiv:2410.06245, 2024.   
[48] Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mvdiffusion: Enabling holistic multi-view image generation with correspondence-aware diffusion. In NeurIPS, 2023.   
[49] Ziyu Tang, Weicai Ye, Yifan Wang, Di Huang, Hujun Bao, Tong He, and Guofeng Zhang. Nd-sdf: Learning normal deflection fields for high-fidelity indoor reconstruction. arXiv preprint arXiv:2408.12598, 2024.   
[50] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023.

[51] Hung-Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia-Bin Huang, and Johannes Kopf. Consistent view synthesis with pose-guided diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 16773–16783. IEEE, 2023.   
[52] Chen Wang, Jiatao Gu, Xiaoxiao Long, Yuan Liu, and Lingjie Liu. GECO: generative image-to-3d within a second. CoRR, abs/2405.20327, 2024.   
[53] Guangcong Wang, Peng Wang, Zhaoxi Chen, Wenping Wang, Chen Change Loy, and Ziwei Liu. Perf: Panoramic neural radiance field from a single panorama. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2024.   
[54] Hai Wang, Xiaoyu Xiang, Yuchen Fan, and Jing-Hao Xue. Customizing 360-degree panoramas through text-to-image diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4933–4943, 2024.   
[55] Hai Wang, Xiaoyu Xiang, Yuchen Fan, and Jing-Hao Xue. Customizing 360-degree panoramas through text-to-image diffusion models. In WACV, pages 4921–4931. IEEE, 2024.   
[56] Jionghao Wang, Ziyu Chen, Jun Ling, Rong Xie, and Li Song. 360-degree panorama generation from few unregistered nfov images. arXiv preprint arXiv:2308.14686, 2023.   
[57] Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. CoRR, abs/2312.02201, 2023.   
[58] Yifan Wang, Di Huang, Weicai Ye, Guofeng Zhang, Wanli Ouyang, and Tong He. Neurodin: A two-stage framework for high-fidelity neural surface reconstruction. arXiv preprint arXiv:2408.10178, 2024.   
[59] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600– 612, 2004.   
[60] Tianhao Wu, Chuanxia Zheng, and Tat-Jen Cham. Ipo-ldm: Depth-aided 360-degree indoor rgb panorama outpainting via latent diffusion model. arXiv preprint arXiv:2307.03177, 2023.   
[61] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint arXiv:2403.14621, 2024.   
[62] Bangbang Yang, Wenqi Dong, Lin Ma, Wenbo Hu, Xiao Liu, Zhaopeng Cui, and Yuewen Ma. Dreamspace: Dreaming your room space with text-driven panoramic texture propagation. In 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), pages 650–660. IEEE, 2024.   
[63] Jiayu Yang, Ziang Cheng, Yunfei Duan, Pan Ji, and Hongdong Li. Consistnet: Enforcing 3d consistency for multi-view images diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7079–7088, 2024.   
[64] Weicai Ye, Shuo Chen, Chong Bao, Hujun Bao, Marc Pollefeys, Zhaopeng Cui, and Guofeng Zhang. IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.   
[65] Weicai Ye, Xinyu Chen, Ruohao Zhan, Di Huang, Xiaoshui Huang, Haoyi Zhu, Hujun Bao, Wanli Ouyang, Tong He, and Guofeng Zhang. Dynamic-Aware Tracking Any Point for Structure from Motion in the Wild. arXiv preprint, 2024.   
[66] Weicai Ye, Xinyue Lan, Shuo Chen, Yuhang Ming, Xingyuan Yu, Hujun Bao, Zhaopeng Cui, and Guofeng Zhang. Pvo: Panoptic visual odometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9579–9589, June 2023.   
[67] Weicai Ye, Xinyue Lan, Ge Su, Hujun Bao, Zhaopeng Cui, and Guofeng Zhang. Hybrid tracker with pixel and instance for video panoptic segmentation. arXiv preprint arXiv:2203.01217, 2022.   
[68] Weicai Ye, Hai Li, Tianxiang Zhang, Xiaowei Zhou, Hujun Bao, and Guofeng Zhang. Superplane: 3d plane detection and description from a single image. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pages 207–215. IEEE, 2021.

[69] Weicai Ye, Hao Li, Yuanyuan Gao, Yalun Dai, Junyi Chen, Dingwen Zhang, Tong He, Guofeng Zhang, and Junwei Han. Fedsurfgs: Scalable 3d surface gaussian splatting with federated learning for large scene reconstruction. arxiv preprint, 2024.   
[70] Hailin Yu, Youji Feng, Weicai Ye, Mingxuan Jiang, Hujun Bao, and Guofeng Zhang. Improving feature-based visual localization by geometry-aided matching. arXiv preprint arXiv:2211.08712, 2022.   
[71] Hailin Yu, Weicai Ye, Youji Feng, Hujun Bao, and Guofeng Zhang. Learning bipartite graph matching for robust visual localization. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 146–155. IEEE, 2020.   
[72] Cheng Zhang, Qianyi Wu, Camilo Cruz Gambardella, Xiaoshui Huang, Dinh Phung, Wanli Ouyang, and Jianfei Cai. Taming stable diffusion for text to 360◦ panorama image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.   
[73] Jinsong Zhang and Jean-François Lalonde. Learning high dynamic range from outdoor panoramas. In Proceedings of the IEEE International Conference on Computer Vision, pages 4519– 4528, 2017.   
[74] Tianxiang Zhang, Chong Bao, Hongjia Zhai, Jiazhen Xia, Weicai Ye, and Guofeng Zhang. Arcargo: Multi-device integrated cargo loading management system with augmented reality. In 2021 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), pages 341–348. IEEE, 2021.   
[75] Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX, volume 12354 of Lecture Notes in Computer Science, pages 519–535. Springer, 2020.   
[76] Haiyang Zhou, Xinhua Cheng, Wangbo Yu, Yonghong Tian, and Li Yuan. Holodreamer: Holistic 3d panoramic world generation from text descriptions. arXiv preprint arXiv:2407.15187, 2024.   
[77] Haoyi Zhu, Yating Wang, Di Huang, Weicai Ye, Wanli Ouyang, and Tong He. Point cloud matters: Rethinking the impact of different observation spaces on robot learning. arXiv preprint arXiv:2402.02500, 2024.

# A Spherical Epipolar Line Computation

Given the pixel coordinates of a point in the panorama of the target view i, the corresponding epipolar line in the panorama of the source view j can be calculated. First, according to Eq. ( 1), the camera coordinates p in the target view i are computed. Based on the relative pose $\{ { \bf \tilde { R } } ^ { i \to j } , { \bf T } ^ { i \to j } \} , { \bf p } ^ { \prime }$ in the source view j are calculated:

$$
\mathbf {p} ^ {\prime} = \mathbf {R} ^ {i \rightarrow j} \mathbf {p} + \mathbf {T} ^ {i \rightarrow j}. \tag {8}
$$

Simultaneously, the coordinates $\mathbf { o } ^ { \prime }$ of the camera origin ${ \mathbf o } = ( 0 , 0 , 0 ) ^ { T }$ in the target view i projected onto the source view j are computed:

$$
\mathbf {o} ^ {\prime} = \mathbf {R} ^ {i \rightarrow j} \mathbf {o} + \mathbf {T} ^ {i \rightarrow j}. \tag {9}
$$

Then, we need to find the plane L containing the three points $\mathbf { p ^ { \prime } }$ , o, and $\mathbf { o } ^ { \prime }$ in the camera coordinate system of the source view $j .$ The intersection of this plane with the spherical surface is the desired epipolar line. The equation corresponding to plane L is:

$$
A X + B Y + C Z + D = 0. \tag {10}
$$

Substituting the coordinates of the three points into the equation yields the coefficients A, B, C, and $D \colon$

$$
A = \frac {z _ {\mathbf {o} ^ {\prime}} \cdot y _ {\mathbf {p} ^ {\prime}} - z _ {\mathbf {p} ^ {\prime}} \cdot y _ {\mathbf {o} ^ {\prime}}}{x _ {\mathbf {p} ^ {\prime}} \cdot y _ {\mathbf {o} ^ {\prime}} - x _ {\mathbf {o} ^ {\prime}} \cdot y _ {\mathbf {p} ^ {\prime}}} \cdot C
$$

$$
B = \frac {z _ {\mathbf {o} ^ {\prime}} \cdot x _ {\mathbf {p} ^ {\prime}} - z _ {\mathbf {p} ^ {\prime}} \cdot x _ {\mathbf {o} ^ {\prime}}}{y _ {\mathbf {p} ^ {\prime}} \cdot x _ {\mathbf {o} ^ {\prime}} - y _ {\mathbf {o} ^ {\prime}} \cdot x _ {\mathbf {p} ^ {\prime}}} \cdot C \tag {11}
$$

$$
D = 0.
$$

To simplify the representation of the equation for plane $L ,$ we introduce new coefficients $a _ { 1 }$ and $a _ { 2 } \colon$

$$
a _ {1} = \frac {z _ {\mathbf {o} ^ {\prime}} \cdot y _ {\mathbf {p} ^ {\prime}} - z _ {\mathbf {p} ^ {\prime}} \cdot y _ {\mathbf {o} ^ {\prime}}}{x _ {\mathbf {p} ^ {\prime}} \cdot y _ {\mathbf {o} ^ {\prime}} - x _ {\mathbf {o} ^ {\prime}} \cdot y _ {\mathbf {p} ^ {\prime}}} \tag {12}
$$

$$
a _ {2} = \frac {z _ {\mathbf {o} ^ {\prime}} \cdot x _ {\mathbf {p} ^ {\prime}} - z _ {\mathbf {p} ^ {\prime}} \cdot x _ {\mathbf {o} ^ {\prime}}}{y _ {\mathbf {p} ^ {\prime}} \cdot x _ {\mathbf {o} ^ {\prime}} - y _ {\mathbf {o} ^ {\prime}} \cdot x _ {\mathbf {p} ^ {\prime}}},
$$

the equation of plane L can be simplified to:

$$
a _ {1} X + a _ {2} Y + Z = 0. \tag {13}
$$

According to Eq.1 and Eq.2, the epipolar line equation in the source view $j$ can be obtained:

$$
y _ {p i x} = H \cdot \left[ \arctan \left(\frac {a _ {1} \sin \left(\frac {2 \pi x _ {p i x}}{W}\right) - \cos \left(\frac {2 \pi x _ {p i x}}{W}\right)}{a _ {2}}\right) / \pi + 0. 5 \right], \tag {14}
$$

where $x _ { p i x }$ and $y _ { p i x }$ are the corresponding pixel coordinates.

# B Experiment Details

Dataset Processing In single-view panorama generation, we select 100 scenes from HM3D [37] and render cube maps from random viewpoints within each scene, which are then stitched into panoramas through interpolation. However, due to the imperfect quality of the corresponding scene meshes, which may have missing parts, we filter out images with a high proportion of zero-depth values based on their corresponding depth maps, ultimately creating a dataset of 8,508 panoramas.

Table 5: Quantitative Perspective Images Comparisons with Baseline Methods 

<table><tr><td>Method</td><td>FID↓</td><td>IS↑</td></tr><tr><td>MVDiffusion</td><td>188.16</td><td>1.82</td></tr><tr><td>PanFusion</td><td>213.19</td><td>2.61</td></tr><tr><td>Pano-SD(Ours)</td><td>164.75</td><td>1.96</td></tr></table>

In the multi-view panorama generation task, we render images based on camera trajectories across scenes. To accommodate the first stage of training, we select camera trajectories with minimal changes. In contrast, for the second stage, the dataset comprises panoramas with more significant camera displacements between consecutive frames. Each training dataset consists of multiple panoramas, their corresponding pose matrices, and text descriptions. To construct a multi-view panorama dataset with nearly identical image content, we select the first few frames from each camera trajectory, as the camera movement between these frames is minimal and they are captured within the same scene. For datasets where the image content changes, the construction needs to be performed across the entire camera trajectory. By projecting the panorama from the subsequent frame onto the viewpoint of the previous frame using spherical projection, we compare the pixel value differences between the two. If more than 40% of the pixel values differ, we consider new content to have been generated between the two frames. After filtering, there are 19,739 data sets with nearly identical image content and 18,704 data sets with differing image content.

# Compare with Baseline Methods

• Text2Light [9]: In the quantitative analysis, the FID, IS, and CS values are referenced from the results reported in PanFusion[72], which were obtained by testing on the same dataset after training. For the qualitative analysis, we directly employ their jointly trained model on both indoor and outdoor datasets to generate LDR (low dynamic range) panoramas based on text descriptions.   
• PanFusion [72]: To compare the quality of the generated panoramas, we also train the model using panoramas stitched from MP3D skybox images. We employ the same dataset split and text descriptions as in [72], which includes 9,820 panoramas for training and 1,092 for evaluation. In the qualitative analysis, we use our own dataset constructed from HM3D[37]. The panoramas in our dataset do not suffer from blurriness at the bottom and top regions, but the overall image quality may be slightly lower.

Compare with MVDream We adopt the method from MVDream[43] by formatting the $B \times$ $N \times \mathbf { \bar {  { H } } } \times W \times C$ tensor input to the self-attention module as $B \times \bar { N } \bar { H W } \times C$ . This allows the model to simultaneously integrate features from all viewpoints during the self-attention computation, thereby improving the consistency among the generated multi-view panoramas. However, since MVDream is designed for multi-view image generation of objects, the camera pose matrices used in their method only contain rotation information without translation. In contrast, when generating camera embeddings, we use pose matrices that include both rotation and translation information, which increases the learning difficulty for the model. This may be one of the reasons why using this method does not yield particularly ideal results.

Transform Panoramas to Perspective Views We converted the generated panoramas into perspective images and conducted quantitative comparisons, shown in Tab. 5. Experiments show that our method achieves the lowest FID, while our method is higher than MVDiffusion in IS and slightly lower than PanFusion. It should be noted that MVDiffusion directly generates perspective images and then stitches them into panoramas. It is not in the ERP format and does not have the top and bottom parts. Our panorama generation speed is faster.

# B.1 Applications

# B.1.1 Text to Single-View Panorama

# B.1.2 Text to Multi-View Panoramas

DiffPano can generate panoramic video frames with large camera pose spans and multi-view consistency based on diverse textual descriptions, thus achieving the effect of text-to-panoramic video, as shown in the Fig.11.

![](images/e3ff9240ebd278eef871d0daf9ecf8bd72ddb5dbec61481036bbcb0fe22d7f31.jpg)

Figure 6: Qualitative Comparisons of Text to Panoramic Videos. Ours vs MVDream vs PanFusion.   
![](images/e703a33ade21950fd427705a40a3794834768cc8df8d5b48e08b40953094e222.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior photos of three modern living spaces—PanFusion, MVDream, and DiffPano (Ours)—showing various furniture and objects in various interior layouts.
</details>

Figure 7: Qualitative Comparisons of Text to Panoramic Videos. Ours vs MVDream vs PanFusion.

# C Network Architecture and Training Details

Network Architecture Our LoRA-based single-view panorama generation model produces panoramas with a resolution of $5 1 2 \times 1 0 2 4$ . In the multi-view panorama generation approach, we generate continuous frames of panoramas with a resolution of $2 5 6 \times 5 1 2 .$ Generating multiple frames of 512×1024 resolution panoramas simultaneously would consume a significant amount of computational resources. Moreover, our experiments reveal that generating multi-frame high-resolution panoramas requires an exceptionally long training time to improve the quality of the generated images. The network architecture of our multi-view panorama generation model is shown in Table 6 and Table7.

Training Details We fine-tuned the Stable Diffusion v1.5 model using the LoRA method for single-view pano-based synthesis. The training was conducted on 6 A100 GPUs with 80GB memory for 100 epochs (approximately 6.5 hours), with a learning rate of 1e-4 and a batch size of 6. For

Table 6: Network architecture of DiffPano-1 

<table><tr><td></td><td>Layer</td><td>Output</td><td>Additional Inputs</td></tr><tr><td>(1)</td><td>Latent Map</td><td> $4 \times 32 \times 64$ </td><td></td></tr><tr><td>(2)</td><td>Conv.</td><td> $320 \times 32 \times 64$ </td><td></td></tr><tr><td colspan="4">CrossAttnDownBlock1</td></tr><tr><td>(3)</td><td>ResBlock</td><td> $320 \times 32 \times 64$ </td><td>Time emb.</td></tr><tr><td>(4)</td><td>AttnBlock</td><td> $320 \times 32 \times 64$ </td><td>Prompt emb.</td></tr><tr><td>(5)</td><td>ResBlock</td><td> $320 \times 32 \times 64$ </td><td>Time emb.</td></tr><tr><td>(6)</td><td>AttnBlock</td><td> $320 \times 32 \times 64$ </td><td>Prompt emb.</td></tr><tr><td>(7)</td><td>DownSampler</td><td> $320 \times 16 \times 32$ </td><td></td></tr><tr><td colspan="4">CrossAttnDownBlock2</td></tr><tr><td>(8)</td><td>ResBlock</td><td> $640 \times 16 \times 32$ </td><td>Time emb.</td></tr><tr><td>(9)</td><td>AttnBlock</td><td> $640 \times 16 \times 32$ </td><td>Prompt emb.</td></tr><tr><td>(10)</td><td>ResBlock</td><td> $640 \times 16 \times 32$ </td><td>Time emb.</td></tr><tr><td>(11)</td><td>AttnBlock</td><td> $640 \times 16 \times 32$ </td><td>Prompt emb.</td></tr><tr><td>(12)</td><td>DownSampler</td><td> $640 \times 8 \times 16$ </td><td></td></tr><tr><td colspan="4">CrossAttnDownBlock3</td></tr><tr><td>(13)</td><td>ResBlock</td><td> $1280 \times 8 \times 16$ </td><td>Time emb.</td></tr><tr><td>(14)</td><td>AttnBlock</td><td> $1280 \times 8 \times 16$ </td><td>Prompt emb.</td></tr><tr><td>(15)</td><td>ResBlock</td><td> $1280 \times 8 \times 16$ </td><td>Time emb.</td></tr><tr><td>(16)</td><td>AttnBlock</td><td> $1280 \times 8 \times 16$ </td><td>Prompt emb.</td></tr><tr><td>(17)</td><td>DownSampler</td><td> $1280 \times 4 \times 8$ </td><td></td></tr><tr><td colspan="4">DownBlock</td></tr><tr><td>(18)</td><td>ResBlock</td><td> $1280 \times 4 \times 8$ </td><td>Time emb.</td></tr><tr><td>(19)</td><td>ResBlock</td><td> $1280 \times 4 \times 8$ </td><td>Time emb.</td></tr><tr><td colspan="4">MidBlock</td></tr><tr><td>(20)</td><td>ResBlock</td><td> $1280 \times 4 \times 8$ </td><td>Time emb.</td></tr><tr><td>(21)</td><td>EAModule</td><td> $1280 \times 4 \times 8$ </td><td></td></tr><tr><td>(22)</td><td>AttnBlock</td><td> $1280 \times 4 \times 8$ </td><td>Prompt emb.</td></tr><tr><td>(23)</td><td>ResBlock</td><td> $1280 \times 4 \times 8$ </td><td>Time emb.</td></tr><tr><td colspan="4">UpBlock</td></tr><tr><td>(24)</td><td>ResBlock</td><td> $1280 \times 4 \times 8$ </td><td>(19), Time emb.</td></tr><tr><td>(25)</td><td>ResBlock</td><td> $1280 \times 4 \times 8$ </td><td>(18), Time emb.</td></tr><tr><td>(26)</td><td>ResBlock</td><td> $1280 \times 4 \times 8$ </td><td>(17), Time emb.</td></tr><tr><td>(27)</td><td>EAModule</td><td> $1280 \times 4 \times 8$ </td><td></td></tr><tr><td>(28)</td><td>UpSampler</td><td> $1280 \times 8 \times 16$ </td><td></td></tr><tr><td colspan="4">CrossAttnUpBlock1</td></tr><tr><td>(29)</td><td>ResBlock</td><td> $1280 \times 8 \times 16$ </td><td>(16), Time emb.</td></tr><tr><td>(30)</td><td>AttnBlock</td><td> $1280 \times 8 \times 16$ </td><td>Prompt emb.</td></tr><tr><td>(31)</td><td>ResBlock</td><td> $1280 \times 8 \times 16$ </td><td>(14), Time emb.</td></tr><tr><td>(32)</td><td>AttnBlock</td><td> $1280 \times 8 \times 16$ </td><td>Prompt emb.</td></tr><tr><td>(33)</td><td>ResBlock</td><td> $1280 \times 8 \times 16$ </td><td>(12), Time emb.</td></tr><tr><td>(34)</td><td>AttnBlock</td><td> $1280 \times 8 \times 16$ </td><td>Prompt emb.</td></tr><tr><td>(35)</td><td>EAModule</td><td> $1280 \times 8 \times 16$ </td><td></td></tr><tr><td>(36)</td><td>UpSampler</td><td> $1280 \times 16 \times 32$ </td><td></td></tr></table>

![](images/7f2cf4c35d7fda6786379878565b1a6c3e4b554ebe8bdfac73e98ab1b925a9af.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior views of a modern architectural space with labeled gardens and plants, including PanFusion, MVDream, and DiffPano (Ours), all without any text overlays.
</details>

Figure 8: Qualitative Comparisons of Text to Panoramic Videos. Ours vs MVDream vs PanFusion.

![](images/03a95b01f2e8ca483a9e789cb52c70a1ad57ade9e157ef8040a58b57bc7b7fe9.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior photos of a modern bathroom with cubby, furniture, and bench designs, showing interior layouts and lighting (no text or symbols in the interior scenes)
</details>

Figure 9: Qualitative Comparisons of Text to Panoramic Videos. Ours vs MVDream vs PanFusion.

multi-view panoramas generation, we conducted training on 8 80G A100 GPUs with a batch size of 1 and a learning rate of 1e-5. Each GPU utilized approximately 50% of its memory. The two-stage training process involved 100 epochs for each stage, with a total training time of approximately 5 days.

# D Societal Impact

Since our method can achieve scalable, consistent, and diverse multi-view panoramas, it has many potential applications, such as unlimited room roaming in VR, interior design preview, embodied intelligent robot exploration, etc.

![](images/274bd433343c26feb97972cad0eb581bfeabfb682bbc3f70e76c7f26e6ff97fb.jpg)

Figure 10: Qualitative Comparisons of Text to Panoramic Videos. Ours vs MVDream vs PanFusion.   
![](images/9bc2162f9cde6e6a20875a7f85b0b5dc5d16462153419c758deeccdf1b59e7e4.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a modern, curved wooden corridor with arched windows and staircases (no visible text or symbols)
</details>

“A spacious house with wooden paneling features a stairway, and multiple doors leading to various rooms. A cozy wooden house with a stairway leading up to the second floor, surrounded by wood floors and adorned with a picture on the wall.”

![](images/c72072cbff7c7b9e46350765e009c73e4c702ac5d4851174fb169df8f13f2ce9.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior views of a modern kitchen and dining area with wooden flooring (no visible text or symbols)
</details>

“A kitchen with white ceiling, hardwood floors, black cabinets and stainless steel appliances. A kitchen with black cabinets, hardwood floors, and a stove, connected to a another room with wood floors.”

![](images/86218953378a2138510613c8c044fb76e91e4509390621f9e193ccdbaf6f1276.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern living room with black sofas, armchairs, and large windows (no visible text or symbols)
</details>

“A spacious living room with a brown couch, a TV, and a desk. A wellfurnished room with a painting, and an open door, featuring a couch, desk, and TV on the wall, with a carpet.”

![](images/614408e975bbec4512dfa7257138bf484249611a9b673a699a878ddd6fea4fc9.jpg)

<details>
<summary>natural_image</summary>

Four-panel interior view of a modern bedroom with wooden flooring, featuring beds, a lamp, and a door (no visible text or symbols)
</details>

“A bedroom with hardwood floors, a ceiling fan, a bed, and a TV on the wall, featuring white wall and brown floor. A two-story house with a small stairway, hardwood floors, and an open doorway leading to a bedroom.”   
Figure 11: Qualitative Results of Text to Panoramic Videos. DiffPano can generate scalable and consistent panorama videos.

![](images/5bae2b4046db1c45401552e81e27233f4af60e67f67d48fb194ccc9e761da211.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a modern, curved retail store with product displays and ambient lighting (no visible text or signage)
</details>

“A remodeled garage with a door open, displaying a cluttered space filled with various items on the walls. A well-lit hallway with a door and various furniture. ”

![](images/4fb703f2aa5b4b63ddd100c787e21f736c9436fa171d56a6a77fe129ef4e0849.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a modern living room with curved white furniture and red accent wall art (no visible text or symbols)
</details>

“A living room with white walls and a white door. The room is furnished with red leather couches and a TV mounted on the wall. A spacious room with hardwood floors, a kitche and dining area, and an old-fashioned touch.”

![](images/6097e9a1e488b0281e6223cfa7777146b5b499fb07762b356f7ffcaa57ad0414.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a modern, curved wooden room with large windows and patterned flooring (no visible text or symbols)
</details>

“A cozy room with wooden floors and a sliding glass door. The room is well-lit with a sink and wooden furniture. A cozy n bedroom with wooden floors anda large bed. The room is furnished with a rug.”

![](images/574786a3bb7b7fe435637585b172001269bbc74be55c53eff5314e56d964317e.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a modern office or service center with curved ceiling design, featuring orange furniture and wall art (no visible text or symbols)
</details>

“A living area with a high ceiling, a light fixture, and a kitchenette. The room is furnished with a dining table and chairs and a painting on the wall, featuring a living room and dining room combination, and a white couch.”   
Figure 12: Qualitative Results of Text to Panoramic Videos. DiffPano can generate scalable and consistent panorama videos.

![](images/88f28a504c341a42166ca607262ffa17e711a915d5bee707f041da06ac847c1a.jpg)

<details>
<summary>natural_image</summary>

Five-panel interior view of a modern office space with curved wooden flooring, featuring minimalist furniture and windows (no visible text or symbols)
</details>

“A home office with a large desk, bookshelves, hardwood floors and a comfortable chair. A hallway with hardwood floors, leading to a home office with a large desk and bookshelves.”

![](images/5bdaed7d36d8a23365ec2160890fe060122cdbac2365493446cfc09c094ef192.jpg)

<details>
<summary>natural_image</summary>

Four-panel interior view of a bedroom with large windows, beds, and lighting fixtures (no visible text or symbols)
</details>

“A bedroom with a bed, and a dresser, featuring a window with curtains and blinds and a TV on top of the dresser.A cozy bedroom with a bed, dresser, and TV, surrounded by a brown carpet.”

![](images/7f3b2288739b660d1a28aac4f49dbd3427ff29b076e2b5ef22d5989c309a33ed.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a modern living room with purple sofas, wooden flooring, and white railings (no visible text or symbols)
</details>

“A cozy living room with a purple couch, a fireplace, and a rug. The ceiling has a light on it. A living room with a white staircase, wooden floors, a couch, and an open door that leads to another room.”

![](images/8e2c8768521cae32873cd7aa766f88577bd876f0fd31a7e9adc32c099293b4c9.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern bedroom with curved wooden doors and a bed, showing interior lighting and furniture (no text or symbols visible)
</details>

“A small room with a wooden door and a bed, featuring a white wall with a picture. A spacious room with a carpeted floor, a hallway leading to a bedroom with wooden doors and a comfortable bed.”   
Figure 13: Qualitative Results of Text to Panoramic Videos. DiffPano can generate scalable and consistent panorama videos.

Table 7: Network architecture of DiffPano-2 

<table><tr><td></td><td>Layer</td><td>Output</td><td>Additional Inputs</td></tr><tr><td colspan="4">CrossAttnUpBlock2</td></tr><tr><td>(37)</td><td>ResBlock</td><td>640 × 16 × 32</td><td>(11), Time emb.</td></tr><tr><td>(38)</td><td>AttnBlock</td><td>640 × 16 × 32</td><td>Prompt emb.</td></tr><tr><td>(39)</td><td>ResBlock</td><td>640 × 16 × 32</td><td>(9), Time emb.</td></tr><tr><td>(40)</td><td>AttnBlock</td><td>640 × 16 × 32</td><td>Prompt emb.</td></tr><tr><td>(41)</td><td>ResBlock</td><td>640 × 16 × 32</td><td>(7), Time emb.</td></tr><tr><td>(42)</td><td>AttnBlock</td><td>640 × 16 × 32</td><td>Prompt emb.</td></tr><tr><td>(43)</td><td>EAModule</td><td>640 × 16 × 32</td><td></td></tr><tr><td>(44)</td><td>UpSampler</td><td>640 × 32 × 64</td><td></td></tr><tr><td colspan="4">CrossAttnUpBlock3</td></tr><tr><td>(45)</td><td>ResBlock</td><td>320 × 32 × 64</td><td>(6), Time emb.</td></tr><tr><td>(46)</td><td>AttnBlock</td><td>320 × 32 × 64</td><td>Prompt emb.</td></tr><tr><td>(47)</td><td>ResBlock</td><td>320 × 32 × 64</td><td>(4), Time emb.</td></tr><tr><td>(48)</td><td>AttnBlock</td><td>320 × 32 × 64</td><td>Prompt emb.</td></tr><tr><td>(49)</td><td>ResBlock</td><td>320 × 32 × 64</td><td>(2), Time emb.</td></tr><tr><td>(50)</td><td>AttnBlock</td><td>320 × 32 × 64</td><td>Prompt emb.</td></tr><tr><td>(51)</td><td>EAModule</td><td>320 × 32 × 64</td><td></td></tr><tr><td>(52)</td><td>GroupNorm</td><td>320 × 32 × 64</td><td></td></tr><tr><td>(53)</td><td>SiLU</td><td>320 × 32 × 64</td><td></td></tr><tr><td>(54)</td><td>Conv.</td><td>4 × 32 × 64</td><td></td></tr></table>