# CUBEDIFF: REPURPOSING DIFFUSION-BASED IMAGE MODELS FOR PANORAMA GENERATION

Nikolai Kalischek ETH Zurich, Google ¨

Michael Oechsle Google

Fabian Manhardt Google

Philipp Henzler Google

Konrad Schindler ETH Zurich¨

Federico Tombari Google

# ABSTRACT

We introduce a novel method for generating 360° panoramas from text prompts or images. Our approach leverages recent advances in 3D generation by employing multi-view diffusion models to jointly synthesize the six faces of a cubemap. Unlike previous methods that rely on processing equirectangular projections or autoregressive generation, our method treats each face as a standard perspective image, simplifying the generation process and enabling the use of existing multi-view diffusion models. We demonstrate that these models can be adapted to produce high-quality cubemaps without requiring correspondence-aware attention layers. Our model allows for fine-grained text control, generates high resolution panorama images and generalizes well beyond its training set, whilst achieving state-of-the-art results, both qualitatively and quantitatively. Project page: https://cubediff.github.io/

![](images/368e580849ba860a22fd92806401a927d9a84458781ff81a0537d7293ab2930a.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph LR
    A["Unfold"] --> B["3D Shape with Light"]
    B --> C["stack"]
    C --> D["CubeDiff"]
    D --> E["unstack"]
    E --> F["3D Shape with Light"]
    F --> G["fold"]
```
</details>

![](images/cc4765d72e7225b54eade80763a7b4e2297773c7d9d26f37e364e6380c5b850a.jpg)

<details>
<summary>natural_image</summary>

Interior view of an old industrial warehouse with arched wooden walls and large windows (no visible text or symbols)
</details>

![](images/2dd1ecac8537e5bceb5b6288188de69a40b93de54a07a3ea5053caa22f11bfff.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a historic stone-walled room with arched ceilings, tables, and large windows (no visible text or symbols)
</details>

![](images/2b9aeffca80be2f5589ae57aa9b1a1a05fd3d3b0d22d3ee242d0dd6bb6bea08f.jpg)

<details>
<summary>natural_image</summary>

Interior view of a cozy room with wooden bookshelves, a central table, and a window (no visible text or symbols)
</details>

![](images/bdc286d78ee8e622eda7c7104c46a386947d41f6fbd08ec5910abf7bd3b3c558.jpg)

<details>
<summary>natural_image</summary>

Panoramic view of a rustic wooden house surrounded by dense greenery and trees (no visible text or symbols)
</details>

![](images/06b8a9f7f925a7a075d491629d14affa9867fc235d2d01d6f8b3f981cb92acf9.jpg)

<details>
<summary>natural_image</summary>

Panoramic view of hot air balloons floating in a desert canyon under a clear blue sky (no text or symbols visible)
</details>

![](images/dbc891b6d5905d881aa098990e8da508a48f7197abd17b1b72dcfbfceb7fe0fe.jpg)

<details>
<summary>natural_image</summary>

Desert landscape with a glowing globe and a tree on cracked earth, labeled 'Outdoors' in the top right corner (no other text or symbols)
</details>

Figure 1: CubeDiff leverages cubmaps to represent 360° panoramas and denoises all faces together in a single pass. In contrast to other works, Cubediff does not need to consider distortions, since it operatkes on common 90° FOV perspective images, maing it possible to directly utilize the internetscale image prior of the underlying diffusion model.

# 1 INTRODUCTION

Recent advances in diffusion-based generative models have seen tremendous progress over the last two years, enabling a wide range of applications from artistic expression and product design to personalized content creation. Beyond generating realistic and diverse images based on text-to-image models (Rombach et al., 2022; Saharia et al., 2022), these models are now capable of more complex tasks such as 3D asset creation (e.g., (Kalischek et al., 2022; Wang et al., 2024; Mohammad Khalid et al., 2022; Poole et al., 2022)), estimating scene properties such as depth or semantics (Ke et al., 2024; Baranchuk et al., 2021), illumination changes (Jin et al., 2024; Zhao et al., 2024; Zeng et al., 2024), and generation of multi-view consistent images (Gao et al., 2024b; Tang et al., 2023).

The latter is particularly interesting in virtual reality, gaming and entertainment, where 3D consistency is crucial for fully immersive experiences and thus user satisfaction. However, synthesizing high-quality, visually coherent panoramas presents unique challenges. First, capturing sufficient panoramic data is tedious and costly, as specialized cameras and/or additional processing are needed to remove stitching artifacts. Consequently, models must be trained in a low-data regime making them prone to overfitting, this limiting their generalization capabilities. Exemplary, a lot of models are restricted to indoor environments only (Wu et al., 2023; Song et al., 2023). Second, panoramas must fulfill additional constraints compared to perspective images. Most notably, the image borders must align to allow a seamless wrap-around. But there are also more intricate, semantic constraints, e.g., the viewing frustum must cover the entire scene. Hence, when generating a panorama of a bedroom, it must contain exactly one bed, at least one door, etc. On the other hand, outdoor panoramas should maintain realistic spatial relationships between elements.

To satisfy those requirements, prior work had to introduce complex additional model components (Gao et al., 2024a; Tang et al., 2023; Yang et al., 2024), or employ dedicated mechanisms such as autoregressive outpainting from a perspective view (causing artifacts like content drift and the Janus effect (Wang et al., 2023), and circular padding to enforce consistent wrap-around (Feng et al., 2023; Wu et al., 2023).

We introduce a simple yet highly effective solution: we generate panoramas using a fine-tuned multiview diffusion model, following recent line of work (Gao et al., 2024b; Tang et al., 2023; Zhang et al., 2023b). This approach leverages the inherent properties of cubemaps, where a 360° × 180° panorama is represented by six perspective images on the faces of a cube. This allows us to fully recycle a pretrained text-to-image model, enabling generalization far beyond the limited training data. Contrary to existing methods, the architectural modifications we require to ensure consistency between cube faces are minimal: all attention layers are inflated by one additional dimension to enable crosstalk between the six faces. This simple modification, combined with fine-tuning on panorama data, achieves state-of-the-art results with significant visual and semantic coherence. Additionally, the model allows for fine-grained text control by training with face-specific image-text pairs, easily generated by prompting an LLM to produce per face text descriptions.

Our key insight is that existing, generative image models can be easily extended to generate highresolution panoramas, by performing diffusion in cubemap space and adding attention mechanisms to other faces within the cubemap, see Figure 1. The resulting model

• enables consistent image generation across all cubemap faces and preserves the internet-scale image prior of the underlying diffusion model to generalize beyond the training panoramas;   
• delivers state-of-the-art results on panorama generation, both qualitatively and quantitatively, and outperforms previous methods in terms of visual fidelity and coherence;   
• enables efficient high-resolution synthesis, benefiting from current and future advances in offthe-shelf image diffusion models;   
• allows for novel fine-grained text control, enabling users to guide the generation with detailed textual descriptions.

# 2 RELATED WORK

Similar to 3D generative modelling, training data for panorama generation is scarce and much effort has been spent on how to repurpose standard perspective image priors for panoramas. The prevalent approach has been to autoregressively outpaint panoramas, more recently multi-view diffusion models have attracted interest. We now discuss relevant works and differences to our approach.

# 2.1 PANORAMA GENERATION.

Most panorama generators operate in equirectangular projection, thus having to deal with it severe nonlinear distortions (especially near the poles). Previous methods either autoregressively outpaint the panorama (Gao et al., 2024a; Lu et al., 2024; Wang et al., 2023) or generate the entire equirectangular image in one shot (Feng et al., 2023; Wu et al., 2023). They are commonly conditioned on either a single narrow field-of-view image (Akimoto et al., 2022) or solely on a text prompt (Chen et al., 2022). The state of the art are diffusion models, which have gradually replaced adversarial approaches (Akimoto et al., 2022; Somanath & Kurz, 2021). Feng et al. (2023) fine-tune a latent diffusion model on a panorama dataset and apply a circular blending strategy in the denoising and decoding stages to enforce consistent wrap-around. Similarly, Wu et al. (2023) stitches the right part of the image to the left part in latent space in each denoising step. Such blending improves the results, but encumbers the inference step. In our method it is not required. Lu et al. (2024) propose to autoregressively outpaint a panorama with a complex architecture of submodules for panoramaaware visual guidance, NFoV guidance and panorama-aware geometric guidance. In Wang et al. (2023), the authors extend the outpainting task to ingest multiple NFoV images of the same scene. A two-stage network predicts their relative rotations, then a diffusion model with ControlNet (Zhang et al., 2023a) outpaints the panorama based on the projected inputs. Recently (Voynov et al., 2023) introduce a diffusion model with control over the rendering geometry, including panoramic outputs. Gao et al. (2024a) additionally incorporate a state space model to aggregate global information into cross-attention layers of the diffusion model, building up the panorama by inpainting empty regions. The present work demonstrates that, with the right representation, high-quality panoramas can be obtained without inflating the complexity and brittleness of the architecture. Related to panorama generation is the more modest strategy to alter existing panoramas by injecting a user-defined style, in either equirectangular or cubemap projection (Yang et al., 2024; Song et al., 2023).

# 2.2 MULTI-VIEW DIFFUSION

Multi-view diffusion models offer a compelling alternative to equirectangular or autoregressive panorama generators. Zhang et al. (2023b) introduces a compositional diffusion scheme that enables the generation of large-scale content, leveraging models trained on smaller constituent parts. That work is based on factor graphs, and demonstrates how the cubemap can be turned into a factor graph in order to train a diffusion model conditioned on segmentation maps. The work most closely related to ours is Tang et al. (2023). It aims to generate cylindrical panoramas (i.e., 360°horizontal field of view, but restricted vertical view angle). The authors propose a sophisticated correspondence-aware cross-attention between local neighborhoods of eight perspective feature maps spaced at 45°angles. Recently, Gao et al. (2024b) and Shi et al. (2023) discovered that expanded attention layers that connect not only features within an image but also across multiple images, are beneficial when handling multiple object-centric views. Our approach turns this setup inside-out and extends a pretrained text-to-image (T2I) model in a similar manner for panorama generation. We instead do not require camera pose or 3D information, due to the fixed viewing geometry of the cubemap.

# 3 PANORAMA REPRESENTATIONS

Panoramic images aim to capture a complete $3 6 0 ^ { \circ } \times 1 8 0 ^ { \circ }$ view of a scene from a fixed view point. There exist several different panorama representations in literature, each with its own advantages and drawbacks. This section briefly discusses the most prominent ones.

Spherical projection maps a 360° view onto a sphere, preserving the geometric relationships between points in the scene. Points are generally defined using longitude and latitude. While conceptually intuitive, directly utilizing a spherical representation for image processing is challenging due to difficulties in representing a sphere on a flat image plane, which often leads to distortions and non-uniform sampling densities in practical implementations.

Equirectangular projection projects the spherical panorama onto a 2D rectangle. To this end, latitude and longitude coordinates on the sphere are mapped to vertical and horizontal coordinates on a rectangle. While widely used due to its simplicity, equirectangular projection suffers from significant distortions, especially near the poles where horizontal stretching becomes extreme. This distortion affects both visual quality and the performance of algorithms processing equirectangular panoramas, as most existing T2I models process images with NFoV images.

![](images/c6c08459c166ed0d0a54bd28d42bfa15493c13a71371c115f8e7b55e0bc03164.jpg)  
Figure 2: An overview of our training pipeline and panorama model. (a) We project all training panoramas onto a cubmap and feed the faces to our frozen VAE encoder with synchronized Group-Norm to obtain the respective latents and enrich them with panorama-specific positional encodings for explicit spatial awareness. (b) We only train the inflated attention layers to be cross-frame aware.

Cubemaps offer an alternative representation where a 360° view is projected onto the six faces of a cube. Each face captures a 90° field of view, resulting in six perspective images that can be seamlessly stitched together. This representation avoids the polar distortions inherent to equirectangular projections, providing more uniform sampling, making it highly applicable to existing diffusion models trained on vast amount of perspective images. However, note that cubemaps introduce discontinuities at the edges of the cube faces, which needs to be handled carefully.

# 4 METHOD

We introduce CubeDiff, a novel approach for generating high-quality, consistent panoramas using the cubemap representation. CubeDiff generates the six perspective views of a cubemap in parallel and context-aware manner, exploiting the strengths of pretrained T2I diffusion models. Below, we delve into the architectural choices that enable CubeDiff to achieve high-quality and consistent panoramas, while retaining strong generalization capabilities inherited from the pretrained model. Similar to Gao et al. (2024b), CubeDiff comprises a variational autoencoder (VAE) and a latent diffusion model (LDM), mirroring the structure of conventional T2I diffusion models. However, we carefully adapt each component for effective multi-view panorama generation.

# 4.1 MODEL ARCHITECTURE

The latents produced by the VAE are used to fine-tune a pretrained LDM operating on a 128x128x8 latent space, initialized with weights from a model trained on a large-scale image dataset. The pretrained LDM consists of an architecture similar to Stable Diffusion (Rombach et al., 2022), which is build with multiple convolutional, self-attention, and cross-attention layers. To enable cross-view awareness and maintain global consistency, we inflate all existing 2D attention layers, i.e. both selfattention and cross-attention for text conditioning. These layers, adapted from (Shi et al., 2023), extend the attention mechanism across all six cube faces, allowing the model to learn relationships and dependencies between different viewpoints. Inflating of layers can be easily conducted by extending the token sequence length from b × (hw) × l to b × (thw) × l, e.g. for self-attention, where b is the batch size, hw the flattened spatial size and t = 6 the cube length. While this is different to more sophisticated attention layers (Tang et al., 2023; Huang et al., 2024), it in turn enables us to retain the original pretrained attention weights, which reduces the risk of overfitting and thus greatly improves overall performance.

The LDM receives two conditioning signals. We incorporate text embeddings, either one common prompt or one prompt for each face, and a single conditional view of the scene (w.l.o.g. we assume the front face of the cube). During training, we concatenate the VAE latents of the conditioning views to the noisy latents of the target views, providing the LDM with complete context information. Furthermore, we incorporate a binary mask channel into the latent representations. This mask distinguishes between conditioning views (provided as clean latents) and target views (subjected to noise injection during training). We show an overview of our model architecture in Figure 2.

# 4.2 SYNCHRONIZED GROUPNORM

Our VAE architecture incorporates synchronized group normalization, a crucial element for achieving consistent color tones across the generated panorama. Since our VAE processes the six faces of a cubemap as a batch of six individual images, standard group normalization can lead to subtle color inconsistencies among different views $( c . f .$ . fig. 6a). This occurs as feature statistics are computed and normalized independently for each image in the batch. Without synchronization, encoding and decoding a panorama results in noticeable shifts, particularly evident in the equirectangular projection. Synchronized group normalization addresses this issue by jointly normalizing feature activations across both spatial and inter-view dimensions. Consequently, synchronized group normalization contributes significantly to the generation of visually harmonious and coherent panoramas. Similar effects have been observed in (He et al., 2023). We further discuss this in Section 5.6 and compare synchronized and unsynchronized results in Figure 6a.

# 4.3 POSITIONAL ENCODING

To provide the LDM with explicit spatial awareness within the cubemap, we augment the latent representations with positional encodings derived from the 3D geometry of the cube. For each point on a cube face, we compute its corresponding UV coordinates on the unit cube, defined by:

$$
u = \arctan 2 (x, z), v = \arctan 2 (y, \sqrt {x ^ {2} + z ^ {2}}), \tag {1}
$$

where $( x , y , z )$ are the 3D coordinates of the point on the cube face, projected onto the unit cube. These UV coordinates are then normalized to [0, 1] and concatenated as two additional channels to the (noisy) latents. This positional encoding scheme provides the model with information about the spatial location of each latent patch within its respective cube face, facilitating the generation of panoramas with consistent geometry and object relationships across views.

# 4.4 OVERLAPPING PREDICTIONS

To further enhance the geometric and color consistency across cube faces, we introduce overlapping predictions during both training and generation. Instead of generating each face with a 90° field of view (FoV), we enlarge the FoV by 2.5° on each side, resulting in an effective FoV of 95° per face. This means each generated face includes a small overlap with its neighboring ones. This overlapping generation strategy serves two purposes. During training, it encourages the model to learn consistent representations across adjacent faces, as the overlapping regions provide additional context and constraints. During panorama assembly, we discard these overlapping regions and only retain the central 90° portion of each generated face. This strategy effectively avoids the need for explicit blending operations at the cube face boundaries, which can sometimes introduce subtle artifacts. The overlaps can be seen at the boundaries of the cubemaps in Figure 3 (e.g., the duplicated fireplace in the right and back views) and in the appendix.

# 4.5 CLASSIFIER-FREE GUIDANCE

We employ classifier-free guidance (CFG) (Ho & Salimans, 2022) on both the text and image conditions during training. Thereby, we randomly drop either the text prompt, the conditional image, or both. When the text prompt is dropped, it is replaced with null tokens in the cross-attention layers; when the conditional image is dropped, its corresponding tokens in the self-attention layers are masked out by setting them to negative infinity, effectively zeroing out their attention weights. This training procedure enables diverse panorama generation scenarios during inference. Users can provide both text and image conditions for maximum control and fidelity or drop both or either condition to explore unconditional generation modes.

# 5 EXPERIMENTS

This section details our experimental setup, followed by quantitative and qualitative evaluations. We compare the performance of CubeDiff against the state-of-the-art and ablate our design choices.

# 5.1 EVALUATION PROTOCOL

# 5.1.1 TRAINING AND INFERENCE SETUP

We finetune our model using Adam (Kingma & Ba, 2014) and train for 30,000 iterations with batch size 64. The learning rate is ramped up to $8 \times 1 0 ^ { - 5 }$ in the first 10,000 steps. During training, we employ classifier-free guidance, dropping conditional signals 10% of the time. We find it important to not only drop the text condition in the cross-attention layers but to also zero out the input condition in the self-attention layers. The diffusion model is finetuned using v-prediction (Salimans & Ho, 2022). We employ DDIM sampling (Song et al., 2020) with 50 steps during inference.

# 5.1.2 DATASETS

Training. We train on a mixture of indoor and outdoor environments by combining multiple publicly available sources, including Polyhaven (polyhaven.com, accessed 09/2024), Humus (Persson, accessed 09/2024), Structured3D (Zheng et al., 2020) and Pano360 Kocabas et al. (2021), giving in total around 48000 panoramas for training. While Humus provides an explicit cubemap representations, all other datasets come with equirectangular panoramas. We thus first generate cubemaps from these panoramas using standard perspective projection, ensuring consistent overlap between adjacent faces. To further enable text-guided panorama generation, we infer textual descriptions for each panorama in the datasets using the publicly available Gemini model (Gemini Team Google, 2023). We explore two captioning strategies: (1) generating a single caption for the entire panorama by providing Gemini with all six cube faces as input and (2) generating individual captions for each face independently, enabling fine-grained text control.

Testing. We evaluate our method on the common Laval Indoor (Gardner et al., 2017) and Sun360 (Xiao et al., 2018) datasets. Laval Indoor consists of over 2100 high quality panorama captures of various indoor environments, Sun360 encompasses around 1000 panoramas including both – indoor and outdoor scenes. Note that we use those datasets only for evaluation, while Diffusion360 also uses Sun360 for training and OmniDreamer even leverages both datasets to train their models. Nonetheless, we decided to use these datasets for the sake of fairness and due to the lack of any proper overlapping test datasets.

# 5.1.3 METRICS

We use various metrics and modalities for evaluation – including perceptual metrics, text alignment, and a user study.

Perceptual Metrics. We use the very common Frechet Inception Distance (FID) (Heusel et al.,´ 2017) metric to measure the similarity between the distribution of real and generated images in a feature space derived from a pretrained Inception network. Lower FID scores indicate greater similarity and, thus, higher image realism; We additionally report the CLIP-FID (Kynka¨anniemi ¨ et al., 2022) metric, replacing the Inception network with CLIP (Radford et al., 2021) to leverage its semantic understanding capabilities through a joint image-text embedding space. This metric captures thus both – visual fidelity and text-image alignment; Finally, we employ the kernel inception distance (KID)(Binkowski et al., 2018). Similar to FID, KID uses features from a pre-trained ´ network, however, it quantifies the difference between real and generated data distributions using the maximum mean discrepancy rather than the Frechet distance. ´

Text Alignment. To measure text alignment we refer to the common CLIP score (Hessel et al., 2021) . The CLIP score computes the cosine similarity within the shared text-image embedding to measure the agreement between generated panoramas and their corresponding text prompts. Hence, a higher CLIP score indicates stronger semantic agreement between image and text.

# 5.1.4 COMPETITORS

We compare CubeDiff to various state-of-the-art panorama generation methods. As for plain text to panorama generation, we employ Text2Light (Chen et al., 2022) and PanFusion (Zhang et al.,

![](images/10d694317fb7c4ec905ea6d68abcedcf9e9e43a79358d40779fe6b2390ddeb7c.jpg)  
Figure 3: Cubemaps and panoramas generated by CubeDiff with image and text condition. We depict a diverse set of generated panoramas including indoor, outdoors, bright and dark scenes. In all settings, CubeDiff produces high quality and realistic panoramas that align with the input image.

2024) to serve as our main competitors. For single image conditioning, we respectively use OmniDreamer (Lu et al., 2024) and PanoDiffusion Wu et al. (2023) as representatives for autoregressive and direct panorama generation based approaches. Finally, we compare against Diffusion360 (Feng et al., 2023) and MVDiffusion (Tang et al., 2023) for text and image conditioning based methods. Note that while Diffusion360 directly outputs panorama images, MVDiffusion instead employs multi-view diffusion models with a custom cross attention mechanism. Overall, the choice of baselines represents a variety of different generation techniques, covering various different tasks. Please note that none of the existing methods besides MVDiffusion offers the possibility to condition specific parts of the panorama on individual text prompts.

# 5.2 QUALITATIVE EVALUATION

In this section, we provide a qualitative evaluation of our method. We first present several conditional image generations of our method, before comparing CubeDiff against the state-of-the-art.

# 5.2.1 CONDITIONAL IMAGE GENERATION.

In Figure 3, we show generated panoramas given text-image pairs as condition. We considered input conditions that cover a broad range of scenes, such as outdoor and indoor scenes, bright and dark settings as well as texture rich and uniformly colored areas. Note that we do not show the text conditions due to limited space, however, we provide them in the appendix. We see that our approach yields high quality results under these diverse input settings. We especially emphasize the level of detail and geometric consistency beyond the input image.

# 5.2.2 QUALITATIVE COMPARISON.

For visual comparison against the state-of-the-art, we show generated panoramas and their respective perspective projections in Figure 4. To this end, we sample random image and text pairs from the LAVAL Indoor dataset. We further group the methods according to their input modalities. Compared to the text-only approach Text2Light, our method is able to produce much more complex panoramas with better details and visual appeal. As for image-only approaches, we see that CubeDiff is capable of producing the most realistic panoramas. In particular, while OmniDreamer suffers from blurry regions, PanoDiffusion is not able to properly transfer the input image appearance across the whole panorama. Finally, also for text and image conditioning our method again produces the best results, especially in terms of geometry. For example, while MVDiffusion is indeed capable of generating high quality images, the method sometimes produces inaccurate geometries as, for example, some walls and hand rails exhibit bending artifacts after perspective projection. Similarly, Diffusion360 occasionally suffers from implausible indoor layouts. To summarize, despite of using different input modalities, CubeDiff always generates high quality panoramas, surpassing all other state-of-the-art works in terms of visual appeal and geometric consistency.

![](images/c5d628531073bdbc09b9aca84756aed77c1b30d580a5012eba71a62f7dd2e72f.jpg)

<details>
<summary>text_image</summary>

Text-only
Text2Light
PanFusion
Text-only
Image-only
OmniDreamer
PanoDiffusion
Image and Text
MVDiffusion
Diffusion360
OurSing+xtt
OurSing+multtxt
</details>

Figure 4: Qualitative comparison between CubeDiff and baselines on the LAVAL Indoor Dataset. Besides Text2Light, all panoramas are generated using the center face as input condition and additional text prompts if applicable. For each sample we show the panorama image as well as two projected images. Please zoom in to compare the different methods.

<table><tr><td rowspan="2"></td><td colspan="5">LAVAL Indoor</td><td colspan="5">SUN360</td></tr><tr><td>FID ↓</td><td>KID (×102)↓</td><td>Clip-FID ↓</td><td>FAED ↓</td><td>CS ↑</td><td>FID↓</td><td>KID (×102)↓</td><td>Clip-FID↓</td><td>FAED ↓</td><td>CS ↑</td></tr><tr><td>Text2Light</td><td>28.3</td><td>1.45</td><td>11.5</td><td>136.1</td><td>25.18</td><td>60.1</td><td>4.31</td><td>31.3</td><td>82.9</td><td>23.27</td></tr><tr><td>PanFusion</td><td>41.7</td><td>2.85</td><td>19.8</td><td>71.7</td><td>26.58</td><td>30.0</td><td>1.42</td><td>7.8</td><td>44.5</td><td>25.28</td></tr><tr><td>OmniDreamer</td><td>71.0</td><td>5.17</td><td>23.9</td><td>19.2</td><td>-</td><td>92.3</td><td>8.89</td><td>51.7</td><td>30.4</td><td>-</td></tr><tr><td>PanoDiffusion</td><td>58.6</td><td>4.08</td><td>26.6</td><td>106.8</td><td>-</td><td>52.9</td><td>3.51</td><td>28.9</td><td>98.0</td><td>-</td></tr><tr><td>Oursimg</td><td>11.7</td><td>0.47</td><td>4.4</td><td>22.0</td><td>-</td><td>27.4</td><td>1.35</td><td>11.5</td><td>8.9</td><td>-</td></tr><tr><td>Diffusion360</td><td>33.1</td><td>2.07</td><td>16.9</td><td>23.7</td><td>26.38</td><td>45.4</td><td>3.73</td><td>18.5</td><td>12.6</td><td>22.89</td></tr><tr><td>Oursimg+txt</td><td>9.5</td><td>0.32</td><td>3.2</td><td>18.4</td><td>27.02</td><td>25.5</td><td>1.33</td><td>8.1</td><td>7.6</td><td>25.00</td></tr><tr><td>MVDiffusion</td><td>25.7</td><td>1.11</td><td>13.5</td><td>-</td><td>27.44</td><td>50.9</td><td>3.71</td><td>15.4</td><td>32.3</td><td>25.54</td></tr><tr><td>Oursimg+multitxt</td><td>10.0</td><td>0.35</td><td>4.1</td><td>21.2</td><td>30.17</td><td>24.1</td><td>1.33</td><td>7.0</td><td>5.7</td><td>28.14</td></tr></table>

Table 1: Quantitative Evaluation on the Laval Indoor and SUN360 dataset. We provide a comparison to various competitors and different input modalities. The first block of rows are text-only methods, the second image-only, the third image and single text description and the last block are image and multi-caption methods. CubeDiff provides the best perceptual quality having the best scores across all methods. Moreover, we find that the performance of CubeDiff remains similar among different input modalities.

# 5.3 QUANTITATIVE EVALUATION

In this section, we provide the results of our quantitative evaluation on the Laval Indoor and the SUN360 dataset. We evaluate all methods on perceptual quality and consistency.

In Table 1 we provide quantitative results for visual quality. Our method outperforms all competitors significantly, regardless of input modalities. For example, we can report a FID score of 9.47 on Laval Indoor, which is a 270% relative improvement compared to the second best performing method MVDIffusion, reporting a score of 25.7. Compared to works that use only image or text as input conditioning, the gap even widens with Text2Light and PanoDiffusion respectively reporting a FID of 28.3 and 58.6. This trend holds across all metrics. Interestingly, CubeDiff performs similarly across different input modalities, demonstrating its strong generalizibility.

However, the provided perceptual metrics can only evaluate the overall realism of the generated panoramas and are not capable of capturing consistency towards input. We next study the alignment to the input text prompt. To this end, we leverage the CLIP score to measure how well the generated panoramas align with the text input. As can be seen in the table our method surpasses the state-of-the-art again by a significant amount for all datasets and modalities, showing how precisely our model respects the textual input.

# 5.4 USER STUDY

We conducted a user study with a two-alternative forced choice (2AFC) survey to evaluate our panorama generation method. Each of the 28 participants was shown 30 pairs of generated panoramas alongside the original conditioning

image and asked to select their preferred option based on quality, composition, style, and alignment with the condition image.

Our method outperformed competitors statistically $( p < 0 . 1$ , binomial test). Specifically, 16.9%, 17.3%, and 19.5% of participants preferred our single-image, multi-image, and no-text variants, respectively. The no-text variant nearly matched the ground truth preference (19.9%), demonstrating our method’s ability to generate realistic and accurate panoramas. In contrast, OmniDreamer, PanoDiffusion, MVDiffusion, and Diffusion360 had significantly lower preference rates of 1.7%, 5.3%, 7.0%, and 12.3%, respectively.

![](images/5ae06114b74e7ecb1dfe5ab4c04f55cb03f91b5450221d5f3159f4e673165724.jpg)

<details>
<summary>natural_image</summary>

Six-panel interior view of a cozy library with wooden bookshelves, reading tables, and a fireplace (no visible text or symbols)
</details>

Figure 5: Fine-grained Text Control. We show an example for finegrained-text control of the back face. Our model is able to change details following the provided prompt. First, we add a golden globe above the fireplace; second, we place a picture above the fireplace; third, we leave the space above empty; last, we instead add a bookshelf above it.

![](images/724a657dacfde204c7dbd9cdf4dad127753466c203e8c5a34f1a0c9068f602d7.jpg)  
(b)   
Figure 6: Ablation on synchronized GN and overlap prediction. (a) Top: Group normalization over the spatial dimension only. Bottom: Additional normalization over the frame dimension. (b) Top: Panoramas without overlapping cube faces. Bottom: Panoramas with our standard 2.5° overlap. Please zoom in to observe the differences.

# 5.5 FINE-GRAINED TEXT CONTROL

Different to all competitors, our method enables complete fine-grained and per-face text control. For example in Figure 5, we show results for providing different text descriptions for the back face. We can always generate visually appealing results, regardless of the object we place above the fireplace.

# 5.6 ABLATIONS

Synchronized Group Norm (GN) Synchronized GN ensures consistency across cube faces by normalizing over both spatial and frame dimensions, as shown in Figure 6a. Without it, models often exhibit color inconsistencies and artifacts at cube face boundaries. While metrics like FAED may not capture these subtle issues, synchronized GN significantly improves visual quality.

Overlapping Prediction Overlapping predictions mitigate discontinuities at cube face boundaries by introducing small overlaps, as illustrated in Figure 6b. This ensures seamless transitions, with non-overlapping regions cropped for the final panorama. The approach leverages global context from full attention, eliminating visible seams without additional VAE finetuning.

# 6 CONCLUSION

This work introduces a novel approach to panorama generation leveraging pretrained text-to-image diffusion models applied to a cubemap representation. By enabling attention across the cube faces, our method achieves state-of-the-art results in terms of visual fidelity and coherence, while requiring minimal architectural changes. This approach not only inherits the strengths of existing diffusion models, including high-resolution synthesis and generalization capabilities, but also unlocks finegrained text control over the generated panorama. This opens up exciting new possibilities for creative applications and paves the way for future research in controllable panorama generation.

# REFERENCES

Naofumi Akimoto, Yuhi Matsuo, and Yoshimitsu Aoki. Diverse plausible 360-degree image outpainting for efficient 3DCG background creation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.   
Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Labelefficient semantic segmentation with diffusion models. arXiv preprint arXiv:2112.03126, 2021.   
Mikołaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD ´ GANs. arXiv preprint arXiv:1801.01401, 2018.   
Zhaoxi Chen, Guangcong Wang, and Ziwei Liu. Text2Light: Zero-shot text-driven HDR panorama generation. ACM Transactions on Graphics (TOG), 41(6), 2022.   
Mengyang Feng, Jinlin Liu, Miaomiao Cui, and Xuansong Xie. Diffusion360: Seamless 360 degree panoramic image generation based on diffusion models. arXiv preprint arXiv:2311.13141, 2023.   
Penglei Gao, Kai Yao, Tiandi Ye, Steven Wang, Yuan Yao, and Xiaofeng Wang. OPa-Ma: Text guided Mamba for 360-degree image out-painting. arXiv preprint arXiv:2407.10923, 2024a.   
Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul Srinivasan, Jonathan T Barron, and Ben Poole. CAT3D: Create anything in 3d with multi-view diffusion models. arXiv preprint arXiv:2405.10314, 2024b.   
Marc-Andre Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Chris- ´ tian Gagne, and Jean-Franc¸ois Lalonde. Learning to predict indoor illumination from a single ´ image. arXiv preprint arXiv:1704.00090, 2017.   
Gemini Team Google. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.   
Yingqing He, Shaoshu Yang, Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang, Ran He, Qifeng Chen, and Ying Shan. ScaleCrafter: Tuning-free higher-resolution visual generation with diffusion models. In International Conference on Learning Representations (ICLR), 2023.   
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.   
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.   
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.   
Zehuan Huang, Hao Wen, Junting Dong, Yaohui Wang, Yangguang Li, Xinyuan Chen, Yan-Pei Cao, Ding Liang, Yu Qiao, Bo Dai, et al. EpiDiff: Enhancing multi-view synthesis via localized epipolar-constrained diffusion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.   
Haian Jin, Yuan Li, Fujun Luan, Yuanbo Xiangli, Sai Bi, Kai Zhang, Zexiang Xu, Jin Sun, and Noah Snavely. Neural gaffer: Relighting any object via diffusion. arXiv preprint arXiv:2406.07520, 2024.

Nikolai Kalischek, Torben Peters, Jan D Wegner, and Konrad Schindler. TetraDiffusion: Tetrahedral diffusion models for 3d shape generation. In European Conference on Computer Vision (ECCV), 2022.   
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.   
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 12 2014.   
Muhammed Kocabas, Chun-Hao P. Huang, Joachim Tesch, Lea Muller, Otmar Hilliges, and ¨ Michael J. Black. SPEC: Seeing people in the wild with an estimated camera. In International Conference on Computer Vision (ICCV), 2021.   
Tuomas Kynka¨anniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of¨ ImageNet classes in Frechet Inception Distance. ´ arXiv preprint arXiv:2203.06026, 2022.   
Zhuqiang Lu, Kun Hu, Chaoyue Wang, Lei Bai, and Zhiyong Wang. Autoregressive omni-aware outpainting for open-vocabulary 360-degree image generation. In AAAI Conference on Artificial Intelligence (AAAI), volume 38, 2024.   
Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. CLIP-Mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia, 2022.   
Emil Persson. Texture from Humus. https://www.humus.name/index.php?page=Textures, accessed 09/2024.   
polyhaven.com. HDRIs. https://polyhaven.com/hdris, accessed 09/2024.   
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. DreamFusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022.   
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.   
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High- ¨ resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.   
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems (NeurIPS), 2022.   
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022.   
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308.16512, 2023.   
Gowri Somanath and Daniel Kurz. HDR environment map estimation for real-time augmented reality. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.   
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.   
Liangchen Song, Liangliang Cao, Hongyu Xu, Kai Kang, Feng Tang, Junsong Yuan, and Yang Zhao. RoomDreamer: Text-driven 3d indoor scene synthesis with coherent geometry and texture. arXiv preprint arXiv:2305.11337, 2023.

Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. MVDiffusion: Enabling holistic multi-view image generation with correspondence-aware diffusion. arXiv preprint arXiv:2307.01097, 2023.   
Andrey Voynov, Amir Hertz, Moab Arar, Shlomi Fruchter, and Daniel Cohen-Or. Curved diffusion: A generative model with optical geometry control. arXiv preprint arxiv:2311.17609, 2023.   
Jionghao Wang, Ziyu Chen, Jun Ling, Rong Xie, and Li Song. 360-degree panorama generation from few unregistered NFoV images. arXiv preprint arXiv:2308.14686, 2023.   
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. ProlificDreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.   
Tianhao Wu, Chuanxia Zheng, and Tat-Jen Cham. PanoDiffusion: 360-degree panorama outpainting via diffusion. In International Conference on Learning Representations (ICLR), 2023.   
J. Xiao, K. A. Ehinger, A. Oliva, and A. Torralba. Recognizing scene viewpoint using panoramic place representation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.   
Bangbang Yang, Wenqi Dong, Lin Ma, Wenbo Hu, Xiao Liu, Zhaopeng Cui, and Yuewen Ma. DreamSpace: Dreaming your room space with text-driven panoramic texture propagation. In IEEE Conference Virtual Reality and 3D User Interfaces (VR), 2024.   
Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. DiLightNet: Fine-grained lighting control for diffusion-based image generation. In ACM SIGGRAPH, 2024.   
Cheng Zhang, Qianyi Wu, Camilo Cruz Gambardella, Xiaoshui Huang, Dinh Phung, Wanli Ouyang, and Jianfei Cai. Taming stable diffusion for text to 360 {\deg} panorama image generation. arXiv preprint arXiv:2404.07949, 2024.   
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In IEEE/CVF International Conference on Computer Vision, 2023a.   
Qinsheng Zhang, Jiaming Song, Xun Huang, Yongxin Chen, and Ming-Yu Liu. DiffCollage: Parallel generation of large content with diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023b.   
Xiaoming Zhao, Pratul P Srinivasan, Dor Verbin, Keunhong Park, Ricardo Martin Brualla, and Philipp Henzler. IllumiNeRF: 3d relighting without inverse rendering. arXiv preprint arXiv:2406.06527, 2024.   
Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In European Conference on Computer Vision (ECCV), 2020.

![](images/aa008fd009ec12990c64804a8e4f4e859cafb79128a2c31f46813a086bae47b3.jpg)

<details>
<summary>natural_image</summary>

Artistic depiction of a planet swirling in deep space with glowing blue rings and stars, no text or symbols present.
</details>

Figure 7: Panorama in a out of training-distribution setting.

# A APPENDIX

# A.1 OUT-OF-DISTRIBUTION EXAMPLES

![](images/bd18aa6ee117f69c9fbd4a89f6f73a88f03ff3cb8ff873620d94e42eb44c3376.jpg)

<details>
<summary>natural_image</summary>

Three-panel artistic painting depicting the Cape of Vidy, Van Gogh, and St. Paul's Palace with dramatic landscapes and no visible text or symbols.
</details>

![](images/763a06c03ff14ba0a5603883c71ec6962e76516fea4a63ec6059b34dafb9c665.jpg)

<details>
<summary>natural_image</summary>

Three-panel illustration of a fantasy-themed room with mushroom-themed walls, a girl standing on a stone path, and a glowing mushroom island in the night sky (no text or symbols)
</details>

Figure 8: Out-of-distribution examples generated by Cubediff. On the right are artistic generations, on the left fantasy sceneries like Alice in Wonderland.

# A.2 TEXT PROMPT EXAMPLES

The text prompts for the qualitative examples are the following:

# City skyline

• A glittering cityscape at night, densely packed high-rise buildings illuminated against a dark sky. The buildings are various heights and designs, reflecting light onto the calm water in the foreground. A hill or mountain range forms a dark backdrop behind the city.   
• Dark, silhouetted mountains stretching across the entire view, with the faintest glow of city lights on the horizon. The sky is a deep, nighttime blue.   
• The side profile of a mountainous region, dark and slightly textured, extending from the foreground to the distant horizon. The city lights are visible in the distance to the right, providing a subtle contrast against the mountain’s blackness.   
• Similar to the right. A darker, less detailed section of the mountain range appears to the left.   
• An expansive, dark night sky with subtle cloud formations. The upper portions of the city’s tallest buildings are faintly visible as a horizontal line against the darkness.   
• The dark, calm water of the bay reflecting the city lights, showing subtle ripples and disturbances. The reflection is most intense closest to the buildings and gradually fades into darkness.

# Green house

• A dense, overgrown greenhouse teems with lush greenery, creating a vibrant, jungle-like scene. Large ferns, broad-leafed tropical plants, and hanging baskets overflowing with foliage dominate the view. Large rocks, blanketed in thick, \*vibrant\* green moss, suggest a path, nearly disappearing beneath the surrounding plants. The glass roof, tinged with green, casts a soft, diffused light.   
• The right side of the greenhouse is a solid wall of plant life. Large, vibrant green leaves and hanging plants create a dense, tropical atmosphere. The rocks, almost entirely smothered in \*bright\* green moss and overflowing greenery, offer only the faintest hint of a path. Wild, untamed growth dominates the scene.   
• The greenhouse extends into the distance, an endless expanse of green fading into shadow. The rocks, now completely obscured by plants and a thick carpet of \*emerald\* moss, suggest a path swallowed by the jungle. The sheer volume of greenery creates a sense of depth and wildness. The glass roof is barely visible.   
• To the left, a tall palm tree rises amidst the dense foliage. Large ferns and other leafy plants create a solid wall of green around it. The rocks, barely discernible beneath the thick layer of \*brilliant\* green moss and creeping vines, continue on this side, obscuring the path almost entirely.   
• Looking up, the glass panes of the greenhouse roof, tinted green with clinging vines and moss, are mostly obscured by the dense canopy. Hanging plants, heavy with ferns and leafy vines, cascade downwards, creating a lush, verdant ceiling. The metal framework is almost entirely hidden.   
• Looking down, the large rocks are almost completely hidden beneath a thick carpet of \*luminescent\*, almost \*glowing\* green moss, fallen leaves, vines, and other plant debris. Only the edges of the rocks peek through the dense greenery, making the path nearly invisible. The texture of the moss appears incredibly soft and velvety, a seamless blanket of vibrant green.

# Living room

• A cozy, modern living room with large windows allowing natural light to flood in. The room is furnished with a soft, gray sofa facing the windows and a wooden coffee table in the center. A bookshelf filled with plants and books stands against the far wall. A fireplace in the corner crackles softly, casting a warm glow across the room.

• A side view of the living room from the right, showing the side of the gray sofa facing toward the large windows. The coffee table is positioned in front of the sofa. To the left, the wall-mounted television is visible above the fireplace, and the wooden floor stretches across the room. A small side table with a lamp sits next to the sofa.

• A view of the back wall of the room, where the bookshelf is the primary focus. The large windows let in a soft light, but the sofa and coffee table are out of sight from this angle. The fireplace is visible on the right side of the room, softly illuminating the space, and the television mounted above it is partially visible.

• A side view from the left side of the living room, showing the bookshelf along the far wall, and the curtains gently swaying in front of the large windows. The coffee table sits on a stylish rug, but the sofa itself is mostly out of view, hidden from this angle. The soft glow of the fireplace on the far side of the room adds warmth to the scene.

• Looking up at the ceiling, the room features modern recessed lighting, casting a soft, even glow across the space. Wooden beams accent the edges of the ceiling, adding a rustic touch. The tops of the windows and the moving curtains are visible from this angle, as the diffuse light from outside fills the room.

• Looking down at the floor, you see polished wooden floorboards and a stylish area rug that lies under the coffee table. The legs of the sofa are visible at the edge of the rug, and a few books and a small potted plant sit on the coffee table. The contrast between the rug and the wooden floor gives the room a balanced, warm feel.

# Snow landscape

• A view of a frozen river winding through a snowy forest. The river is partially frozen, with snow-covered banks on either side and dark, still water visible in the center. Bare, leafless trees line the edges of the river, their branches covered in snow. The overcast sky casts a soft, cold light over the scene, creating a peaceful and serene winter atmosphere.

• A side view of the snowy forest, showing the frozen river cutting through the landscape. Bare trees with snow-covered branches stretch across the scene, and the forest extends into the distance, with the winding river creating a natural divide. The snowy ground and trees give the area a quiet, isolated feel, and the soft light from the sky casts long shadows across the snow.

• A view from behind the river, where the water flows into the distance, disappearing into the snow-covered forest. The winding shape of the river is prominent, with the snow forming smooth, white edges along the banks. The bare trees on either side create a tunnel-like effect as they stretch over the river. The air feels crisp and cold, and the entire landscape is blanketed in a thick layer of snow.

• A left side view of the river, with snow-covered branches hanging over the water. The frozen river winds through the snowy forest, with the bare trees standing tall on either side. The snow is thick and untouched, creating a pristine winter scene. The soft light of the overcast sky adds a calm, cold atmosphere to the landscape.

• An aerial view of the frozen river, showing its winding path through the snow-covered forest. The dark water contrasts sharply with the white snow, and the leafless trees form a web of branches stretching across the landscape. The river curves gently through the scene, and the blanket of snow gives the forest a peaceful, quiet feel.

• Looking down at the frozen river from above, the snow forms a thick blanket along the banks, with patches of dark water visible in the center. The ground is covered in snow, with bare tree branches reaching over the river. The scene feels still and cold, with no signs of movement, and the snow seems to absorb all sound, creating a peaceful, quiet atmosphere.

For completeness, we provide the text prompts used for the qualitative comparison on the Laval Indoor dataset. On the left, we depict the input image and on the right be provide the text prompt.

![](images/d6dcbdb052cadb22623f74dc3333b4afb8cd1a396a5de11429a3e53b7400c29d.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern building with wooden stairs and a large windowsill (no visible text or symbols)
</details>

A concrete stairwell with orange railings leads up to a yellow door with a number 2 on it.

A bedroom with a window overlooking a snowy forest, a bed, a desk, and a dresser.

# A.3 PERSPECTIVE IMAGES FOR EVALUATION

Please note that all perceptual/text alignment metrics require another network to be computed. However, as these networks are trained with perspective images alone, the metrics would not give meaningful results when computed on panoramas. To circumvent this problem, we instead we render 10 random perspective images with a FOV of $9 0 ^ { \circ }$ for each panorama and use those for metric computation. Notice that we do not sample with an elevation of less than ${ } _ { - 4 5 ^ { \circ } }$ or more then $4 5 ^ { \circ }$ as other works such as MVDiffusion do not generate full $3 6 0 ^ { \circ }$ panoramas.

# A.4 USER STUDY

As described in the main paper we perform a two-alternative forced choice (2AFC) considering all competitors and variants of our method. In Figure 9, we show the percentage of wins against all 1258 draws and corresponding confidence intervals. This indicates ours methods performs significantly better in the user study.

![](images/610a16e7d4327a448a0d2a33aa54d3a360c3a12a25405cea567595534a270c28.jpg)

<details>
<summary>scatter</summary>

| Model               | Value |
| ------------------- | ----- |
| GT                  | 0.20  |
| Ours_imgtxtsingle  | 0.17  |
| Ours_notext         | 0.19  |
| Ours_imgmulttxt    | 0.17  |
| panodiffusion       | 0.05  |
| omnidreamer         | 0.02  |
| panoimage           | 0.12  |
| mvdiffusion         | 0.07  |
</details>

Figure 9: Results of User study. In this figure, we show the percentage of wins against all draws including the confidence interval

# A.5 MORE QUALITATIVE RESULTS

We provide more qualtitative results. In Figures 10 and 11, we show the input modalities in green and the generated faces and panorama. We see that our model aligns well to the given text prompts.

![](images/d34187f05d2d15c328f2313833354381fe0cc56a81533710a4d8bc5d90fd92a9.jpg)  
Figure 10: Generated panoramas with multiple text prompts and image condition

![](images/af2d4971581d21f335f1fae3e05a696e7f122d19b36ef085ff1d6b9f2e9e00c1.jpg)  
Figure 11: Generated panoramas with multiple text prompts and image condition

# A.6 MORE RESULTS OF UNSYNCHRONIZED GROUPNORM

![](images/2e3285ad3769628254bea4df0886f9956983136b56e96ba43563d57627e0b238.jpg)

<details>
<summary>natural_image</summary>

Interior view of a large, empty classroom with desks, chairs, and curved ceiling (no visible text or signage)
</details>

![](images/dae51bd089b0b8768d673c472bb26b38caa00f5e5edc6e173ebcfe39b447cd5f.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern living room with sofa, lamp, and curved glass door (no visible text or symbols)
</details>

![](images/ad214e946438ba71bc9b5aeff9fd5354dcd60310eb6a022ffd7cf7b4a562fc78.jpg)

<details>
<summary>natural_image</summary>

Interior view of a cozy bedroom with yellow walls, bookshelves, and a bed (no visible text or symbols)
</details>

![](images/c427cc5b6970de638b86c84916690fadac6cf682c3c1eb09e189e00425441507.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a hallway with curved walls, doors, and wooden flooring (no text or symbols visible)
</details>

![](images/0f707b5e75f5415c7b06c18b2f4dcfb7470fc80c21fe0dc44d42e5ff00f2da41.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern kitchen with curved wooden appliances and dining areas (no visible text or symbols)
</details>

![](images/ea1a10e0ccf4c97020528036bbf099be6c67f323695e2eeada548f8ba2fce1d0.jpg)

<details>
<summary>natural_image</summary>

Interior view of a multi-level library with curved bookshelves filled with books, no visible text or signage
</details>

![](images/7f2ab7ba8a23bdff0590eae99ecf66738ff4965ffc386ee83ad2fcb232ec705a.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern living room with sofa, TV, and coffee table (no visible text or symbols)
</details>

![](images/70e8ad8d0073017a596ab827145f0c9c4fdda240b7b4c023c0c63e6b46c3f6a1.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a long, empty corridor with curved walls, tiled flooring, and a central door (no visible text or symbols)
</details>

![](images/d6a9b1ff7f3ef6832fc4e1b6c15e45444d22ff63f6b2a7415f9396dee147933b.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern living room with sofa, lamp, and door (no visible text or symbols)
</details>

![](images/6a872aa0a0982b5117ef812b0c0fd6553dffd154ebc8d9f9ba936d0f92fd80e7.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a spacious modern building with curved arches and white tiled flooring (no visible text or symbols)
</details>

![](images/2f1a373deeb7829ab667accb706ad19302ec3b969a0075ef6645237b68a5abb8.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern hallway with curved wooden doors and blue carpet (no visible text or symbols)
</details>

![](images/e446b4a2a126f069ed8683e6a4605af4a5876cef83d820a3cd48a06cc2475069.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern meeting room with curved ceiling, wooden doors, and large windows (no visible text or symbols)
</details>

Figure 12: Additional results of predictions with unsynchronized Group Norm.

# A.7 MORE RESULTS OF NON-OVERLAPPING PREDICTIONS

![](images/46407547e3c74be26346ce6044cd04aa91d3395f58cf0303b15dcf55b1d8b2d3.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern hallway with curved ceiling and tiled floor (no visible text or symbols)
</details>

![](images/4ea803d9c9c4ee8c216fb10820a20023b2cce1aff09aa88e05ca0db4db72f686.jpg)

<details>
<summary>natural_image</summary>

Interior view of a retail store with circular shelves stocked with products and a ceiling light (no visible text or signage)
</details>

![](images/82e71b8aeaea61722190b0829fae37d5f2ad5f18006edbafbb5f9906c8041fae.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern kitchen with wooden cabinets and appliances (no visible text or symbols)
</details>

![](images/50064641701d95a47337ef38760219e2126bde27527a61781ca0f5015a7102b0.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern building hallway with curved white walls and large windows, no visible text or symbols
</details>

![](images/75265729da5fdc1773b28c3d21b8850bdcf64f6460a33327ac163a664df4d8a6.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern building hallway with large mural and seating (no visible text or symbols)
</details>

![](images/6192be119999dfa613b22ed1414f6afd250fc889ab055178a38fa3b7309f2925.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern classroom with desks, tables, and ceiling lighting (no visible text or symbols)
</details>

![](images/b403a3a52c8c55bac16b33b9ebe80a6ba97a992cacf99c71c8fad1dc84f26bd7.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern living room with sofa, coffee table, TV, and large window (no visible text or symbols)
</details>

![](images/8431b15961228479d661d1e77760bf2ea086bec5cece418f054376814a4cb695.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern room with curved white walls, a large window, and a blue staircase (no text or symbols visible)
</details>

![](images/1da7f662947511ad6f2aa673d206a6c057ed226f76b6080719b9b731c97072f7.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a long, empty corridor with recessed lighting and no visible text or symbols
</details>

![](images/9702a3ef9fc44b2555d7eb1ad7cb52174f4fa824a2dc83c01f548f2dcd0a6530.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern kitchen and living room with furniture and large windows (no visible text or symbols)
</details>

![](images/dd50855c5f391a0060676fac9f9ee30f57e3009d5ae8250dac75083b0b73df32.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern, minimalist library or study room with wooden furniture and bookshelves (no visible text or symbols)
</details>

![](images/f17a5bedd8b73086ab4427ecde882eae1c8bf8bb5d734c16f0da5770d4f632ed.jpg)

<details>
<summary>natural_image</summary>

Interior view of a bedroom with a bed, blanket, and large window (no visible text or symbols)
</details>

Figure 13: Additional results of non-overlapping predictions.

# A.8 INDIVIDUAL FACE OVERLAPS FROM QUALITATIVE COMPARISON

Here, we depict the individual faces generated by our three CubeDiff methods used in the equirectangular panoramas in Figure 4 in the main paper. We show both uncropped and cropped faces as requested by reviewers. Additionally, we show the ground truth panoramas, the individual textual face descriptions. The corresponding input conditioning image is always the first (front) image of the individual faces (and thus equal for all models).

# Oursimg

![](images/060b0cb1aa6a3c83152312e787c7eb0fdfabc0d02bf2c260b1793524de2e930d.jpg)

Figure 14: Our generated faces with input conditioning image only. Top row shows the uncropped faces, bottom row shows the cropped faces.   
![](images/2a96a197e6858189d13986dff0829435c4c3bcfedc42b1e3f4363d2571af9dce.jpg)  
Figure 15: Our generated faces with input conditioning image. Top row shows the uncropped faces, bottom row shows the cropped faces.

# Oursimg+txt

• A concrete stairwell with orange railings leads up to a yellow door with a number 2 on it.

![](images/fbacc95e697418b568ab827c15ad16545161133ff11a39a9b5a4c917adea4125.jpg)  
Figure 16: Our generated faces with single caption input. Top row shows the uncropped faces, bottom row shows the cropped faces.

• A bedroom with a window overlooking a snowy forest, a bed, a desk, and a dresser.

![](images/ec93b87570b528b5a44a9731318d39862320190c60a580d80bec5e0dffdf0535.jpg)  
Figure 17: Our generated faces with single caption input. Top row shows the uncropped faces, bottom row shows the cropped faces.

# Oursimg+multitxt

![](images/cd824a2a22fde5217cbd6f5e28b57fe633da9b295d2599aff7c1d088661e378c.jpg)  
A concrete stairwell with orange metal handrails and steps leads up to a landing with a light on the wall.   
A concrete hallway with a blue floor leads to a corner with a staircase with orange railings.   
A yellow double door with a small window in each door is set in a concrete wall at the end of a hallway with a blue floor.   
A yellow hallway with a fire hydrant and pipes on the wall.   
A concrete room with a gold ceiling and an orange staircase leading up to a white doorway.   
A black circle is in the center of a tiled floor with a yellow square tile in the bottom right corner.

Figure 18: Our generated faces with multi caption input. Top row shows the uncropped faces, bottom row shows the cropped faces.   
![](images/6452f3efed5b9669d6df46cbecafa2b3a5d646d036654c0ace8f0cf11359145c.jpg)  
A window with a view of a snowy forest and lake is the focal point of a bright, minimalist room with a desk and wooden dresser.   
A bedroom with a green and white striped bedspread, a wooden dresser, and white walls.   
A messy bedroom with a bed, a nightstand, a clothes rack, and a drying rack, with a doorway leading to a black and white tiled bathroom.   
A cluttered white bedroom with a desk, mirror, and plants.   
A white ceiling with a single light bulb and a smoke detector.   
A black circle is in the center of a distorted image of a room with wooden floors, a bed, and a blue bin.

Figure 19: Our generated faces with multi caption input. Top row shows the uncropped faces, bottom row shows the cropped faces.

# Ground truth

![](images/d7bfd379ef3b272da2a4e0520f1b1a0636ed4baa7cf0e1adeb690b07c7f992c5.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern concrete tunnel with arched walls, orange railings, and a staircase (no visible text or symbols)
</details>

![](images/2d29b4eafb8b07ff907fd922fc1dcf741262170afd2a60558f068bafdbe14236.jpg)

<details>
<summary>natural_image</summary>

Interior panoramic view of a bedroom with bed, nightstand, and window (no visible text or symbols)
</details>

Figure 20: Ground truth panoramas from the Laval Indoor dataset.

# A.9 VAE RECONSTRUCTIONS

Below, we present pairs of images: the ground truth perspective images with a $9 5 ^ { \circ }$ field of view (FoV) and their corresponding reconstructed images. The reconstructed images are produced by passing the ground truth images through the encoder of our VAE and then decoding the resulting latent representations using the decoder of the same VAE.

![](images/99285f772243310abfefb82262e726efd32a53cb30411809bcac42d65ea921c1.jpg)

<details>
<summary>text_image</summary>

Original
Reconstructed
Original
Reconstructed
1550D
1550D
1550D
GEORG PRETISCH
GEORG PRETISCH
</details>

Figure 21: Examples of ground truth and encoded-decoded perspective images with a FoV of $9 5 ^ { \circ }$ using our VAE. The VAE is capable of reconstructing perspective images with out loss of quality.

# A.10 DETAILED ARCHITECTURE

We illustrate our latent diffusion model in Figure 22. The model’s input is a concatenation of encoded latents, positional encodings, and an input mask indicating the conditioning image. To generate the initial latents, the input image is encoded with a VAE, while Gaussian noise is sampled for the other five faces. The VAE architecture is identical to that of Stable Diffusion’s VAE, with one modification: all GroupNorm layers are replaced with synchronized GroupNorms, where normalization is computed across both the spatial and frame dimensions.

The combined input is downsampled three times to a resolution of $B \times 6 \times 3 2 \times 3 2$ . The first and last blocks of the model exclude attention layers and operate independently on each face. Once the final layer is computed, the output is processed through the synchronized decoder. Notably, except for the GroupNorm layers, all computations are performed per face, with no awareness of the overall panorama structure.

Despite this simplicity, our approach outperforms existing methods, as demonstrated by our results.

![](images/3935ae1396bf038708e3356c234724a6c86fb4597a39243ec9bf0ea76f5ba775.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    A["Input (Bx6x128x128x11)"] --> B["Reshape (Bx6)x..."]
    B --> C["DownBlock\n2x ResBlock\nDownConv"]
    C --> D["UpBlock\n2x ResBlock\nUpConv"]
    D --> E["Output (Bx6x128x128x8)"]
    E --> F["Reshape"]
    F --> G["DownBlock\n2x \nResBlock\nFullAtt\nDownConv"]
    G --> H["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    H --> I["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    I --> J["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    J --> K["MidBlock\n10x FullAtt"]
    K --> L["DownBlock\n2x \nResBlock\nFullAtt\nDownConv"]
    L --> M["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    M --> N["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    N --> O["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    O --> P["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    P --> Q["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    Q --> R["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    R --> S["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    S --> T["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    T --> U["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    U --> V["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    V --> W["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    W --> X["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    X --> Y["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    Y --> Z["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    Z --> AA["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AA --> AB["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AB --> AC["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AC --> AD["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AD --> AE["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AE --> AF["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AF --> AG["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AG --> AH["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AH --> AI["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AI --> AJ["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AJ --> AK["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AK --> AL["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AL --> AM["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AM --> AN["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AN --> AO["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AO --> AP["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AP --> AQ["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AQ --> AR["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AR --> AS["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AS --> AT["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AT --> AU["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AU --> AV["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AV --> AW["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AW --> AX["UpBlock\n2x \nResBlock\nFullAtt\nUpConv"]
    AX --> AY["MidBlock\n10x FullAtt"]
```
</details>

Figure 22: Illustration of our latent diffusion model.

# A.11 ABLATIONS ON PANORAMIC DATA

To evaluate the impact of dataset size on the performance of our method, we conducted an ablation study by training CubeDiff on three subsets of panoramic data: a tiny dataset containing approximately 700 panoramas from the Polyhaven dataset, a medium dataset of about 20,000 panoramas from the Structured3D dataset (the same dataset PanoDiffusion used and comparable in size to MVDiffusion), and a full dataset with over 40,000 panoramas. The results demonstrate that Cube-Diff performs robustly across all settings. Even the tiny model, trained on only 700 panoramas, achieves competitive results, while the medium model closely matches the performance of the full model and significantly outperforms baseline methods in most metrics. Qualitative results further confirm the ability of the tiny and medium models to generate visually consistent and high-quality panoramas, demonstrating CubeDiff’s robustness even with constrained data. These findings indicate that the superior performance of CubeDiff stems not only from data volume but also from the strength of the cubemap representation and its compatibility with pretrained latent diffusion models.

<table><tr><td rowspan="2"></td><td colspan="3">LAVAL Indoor</td><td colspan="3">SUN360</td></tr><tr><td>FID ↓</td><td>KID (×102)↓</td><td>Clip-FID ↓</td><td>FID↓</td><td>KID (×102)↓</td><td>Clip-FID↓</td></tr><tr><td>Text2Light</td><td>28.3</td><td>1.45</td><td>11.5</td><td>60.1</td><td>4.31</td><td>31.3</td></tr><tr><td>PanFusion</td><td>41.7</td><td>2.85</td><td>19.8</td><td>30.0</td><td>1.42</td><td>7.8</td></tr><tr><td>OmniDreamer</td><td>71.0</td><td>5.17</td><td>23.9</td><td>92.3</td><td>8.89</td><td>51.7</td></tr><tr><td>PanoDiffusion</td><td>58.6</td><td>4.08</td><td>26.6</td><td>52.9</td><td>3.51</td><td>28.9</td></tr><tr><td>Diffusion360</td><td>33.1</td><td>2.07</td><td>16.9</td><td>45.4</td><td>3.73</td><td>18.5</td></tr><tr><td>MVDiffusion</td><td>25.7</td><td>1.11</td><td>13.5</td><td>50.9</td><td>3.71</td><td>15.4</td></tr><tr><td>Ourstiny</td><td>27.3</td><td>1.05</td><td>8.8</td><td>41.7</td><td>2.99</td><td>14.7</td></tr><tr><td>Oursmedium</td><td>13.8</td><td>0.66</td><td>8.5</td><td>23.9</td><td>1.28</td><td>10.7</td></tr><tr><td>Oursfull</td><td>10.0</td><td>0.35</td><td>4.1</td><td>24.1</td><td>1.33</td><td>7.0</td></tr></table>

Table 2: Quantitative Ablation on the Laval Indoor and SUN360 dataset. We train a model $( \mathrm { O u r s } _ { \mathrm { t i n y } } )$ on a tiny dataset and another model $( \mathrm { O u r s } _ { \mathrm { m e d i u m } } )$ on a medium dataset

![](images/cdbd148ecf91b70359b007b6bb4c6e5a3532ad1f6b1728f4818ae565a211266d.jpg)  
Figure 23: Qualitative results of the ablated models. Both the tiny and the medium model are able to generate consistent panoramas.

![](images/cca0ffd4eff9ea449ee95c31507fdfe59afa8d75beaaea6e628c3e68167fa0a2.jpg)  
Figure 24: Example of an OOD generation of our tiny (top row) model, medium model (second row) and MVDiffusion (last row).