# CamFreeDiff: Camera-free Image to Panorama Generation with Diffusion Model

Xiaoding Yuan1,2, Shitao Tang3, Kejie Li2, Alan Yuille1, and Peng Wang2

1 The Johns Hopkins University

2 ByteDance

3 Simon Fraser University

![](images/e4bcfec94d78f385aa0d6432dbb83df2460e074fa878070ef22af2692063f892.jpg)

This is a bathroom with a large mirror and a sink. It has a walk in closet with a wooden door. There is a walk in shower next to the walk in closet and a large window.

![](images/38c5c2f7f103c8e3f9a6176562e03de6ccfe03eb91a489ef803af8c307cceb32.jpg)

<details>
<summary>natural_image</summary>

Interior view of a luxury bathroom with bathtub, shower, and wooden doors (no visible text or symbols)
</details>

This is an empty room with white walk-in closet with a lot of shelves and a wooden floor. There’s a hallway outside with white walls and a white floor.

![](images/2905db480147ad1fac647030b3f2269cf24d68a90dd769f6ca1f49254cd13d88.jpg)  
Camera -free input

![](images/57fce538fa4a0d4d4530d71fe2cdbc468db80eb8e0f43bf9744221d9b202d898.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern kitchen hallway with built-in shelving and doorways (no visible text or symbols)
</details>

Panorama outpainting results   
Fig. 1: Camera-free image-to-panorama generation: our method generates panoramas with reasonable layout and canonical viewpoint with unknown input image camera parameters.

Abstract. This paper introduces Camera-free Diffusion (CamFreeDiff) model for 360-degree image outpainting from a single camera-free image and text description. This method distinguishes itself from existing strategies, such as MVDiffusion, by eliminating the requirement for predefined camera poses. Instead, our model incorporates a mechanism for predicting homography directly within the multi-view diffusion framework. The core of our approach is to formulate camera estimation by predicting the homography transformation from the input view to a predefined canonical view. The homography provides point-level correspondences between the input image and targeting panoramic images, allowing connections enforced by correspondence-aware attention in a fully differentiable manner. Qualitative and quantitative experimental results demonstrate our model’s strong robustness and generalization ability for

360-degree image outpainting in the challenging context of camera-free inputs.

# 1 Introduction

Unlike traditional movies, 360-degree movies create an immersive experience that allows viewers to feel as if they are part of the movies’s environment rather than merely observing it from a fixed perspective. This immersive aspect has been significantly enhanced with the advent and proliferation of AR/VR devices like VisionPro. Creating 360-degree movies typically requires specialized equipment, like 360-degree cameras, making it a highly professional endeavour. An alternative, yet underexplored, approach is outpainting existing movies into 360-degree formats.

Image-based panorama outpainting [2, 2, 4, 14, 27, 29, 32, 33, 35] is a necessary step towards video-based 360 movie creation. The advance of text-toimage diffusion models [9, 17–21] makes it possible to extrapolate an image into 360-degree view. PanoDiffusion [32] proposes a panorama outpainting methods by fine-tuning a pretrained latent diffusion model [20]. However, with a limited amount of training data available, this method disrupts the pre-trained model’s prior knowledge and diminishing its generalization capabilities. MVDiffusion [29], in contrast, maintains generalization by generating multi-view consistent panoramic images using a frozen pre-trained latent diffusion model. This method ensures geometric consistency through correspondence-aware attention, but it requires the input image to have known intrinsic and rotation matrices, limiting its application to panoramas from arbitrary images.

Extending panorama generation to camera-free inputs poses significant challenges. Our method estimates the input camera parameters by estimating the homography transformation from the input image to a predefined canonical view. The homography establishes a correspondence between input views and each panoramic views, allowing for the enforcement of multi-view consistency via correspondence-aware attention [29]. Furthermore, the homographic matrix estimation network is seamlessly integrated with the multi-view diffusion model in a fully differentiable manner. By doing so, the mechanism can effectively mitigate the errors introduced by the homography estimation process.

We fine-tuned our model on high-quality Matterport3D from the pre-trained stable diffusion inpainting model. We demonstrated the strong robustness of our model to random camera parameters. Our model also shows excellent generalization ability when tested on out-of-domain data from Structured3D without fine-tuning.

To summarize, we make the following contributions:

– We propose CamFreeDiff, a panorama generation model capable of handling camera-free input images. To the best of our knowledge, our model is the first of all designed to handle unkown input view and camera parameters.   
– We formulate the camera parameter estimation by predicting the homography transformation from the input image to a predefined canonical view. We

propose a three degrees of freedom(3-DoF) parameterization of homography instead of a standard 8-DoF way in the context of panorama outpainting.

– We propose a novel strategy for incorporating camera prediction into the panorama generation model to enforce multi-view consistency and high visual quality.

# 2 Related Work

Multi-view image generation. The development of large-scale image generation models [17, 20, 21] has paved the way for multi-view image generation [11–13, 23, 24, 28, 29, 31]. A notable advancement in this area is the Zero-1-to-3 model [11], which can deduce novel views of an object given a single image and specified camera pose. Building on this, SyncDreamer [12] introduces a method for generating images at fixed viewpoints through a combination of volumetric representation and epipolar attention, though it occasionally produces artifacts. In an effort to enhance consistency and image fidelity, models such as MVDream [24], ImageDream [31], and Wonder3D [13] have employed comprehensive self-attention mechanisms across all views. Despite their success, these models necessitate extensive fine-tuning of a latent diffusion model’s parameters, which can compromise pre-trained priors. While our method also adopts multi-view generation paradigms, our method is able to outpaint panorama for arbitrary images by freezing the pretrained stable diffusion model.

Panorama generation. Previous methods [1, 4, 25, 26, 29, 32, 34] for generating panoramas typically rely on generative adversarial networks [7], autoregressive [15] or diffusion models. Text2Light [4] emerged as a pioneering approach, utilizing a cascade auto-regressive model for panorama creation, whereas LDM3D [26] and PanoDiffusion [32] have adopted latent diffusion techniques for generating or outpainting panoramas. Despite their innovations, these models often encounter a reduction in their ability to generalize when they are fine-tuned on limited datasets. On the other hand, the process of generating panoramic images inherently benefits from the direct correspondence between different views. Utilizing this, MVDiffusion [29] presents a novel approach by incorporating correspondence-aware attention to maintain multi-view consistency, without modifying the underlying pretrained model.

However, challenges persist with current outpainting techniques, notably their struggle to generate coherent panoramas due to their inability to infer camera parameters. PanoDiff [30] takes a step forward to estimate the longitude and latitude angles of a view, however still assumes camera roll angle and intrinsic. To address these shortcomings, we introduce a novel, fully differentiable pipeline designed to extrapolate from a single camera-free image to a complete 360-degree panorama. This method aims to bridge the gap between existing panoramic generation techniques and the need for accurate, consistent, and high-quality panoramic imagery, leveraging the strengths of direct view correspondence while overcoming the limitations of camera parameter estimation.

![](images/038f00fcf1383a8e855bb2e4c0932122851c9aa7ffe60ae4c9e89cfebdb1e842.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern pool with blue table, TV, and wall art (no visible text or symbols)
</details>

Canonical view

![](images/94aaca4d3d9eaf45a020da8cc6f3d2330ca51c9941748466b9738092f4e03d43.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern meeting room with blue table, wall, and TV (no visible text or symbols)
</details>

$f o v = 6 0 ^ { \circ }$

![](images/3910df4549c24ad4e34d6f4e9f07310c261661893e3b094f79844b51618c1275.jpg)

<details>
<summary>natural_image</summary>

Interior view of a pool table room with monitor, keyboard, and floor area (no visible text or symbols)
</details>

$\phi = 1 0 ^ { \circ }$

![](images/979e6423af6d65d60243d67b999deecec0fcd1f8bd9d7989e44cd6a35fd836a4.jpg)

<details>
<summary>natural_image</summary>

Interior view of a modern pool with blue table, cue stick, and wall art (no visible text or symbols)
</details>

$\psi = 1 0 ^ { \circ }$   
Fig. 2: In the context of panorama generation, we define the camera parameter for a canonical perspective view as $f o v = 9 0 ^ { \circ }$ , x-axis rotation $\phi = 0$ and z-axis rotation $\psi = 0$ . The y-axis rotation can be any θ from $0 ^ { \circ }$ to $3 6 0 ^ { \circ }$ .

# 3 Preliminary

MVDiffusion is a multi-view diffusion model for coherent panoramic image generation. It splits the $3 6 0 ^ { \circ }$ scene into eight perspective views $\mathbf { I } = \{ I _ { 0 } , I _ { 1 } , . . . , I _ { 7 } \}$ at fixed viewpoint, each with $9 0 ^ { \circ }$ horizontal and vertical field of view and $\mathrm { ~ \textmd ~ { ~ } ~ } 4 5 ^ { \circ }$ horizontal overlapping. For image-based panorama outpainting, MVDiffusion employs a latent diffusion impainting model and makes the first view (canonical view), conditional on the input images. Correspondence-aware attention (CAA) blocks are designed to enforce geometry consistency. Consider a pair of corresponding points located at $p _ { s }$ and $p _ { t }$ in the source view $I _ { s }$ and target view $I _ { t }$ respectively, the information is aggregated from a $K \times K$ neighborhood $\mathcal { N } ( p _ { s } )$ centered at $p _ { s }$ in the source view through a cross-attention mechanism:

$$
M = \sum_ {p _ {s} ^ {*} \in \mathcal {N} (p _ {s})} \text { Softmax } \left(\left[ W _ {Q} f (p _ {t}) \right] \cdot \left[ W _ {K} f (p _ {s} ^ {*}) \right]\right) W _ {V} f (p _ {s} ^ {*}), \tag {1}
$$

Where $f ( p _ { t } )$ denotes image feature at location $p _ { t }$ from target view and $f ( p _ { s } ^ { * } )$ represents image feature at location $p _ { s } ^ { * }$ in the neighborhood $\mathcal { N } ( p _ { s } )$ from source view. $W _ { Q } , W _ { K } , W _ { V }$ are the query, key and value projections for cross-attention in CAA blocks. MVDiffusion assumes the camera parameters of the input are the same as the canonical view, and thus cannot handle camera-free image.

# 4 Method

We aim to generate a text-guided panoramic image from a camera-free input, where we assume the camera parameters are unknown. To address this problem, we propose our Camera-free Diffusion(CamFreeDiff) model to predict the camera parameters of the input image by estimating the homography transformation from the input to a predefined canonical perspective view. The predicted homography provides correspondences between multiple target perspective views and the input view, where correspondence-aware attention could be used to enforce geometry consistency for the final panorama generation.

![](images/e72822a617756aeb8a239495d8c6a15633f53b2a8432bd3dc259ca7645ec6a5f.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    A["Input view"] --> B["Homography Predictor"]
    B --> C["fov, φ, ψ"]
    C --> D["H"]
    D --> E["Canonical view 0°"]
    E --> F["+ a series of predefined canonical views"]
    F --> G["pointwise correspondences"]
    G --> H["Input view"]
    H --> I["-45°"]
    I --> J["Canonical view 0°"]
    J --> K["+45°"]
```
</details>

Fig. 3: We formulate the camera parameter estimation as estimating the homography matrix from the input view to a predefined canonical view of the scene. We define the canonical view as the perspective view with an absolute rotation angle of 0◦. We use a 3-DoF parameterization of the homography matrix instead of standard 8-DoF (details in Sec. 4.1).

# 4.1 Homography Matrix Estimation

To generate a panorama based on the input image with an unknown camera, we first estimate the homography $H ^ { 3 \times 3 }$ matrix from the unknown input view to a predefined canonical view. Fig. 2 Shows how we define a canonical view, and we illustrated our method in Fig. 3.

Parameterization of Homography Matrix. Compared to the most straightforward way of parameterizing a homography into elements in the 3x3 matrix, DeTone et al. [5] demonstrated a 4-point parameterization of homography matrix can well reduce the optimization difficulty for a deep neural network, since the 4-points are independent of each other while the 3x3 matrix mixes both the rotational and the translational terms.

In our case, parameters can be further simplified. When expressing homography as $H = \bar { K } _ { 2 } R K _ { 1 } ^ { - 1 }$ , where $K _ { 2 }$ and $K _ { 1 }$ are the camera intrinsics for the canonical view and the input image accordingly, some variables are constant by default for common cameras. Specifically, the intrinsic matrix of a canonical view $K _ { 2 }$ is

$$
\left[ \begin{array}{c c c} f _ {x} & \gamma & c _ {x} \\ 0 & f _ {y} & c _ {y} \\ 0 & 0 & 1 \end{array} \right] = \left[ \begin{array}{c c c} 2 5 6 & 0 & 2 5 5. 5 \\ 0 & 2 5 6 & 2 5 5. 5 \\ 0 & 0 & 1 \end{array} \right] \tag {2}
$$

We also assume the intrinsic of input image satisfies the pinhole camera and thus for $K _ { 1 } ,$ in accord with our canonical view intrinsic, the axis skew coefficient γ for input view also defaults to 0 and principal point offsets $( c _ { x } , c _ { y } )$ default to the centre of the image $( w / 2 , h / 2 )$ .

Accordingly, for our task to generate 360-degree panorama, the homography matrix from input view to the canonical view has three degrees of freedom (3- DoF): 1) camera field of view (f ), 2) camera rotation around the x-axis (ϕ), 3) camera rotation around the z-axis (ψ). Particularly, under the condition of a single input image, predicting the absolute rotation around the y-axis (θ) is considered meaningless since the input view can be mapped to any standard canonical view of a 360-degree panorama with $0 \leq \theta \leq 3 6 0 ^ { \circ }$ .

Therefore, we formulate the model that predicts homography matrix from an input image $I \in \mathbb { R } ^ { H \times W \times 3 }$ to a predefined canonical view as:

$$
\mathcal {M} (I) \rightarrow (f, \phi , \psi) \tag {3}
$$

The input image camera intrinsic $K _ { 1 }$ can be determined by predicted f. Along with known target perspective camera intrinsic and predicted rotation R from $( \phi , \ \psi , \theta = 0 )$ , the homography transformation H can be recovered from the predictions (f, ϕ, ψ).

Homography Estimator We build an MLP classifier with three hidden layers upon a general image encoder. The U-Net encoder pre-trained by stable diffusion model for image generation also serves as our homography estimation image encoder, but with weights frozen for efficiency. Only the MLP classifier is optimized to learn a homography estimator. Feature dimensions for each hidden layer in MLP are set to 5120, 2560 and 1280. SiLU [6] is used as activation functions in MLP block. Cross-entropy loss is applied as learning objectives to fov, ϕ and $\psi ,$ , respectively.

# 4.2 Camera-free 360-Degree Image Outpainting

Homography matrix from the camera-free input image to a predefined canonical image provides pixel-level correspondences between them. Consider the homography transformation from view $I _ { a }$ to view $I _ { b }$ as $H _ { a , b }$ . The projection from a point at location $p _ { a } = ( u _ { a } , v _ { a } )$ in view $I _ { a }$ and its corresponding point at location $p _ { b } = \left( u _ { b } , \ v _ { b } \right)$ in view $I _ { b }$ is formulated as:

$$
\left[ \begin{array}{l} u _ {b} \\ v _ {b} \\ 1 \end{array} \right] = H _ {a, b} \left[ \begin{array}{l} u _ {a} \\ v _ {a} \\ 1 \end{array} \right] \tag {4}
$$

Based on the estimated correspondences, we design three variants to generate 360-degree panoramic images, as shown in Fig. 4.

Variant 1(unwarp image): Initially, the input image is transformed to a canonical view utilizing estimated correspondences, as depicted in Fig. 4(a). Subsequently, we incorporate the latent diffusion inpainting model to generate eight panoramic views (eight diffusion branches with the same weight copy), following the design of MVDiffusion [29] image-based panorama outpainting model.

![](images/81c64298375245b2d4b20490c3ad371f66a405536b6cf5f03a6efd98a68f8f30.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    A["Input image"] --> B["H"]
    B --> C["Rectified image"]
    C -.-> D["replace"]
    D --> E["E"]
    E --> F["Latent Diffusion Denoising"]
    F --> G["D"]
    G --> H["Output image"]
```
</details>

(a)

![](images/e8767824d36331d1d96434cc64ec897c122d2bfc49e7923abb5b8a8a1559124f.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    A["Input latent"] --> B["H"]
    B --> C["Rectified latent"]
    C --> D["replace"]
    D --> E["ε"]
    E --> F["Latent Diffusion Denoising"]
    F --> G["D"]
    G --> H["Output image"]
```
</details>

(b)

![](images/c2552a9c80de300891f52a05d3577304768eeca598119bca29fa79f4223223ce.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    A["H"] --> B["Image Input"]
    B --> C["ε"]
    C --> D["Latent Diffusion Denoising"]
    D --> E["D"]
    E --> F["Output Image"]
    G["provide correspondences"] -.-> C
```
</details>

(c)   
Fig. 4: Different strategies to generate panorama from a camera-free input. After estimating the homography matrix H from the input view to a predefined canonical view, alternatives are: (a) Rectify the input view by unwarping and replacing the original input image. (b) Rectify input view by unwarping and replacing the input latent after image encoding. (c) Provide point-level correspondences to the multi-view generation model to enforce consistency between corresponding points.

For the branch of canonical view 0◦, input comprises a concatenation of noisy latent, the latent of the unwarped image, and a binary mask that identifies the areas requiring inpainting (zero for the visible region and one for the region to inpaint). The inputs for the remaining seven branches consist of the noisy latent, the latent of a purely white image, and a uniformly one-valued mask. It is important to note that the latent diffusion inpainting model is designed to preserve existing image content where the mask value is set to zero, and to generate new content in areas where the mask value is one.

Variant 2(unwarp latent): Unlike Variant 1, which initiates the process by unwarping the input image and subsequently encoding the unwarped image into a latent, in Variant 2, the image is first encoded into a latent space, followed by unwarping this latent into the canonical view 0◦. Aside from this distinction, all other aspects of the design remain consistent with those outlined for Variant 1 as shown in Fig. 4(b).

Variant 3(new view): Variant 3 introduces a novel approach that diverges from the strategies that involve unwarping images or latent employed in variants 1 and 2. This design is structured into one conditional branch and eight generation branches. Unlike Variants 1 and 2, where the canonical view 0◦ branch depends on unwarped images or latent, the branch of canonical view is a generation branch in Variant 3, while the conditional branch depends on the input image. Input settings for other branches align with those specified in Variant 1. Particularly, Variant 3 enforces consistency between views by implementing CAA not only among the generation branches but also between the conditional branch and the generation branchs of canonical views. This strategy enables the effective reduction of the inaccuracies associated with homography estimation. We also illustrate our Variant 3 in Fig. 4(c).

Multi-branch Diffusion Denoising Model   
![](images/1728478aaf3fd069c3a52ae6fc65b501ae8911afe9797150f97879c03c2f0cb5.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    A["Input view"] --> B["U-Net block"]
    C["-45°"] --> B
    D["Canonical 0°"] --> B
    E["+45°"] --> B
    B --> F["× N"]
    G["CAA block"] --> H["Grid of nodes"]
    H --> I["Grid of nodes"]
    I --> J["Grid of nodes"]
    J --> K["Grid of nodes"]
    K --> L["Grid of nodes"]
    L --> M["Grid of nodes"]
    M --> N["Grid of nodes"]
    N --> O["Grid of nodes"]
    O --> P["Grid of nodes"]
    P --> Q["Grid of nodes"]
    Q --> R["Grid of nodes"]
    R --> S["Grid of nodes"]
    S --> T["Grid of nodes"]
    T --> U["Grid of nodes"]
    U --> V["Grid of nodes"]
    V --> W["Grid of nodes"]
    W --> X["Grid of nodes"]
    X --> Y["Grid of nodes"]
    Y --> Z["Grid of nodes"]
    Z --> AA["Grid of nodes"]
    AA --> AB["Grid of nodes"]
    AB --> AC["Grid of nodes"]
    AC --> AD["Grid of nodes"]
    AD --> AE["Grid of nodes"]
    AE --> AF["Grid of nodes"]
    AF --> AG["Grid of nodes"]
    AG --> AH["Grid of nodes"]
    AH --> AI["Grid of nodes"]
    AI --> AJ["Grid of nodes"]
    AJ --> AK["Grid of nodes"]
    AK --> AL["Grid of nodes"]
    AL --> AM["Grid of nodes"]
    AM --> AN["Grid of nodes"]
    AN --> AO["Grid of nodes"]
    AO --> AP["Grid of nodes"]
    AP --> AQ["Grid of nodes"]
    AQ --> AR["Grid of nodes"]
    AR --> AS["Grid of nodes"]
    AS --> AT["Grid of nodes"]
    AT --> AU["Grid of nodes"]
    AU --> AV["Grid of nodes"]
    AV --> AW["Grid of nodes"]
    AW --> AX["Grid of nodes"]
    AX --> AY["Grid of nodes"]
```
</details>

![](images/8c28738fe09a3dd412c71c275b78566afdf9a07cdc898d3e3f0e7a93455496f7.jpg)

<details>
<summary>flowchart</summary>

```mermaid
graph TD
    A["Different groups of corresponding points"] --> B["Correspondence-aware Attention"]
    C["Source (neighbour size K = 3)"] --> D["Target"]
    style A fill:#f9f,stroke:#333
    style B fill:#ccf,stroke:#333
    style C fill:#cfc,stroke:#333
    style D fill:#fcc,stroke:#333
```
</details>

Fig. 5: Our panorama generation pipeline based on multi-view diffusion denoising model. With the predicted homography matrix from the input view to a predefined canonical view, point-wise information can be aggregated from the input view to all target canonical views through correspondence-aware attention(CAA). Note that this figure only shows the cross-attention between one group of corresponding points for clear visualization. However, the same process is applied to all groups of corresponding points.

![](images/42e9878630c5c9b3a1169a859f515f30339f6d9b6a393d30d68709f46a26b840.jpg)  
Fig. 6: Qualitative comparison between different panorama generation strategies. For hard samples, as shown above, The variants involving unwarping image and unwarping latent (Variant 1 and 2) could lead to different issues.

# 5 Experiments

We provide experiment details in this section, including experiment setup (5.1, baselines (5.2), and experiment results (5.3).

# 5.1 Experiment setup

Dataset. We fine-tune our CamFreeView model on the real-world Matterport3D [3] dataset, which contains 90 building-scale indoor scenes with 10,912 highresolution panoramic images. 9820 and 1092 images are split for training and evaluation, respectively, following MVDiffusion [29]. Each room in the dataset provides six distinct non-overlapping perspective images taken from identical

![](images/203c8a326f98b79eaa757e5753d2af4301486d1ebc263e19826a83c5a8ff18f6.jpg)

Fig. 7: Comparison between baseline MVDiffusion and proposed CamFreeDiff. Note that we provide camera parameter estimation results to MVDiffusion since it has not been trained to handle arbitrary camera views. We observe that CamFreeDiff still shows stronger robustness to camera-free input.   
Table 1: We report the panorama generation results of our proposed CamFreeDiff on Matterport3D dataset. For baseline PanoDiffusion and MVDiffusion, we use our homography predictor to rectify the camera-free input image first, and then evaluate their panorama generations only. Therefore, PSNR is not reported for these baselines. 

<table><tr><td>Model</td><td>FID ↓</td><td>IS ↑</td><td>CS↑</td><td>PSNR↑</td></tr><tr><td>PanoDiffusion</td><td>48.7</td><td>3.1</td><td>-</td><td>-</td></tr><tr><td>MVDiffusion</td><td>42.4</td><td>5.4</td><td>21.9</td><td>-</td></tr><tr><td colspan="5">CamFreeDiff</td></tr><tr><td>+ unwarp image</td><td>35.2</td><td>5.5</td><td>23.6</td><td>18.7</td></tr><tr><td>+ unwarp latent</td><td>34.3</td><td>5.6</td><td>22.4</td><td>15.6</td></tr><tr><td>+ new view</td><td>27.0</td><td>5.6</td><td>24.4</td><td>19.3</td></tr></table>

Table 2: We test our model on the Structured3D(S3D) dataset without applying finetuning or domain transfer techniques to demonstrate the generalization ability. Note that PanoDiffusion is trained on Structured3D without text guidance. Therefore, we provide no CLIP score (CS). 

<table><tr><td>Model</td><td>Training</td><td>Cam-free</td><td>FID ↓</td><td>IS ↑</td><td>CS↑</td><td>PSNR↑</td></tr><tr><td>PanoDiffusion</td><td>✓</td><td>×</td><td>35.3</td><td>3.2</td><td>-</td><td>-</td></tr><tr><td>CamFreeDiff</td><td>×</td><td>✓</td><td>31.1</td><td>6.2</td><td>25.5</td><td>19.8</td></tr></table>

camera positions, with each offering a $9 0 °$ field of view. To reach our goal of learning 360-degree image-to-panorama outpainting from camera-free input, we apply a random warp on each perspective image with a field of view from $6 0 ^ { \circ }$ to 110◦ and camera rotations of $\pm 1 5 ^ { \circ }$ to create camera-free images. After random warping, all input images are in $5 1 2 \times 5 1 2$ resolution.

In addition to the primary dataset, we conducted further evaluation of our model on the Structured3D [36] dataset, a photo-realistic compilation of 3,500 indoor scenes encompassing 21,835 rooms, each rendered with panoramic images. Perspective images for each room are also generated from random camera positions and poses. This step aims to rigorously assess the model’s generalization abilities on out-of-domain data. We applied the same random warp approach in Structured3D as in Matterport3D.

BLIP-2 [10] captioning model is used to generate per-view text descriptions for both datasets mentioned above.

Training Details. Our CamFreeDiff model is fine-tuned from the stable diffusion inpainting model. We retain the weights of VAE image encoder/decoder and the latent denoising U-Net blocks frozen as pre-trained. We optimize the MLP block for homography prediction and the CAA blocks for multi-view consistency with a learning rate $2 \times 1 0 ^ { - 4 }$ for 30 epochs.

Evaluation Metrix. Our text-guided 360-degree image outpainting employs a series of standard image generation metrics to evaluate visual quality: Frechet Inception Distance (FID) [8], which quantifies the distance between real and generated images; Inception Score (IS) [22] offering insight into the diversity and quality of generated images; and CLIP score [16] to measure the alignment between text descripition and corresponding images. In addition, we use the Peak signal-to-noise ratio (PSNR) on the corresponding region between the generated and target canonical view $0 ^ { \circ }$ to evaluate our view estimation error. We also use Mean Absolute Error to assess the accuracy of homography estimation only.

# 5.2 Baselines

We consider the following baselines: MVDiffusion and PanoDiffusion.

MVDiffusion is a multi-view text-to-image diffusion model to generate viewconsistent 360-degree scenes. For comparison, our model learns to generate from camera-free input with unknown camera parameters.

PanoDiffusion is designed for RGB-D panorama outpainting with different types of masks. A super-resolution model further enhances the outpainting results with higher resolution.

To the best of our knowledge, no existing panorama generation model was designed or learned to handle input images from different camera poses (rotation) and focal lengths (field of view). Therefore, by default, we provide all baselines with rectified input images using predicted camera parameters by our homography estimator for fair comparison. In addition, we also compare our three variants.

This is a view of a kitchen and living room from a hallway. A potted plant sitting in front of a glass door. The room is empty with white walls and wood floors.

![](images/1fb82a570dbd3d7bb712bc833468b539a1aeaf11e75d14cd95d6fb46ad4db30b.jpg)

![](images/8781e26d8ee740f062a345e731f1e3c24a491bb602546aacef4d509862c059ae.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern building hallway with wooden flooring, large windows, and a potted plant (no visible text or symbols)
</details>

This is a bedroom with a neatly made bed and a mirror on the wall with a white wall and a wooden floor. There’s a living room filled with furniture and a painting on the wall.

![](images/3668d51c89b646581ade0943c70926166dd05c126cdad3f4c8eeb15ae1adf447.jpg)

![](images/4924838d0e209973715090fd8d73f3e0f061fcf72f7977db4a21d49d2e1f0c02.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a bedroom with bed, wall art, and mirror (no visible text or symbols)
</details>

This is a living room filled with furniture and a mirror, and a picture of a painting on a wall . There is a fireplace in the room.

![](images/442f463726200764989fce1ffdcb6618d6b0e58cea4fa23948a4c85fae27d909.jpg)

![](images/8ae712fca9a1d71481846bd133cf78e97b812a117c2bcab498aa094bd96b841c.jpg)

<details>
<summary>natural_image</summary>

Interior and exterior views of a living room with paintings, furniture, and artwork (no visible text or symbols)
</details>

This is a large hotel lobby with a lot of chairs in it. It has a bar and lounge chairs. A lot of artwork on the walls and a lot of light coming in from the ceiling.

![](images/e77d2f80709113cb94d29a56653e71706930a7439b9b4ce7993d962a21cb73bc.jpg)

![](images/81fd0255b37fcc89cc329cbe17ea706483d71e11260044864ce921074784caec.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern lounge or service hall with ambient lighting, featuring a sofa, coffee table, and bar counter (no visible text or signage)
</details>

This is a living room filled with furniture. There is a bed and a flat screen tv. A fireplace in the room and a mirror on the wall. Beside the window there’s a plant in a pot.

![](images/b13f788824aefb9319961d18b524e4854e285dae3b7e7a390e7a36dc620ea947.jpg)

![](images/ac9301d3e9a68e8078f14addf4833668512b8f42ebaa30506272ef1c0c4943ab.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a modern bedroom with bed, wall art, and large windows (no visible text or symbols)
</details>

This is a bedroom with a large four poster bed in it. It is filled with furniture and a fire place. Potted plant is next to sofa and a big window.

![](images/372a1ef8eb010ff6bd62461ece38eb92dc03805c03ce882594a9aed867896dd0.jpg)

![](images/2317039ec7109176d6a544d86fdaded892a261a44d23380bb92a0610979926e2.jpg)

<details>
<summary>natural_image</summary>

Panoramic interior view of a luxurious bedroom with wooden ceiling beams, large windows, and a bed (no visible text or symbols)
</details>

Camera-free input   
Panorama outpainting results   
Fig. 8: More qualitative results of camera-free panorama generation from our Cam-FreeDiff model.

Table 3: Comparison of the architecture design for homography estimator. 

<table><tr><td>Model</td><td>FID ↓</td><td>IS ↑</td><td>CS↑</td><td>PSNR↑</td></tr><tr><td>HomographyNet</td><td>29.2</td><td>5.6</td><td>24.2</td><td>19.2</td></tr><tr><td>SD encoder(frozen) + MLP</td><td>27.0</td><td>5.6</td><td>24.4</td><td>19.3</td></tr></table>

Table 4: Ablation study of different types of homography estimation objectives. Experiments are based on HomographyNet and evaluated on randomly warped images from Matterport3D. Predictions are evaluated in terms of Mean Absolute Error. 

<table><tr><td>Objectives</td><td>fov↓</td><td>phi(φ)↓</td><td>psi(ψ)↓</td></tr><tr><td>MSE (reg.)</td><td>10.6</td><td>2.5</td><td>2.4</td></tr><tr><td>CE (cls.)</td><td>7.9</td><td>1.8</td><td>1.5</td></tr></table>

# 5.3 Results

Qualitative Results In Fig. 7 we show qualitative comparison between baseline MVDiffusion and CamFreeDiff. Both MVDiffusion and CamFreeDiff are trained on Matterport3D dataset. CamFreeDiff is designed and trained to handle arbitrary camera parameters, while MVDiffusion is not. We report qualitative results for different variants of outpainting strategies as introduced in Sec. 4.2. Qualitative comparison for a hard sample is shown in Fig. 6. We observe that the homography prediction error can be propagated into later panorama generation for Variant 1(unwarping image), resulting in inconsistent scene layout. For Variant 2(unwarping latent), inaccurate homography prediction could lead to changed textures for the input view. Compared to the above variants, introducing a new conditional branch for the input view and leveraging correspondence-aware attention(Variant 3) enables the most consistent and robust multi-view generation. We provide more qualitative results for CamFreeDiff in Fig. 8.

Quantitative Results We show the numerical results for panorama generation between baselines and different variants of our CamFreeDiff model in Tab. 1. Among them, CamFreeDiff with Variant 3, treating the input as a new view, achieved the best results in terms of visual quality for panorama generation (FID, IS, CS) and reconstruction quality (PSNR).

To demonstrate the generalization ability of our model, we also tested our model on the Structured3D [36] dataset. Note that our model was never trained on or applied with domain transfer techniques from Structured3D. Results shown in Tab. 2 indicate the strong generalization ability of CamFreeDiff to out-ofdomain data, even surpassing PanoDiffusion, which is trained directly on Structured3D but without learning from camera-free input.

Table 5: Ablation study on the neighbour size K in CAA blocks for information aggregation between two views. 

<table><tr><td>Neighbour Size</td><td>FID ↓</td><td>IS ↑</td><td>CS↑</td><td>PSNR↑</td></tr><tr><td>K=1</td><td>66.4</td><td>5.5</td><td>22.3</td><td>19.2</td></tr><tr><td>K=3</td><td>51.1</td><td>5.8</td><td>23.9</td><td>19.5</td></tr><tr><td>K=5</td><td>48.0</td><td>6.2</td><td>24.1</td><td>19.4</td></tr><tr><td>K=7</td><td>47.3</td><td>6.1</td><td>24.1</td><td>19.4</td></tr></table>

# 5.4 Ablation Study

Homography Estimator We compared classification and regression as different types of homography estimators. Cross-entropy loss is used as the objective for the classifier, and mean squared error (MSE) loss is for the regression model. The classifier gives the best input view estimation results instead of the regression model, as shown in table Tab. 4. In addition, we ablate on the architecture design of the homography estimator. We compared our design of an MLP block built on a frozen stable diffusion image encoder with the HomographyNet, which is designed to predict homography matrix between views. Our designs achieve better generation results as shown in Tab. 3.

Neighborhood Size in CAA blocks Given correspondences between views, Correspondence-aware attention (CAA) aggregates information from source point $p _ { s }$ neighborhood to target point pt, which is the key to yielding consistency between multiple views. The neighborhood in CAA refers to the $K \times K$ neighboring points centered at $p _ { s }$ . We ablate the neighborhood size K for $K = 1 , 3 , 5 , 7$ . From results shown in Tab. 5, we find larger neighborhood size generally leads to better multi-view generation quality, but the improvement is limited. Note that larger K results in more computational and time complexity for CAA operation.

# 6 Conclusion

This paper introduces a novel method for generating panorama from a camerafree input image. We formulate the camera parameter estimation of the input as an estimation of the homography matrix from the input view to a predefined canonical view of the scene. Our method builds upon the MVDiffusion model for multi-view image generation and incorporates correspondences between the input and target canonical views for coherent and consistent panorama generation. We demonstrate the strong robustness of our model to camera-free inputs and generalization ability to out-of-domain data.

# References

1. Akimoto, N., Kasai, S., Hayashi, M., Aoki, Y.: 360-degree image completion by two-stage conditional gans. In: 2019 IEEE International Conference on Image Processing (ICIP). pp. 4704–4708. IEEE (2019) 3   
2. Akimoto, N., Matsuo, Y., Aoki, Y.: Diverse plausible 360-degree image outpainting for efficient 3dcg background creation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11441–11450 (2022) 2   
3. Chang, A., Dai, A., Funkhouser, T., Halber, M., Niessner, M., Savva, M., Song, S., Zeng, A., Zhang, Y.: Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158 (2017) 9   
4. Chen, Z., Wang, G., Liu, Z.: Text2light: Zero-shot text-driven hdr panorama generation. ACM Transactions on Graphics (TOG) 41(6), 1–16 (2022) 2, 3   
5. DeTone, D., Malisiewicz, T., Rabinovich, A.: Deep image homography estimation. arXiv preprint arXiv:1606.03798 (2016) 5   
6. Elfwing, S., Uchibe, E., Doya, K.: Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks 107, 3–11 (2018) 6   
7. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. Communications of the ACM 63(11), 139–144 (2020) 3   
8. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017) 11   
9. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020) 2   
10. Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023) 11   
11. Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., Vondrick, C.: Zero-1-to-3: Zero-shot one image to 3d object. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9298–9309 (2023) 3   
12. Liu, Y., Lin, C., Zeng, Z., Long, X., Liu, L., Komura, T., Wang, W.: Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453 (2023) 3   
13. Long, X., Guo, Y.C., Lin, C., Liu, Y., Dou, Z., Liu, L., Ma, Y., Zhang, S.H., Habermann, M., Theobalt, C., et al.: Wonder3d: Single image to 3d using crossdomain diffusion. arXiv preprint arXiv:2310.15008 (2023) 3   
14. Oh, C., Cho, W., Chae, Y., Park, D., Wang, L., Yoon, K.J.: Bips: Bi-modal indoor panorama synthesis via residual depth-aided adversarial learning. In: European Conference on Computer Vision. pp. 352–371. Springer (2022) 2   
15. Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al.: Conditional image generation with pixelcnn decoders. Advances in neural information processing systems 29 (2016) 3   
16. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021) 11   
17. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical textconditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2), 3 (2022) 2, 3

18. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. In: International Conference on Machine Learning. pp. 8821–8831. PMLR (2021) 2   
19. Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., Lee, H.: Generative adversarial text to image synthesis. In: International conference on machine learning. pp. 1060–1069. PMLR (2016) 2   
20. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022) 2, 3   
21. Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic textto-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems 35, 36479–36494 (2022) 2, 3   
22. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training gans. Advances in neural information processing systems 29 (2016) 11   
23. Shi, R., Chen, H., Zhang, Z., Liu, M., Xu, C., Wei, X., Chen, L., Zeng, C., Su, H.: Zero123++: a single image to consistent multi-view diffusion base model. arXiv preprint arXiv:2310.15110 (2023) 3   
24. Shi, Y., Wang, P., Ye, J., Long, M., Li, K., Yang, X.: Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308.16512 (2023) 3   
25. Somanath, G., Kurz, D.: Hdr environment map estimation for real-time augmented reality. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11298–11306 (2021) 3   
26. Stan, G.B.M., Wofk, D., Fox, S., Redden, A., Saxton, W., Yu, J., Aflalo, E., Tseng, S.Y., Nonato, F., Muller, M., et al.: Ldm3d: Latent diffusion model for 3d. arXiv preprint arXiv:2305.10853 (2023) 3   
27. Sumantri, J.S., Park, I.K.: 360 panorama synthesis from a sparse set of images with unknown field of view. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 2386–2395 (2020) 2   
28. Tang, S., Chen, J., Wang, D., Tang, C., Zhang, F., Fan, Y., Chandra, V., Furukawa, Y., Ranjan, R.: Mvdiffusion++: A dense high-resolution multi-view diffusion model for single or sparse-view 3d object reconstruction. arXiv preprint arXiv:2402.12712 (2024) 3   
29. Tang, S., Zhang, F., Chen, J., Wang, P., Yasutaka, F.: Mvdiffusion: Enabling holistic multi-view image generation with correspondence-aware diffusion. arXiv preprint 2307.01097 (2023) 2, 3, 6, 9   
30. Wang, J., Chen, Z., Ling, J., Xie, R., Song, L.: 360-degree panorama generation from few unregistered nfov images. arXiv preprint arXiv:2308.14686 (2023) 3   
31. Wang, P., Shi, Y.: Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.02201 (2023) 3   
32. Wu, T., Zheng, C., Cham, T.J.: Panodiffusion: 360-degree panorama outpainting via diffusion. In: The Twelfth International Conference on Learning Representations (2023) 2, 3   
33. Ying, Z., Bovik, A.: 180-degree outpainting from a single image. arXiv preprint arXiv:2001.04568 (2020) 2   
34. Zheng, C., Cham, T.J., Cai, J.: Pluralistic image completion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1438– 1447 (2019) 3

35. Zheng, C., Cham, T.J., Cai, J., Phung, D.: Bridging global context interactions for high-fidelity image completion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11512–11522 (2022) 2   
36. Zheng, J., Zhang, J., Li, J., Tang, R., Gao, S., Zhou, Z.: Structured3d: A large photo-realistic dataset for structured 3d modeling. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16. pp. 519–535. Springer (2020) 11, 13