SDOutpainter

Outpaint with stable diffusion inpainting model.
This repository is image generation part of a collaborative drawing application with AI agent.
See front end and back end from here.

Key Feature

SDOutpainter does require prompt and the rims of images that should be the neighbors only.
It creates seamlessly connected image with the limited information.

Demonstration

Input source is 1/8 areas of left image(girl eating pizza) and right image(nurses in hospital) and prompt: "A surreal dreamscape with floating islands, cascading waterfalls into the void, and fantastical creatures soaring through a colorful, ethereal sky."

Challenges I tackled

How to enable seamless inpainting?

Different from usual inpainting, SDOutpainter draws much larger pictures than input image sources.
You cannot expect good performance if you simply create input and mask and execute stable diffusion pipeline.

To enable seamless inpainting, it pre-paint the area to be inpainted.
Workflow is like this.

• Reiterate the periphrals of neighboring images
  Inpainting on empty space does not perform well. So I need to convey the texture of input images. To keep continuity, the arangement of copied images is {original, flipped, original,,,} vertically or horizontally.

• Merge neighbors
  In case neighboring images have different texture, I need to carefully merge the reiterated images to retain each texture even after merged.

I adopted a function to calculate blend proportion.

Note that unavailable input sources which are not passed as arguments should be dismissed.

Here is 3D plot of proportion of left image when left and up has available input. Check out that P ~= 1 when x = 0 (it means w = 0, left side), and P ~= 0 when y = 0 (it means h = 0, up side), and average P ~= 0.5(it means images are merged in balance)

Here are the reiterated images to which the proportion filter is applied. Above is the left image and below is the right image used in demo section.

And here is the merged result. This image have texture of all the input images and the continuity on the edges. Pre-painting will be very helpful for AI's inpainting.