1

I tried to use KerasCV Stable Diffusion API to run inpainting many times. However the generated image always come with a noisy margin on both the upper border and left border of the image. I only use default parameters to run the API functions. Please see attached the code I used, the seed and the output image.

May I know how I can clear the noisy margins?

import keras_cv
import numpy as np
import tensorflow as tf
from PIL import Image, ImageDraw
import numpy as np
from tensorflow import keras
import matplotlib.pyplot as plt

stable_diffusion = keras_cv.models.StableDiffusion()

def plot_(images):
    plt.figure(figsize=(10, 10))
    for i in range(len(images)):
        ax = plt.subplot(1, len(images), i + 1)
        plt.imshow(images[i])
        plt.axis("off")

def create_mask(img):
     w, h = img.shape[0], img.shape[1]
     x = np.roll(np.roll(np.array(img), h // 2, 0), w // 2, 1)
     mask = Image.fromarray(np.zeros_like(x)[:, :, 0])
     draw = ImageDraw.Draw(mask)
     coords = [(w / 2, 0), (w, h / 2), (w / 2, h), (0, h / 2)]
     draw.polygon(coords, fill=255)
     return mask    

 ### Seed Image: im
 size = (512, 512)
 im = im.resize(size)
 im = np.array(im)
 mask = create_mask(im)
 mask = np.where(np.array(mask)==0, 1, 0) 
 mask = np.expand_dims(mask, axis=0)
 im = np.expand_dims(im, axis=0)

 generated = stable_diffusion.inpaint(    
     f" pig on cart",
     image=im, 
     mask=mask,
 )

 plot_(generated)

Seed: https://c0.wallpaperflare.com/preview/87/385/209/man-riding-on-the-skateboard-photography.jpg

Output: Output

Artico
  • 11
  • 2
  • you may generate a bigger picture and then crop margins – Konstantin Purtov Mar 19 '23 at 08:50
  • @KonstantinPurtov do you mean this is a known problem already? – Artico Mar 19 '23 at 08:58
  • No, there is no problem like that exist in base kerasCV StableDiffusion model. but you not sharing your settings at all. please update question with more details about your setup(include "seed" to reproduce it). otherwise you can use my previous recommendation. – Konstantin Purtov Mar 19 '23 at 11:05

1 Answers1

0

This is due to a bug with the KerasCV StableDiffusion VAE Encoder that we thought we resolved last week with https://github.com/keras-team/keras-cv/pull/1582

As it turns out, our fix broke inpainting so we'll need to start over, but there is not currently anything wrong with your code, it's an issue with KerasCV's encoder.

Ian Stenbit
  • 46
  • 4