Alpha ignored in reference images used to train style?

Archived Forums Muse
, ,
1

I’ve just started with the beta. The first thing I tried to do was use a set of around 15 sprites we use in one of our projects to train a style. The sprites look good when rendered with transparency, but the images in the “training set” area appear not to be using the alpha, thus revealing a load of trash pixels that are normally hidden. First of all - and my primary concern - is that this makes me wonder if that rubbish data is used in the training data set? If so, I suppose it is liable to result in a bad model, as that data has nothing to do with what I want to train. I suppose there could be a use case where that data was significant, so I guess ideally it should be optional whether to include those masked pixels in the training data.

My secondary concern (if the primary concern is invalid) would simply be to ensure that the transparency is rendered correctly in the training set, to avoid confusion.

2

An example to illustrate the issue. This is how one of our icons appears in the preview window:

image
image914×436 53.2 KB

But in the training set it shows up like this:

image
image612×608 53.4 KB

3

It’s not that it’s ignored, I think it’s some sort of display glitch. I saw a similar effect when I tried to remove background from an image with the help of neural network. I too had an attempt to train a drawing style on my images and found that due to multiple redraws, I left small pixels in the final image around. It seems the neural network is trying to incorporate them into the training set and it’s interfering with it. I wrote a script that runs through the texture and removes this noise. I think this has improved the result. If I used this trained style when generating with background removal enabled, it did pretty well. Another thing is that I trained it on images with different themes, I think if I trained it on images with only skeletons for example, it would give better results. Now it looks like Mayan drawings :smile:
Noisy training source:
image
Training result:
image
Cleared training source:
image
Training result:
image
Generation by text prompt with trained style:
image
image
image
image

And yes, for some reason sometimes I get a blank image. I think he’s afraid of the word scribble or doesn’t understand it because of the small training set. :face_with_diagonal_mouth:

4

To be clear, the display problem is definitely because it is ignoring the alpha in the display. If I change the import settings for the sprite to ignore alpha, the sprite shows exactly how it also shows when used as a training image - but as a training image it shows that way even if the alpha is not set to be ignored in the import settings. My concern is that the garbage data that ‘should be’ hidden by the alpha may also being used to train the style, which I would expect to cause problems with training.

5

If I understand you correctly, you took an image with alpha enabled (import settings) and added it to the training set. Then you disabled alpha in the import settings of the original image and expect it to change in the training set as well? But it doesn’t. As far as I understand the mechanism, Unity creates a copy of the file when you add it to the training set, it is not a reference to the original project file. And yes, you seem to be right, in the question that the neural network ignores alpha data in training and only uses rgb channels.

6

If I understand you correctly, you took an image with alpha enabled (import settings) and added it to the training set. Then you disabled alpha in the import settings of the original image and expect it to change in the training set as well?

No, that is not what I mean. I take an image with alpha - the one shown in my first image - and add it to the training set. But it shows up in the training set as shown in my second image. This is the same as it shows up if I change the import settings to Alpha Source => None (instead of Input Texture Alpha). I don’t expect the training data to notice this if I change it after adding the image (I was just trying to illustrate that the garbage that can be seen is verifiably what is contained in the source image transparent pixels).

What I expect is:

  1. The RGB color data for pixels that have alpha==0 will be rendered as transparent in the training data set
  2. The RGB color data for pixels that have alpha==0 will be ignored when training the dataset, as they are not relevant to the final image and are likely to be distracting.

I can see that expectation 1. is definitely not being met, and I fear that expectation 2 may also not be.