Fundamental Flaw in Unity's Mip Maps and Atlased Textures

[LaireonGames]

TLDR: Can we have access to UnityEngine.DDSImporter, and if so would it crash if i removed the last few mip map levels?

Hey folks, so as in the title. Basically Unity does not support mip mapping along side models that use a texture atlas. I have tried contacting support about this issue but they recommend I post here (which I disagree with but here it goes).

Ok so the problem, lets start with some pictures:

So this is my terrain for a minecraft engine, it generates meshes and populates each blocks texture from a texture atlas, pretty standard practise. As you can see here, its looking fine at this distance, you can see some artifacts that even ansio level 13 can’t remove but those I can deal with. Lets zoom out:

Ok already its look pretty warped and distorted, note that I have a mip map bias of -5. This is ten times what Unity recommends! However this I can also deal with by making a custom DDS file and blurring this mip.

However lets zoom further:

Now the terrain is just garbled. What is going on here is I have hit mip map 8 and the textures are bleeding. No amount of padding can over come this! Last image:

Utter distorted madness. What’s going on? Well its actually really simple.

What is happening is I am using block textures that are 128 by 128 and these can be chopped in half 7 times so the final mip is a pixel. Awesome mip mapping doing its job. But I place these textures into a 1024 by 1024 block atlas, with a 28 pixel padding to avoid bleeding. The problem here is that the 1024 texture can be chopped 11 times. So after mip map 7 the base textures no longer fit as one pixel, they bleed into each other because there is no more data but Unity keeps generating mip maps until the file is a pixel.

So to me there are two solutions. Either I fill mip maps 8 > 11 with mip map 7. I lose a little peformance by drawing the bigger textures but my terrain looks nicer. Or I delete these mips entirely.

Both solutions are currently impossible. If I try to do either Unity will automatically generate the remaining mip maps again, if they are gone they get recreated, if they are larger than expected they are automatically shrunk.

It looks like my solution is to make a custom texture importer and override this functionality (however I doubt the validity of this as I imagine if the mip maps are not what unity expects I will get some nasty crashes/errors). I can’t do that however, I can see if I try to import my DDS file it fails to import with the standard texture importer since there is a class for that called: UnityEngine.DDSImporter. However attempting to create an instance or cast to this type results in compiler errors.

To me this is a pretty fundamental design flaw! Does anyone have any suggestions on how to overcome this? If not please bump this thread in the hopes a Unity dev see’s it to comment.

Cheers!
Jason

[LaireonGames]

I hit the picture limit above so using this second post to show the atlas I have used to take these screenshots and the mip levels causing the problem.

Atlas:

You can see here the huge padding to prevent bleeding. This also shows why the distrotion is blue since I have repeated a lot of the water texture for this example.

Here is mip 7, where the blocks are nice and clearly 1 pixel each (I have enlarged the following images to make it easier to see what I mean):

There is still some bleeding but with a custom mip map I can easily deal with this. E.G that bottom right pixel is pretty grey, clearly a merge of what should be brown with the blue of the water above it. But This I can edit to be brown manually.

However this next image is mip 8, where its no longer possible to keep the textures in neat 1 pixel blocks so they merge:

With this, I cannot possibly edit the map to define a pixel for each texture. I can turn the entire image grey but it has its own problems and look gross. This is also only mip 8! It gets smaller and worse with mips 9,10 and 11 where the entire atlas is supposed to be represented by 1 pixel.

[bgolus]

Unfortunately Unity does not provide an easy way to access the texture settings beyond those presented in the texture importer window (what shows when you select a texture asset). Excluding the bottom mips, or using alternate forms of supported DXT textures (like DXT1A), aren’t really possible… except you can compress them externally and import in a DDS file directly. The disadvantage is of course you no longer get the automatic texture handling for different platforms. I’ve not used this myself so I don’t know what kind of gotchas this might add, like I don’t know if the quality settings will adjust the max texture size or not, but it should let you exclude the bottom mips.

Another option is to exclude the mip in the shader. You would have to calculate the mip level yourself and use tex2DLod or tex2Dgrad, but then you’ll have absolute control and still have the benefit of Unity’s tools for compression. It also has the benefit of working on pretty much every platform. It does mean a mildly more complex shader.

The best solution however is to not use an atlas at all, but use an array texture. Then you don’t have to worry about mip levels or padding or keeping UVs in a 0.0-1.0 range.

Unity - Scripting API: Texture2DArray

[LaireonGames]

Hey bgolus thanks for the reply.

I’ve not properly looked into Texture2D arrays, I noticed them in the released notes but in all honesty wasn’t entirely sure of the significance of them! My only concern is do you think a Texture2DArray would support an array of DDS files? I do still want custom mip maps for some of my textures to smooth them out.

Otherwise certainly worth looking into thanks

[hippocoder]

Texture arrays were designed to solve these problems. They also allow for many other great things like tiling them.

[bgolus]

Texture2DArray does not support DDS, but that’s because DDS is a container file type and not an image format. However any image format that can be in a DDS that Unity supports as a Texture2D can be in a Texture2DArray. Also because of the way a Texture2DArray is created in Unity you have full, explicit control over what is in the mip maps… but you can’t clamp the min mip level.

The best way to think of Texture2DArray is each element is the same thing as a normal Texture2D and you don’t have to do anything special with. It’s the same thing as if you had an array of Texture2Ds, it just takes a little more work than setting an array of floats.

The main things with Texture2DArray are every texture in the array must be the same size and format, and all share the same sampler filter settings. You also have to construct them yourself either at runtime or in editor as Unity doesn’t have any built-in automation for them (yet?). They’re also not available for DX9 or OpenGL ES 2.0, so if you’re looking to support that you’ll still need the atlas technique as a backup.

[LaireonGames]

Sweet that makes perfect sense thanks for that

[LaireonGames]

That has worked absoutely perfectly thanks so much! Still feels like Unity didn’t properly consider mip mapping with texture atlases until 5.4 which is a little concerning but oh well, fixed now I guess.

[AcidArrow]

Is there any performance overhead when using a Texture2DArray? I’m guessing they need an extra texcoord to store the index, but other than that they should be as costly as sampling a normal texture, right? (and if that’s the case, being able to combine tiled materials should offer a good performance boost)

[bgolus]

I don’t know of any performance issues with Texture2DArray over a common Texture2D atlas. It’s likely the array is slightly slower to sample, but any costs are going to be offset by having simpler shaders and being able to use fewer vertices (because you don’t have to clamp to the bounds of an atlas tile either via the shader or by breaking up mesh UVs).

For the texcoord Unity already treats texcoords as Vector4/float4 values, and shaders internally always pass a float4 worth of data from the vertex to the pixel shader for each vertex output semantic regardless of the data type you specify, so you can just store the full float3 (uv & index) in the mesh, pass that to the pixel shader, and sample the texture array directly for zero additional cost from data transportation over the usual float2 based UV.

[LaireonGames]

I can confirm there is certainly no performance issues with the array. Within my engine it feels a little quicker now and I can generate 11 Million voxels at 40FPS (in the editor) so definitely impressed with its performance overal and no issues with it.

I can’t remember any exact readings before the change for a good comparison though sorry.

[hungrybelome]

@bgolus

For the texcoord Unity already treats texcoords as Vector4/float4 values, and shaders internally always pass a float4 worth of data from the vertex to the pixel shader for each vertex output semantic regardless of the data type you specify

Hi, is this still true? For surface shaders and vert/frag shaders?

[LaireonGames]

Yup, its still Vector4, seems to be the case for everything except vertex data

[bgolus]

There’s two parts to this.

On the Unity application side, everything can be a Vector4 internally, except the normal (which is always a Vector3) and the color which is a Color32. When storing as assets / packaging for the build, different amounts of compression can be applied which affects how the data is stored, and when sending to the GPU some data will be sent as lower precision forms. Color again is always sent and stored as 4 bytes, which matches the internal Color32 representation. Similarly if no shader uses all channels of a texcoord, Unity can send only the “used” channels to the GPU. So you can set a Vector4 on a UV set, or two Vector2 UV sets with no performance difference.

Once the data gets read by the shader, things change. Data semantics passed between the shader stages are always float4s when using Direct3D. This is a graphics API thing, not anything to do with Unity. OpenGL, as well as Vulcan, Metal, and Direct3D 12, don’t have this same limitation. OpenGL auto packs data of variable component sizes, and this happens automatically even when writing your shaders in HLSL. Other APIs also allows a much broader range of data types that can be passed between shader stages. But those are harder to make use of with Unity as you’re more likely to be writing your shaders in the default Direct3D 9 or 11 style HLSL.

[hungrybelome]

@bgolus

OpenGL auto packs data of variable component sizes, and this happens automatically even when writing your shaders in HLSL.

Hi @bgolus , thanks for the explanation! So even if I am writing HLSL surface shaders, if I target OpenGL, declaring float1/2/3/4 UVs in V2F are actually treated differently, and not just as float4?

[LaireonGames]

@bgolus

except the normal (which is always a Vector3)

Oh yeah forgot about normals!

[bgolus]

@hungrybelome

So even if I am writing HLSL surface shaders, if I target OpenGL, declaring float1/2/3/4 UVs in V2F are actually treated differently, and not just as float4?

Correct. Direct3D will expand those to float4s in the compiled shader, but the HLSL to GLSl converter that Unity uses will compact them back down, and then OpenGL packs that data for you. Basically if you have two float2 texcoords in Direct3D, each TEXCOORD will use a full float4. In OpenGL it’ll only pass a single float4 and split them back up afterward.

AFAIK internally on the GPU hardware the data is always passed as 4 component vectors. So if you have a shader that’s passing a float4 position, a float3 normal and a float2 UV, on both Direct3D and OpenGL that uses 3 float4s (as it’s 9 floats total). Some GPUs might be different, so I could be wrong here, but that’s my current understanding. My knowledge of how mobile GPUs handle data internally is also a bit weak here, so it might not be true there either.

[hungrybelome]

@bgolus

So if you have a shader that’s passing a float4 position, a float3 normal and a float2 UV, on both Direct3D and OpenGL that uses 3 float4s (as it’s 9 floats total).

Awesome, thanks for the detailed explanations!

[Prodigga]

We’re running into mipmap issues with atlased textures, but our atlases are lightmaps generated by unity! We don’t have the luxary of using 3d textures, since it is just a blackbox. Any solutions out there? Just reimplement lightmap sampling code but use Tex2dgrad and clamp manually?