GetRawTextureData() Method resulting in flipped image in RViz

[MrMagicMoz]

I am currently publishing an image to ROS from my unity program, with the hope of tracking an AprilTag in the scene. However when viewing the image in RViz the image is flipped along its x axis and the AprilTag package cannot track the tag unless I flip the tag in the scene. What could be problem here ?

using UnityEngine;
using Unity.Robotics.ROSTCPConnector;
using RosMessageTypes.UnityRoboticsDemo;
using RosMessageTypes.Sensor;
using RosMessageTypes.Std;
using RosMessageTypes.BuiltinInterfaces;
using System.Collections;

/// <summary>
///
/// </summary>

[RequireComponent(typeof(ROSClockSubscriber))]
public class ROSCameraDataPublisher : MonoBehaviour
{
    ROSConnection ros;
    public string imageTopic = "/camera_rect/image_rect";
    public string camInfoTopic = "/camera_rect/camera_info";

    public string CompressedImageTopic = "/camera_rect/image_rect_compressed";

    public Camera target_camera;

    public bool compressed = false;

    public float pubMsgFrequency = 30f;

    private float timeElapsed;
    private RenderTexture renderTexture;
    private RenderTexture lastTexture;

    private Texture2D mainCameraTexture;
    private Rect frame;


    private int frame_width;
    private int frame_height;
    private const int isBigEndian = 0;
    private uint image_step = 4;
    TimeMsg lastTime;

    private ROSClockSubscriber clock;

    private ImageMsg img_msg;
    private CameraInfoMsg infoCamera;

    private HeaderMsg header;

    void Start()
    {
        // start the ROS connection
        ros = ROSConnection.instance;

        if(ros)
        {
            ros.RegisterPublisher<ImageMsg>(imageTopic);
            ros.RegisterPublisher<CompressedImageMsg>(CompressedImageTopic);

            ros.RegisterPublisher<CameraInfoMsg>(camInfoTopic);
            clock = GetComponent<ROSClockSubscriber>();
        }
        else
        {
            Debug.Log("No ros connection found.");
        }


        if (!target_camera)
        {
            target_camera = Camera.main;
        }

        if (Camera.main)
        {
            renderTexture = new RenderTexture(Camera.main.pixelWidth, Camera.main.pixelHeight, 0, UnityEngine.Experimental.Rendering.GraphicsFormat.R8G8B8A8_UNorm);
            renderTexture.Create();

            frame_width = renderTexture.width;
            frame_height = renderTexture.height;

            frame = new Rect(0, 0, frame_width, frame_height);

            mainCameraTexture = new Texture2D(frame_width, frame_height, TextureFormat.RGBA32, false);

            header = new HeaderMsg();

            img_msg = new ImageMsg();

            img_msg.width = (uint) frame_width;
            img_msg.height = (uint) frame_height;
            img_msg.step = image_step * (uint) frame_width;
            img_msg.encoding = "rgba8";

            infoCamera = CameraInfoGenerator.ConstructCameraInfoMessage(target_camera, header);

        }
        else
        {
            Debug.Log("No camera found.");
        }
    }

    private void Update()
    {
        if (Camera.main)
        {
            timeElapsed += Time.deltaTime;

            if (timeElapsed > (1 / pubMsgFrequency))
            {
                header.stamp = clock._time;
                infoCamera.header = header;

                img_msg.header = header;
                img_msg.data = get_frame_raw();
           
                ros.Send(imageTopic, img_msg);
                ros.Send(camInfoTopic, infoCamera);

                timeElapsed = 0;
            }
        }
        else
        {
            Debug.Log("No camera found.");
        }

    }

    private byte[] get_frame_raw()
    {       
        Camera.main.targetTexture = renderTexture;
        lastTexture = RenderTexture.active;

        RenderTexture.active = renderTexture;
        Camera.main.Render();

        mainCameraTexture.ReadPixels(frame, 0, 0);
        mainCameraTexture.Apply();

        Camera.main.targetTexture = lastTexture;

        Camera.main.targetTexture = null;

        return mainCameraTexture.GetRawTextureData();;
    }
}

[mrpropellers]

It’s hard to say for sure why exactly it’s being flipped without knowing more specifics about your platform, but it’s likely due to the coordinate convention of the render texture being different than what RViz expects. This is a pretty common issue due to the differences between OpenGL and Direct3D; more on that here. The brute force fix would be to simply copy all the pixels into a new array, in the correct order, before packing them into your Image message. A more optimal solution would be to write a simple shader like the one in the Unity documentation which flips the image vertically, and then use a Graphics.Blit command to write the flipped texture into a temporary texture that you can read into your message. The latter option would require a bit of research and testing to figure out the exact code necessary to work with the version of our rendering pipelines that you’re using, but doing it in a shader would be much faster than manually copying the pixels around off-GPU.

[MrMagicMoz]

Thanks for your help, the Shader option sounds interesting and I’ll be sure to take a look at it. I did briefly try correcting it with some array operations, which resulted in very low FPS so I don’t think that’s feasible.

[mrpropellers]

Yep manually copying an image pixel by pixel is a costly process - I’d expect it to take on the order of 10-100ms to complete on a 1080p image. This is a pretty common issue though, I’ve even been bit by it on personal projects before, so there is a lot of discussion and suggestions about it elsewhere on the Unity Forums. These are the most useful ones I found from a quick search:

[Custom SRP] How to flip outputs for Render Texture?

For example, how do I flip context.DrawSkybox()? ==== The problem: we render to a custom Render Texture instead of Built-in Render Texture (eg. CameraTarget), we need to somehow flip the unity_MatrixVP so that our skybox is not rendered up-side down. I am aware of an API called GL.GetGPUPorjectionMatrix(), but I am not able to inject that change to skybox pass with cmd.SetViewProjectionMatrices(). To be honest I don't know where DrawSkybox() get its view and projection matrices. Because …

Graphics.Blit flips texture

Hey there, For a project I need to render a game on a very wide but short resolution (5124 x 360), but output that to a regular full HD screen. The game is played on two “ribbons”, each taking half of the vertical space (so 5124 x 180). I wrote a screen space shader that wraps those two ribbons so they fit inside the full HD resolution - so far, so good. However, I also need to use that wrapped output as a texture on a model for testing purposes. I figured I could just blit the output of the c…

You may also want to follow a pattern closer to what’s in the example from the docs for ReadPixels and do your read OnPostRender. Note that you should be able to just do something like

renderTexture.ReadTexture(...);
renderTexture.Apply();
var pixels = renderTexture.GetRawTextureData();

without any of that extra shuffling of active texture references around. If RenderTexture.active is null, it should simply read from the main camera backbuffer (screen).

[maciejw94]

Hi,
I used your script to get the image from unity to ros. However, the image shown in rqt misses a bunch of elements (basically it shows only a fracture of objects in the scene (pictures attached). They are visible in the preview in Unity but not rqt. Do you know what may be the issue?
The same thing happens regardless of which camera and camera position I would use. I also thought that it may be the issue of layers so I tweaked the layers of the objects but it is not that.
Thanks for any tips.
Cheers
Maciej

[maciejw94]

@maciejw94

Hi,
I used your script to get the image from unity to ros. However, the image shown in rqt misses a bunch of elements (basically it shows only a fracture of objects in the scene (pictures attached). They are visible in the preview in Unity but not rqt. Do you know what may be the issue?
The same thing happens regardless of which camera and camera position I would use. I also thought that it may be the issue of layers so I tweaked the layers of the objects but it is not that.
Thanks for any tips.
Cheers
Maciej

following the code, changing the depth of the rendering texture to 8 solves everything