Maybe someone can help me with this problem: I need to get the image that the ARCamera is showing in order to broadcast it.
So I tried to get the ARVideoTextureHandles from the ARSession interface, after that I use the UpdateExternalTexture method from Texture2D class for copying it, encoding it to PNG and converting it to base64 so I can send it through a network connection.
When I receive that base64 string in the server and decode it, the image is full of gray pixels.
Can someone help me sending the camera image through the network?
So your just trying to broadcast the devices camera feed, not the AR experience? This line from the tutorial.txt file might help you understand why you are seeing those pixels:
ā7. On the main camera for the scene, add the UnityARVideo MonoBehaviour component, and set the clear material in the inspector to point to the YUVMaterial in the project. You can look in the source at what this does: every frame, it takes the two textures that make up the video that ARKit wants to display from the camera, and uses the YUVMaterial shader to combine them into the background that is rendered by the camera. (see UnityARVideo,cs)ā
I think someone has asked this question before on this thread, if you do a little search a suggestion was posted.
Unity camera that is specified in ARCameraManager will always have its transform updated by the input pose provided by ARKit. If you want to move to another area of the scene, you can instead move the rest of the scene in the opposite direction in relation to the camera as has been described in previous posts. You can use other cameras that are not affected by the transform sent by ARKit, but then basically you will not be using ARKit to āsteerā the camera.
Thank you for confirming my suspicions, and I appreciate the feedback. I was hoping I would not have to move the entire scene.
To clarify, if I update the camera in ARCameraManager with SetCamera midway through a play, will it be controlled by the input pose from ARKit? Maybe also reinitialize the tracking session?
We have added support for three features that were missing.
1.) Add/Remove Anchors
2.) Callbacks for ARKit Session Interruption status
3.) A callback for ARKit tracking changed events.
There is a new example scene named AddRemoveAnchorScene, that uses a new component named UnityARUserAnchorComponent. This component adds/removes anchors to ARKit based on the life cycle of the object. This is one way to use the add/remove anchor API, but donāt feel limited by our implementation. If you want to implement your own version, there are methods on UnityARSessionNativeInterface named AddUserAnchor, AddUserAnchorFromGameObject, and RemoveUserAnchor.
ARKit provides callbacks for when a session is interrupted (app going to background/phone call/etc.), and when the interruption has ended. You can now subscribe to these events through the plugin. The events exist on the UnityARSessionNativeInterface and are named ARSessionInterruptedEvent and ARSessioninterruptionEndedEvent.
ARKit also provides a callback for when the tracking state changes. You can now subscribe to the ARSessionTrackingChangedEvent on UnityARSessionNativeInterface.
The asset store package will take a couple of days to update, but you can get it from the bitbucket repository immediately here.
As always, please let us know if you have any feedback. We will be watching the forums.
Thanks for the great work guys! Just voicing as well that Vuforia support would be really appreciated! Weāre a team building a big installation with multiple users walking around in a VR environment. Weāve been doing lots of other kinds of tracking, but the flexibility and ease of implementation of ARkit is really a plus. The only downside is that weāre afraid of users getting sidetracked. Of course weāll give them extra physical space for the expected drifting (this is art Itās not meant to be perfect), but having different areas where a marker could get the scene back on track would just be so useful!
Iāve done a little test of my own in ARkit, by calibrating 3 predefined locations and setting the scene at their average while rotating the scene towards the green location I could get a room mesh to fit quite well (see result around 53sec):
The green mesh is scanned using a Hololens and was just for reference. Iām sorry it probably looks pretty confusing, but it helped getting an idea of tracking quality. You can especially see drifitng along the staircase and upstairs when looking at the pillar. And later when I go downstairs again the 3 locations are slightly misplaced.
On the plus side I learned that a Hololens scan is 1:1 with the ARkit world.
I actually find this challenge super fun Someone mentioned Beakons as well. Iāve also been considering simply having some dedicated Vive tracking areas and then just using the ARkit as a transportation tracking between the Vive areas. So every time the user gets close to the Vive border the ARkit would take over, while at the same time also do a slight adjustment of the world so it aligns againā¦ of course this is not really easy to scale and implement
hi there, i have the same problem previously and here is how i solve it and my guess.
I have two copy of xcode in my mac (8.1 and 9 beta 3), i manually open the Simulator under
ā/Applications/Xcode-beta.app/Contents/Developer/Applications/ā
close it and restart xcode, and everything works.
i guess it is because xcode 9 beta somehow referring to the old xcode 8 path for the simulator thing.
I was trying to build AR Application with Unity-ARKit-Plugin but it failed to compile because Unity uses some Metal classes, but in Simulator mode, only OpenGL is supported.
Errors in ARSessionNative.mm:
CVMetalTextureCacheRef _textureCache; unknown type name and so on
I donāt have a solution, but I have experienced the same issue. Iāve confirmed that I have the code from this commit, but I still find that the far clipping plane is set to 30. I didnāt dig any deeper.
Hi @theiajsanchez ,
It sounds like you may be sending only one of the textures over the network which is a luminance map. Which would be grayscale. One option would be to render it to a render texture, then read the pixels back as RGB, encode as PNG and then send it over the network. The other option is that you could send both textures over the Y and UV, then combine them on the other side of the connection in a shader like we do in the plugin. Let me know if this helps
Cheers,
Chris
Hey @Staus ,
When you say āVuforia support would be really appreciated!ā Do you mean the features that Vuforia has, or the actual Vuforia sdk? Could you expand on that a bit?
Cheers,
Chris
Hi @christophergoy thanks for answering! Can you give us some help with the steps you mentioned? Actually I donāt know how to render the texture into a render texture, neither read the pixels back as RGB.
This is what we tried (without success):
Code
public class SuperControlador : MonoBehaviour
{
private WebSocketController ws;
private Texture2D tex;
// Use this for initialization
void Start ()
{
Application.runInBackground = true;
ws = WebSocketController.GetInstance();
tex = new Texture2D(100, 100);
}
void Update()
{
RenderTexture rt = new RenderTexture(100, 100, 24, RenderTextureFormat.ARGB32, RenderTextureReadWrite.Default);
rt.antiAliasing = 1;
rt.filterMode = FilterMode.Bilinear;
rt.useMipMap = false;
rt.wrapMode = TextureWrapMode.Clamp;
if (rt.Create())
{
Shader.SetGlobalTexture(Shader.PropertyToID("_textureY"), rt);
RenderTexture.active = rt;
tex.ReadPixels(new Rect(0, 0, rt.width, rt.height), 0, 0);
tex.Apply();
ws.SendMessage(Convert.ToBase64String(tex.EncodeToPNG()), "stream");
}
}
}
Hi @TJUnityBuilder ,
The second crash is due to your plugin being out of date. If you update to the latest (either bitbucket or asset store) you should not get the UnrecognizedARTrackingStateReason assert anymore.
For the first crash, is the iOS 11 build older or newer?
Cheers,
Chris
Weāre a team building a big installation with multiple users walking around in a VR environment. Weāve been doing lots of other kinds of tracking, but the flexibility and ease of implementation of ARkit is really a plus. The only downside is that weāre afraid of users getting sidetracked. Of course weāll give them extra physical space for the expected drifting (this is art 
Itās not meant to be perfect), but having different areas where a marker could get the scene back on track would just be so useful!
