Streaming data (point cloud ) to Unity?


I am trying to tackle 2 problems at the moment and I found no information for either:

  1. How can I stream data to Unity? In this case I want to stream Kinect v2 data, but I am doing some preprocessing on the images that I m getting first, so the question is how do I connect the output of my application to Unity?

  2. The data output type is point clouds. I 've seen some plugins on the asset store, but they are for loading static point clouds (and I don’t get to write it/build it myself that way).

Any tutorials, reading resources, existing projects are extremely welcome. I assume I might have to write a plugin or something, which is way deeper water than everything I 've done so far (simple C# logic games in Unity).

Hey @Merxim - I’m poking about in similar areas at the moment and interested in using Kinect data within Unity for a visualisation/VFX project (although my background is in games).

This is a fair way over my techy/artist head, but there might be some info over here you can kick off with. In short, I think you want to look at the Kinect v2 SDK/Unity examples and fragment shaders.

Check this out too, I can’t find StreamMeUp.unitypackage anywhere, RGBDToolkit seems to have metamorphosed into DepthKit and all the links are broken :frowning: StreamMeUp Raw Pointclouds in Unity3D on Vimeo

There are some other pretty cool Unity examples on the RGBDToolkit Vimeo playlist: RGBDToolkit Projects on Vimeo

Hey I just stumbled on this thread.
I’m attempting something similar but search results are still coming up with very little for this workflow.
Anyone know of any progress made in this area? Specifically the network live streaming of point-cloud data.
Is this possible?

See example with Realsense camera.

  1. Grabbing Realsense data (native c++)
  2. HEVC Main10 hardware encoding (native c++)
  3. streaming (native c++)
  4. receiving (Unity native plugin)
  5. hardware decoding (Unity native plugin)
  6. unprojection (Unity native plugin)
  7. wrapping point cloud in native buffer (Unity, no data copied)
  8. rendering (Unity shader)

The video, the unity side code and the encoder side code.

There are more relevant links in video description.

@steve-o616, @triff, @Merxim

Finished my proj eventually, I did use Kinekt but with Brekel Point Cloud and 3ds Max/Redshift instead of the Unity/UE approaches I was exploring back in 2017. Too Many T's - P A T T E R N S (Official Video) - YouTube