Hi all! It’s been a while since I’ve announced a new API, but I’m really excited for this one. I’d like to introduce NatReader API. NatReader is a lightweight video decoding API. I began working on NatReader because I had use for a transcoder. My other API, NatCorder, already handles the encoding aspect of transcoding. NatReader completes the other half.
Another goal of mine with NatReader, one which I need help from the community to achieve, is to create a video player plugin that works and works well for everyone. Unity’s VideoPlayer has bugs here and there; and AVPro Video is prohibitively expensive. Hopefully, this is a start to a complete featureset.
NatReader Features Include:
Extremely easy thread-safe API. Use a foreach loop to access frames in a media file.
Support for decoding on iOS, Android, and macOS (editor and standalone). WebGL and Windows coming soon.
Support for decoding audio for full video playback
I would love to hear any and all the feedback you have. What features should we look into? What should we prioritize? Let’s create the best solution for video decoding in Unity Engine together.
Hello @Lanre . Downloaded the project from GitHub. Didn’t change a thing (besides opening it with Unity 2018.3.14f1 instead of 2018.3.13f1). The example scene when played in Editor just shows white rectangle and gives this error:
The Windows backend isn’t complete. On Android, you have to pay special attention to the path of the MP4 file being decoded. Check the logs for info on why it isn’t working.
Thanks for the answer. I attach the complete log of what I’m getting on Android from the demo scene. The project is exactly as it is on GitHub, didn’t change a thing, except for slight Unity version difference, but I don’t really think there should be much difference between 2018.3.13f1 and 2018.3.14f1. Correct me if I’m wrong.
Caused by: java.lang.NoSuchMethodError: No static method copyFrame(Ljava/nio/ByteBuffer;IIILjava/nio/ByteBuffer;)V in class Lapi/natsuite/natrender/GLBlitEncoder; or its super classes (declaration of ‘api.natsuite.natrender.GLBlitEncoder’ appears in /data/app/com.jabii.jabiiapp-1/base.apk:classes2.dex)
at api.natsuite.natreader.MP4FrameReader$3.onImageAvailable(MP4FrameReader.java:138)
at android.media.ImageReader$ListenerHandler.handleMessage(ImageReader.java:658)
at android.os.Handler.dispatchMessage(Handler.java:105)
at android.os.Looper.loop(Looper.java:156)
at android.os.HandlerThread.run(HandlerThread.java:61)
@Lanre I’m using NatReader to generate video thumbnail.
My code:
using (var reader = new MP4FrameReader(galleryPath + child.GetComponent<VideoItem>().name))
{
Texture2D frameTexture = new Texture2D(640, 1386, TextureFormat.RGBA32, false);
frameTexture.LoadRawTextureData(pixelBuffer);
frameTexture.Apply();
ShowToast(frameTexture.width.ToString());
int width = frameTexture.width;
int height = frameTexture.height;
var size = width < height ? width : height;
Rect rect = new Rect(0, 0, size, size);
Texture2D t2d = TextureTools.CropWithRect(frameTexture, rect, TextureTools.RectOptions.Center, 0, 0);
child.GetComponent<RawImage>().texture = t2d;
yield return eof;
Destroy(frameTexture);
break;
}
}
I’m reading only the first frame from the reader. I have multiple videos with different resolutions. My doubt is, it possible to get video height and width from the returnded framebuffer?
2020-06-07 12:23:47.361487+0530 ios[2281:1008812] NSURLConnection finished with error - code -1002
2020-06-07 12:23:47.365437+0530 ios[2281:1008069] NatReader Error: MP4FrameReader failed to find video track for asset at ‘/var/mobile/Containers/Data/Application/0752FA99-154B-4FD9-8E9D-939B6887AAED/Documents/files/tmp/recording-1591512349.mp4’
note: The video is present in the path and it is being played by a video player. This error is not occurring in android.
The download finishes with an error code, so the downloaded file is most likely corrupt. This is most likely the cause of the issue. Resolve that and try again.
I’m not downloading the video. I’m recording the video using natcorder and I’m using Natreader to get thumbnail.
more Logs:
→ applicationWillResignActive()
2020-06-07 17:39:56.105978+0530 ios[2332:1055535] FCM: Connected to FCM
→ applicationDidBecomeActive()
2020-06-07 17:39:59.116250+0530 ios[2332:1055535] NatCorder: Prepared MP4Recorder video encoder at resolution 840x1120@25.000000Hz with average bitrate 5909760 and keyframe interval 3s
2020-06-07 17:40:03.910160+0530 ios[2332:1055535] NatCorder: MP4Recorder finishing
2020-06-07 17:40:14.433993+0530 ios[2332:1055535] [Snapshotting] Snapshotting a view (0x12c8237f0, UIInputSetContainerView) that has not been rendered at least once requires afterScreenUpdates:YES.
2020-06-07 17:40:17.692982+0530 ios[2332:1056335] NSURLConnection finished with error - code -1002
2020-06-07 17:40:17.697945+0530 ios[2332:1055535] NatReader Error: MP4FrameReader failed to find video track for asset at ‘/var/mobile/Containers/Data/Application/BE18848E-CC26-4ED6-AB8C-F6543F1F0C7E/Documents/files/tmp/recording-tmp.mp4’
Then why is the video in tmp? NatCorder never saves to a temp folder. Are you modifying NatCorder to save to a different location? Also, you’ll need to prepend the “file://” protocol to your path when creating the frame reader.
Been using NatCorder for our project and have really enjoyed it! Currently trying to figure out a way to merge more than one .mp4 file together with it’s audio. Would love to see audio decoding in the future. I assume ffmpeg is my best option for now.