I’m creating a WebGL application that needs to play extremely large videos at high playback speeds. Playing the videos by URL isn’t possible since they start to lag heavily with normal internet connections around 1.5x playback speed. I attempted to put the video clips in an asset bundle but apparently WebGL does not support playing them back from asset bundles that are downloaded. The application cannot have any access to the local storage, it only has access to memory so everything needs to be done in memory.
I’m currently using a web request to download the entire video when it’s needed and then convert it to a byte array. I’ve tried using libraries like AVPro Video 3 but it no longer supports playing from memory stream in Unity and LibVLCSharp which does not support WebGL and stops builds from succeeding. I’m currently looking into FFMpegCore which can receive streams as an input but I’m confused how to extract frames from it or to play that back as a video. FFMpegCore has a snapshot function but only accepts a local file path. I’m hoping to either find a way to extract frames 30 times per second and output them to a render texture, or to play the byte array as a video to a render texture. Audio isn’t currently an issue, just the video.
If there are ny other ways to play a video in WebGL without building it into the application, not accessing local storage and still being able to achieve at least 4x playback speed on videos around 25000kbps please let me know.
so the video is still small enough that you can load it fully into memory?
should it this work for all users over internet, or the video could be on a local server?
(with possibly faster to streaming/loading)
are there any javascript video players that can do it?
then you could open div on top of webgl and play video there… (instead of trying to do it inside unity webgl)
So essentially how it works is that we start with a video which is like 20GB in total and then split it up into segments that are each around 2GB and store them all in an AWS S3 bucket in their own asset bundles as VideoClips. The application downloads the first segment and the user waits for that then the video plays. While that segment is playing the next one loads in the background and if the user finishes their current segment fast enough then they wait for the download of the next one to finish. When they start the next segment the previous one is discarded so there is only ever like 4GB being used to store the video but the playback speeds can go up to about 10x with a 4K video at 25000kbps beofre it lags. It works perfectly on Windows.
Currently the videos are stored in an S3 bucket so they can be accessed from any network but if it’s completely necessary we could try to do it with local servers. We prefer the S3 bucket since the users will be on many different networks and getting them to do anything locally other than typing a url into their browser is going to be extremely difficult (many different companies, users aren’t always going to be on their companies’ networks, and the companies are unlikely to accept any local software / data storage).
I can see there are ways to do it in JavaScript but the videos are a part of a 3D environment in the application. I don’t think it would be feasible to put the 3D environment and other UI elements on top of the div that would play the video in JavaScript
yeah… could have a look at some javascript ways, if js would have the frame data available,
pass that into unity in chunks… and draw to texture yourself. (probably hefty overhead again…)
could also try this if nothing else comes up,
(i haven’t tested it myself, saw it in forums few years ago)
That’s 25 Mbit/s, or possibly 100 Mbit/s at 4x speed, is that correct?
Have you considered that this may not even be possible to achieve for most users simply due to their Internet speeds not achieving those 25 / 100 Mbit/s?
Since you seem to be tailoring this app for corporate use, it’ll be more difficult to assess what Internet speed each individual users may have. But you can make a best guess. Say a company has 1,000 Mbit/s Internet for 1,000 employees on that location and 10% of employees are actively using the web at any given time (eg most staff is on the assembly line), then each will have roughly 10 Mbit/s available to them. Not unusual since most staff only browse the web or use an online tool, not streaming videos. Of course this varies greatly per company/location.
Consumer streaming services (eg Netflix) work with bitrates below 10 Mbit/s even for 4k content, perhaps some go slightly above 10 but certainly less than 20.
My hunch is that you’re working on an business app where the customer may have too high expectations when it comes to web streaming, or the requirement of using the existing videos over the web was simply accepted as a given and now you work with that requirement. Perhaps the best option is really to recompress those videos from 20 GB down to 2 GB in a modern, web-streaming capable format and accepting a lower quality.
Why is the 4x speed so important? Perhaps a scrubber or “skip to chapter” feature would be preferable.
Even without audio, who is watching videos at 4x speed? This must be either very boring, drawn out content or someone simply requested to have a 4x option because they often use this on their VHS (they never use Youtube so they aren’t aware of what we call “scrubbing”).
I’m just saying this because this is the kind of oddball staff requests I tend to hear from non-media companies. And if your company simply accepted that request, and now you’re faced with a highly challenging technical requirement that may not even make much sense in the first place, perhaps the best course of action is to talk them out of it. Usually this works without much fuzz.
The internet speed limitaiton is why I’m not using the URL function of the video player. It’s just never going to be able to play back videos of this size at higher speeds without dropping frames and lagging. The videos are downloaded completely before they are played and the user waits for the download to finish. But after the download finishes and the video is stored in memory there isn’t any limitation from the internet speed anymore. The video can be played back as fast as the hardware will allow it without dropping frames or lagging. This works perfectly fine on Windows but WebGL doesn’t support playing videos in the video player from anything except a URL. So what I’ve got is a video stored as a byte array in memory and no way to display it in WebGL which is what I’m trying to solve.
As for why it’s important, basically it wasn’t anyone who requested this. The videos aren’t just for watching, they are part of something interactable and the playback speed is essential to that. The experience is so much better when the playback speeds can gohigher. Currently with a good ethernet connection its only possible to get like 1.5x playback speed on a video of 10000kbps. Testing on Windows where we download the videos into memory first we can get above 10x playback speed on good hardware which makes the experience much better. We decided to do this because it makes our application so much better and are trying to find any way to make it work on WebGL since that’s the target platform.
I’m trying to find a way to play a video from a memory stream in WebGL
Why do the videos need to be this large? Did you evaluate various encoding options to find the best tradeoff for size, quality and performance?
This is crucial because the encoding matters a lot, more so on the web.
It would help if you could open the video in a player and find the section that shows all the video encoding details and post that here. Explorer right-click and choosing Properties would also reveal at least the basics of the video format.
The videos need to be this large for the quality. We have tested many different encoding options and resolutions and decided that 4k with 25000kbps was the best quality for our use case. That’s beside the point though, I’m now trying to see if it’s possible to play a video in Unity from a byte or binary stream or buffer since it would improve the application
I’ve just finished trying this idea. I tried downloading the video in the JavaScript and playing it on a video player that was hidden on the document then every frame using a canvas to capture what the video player was showing and convert it to a Base64 string. The string got sent back to C# in Unity and it converted it to a texture.
Unfortunately it’s too inefficient. With a 1080p test video I could only get like 15fps at best
base64 seems inefficient, surely could pass data some other way? (byte arrays directly and marshal if needed), maybe some code here Send byte array from js to Unity
*but honestly like codesmile said,
would “random” user notice difference for example with 25000kbps vs 20000kpbs ?
if the video is in 3d world also, i would except not noticeable difference… (with even lower rates or resolutions…)
What video container format and encoding are you using? If it is something natively supported by the browser you could try to encapsule it in a Blob and create an object URL to play it back.
Something like this could work(haven’t tested it this, consider it as pseudo-code):
mergeInto(LibraryManager.library, {
// Create an object url from video data in a buffer that can be played back by the browser
GetVideoUrl: function (ptr, length) {
var videoData = HEAPU8.subarray(ptr, ptr + length);
// Create a blob with objec url from data
// Change mime type depending on actual video format.
var blob = new Blob([videoData.buffer], { type: "video/mp4" });
var url = URL.createObjectURL(blob);
// Convert JS string to C string to return it
var bufferSize = lengthBytesUTF8(url) + 1;
var buffer = _malloc(bufferSize);
stringToUTF8(url, buffer, bufferSize);
return buffer;
},
// Use this method to clean up url after video is no longer needed
FreeVideoUrl: function (url) {
URL.revokeObjectURL(UTF8ToString(url));
}
});
using UnityEngine;
using UnityEngine.Video;
using System.Runtime.InteropServices;
public class NewBehaviourScript : MonoBehaviour
{
public VideoPlayer videoPlayer;
[DllImport("__Internal")]
private static extern string GetVideoUrl(byte[] ptr, int lengh);
[DllImport("__Internal")]
private static extern void FreeVideoUrl(string url);
void Start()
{
// Create an object url from raw video data and play it back with Unity video player
var url = GetVideoUrl(videoData, videoData.Length);
videoPlayer.url = url;
videoPlayer.Play();
}
I finally got it working. Updated the GetVideoUrl function to the following
mergeInto(LibraryManager.library, {
// Create an object url from video data in a buffer that can be played back by the browser
GetVideoUrl: function (ptr, length) {
var videoData = HEAPU8.subarray(ptr, ptr + length);
// Create a blob with objec url from data
// Change mime type depending on actual video format.
var blob = new Blob([videoData], { type: "video/mp4" });
var url = URL.createObjectURL(blob);
// Convert JS string to C string to return it
var bufferSize = lengthBytesUTF8(url) + 1;
var buffer = _malloc(bufferSize);
stringToUTF8(url, buffer, bufferSize);
return buffer;
},
// Use this method to clean up url after video is no longer needed
FreeVideoUrl: function (url) {
URL.revokeObjectURL(UTF8ToString(url));
}
});
