Correct video format for video playback ?

Unity Engine
1

Hi,
thanks for that video feature, it’s actually pretty cool.

Anybody can tell me how to encode videos to have them with sound? What are supported formats and codecs for this new videoplayer?

Thanks in advance

2

Also, any chance of it working with the Hololens live preview feature?
(as described here: How to share your new reality with the rest of the world | by Dan Thompson | Perficient Digital Labs)

Tried to make it work under the trivial example, and nothing shows up (nothing complains either) but I haven’t spent hours fiddling with it.

3

Hi CatherineP,
Please keep different issues in separate threads. Otherwise they become hard to manage.

4

Hi komodor,
You can find more information about the VideoPlayer and other features here:

https://forum.unity3d.com/threads/preview-of-some-5-6-features.445778/

5

This is actually the same question: which codec does the feature support? (although I should have phrased it more precisely)

I’m not sure if the Hololens is using H.264/AAC for its live streaming, or if it’s a proprietary codec. But given Unity’s positioning on the VR market, I’m hoping someone in the video team has considered that use case and can give me the answer.

6

ty!

edit: actually there’s not much information about it, just the usual stuff like - do it our way it’ll work … however when i use embeded video, there’s no sound, when i use URL (what is exactly what i want), there’s no sound … if i play that video in system videoplayer, it has sound, the unity videoplayer component can see the track (shows 2 channels) …

so i am expanding the question to general:

what has one to do to be able to play video with sound in unity 5.6 ?

7

See this thread.

8

oh, TY!

9

Hi all,

Thanks for having a look at our new VideoPlayer and giving feedback!

As you are realizing, full documentation has yet to be done so in the mean time it’ll be my pleasure to assist.

About codecs:

komodor, your question about supported formats is a frequent one and the answer is multifacteted. Generally speaking, the supported codecs are:

  • H.264 for video with AAC for audio, typically found in .mp4, .m4v and .mov files.
  • VP8 vor video and Vorbis for audio, typically found in .webm files.

These are the codecs that will work in most situations, both in the editor and in the target platform. H.264 is often the best choice as it is supported in hardware on most platforms - with Android also supporting VP8 in hardware. However, we’re trying to have support for whatever the editor platform supports. For instance:

  • On Windows, we’re using Media Foundation for reading/transcoding movie files.
  • On OSX, we’re using AVFoundation for reading/transcoding movie files.
  • On all editor platforms, including Linux, we have a software version of the VP8 video codec and Vorbis audio codec.

So in theory, all formats handled by these APIs are supported in the editor. I’m saying “in theory” because we do have some filtering based on filename extensions that may go away in the future (not sure yet). At this time, here’s the list:

  • All editor platforms: ogv, vp8, webm
  • Additionally, on Windows and OSX: mov, dv, mp4, m4v, mpg, mpeg
  • Additionally, on Windows: avi, asf, wmf

We will try to lift some restrictions and simply let the platform’s native APIs tell us whether a given file format is supported, but this ties into how Unity’s asset pipeline works so we cannot simply claim that every filename extension “might be a movie file”, hence the current limitation.

All target platforms have their own different capabilities that we’d like to encode in the editor so that you don’t have to keep track of them. Each one has different resolution, bitrate, codec, profile and audio restrictions and recommendations. Using Android as an example, the official recommendations are Supported media formats  |  Android media  |  Android Developers. But every Android device / model / OS version has its own specifics so it wouldn’t be appropriate for us to impose limitations based on the general recommendations. Some Android devices like the Nvidia Shield TV have no problem playing 4K videos. We could emit warnings when a movie is found to be “out of spec”, at best.

Then comes the discussion about whether you’ll ask the movie to be transcoded during import. When the importer transcodes the movie, it will encode it in either H.264 or VP8 based on what the target platform is know to support best (this is the “Auto” mode). For instance on iOS, like on many platforms, this would be H.264 but on Linux this would be VP8. Again here, feedback is incomplete so you don’t get to see what “Auto” translates into for the target platform; this will come. You can also override the codec choice if you have a good reason. A good reason could be: on Windows, H.264 works in hardware most of the time, but on certain Windows versions it won’t work above HD. Forcing the codec to VP8 in this case means we’ll revert to the software codec, which works identically on all Windows variants, but comes at a higher CPU usage cost. There’s also the reality that transcoding takes time, and often a huge amount of it depending mostly on resolution. Furthermore, our current asset pipeline doesn’t support aborting the transcode task, which is a (known) major flaw. We’ll try to address this throughout the beta cycle.

So if you decide to not transcode your movie file, then you’re essentially telling Unity that you know what you’re doing and that you know the movie in its current format is supported on the target platform. This is the default choice when you import a movie, which one may find questionable since it may lead to surprises. But given the time it takes to transcode most videos, we still prefered to have this as the default than to send Unity into a uninterruptible transcode each time movie assets are added/reimported.

About error reporting:

As CatherineP points out: error reporting is most of the time absent; we are aware and are working on this. The console messages, when present, are vague at best but this will be improved. One thing that makes good error reporting harder is that many of the APIs we are using will only report errors asynchronously. The request to start decoding a given video will succeed, but a background event gets fired later on telling us about a failure. So one cannot rely on proper exception throwing to catch errors. This is why the scripting API has a “errorReceived” C# event: we hope to funnel as many error descriptions through there as possible, as well as through console logging.

About audio output:

Audio can be read from all supported movie files with the notable exception of .ogv. This has just been completed internally and will appear in a later beta version.

It’s understandable that many will find the audio output hard to figure out initially as the implementation is not yet completed. Here’s the general idea behind the audio output modes:

  • Audio Source: This causes the movie’s audio to be sent to the specified AudioSource(s): one audio source per multi-channel audio track in the movie. This means the movie’s audio can be processed through Unity’s audio engine alongside any other sound in the game. The AudioSource you assign to the track can then be configured normally for sending the samples to the wanted mixer group and all AudioSource controls will work as expected. When an AudioSource is used this way, its AudioClip field can still be used and both the AudioClip and the VideoPlayer’s contribution will be mixed together.
  • Direct: Some APIs - notably AVFoundation on OSX - make it possible to output audio directly from the movie decoding stage. It seemed convenient to expose this capability if all that is needed is to output the audio from the movie without further processing from Unity. This is what the “Direct” mode is about. It will not be possible to support this on all platforms, but we may end up simulating it for completeness sake, and also because this would still represent an economy over forcing the user to have an AudioSource object.

About HoloLens:

And to answer CatherineP’s question about HoloLens specifically: no work has been done to support this platform yet. The current platform lineup is: Windows, OSX, Linux, Android, iOS, WebGL, Tizen and (in the next beta) XboxOne. Although one of the long term goals is to support better the VR workflows, this initial release is more about general hardware-assisted video playback than specifically about VR features/platforms. I’ve asked a developer on the HoloLens team to see if he can get videos to play in there. We’ll add adequate error reporting if needed, or provide a fix if no major development is required. Thanks for trying it out and bringing this to our attention!

Again, thanks for experimenting with this and for your patience while we’re working on the remaining bits and pieces!

Dominique

3 Likes
10

Thanks a lot, Dominique! (and I maintain that you are my favorite technical comment writer ever :slight_smile: )

(Just to be clear, I’m trying to play the output of the Hololens video feed on a Windows platform. But I understand that it might not be part of the initial scope… it would just be very nice if it came to be eventually. I’ll be happy to help if you need a guinea pig)

11

Thanks for that Video Player, I have been waiting for so many years.
First impression is very good. Got 10 Videos running simultaneously on 3xUHD Displays.
Performs beautifully…

12

Thanks for your good words Studiomaurer!

We’re also interested in hearing what you have to say when things work as expected! So if you can let us know what platform and types of videos you’re using, other users will be able to refer to your experience to know what they can expect!

13

CatherineP : I just educated myself on the type of url that is needed to read the stream from the HoloLens, using instructions from this web page : Streaming HoloLens Video to Your Web Browser – Kloud Blog

The type of URL that has to be used (e.g. taken from the page: https://[USERNAME]:[PASSWORD]@192.168.1.6/api/holographic/stream/live_high.mp4?holo=true&pv=true&mic=true&loopback=true ) may very well trip our internal logic. As soon as we can spend time on the device, we’ll see if this is the case.

Another thing that we didn’t have time to try yet is “live” streams, i.e. streams for which there is no known duration. This also could be problematic. At this point, I feel we could continue chatting in another thread (if you don’t mind opening one) as this doesn’t seem codec-related and we’ll be able to have HoloLens-specific discussions. It is hard to get a confirmation that the stream really is using H.264, but the fact that the filename uses the .mp4 extension is a strong hint. I went from the Microsoft doc for the HoloLens portal: Device portal API reference - Mixed Reality | Microsoft Learn

I’ll let you know when we have time to experiment on our side!

1 Like
14

@DominiqueLrx , are you sure that the new VideoPlayer works on Android devices? It crashes on my Android device. You mentioned that Android is supported. If so, then I have to file for a bug report.

15

Hi @hangemhigh , yes it does work for the devices we were able to test with. Please file a bug giving the details of what you are doing that leads to a crash as well as the full device description so we can reproduce and fix.

Sorry it’s not working for you; hopefully we’ll get you a fix quickly!

Dominique

16

I will file a bug report today. This is my 4th bug report for Unity 5.6 beta about the VideoPlayer. Hats off to you and other Unity engineers for bringing us the new VideoPlayer API. That’s what I’ve been waiting for and it works on my Windows. Just few bugs that needs to be fixed.

17

Is the Alpha channel supported in vp8? or in any other? Thanks

18

Do you need more bug reports on this? I experience it crashing on Android as well when starting to play the video, while it runs perfectly on Windows if you do the same things. Here’s the logcat from when I press the button to start the video:

12-22 14:25:41.772 12441-12452/? V/VRManagerServer: (VRManagerService.java:443) yTqX [setThreadSchedFifo pkg=dk.vendeval.world pid=14975 tid=15017 prio=0]
12-22 14:25:41.772 3471-3513/? V/VRManagerService: ---- demoteProcessThread 'dk.vendeval.world' requestPriority(pid=14975, tid=15017, prio=0 ----
12-22 14:25:41.782 12441-12514/? V/VRManagerServer: (VRManagerService.java:443) yTqX [setThreadSchedFifo pkg=dk.vendeval.world pid=14975 tid=15062 prio=0]
12-22 14:25:41.782 3471-4493/? V/VRManagerService: ---- demoteProcessThread 'dk.vendeval.world' requestPriority(pid=14975, tid=15062, prio=0 ----
12-22 14:25:46.202 3153-3153/? A/DEBUG: pid: 14975, tid: 15017, name: UnityMain  >>> dk.vendeval.world <<<
12-22 14:25:46.242 3153-3153/? A/DEBUG:     #01 pc 005e9560  /data/app/dk.vendeval.world-1/lib/arm/libunity.so
12-22 14:25:46.242 3153-3153/? A/DEBUG:     #02 pc 005eafc8  /data/app/dk.vendeval.world-1/lib/arm/libunity.so
12-22 14:25:46.242 3153-3153/? A/DEBUG:     #03 pc 005ea098  /data/app/dk.vendeval.world-1/lib/arm/libunity.so
12-22 14:25:46.252 3153-3153/? A/DEBUG:     #04 pc 005e7f78  /data/app/dk.vendeval.world-1/lib/arm/libunity.so
12-22 14:25:46.252 3153-3153/? A/DEBUG:     #05 pc 009b8dfc  /data/app/dk.vendeval.world-1/lib/arm/libunity.so
12-22 14:25:46.252 3153-3153/? A/DEBUG:     #06 pc 009b6204  /data/app/dk.vendeval.world-1/lib/arm/libunity.so
12-22 14:25:46.252 3153-3153/? A/DEBUG:     #07 pc 009b6038  /data/app/dk.vendeval.world-1/lib/arm/libunity.so
12-22 14:25:46.252 3153-3153/? A/DEBUG:     #08 pc 0081912c  /data/app/dk.vendeval.world-1/lib/arm/libunity.so
12-22 14:25:47.212 3471-15652/? W/ActivityManager:   Force finishing activity dk.vendeval.world/com.unity3d.player.UnityPlayerActivity
12-22 14:25:47.332 3471-3585/? W/ResourcesManager: getTopLevelResources: /data/app/dk.vendeval.world-1/base.apk / 1.0 running in null rsrc of package dk.vendeval.world
12-22 14:25:47.382 3471-4009/? I/WindowState: WIN DEATH: Window{99561d u0 d0 dk.vendeval.world/com.unity3d.player.UnityPlayerActivity}
12-22 14:25:47.412 3471-3585/? D/ISSUE_DEBUG: InputChannelName : 50d8153 Application Error: dk.vendeval.world
12-22 14:25:47.422 3471-4805/? I/ActivityManager: Process dk.vendeval.world (pid 14975)(adj 0) has died(247,1310)
12-22 14:25:47.432 3471-4805/? D/ActivityManager: isAutoRunBlockedApp:: dk.vendeval.world, Auto Run ON
12-22 14:25:47.462 3471-3585/? D/StatusBarManagerService: manageDisableList userId=0 what=0x0 pkg=Window{50d8153 u0 d0 Application Error: dk.vendeval.world}
12-22 14:25:47.502 3471-3585/? V/WindowStateAnimator: Finishing drawing window Window{50d8153 u0 d0 Application Error: dk.vendeval.world}: mDrawState=DRAW_PENDING
12-22 14:25:47.652 3471-4805/? E/WindowState: The process of this window already died: Window{99561d u0 d0 dk.vendeval.world/com.unity3d.player.UnityPlayerActivity} : android.os.DeadObjectException
12-22 14:25:47.652 3471-4805/? W/WindowState: Failed to report 'resized' to the client of Window{99561d u0 d0 dk.vendeval.world/com.unity3d.player.UnityPlayerActivity}, removing this window.
12-22 14:25:47.752 3471-3585/? V/WindowStateAnimator: Finishing drawing window Window{50d8153 u0 d0 Application Error: dk.vendeval.world}: mDrawState=DRAW_PENDING
12-22 14:25:47.792 3471-3585/? V/WindowStateAnimator: Finishing drawing window Window{50d8153 u0 d0 Application Error: dk.vendeval.world}: mDrawState=HAS_DRAWN

It crashes the app during this.

19

That’s not how to file for a bug report. From Unity Editor, go to Help->Report a Bug… Then report the bug.

@DominiqueLrx

Updated to b2 but this bug is still not fixed.

20

Which android device are you using? I tried nexus 5 and samsung galaxy S6 but both didn’t work. They are not crashing, but also don’t run. I have a scene with empty plane or sphere (I tried to use different objects). I tried puting videos directly on VideoPlayer from assets folder, then loading from StreamingAssets and Resources, but nothing works. I tried changing codec to H264. I tried also to stream the video from the link I got in latest documentation, but also no luck. So basically my questions are:
Which device did you use?
Which folder you keep the video in and do you specify the codec?
Is there something specific you need to do to run android because I it works perfectly on iphone and windows.

Thank you very much for your time and I really apreciate your help.
Leo