Battery-preserving low frame-rate but with immediate new frame upon input

[runevision]

This is a follow-up to discussion in the welcome thread about preserving battery by allowing longer frames when there’s no input without getting lag when there is. Taking it here to avoid hi-jacking that thread any longer.

We have talked some more internally about what we could potentially do. We are thinking about this approach:

• Implement a way to reduce the frame rate when there’s no input but get a new frame immediately when there is.
• Implement this functionality only for touch input for a start. It will treat all touch input as events that can interrupt sleep and only touch input. We may expand it with other event types in the future if there is demand for it.

We’re considering different approaches for what the API would be like.

Potentially:

• int Application.targetFrameRate
• int Application.targetSleepFrameRate
• bool Application.sleepAndWakeOnEvent

Or alternatively:

• float Application.minFrameTime
• float Application.maxFrameTime
• bool Application.sleepAndWakeOnEvent

If we at one point support handling different event types than touches for this, we may add API to specify what types of input should unblock.

Example use case:

You’d set targetFramerate to a normal active rate like 30 fps and targetSleepFrameRate to 1 fps. Then, when nothing needs animation etc. and you just want to wait for the next input, you set sleepAndWakeOnEvent to true.

If there’s input, it will trigger the next frame. If the input causes an animation or similar to start you can set sleepAndWakeOnEvent to false. Then when the animation has finished, set it to true again.

Questions for you:
A) Should input always halt a wait immediately, or still ensure a minimum frame time?

If you have set sleepAndWakeOnEvent to true so you only want a new frame quickly if there’s input, do you want:

• To immediately halt the sleep, even if it results in a frame that’s shorter than the normal targetFrameRate?
• To use the normal targetFrameRate to ensure the sleep is halted as soon as a normal frame length has passed?

The former can result in very fast frames during a swipe. If sleepAndWakeOnEvent is set to false as soon as the swipe starts, most of the frames will have normal length, but the initial one could still be shorter.

The latter guarantees a minimum frame time (max frame rate) even when there’s input.

B) For your own current use cases, would you need sleep to resume on any other types of events than touch events?

And if so, which types of input is that? What game is this for? (platform, brief description of style and interaction, etc.)

[Ferazel]

Thanks for making a thread about it. I’m very happy that we’re having a discussion about it.

From what you said, I think it makes more sense to keep the targetFrameRate naming convention.

A) In terms of expected behavior of the sleepAndWakeOnEvent, I feel that an interrupted frame would be better. It would reduce the latency, and the impact on battery life would be minimal. The user is unlikely to notice the uneven frame pacing. The user is also unlikely to trigger significant qty of input frames with touchscreen input. If the game is going into these sleep framerates, I would expect game already is in a state where a normalized frame time isn’t going to be important. I expect only the first frame to have an “incorrect” frame time and every subsequent frame would rely on the targetFrameRate.

B) I am OK with just touch to start, but there are other input mechanisms that I want to be able to register depending on the application. Bluetooth controllers, keyboards, mice (Android), and Apple pencils to name a few. In terms of other systems that would use this functionality, the only other system that I could even potentially think might want to force a frame update would be networking. However, there is such an embedded latency already associated with networking that I don’t think it would be a problem to delay until the next sleep frame.

The rest of my feedback below is mainly just semantics about the API. At the end of the day, I’ll take any API you can give us to give us the behavior we’re looking for.

To me, having a separate sleep frame rate variable or bumping back to targetFrameRate after the first wake event assumes behavior from game logic. I’d argue that setting the target frame rate is more indicative of what you directly want the application to do. When I tell the targetFrameRate to go down I am telling the application to sleep, when I tell the targetFrameRate to go higher I am telling the application to wake-up. It is unlikely that we would be doing this throughout the entire application and have a centralized MonoBehaviour to control it or give what our desired sleep frame rate so storing that frame rate at the application side seems weird. Unless there is some concern with setting the target frame rate frequently or just triggering one frame. I’m not sure I see the point of adding another field on Application. I would argue that just giving the managed scripting an immediate frame should be enough for us to determine if it is input in game logic that would trigger “a wake up” and we would set the targetFrameRate with game logic instead of relying pre-defined Application’s logic. Sure we could enforce our sleep frame rate again by setting the bool with your API, but why have the Application change its frame rate state without being told to explicitly by the game logic scripting? This also muddies the source of truth. Instead of having a single variable for what is the current application render time, you need to check the bool that would then need to check another int field.

I do agree with having a bool flag, but again to match the previous point something like “triggerFrameOnWakeEvent” would fit better. We are designating the type of event that we can define an API for later. For example, the later API it could be something like RegisterFrameRateWakeEvent() and the purpose of this event would be clearly associated with this “WakeEvent” term we’ve defined for this system.

Thanks for your consideration of this functionality. If there is anything else I can do to help clarify our use case, let me know.

[greggtwep16]

@runevision

This is a follow-up to discussion in the welcome thread about preserving battery by allowing longer frames when there’s no input without getting lag when there is. Taking it here to avoid hi-jacking that thread any longer.

We have talked some more internally about what we could potentially do. We are thinking about this approach:

• Implement a way to reduce the frame rate when there’s no input but get a new frame immediately when there is.
• Implement this functionality only for touch input for a start. It will treat all touch input as events that can interrupt sleep and only touch input. We may expand it with other event types in the future if there is demand for it.

We’re considering different approaches for what the API would be like.

Potentially:

• int Application.targetFrameRate
• int Application.targetSleepFrameRate
• bool Application.sleepAndWakeOnEvent

Or alternatively:

• float Application.minFrameTime
• float Application.maxFrameTime
• bool Application.sleepAndWakeOnEvent

If we at one point support handling different event types than touches for this, we may add API to specify what types of input should unblock.

Example use case:

You’d set targetFramerate to a normal active rate like 30 fps and targetSleepFrameRate to 1 fps. Then, when nothing needs animation etc. and you just want to wait for the next input, you set sleepAndWakeOnEvent to true.

If there’s input, it will trigger the next frame. If the input causes an animation or similar to start you can set sleepAndWakeOnEvent to false. Then when the animation has finished, set it to true again.

Questions for you:
A) Should input always halt a wait immediately, or still ensure a minimum frame time?

If you have set sleepAndWakeOnEvent to true so you only want a new frame quickly if there’s input, do you want:

• To immediately halt the sleep, even if it results in a frame that’s shorter than the normal targetFrameRate?
• To use the normal targetFrameRate to ensure the sleep is halted as soon as a normal frame length has passed?

The former can result in very fast frames during a swipe. If sleepAndWakeOnEvent is set to false as soon as the swipe starts, most of the frames will have normal length, but the initial one could still be shorter.

The latter guarantees a minimum frame time (max frame rate) even when there’s input.

B) For your own current use cases, would you need sleep to resume on any other types of events than touch events?

And if so, which types of input is that? What game is this for? (platform, brief description of style and interaction, etc.)

Your alternative with floats for the framerate times is a bit more flexible. 1 FPS saves a decent amount of battery but I would think that anyone doing that actually wants “sleep forever until woken up”. Floats would allow you to enter .5, .25, etc. That being said keeping the ints for backwards compatibility and wrapping them in an enum with one entry for “sleep until input” or something similar.

Responses to specific feedback requested.

A) This depends at what rate your input gathering piece is running at, but assuming its running faster than the framerate then no it should not interrupt the targetFrameRate (normal fast framerate). It should only interrupt the sleep framerate when the bool is active. It’s a bit redundant though the bool should only be active when we’re ready to sleep. I’d expect Unity to set it back to false and our normal framerate when it detected input (meaning the bool wouldn’t be active when running at the fast framerate). Let’s say the user was checking a box, that triggered an animation to play we wouldn’t set it back to true until that was done. Even if the user touched part of the screen we didn’t care about, I’d still expect to have to query the input and determine that and set it back to true.

B) My current use case is mostly mobile but it would help out on laptops as well. I’m making a gcode editor for 3d printing modifications and virtual rendering of imported files with different materials (so you don’t have to waste filament if your slicer messed up). On mobile touch would be all that I need. On laptops the same would be the case just monitoring keyboard keys and the mouse instead (although a unity app takes over an hour to kill a laptop battery, where many times it can take under 45 minutes to kill a mobile battery on some smaller phones).

[greggtwep16]

@Ferazel

I do agree with having a bool flag, but again to match the previous point something like “triggerFrameOnWakeEvent” would fit better. We are designating the type of event that we can define an API for later. For example, the later API it could be something like RegisterFrameRateWakeEvent() and the purpose of this event would be clearly associated with this “WakeEvent” term we’ve defined for this system.

If I’m understanding this right (a callback specifically called when interrupted) I guess this would be an alternative to Unity always setting the sleep bool back to false when it detects input to interrupt sleep, I’m not sure if I’d like it better or not. Having the frame forced is already going to run my whole game loop once with all my other scripts an logic for what to do when buttons were hit, etc. So I already should have a mechanism to decide whether I cared about any of the interrupting input and either sleep again or go to the fast framerate and play an animation or something else. I’d probably prefer that Unity just set the bool back to false every time and leave it up to me to turn it on again.

If you’re just talking about specifying which mediums will invoke the interrupt then I misunderstood.

[superpig]

@greggtwep16

It should only interrupt the sleep framerate when the bool is active. It’s a bit redundant though the bool should only be active when we’re ready to sleep. I’d expect Unity to set it back to false and our normal framerate when it detected input (meaning the bool wouldn’t be active when running at the fast framerate). Let’s say the user was checking a box, that triggered an animation to play we wouldn’t set it back to true until that was done. Even if the user touched part of the screen we didn’t care about, I’d still expect to have to query the input and determine that and set it back to true.

So this suggests another possibility for the API - we could expose a method, maybe something like:

Application.ReduceFrameRateUntilWakeEvent(float frameRate, WakeEventMask wakeEvents = WakeEventMask.AnyInput);

where “frameRate” is the reduced frame rate you want the engine to use, and “wakeEvents” specifies the kind of events that can wake it up (which maybe for now would just be “any input”, but perhaps someday we’d have ways to say “input” vs “networking”, or “mouse movement” vs “mouse clicks” etc). What do you think of this, compared to the properties approach?

BTW, I am in favour of changing all this ‘frame rate’ stuff to be ‘frame time’ instead. It behaves more linearly and means you’re thinking in the same units as both the profiler and input event timestamps. On the other hand, it’s definitely a less common way of describing the overall player loop execution, and would need more explaining to people…

[Ferazel]

@greggtwep16

If I’m understanding this right (a callback specifically called when interrupted) I guess this would be an alternative to Unity always setting the sleep bool back to false when it detects input to interrupt sleep, I’m not sure if I’d like it better or not.

Based on the API that Rune suggested, I’d rather just have the one application frame rate/time with a bool flag to trigger a managed script update ASAP on the player loop in certain wake cases. Then the game logic maintains full control over the frame rate and interrupt mechanics and nothing is changed automatically by the engine. If I tell the engine to go to a sleep frame rate with wakeup events, it should stay in that state until I tell the engine otherwise. For example, lets say I wanted to manage three game frame rates 2fps sleep (pause/static screen), 20 fps (2D frame animation), 30/60 fps fast (3D action sequence). If I handled that API on the game logic side, I would find it clearer if I just had a single frame rate field on Application which fed into the player loop. Any changes I made were explicitly made by game logic. It is purely API preference. I could definitely live with the API either way. If others feel strongly that having a separate sleep/wake frame rates set in the application and it automatically bounces between them, that’s fine too.

In regards to frame rate vs frame time, I really don’t have an opinion. I agree that frame time on the back-end would likely be cleaner, but at the end of the day it should be what the users of the API will understand the most and what has already existed prior (which is frame rate). I personally wouldn’t want my frame rate to go below 1 fps personally, but I could see that being a use case someone could bring up later.

@superpig

So this suggests another possibility for the API - we could expose a method, maybe something like:

That works for me. I like the functionality of adding an input mask, but the goal would be to have the wake functions be driven by the new ActionMap system so that any inputs not on the active ActionMap would be ignored. However, I’ll take what I can get.

[greggtwep16]

@superpig

So this suggests another possibility for the API - we could expose a method, maybe something like:

Application.ReduceFrameRateUntilWakeEvent(float frameRate, WakeEventMask wakeEvents = WakeEventMask.AnyInput);

where “frameRate” is the reduced frame rate you want the engine to use, and “wakeEvents” specifies the kind of events that can wake it up (which maybe for now would just be “any input”, but perhaps someday we’d have ways to say “input” vs “networking”, or “mouse movement” vs “mouse clicks” etc). What do you think of this, compared to the properties approach?

I’m all in favor of this, I had suggested something similar in the 2nd to last paragraph of this post.

https://forum.unity3d.com/threads/welcome-new-input-system-resources-and-info-please-read.397153/page-4#post-3104329

There are some inputs that it would likely never make sense or be very tricky to ever do it on like controller joysticks or accelerometers and gyros and should probably be excluded from “any input”. But other than those setting a mask would make sense.

I do probably favor something like this over having multiple player settings. To me the “low framerate” would always be triggered at runtime, and the new framerate only applies until a matching input is detected.

@superpig

BTW, I am in favour of changing all this ‘frame rate’ stuff to be ‘frame time’ instead. It behaves more linearly and means you’re thinking in the same units as both the profiler and input event timestamps. On the other hand, it’s definitely a less common way of describing the overall player loop execution, and would need more explaining to people…

Whichever is easier for you guys and most people. Obviously internally you guys are already taking the current int and sleeping by 1 / targetFrameRate. I would thing regardless of the naming there should also be a way to specify “sleep forever until input”.

[greggtwep16]

@Ferazel

Based on the API that Rune suggested, I’d rather just have the one application frame rate/time with a bool flag to trigger a managed script update ASAP on the player loop in certain wake cases.

Totally agree. I like the suggestion by @superpig .

[runevision]

Great, it sounds like we a form of consensus on what form this feature could take.

[Ferazel]

Just a note, that if the API is going to look like superpig’s I’d still want a way to wake up the framerate via game logic. Either by calling Application.targetFrameRate = 30 or by another method. There may be a use case where if I set the framerate to sleep, but I receive a network packet or I want to occassionally trigger a particle effect or animation to help convey that the game isn’t frozen. I’d want to manually relinquish my sleep framerate during that period and re-enable it once the animation/particle effect has concluded, and we’re on a static screen again.

[larku]

Sounds great.

I’d be happy with either runevision’s or superpig’s API suggestions.

superpig’s API is my preferred (at first glance) as it looks neat and concise and would suit my needs which are:

• Reduce the target frame rate to a rate I choose (eg 1fps, 10fps, 15fps, 30fps, etc)
• Wake immediately on input (selected by sleep flag) and allow me to either leave in sleep reduced fps or set a new target (eg 60fps)

I also agree with Ferazel that on the awake from sleep event we set the frame rate to resume to - the API used is of no concern to me.

It’s fantastic to hear Unity showing interest in doing this - this will make battery use and thermal load of some games significantly better (that’s an understatement).

I’m currently using a hand crafted solution that reduces down to 15fps while idle - this works but input lag is ugly and certain swipe events etc don’t always register which make my UI feel low quality.

Out of interest - how far away is such a feature? Weeks, months or years?

[User340]

Wow, what a coincidence. I was actually working on just this feature a few months ago. I actually went so far as to snapshot the game view and disable all rendering when no user input was detected. I can post my code if anyone is interested.

[greggtwep16]

@_Daniel

Wow, what a coincidence. I was actually working on just this feature a few months ago. I actually went so far as to snapshot the game view and disable all rendering when no user input was detected. I can post my code if anyone is interested.

How much did you save with just the rendering turned off (obviously everything else is still happening at the specified framerate). I never did a full test, but a long time ago it was always much more effective to just lower the whole framerate.

In either case hopefully this is a thing of the past soon with full support.

[User340]

@greggtwep16

How much did you save with just the rendering turned off (obviously everything else is still happening at the specified framerate). I never did a full test, but a long time ago it was always much more effective to just lower the whole framerate.

In either case hopefully this is a thing of the past soon with full support.

I didn’t test that, but I agree lowering the frame rate is likely much more effective (and simpler) than snapshotting the game view. One issue I ran into with lowering the frame rate was the initial delay of waking up. When I dragged my finger on the screen, there would be a pause until the next frame came. This initial stall was worse the lower I set the targetFramerate. If the new input system could force the next frame to come immediately, that would be perfect.

[Noisecrime]

Its great to see this issue finally getting addressed.

I’m a little confused by the approach though, but maybe i’m just not aware of the intricacies involved under the hood in Unity? For me whenever I’ve needed such a feature all I’ve wanted is to stop unity from rendering, but still maintain targetframerate for registering inputs and other events. I want to stop/pause rendering as that is the biggest drain, the culling, shadow generation, rendering of the scene etc.

I looked into doing this myself on my projects, but I could never find a good and reliable method that wasn’t overly complex to implement - both in my own code and coercing Unity to do what I wanted. So in the end I just shelved it.

I find myself wondering why in your example targetSleepframerate is set to 1, I would have assumed 0 would be more desirable, but again perhaps there is something fundamental within Unity that requires this?

As to your ‘sleepAndWakeOnEvent’ question I think I’d rather have it wait for the normal targetFrameRate to avoid weirdness like a very fast frame. Although most of the time i’m using deltaTime for framerate independence I’d still be a little concerned that getting a ultra fast frame might mess up some logic somewhere and might be hard to track down bugs.

I guess If others favor waking immediately, then perhaps this needs to be provided as a choice/option that developers can choose in Project settings.

As for use case I’d present a recent ( and ongoing) client app I did for QEPrize.
Links to store - Android
Links to store - iOS

Its essentially a 3D modeling/design app where users can construct models/trophies from basic building blocks based on platonic solids. As such like many 3d modelling apps there are frequent and various periods of inaction by the user, yet the app keeps chugging along at 30/60 fps.

In terms of inactivity you have several different scopes

• Long inactive period - User is thinking about what to do next. Range: 2 seconds to infinity
• Short inactive period - User is pressing buttons, but these actions happen between 1s - 2s intervals.
• Very short inactive period - User is tapping to place primitives 0s - 1s intervals.

The suggested solution would have to work with all these without glitches, delta time issues etc.

One problem is that whenever the user taps to place a primitive the camera auto-adjusts to ‘frame’ the model and the newly placed primitive, so during this time I’d need to run at the targetFramerate. I guess I could just toggle sleepAndWakeOnEvent, though I feel being able to send a custom ‘virtual’ touch event would be nicer a solution?

One area that concerns me though is that the user can save and load previous designs. Saving happens within a single frame, but loading is split across multiple frames in order to not lock up the app. There is also a hidden feature where loading is made to take even longer, where only a single primitive ( or more) is loaded per frame, in order to have the model ‘build’ in front of you eyes. This is currently used internally to generate video of designs, but we want to open this up to users in the future. Again I guess during the loading process i’d just disable sleepAndWakeOnEvent and everything should work ok?

Anyway i’d be more than happy to look at retro-fitting this project with the proposed solution once you get an experimental build produced. I could even provide Unity the source if they wanted, though its a complex project and due to time constraints not the best code architecture.

Finally one other big issue I had with the project was the directional light shadow casting. Its a big drain on the gpu, yet most of time the shadow never changes. It would be great to be able to re-use shadowmap texture. At various times I looked into trying to do this myself, but never made any real progress. I want to look at it again sometime since this was back in Unity 5.2 /5.4 and see if its possible now.

[greggtwep16]

@Noisecrime

I’m a little confused by the approach though, but maybe i’m just not aware of the intricacies involved under the hood in Unity? For me whenever I’ve needed such a feature all I’ve wanted is to stop unity from rendering, but still maintain targetframerate for registering inputs and other events. I want to stop/pause rendering as that is the biggest drain, the culling, shadow generation, rendering of the scene etc.

It was a long time ago when I profiled but while preventing rendering helps a bit, don’t underestimate all the other things going on under the hood (physics, scripts, etc.). They drain a lot of battery too.

@Noisecrime

I find myself wondering why in your example targetSleepframerate is set to 1, I would have assumed 0 would be more desirable, but again perhaps there is something fundamental within Unity that requires this?

There have been responses since then and yes sleeping until woken by input is desired in some cases. At that previous point in time, the design was related to current behavior where the sleep time is 1/targetFramerate. Obviously dividing by zero isn’t desired. The runtime method talked about since then should address this concern.

@Noisecrime

As to your ‘sleepAndWakeOnEvent’ question I think I’d rather have it wait for the normal targetFrameRate to avoid weirdness like a very fast frame. Although most of the time i’m using deltaTime for framerate independence I’d still be a little concerned that getting a ultra fast frame might mess up some logic somewhere and might be hard to track down bugs.

Setting this at runtime avoids the problem of ever having interruptions super fast over and over. Your app will have its normal framerate (say 60fps) and it’s up to you to set when to sleep and how long to sleep for. Upon input waking up the sleep Unity would resume the normal framerate and if you didn’t care about the particular input you can set it back to sleep.

[Noisecrime]

@greggtwep16

It was a long time ago when I profiled but while preventing rendering helps a bit, don’t underestimate all the other things going on under the hood (physics, scripts, etc.). They drain a lot of battery too.

True and in my case physics and scripts were not a worry, but how will this work with physics anyway? If you still had physics simulation going on why would you want to stop or reduce rendering updates, wouldn’t that just look worse, even with interpolation? My memory is a bit fuzzy but reducing the framerate wont affect physics, it will still be updating on its own frequency and thus still doing the same amount of work regardless?

Setting this at runtime avoids the problem of ever having interruptions super fast over and over. Your app will have its normal framerate (say 60fps) and it’s up to you to set when to sleep and how long to sleep for. Upon input waking up the sleep Unity would resume the normal framerate and if you didn’t care about the particular input you can set it back to sleep.

Yeah this sounds like a good idea, though I still prefer the idea of being able to use event masks and being able to send a dummy event from my code when needed rather than turning some application.setting on/off.

[greggtwep16]

@Noisecrime

True and in my case physics and scripts were not a worry, but how will this work with physics anyway? If you still had physics simulation going on why would you want to stop or reduce rendering updates, wouldn’t that just look worse, even with interpolation? My memory is a bit fuzzy but reducing the framerate wont affect physics, it will still be updating on its own frequency and thus still doing the same amount of work regardless?

Yeah this sounds like a good idea, though I still prefer the idea of being able to use event masks and being able to send a dummy event from my code when needed rather than turning some application.setting on/off.

With physics or the other stuff it isn’t about having things happening (just like with your rendering) it’s about shutting off the Unity plumbing for rendering, physics, scripts, etc. that happen regardless even if everything is still. Obviously when you’re sleeping it only makes sense if the scene became static and your trying to prevent Unity from doing anything as well, instead of still doing all of it’s bookkeeping draining battery.

Event masks were in the latest design. It’s not an application setting it will be a method in their API.

[Elecman]

Without getting into technicalities at this moment, I would just like to say that I am very happy to see that this issue is finally being taken seriously.

My app has very little frame change, probably only 2%, so I welcome any battery usage improvements.

[tswalk]

imo, I prefer:

• int Application.targetFrameRate

• int Application.targetSleepFrameRate

• bool Application.sleepAndWakeOnEvent

I think this would make more sense… i’m not so keen on the idea of having to come up with my own routines to determine the device per frame time, which can vary wildly. I also like having a way to ‘trigger’ sleep, so that I can control when it should be waiting to sleep.

This would allow me so much more control over power usage… as long as input events are captured. That I believe has been the big problem with controlling framerates… you miss inputs.

I actually created a B2B app with unity… and the biggest complaint from the client that I have not been able to truly address is power usage. The reason is that when I put in my own frame rate controls, inputs get missed… they complain the app isn’t responsive enough >.<

So… in summary, I’ve been using the TouchScript library, which is wonderful. If these updates to Unity API were implemented, I could hook its’ touch events to trigger an Application.sleepAndWakeOnEvent state change. This would awesome sauce and candy land goodness.

One question I have though, will we be able to set a type of delay timer for going “into” sleep mode… which could be a float value? Or are you designing this into sleep so that it gets triggered when no changes to the display are happening?

[edit]
OH, also… could there be overrides on your sleepAndWakeOnEvent to accept a callback function? I was thinking about this because it would be nice to be able to optionally have this for either logging or analytics… something like:

Application.sleepAndWakeOnEvent(OnTriggeredEvent);


private void OnTriggeredEvent(SleepAwakeEventArgs _event)
{
  if(_event.Sleep)
{
  Debug.Log("app is going to sleep");
}
if(_event.Awake)
{
  Debug.Log("app is being awoke");
}

}

…AND, could these please be backported to 5.6x !!! PLEASE