Ive tried what seems like a million ways to get realtime audio sequencing in unity. Unity isnt very good for small MS callbacks, it wont give access to fmods callbacks and all manner of timers (threads, coroutines,system timers) have lag and variable results when down to audio MS range. Probably only hardware bound callbacks (ie buffer requests from audio cards) are accurate enough. However, with some sneaky tricks Ive managed to get triggers to work down to a acceptable range.
Heres the trick ,just in case anyone else has struggled with this.
Run one sample on loop as a metronome.
Run a fixedupdate (or any fast thread) and use this to find the current PCM sample pos of the metronome.
Use this position to detect if the current loop of the metronome sample is nearing its end (the margin you can detect depends on the frequency of your fixedupdate etc).
If its close enough to the next loop then trigger a new sound to play (and stop checking )… BUT offset it by the remaining samples in the current loop (loop sample length- current sample position).
This means it will play on the exact loop point that is coming up. This solution might not work for super slow machines and it less accurate as your tempo increases (but you can adjust the margin for that) but its the best, rather only solution ive found so far.
Awesome work on this project ,its great to see someone pursuing the possibilities of audio sequencing and timings involved . I have been doing the same also and trying to find a way to achieve perfect sync .Its an interesting way u worked the timings and has given me new ideas and unity is a great tool to add visuals too
this was a simple project where we wanted to make someone with no timing knowledge be able to make something sounding good with no skills in music making.
after watchin your sequencing i also got to try the idea out the blocks are representative of an audio loop and i placed them in various places and used collisions to check for hits and if hit start playing the relevant loops still got a lot of work to do on it but gives a basic idea
Hey folks! Stumbled upon this thread after MUCH searching for information on implementing this sort of thing with Unity (meaning triggering things in nearly sample accurate musical time). But, despite the info provided here and the example webplayers, my friend and I have been unable to get this technique that tomnullpointer described to really work correctly, and all our results still have a very audible inaccuracy about the timing that isn’t present in any of the other projects this thread. Could anyone who’s had success with this maybe shed a little light on how to make it work or what to avoid that might fudge it up? A peruse through the forums tells me there’s still a lot of people looking to crack this. The code I’ve come up with so far, based on tomnullpointer’s original description, is as follows:
var checkingTime : boolean = false;
var audioSample : AudioSource;
var audioMetronome : AudioSource;
var checkingSamples : boolean = true;
function FixedUpdate () {
if (checkingSamples == true) {
samplePos = audio.timeSamples;
}
if (samplePos < 20000) {
checkingTime = false;
checkingSamples = true;
}
if (samplePos > 20000) {
checkingTime = true;
}
if (checkingTime == true) {
checkingSamples = false;
audioSample.Play(audioMetronome.audio.clip.samples - audioMetronome.audio.timeSamples);
var offset = audioMetronome.audio.clip.samples - audioMetronome.audio.timeSamples;
print("Sample position is " + samplePos);
print("Offset is " + offset);
}
}
Note: The sample we’re using as a metronome is one beat long and 22050 samples; test we logged showed that 20,000 seems to be an appropriate threshold that always gets caught. The result from this script is that our sample to be triggered plays near each quarter note at the metronome’s tempo, but is always off; the amount of inaccuracy changes from trigger point to trigger point, so it doesn’t seem to be a matter of a static offset for latency or something like that. The two print commands at the end have confirmed that the math is coming out right, and it should be triggering the sounds right at the start of the next 22050 sample loop. Even if folks don’t want to spell this out in their own source, any help would be appreciated about what direction to look in. Thanks!
I’m trying to understand what you are trying to accomplish. It appears that you are timing each consecutive note based on the amount of samples that have passed for the currently playing audio. If I’m right then, you are using the audio itself as a measurement for the tempo. Are you trying to keep the audio in sync or are you trying to play the audio clips from a given point within each audio?
Sorry, typo in that first if statement I think; let me fix and clarify. There’s two audio source variables that are assigned to the two audio sources that are part of the gameobject this is attached to. As per tomnullpointer’s description, there’s one looping sample (audioMetronome) that is one beat long at the desired tempo and used as a master clock. audioSample is the sound effect that is meant to be triggered at the start of every loop of audioMetronome. Does that make a little more sense?
var checkingTime : boolean = false;
var audioSample : AudioSource;
var audioMetronome : AudioSource;
var checkingSamples : boolean = true;
function FixedUpdate () {
if (checkingSamples == true) {
samplePos = audioMetronome.audio.timeSamples;
}
if (samplePos < 20000) {
checkingTime = false;
checkingSamples = true;
}
if (samplePos > 20000) {
checkingTime = true;
}
if (checkingTime == true) {
checkingSamples = false;
audioSample.Play(audioMetronome.audio.clip.samples - audioMetronome.audio.timeSamples);
var offset = audioMetronome.audio.clip.samples - audioMetronome.audio.timeSamples;
print("Sample position is " + samplePos);
print("Offset is " + offset);
}
}
ok I will try this as best as I can. I actually learned something reading the entire forum post :). First off i tried modifying your code:
var checkingTime : boolean = false;
var audioSample : AudioSource;
var audioMetronome : AudioSource;
var checkingSamples : boolean = true;
function Update () { // why not use and update function
if (checkingSamples == true) {
samplePos = audioMetronome.audio.timeSamples;
}
if (samplePos < 20000) {
checkingTime = false;
checkingSamples = true;
}
if (samplePos >= 20000) { // larger than equal
checkingTime = true;
}
if (checkingTime == true) {
checkingSamples = false;
audioSample.Play(audioMetronome.audio.timeSamples); // offset?
var offset = audioMetronome.audio.clip.samples - audioMetronome.audio.timeSamples;
print("Sample position is " + samplePos);
print("Offset is " + offset);
}
}
I used an update because to my knowledge fixedUpdate is usually for physics. Then I changed the offset code. Subtracting the current metronome current position from its total length would only give you the remainder of time left before the metronome finishes. So why not play the audioSample from the same point where the metronome is at. Perhaps that was the problem.
Then I decided to modify my method a bit because at the time I did not use any offsetting of audio samples. I use a a function that invokes itself over and over at a desired rate of time. Then i tried using an offset with time instead of samples. Here it is:
var metronomeSpeed : float = 1.0; //seconds
var audioSample : AudioSource;
function Start(){
metronome();
}
function metronome(){
var timeLapse : float = Time.deltaTime;
var offset : float = timeLapse - metronomeSpeed;
audioSample.Play()
if( offset > 0.0 ){
audioSample.time = offset;
}
Invoke( "metronome", metronomeSpeed );
}
I have not tested my modified version. But that is my insights on how to go keeping sync. With my sequencer I did not use audio sample offsetting. The timing using the invoke appeared to keep sound timing accurate as long as the fps was constant.
var metronomeSpeed : float = 1.0; //seconds
var audioSample : AudioSource;
var timeLapse : float;
function Start(){
metronome();
}
function Update(){
timeLapse += Time.deltaTime;
}
function metronome(){
var offset : float = timeLapse - metronomeSpeed;
audioSample.Play();
if( offset > 0.0 ){
audioSample.time = offset;
}
timeLapse = 0.0;
Invoke( "metronome", metronomeSpeed );
}
updated
EDIT: I just tried it out in unity and it works! I also put a missing semicolon into the code.
2nd Edit: Nevermind there is an extra piece of the audio playing. It probably needs an audio clean up.
LAST EDIT (for sure) : It really does work! I just had “play on awake” selected when it should have be deselected.
I tried your code out, starpaq, and it seems to have about the same result as a lot of our previous tests, in that there are still very noticeable inaccuracies in the timing of the triggered sample (i.e. it doesn’t sound very musical). There still remains a very big difference between the results of my scripts/your suggestion, and the results of tomnullpointer’s example at the top of the thread, which is what I’m mainly confused about.
To answer your other questions:
We cached the remaining samples in the current metronome loop because, while the goal was to trigger the sound effect at the start of every metronome loop, neither Update nor FixedUpdate (which is faster, to my knowledge) updates anywhere near as fast as the sample rate of an audio file (in this case, 44100 Hz), so it would be impossible to cache the metronome’s sample position and simply trigger a sound whenever that position is 0; it simply wouldn’t see 0 most of the time. So by checking once a threshold is hit, it ensures that a new sound will always be triggered once every loop, and then that sound is offset by the number of samples left in the metronome loop, so that the sound plays exactly at the start of the next loop. At least that’s the theory.
I think the accuracy you are looking for is too limited by using Unity because the audio timing is based on updated cycles which are too variable. I was even considering simulating a buffer in code to play the entire beat on delay in order to anticipate playing the sample on time. However, even that seems to be crippled by variable performance. It appears to be not possible given the current access or resources being used in code. If I think of something I will post it.
Perhaps someone else would add to this in order to steer you in a better direction.
I’m not even sure if this level of accuracy is possible programming through middleware or without using a fully accessible fmod plugin.
how does this differ from simply checking millisecs? millisec-based tempo is rock solid on time, I don’t understand how this would be any more accurate than firing events based on how many millisecs have elapsed since the last metronome.
It was my understanding, hippocoder, that actual millisecond based checks aren’t possible or are difficult because of the speed at which Update, FixedUpdate, and coroutines run in Unity. You can tell the script to look for each millisecond, but the engine isn’t checking fast enough to actually see every millisecond (thus why tomnullpointer’s solution at the top of the thread was necessary) as far as I know. As for whether or not that accuracy is possible, there’s two step sequencers out (the one at the top of this thread, and the one on Kongregate made by quickfingers) that seem to indicate it is possible in Unity. Whether or not you need some sort of custom .dll, plugin, or middleware is another question, but the description at the top of the thread seems to indicate it was done without using millisecond counters and without middleware.
Basically, we have yet to see a solution that is actually rock solid on musical timing, and we’ve tried to do a lot of time/second/ms based solutions; entirely possible I’ve missed something, but I’m just trying to get a discussion going on the matter and let ideas bounce around.
Are you using anything specifically to measure the accuracy of timing? As I would like to know because I have not been able to determine the level of accuracy with my own sequencer. I have played my sequencer and personally haven’t noticed (with my ears) any obvious poor timing and I have not received feed back about its approximation of timing. I would appreciate any insight or opinion you have on my sequencer.
I just checked yours out starpaq. Coming from a musical background I can definitely hear some discrepancies in the timing of notes in your step seq., i.e. if I fill every bubble/button of a single row (so 16 notes should play in total), they sometimes sound equidistant from one another in terms of time, and other times they don’t; they sound closer or farther apart in subtle (but still noticeable ways). It’s the same result that I’ve been getting with a lot of my tests. But, to answer your other question, we haven’t been scientifically measuring anything, I’ve just been relying on my ears, and even though I trust them, feel free to take my feedback with a grain of salt.
If you do want to test the timing, I suppose you could record the audio output of your webplayer (probably just by recording the output of your computer somehow), put the resulting file in some audio editing software and measure the milliseconds between the transients (or the start of each sound). Alternatively, you could always have a backing track or metronome start to play when the sequencer starts moving, at the tempo that the seq. is moving at, and see how the triggered sounds line up with the streaming audio. I think even the example at the top of this thread has some slipping of the time between samples, but it’s very small. (Not trying to be down on your work, just giving some honest feedback).
Hey TomNullPointer, I don’t know if you’ll read this but you’re a freaking genius, talk about thinking outside the box, this works perfectly, I just tried it. So thanks a bunch for this ingenious implementation, I’ve been looking around for a solution for seamless sound sequencing.
Cheers
more sounds would be awesome
there is an extra piece of the audio playing. It probably needs an audio clean up.