I’m encountering a technical challenge with a rather unique project, and after numerous experiments, I’m seeking the advice of those more experienced than myself to explore potential solutions I may not have yet considered.
I have a scene with a quad that alternates between black and white with each frame. My objective is to have a pure tone play every time the quad is displayed in white. The desired outcome is to achieve synchronous blinking between the image and the sound.
To accomplish this, I’ve attempted to generate a sound using AudioClip.Create, which continuously plays the tone, and I’ve implemented a command in the Update function to manage the quad’s color switch. I’ve also added audioSource.volume = isWhite ? 1.0f : 0.0f;.
However, the current result is far from satisfactory as the sound does not synchronize with the image, and it feels like there is something too heavy to be managed in real-time.
I really want the sound to depend on the frame, and I’m looking for the most low-level method to achieve optimal results. Do you have any other avenues to consider?
I tried implementing that method. I can generate my pure tone, but it seems I still have a delay that I need to compensate for to ensure the sound plays when the frame is blank. I’m trying to add a delay in the buffer to play the sound with a one-frame delay but at an accurately calibrated position. Do you think this is possible? Currently, my code doesn’t seem to work because no matter what I put in for audioDelayInSeconds, the delay seems to always be the same.
This plot shows the reading of a light sensor placed on the screen and the audio signal from the jack cable:
Here is my code:
using UnityEngine;
public class SimpleColorBlinkAndContinuousToneWithDelay : MonoBehaviour
{
private Material objectMaterial;
private bool isWhite = false;
public int framesBetweenSwitch = 1;
private int frameCounter = 0;
private int sampleRate;
public float toneFrequency = 440f;
private float[] delayBuffer;
private int writeIndex = 0;
private int readIndex = 0;
public float audioDelayInSeconds = 0.1f;
public float activeTime = 5f;
public float pauseTime = 5f;
private float activeTimer;
private float pauseTimer;
private bool isActive = true;
void Start()
{
objectMaterial = GetComponent<Renderer>().material;
sampleRate = AudioSettings.outputSampleRate;
int delaySamples = (int)(audioDelayInSeconds * sampleRate);
delayBuffer = new float[delaySamples];
writeIndex = 0;
readIndex = (writeIndex - delaySamples + delayBuffer.Length) % delayBuffer.Length;
}
void Update()
{
if (isActive)
{
activeTimer -= Time.deltaTime;
if (activeTimer <= 0)
{
isActive = false;
pauseTimer = pauseTime;
}
}
else
{
pauseTimer -= Time.deltaTime;
if (pauseTimer <= 0)
{
isActive = true;
activeTimer = activeTime;
}
}
if (isActive)
{
frameCounter++;
if (frameCounter >= framesBetweenSwitch)
{
frameCounter = 0;
isWhite = !isWhite;
objectMaterial.color = isWhite ? Color.white : Color.black;
}
}
else
{
objectMaterial.color = Color.black;
}
}
void OnAudioFilterRead(float[] data, int channels)
{
for (int i = 0; i < data.Length; i += channels)
{
float sampleValue = Mathf.Sin(2 * Mathf.PI * toneFrequency * i / sampleRate) * (isActive && isWhite ? 1.0f : 0.0f);
delayBuffer[writeIndex] = sampleValue;
for (int channel = 0; channel < channels; channel++)
{
if (i + channel < data.Length)
data[i + channel] = delayBuffer[readIndex];
}
writeIndex = (writeIndex + 1) % delayBuffer.Length;
readIndex = (readIndex + 1) % delayBuffer.Length;
}
}
}
I’m not really sure why it wouldn’t work right there…
A thing we need to keep in mind is that Start and Update are not running on the same thread than OnAudioFilterRead. This latter one is called back from the audio thread so the values of your write/read indices might suffer from that.
I think you should keep readIndex to 0 in the start and wait for the audio buffer to fill at least that much data in your delay buffer before starting to write data to the callback output. Although that’s just a hunch…
Hello @maxronc , I think you can keep your OnAudioFilterRead the same (almost) and sync your flash off of the audio thread.
One way to do that would be to push alternating white/black messages to a queue from the audio thread and have the main thread Update function poll the queue for color changes.
Something like this…
using System.Collections.Concurrent;
public class SimpleColorBlinkAndContinuousToneWithDelay : MonoBehaviour
{
const float SECONDS_BETWEEN_TOGGLE = .5f;
int BuffersBetweenToggle;
int bufferCount;
enum ColorMessage
{
White,
Black,
}
ConcurrentQueue<ColorMessage> colorQueue = new ConcurrentQueue<ColorMessage>();
bool Ready;
...
void Start()
{
...
BuffersBetweenToggle = SECONDS_BETWEEN_TOGGLE *
AudioSettings.outputSampleRate /
AudioSettings.dspBufferSize;
Ready = true;
}
void Update()
{
while (colorQueue.TryDequeue(out transitionColor))
{
if (colorQueue.Count > 0)
{
// Should only have 1 message unless something got paused,
// to be safe, don't let q messages build up
continue;
}
if (transitionColor == ColorMessage.White)
{
objectMaterial.color = Color.white;
}
else
{
objectMaterial.color = Color.black;
}
}
}
void OnAudioFilterRead(float[] data, int channels)
{
if (Ready) {
bufferCount++;
...
if (bufferCount % BuffersBetweenToggle == 0)
{
colorQueue.Enqueue(isActive ? ColorMessage.White : ColorMessage.Black);
isActive = !isActive;
}
}
}
}
You can control the flash rate by changing the value of SECONDS_BETWEEN_TOGGLE (which I have arbitrarily set to 0.5). I have not tested this code so there are certainly bugs
