Greetings,
I started a project for learning purposes exploring the use of MIDI files and sound font playback in Unity. I know very little of the subject to begin with, but I have looked at various C# files around the web for MIDI file parsing and started to use one to extract the NoteOff and On events, delta timing, the channel, and what note number it is. From there I made a simple piano that built a bunch of empty game objects with timers and a piano wave file to use based on the note number. I originally found all 88 key recordings online and imported those into my project and had AudioSource components just play that when their time was up. Another timer would stop the playing clip set to the note off time. I had the piano keys set to events that would change the color of the key to simulate it being pressed.
So I thought that was cool progress, but if I wanted more instruments I had to search for each instrument and each note and if it sounded good. This was very impractical and would be very tedious given I have no synthesizer or recording devices. I discovered sound font files. The .sf2 file looks like it has various instrument samples packed into a single file. If I did not like how it sounded I could swap it out for another one found on the internet. This is where the problems began.
The MIDI file is a binary file that is split into data chunks that is streamed and parsed in the program. There seemed to be a lot of documentation and some tutorials on how to do that in various language. The sound bank file is also a binary file split into chunks, but from what I found there was a lot less documentation to be found and there was mostly just programs to buy that would have these pre-built into them. I found a complete C# source library that claimed to read MIDI and sound bank files as well as format a sample buffer float array ready to be used. I also found another Unity forum from around a decade ago of someone taking that library and importing it into Unity. I downloaded it and opened it up, fixed some compile errors, and ran one of the provided MIDI files. It worked, but when I provided one of my own, the channels were not in sync anymore and some of the instruments were 1 or more measures off. As I tinkered with it I found it primarily supported the .sfz format and I would still have to provide all my own wave files of all the different instruments, so that seemed like a bust.
On the original C# synth website the old project ported over, I noticed there were updates to the library since the last edit on the forum. I downloaded the latest version of it and it looked like it has expanded capabilities and dedicated .sf2 support. I imported the whole thing into my new project and started looking at it and comparing the differences between the two. The new version looked like it was vastly restructured.
I added the FluidR3_GM.sf2 sound bank file to my project as well as some MIDI files. The old Unity project had a game object with an AudioSource and Listener on it and a main test script. This script initializes a MIDI file object and synthesizer, loads a sound bank (.sfz), loads a MIDI file (as a text asset), loads a bunch of GUI things, and there was a OnAudioFilterRead method. This method looked to be how it was making use of the readied sample buffer. I replicated the game object minus the GUI things and just had it start to play after it finished initializing the objects in the Awake method. It uses the new library and an .sf2 sound bank file instead.
Here’s where the differences became apparent and where I need some help, or at least someone else to take a look over. The old Unity project gets a float array to send to the AudioSource where as mine comes back as an empty float array and thus, no audio. This new library has been restructured enough that although there still is a GetNext(float[ ] buffer) method, it actually doesn’t do anything with it when viewing it in the library. There is practically no documentation on this library which lead me to making this post. I think this is where the user is to make edits and determine what they want to return, but I simply want to just get it to work how the old project had. I’ve tried all sorts of things to try and find some kind of method to populate or format the float buffer array to no luck.
So, if anyone has done this before or has some insight as to how to do this, I would be grateful.
Here is where the C# synthesizer library was from (the download archive button gets the latest version):
https://archive.codeplex.com/?p=csharpsynthproject
Here is the old Unity forum:
Here is a snip of the old Unity project’s GetNext looks like in StreamSynthesizer.cs:
//UnitySynth
public void GetNext(float[ ] buffer)
{//Call this to process the next part of audio and return it in raw form.
ClearWorkingBuffer();
FillWorkingBuffer();
for (int x = 0; x < effects.Count; x++)
{
effects[×].doEffect(sampleBuffer);
}
ConvertBuffer(sampleBuffer, buffer);
}
Here is what it looks like in the new library in Synthesizer.cs:
public void GetNext(byte[ ] buffer)
{
CrossPlatformHelper.Assert(sampleBuffer.Length * 2 == buffer.Length, “Output buffer length must equal RawBufferSize!”);
Array.Clear(sampleBuffer, 0, sampleBuffer.Length);
FillWorkingBuffer();
ConvertWorkingBuffer(buffer, sampleBuffer);
}
Here is what I have been trying to use. I copied pasted the one that was there and changed it to accept a float array and removed the ConvertWorkingBuffer line so that it stays as a float:
//custom float get next for audio source
public void GetNext(float[ ] buffer)
{
CrossPlatformHelper.Assert(sampleBuffer.Length * 2 == buffer.Length, “Output buffer length must equal RawBufferSize!”);
Array.Clear(sampleBuffer, 0, sampleBuffer.Length);
FillWorkingBuffer();
//Debug.Log(“something is supposed to go here…”);
}
The debugger has shown that there is MIDI file data and sound bank data being loaded, just I have no idea what is supposed to go here in order to populate the buffer to be used back on the OnAudioFilterRead method. Here is what that looks like on both projects:
private void OnAudioFilterRead(float[ ] data, int channels)
{
midiSynthesizer.GetNext(sampleBuffer);
for (int i = 0; i < data.Length; i++)
{
data = sampleBuffer*;*
}
}
I almost never make forum posts, so if there is anything I missed let me know and I will add it in.
Thank you all.