its not possible to have any editor startup on double click / right click a file with a custom extension. if it were a monobehaviour you could use a custom editor
Thank you Dreamora. After re-reading my question, I realized it might be too vague (someone may have thought I meant another exe entirely that I was hoping to launch), so I thought I would expand on it to make sure (though the answer may very well be the same):
I wrote a custom tool within the Unity editor that is generating a binary data file. What I was hoping to do was to tell Unity that those files with a given extension belong to that tool/plugin and it open my editor window.
The answer is the same.
Unity has no focus nor support / knowledge about custom extension.
Unity focuses its support for its provided classes like ScriptableObject as datastorages and monobehaviours. These would allow you to use the CustomEditor attribute.
but if you do completely own extensions it will be up to you, adding it to the window menu as distinct editors are handled for other topics too