I’m trying to decide between using rotating planes, mesh, and particles. Which approach is best for muzzle flash of RTS perspective camera?
Either one of those would work, and can be done with pretty good realism. The best option depends on a number of factors, mainly what is the visual quality and style of your game and how many resources can the solution take up.
If you have a very high visual quality game, a mesh would probably look best. I would make a mesh object as a child of the weapon, and activate it for the frame or two that the muzzle flash lasts. This way, practically no resources are being consumed while the flash isn’t active, and when it is, it does not last for long.
Planes would be a good option if you needed to worry about performance more (like on a mobile device). You could do a X shape of two planes. It would look fairly decent. You will just need to tweak it to look best from what ever perspective you have.
Particles are a decent option too, and can possibly be a better option than planes. You do have to be careful as particles face the camera and might look odd if you have a directional flash.
There are two other aspects of muzzle flashes that are often not thought about: light and smoke. A muzzle flash is the result of an explosion within the chamber of a gun barrel. The explosion creates fire, which emits light. So, if you have the resources to handle it, I would add a point light on to your muzzle flash. It should be a warm light (somewhat orange) and should not have a large cast radius. From this explosion/fire, smoke can sometimes be created as well. The amount of smoke isn’t very much. It should be a simple smoke emitter that only emits one particle at a time. The smoke should be very subtle. I think that the light and smoke together would create a very visually appealing effect when combined with the right muzzle flash.