I have tried several different ways to get the AI to “see” enemies in its line of view, but every attempt I have made so far has not worked out so very well. How do you create a line of sight for AI characters and have the script make the decision of which enemy is the greatest threat?
There are many different possible answers to this question, depending on your game type and how accurate you want the algorithm to be. I’ll try and share a couple of ideas with you, but you’ll have to make the ultimate decision what makes the most sense for your game.
##Basic Test
A solid first step is to perform a frustum test, this is basically checking to see if the player is even in the viewing plane of the AI (without testing for occlusion, yet). Unity supplies two very handy functions for doing this.
##Raycast Technique##
If you determine that it’s possible for the player to be visible, you have several options. I would probably use some series of raycasts based on the type of game.
Would just a raycast pointed at the head be sufficient? Or just a raycast pointed at the body? This would fail in cases where parts of the player stick out of cover, but for AI in a first person shooter, it might be okay to be a little spongey. It would be frustrating for the player (especially where they can’t see their own bodies) if a finger, or spike of hair, or toe, stuck out and gave away their position. In general, for first person shooters I found that “I can’t see you then you can’t see me” approach works well. A single raycast right into the center of the camera does the trick 90% of the time.
For more accuracy I might try using multiple raycasts. Based on your animations and models, your cover, and your level geometry, determining what targets for your raycasts and how many hit positives determine a “sighting” need to be carefully considered. Head and body? Feet? Hands?
General Purpose Solution
If you need something accurate and all purpose you could try shooting 4 raycasts through the player collider at the top, bottom, and horizontal extremes. If 2 or more of these trigger, then you have a sighting. Attach the main camera to an AI and watch them navigate the world. Pay attention to situations you think should be a “hit” and add additional targets as necessary.
Performance Considerations
Having a few raycasts here may not be as computationally expensive as you might initially think (depending on the number of AI). Because we are first doing a frustum check so we don’t likely need to perform this for every AI. We don’t even necessarily have to calculate these every frame (only when the player or the AI moves). Even still, you probably don’t have to perform them every frame even in motion. Maybe once every half a second would give a reasonable reaction time for a human walking down the hallway not really expecting to see anything. You can play with the timing to get the desired results.
If you have a LOT of AI, all in the same area, and ALL finding the player in their view frustum, you can do a round robin approach. Maybe every “look” update, only 25 AIs at a time actually perform the test. Even with 100 enemies all in a room at once, all with the player in their view frustum at all times but with only one AI who can actually see him it will only take 2 seconds worst case to spot him. In reality it would almost certainly be much shorter. You can improve these results by allowing everyone to do a “quick and dirty” check (a single raycast to center of mass to the player) while throttling the “deep look” (4+ raycasts).
Avoid Premature Optimization###
I wouldn’t optimize with this until you need to though, for most use cases, I think you’ll be fine. If you knew how many raycasts were being fired every frame by the zombies in Left 4 Dead, you’d be shocked that the game runs at all.
##Food for Thought##
Also remember that since it’s AI, it’s okay to let the player get away a little more with it. Interesting study, if you tell a person a game is stealth based, give them the right mood and setting, and give them the tools to be stealthy (crouching, walking, hiding in shadows), but make them completely invisible to the AI at all times, how long will the player think they are just successfully being stealthy before they realize the AI doesn’t really work? How much more satisfying would that initial play session be if the AI was hyper realistic and spotting the player immediately when even slightly visible? I mean really, you just need them to get spotted enough to make it fun, which is probably very far from realistic (like most game mechanics).