Long lived virtual agent simulator : is Unity for me?

Hi folks, hoping to get some feedback from you about the possible uses of Unity outside of straight game development.

I am developing (as part of a research project in my graduate studies program) a long-lived virtual agent, a computer program that will live in a virtual world over a long period of time. I’m not really an expert game programmer, so I don’t really want to code up a physics and rendering engine : my interest is the the AI of the virtual creature.

The idea is that the virtual creature takes the role of the game player : a computer program will be playing my game.

I’ve been thinking that Unity might be a good place to do this, except I have certain requirements that I need to know if Unity can handle.

  1. The agent should be able to pick up and move things in the environment, like in half life 2.

  2. The agent’s code will not be a simple script, it will need to be decoupled from Unity. I noticed that some network code was included on this website, I was thinking perhaps that would be a way to connect an external process up to Unity to fill the role of the player. In the network example the client is a Unity instance, but it seems like this could almost as easily be a C++ or Java program running on a supercomputer.

  3. The agent’s perception need to be direct, things like sounds and images.Ideally, this would simulate an agents sight/hearing or a robots camera/microphone. It is not clear if this is possible with Unity, which would seem more suited to providing the agent with coordinate-based locations, distances to nearby objects etc. Would I be able to get a camera image somehow and make it available directly to the agent? Over the network?

So : wise Unity users, what do you think? Is this the way to go. If not, do you have any suggestions. I’ve considered using the HL2 SDK, but I’m a MAC user and I’d rather not have to reboot to Windows all the time. Also, Unity seems like a better match.

Can people think of a better way for me to control the player? Or perhaps the virtual agent should be a NPC… if this is the case, still : does anyone know how I could get realistic sensors to that agent and control it outside of the Unity framework?

Thanks in advance for any help, comments, or suggestions you have.

-Brian

All that you posted seems possible in Unity. It seems like you are really talking about a bunch of AI type code, although perhaps very sophisticated.

  1. Picking up things should not present any issue
  2. It seems like an external AI is not really all that different than a remote user. Depending on speed of the “game” then you may not need most robust networking code. If reaction time was not very important then you might even be able to get away with the WWW class. Unity 2.0, whenever that comes out, is suppose to have networking as part of its “tools”. Until then you would have to use the WWW class, use .Net, or use an external library via a plugin.
  3. I assume this means a way for your AI/agent to perceive the world they are in. This will require code but you can probably get a long way with collisions, raycasts, using distance, and some vector stuff like dot product. Thinking of your agent as being external to the game might actually help with the design part and abstracting the perception components out from the decision making components of the agent.

You would want to be able to save your world state in case there was a crash or a power failure. This can also be done in Unity.

Whatever you are doing it sounds pretty neat. You could probably get a proof of concept up and running pretty quickly, definitely within the trial period if you have some coding experience and can actually dedicate some time.

I don’t see anything inherently problematic with what you’re doing. I don’t quite understand what you’re doing wrt. question 3, but if you explain a bit better I’m sure we can help you.

You might be interested in the second testimonial on this page:

http://unity3d.com/unity/unity-visualization.html

INNlink, an AI (spiking neural networks) startup located at the Danish Technical University, is using Unity to train virtual agents that are simulated in Unity but controlled by their AI kernel over the network. The entire project is then run on a large array of Apple XServes. The Unity part of their project was achieved in a matter of days.

That sounds somewhat similar to what you are working on.

d.

It’s absolutely similar, and I think the problems they ran into and the challenges they overcame are the same that I am having.

So, first they had to write a bunch of socket code so that their neural network software could communicate with Unity without being inside Unity. Great, that’s what I’m working on. Limited success so far, but it’s only been a day. I can write a java program that connects to a Unity game and sends commands into it. Unity doesn’t always respond how I’d like, but it’s a work in progress :stuck_out_tongue:

About the third part though, the sensors. The sensors that I want for my virtual agent are vision, hearing, and perhaps touch. This correspond to real-world sensors on a robot of a camera and a microphone. If you play certain FPS games, you’ll notice that sometimes physical cameras are setup around the game, and then there are viewscreens through which the player can see what is on the camera. I think what I want is like that, where the camera is on my agent, and my agent’s brain gets to see what the camera sees. Alternatively, you can just think of my creature as actually playing the game as an FPS where he only gets access to the frame buffer instead of access to the locations of various objects on the map. Is this making more sense?

Ok, the “touch” part is probably the easiest one - you get all collision events for “the player” and send them off somehow to your AI.

Additionally, you could cast a lot of rays around the player - these could represent either “will touch soon” or some kind of “vision”.

Casting a grid of rays in the viewable area in front of the player would return you a grid of distances, and additionally the info on which objects were hit. So you at least get rough “depth perception” this way.

I’m not sure if your AI needs the color image visible to the player (maybe just depth is ok?). Right now there’s no way to read back the rendered pixels with Unity scripting API, but of course you could write a small C++ plugin that just does glReadPixels pass them somewhere/somehow.

Getting back precise audio might be tricky. I guess it could be approximated by iterating over all AudioSource objects, checking what sounds are playing, distances to them etc. and then passing something to your AI.

Two possibilities:

  1. Save a screenshot (there’s a thingy somewhere around here to do that) every frame and send that off to your AI separately from Unity.
  2. if you have Unity Pro, you can render to texture, and use GetPixel on that texture to send the data.

GetPixel is for Texture2D; a RenderTexture is not a Texture2D.

Well, that’s what you get when someone with Indie tries to describe Pro features… :wink:

(1 year later…)
Btw Aras, how can i get the pixels of a rendertexture 's single frame?

you use ReadPixels to copy it into a texture.

Hi Joachim,

I wasn’t sure which way to go…
It work perfectly with readPixels.

Tx very much :wink:

63792–2340–$videof1_124.zip (515 KB)