Better machine learning for Unity ML-Agent/Sentis

Community Showcases
, , ,
1

I am researching and figuring out how to train easier and better with ML agents using other APIs. This is just preliminary idea and get it to final stage of the code. If anyone want to try it, please contact me and I will provide code, and see if it is easier for you. When research is done, I will add this to Unity Asset Store.

The summarized idea of the C# script for Unity, OctoAIIntegration.cs(OpenAIIntegration.cs), is designed to facilitate communication between a Unity application and an external AI service provided by OctoAI via its API.
The script sends natural language prompts to the OctoAI API and handles the responses asynchronously. Here’s a breakdown of its main components and functionality:

  • Using Directives

  • using UnityEngine; includes the base Unity namespace which provides access to all the fundamental Unity functionalities.

  • using UnityEngine.Networking; includes the Unity Networking namespace that allows sending and receiving network requests.

  • using System.Collections; includes support for coroutines, which are used for asynchronous operations.

  • OctoAIIntegration Class

  • Inherits from MonoBehaviour, making it a component that can be attached to Unity GameObjects. This allows it to interact with Unity’s update loop and scene elements.

  • Class Variables

  • apiURL: A string containing the URL to the OctoAI/OpenAI API endpoint responsible for processing chat completion requests.

  • apiKey: Your personal API key for authenticating with the OctoAI/OpenAI service. You need to replace “YOUR_API_KEY_HERE” with the actual key provided by OctoAI.

  • SendRequestToOctoAI(OpenAI) Method

  • Public method that takes a string prompt and a callback function of type System.Action. This method initiates the process of sending a request to the OctoAI/OpenAI API by starting the coroutine SendRequestCoroutine with the given prompt and callback.

  • SendRequestCoroutine IEnumerator

  • A coroutine method that asynchronously sends a request to the OctoAI/OpenAI API.

  • It constructs a request body with necessary parameters such as model, prompt, and max_tokens. The model specifies which AI model to use for the completion, prompt is the text input for which you seek a completion, and max_tokens limits the length of the generated response.

  • requestBodyJson: The request body is serialized into a JSON string using JsonUtility.ToJson.

  • UnityWebRequest: This object is used to configure and send the HTTP POST request. The request URL is the apiURL, and the serialized JSON is the body of the request.

  • SetRequestHeader is called to add necessary HTTP headers, including Content-Type as “application/json” and Authorization with the Bearer token (your apiKey).

  • uploadHandler is assigned a new UploadHandlerRaw initialized with the byte array of the JSON request body, allowing the data to be sent as the request body.

  • downloadHandler is assigned a new DownloadHandlerBuffer, which will store the API response.

  • The coroutine then waits (yield return) for the request to complete.

  • Upon completion, if there are no connection or protocol errors (UnityWebRequest.Result.ConnectionError, UnityWebRequest.Result.ProtocolError), it invokes the callback function with the response text as its argument. This allows the calling context to process the API’s response.

  • If an error occurs, it logs the error using Debug.LogError.

Summary

This script is designed to abstract the complexity of communicating with the OctoAI/OpenAI API, handling asynchronous network operations within Unity. It allows Unity developers to send prompts to OctoAI, receive generated text completions, and then use these completions within their Unity projects, such as for controlling NPCs, generating dynamic dialogues, or other creative applications that benefit from natural language processing.

2

The OctoAIChatAgent.cs or OpenAIChatAGent.cs script is designed for Unity’s ML-Agents framework, allowing an agent within Unity to interact with an external AI service through the OctoAIIntegration (OpenAIIntegration) component. The agent seeks advice on movement decisions in a simulated environment, aiming to reach a target position. Here’s a detailed explanation of the code:

Namespaces

  • Unity.MLAgents, Unity.MLAgents.Actuators, Unity.MLAgents.Sensors, and UnityEngine namespaces are included to access the functionality provided by Unity and the ML-Agents toolkit.

OctoAIChatAgent Class

  • Inherits from Agent, a base class provided by ML-Agents, allowing it to function as an intelligent agent within Unity.

Class Variables

  • public Transform Target; references the target GameObject that the agent aims to reach.
  • public float MoveSpeed = 1f; determines the speed at which the agent moves towards the target.
  • private OctoAIIntegration octoAIIntegration; holds a reference to the OctoAIIntegration(OpenAIIntegration) component, which manages requests to the external AI service.

Initialize Method

  • Called when the agent is initialized. It assigns the octoAIIntegration variable by finding an existing OctoAIIntegration (OpenAIIntegration) component in the scene.

OnEpisodeBegin Method

  • Triggered at the start of each training episode. It resets the agent and target positions to start a new episode. The target’s position is randomized within specified bounds.

CollectObservations Method

  • Used to collect the agent’s observations that are fed into the neural network. It adds the agent’s position and the target’s position as observations, providing the neural network with context about their relative locations.

OnActionReceived Method

  • Called when the neural network produces an action. The actions, represented as a continuous vector, dictate the agent’s movement direction. The agent’s position is updated based on these actions, multiplied by the MoveSpeed and the time delta.
  • If the agent gets close enough to the target (distance < 1.42 units), a reward is given, and the episode ends. If not, a small penalty is applied to encourage efficiency.

Heuristic Method

  • Provides a way to manually control the agent for testing purposes. It maps keyboard inputs to actions, allowing human players to control the agent directly.

RequestMovementAdvice Method

  • This method crafts a situation description based on the current positions of the agent and the target. It then requests advice from the external AI service via the OctoAIIntegration component.
  • The response (advice) from the AI service is intended to guide the agent’s movement decisions.

ProcessOctoAIAdvice(ProcessOpenAIAdvice) Method

  • A placeholder method for processing the AI service’s advice. The actual implementation should parse the received advice and translate it into actions or adjustments to the agent’s behavior.
  • As written, it simply logs the received advice, indicating where developers should implement the logic for acting on the advice.

Summary
This script demonstrates how to integrate an external AI-powered chat service into a Unity ML-Agents project. It sets the foundation for creating intelligent agents that can request and use natural language advice to make decisions in a simulated environment. The key aspect missing from the script is the logic for interpreting and acting on the advice received from the OctoAI/OpenAI service, which would be specific to the application’s requirements and the nature of the advice provided by the AI.

3

Free feel to drop a comment or assistance or other suggestion. We are here to discuss as well.

4

Thanks for the feedback. However, I need feedback on the idea of AI and ML agents and how to have smarter agents for AI enemies and NPCs with my code.