I’m training my RL Agent with single-timestep episode, so i call EndEpisode() method after the first action of the episode.
Also, i’m trying to train my agent with external lib(Pytorch), and the script contains env.reset() in every loop. And the example of my training script is below.
‘’’
for episode in range(num_episodes):
env.reset()
decision_steps, terminal_steps = env.get_steps(behavior_name)
if len(decision_steps) > 0:
state = decision_steps.obs[0][0][:state_size]
state_np = np.array(state)
state_tensor = torch.tensor(state_np, dtype=torch.float32).unsqueeze(0)
noise = np.random.normal(0, 0.1, size=(action_size,))
action = agent.select_action(state_tensor, noise=noise)
action_tuple = ActionTuple(continuous=np.array([action]))
env.set_actions(behavior_name, action_tuple)
env.step()
decision_steps, terminal_steps = env.get_steps(behavior_name)
reward = terminal_steps.reward[0] if len(terminal_steps) > 0 else 0.0
next_state = terminal_steps.obs[0][0][:state_size]
done = 1.0 if len(terminal_steps) > 0 else 0.0
‘’’
So, there is an error that the episode reset twice every time. Actually it doesn’t matter, but i hope to fix it. There are two logs when OnEpisodeBegin() method is called.
New Episode Begin
UnityEngine.Debug:Log (object)
MyArmAgent:OnEpisodeBegin () (at Assets/Scripts/MLAgents/MyArmAgent.cs:51)
Unity.MLAgents.Agent:_AgentReset () (at ./Library/PackageCache/com.unity.ml-agents/Runtime/Agent.cs:1350)
Unity.MLAgents.Agent:EndEpisodeAndReset (Unity.MLAgents.Agent/DoneReason) (at ./Library/PackageCache/com.unity.ml-agents/Runtime/Agent.cs:834)
Unity.MLAgents.Agent:EndEpisode () (at ./Library/PackageCache/com.unity.ml-agents/Runtime/Agent.cs:808)
MyArmAgent/d__25:MoveNext () (at Assets/Scripts/MLAgents/MyArmAgent.cs:117)
UnityEngine.SetupCoroutine:InvokeMoveNext (System.Collections.IEnumerator,intptr)
MyArmAgent:OnEpisodeBegin () (at Assets/Scripts/MLAgents/MyArmAgent.cs:51)
Unity.MLAgents.Agent:_AgentReset () (at ./Library/PackageCache/com.unity.ml-agents/Runtime/Agent.cs:1350)
Unity.MLAgents.Academy:ForcedFullReset () (at ./Library/PackageCache/com.unity.ml-agents/Runtime/Academy.cs:560)
Unity.MLAgents.Academy:OnResetCommand () (at ./Library/PackageCache/com.unity.ml-agents/Runtime/Academy.cs:522)
Unity.MLAgents.RpcCommunicator:SendCommandEvent (Unity.MLAgents.CommunicatorObjects.CommandProto) (at ./Library/PackageCache/com.unity.ml-agents/Runtime/Communicator/RpcCommunicator.cs:306)
Unity.MLAgents.RpcCommunicator:UpdateEnvironmentWithInput (Unity.MLAgents.CommunicatorObjects.UnityRLInputProto) (at ./Library/PackageCache/com.unity.ml-agents/Runtime/Communicator/RpcCommunicator.cs:226)
Unity.MLAgents.RpcCommunicator:SendBatchedMessageHelper () (at ./Library/PackageCache/com.unity.ml-agents/Runtime/Communicator/RpcCommunicator.cs:416)
Unity.MLAgents.RpcCommunicator:DecideBatch () (at ./Library/PackageCache/com.unity.ml-agents/Runtime/Communicator/RpcCommunicator.cs:328)
Unity.MLAgents.Policies.RemotePolicy:DecideAction () (at ./Library/PackageCache/com.unity.ml-agents/Runtime/Policies/RemotePolicy.cs:67)
Unity.MLAgents.Agent:DecideAction () (at ./Library/PackageCache/com.unity.ml-agents/Runtime/Agent.cs:1411)
Unity.MLAgents.Academy:EnvironmentStep () (at ./Library/PackageCache/com.unity.ml-agents/Runtime/Academy.cs:590)
MyGameManager:Update () (at Assets/Scripts/MLAgents/MyGameManager.cs:22)
My assumption is env.reset() resets environment, and EndEpisode() also resets. But, when i erase env.reset(), there is an error that terminal_steps is empty.
How can i fix it? and sorry for my english.