Connecting Unity simulation with python controller over TCP

Question

for the purpose of a pilot study i try to connect a simple simulated environment in Unity with a python controller based on neural network. The main character in the simulated Unity environment should learn a behavior based on it's experience:

The main character scans the environment and stores it in a data structure;
This data structure is sent via TCP to a running server in python;
A complex calculation (based on the data received from Unity) is performed in python and it produces a result (action);
The result (action) is sent back via TCP to Unity. The character performs the action corresponding to the result.
Steps 1-4 are repeated until infinity (unless the client or server stops).

This process can be abstracted down to a Ping - Pong message sending via TCP where Unity sends a "Ping", Python does some calculation and produces a "Pong" which is being sent back to Unity which updates the simulation and again produces a "Ping" and sends it to python...

I currently test this process in a toy project. I have a script attached to the main camera in a Unity scene that looks like this:

using UnityEngine;
using System.Collections;
using System;
using System.IO;
using System.Net.Sockets;
using Newtonsoft.Json;
using System.Threading;



public class networkSocketSO : MonoBehaviour
{
    public String host = "localhost";
    public Int32 port = 50000;

    internal Boolean socket_ready = false;
    internal String input_buffer = "";

    TcpClient tcp_socket;
    NetworkStream net_stream;

    StreamWriter socket_writer;
    StreamReader socket_reader;

    private void Start()
    {
        setupSocket();
    }


     void Update()
     {
        string received_data = readSocket();
        switch (received_data)
             {
                 case "pong":
                     Debug.Log("Python controller sent: " + (string)received_data);
                     writeSocket("ping");
                     break;
                 default:
                     Debug.Log("Nothing received");
                     break;
         }
     }

    void OnApplicationQuit()
    {
        closeSocket();
    }

    // Helper methods for:
    //...setting up the communication
    public void setupSocket()
    {
        try
        {
            tcp_socket = new TcpClient(host, port);
            net_stream = tcp_socket.GetStream();
            socket_writer = new StreamWriter(net_stream);
            socket_reader = new StreamReader(net_stream);
            socket_ready = true;
        }
        catch (Exception e)
        {
            // Something went wrong
            Debug.Log("Socket error: " + e);
        }
    }

    //... writing to a socket...
    public void writeSocket(string line)
    {
        if (!socket_ready)
            return;

        line = line + "\r\n";
        socket_writer.Write(line);
        socket_writer.Flush();
    }

    //... reading from a socket...
    public String readSocket()
    {
        if (!socket_ready)
            return "";

        if (net_stream.DataAvailable)
            return socket_reader.ReadLine();

        return "";
    }

    //... closing a socket...
    public void closeSocket()
    {
        if (!socket_ready)
            return;

        socket_writer.Close();
        socket_reader.Close();
        tcp_socket.Close();
        socket_ready = false;
    }
}

And a python server with the following code:

import socket
import json

host = 'localhost' 
port = 50000
backlog = 5 
size = 1024 
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
s.bind((host,port)) 
s.listen(backlog) 

while 1:
    client, address = s.accept() 
    print "Client connected."
    while 1:
        data = client.recv(size)
        if data == "ping":
            print ("Unity Sent: " + str(data))
            client.send("pong")
        else:
            client.send("Bye!")
            print ("Unity Sent Something Else: " + str(data))
            client.close()
            break

If i start the server and then run the Unity project i get weird things going on. Namely, the python server does not seem to understand the "ping" string sent by the Unity client. Moreover, i want the frames in the Unity simulation not to update before it receives the "Pong" message from the python server.

I've read about solutions of similar problems for using coroutines or threads but i fear that it will establish another "clock" and result with no real ping-pong-updateframe-ping-pong-updateframe... process.

Any ideas?

Thanks in advance!!!

P.S. please have in mind that the real messages will not be "Ping" and "Pong" but proper Json objects. This is just a simplification.

EDIT: I first start the python server and then the Unity simulation (in editor). On the python console i get:

Client connected.
Unity Sent Something Else: ping

...which justifies my claim that "python does not understand the strings sent via Unity (C#)"

In Unity console i get:

This is more than one question. For the first one: Does the python server receive anything at all? What does your programm print? — Dschoni, Mar 01 '17 at 12:47
I just edited the original question with the python prints and Unity console logs. And... thanks for the response guys. — thecritter, Mar 01 '17 at 18:38

score 0 · Answer 1 · answered Mar 02 '17 at 08:49

I just found the solution to my first question, namely "python not understanding the string sent from Unity". It seems that the newline sequences caused the problem.

I just removed the:

line = line + "\r\n";

line in the original code. This way, no additional escape character is added to the string. Adding \r\n caused python's socket.recv() to compare ping with ping\r\n and it always triggered the default case in the switch statement.

Other way around (from python) i send pong\n (note the escape character) in order Unity's (C#) streamWriter.ReadLine() to be able to receive clear pong.

Now the mutual message understanding works. However, it does not resolve my second problem - reaching the ping - pong - update frame process.

score 0 · Answer 2 · answered Nov 09 '17 at 22:21

You should make the response async. You can use a callback. Your error is that you are doing continous polling, and the socket available call will be always false.

Try this code in Unity:

using UnityEngine;
using System.Collections;
using System;
using System.Net;
using System.Net.Sockets;

public class Client : MonoBehaviour {

private Socket _clientSocket = new Socket(AddressFamily.InterNetwork,SocketType.Stream,ProtocolType.Tcp);
private byte[] _recieveBuffer = new byte[8142];

void Start() {
    SetupServer ();
}

void Update() {

}

void OnApplicationQuit()
{
    _clientSocket.Close ();
}

private void SetupServer()
{
    try
    {
        _clientSocket.Connect(new IPEndPoint(IPAddress.Loopback,50003));
    }
    catch(SocketException ex)
    {
        Debug.Log(ex.Message);
    }

    _clientSocket.BeginReceive(_recieveBuffer,0,_recieveBuffer.Length,SocketFlags.None,new AsyncCallback(ReceiveCallback),null);

}


private void ReceiveCallback(IAsyncResult AR)
{
    //Check how much bytes are recieved and call EndRecieve to finalize handshake
    int recieved = _clientSocket.EndReceive(AR);

    if(recieved <= 0)
        return;

    //Copy the recieved data into new buffer , to avoid null bytes
    byte[] recData = new byte[recieved];
    Buffer.BlockCopy(_recieveBuffer,0,recData,0,recieved);

    //Process data here the way you want , all your bytes will be stored in recData
    Debug.Log(System.Text.Encoding.Default.GetString(_recieveBuffer));
    SendData (System.Text.Encoding.Default.GetBytes ("ping"));

    //Start receiving again
    _clientSocket.BeginReceive(_recieveBuffer,0,_recieveBuffer.Length,SocketFlags.None,new AsyncCallback(ReceiveCallback),null);
}

private void SendData(byte[] data)
{
    SocketAsyncEventArgs socketAsyncData = new SocketAsyncEventArgs();
    socketAsyncData.SetBuffer(data,0,data.Length);
    _clientSocket.SendAsync(socketAsyncData);
}

}

Connecting Unity simulation with python controller over TCP

2 Answers2