Running a python threaded TCP server in the background forever

Question

Being rather new to socket programming and threading in general, my issue might be stemming from a misunderstanding of how they work. I am trying to create something that acts as both a client and a server using threading.

Following: https://docs.python.org/3/library/socketserver.html#asynchronous-mixins

I created a client class to go with the server and executed both from a main class. The server supposedly launches normally and doesn't give any errors but when I try to connect from the client, it fails on the following:

    # In client connection
    sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    sock.connect((ip, port))

the error I am getting is:

Traceback (most recent call last):
  File "./<project>", line 61, in <module>
    client.start_client(serverInfo)
  File "/home/<username>/Documents/Github/project/client.py", line 54, in <startclient>
    <connectionMethod>(cmd)
  File "/home/<username>/Documents/Github/project/client.py", line 112, in <connectionMethod>
    sock.connect((remoteHOST,remotePORT))
    ConnectionRefusedError: [Errno 111] Connection refused

Even when I modify the server from the code in the referenced python page (just to run on a specific port, 1234), and try to connect to that port with

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
    sock.connect(('127.0.0.1',1234))
    sock.sendall('Test message')

I get the same problem

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ConnectionRefusedError: [Errno 111] Connection refused

Why is the server refusing connections? No firewall rules or iptables are in place, running the example that has the client and socket together as is from the site works but even removing the server.shutdown() line it still kills itself immediately.

Am I missing something?

The behaviour I am expecting is:

./programA
<server starts on port 30000>

./programB
<server starts on port 30001>
<client starts>

--- input/output ---

FROM CLIENT A:

/connect 127.0.0.1 30001
CONNECTED TO 127.0.0.1 30001

ON CLIENT B:

CONNECTION FROM 127.0.0.1 30000

Basically once the clients connect the first time they can communicate with each other by typing into a prompt which just creates another socket targeting the 'remote' IP and PORT to send off the message (instant messaging). The sockets are afterwords left to close because they aren't actually responsible for receiving anything back.

I would appreciate any help I can get on this.

Answer 1

The reason why even removing server.shutdown() is not helping is because when the main thread exists, the server thread exists immediately because you probably have not removed server_thread.daemon=True.
If you change it to server_thread.daemon=False then also even after main thread terminates, the program will not return to terminal immediately but it will not kill the server either even after last line of the main program gets executed.
I am not sure if you can run a server completely in the background forever using simple python script but I do not think it is possible.

You can call the script with '&' option: python3 <file>.py & which will run the python script in background(this works with Ubuntu at least)(https://askubuntu.com/questions/396654/how-to-run-the-python-program-in-the-background-in-ubuntu-machine) but it probably is not the answer you are looking for.

Answer 2

Incidentally, I've made something of a breakthrough. While I'm not entirely sure why it does it, threadedTCPServer (as it is on the python docs) is basically unusable for this purpose and won't act right while actually in a thread. Instead I opted to manually build a socket server like so:

import threading
import socket

def simpleServer():
    HOST = '127.0.0.1'
    PORT = 3000
    sock = socket.socket()
    sock.bind((HOST,PORT))

    s.listen(5)
    while True:
        conn, addr = s.accept()
        msg = str(conn.recv(1024),'utf8')
        c.send(bytes('GOT: {0}'.format(msg),'utf8')
        c.close()

if __name__ == '__main__':

    server_thread = threading.Thread(target=simpleServer)
    server_thread.start()

    # Make connections based on '<ip>:<port>:<msg>'
    while True:
        cmd = input('>> ')

        try:
            parts = cmd.split(':')
            ip = parts[0]
            port = parts[1]
            msg = parts[2]

            with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
                sock.connect((ip,port))
                sock.sendall(bytes(msg,'utf8'))
                print(str(sock.recv(1024),'utf8'))

        except Exception as e:
            print(e)

The code is pretty basic (and probably not the best) but I have just tested it and it is definitely working for now. I adapted this code based on a solution I saw in: Why am I getting the error "connection refused" in Python? (Sockets)

I do have to readapt a large portion of code but I think it should definitely work, I'll report back after implementation.