The presence of connections in the TIME_WAIT state means that your server is the initiator of their closing phase. It prevents against accepting late segments and ensures that the other end has correctly closed the connection.
It shouldn't be a problem to have a lot of them in most cases, it depends on the services that the server provides.
The main issue with a high number of TIME_WAIT connections is the memory consumption, which is only about 10MB for 40 000 of them.
If you think it may cause harm in your specific case, you can use ss from iproute2 to see which tuples are the culprits:
$ ss -tan state time-wait
Then, with a tool such as wireshark, you can easily analyse the TCP flow corresponding to that tuple with a filter similar to:
tcp.port == <port> and ip.addr == <ip>
You can also use the feature "follow TCP stream" from wireshark that will show you every segment from that specific connection.