Well I get a "host was not accessible" error when trying to use a software. I don't know which port they use nor which address they are trying to reach. Is there a more or less easy way to figure that out?
I got a program called wireshark - but the data is just huge. Is there an alternative or way to filter data to get a good overview?
If you have access to the firewall logs look in them.
If it's not in the documentation, you're going to have to sift through a Wireshark or Netmon dump. That being said, a quick call to the mystery application's support staff should resolve your problem.
Are you using Linux? If so, as others mentioned, you access the data using wireshark/tcpdump, filtered for the IP, while using that software.
On the server, you can try running netstat or a similar program depending on your OS that will tell you what ports are being listened to. Use the man page to see if further options (such as what the program command line is for each listened port) can help you narrow it down.
Or you can drop the firewall temporarily and run NMap against the server, scanning all ports, to see what's open. Or scan it from behind your firewall if you're not running a software firewall. Note the open ports and investigate.
If it is on Linux (or some Windows machines) you can look in the config files to see if they have a port specified.
Last, see if you can get in touch with any of the ex-developers from that company, or other customers of that software and see if they can help; if you can find any of them, maybe they know of other customers or have a support forum or other admins that are running the software that can get into a mailing list to help each other out.
Contacting the supplier/manufacturer of the software would seem like a sensible thing to do in this situation.
nestat -nab 1 (in case of Windows, change some parameters in case of Linux)
There are a couple of different options, ranging from "simple and a long shot" to "sure to work but tedious".
You say the network trace is huge. Have you tried to shut down all unnecessary programs and network communication first? A surefire way is to use a Virtual Machine to start the program. This makes it a lot easier for you to trace its activity, since it is limited to a specific IP.
If it is really difficult for you to find the correct network activity, attach a debugger and do a trace on the program. You can filter out calls to well-known application libraries like WinSock, or even set a breakpoint on when the "host was not accessible" error dialog is shown. This is however very tedious and will require some reverse engineering voodoo.
If you have the source code, grep it for the word socket. When you find it, the port will be around that line of code.
