HTTP requests trace

Question

Are there any tools to trace the exact HTTP requests sent by a program?

I have an application which works as a client to a website and facilitates certain tasks (particularly it's a bot which makes automatic offers in a social lending webstite, based on some predefined criteria), and I'm interested in monitoring the actual HTTP requests which it makes.

Any tutorials on the topic?

I'm partial to http://www.charlesproxy.com. – Kylar Apr 30 '13 at 19:19 — Kylar, Apr 30 '13 at 19:19

score 11 · Accepted Answer · edited May 23 '17 at 12:13

11

Some popular protocol/network sniffers are:

Wireshark (previous the famous Ethereal)
Nirsoft SmartSniff (using WinPcap)
Nirsoft SocketSniff (allows you to watch the WinSock activity of the selected process and watch the content of each send or receive call, in Ascii mode or as Hex Dump)
Microsoft's Network Monitor (and a list of video-tutorials here, note video 'Advanced Filtering 2 of 2' where they specifically filter on process)

Wikipedia article 'Comparison of packet analyzers' has a nice overview of some other tools to.

Alternatively you could also look into (man-in-the-middle) proxy tools like:

Both of the above actually record/decrypt/modify/replay HTTPS to!! You'd need to point the application you are monitoring to this proxy. If nothing else uses that proxy the log would be application/process specific and another upside to this approach is that one could also run the monitor/logger on a different machine.

Once you choose a tool, you can easily google a tutorial to go along with it.
However the core idea is usually the same: basically one sets a filter (on capture itself or display of captured data) on things like protocol, network/mac address, portno, etc. Depending on the tool, some can also filter on local application.

Hope this helps!

edited May 23 '17 at 12:13

Community

1
1

answered Apr 29 '13 at 21:23

GitaarLAB

14,536
11
60
80

Sniffing all the entire network just for few requests coming from only 1 program, is it really the best solution? Especially if the asker wants to try it out on the production environment. – MrYoshiji Apr 29 '13 at 21:24
Thanks for the info, but as far as I know, Wireshark captures everything from a chosen network interface. Is it possible to filter the requests coming from a particular program (I haven't seen such option)? – asliwinski Apr 29 '13 at 21:28
@MrYoshiji: since most computers nowadays are behind a switch (that's behind a router/gateway) not a hub, one wouldn't be sniffing the 'entire network' (but in 'worst case' only all traffic that actually reaches the computer). However I understand your point and updated my answer, also addressing the askers comment. – GitaarLAB Apr 29 '13 at 22:47
1

I tried Charles Proxy and Fiddler, and Fiddler finally did the job, as actually I needed to look into HTTPS. I've had some difficulties in configuring Charles Proxy for that purpose, but with Fiddler it went flawless. Thank you so much for your profound answer! :) – asliwinski May 03 '13 at 21:56

score 0 · Answer 2 · answered Jul 24 '19 at 10:22

Take a look at HTTP Toolkit (disclaimer: it's my project).

Totally automatic HTTP & HTTPS interception, with zero setup, isolated to just the code you want to debug.

You can open a browser with it, and see all the traffic from that one window immediately (but no others), or run a terminal and automatically see all traffic only from processes started from that terminal. Built-in HTTPS decryption for everything, with no risky system-wide certificates and no manual setup. Let me know what you think!

HTTP requests trace

2 Answers2