1

I have a port scanning application that uses work queues and threads.

It uses simple TCP connections and spends a lot of time waiting for packets to come back (up to half a second). Thus the threads don't need to fully execute (i.e. first half sends a packet, context switch, does stuff, comes back to thread which has network data waiting for it).

I suspect I can improve performance by modifying the sys.setcheckinterval from the default of 100 (which lets up to 100 bytecodes execute before switching to another thread).

But without knowing how many bytecodes are actually executing in a thread or function I'm flying blind and simply guessing values, testing and relying on the testing shows a measurable difference (which is difficult since the amount of code being executed is minimal; a simple socket connection, thus network jitter will likely affect any measurements more than changing sys.setcheckinterval).

Thus I would like to find out how many bytecodes are in certain code executions (i.e. total for a function or in execution of a thread) so I can make more intelligent guesses at what to set sys.setcheckinterval to.

Dharman
  • 30,962
  • 25
  • 85
  • 135

3 Answers3

3

For higher level (method, class) wise, dis module should help.

But if one needs finer grain, tracing will be unavoidable. Tracing does operate line by line basis but explained here is a great hack to dive deeper at the bytecode level. Hats off to Ned Batchelder.

utku_karatas
  • 6,163
  • 4
  • 40
  • 52
2

Reasoning about a system of this complexity will rarely produce the right answer. Measure the results, and use the setting that runs the fastest. If as you say, testing can't measure the difference in various settings of setcheckinterval, then why bother changing it? Only measurable differences are interesting. If your test run is too short to provide meaningful data, then make the run longer until it does.

Ned Batchelder
  • 364,293
  • 75
  • 561
  • 662
1

" I suspect I can improve performance by modifying the sys.setcheckinterval"

This rarely works. Correct behavior can't depend on timing -- you can't control timing. Slight changes on OS, hardware, patch level of Python or phase of the moon will change how your application behaves.

The select module is what you use to wait for I/O's. Your application can be structured as a main loop that does the select and queues up work for other threads. The other threads are waiting for an entries in their queue of requests to process.

S.Lott
  • 384,516
  • 81
  • 508
  • 779
  • This suggestion is only useful if users have direct control of the I/O operations being performed, in Python or extension code they control. Very often (likely more often), they do not: external drivers for I/O (e.g. to a database) often do not provide support for asynchronous I/O (via select or other async-oriented tools). In situations restricted by such a driver, where the overhead of process-based solutions may not be acceptable, tuning `setcheckinterval` or `setswitchinterval` can be a capable solution. If used thoughtfully, not superstitiously, this can yield massive performance gains. – Zac B Jan 14 '19 at 16:25