Determining modules loaded once program starts

Question

I have a forking server. I load all modules before I fork.

...or do I? I want a message to be logged to STDERR for every module loaded after a certain point in my program.

I was going to use the following, but it's logging some scripts executed using do.

my %ignore = map { $_ => 1 } ( ... );
unshift @INC, sub {
   my (undef, $path) = @_;
   warn("$path wasn't loaded before forking.\n")
      if !$ignore{$path};

   return ();
};

I'm not out of ideas, but before I spend more time on this, I was wondering if there's an existing solution.

I could write a solution that dumps `%INC` on exit, but I'd rather avoid that since this is a long-lived process that only exits by signal. — ikegami, May 05 '14 at 20:22
`%ignore` is there because some modules attempt to load others to determine if they are installed. — ikegami, May 05 '14 at 20:23
Do you control the forking in your own Perl code, or is the fork part of some other beast that you just plug into? — Len Jaffe, May 05 '14 at 20:28
If you know about the ones that are `do`ne, why don't you just pot those in `%ignore`? — simbabque, May 05 '14 at 20:31
Btw I think you wanted to say `unless $ignore{$path}`. With the `if` it only `warn`s for the ones it should ignore. — simbabque, May 05 '14 at 20:39
@simbabque, That might be the simplest. I'd have to introduce regex, though. Perl builtins alone load a number of files using `do`. Honestly, I wish I didn't have to use `%ignore` at all. — ikegami, May 05 '14 at 20:41
If you control the forking, dump %INC before the fork, and then dump %INC in the child, later, and compare... — Len Jaffe, May 06 '14 at 15:33
But you're trying to determine the program's run-time profile. You do it once or twice, collect a bunch of logs to analyze, and then you stop doing it. Combine that with some static analysis to pre-compute all of your includes and their dependencies. — Len Jaffe, May 06 '14 at 17:24
@Len Jaffe, Can't do it "once or twice". The load could happen at any time in its "infinitely" long lifetime. And I really hate the idea of buffering/postponing error logging. — ikegami, May 06 '14 at 17:47
sure you can. keep a counter... only dump INC on the when counter < X. — Len Jaffe, May 06 '14 at 19:17
@Len Jaffe, Let me rephrase: I don't want to do it just once or twice. The load could happen at any time in the program's "infinitely" long lifetime, so you're effectively suggesting what I've already said I don't want to do: Waiting until the process exits. That's not really safe here, and generally speaking, I really hate the idea of buffering/postponing error logging. In short, see my first comment. — ikegami, May 06 '14 at 19:21

ysth · Answer 1 · 2014-05-05T20:52:02.337

4

What things are using do? Is simply

if $path =~ /\.pm\z/ && !$ignore{$path};

good enough to distinguish?

Other options:

Static code analysis with PPI (using or based on Module::Extract::Use).

Dump %INC to a file upon SIGHUP or some other signal.

edited May 05 '14 at 20:52

answered May 05 '14 at 20:44

ysth

96,171
6
121
214

Static analysis won't catch DBD::mysql being loaded by DBI, for example. – ikegami May 06 '14 at 19:25
Sending SIGHUP to the forks would be tricky. I'd have to start managing an externally accessible list of the forks. – ikegami May 06 '14 at 19:27
This is probably what I'll use, but I'd rather get rid of `%ignore` altogether. What if I ignore XML::SAX (which XML::Simple attempts to load) now, but I install it later? It fails hard. I like my programs to fail safe. Looks like there is no existing solution, so I guess I'll have to publish what I come up with. Any ideas for a name? – ikegami May 06 '14 at 19:29
Posted a solution that addresses all the concerns. Thanks anyway. – ikegami May 08 '14 at 18:54

score 0 · Accepted Answer · answered May 08 '14 at 18:26

Solution:

my $old_require =
   defined(&CORE::GLOBAL::require)
      ? \&CORE::GLOBAL::require
      : sub { CORE::require($_[0]) };

my $new_require = sub {
   my $path = shift;
   if ($INC{$path}) {
      return $old_require->($path);
   }

   my $rv = $old_require->($path);
   warn("Loaded $path after fork\n");
   return $rv;
};

no warnings 'redefine';
*CORE::GLOBAL::require = $new_require;

The solution has the following features:

Logs modules loaded by require EXPR.
Logs modules loaded by require BAREWORD.
Logs modules loaded by use BAREWORD.
Ignores code loaded by do EXPR.
Ignores modules that fail to load.
Ignores modules that have already been loaded.
Works even if @INC is manipulated.

The last four were problems experienced by the solution in the OP. I can't find any downsides to this solution.

One downside: A file without a `package` will place symbols in the wrong namespace. But you shouldn't be passing such a file to `require` in the first place! (You should be using `do`.) — ikegami, May 08 '14 at 18:53

Determining modules loaded once program starts

2 Answers2