When too few threads, program hangs

Question

When the following program is run for -t=17 -fn=15, then the program halts in a reasonable time. This means the maximum number of threads is greater than the number of files that are written and concurrently cached.

Update as requested: profiler output when $ perl6 --profile con-test.p6 -t=7 -fn=5

When -t >= -fn then the program fails to halt!

When the program is run for -t=17 -fn=20 --hack the program then runs to completion in a reasonable time.

I do not know whether the program below is failing because I am hitting a problem in CompUnit, or whether there is a thread (or other) issue I am missing.

Assuming that there is some sort of thread situation and that the number of files must be less than the number of threads, what would be the best way to rewrite the hack. At present, an array is filled with Promises, which are then allowed to be kept. But await requires all of them to be kept before emptying the array and refilling it. I think that somehow this should be re-written as a Channel or a Supply. However, I cannot quite work out how to do this.

Thanks in advance.

use v6.c;
use nqp;
use File::Directory::Tree;

my Lock $lock;
my $precomp;
#assume a writable directory files/

sub MAIN(:$t=5, :$fn = 10, :$hack=False ) {
    PROCESS::<$SCHEDULER> = ThreadPoolScheduler.new(initial_threads => 0, max_threads => $t);

    rmtree '.test';
    empty-directory 'files';

    my $precomp-store = CompUnit::PrecompilationStore::File.new(prefix => '.test'.IO );
    $precomp = CompUnit::PrecompilationRepository::Default.new(store => $precomp-store);
    $lock .=new;
    my %files = |gather for ^$fn {
        my $f = "files/name_$_.pod6";
        $f.IO.spurt: data;
        take "name_$_" => %(:key(nqp::sha1($f)), :path($f))
    }
    my @compilations;
    my @compiled;
    for %files.kv -> $source, (:path($path), :key($key)) {
        @compilations.push: start compile( $source, $key, $path );
        if $hack {
           if @compilations.elems %% ($t - 2) {
              @compiled.append: await @compilations;
              @compilations = ()
          }
       }
    }
    @compiled.append: await @compilations;
    for @compiled {
        if .<error>.defined {
            say .<error>
        }
        else {
            say .<source> ~ ' compiled'
        }
    }
}

sub compile( $source, $key, $path ) {
    my ($handle , $error, $status );
    try {
        CATCH {
            default {
                $error = "Compile error in $source:\n\t" ~ .Str
            }
        }
        $lock.protect( {
            $precomp.precompile($path.IO, $key, :force );
            $handle = $precomp.load($key)[0];
        })
    }
    with $handle {
        $status = 'OK';
    }
    else {
        $status = 'Failed';
        $error = 'unknown precomp error' without $error; # make sure that $error is defined for no handle
    }
    %(:$error, :$status, :$source)
}

sub data(-->Str ) {
    q:to/DD/
    =begin pod :tag<self>

    =TITLE Community
    X<|Community>

    =SUBTITLE Information about the people working on and using Perl 6

    =head1 Overview

    "Perl 5 was my rewrite of Perl.  I want Perl 6 to be the community's rewrite
    of Perl and of the community." - Larry Wall

    =head1 The Perl 6 community

    There is a large presence on the C<#perl6> channel on C<freenode.net>,
    who are happy to provide support and answer questions. More resources can be found in the L<perl6.org community page|https://perl6.org/community/>. L<Camelia|https://perl6.org/>, the multi-color butterfly with P 6 in her wings, is the symbol of this diverse and welcoming community. We use extensively
    the L<C<#perl6>|https://perl6.org/community/irc> IRC channel for communication, questions and simply hanging out. Check out this L<IRC lingo|http://www.ircbeginner.com/ircinfo/abbreviations.html> resource for the abbreviations frequently used there.  L<StackOverflow|https://stackoverflow.com/questions/tagged/perl6> is also a great resource for asking questions and helping others with their Perl 6 problems and challenges.

    The Perl 6 community publishes every December an L<Advent Calendar|https://perl6advent.wordpress.com/>, with Perl 6 tutorials every day until Christmas. Organization and assignment of days is done through the different Perl 6 channels and the L<Perl6/mu|https://github.com/perl6/mu> repository. If you want to participate, it starts organization by the end of October, so check out the channels above for that.

    =end pod

    DD
}

Answer 1

The problem is that if there are too many threads required, then the program hangs, so a workaround is to create a queue, allocate as many threads are possible, then when the routine in each thread completes, it pops the data off the queue.

Below I have implemented this strategy, and it handles, without freezing, the case when the number of threads is less than the number of files. Actually, the number of threads must be two less than the number of files (see if +@threads < $t - 2). Reducing the 2 to 1 causes the program to hang again. Not entirely sure why this is.

The following is a rewrite of the program in the question with a queue.

#!/usr/bin/env perl6
use v6.d;
use nqp;
use File::Directory::Tree;

my Lock $lock;
my $precomp;
#assume a writable directory files/

sub MAIN(:$t=15, :$fn = 10 ) {
    say "Threads: $t, Files: $fn, Compiler:", $*PERL;
    PROCESS::<$SCHEDULER> = ThreadPoolScheduler.new(initial_threads => 0, max_threads => $t);

    rmtree '.test';
    empty-directory 'files';

    my $precomp-store = CompUnit::PrecompilationStore::File.new(prefix => '.test'.IO );
    $precomp = CompUnit::PrecompilationRepository::Default.new(store => $precomp-store);
    $lock .=new;
    my %files = |gather for ^$fn {
        my $f = "files/name_$_.pod6";
        $f.IO.spurt: data;
        take "name_$_" => %(:key(nqp::sha1($f)), :path($f))
    }
    my @compilations;
    my @compiled;
    my @threads; 
    for %files.kv -> $source, (:path($path), :key($key)) {
        @compilations.push: ( $source, $key, $path );
        @threads.push( start
            sub ( @queue ) {
                my @params = @queue.pop.list if @queue;
                return unless +@params;
                my $res = compile( |@params );
                $lock.protect({
                    @compiled.append: $res;
                });
                &?ROUTINE( @queue )
            }( @compilations )
        )  if +@threads < $t - 2;
    }
    await @threads;
    for @compiled {
        if .<error>.defined {
            say .<error>
        }
        else {
            say .<source> ~ ' compiled'
        }
    }
}

sub compile( $source, $key, $path ) {
    my ($handle , $error, $status );
    try {
        CATCH {
            default {
                $error = "Compile error in $source:\n\t" ~ .Str
            }
        }
        $precomp.precompile($path.IO, $key, :force );
        $handle = $precomp.load($key)[0];
    }
    with $handle {
        $status = 'OK';
    }
    else {
        $status = 'Failed';
        $error = 'unknown precomp error' without $error; # make sure that $error is defined for no handle
    }
    %(:$error, :$status, :$source)
}

sub data(-->Str ) {
    q:to/DD/
    =begin pod :tag<self>

    =TITLE Community
    X<|Community>

    =SUBTITLE Information about the people working on and using Perl 6

    =head1 Overview

    "Perl 5 was my rewrite of Perl.  I want Perl 6 to be the community's rewrite
    of Perl and of the community." - Larry Wall

    =head1 The Perl 6 community

    There is a large presence on the C<#perl6> channel on C<freenode.net>,
    who are happy to provide support and answer questions. More resources can be found in the L<perl6.org community page|https://perl6.org/community/>. L<Camelia|https://perl6.org/>, the multi-color butterfly with P 6 in her wings, is the symbol of this diverse and welcoming community. We use extensively
    the L<C<#perl6>|https://perl6.org/community/irc> IRC channel for communication, questions and simply hanging out. Check out this L<IRC lingo|http://www.ircbeginner.com/ircinfo/abbreviations.html> resource for the abbreviations frequently used there.  L<StackOverflow|https://stackoverflow.com/questions/tagged/perl6> is also a great resource for asking questions and helping others with their Perl 6 problems and challenges.

    The Perl 6 community publishes every December an L<Advent Calendar|https://perl6advent.wordpress.com/>, with Perl 6 tutorials every day until Christmas. Organization and assignment of days is done through the different Perl 6 channels and the L<Perl6/mu|https://github.com/perl6/mu> repository. If you want to participate, it starts organization by the end of October, so check out the channels above for that.

    =end pod

    DD
}