0

I have a laravel application that uses a while loop to process a txt file(6000 lines) line by line together with various helpers to manipulate each string. At the end of the manipulation it stores a record in db for each line. Everything works fine, the application process the entire file in a minute and returns the view indicated. The problem is that the application than freeze(not sure if this is the appropriate term), it keeps running and the browser ask repeatedly to wait or close the application. I noticed also this message in the laravel server if it may help

[Wed Apr 21 10:28:26 2021] 127.0.0.1:50897 Closed without sending a request; it was probably just an unused speculative preconnection

Thanks to anyone who could help me.

This is the process:

    $log = new UserActivityHelper;
    $log->create(Config::get('constants.user_activities.FTP_FILE_PROCESSED_START'), 'test');

    $file = fopen(Storage::path("1-2021.txt"), 'r');
    while(!feof($file))
    {
        $line = fgets($file);
       
        //call the method that validate the string
        $string = new ValidateStringHelper($line);  //FIRST HELPER
        $string->validate();

        //check the result of the validation
        if($string->validated == false){
            dd("out");
        } elseif ($string->empty == true){
            continue;
        }else{
            //save the validated array of substrings as variable
            $my_arr = $string->validatedChars;
        
            //check if the province exists or create it
            $province = new ProvinceExistsHelper($my_arr['district']); //SECOND HELPER
            $province->check_if_exists_or_create();

            if($province->new_province == true) {
                $log = new UserActivityHelper;
                $log->create(Config::get('constants.ftp_process.FTP_PROVINCE_CREATED'), "Creata nuova provincia con id {$province->district_id}");
                self::message('yellow', "Attenzione. Nell'elaborazione sono state create nuove provincie, controllare le impostazioni.");
            }
            
            //manipolation of name and lastname
            $name = ucwords(strtolower($my_arr['name_lastname']));
            //check if the person already exists or create it (passing fiscal code, name and district_id)
            $person = new PersonExistsHelper($my_arr['fiscal_code'], $name, $province->district_id); //THIRD HELPER
            $person->check_if_exists_or_create();
            
            $newMovement = new Movement;
            $newMovement->person_id = $person->person_id;
            
            if(array_key_exists('level', $my_arr)){
                $newMovement->level = $my_arr['level'];
            }
            ...

            try {
                $newMovement->save();
            } catch (\Throwable $exception) {
                report($exception);
                return;
            }
           
        }

    }
    fclose($file);
    
    $log = new UserActivityHelper;
    $log->create(Config::get('constants.user_activities.FTP_FILE_PROCESSED_END'), 'test');
    return view('ftp.test');

Helpers called:

class ValidateStringHelper
{
    public $line;
    public $empty = false;
    public $validated = false;
    public $validationErrors = array();
    public $validatedChars = array();

    public function __construct($line){
        $this->line = $line;
    }

    public function validate(){

        if ($this->line!="")
        {
            if(strlen($this->line) >= 186)
            {
                //substrings
                $district = trim(substr($this->line, 0, 2));
                $trade_union_tax_code = trim(substr($this->line, 2, 3));
                ...
                //validation
                //check if validated
                
                $this->validatedChars['district'] = $district;                 
                $this->validatedChars['trade_union_tax_code'] = $trade_union_tax_code;
                ...
                
                //filter all non empty string values and not all zero strings
                $filter_arr = array_filter($this->validatedChars, fn($value) => $value !== '' && preg_filter('/^(?!0*$).*$/', '$0', $value));
                $this->validatedChars = $filter_arr;
                $this->validated = true;
                return $this->validated;
                //return $this->validatedChars;
                //eturn $this->validated;

            } else {
                $this->validationErrors[] = "string_length";
                return $this->validated;
            }

        } else {
            $this->empty = true;
            return $this->empty;
        }
    }
class PersonExistsHelper
{
    public $name_lastname;
    public $fiscal_code;
    public $district_id;
    public $new_person = false;
    public $person_id;

    public function __construct(string $fiscal_code, string $name_lastname, string $district_id){
        $this->fiscal_code = $fiscal_code;
        $this->name_lastname = $name_lastname;
        $this->district_id = $district_id;
    }
    public function check_if_exists_or_create()
    {
        $person = Person::where('fiscal_code', '=', $this->fiscal_code)->first();

        if($person == NULL)
        {
            $this->new_person = true;
            $newPerson = new Person;
            $newPerson->name_lastname = $this->name_lastname;
            $newPerson->district_id = $this->district_id;
            $newPerson->fiscal_code = $this->fiscal_code;
            $newPerson->created_by = Config::get('constants.people_creation.FTP_PROCESS');
            $newPerson->save();
            $this->person_id = $newPerson->id;
            return $this->person_id;
        } else{
        $this->person_id = $person->id;  
        return $this->person_id;
        }
    }
}
class ProvinceExistsHelper
{
    public $district_id;
    public $district_code;
    public $new_province = false;

    public function __construct($district_code){
        $this->district_code = $district_code;
    }

    public function check_if_exists_or_create()
    {
        //check if the province is in the Provinces table
        $province = Province::where('code', '=', $this->district_code)->first();
        
        if(!isset($province))
        {
            $this->new_province = true;                
            $newProvince = new Province;
            $newProvince->name = $this->district_code;
            $newProvince->code = $this->district_code;
            $newProvince->save();
            $this->district_id = $newProvince->id;
            return $this->district_id;

        } else {
        //if yes, just return the id of the province
        $this->district_id = $province->id;
        return $this->district_id;
        }
    }  
}
  • "Everything works fine" except your code takes more than one minute to process the file, the whole time your application is KO, a php worker is still processing the file. or at least your code is still running. either increase the php workers number or simply use job/queue to process the file in the background so the application stays online – N69S Apr 21 '21 at 08:56
  • @N69S ok I'll try the queue job to see If it works, thanks! – Alessandro Boscato Apr 21 '21 at 09:27
  • I agree with @N69S, when you have to do some heavy tasks like this one, try to always use `Jobs`. One tip I can share with you is to use `Guard Clause` and ditch the `if (true) { main code } else { exit }` that is absolutely horrible. You will have better code if you start using them! Read more [here](https://engineering.helpscout.com/reducing-complexity-with-guard-clauses-in-php-and-javascript-74600fd865c7) and [here](https://codeinphp.github.io/post/replace-nested-conditions-with-guard-clauses/) – matiaslauriti Apr 21 '21 at 10:17
  • @N69S in addition I would like to say that records are created perfectly (this is what I meant with “everything works fine”) but the process keeps running don’t know for which reason, maybe for an error in the script. Having said that, I’ve implemented queues but now I need to configure supervisor to do it automatically. Before doing so, I would like yo understand if this is strictly necessaire, thanks. – Alessandro Boscato Apr 23 '21 at 05:34
  • The queue agent is meant to keep running till it breaks/restart. It will handle new jobs on the fly automatically. The superviser will start the queue agent if not running (server restart. error exit, ...) That's why the supervisor is daemonized – N69S Apr 23 '21 at 09:20

0 Answers0