How to store data from hash in file and use it for another hash

Question

I'm new in the Perl's world and I need help regarding the save data from a hash in a file and after that use it in another hash.

This is а short example of my code:

#!/usr/local/bin/perl
use FileHandle;
use File::Copy;
use Data::Dumper qw(Dumper);
use Storable;
use warnings;
use strict;

my $user_data;

$user_data->{test_user}->{1256489043}->{STATUS}     =  "RUN";
$user_data->{test_user}->{1256489043}->{MEM}        =  "51591";
$user_data->{test_user}->{1256489043}->{RUN_TIME}   =  "41410";
$user_data->{test_user}->{1256489043}->{PROJ_NAME}  =  "unkown";
$user_data->{test_user}->{1256489043}->{GROUP}      =  "default";
$user_data->{test_user}->{1256489043}->{DATE}       =  "Aug 17 05:23";

$user_data->{test_user_2}->{528562752}->{STATUS}     =  "RUN";
$user_data->{test_user_2}->{528562752}->{MEM}        =  "591";
$user_data->{test_user_2}->{528562752}->{RUN_TIME}   =  "46410";
$user_data->{test_user_2}->{528562752}->{PROJ_NAME}  =  "unkown";
$user_data->{test_user_2}->{528562752}->{GROUP}      =  "default";
$user_data->{test_user_2}->{528562752}->{DATE}       =  "Aug 17 05:23";

store (\$user_data, 'temp_jobs.txt') or die "can't store data to $!";
my $data = retrieve('temp_jobs.txt');

print "Hash 1\n";
print Dumper \$user_data;

print "Hash2\n";
print Dumper \$data;

my @new_array_id;
foreach my $user (keys %{$user_data}) {
   foreach my $job_id (keys %{$user_data->{$user}}) {
      push (@new_array_id, $job_id)
   }
}

my @old_array_id;
foreach my $user (keys %{$data}) {
   foreach my $job_id (keys %{$data->{$user}}) {
      push (@old_array_id, $job_id)
   }
}

These are outputs from Dumper prints:

$VAR1 = \{
            'test_user_2' => {
                               '528562752' => {
                                                'GROUP' => 'default',
                                                'PROJ_NAME' => 'unkown',
                                                'DATE' => 'Aug 17 05:23',
                                                'STATUS' => 'RUN',
                                                'RUN_TIME' => '46410',
                                                'MEM' => '591'
                                              }
                             },
            'test_user' => {
                             '1256489043' => {
                                               'GROUP' => 'default',
                                               'PROJ_NAME' => 'unkown',
                                               'DATE' => 'Aug 17 05:23',
                                               'STATUS' => 'RUN',
                                               'RUN_TIME' => '41410',
                                               'MEM' => '51591'
                                             }
                           }
          };

$VAR1 = \\{
              'test_user' => {
                               '1256489043' => {
                                                 'GROUP' => 'default',
                                                 'PROJ_NAME' => 'unkown',
                                                 'DATE' => 'Aug 17 05:23',
                                                 'STATUS' => 'RUN',
                                                 'RUN_TIME' => '41410',
                                                 'MEM' => '51591'
                                               }
                             },
              'test_user_2' => {
                                 '528562752' => {
                                                  'GROUP' => 'default',
                                                  'PROJ_NAME' => 'unkown',
                                                  'DATE' => 'Aug 17 05:23',
                                                  'STATUS' => 'RUN',
                                                  'RUN_TIME' => '46410',
                                                  'MEM' => '591'
                                                }
                               }
            };

The error is observed at second foreach loop:

foreach my $user (keys %{$data}) {
   foreach my $job_id (keys %{$data->{$user}}) {
      push (@old_array_id, $job_id)
   }
}

output: Not a HASH reference at report.pl`

Actually, I'm not sure about the type of the retrieved data. Could you please help with saving the data in the new hash?

Best Regards, SK

Tip: `store (\$user_data, ...` should be `store ($user_data, ...`. One wants to pass a scalar to `Dumper`, so one uses `\@a` and `\%h` for arrays and hashes, but a scalar is already a scalar, so using `\$s` just complicates things for nothng — ikegami, Aug 17 '20 at 18:36
woops, I didn't realize this was the underlying cause of the problem — ikegami, Aug 17 '20 at 18:37

score 4 · Accepted Answer · answered Aug 17 '20 at 16:25

Your $user_data variable contains a hash reference. A hash reference is a scalar value. I think you're getting confused by the synopsis of the documentation for Storable which contains examples like this:

store \%table, 'file';

In the example above, %table is a hash. You, therefore, need to take a reference to it in order to pass it to store(). Because what you have in $user_data is a hash reference, not a hash, you don't need to take its reference before passing it to store(). In fact, by taking its reference, you've added an extra level of indirection which breaks your code.

You say:

Actually, I'm not sure about the type of the retrieved data.

It's a reference to a hash reference. But your code is expecting just a hash reference.

The simplest fix is to replace:

store (\$user_data, 'temp_jobs.txt') ... ;

with

store ($user_data, 'temp_jobs.txt') ... ;

With that single-character deletion, your two data structures are the same and your code works as expected.

hoffmeister · Answer 2 · 2020-08-17T15:52:49.393

I like to use Yaml::XS for basic data structures, they're easier to look at later. I've included examples of YAML::XS and Storable

Hopfully this helps

#!/usr/bin/env perl

use strict;
use warnings; 
use Storable;
use YAML::XS;
use Data::Dumper;

my $struct = {
'test_user_2' => {
   '528562752' => {
                    'GROUP' => 'default',
                    'PROJ_NAME' => 'unkown',
                    'DATE' => 'Aug 17 05:23',
                    'STATUS' => 'RUN',
                    'RUN_TIME' => '46410',
                    'MEM' => '591'
                  }
 },
'test_user' => {
 '1256489043' => {
                   'GROUP' => 'default',
                   'PROJ_NAME' => 'unkown',
                   'DATE' => 'Aug 17 05:23',
                   'STATUS' => 'RUN',
                   'RUN_TIME' => '41410',
                   'MEM' => '51591'
                 }
}

};


#print Dumper($struct);

#print Dump($struct);

YAML::XS::DumpFile("store.yaml", $struct);
my $read_yml = YAML::XS::LoadFile("store.yaml");
print "yaml out\n";
print Dumper($read_yml);

store $struct, 'store.perl_str';
my $hashref = retrieve('store.perl_str');
print "store out\n";
print Dumper($hashref);

print "looping  data\n";
for my $user_key ( sort keys %{ $hashref } )
{
    my $user = $hashref->{$user_key};
    for my $id_key ( sort keys %{ $user } )
    {
        print "$user_key - $id_key\n";
        print "   " . $user->{$id_key}{"DATE"} . "\n";
        print "   " . $user->{$id_key}{"STATUS"} . "\n";
    }
}

You fixed the OP's problem (`store $struct` rather than `store \$struct`) but you didn't point out what the actual fix was. Switching to another serialisation format would have made no difference if the OP continued to try to serialise `\$struct` and deserialise into `$struct`. — Dave Cross, Aug 17 '20 at 16:33
@dave you're absolutely right. I missed that. I do feel however, if the code can be simplified then the issues disappear. Storing the data in yaml (or json or something else) does make looking at the file directly far easier, which undoubtedly at some point one will want to do. — hoffmeister, Aug 18 '20 at 00:00
I think your example is confusing because it does the Serialization and Deserialization **twice**. `$read_yml` is actually already de deserialized `$hashref` and it is never used. `YAML::XS::LoadFile()` gives you already the `$hashref` you are looking for according to this explanation: https://www.perl.com/article/29/2013/9/17/How-to-Load-YAML-Config-Files/. You should stick to **only 1 Serialization Method** for a clearer code. — Bodo Hugo Barwich, Aug 25 '20 at 14:32

score 2 · Answer 3 · answered Aug 18 '20 at 04:47

As an option you can save hash into JSON file. Following code demonstrates how desired result can be achieved.

use strict;
use warnings;
use feature 'say';

use JSON;
use Data::Dumper;

my $fname = 'datafile.json';
my $user_data;

$user_data->{test_user}{1256489043}{STATUS}      =  "RUN";
$user_data->{test_user}{1256489043}{MEM}         =  "51591";
$user_data->{test_user}{1256489043}{RUN_TIME}    =  "41410";
$user_data->{test_user}{1256489043}{PROJ_NAME}   =  "unkown";
$user_data->{test_user}{1256489043}{GROUP}       =  "default";
$user_data->{test_user}{1256489043}{DATE}        =  "Aug 17 05:23";

$user_data->{test_user_2}{528562752}{STATUS}     =  "RUN";
$user_data->{test_user_2}{528562752}{MEM}        =  "591";
$user_data->{test_user_2}{528562752}{RUN_TIME}   =  "46410";
$user_data->{test_user_2}{528562752}{PROJ_NAME}  =  "unkown";
$user_data->{test_user_2}{528562752}{GROUP}      =  "default";
$user_data->{test_user_2}{528562752}{DATE}       =  "Aug 17 05:23";

my $json = to_json($user_data);

write_json($fname,$json);

my $data = read_json($fname);

say '--- Read from file -----------------';
say Dumper($data);
say '-' x 45;
say Dumper( jobs_array($user_data) );
say '-' x 45;
say Dumper( jobs_array($data) );


sub jobs_array {
    my $data = shift;
    my @array;
    
    for my $user ( keys %{$data} ) {
        for my $job_id ( keys %{$data->{$user}} ) {
            push @array, $job_id;
        }
    }
    
    return \@array;
}


sub write_json {
    my $fname = shift;
    my $data  = shift;
    
    open my $fh, '>', $fname
        or die "Couldn't open $fname";
        
    say $fh $data;
    
    close $fh;
}

sub read_json {
    my $fname = shift;
    
    open my $fh, '<', $fname
        or "Couldn't open $fname";
        
    my $data = do{ local $/; <$fh> };
    
    close $fh;
    
    my $href = from_json($data);
    
    return $href;
}

Output

--- Read from file -----------------
$VAR1 = {
          'test_user' => {
                           '1256489043' => {
                                             'RUN_TIME' => '41410',
                                             'MEM' => '51591',
                                             'PROJ_NAME' => 'unkown',
                                             'STATUS' => 'RUN',
                                             'GROUP' => 'default',
                                             'DATE' => 'Aug 17 05:23'
                                           }
                         },
          'test_user_2' => {
                             '528562752' => {
                                              'GROUP' => 'default',
                                              'STATUS' => 'RUN',
                                              'MEM' => '591',
                                              'PROJ_NAME' => 'unkown',
                                              'DATE' => 'Aug 17 05:23',
                                              'RUN_TIME' => '46410'
                                            }
                           }
        };

---------------------------------------------
$VAR1 = [
          '1256489043',
          '528562752'
        ];

---------------------------------------------
$VAR1 = [
          '1256489043',
          '528562752'
        ];

I agree that JSON (or YAML) would be a better serialisation format here. But changing the format does not change the underlying issue. If the OP serialises a reference to a hash reference and deserialises it, expecting to get a hash reference, then they will have exactly the same problem no matter which format they choose. — Dave Cross, Aug 18 '20 at 07:29

How to store data from hash in file and use it for another hash

3 Answers3