Remove null fields in JSON String in Perl

Question

I am modifying a Perl script that collect some data from various sources, put them into a structured JSON and call a web service passing that JSON file. I have a problem with null values in some of these fields ...

This is a snippet of my code:

  $p->{BAND_ATTEN_DN1} = 'null';
  $p->{BAND_ATTEN_DN2} = 'null';
  $p->{BAND_ATTEN_DN3} = 'null';
  $p->{BAND_ATTEN_DN4} = 'null';
  $p->{BAND_ATTEN_UP0} = 'null';
  $p->{BAND_ATTEN_UP1} = 'null';
  $p->{BAND_ATTEN_UP2} = 'null';
  $p->{BAND_ATTEN_UP3} = 'null';
  $p->{BAND_ATTEN_UP4} = 'null';
  $p->{BAND_SNR_MARGIN_DN1} = 'null';
  $p->{BAND_SNR_MARGIN_DN2} = 'null';
  $p->{BAND_SNR_MARGIN_DN3} = 'null';
  $p->{BAND_SNR_MARGIN_DN4} = 'null';
  $p->{BAND_SNR_MARGIN_UP0} = 'null';
  $p->{BAND_SNR_MARGIN_UP1} = 'null';
  $p->{BAND_SNR_MARGIN_UP2} = 'null';
  $p->{BAND_SNR_MARGIN_UP3} = 'null';
  $p->{BAND_SNR_MARGIN_UP4} = 'null';
  $p->{BAND_SIG_ATTEN_DN1} = 'null';
  $p->{BAND_SIG_ATTEN_DN2} = 'null';
  $p->{BAND_SIG_ATTEN_DN3} = 'null';
  $p->{BAND_SIG_ATTEN_DN4}  = 'null';

  $p->{BAND_SIG_ATTEN_UP0} = 'null';
  $p->{BAND_SIG_ATTEN_UP1} = 'null';
  $p->{BAND_SIG_ATTEN_UP2} = 'null';
  $p->{BAND_SIG_ATTEN_UP3} = 'null';
  $p->{BAND_SIG_ATTEN_UP4} = 'null';
  $p->{ATTAINABLE_DN} = 'null';
  $p->{ATTAINABLE_UP} = 'null';

  $p->{POWER_DN} = 'null';
  $p->{POWER_UP} = 'null';

  for my $k (keys %$q) {
    $p->{$k} =  ($q->{$k}) ? $q->{$k} : 'null';
  }

  my $m = $measure->{$port};
  for my $k (keys %$m) {
    $p->{$k} =  ($m->{$k}) ? $m->{$k} : 'null';
  }

  my $hlog_up_values = &get_values($delt->{UPHLOG}, 9999, -99);
  my $hlog_dn_values = &get_values($delt->{DOWNHLOG}, 9999, -99);
  my $qln_up_values = &get_values($delt->{UPQLN}, 9999, -199);
  my $qln_dn_values = &get_values($delt->{DOWNQLN}, 9999, -199);
  my $snr_up_values = &get_values($delt->{UPSNR}, 9999, -99);
  my $snr_dn_values = &get_values($delt->{DOWNSNR}, 9999, -99);

  my $hlog_cg_size_dn = grep ($delt->{DOWNHLOGGRUOPSIZE} eq $_,(1,2,4,8,16)) ? $delt->{DOWNHLOGGRUOPSIZE} : $delt->{DOWNHLOGGRUOPSIZE}/4.3125;
  my $hlog_cg_size_up = grep ($delt->{UPHLOGGRUOPSIZE} eq $_,(1,2,4,8,16)) ? $delt->{UPHLOGGRUOPSIZE} : $delt->{UPHLOGGRUOPSIZE}/4.3125;
  my $qln_cg_size_dn = grep ($delt->{DOWNQLNGRUOPSIZE} eq $_,(1,2,4,8,16)) ? $delt->{DOWNQLNGRUOPSIZE} : $delt->{DOWNQLNGRUOPSIZE}/4.3125;
  my $qln_cg_size_up = grep ($delt->{UPQLNGRUOPSIZE} eq $_,(1,2,4,8,16)) ? $delt->{UPQLNGRUOPSIZE} :$delt->{UPQLNGRUOPSIZE}/4.3125;
  my $snr_cg_size_dn = grep ($delt->{DOWNSNRGRUOPSIZE} eq $_,(1,2,4,8,16)) ? $delt->{DOWNSNRGRUOPSIZE} :$delt->{DOWNSNRGRUOPSIZE}/4.3125;
  my $snr_cg_size_up = grep ($delt->{UPSNRGRUOPSIZE} eq $_,(1,2,4,8,16)) ? $delt->{UPSNRGRUOPSIZE} : $delt->{UPSNRGRUOPSIZE}/4.3125;

  my $pots_line_length_A    = defined $plt->{POTS_LINE_LENGTH_A} ? $plt->{POTS_LINE_LENGTH_A} : 'null';
  my $pots_line_length_B    = defined $plt->{POTS_LINE_LENGTH_B} ? $plt->{POTS_LINE_LENGTH_B} : 'null';
  my $pots_line_state       = defined $plt->{POTS_LINE_STATE} ? $plt->{POTS_LINE_STATE} : 'null';
  my $pots_line_termination = defined $plt->{POTS_LINE_TERMINATION_TYPE} ? $plt->{POTS_LINE_TERMINATION_TYPE} : 'null';


  #&_trace("STUB GROUP SIZE",$function, TRC_PDNT);
  #$hlog_cg_size_dn = 8;
  #$hlog_cg_size_up = 8;
  #$qln_cg_size_dn = 8;
  #$qln_cg_size_up = 8;
  #$snr_cg_size_dn = 8;
  #$snr_cg_size_up = 8;


  my $json_string = "{
    \"attainable_rate\": {
      \"down_value\": $p->{ATTAINABLE_DN},
      \"up_value\": $p->{ATTAINABLE_UP}
    },
    \"hlog\": {
      \"up_values\": [$hlog_up_values],
      \"down_values\": [$hlog_dn_values]
    },
    \"hlog_cg_size\": {
      \"down_value\": $hlog_cg_size_dn,
      \"up_value\": $hlog_cg_size_up
    },
    \"qln\": {
      \"down_values\": [$qln_up_values],
      \"up_values\": [$qln_dn_values]
    },
    \"qln_cg_size\": {
      \"down_value\": $qln_cg_size_dn,
      \"up_value\": $qln_cg_size_up
    },
    \"snr\": {
      \"up_values\": [$snr_up_values],
      \"down_values\": [$snr_dn_values]
    },
    \"snr_cg_size\": {
      \"down_value\": $snr_cg_size_dn,
      \"up_value\": $snr_cg_size_up
    },
    \"spectrum\": \"$actual_spectrum\",

    \"rtx_status\": {
      \"down_value\": [
        null
      ],
      \"up_value\": [
        null
      ]
    },
    \"line_attenuation\": {
      \"down_values\": [
        $p->{BAND_ATTEN_DN1},
        $p->{BAND_ATTEN_DN2},
        $p->{BAND_ATTEN_DN3},
        $p->{BAND_ATTEN_DN4}
      ],
      \"up_values\": [
        $p->{BAND_ATTEN_UP0},
        $p->{BAND_ATTEN_UP1},
        $p->{BAND_ATTEN_UP2},
        $p->{BAND_ATTEN_UP3},
        $p->{BAND_ATTEN_UP4}
      ]
    },
    \"noise_margin\": {
      \"down_values\": [
        if($p->{BAND_SNR_MARGIN_DN1},
        $p->{BAND_SNR_MARGIN_DN2},
        $p->{BAND_SNR_MARGIN_DN3},
        $p->{BAND_SNR_MARGIN_DN4}
      ],
      \"up_values\": [
        $p->{BAND_SNR_MARGIN_UP0},
        $p->{BAND_SNR_MARGIN_UP1},
        $p->{BAND_SNR_MARGIN_UP2},
        $p->{BAND_SNR_MARGIN_UP3},
        $p->{BAND_SNR_MARGIN_UP4}
      ]
    },
    \"signal_attenuation\": {
      \"down_values\": [
       $p->{BAND_SIG_ATTEN_DN1},
       $p->{BAND_SIG_ATTEN_DN2},
       $p->{BAND_SIG_ATTEN_DN3},
       $p->{BAND_SIG_ATTEN_DN4}
      ],
      \"up_values\": [
       $p->{BAND_SIG_ATTEN_UP0},
       $p->{BAND_SIG_ATTEN_UP1},
       $p->{BAND_SIG_ATTEN_UP2},
       $p->{BAND_SIG_ATTEN_UP3},
       $p->{BAND_SIG_ATTEN_UP4}
      ]
    },
    \"transmit_power\": {
      \"down_value\": $p->{POWER_DN},
      \"up_value\": $p->{POWER_UP}
    }
  }";

For example (I know for sure the problem is present only for line_attenuation, signal_attenuation and noise_margin both up and down)

   \"line_attenuation\": {
      \"down_values\": [
        $p->{BAND_ATTEN_DN1},
        $p->{BAND_ATTEN_DN2},
        $p->{BAND_ATTEN_DN3},
        $p->{BAND_ATTEN_DN4}

if $p->{BAND_ATTEN_DN4} is null I have to pass only the first 3 values and not 3 valid value+1 null .. how can I delete those null values? checking all those fields?

Currently I'm using those library

use strict;
use Data::Dumper;
use Getopt::Long;
use REST::Client;
use DBI;
use JSON;

If it is possible I would like not to import other libraries or so.

I have tryed checking if defined, or putting if in the json code but nothing ...

Thanks for the help

Why are you building your own JSON string? You've got the `JSON` module loaded. What are you using it for? Build up a new Perl data structure step by step, then convert it with the JSON module. — simbabque, Jun 04 '18 at 10:12
i can only modify this script, cannot rework it all .. i know it was better to use the conversion function, but i do not create the script originally — Carlo Viola, Jun 04 '18 at 10:18
Where's the difference between changing a few lines to make it even more complex and changing more lines to make it simpler? — simbabque, Jun 04 '18 at 10:19
I was asked to modify this without changing it radically ... i am not so able in using perl and do not have so much time so cannot read how to do all the changes to make it simpler ... — Carlo Viola, Jun 04 '18 at 10:22
The code is already of low quality. Colleagues who insist on making it worse are not great. I would write some tests for it, and then refactor completely. You'll probably end up with less code that is more readable. — simbabque, Jun 04 '18 at 10:36

simbabque · Accepted Answer · 2018-06-04T12:21:55.100

The smart way to do this would be to use the JSON module to convert your data structure to JSON. I would either build a new Perl data structure up step by step

my $to_json;

$to_json->{foo} = $p->{foo};

if ($p->{bar}) {
    $to_json->{baz} = $p->{baz};
}

or use inline operations with map and grep.

use strict;
use warnings;
use JSON;

my $p = {
    BAND_ATTEN_DN1 => 1,
    BAND_ATTEN_DN2 => 2,
    BAND_ATTEN_DN3 => 3,
    BAND_ATTEN_DN4 => undef,
};

my $to_json = {
    line_attenuaion => {
        down_values => [
            grep defined, map { $p->{$_} } 
                qw/BAND_ATTEN_DN1 BAND_ATTEN_DN2 BAND_ATTEN_DN3 BAND_ATTEN_DN4/,
        ],
        hashref_slice => [
            grep defined,
                @{$p}{qw/BAND_ATTEN_DN1 BAND_ATTEN_DN2 BAND_ATTEN_DN3 BAND_ATTEN_DN4/},
        ],
    },
    # ...
};

print to_json $to_json;

However, you say you cannot make large modifications. You'll have to split up the code that outputs the JSON instead. This is going to produce even uglier code and I strongly recommend not to do it this way, for the sake of maintenance.

# HERE BE DRAGONS...

my $json_string = "{
    \"attainable_rate\": {
        \"down_value\": $p->{ATTAINABLE_DN},
        \"up_value\": $p->{ATTAINABLE_UP}
    },
    \"line_attenuation\": {
        \"down_values\": [
            $p->{BAND_ATTEN_DN1},
            $p->{BAND_ATTEN_DN2},
            $p->{BAND_ATTEN_DN3}";
if ($p->{BAND_ATTEN_DN4}) {
    $json_string .= ",
            $p->{BAND_ATTEN_DN4}";
}
$json_string .= "
        ]
    }
}
";

If you do this, then at least get rid of all the escaping by converting your double quotes "" for the string to either the qq operator or by using HEREDOC.

my $json_string = qq({
    "attainable_rate": {
        "down_value": $p->{ATTAINABLE_DN},
        "up_value": $p->{ATTAINABLE_UP}
    }
}
);

my $heredoc = <<JSON
{
    "attainable_rate": {
        "down_value": $p->{ATTAINABLE_DN},
        "up_value": $p->{ATTAINABLE_UP}
    }
}
JSON

Remove null fields in JSON String in Perl

1 Answers1