left outer join 2 shell output on a common key column using bash

Question

Is there a way I can do a "left outer join" for bash output from 2 different commands based on a common key column (ip address)? For various reasons, I would like to avoid saving the output from either commands into a file and using the filename as inputs into awk/join.

These are the 2 outputs I would like to merge together:

output1=$(oc get pod -lname=elasticsearch -o wide)
output2=$(curl -ks https://elasticsearch.apps.ocp.domain.com/_cat/allocation?v)

Here's the disk utilization of my elasticsearch pods

$ echo "$output1"
shards disk.indices disk.used disk.avail disk.total disk.percent host          ip            node
   347        1.5tb     1.5tb      4.2tb      5.8tb           27 10.1.1.1      10.1.1.1      jxAuPMt
   335        1.7tb     1.7tb        4tb      5.8tb           30 10.2.2.2      10.2.2.2      bwYW1cg
   341        1.3tb     1.9tb    959.7gb      2.9tb           67 10.3.3.3      10.3.3.3      69O4sR-
   343        1.6tb     1.7tb      1.1tb      2.9tb           59 10.4.4.4      10.4.4.4      zCVUS05
   346        1.4tb     1.9tb   1000.6gb      2.9tb           66 10.5.5.5      10.5.5.5      AlvtEtf
   345        1.6tb     1.6tb      1.2tb      2.9tb           57 10.6.6.6      10.6.6.6      pmObkSm
   248          1tb       1tb    374.9gb      1.4tb           74 10.7.7.7      10.7.7.7      OSzB5J1
   332        1.4tb     1.9tb    952.1gb      2.9tb           68 10.8.8.8      10.8.8.8      EShhlrb
   218     1011.8gb       1tb    418.2gb      1.4tb           71 10.9.9.9      10.9.9.9      1rQ7PXD
   329        1.7tb     1.8tb        1tb      2.9tb           63 10.10.10.10   10.10.10.10   73TfUnn
    15                                                                                       UNASSIGNED

Here are the running elasticsearch pods in my environment

$ echo "$output2"
NAME                              READY     STATUS    RESTARTS   AGE       IP              NODE
elasticsearch-4-40700             1/1       Running   0          4d        10.1.1.1        server1b.ocp.domain.com
elasticsearch-4-6m2p1             1/1       Running   0          4d        10.2.2.2        server2b.ocp.domain.com
elasticsearch-4-8n8v5             1/1       Running   0          7d        10.3.3.3        server3b.ocp.domain.com
elasticsearch-4-ffnk0             1/1       Running   0          4d        10.4.4.4        server4b.ocp.domain.com
elasticsearch-4-h5xr6             1/1       Running   0          7d        10.5.5.5        server5b.ocp.domain.com
elasticsearch-4-h6c15             1/1       Running   0          7d        10.6.6.6        server6b.ocp.domain.com
elasticsearch-4-j7cv5             1/1       Running   0          7d        10.7.7.7        server7b.ocp.domain.com
elasticsearch-4-nn3pc             1/1       Running   0          5d        10.8.8.8        server8b.ocp.domain.com
elasticsearch-4-sq5hr             1/1       Running   0          7d        10.9.9.9        server9b.ocp.domain.com
elasticsearch-4-zj7tq             1/1       Running   0          7d        10.10.10.10     server10b.ocp.domain.com
elasticsearch-4-az9sp             1/1       Running   0          7d        10.10.10.11     server11b.ocp.domain.com
elasticsearch-4-bi2c7             1/1       Running   0          7d        10.10.10.12     server12b.ocp.domain.com

Expected results after doing a "left outer join" from output1 and output2 based on the IP address

NAME                              READY     STATUS    RESTARTS   AGE       IP              NODE                      shards disk.indices disk.used disk.avail disk.total disk.percent node
elasticsearch-4-40700             1/1       Running   0          4d        10.1.1.1        server1b.ocp.domain.com      347        1.5tb     1.5tb      4.2tb      5.8tb           27 jxAuPMt 
elasticsearch-4-6m2p1             1/1       Running   0          4d        10.2.2.2        server2b.ocp.domain.com      335        1.7tb     1.7tb        4tb      5.8tb           30 bwYW1cg 
elasticsearch-4-8n8v5             1/1       Running   0          7d        10.3.3.3        server3b.ocp.domain.com      341        1.3tb     1.9tb    959.7gb      2.9tb           67 69O4sR- 
elasticsearch-4-ffnk0             1/1       Running   0          4d        10.4.4.4        server4b.ocp.domain.com      343        1.6tb     1.7tb      1.1tb      2.9tb           59 zCVUS05 
elasticsearch-4-h5xr6             1/1       Running   0          7d        10.5.5.5        server5b.ocp.domain.com      346        1.4tb     1.9tb   1000.6gb      2.9tb           66 AlvtEtf 
elasticsearch-4-h6c15             1/1       Running   0          7d        10.6.6.6        server6b.ocp.domain.com      345        1.6tb     1.6tb      1.2tb      2.9tb           57 pmObkSm 
elasticsearch-4-j7cv5             1/1       Running   0          7d        10.7.7.7        server7b.ocp.domain.com      248          1tb       1tb    374.9gb      1.4tb           74 OSzB5J1 
elasticsearch-4-nn3pc             1/1       Running   0          5d        10.8.8.8        server8b.ocp.domain.com      332        1.4tb     1.9tb    952.1gb      2.9tb           68 EShhlrb 
elasticsearch-4-sq5hr             1/1       Running   0          7d        10.9.9.9        server9b.ocp.domain.com      218     1011.8gb       1tb    418.2gb      1.4tb           71 1rQ7PXD 
elasticsearch-4-zj7tq             1/1       Running   0          7d        10.10.10.10     server10b.ocp.domain.com     329        1.7tb     1.8tb        1tb      2.9tb           63 73TfUnn 
elasticsearch-4-az9sp             1/1       Running   0          7d        10.10.10.11     server11b.ocp.domain.com      
elasticsearch-4-bi2c7             1/1       Running   0          7d        10.10.10.12     server12b.ocp.domain.com

Lorem ipsum · Accepted Answer · 2020-09-08T10:12:20.200

The below produces your desired output, but without headers:

awk 'NR==FNR{a[$8]=$0}FILENAME==ARGV[1]{b[$6]=$2} FILENAME==ARGV[2]{print $0,a[$6]}' <(oc get pod -lname=elasticsearch -o wide) <(curl -ks https://elasticsearch.apps.ocp.domain.com/_cat/allocation?v) | awk 'NR>1' | awk '!($14=$15="")'

The part, wich is a little bit unclean is how this removes the duplicate ip address columns with 'awk '!($14=$15="")''

You might be better off with a little multiline script instead of a one liner to make this process cleaner.

left outer join 2 shell output on a common key column using bash

1 Answers1