how to add multiples ips cidr with ipset save and restore options

Question

i have this loop to block IPs and CIDR with ipset/iptables:

# this is just an example. the actual list IPs/CIDR is very large
cat blockip.txt
13.31.0.254
cat blockcidr.txt
13.32.0.0/15

Loop:

#!/bin/bash
ipset -F
ipset -N -! blacklist hash:net maxelem 1000000
for ip in $(cat blockip.txt blockcidr.txt); do
    ipset -A blacklist $ip
done
iptables -A FORWARD -m set --match-set blacklist dst -j DROP

Note: I have always used the ipset -A option, but I don't know exactly what this option means, since it does not appear in "Man Ipset", and at this point, I'm assuming that add is the same as -A, since the output in both cases is the same.

#!/bin/bash
ipset -F
ipset -N -! blacklist hash:net maxelem 1000000
for ip in $(cat blockip.txt blockcidr.txt); do
    ipset add blacklist $ip -q
done
iptables -A FORWARD -m set --match-set blacklist dst -j DROP

out both cases:

sudo ipset -L
Name: blacklist
Type: hash:net
Revision: 7
Header: family inet hashsize 1024 maxelem 1000000 bucketsize 12 initval 0xbc0136c8
Size in memory: 552
References: 0
Number of entries: 2
Members:
13.31.0.254
13.32.0.0/15

"It works fine", but I have read that adding the IPs and CIDR with ipset -A is very slow. Which is faster using ipset save and restore. But I don't understand how works, And my attempt is failed:

Note: I have not found an explanation of why it is faster to use the save/restore options, instead of add or -A

#!/bin/bash
ipset -F
ipset -N -! blacklist hash:net maxelem 1000000
for ip in $(cat blockip.txt blockcidr.txt); do
    ipset add blacklist $ip # ??
    ipset save blacklist -f newblacklist.txt # ???
done
ipset restore -! < newblacklist.txt # ??
iptables -A FORWARD -m set --match-set blacklist dst -j DROP # ??

out:

sudo ipset -L
Name: blacklist
Type: hash:net
Revision: 7
Header: family inet hashsize 1024 maxelem 1000000 bucketsize 12 initval 0xcb0e583b
Size in memory: 552
References: 0
Number of entries: 2
Members:
13.32.0.0/15
13.31.0.254

cat newblacklist.txt # out wrong
create blacklist hash:net family inet hashsize 1024 maxelem 1000000 bucketsize 12 initval 0xcb0e583b
add blacklist 13.32.0.0/15
add blacklist 13.31.0.254

I would appreciate any help (with a complete answer, including the proposed loop or corrections to my loop)

Answer 1

This worked for me:

# remove any old reference to the ipset
iptables -D FORWARD -m set --match-set blacklist dst -j DROP
ipset destroy

ipset create blacklist hash:net family inet hashsize 1024
ipset save > /tmp/ipset.txt
ipset destroy

cat blockip.txt blockcidr.txt | while read line; do
    if [ "${line:0:1}" = "#" ]; then
        continue
    fi
    echo "add blacklist $line" >> /tmp/ipset.txt
done

ipset restore < /tmp/ipset.txt
iptables -A FORWARD -m set --match-set blacklist dst -j DROP

Please adapt the options in the create statement according to your needs - depending on the size of the set, these options are quite important. This further assumes that the file blockcidr.txt contains only lines like this:

#comment
1.2.3.4/20

afterwards, a restore works just fine.

update
the bottleneck of such a loop is always the creation of a subprocess, ipset in this case. If you have 10000+ entries, the executable ipset is being loaded into memory, options parsed for each line you would like to add...

My loop contains only bash-internal commands, so no executable must be loaded / executed - there is just being some text written into a file. And of course - a single call to ipset is much faster than 10000+ calls to ipset...