1

Here is a list created from YAML file. I want to compare the following contents of each memservers with other merserves in the list:

  1. If the rpc_interface is same.
  2. If rpc_interface is same, is fam_path same?
  3. If rpc_interface is same, is libabric_port same?

The code should be generic i.e., it should work for any number of servers.

provider: sockets
delayed_free_threads: 0
ATL_threads: 0
ATL_queue_size: 1000
ATL_data_size: 1024
Memservers:
  0:
    memory_type: volatile
    fam_path: /dev/shm/vol_path
    rpc_interface: fam5:8793
    libfabric_port: 7500
    if_device: eth0
  1:
    memory_type: volatile
    fam_path: /dev/shm/vol_path
    rpc_interface: fam4:8793
    libfabric_port: 7500
    if_device: eth1
  2:
    memory_type: volatile
    fam_path: /dev/shm/vol_path
    rpc_interface: fam3:8793
    libfabric_port: 7500
    if_device: eth1
Vsdm in
  • 25
  • 4
  • "The length of the list" should read "the length of the dict", ore ven better the length of the mapping (as that is what you have in YAML. How do you determin Ip:port combination. I see a key that might indicate port number, but it doesn't seem to from your 3rd permise, and nothing that looks like an IP address. What are the memorey servers? The values? What dtermines that they are hosted from the same machine? – Anthon Jun 11 '22 at 19:53
  • IP address includes computer number/ip_address + the host port number. Memory_servers are the computer names. It's nothing to worry about. We need to check if the : 1. rpc_interface is the same between any of the two servers given(like 0, 1,...7) 2. if rpc_interface is the same, we have to check if fam_path is the same between any of the two servers given(like 0, 1,...7) 3. if rpc_interface is the same, we have to check if libabric_port is the same between any of the two servers given(like 0, 1,...7) – Vsdm in Jun 12 '22 at 02:29

1 Answers1

0

For each of your requirements, you should just map the value (that should not duplicate) to the machine numbers that you found them in. In the following just done for the first requirement:

from pathlib import Path
import ruamel.yaml

host_port = {}
file_in = Path('fam_memoryserver_config.yaml')

yaml = ruamel.yaml.YAML(typ='safe')  # faster than using yaml.safe_load()
data = yaml.load(file_in)
for machine_nr, config in data.items():
    host_port.setdefault(config['rpc_interface'], set()).add(machine_nr)

# now check if host_port has any values that have more than one machine_nr
for hp, machine_nrs in host_port.items():
    if len(machine_nrs) == 1:
        continue
    print(f'found {hp} in machines: {", ".join([str(x) for x in machine_nrs])}')

which gives:

found fam3:8793 in machines: 1, 2

You should get your terminology clear, or at least define these in your question. Common terminology will help you find answers yourself more easily (here on SO, or googling).

You don't have any lists, you have a mapping at the root level of your YAML document, with mapping for the values of each key. Those load to dicts nested within a dict. there are no lists anywhere.

The above also assumes that the value for key rpc_interfaces is what you refer to as ip:port, however that part before the : in fam5:8793 doesn't look like an IPv4 or an IPv6 address. It looks more like a hostname.

You also refer to checks between all files, but there are only two files: the YAML input and your source code. And comparing between those doesn't make much sense.

Anthon
  • 69,918
  • 32
  • 186
  • 246
  • Thanks for the help, I notice an error as: host_port.setdefault(config['rpc_interface'], set()).add(machine_nr) KeyError: 'rpc_interface. Please let me know how to fix it. Regarding check all the files, what i meant to say is that I want to check among 7 servers that exists. – Vsdm in Jun 12 '22 at 09:42
  • you get that error if some of the machines don't have the key `rpc_interface`. So you need to check on that in the first for loop. Probably print an error message and potentially error out if that need to be fixed ( `if 'rpc_interface' not in config: print(f'machine {machine_nr} has no "rpc_interface"'); sys.exit(1)` or similar. – Anthon Jun 12 '22 at 10:12
  • Thankyou. Now that the rpc_interface is same, I need to check if 'fam_path' and 'libabric_port' are also same from the same machine. how should I do that? – Vsdm in Jun 12 '22 at 11:32
  • Make two more dicts like host_port and make two additional for loops for each of those dicts (adapting the key that you test for). It might help to print the dict after the first for loop so you can see if it filled up as expected. Any entry for which the value is more than one machine_nr doesn't conform. – Anthon Jun 12 '22 at 12:08
  • What if i make a nested yaml code? For example: Memservers: 0: memory_type: volatile fam_path: /dev/shm/vol_path rpc_interface: fam5:8793 libfabric_port: 7500 if_device: eth0 If I run the same code, I'm getting an error as "no rpc_interface found" though it exists. How do i modify my code ? – Vsdm in Jun 23 '22 at 17:56
  • Could you check how to modify if it's a nested code? – Vsdm in Jun 23 '22 at 17:59
  • If you make nested yaml code you either directly access the parent using subscription on `data` and its substructure, or you recursively walk over `data` (assuming you have some way to determine you are at the right level. – Anthon Jun 23 '22 at 19:18