How to form a dictionary of dictionaries with keys/values built from 3 different lists in Ansible-CodePudding

I'm pulling a data file off several remote hosts that has dynamically named columns and then rows of data. I'd like to compute a consolidated data structure. I cannot get json or csv from the source, just a text data file.

NODE1

field1  field2  field3
valueA1 valueA2 valueA3

NODE2

field1  field2  field3
valueB1 valueB2 valueB3

So far, I built 3 lists: The 1st is a list of node names. The 2nd & 3rd are formed from the data file on each host. The column names are the same in each data file, but I currently just build List2 with redundant names throughout as I read the remote data file.

List1

- node1
- node2
- node3

List2 of list

-
    - field1
    - field2
    - field3
-
    - field1
    - field2
    - field3
-
    - field1
    - field2
    - field3

List3 of list

-
    - valueA1
    - valueA2
    - valueA3
-
    - valueB1
    - valueB2
    - valueB3
-
    - valueC1
    - valueC2
    - valueC3

I'd like to end up with a dictionary of dictionaries like this:

{ node1: 
    {field1:valueA1, field2:valueA2, field3:valueA3},
  node2: 
    {field1:valueB1, field2:valueB2, field3:valueB3},
  node3:
    {field1:valueC1, field2:valueC2, field3:valueC3}
}

Suppose I could use a different data structure, but ultimately, I need the data nested like this into a variable for use later in the playbook and ultimately saved to a json file.

CodePudding user response：

this playbook does the job (using jinja2):

- hosts: localhost
  gather_facts: false
  vars:
    list1:
      - node1
      - node2
      - node3
    list2:
      - - field1
        - field2
        - field3
      - - field1
        - field2
        - field3
      - - field1
        - field2
        - field3
    list3:
    - - valueA1
      - valueA2
      - valueA3
    - - valueB1
      - valueB2
      - valueB3
    - - valueC1
      - valueC2
      - valueC3
  tasks:
    - name: loop over lists
      set_fact:
        list4: >-
            {%- set ns = namespace() -%}
            {%- set ns.dico = {} -%}
            {%- for l1 in list1 -%}
            {%- set ll = loop -%}
            {%- set ns.l1 = {} -%}
            {%- for l2 in list2 -%}
            {%- set ns.d2 = {} -%}            
            {%- for sl2 in l2 -%}
            {%- if ns.d2.update({sl2: list3[ll.index0][loop.index0]}) -%}{%- endif -%}                       
            {%- endfor -%}
            {%- if ns.dico.update({l1: ns.d2}) -%}{%- endif -%} 
            {%- endfor -%}
            {%- endfor -%}
            {{ ns.dico }}
    - debug:
        msg: "{{ list4 }}"

result:

ok: [localhost] => {
    "msg": {
        "node1": {
            "field1": "valueA1",
            "field2": "valueA2",
            "field3": "valueA3"
        },
        "node2": {
            "field1": "valueB1",
            "field2": "valueB2",
            "field3": "valueB3"
        },
        "node3": {
            "field1": "valueC1",
            "field2": "valueC2",
            "field3": "valueC3"
        }
    }
}

you could use too a custom filter (full python script):

#!/usr/bin/python
class FilterModule(object):
    def filters(self):
        return {
            'listzip': self.listzip
        }

    def listzip(self, l1, l2, l3):
        result = {
            node: {
                    field: value for field, value in zip(l2[idx], l3[idx])
                } for idx, node in enumerate(l1)
        }
        return result

and you use it like this:

- name: loop
  set_fact:
    list4: "{{ list1 | listzip(list2, list3) }}"

- debug:
    msg: "{{ list4 }}"

CodePudding user response：

First, note that your file format is actually csv compatible and can be read with the read_csv plugin

Given the following inventory for local test purpose:

---
all:
  vars:
    ansible_connection: local
  hosts:
    node1:
    node2:
    node3:

and faking target files as /tmp/test_files/nodeX_file.txt on my local machine (same content as the one you pasted above).

The following playbook:

---
- hosts: all
  gather_facts: false

  vars:
    test_file_path: /tmp/test_files/

  tasks:
    - name: load content from test file on each host
      read_csv:
        delimiter: " "
        skipinitialspace: true
        path: "{{ test_file_path }}/{{ inventory_hostname }}_file.txt"
      register: field_values

    - name: Show the result
      vars:
        relevant_raw_data: "{{ hostvars | dict2items | selectattr('value.field_values', 'defined') }}"
        relevant_keys: "{{ relevant_raw_data | map(attribute='key') }}"
        relevant_values: "{{ relevant_raw_data | map(attribute='value.field_values.list') | map('first') }}"
        dict_result: "{{ relevant_keys | zip(relevant_values) | items2dict(key_name=0, value_name=1) }}"
      debug:
        var: dict_result
      delegate_to: localhost
      run_once: true

gives:

PLAY [all] ********************************************************************************************************************************************************************************************************************

TASK [load content from test file on each host] *******************************************************************************************************************************************************************************
ok: [node2]
ok: [node3]
ok: [node1]

TASK [Show the result] ********************************************************************************************************************************************************************************************************
ok: [node1 -> localhost] => {
    "dict_result": {
        "node1": {
            "field1": "valueA1",
            "field2": "valueA2",
            "field3": "valueA3"
        },
        "node2": {
            "field1": "valueB1",
            "field2": "valueB2",
            "field3": "valueB3"
        },
        "node3": {
            "field1": "valueC1",
            "field2": "valueC2",
            "field3": "valueC3"
        }
    }
}

PLAY RECAP ********************************************************************************************************************************************************************************************************************
node1                      : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
node2                      : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
node3                      : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0