Python XML handling. Subset Etree / remove elements / split etree in two parts based on tags-CodePudding

I have an issue where I loop through a xml response and populate lists. But once in a while the response has unexpected data which needs to be treated differently. I am struggling to split the relevant data into 2 categories. the main tag here "familierelasjon" in theory can go to infinity.

Id like to have two etrees as an output: one containing all data where the parent "familierelasjon" has the child "relatertPersonUtenFolkeregisteridentifikator" and one etree where it does not. See the xml example at the end.

Im using the data to populate lists that are appended and since the data format and content for the two cases are different the irregular responses kills my logic. The lists would generate a Length of values (x) does not match length of index (y) error. If i was able to split the data before looping through I could make two different logics for handling said data based on which tree im processing. Is there a simple way to do this? The BASE64 string contains the XML example data attached under.

Simplified code just for some context and the xml example data:

import base64  # Encoding/decoding Base64
import pandas as pd  # Create and manipulate tables/dataframes
#import numpy as np  # Formatting and array operations
from lxml import etree  # Xml parsing

#from datetime import datetime as datetime
#import datetime as dt
#from dateutil.relativedelta import relativedelta

enc_family_str_main = str('ICAgIDxmYW1pbGllcmVsYXNqb24 CiAgICAJPHJlbGF0ZXJ0UGVyc29uVXRlbkZvbGtlcmVnaXN0ZXJpZGVudGlmaWthdG9yPgogICAgCSAgPG5hdm4 CiAgICAJCTxmb3JuYXZuPkNsb3duLUNsb3duPC9mb3JuYXZuPgogICAgCQk8ZXR0ZXJuYXZuPkNsb3duaWU8L2V0dGVybmF2bj4KICAgIAkgIDwvbmF2bj4KICAgIAkgIDxmb2Vkc2Vsc2RhdG8 MjAyMi0wMS0wMTwvZm9lZHNlbHNkYXRvPgogICAgCSAgPHN0YXRzYm9yZ2Vyc2thcD5GVUtPRjwvc3RhdHNib3JnZXJza2FwPgogICAgCTwvcmVsYXRlcnRQZXJzb25VdGVuRm9sa2VyZWdpc3RlcmlkZW50aWZpa2F0b3I CiAgICAJPHJlbGF0ZXJ0UGVyc29uc1JvbGxlPmVrdGVmZWxsZUVsbGVyUGFydG5lcjwvcmVsYXRlcnRQZXJzb25zUm9sbGU CiAgICAgIDwvZmFtaWxpZXJlbGFzam9uPgogICAgICA8ZmFtaWxpZXJlbGFzam9uPgogICAgCTxyZWxhdGVydFBlcnNvbj4xMjM0NTY3ODkxMDwvcmVsYXRlcnRQZXJzb24 CiAgICAJPHJlbGF0ZXJ0UGVyc29uc1JvbGxlPmJhcm48L3JlbGF0ZXJ0UGVyc29uc1JvbGxlPgogICAgCTxtaW5Sb2xsZUZvclBlcnNvbj5mYXI8L21pblJvbGxlRm9yUGVyc29uPgogICAgICA8L2ZhbWlsaWVyZWxhc2pvbj4=')

"""
######## Parse and create loopable xml trees ########
"""
dec_family_str_main = base64.b64decode(enc_family_str_main   '='*(-len(enc_family_str_main) % 4))

parser = etree.XMLParser(recover=True)
# DSF children info main

tree_main = etree.fromstring(dec_family_str_main, parser=parser)

full_ssn = [] 
applicant = []
relation = []


#populate lists by looping through xml tree data
for element in tree_main.iter():
    if element.tag=='relatertPerson':
        full_ssn.append(element.text)
        applicant.append('Main')
    if element.tag=='relatertPersonsRolle':
        relation.append(element.text)
#if co_applicant==1:
    #for element in tree_co.iter():
        #if element.tag=='relatertPerson':
            #full_ssn.append(element.text)
            #applicant.append('Co')
        #if element.tag=='relatertPersonsRolle':
            #relation.append(element.text)

#create dataframe and fill with list data
df = pd.DataFrame()
df['applicant'] = applicant
df['SSN'] = full_ssn  
df['relation'] = relation

XML example simplified:

<familierelasjon>
    <relatertPersonUtenFolkeregisteridentifikator>
      <navn>
        <fornavn>Clown-Clown</fornavn>
        <etternavn>Clownie</etternavn>
      </navn>
      <foedselsdato>2022-01-01</foedselsdato>
      <statsborgerskap>FUKOF</statsborgerskap>
    </relatertPersonUtenFolkeregisteridentifikator>
    <relatertPersonsRolle>ektefelleEllerPartner</relatertPersonsRolle>
  </familierelasjon>
  <familierelasjon>
    <relatertPerson>12345678910</relatertPerson>
    <relatertPersonsRolle>barn</relatertPersonsRolle>
    <minRolleForPerson>far</minRolleForPerson>
  </familierelasjon>

I'd like the following output so that i have two subset of trees that i can run my iteration logic on.

Tree1: 
<familierelasjon>
    <relatertPersonUtenFolkeregisteridentifikator>
      <navn>
        <fornavn>Clown-Clown</fornavn>
        <etternavn>Clownie</etternavn>
      </navn>
      <foedselsdato>2022-01-01</foedselsdato>
      <statsborgerskap>FUKOF</statsborgerskap>
    </relatertPersonUtenFolkeregisteridentifikator>
    <relatertPersonsRolle>ektefelleEllerPartner</relatertPersonsRolle>
  </familierelasjon>
  all other <familierelasjon> that contains <relatertPersonUtenFolkeregisteridentifikator>

tree2: 
<familierelasjon>
    <relatertPerson>12345678910</relatertPerson>
    <relatertPersonsRolle>barn</relatertPersonsRolle>
    <minRolleForPerson>far</minRolleForPerson>
  </familierelasjon> 
  all other <familerelasjon> that does NOT contain <relatertPersonUtenFolkeregisteridentifikator>

PS I am stupid and not a programmer. I just type stuff and see what happens.

CodePudding user response：

If I understand you correctly, this will get you what you want. Note that, instead of ElementTree, I'm using lxml because of its better xpath support.

Assuming your sample xml is this:

fams = """<root>
   <familierelasjon>
    <relatertPersonUtenFolkeregisteridentifikator>
      <navn>
        <fornavn>Clown1-Clown1</fornavn>
        <etternavn>Clownie1</etternavn>
      </navn>
      <foedselsdato>2022-01-01</foedselsdato>
      <statsborgerskap>FUKOF1</statsborgerskap>
    </relatertPersonUtenFolkeregisteridentifikator>
    <relatertPersonsRolle>ektefelleEllerPartner1</relatertPersonsRolle>
  </familierelasjon>
  <familierelasjon>
    <relatertPerson>12345678910</relatertPerson>
    <relatertPersonsRolle>barn1</relatertPersonsRolle>
    <minRolleForPerson>far1</minRolleForPerson>
  </familierelasjon>

  <familierelasjon>
    <relatertPersonUtenFolkeregisteridentifikator>
      <navn>
        <fornavn>Clown2-Clown2</fornavn>
        <etternavn>Clownie2</etternavn>
      </navn>
      <foedselsdato>3022-01-01</foedselsdato>
      <statsborgerskap>FUKOF2</statsborgerskap>
    </relatertPersonUtenFolkeregisteridentifikator>
    <relatertPersonsRolle>ektefelleEllerPartner2</relatertPersonsRolle>
  </familierelasjon>
  <familierelasjon>
    <relatertPerson>1111111</relatertPerson>
    <relatertPersonsRolle>barn2</relatertPersonsRolle>
    <minRolleForPerson>far2</minRolleForPerson>
  </familierelasjon>
</root>"""

You can use the following:

from lxml import etree

tree1= etree.fromstring('<root></root>')
tree2= etree.fromstring('<root></root>')
#you need a root element for a well formed xml file

source = etree.fromstring(fams)

#first run will get you elements "where the parent "familierelasjon" has the child "relatertPersonUtenFolkeregisteridentifikator""
for fr in source.xpath('//familierelasjon[relatertPersonUtenFolkeregisteridentifikator]'):
    tree1.append(fr)

#second run will get you the opposite
for nfr in source.xpath('//familierelasjon[not(.//relatertPersonUtenFolkeregisteridentifikator)]'):
    tree2.append(nfr)

print(etree.tostring(tree1).decode())
print('----------------------')
print(etree.tostring(tree2).decode())

Output should be your expected output.