RuntimeError dictionary changed size during iteration during dictionary iteration-CodePudding

I am getting following error when I run my code. It started happening with apparently no changes to the code. The code basically tries to rendor the device config using jinja2 template.

Traceback (most recent call last):
  File "vdn_ler_generate_config.py", line 380, in <module>
    lerConfig = lerTemplate.render(config=copy.copy(config.lerDevList[tLer]))
  File "/Users/nileshkhambal/Documents/myansible/lib/python3.8/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/Users/nileshkhambal/Documents/myansible/lib/python3.8/site-packages/jinja2/environment.py", line 925, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "./templates/jinja2/JuniperLerConfigTemplate", line 71, in top-level template code
    {%- for tBgpGrp in config.vrfs[tvrf].bgpProto %}
RuntimeError: dictionary changed size during iteration

The code snippet below inside "try" and "expect" block shows the problem. There is an exactly same code for second vendor and it works fine. Using print statement in expect statement I see that there is an extra group object (with 'default') added in the dictionary during the iteration. According to my YAML config there should be only one such group object that should be there.

    try:
        if tVendor == 'VENDOR1':
            lerTemplate = ENV.get_template('Vendor1LerConfigTemplate')
            lerConfig = lerTemplate.render(config=config.lerDevList[tLer])

            pOutputFile = './configs/'   str(tLer)   '.cfg'

            with open(pOutputFile, "w ") as opHandle:
                #print('Writing configuration for {}'.format(tLer))
                opHandle.write(lerConfig)
                opHandle.write('\n')
    except:
        for aKey in config.lerDevList[tLer].vrfs.keys():
            #print(config.lerDevList[tLer].vrfs[aKey].bgpProto)
            print('What VRF: {} How many BGP Groups: {}'.format(aKey,len(config.lerDevList[tLer].vrfs[aKey].bgpProto.keys())))
            for agrp in config.lerDevList[tLer].vrfs[aKey].bgpProto.keys():
                print(config.lerDevList[tLer].vrfs[aKey].bgpProto[agrp])
        continue

    if tVendor == 'VENDOR2':
        for aKey in config.lerDevList[tLer].vrfs.keys():
            #print(config.lerDevList[tLer].vrfs[aKey].bgpProto)
            for agrp in config.lerDevList[tLer].vrfs[aKey].bgpProto.keys():
                print(config.lerDevList[tLer].vrfs[aKey].bgpProto[agrp])
        lerTemplate = ENV.get_template('Vendor2LerConfigTemplate')
        lerConfig = lerTemplate.render(config=config.lerDevList[tLer])

        pOutputFile = './configs/'   str(tLer)   '.cfg'

        with open(pOutputFile, "w ") as opHandle:
            #print('Writing configuration for {}'.format(tLer))
            opHandle.write(lerConfig)
            opHandle.write('\n')

using some print() statements I can see that, the code creating group using base class object, adds one group but iteration code seems to add or see an extra group with name 'default'. 'default' is the name used in the base class for the group. once the object is instantiated it is assigned a proper group name.

Creating vrf bgp group object for xxx4-bb-pe1 BLUE: AS65YYY-BLUE-V4-PEER
Double checking bgp groups: 1
Creating vrf bgp group object for yyy6-bb-pe1 RED: AS65YYY-RED-V4-PEER
Double checking bgp groups: 1
Creating vrf bgp group object for zzz2-bb-pe1 BLUE: AS4200XXXXXX-BLUE-V4-PEER
Double checking bgp groups: 1
Creating vrf bgp group object for zzz2-bb-pe2 RED: AS4200XXXXXX-RED-V4-PEER
Double checking bgp groups: 1
Creating vrf bgp group object for xyxy2-bb-gw1 BLUE: AS4200XXXXXX-BLUE-V4-PEER
Double checking bgp groups: 1
Creating vrf bgp group object for xyxy2-bb-gw2 RED: AS4200XXXXXX-RED-V4-PEER
Double checking bgp groups: 1
Writing configuration for xxx4-bb-pe1
AS65YYY-BLUE-V4-PEER
Writing configuration for yyy6-bb-pe1
AS65YYY-RED-V4-PEER
Writing configuration for zzz2-bb-pe1
AS4200XXXXXX-BLUE-V4-PEER
Writing configuration for zzz2-bb-pe2
AS4200XXXXXX-RED-V4-PEER
Writing configuration for xyxy2-bb-gw1
What VRF: BLUE How many BGP Groups: 2  <<< extra group
AS4200XXXXXX-BLUE-V4-PEER
default << extra group
Writing configuration for xyxy2-bb-gw2
What VRF: RED How many BGP Groups: 2 <<< extra group
AS4200XXXXXX-RED-V4-PEER
default  <<<< extra group

here is the default group class definition

class bgpGroup():
    def __init__(self):
        self.vrf = ''
        self.grpName = 'default'
        self.grpType = 'internal'
        self.grpDescr = ''
        self.grpLocalAddr = '0.0.0.0'
        self.clusterid = ''
        self.gr_disable = False
        self.remove_private = False
        self.as_override = False
        self.peer_as = ''
        self.local_as = ''
        self.local_as_private = False
        self.local_as_noprepend = False
        self.holdtime = ''
        self.grpFamily = []
        self.grpImport = ''
        self.grpExport = ''
        self.grpNbrList = defaultdict(bgpNbr)
        self.grpLoopDetect = False
        self.grpMltHopTtl = 0
        self.grpInetAsPathPrependReceive = False
        self.grpLabelInet6AsPathPrependReceive = False

    def __repr__(self):
        return self.grpName

CodePudding user response：

I really don't know what the bug is, but have you tried:

for aKey in list(config.lerDevList[tLer].vrfs.keys())

This grabs all the keys from the dictionary at the start, and if another entry is added, it won't matter.

Don't also that you don't need .keys(), though it doesn't hurt. Iterating through a dictionary is iterating through its keys.

CodePudding user response：

Issue was in a jinja2 template code. A wrong variable was referenced inside a statement in for loop. Fixing it fixed the error. Sorry for the noise. Error did point to jinja2 code with the start of the loop line that had error. Did not point to the actual line with an error. All good now.