I am currently using spark on python to extract data from a SQL database. However each field has 4 instances of data, e.g. cat_i1, cat_i2, cat_i3, cat_i4, var_i1, var_i3, var_i4 etc...

The code I am using looks as follows:

field_names = ["cat_i1, var_i1"] ,

df = retrieve_fields(names=field_names, engine=dxdata.connect())

for the instance 1 dataframe

I am looking for a way to get each instance in the dataframe without typing it manually. I am new to python so unsure how I would use loops (or any other method), to do this. Can anyone help? Thank you.

CodePudding user response：

You can just specify the general fields and then create the instances iterating through them:

fields = ["cat", "var"]
inst = [f   "_i{}".format(x) for x in range(1,5) for f in fields]