How to avoid for loops and use an alternative method in this code ( python and postgres sql)?. Below-CodePudding

##Here is my code.I want to get rid of the for loops here. Can you please suggest an alterantive methods in python which can be used for the same logic##

usa_records = cursor.fetchall()
    for k in csv_files:#reading csv files
      with open(k, 'r') as f:
          reader = csv.reader(f)
          next(reader)  # Skip the header row.
          for row in reader:
              status = 0
              for m in usa_records:
                  if (m[6] == row[6]):
                      status = status   1

                      row[0] = m[0] if m[0] is not None else row[0]
                      row[1] = m[1] if m[1] is not None else row[1]
                      row[2] = m[2] if m[2] is not None else row[2]
                      row[3] = m[3] if m[3] is not None else

                      row[17] = m[17] if m[17] is not None else row[17]

                      sql_update_query = """Update usa set technology=%s,   company_name=%s,contact_name=%s,first_name=%s,last_name=%s,title=%s,email=%s,person_linkedin_url=%s,web_address=%s,company_linkedin_url=%s,company_address=%s,city=%s,state=%s,company_phone=%s,employees=%s,industry=%s,country=%s where id = %s"""
                      cursor.execute(sql_update_query,
                                     (row[0], row[1], row[2], row[3], row[4], row[5], row[6], row[7], row[8],
                                      row[9], row[10], row[11], row[12], row[13], row[14], row[15], row[16],
                                      row[17]))
                      conn.commit()

                      row_count = cursor.rowcount
                      print(row_count, "Records Updated")
                      break
                  else:
                      pass
              if status == 0:
                  cursor.execute(
                      "INSERT INTO usa VALUES (%s, %s, %s, %s,%s, %s, %s, %s,%s, %s, %s, %s,%s, %s, %s, %s,%s,%s)",
                      row)

                  conn.commit()
                  print("created new data")

CodePudding user response：

A good approach would be to use clean code principles and divide your logic into smaller functions. This will reduce nesting and make your code more readable.

Here's some pseudocode:

usa_records = cursor.fetchall()
process_csv_files(csv_files, usa_records)


def process_csv_files(csv_files, usa_records):
    for k in csv_files:  #reading csv files
        with open(k, 'r') as f:
            reader = csv.reader(f)
            next(reader)  # Skip the header row.

            process_csv_content(reader, usa_records)

def process_csv_content(reader):
    for row in reader:
        usa_record = find_usa_record_by_id(row[6], usa_records)

        if not usa_record:
            # INSERT in the database
        else:
            # UPDATE the database

def find_usa_record_by_id(id, usa_records):
    # Implement a function that returns the record based on the given id.
    pass

CodePudding user response：

What's the reason you want to remove the for loops? I mean, what's the goal? By knowing the reasons, we can propose different approaches.

Is it about "code readability"? Is it because of performance? Is it just an exercise?

CodePudding user response：

Check out Pandas. It looks like most of what you're doing could be completed with somewhat simple dataframe operations. For example, reading a csv: https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html

And rather than looping over rows, you can use .apply: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html