Home > other >  Python how to realize the fast will read one million data from the XLSX file and inserted into the M
Python how to realize the fast will read one million data from the XLSX file and inserted into the M

Time:01-26

I now have a XLSX format excel file, there are 720000 data, now want to use python implementation from the excel file to read and write MSSQL database

A. XLSX file converts CSV format, and then use bulk insert, tried just about 20 seconds, but there are a few problems, a database login permissions are high enough, and is a CSV file to the SQL server (if it's not safe to go any distance need to share files)

Scheme 2. Pymssql adopts INSERT INTO Score VALUES (102,87), (183180), 104 (13), (102,87), (183180), 104 (13) followed by data in the form of a list, but behind this way only with 1000 data, is about to go wrong, more tried this way need 32 minutes (with an average of 10000 data needs to be 27 seconds), plus XLRD. Open_workbook open the excel file need 5 minutes, a total of 37 minutes.

Plan 3. Pymssql insert data using executemany way, this way can be 10000 data operations together, but found no 2 fast, processing the data of article 10000 of the need for 40 seconds.

Plan 4. Pymssql detailed data insert. This way is obviously more slowly and is not recommended.



So dalao excuse me, plan a ruled out at present, is there any faster than the second scheme, more viable option? Some netizens said with pymssql _mssql faster than pymssql module operation, but the actual test contrast is not obvious.
  • Related