MySQL DB to pandas dataframe-CodePudding

I am handling sql on python, and I'm more familiar with pandas dataframe than SQL statements so I want to import whole data on MySQL DB and handle it on python. And, I'm now a little afraid of my memory error.

Is pandas needs memory for containing all data? or is it enough even if memory is small enough to just contain the results?

import pandas as pd
import sqlalchemy
engine = sqlalchemy.create_engine("mysql://"   "root"   ":"   "password"   "@"   "localhost"   "/"   "mydb")

df = pd.read_sql_table('table1', engine)

# some operations

print(df)

CodePudding user response：

You need more memory than the results because you will do some operations on them, so these operations need memory too. In addition, the memory will not only contain the results of your query, but also a lot of data that already exists. If memory is an issue, you can pull data in small numbers using the LIMIT and OFFSET clauses.

CodePudding user response：

Pandas needs more memory than the size of your data.

Please refer to this thread if you want to query just a part of your database to avoid running out of memory.