How can I work on a large dataset without having to use Pyspark?-CodePudding

I'm trying to work on a dataset with 510,000 rows and 636 columns. I loaded it into a dataframe using the dask dataframe method, but the entries can't be displayed. When i try to get the shape, it results in delays. Is there a way for me to analyze the whole dataset without using big data technologies like Pyspark?

from dask import dataframe
import requests
import zipfile
import os
import pandas as pd

if os.path.exists('pisa2012.zip') == False:
    r = requests.get('https://s3.amazonaws.com/udacity-hosted-downloads/ud507/pisa2012.csv.zip', allow_redirects=True)
    open('pisa2012.zip', 'wb').write(r.content)

if os.path.exists('pisa2012.csv') == False:
    with zipfile.ZipFile('pisa2012.zip', 'r') as zip_ref:
        zip_ref.extractall('./')

df_pisa = dataframe.read_csv('pisa2012.csv')

df_pisa.shape #Output:(Delayed('int-e9d8366d-1b9e-4f8e-a83a-1d4cac510621'), 636)

CodePudding user response：

Firstly, spark, dask and vaex are all "big data" technologies.

it results in delays

If you read the documentation, you will see that dask is lazy and only performs operations on demand, you have to want to. The reason is, that just getting the shape requires reading all the data, but the data will not be held in memory - that is the whole point and the feature that lets you work with bigger-than-memory data (otherwise, just use pandas).

This works:

df_pisa.shape.compute()

Bute, better, figure out what you actually want to do with the data; I assume you are not just after the shape. You can put multiple operations/delayed objects into a dask.compute() to do them at once and not have to repeat expensive tasks like reading/parsing the file.

CodePudding user response：

you can use vaex which is a very good alternative for big data https://vaex.io/ perfect for this kind of problem.