Home > database >  best way to import very large file
best way to import very large file

Time:06-30

enter image description here

The data can be found at https://www.bls.gov/cew/downloadable-data-files.htm. It is the single file found in the middle of the page. I opened up the zip file for the 2021 file and renamed the csv to just 2021 instead of 2021.q1-q4.singlefile. I tried opening it in notepad but it says that it is too large. Also, can't open it in excel. I try the following code and get the error below. Is this error telling me that it is too large or can't read it? Also use the command, read.csv to no avail. Any suggestions? I thought of opening it in Notepad and doing a save as 2021.txt

  msa2021 <-fread("S:/file_path/2021.csv") 

 Warning in (if (.Platform$OS.type == "unix") system else shell)(paste0("(",  :
'(S:/file_path/2021.csv) > 
 C:\Users\someone\AppData\Local\Temp\3\RtmpqCvyUH\file2eac5c0526dd' execution failed with 
error code 1
Warning in fread("S:file_path/2021.csv") 
 :
 File 'C:\Users\someone\AppData\Local\Temp\3\RtmpqCvyUH\file2eac5c0526dd' has size 0. Returning a NULL data.table.

CodePudding user response:

Transcribed from comments:

Occasionally (too often?) I find problems loading files from network drives (perhaps more so reading from within R/python). In cases like this, try copying from the network drive S: to somewhere on your local C: drive and read from there. In this case, your problem has nothing to do with data.table::fread or the data size, merely a glitch accessing that large a file on the network.

  • Related