My goal is to read in zip files directly from the web (opentransportdata.swiss). Each zip file contains multiple .txt files. In my example I am trying to retrieve the data of the routes.txt file.
So my code is the following:
library(tidyverse)
# links
tt_url <- c("https://opentransportdata.swiss/de/dataset/7787e566-03cf-4cd5-8a66-b5af08547e74/resource/4bc9d75e-cdd7-4020-8ee1-9dd494ee8b4c/download/gtfsfp20162016-11-30.zip",
"https://opentransportdata.swiss/de/dataset/587ecf41-eb18-448a-8073-7076bc3cbfeb/resource/e499a630-4e65-4e00-8522-26c5c78b88ca/download/gtfsfp20172017-12-06.zip")
# download zip files
f_get_data <- function(i, data){
url <- tt_url[i]
zip_file <- tempfile(fileext = ".zip")
download.file(url, zip_file, mode = "wb")
df <- read_delim(unzip(zip_file, files = data), delim = ",") %>%
mutate(year = i 2015)
return(df)
}
test_1 <- f_get_data(1, "routes.txt")
head(test_1)
test_2 <- f_get_data(2, "routes.txt")
head(test_2)
head(test_1)
If one applies the function f_get_data(1, "routes.txt) the first time, the retrieved df ,test_1, is correct.
head(test_1)
# A tibble: 6 × 8
route_id agency_id route_short_name route_long_name route_type route_color route_text_color year
<chr> <lgl> <chr> <lgl> <dbl> <lgl> <lgl> <dbl>
1 11-21-j16-1 NA 021 NA 3 NA NA 2016
2 11-22-j16-1 NA 022 NA 3 NA NA 2016
3 16-22-j16-1 NA 022 NA 3 NA NA 2016
4 11-25-j16-1 NA 025 NA 3 NA NA 2016
5 11-41-j16-1 NA 041 NA 3 NA NA 2016
6 11-42-j16-1 NA 042 NA 3 NA NA 2016
If I go onto the next period with f_get_data(2, "routes.txt), the retrieved df, test_2, is also correct.
BUT, after I completed my second iteration, the first df, test_1, corrupts itself:
> head(test_2)
# A tibble: 6 × 7
route_id agency_id route_short_name route_long_name route_desc route_type year
<chr> <chr> <chr> <lgl> <chr> <dbl> <dbl>
1 79-0-j17-1 881 00 NA Bus 700 2017
2 11-61-j17-1 7031 061 NA Bus 700 2017
3 11-62-j17-1 7031 062 NA Bus 700 2017
4 24-64-j17-1 801 064 NA Bus 700 2017
5 24-65-j17-1 801 065 NA Bus 700 2017
6 24-66-j17-1 801 066 NA Bus 700 2017
> head(test_1)
# A tibble: 6 × 8
route_id agency_id route_short_name route_long_name route_type route_color route_text_color year
<chr> <lgl> <chr> <lgl> <dbl> <lgl> <lgl> <dbl>
1 ",\"00\",\"\",\"Bus" NA "00\"\r\n" NA NA NA NA 2016
2 "7031\",\"061\",\"" NA "us\",\"" NA NA NA NA 2016
3 "7-1\",\"7031\",\"" NA ",\"\",\"" NA NA NA NA 2016
4 "-64-j17-1\",\"8" NA "064" NA NA NA NA 2016
5 "\r\n\"24-65-j17-" NA "801\"," NA NA NA NA 2016
6 "700\r\n24-66" NA "-1\",\"" NA NA NA NA 2016
Does anyone know why and especially how this happens? In my opinion, after I have assigned the retrieved data of my function to a certain data frame, it should be independent of the later use of my function.
CodePudding user response：
Cannot replicate this, not coding tidyversish though. Perhaps try my code.
> # links
> tt_url <- c("https://opentransportdata.swiss/de/dataset/7787e566-03cf-4cd5-8a66-b5af08547e74/resource/4bc9d75e-cdd7-4020-8ee1-9dd494ee8b4c/download/gtfsfp20162016-11-30.zip",
"https://opentransportdata.swiss/de/dataset/587ecf41-eb18-448a-8073-7076bc3cbfeb/resource/e499a630-4e65-4e00-8522-26c5c78b88ca/download/gtfsfp20172017-12-06.zip")
> # download zip files
> f_get_data <- function(i, data) {
on.exit(unlink(temp)) ## don't forget to unlink your tempfiles!
temp <- tempfile(fileext='.zip')
url <- tt_url[i]
download.file(url, temp, mode = "wb")
df <- read.csv(unzip(temp, files=data)) |>
transform(year=i 2015)
return(df)
}
>
> test_1 <- f_get_data(1, "routes.txt")
trying URL 'https://opentransportdata.swiss/de/dataset/7787e566-03cf-4cd5-8a66-b5af08547e74/resource/4bc9d75e-cdd7-4020-8ee1-9dd494ee8b4c/download/gtfsfp20162016-11-30.zip'
downloaded 26.1 MB
> head(test_1)
route_id agency_id route_short_name route_long_name route_type route_color route_text_color year
1 11-21-j16-1 NA 021 NA 3 NA NA 2016
2 11-22-j16-1 NA 022 NA 3 NA NA 2016
3 16-22-j16-1 NA 022 NA 3 NA NA 2016
4 11-25-j16-1 NA 025 NA 3 NA NA 2016
5 11-41-j16-1 NA 041 NA 3 NA NA 2016
6 11-42-j16-1 NA 042 NA 3 NA NA 2016
> test_2 <- f_get_data(2, "routes.txt")
trying URL 'https://opentransportdata.swiss/de/dataset/587ecf41-eb18-448a-8073-7076bc3cbfeb/resource/e499a630-4e65-4e00-8522-26c5c78b88ca/download/gtfsfp20172017-12-06.zip'
downloaded 81.1 MB
> head(test_2)
route_id agency_id route_short_name route_long_name route_desc route_type year
1 79-0-j17-1 881 00 NA Bus 700 2017
2 11-61-j17-1 7031 061 NA Bus 700 2017
3 11-62-j17-1 7031 062 NA Bus 700 2017
4 24-64-j17-1 801 064 NA Bus 700 2017
5 24-65-j17-1 801 065 NA Bus 700 2017
6 24-66-j17-1 801 066 NA Bus 700 2017
> head(test_1)
route_id agency_id route_short_name route_long_name route_type route_color route_text_color year
1 11-21-j16-1 NA 021 NA 3 NA NA 2016
2 11-22-j16-1 NA 022 NA 3 NA NA 2016
3 16-22-j16-1 NA 022 NA 3 NA NA 2016
4 11-25-j16-1 NA 025 NA 3 NA NA 2016
5 11-41-j16-1 NA 041 NA 3 NA NA 2016
6 11-42-j16-1 NA 042 NA 3 NA NA 2016
>
CodePudding user response：
The problem is the default behavior of the read_delim() function. In order to improve performance the data is loaded in a lazy manner, meaning the data is only accessed when needed.
So in actuality the return value from "f_get_data" is just a pointer to the data. In this case it is a pointer your temporary file which is overwritten on each call to the function.
To solve this, set lazy to FALSE in the read_delim() function call.
df <- read_delim(unzip(zip_file, files = data), delim = ",", lazy=FALSE) %>%
mutate(year = i 2015)