The site is the CAISO OASIS and they use an old way of querying and returning datasets. 30 days at a time, zip file with enclosed *.csv files. I have tried for 4 days coming up with a way to automate this download first of the zip file. but i cant even get past there.
If I populate the url string I want and enter it myself in the address bar a zip file will start downloading, I cannot for the life of me get it to do the same through code.
I need to use php to accomplish this task. I have tried, fopen, file_get_contents, file_put_contents, cURL, ziparchive class. I have tried changing all sorts of things. The best I get is a download of a corrupt zip file that is empty.
Ideally i would love to extract the *.csv or xml file but I cannot even do the first basic step.
the following variable I populate, $url_caiso_qry, if you take the echo string and past in the browser and enter the zip file will start downloading, but I cannot download it automatically.
///////////////////////////////// start of code///////////////////////////
<?php
define("PRIVATE_PATH", dirname(__FILE__));
define("PROJECT_PATH", dirname(PRIVATE_PATH));
$public_end = strpos($_SERVER['SCRIPT_NAME'], '/public') 13;
$doc_root = substr($_SERVER['SCRIPT_NAME'], 0, $public_end);
define("WWW_ROOT", $doc_root);
// Initialize directory name where
// file will be save
$dir = PROJECT_PATH;
// creates data array for precio de carga SIN
//function create_dataarrayp_chrt($zona) {
/* $hoy = date("Y/m/d", strtotime('now'));
$manana = date("Y/m/d", strtotime('tomorrow'));
$semanapasada = date("Y/m/d",strtotime('-6 days'));
$manana = date("Ymd\TH:i:s", strtotime('tomorrow')); */
$hoy = date("Ymd\T7:00-0000", strtotime('now'));
$ayer = date("Ymd\T7:00-0000", strtotime('now - 1 day'));
$manana = date("Ymd\T7:00-0000", strtotime('tomorrow'));
$semanapasada = date("Ymd\T7:00-0000",strtotime('-6 days'));
$mespasado = date("Ymd\T7:00-0000",strtotime('-30 days'));
$sdate = $mespasado; //startdatetime=
$edate = $ayer; //enddatetime=
$resultformat = '6';
$queryname='SLD_FCST';
$version='1';
$marketrunid='ACTUAL';
$fields = [
'startdatetime' => $sdate,
'enddatetime' => $edate,
'resultformat' => $resultformat,
'queryname' => $queryname,
'market_run_id' => $marketrunid,
'version' => $version,
];
//post fields
$postparam = urldecode(http_build_query($fields));
//PRECIOS url base string
$urlcaiso = 'http://oasis.caiso.com/oasisapi/SingleZip?';
//$urlcaiso = 'http://oasis.caiso.com/oasisapi/SingleZip?resultformat=6&queryname=SLD_FCST&version=1&market_run_id=ACTUAL&startdatetime=' . $sdate . 'T07:00-0000&enddatetime=' . $edate . 'T07:00-0000';
$url_caiso_qry = $urlcaiso . $postparam;
// This variable $caisofile, produces an ugly looking symbol page when echo; then exit();
//I tried "rb" "wb"
//echo $caisofile;
//exit();
$caisofile = file_get_contents($url_caiso_qry, "rb");
//recreate the filename that appears in the response header, adjusted to get matching strings. $tmp_zipfilename
$ptoday = date("Ymd", strtotime('now'));
$sdatef = date("Ymd", strtotime('-31 days'));
$edatef = date("Ymd", strtotime('now - 2 day'));
$stampf = date("H_i_s", strtotime('now - 9 hours - 1 sec'));
//this is to replicate the timestamp that is used when you just enter the url in the address bar, if you just used the url string above as
//defined by $url_caiso_qry when echo 20220108_20220206_SLD_FCST_ACTUAL_20220208_19_03_30_v1.zip; when you enter the $url_caiso_qry string in the
//address bar of chrome, the response is a zip file.
$tmp_file_name = $sdatef . '_' . $edatef . '_' . $queryname . '_' . $marketrunid . '_' . $ptoday . '_' . $stampf . '_v1';
$tmp_zipfilename = $tmp_file_name . '.zip';
//$tmp_file_name_csv = $tmp_file_name. '.csv';
//verify through echo response header
/* echo var_dump($http_response_header);
echo '<br><br>';
echo $http_response_header[1];
echo '<br><br>';
echo $tmp_zipfilename; */
//string to create full url path to replicated filename. did not work when using fopen or file_get_contents.
//$file_name_qry = $file_name_path . '&filename=' . $tmp_zipfilename;
//attempt to write file to destination path, but no file is written. Instead the following warning
//Warning: file_put_contents(/DJworkspace/proj3-ampenergy/private/test/):
// failed to open stream: No such file or directory in C:\xampp\htdocs\DJworkspace\proj3-ampenergy\private\test.php on line 94
//20220108_20220206_SLD_FCST_ACTUAL_20220208_19_21_45_v1.zip
$destination_path = WWW_ROOT . 'proj3-ampenergy/private/test/';
$newfile = file_put_contents($destination_path, $caisofile);
echo $tmp_zipfilename . '<br><br>';
//echo $file_name_qry . '<br><br>';
echo $newfile . '<br><br>';
//I tried 5-6 different cURL statements, but nothing worked
/*
// Initialize the cURL session
$ch = curl_init();
//$timeout = 5;
// It set an option for a cURL transfer
curl_setopt($ch, CURLOPT_URL, $file_name_path);
// curl_setopt($ch, CURLOPT_PORT , 80);
// curl_setopt($ch,CURLOPT_POSTFIELDS, $postparam);
//
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
//curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_FILE, $tmpzip);
// Perform a cURL session
$response = curl_exec($ch);
curl_close($ch);
$zipf = new ZipArchive();
//$zipf->open(WWW_ROOT . 'proj3-ampenergy/private/test/' . $tmpzip);
$zipf->open($response);
$extractPath = WWW_ROOT . 'proj3-ampenergy/private/test/';
$zipf->extractTo($extractPath);
$zipf->close();
// Closes a cURL session and frees all resources
*/
/*
// echo $dir . $data;
echo basename($_SERVER['REQUEST_URI'], '?' . $_SERVER['QUERY_STRING']);
exit();
*/
// Use basename() function to return
// the base name of file
// $file_name = '*.zip';
// Save file into file location
// $save_file_loc = $dir . $file_name;
// Open file
// $fp = fopen($urlcaiso, 'wb');
/* $fp = fopen($dir . $data, 'wb');
$zip = new ZipArchive;
$res = $zip->open($fp);
if ($res === TRUE) {
echo 'ok';
$zip->extractTo($dir);
$zip->close();
} else {
echo 'failed, code:' . $res;
} */
// Close file
// fclose($fp);
?>
CodePudding user response：
The second parameter you pass to file_get_contents describes the use_include_path according to the php documentation.
I tried this simple example to get the file and save it:
<?php
$content = file_get_contents('http://oasis.caiso.com/oasisapi/SingleZip?startdatetime=20220109T7:00-0000&enddatetime=20220207T7:00-0000&resultformat=6&queryname=SLD_FCST&market_run_id=ACTUAL&version=1');
file_put_contents('example.zip', $content);
Running file example.zip gives the following output:
example.zip: Zip archive data, at least v2.0 to extract which indicates success.
So essentially your final code could look something like this:
<?php
// Build url here
$url = 'http://oasis.caiso.com/...';
// Download file
$zipFile = file_get_contents($url);
$public_end = strpos($_SERVER['SCRIPT_NAME'], '/public') 13;
$doc_root = substr($_SERVER['SCRIPT_NAME'], 0, $public_end);
define("WWW_ROOT", $doc_root);
$targetPath = WWW_ROOT . 'proj3-ampenergy/private/test/'; // This could be any other path where your file should be stored
// Write contents to file
file_put_contents($targetPath, $zipFile);
// Write extraction logic here
CodePudding user response：
Clean & tested code. Your csv file will be extracted to your script's directory.
<?php
define('PATH_ZIP', __DIR__ . '/archive.zip');
$url = 'http://oasis.caiso.com/oasisapi/SingleZip?' . http_build_query
([
'startdatetime' => '20220109T7:00-0000',
'enddatetime' => '20220207T7:00-0000',
'resultformat' => '6',
'queryname' => 'SLD_FCST',
'market_run_id' => 'ACTUAL',
'version' => '1',
]);
$data = file_get_contents($url);
file_put_contents(PATH_ZIP, $data);
$zip = new ZipArchive;
if ($zip->open(PATH_ZIP) === true)
{
$zip->extractTo(__DIR__);
$zip->close();
}