How do I send compressed data from Laravel server to frontend JS (Solved)-CodePudding

Solved: Added the solution I used in an answer below

I have a compressed json file in my storage folder with path storage/app/public/data.json.gz I am trying to send this data to my js frontend via a fetch request. Sending the data is working but I am having trouble decompressing it back to JSON with js to use in my js code. I've read that it might be possible for the browser to automatically decompress this but I'm not sure how to enable it. Also the reason I am trying to send over the data compressed is because it is 130 MB of data that shrinks down to 7 MB when compressed and I am hoping that sending less data will speed up the site for users.

Laravel route that sends compressed file

Route::get('/chunks/{index}', function ($index) {
    $path = 'public/chunks_'.$index.'.json.gz';

    if (!Storage::exists($path)) {
        abort(404);
    }
    return Storage::response($path);
});

Currently I am using the fetch API to get the data JS Code

let chunks = await getZipFile("/chunks/0",[]).then((data) => {
    return data;
});

public static async getZipFile(url: string, params: any = {}, method = "GET", headers: any = {
        "Content-Type": "application/zip",
        }){
        
        headers['X-Requested-With'] = 'XMLHttpRequest';
        let options: any = {
           'method': method,
           'headers': headers
        };
        url  = "?"   new URLSearchParams(params).toString();
        
        const result = await fetch(url, options).then((response) => response);
        
        return result;
    };

any help would be appreciated, currently I can retrieve the compressed data and convert it to a string with result.text() but I have not been able to figure out how to decompress it. I tried using Zlib to decompress but got an error Can't resolve './zlib_bindings'. So looking for a solution similar to using Zlib (or something similar) to decompress or figuring out to configure the server/browser to automatically decompress.

CodePudding user response：

You're mixing two ideas. HTTP requests can be compressed and decompressed at a lower level than application level. This means that it will handle compression and decompression by itself, if enabled. See here.

What you have here is a compressed file. No frontend or ajax call would decompress it automatically for you.

Solutions:

Either enable compression for HTTP requests and depend on it to handle compression automatically. So send the uncompressed in this version. This could be helpful.
Or use a frontend library, like 'zip.js', to decompress when you receive the compressed file.

CodePudding user response：

I ended up taking Moradnejad's answer and used zip.js, here is the updated code

Laravel Route: Instead of declaring a route to send the file I used a laravel symbolic link to get a .zip file from my public storage. (https://laravel.com/docs/9.x/filesystem#the-public-disk)

Also incase it is helpful here is the command I wrote to create the .zip file from the files in my storage.

    public function handle()
    {
        $fileNames = [];
        for($i = 0; $i < 10000/1000; $i  ){
            array_push($fileNames,'public/chunks_'.$i.'.json');
        }
        $this->zipFiles($fileNames,'./storage/app/public','./storage/app/public/chunks.zip');

        return 0;
    }

    public function zipFiles($files,$path,$zipFileNameAndPath) {
        $zip = new ZipArchive;
        $zip->open($zipFileNameAndPath, ZipArchive::CREATE);
        foreach ($files as $file) {
        $zip->addFile($path.'/'.$file,$file);
        }
        $zip->close();
    }

Updated JS request code, I used result.blob() to return a blob of the data.

    public static zipRequest(url: string){
        return this.getZipFile(url);
    }

    public static async getZipFile(url: string, params: any = {}, method = "GET", headers: any = {
        "Content-Type": "application/zip",
        }){
        
        headers['X-Requested-With'] = 'XMLHttpRequest';
        let options: any = {
           'method': method,
           'headers': headers
        };

        if ("GET" === method) {
           url  = "?"   new URLSearchParams(params).toString();
        } else {
           //add csrf token to post request
           options.headers["X-CSRF-TOKEN"] = document.querySelector<HTMLElement>('meta[name="csrf-token"]')!.getAttribute('content');
           options.body = JSON.stringify(params);
        }
        const result = await fetch(url, options).then((response) => response);
        
        return result.blob();
    };

Updated JS handle blob result, I am using zip.js to get all 10 json files from the .zip data, and then I am merging the 10 json files together.

import * as zip from "@zip.js/zip.js";


async function getAllChunks() {
    let chunks = await Helper.getZipFile("storage/chunks.zip",[]).then( async (data) => {
        //console.log(data);

        let allChunks: any = [];
        let textReader = new zip.BlobReader(data);
        let zipReader = new zip.ZipReader(textReader);
        let entries = await zipReader.getEntries();

        for(let i = 0; i < entries.length; i  ){
            let entry = entries[i];
            if(entry.getData){
                let textWriter = new zip.TextWriter();
                let jsonString = await entry.getData(textWriter);
                let chunks = await JSON.parse(jsonString);
                allChunks.push.apply(allChunks, chunks);
            }
        }
        return allChunks;
    });
    return chunks;
}