php - Create and download large zip from GCE without reaching memory and response limits

We have a problem downloading large .zip files from GCE, it is either memory limit (512 MB), or response size limit (32 MB).

We could increase memory limit, but we already have 1 GB .zip and it can be 10 GB next time.

Response size limit cannot be increased.

The goal: download product images archived as .zip.

Images vary from 1 MB to 10+ MB, so e.g. 100+ products .zip would be 1 GB in size. In the future there will be catalogs of 1K products, so the size can be 10 GB+.

Project is on Laravel 6.

I've already tried various solutions without success:

• Saving .zip in GCS and then start downloading. Result: reaches memory limits.
• Adding files to .zip without saving, and streaming withZipStream. Result: Response size limit triggered.

--

Code usingZipStream:

        $options->setSendHttpHeaders(true);
        //$options->setContentType('application/octet-stream');
        $options->setZeroHeader(true);

        $zipName = time().'.zip';

        // create a new zipstream object
        $zip = new ZipStream($zipName, $options);

        // Turned off deflate
        $fileOptions = new FileOptions();
        $fileOptions->setMethod(MethodOptions::STORE());

        foreach($files as $file) {
            $zip->addFileFromStream($file['name'], Storage::readStream($file['src'], $fileOptions));
        }

        // finish the zip stream
        $zip->finish();
Copy code

This way it is OK with memory limits, but I get response size (32MB) error which not sure how to overcome.

I've tried saving to GCS, but without any luck, it runs out of memory each time. Even if I could save successfully, it runs out of memory during download, e.g.:

$fs = Storage::getDriver();

        $metaData = $fs->getMetadata($filePath);
        $stream = $fs->readStream($filePath);

        return response()->streamDownload(function () use ($stream) {
            while(ob_get_level() > 0) ob_end_flush();

            fpassthru($stream);
        }, $name, [
            'Content-Type' => $metaData['type'],
            'Content-disposition' => 'attachment; filename="' . $name . '"',
        ]);
Copy code

The solution could be saving to GCS with longer (random) .zip name as public for shorter period of time (for security reasons), and initiate download directly, the problem is that I haven't managed to do so without reaching memory limits:

$zip = new ZipArchive();
        $zipName = time().'.zip';
        $downloadFolderName = 'downloads';
        $zipCreatedDirs = [];

        if (! Storage::exists($downloadFolderName)) {
            Storage::makeDirectory($downloadFolderName);
        }

        $zipPath = tempnam(sys_get_temp_dir(), time());

        if (! $zip->open($zipPath, ZIPARCHIVE::OVERWRITE)) {
            return response()->json([
                'status'  => 'error',
                'message' => 'Failed to create .zip archive.'
            ]);
        }

        // Add files for download one by one to zip
        // and split into different directories grouped by product
        foreach($files as $file) {
            if (! in_array($file['dirName'], $zipCreatedDirs)) {
                $zip->addEmptyDir($file['dirName']);

                $zipCreatedDirs[] = $file['dirName'];
            }

            $zip->addFromString($file['dirName'].'/'.$file['name'], Storage::get($file['src']));
        }

        $zip->close();

        Storage::putFileAs($downloadFolderName, new File($zipPath), $zipName, 'private');

        return response()->json([
            'status' => 'success',
            'downloadFiles' => [
                [
                    'src' => route('download', [base64_encode($downloadFolderName.'/'.$zipName), base64_encode($zipName)]),
                    'name' => $zipName
                ]
            ]
        ], 200, [], JSON_NUMERIC_CHECK);
Copy code

It seems that pretty much any of this doesn't work on GCE compared to Apache (local WAMP environment). If I set memory limit to 512 MB (the same as in GCE), I can download with the script above .zip of 1 GB without reaching memory limits.

Maybe I do something wrong, or don't understand something how it should be done. Any other thoughts or ideas? Any help would be appreciated.

Didn't find the answer?

Our community is visited by hundreds of web development professionals every day. Ask your question and get a quick answer for free.