php - Why does my laravel horizon job fail during file chunk process?

Situation: Job runs fine the 1st time (1 min), fails quietly if I run it again. The issue is somewhere in writing the smaller files as I can see partial files on my local storage. This job can only run once so I'll get the typical: "job has been attempted too many times or run too long" after the actual timeout has been reached.

    $importFileProcessing = ImportFileProcessing::find($this->jobNumber);
    $importFileProcessing->RecordCount = $this->getContent()->count() - 1;
    $importFileProcessing->saveQuietly();

    $basename = basename($this->fileName, '.csv');
    $successFileName = sprintf('%d%s.csv', $this->jobNumber, ImportFileProcessing::SUCCESS_FILE_SUFFIX);
    $errorFileName = sprintf('%d%s.csv', $this->jobNumber, ImportFileProcessing::ERROR_FILE_SUFFIX);

    $recordsArray = iterator_to_array($this->getContent()->getRecords());
    $header = $recordsArray[0];
    unset($recordsArray[0]);
    $chunks = array_chunk($recordsArray, self::FILE_MAX_CHUNK_SIZE);
    $chunkCount = sizeof($chunks);

    // success file
    $writer = Writer::createFromPath(sprintf('%s/app/csv/%s', storage_path(), $successFileName), 'a+');
    $writer->insertOne($header);
    // error file
    $writer = Writer::createFromPath(sprintf('%s/app/csv/%s', storage_path(), $errorFileName), 'a+');
    $writer->insertOne(array_merge($header, ['errors']));

    Redis::incrby(sprintf('ImportCSVCRM:%d:%d:ChunksRemaining', $this->companyId, $this->jobNumber), $chunkCount);
    Redis::expire(sprintf('ImportCSVCRM:%d:%d:ChunksRemaining', $this->companyId, $this->jobNumber), 86400); // 86400 seconds per day

    $customFields = $this->getRequestFieldsByType($this->requestFields, ImportSearchField::FIELD_TYPE_CUSTOM);

    foreach ($chunks as $key => $chunk) {
        $newFileName = sprintf('%s_job_%s_chunk_%04d.csv', $basename, (string) $this->jobNumber, $key);
        $writer = Writer::createFromPath(sprintf('%s/app/csv/%s', storage_path(), $newFileName), 'w+');
        $writer->insertOne($header);
        $writer->insertAll($chunk);
        
        ImportCSVCRM::dispatch(
            $this->companyId,
            $this->jobNumber,
            $newFileName,
            $this->fileName,
            $this->signUpSourceId,
            $this->prospect,
            $this->requestFields,
            $this->requestValidationModifiers,
            $customFields,
            $successFileName,
            sprintf('%s/app/csv/%s', storage_path(), $successFileName),
            $errorFileName,
            sprintf('%s/app/csv/%s', storage_path(), $errorFileName),
            $chunkCount,
            $this->mergeField,
        )->onQueue(ImportCSVCRM::QUEUE_NAME);
    }
Copy code

Didn't find the answer?

Our community is visited by hundreds of web development professionals every day. Ask your question and get a quick answer for free.