N+1 problem in Validator class

[dazza-dev]

Hi guys, I want to know if anyone has the same problem as me.

Validating data with Laravel's Validator class causes an N+1 problem. Example if I send an array and I want to validate that the document number already exists in the database, a query is generated for each record.

I am using lazy collection to do batch validations and this helped a lot but it could be improved more if Laravel allowed batch validations and not iterating on each item

  public function validateDataInBatches($batchData) {
    $batchSize = 1000;
    $totalRecords = count($batchData);
    $batches = ceil($totalRecords / $batchSize);
    $request = new ImportValidationRequest;

    LazyCollection::make($batchData)->chunk(100)
    ->each(function ($batch) use ($request) {
        $request->validateBatch($batch->toArray());
    });
  }

[henzeb]

I suggest moving validation of existence in database up. So each chunk first validate the order numbers, then validate the rest. Validator was more designed for http requests with simple POST data.

[dazza-dev]

I faced the same problem, and since there wasn't a solution available, I decided to create a package: https://github.com/dazza-dev/Laravel-Batch-Validation that solves it. This is the first version, so it may have some bugs, but it would be great if you could try it out and help improve it further. It works well for my use case, but I hope it can be helpful to the Laravel community.