Batched jobs cannot be put on differing queues

[ryangurn]

Laravel Version

^11.9

PHP Version

8.3.13

Database Driver & Version

MariaDB

Description

Before I get started, I am not sure if this is a bug, because it is unclear if what I am trying to do is even expected, we might end up closing this saying "batches were never meant to support that" and that is fine.

I was attempting to do some job batching, in which there are two separate queues that the jobs within the batch are placed within.

For the sake of example, the two queues were "processing" and "outbound" so I can handle prioritizing this properly. I started out using sqlite for my queue and db driver, however quickly switched to mariadb.

During this process, I realized that no matter what I do when I create a batch via a hydration job, I cannot seem to get onQueue() to place the specific job into the batch but on the specified queue. Below is some example code.

Bus::batch([ (new Job1())->onQueue('processing') ])->dispatch();

at a couple points I referred to the Laravel documentation that state, "Alternatively, you may specify the job's queue by calling the onQueue method within the job's constructor:

public function __construct()
{
        $this->onQueue('processing');
}

In my specific case Job1 would hydrate the batch, with more Job1's and Job2's (which in this specific case were supposed to be on the outbound queue).

I noticed that until I stopped batching the items, I was unable to set the queue properly on items within the batch.

Steps To Reproduce

Create a batch, use a recursive job (Job 1) to hydrate the batch with more instances of itself on one queue, meanwhile in Job 1 it will also continue to hydrate the batch with another job (Job 2) but on another queue.

[crynobone]

Hey there, thanks for reporting this issue.

We'll need more info and/or code to debug this further. Can you please create a repository with the command below, commit the code that reproduces the issue as one separate commit on the main/master branch and share the repository here?

Please make sure that you have the latest version of the Laravel installer in order to run this command. Please also make sure you have both Git & the GitHub CLI tool properly set up.

laravel new bug-report --github="--public"

Do not amend and create a separate commit with your custom changes. After you've posted the repository, we'll try to reproduce the issue.

Thanks!

[rodrigopedra]

This is by design:

All batched jobs must execute within the same connection and queue

Reference: https://laravel.com/docs/11.x/queues#batch-connection-queue

[rodrigopedra]

From your use case description, I think you could take a look into dispatching the hydrating job from a job chain, which also allows appending jobs later to it.

You can also append job batches later on.

Jobs/bacthes on a chain can be added to different queues.

On my understanding, this makes more sense, as you need the hydrating job to be executed before the other jobs.

It would be something like this:

flowchart TD
    A[Hydrating job]
    A -->|append to chain| B[Batch on Queue A]
    A -->|append to chain| C[Batch on Queue B]
    A -->|append to chain| D[Batch on Queue C]

Loading

Also, if only the hydrating job is in one queue and all the remaining are in the other queues, you can dispatch the hydrating job as a regular job, and batch the generated jobs from there.

[ryangurn]

This is by design:

All batched jobs must execute within the same connection and queue

Reference: https://laravel.com/docs/11.x/queues#batch-connection-queue

Thank you so much for the information, I missed that in the documentation.

From your use case description, I think you could take a look into dispatching the hydrating job from a job chain, which also allows appending jobs later to it.

You can also append job batches later on.

Jobs/bacthes on a chain can be added to different queues.

On my understanding, this makes more sense, as you need the hydrating job to be executed before the other jobs.

It would be something like this:

flowchart TD
    A[Hydrating job]
    A -->|append to chain| B[Batch on Queue A]
    A -->|append to chain| C[Batch on Queue B]
    A -->|append to chain| D[Batch on Queue C]

Loading

Also, if only the hydrating job is in one queue and all the remaining are in the other queues, you can dispatch the hydrating job as a regular job, and batch the generated jobs from there.

This is basically what I ended up doing, quite poorly at this moment, but alas that is progress. Thank you for explaining this, and it does help explain one nuance of chains, which was unclear from the documentation. That items within a chain can be in various queues, but batches can only exist on one queue at a time. (I kinda want to poke at the source code and see the difference between batch and chain implementation).

Glad to know there is no issue here, closing based on this.