pipelining with python client

[romange]

Hello,

is it possible to run multiplexing/pipelining with the python client?
it seems to me that the sample below does not trigger multiplexing and I did not find the api for the explicit pipeling.

    for i in range(100000):
        await client.set(f"key{i}", "1")

[asafpamzn]

Hi,

We are currently working on pipeline support.
Adding Shoham to add more details.
I suggest to use MSET for this scenario, it will provide the best performance.

[shohamazon]

Hello 🙂 ,

For now, there are two possible solutions to achieve batch operations:

1. mset:

data = {f"key{i}": "1" for i in range(100000)}
await client.mset(data)

2. asyncio.gather: This method allows you to run tasks concurrently.
   However, please note that while tasks are executed concurrently, they are not guaranteed to be scheduled or executed in a specific order.

data = {f"key{i}": "1" for i in range(100000)}
tasks = [client.set(key, value) for key, value in data.items()]
await asyncio.gather(*tasks)

We're working on adding support for pipelines in Glide.

[BoazBD]

Hi @romange
The release candidate for Python with pipeline support is ready.
You can install it with:
pip install valkey-glide==2.0.0rc2
Docs are available here:
https://valkey.io/valkey-glide/python/standalone_batch/
https://valkey.io/valkey-glide/python/cluster_batch/
We would love to get your feedback.
Let me know if you have any questions!

[romange]

I do actually.

1. So with redis-py, the client aggregates everything in python until pipeline.execute() is called. This creates non-uniform load on a backend for long pipelines (say hundreds of commands or more). It is more efficient to flush pipelines prematurely underneath to let backend start processing everything even before execute() is called. Especially since it's async rust underneath, you can probably start parsing responses even before client.exec is called. Have you considered anything like this?
2. What is the second argument in result = await client.exec(transaction, false) in https://valkey.io/valkey-glide/python/standalone_batch/ ?

[shohamazon]

@romange Thanks for the response! 🙂

1. I agree, if its something critical for your workflow, I recommend opening a dedicated GitHub issue to ensure it receives the attention it deserves.
2. The flag in question determines how errors are handled during command execution. Specifically:

raise_on_error=True: An exception is raised if a command fails.
raise_on_error=False: Errors are captured and included in the response without raising exceptions.

For detailed information, please refer to the official documentation:

Cluster Commands exec
Standalone Commands exec

[avifenesh]

@romange 2 questions — what are the recommendations of your service for clients?
How can I catch for a short chat?

[romange]

@avifenesh sure, send me an email to roman at dragonflydb.io
I am available from May 25