Add Operator-Level Parallel Data Processing with Ray Actors

[Cccccc0630]

This PR introduces significant performance optimizations for parallel data processing using Ray Actors and multi-threading. The following changes have been implemented:

1. Dynamic Resource Allocation:

   • Multiple Actors per operator (OP) are now dynamically created based on resource requirements (CPU/GPU).
   • The OP using CUDA will load corresponding model onto to available resources accordingly when its Actors is created.

2. Parallel Data Processing:

   • A data distribution thread is responsible for distributing batches of data to the first operator's actors, supporting data processing for multiple streams.

Benefits:

• Improved Performance: By utilizing multi-threading and Ray Actors for parallel data processing, the system can handle large volumes of data more efficiently.
• Scalability: The dynamic creation of actors based on resource availability allows the system to scale according to the workload.

Future Work (Potential Follow-up PRs)​:

• Support parallel batch processing and implement processing_batched for OPs in the pr_demo.yaml process.
• Achieve multi-Actor parallelism on GPUs to improve GPU utilization and SM% (Streaming Multiprocessor percentage).

Experiment:

The end-to-end efficiency optimization for a small amount of videos is shown in the figure below. Increasing the amount of data can make the advantages of parallel processing more obvious.

[gemini-code-assist]

Summary of Changes

Hello @Cccccc0630, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the data processing capabilities by introducing operator-level parallel execution using Ray Actors. The core change involves a new parallel processing pipeline that dynamically allocates resources, distributes data efficiently across multiple actors, and leverages multi-threading to maximize throughput. This optimization is designed to handle large volumes of data more efficiently and improve scalability, particularly for resource-intensive operations like video processing. The changes provide a robust framework for distributed data transformation, allowing for better utilization of available computing resources.

Highlights

• Parallel Data Processing with Ray Actors: The pull request introduces a new process_parallel method in the RayDataset class, enabling operators to run concurrently using Ray Actors. This method dynamically creates actors for each operator based on their resource requirements (CPU/GPU), significantly improving processing efficiency.
• Efficient Data Distribution and Streaming: A dedicated data distribution thread has been implemented to efficiently distribute data batches to the first operator's actors. This design supports processing multiple data streams simultaneously, ensuring a smooth and continuous flow of data through the parallel pipeline.
• Configurable Parallel Execution: The RayExecutor now includes a configurable flag, op_enable_parallel, allowing users to switch between the traditional sequential processing and the new parallel processing mode. This provides flexibility and control over how data operations are executed.
• Dedicated Ray Actor for Operators: A new Actor class has been added, serving as a remote execution wrapper for individual operators. This class handles the loading of models and the execution of both single and batched operations (mappers and filters) on either CPU or CUDA, facilitating the distributed nature of the processing.
• Operator Adaptations for Parallelism: The base operator (OP) class and specific video processing operators (e.g., video_aesthetics_filter, video_watermark_filter, video_captioning_from_frames_mapper) have been updated to support the new actor-based execution model, including the ability to specify GPU requirements and load models within the actor context.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a significant and complex feature for parallel data processing using Ray Actors, which is a great step towards improving performance. The overall architecture is well-designed, but I've identified several areas for improvement concerning performance, correctness, and maintainability. My review includes feedback on potential performance bottlenecks, bugs that could lead to incorrect behavior, and suggestions for refactoring to improve code quality. Please review the detailed comments on the specific files.

[pan-x-c]

Good job!
Here are some suggestions:

1. Clean up the code, use English comments and fix pre-commit.
2. Move core/ray_actor.py to ops. The current Actor is actually more like an operator wrapper, and can even be renamed to RayOperatorWrapper or something like it.