AI models for graphics editing

[Keavon]

Below is a list of AI model disciplines that will be useful tools in the graphics editing process. Feel free to comment with ideas for more items missing from this list.

• Image generation (to turn inputs like text, images, masks, or other control data into a desired image)
  • Infill (to generate the content in missing areas based on a mask, or uncrop an image)
  • Style transfer (to adapt the subject of one image to the art style of another image)
• Upscaling
• Segmentation (to automatically mask a subject or break a scene apart into multiple subjects)
• Depth estimation (to generate a depth map that can be used for many procedural effects)
• Decomposing into render channels like albedo, normal, depth, irradiance, roughness, metalness (RGB↔X)
• Relighting (to change the direction and color cast of the lighting on a scene or subject)
• Novel view synthesis (to alter the perspective angle or FoV of a subject)
• Altering a scene's focus or deconvolving blur
• Un-smearing a motion-blurred image
• Noise removal (sensor noise, JPEG artifacts)
• SDR to HDR conversion by inferring the extra data that was outside the camera's dynamic range
  • Recovering clipped pixels in overexposed scenes
• Color gamut extension by inferring true WCG colors of a scene beyond the range of the imaging device sensor

[CheeksTheGeek]

Can I possibly submit a GSOC Proposal for this exact mega-task?
It seems like an interesting domain to which I'd like to contribute, can see starting points for 4 out of 5 pointers (except Novel View Synthesis), and would very much like to learn in these fields, and seems like the timeline matches with that of GSOC?

Though I can understand if the maintainers would like someone more specialized to be up for this task, if I get the opportunity, I'll definitely make it up to the quality standards maintained in this great project!

Let me know if it's possible and I'll start working on a proposal right away!

[Keavon]

@CheeksTheGeek please join the Discord and discuss this with me there. The task is described somewhat further in https://graphite.rs/volunteer/guide/student-projects/#machine-learning-architecture. Just to set expectations, I will note that this requires a decently advanced level of understanding in deep learning concepts, including a fluent understanding of how to use libraries like PyTorch or similar to implement concepts based on your readings of research papers.

[akshitgaur2005]

Hello! I'm interested in contributing to this task and had a few questions:

1. Has this task already been assigned?
2. If not, is there a possibility of it being included in GSoC 2025?

I have some experience with Python, PyTorch, and deep learning algorithms. I've also experimented with setting up and using Stable Diffusion, Segment Anything, and llama.cpp. While I don't have Rust experience yet, I'm committed to learning it as both the language and this project domain interest me greatly.
I plan to contribute regardless of GSoC status, but early confirmation would help me prioritize my Rust learning schedule.

Thanks!

[akshitgaur2005]

While waiting for your confirmation, I went ahead and researched a bit about what would be needed for implementing these tools.

1. Upscaling - Upscaling seems simple enough. Candle could be used to implement something like ESRGAN.
2. Image Generation - Candle provides a pre-written implementation of Stable Diffusion as an example which is quite performant in and of itself. Infilling and rest will take more effort to seemlessly integrate.
3. Depth Estimation - Models like MiDaS do exist but there does not seem to be any implementation in rust as of now. It should be possible to implement it in Candle and use the pre-trained model.
4. Segmentation - Implementation for Segment Anything already exists for candle.
5. Relighting - Models such as DiLightNet and IC-Light exist, but I will have to explore this avenue more before I can say anything with surety.
6. Novel View Synthesis - Found a few papers but have to look more, same for deconvolving a blur.

[Keavon]

Hi @akshitgaur2005. It might be best if we discuss this through Discord in the #ai-ml channel. I'll just say that this is an important topic, but also the most speculative because there are many possible approaches and no single straightforward path, which means it's more uncertain as to whether it works out in a way that ends up being useful to Graphite. A big part of it would be pathfinding the right approach and making me feel confident that it's a solid path forward that is likely to result in good long-term outcomes. Now is a good time to get started if you're interested in it because you'd be able to experiment with solutions and make a case for why they'd succeed as a GSoC project. Normally, the default state of this project within the scope of a GSoC proposal would be that it's a risky one and therefore might not top our list of priorities if we have other good proposals that have a more certain pathway towards success given the nature of limited applicant slots we can choose. That said, even though it's a hard problem, it's not one we can continue to neglect for years while the industry marches past us, so some R&D in this direction is of value regardless. I'll say, a few sample projects would be an initial implementation of Segment Anything 2 or Depth Anything, and ideally running in WGPU through either Candle or Burn. The latter seemed more aligned with our long-term goals, but is more removed from the Hugging Face community than Candle which may mean we'd miss out on the value of HF's research adjacency. Let's talk more on Discord if you'd like to dive into the weeds of things.

[Keavon]

Great example from RGB↔X of the useful workflows that are possible:

[j-jasz]

Segmentation model with Apache 2.0 license: https://github.com/xuebinqin/DIS