Probabilistic diffusion models have become the established standard for generative modeling in continuous domains. DALLE is the leader in text-image delivery models. These models have gained importance due to their ability to generate images by training on large, web-scale datasets. The article discusses the recent emergence of text-image delivery models at the forefront of image generation. These models were trained on large-scale unsupervised or weakly supervised text-image datasets. However, due to their unsupervised nature, controlling their behavior in downstream tasks such as optimizing human-perceived image quality, image-to-text alignment, or generating ethical images is a difficult undertaking.
Recent research has attempted to refine diffusion models using reinforcement learning techniques, but this approach is known for its large variance in gradient estimators. In response, the paper presents “AlignProp”, a method that aligns diffusion models with downstream reward functions via end-to-end backpropagation of the reward gradient during the denoising process.
AligneProp’s innovative approach alleviates the high memory requirements that would typically be associated with backpropagation via modern text-image models. It achieves this by fine-tuning low-rank adapter weight modules and implementing gradient checkpoints.
The paper evaluates the performance of AligneProp in fine-tuning diffusion models for various objectives, including semantic image-text alignment, aesthetics, image compressibility, and controllability of the number of objects in the generated images, as well as combinations of these lenses. The results demonstrate that AligneProp outperforms alternative methods by achieving higher rewards in fewer training steps. Additionally, it stands out for its conceptual simplicity, making it a straightforward choice for optimizing diffusion models based on differentiable interest reward functions.
The AlignProp approach uses gradients obtained from the reward function with the aim of refining diffusion models, resulting in improvements in both sampling efficiency and computational efficiency. The conducted experiments consistently demonstrate the effectiveness of AligneProp in optimizing a wide range of reward functions, even for tasks that are difficult to set using prompts alone. In the future, potential research directions could involve extending these principles to broadcast-based linguistic models, with the aim of improving their alignment with human feedback.
Check Paper And Project. All credit for this research goes to the researchers of this project. Also don’t forget to register our SubReddit 31k+ ML, More than 40,000 Facebook communities, Discord Channel, And E-mailwhere we share the latest AI research news, interesting AI projects and much more.
If you like our work, you will love our newsletter.
We are also on WhatsApp. Join our AI channel on Whatsapp.
Janhavi Lande is a graduate in engineering physics from IIT Guwahati, batch of 2023. She is an upcoming data scientist and has been working in the world of ml/ai research for the past two years. Above all, she is fascinated by this constantly evolving world and by the constant demand of humans to keep pace with it. In her hobby, she likes to travel, read and write poems.
