Check out the code for our recent #ICLR2024 paper "ROPIM", a self-supervised learning method for Vision Transformers (ViT). ROPIM forms “continuous” masking/unmasking strategy that creates a rich model with less pre-training cost. CSIRO's Data61 QUT (Queensland University of Technology) #TGIF Github: https://lnkd.in/g_E_aWkJ paper: https://lnkd.in/gyWEuNHa
Can we train AI faster, and with less compute power? Our Embodied AI Cluster researchers have released ROPIM (Random Orthogonal Projection Image Modeling), a ground-breaking self-supervised learning method for Vision Transformers (ViTs). Self-supervised learning for transformers, which teaches AI models to learn from unlabelled data, underpins the recent successes in generative AI. The conventional way for self-supervised Vision Transformers is through Masked Image Modeling (MIM). This technique uses binary masking, randomly blanking out some regions in the input data so that the AI model learns to recover the missing data. In contrast, ROPIM forms a ‘continuous’ masking and unmasking strategy to create a rich model with less pre-training cost. This research was featured as a Spotlight paper (top 5%) at The Twelfth ICLR 2024. The research was led by Maryam Haghighat, along with Peyman Moghadam, Mohamed Shaheer, and Piotr Koniusz. Embodied AI Cluster Leader Dr Moghadam said this research is another critical step toward the team’s outlined vision of AI models that can learn directly from their interactions with the physical world, without human-generated annotation. Read the paper 🔗 https://lnkd.in/gyWEuNHa Source code 🔗 https://lnkd.in/g_E_aWkJ