Self-Supervised Learning Methods

LeJEPA: A new era for Self-Supervised Learning(SSL) - Simple, Provable, Scalable! Self-supervised learning has received a significant upgrade with LeJEPA, developed by Profs Randall Balestriero and Yann LeCun. This framework introduces the first principled, theory-driven JEPA approach that eliminates the heuristics that have long dominated SSL, including stop-gradients, teacher-student networks, EMA schedules, and whitening tricks. The result is clean, stable, and provable learning. Here’s why the research community is excited:- 1. A foundational breakthrough:- The paper demonstrates that to minimize downstream risk across tasks, the optimal embedding distribution is an isotropic Gaussian, providing SSL with its first normative objective rather than just empirical recipes. 2. SIGReg:- LeJEPA introduces Sketched Isotropic Gaussian Regularization (SIGReg), which utilizes directional slicing and characteristic function tests. It operates in linear time and memory, offers bounded gradients for stable training, and completely avoids representation collapse. This is one of the cleanest regularizers seen in the SSL field. 3. One hyperparameter:- LeJEPA trains various architectures, including ResNets, ConvNeXts, and ViTs up to 1.8B parameters across 10+ datasets, all without the need for tuning fragile heuristics. In-domain SSL even outperforms DINOv2 and DINOv3 in transfer learning. 4. Real-world performance:- ViT-H/14 achieves 79% ImageNet linear-probe accuracy, with PCA visualizations showing a clean semantic structure. Training curves remain stable even for billion-scale models. 5. The bigger picture:- By grounding JEPAs in first-principles mathematics and providing a clean, scalable recipe, LeJEPA could lead to a resurgence of principled pretraining in AI, moving beyond brute-force scaling. This paper is a strong reminder that breakthroughs don’t always come from bigger models or more data. Sometimes they come from better theory, cleaner objectives, and a deeper understanding of what representations should look like. LeJEPA is exactly that. #AI #SelfSupervisedLearning #JEPA #LeJEPA #MachineLearningResearch #RepresentationLearning #DeepLearning #YannLeCun #SSL #AIResearch #FoundationModels #ViT #DINOv2 #MLTheory #ScalableAI

LLMs and image generators need much more training data than a human child does to reach comparable competence, and one explanation is that predicting raw tokens—the next word, a masked pixel—wastes effort on surface details rather than the abstract structure underneath. Some recent methods instead train a network to predict its own internal representations of the data, but there has been little theory saying how much this helps or why. The authors of this paper study this using a synthetic language model called the Random Hierarchy Model, where data is built by a tree of hidden symbols of depth L: visible tokens at the bottom, increasingly abstract latent symbols above, with fixed random rules connecting each level to the next. Within this setting they can count exactly how many training examples a method needs to recover the hidden tree. Standard supervised learning and token-level prediction need a number of samples that grows exponentially with the depth L, because the statistical signal connecting a token to distant parts of the tree gets averaged away across many levels. The authors prove that predicting one's own latents instead needs a number of samples that does not grow with depth at all (roughly m³, where m is the number of rules per symbol), because once a level has been decoded the next level becomes just as easy to learn as the first—every level reduces to the same local problem. https://lnkd.in/g7Urcvu7

🚀 𝘃𝟮 𝗼𝗳 𝗼𝘂𝗿 𝗽𝗮𝗽𝗲𝗿 “𝗟𝗟𝗠-𝗝𝗘𝗣𝗔” 𝗶𝘀 𝗼𝘂𝘁 𝗼𝗻 𝗮𝗿𝗫𝗶𝘃! 🔍 𝐖𝐡𝐚𝐭’𝐬 𝐧𝐞𝐰? ✅ 𝗦𝗶𝗴𝗻𝗶𝗳𝗶𝗰𝗮𝗻𝘁𝗹𝘆 𝗹𝗼𝘄𝗲𝗿 𝗰𝗼𝗺𝗽𝘂𝘁𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝗼𝘃𝗲𝗿𝗵𝗲𝗮𝗱 — reduced overhead from 𝟮𝟬𝟬% → 𝟮𝟱% using a simple yet effective 𝗿𝗮𝗻𝗱𝗼𝗺 𝗝𝗘𝗣𝗔-𝗹𝗼𝘀𝘀 𝗱𝗿𝗼𝗽𝗼𝘂𝘁. ✅ 𝗕𝗿𝗼𝗮𝗱𝗲𝗿 𝗮𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀 — extended beyond symmetric 2-view datasets to 𝗡𝗤-𝗢𝗽𝗲𝗻 (Natural Questions for open-domain) and 𝗛𝗲𝗹𝗹𝗮𝗦𝘄𝗮𝗴 (sentence completion), and tested on reasoning models like 𝗤𝘄𝗲𝗻𝟯 and 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸-𝗥𝟭-𝗗𝗶𝘀𝘁𝗶𝗹𝗹𝗲𝗱. ✅ 𝗥𝗶𝗴𝗼𝗿𝗼𝘂𝘀 𝗮𝗯𝗹𝗮𝘁𝗶𝗼𝗻𝘀 — JEPA loss design outperforms alternatives including 𝗟𝟮, 𝗠𝗦𝗘, 𝗽𝗿𝗲𝗽𝗲𝗻𝗱 [𝗣𝗥𝗘𝗗] 𝘁𝗼𝗸𝗲𝗻𝘀, 𝗖𝗼𝗱𝗲→𝗧𝗲𝘅𝘁, and 𝗜𝗻𝗳𝗼𝗡𝗖𝗘 variants. 🧩 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐋𝐋𝐌-𝐉𝐄𝐏𝐀? If you’re seeing this for the first time: LLM-JEPA introduces the 𝗝𝗼𝗶𝗻𝘁 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴 𝗣𝗿𝗲𝗱𝗶𝗰𝘁𝗶𝘃𝗲 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 (𝗝𝗘𝗣𝗔) — a self-supervised learning paradigm proven in vision — as a 𝗿𝗲𝗴𝘂𝗹𝗮𝗿𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗹𝗼𝘀𝘀 for LLMs. Combined with next-token prediction, it enables models to: 🎯 Boost fine-tuning accuracy 🧠 Resist overfitting 🌱 Work in pretraining via 𝗽𝗮𝗿𝗮𝗽𝗵𝗿𝗮𝘀𝗲-𝗯𝗮𝘀𝗲𝗱 𝗝𝗘𝗣𝗔 🌀 Induce 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱 𝗹𝗮𝘁𝗲𝗻𝘁 𝗿𝗲𝗽𝗿���𝘀𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻𝘀 unseen in either base or normally fine-tuned models 🧪 The 𝘃𝟭 𝘄𝗼𝗿𝗸𝘀𝗵𝗼𝗽 𝘃𝗲𝗿𝘀𝗶𝗼𝗻 (accepted to NeurIPS 2025 UniReps + DL4C) received valuable feedback highlighting high compute cost, limited applications, and missing ablations — all fully addressed in this release. Huge thanks to the UniReps and DL4C reviewers for their constructive and insightful comments that helped shape v2. It’s been a privilege to collaborate with Yann LeCun (NYU) and Randall Balestriero (Brown) — few experiences are more inspiring than working alongside the pioneers of modern deep and self-supervised learning. The 𝗰𝗼𝗱𝗲 𝗶𝘀 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲𝗱, and we warmly invite others to experiment with it — and help explore this emerging frontier between 𝗝𝗘𝗣𝗔 𝗮𝗻𝗱 𝗟𝗟𝗠𝘀. 💻 Code: https://lnkd.in/eUX2b8iE 📄 Paper: https://lnkd.in/ers8_yzm Together with Yann and Randall, we’re already exploring new variants and applications — and look forward to sharing more soon. Stay tuned!

A good ending to 2023 is sharing one of my favorite papers from our group (lead by Weicheng Zhu at NYU Center for Data Science) on improving fairness in self supervised learning, at the SSL stage and without relying on any known attributes about the samples or downstream task. This is especially important as we move more and more towards using general purpose SSL-pre-trained foundation models as basis for most tasks. Our focus is on making these foundation models more "fair". Fairness has many definitions. Here, we focus on learning good (discriminative) features for all sub-cohorts during SSL pre-training. It is often the case that variables that describe data have correlations. i.e. Age is correlated with various disease risks. If the correlation is high, e.g. older people have a lot more prevalence of diseases, the SSL pre-trained model may focus only on learning features of old age instead of features of diseases within the old age group. This is called learning spurious correlations and has been observed during SSL. If we knew all these variables and their correlations upfront, we could use conditional SSL, but (1) we often work with truly unlabeled data during SSL, and (2) we don't know good features that describe our data in advance, otherwise we wouldn't need SSL. We have observed that in SSL, for instance contrastive SSL, the samples that are *aligned* with the correlations (i.e. old AND sick) converge faster than those that are in *conflict* with these correlations (i.e. old AND not sick, often in minority). These two samples have measurably different learning speed. We use this insight, and propose a resampling method that dynamically samples training examples with a probability negatively related to their learning speed. This reduces the impact of spurious correlations in the dataset by upsampling the examples that may conflict with spurious correlation. We show evidence that our approach leads to better features in the settings that involve strong spurious correlations. I'm really excited about implications of this work, and extension to various data modalities and settings. Here's our paper: Zhu, W., Liu, S., Fernandez-Granda, C. and Razavian, N., 2023. Making Self-supervised Learning Robust to Spurious Correlation via Learning-speed Aware Sampling. arXiv preprint arXiv:2311.16361. https://lnkd.in/eUPETV6m #deeplearning #selfsupervisedLearning #foundationmodels #fairness #spuriousCorrelations #medicalai #aiinmedicine #happynewyear2024

Learning Accurate Segmentation Purely from Self-Supervision https://lnkd.in/e2PMQRiJ Accurately segmenting objects without any manual annotations remains one of the core challenges in computer vision. In this work, we introduce Selfment, a fully self-supervised framework that segments foreground objects directly from raw images without human labels, pretrained segmentation models, or any post-processing. Selfment first constructs patch-level affinity graphs from self-supervised features and applies NCut to obtain an initial coarse foreground--background separation. We then introduce Iterative Patch Optimization (IPO), a feature-space refinement procedure that progressively enforces spatial coherence and semantic consistency through iterative patch clustering. The refined masks are subsequently used as supervisory signals to train a lightweight segmentation head with contrastive and region-consistency objectives, allowing the model to learn stable and transferable object representations. Despite its simplicity and complete absence of manual supervision, Selfment sets new state-of-the-art (SoTA) results across multiple benchmarks. It achieves substantial improvements on over previous unsupervised saliency detection methods on ECSSD (), HKUIS (), and PASCAL-S (). Moreover, without any additional fine-tuning, Selfment demonstrates remarkable zero-shot generalization to camouflaged object detection tasks (e.g.,  on CHAMELEON and  on CAMO), outperforming all existing unsupervised approaches and even rivaling the SoTA fully supervised methods. --- Newsletter https://lnkd.in/emCkRuA More story https://lnkd.in/eMFcEekQ LinkedIn https://lnkd.in/ehrfPYQ6 #AINewsClips #AI #ML #ArtificialIntelligence #MachineLearning #ComputerVision

Self-supervision continues to revolutionize pre-training strategies in foundation models, offering efficient ways to learn representations without labeled data. This week, our blog-post explores SimSiam, which stands out as a simple yet powerful approach to Representation Learning. What’s covered? ✅Detailed discussion on SimSiam’s architecture and algorithm, which simplifies pretraining by removing the need for Negative Pairs and Momentum Encoders. ✅Thorough Investigation on SimSiam’s Non-Collapsing Dynamics ✅SimSiam's Implicit Optimization Dynamics To make the exploration practical, the blog-post also includes the pretraining code for SimSiam and the code for fine-tuning its learned representations on the downstream classification task. This is a must-read for those eager to understand and implement Self-Supervised Learning! Read the full article here: https://buff.ly/3Wadooe

There are a variety of self-supervised (SSL) approaches to choose from when pretraining a model on unlabeled data. As I’m exploring these techniques for building domain-specific foundation models, my initial focus has been on some of the most commonly-used ones: SimCLR, MoCo, SimSiam, and VicReg. I started by getting each up and running on an easy, smaller dataset (EuroSAT), before branching out. I’m currently working with Patch Camelyon (H&E histopathology of lymph nodes), AML-Cytomorphology LMU (blood cell classification), and Hyper-Kvasir (endoscopy). Results generated so far are in the carousel below, showing that you can train a classifier with significantly fewer labeled examples if your model already knows how to represent images from that domain. These results also show that the SSL approaches perform similarly on some datasets, but differ on others. For Patch Camelyon, MoCoV2 provided a significant improvement over the others. I still have work to do in customizing the image augmentation approach for each domain though. This may change my conclusions, or it may simply be that some families of methods are more suited to a particular imaging domain. Handling labeled data scarcity is just one of the benefits of a domain-specific foundation model. I’ll be highlighting others in future updates – some of which require exploring datasets with more challenging characteristics like distribution shifts or class imbalance. What data challenge would you most like to see overcome with a domain-specific foundation model? #machinelearning #deeplearning #computervision