Transfer Learning Approaches

In the last blog I talked about the importance of Classical ML/DL.  This post focuses on Finetuning image models. (Part 4 of #ArchitectingAI) Pre-trained image models are powerful. Fine-tuning them correctly is the real skill. Transfer learning lets you start with a backbone — ResNet, MobileNet, EfficientNet — already trained on millions of images, and adapt it to your problem. Less data, faster training, better results. I applied this to classify surface defects in industrial steel project. It works well and rewards a meticulous approach. Each wrong decision compounds! 5 key things that actually matter: 1. 𝐓𝐡𝐞 𝐟𝐫𝐨𝐳𝐞𝐧 𝐥𝐚𝐲𝐞𝐫𝐬 𝐚𝐫𝐞 𝐚𝐥𝐫𝐞𝐚𝐝𝐲 𝐝𝐨𝐢𝐧𝐠 𝐦𝐨𝐬𝐭 𝐨𝐟 𝐭𝐡𝐞 𝐰𝐨𝐫𝐤. 𝑭𝒊𝒏𝒆-𝒕𝒖𝒏𝒊𝒏𝒈 𝒊𝒔 𝒓𝒆𝒇𝒊𝒏𝒆𝒎𝒆𝒏𝒕, 𝒏𝒐𝒕 𝒓𝒆𝒍𝒆𝒂𝒓𝒏𝒊𝒏𝒈. Early layers in a pre-trained model capture universal patterns — edges, textures, shapes — that transfer across domains. Later layers are task-specific. Freezing the backbone and training only a new classification head gets you most of the way there. Unfreezing the whole network is rarely worth it. 2. 𝐃𝐨𝐦𝐚𝐢𝐧 𝐠𝐚𝐩 𝐝𝐞𝐭𝐞𝐫𝐦𝐢𝐧𝐞𝐬 𝐲𝐨𝐮𝐫 𝐟𝐢𝐧𝐞-𝐭𝐮𝐧𝐢𝐧𝐠 𝐬𝐭𝐫𝐚𝐭𝐞𝐠𝐲. 𝑻𝒉𝒆 𝒇𝒖𝒓𝒕𝒉𝒆𝒓 𝒚𝒐𝒖𝒓 𝒅𝒂𝒕𝒂 𝒇𝒓𝒐𝒎 𝑰𝒎𝒂𝒈𝒆𝑵𝒆𝒕, 𝒕𝒉𝒆 𝒎𝒐𝒓𝒆 𝒚𝒐𝒖 𝒏𝒆𝒆𝒅 𝒕𝒐 𝒖𝒏𝒇𝒓𝒆𝒆𝒛𝒆. Natural images transfer easily. Industrial textures, medical scans, satellite imagery — larger domain gap, less directly applicable features. Know your gap before deciding how many layers to unfreeze. More gap = more fine-tuning needed. 3. 𝐃𝐚𝐭𝐚 𝐚𝐮𝐠𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 𝐢𝐬 𝐧𝐨𝐭 𝐨𝐩𝐭𝐢𝐨𝐧𝐚𝐥 𝐰𝐢𝐭𝐡 𝐬𝐦𝐚𝐥𝐥 𝐝𝐚𝐭𝐚𝐬𝐞𝐭𝐬. 𝑻𝒉𝒆 𝒎𝒐𝒅𝒆𝒍 𝒘𝒊𝒍𝒍 𝒎𝒆𝒎𝒐𝒓𝒊𝒔𝒆 𝒚𝒐𝒖𝒓 𝒅𝒂𝒕𝒂 𝒊𝒇 𝒚𝒐𝒖 𝒈𝒊��𝒆 𝒊𝒕 𝒕𝒉𝒆 𝒄𝒉𝒂𝒏𝒄𝒆. Augmentation creates diversity the model hasn't seen — rotations, skews, flips, contrast shifts, brightness changes, blur. It forces generalisation over memorisation, critical when domain-specific data is limited. Apply it before model-specific preprocessing — wrong order means augmenting corrupted inputs, silently. 4. 𝐅𝐫𝐞𝐞𝐳𝐞 𝐟𝐢𝐫𝐬𝐭. 𝐅𝐢𝐧𝐞-𝐭𝐮𝐧𝐞 𝐬𝐞𝐜𝐨𝐧𝐝. 𝑨𝒍𝒘𝒂𝒚𝒔 𝒊𝒏 𝒕𝒉𝒂𝒕 𝒐𝒓𝒅𝒆𝒓. A randomly initialised head will undo what you borrowed. New classifier weights start random. Early gradients are large and noisy — if the backbone is already unfrozen, they overwrite representations learned from millions of images. Train the head first, stabilise it, then unfreeze selectively. 5. 𝐓𝐫𝐞𝐚𝐭 𝐩𝐫𝐞-𝐭𝐫𝐚𝐢𝐧𝐞𝐝 𝐰𝐞𝐢𝐠𝐡𝐭𝐬 𝐚𝐬 𝐟𝐫𝐚𝐠𝐢𝐥𝐞 𝐝𝐮𝐫𝐢𝐧𝐠 𝐟𝐢𝐧𝐞-𝐭𝐮𝐧𝐢𝐧𝐠. 𝑼𝒔𝒆 𝒂 𝒎𝒖𝒄𝒉 𝒔𝒎𝒂𝒍𝒍𝒆𝒓 𝒍𝒆𝒂𝒓𝒏𝒊𝒏𝒈 𝒓𝒂𝒕𝒆. A standard learning rate undoes the representations you were trying to preserve. Unfreeze only the last few layers — more layers at a higher learning rate tend to overfit or underfit fast. Transfer learning is powerful because it builds on what's already been learned. Know your domain gap. Know what to freeze. Know when to fine-tune. Do it with care.

Machine learning models trained on geospatial data from one region can easily fail when applied to another. And getting sufficient annotated training data from every region that you want to apply your model to frequently isn't possible. Hao Li et al. proposed an unsupervised method for adapting a model to a new region. They first trained a model on a region where they had plenty of labeled training data. Much smaller datasets are available for other regions, so they used few-shot transfer learning to adapt their model to each new region. To apply this set of models to a new region, they used an ensembling approach and experimented with different weighting methods. When they applied their approach to detecting OpenStreetMap missing buildings across different countries in sub-Saharan Africa, the best weighting turned out to be using self-attention. This approach could provide value to other geospatial applications, as well as medical applications with technical and biological variations. https://lnkd.in/e4skmTCA #RemoteSensing #EarthObservation #MachineLearning #DeepLearning #ComputerVision __________________ Enjoyed this post? Like 👍, comment 💬, or re-post 🔄 to share with others. Click "View my newsletter" under my name ⬆️ to join 1500+ readers.

New peer reviewed paper: how to scale healthcare risk prediction models when they're typically trained on one population and applied to another? We compared nine leading transfer-learning strategies, including source-only and target-only logistic regression, prototypical networks, domain-adversarial neural networks, causal transfer learning, TabTransformer, Enhanced MAML, meta-ensembles, and averaged ensembles. We found that a well-calibrated, source-trained logistic model performed on par with a sophisticated meta-ensemble and outperformed locally trained models built on scarce data. At the same time, unadapted models amplified racial and ethnic disparities in prediction quality, and even limited target-domain data materially improved equalized odds of fairness. https://lnkd.in/gSh3tK6D

Transfer Learning vs. Fine-tuning vs. Multitask Learning vs. Federated Learning, explained visually👇 . . Most ML models are trained independently without any interaction with other models. But real-world ML uses many powerful learning techniques that rely on model interactions. The following animation summarizes four such well-adopted and must-know training methodologies: 1) Transfer learning Useful when: - The task of interest has less data. - But a related task has abundant data. This is how it works: - Train a neural network model (base model) on the related task. - Replace the last few layers on the base model with new layers. - Train the network on the task of interest, but don’t update the weights of the unreplaced layers. Training on a related task first allows the model to capture the core patterns of the task of interest. Next, it can adapt the last few layers to capture task-specific behavior. 2) Fine-tuning Update the weights of some or all layers of the pre-trained model to adapt it to the new task. The idea may appear similar to transfer learning. But here, the whole pretrained model is typically adjusted to the new data. 3) Multi-task learning (MTL) A model is trained to perform multiple related tasks simultaneously. Architecture-wise, the model has: - A shared network - And task-specific branches The rationale is to share knowledge across tasks to improve generalization. In fact, we can also save computing power with MTL: - Imagine training 2 independent models on related tasks. - Now compare it to having a network with shared layers and then task-specific branches. Option 2 will typically result in: - Better generalization across all tasks. - Less memory to store model weights. - Less resource usage during training. 4) Federated learning This is a decentralized approach to ML. Here, the training data remains on the user's device. So in a way, it's like sending the model to data instead of data to model. To preserve privacy, only model updates are gathered from devices and sent to the server. The keyboard of our smartphone is a great example of this. It uses FL to learn typing patterns. This happens without transmitting sensitive keystrokes to a central server. Note: Here, the model is trained on small devices. Thus, it MUST be lightweight yet useful. Model compression techniques are prevalent in such cases. I have linked a detailed guide in the comments. -- If you want to learn AI/ML engineering, I have put together a free PDF (530+ pages) with 150+ core DS/ML lessons. Get here: https://lnkd.in/gi6xKmDc -- 👉 Over to you: What are some other ML training methodologies that I have missed here?

🌟 First pre-print of 2025 🌟 Active and transfer learning with partially Bayesian neural networks for materials and chemicals ✅ We show that partially Bayesian neural networks (PBNNs), where only selected layers have probabilistic weights, can achieve performance on active learning tasks comparable to that of fully Bayesian networks at reduced computational cost. ✅ We investigated the role of probabilistic layer selection, finding somewhat counterintuitively that making earlier layers probabilistic leads to better uncertainty estimates for active learning. ✅ We further enhanced PBNN performance through transfer learning by using weights pre-trained on theoretical calculations to initialize the probabilistic layers' prior distributions, which was particularly effective in the early stages of active learning. ✅ We validated the approach on multiple real-world tasks including predicting molecular properties (hydration energy, solubility) and materials properties (fatigue strength, electrical conductivity, bandgaps), demonstrating PBNNs can be a valuable tool to aid scientific discovery with limited, complex datasets. ✅ We provide an easy to use open-source code for anyone wanting to experiment with PBNNs. See https://lnkd.in/e-t3vz9Y ▶️ While our current work focuses on simple architectures, future directions include extending PBNNs to Graph Neural Networks and Transformers while exploring connections with neural network interpretability. ⏩ We're also developing PBNN applications for autonomous experiments - stay tuned for updates! Preprint: https://lnkd.in/eWYhQTp7

You do not need massive datasets to build powerful models. This week I broke down transfer learning step by step. What it is. How it works. The mistakes to avoid. Today I am going deeper. In this newsletter: → Why transfer learning matters → The 3 core concepts explained simply → Pretrained models, feature extraction, and fine-tuning → A decision framework for when to use which → 5 common mistakes and how to fix them → A simple workflow you can follow Transfer learning is one of the most practical techniques in deep learning. This newsletter gives you everything you need to start using it. Take your time to read it. And let me know what you learnt. I would love to hear Richel makes tech easy 💕

🚀 New paper: “Accelerating Process Synthesis with Reinforcement Learning: Transfer Learning Approach” I'm excited to share our latest open-access article in Computers & Chemical Engineering that explores Reinforcement Learning and Representation Learning for process synthesis. Thanks to Qinghe Gao, Haoyu Yang, and Maximilian Theisen for their great work. 🔍 What we did We developed a reinforcement learning (RL) framework for process synthesis and enhanced its performance using transfer learning. The model has been pre-trained as a Variational Autoencoder (VAE) on existing process flowsheet graphs. This enables models trained on existing processes to effectively jumpstart learning on a different, but related, task. The goal was to leverage existing process flowsheet information but also reduce training time. ⚙️ Why it matters Leveraging existing knowledge/information is increasingly important. There is a lot of knowledge hidden in existing process documentation (e.g., flowsheets). The core research question is: how can we extract information from existing process flowsheets and use it for optimal decision making. AI technologies like VAEs, LLMs, and RL can be part of the solution. We made a first step towards answering this research question. 🧪 Key results Our experiments show that transfer-augmented RL agents converge quicker and require fewer simulation steps compared to those trained from zero. However, this is just a first "baby step" in the right direction. There are plenty of open research questions that need to be answered and there is still a long way until industrial use. We need better algorithms, more data and hybrid solutions that combine AI with engineering knowledge. 🌟 Broader impact The underlying research question has also a large potential impact on many other fields within engineering and science. There is a lot of un-used knowledge hidden in documents. Tailored AI can help to leverage this for future innovations. ✅ Read the full open-access article here: https://lnkd.in/emW4Bjw6 #ProcessSystemsEngineering #ReinforcementLearning #TransferLearning #AIinEngineering #ChemicalEngineering #ProcessDesign #ResearchInnovation