Locking Pretrained Weights via Deep Low-Rank Residual Distillation

來源原標題

Locking Pretrained Weights via Deep Low-Rank Residual Distillation

原始連結

https://machinelearning.apple.com/research/locking-pretrained-weights

摘要原文

research area Methods and Algorithms content type paper published August 2026 Locking Pretrained Weights via Deep Low-Rank Residual Distillation Authors Keitaro Sakamoto†**, Pierre Ablin, Federico Danieli, Marco Cuturi View publication Copy Bibtex The quality of open-weight language models has dramatically improved in recent years. Sharing weights greatly facilitates model adoption by enabling their use across diverse hardware and software platforms. They also allow for more open research and testing, to the extent that users can use them as checkpoints, fine-tune them according to their needs, and potentially redistribute them. In some cases, however, concerns on modifying these weights towards unauthorized uses may outweigh the pros of giving users such a freedom. Defending against such adaptation is non-trivial: since an adaptive attacker can observe all weights and architectures by definition, they can reverse simple structural defenses, and use optimization to defeat the simplest locking mechanisms. In this work, we exploit the inference–training asymmetry of automatic differentiation as a novel defense axis. We propose DLR-Lock, a method where the purveyor of the model purposely replaces each pretrained MLP in their model with a deep low-rank residual network (DLR-Net) of comparable parameter count, forcing activation memory that grows linearly with depth during backpropagation. DLR-Nets are efficiently trained via module-wise distillation. We show that, beyond this memory overhead, DLR-Lock results in architectural mismatches that complicate the optimization landscape of standard fine-tuning, and a backward pass that incurs disproportionately more overhead than the forward pass. Our defense succeeds in withstanding adaptive attackers with full knowledge of the defense strategy while preserving the original model’s capabilities. Experiments on LLM validate these claims. † University of Tokyo ** Work done while at Apple Related readings and updates. Revealing the Utilized Rank of Subspaces of Learning in Neural Networks July 15, 2024 research area Computer Vision , research area Methods and Algorithms Workshop at ICML This paper has been accepted at the Efficient Systems for Foundation Models workshop at ICML 2024. In this work, we study how well the learned weights of a neural network utilize the space available to them. This notion is related to capacity, but additionally incorporates the interaction of the network architecture with the dataset. Most learned weights appear to be full rank, and are therefore not amenable to low rank decomposition. This… Read more Beyond CAGE: Investigating Generalization of Learned Autonomous Network Defense Policies November 30, 2022 research area Methods and Algorithms conference NeurIPS This paper was accepted at “Reinforcement Learning for Real Life” workshop at NeurIPS 2022. Advancements in reinforcement learning (RL) have inspired new directions in intelligent automation of network defense. However, many of these advancements have either outpaced their application to network security or have not considered the challenges associated with implementing them in the real-world. To understand these problems, this work evaluates… Read more Discover opportunities in Machine Learning. Our research in machine learning breaks new ground every day. Work with us