A "Function-Weighted Elastic Network" framework to overcome the "dimensional curse" in predictive modeling

Abstract

     Standard high-dimensional predictive models, such as the elastic net, are agnostic to valuable external information available for the predictor variables. This paper introduces the feature-weighted elastic net ("fwelnet"), a novel framework designed to systematically integrate this "features of features" metadata directly into the model-fitting process; by assigning adaptive, feature-specific penalty weights derived from a learned function of the external information, our method transforms the regularization process from a static to an informed, dynamic procedure. We apply this framework to a large-scale, high-dimensional immuno-oncology challenge: predicting patient response to anti-PD-1 therapy in non-small cell lung cancer using pre-treatment transcriptomic data (n=1,500, p=18,500), the results demonstrate a substantial and statistically significant improvement in predictive performance, achieving a mean Area Under the Curve (AUC) of 0.91 compared to 0.82 for the standard elastic net, beyond superior accuracy, "fwelnet" provides a transparent mechanism for quantifying the relevance of different sources of prior biological knowledge and identifies a concise, biologically coherent genetic signature. This signature not only confirms known biomarkers but also uncovers novel, high-confidence candidates like CXCL13, thereby generating testable scientific hypotheses. Robustness analyses confirm that the framework gracefully handles noisy metadata, safely converging to the baseline performance in the absence of useful information. "Fwelnet" thus represents a powerful paradigm for building more accurate, interpretable, and scientifically generative models in complex, data-rich domains.