The Future of Discovery in the Age of Human-AI Collaboration

Many scientific disciplines, including biology, chemistry and materials science, face challenges from the presence of complex systems and the absence of sufficient theory to enable prediction of system behaviors. This led first to application of “passive” machine learning approaches to learn predictive models from existing experimental data, and then to the growth of “active” machine learning approaches that seek to efficiently improve such models by iterative selection of new experiments to perform. The combination of these approaches with laboratory automation to robotically execute chosen experiments is variously termed “automated science” or “self-driving labs”. The active approaches have largely been restricted to one or two independent experimental variables, and their goals are usually either maximizing model accuracy or optimizing particular properties. For biomedical research, they thus do not provide models that can encompass the dozens to thousands of potential independent variables (e.g., proteins, genes, metabolites) that are involved in even relatively small biological systems. Furthermore, with some exceptions, they do not consider which approach to use to acquire new data (e.g., computational simulations vs. laboratory measurements). I will discuss the premise that the next wave of approaches for automated science will need to simultaneously model large numbers of independent variables and be able to assess the relative value of making different types of measurements that may provide information on different subsets of the system with different levels of accuracy.

In this talk, I will present a brief history of neural scaling laws, with the focus on our recent unifying framework of Broken Neural Scaling Laws that generalizes the prior power laws to a wide range of nonlinear, nonmonotonic scaling behaviors observed for various behavioral metrics (capabilities and alignment), on multiple data modalities, architectures, settings. Besides this standard  “closed-box” approach to neural scaling, we also look inside the model’s training dynamics (an “open-box” approach) to discover scaling patterns predictive of certain “emergent” behaviors such as “grokking”. 

Finally, this talk will provide an overview of open-source foundation models our lab has built and released so far using large-scale INCITE allocation on Summit and Frontier supercomputers, including several 9.6B LLMs trained continually, the first Hindi model Hi-NOLIN,  multimodal vision-text model suite Robin, as well as time-series foundation models. I will highlight the continual pre training paradigm that allows to train models on potentially infinite datasets, as well as approaches to AI ethics and multimodal alignment. See our CERC-AAI project page for more details: https://www.irina-lab.ai/projects.