The Future of Discovery in the Age of Human-AI Collaboration

The successful development of transformative applications of AI for science, medicine and energy research will have a profound impact on the world. The rate of development of AI capabilities continues to accelerate, and the scientific community is becoming increasingly agile in using AI, leading us to anticipate significant changes in how science and engineering goals will be pursued in the future. Frontier AI (the leading edge of AI systems) enables small teams to conduct increasingly complex investigations, accelerating some tasks such as generating hypotheses, writing code, or automating entire scientific campaigns. However, certain challenges remain resistant to AI acceleration such as human-to-human communication, large-scale systems integration, and assessing creative contributions. Taken together these developments signify a shift toward more capital-intensive science, as productivity gains from AI will drive resource allocations to groups that can effectively leverage AI into scientific outputs, while other will lag. In addition, with AI becoming the major driver of innovation in high-performance computing, we also expect major shifts in the computing marketplace over the next decade, we see a growing performance gap between systems designed for traditional scientific computing vs those optimized for large-scale AI such as Large Language Models. In part, as a response to these trends, but also in recognition of the role of government supported research to shape the future research landscape of the U. S. Department of Energy has created the FASST (Frontier AI for Science, Security and Technology) initiative. FASST is a decadal research and infrastructure development initiative aimed at accelerating the creation and deployment of frontier AI systems for science, energy research, national security. I will review the goals of FASST and how we imagine it transforming the research at the national laboratories. Along with FASST, I’ll discuss the goals of the recently established Trillion Parameter Consortium (TPC), whose aim is to foster a community wide effort to accelerate the creation of large-scale generative AI for science. Additionally, I’ll introduce the AuroraGPT project an international collaboration to build a series of multilingual multimodal foundation models for science, that are pretrained on deep domain knowledge to enable them to play key roles in future scientific enterprises.

Rick Stevens is Argonne’s Associate Laboratory Director for the Computing, Environment and Life Sciences (CELS) Directorate and an Argonne Distinguished Fellow. He is also a Professor of Computer Science at the University of Chicago. Stevens is responsible for ongoing research in the computational and computer sciences from high performance computing architecture to the development of tools and methods for bioinformatics, cancer, infectious disease, and other challenges in science and engineering. He has significant responsibility in delivering on the national initiative for exascale computing and developing the U.S. Department of Energy’s (DOE) AI for Science national initiative. Recently, he has focused on developing artificial intelligence methods for a variety of scientific and biomedical challenges.

The field of AI is advancing at unprecedented speed in the past few years,  due to the rise of large-scale, self-supervised pre-trained models (a.k.a. “foundation  models”).  Impressive  few-shot generalization capabilities of such models on a wide range of novel tasks appear to emerge primarily due to scaling the number of model’s parameters, training data and compute resources. Empirical neural scaling laws aim at predicting scaling behaviors of foundation models, and serve as AI  “investment tools” that can  suggest an optimal allocation of compute resources, and select best-scaling models and algorithms, likely to stand the  test-of-time as  larger compute resources become available. Moreover, accurate prediction of emerging behaviors in large-scale AI systems is essential from AI Safety perspective.  

In this talk, I will present a brief history of neural scaling laws, with the focus on our recent unifying framework of Broken Neural Scaling Laws that generalizes the prior power laws to a wide range of nonlinear, nonmonotonic scaling behaviors observed for various behavioral metrics (capabilities and alignment), on multiple data modalities, architectures, settings. Besides this standard  “closed-box” approach to neural scaling, we also look inside the model’s training dynamics (an “open-box” approach) to discover scaling patterns predictive of certain “emergent” behaviors such as “grokking”. 

Finally, this talk will provide an overview of open-source foundation models our lab has built and released so far using large-scale INCITE allocation on Summit and Frontier supercomputers, including several 9.6B LLMs trained continually, the first Hindi model Hi-NOLIN,  multimodal vision-text model suite Robin, as well as time-series foundation models. I will highlight the continual pre training paradigm that allows to train models on potentially infinite datasets, as well as approaches to AI ethics and multimodal alignment. See our CERC-AAI project page for more details: https://www.irina-lab.ai/projects.

Irina Rish is a Full Professor in the Computer Science and Operations Research department at the Université de Montréal (UdeM) and a core member of Mila – Quebec AI Institute. She holds the Canada CIFAR AI Chair and the Canadian Excellence Research Chair in Autonomous AI. She holds MSc and PhD in AI from University of California, Irvine and MSc in Applied Mathematics from Moscow Gubkin Institute. Dr. Rish’s research focus is on machine learning, neural data analysis and neuroscience-inspired AI.

Her current research interests include continual lifelong learning, optimization algorithms for deep neural networks, sparse modelling and probabilistic inference, dialog generation, biologically plausible reinforcement learning, and dynamical systems approaches to brain imaging analysis. Before joining UdeM and Mila in 2019, she was a research scientist at the IBM T. J. Watson Research Center, where she worked on various projects at the intersection of neuroscience and AI, and led the Neuro-AI challenge. Dr. Rish holds 64 patents, has published over 80 research papers, several book chapters, three edited books, and a monograph on Sparse Modelling.

Many scientific disciplines, including biology, chemistry and materials science, face challenges from the presence of complex systems and the absence of sufficient theory to enable prediction of system behaviors.  This led first to application of “passive” machine learning approaches to learn predictive models from existing experimental data, and then to the growth of “active” machine learning approaches that seek to efficiently improve such models by iterative selection of new experiments to perform.  The combination of these approaches with laboratory automation to robotically execute chosen experiments is variously termed “automated science” or “self-driving labs”.  The active approaches have largely been restricted to one or two independent experimental variables, and their goals are usually either maximizing model accuracy or optimizing particular properties.  For biomedical research, they thus do not provide models that can encompass the dozens to thousands of potential independent variables (e.g., proteins, genes, metabolites) that are involved in even relatively small biological systems.  Furthermore, with some exceptions, they do not consider which approach to use to acquire new data (e.g., computational simulations vs. laboratory measurements).  I will discuss the premise that the next wave of approaches for automated science will need to simultaneously model large numbers of independent variables and be able to assess the relative value of making different types of measurements that may provide information on different subsets of the system with different levels of accuracy.

Liza Levina is the Vijay Nair Collegiate Professor and Chair of Statistics at the University of Michigan, and affiliated faculty at the Michigan Institute for Data Science and the Center for the Study of Complex Systems. She received her PhD in Statistics from UC Berkeley in 2002, and has been at the University of Michigan since. Her research interests include statistical inference in high dimensions, statistical network analysis, and applications to imaging, especially in neuroscience. She is a Fellow of the American Statistical Association and the Institute of Mathematical Statistics, and a Web of Science Highly Cited Researcher.

The increasing availability of large-scale datasets that trace the entirety of the scientific enterprise, have created an unprecedented opportunity to explore scientific production and reward. Parallel developments in data science, network science, and artificial intelligence offer us powerful tools and techniques to make sense of these millions of data points. Together, they tell a complex yet insightful story about how scientific careers unfold, how collaborations contribute to discovery, and how scientific progress emerges through a combination of multiple interconnected factors. These opportunities—and challenges that come with them—have fueled the emergence of a multidisciplinary community of scientists that are united by their goals of understanding science and innovation. These practitioners of the science of science use the scientific methods to study themselves, examine projects that work as well as those that fail, quantify the patterns that characterize discovery and invention, and offer lessons to improve science as a whole. In this talk, I’ll highlight some examples of research in this area, hoping to illustrate the promise of science of science as well as its limitations.

Ido is a Professor at the Technion. In his Ph.D. research, Ido discovered new classes of accelerating beams in nonlinear optics and electromagnetism. He was the first Israeli ever to win an APS award for his Ph.D. thesis. As a postdoc at MIT, he established the foundations of macroscopic quantum electrodynamics (MQED) for photonic quasiparticles and used it to find a way to enable forbidden electronic transitions in atoms. As a faculty member, Ido created a paradigm shift in the understanding of free-electron radiation, connecting it to the field of quantum optics. He performed the first experiment on free-electron interaction with quantum light, demonstrating that the quantum statistics of photons can be imprinted on the electron. He studies the frontiers of photonics, quantum optics, and laser-driven electron acceleration, by developing novel theoretical and experimental methods. He is head of the AdQuanta Lab at the Technion, whose research established the foundations of quantum electrodynamics with photonic quasiparticles.

Dr. Seifert’s research interests lie in complex human cognition, including memory, learning, problem solving, and creativity. With her collaborators, she is currently investigating cognitive accounts of creative thinking, and ways to improve the creative process. She is invested in improving learning and pedagogy in design, higher education, and applications in the public interest..

Senior Director for Strategic Artificial Intelligence and Data Science Initiatives at the Information Sciences Institute of the University of Southern California, and Research Professor in Computer Science and in Spatial Sciences. She is also Director of Data Science programs and of the USC Center for Knowledge-Powered Interdisciplinary Data Science. She received her M.S. and Ph. D. degrees in Computer Science from Carnegie Mellon University, with a focus on artificial intelligence. Her research is on intelligent interfaces for knowledge capture and discovery, which she investigates in a variety of projects concerning scientific discovery, knowledge-based planning and problem solving, information analysis and assessment of trust, semantic annotation and metadata, and community-wide development of knowledge bases.

The scientific method has been transformative to humankind. However, there are signs that despite great investment in the area, scientific discovery is approaching a state of stagnation. In the context of scientific discovery, a fundamental problem is to explain natural phenomena in a manner consistent with both (noisy) experimental data, and a body of (possibly inexact and incomplete) background knowledge about the laws of the universe. Historically, models were manually derived in a first-principles deductive fashion. The first-principles approach often offers the derivation of interpretable symbolic models of remarkable levels of universality while being substantiated by little data. Nonetheless, derivation of such models is time-consuming and relies heavily upon domain expertise. Conversely, with the rising pervasiveness of statistical AI and data-driven approaches, automated, rapid construction and deployment of models has become a reality. Many data-driven modeling techniques demonstrate remarkable scalability due to their reliance upon predetermined, exploitable model form (functional form) structures. Such structures, entail non-interpretable models, demand Big Data for training, and provide limited predictive power for out-of-set instances. In this lecture, we will learn about the two breeds of discovery frameworks, explore recent attempts to bridge the divide between them, as to enable (with some optimism) a unified discovery of fundamental laws of nature at scale.

Biological brains demonstrate remarkable capabilities in model building and event prediction using sparse, noisy data and showcasing the intricate complexity of cognitive processes. These processes, powered by intricate networks of neurons, glia, and vascular systems, underscore the potential for developing artificial general intelligence (AGI). We establish the theoretical underpinnings of biological intelligence to steer future AGI advancements, thereby broadening the horizons of human discovery. We delve into a mathematical framework designed to decipher neuronal activity’s emerging causal fractal memory, providing insights into neuronal network topologies, and predicting animal behavior in cognitive tasks. Furthermore, we present a multiwavelet-based neural operator used for deducing and forecasting the dynamics of coupled governing laws from sparse, noisy data. This operator demonstrates superior accuracy over existing machine learning tools. Finally, drawing upon multifractal formalism’s capability to identify phase transitions in biological networks, we advance the understanding self-organization within Large Language Models (LLMs). We quantitatively analyze how self-organization underpins the emergence of sophisticated intelligence patterns in LLMs and offer a glimpse into the complex interplay of neuron interactions and their pivotal role in cognitive emergence.

Paul Bogdan received his Ph.D. degree in Electrical and Computer Engineering from Carnegie Mellon University, Pittsburgh. He is an assistant professor in the Ming Hsieh Department of Electrical Engineering at University of Southern California. His work has been recognized with a number of distinctions, including the 2012 A.G. Jordan Award from the Electrical and Computer Engineering Department, Carnegie Mellon University for outstanding Ph.D. thesis and service, the 2012 Best Paper Award from the Networks-on-Chip Symposium (NOCS), the 2012 D.O. Pederson Best Paper Award from IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, the 2012 Best Paper Award from the International Conference on Hardware/Software Codesign and System Synthesis (CODES+ISSS), the 2013 Best Paper Award from the 18th Asia and South Pacific Design Automation Conference, and the 2009 Roberto Rocca Ph.D. Fellowship. His research interests include performance analysis and design methodologies for multicore systems, the theoretical foundations of cyber-physical systems, the modeling and analysis of bio-inspired computing, and the applications of statistical physics to biological systems and regenerative medicine.

What does flourishing look like for scientists, science communities, and society? Certainly it involves pushing the frontiers of scientific discovery, frontiers that now exist at the intersection between disciplines and that require new approaches to science and collaboration. These approaches are emerging from data science, open science, and social science, while we as scientists, engineers, and society are in the midst of understanding how they relate to one another.

 I am a senior Data Scientist at the NASA Jet Propulsion Laboratory, where I work with the Machine Learning and Instrument Autonomy (MLIA) group to apply data science techniques robustly and responsibly to the Earth and Space Sciences, to cultivate cross-NASA Center collaborations, and to explore more cohesive and plural scientific communities. I am also a core team member for the NASA Transformation to Open Science (TOPS) initiative, improving the accessibility, inclusivity, and reproducibility of science.