Active Projects on Neocortex

Project Abstract:

Recently, we have developed the Network Parameter Outlier (NPO) algorithm, that uses graphical clustering methods to cluster biological event data (e.g., action potentials, sharp-wave ripples) in O(n) time, improving on current methods that require O(nlog n) time. Biologically-relevant events are detected and stored as a graph with connections weighted by corresponding correlation coefficients. A graphical clustering technique (Louvain) partitions the graph into clusters. This process is repeated in a moving-window fashion with new vertices being added and old vertices being dropped. Cluster labels are passed to succeeding windows, generating a consistent clustering across an arbitrarily large data set. Because the windowed-data graph size remains constant, processing time is bounded, which allows the total clustering time to increase linearly with increasing data size. The optimal parameters regarding both computational time and overall performance, however, are unknown due to memory limitations of conventional CPUs. We propose to use the large-memory model of Neocortex to allow graph sizes that are much large than can be supported by current computing hardware. The algorithm has been wrapped in a Singularity container linked to a MySQL database (also running in a Singularity container) to allow portable computation.

Project Abstract:

Many critical applications, such as data mining, social network analytics, cybersecurity, and healthcare, generate massive amounts of multidimensional data as sparse tensors that can be analyzed quickly and efficiently using tensor decomposition (TD). TD algorithms for high-dimensional sparse data are challenging to execute on emerging parallel architectures due to their low arithmetic intensity, irregular memory access, workload imbalance, and synchronization overhead. In TD algorithms, each non-zero element is associated with N indices in an N-dimensional space, which are used to retrieve rows from source matrices that are used to calculate an update for a destination matrix row. This type of processing is potentially well-suited for data flow architectures, where each non-zero element is passed through a series of data accesses and floating-point operations, and the final result associated with the non-zero is accumulated in a serialized/atomic manner. This is particularly true for streaming data, which flows in continuously over time. However, due to the lack of commercially available data flow processors, only a few studies have been done on FGPAs with limited performance improvements. The CS-2 system offers large compute performance, coupled with extremely high memory and interconnect bandwidth, allowing for our algorithm to achieve unprecedented level of performance, while simultaneously offering interesting performance challenges associated with its unique data flow architecture. We propose to develop and implement novel parallel algorithms for various TD algorithms and use it to analyze both static and streaming data.

Project Abstract:

Maps of our Universe help us study how the contents of the universe evolved with time, how galaxies formed and developed, and pinpoint the location of short-lived astrophysical events like supernovae or sources of gravitational waves. To make such maps, astronomers measure a quantity called redshift which is a proxy for galaxy distance. However, high precision measurements of redshifts can only be done for a tiny fraction of galaxies for which we have images. Because of this limitation, it is necessary to infer redshift information from imaging data alone; the resulting measurements are called photometric redshifts. Precise estimates of photometric redshifts, along with associated uncertainties, are key to astronomical research for the coming decade since a majority of the data sets that will be available to the astronomy community will come from large-scale imaging only astronomical surveys and depend heavily on photometric redshifts to measure galaxy distances. We propose to use one such data set of over 32 million galaxy images called the DESI Legacy Imaging Surveys to pre-train a vision transformer-based self-supervised model which will learn a latent representation of the galaxy images. We then plan to fine-tune the model using a small number of known galaxy redshifts from the Dark Energy Spectroscopic Instrument (DESI) to finally infer the redshifts to the galaxy for which we only have images. This map of our Universe will enable the myriad of scientific cases previously mentioned.

Project Abstract:

Automated translation of data to textual narratives is commonly referred to in literature as data-to-text generation. Here, data refers to an entity that is not exclusively linguistic (eg: graphs, tables, knowledge bases). While most existing data-to-text approaches deal with tabular data and graphs, few approaches have focused on textual summaries of time-series data (Time-Series Captioning, or Time-Series Summarization). This has largely been due to the lack of paired time-series and text data. Biosignal time-series summarization has been studied in EEG-to-text and ECG-to-text. Prior work advocates for a human-in-the-loop system to generate the text summaries, stating that automated reports are too erroneous. While we note the merit in human-guided text generation, our proposed method requires far less effort from the associated medical professional. Another overarching limitation observed in all data-to-text generation methods is their evaluation metrics. Almost all methods report standard text generation metrics like BLEU and BERTScore that are simply inadequate in evaluating correctness in domains like healthcare. While many methods perform expert (human) evaluation, we note that there are no standard guidelines on how to evaluate medical text reports. Our proposed method addresses all these limitations.

Project Abstract:

Project Abstract:

The SPIRAL system builds on 20 years of research by PIs Franchetti, Moura, Hoe, and Low, and 9 years of commercial R&D in the SPIRAL effort. SPIRAL has demonstrated across a wide range of hardware architectures that it is able to produce software that outperforms the best human programmers, and was designed to automatically deliver the performance of the best hand-tuned code on a new target platform on day one. SPIRAL has successfully targeted single core/multicore/manycore CPUs, GPUs, DSPs, the Cell BE, Xeon PHI, FPGAs, clusters, up to 128k cores on BlueGene/L/P/Q, the 20 K computer, and in pre-silicon settings (IBM Cell BE and BG/Q, and Intel AVX and Xeon PHI). Work in DARPA BRASS is enabling SPIRAL to run as a just-in-time compiler, and we are extending SPIRAL to support CNN/DNN kernels, graph and sparse matrix algorithms in a GraphBLAS related effort. In DARPA PERFECT we used SPIRAL to program and configure the HAMLeT memory side data reorganization unit we developed. Our work on MEALib in PERFECT demonstrates how standard C++ source code can be interpreted as an embedded DSL program, compiled with SPIRAL, and run on advanced memory-side accelerators without any change to the source code, leading to 150x performance gain and 8000x power efficiency gain. SPIRAL is also used as code generation backend in the DOE ExaScale effort FFTX and SpectralPack and the DARPA DPRIVE program to target homomorphic encryption. In this effort we plan to target the WSE system with SPIRAL to explore how to make our code generation technology compatible with the WSE sysytem.

Project Abstract:

P300 is a histone acetyltransferase that acts on a wide-range of proteins in the cell. Unlike its homolog CREBBP (CBP), aberrant p300 activity is often implicated as critical in driving dysregulation and disease. In the case of Alveolar Rhabdomyosarcoma (aRMS), recruitment of p300 by the fusion protein PAX3-FOXO1 (P3F) is one of the key causes of the transcriptional activation of core regulatory transcription factors driving disease progression. Prior work has shown the limited utility of catalytic inhibitors in slowing disease progression, as well as the strong impacts of p300 degradation in some diseases. However, very little previous work has explored the prospect of targeting the specific protein-protein interaction implicated in driving aRMS. This proposed work will use deep learning enhanced docking techniques to probe the interaction space between p300 and P3F, combined with recent advances in machine-learning based small molecule discovery and refinement to provide a library of promising molecules to disrupt this key interaction. The goals of this work are not only to identify a set of drugs which could be synthesized as first-in-class p300-P3F covalent disrupters, but also to build a cutting-edge pipeline for model-based drug discovery based on structural proteomics and novel chemoinformatics.

Project Abstract:

A major goal of the international math community is to obtain tools for the automated processing and transformation of mathematical documents. These tools do not currently exist in a satisfactory form and extensive research is oriented towards improving on this. The Formal Abstracts Project aims to provide mathematicians with software tools for stating their research results in a human/machine readable format amenable to formal verification. In order to achieve this goal, the Formal Abstracts Project has recognized the need for (1) a comprehensive vocabulary of mathematics, in order to state research results, and for (2) improved automated reasoning tools to aid in processing and formally verifying those statements. Using the startup allocation #TG-DMS190028, we have (1) applied techniques from Natural Language Processing (NLP) to produce a dataset of labeled definitions extracted from the entire arXiv mathematics corpus, and (2) applied deep learning to accelerate SAT solvers, a type of automated reasoning tool. We consider these successes to be promising first steps towards our ultimate goal. We propose a strategy to continue work in this vein, including a machine learning methodology that uses well-established techniques in NLP. This methodology consists of a detailed strategy to obtain and process the relevant data, and has been thoroughly peer-reviewed by the Mathematical Knowledge Management (MKM) community. Our group has received a major grant from the Alfred P. Sloan Foundation (G-2018-10067) to develop software and services for transforming mathematical results into formally structured data that machines can read, process, search, check, compute with, and learn from as logical statements. This puts our group in an ideal position to create this much needed resource.

Project Abstract:

A key upcoming challenge in preserving the Earth’s biosphere will be the continuous monitoring of a variety of animal and plant species present in given ecosystems. Artificial Intelligence (AI) can play an important role for solving this challenge, and very recent advances in vision-language models (VLMs) have the promise to build powerful human-interpretable AI models that can be deployed on a wide array of datasets. However, unlike standard deep neural networks, these models are extremely challenging to train on workstations or even small clusters. This Neocortex project will serve two purposes. First, it will help my lab build the first open-source VLMs for image classification and object detection for ecological monitoring. These will be trained on large-scale image datasets (such as iNaturalist) and will specifically be fine-tuned for robust species tracking and monitoring. Second, all software developed within this project will be open-sourced and will serve as stepping stones in the intersection of machine learning and computational sustainability.

Project Abstract:

 Viral pathogens target ‘receptor’ proteins on the surface of host cells to initiate infection. Recognition of the receptor is coordinated by a viral surface protein, and the strength of binding, determined by the biochemical properties of the amino acid sequences of the viral and host proteins, often dictates the course of disease. Because of the impact on viral fitness during immune host responses caused by viral infection, there is constant evolutionary selective pressure asserted on these viral surface proteins, where mutations that provide a fitness advantage to the virus will outcompete and spread more rapidly than other variants. Because of their accessibility, viral surface proteins are also common targets for vaccines and therapeutic antibodies. Mutations in viral surface proteins can perturb antibody binding and lead to increased infection or even escape from vaccine and therapeutic regimes. Evidence points to this being the case with some of the more recent lineages of SARS-CoV-2 that continues to spread within the US. We have developed a machine learning (ML) approach to study deep mutational data and resulting phenotypes of receptor binding domain variants of the SARS-CoV-2 Spike protein. This ML model can accurately predict the expression level of viral proteins, including those that display combinatorial mutations, as well as predict their affinity to the human ACE2 receptor. We plan to further develop this model and test several alternative model architectures using natural language processing and graph models to increase the explainability of the model. The ultimate goal of this work is to develop explainable AI methodology for studying viral evolutions and associated biothreats.

Project Abstract:

Protein engineering has become a crucial research area in the fields of chemistry and biology, with the primary objective being to design proteins with desired properties. One of the significant challenges in protein engineering is designing novel protein sequences with improved properties, such as increased structural stability or enzyme activity. Recently, large language model~(LLM) becomes more and more popular and it achieves state-of-the-art performance on many natural language processing tasks. Regarding its strong ability to model discrete text sequence, it is straightforward to think about if it is possible to directly adapt LLM to protein sequence design? In this project, we aim to study this problem by guiding GPT-3 language model to design novel protein sequences with improved properties.

Project Abstract:

Neural text detectors aim to decide the characteristics that distinguish neural (machine-generated) from human texts. To challenge such detectors, adversarial attacks can alter the statistical characteristics of the generated text making the detection task more and more difficult. Inspired by the advances of mutation analysis in software development and testing, in this paper, we propose character- and word-based mutation operators for generating adversarial samples to attack state-of-the-art natural text detectors. This falls under white-box adversarial attacks. In such attacks, attackers have access to the original text and create mutation instances based on this original text. The ultimate goal is to confuse machine learning models and classifiers and decrease their prediction accuracy. We introduced a general framework for building the character- and word-level mutation operators. Several operators were demonstrated and evaluated using the text captions of the MS COCO2017 dataset and state-of-the-art neural language models. We believe the proposed mutation-based adversarial attacks can be used as a systematic way to evaluate the robustness of any language analysis models.

Project Abstract:

Large language models have become the mainstay in natural language processing (NLP). These models entail high costs in terms of storage, memory, and computation time and it has motivated a large body of work on model compression to make them smaller and faster to use in real-world applications. One attractive solution to this problem is parameter efficient fine-tuning, in which we need only fine-tuning a subset of the model parameters. However, the question as to which subset to be trained to achieve the best result remains unanswered. In this project, we have initially analyzed different components in the commonly used BERT model to see which one undergoes the most change after fine-tuning. We show that output of LayerNorm changes the most among other model components when fine-tuned with Microsoft Research Paraphrase Corpus (MRPC), as one of the of the General Language Understanding Evaluation (GLUE) tasks. We further show that by only fine-tuning this component can have competitive performance to full fine-tuning and other parameter-efficient fine-tuning approaches. Moreover, we use Fisher information to assess which parameters in LayerNorm are most important in order to have even less parameters involved in parameter-efficient fine-tuning. After getting the resources, we plan to test our hypothesis on the rest of the GLUE tasks before we apply the model to a real-world application of the drug labeling data.

Project Abstract:

Reasoning plays a central role in human cognition and human ability to solve problems, make decisions, and think critically. In the field of deep learning, large language models (LLMs) have recently obtained human-level performance on a variety of tasks including translation, question-answering, and summarization. Despite these successes, recent works such as GPT-4 have highlighted that LLMs still perform poorly in reasoning and mathematical problem solving. Current approaches adapt LLMs to math datasets using continual pretraining, gradient-based finetuning, or prompt engineering to elicit step-by-step reasoning. Though these methods have significantly improved LLM performance on tasks such as grade school and high school math, LLMs still make simple arithmetic errors and struggle to write complex proofs. We aim to improve the performance of LLMs on mathematical problem solving by (1) benchmarking LLMs on challenging reasoning tasks to better understand their failures, and by (2) introducing and evaluating novel finetuning-based methods for adapting LLMs to reasoning tasks. We anticipate that a key to our methods will be teaching LLMs how to plan, brainstorm, and backtrack in the process of writing mathematical arguments.

Project Abstract:

Efficient DL training on large sample sizes (e.g. long-sequence language modeling and large-image vision modeling) would provide wide-reaching improvements to cutting-edge applications in vision (medical, geospatial, and astronomical imaging), and language (Document summarization, extractive Q&A, DNA sequence analysis). However, storing the large samples on accelerator memory is a challenging paradigm for GPU-based HPC systems to tackle due to the limited HBM storage on GPUs. The configurable MemoryX solution provided for Cerebras systems, however, decouples accelerator memory from the accelerator. Therefore, we propose training large transformer-based and CNN-based models on large-sample datasets on the CS-2 architecture. We believe such a pairing will demonstrate the unique strengths of Cerebras hardware on a wide-reaching application domain.

Project Abstract:

Schizophrenia is a severe and chronic brain disorder associated with delusions, hallucinations, disorganized thoughts, and cognitive impairments. Available treatments are symptomatic and do not provide lasting recovery in majority of persons with schizophrenia. Therefore, better elucidation of neurobiology of this illness may help design new treatments. Studies to date clearly support that schizophrenia and related psychotic disorders are dysconnection syndromes. Hence, there is tremendous impetus to understand the nature and causes of dysconnectivity. However, current efforts are directed at examining networks built on one modality of data such as diffusion or functional connectivity. Our lab, the CONCEPT Lab, uses multimodal MRI data, e.g., structural, diffusion-weighted, and functional imaging data to construct multiplex multilayer networks to delineate dysconnectivity related to schizophrenia at different levels. Using this approach, our goal is to understand schizophrenia networks on a nodal and global level as well as at the subject and the group level. Differences between the brain networks of persons with schizophrenia and healthy subjects have been extensively reported. Recent studies have been conducted using graph theoretical approaches. Although this approach provides important leads in elucidating network architecture and provide clues on potential functional impact, it does not provide means to examine the entire graph and characterize the differences. For examples, it does not answer questions on whether there are differences in the patterns of network architecture, what features of nodes tend to affect the strength of connections and whether the edge-centric pattern can help in classifying the networks. Graph Neural Networks (GNNs) are a way to identify graph difference and categorize networks. This set of machine learning approaches can help draw inferences on the nodes, edges, and the graph level characteristics. Using GNN, graphs and nodes can be classified into groups, and we will be able to make edge, or link, predictions. This will allow us to understand which graph connectivity pattern is unique to patients versus controls on a global and nodal level. Further, we are also interested in finding out if there are subgroups within patients since it is well known that schizophrenia is a heterogeneous disorder. These efforts will go a long way to help us understand the underlying pathology and how to better treat each network classification.

Project Abstract:

Dynamical and stochastic processes are ubiquitous in scientific applications and these are often governed by parametrized equations that are deterministic differential systems and/or stochastic processes. We recently developed parameter estimation techniques based on deep neural networks to estimate parameters of such systems. Advantages of our approach are fast parameter predictions, which, as a consequence, accelerate applications with real-time and frequent estimation demands. However, the training phase of deep neural networks poses computational bottlenecks preventing a wider applicability of deep learning-based parameter estimation. Our aim is to overcome these limitations with recent advances in hardware and algorithms tailored to neural networks.

Project Abstract:

The emergence of large multi-modal foundation models has altered the paradigm of machine learning and computer vision. Our project aims to create cutting-edge object tracking algorithms, built upon the pioneering foundation models. By integrating multi-modal cues, we will significantly elevate tracking precision and robustness, driving transformative advancements in the field. In this project, we will start from the vision-language foundation models and fine-tune them specifically for object tracking applications. We will also devise novel multi-model feature fusion modules, employing Transformers to better harness information from the language domain.

Project Abstract:

We propose to expand the Wafer Scale Engine, Field Equation, Application Programming Interface (WFA) by developing new kernels to solve MD and MC problems. Specifically, we propose to implement the spatial-decomposition method (ref 2) for MD simulations on WSE, and algorithms performed on Graphics Processing Units (GPU) (ref 3) for MC simulations. Many kernels, such as force, energy, periodic boundary condition, nearest neighboring link list and link cells, Ewald summation and Wolf’s method to conduct electrostatic interactions, excluded list to exclude specific atom pairs, various integrators to solve second-partial differential equations will be implemented. Note that although the same Newtonian equations are to be solved in both CFD and MD simulations, they are solved using completely different methods. In CFD, an Euler-type method was typically used to solve physical properties (such as temperature, pressure, and fluid velocity) in a fixed space. In contrast, in MD simulations, the Lagrangian method was used to track the moving history of each particle and compute the ensemble average over particles. For MC simulations, three different moves in MC simulations, that is, thermal move, volume change volume, and insertion and deletion of molecules, will be implemented on WSE. It is expected that implementing these MD/MC kernels on WSE will benefit a lot to various industries and academia, such as accelerating drug design, facilitating materials development for carbon capture, battery electrolyte, gas sensor, and beyond. Finally, we expect that researchers can utilize these low-level fundamental MD/MC kernels to develop more advanced simulation methods on WSE. The researchers will use Python to call these kernels to build more advanced simulation methods. This way, researchers do not need to handle and process complex operations on WSE. To start the MC project implementation, we will implement new kernels on WFA to solve the 2-D Ising model problem, which shares many significant features with the more complex atomistic MC simulations, such as the acceptance rule and periodic boundary condition.

Project Abstract:

The National Energy Technology Laboratory (NETL) leads US DOE multi-lab SMART (Scientific Machine learning to Accelerate Real-Time decisions) Initiative, with one of the goals aiming at near real-time forecasting of CO₂ behavior after injection in subsurface storage reservoirs. One of SMART tasks is to explore and research advanced AI/ML techniques and computational technologies. In this proposed work, NETL’s SMART team aims at using CS-2 with the team’s AI/ML models and leveraging NETL’s Wafer scale engine Field equation Application programming language (WFA) to explore HPC-type solutions for subsurface. There is a separate PSC proposal from NETL to focus on maturing the WFA.

Project Abstract:

Experiencing a seizure while alone increases morbidity and mortality. Studies based on national population registries have shown a 3.5 times higher odds of experiencing a sudden, unexpected death for people with epilepsy who sleep alone compared to those who sleep with a partner. This led to the development of seizure alert devices. Currently, over 20 seizure alert devices are commercially available. All are proprietary and none have undergone a clinical trial comparing performance in real time to human epilepsy doctors. The need for bespoke hardware with proprietary seizure alert devices advertises the medical condition of the user, which impacts technology adoption. Our work aims to create a clinically validated, open-source seizure alert system that can function using the accelerometers found in smartwatches. This will allow discreet deployment of models validated in a blinded clinical trial for detecting seizures.

Project Abstract:

The gigantic size of recent transformer language models like GPT3 is due to the use of large dense weight matrices, too large to fit even on a wafer-scale engine like the Cerebras systems. There has been increasing recent interest in exploring more compact representations for weight matrices using various factored representations using tensor networks. In this project, we propose to explore two complementary research questions: 1) Can we develop compact and accurate factored language models that can fit within the CS-1 and achieve comparable accuracy to much larger models using the standard transformer architecture? 2) Can we develop effective customized mapping/scheduling techniques to enable high performance on the Cerebras CS-1 for training and inference with factored tensor networks?

Project Abstract:

Graph Neural Network (GNN) is a promising approach to efficiently learn from graph data structures, having shown advantages in many critical applications such as chemical reaction prediction, traffic state prediction, and text classification. This project is to investigate how to accelerate large-scale Graph Neural Networks by leveraging Cerebras Sparse Linear Algebra Compute Cores and our proposed new graph re-ordering algorithm. This project is well aligned with our NSF-funded large-scale DNN training project (https://www.nsf.gov/awardsearch/showAward?AWD_ID=2034169).

Project Abstract:

The National Energy Technology Laboratory and Cerebras Systems Inc. are developing a domain specific Wafer scale engine Field equation Application programming interface (WFA). It is designed to solve field equations on the Wafer Scale Engine (WSE). Initial results showed the outstanding performance of WFA in speed, power consumption, and carbon footprint. This project aims at further maturing the WFA and investigating the applications in computational fluid dynamics and other related fields. By maturing the WFA, users will be able to form and solve field equations with easy-to-use and high-level Python interface while maintaining high performance of the WSE.

Project Abstract:

Machine Learning has enabled the prediction of various material properties from formation energies, HOMO/LUMO levels to atomic energies and forces. The increasing number of material and molecule datasets available present an opportunity to train Machine Learning models on datasets larger than typically used in materials science including larger sets of descriptors and more model parameters. Computational cost for training typically limits the dataset size and model size. In this work we train Machine Learning models to predict scalar properties of metal hydrides, materials which have been shown to have high temperature superconducting properties as well as molecular datasets such as QM9 important in various chemical processing industries. NeoCortex will allow us to push the limits of the sizes of training sets and models at record training speeds in an attempt to beat the state of the art accuracy on scalar properties.

Project Abstract:

The research project “One Model to Generate Them All: Scaling Multi-modal Diffusion Transformers on CS-2” aims to develop a new generation of infrastructure for Diffusion Transformers, which achieve outstanding performance on image conditional generation tasks and showcase the remarkable scalability of transformers within the diffusion framework. A major benefit of using a transformer backbone is its adaptability to multi-modal learning, allowing for the standardization and unification of architectural backbones across various domains. The project intends to train multi-modal transformer-based foundation models using CS-2, which could revolutionize the research landscape in this area. The proposed multi-modal diffusion transformers will utilize existing transformer blocks in PyTorch and will be extended to support various modalities such as images, audio, and video by designing unique conditioning modules.

Project Abstract:

This proposal studies efficient optimization for transformer-based neural document ranking. Recently transformer- based neural ranking with deep contextual models has been extensively studied in delivering a high relevance score for top k search of text documents. The main challenge is that using such a model to rank or re-rank documents is extremely expensive during the runtime inference. This proposal is focused on developing efficient solutions to perform transformer-based re-ranking computation during ad-hoc query processing. The evaluation process will use public datasets to assess the effectiveness of the proposed techniques in terms of relevance and efficiency.

Project Abstract:

The hardware design industry has traditionally required manual processes for the creation and verification of Verilog RTL (register-transfer level) designs. However, with the emergence of large language models, there is growing interest in utilizing these models to automate various tasks in the hardware design process. Our project aims to establish a foundation model for efficient and trustworthy hardware design through the automation of Verilog RTL tasks using large language models. We plan to pre-train a T5-like large language model on a self-curated dataset consisting of approximately 130 million Verilog code samples. We also aim to modify the T5 model architecture to enhance its performance on downstream tasks such as code summarization and refinement. To further optimize the training process, we plan to design an efficient loss function. Our framework will utilize complementary libraries including Transformers, OpenAI, Pickle, and Datasets among others, in addition to the standard PyTorch and TensorFlow distributions. Our approach has the potential to transform the hardware design industry by utilizing advanced language models and cutting-edge computational resources to create more efficient and reliable systems. By establishing a foundation model for efficient and trustworthy hardware design, our project aims to revolutionize the Verilog RTL design process.