To Örebro University

Being able to predict the cell permeability of cyclic peptides is essential for unlocking their potential as a drug modality for intracellular targets. With a wide range of studies of cell permeability but a limited number of data points, the reliability of the machine learning (ML) models to predict previously unexplored chemical spaces becomes a challenge. In this work, we systemically investigate the predictive capability of ML models from the perspective of their extrapolation to never-before-seen applicability domains, with a particular focus on the permeability task. Four predictive algorithms, namely Support-Vector Machine, Random Forest, LightGBM and XGBoost, jointly with a conformal prediction framework were employed to characterize and evaluate the applicability through uncertainty quantification. Efficiency and validity of the models' predictions with multiple calibration strategies were assessed with respect to several external datasets from different parts of the chemical space through a set of experiments. The experiments showed that the predictors generalizing well to the applicability domain defined by the training data, can fail to achieve similar model performance on other parts of the chemical spaces. Our study proposes an approach to overcome such limitations by the means of improving the efficiency of models without sacrificing the validity. The trade-off between the reliability and informativeness was balanced when the models were calibrated with a subset of the data from the new targeted domain. This study outlines an approach to enable the extrapolation of predictive power and restore the models' reliability via a recalibration strategy without the need for retraining the underlying model. This work outlines peptide predictive model methodology with conformal prediction, focusing on extrapolation task. Calibrating on the unseen chemical space recovers efficiency and validity enabling reliable predictions without retraining the models.

Conformal prediction has seen many applications in pharmaceutical science, being able to calibrate outputs of machine learning models and producing valid prediction intervals. We here present the open source software CPSign that is a complete implementation of conformal prediction for cheminformatics modeling. CPSign implements inductive and transductive conformal prediction for classification and regression, and probabilistic prediction with the Venn-ABERS methodology. The main chemical representation is signatures but other types of descriptors are also supported. The main modeling methodology is support vector machines (SVMs), but additional modeling methods are supported via an extension mechanism, e.g. DeepLearning4J models. We also describe features for visualizing results from conformal models including calibration and efficiency plots, as well as features to publish predictive models as REST services. We compare CPSign against other common cheminformatics modeling approaches including random forest, and a directed message-passing neural network. The results show that CPSign produces robust predictive performance with comparative predictive efficiency, with superior runtime and lower hardware requirements compared to neural network based models. CPSign has been used in several studies and is in production-use in multiple organizations. The ability to work directly with chemical input files, perform descriptor calculation and modeling with SVM in the conformal prediction framework, with a single software package having a low footprint and fast execution time makes CPSign a convenient and yet flexible package for training, deploying, and predicting on chemical data. CPSign can be downloaded from GitHub at https://github.com/arosbio/cpsign.

Scientific contribution: CPSign provides a single software that allows users to perform data preprocessing, modeling and make predictions directly on chemical structures, using conformal and probabilistic prediction. Building and evaluating new models can be achieved at a high abstraction level, without sacrificing flexibility and predictive performance-showcased with a method evaluation against contemporary modeling approaches, where CPSign performs on par with a state-of-the-art deep learning based model.

Oxidative stress is the consequence of an abnormal increase of reactive oxygen species (ROS). ROS are generated mainly during the metabolism in both normal and pathological conditions as well as from exposure to xenobiotics. Xenobiotics can, on the one hand, disrupt molecular machinery involved in redox processes and, on the other hand, reduce the effectiveness of the antioxidant activity. Such dysregulation may lead to oxidative damage when combined with oxidative stress overpassing the cell capacity to detoxify ROS. In this work, a green fluorescent protein (GFP)-tagged nuclear factor erythroid 2-related factor 2 (NRF2)-regulated sulfiredoxin reporter (Srxn1-GFP) was used to measure the antioxidant response of HepG2 cells to a large series of drug and drug-like compounds (2230 compounds). These compounds were then classified as positive or negative depending on cellular response and distributed among different modeling groups to establish structure-activity relationship (SAR) models. A selection of models was used to prospectively predict oxidative stress induced by a new set of compounds subsequently experimentally tested to validate the model predictions. Altogether, this exercise exemplifies the different challenges of developing SAR models of a phenotypic cellular readout, model combination, chemical space selection, and results interpretation.

Receptor-mediated molecular initiating events (MIEs) and their relevance in endocrine activity (EA) have been highlighted in literature. More than 15 receptors have been associated with neurodevelopmental adversity and metabolic disruption. MIEs describe chemical interactions with defined biological outcomes, a relationship that could be described with quantitative structure-activity relationship (QSAR) models. QSAR uncertainty can be assessed using the conformal prediction (CP) framework, which provides similarity (i.e., nonconformity) scores relative to the defined classes per prediction. CP calibration can indirectly mitigate data imbalance during model development, and the nonconformity scores serve as intrinsic measures of chemical applicability domain assessment during screening. The focus of this work was to propose an in silico predictive strategy for EA. First, 23 QSAR models for MIEs associated with EA were developed using high-throughput data for 14 receptors. To handle the data imbalance, five protocols were compared, and CP provided the most balanced class definition. Second, the developed QSAR models were applied to a large data set (∼55,000 chemicals), comprising chemicals representative of potential risk for human exposure. Using CP, it was possible to assess the uncertainty of the screening results and identify model strengths and out of domain chemicals. Last, two clustering methods, t-distributed stochastic neighbor embedding and Tanimoto similarity, were used to identify compounds with potential EA using known endocrine disruptors as reference. The cluster overlap between methods produced 23 chemicals with suspected or demonstrated EA potential. The presented models could be utilized for first-tier screening and identification of compounds with potential biological activity across the studied MIEs.

This investigation shows that successful forecasting models for monitoring forest health status with respect to Larch Casebearer damages can be derived using a combination of a confidence predictor framework (Conformal Prediction) in combination with a deep learning architecture (Yolo v5). A confidence predictor framework can predict the current types of diseases used to develop the model and also provide indication of new, unseen, types or degrees of disease. The user of the models is also, at the same time, provided with reliable predictions and a well-established applicability domain for the model where such reliable predictions can and cannot be expected. Furthermore, the framework gracefully handles class imbalances without explicit over- or under-sampling or category weighting which may be of crucial importance in cases of highly imbalanced datasets. The present approach also provides indication of when insufficient information has been provided as input to the model at the level of accuracy (reliability) need by the user to make subsequent decisions based on the model predictions.

Integration of statistical learning methods with structure-based modeling approaches is a contemporary strategy to identify novel lead compounds in drug discovery. Hepatic organic anion transporting polypeptides (OATP1B1, OATP1B3, and OATP2B1) are classical off-targets, and it is well recognized that their ability to interfere with a wide range of chemically unrelated drugs, environmental chemicals, or food additives can lead to unwanted adverse effects like liver toxicity and drug-drug or drug-food interactions. Therefore, the identification of novel (tool) compounds for hepatic OATPs by virtual screening approaches and subsequent experimental validation is a major asset for elucidating structure-function relationships of (related) transporters: they enhance our understanding about molecular determinants and structural aspects of hepatic OATPs driving ligand binding and selectivity. In the present study, we performed a consensus virtual screening approach by using different types of machine learning models (proteochemometric models, conformal prediction models, and XGBoost models for hepatic OATPs), followed by molecular docking of preselected hits using previously established structural models for hepatic OATPs. Screening the diverse REAL drug-like set (Enamine) shows a comparable hit rate for OATP1B1 (36% actives) and OATP1B3 (32% actives), while the hit rate for OATP2B1 was even higher (66% actives). Percentage inhibition values for 44 selected compounds were determined using dedicated in vitro assays and guided the prioritization of several highly potent novel hepatic OATP inhibitors: six (strong) OATP2B1 inhibitors (IC₅₀ values ranging from 0.04 to 6 μM), three OATP1B1 inhibitors (2.69 to 10 μM), and five OATP1B3 inhibitors (1.53 to 10 μM) were identified. Strikingly, two novel OATP2B1 inhibitors were uncovered (C7 and H5) which show high affinity (IC₅₀ values: 40 nM and 390 nM) comparable to the recently described estrone-based inhibitor (IC₅₀ = 41 nM). A molecularly detailed explanation for the observed differences in ligand binding to the three transporters is given by means of structural comparison of the detected binding sites and docking poses.

Read-across approaches often remain inconclusive as they do not provide sufficient evidence on a common mode of action across the category members. This read-across case study on thirteen, structurally similar, branched aliphatic carboxylic acids investigates the concept of using human-based new approach methods, such as in vitro and in silico models, to demonstrate biological similarity.

Five out of the thirteen analogues have preclinical in vivo studies. Three out of them induced lipid accumulation or hypertrophy in preclinical studies with repeated exposure, which leads to the read-across hypothesis that the analogues can potentially induce hepatic steatosis.

To confirm the selection of analogues, the expression patterns of the induced differentially expressed genes (DEGs) were analysed in a human liver model. With increasing dose, the expression pattern within the tested analogues got more similar, which serves as a first indication of a common mode of action and suggests differences in the potency of the analogues.

Hepatic steatosis is a well-known adverse outcome, for which over 55 adverse outcome pathways have been identified. The resulting adverse outcome pathway (AOP) network, comprised a total 43 MIEs/KEs and enabled the design of an in vitro testing battery. From the AOP network, ten MIEs, early and late KEs were tested to systematically investigate a common mode of action among the grouped compounds.

The targeted testing of AOP specific MIE/KEs shows that biological activity in the category decreases with side chain length. A similar trend was evident in measuring liver alterations in zebra fish embryos. However, activation of single MIEs or early KEs at in vivo relevant doses did not necessarily progress to the late KE “lipid accumulation”. KEs not related to the read-across hypothesis, testing for example general mitochondrial stress responses in liver cells, showed no trend or biological similarity.

Testing scope is a key issue in the design of in vitro test batteries. The Dempster-Shafer decision theory predicted those analogues with in vivo reference data correctly using one human liver model or the CALUX reporter assays.

The case study shows that the read-across hypothesis is the key element to designing the testing strategy. In the case of a good mechanistic understanding, an AOP facilitates the selection of reliable human in vitro models to demonstrate a common mode of action. Testing DEGs, MIEs and early KEs served to show biological similarity, whereas the late KEs become important for confirmation, as progression from MIEs to AO is not always guaranteed.

Machine learning models are widely applied to predict molecular properties or the biological activity of small molecules on a specific protein. Models can be integrated in a conformal prediction (CP) framework which adds a calibration step to estimate the confidence of the predictions. CP models present the advantage of ensuring a predefined error rate under the assumption that test and calibration set are exchangeable. In cases where the test data have drifted away from the descriptor space of the training data, or where assay setups have changed, this assumption might not be fulfilled and the models are not guaranteed to be valid. In this study, the performance of internally valid CP models when applied to either newer time-split data or to external data was evaluated. In detail, temporal data drifts were analysed based on twelve datasets from the ChEMBL database. In addition, discrepancies between models trained on publicly-available data and applied to proprietary data for the liver toxicity and MNT in vivo endpoints were investigated. In most cases, a drastic decrease in the validity of the models was observed when applied to the time-split or external (holdout) test sets. To overcome the decrease in model validity, a strategy for updating the calibration set with data more similar to the holdout set was investigated. Updating the calibration set generally improved the validity, restoring it completely to its expected value in many cases. The restored validity is the first requisite for applying the CP models with confidence. However, the increased validity comes at the cost of a decrease in model efficiency, as more predictions are identified as inconclusive. This study presents a strategy to recalibrate CP models to mitigate the effects of data drifts. Updating the calibration sets without having to retrain the model has proven to be a useful approach to restore the validity of most models.

The adverse outcome pathway (AOP) paradigm has brought mechanism of action in the spotlight of regulatory toxicology, linking biochemical interactions on cellular level (i.e. molecular initiating event, MIE) via key events to adverse outcomes (AOs) on population level. Developments on mechanistic understanding of endocrine disruption (ED) has brought forward MIEs associated with early neurode-velopmental interference  [1] and metabolic disruption [2], describing agonistic and antagonistic interactions with receptors such as constitutive androstane receptor (CAR), estrogen receptor alpha (ERα), farsenoid X receptor (FXR), and glucocorticoid receptor (GR). High confidence on in silico predictions is dictated by high quality training data  on  mechanistically  relevant  endpoints,  where  well-defined  chemistry is covered. Based on Tox21 in vitro assays describing events of agonism and antagonism for 13 receptors linked to ED, 23 in silico models were developed using Random Forest Classification. To quantify measures of uncertainty per prediction a Conformal Prediction framework was employed. In order to assess whether currently available models can confidently predict endocrine disrupting chemicals (EDCs), screening of EURION reference chemicals was conducted. The EURION cluster is a constellation of 8 research consortia aiming to improve endocrine disruption identification. Preliminary results revealed strengths in the use of in silico models for screening of current ED chemical landscape, and data gaps that need to be considered for next steps.