Would a certification process for models published on the DDMoRe repository serve the modelling community?

Aims: The Drug Disease Model Resources (DDMoRe, http://www.ddmore.eu/) consortium was created to improve the quality, efficiency and cost effectiveness of Model-Informed Drug Discovery & Development (MID3) by delivering an open source, integrated framework to enable MID3. The DDMoRe Model Repository (http://repository.ddmore.eu/) is a platform for storage of models relevant for MID3 and includes specific features for models represented by the Pharmacometrics Markup Language, PharmML (1). Together with DDMoRe’s unique Interoperability Framework, collaborative development of computational models can be facilitated. Models are accepted for publication in the DDMoRe Model Repository if they fulfil specified minimal requirements, which are automatically checked (no human intervention) when uploading the model. However, to ensure that models published in the DDMoRe Model Repository can be trustfully used, a Model Qualification Procedure (MQP) has been developed with the aim to assign qualified models with a DDMoRe model certification.

Methods: The MQP details a human-based review process performed by a body of qualified experts, the Modelling Review Group. The model being subject for review must be publicly available on the DDMoRe Model Repository and meet specified eligibility criteria. The certification is then requested by the model submitter. The review aims to ensure the technical validity of the model and its correspondence with the associated publication and, therefore, checks that i) information associated with the model is complete, makes sense and is sufficient to understand the model, ii) the submitted model code represents the original model in the associated publication without modifications, iii) the model can be executed using the DDMoRe Interoperability Framework, iv) additional validation information is sufficient to prove correspondence of the submitted model with the model in the associated publication. The model evaluation is concluded by a decision of granting or rejecting the certification and the DDMoRe model certification is displayed on the repository, with a short report of the model review outcome.

Results: When a model is granted the DDMoRe certification, the review confirms that the information provided by the submitter is sufficient for model interpretation, i.e. complete and comprehensible. Furthermore, successful execution of the model code by the reviewer endorses functioning of the code. Also, the review concludes that results of the model execution (e.g. parameter estimates and uncertainty, goodness of fit statistics) and other validation output generated from the model code (e.g. simulations, plots) are in agreement with the corresponding results and output generated from the original model code presented in the associated publication, thus confirming the adherence of the certified model with the associated publication. An example of the implementation of the MQP will be presented.
The benefits for the model community of such a technical validation procedure ensures that the certification guarantees that the documentation provided with the model is understandable and that the model can be downloaded from the repository and safely used for the purposes for which it was proposed. Furthermore, these models can then serve as useful examples for various modelling approaches.

Conclusion: The current MQP considers the validity of a model from a technical point of view, thus ensuring that a model is executable and corresponds to is the one reported in the associated publication.

This work is on behalf of the DDMoRe project (www.ddmore.eu).

1. Swat MJ et al. Pharmacometrics Markup Language (PharmML): Opening New Perspectives for Model Exchange in Drug Development. CPT Pharmacometrics Syst Pharmacol. 2015 ;4(6):316-9.