memote - the genome-scale metabolic model test suite

Our goal in promoting this tool is to achieve two major shifts in the metabolic model building community:

  1. Models should be version-controlled such that changes can be tracked and if necessary reverted. Ideally, they should be available through a public repository such as GitHub that will allow other researchers to inspect, share, and contribute to the model.
  2. Models should, for the benefit of the community and for research gain, live up to certain standards and minimal functionality.

The memote tool therefore performs four subfunctions:

  1. Create a skeleton git repository for the model.
  2. Run the current model through a test suite that represents the community standard.
  3. Generate an informative report which details the results of the test suite in a visually appealing manner.
  4. (Re-)compute test statistics for an existing version controlled history of a metabolic model.

And in order to make this process as easy as possible the generated repository can easily be integrated with continuous integration testing providers such as Travis CI, which means that anytime you push a model change to GitHub, the test suite will be run automatically and a report will be available for you to look at via GitHub pages for your repository.

Contact

For comments and questions get in touch via

Are you excited about this project? Consider contributing by adding novel tests, reporting or fixing bugs, and generally help us make this a better software for everyone.

Contents

Getting Started

Installation

We highly recommend creating a Python virtualenv for your model testing purposes.

Stable release

To install memote, run this command in your terminal:

$ pip install memote

This is the preferred method to install memote, as it will always install the most recent stable release.

If you don’t have pip installed, this Python installation guide can guide you through the process.

From sources

The sources for memote can be downloaded from the Github repo.

You can either clone the public repository:

$ git clone https://github.com/opencobra/memote.git

Or download the tarball or zip archive:

$ curl  -OL https://github.com/opencobra/memote/archive/master.zip

Once you have a copy of the source files, you can install it with:

$ pip install .

After installation, memote can be employed in two different ways: As a benchmarking tool for ad hoc model assessment and as an automated testing suite to aid with the reconstruction of metabolic models. When using memote to benchmark a model, the tests are run once and a report is generated which describes the status-quo. As an automated testing suite, memote facilitates tracking incremental model changes in a version controlled repository and can enable continuous testing and reporting if desired.

Here, we explain step-by-step the necessary commands to pursue either workflow. Users that have already followed this tutorial once may want to refer to the cheat-sheet flowchart to refresh their memory.

Benchmark

Single Model

To benchmark the performance of a single model, run this command in your terminal:

$ memote --model path/to/model.xml report --one-time

This will generate the performance report as index.html.

The output filename can be changed by adding the following option. To illustrate here it is changed to report.html.

$ memote --model path/to/model.xml --filename "report.html" report --one-time
Comparative

This functionality is coming soon.

Comparing two models against each other and quickly identify the differences.

Reconstruction

When starting a memote repository, users need to provide an SBMLv3-FBC formatted file. Automatic draft reconstruction tools such as Pathway Tools, Model SEED, The RAVEN Toolbox and others are able to output files in this format. Model repositories such as BiGG or BioModels further serve as a great resource for models in the correct format.

With this in mind, starting a local, version-controlled model repository is as simple as running the following command:

$ memote new
CI tested, online and public workflow:

After this, the user will be prompted with a few questions regarding details of the project. If the project is to be kept strictly locally, the user does not need to supply GitHub (or GitLab - not implemented yet) credentials. However, these are a requirement if the project is to use the full benefits of distributed version control such as cloud-based development, remote collaboration and community feedback. It is important to note that furthermore a public repository is needed to set up automatic testing through continuous integration, one of the key features of memote.

Once all the questions following memote new have been answered, a public repository has been created under either the user’s GitHub or GitLab account. To enable continuous integration via Travis CI the following command is executed:

This functionality is coming soon. A manual workaround is outlined in the cookiecutter-memote readme.

$ memote online

Now, after each edit to the model in the repository, the user can generate an update to the continuous model report shown at the project’s gh-pages branch by saving the changes with the following command:

This functionality is coming soon, for now please utilize the steps outlined for advanced users.

$ memote save

For advanced users: memote save is the equivalent of executing git add ., git commit and git push in sequence.

Offline, local or private workflow:

Users that have decided to not to use GitHub (or GitLab Not implemented yet) or those that have decided to set the model repository to private, will need to execute:

$ memote

to run the testing suite on their commit history followed by:

$ memote report

to generate the same type of report that would be shown automatically with continuous integration. After this it is crucial to save the generated test results by running memote save again.

We recommend the public workflow not only to promote open, collaborative science but also to benefit from the full functionality of memote.

Flowchart

This cheat-sheet flowchart servers as a visual guide on how to get up and running with memote! It is essentially the pictographic form of the section Getting Started.

_images/MemoteFlowchart.png

Test Suite

Core Tests

Basic

Perform basic tests on an instance of cobra.Model.

test_basic.test_gene_protein_reaction_rule_presence(read_only_model, store)[source]

Expect all non-exchange reactions to have a GPR.

test_basic.test_genes_presence(read_only_model, store)[source]

Expect that >= 1 genes are present in the read_only_model.

test_basic.test_metabolic_coverage(read_only_model, store)[source]

Expect a model to have high metabolic coverage.

test_basic.test_metabolites_charge_presence(read_only_model, store)[source]

Expect all metabolites to have a formula.

test_basic.test_metabolites_formula_presence(read_only_model, store)[source]

Expect all metabolites to have a formula.

test_basic.test_metabolites_presence(read_only_model, store)[source]

Expect that >= 1 metabolites are present in the read_only_model.

test_basic.test_model_id_presence(read_only_model, store)[source]

Expect that the read_only_model has an ID.

test_basic.test_ngam_presence(read_only_model, store)[source]

Expect a single non growth-associated maintenance reaction.

test_basic.test_reactions_presence(read_only_model, store)[source]

Expect that >= 1 reactions are present in the read_only_model.

Consistency
Biomass
Annotation
Syntax

Contributing

Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.

You can contribute in many ways:

Report Bugs

Report bugs at https://github.com/opencobra/memote/issues.

If you are reporting a bug, please include:

  • Your operating system name and version.
  • Any details about your local setup that might be helpful in troubleshooting.
  • Detailed steps to reproduce the bug.

Fix Bugs

Look through the GitHub issues for bugs. Anything tagged with “bug” and “help wanted” is open to whoever wants to implement it.

Implement Features

Look through the GitHub issues for features. Anything tagged with “enhancement” and “help wanted” is open to whoever wants to implement it.

Write Documentation

memote could always use more documentation, whether as part of the official memote docs, in docstrings, or even on the web in blog posts, articles, and such.

Submit Feedback

The best way to send feedback is to file an issue at https://github.com/opencobra/memote/issues.

If you are proposing a feature:

  • Explain in detail how it would work.
  • Keep the scope as narrow as possible, to make it easier to implement.
  • Remember that this is a volunteer-driven project, and that contributions are welcome :)
Get Started!

Ready to contribute? Here’s how to set up memote for local development.

  1. Fork the memote repo on GitHub.

  2. Clone your fork locally:

    git clone git@github.com:your_name_here/memote.git
    
  3. Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed, this is how you set up your fork for local development:

    mkvirtualenv memote
    cd memote/
    pip install -e .
    
  4. Create a branch for local development using fix or feat as a prefix:

    git checkout -b fix-name-of-your-bugfix
    

    Now you can make your changes locally.

  5. When you’re done making changes, check that your changes pass flake8 and the tests, including testing other Python versions. This is all included with tox:

    tox
    

    You can run all tests in parallel using detox. To get tox and detox, just pip install them into your virtualenv.

  6. Commit your changes and push your branch to GitHub. Please use semantic commit messages:

    git add .
    git commit -m "fix: Your detailed description of your changes."
    git push origin fix-name-of-your-bugfix
    
  7. Submit a pull request through the GitHub website.

Pull Request Guidelines

Before you submit a pull request, check that it meets these guidelines:

  1. The pull request should include tests.
  2. If the pull request adds functionality, the docs should be updated. Put your new functionality into a function with a docstring.
  3. The pull request should work for Python 2.7, 3.4, 3.5 and 3.6. Check https://travis-ci.org/opencobra/memote/pull_requests and make sure that the tests pass for all supported Python versions.

Credits

Development Lead

History

0.3.4 (2017-08-12)

  • Properly configure Travis deployment.

0.3.3 (2017-08-12)

  • Build tags.

0.3.2 (2017-08-12)

  • Enable automatic deployment to PyPi.

0.3.0 (2017-08-12)

  • Greatly extend the core test modules: * basic * consistency * biomass * annotation * syntax
  • Add an Angular-material based report with plotly.
  • Add documentation on readthedocs.io.
  • Make the first release on PyPi.

0.2.0 (2017-02-09)

  • Yet another package structure for supporting functions, their tests, and the model test suite.

0.1.0 (2017-01-30)

  • New package structure and start of joint development

Indices and tables