Development

Development notes for savu-lite

Cooking with Conda

To set-up the basic development environment download Miniconda2-latest-linux-x86_64.sh and install on local machine. Update to get latest versions and then install conda-build, conda-verify, sphinx (for documentation), and anaconda-client (to allow uploading to anaconda cloud).

  1. conda update conda
  2. conda install conda-build
  3. conda install conda-verify
  4. conda install sphinx
  5. conda install anaconda-client

Building savu-lite

Build Sequence

Move to python 2 environment “miniconda2”

conda activate miniconda2

From directory containing meta.yaml and build.sh {the only required files}

conda-build purge conda-build .

Upload resulting file that will be created in linux-64 folder {first time it will create anaconda-project.yml}

anaconda upload linux-64/savu.....

Test by downloading file from test channel into fresh test environment

conda install -c test_channel savu-lite

Run training example and savu_full_test

Upload to savu-dep if testing passes

anaconda upload -u savu-dep linux-64/savu.....

Channels

Default channels used

Additional channels used

Channel Priority

Channel_Priority is one of strict, flexible, disabled. Strict chosen as it should be quickest as it takes stops when it finds a package looking in defined order.

.condarc includes

  • channels:
    • defaults
    • conda-forge
    • astra-toolbox/label/dev
    • ccpi
    • dkazanc
  • auto_activate_base: false
  • show_channel_urls: true
  • unsatisfiable_hints: true

Local Channels Creation

Created a Conda Environment and install conda-build

conda create --name local_channel_name python=2 conda-build

conda install conda-build

To create a local channel for python package “click”:

conda skeleton pypi click

conda-build click

whenever local channels change run the following in the channel root directory:

conda index

Able to pick-up local version of click using:

conda install -c local_channel_name click

Use of PIP with conda

pip installs are not recommended as they mix poorly Conda and should be done after conda installs completed!?

Currently the following files need to be pip installed as part of the build.sh script:

  • gnureadline (8.0.0) - deprecated and no longer recommended for use
  • nvidia-ml-py (7.352.0) - older python 2 version

However Conda and Pip don’t play well together and this is only an interim solution.

Gnureadline

This is reasonably low-risk given the lack of dependencies. The long-term solution is likely to be removal of dependency in Savu. Conda install options listed at: https://anaconda.org/conda-forge/gnureadline do not work.

Nvidia-ml-py

https://files.pythonhosted.org/packages/72/31/378ca145e919ca415641a0f17f2669fa98c482a81f1f8fdfb72b1f9dbb37/nvidia-ml-py-7.352.0.tar.gz

Anaconda Cloud

Anaconda Cloud is best delivery mechanism for savu-lite. Adding the built package to existing savu-dep anaconda cloud repository automatically allows conda install with savu-dep as the channel. This seems much easier that attempting to set-up a continuous integration build on conda-forge that would not be of much use for this application. It also quick, less than 15 minutes to upload and requires no approval from any external entity.

To access Anaconda Cloud

Sign-up at: https://anaconda.org/ Add to conda installation anaconda-client as follows:

conda install anaconda-client

Notes needed to delete and re-add to change documentation URL shown on Anaconda Cloud.

Build with Miniconda 3 or 2 ?

Miniconda2 seemed to have stability issues {some packages no longer supported?!}. Miniconda3 may cause issues when used even though building a python 2 system although this needs to be reconfirmed.

Miscellaneous

Python 2 “end of life”

Any Python 2 solution will be frozen in time as support for dependant libraries will diminish and cease. Currently 2.7.16 used, 2.7.17 is latest with 2.7.18 due in April

numpy is no longer supported for python 2; last available version that can be used is 1.16.6

Virtual Packages exist for: cuda (__cuda) and glibc (__glibc) These can detect presence of and return version numbers for features that cannot be managed directly by conda.

yaml need pyyaml in recipe to import yaml even though yaml is installed

pandoc

A very nice tool that automatically converts between document formats, it is available via conda, and should do most and in simple cases all of the necessary work for likely source formats. As a command line tools it is automation friendly. Details can be found at pandoc