2016-05-09 Exceptional meeting regarding build customization

14:00 UTC

Hangout link: ???

Attendees: Michael Sarahan, Ray Donnelly, Kale Franz, Michael Grant, Ullrich Koethe, Stuart Berg, Phil Elson, Mark Wiebe

Agenda/Minutes:

Version parameterization:

*   MichaelS:  proposing to have dependency resolution become part of meta.yaml  rendering.  May require extra (3rd) jinja rendering pass.

Need to know dependencies before Jinja templates can be fully expressed (main issue in #747)

Phil: not good, because you cant differentiate between a manual entry of a value, and a templated entry of a value.

    *   Dont know which ones are important for build string

            *   Build strings only matter when we have a build matrix; there to disambiguate
*   MichaelS proposes manual specification of entries in build string
*   Ullrich: if build strings just disambiguate, why not use hash?

                    *   MichaelS: hash fine, but needs to be reproducible.

    *   Heart of issue is that matrix build tools need some way to understand what parameterizes the build matrix.

            *   MichaelS  offered that no matter what solution we settle on, there will be  something readily inspectable.  Maybe this can feed into automatically  creating the build string.

If recipe-local config.yaml is needed, Kale asks why not include it directly in meta.yaml

*   Not  sure what use case necessitates recipe-local config.yaml (global one is  agreed upon by all).  MichaelS OK with meta.yaml modifications if need  be.

Phil points out that global config.yaml might still need code changes for new versions of, say, Python.

*   MichaelS  says no, theres surely ways we can express things like latest 2  versions - just question of how to technically achieve.

Ray: filter things out of global config.yaml when they are not used in a recipe (so that matrix dimensions are not needlessly added)

*   Ullrich:  However, one must be careful to not filter out indirect dependencies  that are not explicitly named in meta.yaml, but were intentionally added  to config.yaml. (example: meta.yaml lists h5py as a requirement, and  config.yaml restricts the indirect hdf5 dependency to a particular  variant of the hdf5 package). It is safer to err on the too much side.

Build environment specification:

*   Conda-build 848 has bootstrap environment: user creates an env, that env is captured as build requirements for another recipe.

        *   Phil: this ruins reproducibility, since it depends on the setup of the users system at build time.

                *   Stuart: were already not reproducible, and this bootstrap doesnt make things worse.
    *   MichaelS:  if this is used to hard-pin build-time dependencies, it may actually  help reproducibility.  Hard-pinnning at build time is orthogonal to this  PR, but may be easier to achieve with it.
    *   Ullrich:  if the contents of the build environment are recorded in index.json,  one can recreate the bootstrap environment at any time (or equivalently,  create config.yaml from that information), subject to the degree of  accuracy of the recorded data (e.g. are the source channels recorded, or  only version numbers and build strings?)

        *   MichaelS,  Kale concerned that people may include too many unnecessary packages in  build requirements this way, out of carelessness or ignorance

                *   Ullrich: not a big deal - runtime dependencies are what really matter.

Compiler customization:
```
*   Mark: 848 is OK, but how does it address compiler and build flags?
```
- Conda 1959 is example of build flags. Uses bat/sh script at beginning of bld.bat/build.sh. Similar to Marks effort. Dont like having to add that extra shell script, though - want native support
```
    *   MichaelS  offered integration with conda-build, when use cases are better  understood.  Some kind of hook to call standardized setup script.
```
  - Ullrich: The important aspect is that it should be easy for the conda-build user to specify which setup script to use. Having specialized build packages installed in a bootstrap environment seems to be an easy solution (e.g. conda create -n my_bootstrap_env visual-studio-build=12.0 or conda create -n my_bootstrap_env visual-studio-build=12.0.debug)
  - Ullrich points out that these packages also need to exclude use with other packages, as they are not compatible
- Mark: would like to create easy to use instance of http://vfxplatform.com/ using conda. Wants to be in the loop as we get that functionality worked out.
- Ullrich: Critical requirements for the final solution (whatever it will be):
```
    *   Build dependency customization must be possible for _any_  package, although it would be OK if it were a bit more complicated for  the less common situations. It is crucial to support two cases:

            *   Customize _indirect _dependencies not specified in meta.yaml.
*   Narrow dependencies listed in meta.yaml (e.g. python 3.* => python 3.5.*).

    *   Run  requirements must be specialized during the build such that version  resolution will only select compatible variants, even if active channels  offer newer, yet incompatible variants (right now I usually use -c  ukoethe --override-channels, but this is a pain).
```