3. Subproject includes

We’ve already discussed how we can add optionality to projects and explored how we can perform conditional statements and include fragments of BuildStream YAML in the earlier chapter about optionality and directives.

In this chapter we’re going to explore how we can use include directives to include YAML fragments from a subproject referred to by a junction element, and how project options can be specified in the configuration of your junction.

Note

This example is distributed with BuildStream in the doc/examples/junction-includes subdirectory.

3.1. Overview

It is a goal of BuildStream to provide developers and integrators with the tools they need to maintain software stacks which depend on eachother with least friction as possible, such that one can integrate upgrades of projects one depends on via junction elements regularly and with the least hassle as possible.

Project options and include directives combined form the basis on which projects can maximize on code sharing effectively, and the basis on which BuildStream projects can form reliable APIs.

3.1.1. Project options

The options which a project exposes is a fairly limited API surface, it allows one to configure a limited set of options advertized by the project maintainers, and the options will affect what kind of artifacts will be produced by the project.

This kind of optionality however does not allow consumers to entirely redefine how artifacts are produced and how elements are configured.

On the one hand, this limitation can be frustrating, as one constantly finds themselves requiring a feature that their subproject does not support right now. On the other hand, the limitation of features which a given project chooses to support is what guards downstream project consumers against consuming artifacts which are not supported by the upstream.

Project options are designed to enforce a separation of concerns, where we expect that downstreams will either fork a project in order to support a new feature, or convince the upstream to start supporting a new feature. Furthermore, limited API surfaces for interdependent projects offers a possibility of API stability of projects, such that you can upgrade your dependencies with limited friction.

3.1.2. Includes

The includes which a project might advertize as “public”, form the output of the API exchange between a project and its subproject(s).

Cross-project include files allow a project to inherit configuration from a subproject. Include files can be used to define anything from the variables one needs to have in context in order to build into or link into alternative system prefixes, what special compiler flags to use when building for a specific machine architecture, to customized shell configurations to use when testing out applications in bst shell.

This chapter will provide an example of the mechanics of cross project includes when combined with project optionality.

3.2. Project structure

3.2.1. Project options

This example is comprised of two separate projects, both of which offer some project options. This is intended to emphasize how your toplevel project options can be used to select and configure options to use in the subprojects you depend on.

For convenience, the subproject is stored in the subdirectory of the toplevel project, while in the real world the subproject is probably hosted elsewhere.

First let’s take a look at the options declared in the respective project.conf files.

3.2.1.1. Toplevel project.conf

name: project
min-version: 2.0
element-path: elements

aliases:
  gnu: https://ftp.gnu.org/gnu/automake/

plugins:
- origin: pip
  package-name: buildstream-plugins
  elements:
  - autotools

# Define some options for this project
#
options:
  funky:
    type: bool
    description: Whether this project is funky
    default: False

# Define mirrors
mirrors:
- name: gnu_mirror
  aliases:
    gnu:
    - https://ftpmirror.gnu.org/gnu/automake/

3.2.1.2. Subproject project.conf

name: subproject
min-version: 2.0
element-path: elements

aliases:
  alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/

# Define some options for this project
#
options:
  color:
    type: enum
    description: The color of this runtime
    values:
    - red
    - green
    - blue
    default: blue

As we can see, these two projects both offer some arbitrarily named options.

3.2.2. Conditional configuration of subproject

The toplevel project here does some conditional configuration of the subproject.

3.2.2.1. Toplevel elements/subproject-junction.bst

kind: junction

config:
  # Configure the options for subproject
  #
  # If our project is funky, then it requires
  # a blue subproject, otherwise we use a red one.
  #
  options:
    color: red
    (?):
    - funky == True:
        color: blue

sources:
- kind: local
  path: subproject

Here we can see that projects can use conditional statements to make decisions about subproject configuration based on their own configuration.

In this example, if the toplevel project is funky, then it will configure its subproject with color set to blue, otherwise it will use the red variant of the subproject color.

3.2.3. Including configuration from a subproject

Here there are a couple of aspects to observe, namely how the toplevel project includes files across a junction boundary, and how that include file might be implemented.

3.2.3.1. Toplevel elements/hello.bst

kind: autotools
description: |

    Hello world example from automake

variables:

  # The special paths.bst from our subproject is used to
  # define the paths of some elements in this project.
  #
  (@): subproject-junction.bst:include/paths.bst

  # The hello world example lives in the doc/amhello folder.
  command-subdir: doc/amhello

sources:
- kind: tar
  url: gnu:automake-1.16.tar.gz
  ref: 80da43bb5665596ee389e6d8b64b4f122ea4b92a685b1dbd813cd1f0e0c2d83f

depends:
  - filename: base.bst
    junction: subproject-junction.bst

Here we can see the same element which we discussed in the autotools example, except that we’re including a file from the subproject. As explained in the reference manual, this is done by prefixing the include path with the local junction element name and then a colon.

Note that in this case, the API contract is simply that hello.bst is including paths.bst, and has the expectation that paths.bst will in some way influence the variables, nothing more.

It can be that an include file is expected to create new variables, and it can be that the subproject might declare things differently depending on the subproject’s own configuration, as we will observe next.

3.2.3.2. Subproject include/paths.bst

# When this project color is blue, including this
# file causes installations to be relocated to /opt
#
prefix: /usr
(?):
- color == "blue":
    prefix: /opt

Here, we can see the include file itself is making a conditional statement, in turn deciding what values to use depending on how the project was configured.

This decision will provide valuable context for any file including paths.bst, whether it be an element, a project.conf which applies the variable as a default for the entire project, whether it is being included by files in the local project, or whether it is being included by a downstream project which junctions this project, as is the case in this example.

3.3. Using the project

At this stage, you have probably already reasoned out what would happen if we tried to build and run the project.

Nevertheless, we will still present the outputs here for observation.

3.3.1. Building the project normally

Here we build the project without any special arguments.

user@host:~/junction-includes$ bst build hello.bst

[--:--:--][        ][    main:core activity                 ] START   Build
[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Resolving cached state
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving cached state
[--:--:--][        ][    main:core activity                 ] START   Checking sources
[00:00:00][        ][    main:core activity                 ] SUCCESS Checking sources

BuildStream Version 1.93.1+172.g03ccfcd3b
  Session Start: Friday, 17-04-2020 at 20:53:40
  Project:       subproject (/home/user/junction-includes)
  Targets:       hello.bst

User Configuration
  Configuration File:      /home/user/.config/buildstream.conf
  Cache Directory:         /home/user/.cache/buildstream
  Log Files:               /home/user/.cache/buildstream/logs
  Source Mirrors:          /home/user/.cache/buildstream/sources
  Build Area:              /home/user/.cache/buildstream/build
  Strict Build Plan:       Yes
  Maximum Fetch Tasks:     10
  Maximum Build Tasks:     4
  Maximum Push Tasks:      4
  Maximum Network Retries: 2

Project Options
  funky: 0

Pipeline
   buildable 179c6ae937e8e2ece3192ab8dc2a55053134a591c9ccd37507b56d11885fae23 subproject-junction.bst:base/alpine.bst 
     waiting a41060b8e6ebd32de32417cd4576f3f21e168787c82302df104a8c463a7d1cf0 subproject-junction.bst:base.bst 
     waiting 71d3352ef5aad1fef8d62c07c858ff7984cfe871bf9bdde6df16408e62f2f6a6 hello.bst 
===============================================================================
[--:--:--][179c6ae9][   build:subproject-junction.bst:base/alpine.bst] START   subproject/base-alpine/179c6ae9-build.1902.log
[--:--:--][179c6ae9][   build:subproject-junction.bst:base/alpine.bst] START   Staging sources
[00:00:00][179c6ae9][   build:subproject-junction.bst:base/alpine.bst] SUCCESS Staging sources
[--:--:--][179c6ae9][   build:subproject-junction.bst:base/alpine.bst] START   Caching artifact
[00:00:00][179c6ae9][   build:subproject-junction.bst:base/alpine.bst] SUCCESS Caching artifact
[00:00:00][179c6ae9][   build:subproject-junction.bst:base/alpine.bst] SUCCESS subproject/base-alpine/179c6ae9-build.1902.log
[--:--:--][a41060b8][   build:subproject-junction.bst:base.bst] START   subproject/base/a41060b8-build.1920.log
[--:--:--][a41060b8][   build:subproject-junction.bst:base.bst] START   Caching artifact
[00:00:00][a41060b8][   build:subproject-junction.bst:base.bst] SUCCESS Caching artifact
[00:00:00][a41060b8][   build:subproject-junction.bst:base.bst] SUCCESS subproject/base/a41060b8-build.1920.log
[--:--:--][71d3352e][   build:hello.bst                     ] START   subproject/hello/71d3352e-build.1940.log
[--:--:--][71d3352e][   build:hello.bst                     ] START   Staging dependencies
[00:00:00][71d3352e][   build:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][71d3352e][   build:hello.bst                     ] START   Integrating sandbox
[00:00:00][71d3352e][   build:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][71d3352e][   build:hello.bst                     ] START   Staging sources
[00:00:00][71d3352e][   build:hello.bst                     ] SUCCESS Staging sources
[--:--:--][71d3352e][   build:hello.bst                     ] START   Running configure-commands
[--:--:--][71d3352e][   build:hello.bst                     ] STATUS  Running command

    export NOCONFIGURE=1;
    
    if [ -x ./configure ]; then true;
    elif [ -x ./autogen ]; then ./autogen;
    elif [ -x ./autogen.sh ]; then ./autogen.sh;
    elif [ -x ./bootstrap ]; then ./bootstrap;
    elif [ -x ./bootstrap.sh ]; then ./bootstrap.sh;
    else autoreconf -ivf .;
    fi

[--:--:--][71d3352e][   build:hello.bst                     ] STATUS  Running command

    ./configure --prefix=/usr \
    --exec-prefix=/usr \
    --bindir=/usr/bin \
    --sbindir=/usr/sbin \
    --sysconfdir=/etc \
    --datadir=/usr/share \
    --includedir=/usr/include \
    --libdir=/usr/lib \
    --libexecdir=/usr/libexec \
    --localstatedir=/var \
    --sharedstatedir=/usr/com \
    --mandir=/usr/share/man \
    --infodir=/usr/share/info   

[00:00:03][71d3352e][   build:hello.bst                     ] SUCCESS Running configure-commands
[--:--:--][71d3352e][   build:hello.bst                     ] START   Running build-commands
[--:--:--][71d3352e][   build:hello.bst                     ] STATUS  Running command

    make

[00:00:00][71d3352e][   build:hello.bst                     ] SUCCESS Running build-commands
[--:--:--][71d3352e][   build:hello.bst                     ] START   Running install-commands
[--:--:--][71d3352e][   build:hello.bst                     ] STATUS  Running command

    make -j1 DESTDIR="/buildstream-install" install

[--:--:--][71d3352e][   build:hello.bst                     ] STATUS  Running command

    if false || false; then
      find "/buildstream-install" -name "*.la" -print0 | while read -d '' -r file; do
        if grep '^shouldnotlink=yes$' "${file}" &>/dev/null; then
          if false; then
            echo "Removing ${file}."
            rm "${file}"
          else
            echo "Not removing ${file}."
          fi
        else
          if false; then
            echo "Removing ${file}."
            rm "${file}"
          else
            echo "Not removing ${file}."
          fi
        fi
      done
    fi

[00:00:00][71d3352e][   build:hello.bst                     ] SUCCESS Running install-commands
[--:--:--][71d3352e][   build:hello.bst                     ] START   Running strip-commands
[00:00:00][71d3352e][   build:hello.bst                     ] SUCCESS Running strip-commands
[--:--:--][71d3352e][   build:hello.bst                     ] START   Caching artifact
[00:00:00][71d3352e][   build:hello.bst                     ] SUCCESS Caching artifact
[00:00:04][71d3352e][   build:hello.bst                     ] SUCCESS subproject/hello/71d3352e-build.1940.log
[00:00:05][        ][    main:core activity                 ] SUCCESS Build

Pipeline Summary
  Total:       3
  Session:     3
  Fetch Queue: processed 0, skipped 3, failed 0 
  Build Queue: processed 3, skipped 0, failed 0

3.3.2. Building the project in funky mode

Now let’s see what happens when we build the project in funky mode

user@host:~/junction-includes$ bst --option funky True build hello.bst

[--:--:--][        ][    main:core activity                 ] START   Build
[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Resolving cached state
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving cached state
[--:--:--][        ][    main:core activity                 ] START   Checking sources
[00:00:00][        ][    main:core activity                 ] SUCCESS Checking sources

BuildStream Version 1.93.1+172.g03ccfcd3b
  Session Start: Friday, 17-04-2020 at 20:53:45
  Project:       subproject (/home/user/junction-includes)
  Targets:       hello.bst

User Configuration
  Configuration File:      /home/user/.config/buildstream.conf
  Cache Directory:         /home/user/.cache/buildstream
  Log Files:               /home/user/.cache/buildstream/logs
  Source Mirrors:          /home/user/.cache/buildstream/sources
  Build Area:              /home/user/.cache/buildstream/build
  Strict Build Plan:       Yes
  Maximum Fetch Tasks:     10
  Maximum Build Tasks:     4
  Maximum Push Tasks:      4
  Maximum Network Retries: 2

Project Options
  funky: 1

Pipeline
      cached 179c6ae937e8e2ece3192ab8dc2a55053134a591c9ccd37507b56d11885fae23 subproject-junction.bst:base/alpine.bst 
      cached a41060b8e6ebd32de32417cd4576f3f21e168787c82302df104a8c463a7d1cf0 subproject-junction.bst:base.bst 
   buildable 7250d6eee607e51c1e63ee48a4c1dd28c2a775cc5c92678e85fadc9dca298147 hello.bst 
===============================================================================
[--:--:--][7250d6ee][   build:hello.bst                     ] START   subproject/hello/7250d6ee-build.3495.log
[--:--:--][7250d6ee][   build:hello.bst                     ] START   Staging dependencies
[00:00:00][7250d6ee][   build:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][7250d6ee][   build:hello.bst                     ] START   Integrating sandbox
[00:00:00][7250d6ee][   build:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][7250d6ee][   build:hello.bst                     ] START   Staging sources
[00:00:00][7250d6ee][   build:hello.bst                     ] SUCCESS Staging sources
[--:--:--][7250d6ee][   build:hello.bst                     ] START   Running configure-commands
[--:--:--][7250d6ee][   build:hello.bst                     ] STATUS  Running command

    export NOCONFIGURE=1;
    
    if [ -x ./configure ]; then true;
    elif [ -x ./autogen ]; then ./autogen;
    elif [ -x ./autogen.sh ]; then ./autogen.sh;
    elif [ -x ./bootstrap ]; then ./bootstrap;
    elif [ -x ./bootstrap.sh ]; then ./bootstrap.sh;
    else autoreconf -ivf .;
    fi

[--:--:--][7250d6ee][   build:hello.bst                     ] STATUS  Running command

    ./configure --prefix=/opt \
    --exec-prefix=/opt \
    --bindir=/opt/bin \
    --sbindir=/opt/sbin \
    --sysconfdir=/etc \
    --datadir=/opt/share \
    --includedir=/opt/include \
    --libdir=/opt/lib \
    --libexecdir=/opt/libexec \
    --localstatedir=/var \
    --sharedstatedir=/opt/com \
    --mandir=/opt/share/man \
    --infodir=/opt/share/info   

[00:00:03][7250d6ee][   build:hello.bst                     ] SUCCESS Running configure-commands
[--:--:--][7250d6ee][   build:hello.bst                     ] START   Running build-commands
[--:--:--][7250d6ee][   build:hello.bst                     ] STATUS  Running command

    make

[00:00:00][7250d6ee][   build:hello.bst                     ] SUCCESS Running build-commands
[--:--:--][7250d6ee][   build:hello.bst                     ] START   Running install-commands
[--:--:--][7250d6ee][   build:hello.bst                     ] STATUS  Running command

    make -j1 DESTDIR="/buildstream-install" install

[--:--:--][7250d6ee][   build:hello.bst                     ] STATUS  Running command

    if false || false; then
      find "/buildstream-install" -name "*.la" -print0 | while read -d '' -r file; do
        if grep '^shouldnotlink=yes$' "${file}" &>/dev/null; then
          if false; then
            echo "Removing ${file}."
            rm "${file}"
          else
            echo "Not removing ${file}."
          fi
        else
          if false; then
            echo "Removing ${file}."
            rm "${file}"
          else
            echo "Not removing ${file}."
          fi
        fi
      done
    fi

[00:00:00][7250d6ee][   build:hello.bst                     ] SUCCESS Running install-commands
[--:--:--][7250d6ee][   build:hello.bst                     ] START   Running strip-commands
[00:00:00][7250d6ee][   build:hello.bst                     ] SUCCESS Running strip-commands
[--:--:--][7250d6ee][   build:hello.bst                     ] START   Caching artifact
[00:00:00][7250d6ee][   build:hello.bst                     ] SUCCESS Caching artifact
[00:00:04][7250d6ee][   build:hello.bst                     ] SUCCESS subproject/hello/7250d6ee-build.3495.log
[00:00:04][        ][    main:core activity                 ] SUCCESS Build

Pipeline Summary
  Total:       3
  Session:     1
  Fetch Queue: processed 0, skipped 1, failed 0 
  Build Queue: processed 1, skipped 0, failed 0

As we can see, this time we’ve built the project into the /opt system prefix instead of the standard /usr prefix.

Let’s just take a step back now and summarize the process which went into this decision:

• The toplevel project.conf exposes the boolean funky option

• The toplevel junction subproject-junction.bst chooses to set the subproject color to blue when the toplevel project is funky

• The subproject include/paths.bst include file decides to set the prefix to /opt in the case that the subproject is blue

• The hello.bst includes the include/paths.bst file, in order to inherit its path configuration from the subproject

3.3.3. Running the project in both modes

user@host:~/junction-includes$ bst shell hello.bst -- /usr/bin/hello

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Resolving cached state
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving cached state
[--:--:--][        ][    main:hello.bst                     ] START   Staging dependencies
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][        ][    main:hello.bst                     ] START   Integrating sandbox
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][        ][    main:hello.bst                     ] STATUS  Running command

    /usr/bin/hello

Hello World!
This is amhello 1.0.

user@host:~/junction-includes$ bst --option funky True shell hello.bst -- /opt/bin/hello

[--:--:--][        ][    main:core activity                 ] START   Loading elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
[--:--:--][        ][    main:core activity                 ] START   Resolving elements
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving elements
[--:--:--][        ][    main:core activity                 ] START   Resolving cached state
[00:00:00][        ][    main:core activity                 ] SUCCESS Resolving cached state
[--:--:--][        ][    main:hello.bst                     ] START   Staging dependencies
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Staging dependencies
[--:--:--][        ][    main:hello.bst                     ] START   Integrating sandbox
[00:00:00][        ][    main:hello.bst                     ] SUCCESS Integrating sandbox
[--:--:--][        ][    main:hello.bst                     ] STATUS  Running command

    /opt/bin/hello

Hello World!
This is amhello 1.0.

As expected, the funky variant of the toplevel project installs the hello world program in the /opt prefix, and as such we need to call it from there.

3.4. Summary

In this chapter we’ve discussed how conditional statements and include files play an essential role in the API surface of a project, and help to provide some configurability while preserving encapsulation of the API which a project exposes.

We’ve also gone over the mechanics of how these concepts interact and presented an example which shows how project options can be used in a recursive context, and how includes can help not only to share code, but to provide context to dependent projects about how their subprojects are configured.