2. Strict mode
In this section, we will cover the usage of strict vs non-strict build plans in conjunction with workspaces, and how this can help to improve your edit/compile/test cycles.
Note
This example is distributed with BuildStream in the doc/examples/strict-mode subdirectory.
2.1. Overview
When working with BuildStream to create integrations, it is typical that you have a lot of components to build, and you frequently need to modify a component at various levels of the stack. When developing one or more applications, you might want to open a workspace and fix a bug in an application, or you might need to open a workspace on a low level shared library to fix the behavior of one or more misbehaving applications.
By default, BuildStream will always choose to be deterministic in order to produce the most correct build results as possible. As such, modifying a low level library will result in rebuilding all of it’s reverse dependencies, but this can be very time consuming and inconvenient for your edit/compile/test cycles.
This is when enabling non-strict build plans can be helpful.
To illustrate the facets of how this works, this example will present a project consisting of an application which is linked both statically and dynamically linked to a common library.
2.2. Project structure
This project is mostly based on the integration commands example, as such we will ignore large parts of this project and only focus on the elements which are of specific interest.
To illustrate the relationship of these two applications and the library, let’s briefly take a look at the underlying Makefiles which are used in this project, starting with the library and followed by both Makefiles used to build the application.
2.2.1. files/libhello/Makefile
# Sample makefile for hello library
#
.PHONY: all install
all: libhello.so libhello.a
install: all
install -d ${DESTDIR}${PREFIX}/lib
install -d ${DESTDIR}${PREFIX}/include
install -m 644 libhello.so ${DESTDIR}${PREFIX}/lib
install -m 644 libhello.a ${DESTDIR}${PREFIX}/lib
install -m 644 libhello.h ${DESTDIR}${PREFIX}/include
%.o: %.c %.h
$(CC) -c $< -o $@ -Wall
libhello.a: libhello.o
$(AR) rcs $@ $^
libhello.so: libhello.o
$(CC) -shared -o $@ $<
2.2.2. files/hello/Makefile.dynamic
# Sample makefile for hello.c
#
.PHONY: all install
all: hello
install: all
install -d ${DESTDIR}${PREFIX}/bin
install -m 755 hello ${DESTDIR}${PREFIX}/bin
hello: hello.c
$(CC) -Wall -o $@ $< -lhello
2.2.3. files/hello/Makefile.static
# Sample makefile for hello.c
#
.PHONY: all install
all: hello
install: all
install -d ${DESTDIR}${PREFIX}/bin
install -m 755 hello ${DESTDIR}${PREFIX}/bin
hello: hello.c
$(CC) -Wall -o $@ $< /usr/lib/libhello.a
As we can see, we have a library that is distributed both as the dynamic
library libhello.so and also as the static archive libhello.a.
Now let’s take a look at the two separate elements which build the application, first the dynamically linked version and then the static one.
2.2.4. elements/hello-dynamic.bst
kind: manual
description: |
The dynamically linked hello application
# Depend on the hello library
depends:
- libhello.bst
# Stage the files/hello directory for building
sources:
- kind: local
path: files/hello
# Now configure the commands to run
config:
build-commands:
- make -f Makefile.dynamic PREFIX="%{prefix}"
install-commands:
- make -f Makefile.dynamic -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
Nothing very special to observe about this hello program, just a
manual element quite similar to the one we’ve
already seen in the running commands
example.
2.2.5. elements/hello-static.bst
kind: manual
description: |
The statically linked hello application
# Depend on the hello library with the strict option
#
depends:
- filename: libhello.bst
strict: true
# Stage the files/hello directory for building
sources:
- kind: local
path: files/hello
# Now configure the commands to run
config:
build-commands:
- make -f Makefile.static PREFIX="%{prefix}"
install-commands:
- make -f Makefile.static -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
Almost the same as the dynamic element, except here we have declared the
dependency to the libhello.bst element differently: this time we have enabled
the strict option in the dependency declaration.
The side effect of setting this option is that hello-static.bst will be
rebuilt any time that libhello.bst has changed, even when
non-strict build plans have been enabled.
Tip
Some element plugins are designed to consume the content of their
dependencies entirely, and output an artifact without any transient
runtime dependencies, an example of this is the compose
element.
In cases such as compose, it is not necessary to
explicitly annotate their dependencies as strict.
It is only helpful to set the strict attribute on a
dependency declaration in the case that the
specific dependency relationship causes data to be consumed verbatim,
as is the case with static linking.
2.3. Using the project
For the sake of brevity, let’s assume that you’ve already built all of the
elements of this project, and that you want to make some changes to the
libhello.bst element, and test how it might effect the hello program.
2.3.1. Everything is already built
user@host:~/strict-mode$ bst show hello-static.bst hello-dynamic.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
cached b795a72d92c05ed0810723f5c3ad40c53bba6e8b568a9d5da22a450a11e03659 base/alpine.bst
cached e4a4926893ce660306ca9c812fd8018fa41571c27f37fff4a9e6ab592d0333fe base.bst
cached 82aeedb247804d906e44722cadc8e36e6a12f84518f78e6ed44af56c11aec5fc libhello.bst
cached 7a164ef5f2a2e36ba63d454c1ab9e18b837d327c46f091305814e46ab6c71f2f hello-static.bst
cached 200162f545a171c13e9fa2eb0bf562742d6b8cecaeadbf6a0914748dc98c4dde hello-dynamic.bst
2.3.2. Open a workspace and modify libhello.c
Now let’s open up a workspace on the hello library
user@host:~/strict-mode$ bst workspace open --directory workspace_libhello libhello.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:core activity ] START Checking sources
[00:00:00][ ][ main:core activity ] SUCCESS Checking sources
[--:--:--][ ][ main:core activity ] INFO Creating workspace for element libhello.bst
[--:--:--][ ][ main:libhello.bst ] START Staging sources to /home/user/strict-mode/workspace_libhello
[--:--:--][ ][ main:libhello.bst ] START Staging local files into CAS
[00:00:00][ ][ main:libhello.bst ] SUCCESS Staging local files into CAS
[00:00:00][ ][ main:libhello.bst ] SUCCESS Staging sources to /home/user/strict-mode/workspace_libhello
[--:--:--][ ][ main:core activity ] INFO Created a workspace for element: libhello.bst
And go ahead and make a modification like this:
--- libhello.c
+++ libhello.c
@@ -5,5 +5,5 @@
void hello(const char *person)
{
- printf("Hello %s\n", person);
+ printf("Good morning %s\n", person);
}
2.3.3. Observing hello-dynamic.bst
Let’s take a look at the bst show output for
the dynamically linked hello-dynamic.bst element.
user@host:~/strict-mode$ bst show hello-dynamic.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
cached b795a72d92c05ed0810723f5c3ad40c53bba6e8b568a9d5da22a450a11e03659 base/alpine.bst
cached e4a4926893ce660306ca9c812fd8018fa41571c27f37fff4a9e6ab592d0333fe base.bst
buildable bbf7a5de337c55b6f4d32cbe3d47ed7f39ff9649bb5ff52edd44bd7c0e36e38e libhello.bst Workspace: /home/user/strict-mode/workspace_libhello
waiting fd41ac4b5ebf2fccae28e727602f08d7ee2997fd36b96b597da183f86e2caf32 hello-dynamic.bst
As one might expect, the libhello.bst element is ready to be built
after having been modified, and the hello-dynamic.bst element is
waiting for libhello.bst to be built before it can build.
Now let’s take a look at the same elements if we pass the --no-strict
option to bst:
user@host:~/strict-mode$ bst --no-strict show hello-dynamic.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
cached b795a72d92c05ed0810723f5c3ad40c53bba6e8b568a9d5da22a450a11e03659 base/alpine.bst
cached e4a4926893ce660306ca9c812fd8018fa41571c27f37fff4a9e6ab592d0333fe base.bst
buildable 3ea19fa473f6d9f4a7e67cb3bcafd0306486abdae7e126cd49c13cfb281df882 libhello.bst Workspace: /home/user/strict-mode/workspace_libhello
cached 200162f545a171c13e9fa2eb0bf562742d6b8cecaeadbf6a0914748dc98c4dde hello-dynamic.bst
Note that this time, the libhello.bst still needs to be built,
but the hello-dymamic.bst element is showing up as cached.
Tip
The bst show output will show some cache keys dimmed out in the case that they are not entirely deterministic.
Here we can see that hello-dynamic.bst is dimmed out because
it will not be rebuilt against the changed libhello.bst element,
and it also has a different cache key because of this.
2.3.4. Observing hello-static.bst
Now let’s observe the hello-static.bst element with strict mode
disabled:
user@host:~/strict-mode$ bst --no-strict show hello-static.bst
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
cached b795a72d92c05ed0810723f5c3ad40c53bba6e8b568a9d5da22a450a11e03659 base/alpine.bst
cached e4a4926893ce660306ca9c812fd8018fa41571c27f37fff4a9e6ab592d0333fe base.bst
buildable 3ea19fa473f6d9f4a7e67cb3bcafd0306486abdae7e126cd49c13cfb281df882 libhello.bst Workspace: /home/user/strict-mode/workspace_libhello
waiting 1fe34abbad09b930e86d213b3c21507bf250baecd95effb24fd2223fc6017968 hello-static.bst
Note that in this case the hello-strict.bst is going to be
rebuilt even in strict mode. This is because we annotated the
declaration of the libhello.bst dependency with the strict
attribute.
We did this because hello-strict.bst consumes the input of
libhello.bst verbatim, by way of statically linking to it, instead
of merely being affected by the content of libhello.bst at runtime,
as would be the case of static linking.
2.3.5. Building and running hello-dynamic.bst
Now let’s build hello-dynamic.bst with strict mode disabled.
user@host:~/strict-mode$ bst --no-strict build hello-dynamic.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:core activity ] START Checking sources
[00:00:00][ ][ main:core activity ] SUCCESS Checking sources
BuildStream Version 1.93.1+172.g03ccfcd3b
Session Start: Friday, 17-04-2020 at 20:52:12
Project: strict-mode (/home/user/strict-mode)
Targets: hello-dynamic.bst
User Configuration
Configuration File: /home/user/.config/buildstream.conf
Cache Directory: /home/user/.cache/buildstream
Log Files: /home/user/.cache/buildstream/logs
Source Mirrors: /home/user/.cache/buildstream/sources
Build Area: /home/user/.cache/buildstream/build
Strict Build Plan: No
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Pipeline
cached b795a72d92c05ed0810723f5c3ad40c53bba6e8b568a9d5da22a450a11e03659 base/alpine.bst
cached e4a4926893ce660306ca9c812fd8018fa41571c27f37fff4a9e6ab592d0333fe base.bst
buildable 3ea19fa473f6d9f4a7e67cb3bcafd0306486abdae7e126cd49c13cfb281df882 libhello.bst Workspace: /home/user/strict-mode/workspace_libhello
cached 200162f545a171c13e9fa2eb0bf562742d6b8cecaeadbf6a0914748dc98c4dde hello-dynamic.bst
===============================================================================
[--:--:--][3ea19fa4][ fetch:libhello.bst ] START strict-mode/libhello/3ea19fa4-fetch.31676.log
[00:00:00][3ea19fa4][ fetch:libhello.bst ] SUCCESS strict-mode/libhello/3ea19fa4-fetch.31676.log
[--:--:--][3ea19fa4][ build:libhello.bst ] START strict-mode/libhello/3ea19fa4-build.31680.log
[--:--:--][3ea19fa4][ build:libhello.bst ] START Staging dependencies
[00:00:00][3ea19fa4][ build:libhello.bst ] SUCCESS Staging dependencies
[--:--:--][3ea19fa4][ build:libhello.bst ] START Integrating sandbox
[--:--:--][3ea19fa4][ build:libhello.bst ] STATUS Running command
ldconfig "/usr/lib"
[00:00:00][3ea19fa4][ build:libhello.bst ] SUCCESS Integrating sandbox
[--:--:--][3ea19fa4][ build:libhello.bst ] START Staging sources
[00:00:00][3ea19fa4][ build:libhello.bst ] SUCCESS Staging sources
[--:--:--][3ea19fa4][ build:libhello.bst ] START Running build-commands
[--:--:--][3ea19fa4][ build:libhello.bst ] STATUS Running command
make PREFIX="/usr"
[00:00:00][3ea19fa4][ build:libhello.bst ] SUCCESS Running build-commands
[--:--:--][3ea19fa4][ build:libhello.bst ] START Running install-commands
[--:--:--][3ea19fa4][ build:libhello.bst ] STATUS Running command
make -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][3ea19fa4][ build:libhello.bst ] SUCCESS Running install-commands
[--:--:--][3ea19fa4][ build:libhello.bst ] START Running strip-commands
[00:00:00][3ea19fa4][ build:libhello.bst ] SUCCESS Running strip-commands
[--:--:--][3ea19fa4][ build:libhello.bst ] START Caching artifact
[00:00:00][3ea19fa4][ build:libhello.bst ] SUCCESS Caching artifact
[00:00:01][3ea19fa4][ build:libhello.bst ] SUCCESS strict-mode/libhello/3ea19fa4-build.31680.log
[00:00:01][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 4
Session: 1
Fetch Queue: processed 1, skipped 0, failed 0
Build Queue: processed 1, skipped 0, failed 0
Note that the bst build command completed without
having to build hello-dynamic.bst at all.
And now we can also run hello-dynamic.bst
user@host:~/strict-mode$ bst --no-strict shell hello-dynamic.bst -- hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:hello-dynamic.bst ] START Staging dependencies
[00:00:00][ ][ main:hello-dynamic.bst ] SUCCESS Staging dependencies
[--:--:--][ ][ main:hello-dynamic.bst ] START Integrating sandbox
[--:--:--][ ][ main:hello-dynamic.bst ] STATUS Running command
ldconfig "/usr/lib"
[00:00:00][ ][ main:hello-dynamic.bst ] SUCCESS Integrating sandbox
[--:--:--][ ][ main:hello-dynamic.bst ] STATUS Running command
hello
Good morning stranger
When running hello-dynamic.bst with no-strict mode, we are
actually reusing the old build of hello-dynamic.bst staged against
the new build of the modified libhello.bst element.
2.3.6. Building and running hello-static.bst
Finally, if we build hello-static.bst with strict mode disabled,
we can see that it will be rebuilt regardless of strict mode being
enabled.
user@host:~/strict-mode$ bst --no-strict build hello-static.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:core activity ] START Checking sources
[00:00:00][ ][ main:core activity ] SUCCESS Checking sources
BuildStream Version 1.93.1+172.g03ccfcd3b
Session Start: Friday, 17-04-2020 at 20:52:15
Project: strict-mode (/home/user/strict-mode)
Targets: hello-static.bst
User Configuration
Configuration File: /home/user/.config/buildstream.conf
Cache Directory: /home/user/.cache/buildstream
Log Files: /home/user/.cache/buildstream/logs
Source Mirrors: /home/user/.cache/buildstream/sources
Build Area: /home/user/.cache/buildstream/build
Strict Build Plan: No
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Pipeline
cached b795a72d92c05ed0810723f5c3ad40c53bba6e8b568a9d5da22a450a11e03659 base/alpine.bst
cached e4a4926893ce660306ca9c812fd8018fa41571c27f37fff4a9e6ab592d0333fe base.bst
cached 3ea19fa473f6d9f4a7e67cb3bcafd0306486abdae7e126cd49c13cfb281df882 libhello.bst Workspace: /home/user/strict-mode/workspace_libhello
buildable 1fe34abbad09b930e86d213b3c21507bf250baecd95effb24fd2223fc6017968 hello-static.bst
===============================================================================
[--:--:--][1fe34abb][ build:hello-static.bst ] START strict-mode/hello-static/1fe34abb-build.31821.log
[--:--:--][1fe34abb][ build:hello-static.bst ] START Staging dependencies
[00:00:00][1fe34abb][ build:hello-static.bst ] SUCCESS Staging dependencies
[--:--:--][1fe34abb][ build:hello-static.bst ] START Integrating sandbox
[--:--:--][1fe34abb][ build:hello-static.bst ] STATUS Running command
ldconfig "/usr/lib"
[00:00:00][1fe34abb][ build:hello-static.bst ] SUCCESS Integrating sandbox
[--:--:--][1fe34abb][ build:hello-static.bst ] START Staging sources
[00:00:00][1fe34abb][ build:hello-static.bst ] SUCCESS Staging sources
[--:--:--][1fe34abb][ build:hello-static.bst ] START Running build-commands
[--:--:--][1fe34abb][ build:hello-static.bst ] STATUS Running command
make -f Makefile.static PREFIX="/usr"
[00:00:00][1fe34abb][ build:hello-static.bst ] SUCCESS Running build-commands
[--:--:--][1fe34abb][ build:hello-static.bst ] START Running install-commands
[--:--:--][1fe34abb][ build:hello-static.bst ] STATUS Running command
make -f Makefile.static -j1 PREFIX="/usr" DESTDIR="/buildstream-install" install
[00:00:00][1fe34abb][ build:hello-static.bst ] SUCCESS Running install-commands
[--:--:--][1fe34abb][ build:hello-static.bst ] START Running strip-commands
[00:00:00][1fe34abb][ build:hello-static.bst ] SUCCESS Running strip-commands
[--:--:--][1fe34abb][ build:hello-static.bst ] START Caching artifact
[00:00:00][1fe34abb][ build:hello-static.bst ] SUCCESS Caching artifact
[00:00:01][1fe34abb][ build:hello-static.bst ] SUCCESS strict-mode/hello-static/1fe34abb-build.31821.log
[00:00:01][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 4
Session: 1
Fetch Queue: processed 0, skipped 1, failed 0
Build Queue: processed 1, skipped 0, failed 0
This is of course because we declared its dependency on libhello.bst
as a strict dependency.
And by the same virtue, we can see that when we run the example it has properly relinked against the changed static archive, and has the updated text in the greeting:
user@host:~/strict-mode$ bst --no-strict shell hello-static.bst -- hello
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:hello-static.bst ] START Staging dependencies
[00:00:00][ ][ main:hello-static.bst ] SUCCESS Staging dependencies
[--:--:--][ ][ main:hello-static.bst ] START Integrating sandbox
[--:--:--][ ][ main:hello-static.bst ] STATUS Running command
ldconfig "/usr/lib"
[00:00:00][ ][ main:hello-static.bst ] SUCCESS Integrating sandbox
[--:--:--][ ][ main:hello-static.bst ] STATUS Running command
hello
Good morning stranger
2.4. Summary
In this chapter we’ve explored how to use non-strict build plans in order to avoid rebuilding reverse dependencies of a lower level element you might be working with in a workspace, consequently improving your edit/compile/test experience.
We’ve also explained how to ensure your project still works properly
with non-strict build plans when some elements perform static linking
(or other operations which consume data from their dependencies
verbatim), by annotating dependency declarations
as strict.