3. Overlapping files
In this chapter, we will discuss what happens when files from multiple element artifacts conflict with eachother, and what strategies we can use to resolve these situations.
Note
This example is distributed with BuildStream in the doc/examples/overlaps subdirectory.
3.1. Overview
This project builds on the previous chapter on composition, and as such we’ll only go over what has changed from there, which is not much.
3.2. Project structure
In this example we’ve just extended the libhello.bst and the hello.bst
elements such that they both install an additional file: %{docdir}/hello.txt.
We’ve updated the following Makefiles:
3.2.1. files/libhello/Makefile
# Sample makefile for hello library
#
.PHONY: all install
all: libhello.so
install:
install -d ${DESTDIR}${PREFIX}/lib
install -d ${DESTDIR}${PREFIX}/include
install -d ${DESTDIR}${PREFIX}/share/doc
install -m 644 libhello.so ${DESTDIR}${PREFIX}/lib
install -m 644 libhello.h ${DESTDIR}${PREFIX}/include
install -m 644 hello.txt ${DESTDIR}${PREFIX}/share/doc
%.o: %.c %.h
$(CC) -c $< -o $@ -Wall
libhello.so: libhello.o
$(CC) -shared -o $@ $<
3.2.2. files/hello/Makefile
# Sample makefile for hello.c
#
.PHONY: all install
all: hello
install:
install -d ${DESTDIR}${PREFIX}/bin
install -d ${DESTDIR}${PREFIX}/share/doc
install -m 755 hello ${DESTDIR}${PREFIX}/bin
install -m 644 hello.txt ${DESTDIR}${PREFIX}/share/doc
hello: hello.c
$(CC) $< -o $@ -Wall -lhello
As you can see, this now presents a conflict of sorts, where multiple elements in the pipeline want to install the same file.
3.3. Using the project
In this chapter, we’re only going to present the warning and then discuss how to mitigate this situation.
See what happens when we try to build the runtime-only.bst
compose element:
user@host:~/overlaps$ bst build runtime-only.bst
[--:--:--][ ][ main:core activity ] START Build
[--:--:--][ ][ main:core activity ] START Loading elements
[00:00:00][ ][ main:core activity ] SUCCESS Loading elements
[--:--:--][ ][ main:core activity ] START Resolving elements
[00:00:00][ ][ main:core activity ] SUCCESS Resolving elements
[--:--:--][ ][ main:core activity ] START Resolving cached state
[00:00:00][ ][ main:core activity ] SUCCESS Resolving cached state
[--:--:--][ ][ main:core activity ] START Checking sources
[00:00:00][ ][ main:core activity ] SUCCESS Checking sources
BuildStream Version 1.93.1+172.g03ccfcd3b
Session Start: Friday, 17-04-2020 at 20:53:26
Project: overlaps (/home/user/overlaps)
Targets: runtime-only.bst
User Configuration
Configuration File: /home/user/.config/buildstream.conf
Cache Directory: /home/user/.cache/buildstream
Log Files: /home/user/.cache/buildstream/logs
Source Mirrors: /home/user/.cache/buildstream/sources
Build Area: /home/user/.cache/buildstream/build
Strict Build Plan: Yes
Maximum Fetch Tasks: 10
Maximum Build Tasks: 4
Maximum Push Tasks: 4
Maximum Network Retries: 2
Pipeline
cached afe1fb0c366992c006f58c49017f0b48031f6f397a1e2f0e73756d619de61ffc base/alpine.bst
cached 14a4e96021cd12177192902ea8a2ac5354aa358e2bf9e4d80c1e25639028ca0a base.bst
cached e2b5f713391cfd8a76593cdd689e3b1004ba649f50635ba615f0c865fe4c953a libhello.bst
cached 26037d070846c5adb7ecba45fb5b01b4d97e0535fdd03fba81a3cc86ff30e0e4 hello.bst
buildable af1fc1febe3cbed1ac5a71e48cd048dd6dc919f0a8e4658038480cd526dc001b runtime-only.bst
===============================================================================
[--:--:--][af1fc1fe][ build:runtime-only.bst ] START overlaps/runtime-only/af1fc1fe-build.1682.log
[--:--:--][af1fc1fe][ build:runtime-only.bst ] START Staging dependencies
[--:--:--][af1fc1fe][ build:runtime-only.bst ] WARNING [overlaps]: Non-whitelisted overlaps detected
Staged files overwrite existing files in staging area:
/usr/share/doc/hello.txt: hello.bst is not permitted to overlap other elements, order hello.bst above libhello.bst
[00:00:00][af1fc1fe][ build:runtime-only.bst ] SUCCESS Staging dependencies
[--:--:--][af1fc1fe][ build:runtime-only.bst ] START Computing split
[00:00:00][af1fc1fe][ build:runtime-only.bst ] SUCCESS Computing split
[--:--:--][af1fc1fe][ build:runtime-only.bst ] START Integrating sandbox
[--:--:--][af1fc1fe][ build:runtime-only.bst ] STATUS Running command
ldconfig "/usr/lib"
[--:--:--][af1fc1fe][ build:runtime-only.bst ] INFO Integration modified 367, added 0 and removed 0 files
[00:00:00][af1fc1fe][ build:runtime-only.bst ] SUCCESS Integrating sandbox
[--:--:--][af1fc1fe][ build:runtime-only.bst ] START Creating composition
Including files from domains: runtime
Excluding orphaned files
[--:--:--][af1fc1fe][ build:runtime-only.bst ] INFO Composing 287 files
[00:00:00][af1fc1fe][ build:runtime-only.bst ] SUCCESS Creating composition
[--:--:--][af1fc1fe][ build:runtime-only.bst ] START Caching artifact
[00:00:00][af1fc1fe][ build:runtime-only.bst ] SUCCESS Caching artifact
[00:00:01][af1fc1fe][ build:runtime-only.bst ] SUCCESS overlaps/runtime-only/af1fc1fe-build.1682.log
[00:00:01][ ][ main:core activity ] SUCCESS Build
Pipeline Summary
Total: 5
Session: 1
Fetch Queue: processed 0, skipped 1, failed 0
Build Queue: processed 1, skipped 0, failed 0
Notice the warning message about the conflicting file, it is there to inform the user about which files are overlapping, and also which elements are being staged in which order.
Note also that BuildStream does not discover the overlap until the moment that you build a reverse dependency which will require staging of both artifacts.
Tip
The overlaps warning discussed here can be configured to be
a fatal warning. This is useful
in the case that you want to be strict about avoiding overlapping
files in your project.
3.4. Mitigating overlapping files
Since we recently discussed filtering of artifacts,
we should note that it is of course possible to handle this case by
having hello.bst depend on a filtered version of libhello.bst with the
offending file excluded.
However, working with filter elements just for the
sake of handling a conflicting artifact would be quite inconvenient, so we
have other means.
3.4.1. Whitelisting overlapping files
BuildStream allows explicitly ignoring such errors by adding the files
to an overlap whitelist, you could achieve
this in the given example by adding the following to the hello.bst element:
public:
bst:
overlap-whitelist:
- |
%{docdir}/hello.txt
Note
Note that glob patterns are also
supported in the whitelist.
3.4.2. Artifact munging
Another way around this situation is the munge the artifacts at install time such that there is no conflict.
This is the easiest approach in the case that you might want to keep the
underlying %{docdir}/hello.txt from libhello.bst and discard the
same file from hello.bst.
In this case, we might modify the hello.bst file so that it’s install
command contain an rm statement, as such:
install-commands:
- make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install
- |
# Rid ourselves of the unwanted file at install time
rm -f %{install-root}%{docdir}/hello.txt
This would cause later builds of runtime-only.bst to no longer
conflict on the given file.
3.5. Summary
In this chapter we’ve presented a situation where an artifact can conflict with another artifact by way of providing the same files.
We’ve presented the overlap whitelist public data which is the typical solution for silencing the error when the outcome is desired, and also presented a strategy to deal with cases where you want to keep files from the overlapped artifact instead.