dev-manual, ref-manual: Moved build history info to dev-manual

Fixes [YOCTO #12370]

The section in the ref-manual on build history has been moved to
the dev-manual.  It is more of a "how-to" piece of information than
a reference.

(From yocto-docs rev: 9634bd8dc51e2972e6a5f3a3d3b4256c8ca8749c)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Scott Rifenbark 2018-01-23 11:19:45 -08:00 committed by Richard Purdie
parent c6f8814461
commit 355103f8cf
9 changed files with 576 additions and 515 deletions

View File

@ -146,8 +146,8 @@ TARFILES = dev-style.css dev-manual.html \
figures/source-repos.png figures/yp-download.png \
figures/wip.png
else
TARFILES = dev-style.css dev-manual.html \
figures/dev-title.png \
TARFILES = dev-style.css dev-manual.html figures/buildhistory-web.png \
figures/dev-title.png figures/buildhistory.png \
figures/recipe-workflow.png figures/bitbake-build-flow.png \
eclipse
endif
@ -281,7 +281,6 @@ ifeq ($(DOC),ref-manual)
XSLTOPTS = --xinclude
ALLPREQ = html eclipse tarball
TARFILES = ref-manual.html ref-style.css figures/poky-title.png \
figures/buildhistory.png figures/buildhistory-web.png \
figures/build-workspace-directory.png \
eclipse
MANUALS = $(DOC)/$(DOC).html $(DOC)/eclipse

View File

@ -7432,8 +7432,8 @@ Some notes from Cal:
BUILDHISTORY_COMMIT = "1"
</literallayout>
For information on build history, see the
"<ulink url='&YOCTO_DOCS_REF_URL;#maintaining-build-output-quality'>Maintaining Build Output Quality</ulink>"
section in the Yocto Project Reference Manual.
"<link linkend='maintaining-build-output-quality'>Maintaining Build Output Quality</link>"
section.
</para>
<note>
@ -9089,6 +9089,566 @@ Some notes from Cal:
</section>
</section>
<section id='maintaining-build-output-quality'>
<title>Maintaining Build Output Quality</title>
<para>
Many factors can influence the quality of a build.
For example, if you upgrade a recipe to use a new version of an
upstream software package or you experiment with some new
configuration options, subtle changes can occur that you might
not detect until later.
Consider the case where your recipe is using a newer version of
an upstream package.
In this case, a new version of a piece of software might
introduce an optional dependency on another library, which is
auto-detected.
If that library has already been built when the software is
building, the software will link to the built library and that
library will be pulled into your image along with the new
software even if you did not want the library.
</para>
<para>
The
<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-buildhistory'><filename>buildhistory</filename></ulink>
class exists to help you maintain the quality of your build
output.
You can use the class to highlight unexpected and possibly
unwanted changes in the build output.
When you enable build history, it records information about the
contents of each package and image and then commits that
information to a local Git repository where you can examine
the information.
</para>
<para>
The remainder of this section describes the following:
<itemizedlist>
<listitem><para>
How you can enable and disable build history
</para></listitem>
<listitem><para>
How to understand what the build history contains
</para></listitem>
<listitem><para>
How to limit the information used for build history
</para></listitem>
<listitem><para>
How to examine the build history from both a
command-line and web interface
</para></listitem>
</itemizedlist>
</para>
<section id='enabling-and-disabling-build-history'>
<title>Enabling and Disabling Build History</title>
<para>
Build history is disabled by default.
To enable it, add the following <filename>INHERIT</filename>
statement and set the
<ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_COMMIT'><filename>BUILDHISTORY_COMMIT</filename></ulink>
variable to "1" at the end of your
<filename>conf/local.conf</filename> file found in the
<ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>:
<literallayout class='monospaced'>
INHERIT += "buildhistory"
BUILDHISTORY_COMMIT = "1"
</literallayout>
Enabling build history as previously described causes the
OpenEmbedded build system to collect build output information
and commit it as a single commit to a local
<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#git'>Git</ulink>
repository.
<note>
Enabling build history increases your build times slightly,
particularly for images, and increases the amount of disk
space used during the build.
</note>
</para>
<para>
You can disable build history by removing the previous
statements from your <filename>conf/local.conf</filename>
file.
</para>
</section>
<section id='understanding-what-the-build-history-contains'>
<title>Understanding What the Build History Contains</title>
<para>
Build history information is kept in
<filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-TOPDIR'><filename>TOPDIR</filename></ulink><filename>}/buildhistory</filename>
in the Build Directory as defined by the
<ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_DIR'><filename>BUILDHISTORY_DIR</filename></ulink>
variable.
The following is an example abbreviated listing:
<imagedata fileref="figures/buildhistory.png" align="center" width="6in" depth="4in" />
</para>
<para>
At the top level, a <filename>metadata-revs</filename>
file exists that lists the revisions of the repositories for
the enabled layers when the build was produced.
The rest of the data splits into separate
<filename>packages</filename>, <filename>images</filename>
and <filename>sdk</filename> directories, the contents of
which are described as follows.
</para>
<section id='build-history-package-information'>
<title>Build History Package Information</title>
<para>
The history for each package contains a text file that has
name-value pairs with information about the package.
For example,
<filename>buildhistory/packages/i586-poky-linux/busybox/busybox/latest</filename>
contains the following:
<literallayout class='monospaced'>
PV = 1.22.1
PR = r32
RPROVIDES =
RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
PKGSIZE = 540168
FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
/etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
/usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
/usr/share/pixmaps /usr/share/applications /usr/share/idl \
/usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
/etc/busybox.links.nosuid /etc/busybox.links.suid
</literallayout>
Most of these name-value pairs correspond to variables
used to produce the package.
The exceptions are <filename>FILELIST</filename>, which
is the actual list of files in the package, and
<filename>PKGSIZE</filename>, which is the total size of
files in the package in bytes.
</para>
<para>
A file also exists that corresponds to the recipe from
which the package came (e.g.
<filename>buildhistory/packages/i586-poky-linux/busybox/latest</filename>):
<literallayout class='monospaced'>
PV = 1.22.1
PR = r32
DEPENDS = initscripts kern-tools-native update-rc.d-native \
virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
virtual/libc virtual/update-alternatives
PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
</literallayout>
</para>
<para>
Finally, for those recipes fetched from a version control
system (e.g., Git), a file exists that lists source
revisions that are specified in the recipe and lists
the actual revisions used during the build.
Listed and actual revisions might differ when
<ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink>
is set to
${<ulink url='&YOCTO_DOCS_REF_URL;#var-AUTOREV'><filename>AUTOREV</filename></ulink>}.
Here is an example assuming
<filename>buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev</filename>):
<literallayout class='monospaced'>
# SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
# SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
</literallayout>
You can use the
<filename>buildhistory-collect-srcrevs</filename>
command with the <filename>-a</filename> option to
collect the stored <filename>SRCREV</filename> values
from build history and report them in a format suitable for
use in global configuration (e.g.,
<filename>local.conf</filename> or a distro include file)
to override floating <filename>AUTOREV</filename> values
to a fixed set of revisions.
Here is some example output from this command:
<literallayout class='monospaced'>
$ buildhistory-collect-srcrevs -a
# i586-poky-linux
SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072"
SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072"
SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a"
SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
# x86_64-linux
SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa"
SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf"
SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11"
SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072"
SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3"
SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca"
SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a"
SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff"
SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
# qemux86-poky-linux
SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f"
# all-poky-linux
SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11"
</literallayout>
<note>
Here are some notes on using the
<filename>buildhistory-collect-srcrevs</filename>
command:
<itemizedlist>
<listitem><para>
By default, only values where the
<filename>SRCREV</filename> was not hardcoded
(usually when <filename>AUTOREV</filename>
is used) are reported.
Use the <filename>-a</filename> option to
see all <filename>SRCREV</filename> values.
</para></listitem>
<listitem><para>
The output statements might not have any effect
if overrides are applied elsewhere in the
build system configuration.
Use the <filename>-f</filename> option to add
the <filename>forcevariable</filename> override
to each output line if you need to work around
this restriction.
</para></listitem>
<listitem><para>
The script does apply special handling when
building for multiple machines.
However, the script does place a comment before
each set of values that specifies which
triplet to which they belong as previously
shown (e.g.,
<filename>i586-poky-linux</filename>).
</para></listitem>
</itemizedlist>
</note>
</para>
</section>
<section id='build-history-image-information'>
<title>Build History Image Information</title>
<para>
The files produced for each image are as follows:
<itemizedlist>
<listitem><para>
<filename>image-files:</filename>
A directory containing selected files from the root
filesystem.
The files are defined by
<ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_IMAGE_FILES'><filename>BUILDHISTORY_IMAGE_FILES</filename></ulink>.
</para></listitem>
<listitem><para>
<filename>build-id.txt:</filename>
Human-readable information about the build
configuration and metadata source revisions.
This file contains the full build header as printed
by BitBake.
</para></listitem>
<listitem><para>
<filename>*.dot:</filename>
Dependency graphs for the image that are
compatible with <filename>graphviz</filename>.
</para></listitem>
<listitem><para>
<filename>files-in-image.txt:</filename>
A list of files in the image with permissions,
owner, group, size, and symlink information.
</para></listitem>
<listitem><para>
<filename>image-info.txt:</filename>
A text file containing name-value pairs with
information about the image.
See the following listing example for more
information.
</para></listitem>
<listitem><para>
<filename>installed-package-names.txt:</filename>
A list of installed packages by name only.
</para></listitem>
<listitem><para>
<filename>installed-package-sizes.txt:</filename>
A list of installed packages ordered by size.
</para></listitem>
<listitem><para>
<filename>installed-packages.txt:</filename>
A list of installed packages with full package
filenames.
</para></listitem>
</itemizedlist>
<note>
Installed package information is able to be gathered
and produced even if package management is disabled
for the final image.
</note>
</para>
<para>
Here is an example of <filename>image-info.txt</filename>:
<literallayout class='monospaced'>
DISTRO = poky
DISTRO_VERSION = 1.7
USER_CLASSES = buildstats image-mklibs image-prelink
IMAGE_CLASSES = image_types
IMAGE_FEATURES = debug-tweaks
IMAGE_LINGUAS =
IMAGE_INSTALL = packagegroup-core-boot run-postinsts
BAD_RECOMMENDATIONS =
NO_RECOMMENDATIONS =
PACKAGE_EXCLUDE =
ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \
write_image_manifest ; buildhistory_list_installed_image ; \
buildhistory_get_image_installed ; ssh_allow_empty_password; \
postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ;
IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
IMAGESIZE = 6900
</literallayout>
Other than <filename>IMAGESIZE</filename>, which is the
total size of the files in the image in Kbytes, the
name-value pairs are variables that may have influenced the
content of the image.
This information is often useful when you are trying to
determine why a change in the package or file
listings has occurred.
</para>
</section>
<section id='using-build-history-to-gather-image-information-only'>
<title>Using Build History to Gather Image Information Only</title>
<para>
As you can see, build history produces image information,
including dependency graphs, so you can see why something
was pulled into the image.
If you are just interested in this information and not
interested in collecting specific package or SDK
information, you can enable writing only image information
without any history by adding the following to your
<filename>conf/local.conf</filename> file found in the
<ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>:
<literallayout class='monospaced'>
INHERIT += "buildhistory"
BUILDHISTORY_COMMIT = "0"
BUILDHISTORY_FEATURES = "image"
</literallayout>
Here, you set the
<ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_FEATURES'><filename>BUILDHISTORY_FEATURES</filename></ulink>
variable to use the image feature only.
</para>
</section>
<section id='build-history-sdk-information'>
<title>Build History SDK Information</title>
<para>
Build history collects similar information on the contents
of SDKs
(e.g. <filename>bitbake -c populate_sdk imagename</filename>)
as compared to information it collects for images.
Furthermore, this information differs depending on whether
an extensible or standard SDK is being produced.
</para>
<para>
The following list shows the files produced for SDKs:
<itemizedlist>
<listitem><para>
<filename>files-in-sdk.txt:</filename>
A list of files in the SDK with permissions,
owner, group, size, and symlink information.
This list includes both the host and target parts
of the SDK.
</para></listitem>
<listitem><para>
<filename>sdk-info.txt:</filename>
A text file containing name-value pairs with
information about the SDK.
See the following listing example for more
information.
</para></listitem>
<listitem><para>
<filename>sstate-task-sizes.txt:</filename>
A text file containing name-value pairs with
information about task group sizes
(e.g. <filename>do_populate_sysroot</filename>
tasks have a total size).
The <filename>sstate-task-sizes.txt</filename> file
exists only when an extensible SDK is created.
</para></listitem>
<listitem><para>
<filename>sstate-package-sizes.txt:</filename>
A text file containing name-value pairs with
information for the shared-state packages and
sizes in the SDK.
The <filename>sstate-package-sizes.txt</filename>
file exists only when an extensible SDK is created.
</para></listitem>
<listitem><para>
<filename>sdk-files:</filename>
A folder that contains copies of the files
mentioned in
<filename>BUILDHISTORY_SDK_FILES</filename> if the
files are present in the output.
Additionally, the default value of
<filename>BUILDHISTORY_SDK_FILES</filename> is
specific to the extensible SDK although you can
set it differently if you would like to pull in
specific files from the standard SDK.</para>
<para>The default files are
<filename>conf/local.conf</filename>,
<filename>conf/bblayers.conf</filename>,
<filename>conf/auto.conf</filename>,
<filename>conf/locked-sigs.inc</filename>, and
<filename>conf/devtool.conf</filename>.
Thus, for an extensible SDK, these files get
copied into the <filename>sdk-files</filename>
directory.
</para></listitem>
<listitem><para>
The following information appears under
each of the <filename>host</filename>
and <filename>target</filename> directories
for the portions of the SDK that run on the host
and on the target, respectively:
<note>
The following files for the most part are empty
when producing an extensible SDK because this
type of SDK is not constructed from packages
as is the standard SDK.
</note>
<itemizedlist>
<listitem><para>
<filename>depends.dot:</filename>
Dependency graph for the SDK that is
compatible with
<filename>graphviz</filename>.
</para></listitem>
<listitem><para>
<filename>installed-package-names.txt:</filename>
A list of installed packages by name only.
</para></listitem>
<listitem><para>
<filename>installed-package-sizes.txt:</filename>
A list of installed packages ordered by size.
</para></listitem>
<listitem><para>
<filename>installed-packages.txt:</filename>
A list of installed packages with full
package filenames.
</para></listitem>
</itemizedlist>
</para></listitem>
</itemizedlist>
</para>
<para>
Here is an example of <filename>sdk-info.txt</filename>:
<literallayout class='monospaced'>
DISTRO = poky
DISTRO_VERSION = 1.3+snapshot-20130327
SDK_NAME = poky-glibc-i686-arm
SDK_VERSION = 1.3+snapshot
SDKMACHINE =
SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
BAD_RECOMMENDATIONS =
SDKSIZE = 352712
</literallayout>
Other than <filename>SDKSIZE</filename>, which is the
total size of the files in the SDK in Kbytes, the
name-value pairs are variables that might have influenced
the content of the SDK.
This information is often useful when you are trying to
determine why a change in the package or file listings
has occurred.
</para>
</section>
<section id='examining-build-history-information'>
<title>Examining Build History Information</title>
<para>
You can examine build history output from the command
line or from a web interface.
</para>
<para>
To see any changes that have occurred (assuming you have
<ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_COMMIT'><filename>BUILDHISTORY_COMMIT</filename></ulink><filename>&nbsp;= "1"</filename>),
you can simply use any Git command that allows you to
view the history of a repository.
Here is one method:
<literallayout class='monospaced'>
$ git log -p
</literallayout>
You need to realize, however, that this method does show
changes that are not significant (e.g. a package's size
changing by a few bytes).
</para>
<para>
A command-line tool called
<filename>buildhistory-diff</filename> does exist, though,
that queries the Git repository and prints just the
differences that might be significant in human-readable
form.
Here is an example:
<literallayout class='monospaced'>
$ ~/poky/poky/scripts/buildhistory-diff . HEAD^
Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
/etc/anotherpkg.conf was added
/sbin/anotherpkg was added
* (installed-package-names.txt):
* anotherpkg was added
Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
anotherpkg was added
packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
* PR changed from "r0" to "r1"
* PV changed from "0.1.10" to "0.1.12"
packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
* PR changed from "r0" to "r1"
* PV changed from "0.1.10" to "0.1.12"
</literallayout>
<note>
The <filename>buildhistory-diff</filename> tool
requires the <filename>GitPython</filename> package.
Be sure to install it using Pip3 as follows:
<literallayout class='monospaced'>
$ pip3 install GitPython --user
</literallayout>
Alternatively, you can install
<filename>python3-git</filename> using the appropriate
distribution package manager (e.g.
<filename>apt-get</filename>, <filename>dnf</filename>,
or <filename>zipper</filename>).
</note>
</para>
<para>
To see changes to the build history using a web interface,
follow the instruction in the <filename>README</filename>
file here.
<ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/buildhistory-web/'></ulink>.
</para>
<para>
Here is a sample screenshot of the interface:
<imagedata fileref="figures/buildhistory-web.png" align="center" scalefit="1" width="130%" contentdepth="130%" />
</para>
</section>
</section>
</section>
<section id="performing-automated-runtime-testing">
<title>Performing Automated Runtime Testing</title>

View File

Before

Width:  |  Height:  |  Size: 49 KiB

After

Width:  |  Height:  |  Size: 49 KiB

View File

Before

Width:  |  Height:  |  Size: 44 KiB

After

Width:  |  Height:  |  Size: 44 KiB

View File

@ -1022,8 +1022,8 @@
</para></listitem>
</itemizedlist>
For more information on Build History, see the
"<link linkend='maintaining-build-output-quality'>Maintaining Build Output Quality</link>"
section.
"<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-build-output-quality'>Maintaining Build Output Quality</ulink>"
section in the Yocto Project Development Tasks Manual.
</para>
</section>
@ -2026,8 +2026,8 @@
You should manually remove old "build-id" files from your
existing build history repositories to avoid confusion.
For information on the build history feature, see the
"<link linkend='maintaining-build-output-quality'>Maintaining Build Output Quality</link>"
section.
"<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-build-output-quality'>Maintaining Build Output Quality</ulink>"
section in the Yocto Project Development Tasks Manual.
</para></listitem>
</itemizedlist>
</para>

View File

@ -357,8 +357,8 @@
history of build output metadata, which can be used to detect possible
regressions as well as used for analysis of the build output.
For more information on using Build History, see the
"<link linkend='maintaining-build-output-quality'>Maintaining Build Output Quality</link>"
section.
"<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-build-output-quality'>Maintaining Build Output Quality</ulink>"
section in the Yocto Project Development Tasks Manual.
</para>
</section>

View File

@ -306,8 +306,8 @@
The directory tracks build information into image, packages, and
SDK subdirectories.
For information on the build history feature, see the
"<link linkend='maintaining-build-output-quality'>Maintaining Build Output Quality</link>"
section.
"<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-build-output-quality'>Maintaining Build Output Quality</ulink>"
section in the Yocto Project Development Tasks Manual.
</para>
</section>

View File

@ -1923,8 +1923,8 @@
class, this variable specifies the build history features
to be enabled.
For more information on how build history works, see the
"<link linkend='maintaining-build-output-quality'>Maintaining Build Output Quality</link>"
section.
"<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-build-output-quality'>Maintaining Build Output Quality</ulink>"
section in the Yocto Project Development Tasks Manual.
</para>
<para>

View File

@ -11,511 +11,13 @@
documentation set provide more details on how to use the Yocto Project.
</para>
<section id='maintaining-build-output-quality'>
<title>Maintaining Build Output Quality</title>
<para>
Many factors can influence the quality of a build.
For example, if you upgrade a recipe to use a new version of an upstream software
package or you experiment with some new configuration options, subtle changes
can occur that you might not detect until later.
Consider the case where your recipe is using a newer version of an upstream package.
In this case, a new version of a piece of software might introduce an optional
dependency on another library, which is auto-detected.
If that library has already been built when the software is building,
the software will link to the built library and that library will be pulled
into your image along with the new software even if you did not want the
library.
</para>
<para>
The
<link linkend='ref-classes-buildhistory'><filename>buildhistory</filename></link>
class exists to help you maintain
the quality of your build output.
You can use the class to highlight unexpected and possibly unwanted
changes in the build output.
When you enable build history, it records information about the contents of
each package and image and then commits that information to a local Git
repository where you can examine the information.
</para>
OLD START WAS HERE.
<para>
The remainder of this section describes the following:
<itemizedlist>
<listitem><para>How you can enable and disable
build history</para></listitem>
<listitem><para>How to understand what the build history contains
</para></listitem>
<listitem><para>How to limit the information used for build history
</para></listitem>
<listitem><para>How to examine the build history from both a
command-line and web interface</para></listitem>
</itemizedlist>
</para>
<section id='enabling-and-disabling-build-history'>
<title>Enabling and Disabling Build History</title>
OLD END WAS HERE.
<para>
Build history is disabled by default.
To enable it, add the following <filename>INHERIT</filename>
statement and set the
<link linkend='var-BUILDHISTORY_COMMIT'><filename>BUILDHISTORY_COMMIT</filename></link>
variable to "1" at the end of your
<filename>conf/local.conf</filename> file found in the
<link linkend='build-directory'>Build Directory</link>:
<literallayout class='monospaced'>
INHERIT += "buildhistory"
BUILDHISTORY_COMMIT = "1"
</literallayout>
Enabling build history as previously described causes the
OpenEmbedded build system to collect build output information and
commit it as a single commit to a local
<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#git'>Git</ulink> repository.
<note>
Enabling build history increases your build times slightly,
particularly for images, and increases the amount of disk
space used during the build.
</note>
</para>
<para>
You can disable build history by removing the previous statements
from your <filename>conf/local.conf</filename> file.
</para>
</section>
<section id='understanding-what-the-build-history-contains'>
<title>Understanding What the Build History Contains</title>
<para>
Build history information is kept in
<filename>${</filename><link linkend='var-TOPDIR'><filename>TOPDIR</filename></link><filename>}/buildhistory</filename>
in the Build Directory as defined by the
<link linkend='var-BUILDHISTORY_DIR'><filename>BUILDHISTORY_DIR</filename></link>
variable.
The following is an example abbreviated listing:
<imagedata fileref="figures/buildhistory.png" align="center" width="6in" depth="4in" />
</para>
<para>
At the top level, there is a <filename>metadata-revs</filename> file
that lists the revisions of the repositories for the layers enabled
when the build was produced.
The rest of the data splits into separate
<filename>packages</filename>, <filename>images</filename> and
<filename>sdk</filename> directories, the contents of which are
described below.
</para>
<section id='build-history-package-information'>
<title>Build History Package Information</title>
<para>
The history for each package contains a text file that has
name-value pairs with information about the package.
For example, <filename>buildhistory/packages/i586-poky-linux/busybox/busybox/latest</filename>
contains the following:
<literallayout class='monospaced'>
PV = 1.22.1
PR = r32
RPROVIDES =
RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
PKGSIZE = 540168
FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
/etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
/usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
/usr/share/pixmaps /usr/share/applications /usr/share/idl \
/usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
/etc/busybox.links.nosuid /etc/busybox.links.suid
</literallayout>
Most of these name-value pairs correspond to variables used
to produce the package.
The exceptions are <filename>FILELIST</filename>, which is the
actual list of files in the package, and
<filename>PKGSIZE</filename>, which is the total size of files
in the package in bytes.
</para>
<para>
There is also a file corresponding to the recipe from which the
package came (e.g.
<filename>buildhistory/packages/i586-poky-linux/busybox/latest</filename>):
<literallayout class='monospaced'>
PV = 1.22.1
PR = r32
DEPENDS = initscripts kern-tools-native update-rc.d-native \
virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
virtual/libc virtual/update-alternatives
PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
</literallayout>
</para>
<para>
Finally, for those recipes fetched from a version control
system (e.g., Git), a file exists that lists source revisions
that are specified in the recipe and lists the actual revisions
used during the build.
Listed and actual revisions might differ when
<link linkend='var-SRCREV'><filename>SRCREV</filename></link>
is set to
<filename>${<link linkend='var-AUTOREV'>AUTOREV</link>}</filename>.
Here is an example assuming
<filename>buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev</filename>):
<literallayout class='monospaced'>
# SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
# SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
</literallayout>
You can use the <filename>buildhistory-collect-srcrevs</filename>
command with the <filename>-a</filename> option to
collect the stored <filename>SRCREV</filename> values
from build history and report them in a format suitable for
use in global configuration (e.g.,
<filename>local.conf</filename> or a distro include file) to
override floating <filename>AUTOREV</filename> values to a
fixed set of revisions.
Here is some example output from this command:
<literallayout class='monospaced'>
$ buildhistory-collect-srcrevs -a
# i586-poky-linux
SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072"
SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072"
SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a"
SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
# x86_64-linux
SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa"
SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf"
SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11"
SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072"
SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3"
SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca"
SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a"
SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff"
SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
# qemux86-poky-linux
SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f"
# all-poky-linux
SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11"
</literallayout>
<note>
Here are some notes on using the
<filename>buildhistory-collect-srcrevs</filename> command:
<itemizedlist>
<listitem><para>By default, only values where the
<filename>SRCREV</filename> was
not hardcoded (usually when <filename>AUTOREV</filename>
was used) are reported.
Use the <filename>-a</filename> option to see all
<filename>SRCREV</filename> values.
</para></listitem>
<listitem><para>The output statements might not have any effect
if overrides are applied elsewhere in the build system
configuration.
Use the <filename>-f</filename> option to add the
<filename>forcevariable</filename> override to each output line
if you need to work around this restriction.
</para></listitem>
<listitem><para>The script does apply special handling when
building for multiple machines.
However, the script does place a
comment before each set of values that specifies
which triplet to which they belong as shown above
(e.g., <filename>i586-poky-linux</filename>).
</para></listitem>
</itemizedlist>
</note>
</para>
</section>
<section id='build-history-image-information'>
<title>Build History Image Information</title>
<para>
The files produced for each image are as follows:
<itemizedlist>
<listitem><para><filename>image-files:</filename>
A directory containing selected files from the root
filesystem.
The files are defined by
<link linkend='var-BUILDHISTORY_IMAGE_FILES'><filename>BUILDHISTORY_IMAGE_FILES</filename></link>.
</para></listitem>
<listitem><para><filename>build-id.txt:</filename>
Human-readable information about the build configuration
and metadata source revisions.
This file contains the full build header as printed
by BitBake.</para></listitem>
<listitem><para><filename>*.dot:</filename>
Dependency graphs for the image that are
compatible with <filename>graphviz</filename>.
</para></listitem>
<listitem><para><filename>files-in-image.txt:</filename>
A list of files in the image with permissions,
owner, group, size, and symlink information.
</para></listitem>
<listitem><para><filename>image-info.txt:</filename>
A text file containing name-value pairs with information
about the image.
See the following listing example for more information.
</para></listitem>
<listitem><para><filename>installed-package-names.txt:</filename>
A list of installed packages by name only.</para></listitem>
<listitem><para><filename>installed-package-sizes.txt:</filename>
A list of installed packages ordered by size.
</para></listitem>
<listitem><para><filename>installed-packages.txt:</filename>
A list of installed packages with full package
filenames.</para></listitem>
</itemizedlist>
<note>
Installed package information is able to be gathered and
produced even if package management is disabled for the final
image.
</note>
</para>
<para>
Here is an example of <filename>image-info.txt</filename>:
<literallayout class='monospaced'>
DISTRO = poky
DISTRO_VERSION = 1.7
USER_CLASSES = buildstats image-mklibs image-prelink
IMAGE_CLASSES = image_types
IMAGE_FEATURES = debug-tweaks
IMAGE_LINGUAS =
IMAGE_INSTALL = packagegroup-core-boot run-postinsts
BAD_RECOMMENDATIONS =
NO_RECOMMENDATIONS =
PACKAGE_EXCLUDE =
ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \
write_image_manifest ; buildhistory_list_installed_image ; \
buildhistory_get_image_installed ; ssh_allow_empty_password; \
postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ;
IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
IMAGESIZE = 6900
</literallayout>
Other than <filename>IMAGESIZE</filename>, which is the
total size of the files in the image in Kbytes, the
name-value pairs are variables that may have influenced the
content of the image.
This information is often useful when you are trying to determine
why a change in the package or file listings has occurred.
</para>
</section>
<section id='using-build-history-to-gather-image-information-only'>
<title>Using Build History to Gather Image Information Only</title>
<para>
As you can see, build history produces image information,
including dependency graphs, so you can see why something
was pulled into the image.
If you are just interested in this information and not
interested in collecting specific package or SDK information,
you can enable writing only image information without
any history by adding the following to your
<filename>conf/local.conf</filename> file found in the
<link linkend='build-directory'>Build Directory</link>:
<literallayout class='monospaced'>
INHERIT += "buildhistory"
BUILDHISTORY_COMMIT = "0"
BUILDHISTORY_FEATURES = "image"
</literallayout>
Here, you set the
<link linkend='var-BUILDHISTORY_FEATURES'><filename>BUILDHISTORY_FEATURES</filename></link>
variable to use the image feature only.
</para>
</section>
<section id='build-history-sdk-information'>
<title>Build History SDK Information</title>
<para>
Build history collects similar information on the contents
of SDKs
(e.g. <filename>bitbake -c populate_sdk imagename</filename>)
as compared to information it collects for images.
Furthermore, this information differs depending on whether an
extensible or standard SDK is being produced.
</para>
<para>
The following list shows the files produced for SDKs:
<itemizedlist>
<listitem><para><filename>files-in-sdk.txt:</filename>
A list of files in the SDK with permissions,
owner, group, size, and symlink information.
This list includes both the host and target parts
of the SDK.
</para></listitem>
<listitem><para><filename>sdk-info.txt:</filename>
A text file containing name-value pairs with information
about the SDK.
See the following listing example for more information.
</para></listitem>
<listitem><para><filename>sstate-task-sizes.txt:</filename>
A text file containing name-value pairs with information
about task group sizes
(e.g. <filename>do_populate_sysroot</filename> tasks
have a total size).
The <filename>sstate-task-sizes.txt</filename> file
exists only when an extensible SDK is created.
</para></listitem>
<listitem><para><filename>sstate-package-sizes.txt:</filename>
A text file containing name-value pairs with information
for the shared-state packages and sizes in the SDK.
The <filename>sstate-package-sizes.txt</filename> file
exists only when an extensible SDK is created.
</para></listitem>
<listitem><para><filename>sdk-files:</filename>
A folder that contains copies of the files mentioned in
<filename>BUILDHISTORY_SDK_FILES</filename> if the
files are present in the output.
Additionally, the default value of
<filename>BUILDHISTORY_SDK_FILES</filename> is specific
to the extensible SDK although you can set it
differently if you would like to pull in specific files
from the standard SDK.</para>
<para>The default files are
<filename>conf/local.conf</filename>,
<filename>conf/bblayers.conf</filename>,
<filename>conf/auto.conf</filename>,
<filename>conf/locked-sigs.inc</filename>, and
<filename>conf/devtool.conf</filename>.
Thus, for an extensible SDK, these files get copied
into the <filename>sdk-files</filename> directory.
</para></listitem>
<listitem><para>The following information appears under
each of the <filename>host</filename>
and <filename>target</filename> directories
for the portions of the SDK that run on the host and
on the target, respectively:
<note>
The following files for the most part are empty
when producing an extensible SDK because this
type of SDK is not constructed from packages as is
the standard SDK.
</note>
<itemizedlist>
<listitem><para><filename>depends.dot:</filename>
Dependency graph for the SDK that is
compatible with <filename>graphviz</filename>.
</para></listitem>
<listitem><para><filename>installed-package-names.txt:</filename>
A list of installed packages by name only.
</para></listitem>
<listitem><para><filename>installed-package-sizes.txt:</filename>
A list of installed packages ordered by size.
</para></listitem>
<listitem><para><filename>installed-packages.txt:</filename>
A list of installed packages with full package
filenames.</para></listitem>
</itemizedlist>
</para></listitem>
</itemizedlist>
</para>
<para>
Here is an example of <filename>sdk-info.txt</filename>:
<literallayout class='monospaced'>
DISTRO = poky
DISTRO_VERSION = 1.3+snapshot-20130327
SDK_NAME = poky-glibc-i686-arm
SDK_VERSION = 1.3+snapshot
SDKMACHINE =
SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
BAD_RECOMMENDATIONS =
SDKSIZE = 352712
</literallayout>
Other than <filename>SDKSIZE</filename>, which is the
total size of the files in the SDK in Kbytes, the
name-value pairs are variables that might have influenced the
content of the SDK.
This information is often useful when you are trying to
determine why a change in the package or file listings
has occurred.
</para>
</section>
<section id='examining-build-history-information'>
<title>Examining Build History Information</title>
<para>
You can examine build history output from the command line or
from a web interface.
</para>
<para>
To see any changes that have occurred (assuming you have
<link linkend='var-BUILDHISTORY_COMMIT'><filename>BUILDHISTORY_COMMIT = "1"</filename></link>),
you can simply
use any Git command that allows you to view the history of
a repository.
Here is one method:
<literallayout class='monospaced'>
$ git log -p
</literallayout>
You need to realize, however, that this method does show
changes that are not significant (e.g. a package's size
changing by a few bytes).
</para>
<para>
A command-line tool called <filename>buildhistory-diff</filename>
does exist, though, that queries the Git repository and prints just
the differences that might be significant in human-readable form.
Here is an example:
<literallayout class='monospaced'>
$ ~/poky/poky/scripts/buildhistory-diff . HEAD^
Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
/etc/anotherpkg.conf was added
/sbin/anotherpkg was added
* (installed-package-names.txt):
* anotherpkg was added
Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
anotherpkg was added
packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
* PR changed from "r0" to "r1"
* PV changed from "0.1.10" to "0.1.12"
packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
* PR changed from "r0" to "r1"
* PV changed from "0.1.10" to "0.1.12"
</literallayout>
<note>
The <filename>buildhistory-diff</filename> tool requires
the <filename>GitPython</filename> package.
Be sure to install it using Pip3 as follows:
<literallayout class='monospaced'>
$ pip3 install GitPython --user
</literallayout>
Alternatively, you can install
<filename>python3-git</filename> using the appropriate
distribution package manager (e.g.
<filename>apt-get</filename>, <filename>dnf</filename>, or
<filename>zipper</filename>).
</note>
</para>
<para>
To see changes to the build history using a web interface, follow
the instruction in the <filename>README</filename> file here.
<ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/buildhistory-web/'></ulink>.
</para>
<para>
Here is a sample screenshot of the interface:
<imagedata fileref="figures/buildhistory-web.png" align="center" scalefit="1" width="130%" contentdepth="130%" />
</para>
</section>
</section>
</section>
<section id='speeding-up-the-build'>
<title>Speeding Up the Build</title>