mirror of
git://git.yoctoproject.org/poky.git
synced 2025-07-19 21:09:03 +02:00
test-manual: text and formatting fixes
(From yocto-docs rev: 7c4f616f773bb9071b395e977b2ca9f8ac8f7e2a) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
parent
c24e3b9f67
commit
84b242ef8b
|
@ -14,7 +14,7 @@ release works as intended. All the project's testing infrastructure and
|
||||||
processes are publicly visible and available so that the community can
|
processes are publicly visible and available so that the community can
|
||||||
see what testing is being performed, how it's being done and the current
|
see what testing is being performed, how it's being done and the current
|
||||||
status of the tests and the project at any given time. It is intended
|
status of the tests and the project at any given time. It is intended
|
||||||
that Other organizations can leverage off the process and testing
|
that other organizations can leverage off the process and testing
|
||||||
environment used by the Yocto Project to create their own automated,
|
environment used by the Yocto Project to create their own automated,
|
||||||
production test environment, building upon the foundations from the
|
production test environment, building upon the foundations from the
|
||||||
project core.
|
project core.
|
||||||
|
@ -93,8 +93,8 @@ the following types of tests:
|
||||||
- *Build Testing:* Tests whether specific configurations build by
|
- *Build Testing:* Tests whether specific configurations build by
|
||||||
varying :term:`MACHINE`,
|
varying :term:`MACHINE`,
|
||||||
:term:`DISTRO`, other configuration
|
:term:`DISTRO`, other configuration
|
||||||
options, and the specific target images being built (or world). Used
|
options, and the specific target images being built (or ``world``). This is
|
||||||
to trigger builds of all the different test configurations on the
|
used to trigger builds of all the different test configurations on the
|
||||||
Autobuilder. Builds usually cover many different targets for
|
Autobuilder. Builds usually cover many different targets for
|
||||||
different architectures, machines, and distributions, as well as
|
different architectures, machines, and distributions, as well as
|
||||||
different configurations, such as different init systems. The
|
different configurations, such as different init systems. The
|
||||||
|
@ -120,7 +120,7 @@ the following types of tests:
|
||||||
|
|
||||||
$ bitbake image -c testsdkext
|
$ bitbake image -c testsdkext
|
||||||
|
|
||||||
The tests utilize the :ref:`ref-classes-testsdk` class and the
|
The tests use the :ref:`ref-classes-testsdk` class and the
|
||||||
``do_testsdkext`` task.
|
``do_testsdkext`` task.
|
||||||
|
|
||||||
- *Feature Testing:* Various scenario-based tests are run through the
|
- *Feature Testing:* Various scenario-based tests are run through the
|
||||||
|
@ -131,7 +131,7 @@ the following types of tests:
|
||||||
|
|
||||||
$ bitbake image -c testimage
|
$ bitbake image -c testimage
|
||||||
|
|
||||||
The tests utilize the :ref:`ref-classes-testimage`
|
The tests use the :ref:`ref-classes-testimage`
|
||||||
class and the :ref:`ref-tasks-testimage` task.
|
class and the :ref:`ref-tasks-testimage` task.
|
||||||
|
|
||||||
- *Layer Testing:* The Autobuilder has the possibility to test whether
|
- *Layer Testing:* The Autobuilder has the possibility to test whether
|
||||||
|
@ -151,7 +151,7 @@ the following types of tests:
|
||||||
|
|
||||||
$ bitbake image -c testsdk
|
$ bitbake image -c testsdk
|
||||||
|
|
||||||
The tests utilize the :ref:`ref-classes-testsdk` class and
|
The tests use the :ref:`ref-classes-testsdk` class and
|
||||||
the ``do_testsdk`` task.
|
the ``do_testsdk`` task.
|
||||||
|
|
||||||
- *Unit Testing:* Unit tests on various components of the system run
|
- *Unit Testing:* Unit tests on various components of the system run
|
||||||
|
@ -190,7 +190,7 @@ Tests map into the codebase as follows:
|
||||||
$ bitbake-selftest -v
|
$ bitbake-selftest -v
|
||||||
|
|
||||||
To skip tests that access the Internet, use the ``BB_SKIP_NETTESTS``
|
To skip tests that access the Internet, use the ``BB_SKIP_NETTESTS``
|
||||||
variable when running "bitbake-selftest" as follows::
|
variable when running ``bitbake-selftest`` as follows::
|
||||||
|
|
||||||
$ BB_SKIP_NETTESTS=yes bitbake-selftest
|
$ BB_SKIP_NETTESTS=yes bitbake-selftest
|
||||||
|
|
||||||
|
@ -215,7 +215,7 @@ Tests map into the codebase as follows:
|
||||||
- These tests use OE to test the workflows, which include testing
|
- These tests use OE to test the workflows, which include testing
|
||||||
specific features, behaviors of tasks, and API unit tests.
|
specific features, behaviors of tasks, and API unit tests.
|
||||||
|
|
||||||
- The tests can take advantage of parallelism through the "-j"
|
- The tests can take advantage of parallelism through the ``-j``
|
||||||
option, which can specify a number of threads to spread the tests
|
option, which can specify a number of threads to spread the tests
|
||||||
across. Note that all tests from a given class of tests will run
|
across. Note that all tests from a given class of tests will run
|
||||||
in the same thread. To parallelize large numbers of tests you can
|
in the same thread. To parallelize large numbers of tests you can
|
||||||
|
@ -231,18 +231,18 @@ Tests map into the codebase as follows:
|
||||||
$ oe-selftest -a
|
$ oe-selftest -a
|
||||||
|
|
||||||
- To run a specific test, use the following command form where
|
- To run a specific test, use the following command form where
|
||||||
testname is the name of the specific test::
|
``testname`` is the name of the specific test::
|
||||||
|
|
||||||
$ oe-selftest -r <testname>
|
$ oe-selftest -r <testname>
|
||||||
|
|
||||||
For example, the following command would run the tinfoil
|
For example, the following command would run the ``tinfoil``
|
||||||
getVar API test::
|
``getVar`` API test::
|
||||||
|
|
||||||
$ oe-selftest -r tinfoil.TinfoilTests.test_getvar
|
$ oe-selftest -r tinfoil.TinfoilTests.test_getvar
|
||||||
|
|
||||||
It is also possible to run a set
|
It is also possible to run a set
|
||||||
of tests. For example the following command will run all of the
|
of tests. For example the following command will run all of the
|
||||||
tinfoil tests::
|
``tinfoil`` tests::
|
||||||
|
|
||||||
$ oe-selftest -r tinfoil
|
$ oe-selftest -r tinfoil
|
||||||
|
|
||||||
|
@ -391,14 +391,14 @@ so tests within a given test class should always run in the same build,
|
||||||
while tests in different classes or modules may be split into different
|
while tests in different classes or modules may be split into different
|
||||||
builds. There is no data store available for these tests since the tests
|
builds. There is no data store available for these tests since the tests
|
||||||
launch the ``bitbake`` command and exist outside of its context. As a
|
launch the ``bitbake`` command and exist outside of its context. As a
|
||||||
result, common bitbake library functions (bb.\*) are also unavailable.
|
result, common BitBake library functions (``bb.\*``) are also unavailable.
|
||||||
|
|
||||||
``testimage``
|
``testimage``
|
||||||
-------------
|
-------------
|
||||||
|
|
||||||
These tests are run once an image is up and running, either on target
|
These tests are run once an image is up and running, either on target
|
||||||
hardware or under QEMU. As a result, they are assumed to be running in a
|
hardware or under QEMU. As a result, they are assumed to be running in a
|
||||||
target image environment, as opposed to a host build environment. A
|
target image environment, as opposed to in a host build environment. A
|
||||||
simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains
|
simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains
|
||||||
the following::
|
the following::
|
||||||
|
|
||||||
|
@ -413,19 +413,19 @@ the following::
|
||||||
|
|
||||||
In this example, the ``OERuntimeTestCase`` class wraps
|
In this example, the ``OERuntimeTestCase`` class wraps
|
||||||
``unittest.TestCase``. Within the test, ``self.target`` represents the
|
``unittest.TestCase``. Within the test, ``self.target`` represents the
|
||||||
target system, where commands can be run on it using the ``run()``
|
target system, where commands can be run using the ``run()``
|
||||||
method.
|
method.
|
||||||
|
|
||||||
To ensure certain test or package dependencies are met, you can use the
|
To ensure certain tests or package dependencies are met, you can use the
|
||||||
``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test
|
``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test
|
||||||
in this example would only make sense if python3-core is installed in
|
in this example would only make sense if ``python3-core`` is installed in
|
||||||
the image.
|
the image.
|
||||||
|
|
||||||
``testsdk_ext``
|
``testsdk_ext``
|
||||||
---------------
|
---------------
|
||||||
|
|
||||||
These tests are run against built extensible SDKs (eSDKs). The tests can
|
These tests are run against built extensible SDKs (eSDKs). The tests can
|
||||||
assume that the eSDK environment has already been setup. An example from
|
assume that the eSDK environment has already been set up. An example from
|
||||||
``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following::
|
``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following::
|
||||||
|
|
||||||
class DevtoolTest(OESDKExtTestCase):
|
class DevtoolTest(OESDKExtTestCase):
|
||||||
|
@ -472,9 +472,9 @@ following::
|
||||||
output = self._run(cmd)
|
output = self._run(cmd)
|
||||||
self.assertEqual(output, "Hello, world\n")
|
self.assertEqual(output, "Hello, world\n")
|
||||||
|
|
||||||
In this example, if nativesdk-python3-core has been installed into the SDK, the code runs
|
In this example, if ``nativesdk-python3-core`` has been installed into the SDK,
|
||||||
the python3 interpreter with a basic command to check it is working
|
the code runs the ``python3`` interpreter with a basic command to check it is
|
||||||
correctly. The test would only run if Python3 is installed in the SDK.
|
working correctly. The test would only run if Python3 is installed in the SDK.
|
||||||
|
|
||||||
``oe-build-perf-test``
|
``oe-build-perf-test``
|
||||||
----------------------
|
----------------------
|
||||||
|
@ -520,7 +520,7 @@ an isolated directory.
|
||||||
|
|
||||||
This can delete files from :term:`SSTATE_DIR` which would potentially break
|
This can delete files from :term:`SSTATE_DIR` which would potentially break
|
||||||
other builds running in parallel. If this is required, :term:`SSTATE_DIR` must
|
other builds running in parallel. If this is required, :term:`SSTATE_DIR` must
|
||||||
be set to an isolated directory. Alternatively, you can use the "-f"
|
be set to an isolated directory. Alternatively, you can use the ``-f``
|
||||||
option with the ``bitbake`` command to "taint" tasks by changing the
|
option with the ``bitbake`` command to "taint" tasks by changing the
|
||||||
sstate checksums to ensure sstate cache items will not be reused.
|
sstate checksums to ensure sstate cache items will not be reused.
|
||||||
|
|
||||||
|
|
|
@ -42,16 +42,16 @@ our sstate up to date. On the weekend, there is a master-next build
|
||||||
instead to ensure the test results are updated for the less frequently
|
instead to ensure the test results are updated for the less frequently
|
||||||
run targets.
|
run targets.
|
||||||
|
|
||||||
Performance builds (buildperf-\* targets in the console) are triggered
|
Performance builds (``buildperf-\*`` targets in the console) are triggered
|
||||||
separately every six hours and automatically push their results to the
|
separately every six hours and automatically push their results to the
|
||||||
buildstats repository at:
|
buildstats repository at:
|
||||||
:yocto_git:`/yocto-buildstats/`.
|
:yocto_git:`/yocto-buildstats/`.
|
||||||
|
|
||||||
The 'quick' targets have been selected to be the ones which catch the
|
The "quick" targets have been selected to be the ones which catch the
|
||||||
most failures or give the most valuable data. We run 'fast' ptests in
|
most failures or give the most valuable data. We run "fast" ptests in
|
||||||
this case for example but not the ones which take a long time. The quick
|
this case for example but not the ones which take a long time. The quick
|
||||||
target doesn't include \*-lsb builds for all architectures, some world
|
target doesn't include ``\*-lsb`` builds for all architectures, some ``world``
|
||||||
builds and doesn't trigger performance tests or ltp testing. The full
|
builds and doesn't trigger performance tests or ``ltp`` testing. The full
|
||||||
build includes all these things and is slower but more comprehensive.
|
build includes all these things and is slower but more comprehensive.
|
||||||
|
|
||||||
Release Builds
|
Release Builds
|
||||||
|
@ -67,7 +67,7 @@ that in :ref:`test-manual/test-process:day to day development`, in that the
|
||||||
a-full target of the Autobuilder is used but in addition the form is
|
a-full target of the Autobuilder is used but in addition the form is
|
||||||
configured to generate and publish artifacts and the milestone number,
|
configured to generate and publish artifacts and the milestone number,
|
||||||
version, release candidate number and other information is entered. The
|
version, release candidate number and other information is entered. The
|
||||||
box to "generate an email to QA"is also checked.
|
box to "generate an email to QA" is also checked.
|
||||||
|
|
||||||
When the build completes, an email is sent out using the send-qa-email
|
When the build completes, an email is sent out using the send-qa-email
|
||||||
script in the ``yocto-autobuilder-helper`` repository to the list of
|
script in the ``yocto-autobuilder-helper`` repository to the list of
|
||||||
|
|
|
@ -32,8 +32,8 @@ which looks like::
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
|
||||||
And to expand that, you need the "arch-qemu" entry from
|
And to expand that, you need the ``arch-qemu`` entry from
|
||||||
the "templates" section, which looks like::
|
the ``templates`` section, which looks like::
|
||||||
|
|
||||||
"arch-qemu" : {
|
"arch-qemu" : {
|
||||||
"BUILDINFO" : true,
|
"BUILDINFO" : true,
|
||||||
|
@ -54,11 +54,11 @@ the "templates" section, which looks like::
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
|
||||||
Combining these two entries you can see that "qemux86-64" is a three step build where the
|
Combining these two entries you can see that ``qemux86-64`` is a three step
|
||||||
``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for
|
build where ``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS``
|
||||||
``MACHINE="qemux86-64"`` but with differing :term:`SDKMACHINE` settings. In step
|
for each step; all for ``MACHINE="qemux86-64"`` but with differing
|
||||||
1 an extra variable is added to the ``auto.conf`` file to enable wic
|
:term:`SDKMACHINE` settings. In step 1, an extra variable is added to the
|
||||||
image generation.
|
``auto.conf`` file to enable wic image generation.
|
||||||
|
|
||||||
While not every detail of this is covered here, you can see how the
|
While not every detail of this is covered here, you can see how the
|
||||||
template mechanism allows quite complex configurations to be built up
|
template mechanism allows quite complex configurations to be built up
|
||||||
|
@ -163,8 +163,9 @@ Autobuilder Worker Janitor
|
||||||
--------------------------
|
--------------------------
|
||||||
|
|
||||||
This is a process running on each Worker that performs two basic
|
This is a process running on each Worker that performs two basic
|
||||||
operations, including background file deletion at IO idle (see :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and
|
operations, including background file deletion at IO idle (see
|
||||||
maintenance of a cache of cloned repositories to improve the speed
|
"Run clobberdir" in :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`)
|
||||||
|
and maintenance of a cache of cloned repositories to improve the speed
|
||||||
the system can checkout repositories.
|
the system can checkout repositories.
|
||||||
|
|
||||||
Shared DL_DIR
|
Shared DL_DIR
|
||||||
|
@ -172,7 +173,7 @@ Shared DL_DIR
|
||||||
|
|
||||||
The Workers are all connected over NFS which allows :term:`DL_DIR` to be shared
|
The Workers are all connected over NFS which allows :term:`DL_DIR` to be shared
|
||||||
between them. This reduces network accesses from the system and allows
|
between them. This reduces network accesses from the system and allows
|
||||||
the build to be sped up. Usage of the directory within the build system
|
the build to be sped up. The usage of the directory within the build system
|
||||||
is designed to be able to be shared over NFS.
|
is designed to be able to be shared over NFS.
|
||||||
|
|
||||||
Shared SSTATE_DIR
|
Shared SSTATE_DIR
|
||||||
|
@ -180,8 +181,8 @@ Shared SSTATE_DIR
|
||||||
|
|
||||||
The Workers are all connected over NFS which allows the ``sstate``
|
The Workers are all connected over NFS which allows the ``sstate``
|
||||||
directory to be shared between them. This means once a Worker has built
|
directory to be shared between them. This means once a Worker has built
|
||||||
an artifact, all the others can benefit from it. Usage of the directory
|
an artifact, all the others can benefit from it. The usage of the directory
|
||||||
within the directory is designed for sharing over NFS.
|
within the build system is designed for sharing over NFS.
|
||||||
|
|
||||||
Resulttool
|
Resulttool
|
||||||
----------
|
----------
|
||||||
|
@ -192,7 +193,7 @@ in a given build and their status. Additional information, such as
|
||||||
failure logs or the time taken to run the tests, may also be included.
|
failure logs or the time taken to run the tests, may also be included.
|
||||||
|
|
||||||
Resulttool is part of OpenEmbedded-Core and is used to manipulate these
|
Resulttool is part of OpenEmbedded-Core and is used to manipulate these
|
||||||
json results files. It has the ability to merge files together, display
|
JSON results files. It has the ability to merge files together, display
|
||||||
reports of the test results and compare different result files.
|
reports of the test results and compare different result files.
|
||||||
|
|
||||||
For details, see :yocto_wiki:`/Resulttool`.
|
For details, see :yocto_wiki:`/Resulttool`.
|
||||||
|
@ -206,7 +207,7 @@ are general setup steps that are run once and include:
|
||||||
|
|
||||||
#. Set up any :term:`buildtools` tarball if configured.
|
#. Set up any :term:`buildtools` tarball if configured.
|
||||||
|
|
||||||
#. Call "buildhistory-init" if :ref:`ref-classes-buildhistory` is configured.
|
#. Call ``buildhistory-init`` if :ref:`ref-classes-buildhistory` is configured.
|
||||||
|
|
||||||
For each step that is configured in ``config.json``, it will perform the
|
For each step that is configured in ``config.json``, it will perform the
|
||||||
following:
|
following:
|
||||||
|
@ -258,7 +259,7 @@ it is inevitable that users will end up needing to heavily customise the
|
||||||
``yocto-autobuilder-helper`` repository, particularly the
|
``yocto-autobuilder-helper`` repository, particularly the
|
||||||
``config.json`` file as they will want to define their own test matrix.
|
``config.json`` file as they will want to define their own test matrix.
|
||||||
|
|
||||||
The Autobuilder supports wo customization options:
|
The Autobuilder supports two customization options:
|
||||||
|
|
||||||
- variable substitution
|
- variable substitution
|
||||||
|
|
||||||
|
@ -278,7 +279,7 @@ environment::
|
||||||
$ ABHELPER_JSON="config.json /some/location/local.json"
|
$ ABHELPER_JSON="config.json /some/location/local.json"
|
||||||
|
|
||||||
One issue users often run into is validation of the ``config.json`` files. A
|
One issue users often run into is validation of the ``config.json`` files. A
|
||||||
tip for minimizing issues from invalid json files is to use a Git
|
tip for minimizing issues from invalid JSON files is to use a Git
|
||||||
``pre-commit-hook.sh`` script to verify the JSON file before committing
|
``pre-commit-hook.sh`` script to verify the JSON file before committing
|
||||||
it. Create a symbolic link as follows::
|
it. Create a symbolic link as follows::
|
||||||
|
|
||||||
|
|
Loading…
Reference in New Issue
Block a user