Commit Graph

3757 Commits

Author SHA1 Message Date
Ninette Adhikari
314a220280 oe-build-perf-report: Improve report styling and add descriptions
Styling updates are added including page margin, labels for x and y axis, tooltip, and section descriptions.

(From OE-Core rev: 2a2568fcf1a3f8e467bd814f4fb13dffae8ec61d)

Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-15 16:39:19 +01:00
Ninette Adhikari
99861c17e7 oe-build-perf-report: Display more than 300 commits and date instead of commit number
- This commit updates measurement statistics data to include start_time so that time can be displayed instead of commit numbers on the chart.
- It also updates default commit history length to 300.

(From OE-Core rev: 64275a41c37130aaaacc5e592f94a1afe057119b)

Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-15 16:39:19 +01:00
Ninette Adhikari
836b4b57d6 oe-build-perf-report: Add apache echarts to make report interactive
- Add Apache echarts (https://echarts.apache.org/en/index.html) library to create build performance charts.
- Restructure data to time and value array format so that it can be used by echarts.
- This commit also converts test duration to minutes to map against the values axis.
- Zoom is added to the line charts.

(From OE-Core rev: 63c9321832aae79d20a4ddd199a4a1385f81de53)

Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-15 16:39:19 +01:00
Antonin Godard
12c447b38a devtool: ide-sdk: correct help typo
(From OE-Core rev: 50e043387a2f0f9a5c2f7a5f914c465c830d329b)

Signed-off-by: Antonin Godard <antoningodard@pm.me>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-09 09:48:58 +01:00
Richard Purdie
808f700efc classes/lib/scripts: Initial WORKDIR -> UNPACKDIR updates
Work through the initial issues I found where we need to change WORKDIR
to UNPACKDIR.

(From OE-Core rev: 86fec41b1e809d1a2fa2feadc26d29020df53d39)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-02 15:15:51 +01:00
Alexander Kanavin
ad6470a4c2 scripts/oe-setup-build: write a build environment initialization one-liner into the build directory
With this, users no longer have to know where oe-init-build-env is relative to the
build directory; that information is contained in the one liner and then
it's possible to simply use that:

. /path/to/build/init-build-env

This will particularly help with initializing builds in unpacked
build bundles, as users won't have to know where oe-init-build-env
is in the bundle directory tree - similar to esdk initialization.

(From OE-Core rev: 1cabdf287c2739accdab3a766df060f1bc802b63)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-02 11:47:51 +01:00
Joe Slater
a46e374d43 oe-debuginfod: add option for data storage
Storing the data files under $HOME can be unreliable if debuginfod
is used for several projects, especially if $HOME is shared
between machines.  We provide an option to save files under the
project directory.  The default behavior is unchanged.

(From OE-Core rev: e1e0cf82f559077e2a51447baf137086202c0c4a)

Signed-off-by: Joe Slater <joe.slater@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-30 22:20:16 +01:00
John Ripple
5756ac9936 rootfs.py: Fix logger error message format
--_000_PH7PR17MB61307CB6690EC00DEB4ED9B39C0F2PH7PR17MB6130namp_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

This patch adds a missing "%s" format in a logger.error call. Without
this addition the logger itself would error out and not print a useful
message.

(From OE-Core rev: 2c892e5dd9ba72a51c0a8fb851599cc2dc3a8b5c)

Signed-off-by: John Ripple <john.ripple@keysight.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-25 10:19:28 +01:00
Julien Stephan
28516320d8 devtool: standard: update-recipe/finish: fix update localfile in another layer
When trying to use devtool update-recipe/finish on another layer, with modified
local file we have the following error:

  Traceback (most recent call last):
    File "<..>/poky/scripts/devtool", line 350, in <module>
      ret = main()
            ^^^^^^
    File "<..>/poky/scripts/devtool", line 337, in main
      ret = args.func(args, config, basepath, workspace)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "<..>/poky/scripts/lib/devtool/standard.py", line 1968, in update_recipe
      updated, _, _ = _update_recipe(args.recipename, workspace, rd, args.mode, args.append, args.wildcard_version, args.no_remove, args.initial_rev, dry_run_outdir=dry_run_outdir, no_overrides=args.no_overrides, force_patch_refresh=args.force_patch_refresh)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "<..>/poky/scripts/lib/devtool/standard.py", line 1930, in _update_recipe
      updated, appendf, removed = _update_recipe_patch(recipename, workspace, srctree, crd, appendlayerdir, wildcard_version, no_remove, no_report_remove, initial_rev, dry_run_outdir, force_patch_refresh)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "<..>/poky/scripts/lib/devtool/standard.py", line 1747, in _update_recipe_patch
      patchdir = param.get('patchdir', ".")
                 ^^^^^^^^^
  AttributeError: 'str' object has no attribute 'get'

This was introduced when adding support for git submodules.
No selftest case exists to catch this, so a selftest will be
added in another commit.

(From OE-Core rev: de7ca9f800e15e10271502da7e51e3ae08e0c85b)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-25 10:10:14 +01:00
Vincent Kriek
629063944c devtool: sync: Fix Execution error
When executing devtool sync on a recipe that was extract with devtool
extract earlier the following error occured:

    Traceback (most recent call last):
    [...]
    bb.process.ExecutionError: Execution of 'git fetch file:///home/vin/projects/poky/build/tmp/work/all-poky-linux/netbase/6.4/devtooltmp-figt1jmr/workdir/netbase devtool:devtool' failed with exit code 128:
    fatal: refusing to fetch into branch 'refs/heads/devtool' checked out at '/home/vin/projects/poky/build/netbase-src'

Fix this by adding --update-head-ok and --force to git fetch so it will
override the current head even if it is checked out and has changes.
Possible existing changes in the devtool branch can be retrieved by
checking out the devtool.bak branch

(From OE-Core rev: 4c89b061003a0f4143c6d3838e39fc2643edc8d6)

Signed-off-by: Vincent Kriek <vincent@coelebs.dev>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-25 10:10:14 +01:00
Sven Schwermer
8219eefdd8 recipetool: Handle several go-import tags in go resolver
When dynamically resolving go modules, the HTML page may contain several
go-import meta tags. We must handle all and pick the correct one based
on the module name. An example for such a behaviour is
gonum.org/v1/gonum:

<meta name="go-import" content="gonum.org/v1/exp git https://github.com/gonum/exp">
<meta name="go-import" content="gonum.org/v1/gonum git https://github.com/gonum/gonum">
<meta name="go-import" content="gonum.org/v1/hdf5 git https://github.com/gonum/hdf5">
<meta name="go-import" content="gonum.org/v1/netlib git https://github.com/gonum/netlib">
<meta name="go-import" content="gonum.org/v1/plot git https://github.com/gonum/plot">
<meta name="go-import" content="gonum.org/v1/tools git https://github.com/gonum/tools">

(From OE-Core rev: 9c36a61e29359067165bddc7f2accdf2c4c8a761)

Signed-off-by: Sven Schwermer <sven.schwermer@disruptive-technologies.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-23 13:40:24 +01:00
Sven Schwermer
e4c3483ecf recipetool: Handle unclean response in go resolver
It appears that some go modules repond with a 404 error when trying to
resolve them dynamically. The response body may still contain the
go-import meta tag. An example for such behaviour is gonum.org/v1/gonum.

(From OE-Core rev: 8f2e14ab6562a9a68819a960c66a258ea9dbe246)

Signed-off-by: Sven Schwermer <sven.schwermer@disruptive-technologies.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-23 13:40:24 +01:00
Julien Stephan
834e91ce53 devtool: standard: throws appropriate error if source is in detached HEAD
If source is in detached HEAD, we get the following error when using
detvool finish command:

    [...]
    File "<...>/poky/scripts/lib/devtool/standard.py", line 1938, in _update_recipe
      bb.process.run('git checkout %s' % startbranch, cwd=srctree)
    File "<...>/poky/bitbake/lib/bb/process.py", line 189, in run
      raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
  bb.process.ExecutionError: Execution of 'git checkout (HEAD detached at 9bbf87e)' failed with exit code 2:
  /bin/sh: -c: line 1: syntax error near unexpected token `('
  /bin/sh: -c: line 1: `git checkout (HEAD detached at 9bbf87e)'

Check this and throws an appropriate error in this case

(From OE-Core rev: d9c686b5ff9f591ec6b928ed539084c02df4c8a5)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-12 17:27:53 +01:00
Anibal Limon
7c3f956a0a wic: bootimg-partition allow to set var to get boot files
This is a generic functionality that takes a set of files and create a
boot parition based on IMAGE_BOOT_FILES var, allow to sepcify the
variable name so a BSP plugin can extend this and re-use the
functionality.

(From OE-Core rev: 3a2339455be29a6ff65c8c5f6865a4cd35dd7f63)

Signed-off-by: Anibal Limon <anibal@limonsoftware.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-26 17:10:51 +00:00
Trevor Woerner
f15676592b bmaptool: now part of Yocto Project
The bmaptool (previously: bmap-tools, bmap-tool, bmaptool) has been moved
to be under the Yocto Project umbrella and is now hosted at:

	github.com/yoctoproject/bmaptool

[RP: Added a couple of missing renames]
(From OE-Core rev: 7a036b1a1ec7dcd27dbe18d4c2e703bd2a8af182)

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-05 12:24:50 +00:00
Adrian Freihofer
c82c57621b devtool: ide-sdk prefer sources from workspace
Improve the previous commit:
- log an error if some assumptions are not true
- Use TARGET_DBGSRC_DIR variable
- Do the same for ide none

Why the additional source mapping is required:

For example the cmake-example recipe refers to sources like this:
./recipe-sysroot-native/usr/bin/x86_64-poky-linux/x86_64-poky-linux-readelf \
  -wi image/usr/bin/cmake-example | grep -B1 DW_AT_comp_dir
    ...
    <560>   DW_AT_name        : (indirect line string, offset: 0x1da):
    /usr/src/debug/cmake-example/1.0/oe-local-files/cpp-example.cpp
    ...

Another example is powertop:
./recipe-sysroot-native/usr/bin/x86_64-poky-linux/x86_64-poky-linux-readelf \
  -wi image/usr/sbin/powertop | grep -B1 DW_AT_comp_dir
    ...
    <561>   DW_AT_name        : (indirect line string, offset: 0x1da):
    /usr/src/debug/powertop/2.15/src/devlist.cpp
    ...

For recipes with local files this works. The oe-local-files folder is
not available in the rootfs-dbg and therefore the sources are first
found in the workspace folder. GDB searches for source files in various
places:
https://sourceware.org/gdb/current/onlinedocs/gdb.html/Source-Path.html
However, for the powertop example the sources opened in the editor are
from the rootfs-dbg instead of from the workspace.

Bitbake calls the compiler with
  -fmacro-prefix-map=${S}=${TARGET_DBGSRC_DIR}
where TARGET_DBGSRC_DIR defaults to "/usr/src/debug/${PN}/${PV}".

A source map which maps the recipe specific path from TARGET_DBGSRC_DIR
to the workspace fixes this.
The already existing source map for /usr/src/debug applies for all other
recipes. It finds the sources (read only) in the rootfs-dbg folder.

(From OE-Core rev: 06601632c1879cb80276f9b36de91fb7808311a5)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-27 11:38:46 +00:00
Enguerrand de Ribaucourt
a5b75e29ff devtool: ide-sdk source mapping for vscode
When launching the debug configuration, the source files from the debug
rootfs were openened in the editor instead of the local workspace files.
We add an exception to properly map them to the file being developed and
compiled by the IDE integration. This also more closely matches what the
user would expect compared to native development.

This is also true for the devtool fallback mode.

(From OE-Core rev: 24db2b8d0d7104960c1cdb2c7ee5216c830a6754)

Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-27 11:38:46 +00:00
Adrian Freihofer
4c6f3bbc77 devtool: ide-sdk launch.json per recipe only
If multiple recipes are processed at once, the launch.json and the
tasks.json of the second recipe contains also the configurations for the
binaries of the first recipe.

Example:
  devtool ide-sdk powertop cmake-example oe-selftest-image
generated a launch and a tasks configuration for the cmake-example
recipe which also offers debugging the powertop binary.

(From OE-Core rev: 63986b2c40d90fe96cdc6a46aa649efcf17f6ac2)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-27 11:38:46 +00:00
Enrico Jörns
54c27d84b6 wic: 'empty' plugin: fix typo in comment
(From OE-Core rev: 1a9013c3767d174a970d31e80748bc8d73af0dba)

Signed-off-by: Enrico Jorns <ejo@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-27 11:35:43 +00:00
Simone Weiß
8a3f025b3a patchtest: provide further guidance for failed testcases
Cross-reference the wiki page on patchtest now that it is updated and contains
more information how to address failed testcases. Adding it in patchtest only
is enough as patchtest-send-result already points to the wikipage for failures.

(From OE-Core rev: 51267f3c5d647fc6483ce6b597ed9e25c14bd425)

Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-24 16:10:23 +00:00
Adrian Freihofer
4e3b6c24d5 devtool: ide-sdk python 3.12 escaping
scripts/lib/devtool/ide_sdk.py:709: SyntaxWarning: invalid escape sequence '\.'
  re_so = re.compile('.*\.so[.0-9]*$')

scripts/lib/devtool/ide_plugins/__init__.py:87: SyntaxWarning: invalid escape sequence '\$'
  gdbserver_cmd_start += "test -f \$TEMP_DIR/pid && exit 0; "

scripts/lib/devtool/ide_plugins/__init__.py:88: SyntaxWarning: invalid escape sequence '\$'
  gdbserver_cmd_start += "mkdir -p \$TEMP_DIR; "

scripts/lib/devtool/ide_plugins/__init__.py:89: SyntaxWarning: invalid escape sequence '\$'
  gdbserver_cmd_start += "%s --multi :%s > \$TEMP_DIR/log 2>&1 & " % (

scripts/lib/devtool/ide_plugins/__init__.py:91: SyntaxWarning: invalid escape sequence '\$'
  gdbserver_cmd_start += "echo \$! > \$TEMP_DIR/pid;"

scripts/lib/devtool/ide_plugins/__init__.py:94: SyntaxWarning: invalid escape sequence '\$'
  gdbserver_cmd_stop += "test -f \$TEMP_DIR/pid && kill \$(cat \$TEMP_DIR/pid); "

scripts/lib/devtool/ide_plugins/__init__.py:95: SyntaxWarning: invalid escape sequence '\$'
  gdbserver_cmd_stop += "rm -rf \$TEMP_DIR; "

(From OE-Core rev: e8c64921de7206bf617fc42433286867ae3c931d)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-24 16:10:23 +00:00
Enguerrand de Ribaucourt
e2f8ed72dd devtool: ide: define compilerPath for meson projects
The compile_commands.json file output by meson uses the compiler as if
present in the $PATH. However, when using an IDE, the $PATH used by
bitbake is not there.

The vscode-cpptools now allows to define the compilerPath in addition
to replace the one from compile_commands.json.

(From OE-Core rev: d9f5c27c8beee07c7cbbed11f5d45058e7315846)

Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-23 14:29:03 +00:00
Enguerrand de Ribaucourt
c0d340c52e devtool: ide: vscode: Configure read-only files
When debugging or browsing files, the user may fall into external
sources from other packages in the sysroot or dbg-rootfs. Modifying them
will only lead to confusion since they will be overwritten by Yocto. The
user should open them in a separate devtool modify session if they want
to make changes. Meanwhile, we should prevent write access to them.

(From OE-Core rev: 8a95ad9aecb81732c865e00a987bb8bd3d6cb91d)

Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-20 12:58:41 +00:00
Enguerrand de Ribaucourt
8e84fbba72 devtool: ide_sdk: Use bitbake's python3 for generated scripts
The generated scripts use the sys.path configuration found inside
bitbake. It can be a different python version than the one used on the
host through the IDE.

For instance, when running the generated script
deploy_target_cmake-example-core2-64 from an eSDK generated on another
machine, I got the following exception:
    AssertionError: SRE module mismatch

We need to match the sys.executable to the sys.path.

(From OE-Core rev: 45704319661570b45ef69fddd6b4b4fa22ca80d1)

Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-20 12:58:41 +00:00
Tim Orling
9f7475b4e5 recipetool; add support for python_mesonpy class
* Add support to detect the "mesonpy" build-backend for recipetool create.
* Add oe-selftest case for creating a recipe for "siphash24" from pypi.
  https://pypi.org/project/siphash24/

This is by far the simplest recipe using the mesonpy build backend.

Upstream does not provide LICENSE file(s) and we do not detect the
LICENSE so don't check for that result in the test. Likewise, upstream
does not define HOMEPAGE, so skip that result.

(From OE-Core rev: 256749322671d2f4ea994db671d73c4de10e1723)

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-20 11:39:45 +00:00
Peter Kjellerstedt
4cfd0f7e4e lib/oe/patch: Use git notes to store the filenames for the patches
The old way of keeping track of the filenames for the patches that
correspond to the commits was to add a special comment line to the end
of the commit message, e.g., "%% original patch: <filename>", using a
temporary git hook. This method had some drawbacks, e.g.:

* It caused problems if one wanted to push the commits upstream as the
  comment line had to be manually removed.
* The comment line would end up in patches if someone used git
  format-path rather than devtool finish to generate the patches.
* The comment line could interfere with global Git hooks used to
  validate the format of the Git commit message.
* When regenerating patches with `devtool finish --force-patch-refresh`,
  the process typically resulted in adding empty lines to the end of the
  commit messages in the updated patches.

A better way of keeping track of the patch filenames is to use Git
notes. This way the commit messages remain unaffected, but the
information is still shown when, e.g., doing `git log`. A special Git
notes space, refs/notes/devtool, is used to not intefere with the
default Git notes. It is configured to be shown in, e.g., `git log` and
to survive rewrites (i.e., `git commit --amend` and `git rebase`).

Since there is no longer any need for a temporary Git hook, the code
that manipulated the .git/hooks directory has also been removed. To
avoid potential problems due to global Git hooks, --no-verify was added
to the `git commit` command.

To not cause troubles for those who have done `devtool modify` for a
recipe with the old solution and then do `devtool finish` with the new
solution, the code will fall back to look for the old strings in the
commit message if no Git note can be found.

While not technically motivated like above, the way to keep track of
ignored commits is also changed to use Git notes to avoid having
different methods to store similar information.

(From OE-Core rev: f5e6183b9557477bef74024a587de0bfcc2b7c0d)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 16:03:22 +00:00
Peter Kjellerstedt
85b5e87c7d devtool: Make use of oe.patch.GitApplyTree.commitIgnored()
This makes use of the oe.patch.GitApplyTree.commitIgnored() function to
create commits that shall be ignored by `devtool finish`.

(From OE-Core rev: 4e1b9289450b5e7946bd5999c95a3ee214aab8a5)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 16:03:22 +00:00
Alexander Kanavin
c390b2e615 oe-setup-build: add a tool for discovering config templates and setting up builds
This is another piece of the puzzle in setting up builds from nothing
without having to write custom scripts or use external tools.

After layers have been fetched and placed into their respective locations by
oe-setup-layers, one would surely want to proceed to the actual build, and here's how:

1. Without arguments the tool reads available layers
from .oe-layers.json file (written out by oe-setup-layers or a fallback under scripts/),
prints what templates it has found, and asks the user to select one, as seen below.
This will land the user in a shell ready to run bitbake:

=============================================
alex@Zen2:/srv/work/alex$ ./setup-build
Available build configurations:

1. alex-configuration-gadget
This configuration will set up a build for the purposes of supporting gadget.

2. alex-configuration-gizmo
This configuration allows building a gizmo.

3. poky-default
This is the default build configuration for the Poky reference distribution.

Re-run with 'list -v' to see additional information.
Please choose a configuration by its number: 1
Running: TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/configuration-gadget . /srv/work/alex/poky/oe-init-build-env /srv/work/alex/build-alex-configuration-gadget && /bin/bash
You had no conf/local.conf file. This configuration file has therefore been
created for you from /srv/work/alex/meta-alex/conf/templates/configuration-gadget/local.conf.sample
You may wish to edit it to, for example, select a different MACHINE (target
hardware).

You had no conf/bblayers.conf file. This configuration file has therefore been
created for you from /srv/work/alex/meta-alex/conf/templates/configuration-gadget/bblayers.conf.sample
To add additional metadata layers into your configuration please add entries
to conf/bblayers.conf.

The Yocto Project has extensive documentation about OE including a reference
manual which can be found at:
    https://docs.yoctoproject.org

For more information about OpenEmbedded see the website:
    https://www.openembedded.org/

This configuration will set up a build for the purposes of supporting gadget.
Please refer to meta-alex/README for additional details and available bitbake targets.
==============================================

2. It is also possible to list available configurations without selecting one using
'setup-build list' or to select and setup one non-interactively with 'setup-build setup'.

3. The full set of command line options is:

$ ./setup-build --help
usage: setup-build [-h] [--layerlist LAYERLIST] {list,setup} ...

A script that discovers available build configurations and sets up a build environment based on one of them. Run without arguments to choose one interactively.

positional arguments:
  {list,setup}
    list                List available configurations
    setup               Set up a build environment and open a shell session with it, ready to run builds.

optional arguments:
  -h, --help            show this help message and exit
  --layerlist LAYERLIST
                        Where to look for available layers (as written out by setup-layers script) (default is /srv/work/alex/.oe-layers.json).

$ ./setup-build list --help
usage: setup-build list [-h] [-v]

optional arguments:
  -h, --help  show this help message and exit
  -v          Print detailed information and usage notes for each available build configuration.

$ ./setup-build setup --help
usage: setup-build setup [-h] [-c configuration_name] [-b build_path] [--no-shell]

optional arguments:
  -h, --help            show this help message and exit
  -c configuration_name
                        Use a build configuration configuration_name to set up a build environment (run this script with 'list' to see what is available)
  -b build_path         Set up a build directory in build_path (run this script with 'list -v' to see where it would be by default)
  --no-shell            Create a build directory but do not start a shell session with the build environment from it.

4. There's an an added hint in oe-setup-layers about how to proceed (as it is really not user-friendly
to fetch the layer repos successfully and then exit without a word), and a symlink to the script
from the top level layer checkout directory.

5. The selftest to check layer setup has been adjusted to run a basic check for template
discovery and build setup. The revision of poky to be cloned has been bumped to 4.1,
as that's the first version with a default template in a standard location.

(From OE-Core rev: 1360b64e88cda7dddfb0eca6a64f70c13dafb890)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 11:47:53 +00:00
Alexander Kanavin
01d0ee1b83 scripts/oe-setup-builddir: add support for configuration summaries
They are handled exactly same as conf-notes.txt.

(From OE-Core rev: 6fd2eb619693da4cb918b421f3e31072b48fa85a)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 11:47:53 +00:00
Simone Weiß
0ba7038c09 patchtest: add stronger indication for failed tests
Do not only log that there has been an issue but add WARNING before for local
runs. Hopefully this helps to avoid that people reading the log to quickly miss
issues.

Fixes [YOCTO #15389]

(From OE-Core rev: 661c0a8fd8fe7bff61cea82778c25cf24d791267)

Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 11:39:08 +00:00
Alexander Kanavin
a4b5a2d5b7 scripts/oe-setup-layers: write a list of layer paths into the checkout's top dir
This is beneficial for setting up builds, as this list can be used
to determine reliably where the actual layers are, and discover
available configurations from them.

Also adjust the selftest to check the presence of that file rather
than any specific layer in a hardcoded location.

Sample output (paths are written relative to the file for relocatability
and ease of reading):

{
    "layers": [
        "meta-openembedded/meta-filesystems",
        "meta-openembedded/meta-gnome",
        "meta-openembedded/meta-initramfs",
        "meta-openembedded/meta-multimedia",
        "meta-openembedded/meta-networking",
        "meta-openembedded/meta-oe",
        "meta-openembedded/meta-perl",
        "meta-openembedded/meta-python",
        "meta-openembedded/meta-webserver",
        "meta-openembedded/meta-xfce",
        "poky/bitbake/lib/layerindexlib/tests/testdata/layer1",
        "poky/bitbake/lib/layerindexlib/tests/testdata/layer2",
        "poky/bitbake/lib/layerindexlib/tests/testdata/layer3",
        "poky/bitbake/lib/layerindexlib/tests/testdata/layer4",
        "poky/meta-poky",
        "poky/meta-selftest",
        "poky/meta-skeleton",
        "poky/meta-yocto-bsp",
        "poky/meta"
    ],
    "version": "1.0"
}

(From OE-Core rev: 82743f4f767f8016564be0d9d6c0d8fe9e067740)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 11:39:08 +00:00
Adrian Freihofer
dfe9871372 oe-init-build-env: generate .vscode from template
Provide a reasonable default configuration for VSCode and the
yocto.bitbake extension.
The generated default configuration is generic and minimal. It's mostly
supposed to prevent VSCode from OOM exceptions if the build directory is
in the scope of the indexer plugins of VSCode.
But it also configures the yocto-bitbake plugin to just work without
manual user interaction.

The configuration is only generated if VSCode is installed. Currently,
VSCode is one of many popular editors for Yocto/OE. Removing the check
would mean that the configuration would be generated by e.g. oe-selftest
or for users not using VSCode. If it should prove useful, the check can
be removed later.

Customization for other layers is possible. A layer might provide it's
own oe-setup-build-env script which calls the oe-setup-vscode script
from poky with different folders. But it's also possible to override the
oe-setup-vscode script by another layer with a custom implementation.

(From OE-Core rev: 48829be7ab2edcbc2e4473f81cdaf35889d63f9c)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 11:34:33 +00:00
Peter Kjellerstedt
09839f6a8a devtool: _extract_source: Correct the removal of an old backup directory
Also correct the comment describing what is happening.

(From OE-Core rev: 7d867753fba8d536bef4c72c7bea3f4ed26a1a95)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-18 22:02:40 +00:00
Peter Kjellerstedt
890446392b devtool: standard: Add some missing whitespace
Makes it a little bit easier when reading the code.

(From OE-Core rev: 6a757f9648685448fd18507f6aaf4eed0a57579f)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-18 07:34:42 +00:00
Adrian Freihofer
539c880126 devtool: ide-sdk make deploy-target quicker
(From OE-Core rev: 3b63e7adb5596739b846396304ff815859ce6a74)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-18 07:34:42 +00:00
Adrian Freihofer
3ccb4d8ab1 devtool: new ide-sdk plugin
The new devtool ide plugin provides the eSDK and configures an IDE to
work with the eSDK. In doing so, bitbake should be used to generate the
IDE configuration and update the SDK, but it should no longer play a
role when working on the source code. The work on the source code should
take place exclusively with the IDE, which, for example, calls cmake
directly to compile the code and execute the unit tests from the IDE.

The plugin works for recipes inheriting the cmake or the meson bbclass.
Support for more programming languages and build tools may be added in
the future.

There are various IDEs that can be used for the development of embedded
Linux applications. Therefore, devtool ide-sdk, like devtool itself,
supports plugins to support IDEs.
VSCode is the default IDE for this first implementation. Additionally,
some generic helper scripts can be generated with --ide none instead of
a specific IDE configuration. This can be used for any IDE that
supports calling some scripts.

There are two different modes supported:

- devtool modify mode (default):
  devtool ide-sdk configures the IDE to manage the build-tool used by the
  recipe (e.g. cmake or meson). The workflow looks like:
    $ devtool modify a-recipe
    $ devtool ide-sdk a-recipe a-image
    $ code "$BUILDDIR/workspace/sources/a-recipe"
    Work in VSCode, after installing the proposed plugins

  Deploying the artifacts to the target device and running a remote
  debugging session is supported as well.
  This first implementation still calls bitbake and devtool to copy the
  binary artifacts to the target device. In contrast to compiling,
  installation and copying must be performed with the file rights of the
  target device. The pseudo tool must be used for this. Therefore
  bitbake -c install a-recipe && devtool deploy-target a-recipe
  are called by the IDE for the deployment. This might be improved later
  on.

  Executing the unit tests out of the IDE is supported via Qemu user if
  the build tool supports that. CMake (if cmake-qemu.bbclass is
  inherited) and Meson support Qemu usermode.

- Shared sysroots mode: bootstraps the eSDK with shared sysroots for
  all the recipes passed to devtool ide-sdk. This is basically a wrapper
  for bitbake meta-ide-support && bitbake build-sysroots. The workflow
  looks like:
    $ devtool ide-sdk --share-sysroots a-recipe another-recipe
    vscode where/the/sources/are
  If the IDE and the build tool support it, the IDE gets configured to
  offer the cross tool-chain provided by the eSDK. In case of VSCode and
  cmake a cmake-kit is generated. This offers to use the cross
  tool-chain from the UI of the IDE.

Many thanks to Enguerrand de Ribaucourt for testing and bug fixing.

(From OE-Core rev: 3f8af7a36589cd05fd07d16cbdd03d6b3dff1f82)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-18 07:34:42 +00:00
André Draszik
0684f971c3 sstate-cache-management: fix regex for 'sigdata' stamp files
Given file names like
    6.4.do_fetch.sigdata.821b6c62f9f2bd8b7e1378656b8319697a21f6f6e4a351f98dc325a18ef7ed0f,
I'm pretty sure we want to match the dot here, not any character.

Fixes: 2fa1b25d7485 ("sstate-cache-management: Rewrite in python") in oe-core
Fixes: b723fcaac5 ("sstate-cache-management: Rewrite in python") in poky
(From OE-Core rev: 64281de7e352aaf48aa8c60b68633c54471fe353)

Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-17 18:19:19 +00:00
Trevor Gamblin
122401ce48 patchtest-send-results: add --debug option
Add a -d/--debug option to patchtest-send-results so that the contents
of the constructed raw email can be checked without actually sending
anything to the list or patch author.

(From OE-Core rev: e50827d23fd3789e7a13df6918a06b198bbf4f95)

Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-16 15:14:26 +00:00
Trevor Gamblin
6d6a924718 patchtest: Fix grammar in log output
(From OE-Core rev: ded6eb17dc423988cc975d72c36fe9ba6f28eab8)

Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-16 15:14:26 +00:00
Peter Kjellerstedt
37aa0e077a devtool: modify: Correct appending of type=git-dependency to URIs
A missing space when using :append would lead to run-on URIs if there
was no whitespace at the end of the original SRC_URI.

(From OE-Core rev: 4de0c679e4b1a3cb394f348d625b97ad73f1efe7)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-16 15:14:26 +00:00
Trevor Gamblin
5c8e1e9955 patchtest-send-results: use Message-ID directly
There's no need to use regex for extracting the Message-ID field from
the patch email and mangle it by removing the angle brackets in the
process. Pull it directly from the mbox so that Patchtest's replies have
even fewer differences when compared to other replies. Also add a TODO
so that it's clear this needs adjustment when full series support is
added.

(From OE-Core rev: 1ce51580a23869a9c0f409446ff717d471fb60db)

Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-15 21:41:10 +00:00
Simone Weiß
b83bd953da patchtest: log errors and failures at end
At the moment, running patchtest locally will only print failures and errors
to the log when the not passing test case is executed. This might lead to
people overlooking issues with their patches, so print a log line at the
end if testcases showed issues. This should make it more easy to spot then
before.

Fixes [YOCTO #15389]

(From OE-Core rev: 84ca5a5f5a44de6ed4551ab08e58087aaa7e1369)

Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-15 09:18:10 +00:00
Adrian Freihofer
605ef6f5a2 scripts: python 3.12 regex
All the regexes throw a warning like this:

WARNING: scripts/lib/recipetool/create_buildsys.py:140:
      SyntaxWarning: invalid escape sequence '\s'
      proj_re = re.compile('project\s*\(([^)]*)\)', re.IGNORECASE)

Python 3 interprets string literals as Unicode strings, and therefore
\s is treated as an escaped Unicode character which is not correct.
Declaring the RegEx pattern as a raw string instead of unicode is
required for Python 3.

(From OE-Core rev: 24b0ba00d4f0b4d9834f7693ecb6032dfc534a80)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-13 13:51:41 +00:00
Trevor Gamblin
465c25ce2d patchtest-send-results: Add 'References' header
Based on https://www.rfc-editor.org/rfc/rfc2822#section-3.6.4, to get
proper threading there should be a 'References' field defined in
Patchtest's response headers if its replies are going to thread properly
on Patchwork. This wasn't there before, hence why were were seeing the
responses in the right place on the mailing list but not Patchwork.

Since we only want Patchtest to reply directly to the tested patch,
the 'References' field should only need to contain the same ID as the
'In-Reply-To' one.

(From OE-Core rev: f8a2af973d829c3c7f2dce39ce452daf74f31f52)

Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-10 14:13:24 +00:00
Ross Burton
c472b4ce59 recipetool: don't dump stack traces if a toml parser can't be found
If we can't find tomllib or tomli then we can just tell the user politely
that we can't parse the pyproject.toml file, there's no need to dump
exception stack traces.

Move the parser exception handler to catch the actual parse, as otherwise
it will never be used.

Whilst here, also add some debug statements to make it clear what of the
handlers is being called.

(From OE-Core rev: 1a3ba4c312844d80ae382912b319e60ad8b30737)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-10 14:13:24 +00:00
Adrian Freihofer
233d2d8c0d recipetool: cleanup imports
- os is missing
- glob is not used

(From OE-Core rev: f560bc8972e97d2e497d7b976586e4a79ffc74ec)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-08 23:14:51 +00:00
Adithya Balakumar
e685773513 wic: implement reproducible Disk GUID
GPT based disks have a disk guid apart from the 32-bit disk identifier.
This commit implements reproducible disk guid by using SOURCE_DATE_EPOCH (if available) value as a random seed

(From OE-Core rev: 150e079589e207fe174d2dceb40cd8f3d3972c5a)

Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-08 10:53:13 +00:00
Trevor Gamblin
9c1d226fad patchtest.README: update mailing list
The contributing info still lists the main Yocto list, but patchtest is
in oe-core, so simplify the instructions and put the right mailing list
in.

(From OE-Core rev: 2cfad933c334b3d7f49bc8ca62f9d443428d4523)

Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-06 15:01:16 +00:00
Alexis Lothoré
149f42b59e patchtest-send-results: properly parse test status
patchtest-send-results currently search the word "FAIL" in the whole
testresults file to decide whether it should send a report to patch
submitter or not. This global search can lead to false positives, for
example if the commit subject contains the word "FAIL" (as observed in
[1])

Prevent those false positives by explicitely parsing the test status from
each line. Each test result line, generated by the patchtest script, is
expected to have the following format:
<STATUS>: <some info, depending on the status>

[1] https://lore.kernel.org/openembedded-core/0101018d79bfe020-06f2ce89-ea19-456b-92e7-66ee1c710fd1-000000@us-west-2.amazonses.com/

(From OE-Core rev: 3567c21af8ed65448f9325ee3fe85b8be839e1b5)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-06 10:32:19 +00:00
Alexis Lothoré
d0b817d946 patchtest-send-results: remove unused variable
result_basename is declared but not used in the script

(From OE-Core rev: c2ba125dc30fb1ef0bf96152863db22159f4b31c)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-06 10:32:19 +00:00