Commit Graph

3709 Commits

Author SHA1 Message Date
Alexis Lothoré
149f42b59e patchtest-send-results: properly parse test status
patchtest-send-results currently search the word "FAIL" in the whole
testresults file to decide whether it should send a report to patch
submitter or not. This global search can lead to false positives, for
example if the commit subject contains the word "FAIL" (as observed in
[1])

Prevent those false positives by explicitely parsing the test status from
each line. Each test result line, generated by the patchtest script, is
expected to have the following format:
<STATUS>: <some info, depending on the status>

[1] https://lore.kernel.org/openembedded-core/0101018d79bfe020-06f2ce89-ea19-456b-92e7-66ee1c710fd1-000000@us-west-2.amazonses.com/

(From OE-Core rev: 3567c21af8ed65448f9325ee3fe85b8be839e1b5)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-06 10:32:19 +00:00
Alexis Lothoré
d0b817d946 patchtest-send-results: remove unused variable
result_basename is declared but not used in the script

(From OE-Core rev: c2ba125dc30fb1ef0bf96152863db22159f4b31c)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-06 10:32:19 +00:00
Adrian Freihofer
c2813a4c69 devtool: refactor deploy-target
Make the deploy function independent from d. This allows to call the
function also from Python code not running in bitbake.
This is needed to for the devtool ide plugin which will call the
do_install task and the code from devtool deploy-target independently
from a bitbake server. This allows a much quicker workflow.

(From OE-Core rev: c8697d1132cbd4b2a2502b4c48e7d91fc18de786)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-02 10:38:28 +00:00
Adrian Freihofer
50618e59dd devtool: refactor deploy to use exec_fakeroot_no_d
This is a step towards a deploy function which can be called without
passing the d variable.

(From OE-Core rev: 3e8c41ec373a1766f68932d3b979d00e4e3bb3a9)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-02 10:38:28 +00:00
Adrian Freihofer
c17ce99a2f devtool: refactor exec_fakeroot
Provide a function exec_fakeroot_no_d which does the same like
exec_fakeroot does, but is usable independenlty from bitbake. This
allows to use the fanction from scripts where the d variable is not
available.

(From OE-Core rev: 6f5980d96844559639fc3660bb975b0bdd1aa523)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-02 10:38:28 +00:00
Adrian Freihofer
c9c0974209 devtool: support plugins with plugins
Pass the plugins search path to plugins via context. This allows plugins
to search for their own plugins at the same paths.

(From OE-Core rev: e9c3c996f2e9a52462c4426f394798a211e53ae1)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-02 10:38:28 +00:00
Julien Stephan
facab170b6 devtool: modify: add support for multiple source in SRC_URI
[YOCTO #15162]

when doing devtool modify, sources are extracted into a devtool
temporary workdir. The main source is moved inside
build/workspace/sources/${BPN}/ and local files are moved inside
build/workspace/sources/${BPN}/oe-local-files. Secondary sources are
currently not handled and are lost.

Here is the output of devtool modify/build on bzip2 recipe:

NOTE: bzip2: compiling from external source tree <...>/build/workspace/sources/bzip2
ERROR: bzip2-1.0.8-r0 do_install_ptest_base: ExecutionError('<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368', 1, None, None)
ERROR: Logfile of failure stored in: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/log.do_install_ptest_base.3368
Log data follows:
| DEBUG: Executing shell function do_install_ptest_base
| NOTE: make -j 16 DESTDIR=<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest install-ptest
| sed  -n '/^runtest:/,/^install-ptest:/{/^install-ptest:/!p}' \
|            ../../../../../../workspace/sources/bzip2/Makefile.am      > <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/Makefile
| cp ../../../../../../workspace/sources/bzip2/sample1.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| cp ../../../../../../workspace/sources/bzip2/sample2.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| cp ../../../../../../workspace/sources/bzip2/sample3.ref      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| cp ../../../../../../workspace/sources/bzip2/sample1.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| cp ../../../../../../workspace/sources/bzip2/sample2.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| cp ../../../../../../workspace/sources/bzip2/sample3.bz2      <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/
| ln -s /usr/bin/bzip2          <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2
| cp: cannot stat '<...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/git/commons-compress': No such file or directory
| WARNING: <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368:189 exit 1 from 'cp -r <...>/build/tmp/work/core2-64-poky-linux/bzip2/
1.0.8/git/commons-compress <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/image/usr/lib/bzip2/ptest/bzip2-tests/commons-compress'
| WARNING: Backtrace (BB generated script):
|       #1: do_install_ptest, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 189
|       #2: do_install_ptest_base, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 158
|       #3: main, <...>/build/tmp/work/core2-64-poky-linux/bzip2/1.0.8/temp/run.do_install_ptest_base.3368, line 226
ERROR: Task (<...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base) failed with exit code '1'
NOTE: Tasks Summary: Attempted 776 tasks of which 765 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  <...>/poky/meta/recipes-extended/bzip2/bzip2_1.0.8.bb:do_install_ptest_base

externalsrc class modify SRC_URI to keep only:
* 'file', 'npmsw' and 'crate' sources
* url with type parameter matching 'kmeta' or 'git-dependency'

So by forcing to add type='git-dependency' on secondary sources, we
ensure that when building the recipe, the secondary sources can be
unpacked into WORKDIR.

This allows recipes containing several sources to be built under a
devtool context, but it has some limitations:
* user would not be able to generate patches for the secondary sources
* type="git-dependency" is added for secondary sources even on non git
  sources, so we may want to rename this parameter

(From OE-Core rev: cfd5ee890163a3d975093359016dda104e7b71df)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-24 15:46:19 +00:00
Vyacheslav Yurkov
5eb10c53d8 recipetool: Disregard version in URL for replaced modules
Major module version is a part of name, but not necessary part of the
actual URL (See https://go.dev/ref/mod#module-path).

Nevertheless, name detection function can't handle that suffix, so get
rid of it to determine component name.

For replaced modules that name might be different that the actual module
name defined in go.mod file.

(From OE-Core rev: 0cccfa1041d48f0ae3a2dc89a129cf7884fc08f0)

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-19 12:21:22 +00:00
Vyacheslav Yurkov
0576bd1edd recipetool: Proceed even with a missing license file
Whenever the recipe uses a CLOSED license, the list is going to be
empty. It's a discouraged practice not to have a license, but proceed
anyway to finish recipe generation.

(From OE-Core rev: 5ca920284d0946346f5b06f5e443c80d9d8b85ce)

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-19 12:21:22 +00:00
Vyacheslav Yurkov
0161d08ac9 recipetool: Don't fail on local go modules
Local modules are usually referenced with a 'replace' directive in
go.mod file. If that's the case, remove them from populating SRC_URI.

(From OE-Core rev: 9f220f61e3e44a650a46ee997b47f1d87b7c4ef0)

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-19 12:21:22 +00:00
Alexander Kanavin
f01ea3700e devtool/standard: correctly escape \
python 3.12 points out that:

SyntaxWarning: invalid escape sequence '\*'

(From OE-Core rev: bafb4b4edb4fb7908fdda272b7b2c2cbdef4728b)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-09 22:59:28 +00:00
Jamin Lin
64d5db2f89 devtool: modify: fix exception
Root Cause:
initial_revs is an empty dictionary and do not have "." key.

Traceback (most recent call last):
  File "scripts/devtool", line 349, in <module>
    ret = main()
  File "scripts/devtool", line 336, in main
    ret = args.func(args, config, basepath, workspace)
  File "scripts/lib/devtool/standard.py", line 922, in modify
    if not initial_revs["."]:
KeyError: '.'

Solution:
check key exists, then get its value.

(From OE-Core rev: fb0db5c48abb4d56233a175fdd349d18b972e452)

Signed-off-by: Jamin Lin <jamin_lin@aspeedtech.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-07 12:24:57 +00:00
Mikko Rapeli
339ccc0c2b runqemu: match .rootfs. in addition to -image- for rootfs
Also change path.exists() and !path.isdir() to a single
path.isfile() which should be equal.

Enables running tests against image recipes which are not called
"bla-image" but plain "bla". Currently they fail with do_testimage/runqemu
error:

runqemu - ERROR - Unknown path arg /home/builder/src/base/build/tmp_qemuarm64/deploy/images/qemuarm64/img-qemuarm64.rootfs.wic

Suggested-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(From OE-Core rev: 107d0db3dc3678e6f0ae4035f4c0b86c6b421168)

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-04 23:47:51 +00:00
Trevor Gamblin
3c71605a77 scripts/runqemu: fix regex escape sequences
When invoking runqemu with Python 3.12, the following warning is
encountered:

|SyntaxWarning: invalid escape sequence '\.'

This is because the interpreter scans the string before it is processed
by the regex module, and it interprets the backslash as part of an
escape sequence, but not a standard one. This will be registered as an
error rather than a warning in future Python versions. To avoid the it,
simply add an extra backslash so that Python doesn't misinterpret the
string, while the regex parser still sees an escaped '.' character.

(From OE-Core rev: 0e8a4142bb90a92d175df6b2537d24a372356f98)

Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-02 22:51:01 +00:00
Chen Qi
bd05979aa8 devtool: use straight print in check-upgrade-status output
'devtool check-upgrade-status' is for reporting upgradable
status for recipes. The output should always be printed out. So
we should just use 'print' instead of 'logger.info' as the latter
will be suppressed if '-q' parameter is supplied to devtool.

(From OE-Core rev: 2c7bf9c8a833bec13a1ebabdce30933cbe691108)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-02 22:51:01 +00:00
Richard Purdie
6ce42d73a3 scripts: Drop shell sstate-cache-management
Since there is a faster python version, drop the slower shell one.

(From OE-Core rev: 0551fa2ae3dacf51825ff5c1d081e713a150ddd2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-28 10:57:49 +00:00
Alex Kiernan
b723fcaac5 sstate-cache-management: Rewrite in python
This (should be) a drop in replacement for sstate-cache-management.sh.

(From OE-Core rev: 2fa1b25d7485bfbb92bcc33067beb6751218b36a)

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-28 10:57:49 +00:00
Clay Chang
53b8ae2679 devtool: deploy: provide max_process to strip_execs
Use oe.utils.get_bb_number_threads to get max_process

(From OE-Core rev: f0056dca0a44c374f1f0c5fccbf66ae88e0b1850)

Signed-off-by: Clay Chang <clayc@hpe.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-23 08:46:00 +00:00
venkata pyla
bc8ca0bacc wic: use E2FSPROGS_FAKE_TIME and hash_seed to generate reproducible ext4 images
E2FSPROGS_FAKE_TIME: sets fixed times for the inodes in the file system.
hash_seed: creates reproducible directory indexes in the file system.

Reference commit in e2fsprogs: e1f7100643a46456be107b33098f6034b0835e6d

(From OE-Core rev: bb822ab75de0020572058090439b93cc56bbf7e0)

Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-21 10:38:30 +00:00
Tim Orling
9707dc5549 recipetool: add python_maturin support
Add the new python_maturin PEP-517 backend

Add selftest for 'pydantic-core' pypi package.

(From OE-Core rev: 69b679380616a94a631681caa05d9bf7610f9372)

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-17 19:07:21 +00:00
Peter Kjellerstedt
53d45455a2 devtool: modify: Handle recipes with a menuconfig task correctly
This avoids the following error when running `devtool modify` on a
recipe that has a menuconfig task, but does not have
KCONFIG_CONFIG_ENABLE_MENUCONFIG set.

  .../temp/run.do_configure.4163366: line 152:
  ${@ oe.types.boolean('${KCONFIG_CONFIG_ENABLE_MENUCONFIG}') }: bad substitution
  WARNING: .../temp/run.do_configure.4163366:152 exit 1 from
  '[ ${@ oe.types.boolean('${KCONFIG_CONFIG_ENABLE_MENUCONFIG}') } = True ]'

(From OE-Core rev: e87e6fa84a0c4b5ac8e736dc62f6e08390ba2436)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-16 13:05:42 +00:00
Ross Burton
942f0909f3 runqemu: add qmp socket support
Add support for qmp sockets and defaults to unix:qmp.sock if unspecified

(From OE-Core rev: 380631797f0d63124a8c21efa93ab672dbd79283)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Eilís 'pidge' Ní Fhlannagáin <pidge@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-15 11:14:38 +00:00
Tim Orling
78ef0313ee recipetool: pypi: do not clobber SRC_URI checksums
The pypi change:
"85a2a6f68af recipetool: create_buildsys_python: add pypi support"
deleted all the SRC_URI variables, including the SRC_URI checksums.
These are not generated by the pypi.bbclass (how could they be trusted?)

Without the checksum(s), we are vulnerable to a man-in-the-middle attack
and zero checks on the validity of the downloaded tarball from pypi.org.

Fix by only setting S and SRC_URI to None.

(From OE-Core rev: 560181a52111569f7bc57b09139b42510e0d0325)

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-09 19:17:11 +00:00
Peter Kjellerstedt
9a5c1456ba devtool: modify: Make --no-extract work again
This avoids the following error when using --no-extract, introduced in
commit 900129cbdf (devtool: add support for git submodules):

  Traceback (most recent call last):
    File ".../scripts/devtool", line 349, in <module>
      ret = main()
    File ".../scripts/devtool", line 336, in main
      ret = args.func(args, config, basepath, workspace)
    File ".../scripts/lib/devtool/standard.py", line 995, in modify
      for commit in commits[name]:
  KeyError: '.'

(From OE-Core rev: 6a44fb10ef4b90ee24f470362007d56183abc593)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-08 16:58:34 +00:00
Peter Kjellerstedt
dd2fb8f00a devtool: upgrade: Update all existing checksums for the SRC_URI
In addition to updating the sha256sum and removing the md5sum, update
all other existing checksums. If the only existing checksum is md5sum,
then replace it with the default expected checksums (currently only
sha256sum).

(From OE-Core rev: 8ea8827ee49b7f0443b1c4bd47d1344a689d73a3)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-08 16:58:34 +00:00
Peter Kjellerstedt
45d2f8d4bc recipetool: create: Only include the expected SRC_URI checksums
Rather than including all SRC_URI checksums, include the ones that are
expected. These are the same as are output if no checksums are included
when building the recipe.

(From OE-Core rev: c2af83eb5e8573480179b6c0bcce50606b547099)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-08 16:58:34 +00:00
Peter Kjellerstedt
decf6e66df recipetool: create: Improve identification of licenses
Rather than having a static list of crunched MD5 checksums for some of
the most common licenses, calculate it for all common licenses. This
should improve the identification of license text variantions.

(From OE-Core rev: 377f9513dc56e9b8e5f5813c1535be0206756949)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-08 16:58:34 +00:00
Zoltan Boszormenyi
e2165e26db update_gtk_icon_cache: Fix for GTK4-only builds
Try to execute both gtk-update-icon-cache and
gtk4-update-icon-cache after checking whether the
commands are available.

This attempts to match what gtk-icon-cache.bbclass is doing.

This fixes running update_gtk_icon_cache during do_rootfs
for an image that contains only GTK4 related packages.

(From OE-Core rev: 1e01ef59a5864f1261e9c0ca76fcccee2eb7e7f3)

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-08 16:58:34 +00:00
Julien Stephan
bc9291c5fd recipetool: appendsrcfile(s): add a mode to update the recipe itself
Add a new option -u/--update-recipe to update the recipe itself instead
of creating/updating a bbappend file

(From OE-Core rev: c9a80f55ed0bf9ce249205d6a9ba670fbb7f59ba)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-06 22:55:50 +00:00
Julien Stephan
29dc0d7315 recipetool: appendsrcfile(s): use params instead of extraline
appendsrc function relies on oe.recipeutils.bbappend_recipe to
copy files and add the corresponding entries in SRC_URI.

Currently, appendsrc function build itself the new SRC_URI entry to add the
correct subdir param, and gives it using the extralines parameter.
This has 2 drawbacks:
- oe.recipeutils.bbappend_recipe can already do this if we specify the
  correct params, so we have duplicate code
- the duplicated code is not fully functional: for example, it doesn't
  take into account the -m/--machine parameter

So fix this by not using extralines but give correctly formatted params.

Also remove the check for already existing entries as
oe.recipeutils.bbappend_recipe already implement it

The new bbappend file now have the SRC_URI entry after the
FILESEXTRAPATHS so fix the selftest.

Update test_recipetool_appendsrcfile_existing_in_src_uri_diff_params
test because recipetool appendsrcfiles used to not add new src_uri entry
if the entry already exist even with different parameters while
oe.recipeutils.bbappend_recipe adds it if parameters are different (and
remove the old entry)

(From OE-Core rev: cd5de8d53849a6f3bb6f82e45fb301e39892c789)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-06 22:55:49 +00:00
Julien Stephan
de5ab8e863 recipetool: appendsrcfile(s): add dry-run mode
Add dry-run mode for recipetool appendsrcfile and appendsrcfiles, with
if necessary, a diff of changes

(From OE-Core rev: 456c726e6b52f4dc57ff605e1cf1687097537002)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-06 22:55:49 +00:00
Malte Schmidt
29b2bda786 wic: extend empty plugin with options to write zeros to partiton
Adds features to explicitly write zeros to the start of the
partition. This is useful to overwrite old content like
filesystem signatures which may be re-recognized otherwise.

The new features can be enabled with
'--soucreparams="[fill|size=<N>[S|s|K|k|M|G]][,][bs=<N>[S|s|K|k|M|G]]"'
Conflicting or missing options throw errors.

The features are:
- fill
  Fill the entire partition with zeros. Requires '--fixed-size' option
  to be set.
- size=<N>[S|s|K|k|M|G]
  Set the first N bytes of the partition to zero. Default unit is 'K'.
- bs=<N>[S|s|K|k|M|G]
  Write at most N bytes at a time during source file creation.
  Defaults to '1M'. Default unit is 'K'.

(From OE-Core rev: d19d4529e7a2056caeb526fed980cc1df19a5f6e)

Signed-off-by: Malte Schmidt <malte.schmidt@weidmueller.com>
Signed-off-by: Lukas Funke <lukas.funke@weidmueller.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-06 22:55:49 +00:00
Julien Stephan
85a2a6f68a recipetool: create_buildsys_python: add pypi support
Today, we can use devtool/recipetool to create recipes for python projects
using the github url or the direct release tarball of the project, but the
create_buildsys_python plugin doesn't support the pypi class, since we cannot
know from the extracted source if the package is available on pypi or not.

By implementing the new optional process_url callback, we can detect
that the url is a pypi one (i.e 'https://pypi.org/project/<package>')
and retrieve the release tarball location.
Also detect if the url points to a release tarball hosted on
"files.pythonhosted.iorg" (i.e https://files.pythonhosted.org/packages/...)

In both cases, adds the pypi class, remove 'S' and 'SRC_URIxxx'
variables from the created recipe as they will be handled by the pypi class
and add the PYPI_PACKAGE variable

This helps to produce cleaner recipes when package is hosted on pypi.

If the url points to a github url or a release tarball not coming from
"files.pythonhosted.org", the created recipe is the same as before.
One can also use the newly added "--no-pypi" switch to NOT inherit
from pypi class on matching url, to keep legacy behaviour.

To create a recipe for a pypi package, one can now use one of the
new following syntax (using recipetool create / devtool add):

* recipetool create https://pypi.org/project/<package>
* recipetool create https://pypi.org/project/<package>/<version>
* recipetool create https://pypi.org/project/<package> --version <version>

or the old syntax:
* recipetool create https://files.pythonhosted.org/packages/<...>

(From OE-Core rev: 097a43846cd99a7d74d004efc57f583ce78970a4)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-06 22:55:49 +00:00
Julien Stephan
6c06fb0a43 recipetool: create: add new optional process_url callback for plugins
Add a new process_url callback that plugins can optionally implement if
they which to handle url.

Plugins can implement this callback for example, to:
* transform the url
* add special variables using extravalues
* add extra classes
* ...

If a plugin handles the url, it must append 'url' to the handled
list and must return the fetchuri

No functional changes expected for plugins non implementing this
optional callback

(From OE-Core rev: 3e5ce351e7dbe283562bf1db1f2ce3b121c49b53)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-06 22:55:49 +00:00
Julien Stephan
86b9510bd3 recipetool: create: add trailing newlines
create_recipe() function relies on oe.recipeutils.patch_recipe_lines()
which relies on bb.utils.edit_metadata(). edit_metada expect lines to
have trailing newlines, so add it to each lines before calling
patch_recipe_lines, otherwise edit_metadata will not be able to squash
blank line if there are two consecutive blanks after a removal

(From OE-Core rev: 31b27cc76a1b669e2b126c332608fd772c124d69)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-06 22:55:49 +00:00
Julien Stephan
344e10a21b recipetool: create_buildsys_python.py: initialize metadata
In the case pyproject.toml doesn't contains metadatas, the metadata
variable is not initialized and the plugin throws an error and falls back
to another plugin, which is not the desired behaviour. So just ignore
metadata if we don't have them

(From OE-Core rev: 88d15877ba18309c521740d7a9649e14d77189bc)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-06 22:55:49 +00:00
Kareem Zarka
0a39f1aa09 wic: bootimg-efi: Make kernel image installation configurable
The issue with installing the kernel image to both rootfs
and boot partition is that some systems rely on the kernel image in
rootfs and not in the boot partition.
This leads to duplication of the kernel image, which can cause
unnecessary storage usage.
This patch provides a solution to the problem by adding a new
parameter "install-kernel-into-boot-dir" to the wic kickstart file.
If this parameter is set to 'true', the plugin will install the
kernel image to the boot partition. If the parameter is set to
'false', the plugin will skip installing the kernel image, avoiding
duplication.

(From OE-Core rev: d3599afe5f604ea5afd9411e114934dcb52b2d48)

Signed-off-by: Kareem Zarka <kareem.zarka@huawei.com>
Signed-off-by: Christian Taedcke <christian.taedcke@weidmueller.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-04 11:23:37 +00:00
Julien Stephan
89f1662484 devtool: add support for git submodules
Adding the support of submodules required a lot of changes on the
internal data structures:
* initial_rev/startcommit used as a starting point for looking at new
  / updated commits was replaced by a dictionary where the keys are the
  submodule name ("." for main repo) and the values are the
  initial_rev/startcommit

* the extractPatches function now extracts patch for the main repo and
  for all submodules and stores them in a hierarchical way describing the
    submodule path

* store initial_rev/commit also for all submodules inside the recipe
  bbappend file

* _export_patches now returns dictionaries that contains the 'patchdir'
  parameter (if any). This parameter is used to add the correct
  'patchdir=' parameter on the recipe

Also, recipe can extract a secondary git tree inside the workdir.

By default, at the end of the do_patch function, there is a hook in
devtool that commits everything that was modified to have a clean
repository. It uses the command: "git add .; git commit ..."

The issue here is that, it adds the secondary git tree as a submodule
but in a wrong way. Doing "git add <git dir>" declares a submodule but do
not adds a url associated to it, and all following "git submodule foreach"
commands will fail.

So detect that a git tree was extracted inside S and correctly add it
using "git submodule add <url> <path>", so that it will be considered as a
regular git submodule

(From OE-Core rev: 900129cbdf25297a42ab5dbd02d1adbea405c935)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-01 11:48:25 +00:00
Julien Stephan
17427db136 devtool: tag all submodules
In the case of a repository with submodules, we need to add the
"devtool-base" and "devtool-patched" tag on all submodules in order to
properly detect the added/removed/modified patches

(From OE-Core rev: 241da68805d177d4ec4b302c8a997645cc645286)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-01 11:48:25 +00:00
Julien Stephan
1ae4cc7a11 devtool: finish/update-recipe: restrict mode srcrev to recipes fetched from SCM
When specifying --mode / -m srcrev with devtool finish/update-recipe on
recipes that are not fetched from a SCM repository we get the following
error:

  Traceback (most recent call last):
  [..]
    File "<...>/poky/meta/lib/oe/patch.py", line 49, in runcmd
    raise CmdError(cmd, exitstatus >> 8, "stdout: %s\nstderr: %s" % (stdout, stderr))
oe.patch.CmdError: Command Error: 'sh -c 'git format-patch --no-signature --no-numbered INVALID -o /tmp/oepatchbj7pfmzj -- .'' exited with 0  Output:
stdout:
stderr: fatal: bad revision 'INVALID'

Fix this by adding a check and abort with a proper error message.

(From OE-Core rev: 5b77f60e85d07921ae5e808daa1fd7d8c3dc40ea)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-01 11:48:25 +00:00
Julien Stephan
fcff067407 devtool: fix update-recipe dry-run mode
When running devtool update-recipe with --mode=srcrev AND --append switch
in dry-run, we get the following error:

  Traceback (most recent call last):
  [...]
  Exception: destpath should be set here

Fix this by removing a misplaced else statement in _update_recipe_srcrev

(From OE-Core rev: 85ba125703d5b442133fd7c470b915460ee68ac9)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-30 08:43:04 +00:00
Malte Schmidt
5d12c0a3dc wic: rawcopy: add support for zstd decompression
Add support for zstd decompression in rawcopy plugin. zstd claims
to reach higher, uniform decompression rates.

(From OE-Core rev: 707471b71a417187f1e581d0c540e058825f7242)

Signed-off-by: Malte Schmidt <malte.schmidt@weidmueller.com>
Signed-off-by: Lukas Funke <lukas.funke@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-27 22:54:59 +00:00
Lukas Funke
7de36b25d1 recipetool: Add handler to create go recipes
[RP: Replace python 3.9 use of removeprefix()]
(From OE-Core rev: 049afb933f42851d56d2e6522150a32bcccf2a6f)

Signed-off-by: Lukas Funke <lukas.funke@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-06 16:41:11 +00:00
Lukas Funke
f7f304fa3e recipetool: Ignore *.go files while scanning for licenses
(From OE-Core rev: 1765acc8bdc08e392daa8b77add8b4ce1a9e70b6)

Signed-off-by: Lukas Funke <lukas.funke@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-06 16:41:11 +00:00
Alexis Lothoré
838b8e78dc scripts/resulttool: group all regressions in regression report
Commit c304fcbe05 introduced a grouping when
listing regressions. This grouping has been added only for ptests. It has
been observed that any other kind of tests could benefit from it. For
example, current regression reports can show the following:

1 regression(s) for oescripts.OEGitproxyTests.test_oegitproxy_proxy_dash
    oescripts.OEGitproxyTests.test_oegitproxy_proxy_dash: PASSED -> SKIPPED
1 regression(s) for oescripts.OEPybootchartguyTests.test_pybootchartguy_help
    oescripts.OEPybootchartguyTests.test_pybootchartguy_help: PASSED -> SKIPPED
1 regression(s) for oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output
    oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: PASSED -> SKIPPED
1 regression(s) for oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output
    oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: PASSED -> SKIPPED
1 regression(s) for oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output
    oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: PASSED -> SKIPPED
[...]

This output is not so useful in its current state and should be grouped per
test type too.
Enable grouping for all kind of tests, to make it llok like the following
in reports:
5 regression(s) for oescripts
    oescripts.OEGitproxyTests.test_oegitproxy_proxy_dash: PASSED -> SKIPPED
    oescripts.OEPybootchartguyTests.test_pybootchartguy_help: PASSED -> SKIPPED
    oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_pdf_output: PASSED -> SKIPPED
    oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_png_output: PASSED -> SKIPPED
    oescripts.OEPybootchartguyTests.test_pybootchartguy_to_generate_build_svg_output: PASSED -> SKIPPED

(From OE-Core rev: 982798ef96e3a32bf15341bdd3bb7c4356709412)

Signed-off-by: Alexis Lothoré <alexis.lothore@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-05 08:44:06 +00:00
Alexander Kanavin
81a400a8f5 scripts/esdk-tools: use a dedicated, static directory for esdk tools
This allows easier replication of esdk environment (which provides
a curated, limited set of tools that for example does not include bitbake)
in a standard yocto build. Switchover between various sets can be achieved
via PATH manipulation.

(From OE-Core rev: 20c548f2edca3888152adb63de7b23d84e3848e7)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-05 08:44:06 +00:00
Trevor Gamblin
b53d1287d2 patchtest-send-results: fix sender parsing
Not all mbox 'from' fields will contain angle brackets, so the
re.findall invocation used for getting a reply_address may fail. Use a
simpler reference to the field to get the sender's email address.

(From OE-Core rev: 86e9afe09a346586114133f5a7470304d2ed733f)

Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-02 21:48:30 +00:00
Trevor Gamblin
2bddb6bb97 patchtest-send-results: add In-Reply-To
Rework the script for sending results to use send_raw_email and specify
the 'In-Reply-To' field so that patchtest replies to the emails, rather
than sending them standalone to the submitter and mailing list.

(From OE-Core rev: 0c45c92e7f26aea4edf2cfa577b7ba51384e59d3)

Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-01 19:19:40 +00:00
Trevor Gamblin
a88b6d2452 patchtest-send-results: send results to submitter
Modify patchtest-send-results so that it extracts the submitter's email
address and responds to them with the patch testresults. Also make a
minor adjustment to the suggestions provided with each email and include
a link to the Patchtest wiki page for additional clarification on
specific failures.

(From OE-Core rev: 64ed88e32cf9e04772319ff6e66c602d1cff4fd7)

Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-10-30 16:38:15 +00:00
Ross Burton
28324600cb scripts/contrib/patchreview: fix commit identification
git show-ref looks at the _remote_ ref called HEAD, which is fine when it
matches the local HEAD but problematic when you're iterating a series of
commits.

Use rev-parse to resolve the local name to a proper hash.

(From OE-Core rev: 3c04747b681cf6090ba9c77752f6c2f304dbbe17)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-10-30 11:53:20 +00:00