Commit Graph

3143 Commits

Author SHA1 Message Date
Yeoh Ee Peng
f2d5800563 resulttool/store.py: Enable add extra test environment data
Enable the option to add extra test environment data to the
configuration of each test result (as optional).

Example of optional test environment data include:
	- custom packages included for runtime test
	- detail machine specification used as target
	- detail host environment used for bitbake

(From OE-Core rev: 3cc887a72c5726024c3699e6a3deb72fad4992a9)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:20:59 +00:00
Yeoh Ee Peng
610b02a2fb scripts/resulttool/report: Add total statistic to test result.
Add total passed, failed, and skipped statistic to test result.

(From OE-Core rev: 7ad74f99b6105f23bcf930712627aac279c6af10)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:20:59 +00:00
Yeoh Ee Peng
565b05b0e4 scripts/resulttool/report: Enable output raw test results
In case of debugging, report user need to acccess the raw
test result. Instead of going back to source file/directory/URL
to manually pull out the raw result, provide alternative
way to let report showing raw test results by providing
the result id (optional).

(From OE-Core rev: 8b2485bc0d0fa2f72b37e8bafbfa663d456ea98c)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:20:59 +00:00
Yeoh Ee Peng
3adefee2a1 scripts/resulttool/report: Enable report to use regression_map
By default, report will use the store_map to generate the key
to reference each result set. In some situation when using store_map
with multiple set of tests sharing similar test configurations,
the report will only showing partial result set for results
that having identical result_id (use of multiconfig to run tests
where it generate identical result_id).

Enable report to have the option to use the regression_map (optional)
instead of the default store_map, where it will take larger
set of configurations to generate the key to reference each
result set, this will prevent the report from only showing
partial result set.

(From OE-Core rev: 8577abce6feb1f529537da554d39e86a5b05ca2d)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:20:59 +00:00
Chee Yang Lee
621b10f007 wic: beautify 'wic help'
The Wic help returned to the user is unreadable.

Use a custom ArgumentParser to override argparse help message.

change help message as suggest in
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12205
[YOCTO #12205]

changes applies to 'wic help', 'wic -h', 'wic --h' and 'wic --help'

(From OE-Core rev: d29d553cc096f4e1208c44dc595e1cf365c3dff3)

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-14 13:20:59 +00:00
Haris Okanovic
a98c7c7aa3 isoimage-isohybrid.py: Parameterize ESP partition size
Add "esp_extra_blocks" plugin parameter so that caller may change
ESP's free space from the default 100 blocks.

(From OE-Core rev: db4be094b0d6741ca4f9e2cbb499fc9c224fa690)

Signed-off-by: Haris Okanovic <haris.okanovic@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-10 14:06:10 +00:00
Haris Okanovic
36bb63c3d2 isoimage-isohybrid.py: Parameterize ESP label
Add "esp_label" plugin parameter so that caller may override default
ESP partition label 'EFIimg'.

(From OE-Core rev: 6b3b9042a55d47205b16ef792f93a9bb240fd7cb)

Signed-off-by: Haris Okanovic <haris.okanovic@ni.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-10 14:06:10 +00:00
Yeoh Ee Peng
7035b4b21e scripts/oe-pkgdata-util: Enable list-pkgs to print ordered packages
The list-pkgs currently print packages in unordered format.
Enable list-pkgs to print ordered packages that will ease
viewing.

(From OE-Core rev: ab661f96e2fd529588692c978274a840f34ee7cf)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-04 13:39:05 +00:00
Chee Yang Lee
a94e0e14e1 wic/engine: use 'linux-swap' for swap file system
[YOCTO #13312]
see https://bugzilla.yoctoproject.org/show_bug.cgi?id=13312

wic/engine.Disk._get_part_image was looking at variable fstypes for
supported fstype which is 'swap' but image build with 'linux-swap'.
supported fstype should be 'linux-swap'.

(From OE-Core rev: 7e6da22fe4faf841bcec02e55f376b4dae04d6a8)

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-23 16:30:36 +01:00
Chee Yang Lee
8ec9acb55d wic/help: change 'wic write' help description
update the help desciption for 'wic write' command to
match behaviour.

see commit:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/scripts/wic?id=74b44e59be89c738f702948f9e16e367c0ba43ba

(From OE-Core rev: 521942bde1d49d987c85bf071275e8e7150d012a)

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-23 16:30:36 +01:00
Eduardo Abinader
3e0c8798af devtool: add ssh key option to deploy-target param
(From OE-Core rev: 729da1414098c2dd103d9749cf39117881cafeec)

Signed-off-by: Eduardo Abinader <eduardoabinader@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-23 16:30:35 +01:00
Sai Hari Chandana Kalluri
586175df3b devtool: Add --remove-work option for devtool reset command
Enable --remove-work option for devtool reset command that allows user
to clean up source directory within workspace.

Currently devtool reset command only removes recipes and user is forced
to manually remove the sources directory within the workspace before
running devtool modify again.

Using devtool reset -r or devtool reset --remove-work option, user can
cleanup the sources directory along with the recipe instead of manually
cleaning it.

syntax: devtool reset -r <recipename>
    Ex: devtool reset -r zip

	devtool finish -r <recipename> <layer-name>
    Ex: devtool finish -r zip meta-yocto-bsp

(From OE-Core rev: 0d6dd01b8633ad63d77c24568a4b20852aaa366f)

Signed-off-by: Sai Hari Chandana Kalluri <chandana.kalluri@xilinx.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-23 16:30:35 +01:00
Eugene Smirnov
8d619931ee wic/rawcopy: Support files in sub-directories
If the source file is located in a subdirectory of DEPLOY_DIR
rawcopy will currently fail in sparse_copy function on
open(dst_fname, 'wb'), as the parent directory for destination
file does not exist.

This patch helps to avoid that by recursively creating
parent directories.

(From OE-Core rev: 073c435644091c2801e45c6d02afa917de575082)

Signed-off-by: Eugene Smirnov <evgenii.smirnov@here.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-19 23:18:33 +01:00
Joerg Vehlow
7ed5aa23c7 runqemu: Remove disabling of high resolution timer
The option 'highres=off' sneaked itself into the runqemu script for all
configurations, where the root filesystem type is not 'cpio' or 'cpio.gz'.
See: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13590

(From OE-Core rev: 3ecb6744402332f1ad143eb1eb6a94143e1700c8)

Signed-off-by: Joerg Vehlow <joerg.vehlow@aox-tech.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-15 14:16:11 +01:00
Alexander Kanavin
8c4a9f5b65 runqemu: unset another environment variable for 'egl-headless'
Some host distributions (opensuse for example) are using 'pkgconf',
which, unlike the original pkg-config, appends PKG_CONFIG_SYSROOT_DIR
to every directory from the .pc file.

(From OE-Core rev: 2d5ecc560dcfbaaaff307f8367d789d8c7e00dcf)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-15 14:16:10 +01:00
Richard Purdie
4964e42e19 scripts/gen-lockedsig-cache: Don't list paths which don't exist
This avoids failures seen on the autobuilder when generating eSDKs
and release sstate copies.

(From OE-Core rev: 8ffe6ca984e034976322beea9b16c92d46708d15)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-09 22:28:37 +01:00
Michael Cooper
7a1d68d136 wic/direct: Partition numbering is broken for MBR primary partition #4
When wks describes extra partitions that aren't in the partition
table (e.g. boot loader) and exactly four primary MBR partitions, the
last partition gets added to fstab as partition #5 instead of #4.

[YOCTO #13560]

(From OE-Core rev: 7537580b3dd21bd512fb26e56e92b6553c549fa8)

Signed-off-by: Michael Cooper <michaelcooper81@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-08 11:22:24 +01:00
Jaewon Lee
41d225f4a3 devtool/standard.py: Not filtering devtool workspace for devtool finish
All devtool commands right now are filtering out the devtool workspace
bbappends in build/workspace/appends when calling parse_recipe. While
this may make sense for devtool add and modify, we need devtool finish
to include those appends.

A specific breakage that is caused because devtool finish filters devtool
appends is the cmake/cml1 flow where a file is created in the WORKDIR
that finish needs access to, to commit those files. Particularly for
git packages with SRCPV in PV, SRCPV is only changed to 999 when using
external source, hence when creating the cfg or cmake config files using
for instance bitbake -c diffconfig, these files are created in the
git999 workdir correctly (as in the devtool bbapends, we are inheriting
externalsrc class). But when devtool finish is run, the devtool appends
are not parsed, hence SRCPV is not changed to 999 and devtool is looking
for the fragment files in the wrong WORKDIR.

Changing the parse_recipe call just in devtool finish to not filter out
the devtool workspace.

Fixes [YOCTO #13533]

(From OE-Core rev: aa94f00eec64ef936acc80dfa826fd309daa294f)

Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-08 11:22:24 +01:00
Konrad Scherer
e1c173621a gen-lockedsig-cache: Replace glob lookup with hash to filename lookup
Using the glob function to map signatures to sstate files is very slow
when the sstate is large and accessed over nfs. The lookup now only
loads the necessary prefixes and doesn't use glob as all.

Unfortunately I don't have access to the systems where the performance
isse was noticed and on my test system the glob is fast enough that
the performance numbers aren't useful. I could verify that file list
returned by the new code is the same.

[YOCTO #13539]

(From OE-Core rev: ad36335b8592e0387dd36066920cd5ffefd375f8)

Signed-off-by: Konrad Scherer <Konrad.Scherer@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-02 20:22:25 +01:00
Nathan Rossi
ec1104fd7f oeqa/core/case.py: Encode binary data of log
Do not decode the log content into a string only to re-encode it as
binary data again. Some logs might un-intentionally contain bytes that
do not decode as utf-8, as such preserve the log file content as it was
on disk.

Handle the decoding on the resulttool side, but also handle the failure
to decode the data.

(From OE-Core rev: 20531dc0b8f76a6e37cc856f36cd94077b6aba50)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-30 16:55:21 +01:00
Peter Kjellerstedt
5af64b55f3 devtool: finish: Add suppport for the --no-clean option
This works just like the already existing --no-clean option to the
`devtool reset` command.

(From OE-Core rev: 4433d3133ce53cdf7db17f5e7914fe8c4d78dba3)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27 13:02:17 +01:00
Niclas Svensson
a70e1fdba2 devtool: finish: Keep patches ordered when updating bbappend
The _get_patchset_revs() function returns the patches in an
OrderedDict to keep them ordered. However, this information was lost
when the patches were added to the bbappend file.

(From OE-Core rev: 69b7a2ba3af1280cc220ab236032f8466246ee93)

Signed-off-by: Niclas Svensson <niclas.svensson@axis.com>
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27 13:02:16 +01:00
Alessio Igor Bogani
b404c0cd3c wic: Using the right rootfs size during prepare_rootfs
The commit 8e48b4d6c4 makes wic ignores IMAGE_ROOTFS_SIZE for rootfs
size and makes it uses the computed one only. Re-add support for
IMAGE_ROOTFS_SIZE variable and compute roots size only if the former
is not defined. Moreover the size of a provided directory with
--rootfs-dir="" in the .wks file should always be computed on the fly,
else every partition will be constrained to be the same size as what
ever value was in ROOTFS_SIZE.

(From OE-Core rev: 0217a0392b8cc534fcc0d760b7663a1845b306f5)

Signed-off-by: Alessio Igor Bogani <alessio.bogani@elettra.eu>
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27 13:02:16 +01:00
Richard Purdie
cf37ca58c2 runqemu: Mention snapshot in the help output
This is a useful option but not documented in the help text.

(From OE-Core rev: 95bad67a802f956082bd0af5615c1dbaae01be0f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-18 12:09:02 +01:00
Alexander Kanavin
63f08ef496 runqemu: decouple gtk and gl options
This will allow not having to multiply these options for the sdl
frontend, instead combining them as needed.

(From OE-Core rev: 922eb5012364b1603338cfa617712b941e892bbf)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:44 +01:00
Henning Schild
4b0bf01c14 oe-git-proxy: fix dash "Bad substitution"
The script claims it works with dash, make sure that is actually the
case.

(From OE-Core rev: cb373201464f4a0a90482f62a24a4043abe73fd6)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:44 +01:00
Henning Schild
7ffd9ee9c3 oe-git-proxy: NO_PROXY suffix matching without wildcard for match_host
NO_PROXY can also contain just suffixes that do not start with a "*". We
failed to match those so far. Just add an extra "*" to also match those
suffixes. If one was there we get "**" which does not hurt.

(From OE-Core rev: 12f0cc209aaba48f846c62663e0b9e5efd253d71)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:44 +01:00
Henning Schild
1b970e222e oe-git-proxy: disable shell pathname expansion for the whole script
This truly fixes the issue that cbc148d5d9
wanted to solve, without breaking the iteration over multiple entries.

(From OE-Core rev: 477ee7e673684db988c66a75b6400e33509730b4)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:44 +01:00
Henning Schild
f107c9f815 Revert "oe-git-proxy: Avoid resolving NO_PROXY against local files"
This reverts commit cbc148d5d9.

The quoting causes H to be one string with spaces, so looping over
multiple entries does not work anymore.

(From OE-Core rev: 0fdc12dac6244be135ea519fe9c39109e7cfc6d6)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:44 +01:00
Henning Schild
49bb6cefb5 oe-git-proxy: allow setting SOCAT from outside
This allows to write selftests where we can mock the real socat.

(From OE-Core rev: ea2690e867ff11250d3dd143184432dd03909910)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:43 +01:00
Nathan Rossi
e08f657220 oeqa/core/case.py: Add OEPTestResultTestCase for ptestresult helpers
Add the OEPTestResultTestCase class as a mix-in class to provide helper
functions for interacting with ptestresults within the extraresults
object generated by the test case.

This class also provides default compression of log text and log files.

Also add support to resulttool for decoding/decompressing log files
embedded in the test results.

(From OE-Core rev: 06cba9883a5964320969301fd05eeb6bec3e786d)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:43 +01:00
Nathan Rossi
05e583ae84 resulttool: Handle multiple series containing ptestresults
Handle multiple results series having ptestresults content. The contents
are merged on a per-result basis where duplicates are ignored (with a
warning message printed). The 'ptestresults.sections' collection is also
merged on a per-suite basis.

(From OE-Core rev: 47edd51970ed0c33edbe04fd72abd1cfc6ecd3d1)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-16 23:02:43 +01:00
Joshua Watt
bff47225c6 resulttool: Add reproducible log extraction
Adds an argument to the log subcommand to extract the raw logs from the
reproducible selftest.

To prevent ambiguity, the "--raw" argument has been renamed
"--raw-ptest", although the old "--raw" argument is kept around for
compatibility.

[YOCTO #13324]

(From OE-Core rev: 7a4ebb361ff1efc22e7dafadfa60c98bc8a79ed4)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 14:58:10 +01:00
Drew Moseley
7ded301c3c runqemu: Fix typo in help text.
(From OE-Core rev: c745185b0c112530d1a3d1c57feea0754d68ebb2)

Signed-off-by: Drew Moseley <drew.moseley@northern.tech>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 14:58:10 +01:00
Nathan Rossi
4364a26db1 oe-selftest: Implement console 'keepalive' output
Similar to bitbake, implement a 'keepalive' output to the console to
ensure CI systems do not kill the process. The default timeout for
bitbake is 5000s.

(From OE-Core rev: 77939cca96fa5467c88eafa3ac0db2db4aef09d6)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 08:15:45 +01:00
Alistair Francis
b8c2575002 qemuriscv: Generate a wic rootFS with a larger filesystem
This allows us to generate a rootFS with a large filesystem for use with
QEMU.

(From OE-Core rev: e06439200e44999c1e2f88d7d6c651da13698ca7)

Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-03 09:55:50 +01:00
Peter Kjellerstedt
397d1432cb devtool: Avoid failure for recipes with S == WORKDIR and no local files
When extracting the sources for a recipe that has S == WORKDIR and no
local files in the SRC_URI (which, e.g., can happen for a recipe with
a URI that has the unpack=false attribute), the extraction fails with
the following backtrace:

  Traceback (most recent call last):
    File ".../scripts/devtool", line 344, in <module>
      ret = main()
    File ".../scripts/devtool", line 331, in main
      ret = args.func(args, config, basepath, workspace)
    File ".../poky/scripts/lib/devtool/standard.py", line 762, in
    modify
      initial_rev, _ = _extract_source(srctree, args.keep_temp,
      args.branch, False, config, basepath, workspace,
      args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides)
    File ".../poky/scripts/lib/devtool/standard.py", line 647, in
    _extract_source
      bb.process.run('git %s commit -a -m "Committing local file
      symlinks\n\n%s"' % (' '.join(useroptions),
      oe.patch.GitApplyTree.ignore_commit_prefix), cwd=srctree)
    File ".../poky/bitbake/lib/bb/process.py", line 178, in run
      raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
  bb.process.ExecutionError: Execution of 'git commit -a -m
  "Committing local file symlinks

  %% ignore"' failed with exit code 1:
  On branch devtool
  nothing to commit, working tree clean

This is because no files were found in the oe-local-files directory
and consequently no symbolic links were added using `git add`, but the
`git commit` command was still executed.

(From OE-Core rev: 3fdf304e72a1fb5de8bf9bc21e5b598fefb08648)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-01 22:33:08 +01:00
Alexander Kanavin
58e85c60cd qemu: switch to '-vga std' emulated hardware from vmware/cirrus for x86/mips
This is the qemu default since qemu 2.2, is generally supported better,
and is recommended by upstream. It also has already been in use for arm/risc
and ovmf.

Additional information:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13466
https://www.kraxel.org/blog/2014/10/qemu-using-cirrus-considered-harmful/

'-vga virtio' emulated hardware remains in use when virgl is enabled via a runqemu override.

Also, adjust the error whitelist, as there is a number of new messages
coming from the drivers that are not actual errors.

(From OE-Core rev: 73cb104f3307736f4922f2e0c9648f9b2d3b3b6b)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-01 22:33:07 +01:00
Mark Hatle
b4932a18ab yocto-check-layer: Allow any case for README file detection
It's become more commone for files to be named "readme" or "Readme" on github servers
in recent time.  So adjust the scanning to allow any mix of case.

(From OE-Core rev: afe46eca15b6ddfa15c75cb7b707d6dd9aae3eae)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-30 17:10:28 +01:00
Alexander Kanavin
b0bad37101 qemu: set default RAM to 256M for all machines
There was a discussion about what amount of RAM is appropriate for a
default; the outcome was that for now it is still 256M. Some qemu machine
definitions have however set this to 512M so for the sake of
treating all architectures fairly, they are reset back to 256M.

Also runqemu is adjusted to use 256M if QB_MEM is not set at all.

http://lists.openembedded.org/pipermail/openembedded-core/2019-August/285900.html

(From OE-Core rev: 04c01b6cc5be3e6d45d0e04571640648a5655a8b)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-30 17:10:28 +01:00
Nathan Rossi
6e48f9aa23 scripts/lib/resulttool/report.py: Add more result types
Add additional result types into the dictionary to handle dejagnu style
test results. These include PASS, FAIL, XPASS, XFAIL, UNSUPPORTED,
UNTESTED, UNRESOLVED and ERROR.

(From OE-Core rev: 684fc36402a23760b203f4761f284043031c799c)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-28 16:44:09 +01:00
Jaewon Lee
8f29534ccb devtool: build: Also run deploy for devtool build if applicable
Right now `devtool build` runs populate_sysroot and packagedata tasks.
Adding deploy to this list, if the recipe has the deploy task, so that
the newly built artifacts are available in the deploy directory.
Applicable only for packages with deploy task, such as kernel.

[YOCTO#13382]

(From OE-Core rev: b38a1328f0c7bc4b4102a05daee4058fd3214489)

Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-21 21:52:59 +01:00
Jon Mason
87138c4c2b resulttool: Prevent multiple results for the same test
Currently, if a test occurs multiple times over different series, the
code will sum these.  This can lead to confusion over the desired
results.  Change the code to report the redundant tests and skip adding
an additional values.

(From OE-Core rev: caeaa0648dff64c56a33f52e45e82bfab6719e3e)

Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-21 21:52:59 +01:00
Alistair Francis
23662395e2 scripts/runqemu: Add support for the BIOS variable
Add support for specifying a BIOS the same way that the KERNEL variable
is specified. This includes specifying a QB_DEFAULT_BIOS variable.

(From OE-Core rev: fc2a2260aa22a81da6619b4affaf8ae0b5556a34)

Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-21 15:29:02 +01:00
Richard Purdie
d67fbf68f6 yocto-check-layer: Ensure we use OEBasicHash as the signature handler
The layer checks are designed to work with OEBasicHash so ensure that handler
is in use rather than the new hash equivalency one as an example.

(From OE-Core rev: a10bf92516a4771e2dc49ba9f74323d7a87a1619)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-21 15:29:01 +01:00
Jason Wessel
7365a605ae wic: Make disk partition size consistently computed
When using different root directories with a wks file wic is using the
value from the original ROOTFS_SIZE which is not correct.  Example:

Number  Start   End     Size    File system  Name       Flags
 1      20.5kB  318MB   318MB   fat16        otaefi     legacy_boot, msftdata
 2      318MB   636MB   318MB   ext4         otaboot
 3      636MB   1709MB  1074MB  ext4         otaroot
 4      1709MB  2027MB  318MB   ext4         otaboot_b
 5      2027MB  3101MB  1074MB  ext4         otaroot_b
 6      3101MB  5249MB  2147MB  ext4         fluxdata

The partitions 1, 2, and 3 incorrectly inherit the size, instead of using
a computed size.  With the patch applied it is working properly:

Number  Start   End     Size    File system  Name       Flags
 1      20.5kB  14.5MB  14.5MB  fat16        otaefi     legacy_boot, msftdata
 2      14.5MB  65.3MB  50.8MB  ext4         otaboot
 3      65.3MB  1139MB  1074MB  ext4         otaroot
 4      1139MB  1190MB  50.8MB  ext4         otaboot_b
 5      1190MB  2264MB  1074MB  ext4         otaroot_b
 6      2264MB  4411MB  2147MB  ext4         fluxdata

As for the removal of the bb.warn, if the size is not specified, it is
not something to warn the end user about.  Some of my default images
make use of the head room + a computed directory size or while generating
images.

(From OE-Core rev: 8e48b4d6c4d0ed213089a7449fea63aa0656e786)

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-12 16:23:57 +01:00
Jason Wessel
351a10ccd6 wic: Add partition type for msdos partition tables
In order to create a msdos partition table disk image that can auto
expand after the image is copied to an SD card, wic needs the ability
to have a primary partition as the last entry.  The desired use case
is to be able to create an A/B update partition image scheme with a
/var volume that can be auto expanded to the remainder of the SD card
at run time.

The typical .wks file will look similar to the following:

bootloader --ptable msdos
part / --source rawcopy --sourceparams="file=u-boot.imx" \
   --ondisk mmcblk --no-table --align 1 --size 1
part /boot --source bootimg-partition \
   --ondisk mmcblk --fstype=vfat --label boot --active --align 4 --size 16
part / --source rawcopy --sourceparams="file=imx6_boot.otaimg" \
   --ondisk mmcblk --fstype=ext4 --label otaboot --align 4 --type logical
part / --source rawcopy --sourceparams="file=imx6.otaimg" \
   --ondisk mmcblk --fstype=ext4 --label otaroot --align 4 --type logical
part / --source rawcopy --sourceparams="file=imx6_boot.otaimg" \
   --ondisk mmcblk --fstype=ext4 --label otaboot_b --align 4 --type logical
part / --source rawcopy --sourceparams="file=imx6.otaimg" \
   --ondisk mmcblk --fstype=ext4 --label otaroot_b --align 4 --type logical
part /var --source rawcopy --sourceparams="file=imx6_var.otaimg" \
    --ondisk mmcblk --fstype=ext4 --label fluxdata --align 4

Without the patch applied, wic will generate an SD card image that looks like:

Disk boot.img: 890940s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start    End      Size     Type      File system  Flags
 1      2056s    48001s   45946s   primary   fat16        lba
 2      48008s   132467s  84460s   primary   ext4
 3      132472s  454467s  321996s  primary   ext4
 4      454471s  890939s  436469s  extended               lba
 5      454472s  538931s  84460s   logical   ext4
 6      538936s  860931s  321996s  logical   ext4
 7      860936s  890939s  30004s   logical   ext4         boot

With the patch applied a primary partition can be created at the end
of the image which can be expanded to fill the free space on the media
where the image has been copied, which looks like:

Disk boot.img: 890940s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start    End      Size     Type      File system  Flags
 1      2056s    48001s   45946s   primary   fat16        lba
 2      48007s   860931s  812925s  extended               lba
 5      48008s   132467s  84460s   logical   ext4
 6      132472s  454467s  321996s  logical   ext4
 7      454472s  538931s  84460s   logical   ext4
 8      538936s  860931s  321996s  logical   ext4
 3      860936s  890939s  30004s   primary   ext4         boot

(From OE-Core rev: 56add7cc547e0113cdf980579d1421b14cc233e5)

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-12 16:23:57 +01:00
Ricardo Neri
9b90717e91 runqemu: Add support to handle EnrollDefaultKeys PK/KEK1 certificate
The EnrollDefaultKeys.efi application (distributed in ovmf-shell-image)
expects the hypervisor to provide a Platform Key and first Key Exchange
Key certificate.

For QEMU, this is done by adding an OEM string in the Type 11 SMBIOS
table. The string contains the EnrollDefaultKeys application GUID followed
by the certificate string. For now, the string is passed in the command
line until QEMU understands OEM strings from regular files (please see
https://bugs.launchpad.net/qemu/+bug/1826200).

If runqemu detects it is given an OVMF binary with support for Secure Boot
(i.e., ovmf.secboot* binaries), extract the certificate string from the
OvmfPkKek1.pem certificate and modify the command-line parameters to
provide the key. Such certificate is created when building OVMF with
support for Secure Boot.

Cc: Ross Burton <ross.burton@intel.com>
Cc: Patrick Ohly <patrick.ohly@intel.com>
(From OE-Core rev: 5e47316ae62f7632fb62bc3b8093ac42f9e3541c)

Signed-off-by: Ricardo Neri <ricardo.neri-calderon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-12 16:23:57 +01:00
Changqing Li
f2a951f479 runqemu: fix get portlock fail for multi users
when runqemu with slirp option on same host with different
users, it will report PermissionError: [Errno 13] Permission
denied: '/tmp/qemu-port-locks/2222.lock'
and during handle this exception, another exception happened since
key not exist. Fix by check if key exist first

(From OE-Core rev: 56f30e5377ebe5cc4544f081e001934706a0d8d3)

Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-03 23:56:01 +01:00
Paul Eggleton
bbd51cbc4a scripts/create-pull-request: fix putting subject containing / into cover letter
If a single-commit series had a shortlog containing a "/" character then
that prevented putting the shortlog into the subject of the cover letter
message. Use a different separating character with the sed command (one
much less likely to appear) in order to fix it.

(From OE-Core rev: d4c3f93d7407ac1ea20b33149f20153972d631c0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-31 13:50:46 +01:00
Paul Eggleton
f3649983d3 scripts/create-pull-request: improve handling of non-SSH remote URLs
When attempting to create a pull request, we look at the remote URL in
order to extract information to include in the cover letter. However,
the assumption was that the remote is an SSH URL i.e. containing '@'
which is not always the case (the pull and push URLs might be different,
or we might be pushing via https) - if it wasn't the script just gave up
leaving you to manually edit the URLs in the email. With a few minor
tweaks to the regexes the script will work for these cases as well.

(From OE-Core rev: 675e88e6e0bbd5ab2dcd4bdf97b0de59925a1be6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-31 13:50:46 +01:00
Changqing Li
09af4dafc7 runqemu: add lockfile for port used when slirp enabled
There is race condition when multi qemu starting with slirp,
add lockfile for each port to avoid problem like:

runqemu - ERROR - Failed to run qemu: qemu-system-x86_64: Could not set up host forwarding rule 'tcp::2323-:23'

[YOCTO #13364]

(From OE-Core rev: ceb3555a40ba06e58914465376aaf41392c12a7c)

Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-26 08:41:38 +01:00
Daniel Ammann
d3ed42ef27 image_types: Remove remnants of hdddirect
hdddirect was removed in commit 929ba563f1bc7195c4981b8e139c432b2cc388ea.

(From OE-Core rev: 7a91a9818057d69cba32db2428ffc1a1ebb3876e)

Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-26 08:41:38 +01:00
William Bourque
def9b64e27 wic/plugins: Source that support both EFI and BIOS
Add a source plugin that support both EFI and legacy PC-Bios.
While using this plugin, both bootloaders configurations reside
in the same /boot partitions.
This plugin has very little code : to avoid code duplication,
we simply re-import bootimg-pcbios and bootmg-efi source and
call both their SourcePlugin methods.

(From OE-Core rev: c8545d54139c6c48bffd1dd1d39d79891626c6f7)

Signed-off-by: William Bourque <wbourque@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-23 22:26:28 +01:00
Chen Qi
cedd95e091 devtool: remove temp dir in upgrade
For now, the temp dir is left in system, although the temporary
source directory has been cleaned up. So we clean it up too.

(From OE-Core rev: 8a0602327d5afcf4f36850d3f05c9721305852af)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-22 17:31:03 +01:00
Ricardo Ribalda Delgado
58589bc538 wic: Fix (again) partition files UIDs on multi rootfs images
Commit 450335ba5e73a375eb9932b4c4cf37979640dbfc copies the pseudo
database to the working directory in order to have ownership information
when the filesystem is generated.

Unfortunately this does not work anymore. The filenames on the database
are absolute and there is no information about the new directory.

Instead of fixing the database, we could redo a bit the way we patch the
fstab file. Now I am saving the old contents of fstab, modifying the
file and then reverting the changes on exit.

This is faster than the previous approach, although it can cause
indeterminism if the application is killed before finishing.

(From OE-Core rev: dcbf7b864dd1713b54a172d8714ce1508482f086)

Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-19 08:41:40 +01:00
Chee Yang Lee
b697aba61e wic: add support for kernel with initramfs bundled
when INITRAMFS_IMAGE_BUNDLE and INITRAMFS_IMAGE are set, wic should
look for kernel with initramfs image bundled.

Include required variable MACHINE, INITRAMFS_IMAGE_BUNDLE,
INITRAMFS_IMAGE, INITRAMFS_LINK_NAME and KERNEL_IMAGETYPE in WICVARS.
No longer require default value for variable kernel as KERNEL_IMAGETYPE
is not optional variable and included in WICVARS.
image_types_wic to inherit kernel-artifact-names to obtain default
INITRAMFS_LINK_NAME when INITRAMFS_IMAGE_BUNDLE are set.

update wic.Wic2.test_image_env test case to filter optional
variable INITRAMFS_LINK_NAME, INITRAMFS_IMAGE and INITRAMFS_IMAGE_BUNDLE.

(From OE-Core rev: bac984fbb2d5ad5d13ba3275c8a3e878d8753c58)

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-17 19:25:02 +01:00
Joshua Watt
067d475dbb scripts/buildstats-diff: Add option to filter tasks
Adds a command line option to filter out the buildstats-diff report by
one more more tasks. e.g.:

 buildstats-diff --only-task do_compile A B

will only show the differences for do_compile tasks. The --only-task
option can be specified multiple times to filter out multiple tasks at
once.

(From OE-Core rev: a8c7960d24c48107fd3703e49c38f890e84e2226)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-17 09:36:35 +01:00
Sai Hari Chandana Kalluri
57beed1d30 devtool: provide support for devtool menuconfig command
All packages that support the menuconfig task will be able to run
devtool menuconfig command. This would allow the user to modify the
current configure options and create a config fragment which can be
added to a recipe using devtool finish.

1. The patch checks if devtool menuconfig command is called for a valid
package.
2. It checks for oe-local-files dir within source and creates one if
needed, this directory is needed to store the final generated config
fragment so that devtool finish can update the recipe.
3. Menuconfig command is called for users to make necessary changes.
After saving the changes, diffconfig command is run to generate the
fragment.

Syntax:
	devtool menuconfig <package name>
	 Ex: devtool menuconfig linux-yocto

The config fragment is saved as devtool-fragment.cfg within
oe-local-files dir.

	Ex:
<workspace_path>/sources/linux-yocto/oe-local-files/devtool-fragment.cfg

Run devtool finish to update the recipe by appending the config fragment
to SRC_URI and place a copy of the fragment within the layer where the
recipe resides.
	Ex: devtool finish linux-yocto meta

[YOCTO #10416]

(From OE-Core rev: 417feb559a74b367315e8658d6ba868a4f8d1340)

Signed-off-by: Sai Hari Chandana Kalluri <chandana.kalluri@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-17 09:36:34 +01:00
Sai Hari Chandana Kalluri
3663a5d216 devtool/standard.py: Create a copy of kernel source within work-shared if not present
If kernel source is not already downloaded i.e staging kernel dir is
empty, place a copy of the source when the user runs devtool modify
linux-yocto.  This way the kernel source is available for other packages
that use it.

[YOCTO #10416]

(From OE-Core rev: bb42ab90835e8ec2f1dfbb35056c353784693266)

Signed-off-by: Sai Hari Chandana Kalluri <chandana.kalluri@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-17 09:36:34 +01:00
Sai Hari Chandana Kalluri
015c87d952 devtool/standard.py: Update devtool modify to copy source from work-shared if its already downloaded
In the regular devtool modify flow, the kernel source is fetched by
running do_fetch task. This is an overhead in time and space.

This patch updates modify command to check if the kernel source is
already downloaded. If so, then instead of calling do_fetch, copy the
source from work-shared to devtool workspace by creating hard links
else run the usual devtool modify flow and call do_fetch task.

[YOCTO #10416]

(From OE-Core rev: 3c3a9bae296f849dbfe03942282f44036e6fa1fb)

Signed-off-by: Sai Hari Chandana Kalluri <chandana.kalluri@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-17 09:36:34 +01:00
Ross Burton
b098d0ba84 recipetool: add MD5 hash for the line-wrapped MPL-1.1 license
(From OE-Core rev: d2755523d339a4eba7a034bdef5563657609f563)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-11 09:32:50 +01:00
Paul Eggleton
b52d58b8bb devtool: upgrade: fix handling of errors parsing upgraded recipe
As part of upgrading a recipe we create the upgraded recipe file in the
workspace and then try to parse it so we can then make further
modifications. If for some reason that parsing fails then the failure
was not being handled very well - the broken recipe was being left in
place, breaking parsing until it was removed by hand. Fix that by adding
a call to the cleanup function, and fix the following issues:

* Fix the cleanup function which doesn't look like it has ever worked
  due to a typo in the function call

* Fix double-printing the error message

* Remove usage of DevtoolError in this case (DevtoolError is for simple
  usage errors, not this kind of issue which may be the result of a
  bug).

We're still printing a traceback in this scenario but at least it
doesn't break the build system requiring manual cleanup. I also
introduced a command-line option to preserve the broken upgraded recipe
file(s) for debugging purposes.

(The reproducer for this is "devtool upgrade libnewt-python", however
you need to check out revision b82ea144e1
or earlier since that recipe has now been absorbed into the libnewt
recipe. The libnewt-python recipe was causing an issue with the upgrade
because it actually included the libnewt recipe using ${PV} in the
include statement, and of course PV was changing in the upgrade.)

Fixes [YOCTO #13404].

(From OE-Core rev: c519ac360796675d7fc09a5250d21f0f5b6236fc)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Paul Eggleton
1dffd9c2ec recipetool: ignore zero-length setup.py files
If a setup.py file exists it ought to have something in it before we
consider the source tree to be a Python module and treating it as such.
(A counter-example is https://www.bro.org/downloads/binpac-0.50.tar.gz -
it's not clear why this has a zero-length setup.py in it but we should
pay no attention to it.)

Fixes [YOCTO #12923].

(From OE-Core rev: 548a5c8f42c6ac1b0f7962926d05276e71505678)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Paul Eggleton
9a08720b3a list-packageconfig-flags: print PN instead of P
P (which is ${PN}-${PV}) isn't terribly useful in this context - we
don't really care what the version is, but we do want to know what the
recipe is so we can find it or set PACKAGECONFIG_pn-<PN> in our
configuration, so display ${PN} instead.

(From OE-Core rev: 7facaacd145c2924414ad63ddce07602a72d02c1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Paul Eggleton
12d64d6d1a scripts/contrib/ddimage: be explicit whether device doesn't exist or isn't writeable
Make the error messages a little more friendly.

(From OE-Core rev: 36f308b0134c69b439152c2473a274d96b0fee89)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Paul Eggleton
d3a9f43305 scripts/contrib/ddimage: replace blacklist with mount check
The blacklist, whilst previously useful for safety, is now becoming
obsolete - on my current system, the main storage is at /dev/nvme* and
if I plug in a USB stick it shows up as /dev/sdb which was previously
blacklisted. To make this more flexible, remove the blacklist and
instead check if the specified device is mounted, has a partition
that is mounted, or is otherwise in use according to the kernel, and
show an appropriate error and quit if so.

To make this robust, also ensure we handle where the specified device is
a symlink to another device.

(From OE-Core rev: 49043de1a7716ad612fb92a2e8a52e43d253c800)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Paul Eggleton
8ff741d075 scripts/contrib/ddimage: fix typo
UNKOWN -> UNKNOWN

(From OE-Core rev: adf632c959e7f6595b697726fe64ddda40b9f249)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Richard Purdie
99206cb7b4 package: Build pkgdata specific to the current recipe
This switches the code to build pkgdata specific to the current recipe
which means that its filtered to the recipes dependencies and can perform
better as we can drop the lockfile.

It uses a similar method to the staging code to do this, using BB_TASKDEPDATA
to construct a list of packagedata task output which this recipe should "see".

The original pkgdata store is left unaltered so existing code works.

The lock file was there to prevent files disappearing as they were read or as
directories were listed. Since we have a copy of the data and only access output
from completed tasks (as per their manifests), we can remove the lock.

The lock was causing starvation issues on systems with parallelism.

There was also a potential determinism problem as the current code could "see"
data from recipes which it doesn't depend upon.

[YOCTO #13412]

(From OE-Core rev: 1951132576bfb95675b4879287f8b3b7c47524fa)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-30 22:40:52 +01:00
Robert Yang
b19ce5d2e8 wic: python2 -> python3
(From OE-Core rev: 9303f92599343adf645fee5d2434fadd97e7febb)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-30 22:40:52 +01:00
Robert Yang
69f1b3962e ksum.py: python -> python3
* Testinfo:
  In kernel's builddir:
  $ /path/to/oe-core/scripts/tiny/ksum.py

Collecting object files [DONE]

Totals:

vmlinux:
    text        data            bss             total
    10933110    3824470         1605632         16363212

modules (2004):
    text        data            bss             total
    46144408    3047516         167580          49359504

vmlinux + modules:
    text        data            bss             total
    57077518    6871986         1773212         65722716

(From OE-Core rev: a55955f337b4cebf557f96f5dd7c2edc0bfa6d8a)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-30 22:40:52 +01:00
Chen Qi
dc3b7bd2eb devtool: warn user about multiple layer having the same base name
Currently `devtool finish RECIPE meta' will silently succeed even
if there are multiple layers having the same base name of 'meta'.
e.g. meta layer from oe-core and meta layer from meta-secure-core.

We should at least give user a warning in such case. With the patch,
we will get warning like below.

WARNING: Multiple layers have the same base name 'meta', use the first one '<PROJ_DIR>/oe-core/meta'.
WARNING: Consider using path instead of base name to specify layer:
	 	  <PROJ_DIR>/oe-core/meta
		  <PROJ_DIR>/meta-secure-core/meta

(From OE-Core rev: 2c8740f543c38dbaef3345e40827ef48b3f75405)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-28 13:28:37 +01:00
Ross Burton
e58f930df5 gtk-icon-cache: rename intercept to update_gtk_icon_cache
The intercept is called update_icon_cache which is vague: rename to
update_gtk_icon_cache to make it clearer what it is for, and add a comment
explaining what class caused it to be used.

(From OE-Core rev: 3158adbe684890adc56af11e19af872e90e09d41)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-27 12:20:36 +01:00
Chee Yang Lee
fc0c36a647 wic/bootimg-efi: allow multiple initrd
Allow plugin bootimg-efi to configure with multiple initrd
through source parameter.

Uses ; to separate each initrd.

e.g:
--sourceparams="loader=${EFI_PROVIDER},initrd=initrd1;initrd2"

(From OE-Core rev: c7b0823f9ab6a9959aaa809b8c3f70d199feb64d)

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-19 22:13:39 +01:00
Yeoh Ee Peng
99c821eaaa resulttool/merge: Enable control TESTSERIES and extra configurations
Current QA team need to merge test result files from multiple sources.
Adding TESTSERIES configuration too early will have negative
implication to report and regression. Enable control to add TESTSERIES
when needed. Also enable adding EXECUTED_BY configuration when
needed.

(From OE-Core rev: 651d8d371e78e77599d56681228d5782664f7743)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-12 10:54:40 +01:00
Yeoh Ee Peng
41f10ab3b1 resulttool/store: Enable add EXECUTED_BY config to results
Current results stored does not have information needed to trace who
executed the tests. Enable store to add EXECUTED_BY configuration
to results file in order to track who executed the tests.

(From OE-Core rev: dca2a57d54163a2e63b06e2f140fea3bd49cef0d)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-12 10:54:40 +01:00
Yeoh Ee Peng
588f455082 resulttool/resultutils: Enable add extra configurations to results
Current resultutils library always add "TESTSERIES" configuration
to results. Enhance this to allow control of adding "TESTSERIES"
configuration as well as allow adding extra configurations
when needed.

(From OE-Core rev: 443c0acc14ef2451b10878fc83dd11b46805daf0)

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-12 10:54:40 +01:00
Chee Yang Lee
97feebdf99 wic/plugins: kernel image refer to KERNEL_IMAGETYPE
replaced hardcoded kernel image with KERNEL_IMAGETYPE.
set kernel image to "bzImage" incase KERNEL_IMAGETYPE not set.

(From OE-Core rev: 88a9fef761c5e67b2964fedc85a7e8ad37067564)

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-11 13:26:55 +01:00
Maciej Pijanowski
bb59bcd016 recipetool: add python3 support
Add support for generating python3 recipes using the recipetool / devtool.
Drop python2 support at the same time.

Tested with:

oe-selftest -r recipetool.RecipetoolTest

[YOCTO #13264]

(From OE-Core rev: d8b2f58974482b3b1ccc65c5f93104d0d7ba87bc)

Signed-off-by: Maciej Pijanowski <maciej.pijanowski@3mdeb.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-10 17:38:10 +01:00
Adrian Freihofer
0e4c79a7c4 runqemu: QB_FSINFO to support fstype wic images
wic images are handled as vmtype images. Starting qemu with "-kernel"
parameter and an image of type wic is not supported. Especially for
"-machine virt" the combination of wic with -kernel parameter would
be beneficial.

The new parameter QB_FSINFO allows to pass image type specific flags to
runqemu. QB_FSINFO is a space separated list of parameters. Parameters are
structured according to the following pattern: image-type:flag.

For now two parameters are supported:
- wic:no-kernel-in-fs
  The wic image is treated as rootfs only image. A -kernel option is
  passed to qemu.
- wic:kernel-in-fs
  The wic image is treated as VM image including a bootloader and a
  kernel. This is still the default behavior.

Example:
QB_DEFAULT_FSTYPE = "wic"
QB_FSINFO = "wic:no-kernel-in-fs"
QB_KERNEL_ROOT = "/dev/vda1"
QB_SYSTEM_NAME = "qemu-system-aarch64"
QB_MACHINE = "-machine virt"
...

[YOCTO #13336]

(From OE-Core rev: 2aa79a67affd22dfa37e4c2945c6ab0c86321f98)

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-10 17:38:10 +01:00
Ross Burton
ca4cdfcd18 wic/filemap: handle FIGETBSZ failing
Some file systems don't support fetching the block size (notably the file system
Docker uses for containers), so handle the iotctl() failing and raise the
expected error.

(From OE-Core rev: 3757073726a00c5250556aae3d0daac76b88085e)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-08 16:01:41 +01:00
Chee Yang Lee
cc083399ce wic/engine: include .wks.in in wic search and list
allow wic to list and search for kickstart file in .wks.in extension.
basename show by wic list images to fully exclude extension.

(From OE-Core rev: 2c0a292a790ad069648e37b1b29fcea656fcf3e4)

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-07 09:11:49 +01:00
Kevin Hao
e5f2684d60 runqemu: Add the support to pass multi ports to tcpserial parameter
In some cases(such as the oeqa's qemurunner), we need to setup multi
serial devices via the '-serial 127.0.0.1:xx" and the order of them
is significant. The mixing use of "tcpserial" and "-serial 127.0.0.1:xx"
cause ambiguous issues and we can't fix it by only adjusting the order
of them. So add the support to pass multi ports to the tcpserial
parameter, this will make sure that the order of setting up the serial
is really what we want.

[YOCTO Bug 13309]

(From OE-Core rev: 766c3b56e5071b5a5a64e88df6d3abe5232dd958)

Signed-off-by: Kevin Hao <kexin.hao@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-07 09:11:48 +01:00
Jon Mason
b6ca6ac564 resulttool: Remove prints if no tests occur
Printing the lack of a test is not necessary (per feedback).  Remove
this from the template to quieten it.

(From OE-Core rev: b1fe6ae66360e160eeaeafe456536f335a0eab60)

Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-04 23:09:24 +01:00
sangeeta jain
d9cb1c472c resulttool/manualexecution: Enable creation of test case configuration
Allow the creation of test case configuration file based on user inputs.
Where this testcase configuration file will be used by the the manual
execution to run selected test cases for a module rather than compulsory
run all test cases in manual json file.

(From OE-Core rev: 73d2a747c17779da0ca972da776b3cf02c2e1cbc)

Signed-off-by: sangeeta jain <sangeeta.jain@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-04 09:09:42 +01:00
Jon Mason
cb153e8da8 resulttool: modify to be multi-machine
Currently, the code will sum all of the different machine results into a
single report of the tests results.  This can lead to confusion as to
which machine may be experiencing issues.  Modify the code to store the
results in a per machine basis and report them accordingly.

(From OE-Core rev: 16d4031ea5df8a4ddfdb937d35464c09e1abd10e)

Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-04 09:09:42 +01:00
Richard Purdie
a07a6a7415 runqemu: Add support for kvm on aarch64
The main issue is to make the x86 checks apply to x86 targets only. We may
end up with better checks on other architectures but this adapts the code to
allow for that and its still controlled by whether QB_CPU_KVM is set.

The code needed minor refactoring so the qemu-system-XXX name is set
earlier so the kvm code can use it.

(From OE-Core rev: 06c473a0127f19b76d0f647b87873944add1e331)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-31 15:36:20 +01:00
Chee Yang Lee
01543538d1 wic: bootimg-efi: add label source parameter
Add new source parameter label to allow custom boot.conf/grub.cfg label,
so far it's hardcoded to "Boot".

Default label to "Boot" for systemd-boot and blank for grub-efi when source
parameter label are not set.

(From OE-Core rev: 7a0aab1aa31e66e6bc94c04c2f6c1043b64a8967)

Signed-off-by: Chee Yang Lee <chee.yang.lee at intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-30 12:37:03 +01:00
Chee Yang Lee
92cc79f4e6 wic/bootimg-efi: replace hardcoded volume name with label
volume name should refer to --label in .wks.
Replace the hardcoded volume name  with label.
set "ESP" as default name when no lable specified.

(From OE-Core rev: 5621aceaf39ef0dc097b16c83e73b9882c987a7c)

Signed-off-by: Chee Yang Lee <chee.yang.lee@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-15 17:53:29 +01:00
Richard Purdie
ffae400179 meta/lib+scripts: Convert to SPDX license headers
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.

The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).

More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.

The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.

(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09 16:31:55 +01:00
Joshua Watt
9a97ea263c resulttool: Add option to dump all ptest logs
Adds an option to dump all the ptest logs to individual files in a
specified directory. If multiple test runs are present, the
'--prepend-run' argument will create separate directories for each test
run under the target directory and put the logs there to prevent each
test run from clobbering the others.

[YOCTO #13331]

(From OE-Core rev: f262b9e7482e38f04dfa162f0aaf65b17e86f8b2)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-08 22:56:45 +01:00
Richard Purdie
61c27c1556 scripts/pybootchart/draw: Fix some bounding problems
The chart size extents were being incorrectly reported, not accounting for the
width of the legend. Set a minimum width to account for that (its fixed size).

Also stop printing the chart background off the bottom of the chart extents.

(From OE-Core rev: ef257afb772daf721f12cb3ff9708d39da06f2c9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-08 22:56:45 +01:00
Richard Purdie
cdce17f820 scripts/pybootchart/draw: Clarify some variable names
This tweaks some intermediate variable names to make it clearer what
is being done.

(From OE-Core rev: 50d9af7bcb05823db36e86170ef9ba3a1d6cfeca)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-08 22:56:45 +01:00
Richard Purdie
020911ab59 scripts/pybootchart: Port to python3
This updates the pybootchart code (used for viewing build timing profiles)
to use python3. The bulk of the changes are to use gi instead of pygtk, i.e.
port from gtk+2 to gtk+3.

The main change is to make the bootchart widget inherit gtk.Scrollable
and change the way the scrollbars are implemented to match the new method
upstream. The drawing code used cairo already so can remain unchanged,

(From OE-Core rev: 949144681ad7f536732169351cab6d0612e9c566)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-08 22:56:45 +01:00
Richard Purdie
d13b904305 scripts/pybootchart: Fix mixed indentation
The script had a toxic mix of tabs and spaces, fix this.

(From OE-Core rev: 5eebe246e9444a5a7dbf4d8683ae08c468cfc401)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-08 22:56:45 +01:00
Alex Kiernan
568f0ab886 recipetool: fix unbound variable when fixed SRCREV can't be found
If attempting to find a fixed SRCREV fails because the directory doesn't exit,
avoid failing with:

  Traceback (most recent call last):
    File "/home/vagrant/poky/scripts/recipetool", line 121, in <module>
      ret = main()
    File "/home/vagrant/poky/scripts/recipetool", line 110, in main
      ret = args.func(args)
    File "/home/vagrant/poky/scripts/lib/recipetool/create.py", line 707, in create_recipe
      srcrev = stdout.rstrip()
  UnboundLocalError: local variable 'stdout' referenced before assignment

Fixes: 000480c42797 ("recipetool / devtool: set a fixed SRCREV by default when fetching from git")
(From OE-Core rev: 8e4b301573071ea04911b0df6533c25bc4596b93)

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-08 12:15:17 +01:00
Ross Burton
9a69da4489 wic: add global debug option
Add a global --debug option to assist debugging.

(From OE-Core rev: 4e23b22bdf7b0e1fd16c7e53b6a7232c9c92ce95)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-07 10:12:53 +01:00
Ross Burton
74b44e59be wic: change expand behaviour to match docs
The documentation says that --expand takes a comma-separated list of
partition:size pairs, but the code was splitting on hyphens.

Hyphens are not a transitional separator for a list of items, so change the code
to reflect the documentation.

(From OE-Core rev: a210e28bb3fd5433ebecf50e218fc548013b35dc)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-07 10:12:53 +01:00
Armin Kuster
12d8f77a88 resulttool: add LTP compliance section
(From OE-Core rev: a680d7d15fafbecf4edce9a29cc4eda16c11fd94)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-29 14:16:30 +01:00
Armin Kuster
9c82085036 resulttool: add ltp test support
(From OE-Core rev: 966795aa2c6960aca11a04e87f415256faf26957)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-29 14:16:30 +01:00