* True is default since 2016 and most layers were already updated
not to pass this parameter where not necessary, e.g. oe-core was
updated couple times, first in:
https://git.openembedded.org/openembedded-core/commit/?id=7c552996597faaee2fbee185b250c0ee30ea3b5f
Updated with the same regexp as later oe-core update:
https://git.openembedded.org/openembedded-core/commit/?id=9f551d588693328e4d99d33be94f26684eafcaba
with small modification to replace not only d.getVar, but also data.getVar as in e.g.:
e.data.getVar('ERR_REPORT_USERNAME', True)
and for getVarFlag:
sed -e 's|\(d\.getVarFlag \?\)( \?\([^,()]*, \?[^,()]*\), \?True)|\1(\2)|g' \
-i $(git grep -E 'getVarFlag ?\( ?([^,()]*), ?([^,()]*), ?True\)' \
| cut -d':' -f1 \
| sort -u)
(From OE-Core rev: 26c74fd10614582e177437608908eb43688ab510)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds argument parsing to the conversion script so that the fields that
the script uses to do conversions can be customized on the command line.
The intention is to allows easier customization without having to fork
the script, and allow automated checking on 3rd party layers via CI
without false positives
(From OE-Core rev: b9551f9180bf9f13fb1c480b5b7892fdc831ffcd)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where there isn't a copyright statement, add one to make it explicit.
Also drop editor config lines where they were present and add license
identifiers as MIT if there isn't one.
(From OE-Core rev: deb3ccec53e0bd63bc4235cf2b0d3fc781687361)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sort dict keys in the json output and use tab spacing. This means
when commited into git, the diffs are human readable but it is more
compact filesize than space indentation.
(From OE-Core rev: bde2ecb203d8a1a29715c70ca3ded382982390cf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Save commit data when writing to the json file so the results can
be copared/extended later.
(From OE-Core rev: da761ac1984ee2a06ded905fc4ad878ef7d613e4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use of those tools was removed in b5c131006e3fad0a15e6cdf81f71dc1e96647028
perf-build-test/report: Drop phantomjs and html email reports support
(From OE-Core rev: 33df447affa7a3a360b1da028e6b12fbcd388db6)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
phantomjs isn't reliable and we've moved to sharing the reports via a webserver.
Update the scripts to more match those being used in the autobuilder helper
where the html email support was removed.
(From OE-Core rev: b5c131006e3fad0a15e6cdf81f71dc1e96647028)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The convert-spdx-licenses.py script needs the same file closing
fix as was made to convert-variable-renames to ensure modified
file contents get flushed out.
(From OE-Core rev: 46135c87345c7189053dafbed92c754f9f328c32)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In my test environments (Fedora 35 and Debian 10.10 on AMD 2970WX),
running the convert-variable-renames.py rename script was resulting
in empty files instead of updated ones. From inspection, the new
files are never flushed/closed before moving them into place, which
seems inherently racy. Adding an explicit close to flush the modified
contents out before moving into place fixes the issue for me.
(From OE-Core rev: 187ac1ea0a701a5ba9ec92f6aa32f2a67600a584)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The SYSTEM and USER seperation between variables seems odd and not necessary,
drop it. Avoid the use of whitelist/blacklist and also change "packages" to
"recipes" since that misuse causes confusion.
(From OE-Core rev: 0df0eb6401a02139b9110bc95e21d97a67125ec5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This script searches for a list of variable that have been renamed
and converts them to their more descriptive names. It also searches
for a list of variables that have been removed or deprecated and
prints a message.
It will print a message to inform the user that there are terms that
need to be updated in their files. Many of these changes are context
sensitive and may not be modified as they might be existing calls to
other libraries. This message is informational only.
I have tested this on poky and meta-openembedded so far.
(From OE-Core rev: 50fe7ba8dba05a9681c9095506f798796cfc2750)
(From OE-Core rev: 75f319c105484d0b312a858cc0bd8148728c8622)
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the change to bitbake, update the references in OE-Core to match the updates.
(From OE-Core rev: 193affb9f28b0116c3fd619834f145326fee08c5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're standardising on SPDX license names so it is overdue to change
the ones in the LICENSE fileds to the SPDX values. Add a conversion
script which makes this straightforward on the most part.
Ultimately this allows the core code to be more efficient and not need
to consult the mappings.
(From OE-Core rev: 512cd4ca91bc5107d68b7e721257a8f62f878994)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There may be none, one or more spaces including tab before backslash in
SRC_URI. Use regex to check and update. It helps to avoid malformed uri
such as recipe open-iscsi-user in meta-openstack:
SRC_URI = "git://github.com/open-iscsi/open-iscsi.git;protocol=https ;branch=master \
And help to check more recipes such as concurrent-ruby in the same
layer:
SRC_URI = "git://github.com/ruby-concurrency/concurrent-ruby.git;protocol=https;tag=v1.1.6\
(From OE-Core rev: a69a53573b1987ee5834a6fc27763f9bbf5fe5a4)
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Github are dropping support for git:// protocol fetching. Update the script
to learn about corner cases found in the previous conversion and
support remapping the github urls as needed too.
(From OE-Core rev: e59fe8279b209f67ff79b9d6dbb69389a64db236)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is uncertainty about the default branch name in git going forward.
To try and cover the different possible outcomes, add branch names to all
git:// and gitsm:// SRC_URI entries.
This update was made with the script added to contrib in this patch which
aims to help others convert other layers.
(From OE-Core rev: b51c405faf6f8c0365f7533bfaf470d79152a463)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently lines like below are converted.
e.g.
# IMAGE_INSTALL_append = " A"
But lines without a leading whitespace are not converted.
e.g.
#IMAGE_INSTALL_append = " A"
We should be converting both.
(From OE-Core rev: 1994e3844c1aa6b595c0c18040e4f8240fa04438)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
e.g. VIRTUAL-RUNTIME_com.webos.service.flowmanager_armv4 weren't replaced
with VIRTUAL-RUNTIME_com.webos.service.flowmanager:armv4 or when package
name contains a dot like in:
RDEPENDS:gstreamer1.0-meta-base:remove
(From OE-Core rev: 41bff44bd26c09573eb3139bb6735e5ecfda18b7)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For consistency, use override syntax for these variables as well since
it is more consistent with the rest of the image code. We may be able to use
these as proper overrides in due course.
(From OE-Core rev: 52674c4b1fdf79829095031b2e342d44fb0dc181)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The IMAGE_TYPEDEP variable would make more sense to match the form of the
other image override variables, convert it to use the overrides format.
(From OE-Core rev: 8573f6b2a7af9867da0b21936ffd2cd2a417de1d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add task-configure and few more supported values of TARGET_OS override.
(From OE-Core rev: 1172f9593902f28ddd8da47de6bd51cda9a0f86a)
Signed-off-by: Denys Dmytriyenko <denis@denix.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is handy to be able to have the script convert a single patch file
and it turns out to be straightforward to make it do this.
(From OE-Core rev: 21df7acc969f47d615d1701ee71f19571de94949)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* on bigger layer this script takes couple minutes without showing any progress
* add a version to reference it in the conversion commit (to easily figure out
if it's worth re-converting the layer).
(From OE-Core rev: 39edf868deafe79ffd2ea57620af70840ec4c208)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds a script I've developed to migrate metadata to use the new override
syntax. It is a bit rough but since its for a single use with validation, it
doesn't need to be perfect. It is run simply as:
scripts/contrib/convert-overrides.py <directory>
It is setup and has been tested to work with OE-Core, Bitbake, yocto-docs,
meta-yocto, meta-gplv2 and meta-mingw. For OE-Core, it converts around 10,100
lines with about 34 manual fixes needed.
For other layers it would need updating for override names and exclusions for
functions/variable names with "append", "prepend" or "remove" in them.
(From OE-Core rev: e8fc67f6f6baf54ccdf548b947c35dea926842c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
image-manifest: script to generate product/image specific BOM
The image-manifest script generates image specific reports based on
an image manifest file. Currently there is data generated by buildhistory,
pkgdata, and license manifest but this data is poorly formated and spread
across multiple text files. This script can generate a single JSON output
file that is machine readable by other tools.
The manifest-info collects package information and stores the information
in a tarball. manifest-info can be configured using a json configuration
file. The default configuration including all possible options can be
dumped using the dump-config subcommand.
image-manifest takes an image manifest file as input to get the runtime
dependencies. As an option image-manifest can also use the build dependency
file, pn-buildlist, to get the build dependencies excluding native
packages.
This script extends the oe-image-manifest script [0] done by Paul Eggleton
[0]
https://github.com/intel/clear-linux-dissector-web/blob/master/layerindex/static/files/oe-image-manifest
------------------------------------------------------
usage: image-manifest [-h] [-d] [-q] <subcommand> ...
Image manifest utility
options:
-h, --help show this help message and exit
-d, --debug Enable debug output
-q, --quiet Print only errors
subcommands:
recipe-info Get recipe info
list-depends List dependencies
list-recipes List recipes producing packages within an image
list-packages List packages within an image
list-layers List included layers
dump-config Dump default config
manifest-info Export recipe info for a manifest
Use image-manifest <subcommand> --help to get help on a specific command
Co-developed-by: Paul Eggleton <bluelightning@bluelightning.org>
(From OE-Core rev: ad8fec9ce1704866df925bda18a240d6889b1ed5)
Signed-off-by: Anders Wallin <anders.wallin@windriver.com>
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update after tinfoil API changes in bitake for REQUIRED_VERSION.
(From OE-Core rev: 1645c9b093bebf7ced67cbee0009d81d1a215966)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a filter subcommand to filter a task-depends.dot graph produced by
bitbake -g down to just a subset of targets/tasks.
(From OE-Core rev: a14b274b56676ff0ba55a4048169ad60c9514994)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
argparse makes this a lot easier to extend.
(From OE-Core rev: c751ef8fdc111d1c967029cea7a3ed0f88ce851b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Spaces were previously converted to dashes, and underscores caused the
next character in the title/key to be printed using subscript due to the
enhanced string support in gnuplot.
(From OE-Core rev: 1719f7062988889b7e24b871dc2f0e50c7bff07e)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix indentation in buildstats.sh and buildstats-plot.sh
to convert every line to 4 spaces indentation.
(From OE-Core rev: 3971572042ae9bc8226eb92892bb71010287e0a8)
Signed-off-by: Luis Martins <luis.martins@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add option to filter the stats results for a single recipe, allowing
to easilly visuality the task resource consumption per recipe.
This is specially useful when debugging the performance of
the overall bitbake build system.
(From OE-Core rev: b4bf898d53af60f5d69b50ec5cdd7f9b3870f2dd)
Signed-off-by: Luis Martins <luis.martins@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add option to accumulate stats values per recipe, allowing
to sum related values such as memory (main process + childs).
This is specially useful when debugging the performance of
the overall bitbake build system.
(From OE-Core rev: e6f1aea6260343cb5194f7f8ab70213b705ab441)
Signed-off-by: Luis Martins <luis.martins@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Extend buildstats-plot.sh script to also accept the name
of the tasks as parameter.
This value will be passed directly to buildstats.sh is
already provides this option.
(From OE-Core rev: 495b6d3d85d2d14d54e324d8da43311a23fdfca6)
Signed-off-by: Luis Martins <luis.martins@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix the internal call to buildstats.sh by also providing
the buildstats folder location in the system, which might
differ from the default location.
(From OE-Core rev: 9f4ee2ee8712528ac51ef4ecd17ccde737e8b21d)
Signed-off-by: Luis Martins <luis.martins@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
P (which is ${PN}-${PV}) isn't terribly useful in this context - we
don't really care what the version is, but we do want to know what the
recipe is so we can find it or set PACKAGECONFIG_pn-<PN> in our
configuration, so display ${PN} instead.
(From OE-Core rev: 7facaacd145c2924414ad63ddce07602a72d02c1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make the error messages a little more friendly.
(From OE-Core rev: 36f308b0134c69b439152c2473a274d96b0fee89)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The blacklist, whilst previously useful for safety, is now becoming
obsolete - on my current system, the main storage is at /dev/nvme* and
if I plug in a USB stick it shows up as /dev/sdb which was previously
blacklisted. To make this more flexible, remove the blacklist and
instead check if the specified device is mounted, has a partition
that is mounted, or is otherwise in use according to the kernel, and
show an appropriate error and quit if so.
To make this robust, also ensure we handle where the specified device is
a symlink to another device.
(From OE-Core rev: 49043de1a7716ad612fb92a2e8a52e43d253c800)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.
The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).
More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.
The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.
(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is the older script which the code in lib/oeqa/buildperf replaces so
delete the obsolete script to avoid confusion.
(From OE-Core rev: 9908b7955e75e10d6a1384e4c08417ef534464af)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This tweaks the script to:
* Ensure directories exist and can be written to
* Allow the downloads directory to be specified
* Error early if the phantomjs or optipng dependencies are not installed
* Allow the location of the globalres.log file to be specified
This means that the main build directory can be destroyed and any state
from the script is stored elsewhere allowing it to be triggered
automatically from the autobuilder infrastructure.
(From OE-Core rev: 1de5fbd484e6a747ae6419ccc89d8c2911b9706b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
