All the regexes throw a warning like this:
WARNING: scripts/lib/recipetool/create_buildsys.py:140:
SyntaxWarning: invalid escape sequence '\s'
proj_re = re.compile('project\s*\(([^)]*)\)', re.IGNORECASE)
Python 3 interprets string literals as Unicode strings, and therefore
\s is treated as an escaped Unicode character which is not correct.
Declaring the RegEx pattern as a raw string instead of unicode is
required for Python 3.
(From OE-Core rev: 24b0ba00d4f0b4d9834f7693ecb6032dfc534a80)
Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
git show-ref looks at the _remote_ ref called HEAD, which is fine when it
matches the local HEAD but problematic when you're iterating a series of
commits.
Use rev-parse to resolve the local name to a proper hash.
(From OE-Core rev: 3c04747b681cf6090ba9c77752f6c2f304dbbe17)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move most imports to the top of the file.
(From OE-Core rev: d2c287db0739b249604cd1beaa03ec38512ba718)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The autobuilder scripts post-process the generated JSON to inject recipe
and commit counts into the data. We can do this easily in patchreview
instead.
(From OE-Core rev: 77c96e43090cbf485aec612cc2315b85e5635dda)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A previous patch[1] added the ability to allow the search pattern for
patches to be changed, so that patchreview can be used across the entire
meta-oe repository by changing the patterns.
However, this means the caller needs to write long patterns when calling
patchreview.
Instead, we can see if the specified directory contains a layer by
checking if conf/layer.conf exists. If it does, then search for patches
inside this directory. If it doesn't, assume that the specified
directory is a repository that contains sublayers (such as
meta-openembedded) and look through each of the directories that match
the pattern meta-*.
This means patchreview can both scan either a single layer (eg
.../poky/meta) or a repository of sublayers (eg .../meta-openembedded).
[1] oe-core 599046ea9302af0cf856d3fcd827f6a2be75b7e1
(From OE-Core rev: a3a868519beab1b9cac94fefd7dbeffb09d047e9)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Underscores previously caused the next character in the label to be
printed using subscript due to the enhanced string support in gnuplot.
(From OE-Core rev: 282b48f90f77e0766993018d22fe03dd303febdc)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This introduces support for specifying a search pattern with the -p/--pattern
option in the patchreview.py script. This is designed to accommodate
the directory structure of meta-openembedded.
(From OE-Core rev: 599046ea9302af0cf856d3fcd827f6a2be75b7e1)
Signed-off-by: Mickael RAMILISON <mickael.ramilison@smile.fr>
Reviewed-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* True is default since 2016 and most layers were already updated
not to pass this parameter where not necessary, e.g. oe-core was
updated couple times, first in:
https://git.openembedded.org/openembedded-core/commit/?id=7c552996597faaee2fbee185b250c0ee30ea3b5f
Updated with the same regexp as later oe-core update:
https://git.openembedded.org/openembedded-core/commit/?id=9f551d588693328e4d99d33be94f26684eafcaba
with small modification to replace not only d.getVar, but also data.getVar as in e.g.:
e.data.getVar('ERR_REPORT_USERNAME', True)
and for getVarFlag:
sed -e 's|\(d\.getVarFlag \?\)( \?\([^,()]*, \?[^,()]*\), \?True)|\1(\2)|g' \
-i $(git grep -E 'getVarFlag ?\( ?([^,()]*), ?([^,()]*), ?True\)' \
| cut -d':' -f1 \
| sort -u)
(From OE-Core rev: 26c74fd10614582e177437608908eb43688ab510)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adds argument parsing to the conversion script so that the fields that
the script uses to do conversions can be customized on the command line.
The intention is to allows easier customization without having to fork
the script, and allow automated checking on 3rd party layers via CI
without false positives
(From OE-Core rev: b9551f9180bf9f13fb1c480b5b7892fdc831ffcd)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Where there isn't a copyright statement, add one to make it explicit.
Also drop editor config lines where they were present and add license
identifiers as MIT if there isn't one.
(From OE-Core rev: deb3ccec53e0bd63bc4235cf2b0d3fc781687361)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sort dict keys in the json output and use tab spacing. This means
when commited into git, the diffs are human readable but it is more
compact filesize than space indentation.
(From OE-Core rev: bde2ecb203d8a1a29715c70ca3ded382982390cf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Save commit data when writing to the json file so the results can
be copared/extended later.
(From OE-Core rev: da761ac1984ee2a06ded905fc4ad878ef7d613e4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use of those tools was removed in b5c131006e3fad0a15e6cdf81f71dc1e96647028
perf-build-test/report: Drop phantomjs and html email reports support
(From OE-Core rev: 33df447affa7a3a360b1da028e6b12fbcd388db6)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
phantomjs isn't reliable and we've moved to sharing the reports via a webserver.
Update the scripts to more match those being used in the autobuilder helper
where the html email support was removed.
(From OE-Core rev: b5c131006e3fad0a15e6cdf81f71dc1e96647028)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The convert-spdx-licenses.py script needs the same file closing
fix as was made to convert-variable-renames to ensure modified
file contents get flushed out.
(From OE-Core rev: 46135c87345c7189053dafbed92c754f9f328c32)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In my test environments (Fedora 35 and Debian 10.10 on AMD 2970WX),
running the convert-variable-renames.py rename script was resulting
in empty files instead of updated ones. From inspection, the new
files are never flushed/closed before moving them into place, which
seems inherently racy. Adding an explicit close to flush the modified
contents out before moving into place fixes the issue for me.
(From OE-Core rev: 187ac1ea0a701a5ba9ec92f6aa32f2a67600a584)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The SYSTEM and USER seperation between variables seems odd and not necessary,
drop it. Avoid the use of whitelist/blacklist and also change "packages" to
"recipes" since that misuse causes confusion.
(From OE-Core rev: 0df0eb6401a02139b9110bc95e21d97a67125ec5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This script searches for a list of variable that have been renamed
and converts them to their more descriptive names. It also searches
for a list of variables that have been removed or deprecated and
prints a message.
It will print a message to inform the user that there are terms that
need to be updated in their files. Many of these changes are context
sensitive and may not be modified as they might be existing calls to
other libraries. This message is informational only.
I have tested this on poky and meta-openembedded so far.
(From OE-Core rev: 50fe7ba8dba05a9681c9095506f798796cfc2750)
(From OE-Core rev: 75f319c105484d0b312a858cc0bd8148728c8622)
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the change to bitbake, update the references in OE-Core to match the updates.
(From OE-Core rev: 193affb9f28b0116c3fd619834f145326fee08c5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're standardising on SPDX license names so it is overdue to change
the ones in the LICENSE fileds to the SPDX values. Add a conversion
script which makes this straightforward on the most part.
Ultimately this allows the core code to be more efficient and not need
to consult the mappings.
(From OE-Core rev: 512cd4ca91bc5107d68b7e721257a8f62f878994)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There may be none, one or more spaces including tab before backslash in
SRC_URI. Use regex to check and update. It helps to avoid malformed uri
such as recipe open-iscsi-user in meta-openstack:
SRC_URI = "git://github.com/open-iscsi/open-iscsi.git;protocol=https ;branch=master \
And help to check more recipes such as concurrent-ruby in the same
layer:
SRC_URI = "git://github.com/ruby-concurrency/concurrent-ruby.git;protocol=https;tag=v1.1.6\
(From OE-Core rev: a69a53573b1987ee5834a6fc27763f9bbf5fe5a4)
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Github are dropping support for git:// protocol fetching. Update the script
to learn about corner cases found in the previous conversion and
support remapping the github urls as needed too.
(From OE-Core rev: e59fe8279b209f67ff79b9d6dbb69389a64db236)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is uncertainty about the default branch name in git going forward.
To try and cover the different possible outcomes, add branch names to all
git:// and gitsm:// SRC_URI entries.
This update was made with the script added to contrib in this patch which
aims to help others convert other layers.
(From OE-Core rev: b51c405faf6f8c0365f7533bfaf470d79152a463)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently lines like below are converted.
e.g.
# IMAGE_INSTALL_append = " A"
But lines without a leading whitespace are not converted.
e.g.
#IMAGE_INSTALL_append = " A"
We should be converting both.
(From OE-Core rev: 1994e3844c1aa6b595c0c18040e4f8240fa04438)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
e.g. VIRTUAL-RUNTIME_com.webos.service.flowmanager_armv4 weren't replaced
with VIRTUAL-RUNTIME_com.webos.service.flowmanager:armv4 or when package
name contains a dot like in:
RDEPENDS:gstreamer1.0-meta-base:remove
(From OE-Core rev: 41bff44bd26c09573eb3139bb6735e5ecfda18b7)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For consistency, use override syntax for these variables as well since
it is more consistent with the rest of the image code. We may be able to use
these as proper overrides in due course.
(From OE-Core rev: 52674c4b1fdf79829095031b2e342d44fb0dc181)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The IMAGE_TYPEDEP variable would make more sense to match the form of the
other image override variables, convert it to use the overrides format.
(From OE-Core rev: 8573f6b2a7af9867da0b21936ffd2cd2a417de1d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add task-configure and few more supported values of TARGET_OS override.
(From OE-Core rev: 1172f9593902f28ddd8da47de6bd51cda9a0f86a)
Signed-off-by: Denys Dmytriyenko <denis@denix.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is handy to be able to have the script convert a single patch file
and it turns out to be straightforward to make it do this.
(From OE-Core rev: 21df7acc969f47d615d1701ee71f19571de94949)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* on bigger layer this script takes couple minutes without showing any progress
* add a version to reference it in the conversion commit (to easily figure out
if it's worth re-converting the layer).
(From OE-Core rev: 39edf868deafe79ffd2ea57620af70840ec4c208)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds a script I've developed to migrate metadata to use the new override
syntax. It is a bit rough but since its for a single use with validation, it
doesn't need to be perfect. It is run simply as:
scripts/contrib/convert-overrides.py <directory>
It is setup and has been tested to work with OE-Core, Bitbake, yocto-docs,
meta-yocto, meta-gplv2 and meta-mingw. For OE-Core, it converts around 10,100
lines with about 34 manual fixes needed.
For other layers it would need updating for override names and exclusions for
functions/variable names with "append", "prepend" or "remove" in them.
(From OE-Core rev: e8fc67f6f6baf54ccdf548b947c35dea926842c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
image-manifest: script to generate product/image specific BOM
The image-manifest script generates image specific reports based on
an image manifest file. Currently there is data generated by buildhistory,
pkgdata, and license manifest but this data is poorly formated and spread
across multiple text files. This script can generate a single JSON output
file that is machine readable by other tools.
The manifest-info collects package information and stores the information
in a tarball. manifest-info can be configured using a json configuration
file. The default configuration including all possible options can be
dumped using the dump-config subcommand.
image-manifest takes an image manifest file as input to get the runtime
dependencies. As an option image-manifest can also use the build dependency
file, pn-buildlist, to get the build dependencies excluding native
packages.
This script extends the oe-image-manifest script [0] done by Paul Eggleton
[0]
https://github.com/intel/clear-linux-dissector-web/blob/master/layerindex/static/files/oe-image-manifest
------------------------------------------------------
usage: image-manifest [-h] [-d] [-q] <subcommand> ...
Image manifest utility
options:
-h, --help show this help message and exit
-d, --debug Enable debug output
-q, --quiet Print only errors
subcommands:
recipe-info Get recipe info
list-depends List dependencies
list-recipes List recipes producing packages within an image
list-packages List packages within an image
list-layers List included layers
dump-config Dump default config
manifest-info Export recipe info for a manifest
Use image-manifest <subcommand> --help to get help on a specific command
Co-developed-by: Paul Eggleton <bluelightning@bluelightning.org>
(From OE-Core rev: ad8fec9ce1704866df925bda18a240d6889b1ed5)
Signed-off-by: Anders Wallin <anders.wallin@windriver.com>
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update after tinfoil API changes in bitake for REQUIRED_VERSION.
(From OE-Core rev: 1645c9b093bebf7ced67cbee0009d81d1a215966)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a filter subcommand to filter a task-depends.dot graph produced by
bitbake -g down to just a subset of targets/tasks.
(From OE-Core rev: a14b274b56676ff0ba55a4048169ad60c9514994)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
argparse makes this a lot easier to extend.
(From OE-Core rev: c751ef8fdc111d1c967029cea7a3ed0f88ce851b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Spaces were previously converted to dashes, and underscores caused the
next character in the title/key to be printed using subscript due to the
enhanced string support in gnuplot.
(From OE-Core rev: 1719f7062988889b7e24b871dc2f0e50c7bff07e)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix indentation in buildstats.sh and buildstats-plot.sh
to convert every line to 4 spaces indentation.
(From OE-Core rev: 3971572042ae9bc8226eb92892bb71010287e0a8)
Signed-off-by: Luis Martins <luis.martins@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add option to filter the stats results for a single recipe, allowing
to easilly visuality the task resource consumption per recipe.
This is specially useful when debugging the performance of
the overall bitbake build system.
(From OE-Core rev: b4bf898d53af60f5d69b50ec5cdd7f9b3870f2dd)
Signed-off-by: Luis Martins <luis.martins@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add option to accumulate stats values per recipe, allowing
to sum related values such as memory (main process + childs).
This is specially useful when debugging the performance of
the overall bitbake build system.
(From OE-Core rev: e6f1aea6260343cb5194f7f8ab70213b705ab441)
Signed-off-by: Luis Martins <luis.martins@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Extend buildstats-plot.sh script to also accept the name
of the tasks as parameter.
This value will be passed directly to buildstats.sh is
already provides this option.
(From OE-Core rev: 495b6d3d85d2d14d54e324d8da43311a23fdfca6)
Signed-off-by: Luis Martins <luis.martins@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix the internal call to buildstats.sh by also providing
the buildstats folder location in the system, which might
differ from the default location.
(From OE-Core rev: 9f4ee2ee8712528ac51ef4ecd17ccde737e8b21d)
Signed-off-by: Luis Martins <luis.martins@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
