This function, to tidy a license string, is useful outside of recipetool
so move it to oe.license.
(From OE-Core rev: 9d57b53169bc60b281510c49e54123941a17a8f5)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When creating a recipe there can be cases where there is a class that
does some of the recipe creation (such as cargo-update-recipe-crates).
To avoid duplication of code, look for run_task assignments in the
extravalues dictionary returned by the handler, and if it is set then
call that task after writing the recipe.
(From OE-Core rev: 629184de671cf7b86fe95fcce788dffcc3a4da8a)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that recipes default to S in UNPACKDIR, recipetool and devtool should
do the same.
There was some discussion about changing devtool to simply setting
UNPACKDIR via bbappend to a workspace and running unpack task directly;
currently it has a bunch of convoluted path calculations, substitutions,
moving source trees around and and special casing (devtool-source.bbclass
in particular is an unpleasant hack).
This should definitely be done; but right now we can simply tweak existing
code which at least doesn't make it worse.
(From OE-Core rev: c326ca8aeb2bf0f7719e43921d10efd5dedc7b2a)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removing all the S = ${WORKDIR}/git assignments works because BB_GIT_DEFAULT_DESTSUFFIX
is set to match S from bitbake.conf (which itself is set to match typical tarball
releases).
A few recipes are setting S to a sub-directory of the git tree and need
to be adjusted accordingly.
bzip2 recipe is fetching a tarball and separately cloning tests;
adjust the recipe to put the latter into 'bzip2-tests', instead of 'git'.
devupstream.bbclass no longer needs to rewrite S, and is adjusted accordingly.
Adjust scripts/lib/recipetool/append.py to not hardcode 'git' as unpack
destination.
Adjust kernel-yocto.bbclass to use the git unpack variable instead
of hardcoding 'git' (there's also removal of repetition of
string constants and a correction of workdir/unpackdir mismatch in
one of the if-else branches).
Ensure build-appliance-image recipe does not use 'git' as checkout directory for
poky repo, but rather explicitly name it 'poky'.
Ensure reproducible.py code that looks for git repositories does not
hardcode 'git' but uses the destination set by BB_GIT_DEFAULT_DESTSUFFIX.
Ensure recipetool does not write out unneeded S settings into newly
created recipes that fetch from git.
Adjust selftest to not hardcode 'git' as unpack directory.
(From OE-Core rev: f80c07019ddadaf9c5fb890faabfda7920ecd15e)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Delete the now redundant code, and import oe.license_finder instead.
(From OE-Core rev: 8bba98be5c87dd6749e5cc95e9553dffc23ada73)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are multiple source trees in a project (incredibly common with
go-mod, for example) then the relative path of the LICENSE file from
the source tree could just be "LICENSE", which is not useful when there
are tens of files across the recipe with that name.
Show the parent directory name too, to clarify which file is unknown.
(From OE-Core rev: 9679f4055ad5a077c6b06aa6125cee4e8fa93471)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
create_npm.py duplicated the logic for matching licenses from files and
also finding them. This patch refactors the code to reuse the license
utils. This will make the code more maintainable and also align both
behaviors. For instance, some licenses weren't matched properly because
the duplicate logic did not support the difference in format in the md5
tables for COMMON_LICENSE_DIR and licenses.csv.
This is also faster since the license files were being read twice.
The result is slightly more accurate since the utils have better
implementations, and I was able to reuse the logic for the root PN
package, as well as the base LICENSE variable.
I chose to extract generate_common_licenses_chksums into create.py
since it can be considered a general utility function to allow
other recipetool creators to refer to COMMON_LICENSE_DIR files.
I updated the wording in the code when appropriate.
v3:
- added commit
- this replaces the commit that added all the COMMON_LICENSE_DIR md5
to licenses.csv
(From OE-Core rev: 7bc18bed63b94689890bcde63402d7cc1cedffa9)
Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The npm recipetool handler redefines the license code the could be
unified. In order to do this refactoring, extract the bits we'll
need into separate functions.
guess_license() is renamed to find_licenses() and is split into
find_license_files() and match_licenses().
(From OE-Core rev: f1ec28feaea8ea6a2df894dd4ddba561c8a04ed2)
Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Tweak recipetool and devtool to correctly use UNPACKDIR. This allows some
simplification of the code. This patch makes things basically work but there
are likely deeper improvements that can be made now that WORKDIR != UNPACKDIR.
(From OE-Core rev: d2eeaa88b27a2875c419591d1d91bcc85d7b129c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
All the regexes throw a warning like this:
WARNING: scripts/lib/recipetool/create_buildsys.py:140:
SyntaxWarning: invalid escape sequence '\s'
proj_re = re.compile('project\s*\(([^)]*)\)', re.IGNORECASE)
Python 3 interprets string literals as Unicode strings, and therefore
\s is treated as an escaped Unicode character which is not correct.
Declaring the RegEx pattern as a raw string instead of unicode is
required for Python 3.
(From OE-Core rev: 24b0ba00d4f0b4d9834f7693ecb6032dfc534a80)
Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than including all SRC_URI checksums, include the ones that are
expected. These are the same as are output if no checksums are included
when building the recipe.
(From OE-Core rev: c2af83eb5e8573480179b6c0bcce50606b547099)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than having a static list of crunched MD5 checksums for some of
the most common licenses, calculate it for all common licenses. This
should improve the identification of license text variantions.
(From OE-Core rev: 377f9513dc56e9b8e5f5813c1535be0206756949)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Today, we can use devtool/recipetool to create recipes for python projects
using the github url or the direct release tarball of the project, but the
create_buildsys_python plugin doesn't support the pypi class, since we cannot
know from the extracted source if the package is available on pypi or not.
By implementing the new optional process_url callback, we can detect
that the url is a pypi one (i.e 'https://pypi.org/project/<package>')
and retrieve the release tarball location.
Also detect if the url points to a release tarball hosted on
"files.pythonhosted.iorg" (i.e https://files.pythonhosted.org/packages/...)
In both cases, adds the pypi class, remove 'S' and 'SRC_URIxxx'
variables from the created recipe as they will be handled by the pypi class
and add the PYPI_PACKAGE variable
This helps to produce cleaner recipes when package is hosted on pypi.
If the url points to a github url or a release tarball not coming from
"files.pythonhosted.org", the created recipe is the same as before.
One can also use the newly added "--no-pypi" switch to NOT inherit
from pypi class on matching url, to keep legacy behaviour.
To create a recipe for a pypi package, one can now use one of the
new following syntax (using recipetool create / devtool add):
* recipetool create https://pypi.org/project/<package>
* recipetool create https://pypi.org/project/<package>/<version>
* recipetool create https://pypi.org/project/<package> --version <version>
or the old syntax:
* recipetool create https://files.pythonhosted.org/packages/<...>
(From OE-Core rev: 097a43846cd99a7d74d004efc57f583ce78970a4)
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a new process_url callback that plugins can optionally implement if
they which to handle url.
Plugins can implement this callback for example, to:
* transform the url
* add special variables using extravalues
* add extra classes
* ...
If a plugin handles the url, it must append 'url' to the handled
list and must return the fetchuri
No functional changes expected for plugins non implementing this
optional callback
(From OE-Core rev: 3e5ce351e7dbe283562bf1db1f2ce3b121c49b53)
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
create_recipe() function relies on oe.recipeutils.patch_recipe_lines()
which relies on bb.utils.edit_metadata(). edit_metada expect lines to
have trailing newlines, so add it to each lines before calling
patch_recipe_lines, otherwise edit_metadata will not be able to squash
blank line if there are two consecutive blanks after a removal
(From OE-Core rev: 31b27cc76a1b669e2b126c332608fd772c124d69)
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake has changed to require notification when metadata changes in the middle of tinfoil
sessions. Add the required function calls at the places metadata is changed.
(From OE-Core rev: e5574163ab49a8f51b2b34fd37acfd1cad9b7595)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that SRCPV isn't needed we can simplify things in a few places...
(From OE-Core rev: 843f82a246a535c353e08072f252d1dc78217872)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
native and nativesdk classes are special and must be inherited last :
put them at the end of the gathered classes to inherit.
(From OE-Core rev: a6614fd800cbe791264aeb102d379ba79bd145c2)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove mutable default arguments in Python because they can lead to all
sorts of nasty and horrible bugs.
https://florimond.dev/en/posts/2018/08/python-mutable-defaults-are-the-source-of-all-evil/
Revert `recipetool: Change default paramter fallback_licenses of
function split_pkg_licenses from None to []` and instead check
fallback_licenses before use.
(From OE-Core rev: 99dee60b8db557f54783bf0f61098587badc683c)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some of these are hard to know what to do with since the original source
files for the checksum aren't present. The safe option is to use "-only"
as often the main license is ambiguous and the source files themselves
determine the "or-later" possibility. The "-only" option therefore is
realistically what we need to use in this code.
(From OE-Core rev: 2b0cbafc7854de0308a624b17b8aaba704b031d5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are still a couple of cases where the license may be set as, e.g.,
"GPL" or "GPL-2.0" since there is not enough information to decide the
actual SPDX license. It is then assumed that the developer will have to
correct the information.
(From OE-Core rev: e7df51f8d2361e9fe2d67669d2e17f0a5d01004e)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When creating a new recipe.bb file for a binary, don't use BP which
includes the version information, instead use BPN which is just the
name base Package Name.
Since PB is not specified, it takes the default:
PV = "1.0+git${SRCPV}"
But SRCPV is defined in terms of the SRC_URI, which leads to infinite
recursion (traceback below). Here are the pertinent variables which
cause the recursion:
SRC_URI = "git://github.com/lvc/abi-dumper;protocol=https;subdir=${BP}"
BP = "${BPN}-${PV}"
PV = "1.0+git${SRCPV}"
SRCPV = "${@bb.fetch2.get_srcrev(d)}"
def get_srcrev(d, method_name='sortable_revision'):
# ... trimmed
scms = []
fetcher = Fetch(d.getVar('SRC_URI').split(), d)
# ... trimmed
[YOCTO #14040]
(From OE-Core rev: 3b8d43fc53ee13d39abc3b2a1f706a97fcf752aa)
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Separate licenses with & operator since it should be satisfied most use
cases and it is a reasonable assumption that all the licenses apply.
Furthermore flat, split and sort the licenses to minimize license string
changes.
Separate package licenses with & operator:
-LICENSE:${PN} = "MIT ISC"
+LICENSE:${PN} = "ISC & MIT"
Respect | and brackets in LICENSE:
-LICENSE = "BSD-3-Clause & (ISC & | & MIT)"
+LICENSE = "BSD-3-Clause & (ISC | MIT)"
Sort licenses:
-LICENSE = "MIT & BSD-3-Clause & ISC"
+LICENSE = "BSD-3-Clause & ISC & MIT"
Remove duplicates:
-LICENSE = "MIT & ISC & MIT"
+LICENSE = "ISC & MIT"
(From OE-Core rev: 60a84ecc53d20118c5e7f86dd3e3cafbfed1cf0a)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Sort the output of guess_license function by license file to be
deterministic and to support self-testing.
(From OE-Core rev: ff2a7520813b3f775f96879d1905222e08b8c83f)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The commit 'meta/scripts: Manual git url branch additions (dc53fe75cc)'
sets the branch= parameter too early to master and thereby breaks the
-B/--srcbranch option.
ERROR: branch= parameter and -B/--srcbranch option cannot both be specified - use one or the other
(From OE-Core rev: 34ece8030e7a6a100b5e3e7b94e6c786c0e199a6)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
GitHub URLs are automatically transformed to git: fetches, so handle
GitLab URLs too.
(From OE-Core rev: 651fb951819840fe122458ddbd852ee6c7ec0455)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Following the scripted conversion adding branches to git://
SRC_URI entries, add the remaining references, mainly in the selftests
and recipetool.
(From OE-Core rev: 5340c0d688036c1be6c938f05d8a8c1e3b49ec38)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for linenumbers (begin and end lines) to licenses.csv. Add
an optional linenumbers parameter to get_license_md5sums to support
different use cases.
(From OE-Core rev: d5c4979669f125e73c24dcc73fa3c4f3787bbb62)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a long standing bug where FILES_INFO isn't written into pkgdata
with a package suffix. This means if the files are read into the datastore
as intended, the last one "wins".
Fix this to work as intended. Most of the call sites using the data need
to be updated to handle this and the overrides change correctly.
Also fix some other problematic references noticed along the way.
(From OE-Core rev: a1190903e0a61a12c9854c96af918ae8d12c6327)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is the result of automated script conversion:
scripts/contrib/convert-overrides.py <oe-core directory>
converting the metadata to use ":" as the override character instead of "_".
(From OE-Core rev: 42344347be29f0997cc2f7636d9603b1fe1875ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python now expects regex strings to be prepended with r.
Silence pylint/autopep8 and similar warnings by identifying
these regex patterns as... regex patterns.
(From OE-Core rev: 0eae98a369d80340e48dc690d09a1364cde97973)
Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When explicitly passing a branch using --srcbranch in 'devtool add' or
'recipetool create', the branch name is not included in the params of
bb.fetch2.encodeurl and default 'master' branch is used instead.
(From OE-Core rev: 0424df825f1e509faf6cd44403c0736bb91b57c3)
Signed-off-by: Tuomas Salokanto <tuomas.salokanto@gmail.com>
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit removes the 'noverify' parameter which was added to the url
to fix warnings with the shrinkwrap / lockdown file generation. This is
not needed anymore with the new npm fetcher.
(From OE-Core rev: 401a9818c633373854d3c7dec032b9f455f2b2b4)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit renames the '--fetch-dev' option into '--npm-dev' as it is a
npm only option.
(From OE-Core rev: 2b75cc848ceebee4067788a621299bfd5fb62231)
Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the reproducible_build class is inherited then there may be a
"source-date-epoch" subdirectory in a fetched source tree; devtool
upgrade was not expecting that in the upgraded source. Take a small
snippet of code from recipetool create which already handles this,
and make it a shared function that can be used in both places.
Additionally, fix an assumption that the source is always in a
subdirectory in the cleanup code that blocked debugging this.
[YOCTO #13635]
(From OE-Core rev: 0d642861cd9cf034b8d4951433980addc215d4fd)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The extra diretory created by reproducibile_builds was confusing recipetool
and hence devtool. Ignore this directory to restore the correct behaviour and
allow it to work with the class.
(From OE-Core rev: 188ba30ef290efab6d5a9b95fba8e991b779ab33)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This switches the code to build pkgdata specific to the current recipe
which means that its filtered to the recipes dependencies and can perform
better as we can drop the lockfile.
It uses a similar method to the staging code to do this, using BB_TASKDEPDATA
to construct a list of packagedata task output which this recipe should "see".
The original pkgdata store is left unaltered so existing code works.
The lock file was there to prevent files disappearing as they were read or as
directories were listed. Since we have a copy of the data and only access output
from completed tasks (as per their manifests), we can remove the lock.
The lock was causing starvation issues on systems with parallelism.
There was also a potential determinism problem as the current code could "see"
data from recipes which it doesn't depend upon.
[YOCTO #13412]
(From OE-Core rev: 1951132576bfb95675b4879287f8b3b7c47524fa)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.
The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).
More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.
The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.
(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If attempting to find a fixed SRCREV fails because the directory doesn't exit,
avoid failing with:
Traceback (most recent call last):
File "/home/vagrant/poky/scripts/recipetool", line 121, in <module>
ret = main()
File "/home/vagrant/poky/scripts/recipetool", line 110, in main
ret = args.func(args)
File "/home/vagrant/poky/scripts/lib/recipetool/create.py", line 707, in create_recipe
srcrev = stdout.rstrip()
UnboundLocalError: local variable 'stdout' referenced before assignment
Fixes: 000480c42797 ("recipetool / devtool: set a fixed SRCREV by default when fetching from git")
(From OE-Core rev: 8e4b301573071ea04911b0df6533c25bc4596b93)
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The warn method is deprecated. We should use the documented warning instead.
Quoting from the python's official doc:
"""
Note: There is an obsolete method warn which is functionally identical to warning.
As warn is deprecated, please do not use it - use warning instead.
"""
(From OE-Core rev: f467fd277eb77336097cfc0f5f329bdc8d0f70cb)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
