Add the new python_maturin PEP-517 backend
Add selftest for 'pydantic-core' pypi package.
(From OE-Core rev: 69b679380616a94a631681caa05d9bf7610f9372)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The pypi change:
"85a2a6f68af recipetool: create_buildsys_python: add pypi support"
deleted all the SRC_URI variables, including the SRC_URI checksums.
These are not generated by the pypi.bbclass (how could they be trusted?)
Without the checksum(s), we are vulnerable to a man-in-the-middle attack
and zero checks on the validity of the downloaded tarball from pypi.org.
Fix by only setting S and SRC_URI to None.
(From OE-Core rev: 560181a52111569f7bc57b09139b42510e0d0325)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Today, we can use devtool/recipetool to create recipes for python projects
using the github url or the direct release tarball of the project, but the
create_buildsys_python plugin doesn't support the pypi class, since we cannot
know from the extracted source if the package is available on pypi or not.
By implementing the new optional process_url callback, we can detect
that the url is a pypi one (i.e 'https://pypi.org/project/<package>')
and retrieve the release tarball location.
Also detect if the url points to a release tarball hosted on
"files.pythonhosted.iorg" (i.e https://files.pythonhosted.org/packages/...)
In both cases, adds the pypi class, remove 'S' and 'SRC_URIxxx'
variables from the created recipe as they will be handled by the pypi class
and add the PYPI_PACKAGE variable
This helps to produce cleaner recipes when package is hosted on pypi.
If the url points to a github url or a release tarball not coming from
"files.pythonhosted.org", the created recipe is the same as before.
One can also use the newly added "--no-pypi" switch to NOT inherit
from pypi class on matching url, to keep legacy behaviour.
To create a recipe for a pypi package, one can now use one of the
new following syntax (using recipetool create / devtool add):
* recipetool create https://pypi.org/project/<package>
* recipetool create https://pypi.org/project/<package>/<version>
* recipetool create https://pypi.org/project/<package> --version <version>
or the old syntax:
* recipetool create https://files.pythonhosted.org/packages/<...>
(From OE-Core rev: 097a43846cd99a7d74d004efc57f583ce78970a4)
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the case pyproject.toml doesn't contains metadatas, the metadata
variable is not initialized and the plugin throws an error and falls back
to another plugin, which is not the desired behaviour. So just ignore
metadata if we don't have them
(From OE-Core rev: 88d15877ba18309c521740d7a9649e14d77189bc)
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
One of the newer PEP-517 backends to be added was python_hatchling.bbclass
but it was not included in the recent improvements.
Add selftest for 'jsonschema' pypi package.
(From OE-Core rev: d99b4883b4fee82bc588fd235ba90fedf1550cb8)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add basic support for PEP517 [1] for the 3 following backends that are
supported by bitbake:
- setuptools.build_meta
- poetry.core.masonry.api
- flit_core.buildapi
If a pyproject.toml file is found, use it to create the recipe,
otherwise fallback to the old setup.py method.
Some projects can declare a minimal pyproject.toml file, and put all
the metadata in setup.py/setup.cfg/requirements.txt .. theses cases
are not handled. If a pyproject.toml file is found, assumes it has all
necessary metadata.
As for the old setup.py method, version numbers for dependencies are not
handled.
Some features may be missing, such as the extra dependencies.
[YOCTO #14737]
[1]: https://peps.python.org/pep-0517/
(From OE-Core rev: c7d8d15b2d0a9ecd210bd247fa0df31d9f458873)
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to prepare the support for pyproject.toml (PEP517 [1]) enabled
projects, refactor the code and move setup.py specific code into a
specific class in order to allow sharing the PythonRecipeHandler class
No functionnal changes expected
[1]: https://peps.python.org/pep-0517/#source-tree
(From OE-Core rev: 2281e93347da4129062cfb40710df03c87c63168)
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By convention, all python recipes start with "python3-" so update
create_buildsys_python to do this
This rule doesn't apply for packages already starting with "python"
Update recipetool's selftest accordingly
(From OE-Core rev: b0d87440e610b80f763d09784d4a90a148bb3e7b)
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
License field of setup is not always standardized, so we usually use the
classifier to determine the correct license format to use in the recipe.
A warning note is added above the LICENSE field of the create recipe
in case a license is provided in setup. But when the plugin is called,
"LICENSE =" is not yet present so we can never display this note.
Replace the "LICENSE =" condition with "##LICENSE_PLACEHOLDER##"
to actually be able to display the note message
(From OE-Core rev: b7c26ca2028aa60f740464de85a11a01a531f32e)
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
'imp' was deprecated in Python 3.4 and removed in 3.12. The
piece of importlib we use has been around since 3.3.
(From OE-Core rev: 457f0dad87b4e45a53865b5ad2c150215bd74019)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Substitute expressions or whitespace from python egg requires.txt when
generating PACKAGECONFIG
Pysetuptools sees the uvicorn.egg-info/requires.txt as extra requirements.
Recipetool parses this information to generate the PACKAGECONFIG.
These extra requirements contain expressions and whitespace, which are not allowed in PACKGAGECONFIG.
This patch substitute them by hyphens to make PACKAGECONFIG parsable and readable.
Also adding an oe-selftest for this.
[YOCTO #14446]
(From OE-Core rev: a854d95a79e64f3f82abfa4cc1daec750abf4249)
Signed-off-by: Thomas Roos <throos@amazon.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are still a couple of cases where the license may be set as, e.g.,
"GPL" or "GPL-2.0" since there is not enough information to decide the
actual SPDX license. It is then assumed that the developer will have to
correct the information.
(From OE-Core rev: e7df51f8d2361e9fe2d67669d2e17f0a5d01004e)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add all OSI approved licenses from https://pypi.org/classifiers/. Also
add support for Other/Proprietary (Proprietary) and Public Domain (PD).
(From OE-Core rev: 99ef134d1019e5b98b845cf71f3eb39871218f9d)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
'distutils' is deprecated in Python 3.10 with removal in Python 3.12
(~October 2023). Replace 'distutils.command.build_py' with
'setuptools.command.build_py'.
To avoid an AttributeError, we call super().__init__() which provides
the missing 'distribution' attribute. However, for some reason, __init__()
in 'setuptools.command.build_py.build_py' class requires a 'dist' positional
argument which must be a 'Distribution' instance. It is not clear why
'distutils.command.build_py.build_py' class does not require this.
There is still a check which decides to inherit setuptools3 vs distutils3
that will need to be refactored when we add pyproject.toml and setup.cfg
support for more modern PEP 517 packaging.
Once distutils3.bbclass is dropped, any recipe inheriting distutils3
will throw a parsing error. The plan is to move distutils*.bbclasses to
meta-python. However if meta-python is not in bblayers, the parsing
error would still occur.
[YOCTO #14610]
(From OE-Core rev: 619a3eb1266459daf16e10386113e9201fbf9cf5)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The PyPI license classifiers include "OSI Approved: BSD", which does not
specify which of the many variations of BSD license it actually means.
The generic "BSD" license in the oe-core set is actually BSD-3-Clause.
>From a random sample of ten PyPI modules that use "BSD", they are all
BSD-3-Clause. As we expect the recipe maintainer to verify the license
anyway, and this matches the previous license text, explicitly set the
license to BSD-3-Clause.
(From OE-Core rev: a879fff9af31e45b1acc3f19a3c2a7eaf6319ad4)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a long standing bug where FILES_INFO isn't written into pkgdata
with a package suffix. This means if the files are read into the datastore
as intended, the last one "wins".
Fix this to work as intended. Most of the call sites using the data need
to be updated to handle this and the overrides change correctly.
Also fix some other problematic references noticed along the way.
(From OE-Core rev: a1190903e0a61a12c9854c96af918ae8d12c6327)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is the result of automated script conversion:
scripts/contrib/convert-overrides.py <oe-core directory>
converting the metadata to use ":" as the override character instead of "_".
(From OE-Core rev: 42344347be29f0997cc2f7636d9603b1fe1875ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a setup.py file exists it ought to have something in it before we
consider the source tree to be a Python module and treating it as such.
(A counter-example is https://www.bro.org/downloads/binpac-0.50.tar.gz -
it's not clear why this has a zero-length setup.py in it but we should
pay no attention to it.)
Fixes [YOCTO #12923].
(From OE-Core rev: 548a5c8f42c6ac1b0f7962926d05276e71505678)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for generating python3 recipes using the recipetool / devtool.
Drop python2 support at the same time.
Tested with:
oe-selftest -r recipetool.RecipetoolTest
[YOCTO #13264]
(From OE-Core rev: d8b2f58974482b3b1ccc65c5f93104d0d7ba87bc)
Signed-off-by: Maciej Pijanowski <maciej.pijanowski@3mdeb.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.
The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).
More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.
The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.
(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a setup dict in a python setup.py file pulled in the contents of
another dict (e.g. **otherdict), then we got an error when mapping
the keys because the key is None in that case. Skip those keys to avoid
the error (we pick up the values directly in any case).
A quick reproducer for this issue:
recipetool create https://files.pythonhosted.org/packages/source/p/pyqtgraph/pyqtgraph-0.10.0.tar.gz
(From OE-Core rev: ae62a9953e219df5147ed4a5ae3f4163d51cff28)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(From OE-Core rev: 0a36bd96e6b29fd99a296efc358ca3e9fb5af735)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Yet another instance of us expecting a string back from subprocess when
in Python 3 what you get back is bytes. Just decode the output within
run_command() so we avoid this everywhere.
(From OE-Core rev: 103faae78cdff5280c7b7cdb7ca01e0868d02ec9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Try to ensure that for Apache, GPL and LGPL where the values extracted
from the "Classifiers" field may not be version-specific, if there is a
versioned license in the free-form license field then use that instead.
Also insert the free-form license field as a comment in the recipe for
the user's reference.
(From OE-Core rev: 237f66042eedd906f654827b53bf9269738267ab)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make use of the extravalues dict to send back other variable values from
the python handling plugin, and enable passing back PV and PN. This not
only places variable values in the final recipe a bit more consistently
with other types of source, it also allows the name and version to be
picked up fron a local source tree and not just when the recipe is
fetched from a remote URL that happens to have those in it.
(From OE-Core rev: 3e7029f28c6ea9bb1d283bcdc3fdfee11455af8e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When trying to map python module dependencies to the packages that
provide them, if we're looking for .so files that satisfy
dependencies then we need to exclude files found under the .debug
directory, otherwise the dependency will get mapped to the python-dbg
package which isn't correct.
For example, this fixes creating a recipe for pyserial and not getting
python-fcntl in RDEPENDS_${PN}, leading to errors when trying to use the
serial module on the target.
(From OE-Core rev: 46a068ca35975988a8e9c0310f71fdcee55937a4)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Converted return value of items() keys() and values() to
lists when dictionary is modified in the loop and when
the result is added to the list.
(From OE-Core rev: 874a269eb1d70060c2f3b3f8b70800e2aea789f4)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Moved call of decode('utf-8') as close as possible to
call of subprocess API to avoid calling it in a lot of
other places.
Decoded binary data to utf-8 where appropriate to fix devtool
and recipetool tests in python 3 environment.
(From OE-Core rev: 30d02e2aa2d42fdf76271234b2dc9f37bc46b250)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python 3 doesn't have basestring type as all string
are unicode strings.
(From OE-Core rev: e8cfab060f4ff3c4c16387871354d407910e87aa)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a means of extending the dependency extraction for autotools and
cmake.
Note: in order to have this work, you need to have an __init__.py in the
lib/recipetool directory within your layer along with the module
implementing the handlers, and the __init__.py needs to contain:
# Enable other layers to have modules in the same named directory
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
(From OE-Core rev: 915dea9f89cd737e5ba167c384e8d314c5c23c49)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some build systems (notably autotools) support declaring the name and
version of the program being built; since we need those for the recipe
we can attempt to extract them. It's a little fuzzy as they are often
omitted or may not be appropriately formatted for our purposes, but it
does work on a reasonable number of software packages to be useful.
(From OE-Core rev: 3b3fd33190d89c09e62126eea0e45aa84fe5442e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If SRC_URI happened not to be in the pre-generated lines then this code
would error out. This is unlikely to happen with the way the create code
is structured at the moment, but handle it just in case.
(From OE-Core rev: 95d33e90f2d5d9dd5ccc950856b8a939fefb831e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
So far all devtool and recipetool plugins were expected
to have plugin_init function. This function is empty in
most of plugins as they don't require initialisation.
Making plugin_init optional would allow not having empty
plugin_init in every plugin.
(From OE-Core rev: b99640c89f067866b264f4ee4030fae4c2f338c0)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This uses a standalone python script named `pythondeps` which now lives in
scripts. It supports scanning for provided packages and imported
modules/packages, the latter via the python ast. It's not perfect, and
obviously conditional imports and try/except import blocks are handled
naively, listing all the imports even if they aren't all used at once, but it
gives the user a solid starting point for the recipe.
Currently `python_dir` from setup.py isn't being handled in an ideal way. This
is easily seen when testing the python-async package. There, the root of the
project is the async package, so the root has __init__.py and friends, and the
python provides scanning currently just assumes the basedir of that dir is the
package name in this case, which is not correct. Forthcoming patches will
resolve this.
(From OE-Core rev: cb093aca3b78f130dc7da820a8710342a12d1231)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
