Line lengths, remove duplication, and use the PYTHON variable provided by
pythonnative.bbclass.
Coincidentally fixes a dormant defect in distutils3.bbclass in which we were
sedding for STAGING_BINDIR_NATIVE/python-python3/python3.
(From OE-Core rev: 7b8dd17c65e2d7d163f452833f21469918bf222e)
Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The python distutils generate a python wrapper script for each package,
containing shebang lines pointing to the python executable.
In our case, this is a fully-qualified path to python-native in the
recipe-sysroot-native.
Ubuntu 18.04 restricts the useful length of the shebang line to 125
characters, and Ubuntu 16.04 restricts it to 77. In both cases, the
staged python script fails to run due to the length of the path to
the python-native executable.
Replace the shebang line with nativepython or nativepython3 as appropriate.
The nativepython symlink is installed by the python-native recipe:
#!/usr/bin/env nativepython
We were already doing this for on-target distutils components.
This change applies the sed-line to -native distutils components as well.
In this way, -native clients of these components can invoke the wrapper scripts
directly, without themselves needing to inherit pythonnative.
This works around a known setuptools issue:
https://github.com/pypa/setuptools/issues/494
Even once this issue has been resolved upstream,
we will still need to replace `python` with `nativepython`
(From OE-Core rev: 0a5bd4f30182a3d55153e5ade76f1dfd5b52b482)
Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For non-python developers it's difficult to identify where start
searching in case of errors. Fixing and marking the string to
grep for might help finding some root causes of issues slightly
quicker.
(From OE-Core rev: 44fff749ee7bfe47e88286aec1e8a22e63079ca7)
Signed-off-by: Jens Rehsack <sno@netbsd.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For distutils3, any setup.py invoking will cause setup_requires
argument to trigger a code fetching. Since the following commit
applied in oe-core, code fetching occurs during do_confugire
before the do_compile.
...
b805cef distutils: clean the build tree in do_configure
...
Refer what do_compile did, add var-NO_FETCH_BUILD to do_configure.
Sync with distutils3, add do_configure to distutils also.
[YOCTO #12084]
(From OE-Core rev: 9d556092fcc6f04c487afd126d52935bac133165)
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- Unify var-DISTUTILS_INSTALL_ARGS in distutils and setuptools
- Supply "--root" directory to the "install" command, and use
it as a prefix to strip off the purported filename encoded
in bytecode files. (It strips build path prefix from .pyc files)
[YOCTO #8446]
[YOCTO #12084]
(From OE-Core rev: 21f0c5a50e00ac34975d79fd48f08730d8f19b15)
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a python recipe is using setuptools and the setup_requires argument, where
setuptools will use easy_install to fetch the module if it isn't
present.
The build failed on a machine where a proxy was required, but succeeded on a
machine which had direct access to the internet
Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not
allow to fetch code from internet during do_compile.
Example result:
...
ERROR: Do not try to fetch `pytest-runner1' for building. Please add its native recipe to DEPENDS.
Traceback (most recent call last):
File "setup.py", line 56, in <module>
...
The improvement is flexible for test_requirements argument (used at
`setup.py test'), where use easy_install also.
[YOCTO #12084]
(From OE-Core rev: 57414f4a1c549e6faaa110e6e95ff601b05b7361)
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit 11229de (distutils: Replacing path to native python by path
to python in the image to support python packages with
console-script setup) replaces path to python-native with path
to the python interpreter in the target image.
However on nativesdk builds ${bindir} expanded to the
${SDKPATHNATIVE}${prefix_nativesdk} making shebang line
pointing to the absolute path to env(1) in SDKPATHNATIVE
which may not be present if coreutils isn't part of nativesdk.
On the other hand commit cf63d90 (bitbake.conf: Define USRBINPATH
globally instead of individually) introduces USRBINPATH variable
which has correct value regarding build class and intended for
this use case.
v2: Rebased on top of new head.
Cc: XE-Linux <xe-linux-external@cisco.com>
(From OE-Core rev: 2a83c22a510e10b169015ce936eb51a6fc959ec1)
Signed-off-by: Serhii Popovych <spopovyc@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In entire meta/classes/ directory, replace shell tests of the form
"if test -? ..." with POSIX tests of the form "if [ -? ...
(From OE-Core rev: 78928016f4cf38cf6751cb089200bf950d07ae93)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code that utilized them was superseded by the code (in the same patch!)
that is utilizing STAGING_LIBDIR/STAGING_INCDIR, and wasn't correct in the
first place as HOST_SYS is not necessarily the same as the sysroot directory
name.
(From OE-Core rev: 8834e81a38c24a066bb4fefa93da61011d0db244)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to see the output of the setup script in the error message
because what we are having bbfatal print here doesn't completely
describe what went wrong, thus we should use bbfatal_log here and not
bbfatal.
(From OE-Core rev: 970a02061b474a238f57cd0cc4db7b62e36dfa3d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When installing backport modules they stumble upon each other, complaining with
the following error:
ERROR: The recipe python-backports-ssl is trying to install files into a shared
area when those files already exist.
This is the correct behavior since thats just the way they were designed, all backport
modules provide an __init__.py file (the same among all packages), and without it they
simply wouldnt work.
distutils handles FILES_${PN}* variables for python packages, but it uses wildcards
to include the required files, hence removing the __init__.py files from each backport
package during build time is impossible since it doenst actually contain that value,
this patch simply removes the __init__.py* files from the staging area if they already
exist on sysroot, this way, these are not included in FILES_${PN} anymore, fixing the
issue mentioned above.
[YOCTO #8207]
(From OE-Core rev: e607a89267949b3d5e199b139f752db6000eac25)
Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
py files are edited by sed and therefore *.pyc files are recreated on first boot, but if you have a read-only filesystem this is not possible. This patch creates pyc files directly after the py files are modified.
[YOCTO #7722]
(From OE-Core rev: a0460ac8a2595d4b064b483ca1f282a255ae6411)
Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Thomas Roos <roosesweb@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
setup.py was called with build as argument, but the error message
would report build_ext.
(From OE-Core rev: 898b286b92b031db64c2143e81a79fbb20da481a)
Signed-off-by: Mario Domenech Goulart <mario@ossystems.com.br>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
read -d is a bashism. Replace with a direct exec to avoid the problem
in this case. This fixes silent build failures in do_install of
tasks on systems with dash as /bin/sh.
Also merge the fix to distutils for only changing necessary files
to disutils3 as well.
(From OE-Core rev: 7d61661348cf48cbe379ae600565840ea08664b1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently sed command touches every single *.py file. This modifies the
timestamp of the file. All *.pyo files will be recompiled during the first
boot, because timestamp will not match. This should be only necessary if
sed command changes the file.
(From OE-Core rev: 2d01c5a4989dcf03a202c27730a2a8f334e0c37a)
Signed-off-by: Radek Dostal <radek.dostal@streamunlimited.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is needed to accomodate python3 alongside
python2
(From OE-Core rev: ae931c2cf9e48e1fb74b4b727dbf668ea880023f)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Multiple recipes package this generated file and will clash during installation with:
| Collected errors:
| * check_data_file_clashes: Package python-nose wants to install file /build/v2013.06/build/tmp-angstrom_v2013_06-eglibc/work/beaglebone-angstrom-linux-gnueabi/cloud9-gnome-image/1.0-r0/rootfs/usr/lib/python2.7/site-packages/site.pyc
| But that file is already provided by package * python-setuptools
| * check_data_file_clashes: Package python-nose wants to install file /build/v2013.06/build/tmp-angstrom_v2013_06-eglibc/work/beaglebone-angstrom-linux-gnueabi/cloud9-gnome-image/1.0-r0/rootfs/usr/lib/python2.7/site-packages/site.py
| But that file is already provided by package * python-setuptools
(From OE-Core rev: e53192d98ff8cdc4abe85b42e792da2759b22f84)
Signed-off-by: Koen Kooi <koen@dominion.thruhere.net>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
distutils: Replacing path to native path only to be done to non-native python packages
distutils: Replacing path to native python by path to python in the
image to support python packages with console-script setup resulted in
a "bad interpreter" error message because coreutils-native is not
a specified dependency of a number of native python packages.
We modify the change to apply specifically to non-native packages.
(From OE-Core rev: 312b6b33dca565153bc2e92d7ff6dd2974db4edb)
Signed-off-by: Amy Fong <Amy.Fong@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using distutils for a python package based on a python-setuptools
installation script that sets up a console script, the header
of the console script created by setuptools points to the
python-native path.
The console scripts are commonly executed in the image, but not
in the sysroot environment. Therefore, the header of the
console scripts should point to the python interpreter in the
image.
Setuptools does not allow to set the path of the python
interpreter via some command-line argument.
Hence after the installation script ran, the distutils
class replaces the path in the console script files created by
the installation.
(From OE-Core rev: 11229def87d048c51190b9bd275c73d1f8bf6007)
Signed-off-by: Lukas Bulwahn <Lukas.Bulwahn@oss.bmw-carit.de>
sgw - added \ to protect the space.
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* fixes a lot of QA warnings about unpackaged /usr/share/share
(From OE-Core rev: 10457e343b27e63b35e0278b00e9c25106b0cc1b)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The last two distutils changes progressivly broke the builds. Firstly they
moved things from the site_packages directory to being higher up the tree
which introduced package QA warnings as a side effect. Secondly, it interacts
badly with setuptools which passes in --root=${D} itself.
This patch restores the original directory layout, hence fixing the QA
warnings and also passes extra options to setuptools to deal with the
--root option it passes.
(From OE-Core rev: bed18d5df7915e4127a538be9c7550e185c8c850)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This let's the user override install-lib argument again if it needs
to be something else, otherwise things like python-setuptools
won't be able to modify the install-lib dir
This fixes a new issue exposed by my previous distutils patch
that fixed the python modules default install location. Also,
it removes running the install step twice which was inadvertant
(From OE-Core rev: 3b23feca31480cc56f55301fd0274e622c40b522)
Signed-off-by: Matthew McClintock <msm@freescale.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this some modules will be intalled in /usr/lib/python2.6/
instead of /usr/${libdir}/python2.6
(From OE-Core rev: bc6bd774aa8a3e085e9cabcefb11c3fc537139d5)
Signed-off-by: Matthew McClintock <msm@freescale.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update python-native to install the binaries in the python-native directory,
add pythonnative.bbclass to let recipes that need python-native use the
binaries and update disutils access the new binaries. Update distutils-base
to inherit pythonnative.
(From OE-Core rev: a2e554f731437545e9483a7a73e6847c03f6f48a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Deleting .pyo files causes them to get compiled on the target.
* First boot gets *really* slow for python based projects.
* No space gets saved on the target.
* The package manager doesn't know about the files and
therefore fails to uninstall them, occupying space and causing
uninstalled python scripts to remain executable.
* It's inconsistent, because python itself and autotools based
projects already ship .pyo files.
* Probably .pyo files were deleted because .pyc files were
available earlier, but this has changed and OE-Core's python
now only generates optimized .pyo files. Deletion of .pyo was
introduced in 2008, python/04-default-is-optimized.patch
was introduced in 2009.
(From OE-Core rev: 25e186ad5b75bd2f93435857580bd16698e18e21)
Signed-off-by: Andreas Oberritter <obi@opendreambox.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this fix, when packages are being built using distutils and
the python packages were deployed from sstate-cache is it possible
that the LD command will contain an invalid sysroot override.
We can fix this by always exported LDSHARED, which is the env var
that distutil looks for to override creating shared libraries.
(From OE-Core rev: 3f6b859a29ba7f570b9dae3b5bb7ab4bd7b8cee4)
Signed-off-by: Matthew McClintock <msm@freescale.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The new bash logging class provides bbnote, bbwarn, bbfatal, and bbdebug
replacements (as well as bbplain and bberror) for the oe* equivalents. Use the
new bb* API in preparation to delete the oe* logging API.
This patch was automatically generated by a sed script. The result has been
visually inspected and used to build core-image-sato for qemux86.
(From OE-Core rev: a1f09fce5caba389d0484b169f0cde85d64514fa)
Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using python recipes imported from OE (e.g. python-cheetah) parsing only succeeds when python has already been built due to the PYTHON_DIR references.
This commit syncs the classes with OE to make it work, but keeps the *.pyo removal from yocto.
Signed-off-by: Koen Kooi <koen@dominion.thruhere.net>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Remove layout_* variables and replace them with variables specific to the
different classes. The layout variables were only useful for the native/cross
classes and caused more confusion than they solved. They didn't scale to the
sdk class. It now clear a small set of native/cross variables fulfil the needs.
This patch also changes native.bbclass to use "/" as the STAGING_DIR which makes
sense since we're installing binaries into the locations we're compiling them for.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
