Commit Graph

240 Commits

Author SHA1 Message Date
Denys Dmytriyenko
b26e9eed52 convert-overrides.py: handle few more cases of overrides
Add task-configure and few more supported values of TARGET_OS override.

(From OE-Core rev: 1172f9593902f28ddd8da47de6bd51cda9a0f86a)

Signed-off-by: Denys Dmytriyenko <denis@denix.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-04 20:45:41 +01:00
Richard Purdie
54b9a58c20 convert-overrides: Allow script to handle patch/diffs
It is handy to be able to have the script convert a single patch file
and it turns out to be straightforward to make it do this.

(From OE-Core rev: 21df7acc969f47d615d1701ee71f19571de94949)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-02 15:44:11 +01:00
Martin Jansa
926fae24e1 convert-overrides.py: allow specifying multiple target dirs
(From OE-Core rev: e6aa4b1f6f4f174cba027ee096db174541815ff0)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-30 11:31:09 +01:00
Martin Jansa
812ca4cc91 convert-overrides.py: 0.9.1 include '(' as delimiter for shortvars
(From OE-Core rev: ffe91649fafa84f814e32c9979e6a2de40a1bb25)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-30 11:31:09 +01:00
Martin Jansa
b4d63248f0 convert-overrides.py: remove base_dep_prepend and autotools_dep_prepend exception
* the functions were renamed in:
  https://git.openembedded.org/openembedded-core/commit/?id=9d002acae720b0a8e96a6734424a142b86880461

(From OE-Core rev: 43b7cbfb46701a130c248842426370372c88553b)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-30 10:55:13 +01:00
Martin Jansa
1a1bb6ce51 convert-overrides.py: show processed file and version of this script
* on bigger layer this script takes couple minutes without showing any progress
* add a version to reference it in the conversion commit (to easily figure out
  if it's worth re-converting the layer).

(From OE-Core rev: 39edf868deafe79ffd2ea57620af70840ec4c208)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-30 10:55:13 +01:00
Richard Purdie
82c1581e48 scripts/contrib: Add override conversion script
This adds a script I've developed to migrate metadata to use the new override
syntax. It is a bit rough but since its for a single use with validation, it
doesn't need to be perfect. It is run simply as:

scripts/contrib/convert-overrides.py <directory>

It is setup and has been tested to work with OE-Core, Bitbake, yocto-docs,
meta-yocto, meta-gplv2 and meta-mingw. For OE-Core, it converts around 10,100
lines with about 34 manual fixes needed.

For other layers it would need updating for override names and exclusions for
functions/variable names with "append", "prepend" or "remove" in them.

(From OE-Core rev: e8fc67f6f6baf54ccdf548b947c35dea926842c3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-30 00:11:51 +01:00
Anders Wallin
9e0f210179 scripts/contrib/image-manifest: add new script
image-manifest: script to generate product/image specific BOM

The image-manifest script generates image specific reports based on
an image manifest file. Currently there is data generated by buildhistory,
pkgdata, and license manifest but this data is poorly formated and spread
across multiple text files. This script can generate a single JSON output
file that is machine readable by other tools.

The manifest-info collects package information and stores the information
in a tarball. manifest-info can be configured using a json configuration
file. The default configuration including all possible options can be
dumped using the dump-config subcommand.

image-manifest takes an image manifest file as input to get the runtime
dependencies. As an option image-manifest can also use the build dependency
file, pn-buildlist, to get the build dependencies excluding native
packages.

This script extends the oe-image-manifest script [0] done by Paul Eggleton

[0]
https://github.com/intel/clear-linux-dissector-web/blob/master/layerindex/static/files/oe-image-manifest

------------------------------------------------------
usage: image-manifest [-h] [-d] [-q] <subcommand> ...

Image manifest utility

options:
  -h, --help     show this help message and exit
  -d, --debug    Enable debug output
  -q, --quiet    Print only errors

subcommands:
  recipe-info    Get recipe info
  list-depends   List dependencies
  list-recipes   List recipes producing packages within an image
  list-packages  List packages within an image
  list-layers    List included layers
  dump-config    Dump default config
  manifest-info  Export recipe info for a manifest
Use image-manifest <subcommand> --help to get help on a specific command

Co-developed-by: Paul Eggleton <bluelightning@bluelightning.org>
(From OE-Core rev: ad8fec9ce1704866df925bda18a240d6889b1ed5)

Signed-off-by: Anders Wallin <anders.wallin@windriver.com>
Signed-off-by: Saul Wold <saul.wold@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:37:25 +01:00
Khem Raj
a7e1bbaf6d documentation-audit.sh: Fix typo in specifying LICENSE_FLAGS_WHITELIST
(From OE-Core rev: 410a45639d84a3d69a65133593da32062196dd59)

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-26 11:34:33 +00:00
Richard Purdie
e9aac299c7 scripts/contrib/list-packageconfig-flags: Upate for tinfoil API changes
Update after tinfoil API changes in bitake for REQUIRED_VERSION.

(From OE-Core rev: 1645c9b093bebf7ced67cbee0009d81d1a215966)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-15 14:49:09 +00:00
Paul Eggleton
b87f0746b0 graph-tool: add filter subcommand
Add a filter subcommand to filter a task-depends.dot graph produced by
bitbake -g down to just a subset of targets/tasks.

(From OE-Core rev: a14b274b56676ff0ba55a4048169ad60c9514994)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-16 23:39:28 +01:00
Paul Eggleton
8f7c45c183 graph-tool: switch to argparse
argparse makes this a lot easier to extend.

(From OE-Core rev: c751ef8fdc111d1c967029cea7a3ed0f88ce851b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-16 23:39:28 +01:00
Peter Kjellerstedt
325b820dae buildstats-plot.sh: Show spaces and underscores correctly in titles/keys
Spaces were previously converted to dashes, and underscores caused the
next character in the title/key to be printed using subscript due to the
enhanced string support in gnuplot.

(From OE-Core rev: 1719f7062988889b7e24b871dc2f0e50c7bff07e)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-29 12:05:31 +01:00
Luis Martins
719c14c984 buildstats*.sh: fix spacing indentation in the files
Fix indentation in buildstats.sh and buildstats-plot.sh
to convert every line to 4 spaces indentation.

(From OE-Core rev: 3971572042ae9bc8226eb92892bb71010287e0a8)

Signed-off-by: Luis Martins <luis.martins@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24 16:39:40 +00:00
Luis Martins
41b145b2b1 buildstats*.sh: add recipe parameter to scripts
Add option to filter the stats results for a single recipe, allowing
to easilly visuality the task resource consumption per recipe.
This is specially useful when debugging the performance of
the overall bitbake build system.

(From OE-Core rev: b4bf898d53af60f5d69b50ec5cdd7f9b3870f2dd)

Signed-off-by: Luis Martins <luis.martins@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24 16:39:40 +00:00
Luis Martins
3bd3e23304 buildstats*.sh: add accumulate parameter to scripts
Add option to accumulate stats values per recipe, allowing
to sum related values such as memory (main process + childs).
This is specially useful when debugging the performance of
the overall bitbake build system.

(From OE-Core rev: e6f1aea6260343cb5194f7f8ab70213b705ab441)

Signed-off-by: Luis Martins <luis.martins@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24 16:39:40 +00:00
Luis Martins
fe1e4a4771 buildstats-plot.sh: filter by task
Extend buildstats-plot.sh script to also accept the name
of the tasks as parameter.
This value will be passed directly to buildstats.sh is
already provides this option.

(From OE-Core rev: 495b6d3d85d2d14d54e324d8da43311a23fdfca6)

Signed-off-by: Luis Martins <luis.martins@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24 16:39:40 +00:00
Luis Martins
5ea725cd5a buildstats-plot.sh: fix invoking buildstats.sh
Fix the internal call to buildstats.sh by also providing
the buildstats folder location in the system, which might
differ from the default location.

(From OE-Core rev: 9f4ee2ee8712528ac51ef4ecd17ccde737e8b21d)

Signed-off-by: Luis Martins <luis.martins@criticaltechworks.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-24 16:39:40 +00:00
Paul Eggleton
9a08720b3a list-packageconfig-flags: print PN instead of P
P (which is ${PN}-${PV}) isn't terribly useful in this context - we
don't really care what the version is, but we do want to know what the
recipe is so we can find it or set PACKAGECONFIG_pn-<PN> in our
configuration, so display ${PN} instead.

(From OE-Core rev: 7facaacd145c2924414ad63ddce07602a72d02c1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Paul Eggleton
12d64d6d1a scripts/contrib/ddimage: be explicit whether device doesn't exist or isn't writeable
Make the error messages a little more friendly.

(From OE-Core rev: 36f308b0134c69b439152c2473a274d96b0fee89)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Paul Eggleton
d3a9f43305 scripts/contrib/ddimage: replace blacklist with mount check
The blacklist, whilst previously useful for safety, is now becoming
obsolete - on my current system, the main storage is at /dev/nvme* and
if I plug in a USB stick it shows up as /dev/sdb which was previously
blacklisted. To make this more flexible, remove the blacklist and
instead check if the specified device is mounted, has a partition
that is mounted, or is otherwise in use according to the kernel, and
show an appropriate error and quit if so.

To make this robust, also ensure we handle where the specified device is
a symlink to another device.

(From OE-Core rev: 49043de1a7716ad612fb92a2e8a52e43d253c800)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Paul Eggleton
8ff741d075 scripts/contrib/ddimage: fix typo
UNKOWN -> UNKNOWN

(From OE-Core rev: adf632c959e7f6595b697726fe64ddda40b9f249)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Richard Purdie
ffae400179 meta/lib+scripts: Convert to SPDX license headers
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.

The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).

More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.

The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.

(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09 16:31:55 +01:00
Richard Purdie
f9fcd07e47 scripts/contrib/build-perf-test.sh: Remove it
This is the older script which the code in lib/oeqa/buildperf replaces so
delete the obsolete script to avoid confusion.

(From OE-Core rev: 9908b7955e75e10d6a1384e4c08417ef534464af)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-20 22:24:18 +00:00
Richard Purdie
997615921e scripts/contrib/build-perf-test-wrapper.sh: Improve interaction with autobuilder automation
This tweaks the script to:

* Ensure directories exist and can be written to
* Allow the downloads directory to be specified
* Error early if the phantomjs or optipng dependencies are not installed
* Allow the location of the globalres.log file to be specified

This means that the main build directory can be destroyed and any state
from the script is stored elsewhere allowing it to be triggered
automatically from the autobuilder infrastructure.

(From OE-Core rev: 1de5fbd484e6a747ae6419ccc89d8c2911b9706b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-20 11:54:36 +00:00
Ross Burton
71460428e4 patchreview: Various fixes/improvements
Add various fixes and improvements including the ability to export
patch statsitics as json data.

(From OE-Core rev: aa4a4b3ca799948047337e006ee9bf482be7b409)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-05 12:37:02 +00:00
Ross Burton
befef28652 scripts/patchreview: tighten patch detection
Only look in recipes-* folders for patches, to stop finding patches in
e.g. meta/lib/oeqa/manual.

(From OE-Core rev: ed912e10c61ea9d45ca648e03eebafeebcf7e9b0)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-16 20:35:43 +01:00
Chen Qi
869e501544 logging: use warning instead warn
The warn method is deprecated. We should use the documented warning instead.

Quoting from the python's official doc:
"""
Note: There is an obsolete method warn which is functionally identical to warning.
      As warn is deprecated, please do not use it - use warning instead.
"""

(From OE-Core rev: f467fd277eb77336097cfc0f5f329bdc8d0f70cb)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-26 13:16:40 +01:00
Richard Purdie
4c67ffef2e scripts/contrib/mkefidisk.sh: Remove
This has been printing a deprecation warning for a while, now remove it
in favour of wic's efi functionality.

(From OE-Core rev: 1f279cf1ad2f0a20495780b210a987416650f40f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24 11:52:27 +01:00
Ross Burton
076d8fcdd2 patchreview: don't disable malformed SoB check
We cleaned up the metadata so this can be enabled again.

(From OE-Core rev: 9611485bba03ef77ff31121e3b1da7cd57990c3e)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-15 10:56:51 +01:00
Richard Purdie
c2e722f6a6 scripts/contrib/build-perf-test: Fix fetchall reference to use runall
(From OE-Core rev: 028ab7eb11f78c02900389197eace81824e6553f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-20 22:25:10 +00:00
Alejandro Hernandez
b6777878ff python3: Restructure python3 packaging and replace it with autopackaging
See previous commit (python2 version) for more info, since mostly
everything applies here as well.

Old manifest file had several issues:
 - Its unorganized and hard to read and understand it for an average
   human being.
 - When a new package needs to be added, the user actually has to modify
   the script that creates the manifest, then call the script to create
   a new manifest, and then submit a patch for both the script and the
   manifest, so its a little convoluted.
 - Git complains every single time a patch is submitted to the manifest,
   since it violates some of its guidelines.
 - It changes or may change with every release of python, its impossible
   to know if the required files for a certain package have changed
   (it could have more or less dependencies), the only way of doing so
   would be to install and test them all one by one on separate individual
   images, and even then we wouldnt know if they require less dependencies,
   we would just know if an extra dependency is required since it would
   complain, lets face it, this isnt feasible.
 - The same thing happens for new packages, if someone wants to add a new
   package, its dependencies need to be checked manually one by one.

Features/Fixes:
 - A new manifest format is used (JSON), easy to read and understand.
   This file is parsed by the python recipe and python packages
   read from here are passed directly to bitbake during parsing time.
 - It provides an automatic manifest creation task (explained on previous
   commit), which automagically checks for every package dependencies and
   adds them to the new manifest, hence we will have on each package
   exactly what that package needs to be run, providing finer granularity.
  - Dependencies are also checked automagically for new packages
   (explained on previous commit).

This patch has the same features as the python2 version but it differs
in the following ways:
 - Python3 handles precompiled bytecode files  (*.pyc) differently.
   for this reason and since we are cross compiling, wildcards couldnt be
   avoided on python3 (See PEP #3147 [1]).
   Both the manifest and the manifest creation script handle this
   differently, the manifest for python3 has an extra field for cached
   files, which is how it lets the user install the cached files or not
   via : INCLUDE_PYCS = "1" on their local.conf.
 - Shared libraries nomenclature also changed on python3, so again, we
   use wildcards to deal with this issue ( See PEP #3149 [2]):
 - Fixes python3 manifest, python3-core should be base and everything
   should depend on it, hence several packages were deleted:
   python3-enum, re, gdbm, subprocess, signal, readline.
 - When building python3-native it adds as symlink to it called
   nativepython3, which is then isued by the create_manifest task.
 - Fixes [YOCTO #11513] while were at it.

References:
[1] https://www.python.org/dev/peps/pep-3147/
[2] https://www.python.org/dev/peps/pep-3149/

(From OE-Core rev: 54ac820b8a639950ccb534dcd9d6eaf8b2b736e0)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-20 22:31:56 +00:00
Alejandro Hernandez
8d94b9db22 python: Restructure python packaging and replace it with autopackaging
The reason we have a manifest file for python is that our goal is to
keep python-core as small as posible and add other python packages only
when the user needs them, hence why we split upstream python into several
packages.

Although our manifest file has several issues:
 - Its unorganized and hard to read and understand it for an average
   human being.
 - When a new package needs to be added, the user actually has to modify
   the script that creates the manifest, then call the script to create
   a new manifest, and then submit a patch for both the script and the
   manifest, so its a little convoluted.
 - Git complains every single time a patch is submitted to the manifest,
   since it violates some of its guidelines.
 - It changes or may change with every release of python, its impossible
   to know if the required files for a certain package have changed
   (it could have more or less dependencies), the only way of doing so
   would be to install and test them all one by one on separate individual
   images, and even then we wouldnt know if they require less dependencies,
   we would just know if an extra dependency is required since it would
   complain, lets face it, this isnt feasible.
 - The same thing happens for new packages, if someone wants to add a
   new package, its dependencies need to be checked manually one by one.

This patch fixes those issues, while adding some additional features.

Features/Fixes:

 - A new manifest format is used (JSON), easy to read and understand.
   This file is parsed by the python recipe and python packages read
   from here are passed directly to bitbake during parsing time.
 - It provides an automatic manifest creation task (explained below),
   which automagically checks for every package dependencies and adds
   them to the new manifest, hence we will have on each package exactly
   what that package needs to be run, providing finer granularity.
 - Dependencies are also checked automagically for new packages (explained below).
 - Fixes the manifest in the following ways:
   * python-core should be base and all packages should depend on it,
     fixes lang, string, codecs, etc.
   * Fixes packages with repeated files (e.g. bssdb and db, or
     netclient and mime, and many others).
 - Sitecustomize was fixed since encoding was deprecated.
 - The JSON manifest file invalidates bitbake's cache, so if it changes
   the python package will be rebuilt.
 - It creates a solution for users that want precompiled bytecode files
   (*.pyc) INCLUDE_PYCS = "1" can be set by the user on their local.conf to
   include such files, some argument they get faster boot time, even when the
   files would be created on their first run?, but they also sometimes give a
   magic number error and take up space, so we leave it to the user to
   decide if they want them or not.
 - Fixes python-core dependencies, e.g.
   When python is run on an image, it TRIES to import everything it needs,
   but it doesnt necessarily fails when it doesnt find something, so even if
   we didnt know, we had errors like (trimmed on purpose):
   # trying /usr/lib/python2.7/_locale.so
   # trying /usr/lib/python2.7/lib-dynload/_locale.so
   # trying /usr/lib/python2.7/_sysconfigdata.so

   while it didnt complain about _locale it should have imported it,
   after creating a new manifest with the automated script we get:

   # trying /usr/lib/python2.7/lib-dynload/_locale.so
   dlopen("/usr/lib/python2.7/lib-dynload/_locale.so", 2);
   import _locale # dynamically loaded from /usr/lib/python2.7/lib-dynload/_locale.so

How to use (after a new release of python, or maybe before every OE
release):
 - A new task called create_manifest was added to the python package,
   which may be invoked via:

 $ bitbake python -c create_manifest

 This task runs a script on native python on our HOST system, and since
 the python and python-native packages come from the same source, we can
 use it to know the dependencies of each module as if we were doing it
 on an image, this script is called create_manifest.py and in a very
 simplistic way it does the following:

 1. Reads the JSON manifest file and creates a dictionary data structure
    with all of our python packages, their FILES, RDEPENDS and SUMMARY.
 2. Loops through all of them and runs every module listed on them
    asynchronously, determining every dependency that they have.
 3. These module dependencies are then handled, to be able to know which
    packages contain those files and which should RDEPEND on one another.
 4. The data structure that comes out of this, is then used to create a
    new manifest file which is automatically copied onto the user's python
    directory replacing the old one.

 Create_manifest script features:
 - Handles modules which dont exist anymore (new release for example).
 - Handles modules that are builtin.
 - Deals with modules which were not compiled (e.g. bsddb or ossaudiodev)
 - Deals with packages which include folders.
 - Deals with packages which include FILES with a wildcard.
 - The manifest can be constructed on a multilib environment as well.
 - This method works for both python modules and shared libraries used
   by python.

How to add a new package:
 - If a user wants to add a new package all that has to be done is
   modify the python2-manifest.json file, and add the required file(s)
   to the FILES list, the script should handle all the rest.
   Real example:
   We want to add a web browser package, including the file webbrowser.py
   which at the moment is on python-misc.
   "webbrowser": {
       "files": ["${libdir}/python2.7/lib-dynload/webbrowser.py"],
       "rdepends": [],
       "summary": "Python Web Browser support"}

 Run bitbake python -c create_manifest and the resulting manifest
 should  be completed after a few seconds, showing something like:
   "webbrowser": {
      "files": ["${libdir}/python2.7/webbrowser.py"],
      "rdepends": ["core","fcntl","io","pickle","shell","subprocess"],
      "summary": "Python Web Browser support"}

Known errors/issues:
 - Some special packages are handled differently: core, misc,
   modules,dev, staticdev.
   All these should be handled manually, because they either include
   binaries, static libraries, include files, etc. (something that we
   cant import).
   Specifically static libraries are not not supported by this method
   and have to be handled by the user.
 - The change should be transparent to the user, other than the fact
   that now we CANT build python-foo (it was pretty dumb anyway, since
   what building python-foo actually did was building the whole python
   package anyway), but doing IMAGE_INSTALL_append = " python-foo"
   would create an image with the requested package with no issues.

[YOCTO #11510] [YOCTO #11694] [YOCTO #11695]

(From OE-Core rev: 6959e2e4dba5bbfa6ffd49c44e738cc1c38bc280)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-20 22:31:56 +00:00
Yi Zhao
cf75fd5ae0 contrib/yocto-bsp-kernel-update.sh: remove this script
This script was unused since yocto-bsp tool had been removed.

(From meta-yocto rev: 594ced59dc80c11d573ae4a16e1aff910a8d9300)

Signed-off-by: Yi Zhao <yi.zhao@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-18 13:05:56 +00:00
Tim Orling
8aa71adf49 python-*-manifest/generators: add runpy; python3-plistlib
* runpy allows running modules/scripts with 'python -m foo'
* python3-setuptools RDEPENDS on plistlib (present in python2)
* pip3 RDEPENDS on _markupbase (add to python3-core)

(From OE-Core rev: d95f1005c35bd9c7e22c40c7c17d264fe9435c6b)

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-13 14:00:51 +00:00
Tim Orling
69a910d2e7 python-*-manifest/generators: fix long line lengths
The generators create python-*-manifest.inc files with
lines over 2500 characters long which breaks sending
patches via git send-email (because of smtp limitation).

This patchset formats all the long lines into multiple lines.

(From OE-Core rev: 3a1900a5691466a04b24907067f43117b318ca7e)

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-13 14:00:51 +00:00
Amanda Brindle
4af421a469 scripts/contrib/bbvars.py: Remove dead code
Removed the -m option since this script now searches through all
recipes in the configuration. Also removed dead code, which includes
the functions recipe_bbvars() and collect_bbvars().

(From OE-Core rev: dac6515fcd23ea9cde5308c1d08a7a928efbb4d6)

Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-02 11:25:32 +00:00
Markus Lehtonen
90ba1869ed oe-build-perf-report-email.py: add images as MIME objects
Add images as separate MIME objects instead of directly embedding images
in the html (as base64 encoded pngs). This makes the emails better
suited for certain email servers/clients.

(From OE-Core rev: 6b61126c386a0a7334cdf475d349b830c436ed82)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-08 22:24:04 +00:00
Amanda Brindle
70b3fffe5a scripts/contrib/bbvars.py: Rewrite to use tinfoil
Use tinfoil to collect all variable names globally and in each recipe.

No longer show the count of variables if they are undocumented.

Fixes [YOCTO #2086]

(From OE-Core rev: b947de583964020b488cb1c382877624e96d4319)

Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-08 22:24:04 +00:00
Joshua Lock
eac8a5cf42 oe-build-perf-report-email.py: add cc and bcc options
Enable carbon copy and blind carbon copy recipients for the performance
report emails.

(From OE-Core rev: df5ae8143ff1764b6ed5973ed3d6f1a83ecf45ee)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-21 23:16:55 +01:00
Mark Asselstine
c760d99466 python: don't include -tests with modules
Although 'test' is listed in the python module list
(https://docs.python.org/3/py-modindex.html) it is meant only to be
used 'internally' by folks developing python itself. Per the
documentation:

  Note The test package is meant for internal use by Python only. It
  is documented for the benefit of the core developers of Python. Any
  use of this package outside of Python’s standard library is
  discouraged as code mentioned here can change or be removed without
  notice between releases of Python.

Per the above it is best not to include this module to discourage
folks who might not head the above warnings. Additionally this module
is one of the largest py modules going, by dropping this unneeded
module from the 'modules' package we can reduce overall image size,
something which is important for many embedded deployments.

The generator scripts as well as the manifests have thus been modified
accordingly, providing a generic mechanism to exclude modules from the
'all modules' package.

(From OE-Core rev: a5bb13a5d7d7a668ca61da6b17884e3b05b95355)

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-03 11:14:13 +01:00
Saul Wold
7fd1491d2f mkefidsk: fix bash/dash shell quoting problem
mkefidsk currently writes a startup.nsh with embedded control characters.
This happens because \b etc are control sequences to the shell echo
command when using dash. The resulting startup.nsh causes the bootup
to fail, and the user is dropped into the EFI shell to manually run
startup.nsh.

Patch originally provided by Troy D. Hanson <troy.hanson@jhuapl.edu>

[YOCTO #9665]

(From OE-Core rev: 008d6cb5bb4969f53a228893c502be8c9420ecb0)

Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-06 14:38:13 +01:00
Ross Burton
027bdb0560 scripts/contrib/patchreview: add new script
This script analyses the patches we apply and can sanity check or output
statistics.

(From OE-Core rev: de7914954571ea8e717f56b6d6df13157b0973bc)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-29 13:01:38 +01:00
Leonardo Sandoval
f9b476ebcd scripts/contrib/patchtest: run patchtest on local branch
The script run patchtest on local branch commits, printing results into
stdout. This script is useful to test patches before sending to the
mailing list.

Examples:

$ git checkout master-next-1.9
Branch master-next-1.9 set up to track remote branch master-next-1.9 from origin.
Switched to a new branch 'master-next-1.9'

  $ ~/scripts/contrib/patchtest.sh
  166e70e: Robert Yang: Thu Apr 2 12:01:37 2015 +0100: patch: fix CVE-2015-1196: FAIL
   Issue             Missing or incorrectly formatted CVE tag in commit message [test_cve_presence_in_commit_message]
    Suggested fix    Include a "CVE-xxxx-xxxx" tag in the commit message
   Issue             Missing or incorrectly formatted CVE tag in included patch file [test_cve_tag_format]
    Suggested fix    Correct or include the CVE tag on cve patch with format: "CVE: CVE-YYYY-XXXX"

  eaa4536: Robert Yang: Thu Apr 2 12:01:37 2015 +0100: wget: 1.16.1 -> 1.16.2: OK

  3c29ce3: Robert Yang: Thu Apr 2 12:01:38 2015 +0100: git: 2.3.0 -> 2.3.1: OK

  85491f6: Khem Raj: Thu Apr 2 12:01:38 2015 +0100: gdb: Upgrade 7.8.1 -> 7.9: OK

  f701142: Robert Yang: Thu Apr 2 12:01:38 2015 +0100: binutils: upgrade to 2.25: OK

  385d0b1: Khem Raj: Thu Apr 2 12:01:39 2015 +0100: binutils: Fix ICE in gold: OK

[YOCTO #10720]

(From OE-Core rev: 4ba1334ab7c9b32502a4f8b3e656fb4f8242ccdd)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-23 11:44:12 +01:00
Martin Jansa
d973074bf6 python-3.5: Move bz2.py, lzma.py and _compression.py from python3-misc to python3-compression
* the /usr/lib/python3.5/_compression.py file is possibly incorrectly included
  in python3-misc. This runtime dependency is needed in order to use e.g. gzip.py in runtime:

  >>> import tarfile, zlib, gzip
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib/python3.5/gzip.py", line 12, in <module>
      import _compression
  ImportError: No module named '_compression'

* at least python3-tests and lzma and bz2 still in python3-misc are using this as well:
  $ grep -R import.*_compression tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/
  tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-compression/usr/lib/python3.5/gzip.py:import _compression
  tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-misc/usr/lib/python3.5/lzma.py:import _compression
  tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-misc/usr/lib/python3.5/bz2.py:import _compression
  tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-tests/usr/lib/python3.5/test/test_bz2.py:import _compression
  tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-tests/usr/lib/python3.5/test/test_lzma.py:import _compression

  and python3-tests are using it as well, so add new runtime dependency
  on python3-compression

(From OE-Core rev: 987363c3c720b3764f4d64976d7455f6b0bae99c)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14 14:53:58 +01:00
Ross Burton
f1333958dd python3: add python3-modules-native to RPROVIDES for python3-native
Also clean up the logic in the script to be more Pythonic.

(From OE-Core rev: fae66dd3633aa8a6aa633fcfd7c4b9a728dee7a4)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-06 19:52:26 +01:00
Ross Burton
b39d1c6842 python: add python-modules-native to RPROVIDES for python-native
Also clean up the logic in the script to be more Pythonic.

(From OE-Core rev: e5ac43e1b549e637f1820a03dd0a633fbecd395c)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-06 19:52:26 +01:00
Li Zhou
1c0ce38ba0 python3: Move config-*/Makefile from misc package to dev package
Move config-*/Makefile in libdir from misc package to dev package for
python3, because it is only needed in development process.

(From OE-Core rev: d715dc422ce1723c8d05af7ad4183eeeb36bc2ec)

Signed-off-by: Li Zhou <li.zhou@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-05-29 15:15:20 +01:00
Markus Lehtonen
4fdb6aa28d build-perf-test-wrapper.sh: support uploading test reports
Implement new '-R' command line option for specifying an rsync
destination where to upload test reports.

[YOCTO #5049]

(From OE-Core rev: ebc37ede5b247347483128f091b505fe33356591)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-05-18 14:01:47 +01:00
Markus Lehtonen
11747280b5 oe-build-perf-report-email.py: use pwd for getting user name
Use pwd data instead of os.getlogin() to get the username for fallback
email address. os.getlogin() basically returns the name of the user
logged in on the controlling terminal of the process and raises an
exception on some systems if the process does not have a controlling
terminal - when executed from a cron script, for example.

(From OE-Core rev: 66b7f495bb000c043ae24176dcdec0cb087eeecf)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:10 +01:00