Commit Graph

264 Commits

Author SHA1 Message Date
Ross Burton
befef28652 scripts/patchreview: tighten patch detection
Only look in recipes-* folders for patches, to stop finding patches in
e.g. meta/lib/oeqa/manual.

(From OE-Core rev: ed912e10c61ea9d45ca648e03eebafeebcf7e9b0)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-16 20:35:43 +01:00
Chen Qi
869e501544 logging: use warning instead warn
The warn method is deprecated. We should use the documented warning instead.

Quoting from the python's official doc:
"""
Note: There is an obsolete method warn which is functionally identical to warning.
      As warn is deprecated, please do not use it - use warning instead.
"""

(From OE-Core rev: f467fd277eb77336097cfc0f5f329bdc8d0f70cb)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-26 13:16:40 +01:00
Richard Purdie
4c67ffef2e scripts/contrib/mkefidisk.sh: Remove
This has been printing a deprecation warning for a while, now remove it
in favour of wic's efi functionality.

(From OE-Core rev: 1f279cf1ad2f0a20495780b210a987416650f40f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-24 11:52:27 +01:00
Ross Burton
076d8fcdd2 patchreview: don't disable malformed SoB check
We cleaned up the metadata so this can be enabled again.

(From OE-Core rev: 9611485bba03ef77ff31121e3b1da7cd57990c3e)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-15 10:56:51 +01:00
Richard Purdie
c2e722f6a6 scripts/contrib/build-perf-test: Fix fetchall reference to use runall
(From OE-Core rev: 028ab7eb11f78c02900389197eace81824e6553f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-20 22:25:10 +00:00
Alejandro Hernandez
b6777878ff python3: Restructure python3 packaging and replace it with autopackaging
See previous commit (python2 version) for more info, since mostly
everything applies here as well.

Old manifest file had several issues:
 - Its unorganized and hard to read and understand it for an average
   human being.
 - When a new package needs to be added, the user actually has to modify
   the script that creates the manifest, then call the script to create
   a new manifest, and then submit a patch for both the script and the
   manifest, so its a little convoluted.
 - Git complains every single time a patch is submitted to the manifest,
   since it violates some of its guidelines.
 - It changes or may change with every release of python, its impossible
   to know if the required files for a certain package have changed
   (it could have more or less dependencies), the only way of doing so
   would be to install and test them all one by one on separate individual
   images, and even then we wouldnt know if they require less dependencies,
   we would just know if an extra dependency is required since it would
   complain, lets face it, this isnt feasible.
 - The same thing happens for new packages, if someone wants to add a new
   package, its dependencies need to be checked manually one by one.

Features/Fixes:
 - A new manifest format is used (JSON), easy to read and understand.
   This file is parsed by the python recipe and python packages
   read from here are passed directly to bitbake during parsing time.
 - It provides an automatic manifest creation task (explained on previous
   commit), which automagically checks for every package dependencies and
   adds them to the new manifest, hence we will have on each package
   exactly what that package needs to be run, providing finer granularity.
  - Dependencies are also checked automagically for new packages
   (explained on previous commit).

This patch has the same features as the python2 version but it differs
in the following ways:
 - Python3 handles precompiled bytecode files  (*.pyc) differently.
   for this reason and since we are cross compiling, wildcards couldnt be
   avoided on python3 (See PEP #3147 [1]).
   Both the manifest and the manifest creation script handle this
   differently, the manifest for python3 has an extra field for cached
   files, which is how it lets the user install the cached files or not
   via : INCLUDE_PYCS = "1" on their local.conf.
 - Shared libraries nomenclature also changed on python3, so again, we
   use wildcards to deal with this issue ( See PEP #3149 [2]):
 - Fixes python3 manifest, python3-core should be base and everything
   should depend on it, hence several packages were deleted:
   python3-enum, re, gdbm, subprocess, signal, readline.
 - When building python3-native it adds as symlink to it called
   nativepython3, which is then isued by the create_manifest task.
 - Fixes [YOCTO #11513] while were at it.

References:
[1] https://www.python.org/dev/peps/pep-3147/
[2] https://www.python.org/dev/peps/pep-3149/

(From OE-Core rev: 54ac820b8a639950ccb534dcd9d6eaf8b2b736e0)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-20 22:31:56 +00:00
Alejandro Hernandez
8d94b9db22 python: Restructure python packaging and replace it with autopackaging
The reason we have a manifest file for python is that our goal is to
keep python-core as small as posible and add other python packages only
when the user needs them, hence why we split upstream python into several
packages.

Although our manifest file has several issues:
 - Its unorganized and hard to read and understand it for an average
   human being.
 - When a new package needs to be added, the user actually has to modify
   the script that creates the manifest, then call the script to create
   a new manifest, and then submit a patch for both the script and the
   manifest, so its a little convoluted.
 - Git complains every single time a patch is submitted to the manifest,
   since it violates some of its guidelines.
 - It changes or may change with every release of python, its impossible
   to know if the required files for a certain package have changed
   (it could have more or less dependencies), the only way of doing so
   would be to install and test them all one by one on separate individual
   images, and even then we wouldnt know if they require less dependencies,
   we would just know if an extra dependency is required since it would
   complain, lets face it, this isnt feasible.
 - The same thing happens for new packages, if someone wants to add a
   new package, its dependencies need to be checked manually one by one.

This patch fixes those issues, while adding some additional features.

Features/Fixes:

 - A new manifest format is used (JSON), easy to read and understand.
   This file is parsed by the python recipe and python packages read
   from here are passed directly to bitbake during parsing time.
 - It provides an automatic manifest creation task (explained below),
   which automagically checks for every package dependencies and adds
   them to the new manifest, hence we will have on each package exactly
   what that package needs to be run, providing finer granularity.
 - Dependencies are also checked automagically for new packages (explained below).
 - Fixes the manifest in the following ways:
   * python-core should be base and all packages should depend on it,
     fixes lang, string, codecs, etc.
   * Fixes packages with repeated files (e.g. bssdb and db, or
     netclient and mime, and many others).
 - Sitecustomize was fixed since encoding was deprecated.
 - The JSON manifest file invalidates bitbake's cache, so if it changes
   the python package will be rebuilt.
 - It creates a solution for users that want precompiled bytecode files
   (*.pyc) INCLUDE_PYCS = "1" can be set by the user on their local.conf to
   include such files, some argument they get faster boot time, even when the
   files would be created on their first run?, but they also sometimes give a
   magic number error and take up space, so we leave it to the user to
   decide if they want them or not.
 - Fixes python-core dependencies, e.g.
   When python is run on an image, it TRIES to import everything it needs,
   but it doesnt necessarily fails when it doesnt find something, so even if
   we didnt know, we had errors like (trimmed on purpose):
   # trying /usr/lib/python2.7/_locale.so
   # trying /usr/lib/python2.7/lib-dynload/_locale.so
   # trying /usr/lib/python2.7/_sysconfigdata.so

   while it didnt complain about _locale it should have imported it,
   after creating a new manifest with the automated script we get:

   # trying /usr/lib/python2.7/lib-dynload/_locale.so
   dlopen("/usr/lib/python2.7/lib-dynload/_locale.so", 2);
   import _locale # dynamically loaded from /usr/lib/python2.7/lib-dynload/_locale.so

How to use (after a new release of python, or maybe before every OE
release):
 - A new task called create_manifest was added to the python package,
   which may be invoked via:

 $ bitbake python -c create_manifest

 This task runs a script on native python on our HOST system, and since
 the python and python-native packages come from the same source, we can
 use it to know the dependencies of each module as if we were doing it
 on an image, this script is called create_manifest.py and in a very
 simplistic way it does the following:

 1. Reads the JSON manifest file and creates a dictionary data structure
    with all of our python packages, their FILES, RDEPENDS and SUMMARY.
 2. Loops through all of them and runs every module listed on them
    asynchronously, determining every dependency that they have.
 3. These module dependencies are then handled, to be able to know which
    packages contain those files and which should RDEPEND on one another.
 4. The data structure that comes out of this, is then used to create a
    new manifest file which is automatically copied onto the user's python
    directory replacing the old one.

 Create_manifest script features:
 - Handles modules which dont exist anymore (new release for example).
 - Handles modules that are builtin.
 - Deals with modules which were not compiled (e.g. bsddb or ossaudiodev)
 - Deals with packages which include folders.
 - Deals with packages which include FILES with a wildcard.
 - The manifest can be constructed on a multilib environment as well.
 - This method works for both python modules and shared libraries used
   by python.

How to add a new package:
 - If a user wants to add a new package all that has to be done is
   modify the python2-manifest.json file, and add the required file(s)
   to the FILES list, the script should handle all the rest.
   Real example:
   We want to add a web browser package, including the file webbrowser.py
   which at the moment is on python-misc.
   "webbrowser": {
       "files": ["${libdir}/python2.7/lib-dynload/webbrowser.py"],
       "rdepends": [],
       "summary": "Python Web Browser support"}

 Run bitbake python -c create_manifest and the resulting manifest
 should  be completed after a few seconds, showing something like:
   "webbrowser": {
      "files": ["${libdir}/python2.7/webbrowser.py"],
      "rdepends": ["core","fcntl","io","pickle","shell","subprocess"],
      "summary": "Python Web Browser support"}

Known errors/issues:
 - Some special packages are handled differently: core, misc,
   modules,dev, staticdev.
   All these should be handled manually, because they either include
   binaries, static libraries, include files, etc. (something that we
   cant import).
   Specifically static libraries are not not supported by this method
   and have to be handled by the user.
 - The change should be transparent to the user, other than the fact
   that now we CANT build python-foo (it was pretty dumb anyway, since
   what building python-foo actually did was building the whole python
   package anyway), but doing IMAGE_INSTALL_append = " python-foo"
   would create an image with the requested package with no issues.

[YOCTO #11510] [YOCTO #11694] [YOCTO #11695]

(From OE-Core rev: 6959e2e4dba5bbfa6ffd49c44e738cc1c38bc280)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-20 22:31:56 +00:00
Yi Zhao
cf75fd5ae0 contrib/yocto-bsp-kernel-update.sh: remove this script
This script was unused since yocto-bsp tool had been removed.

(From meta-yocto rev: 594ced59dc80c11d573ae4a16e1aff910a8d9300)

Signed-off-by: Yi Zhao <yi.zhao@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-18 13:05:56 +00:00
Tim Orling
8aa71adf49 python-*-manifest/generators: add runpy; python3-plistlib
* runpy allows running modules/scripts with 'python -m foo'
* python3-setuptools RDEPENDS on plistlib (present in python2)
* pip3 RDEPENDS on _markupbase (add to python3-core)

(From OE-Core rev: d95f1005c35bd9c7e22c40c7c17d264fe9435c6b)

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-13 14:00:51 +00:00
Tim Orling
69a910d2e7 python-*-manifest/generators: fix long line lengths
The generators create python-*-manifest.inc files with
lines over 2500 characters long which breaks sending
patches via git send-email (because of smtp limitation).

This patchset formats all the long lines into multiple lines.

(From OE-Core rev: 3a1900a5691466a04b24907067f43117b318ca7e)

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-13 14:00:51 +00:00
Amanda Brindle
4af421a469 scripts/contrib/bbvars.py: Remove dead code
Removed the -m option since this script now searches through all
recipes in the configuration. Also removed dead code, which includes
the functions recipe_bbvars() and collect_bbvars().

(From OE-Core rev: dac6515fcd23ea9cde5308c1d08a7a928efbb4d6)

Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-02 11:25:32 +00:00
Markus Lehtonen
90ba1869ed oe-build-perf-report-email.py: add images as MIME objects
Add images as separate MIME objects instead of directly embedding images
in the html (as base64 encoded pngs). This makes the emails better
suited for certain email servers/clients.

(From OE-Core rev: 6b61126c386a0a7334cdf475d349b830c436ed82)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-08 22:24:04 +00:00
Amanda Brindle
70b3fffe5a scripts/contrib/bbvars.py: Rewrite to use tinfoil
Use tinfoil to collect all variable names globally and in each recipe.

No longer show the count of variables if they are undocumented.

Fixes [YOCTO #2086]

(From OE-Core rev: b947de583964020b488cb1c382877624e96d4319)

Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-08 22:24:04 +00:00
Joshua Lock
eac8a5cf42 oe-build-perf-report-email.py: add cc and bcc options
Enable carbon copy and blind carbon copy recipients for the performance
report emails.

(From OE-Core rev: df5ae8143ff1764b6ed5973ed3d6f1a83ecf45ee)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-21 23:16:55 +01:00
Mark Asselstine
c760d99466 python: don't include -tests with modules
Although 'test' is listed in the python module list
(https://docs.python.org/3/py-modindex.html) it is meant only to be
used 'internally' by folks developing python itself. Per the
documentation:

  Note The test package is meant for internal use by Python only. It
  is documented for the benefit of the core developers of Python. Any
  use of this package outside of Python’s standard library is
  discouraged as code mentioned here can change or be removed without
  notice between releases of Python.

Per the above it is best not to include this module to discourage
folks who might not head the above warnings. Additionally this module
is one of the largest py modules going, by dropping this unneeded
module from the 'modules' package we can reduce overall image size,
something which is important for many embedded deployments.

The generator scripts as well as the manifests have thus been modified
accordingly, providing a generic mechanism to exclude modules from the
'all modules' package.

(From OE-Core rev: a5bb13a5d7d7a668ca61da6b17884e3b05b95355)

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-03 11:14:13 +01:00
Saul Wold
7fd1491d2f mkefidsk: fix bash/dash shell quoting problem
mkefidsk currently writes a startup.nsh with embedded control characters.
This happens because \b etc are control sequences to the shell echo
command when using dash. The resulting startup.nsh causes the bootup
to fail, and the user is dropped into the EFI shell to manually run
startup.nsh.

Patch originally provided by Troy D. Hanson <troy.hanson@jhuapl.edu>

[YOCTO #9665]

(From OE-Core rev: 008d6cb5bb4969f53a228893c502be8c9420ecb0)

Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-06 14:38:13 +01:00
Ross Burton
027bdb0560 scripts/contrib/patchreview: add new script
This script analyses the patches we apply and can sanity check or output
statistics.

(From OE-Core rev: de7914954571ea8e717f56b6d6df13157b0973bc)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-29 13:01:38 +01:00
Leonardo Sandoval
f9b476ebcd scripts/contrib/patchtest: run patchtest on local branch
The script run patchtest on local branch commits, printing results into
stdout. This script is useful to test patches before sending to the
mailing list.

Examples:

$ git checkout master-next-1.9
Branch master-next-1.9 set up to track remote branch master-next-1.9 from origin.
Switched to a new branch 'master-next-1.9'

  $ ~/scripts/contrib/patchtest.sh
  166e70e: Robert Yang: Thu Apr 2 12:01:37 2015 +0100: patch: fix CVE-2015-1196: FAIL
   Issue             Missing or incorrectly formatted CVE tag in commit message [test_cve_presence_in_commit_message]
    Suggested fix    Include a "CVE-xxxx-xxxx" tag in the commit message
   Issue             Missing or incorrectly formatted CVE tag in included patch file [test_cve_tag_format]
    Suggested fix    Correct or include the CVE tag on cve patch with format: "CVE: CVE-YYYY-XXXX"

  eaa4536: Robert Yang: Thu Apr 2 12:01:37 2015 +0100: wget: 1.16.1 -> 1.16.2: OK

  3c29ce3: Robert Yang: Thu Apr 2 12:01:38 2015 +0100: git: 2.3.0 -> 2.3.1: OK

  85491f6: Khem Raj: Thu Apr 2 12:01:38 2015 +0100: gdb: Upgrade 7.8.1 -> 7.9: OK

  f701142: Robert Yang: Thu Apr 2 12:01:38 2015 +0100: binutils: upgrade to 2.25: OK

  385d0b1: Khem Raj: Thu Apr 2 12:01:39 2015 +0100: binutils: Fix ICE in gold: OK

[YOCTO #10720]

(From OE-Core rev: 4ba1334ab7c9b32502a4f8b3e656fb4f8242ccdd)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-23 11:44:12 +01:00
Martin Jansa
d973074bf6 python-3.5: Move bz2.py, lzma.py and _compression.py from python3-misc to python3-compression
* the /usr/lib/python3.5/_compression.py file is possibly incorrectly included
  in python3-misc. This runtime dependency is needed in order to use e.g. gzip.py in runtime:

  >>> import tarfile, zlib, gzip
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib/python3.5/gzip.py", line 12, in <module>
      import _compression
  ImportError: No module named '_compression'

* at least python3-tests and lzma and bz2 still in python3-misc are using this as well:
  $ grep -R import.*_compression tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/
  tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-compression/usr/lib/python3.5/gzip.py:import _compression
  tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-misc/usr/lib/python3.5/lzma.py:import _compression
  tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-misc/usr/lib/python3.5/bz2.py:import _compression
  tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-tests/usr/lib/python3.5/test/test_bz2.py:import _compression
  tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-tests/usr/lib/python3.5/test/test_lzma.py:import _compression

  and python3-tests are using it as well, so add new runtime dependency
  on python3-compression

(From OE-Core rev: 987363c3c720b3764f4d64976d7455f6b0bae99c)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14 14:53:58 +01:00
Ross Burton
f1333958dd python3: add python3-modules-native to RPROVIDES for python3-native
Also clean up the logic in the script to be more Pythonic.

(From OE-Core rev: fae66dd3633aa8a6aa633fcfd7c4b9a728dee7a4)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-06 19:52:26 +01:00
Ross Burton
b39d1c6842 python: add python-modules-native to RPROVIDES for python-native
Also clean up the logic in the script to be more Pythonic.

(From OE-Core rev: e5ac43e1b549e637f1820a03dd0a633fbecd395c)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-06 19:52:26 +01:00
Li Zhou
1c0ce38ba0 python3: Move config-*/Makefile from misc package to dev package
Move config-*/Makefile in libdir from misc package to dev package for
python3, because it is only needed in development process.

(From OE-Core rev: d715dc422ce1723c8d05af7ad4183eeeb36bc2ec)

Signed-off-by: Li Zhou <li.zhou@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-05-29 15:15:20 +01:00
Markus Lehtonen
4fdb6aa28d build-perf-test-wrapper.sh: support uploading test reports
Implement new '-R' command line option for specifying an rsync
destination where to upload test reports.

[YOCTO #5049]

(From OE-Core rev: ebc37ede5b247347483128f091b505fe33356591)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-05-18 14:01:47 +01:00
Markus Lehtonen
11747280b5 oe-build-perf-report-email.py: use pwd for getting user name
Use pwd data instead of os.getlogin() to get the username for fallback
email address. os.getlogin() basically returns the name of the user
logged in on the controlling terminal of the process and raises an
exception on some systems if the process does not have a controlling
terminal - when executed from a cron script, for example.

(From OE-Core rev: 66b7f495bb000c043ae24176dcdec0cb087eeecf)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:10 +01:00
Li Zhou
a2d5c0b602 python2/3: Move config/Makefile from core package to dev package
Move config/Makefile in libdir from core package to dev package for
python, because it is only needed in development process.

(From OE-Core rev: 8b55d055f046677c18eeaefe3ca18869eedeb14d)

Signed-off-by: Li Zhou <li.zhou@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:10:18 +01:00
Markus Lehtonen
182dd45f1f oe-build-perf-report-email.py: use proper fallback email address
Use properly formatted fallback email address instead of just the
username.

(From OE-Core rev: 5e07504d7a74f0641e2a374b9d12590ce9c9cc89)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:44 +01:00
Markus Lehtonen
60658499c2 build-perf-test-wrapper.sh: support extra args for email script
Make it possible to provide (extra) command line arguments to the
oe-build-perf-test-email script via a new environment variable
OE_BUILD_PERF_REPORT_EMAIL_EXTRA_ARGS.

(From OE-Core rev: 9cdd4ea5e006fe326bdf39ea437b9ba61a66b778)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:44 +01:00
Dmitry Rozhkov
e9cf698d09 python3: fix run-time deps for core python3 libraries
The http.server module from python3-netclient imports the html module
which is in python3-html. Also xmlrpc.server imports pydoc which is a
part of python3-pydoc. But those run-time dependencies are missing
from python3-netclient and python3-xmlrpc respectively.

Add the missing run-time dependencies.

(From OE-Core rev: 8e30b726c44f873e5fd9d3f36c3464a29b97abd8)

Signed-off-by: Dmitry Rozhkov <dmitry.rozhkov@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:43 +01:00
Markus Lehtonen
f81a532dd6 oe-build-perf-report-email.py: fix one file path
Sending report email was not working correctly if the script was given
an html report path that contained directory components.

(From OE-Core rev: 1da641661bb5963fcbd7ac2c20bc997c3eae6f18)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:42 +01:00
Markus Lehtonen
86eaa6d83f build-perf-test-wrapper.sh: support sending email reports
Add new '-E' command line option for sending an email report to
specified recipient.

(From OE-Core rev: 46e76ffd460933ab35da4cfd3509f7c5de5ecd93)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-05 23:22:13 +01:00
Markus Lehtonen
7e7c31fd00 build-perf-test-wrapper.sh: support pushing to remote Git
Implement new '-P' option for spefifying a Git remote where to push
results after committing to a local Git repository.

(From OE-Core rev: d8e14df29d28bfe805dc746f43c9f3a7726e57ce)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-05 23:22:13 +01:00
Markus Lehtonen
5576ce2de3 build-perf-test-wrapper.sh: correctly check test script exit status
Test of the exit code was accidentally moved to wrong place when
oe-git-archive was taken into use.

(From OE-Core rev: ed43b2dfe019f35086967a0c8dc605bc6629c75b)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-05 23:22:13 +01:00
Markus Lehtonen
d3d24ef2f2 build-perf-test-wrapper.sh: make it possible to specify Git branch name
Support <branch>:<commit> format for the -c argument. This makes it
possible to test older commits of a certain branch (not just the tip of
it) so that the branch name will still be correctly recorded in the test
report data.

(From OE-Core rev: be3d1718a99e59e636f349586e0a64eb8e2824a4)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-05 23:22:13 +01:00
Markus Lehtonen
034702f520 scripts/contrib: add oe-build-perf-report-email
Script for sending build perf test reports as an email. Mangles an html
report, generated by oe-build-perf-report, into a format suitable for
html emails. Supports multipart emails where a plaintext alternative can
be included in the same email.

Dependencies required to be installed on the host:
- phantomjs
- optipng

[YOCTO #10931]

(From OE-Core rev: 9e97ff174458f7245fc27a4c407f21a9d2e317ab)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-01 23:28:20 +01:00
Bob Cochran
1771bfd1a7 python: remove stale link to "Python for Embedded Systems Site"
Reference url is a stale, non existent site that returns a 404, so get rid of it

Change impacts both the manifest files and the scripts that generate the manifests

Run the following from within recipes-devtools/python

../../../scripts/contrib/python/generate-manifest-2.7.py > python-2.7-manifest.inc
../../../scripts/contrib/python/generate-manifest-2.7.py -n > python-native-2.7-manifest.inc

../../../scripts/contrib/python/generate-manifest-3.5.py > python-3.5-manifest.inc
../../../scripts/contrib/python/generate-manifest-3.5.py -n > python-native-3.5-manifest.inc

(From OE-Core rev: ae13f580b759211c1a6b59a276f75d589f1db11c)

Signed-off-by: Bob Cochran <openembedded@mindchasers.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-01 23:28:19 +01:00
Leonardo Sandoval
7ddcbc5eca scripts/contrib: scripts that updates kernel templates
Updating the kernel templates (those use by the yocto-bsp script) is a mechanical
process (and prone to errors) which consists of copying latest kernel template then
applying string replacements from old to new kernel version. This script collects
these commands allowing quick updates in the future.

(From meta-yocto rev: 450313d9d8bb1e728ed5a7208decd30f50633be1)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-27 08:15:24 +01:00
Markus Lehtonen
5e3eaf7c57 build-perf-test-wrapper.sh: store buildstats under git-notes
In order to dramatically reduce the data footprint of the result
branches. This makes cloning/pulling result repositories a lot faster.
Buildstats can still be accessed, when needed by doing
git fetch refs/notes/buildstats/*:refs/notes/buildstats/*

[YOCTO #10582]

(From OE-Core rev: 5b52bbf09bb2fbec9d2d54f3c3c46c74a09fbfd0)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-27 08:15:06 +01:00
Markus Lehtonen
28376f9552 build-perf-test-wrapper.sh: support xml report format
Add new command line option '-x' that enbles xml-formatted reports.

(From OE-Core rev: 1aa909991c7c6cd484cae35fcc742fbe7af3f8e8)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-04 23:18:18 +00:00
Ming Liu
5230e490f8 generate-manifest-2.7.py: fix python-tests rdepends
A potential flaw was introduced by commit a8dc4bc0:
[ python: fix python-tests rdepends ]

It made python-tests RDEPENDS on python-modules which is correct, but
this should be done in generate-manifest-2.7.py, and re-generate the
manifest.

Fixed by adding this dependency in generate-manifest-2.7.py as well.

(From OE-Core rev: 13adc5bbfd2ef64133a086ec6c917ac3ef274087)

Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-01 23:27:10 +00:00
Ming Liu
2f821f3c09 generate-manifest-3.5.py: add logic to generate native manifest
python3-native supposes to RPROVIDE all native packages as added in
generate-manifest-3.5.py, but it does not so far, this leads a problem
that sometimes bitbake cant find a runtime provider for a python3-*-native
when a new runtime dependency on it being required, this usualy happens
after a new native python3-* recipe is created or the old native python3-*
recipes are upgraded.

To avoid manually extending RPROVIDE every time when a new runtime
dependency is introduced, an argument '-n/--native' is added to the
manifest generator, allowing it create a native python3 manifest, with a
RPROVIDE line only, the RPROVIDE should contain all the sub-packages.

The generated python-native-3.5-manifest.inc is also added which is
included by python3-native recipe.

(From OE-Core rev: 800753069f667cd1664d70b3779150c467e3b3fe)

Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-01 23:27:10 +00:00
Ming Liu
e720e3b4af generate-manifest-2.7.py: add logic to generate native manifest
python-native supposes to RPROVIDE all native packages as added in
generate-manifest-2.7.py, but it does not so far, this leads a problem
that sometimes bitbake cant find a runtime provider for a python-*-native
when a new runtime dependency on it being required, this usualy happens
after a new native python-* recipe is created or the old native python-*
recipes are upgraded.

To give a example, the following commit is trying to address such a issue:
commit 4583cd1bb15306e8f0ab7bcd80732e6f35aa4533:
[ python-native: Make python-native also RPROVIDE python-unittest-native ]

To avoid manually extending RPROVIDE every time when a new runtime
dependency is introduced, an argument '-n/--native' is added to the
manifest generator, allowing it create a native python manifest, with a
RPROVIDE line only, the RPROVIDE should contain all the sub-packages.

The generated python-native-2.7-manifest.inc is also added which is
included by python-native recipe.

(From OE-Core rev: 0cb15d9559e34faffea1ac0be825d0602f225ba9)

Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-01 23:27:10 +00:00
Markus Lehtonen
f4ba140c59 build-perf-test-wrapper.sh: fix issues with non-existing path arguments
Without the '-s' option realpath will error out if the given path does
not exist.

(From OE-Core rev: b80aba08ba56c7e8f847966b3593f6cedd1b1ee5)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 20:06:41 -08:00
Markus Lehtonen
37c9f3f180 build-perf-test-wrapper.sh: use oe-git-archive
Start to use the new helper script for archiving results data in Git.

[YOCTO #10582]

(From OE-Core rev: 3d8a1df37407686ed699485c37c8517f1d79755d)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 20:06:41 -08:00
Anders Darander
8243ebdbb7 python-3-manifest: split out typing
This allows us to use typing.py without having to add the whole
python3-misc package.

(From OE-Core rev: 66c282541a13f2d1224d3ba933a953c0f613fb2a)

Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-05 09:22:16 +00:00
Anders Darander
5776eea84d python-3-manifest: add ipaddress to python3-io
This allows us to use ipaddress without requiring the add the whole
python3-misc.

(From OE-Core rev: d0e9013f6c68104369de940f5dd8a6ef76cd0210)

Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-05 09:22:16 +00:00
Anders Darander
db6977f72c python-3-manifest: add _compat_pickle to python3-pickle
This allows us to depend on _compat_pickle.* wihtout having to add the whole
python3-misc.

(From OE-Core rev: 6c435672b27d1c97ce3776c98d073a92fde887cd)

Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-05 09:22:16 +00:00
Anders Darander
64ab82509d python-3-manifest: add socketserver to netserver
socketserver.* should be part of python3-netserver.

(From OE-Core rev: 8aa71c7133f996f6ed09a3c094e553bed7f33c7a)

Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-05 09:22:16 +00:00
Anders Darander
ce5661ddc1 python-3-manifest: add argparse to RDEPENDS for netclient
http/server.py requires argparse.

(From OE-Core rev: 04d3e78b31c247ee521a2d4c15286a8c030637cb)

Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-05 09:22:16 +00:00
Anders Darander
67127bec02 python3-manifest: move htlm.py to python3-html
This allows us to use html.py without importing misc.

(From OE-Core rev: d1fefcaa10b85837d18458e291a3df9093b3a6ea)

Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-05 09:22:16 +00:00
Anders Darander
cf610aa69f python-3-manifest: fix adding imp to importlib
Commit: 512334f102a33833d39af53467894315f0715d07
	"python-3.5-manifest: Add imp to importlib"

added imp to importlib in the generated manifest, but not in the generator script.

(From OE-Core rev: bfe1d948ab034017622110c8365be7026e139fbf)

Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-05 09:22:16 +00:00
Markus Lehtonen
42eace0770 build-perf-test-wrapper.sh: implement locking
In order to prevent multiple instances of the script running at the same
time.

(From OE-Core rev: 96a194de890f7ef1e6e5e036b32848e0f9d1bcf5)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-02 17:37:45 +00:00
Derek Straka
413e11ba98 python-3.5-manifest: Add http module to the netclient package
Adding http module from Python's standard library. This allow use
of the http module without installing all python-misc modules.

(From OE-Core rev: 3d0060f9703ba39fbdaafcbdf91e0c319b56f7b3)

Signed-off-by: Derek Straka <derek@asterius.io>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-16 18:05:12 +00:00
Alejandro Hernandez
2dce2648e3 gummiboot: Remove/change gummiboot references with systemd-boot
After systemd-boot was introduced, its been tested for a while with no major
issues being found until now, this patch completely replaces all gummiboot
instances with systemd-boot ones, taking the next step into cleaning
up systemd-boot/gummiboot.

[YOCTO #10332]

(From OE-Core rev: f9a61d3400ad9068a6d83b8eb6aefe3098c58e68)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09 13:39:11 +00:00
Joshua Lock
c0f2890c01 scripts: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(From OE-Core rev: 0a36bd96e6b29fd99a296efc358ca3e9fb5af735)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Leonardo Sandoval
70f99b427b bb-perf: plot histograms base on buildstats data
Scripts that produces script data to be consumed by gnuplot.
There are two possible plots depending if either the
-S parameter is present or not:

    * without -S: Produces a histogram listing top N recipes/tasks versus
      stats. The first stat defined in the -s parameter is the one taken
      into account for ranking
    * -S: Produces a histogram listing tasks versus stats.  In this case,
      the value of each stat is the sum for that particular stat in all recipes found.
      Stats values  are in descending order defined by the first stat defined on -s

EXAMPLES

1. Top recipes' tasks taking into account utime

    $ buildstats-plot.sh -s utime | gnuplot -p

2. Tasks versus utime:stime

    $ buildstats-plot.sh -s utime:stime -S | gnuplot -p

3. Tasks versus IO write_bytes:IO read_bytes

    $ buildstats-plot.sh -s 'IO write_bytes:IO read_bytes' -S | gnuplot -p

(From OE-Core rev: 74408c19fba89de54c093fccf65b3a072d6a197b)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:05 +00:00
Dominic Sacré
3f2ab3f7cc python3: Build and package precompiled modules
Remove the patch that was applied in the python3 and python3-native
recipes to skip compilation of python modules.

Modify generate-manifest-3.5.py to match '__pycache__' directories in
FILES_*.
This is necessary because Python3 puts .pyc files in '__pycache__'
subdirectories one level below the corresponding .py files, whereas in
Python2 they used to be right next to the sources.

This change significantly reduces the startup overhead of Python3
scripts. For example, on a Cortex-A9, "python3 -c pass" took 0.40s
before, and 0.19s after.

(From OE-Core rev: bb4d689769703177dbb0df0935e15016b879f42b)

Signed-off-by: Dominic Sacré <dominic.sacre@gmx.de>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:05 +00:00
Leonardo Sandoval
4bc5353c92 scripts: Specify the stats to take into account
There are many more stats on buildstats that 'Elapsed time', so make the script
more flexible to support all stats. Some cmd line examples:

$ buildstats.sh -s 'utime'

Buildstats' data covers proc's stats in different areas, including CPU times,
IO, program system resources and child program system resources. In order
to print values on each of these sets from command line, one can use the
following:

$ buildstats.sh -H -s 'TIME' | less

$ buildstats.sh -H -s 'IO' | less

and 'RUSAGE' and 'CHILD_RUSAGE' for program and program's child system
resources.

(From OE-Core rev: 81479b191287ccbf4cf94fa2d0ad46813091bca1)

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 11:10:14 +00:00
Ed Bartosh
2d4778b1c8 mkefidisk.sh: add deprecation warning to the output
mkefidisk.sh will soon be deprecated in favor of .wic images.

Added deprecation warning to the script to inform users that
this script will soon be removed from the codebase.

(From OE-Core rev: ccef84fac7b20d483df87aac8c620459fe31b6af)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-06 23:35:32 +00:00
Markus Lehtonen
a17ce357fe build-perf-test-wrapper.sh: accept test case failures
Utilize the new return value (2) from oe-build-perf-test. Do not exit
with an error in case some individual tests fail. Even if some tests
fail we still want to complete successfully, that is, display and
archive the results and do cleanup. The individual tests do not depend
on each other anymore so test failures shouldn't affect the results of
successful tests.

(From OE-Core rev: e3c7d8a98a261a6a8c913e7fcd19264df501636d)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-01 21:45:56 +01:00
Markus Lehtonen
4df40248b3 build-perf-test-wrapper.sh: show defaults for '-a' and '-w'
Display default values for '-a' and '-w' command line arguments in the
usage help text.

(From OE-Core rev: 580708398f22333bc4b5899e4129a8939fb7ce12)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-01 21:45:56 +01:00
Markus Lehtonen
eb1c8cd1d9 build-perf-test-wrapper.sh: check for positional arguments
Stricter checking of command line arguments. The script doesn't use any
positional arguments so don't accept any and error out if those are
found.

(From OE-Core rev: 4725ee8e4e4837446dfa3a319eb68cc9572c55eb)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-01 21:45:56 +01:00
Markus Lehtonen
91ae03931e build-perf-test-wrapper.sh: fix handling of -C argument
Not specifying -C caused oe-build-perf-test to try to commit results to
the build directory.

(From OE-Core rev: 6f4786f5522c366a7fd92f630c3f32629a9f9471)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-05 11:56:02 +01:00
Paul Eggleton
e616beba1c scripts: ensure tinfoil is shut down correctly
We should always shut down tinfoil when we're finished with it, either
by explicitly calling the shutdown() method or by using it as a
context manager ("with ...").

(From OE-Core rev: 5ec6d9ef309b841cdcbf1d14ac678d106d5d888a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 23:45:54 +01:00
Paul Eggleton
4a5aa7ea4d scripts/contrib: update scripts for changes to internal API
The multiconfig changes altered some of the functions being called here,
so update the calls. Make use of the new Tinfoil.parse_recipe_file()
function to make parsing easier.

(From OE-Core rev: 95b6ceffd947271f315d8a7660797ab371adfbb9)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-03 23:45:52 +01:00
Markus Lehtonen
96e68f15f0 build-perf-test-wrapper.sh: make workdir configurable
New command line argument '-w' may be used to specify work dir other
than the default <GIT_DIR>/build-perf-test.

(From OE-Core rev: 824284895f25146520a624b7b97f7475d0135814)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:48 +01:00
Markus Lehtonen
ee4c5f6171 build-perf-test-wrapper.sh: make archive dir configurable
Add new command line argument '-a' that can be used to define the
directory where results (tarballs) are archived. Giving an empty string
disables archiving which makes sense if you store results in Git.

(From OE-Core rev: d53cf92847aa80724be4412801c993948a09cd27)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:48 +01:00
Markus Lehtonen
a34fd3cf27 build-perf-test-wrapper.sh: allow saving results in Git
Add new command line argument '-C' that allows saving results in a Git
repository.

(From OE-Core rev: 3d06795d8cd9017b042a7283c16ac71d4f6317a6)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:48 +01:00
Markus Lehtonen
dc3025215b build-perf-test-wrapper.sh: parse args with getopts
Use getopts for parsing the command line. This changes the usage so that
if a commit (to-be-tested) is defined it must be given by using '-c',
instead of a positional argument.

(From OE-Core rev: b1f77ba41033397a2b25977963682b86f2f76471)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:48 +01:00
Fabio Berton
e484378325 python-3.5-manifest: Add argparse module
Adding argparse module from Python's standard library. This allow use
argparse without installing all python-misc modules. For compatibility,
add python3-argparse as RDEPENDS to python3-misc.

(From OE-Core rev: f2b96001e074d26f5eb8711c2217a695fb02de4c)

Signed-off-by: Fabio Berton <fabio.berton@ossystems.com.br>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:46 +01:00
Fabio Berton
4c184eb757 python-3.5-manifest: Rename Queue module to queue
The Queue module has been renamed to queue in Python 3.

(From OE-Core rev: e19a430da2ef60b2c6cf6a67210ec1a7b292c8ca)

Signed-off-by: Fabio Berton <fabio.berton@ossystems.com.br>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-25 23:03:46 +01:00
Robert Yang
fd23044c46 python-3.5-manifest.inc: the signal module RDEPENDS on enum
Fixed:
$ python3
>>> import signal
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/to/sdk/sysroots/x86_64-pokysdk-linux/usr/lib/python3.5/signal.py", line 4, in <module>
    from enum import IntEnum as _IntEnum
ImportError: No module named 'enum'

(From OE-Core rev: 6306dc8351c19059c4c2a8e75bb5733e64532732)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-23 17:44:41 +01:00
Alejandro Hernandez
0cff756e15 python3.5-manifest: Fixes several dependencies on the newest python3
This patch adds the following packages: python3-enum (needed by python3-git),
python3-selectors (needed by python3-subprocess), python3-signal (needed by python3-subprocess),
and it also fixes the following ones with missing dependencies: python3-subprocess,
python3-compression, python3-datetime

[YOCTO #10127] [YOCTO #10124] [YOCTO #10122]

(From OE-Core rev: 0575e8c9fb52a7b594025fd20445a2edd06e3c69)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:35:43 +01:00
Ross Burton
a56f14e5fc graph-tool: update to new networkx API, be iterative
Update the dot parser to the new networkx API (using pydotplus to parse).

Also, switch the path display to output the paths as they are found instead of
collecting them into a list, so output appears sooner.

(From OE-Core rev: c91898b07465fdd5f3629babb7ff9226454de24e)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:35:42 +01:00
Kyle Russell
a1f39e5117 python-3.5-manifest: Add some missing RDEPENDS
ctype's util.py needs subprocess
lang's inspect.py needs importlib.machinery
math's random.py needs crypt's hashlib
subprocess imports threading

(From OE-Core rev: 38f9d7910fb5b2be5f7b1f62c4c7631d9e7138eb)

Signed-off-by: Kyle Russell <bkylerussell@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:35:42 +01:00
Ed Bartosh
f4709b1960 uncovered: list uncovered python modules
This bash script prints list of modules uncovered by oe-selftest
or any other test that produces coverage report.

It expects coverage report on its stdin and a directory to look
for python modules as a command line parameter, e.g.
    coverage report --rcfile=build/.coveragerc | ./scripts/contrib/uncovered bitbake/
should print list of uncovered python modules from bitbake/
directory tree to stdout.

[YOCTO #9809]

(From OE-Core rev: 00d9df2b70d00b9767f32f172192f33cbf3aca0e)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-01 11:47:11 +01:00
Derek Straka
89eab25990 python3: update manifest RDEPENDS for importlib and compression packages
zipfile.py has dependencies on importlib, threading, and shell
importlib has a dependency on lang
operator and contextlib added to the lang package instead of falling into misc

(From OE-Core rev: 769ad8e114fda1fe112d3747408edbeb7b066a85)

Signed-off-by: Derek Straka <derek@asterius.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:28:47 +01:00
Jussi Kukkonen
fd08714c26 python3: Add compression to python3-shell dependencies
python3-shell needs python3-compression for tarfile.

(From OE-Core rev: fe5979534bd4fc1f3e5401c9a86e4aff571aec24)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:16 +01:00
Maxin B. John
2c9291dfb9 graph-tool: convert to python3
move graph-tool to python3

(From OE-Core rev: 0d0864ae0ff9e53623ad1c7146b071f2a046f21f)

Signed-off-by: Maxin B. John <maxin.john@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:15 +01:00
Paul Eggleton
baed9bd848 scripts/contrib/devtool-stress: exclude more recipes by default
These recipes can't be used with devtool because they can't be unpacked
in the normal way.

(From OE-Core rev: b2cf098969b8b800a78d650cf60c0b5ad31c85b5)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:15 +01:00
Paul Eggleton
89f8348dc5 scripts/contrib/devtool-stress: skip incompatible recipes
If devtool returns exit code 4 then record the recipes as "skipped"
rather than "failed" - these are recipes we know cannot work (usually
because they don't provide any source).

(From OE-Core rev: 8fc109f1cb6eb437c12d2d11a6937de6f035e296)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-12 23:10:15 +01:00
Markus Lehtonen
678a02e112 scripts/contrib: introduce build-perf-test-wrapper.sh
A shell script wrapper around oe-build-perf-test script. The purpose of
this wrapper is to act as a executor script, making it possible to run
the tests with a single command. The wrapper script initializes the
build environment, runs oe-build-perf-test and archives the results.

(From OE-Core rev: 946a076c2ce20dd8f7cfa1acbdab1268d406d3e1)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-01 16:22:48 +01:00
Paul Eggleton
b41b2fa4dd scripts: ensure not specifying subcommand shows help text
With Python 2, argparse subparsers behaviour in Python 2 was to print
the usage information if the subparsers argument wasn't specified.
However, with Python 3.2.3 and later a subparsers argument is not
required by default, leading to errors when no arguments are specified:

  AttributeError: 'Namespace' object has no attribute 'func'

Restore the previous desired behaviour of showing the help text for
devtool, recipetool and the devtool-stress script by setting
subparsers.required to True.

(From OE-Core rev: d36fdea1a7f32d97187e0e9e6d701ae8fa304e8f)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-15 08:35:01 +01:00
Ed Bartosh
cdff6bc0c1 scripts: python3: change python to python3 in shebang
(From OE-Core rev: 4b544ff388497cac82b0585f237900595523e1cb)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-03 13:13:29 +01:00
Ed Bartosh
f6f10858e5 scripts: python3: fix urllib imports
Some functions and classes have been moved from urllib[2]
to urllib.request and urllib.error in python 3.

Used new imports to make the code working in python 3.

(From OE-Core rev: ec3f1759e8b491a44a1fc1ecb6f89919dd30da97)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-03 13:13:29 +01:00
Ed Bartosh
ee31bad762 scripts: python3: use new style except statement
Changed old syle except statements 'except <exception>, var'
to new style 'except <exception> as var' as old style is not
supported in python3.

(From OE-Core rev: 438eabc248f272e3d272aecaa4c9cec177b172d5)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-03 13:13:28 +01:00
Ed Bartosh
fa4275bbc0 scripts: Rename ConfigParser -> configparser for python3
The ConfigParser API was renamed to configparser in python 3.
Renamed ConfigParser -> configparser in scripts/ to make the
code working in python 3.

(From OE-Core rev: de6e98f272e623ce72e724e66920eecf10cb2d41)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:01 +01:00
Ed Bartosh
7eab022d4b scripts: Fix deprecated dict methods for python3
Replaced iteritems -> items, itervalues -> values,
iterkeys -> keys or 'in'

(From OE-Core rev: 25d4d8274bac696a484f83d7f3ada778cf95f4d0)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:01 +01:00
Ed Bartosh
2e388048b6 scripts: python3: Use print function
Used print function instead of print statement to make
the code work in python 3.

(From OE-Core rev: 80fecc44761fa38ccf2e4dc6897b9f1f0c9c1ed0)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-01 12:47:10 +01:00
Richard Purdie
88972114a6 meta/scripts: python3: rename file -> open
file() API doesn't exist in python 3, convert to open(). Also handle
some cases where files aren't closed. Compatible with python 2.7.

[Contributions from Ed and Richard]

(From OE-Core rev: 0f4ec13e11bb8abe21aba2a28547dfb9372bc377)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-21 22:26:33 +01:00
Raymond Tan
37abaa362c mkefidisk.sh: mount images as read-only
Mount the hddimg and rootfs.img as read-only when creating the bootable
image on the medium. Otherwise, the md5 checksum values of the hddimg will
be altered. As this changed checksum value might cause issue for users
whom would reuse the hddimg.

(From OE-Core rev: a1391c8a603f0ed972ee0bcc8c74999f5f43be43)

Signed-off-by: Raymond Tan <raymond.tan@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-19 09:05:18 +01:00
Christopher Larson
6e4b817776 ddimage: if 'pv' is installed, use it
This gives us a progress bar for the image write, which is quite helpful. See
https://www.ivarch.com/programs/pv.shtml.

(From OE-Core rev: 5b20cb2db929daaf0bf81c05368174e9c364ffab)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:31:15 +01:00
Ross Burton
f737af453d build-perf-test: add eSDK installed size to metrics
(From OE-Core rev: 57040305dea7e2167adb47c136a7b85f09ee7d24)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-09 23:00:45 +01:00
Ross Burton
9e69963f11 python: move ast module into python-core
The compile() method returns objects that inherit from ast.AST so it's best that
python-core contains this class.

[YOCTO #8684]

(From OE-Core rev: 48bc643a4ebe74d0fa49fedbe7b0fd63fd0003e8)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-12 22:11:47 +00:00
Ross Burton
9237097b79 python: merge python-elementtree into python-xml
python-xml depends on python-elementtree as the latter just contains a C library
used by the former.  However there's no point to this split apart from
increasing the number of packages, so merge -elementtree into python-xml.

(From OE-Core rev: 5f7206eba3953b7f29148ecfb791995773ee5fc7)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11 12:27:45 +00:00
Dominique Hunziker
149cb1769b python-distutils: add missing dependency on python-email
(From OE-Core rev: 1a8efbf4f06d9274cd60ada61d34b6bf42c15dc0)

Signed-off-by: Dominique Hunziker <dominique.hunziker@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04 23:39:02 +00:00
Markus Lehtonen
7a52f67e59 build-perf-test.sh: add eSDK testing
Add simple initial eSDK test. Currently, only download size and
installation time of eSDK is measured. The eSDK to be tested is
generated from the same image that the other tests are run for. This
patch will add two new fields to the global results log and that needs
to be taken into account when examining the results.

(From OE-Core rev: c903c1e1f36a4dd1dc1b7a621fa7a6ffe7411119)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-25 12:44:13 +00:00
Markus Lehtonen
5c367eccf8 build-perf-test.sh: more generic timing function
Make it possible to time also other than bitbake commands. The name of
the log file is changed from bitbake.log to commands.log.

(From OE-Core rev: 7b355dc96255b06f3108a7d02ab0ed408d64bf1b)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-25 12:44:13 +00:00
Alejandro Hernandez
2a608cce9f python-setuptools: Upgrade to 19.2
Upgrades python-setuptools to 19.2, easy_install works out of the box
adds the package python-plistlib to the manifest as it is needed by
setuptools now, and also updates runtime dependencies

(From OE-Core rev: 25efefac9f68d34bbb109645a515010b846c3a8b)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-20 17:07:14 +00:00
Trevor Woerner
b68d947e7c mkefidisk.sh: add boot log on console
Hooking up a serial console is a "developer mode", the chances are pretty good
developers are interested in watching the kernel boot log on the console so
they can spot any problems or diagnose any failed boots (e.g. can't find root
fs).

(From OE-Core rev: d03e399623b9320268a2b56e4928bb7effa9146c)

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-07 13:40:17 +00:00
Trevor Woerner
62d7c97912 mkefidisk.sh: add startup script for automated boot
Regardless of which image is built using which layers, try to ensure the image
on the SD device being prepared is the one that is booted automatically when
the board is powered.

(From OE-Core rev: 51dc79d55d0985e64838a08d39a22e8cd3fcd59a)

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-07 13:40:17 +00:00