Commit Graph

2824 Commits

Author SHA1 Message Date
California Sullivan
46bb764616 wic: add wks file to make use of exclude-path functionality
Using --exclude-path and a wks.in file we can create an image that takes
the /boot/ directory for the boot partition, has an empty /boot/
directory in the rootfs partition. The boot partition gets mounted to
/boot/ after startup.

(From OE-Core rev: db904053e8ee80fb6930c5e7e22287927e0f25e2)

Signed-off-by: California Sullivan <california.l.sullivan@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-06 06:35:42 -08:00
Paulo Neves
2327e0f585 wic: Set a miniumum FAT16 volume size.
Set a miniumum FAT16 volume size because images are
not valid to parted if this minimum value is not respected.

The value set is determined experimentally forr a logical
sector size of 512. This fixed my local problem but, there
may be better solutions.

(From OE-Core rev: f7dfb4d43247d3c13a4e0a3853007d63b9512b83)

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-04 11:35:40 +00:00
Paulo Neves
5796b037d5 wic: partition plugin wrongly assumes it is rootfs
The partition plugin is used as the base for other plugins.
One of the methods the plugins use, is the prepare_rootfs
method.

The prepare_rootfs method wrongly assumes that the value
ROOTFS_SIZE from bitbake datastore is relevant to every
invocation of prepare_rootfs, which it clearly is not, for
example in the bootimg-partition case.

This commit adds an optional argument to prepare_rootfs
where a caller can tell prepare_rootfs if it is an actual
rootfs and whether related rootfs information retrieved from
bitbake is valid. The default behavior of this optional
argument is to assume that the invocation is an actual
rootfs, to maintain compatibility with previous
implementations.

(From OE-Core rev: 654d72d55194ec41bc1aacfcc6b2c8c9a305b042)

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-04 11:35:40 +00:00
Parthiban Nallathambi
8eb7e66246 wic: Add post operation hook for SourcePlugin
do_post_partition hook is needed if some operations like security signing
the parition needs to be done. source plugins can make use of this to implement
post operatiosn in do_post_partition. do_post_partition is called after
do_prepare_partition if present.

(From OE-Core rev: 5055489b9ab3fda32a285d0d165d080d11a4d432)

Signed-off-by: Parthiban Nallathambi <pn@denx.de>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-04 11:35:40 +00:00
Paul Eggleton
80feb637a4 devtool: search: tweak help text
* We now match on more than just target recipes, so don't specify that
  only target recipes are searched.
* We're printing the SUMMARY value in addition to the name, so mention
  that so it's clear where that text is coming from.
* Remind users that they should use quotes around the keyword to avoid
  shell expansion when using regular expressions.

(From OE-Core rev: cc68971557fe065e59ff47657f650051eb85db3c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-01 22:13:56 +00:00
Paul Eggleton
e8c7c36c60 devtool: search: also look in recipe cache
If pkgdata isn't present or is incomplete, then you get either a
traceback or you don't see the results you were hoping for. The recipe
cache that bitbake collects during startup contains some useful
information for each recipe that we could search through as well, and
we can access it easily using tinfoil's all_recipes() API function,
so add some code that does that. (We still show a warning if pkgdata
isn't present, as there are certain dynamic packages that are generated
at packaging time that won't show up in the cache).

One side-effect of this is that we will start showing non-target
recipes - that's actually a good thing, since seeing those is useful,
however we exclude nativesdk recipes when in the eSDK to avoid confusion
since nativesdk isn't directly applicable there.

Fixes [YOCTO #12356].

(From OE-Core rev: b8406383886d09a80a9a002150dcf364fa9902d7)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-01 22:13:56 +00:00
Paul Eggleton
3bb00af9ab devtool: deploy-target: don't specify ssh/scp port unless user does
If the user doesn't specify a port then we should avoid specifying one
on the ssh/scp command line in case the user has configured one for the
host they are connecting to, which was being overridden unnecessarily.

Fixes [YOCTO #12381].

(From OE-Core rev: f1020eef09fada7ef2231d5528576553f3f6bfe1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-01 22:13:56 +00:00
Paul Eggleton
0465558e83 devtool: finish: fix erroneously creating bbappend for relative paths
After OE-Core rev 5e3fe00a0233d563781849a44f53885b4e924a9c we call
os.path.abspath() on the original layer path, but we later compare that
to the destination layer path. If that layer path isn't absolute but is
effectively the same path, it should be writing to the original recipe
but because we weren't making it absolute we were writing a bbappend
instead. Call os.path.abspath() on the destination path as well to avoid
that.

(From OE-Core rev: a85a78dcf226d160e9b504bfa67b306a9175ac29)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-01 22:13:56 +00:00
Paul Eggleton
e295135ba1 devtool: reset: delete bbappend file if _check_preserve() doesn't
If the .devtool_md5 file doesn't contain a reference to the bbappend
file (e.g. because devtool was interrupted before it could write that
out) then _check_preserve() won't delete it, so we need to delete it
separately because otherwise the recipe won't actually be reset.

(From OE-Core rev: 751d27600a3df18d96baaa48696acf33ee964bec)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-01 22:13:56 +00:00
Paul Eggleton
2f40a87424 devtool: fix poor handling of upgraded BBCLASSEXTENDed recipes
Fix two aspects of handling BBCLASSEXTENDed targets (e.g.
openssl-native) that have been run through "devtool upgrade":

* Fix recipe name not showing up in "devtool status"
* Fix "devtool reset" not deleting empty directories under the recipe
  directory within the workspace, which may lead to problems if you
  subsequently run "devtool upgrade" on the same target again

(From OE-Core rev: 56e04260d64de9c5b83893d97cf41b7ea9a45878)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-01 22:13:56 +00:00
Enrico Scholz
98aff12d9c oe-init-env: search 'bitbake' outside of oe-core
atm, 'oe-init-build-env' expects 'bitbake' to be within the OE core git
repository.  This complicates the project setup because you have to
manage the 'bitbake' directory or symlink manually (or specify the
bitbake location explicitly).

Looking for 'bitbake' outside the main git repository will ease project
management significantly.  Now, you can put everything into git submodules,
clone the project with

| git clone --recursive ...

and continue immediately with

| ..../oe-init-build-env

E.g. when you had previously

| .
| |-- build/
| `-- sources/
|     `-- org.openembedded.core/
|         `-- bitbake -> ../bitbake

(where 'bitbake' must be created manually after cloning the project),
you can have now

| .
| |-- build/
| `-- sources/
|     |-- bitbake/
|     `-- org.openembedded.core/

which is completely managed by 'git'.

Patch adds $OEROOT/.. to the search path for 'bitbake'

(From OE-Core rev: 5b3afc9cfe38a9fb435fbe5fcabc59b9a60f4657)

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-01 22:13:56 +00:00
Anuj Mittal
1b7a9d4f63 buildhistory-diff: honour report_all flag
Make sure that we're passing a bool value. Without this, buildhistory
shows all the output for all the keys/fields when it shouldn't be by
default.

(From OE-Core rev: b30153a15715a83c0f9a7d7d1883a15404992a19)

Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-24 10:35:59 +00:00
Richard Purdie
c2e722f6a6 scripts/contrib/build-perf-test: Fix fetchall reference to use runall
(From OE-Core rev: 028ab7eb11f78c02900389197eace81824e6553f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-20 22:25:10 +00:00
Ross Burton
45f2a2f155 oe-pkgdata-util: add --long option to list-pkg-files to show sizes
(From OE-Core rev: 1f3a5acb825a9f707c1ab780131e009f9ce21451)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-16 18:05:38 +00:00
Paul Eggleton
a1179425a9 devtool: set up git repos so that singletask.lock is ignored
singletask.lock is written out while certain tasks execute for recipes
that have externalsrc.bbclass enabled - this includes recipes in
devtool's workspace. It appears that there's a race where
singletask.lock will be there one minute and then when we try to get the
file checksum of it (since we want to know if anything in the source
tree has changed) it will be gone, and git chokes. To fix that, add
singletask.lock to .git/info/exclude in the repository, regardless of
whether we created the repository or not. In any case singletask.lock
should never be tracked by git, so this is a good thing to be doing for
that reason as well.

This fixes oe-selftest failures in test_devtool_modify that we've seen
on the Yocto Project autobuilder:

bb.data_smart.ExpansionError: Failure expanding variable
do_compile[file-checksums], expression was ${@srctree_hash_files(d)}
which triggered exception CalledProcessError: Command
'['git', 'add', '-A', '.']' returned non-zero exit status 128.

Note that this only fixes this issue for devtool; if you are using
externalsrc independently of devtool there's a chance this will still
be an issue unless you add singletask.lock to your .gitignore.

(From OE-Core rev: 334ba846c795fc0d8c73ce05a1b0882739c86650)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-06 11:06:27 +00:00
Alexander Kanavin
cb63b4bfa0 postinst_intercepts: add a delay_to_first_boot hook
This allows postinsts scripts to explicitly request a deferral to first boot
(by calling 'postinst_intercept delay_to_first_boot') instead of 'exit 1' which
should be used only to indicate actual script failures.

(From OE-Core rev: 853ac4a061e47c70b61e255c3cf6109d2058d2cc)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-30 11:50:12 +00:00
Tan Shen Joon
4984b60c76 distrodata: add a utility script to compare list of recipes
distrocompare.sh is added to compare the added list of recipes
between two releases. The output of the script will share the
information of the licenses required and other distributions
that are using the package.

If a single input is provided, it will compare the current
branch with the provided branch/commit-ish package list.

To run : distrocompare.sh <older hash> <newer hash>
E.g. distrocompare.sh morty 92aa0e7
E.g. distrocompare.sh morty pyro
E.g. distrocompare.sh morty

output : The script will produce a file ending with
new_recipe_list.txt preceeded by the branch name from input

(From OE-Core rev: 32b363c2ba91fde4f10e5fe2c898b2fc2702aa85)

Signed-off-by: Tan Shen Joon <shen.joon.tan@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-29 08:49:50 +00:00
Amanda Brindle
d48c2c6f38 oe-pkgdata-util: Include colon when searching for variables
When searching for variables, include colon to ensure the script doesn't
find a variable that starts with the same name.

(From OE-Core rev: 3d2c87c4f4115b01534ab198c27682c7e4c5f31f)

Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-29 08:49:49 +00:00
Amanda Brindle
3f408bae9f oe-pkgdata-util: Add support for RPROVIDES
In lookup_recipe, package_info, and list_pkg_files, check if the package
name exists in runtime-rprovides. If so, and the provider package has a
different name than the inputted package, print a note that says the
specified package is in another package's RPROVIDES. If the provider
package has the same name as the inputted package, continue as before.

Fixes [YOCTO 11943]

(From OE-Core rev: f78478f0d0379ea02727c81ad2455207c70d140b)

Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-22 10:39:09 +00:00
Amanda Brindle
76cbeffd2f oe-pkgdata-util: Refactor functions for consistency
Refactor functions lookup_recipe and package_info to be consistent with
list_pkg_files. Print the appropriate information as soon as it's found,
rather than storing it in a mappings variable and wait to print after
searching all packages.

(From OE-Core rev: 64d3ce83d5c48d479709b4c1355e23b3768493fb)

Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-22 10:39:09 +00:00
Alejandro Hernandez
b6777878ff python3: Restructure python3 packaging and replace it with autopackaging
See previous commit (python2 version) for more info, since mostly
everything applies here as well.

Old manifest file had several issues:
 - Its unorganized and hard to read and understand it for an average
   human being.
 - When a new package needs to be added, the user actually has to modify
   the script that creates the manifest, then call the script to create
   a new manifest, and then submit a patch for both the script and the
   manifest, so its a little convoluted.
 - Git complains every single time a patch is submitted to the manifest,
   since it violates some of its guidelines.
 - It changes or may change with every release of python, its impossible
   to know if the required files for a certain package have changed
   (it could have more or less dependencies), the only way of doing so
   would be to install and test them all one by one on separate individual
   images, and even then we wouldnt know if they require less dependencies,
   we would just know if an extra dependency is required since it would
   complain, lets face it, this isnt feasible.
 - The same thing happens for new packages, if someone wants to add a new
   package, its dependencies need to be checked manually one by one.

Features/Fixes:
 - A new manifest format is used (JSON), easy to read and understand.
   This file is parsed by the python recipe and python packages
   read from here are passed directly to bitbake during parsing time.
 - It provides an automatic manifest creation task (explained on previous
   commit), which automagically checks for every package dependencies and
   adds them to the new manifest, hence we will have on each package
   exactly what that package needs to be run, providing finer granularity.
  - Dependencies are also checked automagically for new packages
   (explained on previous commit).

This patch has the same features as the python2 version but it differs
in the following ways:
 - Python3 handles precompiled bytecode files  (*.pyc) differently.
   for this reason and since we are cross compiling, wildcards couldnt be
   avoided on python3 (See PEP #3147 [1]).
   Both the manifest and the manifest creation script handle this
   differently, the manifest for python3 has an extra field for cached
   files, which is how it lets the user install the cached files or not
   via : INCLUDE_PYCS = "1" on their local.conf.
 - Shared libraries nomenclature also changed on python3, so again, we
   use wildcards to deal with this issue ( See PEP #3149 [2]):
 - Fixes python3 manifest, python3-core should be base and everything
   should depend on it, hence several packages were deleted:
   python3-enum, re, gdbm, subprocess, signal, readline.
 - When building python3-native it adds as symlink to it called
   nativepython3, which is then isued by the create_manifest task.
 - Fixes [YOCTO #11513] while were at it.

References:
[1] https://www.python.org/dev/peps/pep-3147/
[2] https://www.python.org/dev/peps/pep-3149/

(From OE-Core rev: 54ac820b8a639950ccb534dcd9d6eaf8b2b736e0)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-20 22:31:56 +00:00
Alejandro Hernandez
8d94b9db22 python: Restructure python packaging and replace it with autopackaging
The reason we have a manifest file for python is that our goal is to
keep python-core as small as posible and add other python packages only
when the user needs them, hence why we split upstream python into several
packages.

Although our manifest file has several issues:
 - Its unorganized and hard to read and understand it for an average
   human being.
 - When a new package needs to be added, the user actually has to modify
   the script that creates the manifest, then call the script to create
   a new manifest, and then submit a patch for both the script and the
   manifest, so its a little convoluted.
 - Git complains every single time a patch is submitted to the manifest,
   since it violates some of its guidelines.
 - It changes or may change with every release of python, its impossible
   to know if the required files for a certain package have changed
   (it could have more or less dependencies), the only way of doing so
   would be to install and test them all one by one on separate individual
   images, and even then we wouldnt know if they require less dependencies,
   we would just know if an extra dependency is required since it would
   complain, lets face it, this isnt feasible.
 - The same thing happens for new packages, if someone wants to add a
   new package, its dependencies need to be checked manually one by one.

This patch fixes those issues, while adding some additional features.

Features/Fixes:

 - A new manifest format is used (JSON), easy to read and understand.
   This file is parsed by the python recipe and python packages read
   from here are passed directly to bitbake during parsing time.
 - It provides an automatic manifest creation task (explained below),
   which automagically checks for every package dependencies and adds
   them to the new manifest, hence we will have on each package exactly
   what that package needs to be run, providing finer granularity.
 - Dependencies are also checked automagically for new packages (explained below).
 - Fixes the manifest in the following ways:
   * python-core should be base and all packages should depend on it,
     fixes lang, string, codecs, etc.
   * Fixes packages with repeated files (e.g. bssdb and db, or
     netclient and mime, and many others).
 - Sitecustomize was fixed since encoding was deprecated.
 - The JSON manifest file invalidates bitbake's cache, so if it changes
   the python package will be rebuilt.
 - It creates a solution for users that want precompiled bytecode files
   (*.pyc) INCLUDE_PYCS = "1" can be set by the user on their local.conf to
   include such files, some argument they get faster boot time, even when the
   files would be created on their first run?, but they also sometimes give a
   magic number error and take up space, so we leave it to the user to
   decide if they want them or not.
 - Fixes python-core dependencies, e.g.
   When python is run on an image, it TRIES to import everything it needs,
   but it doesnt necessarily fails when it doesnt find something, so even if
   we didnt know, we had errors like (trimmed on purpose):
   # trying /usr/lib/python2.7/_locale.so
   # trying /usr/lib/python2.7/lib-dynload/_locale.so
   # trying /usr/lib/python2.7/_sysconfigdata.so

   while it didnt complain about _locale it should have imported it,
   after creating a new manifest with the automated script we get:

   # trying /usr/lib/python2.7/lib-dynload/_locale.so
   dlopen("/usr/lib/python2.7/lib-dynload/_locale.so", 2);
   import _locale # dynamically loaded from /usr/lib/python2.7/lib-dynload/_locale.so

How to use (after a new release of python, or maybe before every OE
release):
 - A new task called create_manifest was added to the python package,
   which may be invoked via:

 $ bitbake python -c create_manifest

 This task runs a script on native python on our HOST system, and since
 the python and python-native packages come from the same source, we can
 use it to know the dependencies of each module as if we were doing it
 on an image, this script is called create_manifest.py and in a very
 simplistic way it does the following:

 1. Reads the JSON manifest file and creates a dictionary data structure
    with all of our python packages, their FILES, RDEPENDS and SUMMARY.
 2. Loops through all of them and runs every module listed on them
    asynchronously, determining every dependency that they have.
 3. These module dependencies are then handled, to be able to know which
    packages contain those files and which should RDEPEND on one another.
 4. The data structure that comes out of this, is then used to create a
    new manifest file which is automatically copied onto the user's python
    directory replacing the old one.

 Create_manifest script features:
 - Handles modules which dont exist anymore (new release for example).
 - Handles modules that are builtin.
 - Deals with modules which were not compiled (e.g. bsddb or ossaudiodev)
 - Deals with packages which include folders.
 - Deals with packages which include FILES with a wildcard.
 - The manifest can be constructed on a multilib environment as well.
 - This method works for both python modules and shared libraries used
   by python.

How to add a new package:
 - If a user wants to add a new package all that has to be done is
   modify the python2-manifest.json file, and add the required file(s)
   to the FILES list, the script should handle all the rest.
   Real example:
   We want to add a web browser package, including the file webbrowser.py
   which at the moment is on python-misc.
   "webbrowser": {
       "files": ["${libdir}/python2.7/lib-dynload/webbrowser.py"],
       "rdepends": [],
       "summary": "Python Web Browser support"}

 Run bitbake python -c create_manifest and the resulting manifest
 should  be completed after a few seconds, showing something like:
   "webbrowser": {
      "files": ["${libdir}/python2.7/webbrowser.py"],
      "rdepends": ["core","fcntl","io","pickle","shell","subprocess"],
      "summary": "Python Web Browser support"}

Known errors/issues:
 - Some special packages are handled differently: core, misc,
   modules,dev, staticdev.
   All these should be handled manually, because they either include
   binaries, static libraries, include files, etc. (something that we
   cant import).
   Specifically static libraries are not not supported by this method
   and have to be handled by the user.
 - The change should be transparent to the user, other than the fact
   that now we CANT build python-foo (it was pretty dumb anyway, since
   what building python-foo actually did was building the whole python
   package anyway), but doing IMAGE_INSTALL_append = " python-foo"
   would create an image with the requested package with no issues.

[YOCTO #11510] [YOCTO #11694] [YOCTO #11695]

(From OE-Core rev: 6959e2e4dba5bbfa6ffd49c44e738cc1c38bc280)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-20 22:31:56 +00:00
Robert Yang
1b96585b92 scripts/oe-depends-dot: add it to handle dot files
Add it to handle recipe-depends.dot and task-depends.dot. E.g.:

* Print why rpm is built
  $ oe-depends-dot -k rpm --why/-w recipe-depends.dot
  Because: core-image-sato libdnf libsolv dnf

* Print bzip2-native's depends
  $ oe-depends-dot -k bzip2-native --depends/-d recipe-depends.dot
  Depends: automake-native gnu-config-native libtool-native quilt-native autoconf-native

* Remove duplicated dependencies to reduce the size of the dot files.
  For example, A->B, B->C, A->C, then A->C can be removed. The dot files are too
  big, we nearly couldn't use 'dot -T' to generate pictcures for target recipes,
  remove the duplicated dependencies makes is it possible.
  $ bitbake core-image-sato -g
  $ oe-depends-dot -r recipe-depends.dot
  Saving reduced dot file to recipe-depends-reduced.dot
  $ du -sh recipe-depends*.dot
  608K    recipe-depends.dot
  32K     recipe-depends-reduced.dot

  It has been recuded from 608K to 32K, now we can generate a picture,
  otherwise, it is too big:
  $ dot -Tpng recipe-depends-reduced.dot -O

It also can handle task-depends.dot.

(From OE-Core rev: 7dc7860691304d63e7ad728d2180474906fe0a5c)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-19 12:37:13 +00:00
Yi Zhao
cf75fd5ae0 contrib/yocto-bsp-kernel-update.sh: remove this script
This script was unused since yocto-bsp tool had been removed.

(From meta-yocto rev: 594ced59dc80c11d573ae4a16e1aff910a8d9300)

Signed-off-by: Yi Zhao <yi.zhao@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-18 13:05:56 +00:00
Ross Burton
1c88db7a1c buildhistory-diff: add support for colourising the output
Colour value removes in red and additions in green, making it easier to scan the
output for relevant changes.

This adds a --colour option to specify whether colouring should be on, off, or
detected.  The default is detected, and depends on whether stdout is a TTY (same
behaviour as git).

(From OE-Core rev: 4208f1546c92f069e432d1865269ce539db8cea7)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-18 12:50:37 +00:00
Alexander Kanavin
c1058ac4f2 devtool: rename the suggested license change tag to 'License-Update:'
(From OE-Core rev: 3905ae20330f204f3c8997e2d5aaf15dcf5a860c)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-18 12:50:37 +00:00
Dogukan Ergun
bab3b2e939 wic: if we can't get from ioctl, try from os.stat()
Under some conditions, ioctl FIGETBSZ can't return real value.
We can try to use fallback via os.stat() to get block size.

Source of patch:
17365f4fe9

(From OE-Core rev: d8f7cf2d38934c248be91101236f7537d0d31ea7)

Signed-off-by: Dogukan Ergun <dogukan.ergun@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-14 09:11:57 +00:00
Chang Rebecca Swee Fun
422e16fae3 scripts/wic: explicitly set BUILDDIR within eSDK
When we run wic within eSDK:
$ wic create mkefidisk -e core-image-minimal

ERROR: BUILDDIR not found, exiting. (Did you forget to source oe-init-build-env?)

In order to figure out variable values, one must have sourced
the OE build environment setup script. However, when we are in
within the eSDK environment which isn't initialised like the
normal OE build environment, we can't use wic utility with eSDK.

Reference:
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#wic-requirements

While wic ought to be fixed to be able to run without bitbake
& native tools [YOCTO #11281], but this is a workaround to set
BUILDDIR in the environment so that bitbake environment is setup
for wic to build its required native tools.

(From OE-Core rev: 03fa13a269d2887cc5d13fd474fb39a2be037f2c)

Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-13 10:15:20 +00:00
Chang Rebecca Swee Fun
1d52d1a46a scripts/wic: fix error of import wic module in eSDK environment
wic modules in scripts/lib/ are needed for wic to work, but path to
the python module is not exported in eSDK environment and we were
using an absolutized path of wic script within the sysroots.

We now changed to use real script path instead, where the wic modules
are located. This will also resolved the tracebacks found when running
wic from within the eSDK environment.

Traceback (most recent call last):
  File "/tmp/deploy/sdk/poky_sdk/sysroots/x86_64-pokysdk-linux/usr/bin/wic", line 58, in <module>
    from wic import WicError
ImportError: No module named 'wic'

(From OE-Core rev: dcea30b885797ece3439cf1201795a975628d664)

Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-13 10:15:20 +00:00
Chang Rebecca Swee Fun
7006ff99b0 scripts/wic: append bitbake executable file path in eSDK environment
wic needs a set of tools to be available from sysroots.
wic will find bitbake executable within the environment,
and wic was unable to locate bitbake executable within eSDK
because it wasn't setup with the OE build environment script.
Hence, we need to add bitbake file path into the environment
PATH for wic to be able to discover it and import bb modules.

(From OE-Core rev: 89df0d31c9dd22ceba4c95a2a56ca78e58d871a8)

Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-13 10:15:20 +00:00
Chang Rebecca Swee Fun
be5616a27c scripts/wic: use scriptpath module to find bitbake path and oe lib path
Use the scriptpath module in order to standardize the adding of
bitbake and meta/lib path to sys.path.

(From OE-Core rev: 8aba1fd023ce3c6767bf42b9faf9ec14fd7c4d02)

Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-13 10:15:20 +00:00
Richard Purdie
d18003726f runqemu: Add workaround for APIC hang on pre 4.15 kernels on qemux86
On pre 4.15 host kernels, an APIC window emulation bug can cause qemu
to hang. On 64 bit we can use the x2apic, for 32 bit, we just have to
disable the other timer sources and rely on kvm-clock.

[YOCTO #12301]

(From OE-Core rev: 82e67b82ea8e12aa0b7b9db1d84fec0436dec71b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-02 17:24:39 +00:00
Adrian Fiergolski
faae2df397 wic: Fix a path to a psuedo state directory (PSEUDO_LOCALSTATEDIR).
In case of 'new_rootfs' the psuedo directory is not copied. Thus
PSEUDO_LOCALSTATEDIR should still point to the dsa
'native_sysroot'/../pseudo. Otherwise PSEUDO_LOCALSTATEDIR points to a not
existing director ('new_rootfs'/../pseudo) and UID and GUID attributes are not
applied to files of the image.

(From OE-Core rev: 19642e2d6e015072e4a413f4f57aee65df757cb9)

Signed-off-by: Adrian Fiergolski <adrian.fiergolski@cern.ch>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-02 17:24:37 +00:00
Tom Rini
9ebc830196 wic: Introduce --fsuuid and have --use-uuid make use of UUID too
First, allow for wic to be given a filesystem UUID to be used when
creating a filesystem.  When not provided, wic will generate the UUID to
be used.  Next, when --use-uuid is passed, we update the fstab to mount
things via UUID (and if not found, then use PARTUUID) as UUID is more
portable.

(From OE-Core rev: 9256b8799495634ee8aee5d16ff71bd6e6e25ed4)

Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-18 18:03:58 +00:00
Tom Rini
20748d8ddc wic: partition.py: Update comments slightly
First, we support squashfs as root, so mention that.  Second, the btrfs
rootfs creation function had a copy/paste of the previous function
comment, remove the irrelevant line.

(From OE-Core rev: 7cdd4034b3e6ff4e13d491dfba24906afe495e2d)

Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-18 18:03:58 +00:00
Tom Rini
cce54f6a19 wic: kparser.py: Check for SquashFS and use-uuid
The SquashFS filesystem does not support UUIDs so make this combination
be an error.

(From OE-Core rev: 2fbdcf4e59c835af0f4041bc34243decb42321ef)

Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-18 18:03:58 +00:00
Alexander Kanavin
2de2a19a00 devtool: add license checksum change handling to 'devtool upgrade'
Specifically, 'devtool upgrade' will now do these things:

1) determine if any of the license checksums need updating; if so,
write the new checksums into the LIC_FILES_CHKSUM value in the recipe
that is written to the workspace;

2) print a notice to the standard output:

NOTE: New recipe is /home/ak/development/poky/build/workspace/recipes/puzzles/puzzles_git.bb
NOTE: License checksums have been updated in the new recipe; please refer to it for the difference between the old and the new license texts.

3) and the cool part: devtool will create a diff of the old and new licenses,
and write the diff into the workspace recipe as a comment, like this:

======
 FIXME: the LIC_FILES_CHKSUM values have been updated by 'devtool upgrade'.
 The following is the difference between the old and the new license text.
 Please update the LICENSE value if needed, and summarize the changes in
 the commit message via 'License-checksum-change:' tag.
 (example: 'License-checksum-change: copyright years updated.')

 The changes:

 --- LICENCE
 +++ LICENCE
 @@ -1,8 +1,9 @@
  This software is copyright (c) 2004-2014 Simon Tatham.

  Portions copyright Richard Boulton, James Harvey, Mike Pinna, Jonas
 -Klker, Dariusz Olszewski, Michael Schierl, Lambros Lambrou, Bernd
 -Schmidt, Steffen Bauer, Lennard Sprong and Rogier Goossens.
 +Kölker, Dariusz Olszewski, Michael Schierl, Lambros Lambrou, Bernd
 +Schmidt, Steffen Bauer, Lennard Sprong, Rogier Goossens and Michael
 +Quevillon.

  Permission is hereby granted, free of charge, to any person
  obtaining a copy of this software and associated documentation files

======

(From OE-Core rev: ccb37f9e81eb78ed0eb2a238d7c3e196db9b2f72)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-18 18:03:58 +00:00
Alexander Kanavin
b52a9cf310 devtool: provide useful defaults for version/commit when upgrading recipes
Specifically, 'devtool upgrade' will use the latest upstream release if available
or latest commit if upstream never makes releases.

(From OE-Core rev: 45b4242b105ad36e94ae15a96d588a58b917b8e8)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-18 18:03:57 +00:00
Alexander Kanavin
5b61268fca devtool: add a 'latest-version' command
This command queries the upstream server for what the latest release is and prints
the output; it is a much neater way to find out these things than fumbling with distrodata,
'bitbake -c checkpkg' and awkward to read csv output in a file.

Examples:

python3 (tarballs):
NOTE: Current version: 3.5.3
NOTE: Latest version: 3.6.3

rpm (git):
NOTE: Current version: 4.13.90
NOTE: Latest version: 4.14.0
NOTE: Latest version's commit: da3720f62e57648fb1dc2a632744d38866139971

puzzles (git without version tags):
NOTE: Latest commit: ee8ea9b9785964694cb2b3ad77c3fb2460f49510

(From OE-Core rev: e8f5b5cc25ce7a9882f21473cefc47edcebf77d4)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-18 18:03:57 +00:00
Chang Rebecca Swee Fun
05bcd72f9f recipetool: create: fix conflict between SRCREV and tag
If you specify 'tag=' for a git URL and passed to recipetool create, you
will get into Bitbake expansion error shown below:

----- snip -----
$ devtool add --version 2.4.2 mbedtls "git://github.com/ARMmbed/mbedtls;tag=mbedtls-2.4.2"
...
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Conflicting revisions (abeccb9dbd7e19ae91ac50e1edd3803111c5f9b6 from SRCREV and mbedtls-2.4.2 from the url) found, please specify one valid value
----- snip -----

Assuming the tag is valid, we should get the tag commit hash and
drop the usage of 'tag=' from SRC_URI. By using a commit hash
corresponding to the tag will prevent bitbake from accessing
remote repository in order to expand SRCPV.

(From OE-Core rev: 53f8effa3eb07dc7035ff9933e7918318f242579)

Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-18 18:03:57 +00:00
Tim Orling
8aa71adf49 python-*-manifest/generators: add runpy; python3-plistlib
* runpy allows running modules/scripts with 'python -m foo'
* python3-setuptools RDEPENDS on plistlib (present in python2)
* pip3 RDEPENDS on _markupbase (add to python3-core)

(From OE-Core rev: d95f1005c35bd9c7e22c40c7c17d264fe9435c6b)

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-13 14:00:51 +00:00
Tim Orling
69a910d2e7 python-*-manifest/generators: fix long line lengths
The generators create python-*-manifest.inc files with
lines over 2500 characters long which breaks sending
patches via git send-email (because of smtp limitation).

This patchset formats all the long lines into multiple lines.

(From OE-Core rev: 3a1900a5691466a04b24907067f43117b318ca7e)

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-13 14:00:51 +00:00
Martin Hundebøll
23825ed16a wic: support filesystem label for rawcopy
The '--label' argument should work for '--source rawcopy' as it does for
'--source rootfs', so add a method in RawCopyPlugin to update the label
on the temporary filesystem images.

(From OE-Core rev: 303d6ca5ae986acd2e633b0dc5e386ee7771f8ab)

Signed-off-by: Martin Hundebøll <mnhu@prevas.dk>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-10 22:45:19 +00:00
Alistair Francis
fae2e2f42d runqemu: Allow the user to override the device tree option
Update the runqemu script to allow the user to specify a device tree
to boot when calling runqemu.

This involves creating a seperate check_dtb() function incase the user
has specified 'none' for the kernel but still wants a device tree.

(From OE-Core rev: 867ac1370b294bfd1ee31f94abb63688f77081a1)

Signed-off-by: Alistair Francis <alistair.francis@xilinx.com>
Reviewed-by: Ben Levinsky <ben.levinsky@xilinx.com>
Cc: Ben Levinsky <ben.levinsky@xilinx.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-02 11:25:34 +00:00
Alistair Francis
7159b281e7 runqemu: Allow the user to override the Kernel option
Update the runqemu script to allow the user to specify a Kernel to boot
when calling runqemu.

(From OE-Core rev: eaf2793a98cb27d82561da0f8993f2b4b304ecc2)

Signed-off-by: Alistair Francis <alistair.francis@xilinx.com>
Cc: Ben Levinsky <ben.levinsky@xilinx.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-02 11:25:34 +00:00
Amanda Brindle
4af421a469 scripts/contrib/bbvars.py: Remove dead code
Removed the -m option since this script now searches through all
recipes in the configuration. Also removed dead code, which includes
the functions recipe_bbvars() and collect_bbvars().

(From OE-Core rev: dac6515fcd23ea9cde5308c1d08a7a928efbb4d6)

Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-12-02 11:25:32 +00:00
Robert Yang
8082fd8d52 runqemu: print command search result when not found
This makes debug easier.

(From OE-Core rev: a453639e19fb2a9f9fb63fddd0b3ee26c0116d91)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-30 10:49:20 +00:00
Ross Burton
a67c4f3f6f verify-bashisms: use argparse, add verbose option
(From OE-Core rev: 5168ecf6545ddde03bb801e4200d8a6563789be3)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-30 10:49:20 +00:00
Richard Purdie
72867393fe runqemu: Also specialcase resolution of '.' to the file's location
Similarly to handling "../", handle "." to resovle to the qemuconf
file's current directory.

(From OE-Core rev: 33418ed064fe9cff5b4803f09135a81d9170c189)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-21 17:58:41 +00:00
Richard Purdie
3ee7dcf68b runqemu: Improve relative path handling in qemuconf files
If a variable starts with "../", its likely its a path and we want to
set it to an absolute path relative to the qemuconf file.

This means we don't have to use bitbake as often to figure out variables.

(From OE-Core rev: dfc7940900d798aa47716288338107e1d46a3972)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-21 13:11:03 +00:00