Commit Graph

3881 Commits

Author SHA1 Message Date
Joshua Watt
4fd9ee4670 bitbake: bitbake: Add --skip-setscene option
Adds an option to skip _setscene only if they would normally be
executed, without ignoring sstate completely.

Previously, '--no-setscene' would allow a build that completely ignored
sstate and _setscene tasks, and '--setscene-only' would allow a build
that only ran _setscene tasks, but there was no option do a build that
would respect tasks previously restored from sstate and build everything
else. Now one can run:

 bitbake --setscene-only IMAGE; bitbake --skip-setscene IMAGE

which is functionally equivalent to:

 bitbake IMAGE

The indented use is to allow a build to complete successfully in the
presence of _setscene task failures by splitting apart the two phases
e.g.:

 (bitbake -k --setscene-only IMAGE || true) && bitbake --skip-setscene IMAGE

(Bitbake rev: 813ba5b7c13b573a0b813b628a819bdbf0627540)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Richard Purdie
97e19bf73b bitbake: main: Alter EOFError handling
If the server shuts down 'cleanly' due to some issue, the socket will close.
A recently reported example was an invalid PRSERV being set. Doing this silently
and without changing the retries count will case the server startup to loop
infinitely.

Change the code so it triggers the usual retries note messages and times
out eventually pointing the user at the cooker log file.

[YOCTO #12984]

(Bitbake rev: bb696636ef0c59f9e9640bb9460e7cce323cc785)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Richard Purdie
53138165bc bitbake: main: Ensure log messages are printed when no UI starts
If the UI hasn't started, no messages are printed to the console
until the server starts. This is confusing, particularly if the server
never starts. Flush the UI queue through the simply handler upon connection
retry so the user sees the messages they're supposed to be seeing.

Also point the user at the logfile for hints about why this may be.

(Bitbake rev: 4b9ab675cebb427ab8ad0c56c7b37eed50a2a39e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Richard Purdie
0a0cc5ff2e bitbake: event: Clear ui_queue after handling it
(Bitbake rev: 55631da1336589e583e8341a655179f7714ab3fe)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-03 17:00:57 +01:00
Robert Yang
5161799993 bitbake: bitbake: lib: Cleanup /usr/bin/env python
(Bitbake rev: cc712f3257904960247a7532cfc4611f3dccd36c)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-28 13:29:04 +01:00
Luca Boccassi
d8551bd029 bitbake: tests/fetch.py: add missing skipIfNoNetwork tags to tests that try to git clone
test_git_submodule_dbus_broker, test_git_submodule_CLI11, test_git_submodule_update_CLI11,
test_git_submodule_aktualizr and test_git_submodule_iotedge try to access the network via
git clone, which fails when there is no network available. Add the relevant skip tag.

(Bitbake rev: 9b0538753da0514e6518723dac537007abf7a649)

Signed-off-by: Luca Boccassi <luca.boccassi@microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-28 13:29:04 +01:00
Martin Jansa
50d272863d bitbake: tests/utils.py: add one more test cases for bb.utils.vercmp_string
* this is just another test case for issue already fixed in:
    commit fef56d28c3efec4876c379898cbc4d4c65303aee
    Author: Alexander Kanavin <alex.kanavin@gmail.com>
    Date:   Sun Feb 24 21:07:28 2019 +0100
    Subject: bitbake: fix version comparison when one of the versions ends in .

* The TypeError is triggered not by '.' at the end, but from the extra
  numberic component in one of the versions.

* When one version has fewer elements, it's extended by another (0, None)
  element where 0 means numeric component. Then the result cannot be
  decided by comparing the types (oa < ob, ob > oa) and it continues
  to compare values (ca < cb) which fails when one of them is the None
  from (0, None) appended before.

  ======================================================================
  ERROR: test_vercmpstring (bb.tests.utils.VerCmpString)
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "lib/bb/tests/utils.py", line 32, in test_vercmpstring
      result = bb.utils.vercmp_string('1a', '1a1')
    File "lib/bb/utils.py", line 131, in vercmp_string
      return vercmp(ta, tb)
    File "lib/bb/utils.py", line 123, in vercmp
      r = vercmp_part(va, vb)
    File "lib/bb/utils.py", line 112, in vercmp_part
      elif ca < cb:
  TypeError: '<' not supported between instances of 'NoneType' and 'int'

  ----------------------------------------------------------------------
  Ran 3 tests in 0.002s

(Bitbake rev: 9767fffe3115a1f1afa3c6a2b39720fefb8dc4d5)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-21 00:33:32 +01:00
Oleksandr Kravchuk
8a5501110e bitbake: cooker: list all nonexistent bblayer directories
Check existence of all the bblayer direcotories at once and print them
all, so if there are multiple nonexistent directories, user does not
have to correct bblayers.conf and restart bitbake multiple times.

[YOCTO #11647]

(Bitbake rev: 19291f7c4d17086ebb6a7b80c3cb06333d7fd55b)

Signed-off-by: Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18 11:23:48 +01:00
Richard Purdie
9e5a3f40ca bitbake: cooker: Ensure mcdeps are processed even if only one multiconfig
If you have no BBMULTICONFIG set but set mcdepends, they're currently
ignored. We can handle them correctly with this small tweak.

(Bitbake rev: 578f0c02f6a13f4315e7c2ce8b5e876dd2025055)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-11 13:27:19 +01:00
Chris Laplante
f77badd46a bitbake: build: implement custom progress handlers injected via OE_EXTRA_IMPORTS
A separate patch to base.bbclass (in poky) will add the OE_EXTRA_IMPORTS
variable. The contents are appended into OE_IMPORTS. This provides a
mechanism by which layers (in their layer.conf) can make custom progress
handlers available.

As a backup, individual recipes can inject progress handlers into
__builtins__.

Custom handlers are expected to have this __init__ signature:

    def __init__(self, d, outfile=None, otherargs=None):

Recipes can then use the handlers like this:

    do_task[progress] = "custom:mylayer.util.ProgressHandler[:args]"

The last part (everything after and including the second colon) is
optional. If provided, it is passed to HandlerClass's __init__ as
otherargs="args". Otherwise, otherargs=None.

(Bitbake rev: 20289d62c84c393990dd3deb0cca1b17c09092e6)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-11 13:27:19 +01:00
Chris Laplante
7687469590 bitbake: build/progress: use context managers for progress handlers
It seems context management support was half-implemented, but never
finished. For example, LogTee has __enter__ and __exit__ but they
haven't been exercised until now.

(Bitbake rev: bf522ad3e0c52cdb69b406226840d870ff4f2766)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-11 13:27:19 +01:00
Chris Laplante
575ea5532f bitbake: build: extract progress handler creation logic into its own method
(Bitbake rev: a841efa50d3aaf7c57446806327b2b687371cb29)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-11 13:27:19 +01:00
Chris Laplante
8c15a93553 bitbake: knotty: allow progress rate for indeterminate bars
(Bitbake rev: 85f0b443b7ab1848abc6eb658be489fc1718004c)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-11 13:27:19 +01:00
Richard Purdie
79ef0eab35 bitbake: cooker: Add compability handling for multiconfig: prefix migration
This allows "multiconfig:" targets to continue to work by internally
mapping them to the new "mc:" naming, allowing older builds to work
as before.

(Bitbake rev: c4d90890547af642e99cc541af3415df3559563e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-10 14:46:38 +01:00
Richard Purdie
1f68b0bd98 bitbake: multiconfig: Switch from 'multiconfig' -> 'mc'
After real world use its clear the "multiconfig:" prefix to multiconfig tasks,
whilst clear, is also clumbersome. Switch to use the short version instead.

mcdepends will continue to work with "multiconfig:" for now as well. The commandline
will only accept mc: going forward.

[YOCTO #11168]

(Bitbake rev: 821daf093b76504067a8b77dfa4b181af6ec92b4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-10 14:46:38 +01:00
Matthias Schiffer
482da97cfc bitbake: fetch2: runfetchcmd(): unset _PYTHON_SYSCONFIGDATA_NAME
Since warrior, python3native.bbclass sets _PYTHON_SYSCONFIGDATA_NAME;
unfortunately, this also affects Python scripts run as fetch commands like
git-make-shallow, breaking it with a message like

    Failed to import the site module
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site.py", line 570, in <module>
        main()
      File "/usr/lib/python3.6/site.py", line 556, in main
        known_paths = addusersitepackages(known_paths)
      File "/usr/lib/python3.6/site.py", line 288, in addusersitepackages
        user_site = getusersitepackages()
      File "/usr/lib/python3.6/site.py", line 264, in getusersitepackages
        user_base = getuserbase() # this will also set USER_BASE
      File "/usr/lib/python3.6/site.py", line 254, in getuserbase
        USER_BASE = get_config_var('userbase')
      File "/usr/lib/python3.6/sysconfig.py", line 607, in get_config_var
        return get_config_vars().get(name)
      File "/usr/lib/python3.6/sysconfig.py", line 550, in get_config_vars
        _init_posix(_CONFIG_VARS)
      File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix
        _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
    ModuleNotFoundError: No module named '_sysconfigdata'

on an Ubuntu 18.04 system (and likely others) when building with
BB_GIT_SHALLOW and BB_GENERATE_SHALLOW_TARBALLS.

Unset _PYTHON_SYSCONFIGDATA_NAME in runfetchcmd() to work around this.

(Bitbake rev: d94ccd506d04aff182ab48f501f6f366d5dd14f5)

Signed-off-by: Matthias Schiffer <matthias.schiffer@ew.tq-group.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-30 12:37:03 +01:00
Richard Purdie
cc67ce63d5 bitbake: main: Fix error message typo
(Bitbake rev: a1330b8a15a3f12516e3e293ffb5d840f362375d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-30 12:37:03 +01:00
Joshua Watt
88f5abb028 bitbake: bitbake: Show base multiconfig environment
Adds support to the 'bitbake -e' command so that it can display the base
environment for a multiconfig. It was previously possible to get the
base environment for the main environment by running "bitbake -e", but
there was no support for getting the base environment for a multiconfig
without specifying a recipe. A user can now print the base environment
for the multiconfig "foo" by running:

 $ bitbake -e multiconfig:foo

(Bitbake rev: 3d657af8a6120193d45d01968605b30075a56198)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-30 12:37:03 +01:00
Joshua Watt
66fd4c13d9 bitbake: bitbake: cooker: Rename __depends in all multiconfigs
The renaming of the __depends variable to __base_depends and file
watches needs to occurs for all multiconfigs, not just the base config.
Failing to do this for all multiconfigs will result in a huge increase
in the size of the parsing cache (about 5x for a single mulitconfig)
because all multiconfig caches will still depend on the base config
files. This will also seen a similar jump in the amount of time required
to load the parsing cache from memory, both because the cache is larger
and because of explosion of additional existence checks that must be
done for the base files.

[YOCTO #13359]

(Bitbake rev: 34137a00f60280e9e806070c6507a0fc6284b834)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-30 12:37:03 +01:00
Jean-Marie LEMETAYER
a35abe31dc bitbake: bitbake: fetch2/npm: fix npw view parsing
Fixes [YOCTO #13344]

When parsing manually the 'npm view --json' ouput, an extra closing
brackets in a JSON string can leads the fetcher to fail with a
JSONDecodeError exception.

This commit use the JSON parser to extract:
 - The last object in the returned array if there are multiple results.
 - The returned object if there is only one result.

(Bitbake rev: 3d319c79981811d3cfd4732885057db4fd5afcc2)

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-21 12:56:34 +01:00
Mark Hatle
a1c1f3f7c5 bitbake: svn.py: Stop SVN from directly pulling from an external layer w/o fetcher
Add a new option to the svn fetcher url "externals=allowed".  This will allow
a user to enable svn co w/ externals.  However, this does avoid the fetcher,
network access and mirror systems.

By default we no longer allow externals in the checkout.  This ensures a
deterministic download.  The system does attempt to identify SVN repos that
have externals enabled, and will warn the user.  It is up to the user to
determine if these are necessary for the recipe.  They may disable the warning
by adding "externals=nowarn" to the url.

In the future we would like to parse this list and see if the items are already
in the SRC_URI for that recipe, but with SVN being in limited use these days
that extra work is likely not worth the trouble.

Add test cases that generated a local SVN tree, with an external source
set to github bitbake in svn format.  One test case checks that externals are
ignored, and one checks that they in downloaded.

(Bitbake rev: bf53f07c3647e57d8452a7743a2b04bcb72c80d6)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-21 12:56:34 +01:00
Mark Hatle
7a73a435e0 bitbake: gitsm: Fix a bug where the wrong path was used for the submodule init
Because we are trying to avoid network activity and use our own fetcher,
the system emulates the behavior of 'git submodule init'.

git submodule init uses the .gitmodules file, where typically the module
name and path are the same.  However, in this case the module name and
path (in the tree) were different. i.e.:

[submodule "edgelet/hsm-sys/azure-iot-hsm-c/deps/azure-c-shared-utility"]
        path = edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared
        url = https://github.com/Azure/azure-c-shared-utility.git

Previously the code assumed the 'path' was both the checkout location
under .git/modules, as well as the path to extract the components.  This
proved to be incorrect as the .git/modules path needs to match the submodule
'name'.  This causes the components that were fetched to be initialized in
the wrong location, which later caused the 'git submodule update' process to
skip not properly initialized modules.

A test case was added for this specific case to ensure a regression does
not appear in the future.

(Bitbake rev: fd27ab60d33553dba13de39394edaaac04e446b3)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-16 09:17:15 +01:00
Richard Purdie
9501864db8 bitbake: bitbake: Strip old editor directives from file headers
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.

(Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
cf9c0be3f6 bitbake: bitbake: Drop duplicate license boilerplace text
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.

(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
79834a7144 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:04 +01:00
Richard Purdie
a85601aae1 bitbake: build: Disable warning about dependent tasks for now
This breaks with rm_work so disable the warning until we find a better
solution (and change the test accordingly too).

(Bitbake rev: 93e94c06baf013e3d072465a55bddd1fe61c0772)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-01 23:09:08 +01:00
Richard Purdie
dabc5ae13a bitbake: build: Ensure warning for invalid task dependencies is useful
WARNING: elfutils: dependent task do_rm_work does not exist

is much less useful than

WARNING: elfutils: dependent task do_rm_work for do_deploy does not exist

(Bitbake rev: e034c6f75e3d7730ff16a8d1bd0cba03beda0af8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-01 23:09:08 +01:00
Richard Purdie
fcec90de9c bitbake: knotty: Implement console 'keepalive' output
CI systems like jenkins and buildbot will timeout applications which haven't had console output
in some period of time. Add 'keepalive' output to knotty which gives output every 5000s if not
other output was made and tasks are still running. This reduces some problems encountered
with our CI testing.

(Bitbake rev: aa4f31e5741dd98acec73f16f6028e52f4c22d6f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-01 23:09:08 +01:00
Jacob Kroon
be75acf17f bitbake: knotty: Pretty print task elapsed time
A task's runtime is currently printed in seconds. Change it to
include minutes and hours for easier reading.

(Bitbake rev: c593ae5ec9fecd4bde823948024e4d56314a60ce)

Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-30 12:05:24 +01:00
Stefan Kral
34d526a79b bitbake: build: Add verbnote to shell log commands
Add the missing bbverbnote fifo cmd for logging from shell.

(Bitbake rev: cf11fdb30c405e1a4521a7299f84816c0e13a881)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-30 12:05:24 +01:00
Robert Yang
c761fb126d bitbake: tests/parse.py: Add testcase for addtask and deltask
(Bitbake rev: 4ac388646624e08bef848b560fa52deacf2ff4fb)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-30 12:05:24 +01:00
Robert Yang
e236735d45 bitbake: build.py: check dependendent task for addtask
The following command is incorrect, but was ignored silently, that may suprise
users:

addtask task after task_not_existed

This patch can check and warn for it. It would be better to also check "before"
tasks, but there is no easier way to do it.

[YOCTO #13282]

(Bitbake rev: b1ad36169c2ad189d84c53c707b66c12d5ff812f)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-30 12:05:24 +01:00
Robert Yang
82300691b0 bitbake: BBHandler: Fix addtask and deltask
The following commands are not supported, but they were ignored silently, that
may suprise users:

* addtask task1 task2
  task2 is ignored

* addtask task1 before task2 before task3
  Should be: addtask task1 before task2 task3

* addtask task1 after task2 after task3
  Should be: addtask task1 after task2 task3

* deltask task1 task2
  task2 is ignore

This patch can check and warn for them.

[YOCTO #13282]

(Bitbake rev: 675689aa7cc7287efecf8ef775ca2059369167f1)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-30 12:05:24 +01:00
Naveen Saini
e7c2bc67b5 bitbake: bitbake: fetch2/git: git-lfs check
Build will fail if repository has lfs contents in absense of git-lfs tool on host.
Build will pass if repository may or may not contains lfs content if host has git-lfs installed.

[YOCTO #13198]

(Bitbake rev: 3f2c2eb2f59707828bdcdd6414db837da8dc3b0e)

Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-29 14:16:31 +01:00
Robert Yang
7f7a7cba4d bitbake: bb: siggen: Print more info when basehash are mis-matched
This is useful for debugging.

(Bitbake rev: 07b6054b32b37375e2de4b1276bf6fe3f9ad0b04)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-23 23:30:45 +01:00
Robert Yang
3c06699da3 bitbake: bb: siggen: Make dump_sigfile and compare_sigfiles print uuid4
This can make people easier to understand bitbake-diffsigs/dumpsig's output,
otherwise, it's hard to know it is a random uuid unless look into the code.

E.g.:
$ bitbake bc-native -ccleansstate -Snone
$ bitbake bc-native -ccleansstate -Snone
$ bitbake-diffsigs tmp/stamps/x86_64-linux/bc-native/1.07.1-r0.do_cleansstate.sigdata.*

* Before:
  Taint (by forced/invalidated task) changed from nostamp:fe79d162-c4a8-4174-8007-f6d4aa09abdc to nostamp:28192187-5021-40c1-9e21-45483b62c910

* Now:
  Taint (by forced/invalidated task) changed from nostamp(uuid4):fe79d162-c4a8-4174-8007-f6d4aa09abdc to nostamp(uuid4):28192187-5021-40c1-9e21-45483b62c910

(Bitbake rev: 724b4a5cec8c611d53350f3e5a3988ec3222684b)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-23 23:30:45 +01:00
Robert Yang
5c5a0938de bitbake: bitbake-diffsigs: Use 4 spaces as indent for recursecb
It used 2 spaces as indent which wasn't clear enough, and might cause
confusions, people might think it was in wrong format.

Fixed:
$ bitbake bc-native -ccleansstate -Snone
$ bitbake bc-native -ccleansstate -Snone
$ bitbake-diffsigs tmp/stamps/x86_64-linux/bc-native/1.07.1-r0.do_cleansstate.sigdata.*

* Before:
Hash for dependent task bc/bc_1.07.1.bb.do_clean:virtual:native changed from [foo]
  Taint (by forced/invalidated task) changed from [foo]
Taint (by forced/invalidated task) changed from [foo]

* Now
Hash for dependent task bc/bc_1.07.1.bb.do_clean:virtual:native changed from [foo]
    Taint (by forced/invalidated task) changed from [foo]
Taint (by forced/invalidated task) changed from [foo]

(Bitbake rev: 5127a8d8e6d53f5f43a6ada7fd09b6b0c24ae989)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-23 23:30:45 +01:00
Richard Purdie
03edd0bd23 bitbake: bitbake: Post release version bumnp to 1.43
(Bitbake rev: aae15f3c16fb463962eee100a8b0bcd5fc01ad96)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 13:59:52 +01:00
Richard Purdie
0c3423aae9 bitbake: bitbake: Update version to 1.42.0
(Bitbake rev: 00b133af009f9e7c1a4c751b6ef4902f47a81fe2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 13:48:04 +01:00
Andre Rosa
9be9876760 bitbake: utils: Let mkdirhier fail if existing path is not a folder
Let mkdirhier fail if existing path is not a folder instead of assuming a
directory hierarchy already exists.

(Bitbake rev: a8d9b82ccf93dcb74258693f62d88be380b1c0b7)

Signed-off-by: Andre Rosa <andre.rosa@lge.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 13:48:04 +01:00
Andre Rosa
27cbc8c5f0 bitbake: utils: Make prune_suffix prune a suffix
Make prune_suffix prune a suffix instead of replacing a substring that could
happen more than once and not only when it ends with it.

(Bitbake rev: 57e765e38c6382a9b36d5ee2a6f3fa96ac905b82)

Signed-off-by: Andre Rosa <andre.rosa@lge.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 13:45:15 +01:00
Mark Hatle
2030e815bb bitbake: gitsm: Add need_update method to determine when we are going to a new SRCREV
If the system had previously fetched a source repository for use by gitsm,
and then the SRCREV was updated and the new commit already existed, the system
would not re-evaluate the submodules and update them accordingly.

The cause of this issue was that need_update was being used, unmodified, from
the base git fetcher.  It did not have any knowledge, nor did it care if we
were moving commits and needed to re-evaluate what was happening due to this
switch.

To fix the issue, during the download process we add all processed (by
gitsm) srcrevs to the git config file, as bitbake.srcrev.  This allows us to
use a new need_update function that not only checks if the git commit is
present, but if we have previously processed this commit to ensure all of the
submodule components are also present.

This approach is used, instead of iterating over the submodules in need_update
to avoid a potential race condition that has affected us in the past.  The
need_update is called only with the parent locking.  Any time we need to dive
into the submodules, we need to lock, and unlock them, at each stage.  This
opens the possibility of errors in either the code, or unintended race
conditions with rm_work.

This issue was discovered by William A. Kennington III <wak@google.com>.  The
included test case was also written by him, and included unmodified.

(Bitbake rev: 30fe86d22c239afa75168cc5eb262b880886ef8a)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-01 14:48:44 +01:00
Robert Yang
b1641008f0 bitbake: fetch2: Use bb.utils.to_boolean() for BB_NO_NETWORK
Make it consistent with BB_FETCH_PREMIRRORONLY.

(Bitbake rev: 4c6013f47ef36b03f590c909d7c9a2f50b698620)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 14:02:28 +00:00
Robert Yang
64c21baf63 bitbake: fetch2: Unify BB_FETCH_PREMIRRORONLY
The fetch2/__init__.py checks whether "BB_FETCH_PREMIRRORONLY" == "1", but
fetch2/git.py and hg.py checks whether it is None, this makes it discontinuous,
and BB_FETCH_PREMIRRORONLY = "0" doens't work as expected in the later case,
so unify it to the previous one. (As BB_NO_NETWORK does).

And also use bb.utils.to_boolean() to make them consistent.

(Bitbake rev: 85a0d22835588e9ad8ec29d88a8115227e88420c)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 14:02:28 +00:00
Robert Yang
acc242c7ec bitbake: fetch2: Print SCMs list when SRCREV_FORMAT is not set
This makes it easier to debug, especially when multipe SCMs like gitsm,
otherwise we don't know why there are multiple SCMs.

(Bitbake rev: 313fe5e86b254eadfdead706be4bd7b274d5e3c0)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
Robert Yang
db91c23871 bitbake: fetch2/git: Fix clean to remove clonedir
The localpath is a symlink to clonedir when it is cloned from a mirror, for
example:
$ bitbake systemtap-native -cfetch
$ ls downloads/git2
sourceware.org.git.systemtap.git -> /path/to/downloads/git2/mirror.path.git.sourceware.org.git.systemtap.git
mirror.path.git.sourceware.org.git.systemtap.git

There are both sourceware.org.git.systemtap.git and
mirror.path.git.sourceware.org.git.systemtap.git in DL_DIR/git2, the symlink
sourceware.org.git.systemtap.git is created by try_mirror_url(), but
do_cleanall" only removed the symlink, didn't remove the real dir
mirror.path.git.sourceware.org.git.systemtap.git, this may cause confusions,
for example, I assumed that do_cleanall removed everything, but it didn't, and
it would the re-used next time when do_fetch. This patch fixes the problem.

(Bitbake rev: 452e2200ad2c29dec3753f5f7a8cbc9183ec7dd8)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
Robert Yang
4b2b3339df bitbake: cooker: Fix bbfile_config_priorities when BBFILE_PATTERN is empty
The layer was not in bbfile_config_priorities when BBFILE_PATTERN is empty,
this caused "bitbake-layers show-layers" can't show these layers, this was
incorrect since these layer did exist. Add these layer to
bbfile_config_priorities can fix the problem.

Fixed:
Add BBFILE_PATTERN_core = "" in oe-core/meta/conf/layer.conf
$ bitbake show-layers | grep oe-core

There was nothing, now the layer is shown

(Bitbake rev: 0ff5cdb0cca9266ca29127639494bcfd95e36831)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
Robert Yang
a7f3cbb4a2 bitbake: fetch2: runfetchcmd(): Print workdir in debug message
The workdir is an important message when run git command, for exmample, before the patch:
Running git -c core.fsyncobjectfiles=0 branch --contains [snip]

We don't know where it is running, now it is:
Running 'git -c core.fsyncobjectfiles=0 branch --contains [snip]' in /path/to/git.opensvc.com.multipath-tools..git

Which is easier to debug.

(Bitbake rev: f25c05b73218d0e8edf2dd63b36bc09e547c5455)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
Richard Purdie
c96d8de352 bitbake: fetch2: Fix undefined variable issues
Various refactors have left hanging variables, often in debug messages, hence
why they haven't been spotted. Fix them (pylint spotted the issues).

(Bitbake rev: e1f252408982a62485bbf874a01fd30a02aeed32)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00
Richard Purdie
28fb15c56f bitbake: fetch2/wget: Remove pointless lambda function
(Bitbake rev: 329986c3f2d98c4cd1a43b725194003575e718d5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00