Commit Graph

3862 Commits

Author SHA1 Message Date
Richard Purdie
d4b57c68b2 bitbake: fetch/git: Handle github dropping git:// support
github is dropping support for git protocol in Git urls. Add code to remap
this to https in a way that could be used in older bitbake versions.

(Bitbake rev: af479e239d1e4dc93f0162c5936369a94c1c529d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 11:31:54 +00:00
Richard Purdie
d588ff814d bitbake: tests/fetch: Allow wget upgrade tests to run against a local server
Currently these tests rely upon multiple uptream webservers which may change
or be unavailable. Add local copies of the test data, copy the httpserver
from OE-Core (used for testing there) and run these tests against a local
server instead.

(Bitbake rev: b4860b292406698f3c8af9a3708a09df21c96300)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-09 09:44:49 +01:00
Richard Purdie
b65fc3d950 bitbake: fetch2: Change git fetcher not to destroy old references
It looks like we're about to see a lot of changes in branch names in repos. If
we have the prune option here, those old names are lost, the changes propagate
to our source mirrors and our old releases break.

We have the force option so any replaced references should be replaced, its only
orphaned branches which will now be preserved.

I believe this behaviour will cause us fewer problems given the changes that
look likely to happen.

(Bitbake rev: 4c7e0352c56dc956fc19439713e8dce9f6248c86)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-08 14:46:09 +01:00
Richard Purdie
5ea39eac95 bitbake: tests/fetch: Switch from git.infradead.org to a YP mirror
Upstream is unavailable, breaking tests. Switch to a YP mirror since
if we can't reach that there are bigger problems. This should remove
a source of intermittent failures on the autobuilder.

(Bitbake rev: 417f635fca1aa2484c7ccba9ffbbfe9601d41d66)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-02 16:11:35 +01:00
Daisuke Yamane
4c773c7b03 bitbake: lib/bb/utils.py: Preserve ownership of symlink
Yocto Bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13806

Uncomment lchown() to preserve ownership of symlink.

(Bitbake rev: 7f7126211170439ac1d7d72e980786ce0edb7bb7)

Signed-off-by: Daisuke Yamane <daisuke.yamane@cybertrust.co.jp>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-06 17:25:34 +00:00
Richard Purdie
2ce6c7b190 bitbake: cooker/siggen: Empty siggen cache during parsing
When parsing recipes its apparent the memory usage of bitbake rises linearly
with number of recipes parsed. It shouldn't.

Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the
dependency information left behind in siggen was the culprit. Add a new
method to allow us to drop this information. We don't need it after the recipe
has been parsed and hashes calculated (at runtime its different but only the
currently executing task would be in memory).

This should give signficant memory usage improvements for bitbake and that
in turn should help speed on more constrained systems, as well as when used in
multiconfig environments.

(Bitbake rev: d411b097d810c386d35dc561f8812bb3f35c9a36)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-18 13:59:11 +00:00
Peter Kjellerstedt
2bd2f56d41 bitbake: cooker: Keep track of watched files using a set instead of a list
When there are many watched files, keeping track of them using lists
is suboptimal. Using sets improves the performance considerably.

(Bitbake rev: aa770081d7d3ff678b18117a4b4e9b3da09b72be)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1e96df260e47d160dbd36bfc92c31ef06266f662)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-28 11:51:10 +00:00
Richard Purdie
d9879a1f60 bitbake: fetch2: Ensure cached url data is matched to a datastore
There was a weird error in OE-Core where "devtool modify virtual/kernel"
was showing basehash mismatch errors. This was due to SRCPV sometimes being:
AUTOINC+b867b78b50_47b80ef7bd and sometimes AUTOINC+b867b78b50_255a750d28.

The latter hash comes from KBRANCH and meant sometimes the correct branch
was seen, sometimes it was not. The issue was complicated by the execution
using a remote datastore over tinfoil.

The problem turns out to be a fetcher caching error. If the datastore
changes, the cached url data may not be valid.

We therefore ensure we match cached url data against the datastore that
generated it, which appears to fix this issue.

(Bitbake rev: 4ce50bbd34eefeabfeca89a6a66c71598d3c58f6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-11-13 22:00:27 +00:00
Joshua Watt
07d4569049 bitbake: cookerdata: Add mc conffiles hashes to cache hash
The variable values that result from parsing multiconfig should be
included in the cooker data hash, otherwise changes to these files won't
be detected, which will allow the parsing cache to be loaded with the
old values for the multiconfigs. This can either manifest as the
variable values simply not updating, or getting basehash changed errors
when building.

This bug was previously undetected because all of the multiconfig base
files were a direct file dependency in all parsed recipes. This was
fixed in 34137a00f60 ("bitbake: bitbake: cooker: Rename __depends in all
multiconfigs"), exposing this bug.

[YOCTO #13541]

(Bitbake rev: 75d6648f232a06b99c54a1e33324a7fc1cd15b38)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-10 16:53:33 +01:00
Armin Kuster
6d2e12e792 bitbake: tests/fetch: Resolve fetch error in bitbake-selftest
FAIL: test_wget_latest_versionstring (bb.tests.fetch.FetchLatestVersionTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest/build/bitbake/lib/bb/tests/fetch.py", line 1229, in test_wget_latest_versionstring
      self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0])
      AssertionError: '' is not true : Could not find upstream version for db

[YOCTO #13496]

The Oracle UPSTREAM_CHECK_URI used changed and does not work with logic in wget.

Update UPSTREAM_CHECK_URI and UPSTREAM_CHECK_REGEX to match the ones used in the
recipe. Also change the version being checked.

(Bitbake rev: 164c0df429fc9be771c54c241b17d7afb8849afc)

Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-09 09:58:02 +01:00
Martin Jansa
78ac8af785 bitbake: fetch2: show warning when renaming the archive with bad checksum failed
* noticed on read-only sshfs premirror
* it was showing the warning about renaming the file:
  WARNING: laser-geometry-1.6.4-r0 do_fetch: Renaming /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441

  and then failed because of movefile() issue with python3 (fixed in previous commit):
  ERROR: laser-geometry-1.6.4-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated:

  with movefile() fixed, it let do_fetch continue and re-fetch locally with the right
  checksum, but still the renamed file didn't exist, because of movefile failure - add
  another warning when the movefile fails - for whatever reason - unfortunately movefile
  prints error messages with just print() so the real error is hidden only in log.do_fetch
  in this case:
  movefile: Failed to move /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 [Errno 30] Read-only file system: '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz' -> '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441'

(Bitbake rev: 24e44c412ac36987c1f65ced50030064ec58100d)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 11:58:50 +01:00
Martin Jansa
6cd3eee5fa bitbake: utils: Fix movefile() exception handling with python3
* with python3 this fails with:
  File: 'bitbake/lib/bb/utils.py', lineno: 799, function: movefile
       0795:        try:
       0796:            os.rename(src, destpath)
       0797:            renamefailed = 0
       0798:        except Exception as e:
   *** 0799:            if e[0] != errno.EXDEV:
       0800:                # Some random error.
       0801:                print("movefile: Failed to move", src, "to", dest, e)
       0802:                return None
       0803:            # Invalid cross-device-link 'bind' mounted or actually Cross-Device
  Exception: TypeError: 'OSError' object is not subscriptable

(Bitbake rev: 16415c6e0f9e3d5c9fd81c9aabaea11d61b14187)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-06 11:58:50 +01:00
Richard Purdie
048c1467e2 bitbake: bitbake: Drop duplicate license boilerplace text
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.

(Bitbake rev: 34ed28a412af642a993642c14bd8b95d5ef22cd8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-19 18:04:01 +01:00
Richard Purdie
e1db6a1690 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ac556588fac55e91b7ce4839a975eb9ebb5aa192)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-19 18:04:01 +01:00
Matthias Schiffer
b164f0dc3e bitbake: fetch2: runfetchcmd(): unset _PYTHON_SYSCONFIGDATA_NAME
Since warrior, python3native.bbclass sets _PYTHON_SYSCONFIGDATA_NAME;
unfortunately, this also affects Python scripts run as fetch commands like
git-make-shallow, breaking it with a message like

    Failed to import the site module
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site.py", line 570, in <module>
        main()
      File "/usr/lib/python3.6/site.py", line 556, in main
        known_paths = addusersitepackages(known_paths)
      File "/usr/lib/python3.6/site.py", line 288, in addusersitepackages
        user_site = getusersitepackages()
      File "/usr/lib/python3.6/site.py", line 264, in getusersitepackages
        user_base = getuserbase() # this will also set USER_BASE
      File "/usr/lib/python3.6/site.py", line 254, in getuserbase
        USER_BASE = get_config_var('userbase')
      File "/usr/lib/python3.6/sysconfig.py", line 607, in get_config_var
        return get_config_vars().get(name)
      File "/usr/lib/python3.6/sysconfig.py", line 550, in get_config_vars
        _init_posix(_CONFIG_VARS)
      File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix
        _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
    ModuleNotFoundError: No module named '_sysconfigdata'

on an Ubuntu 18.04 system (and likely others) when building with
BB_GIT_SHALLOW and BB_GENERATE_SHALLOW_TARBALLS.

Unset _PYTHON_SYSCONFIGDATA_NAME in runfetchcmd() to work around this.

(Bitbake rev: 6783104f18a8a14ddd88932720cd80dffeada63d)

Signed-off-by: Matthias Schiffer <matthias.schiffer@ew.tq-group.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18 17:00:37 +01:00
Mark Hatle
12a8a51e96 bitbake: gitsm: Fix a bug where the wrong path was used for the submodule init
Because we are trying to avoid network activity and use our own fetcher,
the system emulates the behavior of 'git submodule init'.

git submodule init uses the .gitmodules file, where typically the module
name and path are the same.  However, in this case the module name and
path (in the tree) were different. i.e.:

[submodule "edgelet/hsm-sys/azure-iot-hsm-c/deps/azure-c-shared-utility"]
        path = edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared
        url = https://github.com/Azure/azure-c-shared-utility.git

Previously the code assumed the 'path' was both the checkout location
under .git/modules, as well as the path to extract the components.  This
proved to be incorrect as the .git/modules path needs to match the submodule
'name'.  This causes the components that were fetched to be initialized in
the wrong location, which later caused the 'git submodule update' process to
skip not properly initialized modules.

A test case was added for this specific case to ensure a regression does
not appear in the future.

(Bitbake rev: 2a823584a5dbc2427060352f3b4798dbb2640cfd)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18 11:43:22 +01:00
Joshua Watt
d6462e0149 bitbake: bitbake: cooker: Rename __depends in all multiconfigs
The renaming of the __depends variable to __base_depends and file
watches needs to occurs for all multiconfigs, not just the base config.
Failing to do this for all multiconfigs will result in a huge increase
in the size of the parsing cache (about 5x for a single mulitconfig)
because all multiconfig caches will still depend on the base config
files. This will also seen a similar jump in the amount of time required
to load the parsing cache from memory, both because the cache is larger
and because of explosion of additional existence checks that must be
done for the base files.

[YOCTO #13359]

(Bitbake rev: 2ea1e6720c66841baa5e01c523c86f8ecb03eb7d)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-11 13:28:26 +01:00
Richard Purdie
38a6b64f1c bitbake: cooker: Ensure mcdeps are processed even if only one multiconfig
If you have no BBMULTICONFIG set but set mcdepends, they're currently
ignored. We can handle them correctly with this small tweak.

(Bitbake rev: 2aaea2f72e0c37451e7c7851b3a32abd733b38e1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-11 13:28:26 +01:00
Naveen Saini
6e3a4d7926 bitbake: bitbake: fetch2/git: git-lfs check
Build will fail if repository has lfs contents in absense of git-lfs tool on host.
Build will pass if repository may or may not contains lfs content if host has git-lfs installed.

[YOCTO #13198]

(Bitbake rev: 7107c1953b4c949b81491295e03736beb085b055)

Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-15 21:52:57 +01:00
Richard Purdie
0c3423aae9 bitbake: bitbake: Update version to 1.42.0
(Bitbake rev: 00b133af009f9e7c1a4c751b6ef4902f47a81fe2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 13:48:04 +01:00
Andre Rosa
9be9876760 bitbake: utils: Let mkdirhier fail if existing path is not a folder
Let mkdirhier fail if existing path is not a folder instead of assuming a
directory hierarchy already exists.

(Bitbake rev: a8d9b82ccf93dcb74258693f62d88be380b1c0b7)

Signed-off-by: Andre Rosa <andre.rosa@lge.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 13:48:04 +01:00
Andre Rosa
27cbc8c5f0 bitbake: utils: Make prune_suffix prune a suffix
Make prune_suffix prune a suffix instead of replacing a substring that could
happen more than once and not only when it ends with it.

(Bitbake rev: 57e765e38c6382a9b36d5ee2a6f3fa96ac905b82)

Signed-off-by: Andre Rosa <andre.rosa@lge.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 13:45:15 +01:00
Mark Hatle
2030e815bb bitbake: gitsm: Add need_update method to determine when we are going to a new SRCREV
If the system had previously fetched a source repository for use by gitsm,
and then the SRCREV was updated and the new commit already existed, the system
would not re-evaluate the submodules and update them accordingly.

The cause of this issue was that need_update was being used, unmodified, from
the base git fetcher.  It did not have any knowledge, nor did it care if we
were moving commits and needed to re-evaluate what was happening due to this
switch.

To fix the issue, during the download process we add all processed (by
gitsm) srcrevs to the git config file, as bitbake.srcrev.  This allows us to
use a new need_update function that not only checks if the git commit is
present, but if we have previously processed this commit to ensure all of the
submodule components are also present.

This approach is used, instead of iterating over the submodules in need_update
to avoid a potential race condition that has affected us in the past.  The
need_update is called only with the parent locking.  Any time we need to dive
into the submodules, we need to lock, and unlock them, at each stage.  This
opens the possibility of errors in either the code, or unintended race
conditions with rm_work.

This issue was discovered by William A. Kennington III <wak@google.com>.  The
included test case was also written by him, and included unmodified.

(Bitbake rev: 30fe86d22c239afa75168cc5eb262b880886ef8a)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-01 14:48:44 +01:00
Robert Yang
b1641008f0 bitbake: fetch2: Use bb.utils.to_boolean() for BB_NO_NETWORK
Make it consistent with BB_FETCH_PREMIRRORONLY.

(Bitbake rev: 4c6013f47ef36b03f590c909d7c9a2f50b698620)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 14:02:28 +00:00
Robert Yang
64c21baf63 bitbake: fetch2: Unify BB_FETCH_PREMIRRORONLY
The fetch2/__init__.py checks whether "BB_FETCH_PREMIRRORONLY" == "1", but
fetch2/git.py and hg.py checks whether it is None, this makes it discontinuous,
and BB_FETCH_PREMIRRORONLY = "0" doens't work as expected in the later case,
so unify it to the previous one. (As BB_NO_NETWORK does).

And also use bb.utils.to_boolean() to make them consistent.

(Bitbake rev: 85a0d22835588e9ad8ec29d88a8115227e88420c)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 14:02:28 +00:00
Robert Yang
acc242c7ec bitbake: fetch2: Print SCMs list when SRCREV_FORMAT is not set
This makes it easier to debug, especially when multipe SCMs like gitsm,
otherwise we don't know why there are multiple SCMs.

(Bitbake rev: 313fe5e86b254eadfdead706be4bd7b274d5e3c0)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
Robert Yang
db91c23871 bitbake: fetch2/git: Fix clean to remove clonedir
The localpath is a symlink to clonedir when it is cloned from a mirror, for
example:
$ bitbake systemtap-native -cfetch
$ ls downloads/git2
sourceware.org.git.systemtap.git -> /path/to/downloads/git2/mirror.path.git.sourceware.org.git.systemtap.git
mirror.path.git.sourceware.org.git.systemtap.git

There are both sourceware.org.git.systemtap.git and
mirror.path.git.sourceware.org.git.systemtap.git in DL_DIR/git2, the symlink
sourceware.org.git.systemtap.git is created by try_mirror_url(), but
do_cleanall" only removed the symlink, didn't remove the real dir
mirror.path.git.sourceware.org.git.systemtap.git, this may cause confusions,
for example, I assumed that do_cleanall removed everything, but it didn't, and
it would the re-used next time when do_fetch. This patch fixes the problem.

(Bitbake rev: 452e2200ad2c29dec3753f5f7a8cbc9183ec7dd8)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
Robert Yang
4b2b3339df bitbake: cooker: Fix bbfile_config_priorities when BBFILE_PATTERN is empty
The layer was not in bbfile_config_priorities when BBFILE_PATTERN is empty,
this caused "bitbake-layers show-layers" can't show these layers, this was
incorrect since these layer did exist. Add these layer to
bbfile_config_priorities can fix the problem.

Fixed:
Add BBFILE_PATTERN_core = "" in oe-core/meta/conf/layer.conf
$ bitbake show-layers | grep oe-core

There was nothing, now the layer is shown

(Bitbake rev: 0ff5cdb0cca9266ca29127639494bcfd95e36831)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
Robert Yang
a7f3cbb4a2 bitbake: fetch2: runfetchcmd(): Print workdir in debug message
The workdir is an important message when run git command, for exmample, before the patch:
Running git -c core.fsyncobjectfiles=0 branch --contains [snip]

We don't know where it is running, now it is:
Running 'git -c core.fsyncobjectfiles=0 branch --contains [snip]' in /path/to/git.opensvc.com.multipath-tools..git

Which is easier to debug.

(Bitbake rev: f25c05b73218d0e8edf2dd63b36bc09e547c5455)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
Richard Purdie
c96d8de352 bitbake: fetch2: Fix undefined variable issues
Various refactors have left hanging variables, often in debug messages, hence
why they haven't been spotted. Fix them (pylint spotted the issues).

(Bitbake rev: e1f252408982a62485bbf874a01fd30a02aeed32)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00
Richard Purdie
28fb15c56f bitbake: fetch2/wget: Remove pointless lambda function
(Bitbake rev: 329986c3f2d98c4cd1a43b725194003575e718d5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00
Richard Purdie
588a4c58e8 bitbake: fetch2/wget: Clean up whitespace/comments
Clean up the whitespace and comments style, keep pylint happy.

(Bitbake rev: ee59fdaae68543ade03cacfdbbf14fdc7e469412)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00
Richard Purdie
9a1faba274 bitbake: fetch2/wget: Fix typo in proxy handling
Fix a typo in a variable which would affect proxy handling spotted by
pylint.

(Bitbake rev: 19f4ca0119e33df64da629253d5002fcdb385f4f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00
Richard Purdie
202b7d0521 bitbake: fetch2/wget: Cleanup module imports
(Bitbake rev: ef062a3251af474fdeb480149856be3887b8c1c9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00
Stefan Klug
0a9161855c bitbake: fetch2/wget: Fix authentication in checkstatus() of the wget fetcher
I wonder how this used to work for anybody.

(Bitbake rev: 9481e9cd688f635a7fe2ac60d5826ac26f0cb9ba)

Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 15:29:31 +00:00
Alexander Kanavin
984a4def83 bitbake: bitbake: fix version comparison when one of the versions ends in .
Previously, this would happen:

======================================================================
ERROR: test_vercmpstring (bb.tests.utils.VerCmpString)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alexander/development/poky/bitbake/lib/bb/tests/utils.py", line 45, in test_vercmpstring
    result = bb.utils.vercmp_string('1.', '1.1')
  File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 143, in vercmp_string
    return vercmp(ta, tb)
  File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 135, in vercmp
    r = vercmp_part(va, vb)
  File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 124, in vercmp_part
    elif ca < cb:
TypeError: '<' not supported between instances of 'NoneType' and 'int'

----------------------------------------------------------------------

(Bitbake rev: fef56d28c3efec4876c379898cbc4d4c65303aee)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 16:35:33 +00:00
Richard Purdie
4b02d1fa21 bitbake: cooker: Tweak multiconfig dependency resolution
There were a couple of problems with the multiconfig dependency resolution:

- the "if mc" condition triggering this code wasn't correct, it needs
  to be "if more than one multiconfig" configured
- after adding providers we need to call add_unresolved again
  and rebuild mcdeps within the "while new" loop

By fixing these issues we allow various other combinations of multiconfig
builds to work which previously didn't.

[YOCTO #13090]
[YOCTO #13130]

(Bitbake rev: 4359b037de578095db2595f119dfb8e3340e1414)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 10:43:31 +00:00
Richard Purdie
3bf0602713 bitbake: siggen: Fix multiconfig corner case
There was already a fix to ignore some multiconfig dependencies but its
'opposite' case wasn't covered. Cover that combination to so as to avoid
tracebacks in multiconfig builds.

[YOCTO #13090]
[YOCTO #13130]

(Bitbake rev: c883dfe378af9dfc192a8e392e84325d68648806)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 10:43:31 +00:00
Richard Purdie
0b19e52b85 bitbake: runqueue: Filter out multiconfig dependencies from BB_TASKDEPDATA
The consumers of BB_TASKDEPDATA in OE metadata can't cope with multiconfig
dependencies. The choice is either to start adding code to each of them to
filter out multiconfig dependencies, or do this at source.

After consideration we've decided to do this at source as doing otherwise
is code duplication and error prone and in any case we've looked at, they
don't make sense.

[YOCTO #13090]
[YOCTO #13130]

(Bitbake rev: 531dcd221a10853f45cc057b52bb2d5083e0ee42)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 10:43:31 +00:00
Richard Purdie
8dac3ce424 bitbake: runqueue: Fix dependency loop analysis 'hangs'
Currently the mechanism for breaking out of the dependnecy loop analysis
code is broken and doesn't work leading to bitbake appearing to hang.

Add in a custom exception for this purpose and fix the code to exit
as intended, fixing the hang and making the dependency loop code
usable again.

(Bitbake rev: 8756e4ade67c16e35269ea0659e10b9ebaa6117f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-17 10:20:21 +00:00
Joshua Watt
9025bd11ec bitbake: bitbake: Allow arguments in FAKEROOTCMD
Changes FAKEROOTCMD so that it can accept additional arguments to pass
to the fakeroot implementation instead of being treated as a simple
command

(Bitbake rev: 4fa51afb56b090cf1f746842acd602c9536715d5)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-15 08:18:19 +00:00
Robert Yang
5fa3b5b152 bitbake: bitbake: cookerdata: Check duplicated BBFILE_COLLECTIONS
It shouldn't work when there are duplicated BBFILE_COLLECTIONS.

(Bitbake rev: a1f251e5ab859d6d4a2cb908408d4ddcab5a5de1)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-14 11:21:58 +00:00
Alexander Kanavin
2f17b87f97 bitbake: build.py: add unhandled exception information to the logger when executing python tasks
Previously this information was simply discarded, which in some cases
resulted in generic, unhelpful failures.

With this change the user would see what the exception is, but
without ugly tracebacks or bulky, irrelevant information from the log file.

(Bitbake rev: 8da9242702fbef8cf3156b95a1076802e0f653c7)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-14 11:21:58 +00:00
André Draszik
2fcbc079e2 bitbake: bitbake: remove True option to getVar calls
getVar() has been defaulting to expanding by default for
a long time (2016), thus remove the True option from
getVar() calls with a regex search and replace.

Search & replace made using the following command:
    sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \
        -i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \
             | cut -d':' -f1 \
             | sort -u)

(Bitbake rev: 3bba0dbd524cf72176a765957adff544ae5c255a)

Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-12 22:01:10 +00:00
Richard Purdie
65a7662c5b bitbake: server/process: Add missing exception raise
The intent of the code was to catch one kind of error, it was actually swallowing
all exceptions and looping indefinitely. Fix it to work as intended.

This explains some mystery hangs we've been seeing.

(Bitbake rev: d73dbc3580faa1225d95ae4cefac4879ca3c1b2f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 08:24:50 +00:00
Robert Yang
2c15d8000d bitbake: bitbake: ConfHandler: Don't strip leading spaces
Fixed:
- Add the following lines to conf/local.conf:
  FOO = "BAR1"
  FOO_append = "\
      BAR2"

  $ bitbake -e | grep '^FOO'
  FOO="BAR1BAR2"

  The leading spaces in the second line have been removed.

- But if add the previous two lines to base.bbclass:
  $ bitbake -e | grep '^FOO'
  FOO="BAR1    BAR2"

  The leading spaces in the second line are preserved, this is inconsistent, now
  fix ConfHandler to preserve leading spaces.

[YOCTO #12380]

(Bitbake rev: 8c3bc15a7b5e0a81d7b6c9d3fe43fbff63207156)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 08:24:50 +00:00
Mark Hatle
a7774aced0 bitbake: gitsm: The fetcher did not process some recursive submodules properly.
Move the submodule processing outside of the if statement to avoid any
optimizations that may happen.

Update the test cases to include the additional case, and split the other
test cases into individual tests to make it easier to figure out what
the failure may be.

(Bitbake rev: 0ec98c01ae50f95c9c74acf53013ac59e0e72b08)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-28 23:14:27 +00:00
Mark Hatle
1979d9162a bitbake: gitsmy.py: Fix unpack of submodules of submodules
If the submodule is in a subdirectory, it needs to have that structure
preserved.  This means the unpack path needs to be in the 'dirname' of the
final path -- since the unpack directory name is specified in the URI.

Additional specific test cases were added to ensure this is working properly
based on two recent error reports.

(Bitbake rev: 8c8ecec2a722bc2885e2648d41ac8df07bdf660d)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-24 17:45:49 +00:00
Mark Hatle
9238a72589 bitbake: gitsm.py: Fix relative URLs
Prior code happened to work with relative URLs, when the code was recently
restructured it caused all relative urls to no longer work.  Restore the prior
code flow for relative support and better comment why that code is there.

(Bitbake rev: 14bb383af0ca98e0e04ec217e537b6a899f3658e)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-24 17:45:49 +00:00
Christopher Larson
6fd870e6a1 bitbake: bb.tests.codeparser: add parameter expansion modifiers test
We don't want references including shell parameter expansion modifiers
(i.e. `:-`, `#`, `%%`, etc) to be added to our vardeps, so add a test to
ensure this.

YOCTO #12987

(Bitbake rev: be022085fe1ea1b9a9d519f0455883e2da363d2c)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-21 23:44:34 +00:00