Commit Graph

158 Commits

Author SHA1 Message Date
Luca Boccassi
d8551bd029 bitbake: tests/fetch.py: add missing skipIfNoNetwork tags to tests that try to git clone
test_git_submodule_dbus_broker, test_git_submodule_CLI11, test_git_submodule_update_CLI11,
test_git_submodule_aktualizr and test_git_submodule_iotedge try to access the network via
git clone, which fails when there is no network available. Add the relevant skip tag.

(Bitbake rev: 9b0538753da0514e6518723dac537007abf7a649)

Signed-off-by: Luca Boccassi <luca.boccassi@microsoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-28 13:29:04 +01:00
Martin Jansa
50d272863d bitbake: tests/utils.py: add one more test cases for bb.utils.vercmp_string
* this is just another test case for issue already fixed in:
    commit fef56d28c3efec4876c379898cbc4d4c65303aee
    Author: Alexander Kanavin <alex.kanavin@gmail.com>
    Date:   Sun Feb 24 21:07:28 2019 +0100
    Subject: bitbake: fix version comparison when one of the versions ends in .

* The TypeError is triggered not by '.' at the end, but from the extra
  numberic component in one of the versions.

* When one version has fewer elements, it's extended by another (0, None)
  element where 0 means numeric component. Then the result cannot be
  decided by comparing the types (oa < ob, ob > oa) and it continues
  to compare values (ca < cb) which fails when one of them is the None
  from (0, None) appended before.

  ======================================================================
  ERROR: test_vercmpstring (bb.tests.utils.VerCmpString)
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "lib/bb/tests/utils.py", line 32, in test_vercmpstring
      result = bb.utils.vercmp_string('1a', '1a1')
    File "lib/bb/utils.py", line 131, in vercmp_string
      return vercmp(ta, tb)
    File "lib/bb/utils.py", line 123, in vercmp
      r = vercmp_part(va, vb)
    File "lib/bb/utils.py", line 112, in vercmp_part
      elif ca < cb:
  TypeError: '<' not supported between instances of 'NoneType' and 'int'

  ----------------------------------------------------------------------
  Ran 3 tests in 0.002s

(Bitbake rev: 9767fffe3115a1f1afa3c6a2b39720fefb8dc4d5)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-21 00:33:32 +01:00
Mark Hatle
a1c1f3f7c5 bitbake: svn.py: Stop SVN from directly pulling from an external layer w/o fetcher
Add a new option to the svn fetcher url "externals=allowed".  This will allow
a user to enable svn co w/ externals.  However, this does avoid the fetcher,
network access and mirror systems.

By default we no longer allow externals in the checkout.  This ensures a
deterministic download.  The system does attempt to identify SVN repos that
have externals enabled, and will warn the user.  It is up to the user to
determine if these are necessary for the recipe.  They may disable the warning
by adding "externals=nowarn" to the url.

In the future we would like to parse this list and see if the items are already
in the SRC_URI for that recipe, but with SVN being in limited use these days
that extra work is likely not worth the trouble.

Add test cases that generated a local SVN tree, with an external source
set to github bitbake in svn format.  One test case checks that externals are
ignored, and one checks that they in downloaded.

(Bitbake rev: bf53f07c3647e57d8452a7743a2b04bcb72c80d6)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-21 12:56:34 +01:00
Mark Hatle
7a73a435e0 bitbake: gitsm: Fix a bug where the wrong path was used for the submodule init
Because we are trying to avoid network activity and use our own fetcher,
the system emulates the behavior of 'git submodule init'.

git submodule init uses the .gitmodules file, where typically the module
name and path are the same.  However, in this case the module name and
path (in the tree) were different. i.e.:

[submodule "edgelet/hsm-sys/azure-iot-hsm-c/deps/azure-c-shared-utility"]
        path = edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared
        url = https://github.com/Azure/azure-c-shared-utility.git

Previously the code assumed the 'path' was both the checkout location
under .git/modules, as well as the path to extract the components.  This
proved to be incorrect as the .git/modules path needs to match the submodule
'name'.  This causes the components that were fetched to be initialized in
the wrong location, which later caused the 'git submodule update' process to
skip not properly initialized modules.

A test case was added for this specific case to ensure a regression does
not appear in the future.

(Bitbake rev: fd27ab60d33553dba13de39394edaaac04e446b3)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-16 09:17:15 +01:00
Richard Purdie
9501864db8 bitbake: bitbake: Strip old editor directives from file headers
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.

(Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
cf9c0be3f6 bitbake: bitbake: Drop duplicate license boilerplace text
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.

(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
79834a7144 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:04 +01:00
Richard Purdie
a85601aae1 bitbake: build: Disable warning about dependent tasks for now
This breaks with rm_work so disable the warning until we find a better
solution (and change the test accordingly too).

(Bitbake rev: 93e94c06baf013e3d072465a55bddd1fe61c0772)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-01 23:09:08 +01:00
Richard Purdie
dabc5ae13a bitbake: build: Ensure warning for invalid task dependencies is useful
WARNING: elfutils: dependent task do_rm_work does not exist

is much less useful than

WARNING: elfutils: dependent task do_rm_work for do_deploy does not exist

(Bitbake rev: e034c6f75e3d7730ff16a8d1bd0cba03beda0af8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-01 23:09:08 +01:00
Robert Yang
c761fb126d bitbake: tests/parse.py: Add testcase for addtask and deltask
(Bitbake rev: 4ac388646624e08bef848b560fa52deacf2ff4fb)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-30 12:05:24 +01:00
Mark Hatle
2030e815bb bitbake: gitsm: Add need_update method to determine when we are going to a new SRCREV
If the system had previously fetched a source repository for use by gitsm,
and then the SRCREV was updated and the new commit already existed, the system
would not re-evaluate the submodules and update them accordingly.

The cause of this issue was that need_update was being used, unmodified, from
the base git fetcher.  It did not have any knowledge, nor did it care if we
were moving commits and needed to re-evaluate what was happening due to this
switch.

To fix the issue, during the download process we add all processed (by
gitsm) srcrevs to the git config file, as bitbake.srcrev.  This allows us to
use a new need_update function that not only checks if the git commit is
present, but if we have previously processed this commit to ensure all of the
submodule components are also present.

This approach is used, instead of iterating over the submodules in need_update
to avoid a potential race condition that has affected us in the past.  The
need_update is called only with the parent locking.  Any time we need to dive
into the submodules, we need to lock, and unlock them, at each stage.  This
opens the possibility of errors in either the code, or unintended race
conditions with rm_work.

This issue was discovered by William A. Kennington III <wak@google.com>.  The
included test case was also written by him, and included unmodified.

(Bitbake rev: 30fe86d22c239afa75168cc5eb262b880886ef8a)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-01 14:48:44 +01:00
Alexander Kanavin
984a4def83 bitbake: bitbake: fix version comparison when one of the versions ends in .
Previously, this would happen:

======================================================================
ERROR: test_vercmpstring (bb.tests.utils.VerCmpString)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alexander/development/poky/bitbake/lib/bb/tests/utils.py", line 45, in test_vercmpstring
    result = bb.utils.vercmp_string('1.', '1.1')
  File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 143, in vercmp_string
    return vercmp(ta, tb)
  File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 135, in vercmp
    r = vercmp_part(va, vb)
  File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 124, in vercmp_part
    elif ca < cb:
TypeError: '<' not supported between instances of 'NoneType' and 'int'

----------------------------------------------------------------------

(Bitbake rev: fef56d28c3efec4876c379898cbc4d4c65303aee)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25 16:35:33 +00:00
André Draszik
2fcbc079e2 bitbake: bitbake: remove True option to getVar calls
getVar() has been defaulting to expanding by default for
a long time (2016), thus remove the True option from
getVar() calls with a regex search and replace.

Search & replace made using the following command:
    sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \
        -i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \
             | cut -d':' -f1 \
             | sort -u)

(Bitbake rev: 3bba0dbd524cf72176a765957adff544ae5c255a)

Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-12 22:01:10 +00:00
Mark Hatle
a7774aced0 bitbake: gitsm: The fetcher did not process some recursive submodules properly.
Move the submodule processing outside of the if statement to avoid any
optimizations that may happen.

Update the test cases to include the additional case, and split the other
test cases into individual tests to make it easier to figure out what
the failure may be.

(Bitbake rev: 0ec98c01ae50f95c9c74acf53013ac59e0e72b08)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-28 23:14:27 +00:00
Mark Hatle
1979d9162a bitbake: gitsmy.py: Fix unpack of submodules of submodules
If the submodule is in a subdirectory, it needs to have that structure
preserved.  This means the unpack path needs to be in the 'dirname' of the
final path -- since the unpack directory name is specified in the URI.

Additional specific test cases were added to ensure this is working properly
based on two recent error reports.

(Bitbake rev: 8c8ecec2a722bc2885e2648d41ac8df07bdf660d)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-24 17:45:49 +00:00
Mark Hatle
9238a72589 bitbake: gitsm.py: Fix relative URLs
Prior code happened to work with relative URLs, when the code was recently
restructured it caused all relative urls to no longer work.  Restore the prior
code flow for relative support and better comment why that code is there.

(Bitbake rev: 14bb383af0ca98e0e04ec217e537b6a899f3658e)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-24 17:45:49 +00:00
Christopher Larson
6fd870e6a1 bitbake: bb.tests.codeparser: add parameter expansion modifiers test
We don't want references including shell parameter expansion modifiers
(i.e. `:-`, `#`, `%%`, etc) to be added to our vardeps, so add a test to
ensure this.

YOCTO #12987

(Bitbake rev: be022085fe1ea1b9a9d519f0455883e2da363d2c)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-21 23:44:34 +00:00
Mark Hatle
7d715ae071 bitbake: gitsm.py: Refactor the functions and simplify the class
The update_submodules and unpack_submodules functions were nearly indentical,
so we made a common function where the different behavior could be passed
in by the download and unpack users.  The new function is process_submodules.

Moved the parse_gitmodules function under the new process_submodules, since
there are no external callers.

Refactor the file relative path processing to the URL translation code.
We also add a warning to the translation if a relative ssh URL has been
detected.  Since this can cause a problem.

In the case of a relative URL that does not work after being translated,
it should be possible to use the MIRROR functions to manual translate the
generated relative URL into one that works properly.

Remove 'git config' processing on download contents.  It turns out this is not
necessary since all of the later components work using the git fetcher.

Limit the 'git submodule update' call to only when unpacking a non-bare
repository.  Submodules are always loaded as bare, so this prevents
intermediate unpacks from being attempted.

Finally, the test cases were updated and the new commit ids in the test
repository were updates as well.

(Bitbake rev: 610dbee5634677f5055e2b36a3043cd197fb8c51)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:08 +00:00
Mark Hatle
a9cf611e7a bitbake: gitsm.py: Rework the shallow fetcher and test case
A custom shallow submodule is no longer necessary, as the regular git
fetcher is used and shallow handling works with the same code.

The only general difference between the regular change is simply declaring a
clone as shallow, when appropriate.

This also removes a potential race condition in copying repositories
vs cloning them.

The gitsm shallow fetcher test was revised to verify that the submodule
is shallow cloned along with the primary repository.

The first step of this change was to be sure to clean the gitsubmodule download
directory, as was previously done with the may gitsource directory.

Additional test components were added to verify commit counts, and an
obsolete (and likely incorrect) test for the .git/modules directory to be
empty was also removed.

(Bitbake rev: f9cc4684dcf4281acc557cda8cb35602354ac3d6)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:08 +00:00
Mark Hatle
e46947e17b bitbake: tests/fetch.py: Add alternative gitsm test case
In order to test the ssh processing in gitsm, we add an alternative
testcase that can be downloaded from git.yoctoproject.org.  However,
this test case requries (read) access, via ssh, to git.yoctoproject.org.

(Bitbake rev: c8554cdc1287620fe8e8960561e614567879a010)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:08 +00:00
Richard Purdie
94b37d134d bitbake: tests.data: Add underscore+numeric in overrides datastore test
Add a test for x86_64 in overrides where is was being incorrectly handled.
There was a previous fix (3a3be518536acc868c7eeb3c1111ad1b321480b7) but
this ensures we don't regress.

(Bitbake rev: 08c314eac231ac9292c8c95f1c5a6fc2023fe749)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-08 11:16:03 +00:00
Joshua Watt
0b8b2f1565 bitbake: tests/persist_data: Add tests
Adds a test suite for testing the persistent data cache

[YOCTO #13030]

(Bitbake rev: 96a4155049e834af17069d981cc2215e50d18c1a)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-26 11:10:33 +00:00
Richard Purdie
0c5e19c96d bitbake: tests/fetch: Update after recent gitsm message changes
Also use assetIn instead of assertTrue which aids debugging failures.

(Bitbake rev: 19dee675bb9ad012d28e1e57a888931355a831cb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-22 12:01:59 +00:00
Richard Purdie
a9dc2ac9e0 bitbake: data_smart: Fix removal handling interaction issue with overrides
If a variable has a _remove applied to it but that variable is in turn
'renamed' through OVERRIDES, the removal gets lost with the current code.

TEST = "foo"
TEST_someval = "bar"
TEST_someval_remove = "bar"
OVERRIDES = "someval"

currently gives "bar" for TEST but should give "".

This fixes the code to track the removal and adds a test case to ensure this
doesn't regress again.

(Bitbake rev: 8f55010c18057be040f073d8bcb4c5c2c311d809)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-20 14:27:03 +01:00
Richard Purdie
4ef016683d bitbake: test/data: Add new tests for task checksum changing/not changing
This adds some basic tests for task checksums to ensure that the
checksums:

* change when variables change
* change when active _remove operators are present
* don't change when the _remove operators are not active
* change when an active contains() expression is present
* dont' change a contains() expression isn't active

There is a lot of other functionality which should be added to this
test but its a start.

(Bitbake rev: 5463c16e3619d324aed137f47f93f0997a227d29)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:27 +01:00
Urs Fässler
a8a077cafb bitbake: fetch2/git: provide information about missing sources
Provide more information in the case the sources are not found in the
unpack step.

(Bitbake rev: 27a2214bf6f2e7c61bfc422a20959a55f7e0d25d)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:27 +01:00
Urs Fässler
0d4fe0602e bitbake: fetch2/git: prevent access to non-existing clonedir
A user friendly error is throw when neither the clonedir nor
fullshallow exist. Without the check, a difficult to interpret error
is throw from within the fetch command.

(Bitbake rev: 30cf2506007d25162f0805051212f54c39034ff3)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:26 +01:00
Urs Fässler
9a058e4421 bitbake: fetch2/git: use intention revealing names for premirror tests
(Bitbake rev: 3434e64e7c077c1ecff7b36f02b6c6b59a7d1fe9)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:26 +01:00
Urs Fässler
36fcd5eda9 bitbake: fetch2/git: add tests to capture existing behavior wrt. naming of git shallow tarball
The mapping of the URLs to the local shallow tarballs is not obvious. For
easier understanding, we add this tests to explicitly showing the mapping.

(Bitbake rev: 05f2ac8e19a027d103921b5ae0070db609a83042)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:26 +01:00
Urs Fässler
b6a972922b bitbake: fetch2/git: add tests to capture existing behavior wrt. naming of mirror tarball
The mapping of the URLs to the local tarballs is not obvious. For easier
understanding, we add this tests to explicitly showing the mapping.

(Bitbake rev: c604d9402b4c9ad7eb5c794ab24f2f348d9719a9)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:26 +01:00
Urs Fässler
ffac9c20f5 bitbake: fetch2/git: add tests to capture existing behavior wrt. naming of clone directories
The mapping of the URLs to the local directory is not obvious. For easier
understanding, we add this tests to explicitly showing the mapping.

(Bitbake rev: 5f92682389fee437d6df2ff7718c571b7444e179)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:26 +01:00
Richard Purdie
a9aa1d24c7 bitbake: data: Fix whitespace on _remove operations
We have some slightly odd behaviours with the current implementation of
_remove operations. For example:

TEST = " A B"
TEST_remove = "C"

would trigger TEST to become "A B" even thought it doesn't contain "C".

In particular, this means that an inactive remove operator added in a
bbappend could change the task checksum which is not desireable.

Fix the operation to preserve whitespace, adding new tests to make this
explict and test further corner cases. Also update the manual to match.

(Bitbake rev: c0a23dd9155c50a6b7df796980bc7b612cac7994)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:26 +01:00
Urs Fässler
b907303b92 bitbake: fetch2/git: prevent access to non-existing fullshallow tarball
(Bitbake rev: b7f00a8c11672a2ee0408e210fb174cda3384e3f)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04 23:00:24 +01:00
Ross Burton
9a47c067a6 bitbake: test/fetch: use canonical GitLab URLs
GitLab returns a redirect from /foo to /foo.git which older releases of Git
don't handle when cloning.  These tests don't clone so the redirect works, but
let's be consistant with the structure of these URLs and add the .git suffix.

(Bitbake rev: d47febbae4ff24be259037f12bafbc14b9e2d6a8)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04 14:22:00 +01:00
Mark Hatle
5cd00e3e53 bitbake: fetch2/gitsm.py: Rework the git submodule fetcher
The prior fetcher did not know how to work with MIRRORS, and did not
honor BB_NO_NETWORK and similar.

The new fetcher approach recursively calls 'gitsm' download on each
submodule detected.  This ensures that it will go throug the
standard download process.

Each downloaded submodule is then 'attached' to the original download in
the 'modules' directory.  This mimics the behavior of:

    git submodule init

but there is no chance it will contact the network without permission.

It then corrects upstream reference URIs.

The unpack steps simply copies the items from the downloads to the destdir.
Once copied the submodules are connected and we then run:

    git submodule update

According to the git documentation, git submodule init can and will modify
the project configuration and may connect to the network.  Doing the
work manually prevents this.  (This manual process is allowed based
on my reading of the documentation.)

See: https://git-scm.com/book/en/v2/Git-Tools-Submodules

The small change to the existing test is due to this new code always assuming
the code is from a remote system, and not a 'local' repository.  If this
assumption proves to be incorrect -- code will need to be added to deal
with local repositories without an upstream URI.

(Bitbake rev: 9c6b39adf9781fa6745f48913a97c859fa37eb5b)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-26 15:14:33 +01:00
Mark Hatle
5fae3cb56e bitbake: lib/bb/tests/parse.py: Test case was changing chdir
The test case was changing the current directory, but was never restoring it
to the original location.  This causes occasional failures in later test cases.

(Bitbake rev: 8c222c45148d1f21c2390d66ddd9d3e33b397f05)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-26 15:14:33 +01:00
Mark Hatle
b7e26bedc2 bitbake: cooker.py: Fix incorrect bb files matched warning
In the case of a sublayer of an existing layer, where the sublayer and
main layer share a path, the system may not match the paths properly resulting
in:

    No bb files matched BBFILE_PATTERN_sublayer '^/path/main/sublayer'

because it has already matched the main layer.

Fix this issue by sorting the collection items based on the pattern, using
longest to shortest.  Obviously regex wildcards could still be an issue
but these are typically not used, so this simply fix should work in the
existing cases.

(Bitbake rev: 1787cef7221b88f6920ea70fadaffc117d84c7aa)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-05 18:01:37 +01:00
Richard Purdie
4434777705 bitbake: tests/fetch: Update gnome.org urls after upstream changes
(Bitbake rev: 43ff74d675baed13fc17e1d12c0e013b16ba249b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-29 10:43:23 +01:00
Nikolai Merinov
070a06ec53 bitbake: fetch2: Use correct check in parameter handling in uri_replace()
If uri_find contain parameters then original URI parameters should
be checked against parameters from uri_find instead of parameters
from uri_replace.

(Bitbake rev: 8efa7826a61501589afa33eb698c0ab3a622bf2e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-21 09:34:40 +01:00
Jakub Dębski
ef7222c6d1 bitbake: fetch2/__init__.py: Fixed handling of uris with empty path
For mirrors or premirrors defined like: "http://.*/.* http://somewhere.org"
fetching ends with errors because function fetch2/__init__.py:encodeurl()
creates url like "http://somewhere.orgsomefile.tar.gz".

It happens because function fetch2/__init__.py:decodeurl()
for url "http://somewhere.org" returns
['http', 'somewhere.org', '', '', '', {}]
and then in function fetch2/__init__.py:uri_replace()
variable result_decode will be
['http', 'somewhere.org', 'somefile.tar.gz', '', '', {}]
(because of line: result_decoded[loc] = os.path.join(result_decoded[loc], basename))
for which encodeurl returns "http://somewhere.orgsomefile.tar.gz".

In addition for mirror "http://.*/.* http://somewhere.org/"
everything works fine.

(Bitbake rev: d822ae24ef5485e550804cbd9130ebd73b2aa48e)

Signed-off-by: Jakub Dębski <jdebski@enigma.com.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-22 13:13:19 +01:00
Ross Burton
d04792b2ed bitbake: tests/fetch: state which upstream failed
(Bitbake rev: 21098de09ee2f7a9f0b3f895bf2ffbdeb8c9ded5)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-28 12:51:52 +01:00
Nathan Rossi
f2d5967982 bitbake: fetch2: Handle missing donestamp file when content is valid
In order to allow users to manually populate the download directory with
valid content change the assumption that missing the donestamp file
means unfetched content.

This allows users to populate the download dir without needing to create
dummy .done files such that a user does not need a PREMIRROR when using
BB_NO_NETWORK to provide valid content files in the download directory.

To ensure the correct result this change also fails first if the
localpath does not exist. This prevents further parts of the function
attempting to calculating the checksum on non-existent files. This also
fixes some edge conditions around where if the donestamp exists but the
localpath does not it returns, and did not remove the donestamp.

Also added test cases to cover this use case and additional use cases
where for example the fetcher does not support checksums.

(Bitbake rev: a335dbbb65d5b56e71d98cf3e4fa9bfbec1dcde6)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-14 15:26:03 +00:00
Nathan Rossi
9c44aee83d bitbake: fetch/git: Use 'git-make-shallow' from bin directory
Move the code that existed in tests/fetch.py for determining the path to
'git-make-shallow' into the git module and reference it.

This ensures that 'git-make-shallow' is always available and the desired
version regardless of the path variable or whether git exposes the
command.

(Bitbake rev: 6b508ab8fd5aa796c1c00c970e81e5e93f84d35d)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-14 15:26:03 +00:00
Richard Purdie
1103a03742 bitbake: tests/fetch: Allow bit/bitbake-selftest to function correctly
Without this the paths to git-make-shallow are incorrect and cause test
failures if bitbake isn't executed from cwd or PATH.

(Bitbake rev: 643eacb162b8710330ef292bfda21cfeab97f95c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-07 12:14:34 +00:00
Jair Gonzalez
e95ce3fe3b bitbake: tests/event: tests: Add event classes test class
This change adds EventClassesTest class to bb/tests/event.py,
including 47 new test cases for the public interfaces of the
bitbake event and related helper classes.

[YOCTO #10773]

(Bitbake rev: ee5fe4431713b8a29bdb424a29460965374b3234)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-07 12:14:34 +00:00
Jair Gonzalez
35226b3307 bitbake: tests/event: Remove empty assignments from event tests
Remove assignments from non-returning calls.

(Bitbake rev: a8cac827dd15227a3940ea25c673d91b5e2c2a75)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-07 12:14:34 +00:00
Jair Gonzalez
fda60b31ec bitbake: tests/event: Add test for bb.event.worker_fire
Test the triggering of bb.event.worker_fire callback.

(Bitbake rev: daa59a2057c811b20d75235526ac6c2079ac6e10)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-07 12:14:34 +00:00
Jair Gonzalez
61f409baff bitbake: tests/event: Include tests for event filtering features
This change includes unit tests for the following functions,
helper class and methods in bitbake.event:

- set_eventfilter(func)
- set_UIHmask(handlerNum, level, debug_domains, mask)
- getName(e)
- class UIEventFilter(object)
    def __init__(self, level, debug_domains)
    def update(self, eventmask, level, debug_domains)
    def filter(self, event)

[YOCTO #10773]

(Bitbake rev: 4a19dde704fd0bf262ea991ef530f991a4897d31)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-07 12:14:34 +00:00
Richard Purdie
2e1eb9ab12 bitbake: tests/fetch: Add ftp test url
Add in a tets ftp url so we ensure ftp urls contnue to work after the loss
of the ftp.gnu.org ones.

(Bitbake rev: e1e8565b5e19dd3f7ef6e7e41932456adaa3df81)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-07 15:02:19 +00:00
Ross Burton
30b25d216e bitbake: tests/fetch: use subtests in the wget tests
As we test multiple URLs in this these tests and one failing abandons the test,
use subtests so all URLs are tested. This should help us identify patterns in
the failing URLs.

(Bitbake rev: c4c4465b32e82d4b6e46a44e776be5039aef6b18)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-07 15:02:19 +00:00