Commit Graph

598 Commits

Author SHA1 Message Date
Matthias Schiffer
482da97cfc bitbake: fetch2: runfetchcmd(): unset _PYTHON_SYSCONFIGDATA_NAME
Since warrior, python3native.bbclass sets _PYTHON_SYSCONFIGDATA_NAME;
unfortunately, this also affects Python scripts run as fetch commands like
git-make-shallow, breaking it with a message like

    Failed to import the site module
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site.py", line 570, in <module>
        main()
      File "/usr/lib/python3.6/site.py", line 556, in main
        known_paths = addusersitepackages(known_paths)
      File "/usr/lib/python3.6/site.py", line 288, in addusersitepackages
        user_site = getusersitepackages()
      File "/usr/lib/python3.6/site.py", line 264, in getusersitepackages
        user_base = getuserbase() # this will also set USER_BASE
      File "/usr/lib/python3.6/site.py", line 254, in getuserbase
        USER_BASE = get_config_var('userbase')
      File "/usr/lib/python3.6/sysconfig.py", line 607, in get_config_var
        return get_config_vars().get(name)
      File "/usr/lib/python3.6/sysconfig.py", line 550, in get_config_vars
        _init_posix(_CONFIG_VARS)
      File "/usr/lib/python3.6/sysconfig.py", line 421, in _init_posix
        _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
    ModuleNotFoundError: No module named '_sysconfigdata'

on an Ubuntu 18.04 system (and likely others) when building with
BB_GIT_SHALLOW and BB_GENERATE_SHALLOW_TARBALLS.

Unset _PYTHON_SYSCONFIGDATA_NAME in runfetchcmd() to work around this.

(Bitbake rev: d94ccd506d04aff182ab48f501f6f366d5dd14f5)

Signed-off-by: Matthias Schiffer <matthias.schiffer@ew.tq-group.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-30 12:37:03 +01:00
Jean-Marie LEMETAYER
a35abe31dc bitbake: bitbake: fetch2/npm: fix npw view parsing
Fixes [YOCTO #13344]

When parsing manually the 'npm view --json' ouput, an extra closing
brackets in a JSON string can leads the fetcher to fail with a
JSONDecodeError exception.

This commit use the JSON parser to extract:
 - The last object in the returned array if there are multiple results.
 - The returned object if there is only one result.

(Bitbake rev: 3d319c79981811d3cfd4732885057db4fd5afcc2)

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-21 12:56:34 +01:00
Mark Hatle
a1c1f3f7c5 bitbake: svn.py: Stop SVN from directly pulling from an external layer w/o fetcher
Add a new option to the svn fetcher url "externals=allowed".  This will allow
a user to enable svn co w/ externals.  However, this does avoid the fetcher,
network access and mirror systems.

By default we no longer allow externals in the checkout.  This ensures a
deterministic download.  The system does attempt to identify SVN repos that
have externals enabled, and will warn the user.  It is up to the user to
determine if these are necessary for the recipe.  They may disable the warning
by adding "externals=nowarn" to the url.

In the future we would like to parse this list and see if the items are already
in the SRC_URI for that recipe, but with SVN being in limited use these days
that extra work is likely not worth the trouble.

Add test cases that generated a local SVN tree, with an external source
set to github bitbake in svn format.  One test case checks that externals are
ignored, and one checks that they in downloaded.

(Bitbake rev: bf53f07c3647e57d8452a7743a2b04bcb72c80d6)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-21 12:56:34 +01:00
Mark Hatle
7a73a435e0 bitbake: gitsm: Fix a bug where the wrong path was used for the submodule init
Because we are trying to avoid network activity and use our own fetcher,
the system emulates the behavior of 'git submodule init'.

git submodule init uses the .gitmodules file, where typically the module
name and path are the same.  However, in this case the module name and
path (in the tree) were different. i.e.:

[submodule "edgelet/hsm-sys/azure-iot-hsm-c/deps/azure-c-shared-utility"]
        path = edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared
        url = https://github.com/Azure/azure-c-shared-utility.git

Previously the code assumed the 'path' was both the checkout location
under .git/modules, as well as the path to extract the components.  This
proved to be incorrect as the .git/modules path needs to match the submodule
'name'.  This causes the components that were fetched to be initialized in
the wrong location, which later caused the 'git submodule update' process to
skip not properly initialized modules.

A test case was added for this specific case to ensure a regression does
not appear in the future.

(Bitbake rev: fd27ab60d33553dba13de39394edaaac04e446b3)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-16 09:17:15 +01:00
Richard Purdie
9501864db8 bitbake: bitbake: Strip old editor directives from file headers
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.

(Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
cf9c0be3f6 bitbake: bitbake: Drop duplicate license boilerplace text
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.

(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
79834a7144 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:04 +01:00
Naveen Saini
e7c2bc67b5 bitbake: bitbake: fetch2/git: git-lfs check
Build will fail if repository has lfs contents in absense of git-lfs tool on host.
Build will pass if repository may or may not contains lfs content if host has git-lfs installed.

[YOCTO #13198]

(Bitbake rev: 3f2c2eb2f59707828bdcdd6414db837da8dc3b0e)

Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-29 14:16:31 +01:00
Mark Hatle
2030e815bb bitbake: gitsm: Add need_update method to determine when we are going to a new SRCREV
If the system had previously fetched a source repository for use by gitsm,
and then the SRCREV was updated and the new commit already existed, the system
would not re-evaluate the submodules and update them accordingly.

The cause of this issue was that need_update was being used, unmodified, from
the base git fetcher.  It did not have any knowledge, nor did it care if we
were moving commits and needed to re-evaluate what was happening due to this
switch.

To fix the issue, during the download process we add all processed (by
gitsm) srcrevs to the git config file, as bitbake.srcrev.  This allows us to
use a new need_update function that not only checks if the git commit is
present, but if we have previously processed this commit to ensure all of the
submodule components are also present.

This approach is used, instead of iterating over the submodules in need_update
to avoid a potential race condition that has affected us in the past.  The
need_update is called only with the parent locking.  Any time we need to dive
into the submodules, we need to lock, and unlock them, at each stage.  This
opens the possibility of errors in either the code, or unintended race
conditions with rm_work.

This issue was discovered by William A. Kennington III <wak@google.com>.  The
included test case was also written by him, and included unmodified.

(Bitbake rev: 30fe86d22c239afa75168cc5eb262b880886ef8a)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-01 14:48:44 +01:00
Robert Yang
b1641008f0 bitbake: fetch2: Use bb.utils.to_boolean() for BB_NO_NETWORK
Make it consistent with BB_FETCH_PREMIRRORONLY.

(Bitbake rev: 4c6013f47ef36b03f590c909d7c9a2f50b698620)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 14:02:28 +00:00
Robert Yang
64c21baf63 bitbake: fetch2: Unify BB_FETCH_PREMIRRORONLY
The fetch2/__init__.py checks whether "BB_FETCH_PREMIRRORONLY" == "1", but
fetch2/git.py and hg.py checks whether it is None, this makes it discontinuous,
and BB_FETCH_PREMIRRORONLY = "0" doens't work as expected in the later case,
so unify it to the previous one. (As BB_NO_NETWORK does).

And also use bb.utils.to_boolean() to make them consistent.

(Bitbake rev: 85a0d22835588e9ad8ec29d88a8115227e88420c)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 14:02:28 +00:00
Robert Yang
acc242c7ec bitbake: fetch2: Print SCMs list when SRCREV_FORMAT is not set
This makes it easier to debug, especially when multipe SCMs like gitsm,
otherwise we don't know why there are multiple SCMs.

(Bitbake rev: 313fe5e86b254eadfdead706be4bd7b274d5e3c0)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
Robert Yang
db91c23871 bitbake: fetch2/git: Fix clean to remove clonedir
The localpath is a symlink to clonedir when it is cloned from a mirror, for
example:
$ bitbake systemtap-native -cfetch
$ ls downloads/git2
sourceware.org.git.systemtap.git -> /path/to/downloads/git2/mirror.path.git.sourceware.org.git.systemtap.git
mirror.path.git.sourceware.org.git.systemtap.git

There are both sourceware.org.git.systemtap.git and
mirror.path.git.sourceware.org.git.systemtap.git in DL_DIR/git2, the symlink
sourceware.org.git.systemtap.git is created by try_mirror_url(), but
do_cleanall" only removed the symlink, didn't remove the real dir
mirror.path.git.sourceware.org.git.systemtap.git, this may cause confusions,
for example, I assumed that do_cleanall removed everything, but it didn't, and
it would the re-used next time when do_fetch. This patch fixes the problem.

(Bitbake rev: 452e2200ad2c29dec3753f5f7a8cbc9183ec7dd8)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
Robert Yang
a7f3cbb4a2 bitbake: fetch2: runfetchcmd(): Print workdir in debug message
The workdir is an important message when run git command, for exmample, before the patch:
Running git -c core.fsyncobjectfiles=0 branch --contains [snip]

We don't know where it is running, now it is:
Running 'git -c core.fsyncobjectfiles=0 branch --contains [snip]' in /path/to/git.opensvc.com.multipath-tools..git

Which is easier to debug.

(Bitbake rev: f25c05b73218d0e8edf2dd63b36bc09e547c5455)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
Richard Purdie
c96d8de352 bitbake: fetch2: Fix undefined variable issues
Various refactors have left hanging variables, often in debug messages, hence
why they haven't been spotted. Fix them (pylint spotted the issues).

(Bitbake rev: e1f252408982a62485bbf874a01fd30a02aeed32)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00
Richard Purdie
28fb15c56f bitbake: fetch2/wget: Remove pointless lambda function
(Bitbake rev: 329986c3f2d98c4cd1a43b725194003575e718d5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00
Richard Purdie
588a4c58e8 bitbake: fetch2/wget: Clean up whitespace/comments
Clean up the whitespace and comments style, keep pylint happy.

(Bitbake rev: ee59fdaae68543ade03cacfdbbf14fdc7e469412)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00
Richard Purdie
9a1faba274 bitbake: fetch2/wget: Fix typo in proxy handling
Fix a typo in a variable which would affect proxy handling spotted by
pylint.

(Bitbake rev: 19f4ca0119e33df64da629253d5002fcdb385f4f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00
Richard Purdie
202b7d0521 bitbake: fetch2/wget: Cleanup module imports
(Bitbake rev: ef062a3251af474fdeb480149856be3887b8c1c9)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 23:54:41 +00:00
Stefan Klug
0a9161855c bitbake: fetch2/wget: Fix authentication in checkstatus() of the wget fetcher
I wonder how this used to work for anybody.

(Bitbake rev: 9481e9cd688f635a7fe2ac60d5826ac26f0cb9ba)

Signed-off-by: Stefan Klug <stefan.klug@baslerweb.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19 15:29:31 +00:00
André Draszik
2fcbc079e2 bitbake: bitbake: remove True option to getVar calls
getVar() has been defaulting to expanding by default for
a long time (2016), thus remove the True option from
getVar() calls with a regex search and replace.

Search & replace made using the following command:
    sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \
        -i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \
             | cut -d':' -f1 \
             | sort -u)

(Bitbake rev: 3bba0dbd524cf72176a765957adff544ae5c255a)

Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-12 22:01:10 +00:00
Mark Hatle
a7774aced0 bitbake: gitsm: The fetcher did not process some recursive submodules properly.
Move the submodule processing outside of the if statement to avoid any
optimizations that may happen.

Update the test cases to include the additional case, and split the other
test cases into individual tests to make it easier to figure out what
the failure may be.

(Bitbake rev: 0ec98c01ae50f95c9c74acf53013ac59e0e72b08)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-28 23:14:27 +00:00
Mark Hatle
1979d9162a bitbake: gitsmy.py: Fix unpack of submodules of submodules
If the submodule is in a subdirectory, it needs to have that structure
preserved.  This means the unpack path needs to be in the 'dirname' of the
final path -- since the unpack directory name is specified in the URI.

Additional specific test cases were added to ensure this is working properly
based on two recent error reports.

(Bitbake rev: 8c8ecec2a722bc2885e2648d41ac8df07bdf660d)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-24 17:45:49 +00:00
Mark Hatle
9238a72589 bitbake: gitsm.py: Fix relative URLs
Prior code happened to work with relative URLs, when the code was recently
restructured it caused all relative urls to no longer work.  Restore the prior
code flow for relative support and better comment why that code is there.

(Bitbake rev: 14bb383af0ca98e0e04ec217e537b6a899f3658e)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-24 17:45:49 +00:00
Mark Hatle
7d715ae071 bitbake: gitsm.py: Refactor the functions and simplify the class
The update_submodules and unpack_submodules functions were nearly indentical,
so we made a common function where the different behavior could be passed
in by the download and unpack users.  The new function is process_submodules.

Moved the parse_gitmodules function under the new process_submodules, since
there are no external callers.

Refactor the file relative path processing to the URL translation code.
We also add a warning to the translation if a relative ssh URL has been
detected.  Since this can cause a problem.

In the case of a relative URL that does not work after being translated,
it should be possible to use the MIRROR functions to manual translate the
generated relative URL into one that works properly.

Remove 'git config' processing on download contents.  It turns out this is not
necessary since all of the later components work using the git fetcher.

Limit the 'git submodule update' call to only when unpacking a non-bare
repository.  Submodules are always loaded as bare, so this prevents
intermediate unpacks from being attempted.

Finally, the test cases were updated and the new commit ids in the test
repository were updates as well.

(Bitbake rev: 610dbee5634677f5055e2b36a3043cd197fb8c51)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:08 +00:00
Mark Hatle
a9cf611e7a bitbake: gitsm.py: Rework the shallow fetcher and test case
A custom shallow submodule is no longer necessary, as the regular git
fetcher is used and shallow handling works with the same code.

The only general difference between the regular change is simply declaring a
clone as shallow, when appropriate.

This also removes a potential race condition in copying repositories
vs cloning them.

The gitsm shallow fetcher test was revised to verify that the submodule
is shallow cloned along with the primary repository.

The first step of this change was to be sure to clean the gitsubmodule download
directory, as was previously done with the may gitsource directory.

Additional test components were added to verify commit counts, and an
obsolete (and likely incorrect) test for the .git/modules directory to be
empty was also removed.

(Bitbake rev: f9cc4684dcf4281acc557cda8cb35602354ac3d6)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:08 +00:00
Mark Hatle
cd1430e379 bitbake: gitsm.py: revise unpack
Greatly simply the unpack rule by copying the general functionality of
update_submodules as unpack_submodules.  This will recursively construct
a set of urls and unpack them using the standard system behaviors.

The overall code may be slightly bigger, but this ensures that all of the
standard locks are inplace, ensuring the code doesn't change out from
under the unpack function.  (This could have happened before due to using
'cp' instead of further unpacks on submodules.  This may still happen in
shallow clones.)

(Bitbake rev: 7d7ee630f1c65e7dd234f945edf5e3b3bcb0fc30)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:08 +00:00
Mark Hatle
80f41edbf7 bitbake: gitsm.py: Optimize code and attempt to resolve locking issue
It was reported that a race condition on a shared download directory could
occur with the gitsm fetcher, the result happened with a call to

  git config

that occured within the update_submodules.  Since the fetch is locked by the
upper level, it was probably the prior need_update(...) function causing this
because of some old code.

The gitsm class inherits the git class.  The need_update was overridding the
version in gitsm, so that it forceably checked the submodules.

It's clear we can optimize the code by only updating if the primary repository
needs updating.  Since we don't care if the submodule repository has changed
because if the primary hasn't, references to the submodule won't change.

(Bitbake rev: 346338667edca1f58ace769ad417548da2b8d981)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:08 +00:00
Mark Hatle
3974e5054f bitbake: gitsm.py: Add support for alternative URL formats from submodule files
The following appear to be the git supported formats:

  proto://user:pass@host/path  (URI format)
  user@host:path (SSH format)
  /path or ./path or ../path (local file format)

We adjust the parsing to find out if we have a URI format or not.
When we are NOT in URI format, we do our best to determine SSH or
file format by looking for a ':' in the overall string.  If we find
a ':' we assume SSH format and adjust accordingly.

Note, in SSH format we simply replace the ':' with a '/' when constructing
the URL.  However, if the original path was ":/...", we don't want '//' so
we deal with this corner case as well.

(Bitbake rev: dcac05e7dc6b0c5f8e63d36ad105b7eab41f0016)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:07 +00:00
Mark Hatle
566d31960c bitbake: gitsm.py: Fix when a submodule is defined, but not initialized
It is possible for a submodule to be defined in the .gitmodules file, but
never initialized in the repository itself.  This shows itself when searching
for the defined module hash you will get back a empty value.

Similarly we need to identify and skip defined but not initialized submodules
during the unpack stages as well.

Thanks to raphael.lisicki@siemens.com for their help is figuring out how
to resolve this issue.

Additionally a problem was found where, while unlikely, it may be possible
for the wrong revision to have been searched using ls-tree.  This has been
resolved in the update_submodules function by keeping the correct revision
along with the submodule path.

(Bitbake rev: 49e1ff920143dcd4e7f73933d7ab8a84b8ffa1a3)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:07 +00:00
Richard Purdie
e5455662a9 bitbake: bitbake: Fix Deprecated warnings from regexs
Fix handling of escape characters in regexs and hence fix python
Deprecation warnings which will be problematic in python 3.8.

(Bitbake rev: c1fcc46e2498ddd41425d8756754f814d682aba3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:07 +00:00
Richard Purdie
e74f80433e bitbake: fetch2: Fix os.errno references
os.errno used to happen to work but is invalid. Correct to use errno.

[YOCTO #13068]

(Bitbake rev: 03aae96a8859409b1ce7e3d5c1197371fa96bd14)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-08 17:17:37 +00:00
Richard Purdie
3630328d05 bitbake: fetch2/wget: Fix typo in exception name
This could clearly never have worked since the python3 migration
but as its in an error path, it doesn't get tested/used much.

(Bitbake rev: 704f27dc28d50a6dc02c8b64274ee4ecb3058c4a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-08 17:17:37 +00:00
Ulf Magnusson
db20fe6f0a bitbake: fetch2/svn: Fix SVN repository concurrent update race
The ${DL_DIR}/svn directory is used by BitBake to keep checked-out SVN
repositories from which tarballs are generated. These repositories were
protected from concurrent update with a lock on the tarballs. However,
the tarballs are specific to the SRCREV and module checked out (many
tarballs can come from the same repository), meaning a repository could
be modified concurrently if two recipes checked out two different
SRCREVs or modules from it in parallel. This caused errors like the
following:

ERROR: Fetcher failure: Fetch command failed with exit code 1, output:
svn: E155004: Run 'svn cleanup' to remove locks (type 'svn help cleanup' for details)
svn: E155004: Working copy '/home/foo/downloads/svn/repo/trunk' locked.
svn: E155004: '/home/foo/downloads/svn/repo/trunk' is already locked.

Fix it by adding a per-repository lock that's independent of the module
and SRCREV.

(Bitbake rev: 3f1f183a17bf3580da8a4ffd6dab30b62c2654a8)

Signed-off-by: Ulf Magnusson <Ulf.Magnusson@bmw.de>
Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-01 11:32:08 +00:00
Richard Purdie
41d8955262 bitbake: fetch2: Avoid warning about incorrect character escaping in regex
Fixes:

lib/bb/fetch2/__init__.py:259: DeprecationWarning: invalid escape sequence \w
  re.compile("^\w+:(?!//)").match(uri):

(Bitbake rev: 2f0b6cea2be7e89ef01181b10899b3e7f11e8f20)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-27 22:15:34 +00:00
Ross Burton
3786729785 bitbake: fetch: don't use shell=True when listing ar files
(Bitbake rev: 7414b3537e8adfb41a9581d70bf8296c4f7d38c0)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-23 16:10:37 +00:00
Christopher Larson
e3e88600ce bitbake: fetch/git: fix AttributeError in shallow extraction logic
This code checks to see if shallow is either disabled or the tarball is
missing, but the else block tries to print the tarball filename, and
this attribute doesn't exist at all when shallow is disabled. Handle the
two cases separately to give sane errors for both cases without the
exception:

    Exception: AttributeError: 'FetchData' object has no attribute 'fullshallow'

(Bitbake rev: bdbb558342ebb4e64384c9838d2485d9299d91a6)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-22 10:45:14 +00:00
Parthiban Nallathambi
83b587e77b bitbake: fetch2/npm.py: Allow shrinkwrap resolved relative URL which startswith 'http' (e.g http-proxy)
shrinkwrap resolved relative URL can start with http. For example,
"resolved: http-proxy/-/http-proxy-${PV}.tgz" is still relative URL
to npm registry, but starts with http.

Current if statement compares the startswith 'resolved' to 'http',
which makes impossible to use npm download. Condtional comparison
now strictly checks for "http://" and "https://"

(Bitbake rev: f76075aa1a5159fd4d62949cb588346888b9fe60)

Signed-off-by: Parthiban Nallathambi <pn@denx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-21 11:54:47 +00:00
Mike Crowe
b322b69a97 bitbake: fetch: Also export GIT_SSH environment variable
Support users who want to specify a custom ssh to Git using an environment
variable. (This lets build systems work without requiring users to
configure Git explicitly.)

(Bitbake rev: db46fb2755a4b1033ab60051ce511cc9dd7e34fb)

Signed-off-by: Mike Crowe <mac@mcrowe.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-07 23:08:55 +00:00
Mark Hatle
a042179a78 bitbake: fetch2/gitsm.py: Fix the references when the module and path are different
Git does not require the module and target path to be the same in the
.gitmodules file.  This incorrect assumption was being made previously
causing various unpack failures.

An example .gitmodule showing this issue:

   [submodule "plugins/WaveShaper/Libs/inih"]
        path = plugins/wolf-shaper/Libs/inih
        url = https://github.com/pdesaulniers/inih.git

The unpack function also needed to work in a loop on the overall
submodules_queue.  Before it could have missed items that were not in the
primary repository.

(Bitbake rev: 5a7009c204f4d2254e3b2d83ad9319ac23f1cf4d)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-07 23:08:55 +00:00
Mark Hatle
a28637abbc bitbake: fetch2/gitsm.py: Disable branch checking on submodules
Submodules by definition refer to a specific commit, not branch.  If we don't
ignore the branch, then any commits on a submodule on a branch different then
the original module will trigger a failure that the commit is not on the
branch.

(Bitbake rev: fdc1dbf96f153b496de52acd8263366a1ff303ad)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-07 23:08:55 +00:00
Richard Purdie
24139e27c6 bitbake: fetch2: Avoid incorrect getVarFlag call
Calling getVarFlag with flag=None makes no sense, don't do it. Bitbake
used to silently ignore this, it now warns so avoid the warning.

(Bitbake rev: a68de8ace62eaba23856bfb301efbbe1824322aa)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:27 +01:00
Urs Fässler
a8a077cafb bitbake: fetch2/git: provide information about missing sources
Provide more information in the case the sources are not found in the
unpack step.

(Bitbake rev: 27a2214bf6f2e7c61bfc422a20959a55f7e0d25d)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:27 +01:00
Urs Fässler
8553b2c111 bitbake: fetch2/git: explicitly show the decision logic to select the source in code
(Bitbake rev: 78d8fb3a7899e1404a513be1c2b2b4440da8b12d)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:26 +01:00
Urs Fässler
0d4fe0602e bitbake: fetch2/git: prevent access to non-existing clonedir
A user friendly error is throw when neither the clonedir nor
fullshallow exist. Without the check, a difficult to interpret error
is throw from within the fetch command.

(Bitbake rev: 30cf2506007d25162f0805051212f54c39034ff3)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:26 +01:00
Urs Fässler
49f25eceeb bitbake: fetch2: extract the function which ensures that a valid symlink exists
For better readability and future use, we extract the function which
ensures that a given symlink exists.

(Bitbake rev: 5e69ca56533666a097bb23d09ab673e5c862051c)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:26 +01:00
Peter Marko
212cbc0382 bitbake: fetch2: avoid circular recursion with SRCPV in PR
Some recent changes broke SRCPV and workaround was introduced
to avoid circular dependency if SRCPV is in PV.
However there is still the same error if SRCPV is in PR.

(Bitbake rev: 05ee4845f925b8528a7ce9cffb4bae425b8fa1e9)

Signed-off-by: Peter Marko <peter.marko@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:26 +01:00
Mark Hatle
8c3db8cb9e bitbake: gitsm: Allow relative URIs when fetching
Some repositories may specify a relative submodule path.  If this happens,
it is our responsibility to use the parents URL (ud) and handle any relative
processing ourselves.

(Bitbake rev: fd9893c338df7828b2c01f1d548aa86dfcf7c50a)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04 23:00:24 +01:00
Laurent Bonnans
41e6a3efac bitbake: fetch2/gitsm: checkout submodules recursively
The new fetcher did not run 'git submodule update' recursively.

(Bitbake rev: 377ed943357b6d47d41d84edbf971741f44affa9)

Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04 23:00:24 +01:00
Laurent Bonnans
c888446d12 bitbake: fetch2/gitsm: handle submodules nested inside subdirs
The new gitsm fetcher assumed that submodules were living directly in
the 'modules' directory, whereas they can be arbitrarily nested inside
subdirectories.

Solve it by first creating the parent of the destination directory for
the symlink and copy steps.

(Bitbake rev: 3dbc84e9c572f43209b79f3656d7dc35a6d982ba)

Signed-off-by: Laurent Bonnans <laurent.bonnans@here.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04 23:00:24 +01:00