Commit Graph

148 Commits

Author SHA1 Message Date
Richard Purdie
79834a7144 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:04 +01:00
Naveen Saini
e7c2bc67b5 bitbake: bitbake: fetch2/git: git-lfs check
Build will fail if repository has lfs contents in absense of git-lfs tool on host.
Build will pass if repository may or may not contains lfs content if host has git-lfs installed.

[YOCTO #13198]

(Bitbake rev: 3f2c2eb2f59707828bdcdd6414db837da8dc3b0e)

Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-29 14:16:31 +01:00
Robert Yang
64c21baf63 bitbake: fetch2: Unify BB_FETCH_PREMIRRORONLY
The fetch2/__init__.py checks whether "BB_FETCH_PREMIRRORONLY" == "1", but
fetch2/git.py and hg.py checks whether it is None, this makes it discontinuous,
and BB_FETCH_PREMIRRORONLY = "0" doens't work as expected in the later case,
so unify it to the previous one. (As BB_NO_NETWORK does).

And also use bb.utils.to_boolean() to make them consistent.

(Bitbake rev: 85a0d22835588e9ad8ec29d88a8115227e88420c)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 14:02:28 +00:00
Robert Yang
db91c23871 bitbake: fetch2/git: Fix clean to remove clonedir
The localpath is a symlink to clonedir when it is cloned from a mirror, for
example:
$ bitbake systemtap-native -cfetch
$ ls downloads/git2
sourceware.org.git.systemtap.git -> /path/to/downloads/git2/mirror.path.git.sourceware.org.git.systemtap.git
mirror.path.git.sourceware.org.git.systemtap.git

There are both sourceware.org.git.systemtap.git and
mirror.path.git.sourceware.org.git.systemtap.git in DL_DIR/git2, the symlink
sourceware.org.git.systemtap.git is created by try_mirror_url(), but
do_cleanall" only removed the symlink, didn't remove the real dir
mirror.path.git.sourceware.org.git.systemtap.git, this may cause confusions,
for example, I assumed that do_cleanall removed everything, but it didn't, and
it would the re-used next time when do_fetch. This patch fixes the problem.

(Bitbake rev: 452e2200ad2c29dec3753f5f7a8cbc9183ec7dd8)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
André Draszik
2fcbc079e2 bitbake: bitbake: remove True option to getVar calls
getVar() has been defaulting to expanding by default for
a long time (2016), thus remove the True option from
getVar() calls with a regex search and replace.

Search & replace made using the following command:
    sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \
        -i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \
             | cut -d':' -f1 \
             | sort -u)

(Bitbake rev: 3bba0dbd524cf72176a765957adff544ae5c255a)

Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-12 22:01:10 +00:00
Richard Purdie
e5455662a9 bitbake: bitbake: Fix Deprecated warnings from regexs
Fix handling of escape characters in regexs and hence fix python
Deprecation warnings which will be problematic in python 3.8.

(Bitbake rev: c1fcc46e2498ddd41425d8756754f814d682aba3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:07 +00:00
Christopher Larson
e3e88600ce bitbake: fetch/git: fix AttributeError in shallow extraction logic
This code checks to see if shallow is either disabled or the tarball is
missing, but the else block tries to print the tarball filename, and
this attribute doesn't exist at all when shallow is disabled. Handle the
two cases separately to give sane errors for both cases without the
exception:

    Exception: AttributeError: 'FetchData' object has no attribute 'fullshallow'

(Bitbake rev: bdbb558342ebb4e64384c9838d2485d9299d91a6)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-22 10:45:14 +00:00
Urs Fässler
a8a077cafb bitbake: fetch2/git: provide information about missing sources
Provide more information in the case the sources are not found in the
unpack step.

(Bitbake rev: 27a2214bf6f2e7c61bfc422a20959a55f7e0d25d)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:27 +01:00
Urs Fässler
8553b2c111 bitbake: fetch2/git: explicitly show the decision logic to select the source in code
(Bitbake rev: 78d8fb3a7899e1404a513be1c2b2b4440da8b12d)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:26 +01:00
Urs Fässler
0d4fe0602e bitbake: fetch2/git: prevent access to non-existing clonedir
A user friendly error is throw when neither the clonedir nor
fullshallow exist. Without the check, a difficult to interpret error
is throw from within the fetch command.

(Bitbake rev: 30cf2506007d25162f0805051212f54c39034ff3)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:26 +01:00
Urs Fässler
82f32d99f7 bitbake: fetch2/git: use correct check to decide if the shallow tarball should be unpacked
The shallow_tarball check is never true due a check on the caller side.
The tarball check is not related to the code on the caller side.

(Bitbake rev: 086eddcf8c7520ff5c52ce2a11ca9bf5b5fe5d7e)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04 23:00:24 +01:00
Urs Fässler
b907303b92 bitbake: fetch2/git: prevent access to non-existing fullshallow tarball
(Bitbake rev: b7f00a8c11672a2ee0408e210fb174cda3384e3f)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04 23:00:24 +01:00
Urs Fässler
073c62cb56 bitbake: fetch2/git: improve readability of method need_update
To improve the readability we extract the different scenarios of why
the clonedir needs an update.

(Bitbake rev: 9038e029f4f0ab413727de76c74248cbb3cdc9ea)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04 23:00:24 +01:00
Paulo Neves
a8368651ff bitbake: fetcher: Fixed remote removal not throwing exception.
Before this fix it is assumed that the removal of the
remote can only fail because there is not remote to remove. This
is a false assumption. Example error which would be ignored:

git -c core.fsyncobjectfiles=0 remote rm origin failed with exit code 1, output:
Note: A branch outside the refs/remotes/ hierarchy was not removed;
to delete it, use:
  git branch -d master
error: could not lock config file config
error: Could not remove config section 'remote.origin'

Due to the masking of this error a stranger error will be
presented to the user, because this time we do not mask the
exception:

git -c core.fsyncobjectfiles=0 remote add --mirror=fetch origin https://github.com/ptsneves/tl-wn722.git failed with exit code 128, output:
fatal: remote origin already exists.

The most likely reason that the remote cannot be removed nor
modified is that the DL_DIR/git2 does not have permissions
compatible with the user running bitbake.

This commit fixes:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12728

(Bitbake rev: 9c86c582a10c9b23abad7d34b6cbf12f7086294d)

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28 10:32:08 +01:00
Andre McCurdy
7838294271 bitbake: fetch2: unify the way fetchers determine DL_DIR and FETCHCMD
Currently there is quite some variation between the fetchers in terms
of how they determine the subdirectory within DL_DIR and the base
fetch command to run. Some rely on variables being set externally
(e.g. from bitbake.conf in oe-core), some respect these external
variables but provide fallback defaults and some use only hardcoded
internal values. Try to unify the approach used across the various
fetchers.

(Bitbake rev: efd5e35af4b08501c67e8b30f30d9457f6fdf610)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-15 17:57:30 +01:00
Andre McCurdy
b68ed05be6 bitbake: fetch/git: make fewer calls to _contains_ref() from download()
Updating a local git repo clone currently results in multiple calls
to self._contains_ref(), some of which appear to be redundant and can
be eliminated by minor tweaks to the logic in download().

Also drop redundant calls to os.path.exists(ud.clonedir) before
self.need_update(), since need_update() includes its own built-in
check for the existance of ud.clonedir.

(Bitbake rev: 61b0df5523afc8f805043f3adc9c106690e6f133)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-22 13:13:19 +01:00
Ross Burton
f500504b81 bitbake: fetch2/git: log exception if ls-remote fails
(Bitbake rev: 8212446de11c0e370c55f88cde86334b760cd939)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-28 12:51:52 +01:00
Nathan Rossi
9c44aee83d bitbake: fetch/git: Use 'git-make-shallow' from bin directory
Move the code that existed in tests/fetch.py for determining the path to
'git-make-shallow' into the git module and reference it.

This ensures that 'git-make-shallow' is always available and the desired
version regardless of the path variable or whether git exposes the
command.

(Bitbake rev: 6b508ab8fd5aa796c1c00c970e81e5e93f84d35d)

Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-14 15:26:03 +00:00
Richard Purdie
cf8425e247 bitbake: fetch/git: Add pack-refs command to mirror handling code
We've noticed issues on our infrastucture iterating over the many
tag/branch/head reference files that some git repositories may contain.

By issuing the pack-refs command, we move these all to a single file
which speeds up operations with the mirror repos in the downloads
directory in general.

(Bitbake rev: f8126aaf774186a6eaf0bd4067b89c074594886c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-29 08:50:08 +00:00
Andre Rosa
20d0282c60 bitbake: Replace deprecated git branch parameter "--set-upstream"
Since 2017-08-17 (git version 2.14.1.473.g3ec7d702a) using deprecated
git branch parameter "--set-upstream" causes a fetcher error. Replace
it by "--set-upstream-to".

https://git.kernel.org/pub/scm/git/git.git/commit/?id=52668846ea2d41ffbd87cda7cb8e492dea9f2c4d
says, it's deprecated since 2012-08-30 so hopefully all still supported
host distributions have new enough git to support "--set-upstream-to".

ERROR: PACKAGE do_unpack: Fetcher failure: ...;
git -c core.fsyncobjectfiles=0 branch --set-upstream master origin/master failed with exit code 128, output:
fatal: the '--set-upstream' option is no longer supported. Please use '--track' or '--set-upstream-to' instead.

ERROR: PACKAGE do_unpack: Function failed: base_do_unpack

(Bitbake rev: 2ab50074c1a6c56a8a178755de108447d7b7acaf)

Signed-off-by: Andre Rosa <andre.rosa@lge.com>
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-11 17:19:37 +01:00
Christopher Larson
35ecff3cf0 bitbake: fetch/git: add support for removing arbitrary revs for shallow
In certain cases, it's valuable to be able to exert more control over what
history is removed, beyond srcrev+depth. As one example, you can remove most
of the upstream kernel history from a kernel repository, keeping predominently
the non-publically-accessible content. If the repository is private, the
history in that repo couldn't be restored via `git fetch --unshallow`, but
upstream history could be.

Example usage:

    # Remove only these revs, not at a particular depth
    BB_GIT_SHALLOW_DEPTH_pn-linux-foo = "0"
    BB_GIT_SHALLOW_REVS_pn-linux-foo = "v4.1"

(Bitbake rev: 97f856f0455d014ea34c28b1c25f09e13cdc851b)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
8144f6f408 bitbake: fetch/git: add support for keeping extra refs for shallow
By default, all unused refs (branches & tags) are removed from the repository,
as shallow processing scales with the number of refs it has to process. Add
the ability to explicitly specify additional refs to keep. This is
particularly useful for recipes with custom checkout processes, or whose
git-based versioning requires a tag be available (i.e. for `git describe
--tags`). The new `BB_GIT_SHALLOW_EXTRA_REFS` variable is a space-separated
list of refs, fully specified, and support wildcards.

Example usages:

    BB_GIT_SHALLOW_EXTRA_REFS = "refs/tags/v1.0"
    BB_GIT_SHALLOW_EXTRA_REFS += "refs/heads/*"

(Bitbake rev: 1771934cd9f8b5847c6fcae0a906fb99d6b0db16)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
bf87c5cd19 bitbake: fetch/git: support per-branch/per-url depths for shallow
Allow the user to explicitly adjust the depth for named urls/branches. The
un-suffixed BB_GIT_SHALLOW_DEPTH is used as the default.

Example usage:

    BB_GIT_SHALLOW_DEPTH = "1"
    BB_GIT_SHALLOW_DEPTH_doc = "0"
    BB_GIT_SHALLOW_DEPTH_meta = "0"

(Bitbake rev: 9dfc517e5bcc6dd203a0ad685cc884676d2984c4)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
27d56982c7 bitbake: fetch/git: add support for shallow mirror tarballs
This adds support to the git fetcher for fetching, using, and generating
mirror tarballs of shallow git repositories. The external git-make-shallow
script is used for shallow mirror tarball creation.

This implements support for shallow mirror tarballs, not shallow clones.
Supporting shallow clones directly is not really doable for us, as we'd need
to hardcode the depth between branch HEAD and the SRCREV, and that depth would
change as the branch is updated.

When BB_GIT_SHALLOW is enabled, we will always attempt to fetch a shallow
mirror tarball. If the shallow mirror tarball cannot be fetched, it will try
to fetch the full mirror tarball and use that. If a shallow tarball is to be
used, it will be unpacked directly at `do_unpack` time, rather than extracting
it to DL_DIR at `do_fetch` time and cloning from there, to keep things simple.
There's no value in keeping a shallow repository in DL_DIR, and dealing with
the state for when to convert the clonedir to/from shallow is not worthwhile.

To clarify when shallow is used vs a real repository, a current clone is
preferred to either tarball, a shallow tarball is preferred to an out of date
clone, and a missing clone will use either tarball (attempting the shallow one
first).

All referenced branches are truncated to SRCREV (that is, commits *after*
SRCREV but before HEAD are removed) to further shrink the repository. By
default, the shallow construction process removes all unused refs
(branches/tags) from the repository, other than those referenced by the URL.

Example usage:

    BB_GIT_SHALLOW ?= "1"

    # Keep only the top commit
    BB_GIT_SHALLOW_DEPTH ?= "1"

    # This defaults to enabled if both BB_GIT_SHALLOW and
    # BB_GENERATE_MIRROR_TARBALLS are enabled
    BB_GENERATE_SHALLOW_TARBALLS ?= "1"

(Bitbake rev: 5ed7d85fda7c671be10ec24d7981b87a7d0d3366)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Christopher Larson
ab4e578b86 bitbake: fetch: support multiple mirror tarball filenames
Remove ud.mirrortarball in favor of ud.mirrortarballs. Each tarball will be
attempted, in order, and the first available will be used. This is needed for
git shallow mirror tarball support, as we want to be able to use either
a shallow or full mirror tarball.

(Bitbake rev: 02eebee6709e57b523862257f75929e64f16d6b0)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-02 13:36:57 +01:00
Paul Eggleton
f41740ee8e bitbake: fetch2/git: prevent recursion on getting latest revision
We call git ls-remote to get the latest revision from a git repository,
however by calling runfetchcmd() we can end up recursively running
git ls-remote a number of times with OE e.g. if ${SRCPV} is in PV, ${PV}
is in WORKDIR, and ${WORKDIR} is in PATH (as a result of recipe-specific
sysroots), our call to runfetchcmd() exports PATH so _lsremote() will
get called again - with the end result that we run git ls-remote 30
times in quick succession (!). Prevent that from happening by using a
guard variable and returning a dummy value if it's called recursively.

Fixes [YOCTO #11185].

(Bitbake rev: ff1ccd1db5d70b3fc9ad0d3e8f3d7b804c22bf36)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-27 20:23:19 +01:00
Christopher Larson
f4a9248975 bitbake: fetch/git: fix FetchError reference
FetchError isn't defined, use bb.fetch2.FetchError in this context.

(Bitbake rev: 945fa980e027753df2c21d84eb63dcaddb2caaee)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22 11:35:23 +00:00
Christopher Larson
754e98c72e bitbake: fetch/git: drop pointless os.path.join, workdir=
The touch of .done explicitly specifies the path, so there's no need for
workdir=, and "os.path.join('.')" is identical to just '.'.

(Bitbake rev: 955cbfdaa2400d15ec428b65848e6835c9f44860)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22 11:35:23 +00:00
Christopher Larson
5ee3bae3bb bitbake: fetch/git: kill pointless quotes around single % args
(Bitbake rev: 4aebf12153369364eae2e6e773e2a921e9c91f72)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22 11:35:23 +00:00
Christopher Larson
b079a2da36 bitbake: fetch/git: use enumerate for ud.names
list.index() isn't a particularly efficient operation, so keep track of our
position via enumerate() instead, which is more pythonic as well.

(Bitbake rev: dec6e90a4d27ee335e9c78aeebd277098fec94d1)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22 11:35:23 +00:00
Andre McCurdy
b16192c938 bitbake: fetch2: don't use deprecated bb.data APIs
Cleanup some more usage of bb.data APIs in the fetchers.

(Bitbake rev: 9752fd1c10b8fcc819822fa6eabc2c1050fcc03b)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-01 11:16:07 +00:00
Robert P. J. Day
6c611d697f bitbake: fetch2: Rename "setup_revisons" to "setup_revisions"
For spelling's sake, rename Python routine "setup_revisons" to
"setup_revisions."

(Bitbake rev: 4df59b027c02ef39d72476251ccd3fd62fc20bf6)

Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-19 06:28:47 -08:00
Robert P. J. Day
1bd4f040a3 bitbake: fetch2: Correct two examples of typo "revsion."
(Bitbake rev: 05f5421b2e44cd58c5912848de43d5884d070150)

Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-19 06:28:46 -08:00
Markus Lehtonen
38438b6cf4 bitbake: fetch2: obey BB_ALLOWED_NETWORKS when checking network access
[YOCTO #10508]

(Bitbake rev: ddd3bc2d64d7240ecb6b6e4a1ae29b1faef6cc22)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:10 +00:00
Joshua Lock
1fce7ecbbb bitbake: bitbake: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:09 +00:00
Richard Purdie
8de811ae76 bitbake: lib/bb: Don't use deprecated bb.data.getVar/setVar API
The old style bb.data.getVar/setVar API is obsolete. Most of bitbake
doesn't use it but there were some pieces that escaped conversion. This
patch fixes the remaining users mostly in the fetchers.

(Bitbake rev: ff7892fa808116acc1ac50effa023a4cb031a5fc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23 10:59:56 +00:00
Matt Madison
ab09541d55 bitbake: fetch2: preserve current working directory
Fix the methods in all fetchers so they don't change
the current working directory of the calling process, which
could lead to "changed cwd" warnings from bitbake.

(Bitbake rev: 6aa78bf3bd1f75728209e2d01faef31cb8887333)

Signed-off-by: Matt Madison <matt@madison.systems>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-20 16:08:59 +01:00
Richard Purdie
c04468d113 bitbake: git: Allow local repos to use HEAD
Introduce a new 'usehead' url parameter for git repositories. Specifying
usehead=1 causes bitbake to use whatever commit the repository HEAD is
pointing to. Usage of usehead=1 is only allowed for local git
repositories, i.e. it must always be accompanied with protocol=file url
parameter.

[YOCTO #9351]

(Bitbake rev: 2673fac5a9d06de937101e3fb2ddf1e60ff99abf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-20 16:08:59 +01:00
Paul Eggleton
45baa90c8c bitbake: fetch2: implement progress support
Implement progress reporting support specifically for the fetchers. For
fetch tasks we don't necessarily know which fetcher will be used (we
might initially be fetching a git:// URI, but if we instead download a
mirror tarball we may fetch that over http using wget). These programs
also have different abilities as far as reporting progress goes (e.g.
wget gives us percentage complete and rate, git gives this some of the
time depending on what stage it's at). Additionally we filter out the
progress output before it makes it to the logs, in order to prevent the
logs filling up with junk.

At the moment this is only implemented for the wget and git fetchers
since they are the most commonly used (and svn doesn't seem to support
any kind of progress output, at least not without doing a relatively
expensive remote file listing first).

Line changes such as the ones you get in git's output as it progresses
don't make it to the log files, you only get the final state of the line
so the logs aren't filled with progress information that's useless after
the fact.

Part of the implementation for [YOCTO #5383].

(Bitbake rev: 4027649f422ee64b1c4e1ad8d48ac295050afbff)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-19 15:04:37 +01:00
Peter Kjellerstedt
08c6808d01 bitbake: fetch2/git.py: References must match exactly
Previously the code used to match a reference to its SHA-1 in
_latest_revision() used the Python "in" operator, which made it match
if the reference matched the beginning of an existing tag or
branch. This test, however, must be exact. I.e., either the reference
matches a tag or branch exactly, or it does not match at all.

(Bitbake rev: e5986c78a6108fd7578989c20efcbf0b81c97e03)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-03 13:13:30 +01:00
Robert Yang
c1157400ab bitbake: fetch2/git.py: remove .indirectiondir workaround
It was used for workaround git 1.7.9.2 which was released in 2012 which
should not be existed on nowadays host, so remove it to avoid
confusions.

(Bitbake rev: 6140d0cc9aecf1029ca16fed47071dfcc92f4269)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-06 10:29:47 +01:00
Richard Purdie
f8508deb01 bitbake: Revert "fetch/git: Change to use clearer ssh url syntax for broken servers"
This reverts commit 4193e99adce8e88f12ac88d7578ad39575f7e346.

It seems the underlying issue was caused by ":" in the url which isn't
supported. The patch was therefore incorrect.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-15 15:02:40 +00:00
Kristian Amlie
1724ffd32d bitbake: fetch2/git.py: Add missing "errno" module import.
This goes undetected most of the time, but when updating a repository,
if the ud.fullmirror file is not present, you end up getting an
exception instead of carrying on because the errno module is not
loaded (specifically "if exc.errno != errno.ENOENT").

(Bitbake rev: e6fca8480731ce817df9bee61438347a5e3d3017)

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:28:55 +00:00
Richard Purdie
221d864042 bitbake: fetch/git: Change to use clearer ssh url syntax for broken servers
Some servers, e.g. bitbucket.org can't cope with ssh:// as part of
the git url syntax. git itself is happy enough with this but you
get server side errors when using it.

This changes the git fetcher to use the more common ssh url format
which also means we need a : before the path.

Seems a shame to have to do this due to broken servers however
it should be safe enough since this other form is the one most people
use on the commandline so it should be safe enough.

[YOCTO #8864]

(Bitbake rev: 4193e99adce8e88f12ac88d7578ad39575f7e346)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08 13:58:08 +00:00
Alexander Kanavin
3745479596 bitbake: bitbake: rename REGEX, REGEX_URI, and GITTAGREGEX.
Rename REGEX to UPSTREAM_CHECK_REGEX, REGEX_URI to UPSTREAM_CHECK_URI, and
GITTAGREGEX to UPSTREAM_CHECK_GITTAGREGEX to better reflect their purpose
and to reflect a common namespace.

(Bitbake rev: f0a9e783f9969573fd74edfa241ef14f18ac684e)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-08 10:20:31 +00:00
Romain Perier
7b86c771c8 bitbake: bitbake: bb.fetch2.git: Import errno module
Currently this module is dereferencing errno.ENOENT but the python module "errno"
is not imported, which causes a crash when fetching from a git repository.

(Bitbake rev: 93e4c9bb2393b1074f5a01e7eaaac742a59d8086)

Signed-off-by: Romain Perier <romain.perier@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-19 22:38:44 +01:00
Christopher Larson
3a5e46b6ed bitbake: bb.fetch2.{git, hg}: remove tarball if it needs updating
We were maintaining state in the form of ud.repochanged to determine whether
we need to write the tarball in the case where the tarball already exists, but
this state is maintained only in memory. If we need to update the git repo,
set ud.repochanged to True, and then are interrupted or killed, the tarball
will then be out of date.

Rather than maintaining this state, simply remove the out of date tarball when
we update the git repo, and it will be re-created with updated content in
build_mirror_data.

[YOCTO #6366]

(Bitbake rev: eaaa81393f181432c8586b17ade623f42c9fed2e)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18 09:05:36 +01:00
Christopher Larson
abba11daf9 bitbake: bb.fetch.git: remove leading '.' from gitsrcname
When using an absolute file URI, there's no host, and the path starts with
'/', the dir under ${DL_DIR}/git2/ ends up starting with '.', so is hidden.
Remove any leading '.' to fix this.

(Bitbake rev: 8dce6964d56b36a77fb113f2ad496cc992a5ff36)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-04 16:23:59 +01:00
Aníbal Limón
bd5167d154 bitbake: fetch2/git.py: latest_versionstring now returns (version, revision)
We need to know the revision associated with the upstream version in
SCM like git, this change broke return convention, oe-core recipeutils
get_recipe_upstream_version need to be updated.

tests/fetch.py: Updated git latest_versionstring test for comply new
convention.

[YOCTO #7605]

(Bitbake rev: fd175dc90024c503134c11cbd83e77d88c406ac8)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-22 08:23:49 +01:00
Aníbal Limón
4bbc5dd6e7 bitbake: fetch2/git.py: latest_versionstring search in all tags
Don't limit the tag search for only tags end with ^{}.

(Bitbake rev: 7006ab313766344cf33481228465082ed5977d28)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-22 08:23:49 +01:00