Commit Graph

183 Commits

Author SHA1 Message Date
Ross Burton
d1cd7dc31c bitbake: fetch2/git: don't set core.fsyncobjectfiles=0
This git configuration variable is deprecated in 2.36.0 onwards, so git
warns in the logs for every git call.

Luckily the default value has always been false[1], so we can just remove
this.

[ YOCTO #14939 ]

[1] aafe9fbaf4

(Bitbake rev: 8ad310633e0c5d5593631c1196cbdde30147efce)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-10-29 16:22:09 +01:00
Paulo Neves
a6b647af79 bitbake: fetch2: Remove unneeded conditional
The condition will always evaluate to true and
thus is redundant.

(Bitbake rev: be1ee681e8a566564549068dcf90c95c36544815)

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-21 20:20:22 +01:00
Martin Jansa
f594264fcc bitbake: fetch2/git: show SRCREV and git repo in error message about fixed SRCREV
* in recipe with 17 git repos in SRC_URI I've accidentally pasted one SRCREV to
  be one character shorter and because fetcher uses:
    if not ud.revisions[name] or len(ud.revisions[name]) != 40  or (False in [c in "abcdef0123456789" for c in ud.revisions[name]]):
  to decide which SRCREV values are fixed SRCREVs this one was
  considered as tag or branch name, because it was only 39 chars long

  The original error message wasn't very helpful as it doesn't show
  which repo or which SRCREV was considered missing:

  do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None)

  with SRCPV included in PV as error recomments it's a bit better:

  bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve '0a92994d729ff76a58f692d3028ca1b64b145d9' in upstream git repository in git ls-remote output for github.com/Maratyszcza/FP16
  The variable dependency chain for the failure is: SRCPV -> PV -> WORKDIR -> T

  with this change the first error will read:

  do_fetch: Bitbake Fetcher Error: FetchError("Recipe uses a floating tag/branch '0a92994d729ff76a58f692d3028ca1b64b145d9' for repo 'github.com/Maratyszcza/FP16' without a fixed SRCREV yet doesn't call bb.fetch2.get_srcrev() (use SRCPV in PV for OE).", None)

(Bitbake rev: 04dc17bef9b762cef9eecdf91c9f37738d8ae44d)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-07-08 00:09:42 +01:00
Pavel Zhukov
53ed421226 bitbake: fetch2: Honour BB_FETCH_PREMIRRORONLY option
This should fix [Yocto 13353] and related to [Yocto 13233] as well.
Previously if git repo mirror has been updated in between of two builds
fetcher of the second build didn't try updated mirror but switched to
git clone from upstream instead. This is problem for offline builds.
Fix this to raise MirrorException if BB_FETCH_PREMIRRORONLY has been
specified by the mirror doesn't contain SRC_REV.

[YOCTO #13353]
[YOCTO #13233]

(Bitbake rev: b47ecab3e3aad5c5c376ec023aa82a51aa0f3b86)

Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-08 21:53:15 +01:00
Richard Purdie
784d17608a bitbake: fetch/git : Use cat as pager
We don't have less in HOSTTOOLS in OE and this can confuse git. Force the
pager to cat to be consistent and minimal everywhere.

(Bitbake rev: d3d406e8552fdd865dc58b419a84411736475ad2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-08 23:52:29 +01:00
Peter Kjellerstedt
51ed090de1 bitbake: fetch2/git: Simplify the validation of SHA-1 revisions
Also correct two comments, and move slash_re from _revision_key() to
the module top level (together with the new sha1_re).

(Bitbake rev: 98cad8636e9c82bc40a033bb83633ec994758eb0)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-19 14:15:09 +01:00
Olaf Mandel
d1f254d56f bitbake: fetch2/git: canonicalize ids in generated tarballs
Change the owner information in the mirror tarballs generated using
BB_GENERATE_MIRROR_TARBALLS="1". This is an extension of commit
0178ab83, which used the original pokybuild:user information, but failed
to clean up the numerical user and group ids. Now set the more canonical
values of oe:oe and 0:0.

(Bitbake rev: 37437115d3fb1a9f5d8ed7356a0fc01a408e4f8c)

Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
CC: Marek Vasut <marex@denx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-14 09:48:33 +01:00
Olaf Mandel
5cbdd2b483 bitbake: fetch2/git: stop generated tarballs from leaking info
When using BB_GENERATE_MIRROR_TARBALLS="1" to generate mirror tarballs
of git repositories, they leaked local information: username, group and
time of the last fetch. Remove all these by setting fixed information:

 * uname = pokybuild
 * gname = users
 * mtime = committer time of newest commit in repo

The username and group value were taken from the archives available on
the downloads.yoctoproject.org mirror. The modification time is chosen
so it still retains some relationship to the contents of the archive.

(Bitbake rev: 0178ab83e6312e97e528aa8c5e12105f5165d896)

Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-26 09:27:43 +00:00
Zygmunt Krynicki
e03ac523e4 bitbake: fetch2/git: Remove duplicate "the"
(Bitbake rev: 02e29e90eac56e46635e9ae44e5c5da684b70ed2)

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-20 16:45:24 +00:00
Richard Purdie
ebfa1700f4 bitbake: fetch2/cooker: Fix source revision handling with floating upstreams
Where a git url uses a tag instead of a full source revision, breakage
can currently occur in builds. Issues include:

* the revision being looked up in multiple tasks (fetch and unpack)
* the risk a different revision may be obtained in those tasks
* that some tasks may not be allowed to access the network
* that a revision may not be consistent throughout a given build
* rerunning a specific task may given inconsistent results

To fix this, stop the workers from cleaning out the source revision store. This
should only be done in the cooker itself (based on current policy).

Also, where the code "sees" an upstream access, mark the recipe as not to be
cached. The reparse re-triggers the upstream lookup by the server.

Add a test to ensure that if get_srcrev isn't called, the user is told they're
using a configuration that is known to break.

(Bitbake rev: 4b5eed1626709ef3dc06b32fd55d40a2a6edd179)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-17 17:51:03 +00:00
Richard Purdie
5fb6b453f6 bitbake: fetch/git: Show warning for invalid github urls
On master, tell the users they need to update their urls for github.

(Bitbake rev: 42526a402357e04794f4cb6f21ac18f562220a9b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
489fe8ff39 bitbake: fetch/git: Handle github dropping git:// support
github is dropping support for git protocol in Git urls. Add code to remap
this to https in a way that could be used in older bitbake versions.

(Bitbake rev: f05e19135b3ddee509c0cb427b3b9376bb4738d1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
d576d9ca3a bitbake: fetch2/git: Add a warning asking users to set a branch in git urls
There is much uncertainty around what tools and hosting providers will
do about default git branch naming in the future. To help ensure we
can handle the various scenarios, we will make branch names required in
SRC_URI. To start that process, show users a warning if it isn't set.
This may also allow us to change the default at some point in the future.

(Bitbake rev: 86a9c26828479be55865bcce72bcc7e12b93caa7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
5acc697506 bitbake: fetch2/git: Allow git fetcher to support subdir param
The git fetcher is odd in that it supports destsuffix as a parameter but not
the default documented subdir parameter. destsuffix is more limited as it can't
take absolute paths. Rework the code to correctly support subdir.

Also cleanup to use the None default .get() values and be a bit more pythonic
and use subpath as the variable name for subpath for code clarity.

We could consider dropping destsuffix as a parameter as some future point.

Also fix the tests not to pass in a subdir parameter which was never used
but now causes errors.

(Bitbake rev: 66953f06fe822e4001efabd9fc1c985ea2b03f96)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
435bce9bf9 bitbake: fetch2/git: Use os.rename instead of mv
os.rename will overwrite the destination file if present so we can use this
instead of the process call overhead.

(Bitbake rev: b3cccaa6a896c41d8c9be5eebc327f726542d16b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-21 23:44:08 +01:00
Richard Purdie
626e949292 bitbake: fetch2/git: Avoid races over mirror tarball creation
There is a potential race over the mirror tarballs where a partial git repo
could be extracted causing fetcher failures if the tarball is being rewritten
whilst another build accesses it.

Create the mirror tarball atomically to avoid this.

[YOCTO #14441]

(Bitbake rev: 3250bc950c56bd7dd2114df26e5a8e13b04ceac8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-17 07:26:23 +01:00
Adam Romanek
6cd8032e35 bitbake: fetch/git: run gc in foreground to avoid race with tar
It looks like git gc can interrupts the package creation when
BB_GENERATE_MIRRORS_TARBALL is in use.

Log excerpts:

tar -czf TOPDIR/../downloads/git2_bitbucket.name-hidden.git.tar.gz . failed with exit code 1, output:
tar: ./objects/pack/pack-89a1d76f6c08f53172ef1d02ff851d90564362c4.pack: file changed as we read it
tar: ./objects/pack/pack-b4a48ada355d333630fdf6b4f67205b7c264dc2c.idx: File removed before we read it

Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.

(Bitbake rev: a8d8cb847063862d1a7998963dd8b767ff73d877)

Signed-off-by: Adam Romanek <romanek.adam@gmail.com>
Signed-off-by: Damian Wrobel <dwrobel@ertelnet.rybnik.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-10 22:43:41 +01:00
Niels Avonds
d12a06cd49 bitbake: fetch/gitsm: Fix crash when using git LFS and submodules
Gitsm fetcher crashes when cloning a repository that contains LFS files.
This happens because the unpack method is called during download, but the
submodules have not been downloaded yet at this point.

This issue was introduced in this
commit: 977b7268bf

[YOCTO #14283]

(Bitbake rev: 26caedc4d2e9b5a0f1d57f9291754a7f6c5e437e)

Signed-off-by: Niels Avonds <niels@codebits.be>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-20 13:57:50 +01:00
Henning Schild
2cb6ce7788 bitbake: fetch/git: add support for disabling shared clones on unpack
By default the unpacker will create a "shared" clone when cloning from
the DL_DIR to the WORKDIR. This patch introduces an option to control
that behaviour.

Imagine some recipe steps are executed in a namespace that is different
from the one your downloader and unpacker ran in. (chroot) Because a
"shared" clone has an absolute reference to its "alternate" you now
have to make that "alternate" visible in that new namespace (chroot) at
the exact place.

With this patch you can unpack "noshared" and get a stand-alone copy.
This copy will also work if the "alternate" is not visible or existant.

The switch is a global bitbake switch and will affect all git urls.
Build systems that need "noshared" most likely need it for everything
they do with git.

(Bitbake rev: 6ae6f1865d5e666ebc670f70b7401a7b41648102)

Signed-off-by: Henning Schild <henning.schild@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-18 11:38:22 +01:00
Matt Hoosier
45c02843d1 bitbake: fetch/git: download LFS content too during do_fetch
Insert an explicit pass to fetch all blobs needed by Git LFS, during the
fetch() function. This avoids the default behavior of Git LFS to wait
until 'git checkout' to begin downloading the blobs pointed to by LFS records.
Network access is not allowed at that point in the recipe's lifecycle.

[YOCTO #14191]

(Bitbake rev: 0efac66043662e7a2027192f50e92e982db2ba1c)

Signed-off-by: Matt Hoosier <matt.hoosier@garmin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-23 17:10:46 +00:00
Joey Degges
10c69538c0 bitbake: fetch/git: Fix usehead for non-default names
The usehead url parameter for git repositories causes bitbake to use
whatever commit the repository HEAD is pointing to if the repository
happens to have the name 'default'. This is the default name so in many
cases it works just fine, but if a different name is specified with the
url parameter 'name=newName' then it will fail to parse the recipe with
an error along the lines of:

ERROR: ExpansionError during parsing /path/to/my/recipe.bb
Traceback (most recent call last):
  File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 235, in Git.urldata_init:
    >        ud.setup_revisions(d)
  File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1302, in FetchData.setup_revisions:
             for name in self.names:
    >            self.revisions[name] = srcrev_internal_helper(self, d, name)
  File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1167, in srcrev_internal_helper(name='newName'):
         if srcrev == "AUTOINC":
    >        srcrev = ud.method.latest_revision(ud, d, name)
  File "/path/to/poky/bitbake/lib/bb/fetch2/__init__.py", line 1562, in Git.latest_revision(name='newName'):
             except KeyError:
    >            revs[key] = rev = self._latest_revision(ud, d, name)
                 return rev
  File "/path/to/poky/bitbake/lib/bb/fetch2/git.py", line 650, in Git._latest_revision(name='newName'):
             raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \
    >            (ud.unresolvedrev[name], ud.host+ud.path))
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'master' in upstream git repository in git ls-remote output for /path/to/local/git/repo

Let's fix this by setting the unresolved rev of _all_ repository names
to 'HEAD' when the usehead url parameter is specified. Update the
currently failing test, test_local_gitfetch_usehead_withname, to now
expect success.

This change preserves existing behavior that allows usehead to be
overridden by a valid looking revision if one happens to be specified
instead of AUTOREV.

(Bitbake rev: 01e901c44ab0f496606b1d45c8953dc54970204c)

Signed-off-by: Joey Degges <jdegges@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-08 10:11:42 +00:00
Richard Purdie
bd98ef572d bitbake: fetch2/git: Document that we won't support passwords in git urls
This keeps coming up, document why we don't do this and the alternatives.

(Bitbake rev: aded964eed4ce5a725ed1ab477efabc86b1aa481)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-03 08:21:10 +00:00
Charlie Davies
b50b6007e5 bitbake: bitbake: fetch/git: use shlex.quote() to support spaces in SRC_URI url
This commit replaces the instances where escaped double quotes
are used to support SRC_URI url containing spaces with the more
pythonic shlex.quote().

(Bitbake rev: 4f9ba9c794de55bea0343267467bddea99844374)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-17 12:36:14 +01:00
Charlie Davies
9ef2bc360d bitbake: bitbake: fetch/git: add support for SRC_URI containing spaces in url
Microsoft's TFS VCS system allows for spaces in a git repository url.
An example of a valid url is:

ssh://tfs-my-company.org:22/tfs/My Projects/FooBar

This commit adds support for such urls by implementing two changes.
Firstly, when bitbake makes a git command line call the url is
surrounded by quotes so that the url, regardless of spaces, is
treated as one argument. Secondly, additional parsing of various
filepath variables, which are based off of the url, are now
completed with any spaces in the url replaced with underscores.

(Bitbake rev: eb38b6f0935763f7ba19e5618f376fcae1dac41a)

Signed-off-by: Charlie Davies <charles.davies@whitetree.xyz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-09-23 20:55:53 +01:00
Richard Purdie
61931d14df bitbake: fetch2: Change git fetcher not to destroy old references
It looks like we're about to see a lot of changes in branch names in repos. If
we have the prune option here, those old names are lost, the changes propagate
to our source mirrors and our old releases break.

We have the force option so any replaced references should be replaced, its only
orphaned branches which will now be preserved.

I believe this behaviour will cause us fewer problems given the changes that
look likely to happen.

(Bitbake rev: 820ab886e79eea516560c0c008e4cf059c6e11a3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-07-12 11:53:00 +01:00
Mauro Queirós
d648096794 bitbake: git.py: Use the correct branch to check if the repository has LFS objects.
Function "contains_lfs" was only looking at the master branch when searching for LFS
content. LFS may be configured in specific branches only, so we need to use the
correct branch.

(Bitbake rev: 4fa67c2af830035a1ddedc14592ee25a15ebff22)

Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04 13:28:16 +01:00
Mauro Queirós
3f8b8f8bca bitbake: git.py: LFS bitbake note should not be printed if need_lfs is not set.
The message "Repository %s has LFS content but it is not being fetched" was
being printed, even when Git-LFS was available and "lfs=1" was set. In those
situations, we want to fetch LFS content, so that message would not make sense.

(Bitbake rev: 45028dfda5a29a34ab408cb3f11d72ae17963340)

Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04 13:28:16 +01:00
Mauro Queirós
8051b79733 bitbake: git.py: skip smudging if lfs=0 is set
Git-LFS objects were being fetched even when lfs=0 was not set.
This patch disables LFS smudging when lfs=0. That way, only the LFS pointers
are downloaded during checkout.

(Bitbake rev: 646d86df7de774255246a3d7051c308e43eb257d)

Signed-off-by: Mauro Queiros <maurofrqueiros@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04 13:28:16 +01:00
Chris Laplante
96adf215cc bitbake: fetch2/git: _revision_key: collapse adjacent slashes
>From a SRCREV caching point of view, there is no reason to treat the
following upstreams as different:

    SRC_URI = "git://github.com/file/file.git"
    SRC_URI = "git://github.com//file/file.git"

(Bitbake rev: 425e21c14955dd38868c6e97637df3bbe0f89fac)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19 13:31:05 +00:00
Frazer Clews
fa5524890e bitbake: lib: amend code to use proper singleton comparisons where possible
amend the code to handle singleton comparisons properly so it only checks
if they only refer to the same object or not, and not bother
comparing the values.

(Bitbake rev: b809a6812aa15a8a9af97bc382cc4b19571e6bfc)

Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19 13:31:05 +00:00
Christopher Larson
d52cb2f01a bitbake: fetch2/git: fetch shallow revs when needed
When bitbake determines if a git clone needs updating, it only checks for the
needed srcrevs, not the revs listed in BB_GIT_SHALLOW_REVS, which will fail if
using shallow and the needed rev was added to the upstream git repo after a
previous fetch. Ensure that we also check for shallow revs.

[YOCTO #13586]

(Bitbake rev: 93b65e3c6ca64f644946953980595c44fbbcc748)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-28 05:31:53 +00:00
Ross Burton
560358e922 bitbake: fetch2/git: refactor check for git-lfs command
Refactor the git-lfs checking: this means both clearer code in the download()
function and allows unit testing to monkeypatch the functionality.

(Bitbake rev: 33cf9172ded50a869f7201ba463ab9ecc69b8252)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-27 13:02:19 +01:00
Ross Burton
e35be7aa7d bitbake: fetch2/git: add git-lfs toggle option
Add a new 'lfs' option to the git fetcher so that we can optionally not fetch
git-lfs content, for repositories that contain LFS data that we don't
actually need for building.

By default lfs is set to 1, so if the repository has LFS content then git-lfs is
required.  Setting lfs to 0 will mean that git-lfs won't be required to fetch,
and some files will be missing.

(Bitbake rev: be0b78ccfc5ede98041bc0545a15092494b12b26)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-09-19 20:30:35 +01:00
Richard Purdie
9501864db8 bitbake: bitbake: Strip old editor directives from file headers
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.

(Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
cf9c0be3f6 bitbake: bitbake: Drop duplicate license boilerplace text
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.

(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
79834a7144 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:04 +01:00
Naveen Saini
e7c2bc67b5 bitbake: bitbake: fetch2/git: git-lfs check
Build will fail if repository has lfs contents in absense of git-lfs tool on host.
Build will pass if repository may or may not contains lfs content if host has git-lfs installed.

[YOCTO #13198]

(Bitbake rev: 3f2c2eb2f59707828bdcdd6414db837da8dc3b0e)

Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-29 14:16:31 +01:00
Robert Yang
64c21baf63 bitbake: fetch2: Unify BB_FETCH_PREMIRRORONLY
The fetch2/__init__.py checks whether "BB_FETCH_PREMIRRORONLY" == "1", but
fetch2/git.py and hg.py checks whether it is None, this makes it discontinuous,
and BB_FETCH_PREMIRRORONLY = "0" doens't work as expected in the later case,
so unify it to the previous one. (As BB_NO_NETWORK does).

And also use bb.utils.to_boolean() to make them consistent.

(Bitbake rev: 85a0d22835588e9ad8ec29d88a8115227e88420c)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-26 14:02:28 +00:00
Robert Yang
db91c23871 bitbake: fetch2/git: Fix clean to remove clonedir
The localpath is a symlink to clonedir when it is cloned from a mirror, for
example:
$ bitbake systemtap-native -cfetch
$ ls downloads/git2
sourceware.org.git.systemtap.git -> /path/to/downloads/git2/mirror.path.git.sourceware.org.git.systemtap.git
mirror.path.git.sourceware.org.git.systemtap.git

There are both sourceware.org.git.systemtap.git and
mirror.path.git.sourceware.org.git.systemtap.git in DL_DIR/git2, the symlink
sourceware.org.git.systemtap.git is created by try_mirror_url(), but
do_cleanall" only removed the symlink, didn't remove the real dir
mirror.path.git.sourceware.org.git.systemtap.git, this may cause confusions,
for example, I assumed that do_cleanall removed everything, but it didn't, and
it would the re-used next time when do_fetch. This patch fixes the problem.

(Bitbake rev: 452e2200ad2c29dec3753f5f7a8cbc9183ec7dd8)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-24 17:01:34 +00:00
André Draszik
2fcbc079e2 bitbake: bitbake: remove True option to getVar calls
getVar() has been defaulting to expanding by default for
a long time (2016), thus remove the True option from
getVar() calls with a regex search and replace.

Search & replace made using the following command:
    sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \
        -i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \
             | cut -d':' -f1 \
             | sort -u)

(Bitbake rev: 3bba0dbd524cf72176a765957adff544ae5c255a)

Signed-off-by: André Draszik <andre.draszik@jci.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-12 22:01:10 +00:00
Richard Purdie
e5455662a9 bitbake: bitbake: Fix Deprecated warnings from regexs
Fix handling of escape characters in regexs and hence fix python
Deprecation warnings which will be problematic in python 3.8.

(Bitbake rev: c1fcc46e2498ddd41425d8756754f814d682aba3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-16 15:35:07 +00:00
Christopher Larson
e3e88600ce bitbake: fetch/git: fix AttributeError in shallow extraction logic
This code checks to see if shallow is either disabled or the tarball is
missing, but the else block tries to print the tarball filename, and
this attribute doesn't exist at all when shallow is disabled. Handle the
two cases separately to give sane errors for both cases without the
exception:

    Exception: AttributeError: 'FetchData' object has no attribute 'fullshallow'

(Bitbake rev: bdbb558342ebb4e64384c9838d2485d9299d91a6)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-22 10:45:14 +00:00
Urs Fässler
a8a077cafb bitbake: fetch2/git: provide information about missing sources
Provide more information in the case the sources are not found in the
unpack step.

(Bitbake rev: 27a2214bf6f2e7c61bfc422a20959a55f7e0d25d)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:27 +01:00
Urs Fässler
8553b2c111 bitbake: fetch2/git: explicitly show the decision logic to select the source in code
(Bitbake rev: 78d8fb3a7899e1404a513be1c2b2b4440da8b12d)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:26 +01:00
Urs Fässler
0d4fe0602e bitbake: fetch2/git: prevent access to non-existing clonedir
A user friendly error is throw when neither the clonedir nor
fullshallow exist. Without the check, a difficult to interpret error
is throw from within the fetch command.

(Bitbake rev: 30cf2506007d25162f0805051212f54c39034ff3)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-18 10:59:26 +01:00
Urs Fässler
82f32d99f7 bitbake: fetch2/git: use correct check to decide if the shallow tarball should be unpacked
The shallow_tarball check is never true due a check on the caller side.
The tarball check is not related to the code on the caller side.

(Bitbake rev: 086eddcf8c7520ff5c52ce2a11ca9bf5b5fe5d7e)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04 23:00:24 +01:00
Urs Fässler
b907303b92 bitbake: fetch2/git: prevent access to non-existing fullshallow tarball
(Bitbake rev: b7f00a8c11672a2ee0408e210fb174cda3384e3f)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04 23:00:24 +01:00
Urs Fässler
073c62cb56 bitbake: fetch2/git: improve readability of method need_update
To improve the readability we extract the different scenarios of why
the clonedir needs an update.

(Bitbake rev: 9038e029f4f0ab413727de76c74248cbb3cdc9ea)

Signed-off-by: Urs Fässler <urs.fassler@bbv.ch>
Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-04 23:00:24 +01:00
Paulo Neves
a8368651ff bitbake: fetcher: Fixed remote removal not throwing exception.
Before this fix it is assumed that the removal of the
remote can only fail because there is not remote to remove. This
is a false assumption. Example error which would be ignored:

git -c core.fsyncobjectfiles=0 remote rm origin failed with exit code 1, output:
Note: A branch outside the refs/remotes/ hierarchy was not removed;
to delete it, use:
  git branch -d master
error: could not lock config file config
error: Could not remove config section 'remote.origin'

Due to the masking of this error a stranger error will be
presented to the user, because this time we do not mask the
exception:

git -c core.fsyncobjectfiles=0 remote add --mirror=fetch origin https://github.com/ptsneves/tl-wn722.git failed with exit code 128, output:
fatal: remote origin already exists.

The most likely reason that the remote cannot be removed nor
modified is that the DL_DIR/git2 does not have permissions
compatible with the user running bitbake.

This commit fixes:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12728

(Bitbake rev: 9c86c582a10c9b23abad7d34b6cbf12f7086294d)

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-28 10:32:08 +01:00
Andre McCurdy
7838294271 bitbake: fetch2: unify the way fetchers determine DL_DIR and FETCHCMD
Currently there is quite some variation between the fetchers in terms
of how they determine the subdirectory within DL_DIR and the base
fetch command to run. Some rely on variables being set externally
(e.g. from bitbake.conf in oe-core), some respect these external
variables but provide fallback defaults and some use only hardcoded
internal values. Try to unify the approach used across the various
fetchers.

(Bitbake rev: efd5e35af4b08501c67e8b30f30d9457f6fdf610)

Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-06-15 17:57:30 +01:00