Commit Graph

227 Commits

Author SHA1 Message Date
Ross Burton
43a2be2897 bitbake: tests/fetch: add a test for download paths without a proper filename
For example the miniupnpd recipe has a SRC_URI like this:

    http://miniupnp.tuxfamily.org/files/download.php?file=${BP}.tar.gz

In this case the path is /files/download.php, which isn't useful when
the latest_upstream logic bails early if there is no version in the path.

The logic now also checks in the downloadfilename, so add a test that
this works as expected.

(Bitbake rev: fffbf5d5e1c8556cddf0794e0b303bb0106747a0)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 13:40:31 +00:00
Ross Burton
58c015cdd7 bitbake: tests/fetch: support setting PV in the wget fetcher
Some code paths in latest_versionstring() need PV to be set correctly.

(Bitbake rev: 0a9f90ff658e09feda63b398ec35715a65ff6193)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 13:40:31 +00:00
Ross Burton
63246241f9 bitbake: tests/fetch: use a namedtuple for the wget test data
Use a named tuple so the test can access named members instead of just
accessing the data via index, which is harder to understand.

(Bitbake rev: 4b15652c84b06f0506c757e2647875a9b1cc7bfe)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 13:40:31 +00:00
Ross Burton
c3fc67960e bitbake: bb/tests: use subtests when looping in a test case
Marking the test iterations as subtests means that when one fails, it
can identify clearly which iteration has failed.

(Bitbake rev: 52c55e681332d7cdbe06f3c9d9c8d77cb0cb93f6)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 13:40:31 +00:00
Christian Lindeberg
c00ad319d4 bitbake: fetch2/gomod: Fix mirroring problem
Build the 'downloadfilename' parameter by replacing path separators in
the module path like the git fetcher builds the mirror tar ball name.
Copy the downloaded file in the fetcher's unpack method like the crate
fetcher instead of calling the base fetcher's unpack method.

(Bitbake rev: 7762cea087597019460d66b04268757bd46befdf)

Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-27 11:19:33 +00:00
Richard Purdie
bdd2b0fee1 bitbake: fetch: Drop multiple branch/revision support for single git urls
We used to use this for bare clones where a single git url could handle multiple
revisions (which implied multiple branches).

We don't use this any more and I doubt we'd want to go back to it. If we
remove it, we can simplfy the looping in the code which seems desireable.

This patch does change the warning for missing branch parameters to a
error. The message has hinted about that for long enough.

Some test cases are removed since they are no longer needed.

(Bitbake rev: 2515fbd10824005fa7f34e87706000c079920366)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-20 13:50:40 +00:00
Richard Purdie
cc5e31ceab bitbake: tests/fetch: Fix typo in npm test
(Bitbake rev: 79b04f61236117d310c12c1b1378ae63afb931ff)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-19 11:46:31 +00:00
Richard Purdie
122de6fd0f bitbake: tests/fetch: Add git tag verification tests
Add tests for git tag verification in both standard and shallow clones.

(Bitbake rev: f47127066d67e2ad80974fa1e7c0fcc7409161af)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-19 11:46:31 +00:00
Richard Purdie
18a533bfe6 bitbake: fetch/git: Rework tag parameter handling
Currently bitbake disallows tag parameters along with revision parameters.
This isn't great since quite often, we'd like to verify that a given revision
does match some tag. At the same time we don't want to or need to access
the network to verify this, which normally a tag would require.

Rework the code so that tag and revisions can both be specified together.
Verify that any tag specified matches the revision in use at unpack time.

This means we can start requiring people to put tags in git SRC_URIs
when revisions are used, making review a little easier that it isn't
some random revision.

The test that is dropped looks like a different test but the comment
is a copy and paste error. The SRCREV/rev mismatch test remains, this
removes the rev and tag set test.

(Bitbake rev: d591d7633fe8d739ec00395920e44910b0b77e27)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-19 11:46:30 +00:00
Richard Purdie
e8e34a04ce bitbake: tests/fetch: Add missing branch parameter to tests
(Bitbake rev: fd01e8e3a5a757d5f506095fc1ac4e45d888ae78)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-19 11:46:30 +00:00
Patrik Nordvall
8180865c22 bitbake: fetch2/git: Restore escape quoting for the git url when used
This fixes a bug where escapes in the url path would not be properly
restored for the git commands in the git fetcher. For example, a
space which is encoded as '%20' was not properly encoded before the
clone command.

e.g.

SRC_URI="git://git.openembedded.org/bitbake%20example/bitbake;protocol=https"

resulted in
git clone 'https://git.openembedded.org/bitbake example/bitbake'

instead of
git clone 'https://git.openembedded.org/bitbake%20example/bitbake'

(Bitbake rev: be48024253b93215bb110cd1d05925e789ec9680)

Signed-off-by: Patrik Nordvall <patrik.nordvall95@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-18 11:03:17 +00:00
Peter Kjellerstedt
55e65e6d70 bitbake: tests/fetch: Verify regular expression "URLs" that contain a ?
A regular expression "URL" in PREMIRRORS and MIRRORS may contain a ? as
part of the regular expression. Make sure this does not cause
problems.

(Bitbake rev: 5af7fe4473cd7e75d4eb7f8b93c499bd157ff156)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-07 15:38:15 +00:00
Stefan Koch
543b1e6ffb bitbake: tests/fetch: Add an additional test case to check whether the fast fetch is shallow
(Bitbake rev: 16f1961e077c525ccfc12496a3deca944df89fc6)

Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-06 11:16:46 +00:00
Stefan Koch
db267f2c94 bitbake: tests/fetch: Adapt test cases for fast shallow fetches
- Address the absence of an initial full bare clone
- Utilize the initial shallow clone
- Modify existing test cases for this behavior
- Remove incompatible test cases

(Bitbake rev: 599fedacd7782dcb52825c22200f35344c102548)

Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-03-06 11:16:46 +00:00
Stefan Herbrechtsmeier
851b24cf81 bitbake: tests: fetch: quote URI password as per RFC3986
The password must be quoted as per RFC3986 because it can contain
reserved characters.

(Bitbake rev: 5c53dbf0ad4385e1de7f2eef66565a1f05dae67a)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Stefan Herbrechtsmeier
34bb313e90 bitbake: tests: fetch: use lower case hostnames
Do not use upper case hostnames because the hostname is case insensitive
and maybe decoded into lower case to follow the common style.

(Bitbake rev: 554b7048412c4c67bf895a8b98822b54ac3a66db)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Stefan Herbrechtsmeier
e69304675d bitbake: tests: fetch: add test for file URI with @
(Bitbake rev: f169704ef48f04da33aaca8b1ad5d8d7eabf8fd6)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-11 21:18:39 +00:00
Richard Purdie
c996cfea52 bitbake: tests/fetch: Fix git shallow test failure with git >= 2.48
From git 2.48 release notes:

"""
When "git fetch $remote" notices that refs/remotes/$remote/HEAD is
missing and discovers what branch the other side points with its
HEAD, refs/remotes/$remote/HEAD is updated to point to it.
"""

This means with git 2.48 onwards, there is a mystery "HEAD" revision
appearing in some of our shallow clone tests. We can avoid this by
using the same canonicalization as used for the reference revisions.

This resolves autobuilder failures on the Fedora 40 workers.

(Bitbake rev: c83444d1210740e27b1744d3aa7c5cad4e28db2f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-23 00:03:06 +00:00
Stefan Herbrechtsmeier
d7c6afc9a9 bitbake: fetch2: do not prefix embedded checksums
The fetcher support entries with an embedded checksum like 'sha256sum'
in the SRC_URI. It adds the parameter 'name' as prefix to the checksums
if the parameter is set. This behavior is unexpected and leads to hacks
in fetchers. Fallback to the checksum without the useless prefix and
set the parameter 'name' in the gomod fetcher unconditional.

(Bitbake rev: 7a86c5a20ea2586f1ae240613644e065e7b21683)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 11:34:04 +00:00
Stefan Herbrechtsmeier
e040b45ad2 bitbake: tests: fetch: replace [url] with urls for npm
Replace [url] with urls to simplify future modifications.

(Bitbake rev: 212f9406585a2a6ce6e128bb5975ea79f47c5522)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 11:34:04 +00:00
Stefan Herbrechtsmeier
c691d4d53f bitbake: tests: fetch: update npmsw tests to new lockfile format
Update npmsw test cases to new lockfile format. The old lockfile format
is required by npm 6 / Node.js 14 which is out of maintenance [2].

[1] https://docs.npmjs.com/cli/v6/configuring-npm/package-lock-json
[2] https://nodejs.org/en/about/previous-releases

(Bitbake rev: bb59d4fdaf8fb81aa3529e431dcdd4c3d665d742)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-08 11:34:04 +00:00
Stefan Herbrechtsmeier
f1745ce249 bitbake: tests: fetch: add npmsw test case for bundled dependencies
The npm package lock and shrinkwrap file list bundled dependencies which
are supplied together with the parent dependency. The bundled
dependencies are marked by a flag. The flag and thereby test depends on
the lock file version. The old lock file version uses a `bundled` flag
and stores dependencies in the `dependencies` list. The new lock file
version uses an `inBundle` flag and stores dependencies in the
`packages` list.

(Bitbake rev: 34fd8ea6abe755e04220fe70b082aa620ae15f86)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-20 16:18:59 +00:00
Stefan Herbrechtsmeier
527e4a7958 bitbake: tests: fetch: do not skip plain npmsw tests
The npmsw fetcher doesn't use the npm command. Remove the
`@skipIfNoNpm()` decorator from plain npmsw tests which don't use the
npm fetcher.

(Bitbake rev: 48f9964a19b8e2e3a73667c9889908eceecf7719)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-20 16:18:59 +00:00
Stefan Herbrechtsmeier
243f9ea060 bitbake: tests: fetch: adapt npmsw tests to fixed unpack behavior
Adapt the npmsw tests to commit 84f102954e10 ("fetch/npmsw: The fetcher
shouldn't have any knowledge of S").

(Bitbake rev: 8194178291d9b47efb2079fe81b704ba6171efc1)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-20 16:18:59 +00:00
Richard Purdie
59d4ee01ca bitbake: tests/fetch: Use our own mirror of mobile-broadband-provider to decouple from gnome gitlab
GNOME gitlab has occasional downtime which impacts bitbake-selftest
and causes autobuilder failures. Switch to our own mirror for test
purposes to avoid those issues.

(Bitbake rev: 91e268b11ed683bd197026f9b36001f6d54ee05c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-29 22:27:45 +00:00
Richard Purdie
2abe7cedbf bitbake: tests/fetch: Use our own mirror of sysprof to decouple from gnome gitlab
GNOME gitlab has occasional downtime which impacts bitbake-selftest
and causes autobuilder failures. Switch to our own mirror for test
purposes to avoid those issues.

(Bitbake rev: 008808755ed6cfeb6c41273e69ce718f0833c26c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-28 14:08:33 +00:00
Christian Lindeberg
6ae52edfbb bitbake: tests/fetch: Update GoModTest and GoModGitTest
Add test cases for a module path consisting of only a hostname and add
checksum verification of the unpacked go.mod files.

(Bitbake rev: 9380859b59923dee17469348f472a22e11be1779)

Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-09-20 13:57:32 +01:00
Christian Lindeberg
87c29b5a94 bitbake: fetch2: Add gomodgit fetcher
Add a go module fetcher for downloading module dependencies to the
module cache directly from a git repository. The fetcher can be used
with the go-mod class in OE-Core.

A module dependency can be specified with:

  SRC_URI += "gomodgit://golang.org/x/net;version=v0.9.0;srcrev=..."

(Bitbake rev: 29ff38ccf0d5389a5bee81e252a78548361a9d7c)

Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-09-10 13:05:00 +01:00
Christian Lindeberg
dd7631426c bitbake: fetch2: Add gomod fetcher
Add a go module fetcher for downloading module dependencies to the
module cache from a module proxy. The fetcher can be used with the
go-mod class in OE-Core.

A module dependency can be specified with:

  SRC_URI += "gomod://golang.org/x/net;version=v0.9.0;sha256sum=..."

(Bitbake rev: 5ff4694bf305e266ebf0abab5d9745c6b6d07d67)

Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-09-10 13:05:00 +01:00
Robert Yang
7bc521ed34 bitbake: bitbake: tests/fetch: Update GitShallowTest for clone_shallow_local()
Update the test cases since the implementation is changed:

* test_shallow_multi_one_uri()
  The a_branch and v0.0 had the same revision, and it required fetch a_branch
  and remove histories of v0.0 which were conflicted, and bitbake reported:
  fatal: no commits selected for shallow requests

  Make a_branch and v0.0 have different revs to fix the problem.

  And now the 'rev^' is not needed, so update self.assertRevCount() as well.

* test_shallow_multi_one_uri_depths()
  Update self.assertRevCount(), now git only fetches the required revs.

* test_shallow_fetch_missing_revs()
  The command is:
  $ git fetch --shallow-exclude=v0.0 master

  But master and v0.0 uses the same revision, so there is no commit to fetch.

* test_shallow_fetch_missing_revs_fails()
  Two unneeded committs are not fetched now:
  - rev^
  - One not specified or required tag.

  So update self.assertRevCount()

(Bitbake rev: 48eff9d9a660ad6b9bd8b53a7dcec600ef42b1d1)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-07-13 23:30:07 +01:00
Richard Purdie
b6b66ca07b bitbake: tests/fetch: Tweak test to match upstream repo url change
Upstream changed their urls, update our test to match.

(Bitbake rev: dc391b86540ec5e0a0f1d811c776a22d443b1c06)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-06-05 18:33:42 +01:00
Richard Purdie
f24ffc087b bitbake: tests/fetch: Tweak to work on Fedora40
On Fedora40, "localhost" sometimes resolves to ::1 and sometimes to 127.0.0.1
and python only binds to one of the addresses, leading to test failures.

Use 127.0.0.1 explicitly to avoid problems of the name resolution, we're trying
to test things other than the host networking.

(Bitbake rev: 9adc6da42618f41bf0d6b558d62b2f3c13bedd61)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-31 16:56:25 +01:00
Kari Sivonen
9925db0c4f bitbake: fetch2/svn: Fix mirroring issue with svn
Add return false to supports_checksum for svn fetcher which fhis
fixes MIRROR usage for svn uris. Also add a testcase.

[YOCTO #15473]

(Bitbake rev: 21cfc7ae9a19f39ac8904e1c3466e7e499ac523f)

Signed-off-by: Kari Sivonen <kari.sivonen@live.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-08 14:51:43 +01:00
Alexander Kanavin
551fdabc54 bitbake: fetch2/crate: add upstream latest version check function
This is actually rather easy: crate web API provides a json
with all the versions, for example:
https://crates.io/api/v1/crates/cargo-c/versions

(Bitbake rev: f6c2755db9a1f88c8534193b420fa31d135945e6)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-05-08 14:51:43 +01:00
Ross Burton
e20ee877ed bitbake: fetch2: handle URIs with single-valued query parameters
Whilst typically the URI query is a list of key-value pairs, that's not
actually required by the URI specification.

For example:  http://example.com/foo?bar is a valid query, but this will
result in the fetcher raising an exception:

  File "bitbake/lib/bb/fetch2/__init__.py", line 265, in __init__
    self.query = self._param_str_split(urlp.query, "&")
  File "bitbake/lib/bb/fetch2/__init__.py", line 293, in _param_str_split
    for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) if x]:
ValueError: not enough values to unpack (expected 2, got 1)

In this case the query is just "bar", but the fetcher is trying to split
it into a key-value pair.

The URI object exposes the parsed query explicitly as a dictionary of
key-value pairs, so we have to be a little creative here: if a value is
None then it isn't a key-value pair, but a bare key.

Fix this by handling elements without the deliminator in _param_str_split()
(by assigning the value to None), and handle a None value when formatting
the query in _param_str_join().

(Bitbake rev: eac583bd4c46f3bb9661852cb6a1448f16147ff1)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-22 16:26:45 +00:00
Philip Lorenz
0e3bcc5103 bitbake: fetch2: Ensure that git LFS objects are available
The current implementation only performs a git lfs fetch alongside of a
regular git fetch. This causes issues when the downloaded revision is
already part of the fetched repository (e.g. because of moving back in
history or the updated revision already being part of the repository at
the time of the initial clone).

Fix this by explicitly checking whether the required LFS objects are
available in the downloade directory before confirming that a downloaded
repository is up-to-date.

This issue previously went unnoticed as git lfs would silently fetch the
missing objects during the `unpack` task. With network isolation turned
on, this no longer works, and unpacking fails.

(Bitbake rev: cfae1556bf671acec119a6c8bbc4b667a856b9ae)

Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-23 14:34:05 +00:00
Peter Kjellerstedt
d25f6b970f bitbake: fetch2/git: Make latest_versionstring extract tags with slashes correctly
Before, everything up to the last slash was removed when extracting the
names of the tags. This would lead to that a tag such as "agent/11.0.0"
would be incorrectly identified as "11.0.0", which would then be treated
as a correct version matching "^(?P<pver>\d+(\.\d+)+)".

(Bitbake rev: 8b21024b9966d5158ac4a77e87ffb935c2a57764)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 15:08:30 +00:00
Peter Kjellerstedt
21603031df bitbake: tests/fetch: Make test_git_latest_versionstring support a max version
In some cases, the version found by latest_versionstring() may be higher
than the real version. Make it possible to specify a maximum version so
that this case can be detected.

(Bitbake rev: 9134d4777109bc78410c3e641420d9a78b485e33)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 15:08:30 +00:00
Chen Qi
09ecbcd307 bitbake: tests/fetch.py: add test case for using premirror in restricted network
We had issue when BB_ALLOWED_NETWORKS is set and `bitbake grpc-native -c fetch'
failed even with all contents available in PREMIRRORS.

Add a test case to ensure no regression in the future.

(Bitbake rev: 80c91ceb81b1cae203067af58d3f1fe9c619ae83)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 15:08:30 +00:00
Pavel Zhukov
de27ecd228 bitbake: tests/fetch.py: add multiple fetches test
Fetch from premirror few times to emulate multiple machines sharing same
clonedir or few rebuilds of the package from (pre)mirror
Regression test for [Yocto #15369]

(Bitbake rev: 7fcbac574c68f16b95ab7abb2874931d168d3c9e)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 12:52:30 +00:00
André Draszik
577e73606a bitbake: tests/fetch: support git's safe.bareRepository
When git is configured with safe.bareRepository=explicit [1], the
bitbake selftests fail miserably. LWN has an article about the
problem that this configuration option addresses and why it is useful
in [2].

It also seems that it is being rolled out in some environments as a
default for users.

In order to allow having this configuration turned on for a user's
environment in general, the fetcher has to be tought to use --git-dir=
for all relevent git operations.

The alternative, implemented here, is to forcibly turn off that option
for all git operations. In the future, we could look into converting
these to using the --git-dir= command line argument instead.

Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1]
Link: https://lwn.net/Articles/892755/ [2]
(Bitbake rev: a45e14a7343e36101e45639931322e5649587f57)

Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 12:00:59 +00:00
Martin Jansa
5276b7f548 bitbake: tests: fetch.py: use real subversion repository
* github no longer provides support for subversion clients:
  https://docs.github.com/en/enterprise-server@3.11/get-started/working-with-subversion-on-github/support-for-subversion-clients
  it was shut down on 2024-01-08:
  https://github.blog/2023-01-20-sunsetting-subversion-support/

  and this test was now failing with:

======================================================================
ERROR: test_external_svn (bb.tests.fetch.SVNTest.test_external_svn)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/OE/layers/bitbake/lib/bb/tests/fetch.py", line 1287, in test_external_svn
    fetcher.download()
  File "/OE/layers/bitbake/lib/bb/fetch2/__init__.py", line 1896, in download
    raise FetchError("Unable to fetch URL from any source.", u)
bb.fetch2.FetchError: Fetcher failure for URL: 'svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2'. Unable to fetch URL from any source.

Stdout:
Fetch svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2
Failed to fetch URL svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2, attempting MIRRORS if available
Fetcher failure: Fetch command export PSEUDO_DISABLED=1; /usr/bin/env svn --non-interactive --trust-server-cert co --no-auth-cache -r 2 file:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project/trunk@2 trunk failed with exit code 1, output:
A    trunk/README.md
 U   trunk
Checked out revision 2.

svn: warning: W205011: Error handling externals definition for 'trunk/bitbake':
svn: warning: W170013: Unable to connect to a repository at URL 'https://github.com/PhilipHazel/pcre2.git'
svn: E205011: Failure occurred processing one or more externals definitions

  in the rare cases where subversion was still installed on the host
  running bitbake-selftest :).

  to avoid this use still alive repository from https://svn.apache.org/
  and pick something rather small and only the trunk subdirectory which
  is fast to fetch:
  svn co https://svn.apache.org/repos/asf/serf/trunk
  takes just 2 sec here

  adjust expected dir/file to use "protocols/fcgi_buckets.h" instead of
  "trunk/README"

(Bitbake rev: a735898abcf056f897c9350bb128a5637e6b4617)

Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-08 10:59:32 +00:00
Desone Burns
7aab731f1e bitbake: bitbake: fetch2: git: Update Git-LFS download and tests
When downloading a Git repository containing an LFS, the Git hooks
are not always initialized correctly to perform the download. This
change updates the Git downloader to run the "git lfs install" command
in order to "smudge" the LFS files. The tests have also been updated
to do checks for situations in which git lfs is not installed, as
the application was required to be installed for any of the tests
to run previously.

The Git LFS functionality was working to some degree previously,
but this change also updates the fetcher to allow LFS downloads for
nobranch SRC_URIs.

(Bitbake rev: 05f8529fb439db93d85a892704b6f2f0ac0c9217)

Signed-off-by: Desone Burns II <dburns@seegrid.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-23 17:37:25 +00:00
Pavel Zhukov
695ba46528 bitbake: tests/fetch.py: Add tests to cover multiple branch/name parameters
Create repository with few branches and test if fetcher can work
with such repository as PREMIRROR

(Bitbake rev: a1737610e5d5b61e126ec3632d7f27b337a87818)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-10-23 11:46:36 +01:00
Richard Purdie
b8b5c786e7 bitbake: selftest/fetch: Ensure top level directory timestamp doesn't break test
The infrastructure changes highlighed an issue where the top level directory
mtime stamp isn't included in the tarballs. This isn't an issue the test is
meant to check for so ignore it.

(Bitbake rev: 5a1e5b8dc55d6bb4d93bac3492f8c43ff957e712)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-10-06 11:42:46 +01:00
Dit Kozmaj
afed7e8747 bitbake: fetch2: Set maxsplit to match expected variables
Set the maxsplit value to match the expected number of variables.
This also avoids an unnecessary split as the parameters are in the form 'key=value'
and the 'value' could contain the '=' character.

(Bitbake rev: 3b17a7ed9bf6cd2808946c2d9c3ed9961af11f19)

Signed-off-by: Dit Kozmaj <dit.kozmaj@kynetics.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-07-27 21:09:14 +01:00
Peter Hoyes
fd1d2c6dc7 bitbake: bitbake: tests: Use assertLogs to test logging output
By default, pytest captures all stdout and exposes it using its built-in
fixtures (capsys, caplog etc), so stdout does not support getvalue().

To support running tests using both unittest and pytest, use assertLogs
to capture logging and assert on the log output instead.

(Bitbake rev: 2d28caa01bab9540d2bbaf713ae3e5c563d003f5)

Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-06-27 15:28:56 +01:00
Peter Hoyes
8b0fc4850d bitbake: bitbake: tests/fetch: Set git config if not already set
git config returns an error code if user.email or user.name are not set,
so bb.process.run raises an ExecutionError, failing the test.

Improve the logic by catching the ExectionError and using this to set a
default value.

Change a direct invocation of 'git init' to use self.git_init so that
it follows the code path above.

Remove the related comment from the README now that git sets up the user
details automatically.

(Bitbake rev: 0c7079d0609a13713b890a9675abd9fba032f199)

Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-06-27 15:28:56 +01:00
Peter Hoyes
baf345598d bitbake: bitbake: tests/fetch: Rename assertRaisesRegexp to assertRaisesRegex
TestCase.assertRaisesRegexp was renamed to assertRaisesRegex in Python
3.2, so rename to fix a warning during test execution.

(Bitbake rev: 9bc2ca3197bff7a5ef6e0ece6bfef0cda68b808e)

Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-06-27 15:28:56 +01:00
Peter Hoyes
a7b74c0b73 bitbake: bitbake: tests/fetch: Mark TestTimeout as not a test suite
pytest test collection attempts to collect all classes containing
"Test". TestTimeout is not a test class so add the __test__ = False
attribute to indicate this.

(Bitbake rev: b5455b98093a3eaf122ebe3d29c3bef949fd2440)

Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-06-27 15:28:56 +01:00