Commit Graph

177 Commits

Author SHA1 Message Date
Bergin, Peter
5c3b6b6fd1 bitbake: lib/bb/tests/fetch.py: set initial branch
If you have a host where the git config for initial branch
name is something else than 'master' the unittest will fail
as they assume the default branch name is 'master'. Fix this
by explicitly set the intial branch name at 'git init'.

(Bitbake rev: 5bcd213c23da30a84baba88b775b1740e6bb77d0)

Signed-off-by: Peter Bergin <peter.bergin@windriver.com>
Signed-off-by: Peter Bergin <peter@berginkonsult.se>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-05-25 10:33:15 +01:00
Peter Kjellerstedt
5b4c7dc600 bitbake: fetch2/crate: Correct unpack for a crate that matches the recipe name
The crate fetcher handles a crate with a name that matches the recipe's
name specially by placing the unpacked code in the current directory
(which typically is ${S}) rather than together with the sources for the
other crates. This broke when the URI names for all crates were changed
recently to include the version in the name.

Correct the crate fetcher to test against ${BP} instead of ${BPN}.
Also add a test case to the selftests to avoid this breaking again.

[Yocto #15012]

(Bitbake rev: 9af52b748357d0c843ce2507ce4d119fd9c37008)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-05-25 10:33:15 +01:00
Richard Purdie
e53cf94d6e bitbake: tests/fetch: Default to https git protocol where possible
The recommendation from server maintainers is that the https protocol
is both faster and more reliable than the dedicated git protocol at this point.
Switch to it where possible.

(Bitbake rev: 879b830fcc60a022682ef22f58c04e7e07d03c63)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-05-05 11:07:42 +01:00
Enrico Scholz
b92918eb0e bitbake: fetch2/crate: create versioned 'name' entries
It is common for rust packages to depend on different versions of the
same crate.  E.g.

|    crate://crates.io/windows_x86_64_msvc/0.42.2 \
|    crate://crates.io/windows_x86_64_msvc/0.48.0 \

Identification only by the plain crate name makes the sha256sum
ambiguous

| SRC_URI[windows_x86_64_msvc.sha256sum] = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0"
| SRC_URI[windows_x86_64_msvc.sha256sum] = "1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a"

and requires lot of manual work to fix the SRC_URI list.

Making the 'name' property unique by appending the version allows
direct copy & paste of reported sha256sum errors to complete the
crates list.

(Bitbake rev: ae2efb05196f9e29ef56ad9a84e2eae5fbdd8030)

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-06 14:28:59 +01:00
Frederic Martinsons
2b12c58724 bitbake: fetch2: Add checksum capability for crate fetcher
This change brings checksum verification of each crate
in a recipe, e.g

| SRC_URI += " \
|     crate://crates.io/aho-corasick/0.7.20 \
|     crate://crates.io/atomic-waker/1.1.0 \
|     crate://crates.io/cc/1.0.79 \
| "
|
| SRC_URI[aho-corasick.sha256sum] = "cc936419f96fa211c1b9166887b38e5e40b19958e5b895be7c1f93adec7071ac"
| SRC_URI[atomic-waker.sha256sum] = "debc29dde2e69f9e47506b525f639ed42300fc014a3e007832592448fa8e4599"
| SRC_URI[cc.sha256sum] = "50d30906286121d95be3d479533b458f87493b30a4b5f79a607db8f5d11aa91f"

That will require to move the checksum initialization
after the possible call to urldata_init method in order
for the crate fetcher to parse the url.

Another way of doing could have been implementing a decodeurl
method that would have been specific for each fetcher class.

(Bitbake rev: 4920686c13dd66f9bfa4f7dd38d6e955f153eeec)

Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-30 10:51:54 +01:00
Richard Purdie
2b8761e5fc bitbake: fetch2: Rename __BBSEENSRCREV -> __BBSRCREV_SEEN
For readability of following patches, rename this internal variable to
allow for others in a similar format.

(Bitbake rev: 450b4a00895ed1f9396a8ff859dc1cc0eccc838f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-22 13:57:04 +00:00
Frederic Martinsons
ee5e6a86fd bitbake: crate.py: authorize crate url with parameters
This allow to have classic fetch parameters
(like destsuffix, sha256, name ...) not being
considered by crate fetcher itself (and so mess
up its download)

Moreover, it allow to overload the name of the downloaded
crate (maybe usefull if there is a naming clash between
two crates coming from different repositories)

(Bitbake rev: 278bd2f1758b8af97552af8d23d16ffb5127a131)

Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-17 17:20:05 +00:00
Mark Asselstine
67d5541f65 bitbake: fetch/npmsw: add more short forms for git operations
>From the npm-install documentation [1] the CLI provides a set of
short forms when the install fetches from git. These include

"github:"
example: npm install github:mygithubuser/myproject

"gist:"
example: npm install gist:101a11beef

"gitlab:"
example: npm install gitlab:mygitlabuser/myproject

"bitbucket:"
example: npm install bitbucket:mybitbucketuser/myproject

Commit 1d8af6aed0a9 [fetch2: npmsw: Add support for github prefix in
npm shrinkwrap version] by Stefan Herbrechtsmeier added support for
the "github:" but the others would marked as 'Unsupported dependency'.

The other prefixes are added in this commit, along with extending the
tests to cover some of these.

However, there is one more short form for github which npm-install
allows which forgoes the prefix altogether.

example: npm install mygithubuser/myproject

Unfortunately this format is a bit problematic as it lacks any easily
identifiable 'marker' to match against, and it could be either the
github short form or install from folder format. Experimentation shows
that the folder format requires a leading './' or '/', so we use this
to rule out the ambiguity.

If this approach to folder and github formats disambiguation is
incorrect it won't matter anyways as the folder format is unrecognized
by the code as-is and thus with this change or without, things would
fail.

Since we have to be less strict in the check for git operations we
move it to be the last install format which we check, such that the
less ambiguous formats can be sorted out first.

[1] https://docs.npmjs.com/cli/v9/commands/npm-install

[Yocto #14236]

(Bitbake rev: 0ac6f6cb5d807919ed13a8b7bb3fb551b79c5a71)

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-26 11:49:41 +00:00
Paulo Neves
ffcc20770d bitbake: tests/fetch: Add real git lfs tests and decorator
Added tests that verify that git-lfs works with an actual
real git-lfs server. This was not previously the case because
the repo in the test was a simulation of git-lfs but not
a real git lfs repo.

The 2 added tests are almost the same but test that the
git lfs file checkout is successfult with or without the
lfs=1 flag. The lfs=1 URI parameter is a quirk that triggers
2 different code paths for git lfs.

lfs=1, when used on git lfs repositories triggers the git lfs
downloading at the fetch bare stage.

lfs query parameter unset triggers the git lfs downloading only
on checkout as an implicit behavior of git. This leads to possible
network access on the unpack stage and outside the DL_DIR.

lfs=0 actually disables git-lfs functionality even if supported.

(Bitbake rev: d2be7f7f652360f13cd66d0850f3e19ffe2afb0a)

Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-22 12:03:39 +00:00
Paulo Neves
41c86ad30e bitbake: tests/fetch: git-lfs restore _find_git_lfs
Not restoring the mocked _find_git_lfs leads to other tests
failing.

(Bitbake rev: 70f848631450bd723c223227c21c60e815ee033d)

Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-22 12:03:39 +00:00
Pavel Zhukov
f962c586fc bitbake: fetch2/tests: Add parameter to recipe_uri
While the parameter is not required it allows testing of possible
regression in fetcher code when parameter specified and mirrors are
used.

(Bitbake rev: 1a21918049091a6d77426dbf8868ffdc14ba1003)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-20 17:49:26 +00:00
Pavel Zhukov
2ac1235b9b bitbake: fetch2/tests: Add test for Mercurial
Basic test for mercurial functionality including PREMIRRORONLY with
disabled network

(Bitbake rev: c0e349a5fc92e9c90afc53800c7e83495393ff79)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-18 16:42:27 +00:00
Alexander Kanavin
6257744d79 bitbake: fetch2/wget.py: correctly match versioned directories
When obtaining latest upstream versions, the code needs
to check if the existing tarball is in a versioned directory
(e.g. component-name/x.y/component-name-x.y.z.tar.gz) and
if it is, it needs to first obtain the list of all
such versioned directories and then check all of them by going
one step up in the directory hierarchy.

Existing code was returning a correct match when the component
name did not have numbers, e.g. a check on 'source/epiphany/43/'
would return 43, but was stopping too soon when the component
name itself had numbers ('source/libxml2/2.10/' would return libxml2).

This change ensures the last match is taken instead of the first.

Also, adjust the fetcher tests to check that versioned directories
are correctly traversed in this case (e.g. the step to go one level
up is taken and a new tarball is discovered in a different versioned
directory).

(Bitbake rev: b6601be22c6d776327acdcd1fa931400f41ac786)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-11 16:33:19 +00:00
Richard Purdie
65779419ad bitbake: tests/fetch: Allow handling of a file:// url within a submodule
CVE-2022-39253 in git meant file:// urls within submodules were disabled. Add
a parameter to the commands in the tests to allow this to continue to work.

(Bitbake rev: 209f7ba352b60722830157054e3fc56cb9c693eb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-10-26 23:02:13 +01:00
Mark Asselstine
cba4d320dc bitbake: tests: bb.tests.fetch.URLHandle: add 2 new tests
Add a test for special characters in user and password to qualify
decodeurl() inspired by a bug report describing that '=' signs in a
password was problematic.

Add a second test to qualify decodeurl() as related to the change in
commit 628c4bf6c89b [fetch2/__init__: handle @ in package names].

Relates to [YOCTO #14476]

(Bitbake rev: ee04cf09c7022168c035affa654773652a49793e)

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-10-26 12:32:08 +01:00
Pavel Zhukov
af64b9b31b bitbake: tests: Add test for possible gitsm deadlock
(Bitbake rev: b0506480baa9bcf3ef645b0aed5a07ad9950245c)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-01 10:07:00 +01:00
Pavel Zhukov
492ec14a37 bitbake: tests: Add Timeout class
The class and exception aim to test rare cases there deadlocks are
possible.
Can be used in context managers:
with Timeout(<value>):
        do_deadlock()

(Bitbake rev: c5fcdd804d422f959a189b270d72123a50e74da6)

Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-01 10:07:00 +01:00
Richard Purdie
63935febda bitbake: fetch2: Drop DL_DIR fallback for local file fetcher
A long time ago, we made DL_DIR a final fallback for the local fetcher.
Since then we added checksum support and task hashes and the world
has changed. There were warnings added some time ago if this fallback
triggers and it is now time to drop it entirely.

The original use case was for sstate however the sstate code now sets
FILESPATH correctly so DL_DIR is no longer needed.

There have been a few small bugs exposed by this change, missing mkdir
calls and some minor test issues that needed tweaks. In general this
simplifies and improves the fetcher code flow though.

This completes a cleanup that ensures local files are correctly covered
at parse time which ensures rebuilds and reparses happen at the right
times.

(Bitbake rev: 3e1444e536c71d3885ef6b9d428807163c309640)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-07-15 12:22:08 +01:00
Paulo Neves
9b398909bd bitbake: fetch: bb.fatal when trying to checksum non-existing files
If the local fetcher was not able to find the file anywhere but it
was included in the SRC_URI for checksumming just make it a fatal
error. Ensure a list of searched locations is included too to match
the runtime error that would have resulted.

(Bitbake rev: 5e3b2ad90d9cd0f248b1cb740637caa24442d101)

Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-07-15 12:22:08 +01:00
Pavel Zhukov
62af35ffb4 bitbake: tests/fetch: Add test for broken mirror tarball
With PREMIRRORS set and BB_NO_NETWORK = "1" bitbake should not try to
fetch into non-initialized git directory if tarball is broken (or not in
gzip format)

[Yocto 14822]

(Bitbake rev: c9aaca3dd2dfdf4a291d6e1f6263037e0f54b4b6)

Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-21 20:50:57 +01:00
Pavel Zhukov
6c5944c114 bitbake: tests/fetch: Add tests for premirror using real project
Existing test uses fake local repo for better performance. Adding
test which uses real (but still small) project hosted on
yoctoproject.org.

(Bitbake rev: 247f3536a691cdaa3f96768d1b42653b1da9ae84)

Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-11 10:30:56 +01:00
Pavel Zhukov
84087dfade bitbake: tests/fetch: Drop unnecessary duplicated function
Function "git" is defined in super class FetcherTest already.

(Bitbake rev: bfed4ad2d50bcd635bb2ef99977e03e879fc79a2)

Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-11 10:30:56 +01:00
Pavel Zhukov
c6afc6d6ed bitbake: Add tests to cover BB_FETCH_PREMIRRORONLY functionality
Basic test to cover functionality of BB_FETCH_PREMIRRORONLY using
local git repository. Local repository has been chosen to allow easy
manipulation with the repo to simulate behaviour reported in
[Yocto 13233]

(Bitbake rev: 773e0815ba0c2183afcb169cda525b7625e60e42)

Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-08 21:53:15 +01:00
Olaf Mandel
d1f254d56f bitbake: fetch2/git: canonicalize ids in generated tarballs
Change the owner information in the mirror tarballs generated using
BB_GENERATE_MIRROR_TARBALLS="1". This is an extension of commit
0178ab83, which used the original pokybuild:user information, but failed
to clean up the numerical user and group ids. Now set the more canonical
values of oe:oe and 0:0.

(Bitbake rev: 37437115d3fb1a9f5d8ed7356a0fc01a408e4f8c)

Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
CC: Marek Vasut <marex@denx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-14 09:48:33 +01:00
Olaf Mandel
5cbdd2b483 bitbake: fetch2/git: stop generated tarballs from leaking info
When using BB_GENERATE_MIRROR_TARBALLS="1" to generate mirror tarballs
of git repositories, they leaked local information: username, group and
time of the last fetch. Remove all these by setting fixed information:

 * uname = pokybuild
 * gname = users
 * mtime = committer time of newest commit in repo

The username and group value were taken from the archives available on
the downloads.yoctoproject.org mirror. The modification time is chosen
so it still retains some relationship to the contents of the archive.

(Bitbake rev: 0178ab83e6312e97e528aa8c5e12105f5165d896)

Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-03-26 09:27:43 +00:00
Richard Purdie
f4bfa6df20 bitbake: tests/fetch: Update for master -> main change upstream
(Bitbake rev: f0fc0fe94161d4dd4f34df8426222ac590ef6736)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-23 18:12:10 +00:00
Richard Purdie
ebfa1700f4 bitbake: fetch2/cooker: Fix source revision handling with floating upstreams
Where a git url uses a tag instead of a full source revision, breakage
can currently occur in builds. Issues include:

* the revision being looked up in multiple tasks (fetch and unpack)
* the risk a different revision may be obtained in those tasks
* that some tasks may not be allowed to access the network
* that a revision may not be consistent throughout a given build
* rerunning a specific task may given inconsistent results

To fix this, stop the workers from cleaning out the source revision store. This
should only be done in the cooker itself (based on current policy).

Also, where the code "sees" an upstream access, mark the recipe as not to be
cached. The reparse re-triggers the upstream lookup by the server.

Add a test to ensure that if get_srcrev isn't called, the user is told they're
using a configuration that is known to break.

(Bitbake rev: 4b5eed1626709ef3dc06b32fd55d40a2a6edd179)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-17 17:51:03 +00:00
Richard Purdie
82e6816024 bitbake: tests/fetch: Add missing branch param for git urls
We started to require the branch parameter for git urls. This updates
the tests to match and quieten some noise.

(Bitbake rev: e5144887f2b6ae266f7b52151005619480e80ec6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-12 17:05:58 +00:00
Peter Kjellerstedt
bef3fe4969 bitbake: fetch2: npm: Put all downloaded files in the npm2 directory
Previously npm files that specify downloadfilename= in the SRC_URI
would be downloaded to the root of ${DL_DIR} rather than in the
${DL_DIR}/npm2 directory where all other npm files are downloaded.
This should make it simpler when setting up and configuring a
premirror with the downloaded npm packages.

(Bitbake rev: 73fa376d6502ab6f2cccfb25a1193d9b1c3c3bc8)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05 17:46:44 +00:00
Peter Kjellerstedt
88993ae030 bitbake: fetch2: Correct handling of replacing the basename in URIs
The solution implementated in commit 96c30007 (fetch2: fix
downloadfilename issue with premirror) missed two corner cases. The
first is if the basename of the original URI also appears somewhere
else in the replacement URI, in which case it would also be replaced.
The second is if the basename of the original URI partially matches
the basename of the replacement URI, in which case the inital part of
the basename from the replacement URI would be left behind.

The second case caused test_npm_premirrors_with_specified_filename to
fail.

The solution is to prefix the basename with a slash when matching to
avoid partial matches, and only replace the basename at the end of the
URI.

This also adds two test cases that test for these problems. Before
this they would give the following errors:

  - ['file:///mirror/example/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
  + ['file:///mirror/some-example-1.0.0.tgz/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
  ?                  +++++       ++++++++++

  - ['file:///mirror/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
  + ['file:///mirror/some-some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
  ?                  +++++

(Bitbake rev: 5924c6f007519cd8ea6cc8b316814d17b43048ca)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05 17:46:44 +00:00
Peter Kjellerstedt
84963eb74a bitbake: tests/fetch: Make test_npm_premirrors work with the current fetcher
There are two totally opposite use cases for how a premirror is
expected to behave in combination with specifying a downloadfilename=
parameter in the SRC_URI. On the one hand there is the expectation
that it works like any other mirror, which means the premirror is
expected to contain a file with the original name specified in the
SRC_URI. On the other hand there is the expectation that one can use
the artefacts downloaded by bitbake in ${DL_DIR} as a premirror, in
which case it is expected to contain a file with the name from the
downloadfilename= parameter.

The latter case has been how downloaded files have been handled until
commit 8a3ff9f3 (fetch2: fix premirror URI when downloadfilename
defined), where the fetcher was changed to store files as per the
first case. This is also when the test_npm_premirrors test case was
modified in commit 5ba191a0 (tests/fetch: add and fix npm tests) to
expect the first case.

The above change was later reverted in commit 96c30007 (fetch2: fix
downloadfilename issue with premirror). However the
test_npm_premirrors test case was not updated to match, and has been
failing ever since. This has probably gone unnoticed because the npm
related test cases require that npm is installed on the host.

This commit updates test_npm_premirrors to expect that premirrors use
the filenames specified by downloadfilename= as this matches the
current fetcher implementation and also is the most likely use case
for premirrors. It also tries to mimic how one typically might setup
the premirror directory by simply copying the download directory.

(Bitbake rev: 9e913ade70474aaeb928814d4763e7105569d63a)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05 17:46:44 +00:00
Peter Kjellerstedt
d715e9105f bitbake: tests/fetch: Only set the Git user name/email if they are not already set
If they are set, assume they are correct.

(Bitbake rev: a3a3a40b96c29051d97d0c96eb01784cbd1f5420)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05 17:46:44 +00:00
Peter Kjellerstedt
d40357665f bitbake: tests/fetch: Unify how git commands are run
This unifies the three separate implementations of git() from
GitMakeShallowTest, GitShallowTest and GitLfsTest into one
implementation in FetcherTest. It also makes use of this in
FetcherLocalTest and FetcherNetworkTest.

(Bitbake rev: 8c8b418c3e6f8ab2535ad405e8006ee1012f3484)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05 17:46:44 +00:00
Peter Kjellerstedt
c1117a8eaa bitbake: tests/fetch: Improve the verbose messages for skipped tests
This changes:

  test_foo .. skipped 'Network tests being skipped'
  test_foo .. skipped 'npm not installed, tests being skipped'

to:

  test_foo .. skipped 'network test'
  test_foo .. skipped 'npm not installed'

to avoid double "skipped" in the output.

(Bitbake rev: e551328a4595c220b94ab5002d978ddb4deeebcf)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05 17:46:44 +00:00
Peter Kjellerstedt
803aff8136 bitbake: tests/fetch: Remove unnecessary \n from mirror variables
Since commit 044fb04d (fetch2: Allow whitespace only mirror entries)
there is no need to separate the entries in mirror variables with
"\n".

(Bitbake rev: ed22b078aae25e804ca5edf062e2c905d3e49426)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05 17:46:44 +00:00
Peter Kjellerstedt
66a7541b4f bitbake: tests/fetch: Skip the crate tests if network tests are disabled
Both test_crate_url and test_crate_url_multi require Internet access.

(Bitbake rev: 34e9daed6a7f766bb6e27c7e02c94b545f4791d8)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-02-05 17:46:44 +00:00
Richard Purdie
12c76723fb bitbake: fetch2: Add crate fetcher
This imports the crate fetcher from OE-Core to resolve various module issues
and adds some very very basic tests of that new fetcher.

(Bitbake rev: 1f06f326fa8b47e2a4dce756d57a9369a2225201)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-01-19 23:03:43 +00:00
Richard Purdie
4a1ba69eab bitbake: tests/fetch: Drop gnu urls from wget connectivity test
These urls are no longer adding much to the test coverage but the intermittent
network issues connecting to them are painful. Drop the urls.

(Bitbake rev: bdf5739c5d831dc97a7d81568f94a0953c71017f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-20 14:39:29 +00:00
Alexander Kanavin
8e03013e19 bitbake: fetch: add a test for version check where compression changes
(Bitbake rev: b6f0c29346ad6463c0e521248633e71886bfb5dc)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-12-09 10:56:49 +00:00
Stefan Herbrechtsmeier
a0320f3944 bitbake: test/fetch: Add striplevel unpack parameter test
(Bitbake rev: 7e1ca7ab50e3c6b642c3c11504c7c8f52cfa4528)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-26 17:01:37 +00:00
Richard Purdie
ffe4af4feb bitbake: fetch2: Fix url remap issue and add testcase
Using "" as a target for .replace() is a really bad idea as it duplicates the replacement
for every character in the string. Add a testcase which triggered this and correct the
code to return the correct result.

(Bitbake rev: 3af1ecf049d2eed56f6d319dc7df6eb4a3d4eebc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-21 11:05:51 +00:00
Richard Purdie
d8e5d5fc16 bitbake: tests/fetch: Update pcre.org address after github changes
vcs.pcre.org was a redirect to github which we use for subversion testing.
With the protocol changes at github and the removal of the redirect, use a
direct address for github.

(Bitbake rev: 6230ca71eb7eb2a6db162e28a01727d00af5299b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-07 15:50:32 +00:00
Richard Purdie
b3dcb6b179 bitbake: tests/fetch: Add test for url parameter rewriting
Add a test to ensure that a parameter like protocol=git can be rewritten
to a different url and protocol.

(Bitbake rev: 69b4f9a09ff74378788cc2ec1ad58cd66b27ca59)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
a4c1325251 bitbake: fetch: Handle mirror user/password replacements correctly
Username or password replacements in URIs were being appended rather than
replaced in mirror url remapping. Fix this and add a test case.

[YOCTO #13823]

(Bitbake rev: 66ad58bb87e5158aced572be4f1d5726bc97fcce)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
2cf162f5fe bitbake: tests/fetch: Update github urls
(Bitbake rev: 07fca7e3ab696ba985b3ef86ab9031d688bf2df2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Richard Purdie
5acc697506 bitbake: fetch2/git: Allow git fetcher to support subdir param
The git fetcher is odd in that it supports destsuffix as a parameter but not
the default documented subdir parameter. destsuffix is more limited as it can't
take absolute paths. Rework the code to correctly support subdir.

Also cleanup to use the None default .get() values and be a bit more pythonic
and use subpath as the variable name for subpath for code clarity.

We could consider dropping destsuffix as a parameter as some future point.

Also fix the tests not to pass in a subdir parameter which was never used
but now causes errors.

(Bitbake rev: 66953f06fe822e4001efabd9fc1c985ea2b03f96)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-03 10:12:42 +00:00
Chen Qi
1e5744c46d bitbake: tests/fetch.py: add test case to ensure downloadfilename is used for premirror
Add a test case test_fetch_premirror_use_downloadfilename_to_fetch to ensure
that 'downloadfilename' is used when fetching from premirror.

Although the other two previous test cases, test_fetch_premirror_specify_downloadfilename_regex_uri
and test_fetch_premirror_specify_downloadfilename_specific_uri already
implicitly contain such verification, we still need to add a very clear
case to ensure no regression.

(Bitbake rev: 20aabc3d53f69949810ecf02295725db947ffef8)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-01 11:12:16 +00:00
Chen Qi
e36765222d bitbake: tests/fetch.py: fix premirror test cases
When downloadfilename is specified, it is used to fetch from premirror.
So fix the test cases accordingly.

(Bitbake rev: 3b4d2e3b5024324058360a2a28f33c34114218d0)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-01 11:12:16 +00:00
Richard Purdie
4359dba4fe bitbake: test/fetch: Update urls to match upstream branch name changes
(Bitbake rev: 473e2a5486bd972ad0f808db089abcb8945d3a48)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-17 22:51:59 +01:00
Jon Mason
2e78ec95be bitbake: bitbake: replace http with https for URLs
https has been the preferred way to access websites for many years now.
Change all of the URLs with a _working_ https server/certificate to use
that URL.

(Bitbake rev: da543cdaf88a387675e25d3555765f1146e4105e)

Signed-off-by: Jon Mason <jdmason@kudzu.us>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-08 17:04:59 +01:00