Some .deb files contain uncompressed data tarballs which do not have an
additional file extension after `.tar`. Add support for such cases while
preserving the existing behaviour.
(Bitbake rev: e3834deb427ceb1d1ae9a96c1e0dec990d7f3805)
Signed-off-by: Isaac True <isaac.true@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When invoking the original git fetcher after downloading a mirror
tarball, BB_FETCH_PREMIRRORONLY is ignored. This leads to git fetch
commands targeting the upstream source being executed silently. Ensure
setting BB_NO_NETWORK before invoking the original fetcher. While this
was only observed for git, setting this in general for all fetcher
types makes sense at this location.
(Bitbake rev: 1b1321f2b60c0a66159e3f20c6befcb0b3ccc4c7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We used to use this for bare clones where a single git url could handle multiple
revisions (which implied multiple branches).
We don't use this any more and I doubt we'd want to go back to it. If we
remove it, we can simplfy the looping in the code which seems desireable.
This patch does change the warning for missing branch parameters to a
error. The message has hinted about that for long enough.
Some test cases are removed since they are no longer needed.
(Bitbake rev: 2515fbd10824005fa7f34e87706000c079920366)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bitbake disallows tag parameters along with revision parameters.
This isn't great since quite often, we'd like to verify that a given revision
does match some tag. At the same time we don't want to or need to access
the network to verify this, which normally a tag would require.
Rework the code so that tag and revisions can both be specified together.
Verify that any tag specified matches the revision in use at unpack time.
This means we can start requiring people to put tags in git SRC_URIs
when revisions are used, making review a little easier that it isn't
some random revision.
The test that is dropped looks like a different test but the comment
is a copy and paste error. The SRCREV/rev mismatch test remains, this
removes the rev and tag set test.
(Bitbake rev: d591d7633fe8d739ec00395920e44910b0b77e27)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This partial reverts commit a5d569c94700f04b8193c6bccae5af619931b00f
which changes decodeurl() to use the URI class to parse the URL instead
of doing it itself. While reusing code is generally a good idea, using
urllib.parse.urlparse() (which the URI class does) to parse the regular
expression "URLs" that are used in PREMIRRORS and MIRRORS does not work.
A regular expression URL containing https?://... would be silently
ignored, while a URL using a negative lookahead such as
git://(?!internal\.git\.server).*/.* would result in a cryptic error:
Exception: re.error: missing ), unterminated subpattern at position 0
The problem is that urllib.parse.urlparse() treats the ? as the start of
URL parameters and thus stops parsing whatever part of the URL it was
parsing. Restore the old function and use it in the PREMIRRORS and
MIRRORS code.
(Bitbake rev: f8a7712754e6d0199a0d227fca288307b935368d)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using BB_FETCH_PREMIRRORONLY we write to the datastore to disable the network.
This change needs to be undo when handling later urls, so operate on a copy of the
datastore to allow this.
Reported by Julian Haller <julian.haller@philips.com>
(Bitbake rev: 67a5ede8ae92ed7dcad29fd0dcfd62c6640b10b2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the path variable from the fetch data instead of decoding the path
manually from the plain unexpanded url.
(Bitbake rev: ad3a29fa6ea53741d4e1786de35f8e7fc4292e7a)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetch data class already expands the type, host, path, user, pswd
and parm variables. The fetcher classes already expand the localfile
variable. The getVar function expands the returned string per default.
Remove unnecessary expand function calls to simplify the code.
(Bitbake rev: 1b1eb037b861fbf20491ac17e519e9eaf232b858)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use the URI class to decode and encode an URL. Remove duplicate code and
unify the behavior.
(Bitbake rev: a5d569c94700f04b8193c6bccae5af619931b00f)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A file URI can't contain a user. Do not treat the @ as reserved
character for a file URI.
(Bitbake rev: 11cf4062f48536547a352e24d6e963d91fdd1190)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is the following warning when executing to bitbake linux-yocto:
bitbake/lib/bb/fetch2/__init__.py:464: DeprecationWarning: 'count' is passed as positional argument
This is because the 4th parameter of re.sub(pattern, repl, string, count=0, flags=0)
is a keyword parameter. We use keyword arguments for parameters that are not positional.
(Bitbake rev: 4bd62639f7023eec9f55ad7909fa9250538de936)
Signed-off-by: Bin Lan <bin.lan.cn@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for the Go mod h1 hash. The hash is based on the Go dirhash
package. The package defines hashes over directory trees and is uses for
Go mod files and zip archives.
(Bitbake rev: deefb01592f717efba68e3997fefd04dc7611d88)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher support entries with an embedded checksum like 'sha256sum'
in the SRC_URI. It adds the parameter 'name' as prefix to the checksums
if the parameter is set. This behavior is unexpected and leads to hacks
in fetchers. Fallback to the checksum without the useless prefix and
set the parameter 'name' in the gomod fetcher unconditional.
(Bitbake rev: 7a86c5a20ea2586f1ae240613644e065e7b21683)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
do_unpack generates a 'Bitbake Fetcher Error: UnpackError' when
uncompressing a single 'file.zst', zstd command line returns the error
'unexpected end of file', and the input file 'file.zst' becomes empty.
The command line is 'zstd --decompress --stdout file.zst > file.zst'.
So the issue is that the input and output filenames are the same.
Fix the output filename when using zstd for a single file.
(Bitbake rev: 5881ca0e0d53937cd25a57ff682a3b7577b0b7f8)
Signed-off-by: Pascal Eberhard <pascal.eberhard@se.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
GitHub Codespaces allow users to use GitHub-hosted pre-configured Docker
containers to work on GitHub repositories in VS Code within the browser
or using a locally running instance of VS Code [1].
Users can define access permissions for Codespaces containers, allowing
them to clone or work with other private repositories [2].
To do this, a git credential helper is injected into the container at
/.codespaces/bin/gitcredential_github.sh, which contains the following
lines:
#!/bin/sh
echo protocol=https
echo host=github.com
echo path=
echo username=PersonalAccessToken
echo password=$GITHUB_TOKEN
The `GITHUB_TOKEN` environment variable is automatically set when the
Codespaces container is created. If authorized by the user, it has the
specified permissions. It is thus required to be able to fetch private
git repos from a Codespace container, so add it to `FETCH_EXPORT_VARS`
to make it available for invocations of git.
[1] https://docs.github.com/en/codespaces/overview
[2] https://docs.github.com/en/codespaces/managing-your-codespaces/managing-repository-access-for-your-codespaces
(Bitbake rev: 71ca83bdd541f70737a3a85f05e40c222611af2f)
Signed-off-by: Oliver Kästner <git@oliver-kaestner.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The sqlite connection handling is causing problems with python 3.13. The
connection can be closed at gc time which causing warnings and those
can appear at 'random' points and break output, causing weird failures
in different tinfoil tools and other tests.
Using sqlite as an IPC was never a great idea so drop that usage entirely
and just use the standard cache mechanism we already have for other
situations.
(Bitbake rev: fdc55bb649cb77456d0ac48a9600ef289a52af18)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>From re on python 3.13 onwards: "Passing count and flags as positional arguments is deprecated.
In future Python versions they will be keyword-only parameters."
Avoid the warning.
(Bitbake rev: 016d872dc3c36792af5c84da1d35a5c86996d795)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When copying files as part of the unpack we currently use cp -p, which
is a shortcut for --preserve=mode,ownership,timestamps.
We do want to preserve timestamps, because some fetchers set these
explicitly.
We don't care about ownership. If the files are owned by us then they
ill remain owned by us, and if they're not then the attempt to change
ownership will be silently ignored. In a shared DL_DIR where files have
group ownership this group access isn't relevant in the single-user build
tree.
We do want to preserve executable bits in the mode, but cp always does
this. The difference between --preserve=mode and no --preserve is that
the mode isn't preserved exactly (no sticky bits, no suid, umask is
applied) but this also isn't a relevant difference in a build tree.
Also expand the arguments to be clearer about what options are being
passed.
The impetus for this is that coreutils 9.4 includes a change in gnulib[1]
and will now try to preserve permission-based xattrs if asked to preserve
the mode. This can result in cp failing when copying a file from a
NFSv4 server with ACLs stored in xattrs to a non-NFS directory where
those xattrs cannot be written:
cp: preserving permissions for ‘./jquery-3.7.1.js’: Operation not supported
The error comes from the kernel refusing to write a system.nfs4_acl
xattr to a file on ext4.
This situation doesn't appear on all systems with coreutils 9.4, at the
time of writing it fails on Ubuntu 24.04 onwards but not Fedora 40. This
is because /etc/xattr.conf is used to determine which xattrs describe
permissions, and Fedora 40 has removed the NFSv4 attributes[2].
Also, use long-form options to make the cp command clearer.
[1] https://github.com/coreutils/gnulib/commit/eb6a8a4dfb
[2] https://src.fedoraproject.org/rpms/attr/blob/rawhide/f/0003-attr-2.4.48-xattr-conf-nfs4-acls.patch
[ YOCTO #15596 ]
(Bitbake rev: 2f35dac0c821ab231459922ed98e1b2cc599ca9a)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a go module fetcher for downloading module dependencies to the
module cache directly from a git repository. The fetcher can be used
with the go-mod class in OE-Core.
A module dependency can be specified with:
SRC_URI += "gomodgit://golang.org/x/net;version=v0.9.0;srcrev=..."
(Bitbake rev: 29ff38ccf0d5389a5bee81e252a78548361a9d7c)
Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a go module fetcher for downloading module dependencies to the
module cache from a module proxy. The fetcher can be used with the
go-mod class in OE-Core.
A module dependency can be specified with:
SRC_URI += "gomod://golang.org/x/net;version=v0.9.0;sha256sum=..."
(Bitbake rev: 5ff4694bf305e266ebf0abab5d9745c6b6d07d67)
Signed-off-by: Christian Lindeberg <christian.lindeberg@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a command is run with a non-null log, errors are only output to the
log and are not returned in the exception. In that case direct users to
that logfile instead of telling the command had no output.
(Bitbake rev: 944fe0a77932a5559e01ae6035c4bffa5185ea6a)
Signed-off-by: Viswanath Kraleti <quic_vkraleti@quicinc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst typically the URI query is a list of key-value pairs, that's not
actually required by the URI specification.
For example: http://example.com/foo?bar is a valid query, but this will
result in the fetcher raising an exception:
File "bitbake/lib/bb/fetch2/__init__.py", line 265, in __init__
self.query = self._param_str_split(urlp.query, "&")
File "bitbake/lib/bb/fetch2/__init__.py", line 293, in _param_str_split
for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) if x]:
ValueError: not enough values to unpack (expected 2, got 1)
In this case the query is just "bar", but the fetcher is trying to split
it into a key-value pair.
The URI object exposes the parsed query explicitly as a dictionary of
key-value pairs, so we have to be a little creative here: if a value is
None then it isn't a key-value pair, but a bare key.
Fix this by handling elements without the deliminator in _param_str_split()
(by assigning the value to None), and handle a None value when formatting
the query in _param_str_join().
(Bitbake rev: eac583bd4c46f3bb9661852cb6a1448f16147ff1)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently when git fetches fail, it destroys all the existing local clone data.
For large repositories this can introduce long build delays when for example,
you just typo'd the git revision hash.
The git fetcher should be able to recover most directories so when the fetch is
for a git repo, avoid removing things unless clean is explicitly called
(e.g. a -c cleanall task).
(Bitbake rev: 1b3cd039fe19b24bd4be9a0202a98cdcbb0e9443)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Support AWS_SESSION_TOKEN which is used in temporary
AWS credentials.
Fixes [YOCTO #15384].
(Bitbake rev: ae1e4c90bbc2002cb2728c64649c095c00220ceb)
Signed-off-by: Toni Lammi <toni.lammi@kone.com>
Reported-by: Toni Lammi <toni.lammi@tl-software.fi>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, there was duplicate clutter in the output, particularly if the
URI points to sstate cache items:
bb.fetch2.FetchError: Fetcher failure for URL: {uri}. URL {uri} doesn't work
(Bitbake rev: 61537b8a98b963e4af265e046d41407b32fa5935)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the following variables to the allow-list, which are
used in the "IAM roles for AWS when using the GitLab chart":
- AWS_ROLE_ARN
- AWS_WEB_IDENTITY_TOKEN_FILE
These variables are set in the CI job environment and are needed to
access the sstate cache artifacts in a connected S3 bucket.
[1] https://docs.gitlab.com/charts/advanced/external-object-storage/aws-iam-roles.html
Reported-by: Zhi Bin Dong <zhibin.dong@siemens.com>
(Bitbake rev: c534526ea73805ee7cc16f3168b05ece10e0c03c)
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds an API to bb.fetch2 to enable users to plug in an unpack
tracer that can trace each source file back to its corresponding
upstream source url, even when multiple upstream sources are combined
together in the same unpack directory. This may be required for software
composition analysis, license compliance, and detailed SBoM generation.
This patch provides only the needed hooks in bb.fetch2 code and a dummy
abstract class defining the API; users may load their own unpack tracer
class by setting the BB_UNPACK_TRACER_CLASS config parameter.
(Bitbake rev: 05051152cc42acc52bcf9af9a696f632fac4307f)
Signed-off-by: Alberto Pianon <alberto@pianon.eu>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Visual studio code is injecting a git credential helper
into every dev container. This helper is forwarding the git credentials
from the host into the container.
In order for this helper to work the REMOTE_CONTAINERS_IPC
needs to be available in the environment for all git operations
that require authentication.
(Bitbake rev: 612bdf09416ff96d08b22cf92b077516d23d0963)
Signed-off-by: Matthias Schnelte <develop@schnelte.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add new functions to return some of the get_srcrev data in new and different
ways. We need two different forms of the data, one is a string to inject into
PKGV, the other is the full revisions as a string to include in hash computations
so that the hash changes when the input revisions change.
This allows us to clean up and simplify the code in OE-Core and move the
version information from PV to PKGV.
(Bitbake rev: ae4dfa2a31c74c0c6c2b14cece822ed1f3d79723)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fetcher allows BitBake to fetch from a Google Cloud Storage
bucket. The fetcher expects a gs:// URI of the following form:
SSTATE_MIRRORS = "file://.* gs://<bucket name>/PATH"
The fetcher uses the Google Cloud Storage Python Client, and
expects it to be installed, configured, and authenticated prior
to use.
If accepted, this patch should merge in with the corresponding oe-core
patch titled "Add GCP fetcher to list of supported protocols".
Some comments on the patch:
There is also documentation for the fetcher added to the User
Manual. I'm still not completely sure about the recommends_checksum()
being set to True. As I've noted in the mailing list, it will throw
warnings if the fetcher is used in recipes without specifying a
checksum. Please let me know if this is intended behavior or if it
should be modified.
Here is how this fetcher conforms to the fetcher expectations
described at this link:
https://git.yoctoproject.org/poky/tree/bitbake/lib/bb/fetch2/README
a) Yes, network fetching only happens in the fetcher
b) The fetcher has nothing to do with the unpack phase so there is no
network access there
c) This change doesn't affect the behavior of DL_DIR. The GCP fetcher
only downloads to the DL_DIR in the same way that other fetchers,
namely the S3 and Azure fetchers do.
d) The fetcher is identical to the S3 and Azure fetchers in this
context
e) Yes, the fetcher output is deterministic because it is downloading
tarballs from a bucket and not modifying them in any way.
f) I set up a local proxy using tinyproxy and set the http_proxy
variable to test whether the Python API respected the proxy. It
appears that it did as I could see traffic passing through the
proxy. I also did some searching online and found posts indicating
that the Google Cloud Python APIs supported the classic Linux proxy
variables, namely:
- https://github.com/googleapis/google-api-python-client/issues/1260
g) Access is minimal, only checking if the file exists and downloading
it if it does.
h) Not applicable, BitBake already knows which version it wants and
the version infomation is encoded in the filename. The fetcher has no
concept of versions.
i) Not applicable
j) Not applicable
k) No tests were added as part of this change. I didn't see any tests
for the S3 or Azure changes either, is that OK?
l) I'm not 100% familiar but I don't believe this fetcher is using any
tools during parse time. Please correct me if I'm wrong.
(Bitbake rev: 8e7e5719c1de79eb488732818871add3a6fc238b)
Signed-off-by: Emil Ekmečić <eekmecic@snap.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add one more verification that checks if localpath is None, because
we can't compute checksum of a None.
(Bitbake rev: 47ab9d21171a834cbac3d1ce368d59fd71d14452)
Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set the maxsplit value to match the expected number of variables.
This also avoids an unnecessary split as the parameters are in the form 'key=value'
and the 'value' could contain the '=' character.
(Bitbake rev: 3b17a7ed9bf6cd2808946c2d9c3ed9961af11f19)
Signed-off-by: Dit Kozmaj <dit.kozmaj@kynetics.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To avoid confusion, rename the local fetcher localpaths function to
something more specific so it doesn't appear similar to the localpaths
top level function.
(Bitbake rev: ada5ea712cc42fff8c55bd261ee6d7579efd4a68)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of only display the first and stop.
For recipe (crago based) that can contains several artifacts
to fetch with their checksum, it will particularly handy
to display all of missing one.
An example of error message would be
NOTE: Executing Tasks
ERROR: zvariant-3.12.0-r0 do_fetch: Missing SRC_URI checksum, please add those to the recipe:
SRC_URI[anyhow-1.0.70.sha256sum] = "7de8ce5e0f9f8d88245311066a578d72b7af3e7088f32783804676302df237e4"
SRC_URI[chrono-0.4.24.sha256sum] = "4e3c5919066adf22df73762e50cffcde3a758f2a848b113b586d1f86728b673b"
SRC_URI[serde-1.0.158.sha256sum] = "771d4d9c4163ee138805e12c710dd365e4f44be8be0503cb1bb9eb989425d9c9"
SRC_URI[system-deps-1.3.2.sha256sum] = "0f3ecc17269a19353b3558b313bba738b25d82993e30d62a18406a24aba4649b"
ERROR: zvariant-3.12.0-r0 do_fetch: Bitbake Fetcher Error: BBFetchException('There was some missing checksums in the recipe')
ERROR: Logfile of failure stored in: /home/jenkins/yocto-poky-master/poky/build/tmp/work/core2-64-poky-linux/zvariant/3.12.0-r0/temp/log.do_fetch.1025
ERROR: Task (/home/jenkins/yocto-poky-master/poky/meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb:do_fetch) failed with exit code '1'
(Bitbake rev: dafa07c080e05975b6319b5adf78a9691c6b6643)
Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change brings checksum verification of each crate
in a recipe, e.g
| SRC_URI += " \
| crate://crates.io/aho-corasick/0.7.20 \
| crate://crates.io/atomic-waker/1.1.0 \
| crate://crates.io/cc/1.0.79 \
| "
|
| SRC_URI[aho-corasick.sha256sum] = "cc936419f96fa211c1b9166887b38e5e40b19958e5b895be7c1f93adec7071ac"
| SRC_URI[atomic-waker.sha256sum] = "debc29dde2e69f9e47506b525f639ed42300fc014a3e007832592448fa8e4599"
| SRC_URI[cc.sha256sum] = "50d30906286121d95be3d479533b458f87493b30a4b5f79a607db8f5d11aa91f"
That will require to move the checksum initialization
after the possible call to urldata_init method in order
for the crate fetcher to parse the url.
Another way of doing could have been implementing a decodeurl
method that would have been specific for each fetcher class.
(Bitbake rev: 4920686c13dd66f9bfa4f7dd38d6e955f153eeec)
Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Bitbake expects a consistent metadata environment when it parses. There
are plenty of ways you can set a recipe to autorev at a point where parts
of the fetcher have already been triggered leading to obsure bugs which
I struggled to debug, let alone anyone not familar with the code.
If anyone is running into the message from the commit, the issue is
likely one of timing. Keep in mind that the anonymous python code
in base.bbclass will expand variables like FILESPATH, WORKDIR and others
which contain PV. The recipe needs to be set to AUTOREV before that
anonymous python runs.
In particular, that means you can't set SRCREV = "${AUTOREV}" in other
anonymous python, it needs to happen earlier.
(Bitbake rev: 4d9ec332d5bfc8b60b54f8ec2a17d34e35aa903a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For readability of following patches, rename this internal variable to
allow for others in a similar format.
(Bitbake rev: 450b4a00895ed1f9396a8ff859dc1cc0eccc838f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These environment variables are needed by gclient and needed to be
passed into fetcher.
(Bitbake rev: 9414ba62454c6b911addf6b0bc02af2afc69b926)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you are behind a corporate proxy, the npm fetcher uses
the proxy IP address already passed in the list of exports.
However, it will error if the proxy injects its own root
CA certificate. Thus, the NODE_EXTRA_CA_CERTS environment
variable must be passed so the user can include their
company's root CA as a trusted CA inside node's
certificate store.
(Bitbake rev: 5daecc25a59ce5bf6c31f5085e6caa7cb543517e)
Signed-off-by: George Kelly <george.kelly1097@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the codeparser cache is set from CACHE, which is typically in
bitbake.conf which means we can't read/write any cache until it is found/read.
We may well have python expressions to parse before that happens.
The net result is suboptimal functioning of the codeparser cache since it will
often be invalidated by data that is never written.
This patch changes the codeparser and filechecksum caches to use BB_CACHE as
their setting and defaults it to ${TOPDIR}/cache.
The patch doesn't change where the "persistent" data such as prserver and
hash-equiavalance resides (PERSISTENT_DIR) or where the metadata parsing
cache resists (still currently CACHE). I've left those for a later patch.
The patch does ensure data parsed by the core datastore parsing calls is
written back since this is now much more useful after this change.
(Bitbake rev: ee89ade5b5a4cf9c53f336d8b800e06fbe436628)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
local file fetches now validate checksums. The checksums for mirror
tarballs of repositories will not match so ignore these checksums.
(Bitbake rev: 6424f4b7e9c1ba8db81346e8b3a806dd035d4551)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If two recipes have conflicting checksums for a file, the code will currently
remove the existing file when a mismatch is downloaded, even if another task
successfully fetched it.
This changes the code to verify the checksum (if possible) before replacing
the file. This removes a potential race window and stops builds failing
everywhere from one incorrect checksum.
To make this work, we need to be able to override localpath and avoid
NoChecksum errors being logged.
(Bitbake rev: 4b8de2e7d12667d69d86ffe6e9f85a7932c4c9a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PN is correct here, bitbake has no knowledge of BPN.
This reverts commit d613e48c07d4b12219270c1359cbf2f390b848dd.
(Bitbake rev: cffcfacb747d41304c857b17bfea646e220b2389)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When checking for the non-existing file, BPN is actually the acutal recipe
name. And we should use BPN for the error message and it also fix the below
test when multilib is enabled.
$ oe-selftest -r bbtests.BitbakeTests.test_invalid_recipe_src_uri
(Bitbake rev: d613e48c07d4b12219270c1359cbf2f390b848dd)
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A long time ago, we made DL_DIR a final fallback for the local fetcher.
Since then we added checksum support and task hashes and the world
has changed. There were warnings added some time ago if this fallback
triggers and it is now time to drop it entirely.
The original use case was for sstate however the sstate code now sets
FILESPATH correctly so DL_DIR is no longer needed.
There have been a few small bugs exposed by this change, missing mkdir
calls and some minor test issues that needed tweaks. In general this
simplifies and improves the fetcher code flow though.
This completes a cleanup that ensures local files are correctly covered
at parse time which ensures rebuilds and reparses happen at the right
times.
(Bitbake rev: 3e1444e536c71d3885ef6b9d428807163c309640)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the mirrors code is trying to create a symlink and the
parent directory doesn't exist, as might be the case for sstate
mirrors where the fetch is into a subdir, it can silently fail.
Ensure the directory exists in this case to avoid issues.
(Bitbake rev: eff16e474ee7dc49ae433420a4c8d15d3314a618)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the local fetcher was not able to find the file anywhere but it
was included in the SRC_URI for checksumming just make it a fatal
error. Ensure a list of searched locations is included too to match
the runtime error that would have resulted.
(Bitbake rev: 5e3b2ad90d9cd0f248b1cb740637caa24442d101)
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
GIT_SSH_COMMAND is more convinient to use if arguments have to be passed
and the user doesn't want to create a wrapper script around ssh.
(Bitbake rev: 5e746cb9d26ce87d6c9d52d9022122081a9811c5)
Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
