The ftp server at ftp.gnu.org is likely to be retired at some point soon
so siwtch over to the http/https services.
This means bitbake-selftest doesn't have ftp test urls, however finding stable
ftp test servers is proving increasingly hard.
(Bitbake rev: 892a08245ddb21a464aeb37d3e32377e99dd7e2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of not even having the test functions if network tests are disabled, use
a custom decorator to mark the network tests and skip them.
(Bitbake rev: cc420f430b1dafd9ca944bea259a564aaab34595)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is a network failure the return value from latest_versionstring() is
('','') which later causes an exception when comparing versions.
Improve this by checking the return value and failing the test early.
[ YOCTO #12053 ]
(Bitbake rev: 3f034d2172bf64ecc43577b43e0cf032a54b1358)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
`git fetch --tags` seems to interact badly with `mirror=fetch`, resulting in
the regular branches not being fetched, so drop the unnecessary `--tags`. This
fixes this unit test failure: `bb.fetch2.FetchError: Fetcher failure: Unable
to resolve 'master' in upstream git repository in git ls-remote output for
/tmp/tmp4ag_mgmn/gitsource`
[YOCTO #11698]
(Bitbake rev: 2d0203fae08c5ff8dc3e9afaa9a819abc4a1af6f)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change adds a new unit test module (bb.tests.event)
for bitbake event.
It includes the following items:
- Client and server stubs setup
- Testing the module's main functions including:
- get_class_handlers
- set_class_handlers
- clean_class_handlers
- enable_threadlock
- disable_threadlock
- get_handlers
- set_handlers
- execute_handler
- fire_class_handlers
- print_ui_queue
- fire_ui_handlers
- fire
- fire_from_worker
- register
- remove
- register_UIHhandler
- unregister_UIHhandler
- Testing event handling using:
- class Event(object)
- class OperationStarted(Event)
- class OperationCompleted(Event)
- class OperationProgress(Event)
- class ConfigParsed(Event)
[YOCTO #10368]
(Bitbake rev: 0be3ad391adc73cc0dff81bd0ed7874f2c6a00be)
Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 22bb7c9270f02ddae72e13d849375feee5f4a98b)
Signed-off-by: Ola x Nilsson <olani@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In certain cases, it's valuable to be able to exert more control over what
history is removed, beyond srcrev+depth. As one example, you can remove most
of the upstream kernel history from a kernel repository, keeping predominently
the non-publically-accessible content. If the repository is private, the
history in that repo couldn't be restored via `git fetch --unshallow`, but
upstream history could be.
Example usage:
# Remove only these revs, not at a particular depth
BB_GIT_SHALLOW_DEPTH_pn-linux-foo = "0"
BB_GIT_SHALLOW_REVS_pn-linux-foo = "v4.1"
(Bitbake rev: 97f856f0455d014ea34c28b1c25f09e13cdc851b)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we're building from a shallow mirror tarball, we don't want to do
anything with ud.clonedir, as it's not being used when we unpack. As such,
disable updating annex in that case. Also include annex files in the shallow
tarball.
(Bitbake rev: ca0dd3c95502b22c369fbf37f915f45e02c06887)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we're building from a shallow mirror tarball, we don't want to do
anything with ud.clonedir, as it's not being used when we unpack. As such,
disable updating the submodules in that case. Also include the repositories in
.git/modules in the shallow tarball. It does not actually make the submodule
repositories shallow at this time.
(Bitbake rev: 6c0613f1f2f9d4f009545f82a9173e80396f9d34)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
By default, all unused refs (branches & tags) are removed from the repository,
as shallow processing scales with the number of refs it has to process. Add
the ability to explicitly specify additional refs to keep. This is
particularly useful for recipes with custom checkout processes, or whose
git-based versioning requires a tag be available (i.e. for `git describe
--tags`). The new `BB_GIT_SHALLOW_EXTRA_REFS` variable is a space-separated
list of refs, fully specified, and support wildcards.
Example usages:
BB_GIT_SHALLOW_EXTRA_REFS = "refs/tags/v1.0"
BB_GIT_SHALLOW_EXTRA_REFS += "refs/heads/*"
(Bitbake rev: 1771934cd9f8b5847c6fcae0a906fb99d6b0db16)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow the user to explicitly adjust the depth for named urls/branches. The
un-suffixed BB_GIT_SHALLOW_DEPTH is used as the default.
Example usage:
BB_GIT_SHALLOW_DEPTH = "1"
BB_GIT_SHALLOW_DEPTH_doc = "0"
BB_GIT_SHALLOW_DEPTH_meta = "0"
(Bitbake rev: 9dfc517e5bcc6dd203a0ad685cc884676d2984c4)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This adds support to the git fetcher for fetching, using, and generating
mirror tarballs of shallow git repositories. The external git-make-shallow
script is used for shallow mirror tarball creation.
This implements support for shallow mirror tarballs, not shallow clones.
Supporting shallow clones directly is not really doable for us, as we'd need
to hardcode the depth between branch HEAD and the SRCREV, and that depth would
change as the branch is updated.
When BB_GIT_SHALLOW is enabled, we will always attempt to fetch a shallow
mirror tarball. If the shallow mirror tarball cannot be fetched, it will try
to fetch the full mirror tarball and use that. If a shallow tarball is to be
used, it will be unpacked directly at `do_unpack` time, rather than extracting
it to DL_DIR at `do_fetch` time and cloning from there, to keep things simple.
There's no value in keeping a shallow repository in DL_DIR, and dealing with
the state for when to convert the clonedir to/from shallow is not worthwhile.
To clarify when shallow is used vs a real repository, a current clone is
preferred to either tarball, a shallow tarball is preferred to an out of date
clone, and a missing clone will use either tarball (attempting the shallow one
first).
All referenced branches are truncated to SRCREV (that is, commits *after*
SRCREV but before HEAD are removed) to further shrink the repository. By
default, the shallow construction process removes all unused refs
(branches/tags) from the repository, other than those referenced by the URL.
Example usage:
BB_GIT_SHALLOW ?= "1"
# Keep only the top commit
BB_GIT_SHALLOW_DEPTH ?= "1"
# This defaults to enabled if both BB_GIT_SHALLOW and
# BB_GENERATE_MIRROR_TARBALLS are enabled
BB_GENERATE_SHALLOW_TARBALLS ?= "1"
(Bitbake rev: 5ed7d85fda7c671be10ec24d7981b87a7d0d3366)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This script will be used by the git fetcher to create shallow mirror tarballs.
usage: git-make-shallow [-h] [--ref REF] [--shrink] REVISION [REVISION ...]
Remove the history of the specified revisions, then optionally filter the
available refs to those specified.
positional arguments:
REVISION a git revision/commit
optional arguments:
-h, --help show this help message and exit
--ref REF, -r REF remove all but the specified refs (cumulative)
--shrink, -s shrink the git repository by repacking and pruning
While git does provide the ability to clone at a specific depth, and fetch all
remote refs at a particular depth, the depth is across all branches/tags, and
doesn't provide the flexibility we need, hence this script.
Refs (branches+tags) can be filtered, as the process of history removal scales
up rapidly with the number of refs. Even the existing `git fetch --depth=` is
extremely slow on an upstream kernel repository with all the branches and tags
kept.
This uses the same underlying mechanism to implement the history removal which
git itself uses (.git/shallow), and the results, when configured similarly, are
in line with the results git itself produces with `fetch --depth`.
(Bitbake rev: 0254020f0e1911c0eaf99111b91828d2a74a4ee1)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We clear append/prepend on newly set variables, we should also clear
remove operations. If we don't do this, there is no way we can actually
delete a remove operation. Bitbake internally uses parsing=True to avoid
these side effects when making its own internal calls.
Also add a testcase to bitbake-selftest to ensure we remain consistent going
forward from here.
(Bitbake rev: 3a319f079d699c870d8531e051ab65e6278d1fa5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some tests to verify that we are extracting "contains" information
from python expressions in the code in the bb.data and bb.codeparser
modules.
(Bitbake rev: 88fda492df875dd79b7aecf1f34b38517fc1eb33)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enable the following operations from a remote datastore to affect the
other end:
* setVarFlag()
* delVar()
* delVarFlag()
* renameVar()
In practice I don't expect these to be used much, but they should be
present so that the implementation is at least reasonably filled out
and that the tests pass.
Also add tests for the interface, mostly by subclassing the existing
local test classes so that they are using a remote datastore. (These
don't actually test remote usage via tinfoil, just that the
datastore's interface can be used.)
(Bitbake rev: 282dc0719d22a39df746eea762ebe05c66aa8f8a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a huge gap in the remote datastore code introduced in the
tinfoil2 rework - we weren't handling overrides at all, since these are
stored separately from the actual data in the DataSmart object. Thus,
when a datastore actually represents a remote datastore we need to go
back to that remote datastore to get the override data as well, so
introduce code to do that.
To avoid a second round-trip I had to modify the _findVar() function to
return the override data as well. This will increase the overhead a
little when that data is superfluous, but without making the function
even uglier I don't think there's a way to avoid that.
(Bitbake rev: 4f9d6f060ed247fb6fa2f45668a892a1788d3f91)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In 2bc4d35fb32defc59cd6ed1fc87e35924c201a5c we fixed cases where an inactive
remove override was influecing the value of a variable. This adds a test
case for this issue.
(Bitbake rev: 1ff9b3c669fa187f152de7b8b57d14c2468d926c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that the datastore works dynamically we don't need the update_data calls
so we can just remove them. They're not actually done anything at all for
a while.
(Bitbake rev: 2300beb50333bb620013b058a7309e7f2042101d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(a follow on patch to fix up a few recent introductions)
(Bitbake rev: f9961fd5beb31d5ab9656a5be59f7ab3effef2f0)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The COW object used within VariableHistory can't be serialised itself,
but we can convert it to a dict when serializing and then back when
deserialising. This finally allows DataSmart objects to be serialized.
NOTE: "serialisation" here means pickling, not over XMLRPC or any other
transport.
(Bitbake rev: bbbb2a53d5decf3b613a92c4ff77c84bfc5d4903)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you're expanding a value that refers to the value of a variable in
python code, we need to ensure that the datastore that gets used to get
the value of that variable is the client-side datastore and not just the
part of it that's on the server side. For example, suppose you are in
client code doing the following:
d.setVar('HELLO', 'there')
result = d.expand('${@d.getVar("HELLO", True)}')
result should be "there" but if the client part wasn't taken into
account, it would be whatever value HELLO had in the server portion of
the datastore (if any).
(Bitbake rev: cbc22a0a9aadc8606b927dbac0f1407ec2736b35)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows you to maintain a local reference to a remote datastore. The
actual implementation of the remote connection is delegated to a
connector object that the caller must define and supply. There is
support for getting variable values and expanding python references
(i.e. ${@...} remotely, however setting variables remotely is not
supported - any variable setting is done locally as if the datastore
were a copy (which it kind of is).
Loosely based on an earlier prototype implementation by Qing He.
(Bitbake rev: a3edc3eefa2d03c4ad5d12187b32fa4dc495082a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVarFlag() now defaults to expanding by default, thus remove the
True option from getVarFlag() calls with a regex search and
replace.
Search made with the following regex:
getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\)
(Bitbake rev: c19baa8c19ea8ab9b9b64fd30298d8764c6fd2cd)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The old style bb.data.getVar/setVar API is obsolete. Most of bitbake
doesn't use it but there were some pieces that escaped conversion. This
patch fixes the remaining users mostly in the fetchers.
(Bitbake rev: ff7892fa808116acc1ac50effa023a4cb031a5fc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you use the subdir parameter in a SRC_URI and pass an absolute path
then it gets appended to the unpack directory instead of being used directly.
This is inconvenient as it may be useful to use ${S} when you want to unpack a
file into the source tree.
Change this behaviour so that absolute paths are used directly instead of being
appended to the root directory. To ensure that recipes cannot write files to an
arbitrary location enforce that the subdir starts with the unpack root.
(Bitbake rev: c3873346c6fa1021a1d63bddd9b898a77c618432)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set this env variable to 'yes' to preserve temporary directories used by
the fetcher tests. Useful for debugging tests.
(Bitbake rev: 04132b261df9def3a0cff14c93c29b26ff906e8b)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
unset VAR
will clear variable VAR
unset VAR[flag]
will clear flag "flag" from var VAR
(Bitbake rev: bedbd46ece8d1285b5cd2ea07dc64b4875b479aa)
Signed-off-by: Jérémy Rosen <jeremy.rosen@openwide.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The CUPS ipptool URL we were checking now redirects to github where the tarball
isn't present, so remove it from the test suite.
(Bitbake rev: 4b50895fb3462b21e3874a2e99c363c8d05e89e6)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.
(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this, the dict can reorder causing sanity test failures.
(Bitbake rev: ca8c91acc9396385834b266d4e8b84d917e5e298)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The preferred form is assertEqual, assertEquals is deprecated and
not present in python v3.
This is v2.7 safe.
(Bitbake rev: b60261bf8ade14aca31238b50c243c01adcabc59)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There were no tests that verified the value of origvalue in the callback
routines used by edit_metadata(). This patch adds one for a simple
multiline variable.
(Bitbake rev: ece3a4d02d8162dee78c2062c10291b5fd625c36)
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
edit_metadata() would corrupt a variable that was multiline, but
had the ending quotes on the same line as the last value. For example:
TEST_VAR = " foo \
bar"
would become " foo ba" because the code would always delete the last
character on the line and then do it again if the line ended in the
quote. This however doesn't show up if you have:
TEST_VAR = " foo \
bar \
"
which is how all the test cases were written.
This patch fixes that bug and adds and fixes a test that matched the bugs
behavior rather than the expected behavior.
(Bitbake rev: 14f05cbdc2ad8d59a94af1c8816567d93c39c88c)
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The tests were skipped when running without network even though they
didn't require network. This commit also adds a test case for URLs with
ports in them (the ports should not be considered when doing trusted
network checks).
(Bitbake rev: 77747de6b20538063eef3b188489a35bef225359)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update the upstream url used for testing cups versions after upstream website
changes.
(Bitbake rev: 5f06041d4936fc22297945bbbad7020bfa9083c6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When in SRC_URI appears file://dir;subdir=foo unpacker copies 'dir' to ${WORKDIR}, not
${WORKDIR}/foo as it should be.
These changes are fixing following bugs as well:
Bug 6128 - Incorrect wildcard unpack behaviour in fetcher
Bug 6129 - Local directories unpack to a different location than local files
(Bitbake rev: e659a3b0c2771679057ee3e13cd42e6c62383ff2)
Signed-off-by: Alexander Shashkevich <alex@stunpix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we specify urls such as npm://somehost;someparams the fetcher currently
does a poor job of handling mirrors of these urls due to deficiencies in the
way decodeurl works. This is because "somehost" is returned as a path, not
a host.
This tweaks the code so that unless its a file url, the host is returned
correctly.
This patch also adds test cases for these urls to the exist set of test
urls.
We need to tweak the URI() class since this thinks this is a relative url
which is clearly isn't. We also need to handle the fact that encodeurl will
error if passed a url of this form (it would want the path to be '/'.
(Bitbake rev: 83203cd2e677706e0111892a7843b83263cb8bd9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This sets the scene for removing the default False for expansion from
getVarFlag. This would later allow True to become the expand default.
On the most part this is an automatic translation with:
sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g' -i `grep -ril getVar *`
There should be no functional change from this patch.
(Bitbake rev: 7c3b99c6a716095af3ffce0b15110e91fb49c913)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The core change here is to fall back to GET requests if HEAD is rejected in the
checkstatus() method, as you can't do a HEAD on Amazon S3 (used by Github
archives). This meant removing the monkey patch that the default method was GET
and adding a fixed redirect handler that doesn't reset to GET.
Also, change the way the opener is constructed from an if/elif cluster to a
conditionally constructed list.
(Bitbake rev: 6ec70d5d2e330b41b932b0a655b838a5f37df01e)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If variables are unset, the code simply doesn't expand them, there
aren't errors. If the code is a python expression, this can get a bit
messy, see the attached test case. The python expansion code sees the }
of the unexpanded value rather than the close of the python expression
and then raises a SyntaxError exception.
Ideally, we'd update the code to match pairs of brackets. I don't know
how to do that with the current regex and this is unfortunately a
performance sensitive piece of code. We also run the risk of breaking
existing code in OE-Core where there are "{" characters but not "}"
to close them (PKGE and PE).
Rather than raising the exception, matching the existing "just return
the expression" behaviour seems more consistent with the standard
variable behaviour.
This addresses an issue found in the recent image.bbclass code where
there are some variables we choose not to expand (TMPDIR/DATETIME).
This patch also adds a test case for this behaviour. It wouldn't preclude
improved bracket matching code in the future either.
(Bitbake rev: d80d39e73223a50fda0090784303d2c57167bb4c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A recent change in bitbake added filename/lineno information to the
parameters of bb.data.build_dependencies(). The codeparser tests
required a little adaption to the changes, adding the flags to the FOO
variable used in the tests.
The error seen when running the tests is a TypeError exception raised
in bb.codeparser:
TypeError: int() argument must be a string or a number, not 'NoneType'
(Bitbake rev: f1fe674397ac5cd355696d5b4cc90b7cfa6c867f)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rename REGEX to UPSTREAM_CHECK_REGEX, REGEX_URI to UPSTREAM_CHECK_URI, and
GITTAGREGEX to UPSTREAM_CHECK_GITTAGREGEX to better reflect their purpose
and to reflect a common namespace.
(Bitbake rev: f0a9e783f9969573fd74edfa241ef14f18ac684e)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make the following improvements to edit_bblayers_conf():
* Support ~ in BBLAYERS entries
* Handle where BBLAYERS items are added over multiple lines with +=
instead of one single long item
Also add some comments documenting the function arguments and return
values as well as a set of bitbake-selftest tests.
(This function is used by the bitbake-layers add, remove and
layerindex-fetch subcommands, as well as devtool when adding the
workspace layer).
(Bitbake rev: e9a0858023c7671e30cc8ebb08496304b7f26b31)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of reproducessing the same line over and over and over, we remove the
current line from the mirror list. This permits us to re-evaluate the list
while excluding all matches that have previousily occured.
Without this fix, adding this test results in a failure:
RuntimeError: maximum recursion depth exceeded in cmp
(Bitbake rev: 24a8e9a5b0ba145ae589178d74365c986ebca325)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
One data store changing a variable poked through into a different data
store. This test case replicates that issue where the value 'B' would
become unset/disappear.
We also enhance parsehelper to generate files with an optional suffix
such as bbclass.
(Bitbake rev: 5c4179f58a4e04f1c354df5f17d1860eb403f0ac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change the test name to be more specific on what is tested.
(Bitbake rev: 3e39156bc330c4c726058a5b9c13d33e2daad89f)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now latest_versionstring method returns (version, revision) for comply
the new return convention needed by SCM's like git get the current
revision.
bb/tests/fetch.py: Updated wget latest_versionstring test for comply new
convention.
[YOCTO #7605]
(Bitbake rev: 8d454646cbe1b04758ca178d8c6fcfd02b818b7b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to know the revision associated with the upstream version in
SCM like git, this change broke return convention, oe-core recipeutils
get_recipe_upstream_version need to be updated.
tests/fetch.py: Updated git latest_versionstring test for comply new
convention.
[YOCTO #7605]
(Bitbake rev: fd175dc90024c503134c11cbd83e77d88c406ac8)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a variety of tests which were found to be useful when working
on the data store recently.
(Bitbake rev: 5c5f8da509f6bbc1fad263462142519ef3d54a35)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.
This patch was mostly made using the command:
sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`
(Bitbake rev: 659ef95c9b8aced3c4ded81c48bcc0fbde4d429f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures basic key expansion works and that overlapping
keys generate a log message.
(Bitbake rev: ed5a8954ac923eda9750a636c5bb5b95ffce664f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows us to write tests which ensure a particular action generates
a particular log message.
(Bitbake rev: b30ee0aba51a35a194a4338b988f93ece1ed281c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix several bugs and add some useful enhancements to make this into a
more generic metadata editing function:
* Support modifying function values (name must be specified ending with
"()")
* Support dropping values by returning None as the new value
* Split out edit_metadata() function to provide same functionality
on a list/iterable
* Pass operation to callback and allow function to return them
* Pass current output lines to callback so they can be modified
* Fix handling of single-quoted values
* Handle :=, =+, .=, and =. operators
* Support arbitrary indent string
* Support indenting by length of assignment (by specifying -1)
* Fix typo in variablename - intentspc -> indentspc
* Expand function docstring to cover arguments / usage
* Add a parameter to enable matching names with overrides applied
* Add some bitbake-selftest tests
Note that this does change the expected signature of the callback
function. The only known caller is in lib/bb/utils.py itself; I doubt
anyone else has made extensive use of this function yet.
(Bitbake rev: 20059e4d5ab9bf0f32c781ccb208da3c95818018)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some tests of mirrors or mirrors to the fetcher unittests.
(Bitbake rev: e33d82bc10283d533f928836d56a6f0af80ea5c1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher calls os.chdir() in a number of places, which can affect
other tests (since the directory it changes into gets deleted) - let's
just put the current directory back to where it was when we're done.
(This fixes bb.tests.Path.test_unsafe_delete_path failing if it was run
as part of a full bitbake-selftest run, where the fetcher tests get to
run before it.)
(Bitbake rev: b1653855c74f86909c9f329ed6d2b10391c28395)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_ALLOWED_NETWORKS is a list of hosts that the fetcher will be allowed
to use when BB_NO_NETWORK is not set.
If BB_NO_NETWORK is set, then networking is still disabled.
If BB_ALLOWED_NETWORKS is not set, the behaviour remains the same as
today.
If BB_NO_NETWORK is NOT set, and BB_ALLOWED_NETWORKS is configured, then
only the hosts in the list are usable by the fetcher.
eg:
BB_ALLOWED_NETWORKS="yoctoproject.org git.gnu.org"
The fetcher will be able to download from yoctoproject.org, git.gnu.org,
but not ftp.gnu.org or any other hostname that is not in the list.
There is also limited support for wildcards on the beginning of the
hosts, so BB_ALLOWED_NETWORKS="*.gnu.org" with match git.gnu.org and
ftp.gnu.org as well as foo.git.gnu.org
(Bitbake rev: c7263096ba31ba45daeeb9de90c1cb9ebef24a28)
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some very basic safeguard against recursively deleting paths such
as / and /home in the event of bugs or user mistakes.
Addresses [YOCTO #7620].
(Bitbake rev: 56cddeb9e1e4d249f84ccd6ef65db245636e38ea)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add set of PN in data because now latest_versionstring use it for
validate version directory searching.
(Bitbake rev: 2e4a03db967ac1459b2764108fc54c4566a7e371)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've had versioned dependency support in LAYERDEPENDS for quite a long
time, but I can say with pretty good certainty that almost nobody has
used it up to now because it was too strict - the specified version had
to exactly match the version in your configuration or you would get an
error; there was no "greater than or equal" option, which is usually
what you will want given that LAYERVERSION does get bumped from time to
time.
However, users mismatching layer branches and then having their builds
fail later on with some incomprehensible error is still a pretty common
problem. We can't simply use the git branch because not everyone is
always on a branch and the branch names don't always match up (and
that's not an issue). To provide a practical means to address branch
mismatching, I have reworked LAYERDEPENDS version specifications to use
the more familiar "dependency (>= version)" syntax as used with package
dependencies, support non-integer versions, and clarified the error
message a little. If we then take care to bump the version on every
breaking change, it is at least possible to have layers depend on these
changes when they update to match; we can now even support a major.minor
scheme to allow retrospectively adding a version limiter to old branches
when a new branch is created and yet still allow the old branch minor
version to be bumped if needed.
Fixes [YOCTO #5991].
(Bitbake rev: 408be9cdf2b1e32e64ea488d8051a546fb54c144)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This scheme is used for versioning recipes that are pre-release (alpha,
beta, etc.) within OpenEmbedded, so add some tests to ensure the
appropriate comparison results still hold true.
(Bitbake rev: 3a9eefe27f29a4593d6298f0427ac5f3e9183377)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The _remove operator isn't working correctly when used with a variable that
expands to several items, so add a test case to exercise this path.
(Bitbake rev: cb2a62a5fbffb358528a85b46c1fc6383286cb9d)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you copy the datastore, then delete a key, it should not exist in
d.keys(). This adds a test to cover the recently found data store bug.
(Bitbake rev: 16d5f40ad20fd08bf7a4d0e36200c739b5a9f59e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The versionstring tests hit the network so should only run when
network tests are enabled.
Also remove the print statement which confuses the test output and
add it to the test failure message instead.
(Bitbake rev: 6c046660cfc4fb3792a42aeafff91a13f68a2e89)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update test case for cups is needed because match only 2.0.0
versions see VERSION=2\.0\.0 in the previous string.
(Bitbake rev: 148aba30155f4de17f6c6fb9b4c37f08a2db202b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being able to query whether updated versions of a url are available
is useful, not least for the package reporting system. Since such code
is closely linked to the url type and the url itself, the fetcher
makes a locical place to contain this code.
For wget based urls this means taking upstream directory listings
and searching those for later versions, returning those that are
found.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
(Bitbake rev: a8272e22b7819e0e8afd8e291d276f5f28fc0007)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being able to generate a version string representing the most recent git commit
given git is useful, not least for the package reporting system.
This adds in a latest_versionstring method to the git fetcher
which allows users to query the latest version using ls-remote
and filtering the responses.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
(Bitbake rev: f71c8c0354e87fed80bc845db6728e6e18ce9c4d)
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This fixes urls of the form file://some/path/file;subdir=b. It also
adds in a couple of tests so we now tests these corner cases.
(Bitbake rev: 46306912a96444790efa9418d934dfdd36773ba1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have:
FOO = "${bindir}/X Y"
FOO_remove = "${bindir}/X"
the expected result is "Y". Currently this doesn't work since the removed
expressions are not expanded first. This patch adjusts things so the
expressions are expanded before being processed for removal.
Also add a test to ensure this case continues to work.
[YOCTO #6624]
(Bitbake rev: 72a1ca4a104ccab73d6abcbd44db9c2636a58572)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
contains_any is buggy so write a test case to demonstrate this.
(Bitbake rev: 25d10c349aab77bf99745c0a90dd8f9b90abccac)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you try:
Y = ""
Y_remove = "X"
in OE-Core, bitbake will crash with a KeyError during expansion. The reason
is that no expansion of the empty value is attempted but removal from is it
and hence no varparse data is present for it in the expand_cache.
If the value is empty, there is nothing to remove so the best fix is simply
not to check for None but check it has any value.
Also add a test for this error so it doesn't get reintroduced.
(Bitbake rev: af3ce0fc0280e6642fa35de400f75fdbabf329b1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add some explicit tests for unpacking local files to the appropriate
location. Some of these tests are actually testing for broken behaviour;
these have been called out in the comments, and associated bugs have
been filed.
(Bitbake rev: ca921c773c52392a5a338b2f493ad38c8132f708)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We currently don't have test cases for _append, _prepend and _remove. This
patch adds some basic tests and includes a test case for a recently reported
issue with the _remove operator.
(Bitbake rev: 93291bd90e18808c7a1c8c692949396bbc7e4348)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When setting the username after already having set the password, the password
was unexpectedly reset. This change fixes this issue and introduces unit tests
to make sure it doesn't happen again.
(Bitbake rev: 25faef3a047f9c7564089463d7c96f6910b640cb)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a case in meta-intel where a SRC_URI contains both a query string and
URI parameter:
https://edc.intel.com/Download.aspx?id=6190;downloadfilename=LIN_IEMGD_1_14_GOLD_2443.tgz
Python's urlparse thought the URI parameters were part of the query parameter
value, but in the bitbake context this is obviously not the case. As bitbake's
usage isn't really RFC compliant, we have to extract and remove the URI parameters
*before* urlparse sees the URI.
(Bitbake rev: c2f27ae4271985b48f957747b6ea372c8699cb49)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change introduces the .query property of the URI class. It is a
read/write dict of the parameters supplied in the query string of the
URI. E.g.:
http://example.com/?foo=bar => .query = {'foo': 'bar'}
(Bitbake rev: 1cb2b3c458c8c5521591d2c8f2e0058143fc77bb)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using the "execs" information in new code, it became clear that
the returned data was incorrect and there were missing exec'd functions.
This corrects the error and changes one of the test results to match
the correct behaviour.
(Bitbake rev: 8a24f2d3b735bbc59ca4a09670cabbadb1868c1a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use a newly created "git-submodule-test" repo on git.yoctoproject.org
which currently contains one submodule (the bitbake repository).
(Bitbake rev: a750c57242928c546a5aace632543e956ee908eb)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was found when trying to run the data tests individually.
(Bitbake rev: e4bf4ad4b99978483541a719105c98ea124e8a34)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a sanity check so that if some SRCREV is set and a rev parameter is given
to the url, the revision given should match.
Any tag parameter behaves the same as rev. If both are specified, error to
tell the user we're confused rather than do something which may or may not
be what they intended.
Also add some unittests for this.
(Bitbake rev: e82a4ab48991035866da9914c8b75a9bfbc9a7fc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* currently decode_url regexp parses branch=@foo as username so it ends like this:
- ('git', '', 'foo', 'git.openembedded.org/bitbake;branch=', '', {})
+ ('git', 'git.openembedded.org', '/bitbake', '', '', {'branch': '@foo'})
* http://hg.python.org/cpython/file/2.7/Lib/urlparse.py also assumes
that there is at least one '/' as separator between netloc and path,
params, so it looks reasonable to prevent including '/' in username
(Bitbake rev: 2c82742114091cb55055328b54223686816582f2)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The parser never has supported it, the datastore API did happen to work
but whitespace in key names is a really bad idea and not something I think we
should encourage or support.
Fix test case failures after excplitly ignoring it for variable expansion
purposes.
(Bitbake rev: a2074ddaba6f53962d6caf34dbd27bdbc259935b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The skipIf decorator is unavailable under python 2.6 and finding a syntax
works there is hard. This patch does allow network tests under 2.6.
(Bitbake rev: 565b3e31e0226c6e193ee0b031bd5e7bef25591e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the fetch tests fail on python 2.6 since check_output
doesn't exist on this version of python. Use bb.process instead
to avoid this problem.
(Bitbake rev: a3637155e829e06bbfe50fb8753de3de7ee8c22e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With this change, you can opt-out to skip unit tests that require an
internet connection. To opt-out, you'll set the environment variable
BB_SKIP_NETTESTS to 'yes'.
(Bitbake rev: 9ff5f172096a4f51b6b085307506473405dc4f59)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a circular dependency issue where bb.data can't directly depend on
bb.parse. The tests were failing due to this issue which is misleading. This
patch hacks around it for now. I'd rather that than not running tests at all.
(Bitbake rev: a206ae0d7769a41ff3666d0f53ff9cf422dfa518)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
