If the user puts universe on the commandline, they don't really want warnings
so use the new verbnote level instead.
(Bitbake rev: 0c87ade5678e503899e3a6cdda5329f6fc212b63)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It has become apparant we need a log level which reaches the console but
isn't a warning/error. Add "verbnote" as a way of doing this, behaves as
a note but with a higher priority.
(Bitbake rev: 2076f12cc2f809345108b1606bd6201f41287505)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There were various problems in the server startup loggin:
a) stdout/stderr were not being flushed before forking which could potentially
duplicate output
b) there were separate buffers for stdout/stderr leading to confusing logs
where the entries could be reordered. This was particularly confusing
due to the separator the logs use to idendify new messages
c) an fd wasn't being closed during server startup meaning if the
server failed to start, the closed fd wasn't detected as it was held
open by the other reference
d) If the pipe was detected as being closed, the code incorrectly retried
server startup
e) The event code would remap stdout/stderr without flushing them, leading
to lose log messages
(Bitbake rev: 0594faa0b52ce5dbd948d836d88617d38d9862d1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the case of a sublayer of an existing layer, where the sublayer and
main layer share a path, the system may not match the paths properly resulting
in:
No bb files matched BBFILE_PATTERN_sublayer '^/path/main/sublayer'
because it has already matched the main layer.
Fix this issue by sorting the collection items based on the pattern, using
longest to shortest. Obviously regex wildcards could still be an issue
but these are typically not used, so this simply fix should work in the
existing cases.
(Bitbake rev: 1787cef7221b88f6920ea70fadaffc117d84c7aa)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before this fix it is assumed that the removal of the
remote can only fail because there is not remote to remove. This
is a false assumption. Example error which would be ignored:
git -c core.fsyncobjectfiles=0 remote rm origin failed with exit code 1, output:
Note: A branch outside the refs/remotes/ hierarchy was not removed;
to delete it, use:
git branch -d master
error: could not lock config file config
error: Could not remove config section 'remote.origin'
Due to the masking of this error a stranger error will be
presented to the user, because this time we do not mask the
exception:
git -c core.fsyncobjectfiles=0 remote add --mirror=fetch origin https://github.com/ptsneves/tl-wn722.git failed with exit code 128, output:
fatal: remote origin already exists.
The most likely reason that the remote cannot be removed nor
modified is that the DL_DIR/git2 does not have permissions
compatible with the user running bitbake.
This commit fixes:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12728
(Bitbake rev: 9c86c582a10c9b23abad7d34b6cbf12f7086294d)
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Enable the ability to select an existing build directory into Toaster.
This opens to the user the backend features of 12823, for command line
user compatibility.
Enable the ability to select saving Toaster settings in the regular
"bblayers.conf" and "local.conf" instead of the default
"toaster_bblayers.conf" and "toaster.conf". This opens to the user the
backend features of 12821, for command line user compatibility.
[YOCTO #12902]
(Bitbake rev: 8ce51fbd92ab42365a38e0c1c260bb4979377a89)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The local layer info (provided through custom fixtures) should
not be hidden. It is better to handle it in the same manner
as an imported layer, otherwise the layer path and dependency
info is not shown. The layer editing fields are handled in the
html side of things appropriately so this does not harm that
implementation.
[YOCTO #12891]
(Bitbake rev: aa79967920b2617ce007f66c89f3343f1a7f34b4)
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In a case where the layer source is local only and the recipe
is not yet built, we can search for the path with layer's
local_source_dir, and if available that should be used rather
than just skipping the scenario.
[YOCTO #12891]
(Bitbake rev: 59f3e04122fca43835603779ac2d201464fbaebf)
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change allows the CUSTOM_XML_ONLY toaster setting to be
provided through the environment so the user can do this without
mingling with the settings.xml, for scenarios where modifying
settings.xml is not achievable.
[YOCTO #12891]
(Bitbake rev: 55333f1c3ded1c53120e6cb32b440cc707521e2b)
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The release drop down divs are not being closed
appropriately, which showed adverse reactions on
the UI that aligned the "Create project" button
with the left edge of the screen without any
margins. This fixes these divs which in turn
aligns the button appropriately.
[YOCTO #12891]
(Bitbake rev: a6e20bddddd083fbecb5ed8d5f4824da640fdd44)
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There can be cases where the variables being used
to divide in build percentage expressions can be
zero. For example, a setup consisting of only local
repos will have repos_to_clone=0 and will generate
a divide by zero scenario.
Fix this by checking the divisor in such cases.
[YOCTO #12891]
(Bitbake rev: 679c70aa32d23e9247f8a68efcb579ad733af84b)
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current mechanism for finding the bitbake binary
assumes a directory structure which is identical to
poky, where oe-core's meta and bitbake directories are
at the same level. There can be a case where bitbake
is used from elsewhere and in such cases the above
mentioned assumption fails to hold, whereas this is
totally allowed by the oe-init-build-env script which
can take bitbakedir as an argument.
The better approach is to allow bitbake to be derived
from PATH, while keeping the older mechanism in place so
it can be removed after tests are done in various
environments. This makes more sense as toaster has
also been launched from the same bitbake instance
that is the one in PATH.
[YOCTO #12891]
(Bitbake rev: 365d8d94ae3e4e0f95e0806dbcb7c77c20a55d2d)
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster depends on pokydirname for identifying the location of
the oe-init-build-env script (and there might be other purposes
in the future). The problem with current approach is that it
only checks/sets the variable with git based repos, whereas
toaster provides mechanisms to allow having layers that are all
locally available. The evaluation of the variable fails in such
scenarios, so use a more flexible mechanism in this case and
try to locate poky in the local layers as well, if not already
set.
[YOCTO #12891]
(Bitbake rev: 6c3c196b28603591371ec7e62871fbb4296f2c71)
Signed-off-by: Awais Belal <awais.belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In finalize() we save event handlers, register the ones relevant to the
recipe being finalised, trigger events, and then restore the handlers so
that one recipe's custom handlers (actually implemented within a class
inherited by the recipe) do not affect other recipes. However, if an
exception occurs during parsing, the saved handlers were not being
restored. Use a try...finally block to ensure that the handlers are
always restored.
This issue became apparent since in OpenEmbedded-Core we have recently
introduced a find_intercepts() handler for the
bb.event.RecipePreFinalise event in image-postinst-intercepts.bbclass
that images and old-style SDK recipes will end up inheriting. So far it
doesn't seem that the the error has manifested itself in normal builds,
but when parsing OE-Core recipes in the OE layer index it has:
core-image-rt-* image recipes were parsed which in the default
configuration raise SkipRecipe. The next non-image recipe that is parsed
will trigger a real exception, because the find_intercepts() handler is
still registered and gets fired, but in the context of the new recipe
the POSTINST_INTERCEPTS_PATHS variable is not set, and the code in
find_intercepts() is written with the reasonable assumption that that
isn't possible given that the class itself sets a default, and thus it
fails.
(Bitbake rev: e5f1f8fa201774e0c3c554d59b277baa2128708f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In spite of a comment suggesting otherwise, os._exit() does not flush
buffered output from file descriptors before exiting the process like
os.exit() does. This means that any un-flushed output is lost in the
daemon process, in particular the traceback from any thrown exceptions,
making debugging exceptions in the daemon quite difficult.
The solution is to flush stdout and stderr before exiting.
(Bitbake rev: fcc8e1ff53696f78dd64b4ee32f3c433b7a47df0)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Use == instead of = when comparing task outcome to OUTCOME_FAILED.
Prior to this fix the recipe template would cause a TemplateSyntaxError
exception.
(Bitbake rev: a53ffec4ed3d0f9221bca398e20e8f480fb2b325)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Here is the primary driving enhancement:
* Bug 12785 - Support Project Specific configuration for external
tools (e.g. ISS, Eclipse)
- Isolated project-specific configuration page (full Toaster context
hidden)
- Support for new project, reconfigure existing project, and import
existing command line project
- Ability to define variables (e.g. image recipe) and pass them back
to external GUI
- Ability to execute the cloning phase, so that external GUI receive
a buildable project
- Ability to call back to the external GUI when updates are completed
and ready
- Compatibility of above projects with the normal full Toaster interface
- Ability to pass to a 'complete' or 'cancel' web page so that the
external GUI can immediately stop that Toaster instance, and not
leave dangling servers nor edit sessions open
Here are the supporting enhancements, where at least the
back end is implemented:
* Bug 12821 - Make Toaster conf changes compatible with command line usage
* Bug 12822 - Support importing user changes to conf files into Toaster
* Bug 12823 - Support importing user build directories into Toaster
* Bug 12824 - Scan imported layers for content so that they are
immediately available
* Bug 12825 - show layer clone item in progress bar
Here are defects fixed:
* Bug 12817 - builddelete.py requires explicit 'add_arguments'
* Bug 12818 - Remove orphaned imported layers when project is deleted
* Bug 12826 - fix imported layer management
* Bug 12819 - build using selected bitbake env, not Toaster's env
* Bug 12820 - Toaster randomizes the layer order in toaster_bblayers.conf
[YOCTO #12785]
(Bitbake rev: 985d6cec290bdd80998a63483561a73c75d82d65)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bb.utils.copyfile is called in a few places with identical src and dst
in order to create an st_nlinks==1 version of the file. That that even
works relies on an implementation detail of copyfile (namely, that it
creates a temporary file and then does a rename). Moreover, it's a waste
of time if the file already has st_nlinks==1.
So create a helper that optimizes away the copy in the st_nlinks==1
case. Of course, this helper relies on the same implementation detail,
but that's now contained within bb.utils itself.
To test that we do at least sometimes hit the no-copy path, I tested
locally with
if sstat[stat.ST_NLINK] == 1:
+ bb.note("Woohoo, 2*%d bytes I/O avoided" % sstat[stat.ST_SIZE])
return True
(and the obvious places in oe-core patched), and the do_package log files
are indeed filled with woohoo notes.
(Bitbake rev: 7ae93cf40ab91965147055100432961436bce46c)
Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Opening a file in binary mode and iterating it seems like the simple solution
but will still break on newlines, which for binary files isn't really useful as
the size of the chunks could be huge or tiny.
Instead, let's be a bit more clever: we'll be MD5ing lots of files, but we don't
want to fill up memory: use mmap() to open the file and read the file in 8k
blocks.
(Bitbake rev: a0ac8d67f1471a0c611d691b856fede67efb53f6)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In case something goes tragically wrong, catch a request to checksum / and
refuse.
(Bitbake rev: e7cd4c86ef8a2c2bbf068e84c83fdc9e052b6e3d)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: d49483eb733ee27325349246fa37a84140c0941d)
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is an oversight in the current hash validation API in that the
function can't know how many setscene tasks already completed. Rather
than trying to add additional parameters to the function, causing
incompatibilities, store the value in the datastore.
This is useful to allow build status reporting to the user for
figures on sstate reusage and build completion.
(Bitbake rev: ec037d3e49264037b81212f498d98e292ae7c334)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change lsupdates.py to use the new layerindexlib module to load the data from
the public layer index. It still does all of the manual parsing. This
is intended to be a stop gap until the toaster can use the module itself to
manage the data.
Everything else is functionally equivalent to the prior version.
(Bitbake rev: 8e482342c652e298b5f5ea58eda72c5eb14ce2bd)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These don't need to access recipe information, so let's not waste the
user's time parsing all recipes.
(Bitbake rev: 6a242a399ae93393c3fc60eda541c1f8f77fed57)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Display changes:
The output will now include references to the layers that the user already
has on their system. It does this by querying the cooker derived index.
The code that enables this behavior is labeled as 'TODO' currently. As
part of the work we need to make a final determination if this is the
desired output.
Also changed the default branch to no longer define itself as 'master'.
When the user does NOT set a branch, the default is now the
'LAYERSERIES_CORENAMES', and if that doesn't exist 'master'. This is
subtly different in behavior, but more consistent with user expectations.
(Bitbake rev: 478c4b5489943f756ae03d6b6d3b5d665cedbce0)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The layer index module is expected to be used by various parts of the system
in order to access a layerindex-web (such as layers.openembedded.org) and
perform basic processing on the information, such as dependency scanning.
Along with the layerindex implementation are associated tests. The tests
properly honor BB_SKIP_NETTESTS='yes' to prevent test failures.
Tests Implemented:
- Branch, LayerItem, LayerBranch, LayerDependency, Recipe, Machine and
Distro objects
- LayerIndex setup using the layers.openembedded.org restapi
- LayerIndex storing and retrieving from a file
- LayerIndex verify dependency resolution ordering
- LayerIndex setup using simulated cooker data
(Bitbake rev: fd0ee6c10dbb5592731e56f4c592fe687682a3e6)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When a layer is added it needs to be in a list, otherwise the system will
error such as:
Specified layer directory / doesn't contain a conf/layer.conf file
Additionally, instead of calling the add layer function over and over, it
is better to add all of the new content in one command. Otherwise the
order is important as the system now checks if the layer can be added. For
instance, trying to add meta-python:
Layer Required by Git repository Subdirectory
===================================================================================================================
meta-python - git://git.openembedded.org/meta-openembedded meta-python
meta-oe meta-python git://git.openembedded.org/meta-openembedded meta-oe
openembedded-core meta-python git://git.openembedded.org/openembedded-core meta
Adding layer "meta-python" (.../oe-core/meta-openembedded/meta-python) to conf/bblayers.conf
ERROR: Layer 'meta-python' depends on layer 'openembedded-layer', but this layer is not enabled in your configuration
The system would try to add meta-python before the dependent meta-oe. Adding
them both at the same time resolves this issue.
(Bitbake rev: 8aeaabf13db645f33495e00b82117327e153d70a)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When adding multiple layers in a chain of commands, reparsing all recipes
each time can be painfully slow. Instead just parse the base configuration
which gives some confidence things worked out correctly without as much
overhead.
(Bitbake rev: cfd1302031f3cca96300a0e445a47b1614ecd00c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the capability for tasks from different
multiconfigs to depend on one another.
These dependencies can be enabled using the following format:
task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-on"
For the sake of simplicity consider the following example:
Assuming we have set up multiconfig builds, one for qemux86 and one for
qemuarm, named x86 and arm respectively.
Adding the following line to an image recipe (core-image-sato):
do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs"
Would state that core-image-sato:do_image from x86 will depend on
core-image-minimal:do_rootfs from arm so it can be executed.
This patch makes modifications to:
- cooker: To glue both multiconfigs in one place and make sure
the dependencies can be provided.
- taskdata: To parse and add a new kind of dependency (mcdepends) to
the taskdata object.
- runqueue: To differentiate tasks from different multiconfigs,
add the specified dependencies to the corresponding tasks, and
create a working runqueue that contains tasks from both multiconfigs.
- siggen: To avoid looking for tasks from different multiconfigs on
objects where they dont belong.
The taskdata objects are still not aware of the concept of multiconfig,
so each object doesnt know which multiconfig its building, hence why
the mcdepends are added to all taskdata objects equally (we really
dont expect many of these), but the actual dependencies are added only
to the required tasks by the runqueue.
(Bitbake rev: da8cb8633504bdc815bdcefc538340b9bce5065d)
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a callback that lets you modify or remove items in addition to the
current scheme where you can only add or remove. This enables you to for
example replace a layer with a temporary copy (which is what we will use
this for first in OE's oe-selftest).
(Bitbake rev: bfedb4e85a84e817dbe5d8694b8f8fcdd6f2f22a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the bitbake-cookerdaemon.log is placed into cwd. This seems like a
bad idea, we can place it in the build directory alongside the lockfile that
represents the server instead.
(Bitbake rev: 1620dbc48ffb2a882371cf9174a7b12648befc8a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The file_name parameter passed to bb.parse.siggen.invalidate_task
should be a virtual file name instead of a real file name, or else you
will encounter a following error, for instance, when you run:
$ bitbake nativesdk-lzip -c unpack -f
the error arise:
| ERROR: An uncaught exception occurred in runqueue
| if file_name:
| > taintfn = d.stamp[file_name] + '.' + task + '.taint'
| else:
| KeyError: 'virtual:nativesdk:/opt/poky/meta/recipes-extended/lzip/lzip_1.19.bb'
when multilib builds are used on OE.
(Bitbake rev: da37bdad46e11e7ce93ba7a59d58757b769dc16b)
Signed-off-by: Ming Liu <liu.ming50@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If uri_find contain parameters then original URI parameters should
be checked against parameters from uri_find instead of parameters
from uri_replace.
(Bitbake rev: 8efa7826a61501589afa33eb698c0ab3a622bf2e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
On high core machines, in do_fetch, it is possible to DDoS your own machine.
A method to limit any arbitrary task type to a certain number of simultaneous
threads is needed. (Similar to how BB_NUMBER_THREADS works in the general
case.) The format of this new limitation is:
do_fetch[number_threads] = "2"
This should be set globally. If it is set in individual recipes it could
result in unpredictable behavior.
Note: a value for number_threads > BB_NUMBER_THREADS will have no effect.
(Bitbake rev: 055865047c63b9c3b213b47a1884924ce0adeda0)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The RunQueueStats:taskCompleted and RunQueueStats:taskSkipped can take
multiple arguments. However, nowehere in bitbake are multiple arguments used.
Change this to match the behavior of the other APIs where it needs to be
called once for each task.
Additionally, these two functions were usually called in tandem, however in
the wrong order. It really doesn't matter as there is no specific preemption
point between the calls. But the taskSkipped should be called first to
increment the 'active' count, and then taskCompleted called to decrement it.
(Bitbake rev: 26d5ea9bb892bd6a2e1fd29a9023e0b0644edc16)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently there is quite some variation between the fetchers in terms
of how they determine the subdirectory within DL_DIR and the base
fetch command to run. Some rely on variables being set externally
(e.g. from bitbake.conf in oe-core), some respect these external
variables but provide fallback defaults and some use only hardcoded
internal values. Try to unify the approach used across the various
fetchers.
(Bitbake rev: efd5e35af4b08501c67e8b30f30d9457f6fdf610)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Although the submodules' histories have been fetched during the
do_fetch() phase, the mechanics used to clone the workdir copy
of the repo haven't been transferring the actual .git/modules
directory from the repo fetched into downloads/ during the
fetch task.
Fix that, and for good measure also explicitly tell Git to avoid
hitting the network during do_unpack() of the submodules.
[YOCTO #12739]
(Bitbake rev: 11b6a5d5c1b1bb0ce0c5bb3983610d13a3e8f84a)
Signed-off-by: Matt Hoosier <matt.hoosier@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When running bitbake command with Python 3.6.5 always result in
import error causing by the change of distutils module.
This patch replaces the method to search executable in PATH by
"/usr/bin/env <command>".
(Bitbake rev: bd9a1b063633af2936ba1dd87b19202424900151)
Signed-off-by: Tzu Hsiang Lin <t9360341@ntut.org.tw>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For mirrors or premirrors defined like: "http://.*/.* http://somewhere.org"
fetching ends with errors because function fetch2/__init__.py:encodeurl()
creates url like "http://somewhere.orgsomefile.tar.gz".
It happens because function fetch2/__init__.py:decodeurl()
for url "http://somewhere.org" returns
['http', 'somewhere.org', '', '', '', {}]
and then in function fetch2/__init__.py:uri_replace()
variable result_decode will be
['http', 'somewhere.org', 'somefile.tar.gz', '', '', {}]
(because of line: result_decoded[loc] = os.path.join(result_decoded[loc], basename))
for which encodeurl returns "http://somewhere.orgsomefile.tar.gz".
In addition for mirror "http://.*/.* http://somewhere.org/"
everything works fine.
(Bitbake rev: d822ae24ef5485e550804cbd9130ebd73b2aa48e)
Signed-off-by: Jakub Dębski <jdebski@enigma.com.pl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Updating a local git repo clone currently results in multiple calls
to self._contains_ref(), some of which appear to be redundant and can
be eliminated by minor tweaks to the logic in download().
Also drop redundant calls to os.path.exists(ud.clonedir) before
self.need_update(), since need_update() includes its own built-in
check for the existance of ud.clonedir.
(Bitbake rev: 61b0df5523afc8f805043f3adc9c106690e6f133)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add Sumo (YP-2.5) to the release selection for new projects.
[YOCTO #12713]
(Bitbake rev: 76b17ffcea5c7275c2f9735a058256ba909b1a75)
Signed-off-by: David Reyna <david.reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prevent movefile from falsely setting the source file's owner and
permissions on the destination directory instead of the destination
file when moving between devices.
This bug caused the last file moved into a directory to dictate the
directory's owner and permissions.
(Bitbake rev: 82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40)
Signed-off-by: Mattias Hansson <mattias.hansson@axis.com>
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We'd like layers to set this variable so that we know which layers are compatible
with which others, even if the branch is a generic un-updated "master" branch.
Start printing a warning to highlight this issue.
(Bitbake rev: cca81e33b58c390dcf5cc3a31555a43b79177166)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was removed in the bitbake repository back in 2010 (bitbake
revision 24857e2ceb405916b0b0b3e75c6c2375a909b9ba) - it's not clear
to me what happened, but for some reason the file remained in the poky
repository. However it does not appear that it has been used since then
and the reasons for removing even more valid now than they were then -
the code has moved on even further, we have memory resident bitbake
server, and if we were to re-implement this we would do it in a
different way. Drop the file and bring us back in sync with the bitbake
repo.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Explicitly capture and ignore errors when trying to load the optional
'custom.xml' fixture file.
[YOCTO #12554]
(Bitbake rev: 132458939d3987ebc58685397714af3d6d5cd8fd)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before, explode_dep_versions2 would sort the OrderedDict before
returning. This function will still sort the OrderedDict by default, but
will now have the option to return the OrderedDict unsorted. This option will
allow us to check if the order of the package list has changed.
(Bitbake rev: 39d6a30a28f66c599e18beddbd847f40dcff623c)
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The base configuration needs key expansion and anon python execution,
the parsed configurations do not. Fix this consistently, its been
broken and causing double key expansion for a while, only relised
when we started double anonymous python exeution too.
(Bitbake rev: 6138897de5ac6becf3bff56ce7a78f3ec208fcdf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The output of bitbake -e can differ from what actually is used due
to anonymous python making changes to the data store. Execute any
anonymous python added in the base configuration to make things
more consistent.
(Bitbake rev: bcdc2f73e3b4a10b1e479c2891f251d9507a9e30)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows us to call this code from other contexts without
duplicating it.
(Bitbake rev: c6be487f9bd5d95915f2495d555b9f539adb1d44)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a standalone tasks adds a dependency on X:do_build, the code in runqueue would
currently remove it if that do_build was part of an image recipe which uses
recrdeptask on do_build.
Such individual tasks shouldn't do this, therefore tweak the recursive reference code
to only process recurseive tasks, not all tasks.
(Bitbake rev: 4cfca360891e1ed876a9c19487b4f6210686af26)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The '--all'' fetches all remotes, including the ones in $HOME/.gitconfig, which
causes the fetching very slow, so don't use "--all", git fetch should be
enough.
(Bitbake rev: 038a5f4b767df7944e73b2e859ac3af2d3a08bf1)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to allow users to manually populate the download directory with
valid content change the assumption that missing the donestamp file
means unfetched content.
This allows users to populate the download dir without needing to create
dummy .done files such that a user does not need a PREMIRROR when using
BB_NO_NETWORK to provide valid content files in the download directory.
To ensure the correct result this change also fails first if the
localpath does not exist. This prevents further parts of the function
attempting to calculating the checksum on non-existent files. This also
fixes some edge conditions around where if the donestamp exists but the
localpath does not it returns, and did not remove the donestamp.
Also added test cases to cover this use case and additional use cases
where for example the fetcher does not support checksums.
(Bitbake rev: a335dbbb65d5b56e71d98cf3e4fa9bfbec1dcde6)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At least the cli-color node module has dependencies that have
cyclic dependency among themselves. npm.py is prepared to deal
with such a case but the condition is handled only for downloading
or not a dependency again, but then it goes checking the its
dependency which causes an infinite loop in _getdependencies().
Make this function simply return when a dependency is already
downloaded and only download and check its dependencies when not.
(Bitbake rev: 545540420112992e53f4a83104af10452df168d0)
Signed-off-by: Zoltán Böszörményi <zboszor@pr.hu>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The resolve_file() calls mark_dependency(), so the one which calls
resolve_file() doesn't need call mark_dependency() again.
(Bitbake rev: 4682571107323a39b42cd9ec8ee67419e7f15acc)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Print message when cachefile is found/not can help debug.
* Update "Using cache in" to "Cache dir:" since it was the same as the debug
message of "codeparser & file checksum caches", which caused confusion. And
whether the cache file will be used or not is still unknown at that time, so
just print the cache dir.
(Bitbake rev: c8d3a2016f432e8ed9e99d9c28850149ab6fd6d8)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It should be live/exited with server rather than cooker, fixed:
$ bitbake --server-only -T -1
Set MACHINE = "qemux86" in conf/local.conf
$ bitbake quilt
Set MACHINE = "qemuppc" in conf/local.conf
$ bitbake quilt
[snip]
ERROR: When reparsing /workspace1/lyang1/poky/meta/recipes-connectivity/openssl/openssl_1.0.2m.bb.do_package, the basehash value changed from c216f7f4fdd3cf4a0b10b975a636426c to d5a8e9431ab261381752d7a64c7b2fa9. The metadata is not deterministic and this needs to be fixed.
[snip]
This is because the server doesn't know local.conf is changed since the
notifiers are stopped, so it doesn't reparse, and then we would get the errors,
let the notifiers live/exited with server can fix the problem.
(Bitbake rev: a6a641cb9c5f3abe901b150da915372e295383d7)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The old code couldn't handle nestled layers correctly, e.g.:
parent_layer/sub_layer/foo.bb
Note there are two layers, parent_layer and sub_layer.
And in parent_layer/conf/layer.conf:
BBFILE_PATTERN_parent_layer = ""^${LAYERDIR}/"
This setting is incorrect since it also matches parent_layer/sub_layer/foo.bb,
so it warns that no files matched sub_layer, this is the expected behavior, but
it doesn't warn when there is a parent_layer/sub_layer/bar.bbappend, this was
incorrect since the bbappend is also matched by BBFILE_PATTERN_parent_layer, it
should warn and let the user fix the problem. Check the bbappend in already
"matched set" before return it as matched by "unmatched set" can fix the problem.
(Bitbake rev: ec90245d28e52ea718d2ce084eb304cdc4355c9c)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is nothing to do when BBFILE_PATTERN is empty.
(Bitbake rev: b7da31641c24c53d47ea45a7119d1bd353011b39)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the code that existed in tests/fetch.py for determining the path to
'git-make-shallow' into the git module and reference it.
This ensures that 'git-make-shallow' is always available and the desired
version regardless of the path variable or whether git exposes the
command.
(Bitbake rev: 6b508ab8fd5aa796c1c00c970e81e5e93f84d35d)
Signed-off-by: Nathan Rossi <nathan@nathanrossi.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The runall commandline option was confusing people. There are in fact two
different behaviours people may want.
a) For a given target (or set of targets) look through the task graph and
run task X only if its present and would have been built.
b) For a given target (or set of targets) look through the task graph and
run task X if any recipe in the taskgraph has such a target even if it wasn't
in the original task graph.
I've decided to interpret the existing "runall" option as b), even if right
now if behaves like a). For a), which is a valid use case, this patch adds
a "runonly" option.
With both behaviours present, I'm hoping we can then kill off the "fetchall",
"checkuriall" and other tasks from OE metadata and replace them with this
option. This would significantly speed up task graph processing.
(Deleting the checkuriall and fetchall tasks takes "bitbake core-image-sato -g"
from 22s to 8s).
(Bitbake rev: 546a662c877b2d3af35e3996950582ed2df41fe4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a performance sensitive piece of code and the shear number
of recursive loops is causing a significant and unscalable performance
pain point.
This change moves to a two step approach, firstly generating a list of recursive
dependencies for any task, then applying this to the recursive tasks, iterating
over things until no further dependencies are added.
It was noticed an optimisation is possible and the list of recursive tasks need not
contain the taskname, only the base task id. This allows a significant performance
improvement and limits the size of the resursive task lists, improving speed.
(Bitbake rev: eba738ac5672556eaab4f3374c8025c322761c4a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can optimise the loops slightly so we only process given substrings
once rather than many times. This means expanding out add_resolved_dependencies.
Also add a function which allows replacement of the task element of a
task id, reducing the amount of string handling we're doing in a performance
critical loop.
Its also clear that later code adds to the tasks depends so we don't need
to add .depends() to extradeps at the start.
(Bitbake rev: 4ad281224e92b5f94e3a9c17e8898ec8f1086cdc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we only run through the recidepends/recrdepends code once. This
means that we can miss some expansions of dependency trees where one
rec{r,i}depends tasks depends on another rec{r,i}depends task.
In reality we need to iterate over the data until we stop adding
dependencies.
In doing this we can't show quite so granular progress information since
we don't know how many times we'll need to do this.
This does slow down the runqueue prepare phase however some optimisations
are possible and can be handled in subsequent patches.
This fix means some missing dependencies, such as:
<image>:do_fetchall -> <image>:do_rootfs -> <pkgs>:do_package_write_X
-> <ca-certs>:do_package_write_X -> debianutils-native
(via PAKAGE_WRITE_DEPS)
are now found/added.
[YOCTO #12510]
(Bitbake rev: aec2f07d56a19b97b6515897532b113cdead8338)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We've noticed issues on our infrastucture iterating over the many
tag/branch/head reference files that some git repositories may contain.
By issuing the pack-refs command, we move these all to a single file
which speeds up operations with the mirror repos in the downloads
directory in general.
(Bitbake rev: f8126aaf774186a6eaf0bd4067b89c074594886c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This means two different graph files can easily be compared, currently
you'd have to sort them as the output is randomized.
(Bitbake rev: 5f7c6ec785f70beb1a4a1bbc0eb83cfa6cd7740d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a fetcher, e.g., git, is run when pseudo is active it will think it
is running as root. If it in turn uses ssh (as git does), ssh too will
think it is running as root. This will cause it to try to read root's
ssh configuration from /root/.ssh which will fail. If ssh then needs to
ask for credentials it will hang indefinitely as there is nowhere for it
to ask the user for them (and even if there was it would not access the
correct private keys).
The solution to the above is to temporarily disable pseudo while
executing any fetcher commands. There should be no reason for them to be
executed under pseudo anyway so this should not be a problem.
RP Ammendum:
We finally did get more information about how to reproduce this problem,
something needs to trigger bb.fetch2.get_srcrev() in a pseudo context,
for example when AUTOREV is in use or the recipe doesn't have a defined
SRCREV. That SRC_URI needs to be using protocol=ssh. This would trigger
an ls-remote of the remote repo and if that happens under pseudo, the
wrong ssh credentials may be attempted which can hang.
[YOCTO #12464]
(Bitbake rev: ceaca281cafa662aa2385b95641bce309dce843d)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The cfg property on events doesn't add much, all code appears to access
"data" at this point. Remove it to clean up the interface.
(Bitbake rev: bd4b9e4460b60f142c3bf346fb04e360e512eaee)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Without this the paths to git-make-shallow are incorrect and cause test
failures if bitbake isn't executed from cwd or PATH.
(Bitbake rev: 643eacb162b8710330ef292bfda21cfeab97f95c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change adds EventClassesTest class to bb/tests/event.py,
including 47 new test cases for the public interfaces of the
bitbake event and related helper classes.
[YOCTO #10773]
(Bitbake rev: ee5fe4431713b8a29bdb424a29460965374b3234)
Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Test the triggering of bb.event.worker_fire callback.
(Bitbake rev: daa59a2057c811b20d75235526ac6c2079ac6e10)
Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This change includes unit tests for the following functions,
helper class and methods in bitbake.event:
- set_eventfilter(func)
- set_UIHmask(handlerNum, level, debug_domains, mask)
- getName(e)
- class UIEventFilter(object)
def __init__(self, level, debug_domains)
def update(self, eventmask, level, debug_domains)
def filter(self, event)
[YOCTO #10773]
(Bitbake rev: 4a19dde704fd0bf262ea991ef530f991a4897d31)
Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix a fatal error introduced by a corrupted patch file
submission.
[YOCTO #12459]
(Bitbake rev: dcd58d351c0478ba7b9fe6c0e5b6a97098c1eb21)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If for example you deltask do_build but it has recrdeps set, its confusing
to have that list returned when the task no longer exists (same would apply
to deps too if it was set after the deltask).
(Bitbake rev: b7a81cb91e82ba64b63c9153dc161a2ab4696715)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a 'nobuild' option for starting Toaster without the project
and hosted builds support. This allows a Toaster host to provide
local build statistics without opening the host to external users
building projects.
[YOCTO #12315]
(Bitbake rev: 2d14d6004b6add5ce07295fff1144ade2e54e1c9)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The Toaster initial landing page needs to show that it still has
the command line build capture support in addition to the new
project support.
[YOCTO #12316]
(Bitbake rev: d0358432ca9dd3deef623f6d0585d99d23b3aede)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow for custom environment additions for git cloning, for example
for anspass support.
[YOCTO #12193]
(Bitbake rev: b4717888c55681a49803c4842140af644a5cdc71)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow for "SinglePackageInfo" events that do not include package data,
for example OPKGN equal 'lib32-*' or 'lib64-*'.
[YOCTO #12204]
(Bitbake rev: 567f072ff260614cde3da220a40a95d5a8b9ab92)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster needs to accomodate API changes in Django 1.11.
[YOCTO #12192]
(Bitbake rev: 1b34e3c0075b4bb8a4800fef3e12c3f39743973c)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The latest build enviroment script is now setting BBPATH. This breaks
building previous releases from a master-based Toaster, because with this
newly inherited BBPATH value the 'bitbake.lock' file ends up in the
Toaster build directory instead of the project's build directory.
Toaster should always clear BBPATH so that the bbserver's environment
is clean (enough).
[YOCTO #12363]
(Bitbake rev: 21dde782c049108dd9455ffbf431de214437e800)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We saw builds where runtime providers were sometimes changing order and the
build result was therefore non-deterministic. For example it could show:
DEBUG: providers for lib32-initd-functions are: ['lib32-lsbinitscripts', 'lib32-initscripts']
or
DEBUG: providers for lib32-initd-functions are: ['lib32-initscripts', 'lib32-lsbinitscripts']
which could cause a test to pass or fail.
This change ensures we don't rely on the random order of dictonaries in
memory and act deterministically.
(Bitbake rev: ebce92bf8d71f8a6e8af1c6cf6ba335faf9d67c8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
.txz is the same as .tar.xz, and can be found in the wild.
(Bitbake rev: 2ba8a6b25ccc12e7b543e8450121e5311c7a701d)
Signed-off-by: André Draszik <git@andred.net>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The connection cache class uses a dummy file object but it doesn't have a closed
attribute, so we can't use it in a context manager.
(Bitbake rev: 7b072ef91d16331eae11bd60f229ce1f0c175995)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a matching recipe is not found then return Null instead of raising
KeyError because we were blindly using None as a key for pkg_fn.
(Bitbake rev: 431e89e322850a2497157c3c0843da9df6bc9a3e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The -i option supports more than one class, but the help didn't mention
that.
(Bitbake rev: 1060955c4aa2ef66cdb4f0549f9bd8c1c332673c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you can add multiple layers at once, it stands to reason that you
should also be able to remove more than one at a time.
(Bitbake rev: 2f2033836a5ce4064d9e4f263788a563001bc008)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Allow specifying multiple layers with bitbake-layers add-layer so that
you can add more than one in a single command. This is not just useful,
it's actually pretty important if you need to add a layer and its
dependencies at the same time - since we now go through a parse process
when the layer is added, without this you have to add them all in just
the right order and wait for the parse each time which is somewhat
painful.
(Bitbake rev: ad6b14f01aa326a1c6baa31bfac33be238bce805)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is primarily paranoid but ensure we remove any loggers we setup
either directly or indirectly so the initial state is restored after
we exit.
(Bitbake rev: af7d63b1f76fd3f7fa92ed15ae61ca47d9e13472)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst we're likely exiting in this case, clean up the loggers we add
so that in the case of certain server retries there is no possibility
multiple loggers stack up.
(Bitbake rev: 25b7bf6672be66bcbfe5760610dce7d3e866cdcc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add in a tets ftp url so we ensure ftp urls contnue to work after the loss
of the ftp.gnu.org ones.
(Bitbake rev: e1e8565b5e19dd3f7ef6e7e41932456adaa3df81)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As we test multiple URLs in this these tests and one failing abandons the test,
use subtests so all URLs are tested. This should help us identify patterns in
the failing URLs.
(Bitbake rev: c4c4465b32e82d4b6e46a44e776be5039aef6b18)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The ftp server at ftp.gnu.org is likely to be retired at some point soon
so siwtch over to the http/https services.
This means bitbake-selftest doesn't have ftp test urls, however finding stable
ftp test servers is proving increasingly hard.
(Bitbake rev: 892a08245ddb21a464aeb37d3e32377e99dd7e2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After cleaning deprecated API usage repo fetcher is missing
logger as it was indirectly imported via deprecated bb.data.
Fix this by importing logger directly.
Fixes: 9752fd1c10b8 ("fetch2: don't use deprecated bb.data APIs")
(Bitbake rev: f8e027d26603db2f1fe757dca767ea35d95174c7)
Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-layers show-recipes and show-appends supported listing all recipes
or one recipe. Adjust the system to permit specifying more then one recipe.
Also update show-appends to match the --help description and support file
style wildcards for selecting the recipe to display.
(Bitbake rev: d72c1a91c261d78004d80e2fe5634f0e5f1ef947)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of not even having the test functions if network tests are disabled, use
a custom decorator to mark the network tests and skip them.
(Bitbake rev: cc420f430b1dafd9ca944bea259a564aaab34595)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you run bitbake in an invalid directory, the user experience
is poor:
birbake/lib/bb/main.py", line 427, in setup_bitbake
topdir, lock = lockBitbake()
File "./bitbake/lib/bb/main.py", line 494, in lockBitbake
lockfile = topdir + "/bitbake.lock"
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
This ensures we exit straight away with a better error message.
[YOCTO #12163]
(Bitbake rev: 562f9ee674a8b392437096422b9cceab9c3cba2e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In Python 3.6, glob.glob() was reimplemented to use os.scandir() (which
itself appeared in Python 3.5), thus our monkey patching of os.listdir()
here was no longer effective. The end result was not only that bitbake
wouldn't notice added recipes or bbappends with BB_SERVER_TIMEOUT set
when being run with Python 3.6 (the shipped Python version on Fedora 26
and some other distribution versions), it also broke devtool modify,
devtool upgrade and devtool extract since they rely on the ability to
create a bbappend on the fly and have bitbake pick it up.
To fix it, do the same monkey patching for os.scandir(), which needs to
be conditional upon that actually existing since we have to support
Python 3.4 that doesn't have it. Long term we should probably look for a
better way to handle this that doesn't involve monkey patching Python
library code.
Fixes [YOCTO #12185].
(Bitbake rev: d02e90db32e7ee341c2ba3be79b0627d8796bdd6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The newly added file in the last commit was corrupted, fix it.
(Bitbake rev: be393f247a08c0a4a50a6a76b8fd57f78295d2a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove "prettify.js" and "prettify.css" due to license issues with Apache2.
Replace with "highlight.pack.js" with its BSD3 License.
[YOCTO #12206]
(Bitbake rev: 6361698819530382541506b06a61f2c76dde59cb)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Brian Avery <brian.avery@intel.com>
Signed-off-by: David Reyna <david.reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding layer "meta-signing-key" to conf/bblayers.conf
Traceback (most recent call last):
File "/local/build/project/build/poky/bitbake/bin/bitbake-layers",
line 103, in <module>
ret = main()
File "/local/build/project/build/poky/bitbake/bin/bitbake-layers",
line 96, in main
return args.func(args)
File
"/local/build/project/build/poky/bitbake/lib/bblayers/layerindex.py",
line 250, in do_layerindex_fetch
self.do_add_layer(localargs)
File
"/local/build/project/build/poky/bitbake/lib/bblayers/action.py", line
44, in do_add_layer
if not (args.force or notadded):
AttributeError: 'Namespace' object has no attribute 'force'
(Bitbake rev: 4325f7a7df67eaf4b51af03b453e84bf88fae408)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake server changed such that the Toaster custom settings from
'toaster.conf' and 'toaster-bblayers.conf' that were set when
the '--server-only' mode was started were lost when the subsequent build
request happened, resulting in builds missing all custom changes.
This patch asserts those environment settings in both server calls.
[YOCTO #12194]
(Bitbake rev: fa767d85f19a7af92a44fe11fdfb38633009ad71)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the destination is a directory, building the the destination file
path is always needed. That's because even if the copy fallback is
taken, it's always followed by a rename.
(Bitbake rev: 14c17480827ced2e03c1b62dc839696421fc4de8)
Signed-off-by: Alberto Mardegan <amardegan@luxoft.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were seeing cases where we could hit the 5s timeout on large/fast machines
running many different tasks at once. Increase this to 30s since the main
connection timeout path should no longer hit this slow path.
[YOCTO #12116]
(Bitbake rev: 131d4b8a5834781a93ed41e2967d8dcd4d80f29a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The dot '.' character should be allowed in the user paths for
local non-git layers, DL_DIR, and SSTATE_DIR.
[YOCTO #10650]
(Bitbake rev: 9c0f6e81dae0b91b425e6c0cdf64caa5c15d92b5)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The HEAD reference in Toaster layers are reserved for the
"Local Yocto Project" layers, stored at the top directory.
Imported layers are not allowed to use this since they are
managed differently - for example the 'remotes' will collide.
Fix the add layer handler to not drop the data fields when it
is a git repo.
Explicitly inform the user when an internal Toaster error is
returned via AJAX, so that they know why clicking the layer add
button did not do anything.
[YOCTO #9924]
(Bitbake rev: 04bd9c9341e8390a7923fa0122cd4cb0befa569f)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In collect_bbfiles() we're monkey-patching os.listdir in order to find
which directories to watch, and then undoing that when we're finished -
however if an exception occurred for any reason there was nothing to
ensure the latter occurred. This may not have caused any issues, but as
this kind of thing really ought to be secured using try...finally just
in case, so do that.
(Bitbake rev: 013047484a03185c0ce281c53c1db4949cdc4e69)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The code that was supposed to watch directories along BBFILES for
creation of new files wasn't working in the case where the directory did
not initially contain any matching files - since in updateCache() we are
passing the directory path to add_filewatch() and the latter function
calls os.path.dirname() on the path on the assumption that it is a file
path, and thus the parent of the directory got watched but not the
directory itself. (If the directory wasn't empty everything worked fine
since add_filewatch() was called elsewhere with the path to one of the
files in that directory, and thus the directory got watched). Add a
parameter to add_filewatch() to tell it we are passing it directory
path(s) rather than file path(s).
(Bitbake rev: 47a34dee08fcc25d896a1bdf16fa86267f0b898f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Printing the last 10 lines of bitbake-cookerdaemon.log when the server
fails to start can sometimes result in printing the output from a
previous run, which could lead the user completely down the wrong path
in terms of the cause of the failure. Use a known start text containing
the time which we can then look for when scanning through the log, and
then grab the last 10 lines of that part instead.
Fixes [YOCTO #11903].
(Bitbake rev: 567f2cf1bc455b4f3cfb1cbd7f25145360b05a62)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The whitelist shouldn't have to be populated in order for the
enforcement to work properly - check if the list is not None in order to
determine whether the functionality is enabled or not since that is how
the function that sets up the list behaves.
(Bitbake rev: 7b1e79c352ca6eef1693d8abfacf7505544f1caa)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Issues in start are not being correctly detected by the current readypipe
code. Change it to use specific "ready" or "fail" messages to correctly
determine the correct failure mode and avoid bitbake seeming to hang
(it does currently timeout eventually).
[YOCTO #12062]
(Bitbake rev: 60d4791e3dd05729d2a2adf6f3b203c80d466a73)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update the Toaster stable release selection to 'Rocko' and
the bitbake version to '1.36'.
[YOCTO #12037]
(Bitbake rev: ed3aab90dbf9b6efefd9eb66d81123cb94d3c7da)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Stop the pending build and report to the user if there is
an internal exception due to a git error, a bitbake server
error, or if the server-only mode halts due to for example
a user syntax error in a layer or recipe.
These exceptions were not caught because they occure before
the normal toastergui processing was started.
[YOCTO #12056]
(Bitbake rev: 2cd664097c29ad07b08c82d07a239ca199abbc9a)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is a network failure the return value from latest_versionstring() is
('','') which later causes an exception when comparing versions.
Improve this by checking the return value and failing the test early.
[ YOCTO #12053 ]
(Bitbake rev: 3f034d2172bf64ecc43577b43e0cf032a54b1358)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since 2017-08-17 (git version 2.14.1.473.g3ec7d702a) using deprecated
git branch parameter "--set-upstream" causes a fetcher error. Replace
it by "--set-upstream-to".
https://git.kernel.org/pub/scm/git/git.git/commit/?id=52668846ea2d41ffbd87cda7cb8e492dea9f2c4d
says, it's deprecated since 2012-08-30 so hopefully all still supported
host distributions have new enough git to support "--set-upstream-to".
ERROR: PACKAGE do_unpack: Fetcher failure: ...;
git -c core.fsyncobjectfiles=0 branch --set-upstream master origin/master failed with exit code 128, output:
fatal: the '--set-upstream' option is no longer supported. Please use '--track' or '--set-upstream-to' instead.
ERROR: PACKAGE do_unpack: Function failed: base_do_unpack
(Bitbake rev: 2ab50074c1a6c56a8a178755de108447d7b7acaf)
Signed-off-by: Andre Rosa <andre.rosa@lge.com>
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the UI disconnects, we can throw away any server-side remote
datastores we created in response to calls from the UI, and we *must*
drop everything in extraconfigdata or it will taint any future
operations.
Dropping extraconfigdata upon disconnect fixes taskhash mismatch errors
when running devtool.DevtoolTests.test_devtool_update_recipe_local_files
within oe-selftest with BB_SERVER_TIMEOUT=100 in OpenEmbedded.
(Bitbake rev: 1ca2eec459424892391f060442ef38cf28d6a54a)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you have a layer with a blank BBFILE_PATTERN the layer was ignored
when processing the list of layers with priorities. This list is not
only used for processing recipes, but also by additional programs such
as bitbake-layers show-layers.
Without this change, a layer that provides configuration or classes
only does now show up in show-layers, which is used by the
yocto-compat-layer.py script. This causes a failures in the compatibility
check.
(Bitbake rev: a0eaf8c0f228f984bafff09e4e9739f758dc1a9b)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're implementing reset() in SignatureGenerator at all (and we need
to for a basic non-OE BitBake setup where that is the default signature
generator), then we need it to be clearing out the internal values
properly.
(Bitbake rev: 13f52d38fdbcb84c2a0c46f85baa44b22d53fdc1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The list of columns in the many 'Edit Columns' pop-ups became unsorted
with the 'Toaster Table' implementation. These entries need to be
gathered and sorted in the column processing.
[YOCTO #12004]
(Bitbake rev: 17aa1ef8f0a00dd3456aac199e558a2f96bf7ad9)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The default layers are missing the recipe link definitions in
the fixture files, and because they are predefined they do not
get the updated information from the Layer Index.
[YOCTO #12006]
(Bitbake rev: 2ff5592baf011de9c778d3c2481b8ed3912f1a4b)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The file "lsupdates.py" is missing the import of 'ToasterSetting',
and that breaks setting up and updating Toaster and its database.
[YOCTO #12036]
(Bitbake rev: d4aef1f1f03d88acc76423b55d8bd6c137f66f44)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If no external layers are defined, pokydirname is not set. Rectify
this by taking the 'be.sourcedir' as the pokydirname.
[YOCTO #12015]
(Bitbake rev: 3b0f04c7b00aaf44ba146c432fcb5d6fd7dafcea)
Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
One of the debug messages during build contains a list of all layers
but without spaces or other separators between them. Use pformat
instead.
[YOCTO #12014]
(Bitbake rev: 9fe38f94b54a8644ac6f493c49e63dd6da5bfbdf)
Signed-off-by: Olaf Mandel <o.mandel@menlosystems.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The build page provides tables related to performance that
cover build time, CPU time and disk IO. The "Edit columns"
drop down does not allow selection of the Order column and
makes it hidden as well which is not accurate from user
interaction point of view.
This patch enables the hideable property for the Order column
so it is hidden by default but the user can enable it through
the drop down if need be.
[YOCTO #11040]
(Bitbake rev: 1f1cc1edb21aeec684ef7323554794b33d84d414)
Signed-off-by: Awais Belal <awais_belal@mentor.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There must be at least one FSTYPE selected in the Toaster bitbake
variable editor page. When the user deselects all the "Save"
button gets disabled, but the error message is missing.
[YOCTO #8126]
(Bitbake rev: 193577655b7491126ca5fa91fa76d79329e900c2)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_SERVER_TIMEOUT=100 oe-selftest -r bblayers
was failing and highlighted that since parseConfiguation clears data
structures, it needs to also clear parsecache_valid as it no longer
contains correct data.
(Bitbake rev: 7234f33a7eb38ad51a8345f6689bc26e29f29f92)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I can't actually see how this was working, nothing connected the commandline option
to the data in TaskData(). Drop the remaining pieces of this option, it was a relic
from a decade ago and we want deterministic builds, not random tries until something
might work.
(Bitbake rev: 767c7ba8fc76ec667ac1567de1c971c3575f2ecd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix failure after commit "cooker/siggen: Reset siggen when reparsing"
(e4c6ca9440f63761560b49bbe12654441f54687e) when executing without
specifying a BB_SIGNATURE_HANDLER.
(Bitbake rev: 2a78c2d09aea0323632bbc927f370f1d3c9c249e)
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_SERVER_TIMEOUT=100 oe-selftest -r devtool.DevtoolTests.test_devtool_build_image
fails, the reason is that internally, the limited_deps flag is set in
the bitbake server and then never cleared. This causes the sysroots to
be setup incorrectly (as per the limited dependency case) and builds
break. There is also potential for corruption of recipecaches.
Add shutdown/cleanup code to ensure these effects don't 'stick'.
This bug is particularly nasty as you can destroy TMPDIR with large
sysroots in build work directories which are prone to break.
Also ensure mtime cache is cleared (to match buildTargets) and
that no lasting changes are made to siggen either which ensures:
BB_SERVER_TIMEOUT=100 oe-selftest -r devtool.DevtoolTests.test_devtool_upgrade_git devtool.DevtoolTests.test_devtool_virtual_kernel_modify
works.
(Bitbake rev: 0a7ee8c8378bba9877c260b1aee782878f1935b4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When fetching source for the first time within scripts such as
OpenEmbedded's recipetool, we don't want to be showing warnings about
NPM_SHRINKWRAP or NPM_LOCKDOWN not being set since there's no way we
could have set them in advance. Previously we were using
ud.ignore_checksums to suppress these but since we are now using a more
standard task-based path to fetch the source, we need to disable these
through the metadata. Look for a "noverify" parameter set on the npm URL
and skip the checks if it is set to "1".
(Bitbake rev: 8c4b35d1e4d31bae9fddd129d5ba230acb72c3bb)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you run some other operations that result in the cache being
populated, and then call buildFileInternal(), then you can end up in a
situation where the cache already contains information about the recipe.
For example in OE this can now happen when you use devtool upgrade.
Normally this doesn't cause any problems, unless you have a non-absolute
path in BBLAYERS - in buildFileInternal() we are calling matchfile() which
will convert the filename to absolute, but later when taskdata goes to find
the providers of the recipe it finds the non-absolute path, sets up the
task information using this and then the runqueue can't find any tasks
matching the absolute path. To fix this, back out the optimisation I did
earlier in bitbake rev ba53e067a2d448dd63b4ca252557ce98aa8e6321 to avoid
calling parseConfiguration() again, which is unfortunate but does result
in the cached information being that causes the problem being cleared
out.
This fixes "Task do_unpack does not exist for target ..." running
devtool upgrade within intel-iot-refkit.
(Bitbake rev: f120355eaec4571ba6d60fc5f7ae9e1f31d846d1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you set tracking=True when creating the tinfoil object, that ensures
history is collected for the main datastore, but at the end of parsing
the configuration, history tracking gets turned off to save time with
the result that we don't collect history for any recipes we parse.
Enable tracking when we parse a recipe (and disable it afterwards if we
enabled it) in order to fix this.
This fixes functionality in OE's devtool that relies upon variable
history (such as devtool upgrade updating PV when it's set within a
recipe).
(Bitbake rev: cc8b4c81bb589fb70774a0151f87a8d277f40f06)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a task fails during build_targets(), we need to print out the log
lines as knotty does or the user will be missing information about the
failure.
(This should get some deeper refactoring, but now isn't the time for
that.)
(Bitbake rev: 24879df071d4803db3d39ae1d5cad852daa92f28)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With config_only=False we launch the UI and it sets up a logger, whereas
when config_only=True we don't, with the result that with True we are
seeing log messages from both our logger and the one set up by the UI.
Suppress our loggers with config_only=True to avoid this.
Fixes [YOCTO #11275] (again).
(Bitbake rev: b5e3b28b7c982dd8a3991d727f25710dbf58bb80)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the server exits quickly its PID may no longer exist. Handle
this gracefully.
(Bitbake rev: c1b00a9265fa4146b8db8b7d03a51bf2bfcf9f51)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
self has a quit function and a variable. Separate this to two different
things as the current setup is prone to breakage.
(Bitbake rev: ba7e3c73d8f4d2bd1d7434b97c326e7ab935231a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Shutdown from SIGTERM currently has to wait for the handler thread to timeout.
Add a sentinel value which triggers it to loop and allows for a quick exit.
(Bitbake rev: a7591ef34ce70ff1d7aa9362d7473e6f16fbd10f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If for example you run:
bitbake -r somefile.inc
rm somefile.inc
bitbake -e
bitbake will crash with an error about not being able to find somefile.inc. This
is because it tries to reparse the base config for the early getVariable requests
before it sees the updated missing -r option.
Send the updateConfig command earlier to avoid this.
(Bitbake rev: a38164620ebdc770690c5f39ff9ed69d3f82719e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cooker.reset() can be called before we've actually setup the datastore. Gracefully
handle this case instead of the current traceback+exit.
(Bitbake rev: 8fd30ca6d271c125a8ea03ef0c5d7ab176900701)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we reload the configuration, PRSERV_HOST can change. Therefore
restart the PR Server depending on the new configuration at reparse.
Note that the server has to be started from the right process, it
can't be in the UI which shuts down as that shutdown triggers a shutdown
of its children and the PR Server shuts down too. This is why we need
pre_serve() which ensures its executed in the right context.
(Bitbake rev: 971272e84f4efe7ebd0037e164ba54f013a2a34e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows for cleaner code in cooker as any existing server is dealt
with before a new one is started.
(Bitbake rev: b8616931bc0e523a3a3bb23b4f623f8b6e71d690)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the original prefile and postfile passed when starting bitbake
server are 'sticky'. With the new memory resident model this doesn't make
sense as the server the system is started with isn't special.
This patch changes the code so the prefile/postfile are used if specified
on the commandline and not used otherwise. This makes the behaviour much
more predictable and expected and as an added bonus simplifies the code.
(Bitbake rev: 638d366234fad78f283d3a13a12b07cb0ccbe914)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current passing of "tracking" backwards and forwards, client to server
is ugly and complex and error prone. Instead, set this during showEnvironment
commands triggering a reset there if/as required.
(Bitbake rev: 9dc7f384db0479569ff93a76a623d5395fecaf47)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop pointless unused function parameter.
(Bitbake rev: 8104b33656de0b619943bd7a9884eb650ccafbf4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The 'needconfig' flag was meant to be the default as most commands do need
it and the "False" cases were the exception. The code was written backwards
with a default False value. Invert this to match the intent, resulting in
the config being reparsed if metadata has changed.
Also ensure the second level configuration is parsed for the getLayerPriorities
command as otherwise it can return stale info.
With these changes:
BB_SERVER_TIMEOUT=100 oe-selftest -r bblayers.BitbakeLayers.test_bitbakelayers_add_remove
passes instead of fails.
(Bitbake rev: af3c8928a69f204d5ced02c947485990ac04a776)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than just waiting for 5s, try and get the lockfile. If we gain
the lock, we know we're ready to retry and can skip any remaining timeout.
(Bitbake rev: 8a60106c6f7d586c793b965c5e9460b6016fab15)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Its possible for a connection to connect to the server as its shutting down
but before its removed the socket file. This patch:
a) Removes the socket file earlier to avoid connections.
b) Handles EOFError in initial connections gracefully. These occur if the
socket is closed during the server shutdown.
c) Ensure duplicate events aren't shown on the console. This makes debugging
these issues very very confusing.
With these changes the backtrace that was concerning users is hidden and the
server works as expected with a reconnect when it catches it in a bad state.
(Bitbake rev: f45196cf84669723382730944dddc7eaf50826f2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
URLs do not have to have a path; currently our npm URLs don't, so
encodeurl() needs to handle if the path element isn't specified. This
fixes errors using OpenEmbedded's devtool add / recipetool create on an
npm URL after OE-Core revision ecca596b75cfda2f798a0bdde75f4f774e23a95b
that uses decodeurl() and encodeurl() to change URL parameter values.
(Bitbake rev: d5cab2dbf5682d2fd08e58316a3bf39a10f63df2)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The underlying model is already sorted for use in the other view,
add a sorting model for the 'Dependent Tasks' view.
(Bitbake rev: 27ca94c33234f0ef9753f8285213dde2871a3fcf)
Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster needs to allow the custom fixture file to specific an
alternate Layer Index URL for users that host their own Layer
Index Server via the 'CUSTOM_LAYERINDEX_SERVER' in
'custom.xml'.
Toaster also needs to allow the ability to completely override
the default fixture files, otherwise that content can leak into
the custom environment (by default the custom fixture is an
overlay that cannot remove existing values from the default
fixture) via the 'CUSTOM_XML_ONLY' value in 'setting.xml'.
[YOCTO #11938]
(Bitbake rev: ac29d4a9078494544a2627d8b6b021096b49cb34)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Toaster needs the ability to allow custom extensions to execute when
Toaster is started and stopped. Toaster will look for a custom
extension script in the fixtures directory and execute any applicable
hooks.
[YOCTO #11938]
(Bitbake rev: 12a73f6914488029f2b9dd680e004fba7dde41af)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When Toaster removes a package from a custom image, it must
also always remove the advised reverse-dependent recipes.
Similarly, when adding a package it must always add its advised
the packages it depends on. This code must be un-indented so
that it applies to all respective added or removed packages.
Toaster normally waits until a new custom image is built before
creating the custom layer and the recipe. However, an intermediate
different build can fail because the recipe has already been added
to the project, so the image's default recipe must be created
when the image is created.
[YOCTO #11915]
(Bitbake rev: bcd68fd7231f166baff875fb88e4f9ce0d9bf91d)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is an interesting bug in the current code where a sync command
is not seen until the current async command completes, by which time
the UI may have shut down.
The reason is that if there are idle commands, we may not end up sleeping
in the select call at all, partiularly under heavy load like parsing.
Fix this by calling select with a zero timeout so that we see active
fds and know to read from them. This fixes various problems toaster was
having with the recent server changes.
[YOCTO #11898]
(Bitbake rev: bbcce58e824b2793abf50efa52db158ae16e23e4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We're seeing the server fail to start within 8s on heavily loaded
autobuilders so increase this timeout to 30s which should be more
than enough time.
(Bitbake rev: 8d4c120ec46d6d7a54947c64d33e18cb60b60505)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There have been cases where the server could loop indefinitely and incorrectly
handle client disconnects. In the EOFError case, ensure a full disconnect
happens in the alternative disconnect path to avoid this.
(Bitbake rev: 5e267f14bb0155889615f567a920af4a37eb3c6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Writes to the cookerdaemon log and/or the lockfile were meaning the parser
cache was always being invalidated and reparsed. This is unnecessary so
spot accesses to these two common cases and ignore the files from a reparse
perspective.
This doesn't remove many sources of reparse but does improve several
common cases.
(Bitbake rev: 218e4b6418992588312b8ef5949b84ef43263d1a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a proper function for clearing the mtime cache. Clean up the inotify
event overflow case to err on the side of caution and clear any potentially
now out of sync caches.
(Bitbake rev: ec60459fe2ba16966544eebff43b061abb7ed3ba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The inotify watch events are handled "at idle" which could in theory mean
a command could run before any preceeding inotify events have been processed.
This leads to a theoretical race window where those events may have a
signficicant effect on the command.
Add a mechanism to allow us to ensure all pending events are processed before
running commands.
(Bitbake rev: bf76cd7e5881adf264b8ba64e27a5b6ca9df4fde)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some of the directories searched by BBFILES are not currently being added
to the inotify watch list. This can mean that added append files are not
noticed leading to misleading metadata results when using
BB_SERVER_TIMEOUT != 0.
We use glob to expand the BBFILES references and without writing our own
implentation, figuring out the directories it searches is hard. We use
some horrible hacks here to intecept the listdir calls, I'm open
to better ways to handle this but this does ensure we have the right
watches set.
(Bitbake rev: 4d508d35a224e3a25d2d59c8415ab7985964b14f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The server currently crashes if we hit an EOFError due to controllersock
still being in ready and the continue meaning ready isn't re-evaluated.
Setting the value to False can mean the shutdown code doesn't handle the
situation cleanly.
Clear ready to avoid the crash/loop instead and handle any OSError whilst
we're in here.
(Bitbake rev: 2bc47e887c3b41417edaa89a8708c223fd2085de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we don't do this, we get basehash mismatch errors occurring from the reparse
which would then set bitbake's error exit code.
This for example would cause oe-selftest -r bbtests.BitbakeTests.test_bbappend_order
to fail with a non-zero BB_SERVER_TIMEOUT.
(Bitbake rev: e4c6ca9440f63761560b49bbe12654441f54687e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make BB_SERVER_TIMEOUT = -1 mean no unload forever.
(Bitbake rev: 923d5567be6fd9969ca74c166f36817ec09305e3)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
$ export BB_SERVER_TIMEOUT=10000
$ bitbake --server-only
$ bitbake --status-only
[snip]
File "/buildarea/lyang1/poky/bitbake/lib/bb/server/process.py", line 472, in recvfds
msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
OSError: [Errno 9] Bad file descriptor
And:
$ export BB_SERVER_TIMEOUT=10000
$ bitbake --server-only -B localhost:-1
$ bitbake --status-only # Everything is fine in first run
$ bitbake --status-only
[snip]
File "/buildarea/lyang1/poky/bitbake/lib/bb/server/process.py", line 472, in recvfds
msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
OSError: [Errno 9] Bad file descriptor
This was because self.controllersock was not set to False, so it still ran
sock.recvmsg() when sock was closed.
And also need set command_channel to Flase, otherwise the
self.command_channel.get() will always run when EOF, and cause infinite loop.
(Bitbake rev: 7b739a38601b053d9bea4df2c0b44a952ab670c4)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is already a type=float, so the float() is not needed, which also makes
the error clearer:
$ export BB_SERVER_TIMEOUT=10000AA
With float():
$ bitbake quilt-native
[snip]
ValueError: could not convert string to float: '10000AA'
Without float():
$ bitbake quilt-native
[snip]
optparse.OptionValueError: option --idle-timeout: invalid floating-point value: '10000AA'
The second one tells clearly where is wrong.
(Bitbake rev: 7a4ea1e6a10f20d80009a78f4f0aebf8f90095fd)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In combination with the recent server reworking, this change actually
prevents messages sent from tasks from being logged properly. This will
of course give us the duplicated messages back, and I really hate to do
that effectively a second time, but that's better than seeing no error
at all in the case of a failure - we'll have to find the proper way of
avoiding the duplication that doesn't result in some messages going
missing.
This reverts commit 8a5bae76f91f2411187c638a42fa3c762052cf11.
(Bitbake rev: 645c8dd15762516ae5ab64a1df47fadb95d072d1)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there are events in the UI queue we want to print them regardless of
whether we're handling an exception or something like SystemExit.
This improves error messages for some other failure cases where currently
no logging would get printed and leave the user confused as to what went
wrong.
(Bitbake rev: 76c4f2c20216719736766e8ae7d089ccd061b71b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"bitbake -g m4-native -c do_unpack" doesn't list any dependencies in
task-depends.dot. This is because no header describing the task was
added unless a task has dependencies. Tweak the code to fix this.
[YOCTO #10893]
(Bitbake rev: e2fbe728769f09ef4178e2ef4d69a2f60ffe3ebb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This was originally based on some other code but its mostly misleading
comments now. Massively clean this up and accept its now a total fork.
There are no funciton changes here, just cleanup.
(Bitbake rev: 3daa9001269e5b9156d0eb3299d2628d2d219fff)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we create the child, if an exception occurred it was transfering
back into the parent context. We don't want to do that us use a try/finally
to ensure we exit.
We need to ensure a traceback is printed and any queued UI messages which
may not have made it to the client UI at this point.
(Bitbake rev: dec1d2c26f6cb3ffeb44beaab0129cd531a6d08b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have cases where the server is being started but we're not seeing any messages
from it. Add some earlier logging so we can try and better understand where
issues may be occurring.
(Bitbake rev: 10a6a48c0c5a2fe2051e90143e66075356853971)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the connection control socket and the command channel close together,
we can race and hit EOFError exceptions before we close the channel.
Reorder the code to handle this in the correct order and ignore the
EOFError exceptions as they mean the client is disconnecting and shouldn't
terminate the server.
(Bitbake rev: 974281ed72d8366baa16ee85f7e93970689b5956)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
My previous assertion about FusionForge appears to have been wrong, or
FusionForge has changed behaviour, or both.
FusionForge now mandates that downloads have the Accept header set, despite that
header being optional, and returns a 406 Not Acceptable error if it isn't set.
As we were pretending that 406 was actually 405 (Moved) and tried to handle it as a
redirect this results in an infinite loop until Python kills the recursion.
Delete the handling of 406 as 405, and pass Accept: */* in the headers.
(Bitbake rev: bb70ae0c9aac5ec688026d23a64ac0cac1947187)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bitbake option "-t SERVERTYPE" was deprecated and can be
removed since the desired XMLRPC listener now the default.
The bitbake server port cannot be "-1" anymore and must be
explicitly passed.
There is a race condition for the bblock file to not only
be created but to actually be populated, so a delay test loop
is required (usually only one pass).
The 'xmlrpcclient' is now the class that allows Toaster to for
example kill builds.
The events for populating the recipe parsing now either show only
the final result or are skipped entiredly, so the progress
calculator needs to be changed to not block on the parsing phase.
[YOCTO #11851]
(Bitbake rev: 2aa7ad38f760ec003fb18faa5aa0014cff191a7a)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add support for Toaster aggregators with a set of api links that
return JSON data for (a) builds in progress, (b) builds completed,
(c) specific build data, and (d) an is-alive health ping link.
[YOCTO #11794]
(Bitbake rev: d8e79661c69671dd424dca5cc3f7f2f855b0afed)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having this as one huge long line isn't easy to manipulate, split it into
multiple lines for ease of debugging issues.
(Bitbake rev: 5753fe81194f75fbcf4ccdc733cc585d02794cb1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The checkstatus() code was expecting checkstatus to throw exceptions if it
failed, but in general it should return False.
(Bitbake rev: 57be5cc6228518e60f564570a39cebbeb6cf564e)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
`git fetch --tags` seems to interact badly with `mirror=fetch`, resulting in
the regular branches not being fetched, so drop the unnecessary `--tags`. This
fixes this unit test failure: `bb.fetch2.FetchError: Fetcher failure: Unable
to resolve 'master' in upstream git repository in git ls-remote output for
/tmp/tmp4ag_mgmn/gitsource`
[YOCTO #11698]
(Bitbake rev: 2d0203fae08c5ff8dc3e9afaa9a819abc4a1af6f)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the server isn't running return 0 as to do otherwise complicates
scripts which do cleanup of bitbake servers which would potentially
be memory resident.
(Bitbake rev: 0f85c54385f325f3ebbf911b35bbac11b8863b6b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This timeout path was commonly hit due to errors starting the server. Now we
have a better way to handle that, the retry logic can be improved and cleaned
up. This patch:
* Makes the timeout 5s rather than intervals of 1s with a message. Paul
noted some commands can take around 1s to run on a server which has just
been started on a loaded system.
* Allows a broke connection to exit immediately rather than retrying something
which will never work.
* Drops the Ctrl+C masking, we shouldn't need that anymore and any issues
would be better handled in other ways.
This should make things clearer and less confusing for users and is much cleaner
code too.
(Bitbake rev: 8633b7cd03cfaba3e0359aa5da22fc76b66768c7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we reconnect to the server, stop leaking pipes and clean up
after ourselves.
(Bitbake rev: f41e4e971e807157be68cf4496580494b8b60643)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if the server fails to start, the user sees no error message and
the server will be repeatedly attempted to be started until some longer
timeouts expire. There are error messages in the cookerdeamon log but
nobody thinks to look there.
Add in a pipe which can be used to tell the starting process whether the cooker
did actually start or not. If it fails to start, no further attempts can be
made and if present, the log file can be shown to the user.
[YOCTO #11834]
(Bitbake rev: 57000d44beb1aeba37dfc70782b0d6418943acc5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This cleans up the socket keep alive into better class structured code
and adds cleanup of the open file descriptors upon shutdown.
(Bitbake rev: 77fd3a3a29a569e212374b27aea742ddbaafcdd5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the monitoring thread exits with some error code or runs indefinitely. Allow
closure of the pipe its monitoring to have the thread exit cleanly/silently.
(Bitbake rev: 930d077637928213e13a07c78fee3bf7a8c37ebf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Expose the underlying close() and fileno() methods which allow connection
monitoring and cleanup.
(Bitbake rev: f79187f4ebfad7969be47b429995e7f7a3e33c1e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code would duplicate messages to stdout and stderr when no UI connected
and there were error level messages.
Rework the code so it either uses stderr (for errors and above) or
stdout for warnings/debug but not both for the same messages.
(Bitbake rev: 45cff5734ba2ba8c8d36d17d722a5804d39b258b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were bridging the gap between the server and UI here by calling a
bb.siggen.find_siginfo, a function defined and set on that module from
the metadata. This worked from the UI side before but since the recent
server changes is no longer accessible. Create a new command so this can
execute on the server side and return the result by way of a new event.
(We're still running compare_sigfiles() on the signature generator but
that isn't quite the same thing and does still work.)
Fixes [YOCTO #11844].
(Bitbake rev: fdcea991baa4f83d9c98d468d7b49c8c388a4a15)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When wget is fetching a listing for a directory over FTP it writes to a
temporary file called .listing in the current directory. If there are many such
operations happening in parallel - for example during 'bitbake world -c
checkpkg' - then up to BB_NUMBER_THREADS instances of wget will be racing to
write to, read, and delete the same file.
This results in various failures such as the file disappearing before wget has
processed it or the file changing contents, which causes checkpkg to randomly
fail.
Mitigate the race condition by creating a temporary directory to run wget in
when doing directory listings.
[ YOCTO #11828 ]
(Bitbake rev: 91d4ca93df092cf86ab84faaa94cc66ff9f43057)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The warning message currently shown can occur more frequently than previously
if a previous bitbake server is shutting down and we're reconnecting to a new
server. Change it to a note message to match the higher level connection
logging retry messages and so as not to interfer with selftests.
(Bitbake rev: b7514340cd6a2753eb217b059229bb279c3849ec)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that there is only one server, this abstraction is no longer needed
and causes indrection/confusion. The server shutdown is also broken with
the cooker post_server calls happening too late, leading to "lock held"
warnings in the logs if PRServ is enabled.
Remove the abstraction and put the shutdown calls in the right order
with respect to the locking.
(Bitbake rev: c0ddde7cf680225127d6285685652b905ed176c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The server shutdown is currenlty laggy and race prone. This patch:
* adds a waitpid so that no zombie server is left around if its not
running in daemon mode.
* adds a quit "sentinal" using a pipe so that we're not sitting in
a socket poll() until timeout in order just to quit.
* use a select() call to poll the socket and the pipe for a quit signal.
The net result of this change is that the prserv exits with the cooker server
and it does so immediately and doesn't wait for the select/poll calls to
timeout. This makes bitbake a lot more responsive for startup/shutdown and
doesn't cause UI timeout errors as often when prserv is used.
(Bitbake rev: 0b5a837477d18442caf97dd0fa14a806c54f5842)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the sstate is accessed via HTTP, the existence check can fail due
to network issues, in which case bitbake silently continues without
sstate.
One such network issue is an HTTP server like Python's own SimpleHTTP
which closes the TCP connection despite an explicit "Keep-Alive" in
the HTTP request header. The server does that without a "close" in the
HTTP response header, so the socket remains in the connection cache,
leading to "urlopen failed: <urlopen error [Errno 9] Bad file
descriptor>" (only visible in "bitbake -D -D" output) when trying to
use the cached connection again.
The connection might also get closed for other reasons (proxy,
timeouts, etc.), so this is something that the client should be able
to handle.
This is achieved by checking for the error, removing the bad
connection, and letting the check_status() method try again with a new
connection. It is necessary to let the second attempt fail
permanently, because bad proxy setups have been observed to also lead
to such broken connections. In that case, we need to abort for real
after trying twice, otherwise a build would just hang forever.
[YOCTO #11782]
(Bitbake rev: 6fa07752bbd3ac345cd8617da49a70e0b2dd565f)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trim the entered built target value so that Toaster is not
confused with no real targets nor a ghost second target.
[YOCTO #11727]
(Bitbake rev: dd83c732b945ba6f9b7cdf66f6e88cfd15a745a0)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Set the clone progress to be off by default for the benefit of
command line projects and 'Local Yocto' builds. For Toaster managed
projects that do use the clone feature the clone progress status
is already explicitly set by the existing code and thus displayed.
[YOCTO #11744]
(Bitbake rev: d25a3088150b2a277242027c12becdeb88c45517)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Change the task progress counting from the task order field
which excludes setscene to the task completion field which
counts all completed tasks regardless of type.
[YOCTO #9971]
(Bitbake rev: 485b328430dcdfd0b52f09f01ebd490d09459ea3)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If BB_STRICT_CHECKSUMS is set to anything other than "1" i.e. we're not
going to raise an error, then fire an event so that scripts can listen
for it and get the checksums.
(Bitbake rev: 8b2ccb4b865f2df118ef668847df682a83f9c500)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If BB_STRICT_CHECKSUMS is set to "ignore" then don't display a warning
if no checksums are specified in the recipe. This is not intended to be
used from recipes - it is needed when we move to using more standard
code paths to fetch new files from scripts i.e. where we don't know what
the checksums are in advance.
(Bitbake rev: f15ca7339de8a448a93a14cf6130b3925178a920)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We want this API to be easier to use, so add missing function
documentation to help with that.
(Bitbake rev: 3e0e002d6497caa987f327cd83ad4db82cca6810)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A common task for tinfoil-using scripts is to iterate over all recipes.
This isn't too difficult with the current API, but the pkg_* variables
are a little awkward and are really designed for bitbake's internal
usage - and it gets a bit more difficult when you want to access some of
the other information such as packages and rprovides. To resolve this,
create a new recipe info class and add an all_recipes() function to
generate this for all recipes. Also add a get_recipe_info() function to
get the information for a specific recipe (by PN).
(It might perhaps be suggested that we already have a structure similar
to this in the cache, however the one we add here is designed for
external use and allows the internal structures to change if needed
without affecting the API).
(Bitbake rev: 308994028e59735ca726c5d2c1f0f85baccfe89d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add access to fn_provides, packages, packages_dynamic and rproviders on
the recipecache object. This requires an additional corresponding
command plumbing to be added.
(Bitbake rev: 3df9b7c615174a6557581f3cd157842a28f6bb26)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Up to this point, if you wanted to run build tasks in the normal way
they get run from a python script, there was no other way than to shell
out to bitbake. Worse than that, you couldn't have tinfoil active during
that because only one bitbake instance could be running at once. As long
as we're prepared to handle the events produced, we can create a wrapper
around calling the buildTargets command. Borrow code from knotty to do
this in such a way that we get the expected running task display
(courtesy of TermFilter) and Ctrl+C handling.
(Bitbake rev: 43761eee756be52a1021be53a40dc591a6c35fa7)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This isn't useful for knotty itself, but for use from tinfoil
in case we can't get access to either the console or errconsole, allow
either to be unspecified (None).
(Bitbake rev: 7544de437fc66b81502ecdb5db859182c45827cb)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It is useful for the caller to know whether the uihelper has handled the
event passed so that it can skip other event handling code if so.
(Bitbake rev: be498abfbbb19bdd31e5b53713a74049007e3737)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We really ought to have just one place where the string representation
of these events is produced. This doesn't take any real control away
from the UI - if an alternative representation is desired, that can
still be made.
(Bitbake rev: cb15db2a799be6d8eab9a2a43a9a573f89229cff)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make it easy to determine if recipes are parsed (and thus information
about available recipes is in memory).
(Bitbake rev: 7efde2df2ff25063d36ac015146f1975284a69ff)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python style recommends underscore based naming rather than camelCase,
and thus the former has been used for most of tinfoil's functions. Add
an underscored version of parseRecipes() for consistency and change the
one place we call it to use the new version.
(Bitbake rev: 821f6c41d850752d2bcc2ccd4f8e75b2897a0a3e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In OE's devtool we want to repeatedly run build_file() without showing
unnecessary messages and triggering buildhistory for each call.
build_file() is just a wrapper around the buildFile command. Change
the final "hidewarning" parameter of the buildFile command to "internal"
and have this call a new buildFileInternal() function without triggering
any of the normal build events, silencing the normal info messages from
the runqueue ("Executing RunQueue Tasks", "Tasks Summary" etc.) and
avoiding calling parseConfiguration() which we've already done at this
point.
(Bitbake rev: ba53e067a2d448dd63b4ca252557ce98aa8e6321)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
$ bitbake --observe-only
FATAL: '--observe-only' can only be used by UI clients connecting to a server.
And even:
$bitbake --observe-only -B localhost:-1
FATAL: '--observe-only' can only be used by UI clients connecting to a server.
This was beucase the brackets were in a wrong position.
(Bitbake rev: 4ffc91a2b3eb13e98078e6b1913f056a0c1797bc)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed 1:
$ . ../poky/oe-init-build-env-memres .
$ bitbake -m
$ bitbake -m # shutdown it again
$ bitbake -m
Starting bitbake server...
Terminated bitbake server.
It starts the server and kill it which is redundant.
$ bitbake -m
$ bitbake --status-only
Starting bitbake server...
It starts the server which is not what we need.
(Bitbake rev: 335fd0106359dddfe1eeb0c7d0bc03f8454b895c)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fixed:
$ bitbake --server-only -B localhost:-1
Infinite loop
Reconnecting to bitbake server...
[snip]
Break the loop when server-only can fix the problem.
(Bitbake rev: c4d5c3dab5b35f3c6713efad1c8a23229ce4e56b)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no global var "configuration", so the old code hang at
self.bitbake_lock.write(), and nothing wrote to bitbake.lock. I didn't
figure out why it hang (but not print errors).
Reproducer:
$ bitbake -B localhost:-1 world -k
Check bitbake.log, there was nothing, now fixed.
(Bitbake rev: dbdd9010663cd8dcb328e1b6fb40e3a777293bc5)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This changes the way bitbake server works quite radically. Now, the
server is always a process based server with the option of starting
an XMLRPC listener on a specific inferface/port.
Behind the scenes this is done with a "bitbake.sock" file alongside
the bitbake.lock file. If we can obtain the lock, we know we need
to start a server. The server always listens on the socket and UIs
can then connect to this. UIs connect by sending a set of three file
descriptors over the domain socket, one for sending commands, one for
receiving command results and the other for receiving events.
These changes meant we can throw away all the horrid server abstraction
code, the plugable transport option to bitbake and the code becomes
much more readable and debuggable. It also likely removes a ton of
ways you could hang the UI/cooker in weird ways due to all the race
conditions that existed with previous processes.
Changes:
* The foreground option for bitbake-server was dropped. Just tail
the log if you really want this, the codepaths were complicated enough
without adding one for this.
* BBSERVER="autodetect" was dropped. The server will autostart and
autoconnect in process mode. You have to specify an xmlrpc server
address since that can't be autodetected. I can't see a use case
for autodetect now.
* The transport/servetype option to bitbake was dropped.
* A BB_SERVER_TIMEOUT variable is added which allows the server
to stay resident for a period of time after the last client
disconnects before unloading. This is used if the -T/--idle-timeout
option is not passed to bitbake.
This change is invasive and may well introduce new issues however I
believe the codebase is in a much better position for further
development and debugging.
(Bitbake rev: 72a3dbe13a23588e24c0baca6d58c35cdeba3f63)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if this code is used with something like oeqa's xml logging
it fails as sys.stdout is an io stream. Add in try/except to handle
this case.
Add a waitpid() call to remove a zombie whilst forking.
Also, append to the logfile, don't overwrite it (otherwise
debugging can be a real pain when the server is restarting for
unknown reasons).
(Bitbake rev: 4e780fa19fc720948cf73133f56f7d837ad9283a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Finding the top level build directory is currently hard and relies on
having a complete cooker being setup. Add a helper function which
does the same thing without all the extra overhead. This is needed
to be able to locate the bitbake lockfile and hence the socket
for connecting clients in the new server model.
(Bitbake rev: d196afe68032898c31a8599ca7d3ceba58d96b0a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In preparation for rewriting this code, expand the relatively useless
base classes into the code itself.
(Bitbake rev: a1c6151420d86bac658c08ae714647062edd6ef2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The UI may want to change its event mask however to do this, it needs the
event handler's ID. Tweak the code to allow this to be stored and add
a command to query it.
Use the new command in the process server backend.
(Bitbake rev: f8cf2cb58b80ce74f756a11a9773b6b0e78d51ee)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix one incorrect indentation and several lines with trailing spaces.
(Bitbake rev: 4dbf4f1907a32f303c2e9acf2f27817cf3c9ec38)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If a signal is sent like SIGWINCH the select could be interrupted
so ignore the InterruptError like in XMLRPC server [1].
[1]
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/server/xmlrpc.py#n307
(Bitbake rev: 96bb174325493764718c61c12c943c37b882cd61)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
To quote Paul:
Not that long ago we added a prefix to logged messages to allow us to see
where the message has been generated (recipe / task). This is undoubtedly
useful for errors and warnings, however, I'm not sure it's really appropriate
for bb.plain(). As an example, see the output for -c listtasks now:
...
NOTE: Executing RunQueue Tasks
nodejs-native-4.5.0-r0 do_listtasks: do_addto_recipe_sysroot
nodejs-native-4.5.0-r0 do_listtasks: do_build Default task for a recipe - depends on all other normal tasks required to 'build' a recipe
nodejs-native-4.5.0-r0 do_listtasks: do_checklicense
nodejs-native-4.5.0-r0 do_listtasks: do_checklicenseall
nodejs-native-4.5.0-r0 do_listtasks: do_checkpkg
...
This patch excludes PLAIN messages from this prefixing making the log output
neater.
[YOCTO #11457]
(Bitbake rev: 4a14b44b3e4fad3a3e5e53461aa8ba9929a515b8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When heartbeat event support was added it was only added to process.py. Add
it to server/xmlrpc too. There is duplicated code however since we're likely
to combine the server abstractions soon its not worth worrying about now.
This ensures the backends have the same event support.
[YOCTO #10741]
(Bitbake rev: 00bf2e60222767b4dee84fb3f958732a83544e80)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Messages printed when no UI is connected (e.g. memres) are currently lost.
Use the existing queue mechanism to queue these until a UI attaches, then
replay them. This isn't ideal but better than the current situation of
losing them entirely.
(Bitbake rev: cb241fb8544dfb05646dbae1a1b04e17878a466c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
You might think Queue.Queue.get(True, 0) would return an event immediately
if present and otherwise return. It doesn't, it immediately "times out"
and returns with nothing from the queue.
The behaviour we want is not to wait but return anything present which is
what .get(False) does so map to this.
This fixes some odd behaviour observed in some of the tinfoil selftests.
(Bitbake rev: 412bfab8721ea317898a1974f6a7a0d0bea763df)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The datastore can assume internal API, this just removes the function indirection
overhead involved in this very common codepath (800,000 calls in parsing OE-Core).
(Bitbake rev: 9a36531ed2b2881a65e5d39ee4b68d2bb392ed78)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently builds in multiple TMPDIRs with multiconfig can break
since the BuildStarted event is used to create directory strutures in several
cases (e.g. buildstats.bbclass) and there is only on BuildStarted event
generated in a multiconfig build.
We have two options, a) to add a new MultiConfigBuildStarted event which is
generated once per multiconfig, or b) allow multiple BuildStarted events.
Having reviewed the code and current users of BuildStarted, sending one event
per multiconfig seems like its the best way forward and the existing code looks
able to cope with the duplication of events. I did also check toaster and I think
that can handle this issue too (multiconfig builds may have other issues there).
I'm therefore proposing we send multiple BuildStarted events for multiconfig
and for consistency, send multiple BuildCompleted events too.
We need to ensure that BUILDNAME, BUILDSTART and DATE/TIME are set consistently
in all the different multiconfig datastores. These events can write to the
datastore so copies are not used. buildFile was also cleaned up to ensure it
uses the right datastore in various places.
(Bitbake rev: 0b00f0382780ab5390a5c3f756a9b4efafe0aec8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
