Commit Graph

120 Commits

Author SHA1 Message Date
Richard Purdie
048c1467e2 bitbake: bitbake: Drop duplicate license boilerplace text
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.

(Bitbake rev: 34ed28a412af642a993642c14bd8b95d5ef22cd8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-19 18:04:01 +01:00
Richard Purdie
e1db6a1690 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ac556588fac55e91b7ce4839a975eb9ebb5aa192)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-19 18:04:01 +01:00
Richard Purdie
74feaddda3 bitbake: server/process: Various server startup logging fixes
There were various problems in the server startup loggin:

a) stdout/stderr were not being flushed before forking which could potentially
   duplicate output

b) there were separate buffers for stdout/stderr leading to confusing logs
   where the entries could be reordered. This was particularly confusing
   due to the separator the logs use to idendify new messages

c) an fd wasn't being closed during server startup meaning if the
   server failed to start, the closed fd wasn't detected as it was held
   open by the other reference

d) If the pipe was detected as being closed, the code incorrectly retried
   server startup

e) The event code would remap stdout/stderr without flushing them, leading
   to lose log messages

(Bitbake rev: 0594faa0b52ce5dbd948d836d88617d38d9862d1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-05 18:01:37 +01:00
Kai Kang
5581b374b9 bitbake: event.py: fix typo
(Bitbake rev: d49483eb733ee27325349246fa37a84140c0941d)

Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-08-14 16:32:48 +01:00
Richard Purdie
869ae2ce2a bitbake: events: Drop unused cfg property
The cfg property on events doesn't add much, all code appears to access
"data" at this point. Remove it to clean up the interface.

(Bitbake rev: bd4b9e4460b60f142c3bf346fb04e360e512eaee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-08 08:48:54 +00:00
Richard Purdie
a04ee42393 bitbake: event: Ensure we clean up loggers
Whilst we're likely exiting in this case, clean up the loggers we add
so that in the case of certain server retries there is no possibility
multiple loggers stack up.

(Bitbake rev: 25b7bf6672be66bcbfe5760610dce7d3e866cdcc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-09 12:24:40 +00:00
Richard Purdie
25e52d34d0 bitbake: event: Don't write duplicate logs to stdout and stderr in no UI case
This code would duplicate messages to stdout and stderr when no UI connected
and there were error level messages.

Rework the code so it either uses stderr (for errors and above) or
stdout for warnings/debug but not both for the same messages.

(Bitbake rev: 45cff5734ba2ba8c8d36d17d722a5804d39b258b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:43:36 +01:00
Paul Eggleton
d4c3ace097 bitbake: bitbake-diffsigs: fix regression after recent server changes
We were bridging the gap between the server and UI here by calling a
bb.siggen.find_siginfo, a function defined and set on that module from
the metadata. This worked from the UI side before but since the recent
server changes is no longer accessible. Create a new command so this can
execute on the server side and return the result by way of a new event.

(We're still running compare_sigfiles() on the signature generator but
that isn't quite the same thing and does still work.)

Fixes [YOCTO #11844].

(Bitbake rev: fdcea991baa4f83d9c98d468d7b49c8c388a4a15)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-27 15:14:20 +01:00
Paul Eggleton
21bb330f46 bitbake: lib/bb/event: refactor printing events
We really ought to have just one place where the string representation
of these events is produced. This doesn't take any real control away
from the UI - if an alternative representation is desired, that can
still be made.

(Bitbake rev: cb15db2a799be6d8eab9a2a43a9a573f89229cff)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:12 +01:00
Richard Purdie
dd71707d5a bitbake: event/command: Allow UI to request the UI eventhander ID
The UI may want to change its event mask however to do this, it needs the
event handler's ID. Tweak the code to allow this to be stored and add
a command to query it.

Use the new command in the process server backend.

(Bitbake rev: f8cf2cb58b80ce74f756a11a9773b6b0e78d51ee)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Richard Purdie
35846a3961 bitbake: event: Queue offline events for the UI
Messages printed when no UI is connected (e.g. memres) are currently lost.
Use the existing queue mechanism to queue these until a UI attaches, then
replay them. This isn't ideal but better than the current situation of
losing them entirely.

(Bitbake rev: cb241fb8544dfb05646dbae1a1b04e17878a466c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08 13:29:54 +01:00
Richard Purdie
bca4396106 bitbake: bitbake: Add MultiConfigParsed event
There are some cases where the metadata needs to be aware a multiconfig build
is happening and have access to the multiconfig data stores to merge data into
the common build. This adds such an event allowing access to these datastores.

(Bitbake rev: 160e47f5df90850e64dcb857c81a5039abc9235f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08 13:29:54 +01:00
Jair Gonzalez
d591b00af0 bitbake: event: remove mapping for deleted event handlers
The current remove method for class event handlers does not update the
event mapping after an event handler is deleted from the main dictionary.

This change enhances the remove method by also updating the event mapping
accordingly.

This was detected after creating the bb.tests.event module.

[YOCTO #10368]

(Bitbake rev: 839f83dea1e99d1d182b138e5aea1df1c45a92fc)

Signed-off-by: Jair Gonzalez <jair.de.jesus.gonzalez.plascencia@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-23 14:14:17 +01:00
Paul Eggleton
b367cd91d1 bitbake: event: drop some unused events
These were all used by support code for the Hob UI which has been
removed - nothing will currently fire or respond to these events.

(Bitbake rev: 03ba8db0595723286ad462fa197f862e1efdf0de)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-22 09:23:38 +01:00
Richard Purdie
631229df72 bitbake: event/ast: Add RecipeTaskPreProcess event before task finalisation
There are various pieces of code which need to run after the tasks are
finalised but before bitbake locks in on the task dependencies. This
adds such an event so dependency changes in anonymous python can
be accounted for and acted upon by these specific event handlers.

(Bitbake rev: 4dcd0e53f5ff4bf4f2d6cbdc51ff33a5f5f206af)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:52:52 +00:00
Richard Purdie
840e221ea7 bitbake: event: Replace slightly elderly bbmake references!
bbmake hasn't existed for a while (a decade+) so we should
probably remove the references.

(Bitbake rev: 4170cf874850b950f31d2e36c895c110c0b096c3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09 13:39:13 +00:00
Patrick Ohly
268bf22263 bitbake: monitordisk: add event
The current disk usage is interesting and may be worth logging over
time as part of the build statistics. Instead of re-implementing the
code and the configuration option (BB_DISKMON_DIRS), the information
gathered by monitordisk.py is made available to buildstats.bbclass via
a new event.

This has pros and cons:
- there is already a useful default configuration for "interesting" directories
- no code duplication
- on the other hand, users cannot configure recording separately from
  monitoring (probably not that important)

(Bitbake rev: f065ac17d0031dca6309ddbff18c8792630de865)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:42:22 +00:00
Patrick Ohly
083365143e bitbake: cooker process: fire heartbeat event at regular time intervals
The intended usage is for recording current system statistics from
/proc in buildstats.bbclass during a build and for improving the
BB_DISKMON_DIRS implementation.

All other existing hooks are less suitable because they trigger at
unpredictable rates: too often can be handled by doing rate-limiting
in the event handler, but not often enough (for example, when there is
only one long-running task) cannot because the handler does not get
called at all.

The implementation of the new heartbeat event hooks into the cooker
process event queue. The process already wakes up every 0.1s, which is
often enough for the intentionally coarse 1s delay between
heartbeats. That value was chosen to keep the overhead low while still
being frequent enough for the intended usage.

If necessary, BB_HEARTBEAT_EVENT can be set to a float specifying
the delay in seconds between these heartbeat events.

(Bitbake rev: 7cf22ea057d28c54bd98dc1ab7a43402a29ff1f5)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:42:22 +00:00
Aníbal Limón
924ca1037c bitbake: bb.event: fix infinite loop on print_ui_queue
If bitbake ends before _uiready and bb.event.LogHandler was add
to the bitbake logger it causes an infinite loop when logging
something.

The scenario is print_ui_queue is called at exit and executes
the log handlers [2] one of them is bb.event.LogHandler this handler
appends the same entry to ui_queue causing the inifine loop [3].

In order to fix a new copy of the ui_queue list is created when iterate
ui_queue.

[YOCTO #10399]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=10399#c0
[2] http://git.openembedded.org/bitbake/tree/lib/bb/event.py?id=41d9cd41d40b04746c82b4a940dca47df02514fc#n156
[3]
http://git.openembedded.org/bitbake/tree/lib/bb/event.py?id=41d9cd41d40b04746c82b4a940dca47df02514fc#n164

(Bitbake rev: 46fecca9d531a07788b5cac8b2dc6a8267d8b6d0)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-15 10:01:44 +01:00
Joshua Lock
1110dde73a bitbake: event: prevent unclosed file warning in print_ui_queue
Use logger.addHandler(), rather than assigning an array of Handlers
to the loggers handlers property directly, to avoid a warning from
Python 3 about unclosed files:

$ bitbake
Nothing to do.  Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.
WARNING: /home/joshuagl/Projects/poky/bitbake/lib/bb/event.py:143: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/joshuagl/Projects/poky/build/tmp/log/cooker/qemux86/20161004094928.log' mode='a' encoding='UTF-8'>
  logger.handlers = [stdout]

(Bitbake rev: 1e23b1f1a80066223b98e18b163840051ac74944)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-05 10:25:34 +01:00
Aníbal Limón
79012f943d bitbake: bb/event.py: fire_ui_handlers enable threading lock support
In some cases there is a need to fire bb events into multiple
python threads so locking is needed (writing to a fd/socket).

Adding a helper functions for disable/enable by request to avoid
overhead.

[YOCTO #10330]

(Bitbake rev: a583dc0b296415ec904c081c4de96ceef46732a8)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-05 10:10:11 +01:00
Ed Bartosh
d649c64beb bitbake: event.py: output errors and warnings to stderr
All logging messages are printed on stdout when processing
UI event queue. This makes it impossible to distinguish between
errors and normal bitbake output. Output to stderror or stdout
depending on log level should fix this.

(Bitbake rev: 56ac0d4c7a5f47aeb707b15a0c305d9f73aae945)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-15 12:15:07 +01:00
Elliot Smith
c471740f5b bitbake: cooker: add BuildInit event
In situations where a bitbake run fails before the build
properly starts and BuildStarted is fired, a UI has no way
to get at the targets passed to the build. This makes it
difficult for the UI to report on the targets which failed.

Fire a BuildInit event before running buildTargets() or
buildFile(). This enables a UI to capture targets passed to
buildTargets(), even if the build fails (e.g. the targets
themselves are invalid).

[YOCTO #8440]

(Bitbake rev: ac02fda870965bf7d44ff5688eda54d2d11ab9c7)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-11 00:09:26 +01:00
Paul Eggleton
8d56d596bb bitbake: knotty: add code to support showing progress for sstate object querying
Add support code on the BitBake side to allow sstate.bbclass in
OpenEmbedded to report progress when it is checking for availability of
artifacts from shared state mirrors.

Part of the implementation for [YOCTO #5853].

(Bitbake rev: 070ae856da0715dbaf4c560c837ea796ffc29f00)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Richard Purdie
ff6d458f9a bitbake: parse/ast, event: Ensure we reset registered handlers during parsing
When parsing, we should reset the event handlers we registered when
done. If we don't do this, parse order may change the build, depending
on what the parse handlers do to the metadata.

This issue showed up as a basehash change:

ERROR: Bitbake's cached basehash does not match the one we just generated (
/media/build1/poky/meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb.do_unpack)!

This is due to the eventhandler in nativesdk.bbclass being run, despite
this .bb file not inheriting nativesdk.bbclass. The parse order was
different between the signature generation and the main multithreaded
parse.

Diffsigs showed:

bitbake-diffsigs 1.0-r2.do_unpack.sigbasedata.*
basehash changed from 887d1c25962156cae859c1542e69a8d7 to cb84fcfafe15fc92fb7ab8c6d97014ca
Variable PN value changed from 'nativesdk-buildtools-perl-dummy' to '${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}'

with PN being set by the event handler.

(Bitbake rev: 0219271d4130c1f4cf071c7577a4101c54c04921)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-15 08:35:07 +01:00
Richard Purdie
0f2c59367a bitbake: bitbake: Convert to python 3
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.

(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00
Richard Purdie
a46d485856 bitbake: event/msg: Pass formatted exceptions
python3 can't cope with the previous approach we were using to pass
exceptions through the RPC. Avoid this by creating a formatted exception
on the sender side.

(Bitbake rev: d7db75020ed727677afbad07a90fb3eac0bf2c45)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-13 13:41:32 +01:00
Christopher Larson
cd20dd057b bitbake: bb.event: handle __builtins__ as a module
Fixes pypy support.

(Bitbake rev: a3e5d9337f5400aab13df63f261e750178f8a661)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:18:12 +01:00
Richard Purdie
0381b78aa4 bitbake: event/utils/methodpool: Add a cache of compiled code objects
With the addition of function line number handling, the overhead of
the compile functions is no longer negligible. We tend to compile
the same pieces of code over and over again so wrapping a cache around
this is beneficial and removes the overhead of line numbered functions.

Life cycle of a cache using a global like this is in theory problematic
although in reality unlikely to be an issue. It can be dealt with
if/as/when we deal with the other global caches.

(Bitbake rev: 98d7002d1dca4b62042e1589fd5b9b3805d57f7a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-22 00:02:05 +00:00
Richard Purdie
0019edc818 bitbake: ast/event/utils: Improve tracebacks to include file and line numbers more correctly
Currently bitbake tracebacks can have places where the line numbers are
inaccurate and filenames may be missing. These changes start to try and
correct this.

The only way I could find to correct line numbers was to compile as a
python ast, tweak the line numbers then compile to bytecode. I'm open
to better ways of doing this if anyone knows of any.

This does mean passing a few more parameters into functions, and putting
more data into the data store about functions (i.e. their filenames
and line numbers) but the improvement in debugging is more than worthwhile).

Before:
----------------
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
  File "run_buildstats(e)", line 43, in run_buildstats(e=<bb.build.TaskStarted object at 0x7f7b7c57a590>)
NameError: global name 'notexist' is not defined

ERROR: Build of do_patch failed
ERROR: Traceback (most recent call last):
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 560, in exec_task
    return _exec_task(fn, task, d, quieterr)
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 497, in _exec_task
    event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 170, in fire
    fire_class_handlers(event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 109, in fire_class_handlers
    execute_handler(name, handler, event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 81, in execute_handler
    ret = handler(event)
  File "run_buildstats(e)", line 43, in run_buildstats
NameError: global name 'notexist' is not defined

----------------

After:
----------------
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
  File "/media/build1/poky/meta/classes/buildstats.bbclass", line 143, in run_buildstats(e=<bb.build.TaskStarted object at 0x7efe89284e10>):
         if isinstance(e, bb.build.TaskStarted):
    >        trigger = notexist
             pn = d.getVar("PN", True)
NameError: global name 'notexist' is not defined

ERROR: Build of do_package failed
ERROR: Traceback (most recent call last):
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 560, in exec_task
    return _exec_task(fn, task, d, quieterr)
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 497, in _exec_task
    event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 170, in fire
    fire_class_handlers(event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 109, in fire_class_handlers
    execute_handler(name, handler, event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 81, in execute_handler
    ret = handler(event)
  File "/media/build1/poky/meta/classes/buildstats.bbclass", line 143, in run_buildstats
    trigger = notexist
NameError: global name 'notexist' is not defined
----------------

(Bitbake rev: 1ff860960919ff6f8097138bc68de85bcb5f88b0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Richard Purdie
5de3800c30 bitbake: event: Fix subprocess event error traceback failures
If subprocess raises a CalledProcessError() error, e.g. from a call
like subprocess.check_call("false"), bitbake would try and pass the
object over IPC and fail, leading to an unusual error:

('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())%

To avoid this, we turn the value into a string which prevents the
issues the IPC has trying to deal with the object (for the same reason
we deal with tracebacks here too).

[YOCTO #8752]

(Bitbake rev: 05695424b918fc81b16cbac70d79d8271a0b6045)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Richard Purdie
4b9de767f0 bitbake: event/server: Add _uiready flag to handle missing error messages
If you start and suspend a bitbake execution so the bitbake lock is held,
then try and run "bitbake -w '' X", you will see bitbake return an error exit
code but print no message about what happened at all.

The reason is that the -w option creates a "UI" which swallows the messages. The
code which handles this exit failure mode thinks a UI has printed the messages
and therefore doesn't do so.

This adds in an extra parameter to the UI registration code so that we
can figure out whether its a primary UI or not and base decisions on whether
to display information on that instead. This fixes the error shown above and
some bizarre failures on the Yocto Project Autobuilder.

[YOCTO #8239]

(Bitbake rev: d1d60a68c2de40c2984d5040d14251c1be121b0b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-04 16:23:56 +01:00
Paul Eggleton
294948c732 bitbake: lib/bb/cooker: add interrupted flag to BuildCompleted event
Allow any listeners for this event (such as buildhistory.bbclass in
OpenEmbedded) to find out if the build was interrupted rather than
completing normally. The value will be 0 if not interrupted, 1 if
interrupted waiting for remaining tasks to complete, or 2 if force
interrupted (stopping any running tasks immediately).

(Bitbake rev: df2b778efd2ecc48f6c5a3ed446f6459f2250035)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-29 14:03:49 +01:00
Paul Eggleton
715d857174 bitbake: Fix default function parameter assignment to a list
With python you should not assign a list as the default value of a
function parameter - because a list is mutable, the result will be that
the first time a value is passed it will actually modify the default.
Reference:

http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments

(Bitbake rev: 7859f7388f2e3f675d0e1527cfde18625f36f637)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:44 +01:00
Richard Purdie
8c3dd2d9eb bitbake: event: Handle recursive events and the data store better
Events can call each other recursively, e.g. an event handler can call
bb.note which in turn generates another event. If these loop, it
can lead to multiple deletions of 'd' from __builtins__ which
can fail since __builtins__ is global scope.

Add handling to only remove 'd' when we added it and it wasn't already
present.

(Bitbake rev: b45952650ce8f470f124df36185b79e0d3a1783a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-26 09:27:34 +01:00
Richard Purdie
df123ddab4 bitbake: event: Inject 'd' into event handlers
To quote Chris Larson:

"""
e.data.getVar() gets a bit old in a large event handler, and it means a
simple handler has to be modified if switching between an event handler
(e.g. RecipeParsed) and anonymous python. I think it would make sense
to restore the 'd' convention here to align with python elsewhere.
It'd just be a convenience, d==e.data, to avoid the common pattern of
setting it at the top of the event handler.
"""

I couldn't find a way to inject 'd' via locals/globals due to the use
of a function parameter so this left __builtins__ as the only way
I could find to make this work.

[YOCTO #7668]

(Bitbake rev: 44ac81e5281fb62ad00e2f79a9d754118ea62526)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-23 11:57:57 +01:00
Richard Purdie
4bedfeffa2 bitbake: cooker/event: Add an event which lists all stamps reachable after parsing
The metadata can potentially use such an event to clean up any
"unreachable" data, solving several problems we currently have
where obsolete data may continue to exist in the shared areas.

(Bitbake rev: c5e6f929f3d5eeb7954660dea62611c58b795ff8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-09 13:30:09 +01:00
Paul Eggleton
9c02ad9fc9 bitbake: event: fix resetting class handlers object
If you don't explicitly specify to use a global variable when doing an
assignment, you will be setting a local variable instead, which means
this function wasn't working at all. It explains some odd behaviour we
have seen in the layer index where event handlers were sometimes
bleeding into other contexts where they should not have been.

(Bitbake rev: ac45ea848901b0f6cd23087b662dde8ce9cd807e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-12-09 22:15:49 +00:00
Paul Eggleton
c1ef10afb6 bitbake: event: add a means of filtering events internally
When using external tinfoil-based utilities, it is useful to be able to
turn off most of the event handlers; for example sstate_eventhandler
doesn't like being sent events for any recipe which has been skipped.

(Bitbake rev: 41236c28985a3b66b3737382a94e39dbf6480160)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-12-09 22:15:48 +00:00
Robert P. J. Day
6d08e5bb09 bitbake: lib/bb/*.py: Typo fixes/grammar/comment fixes, nothing functional.
(Bitbake rev: 587b144ee409d444494d8d7f2d1c53ede8f7c953)

Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-08-25 08:57:45 +01:00
Alexandru DAMIAN
f4456df240 bitbake: toasterui: fixing event.data clash
This patch fixes a name collision on the event.data
in the MetadataEvent class. event.data is used in the
event system as a copy of "d" in certain situations,
and this collision triggered a bug leading to data loss.

[YOCTO #6332]

(Bitbake rev: 3f191b7cfe95aea4d4e96babf001d62d45dd3aaa)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-07-23 20:07:48 +01:00
Richard Purdie
91a164cb87 bitbake: event: Add SkipRecipe event to replace SkipPackage
In the depths of time we were rather confused about naming. bb files
are recipes, the event to skip parsing them should be SkipRecipe,
not SkipPackage. This changes bitbake to use the better name but
leaves the other around for now. We can therefore start removing
references to it from the metadata.

(Bitbake rev: 98d9e6e0f514a7cb7da1d99bf4bd5602b89426d6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-06-01 14:29:31 +01:00
Richard Purdie
044296ce4c bitbake: event.py: Allow passthrough of BBHandledException events
We need BBHandledException events to be passed through to the higher
layers, they don't need addition of any traceback since they've already
been reported to the user.

(Bitbake rev: f63b61f8997862439519f474cc09f3e05e95288c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-06-01 14:29:31 +01:00
Richard Purdie
a26667054b bitbake: event: Fix event handlers to raise SkipPackage
If an event handler triggers a SkipPackage event, we really want that
event to be received and processed by the higher code levels. Currently
it was getting caught and ignored which was leading to recipes
being present when they clearly shouldn't have been.

In general this exception catching looks to be doing the wrong thing. It
was introduced in
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/bitbake/lib/bb/event.py?id=37cb4cc02b2e2b6c338c5943747e0a1ef15176b3
but we likely want exceptions to pass through to the higher layers.

This patch therefore removes that code.

(Bitbake rev: 79211b3434855942f9fad4a1db69ce7be911327c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-06-01 14:29:31 +01:00
Richard Purdie
8c1e43ca58 bitbake: cooker/event: Overhaul sanity test mechanism
Sanity tests are currently a pain as its hard to control when they run. This results
in issues where for example the bitbake -e output is not useful as the sanity tests
prevent it from executing. The sanity tests should run later than the base configuration.

This patch changes the sanity tests to always be event triggered with the option of
returning either events on the status, or raising errors. A new cooker feature is used
to change the behaviour depending on the controlling UI.

This does need a change to sanity.bbclass in the OE metadata but its worth the pain
for the increased flexibility and control this offers UIs and the improvement to the
user experience.

(Bitbake rev: 32e171bcc92c6e27fefee971e8714ddf8e1a8ac1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-03-27 09:42:08 +00:00
Alexandru DAMIAN
24ae636fde bitbake: bitbake: event: adding generic event for metadata usage
Adding the generic bb.event.MetadataEvent that is
targeted specifically at metadata usage. This is
needed in order to let the metadata code send and receive
events during asynchrous execution without having
to define each event specifically in Bitbake.

Metadata code should subscribe to and fire the MetadataEvent
in order to communicate asynchronously, and identify
the object using event.type field, and parse the
data in the event.data field.

Knotty UI will ignore these event by default.

This deprecates RequestPackageInfo/PackageInfo, and that
event pair will be removed in the future.

(Bitbake rev: ae1ea51aaab73e010d1c3db39df058bebebc11dd)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-22 12:19:44 +01:00
Cristiana Voicu
8d134e999e bitbake: bitbake/event.py: UIhandler filter should work without a mask
The default for the mask will be * (all the handlers)

(Bitbake rev: 4c95e5f46cf2a656100bbf5a0e5a09d506abf9b9)

Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-27 11:16:58 +01:00
Richard Purdie
69aab78dd8 bitbake: bitbake: Add ui event handlers filtering
Add functionality to allow UIs to update and change the types of events they
recieve. To do this we need to add a new command and also need to be able
to obtain the current event hander ID. In the case of xmlrpc, this is
straightforward, in the case of the process server we need to save the result
in a multiprocessing.Value() so we can retrive it. An excplit command
was added to the server API to facilitate this.

The same function can also be used to mask or unmask specific log messages,
allowing the UI to optionally differ from the standard set of message
filtering.

Based upon work by Cristiana Voicu <cristiana.voicu@intel.com>

(Bitbake rev: ba5a6c88785d9889d4172ec79937ac2a5555327e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-26 11:29:46 +01:00
Richard Purdie
25e410b743 bitbake: event/msg: Add primitive server side UI log record filtering
Currently one of the bigger bottlenecks in bitbake is passing all the
log messages over IPC to the UI. This is worthwhile if the UI is going
to use them, pointless otherwise. The memory resident bitbake suffers
from this performance issue particularly badly.

This patch filters the log events on the server side with the global
log levels and hence reduces the traffic. This speeds up parsing
(18.5s down to 17s) and bitbake general command overhead is reduced
(7.3s for a NOP to 6.2s).

What isn't added here is general event filtering or the ability to
change the log levels once set. Provision is made for adding this
in a follow up patch though.

(Bitbake rev: 1bf0e88f57ba0bca62532e81d0d62cf88e2abcbb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-23 17:29:54 +01:00
Paul Eggleton
840a427ed2 bitbake: taskdata: report close matches with NoProvider errors
Assuming there is no known reason why an item is not provided, show
close matches on the assumption that it might have been a typo or
other mistake.

(Bitbake rev: ed81b0856b4a3892b53d39871eaaa6273390ea75)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-16 11:21:02 +01:00