Commit Graph

158 Commits

Author SHA1 Message Date
Joshua Watt
b2d55ec1eb bitbake: knotty: Update hash equivalence logging
Updates hash equivalence logging so that the interesting VERBOSE
messages are always logged to the consolelog file so that issues in
individual user builds can be diagnosed. The autobuilder logging config
then updates this so that they also are shown on stdout, since the
consolelog file is not capture there.

In order to facilitate this, 2 new logging handlers were added,
"BitBake.verbconsole" and "BitBake.verbconsolelog". Neither of these
handlers are attached to anything by default, but they will log any
messages that wouldn't otherwise be logged by the normal console or
consolelog handlers. Users can attach whatever loggers the desire to
this handler to get them to appear on the console or in the consolelog,
as demonstrated by the autobuilderlog.json file.

(Bitbake rev: 766587c0baaaeb5cda3e9454395edbb70e33f756)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
c8b2694cd4 bitbake: knotty: Add logging cleanup
Adds code to close all loggers when bitbake exits. This prevents
unclosed file ResourceWarnings. A form of this closing existed
previously, but was removed in the new logging code.

(Bitbake rev: b3f3779adf63c0d970462a558a6205da1d30c0ed)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
1c9874b9f3 bitbake: knotty: Setup logs with config helper
Sets up logging in knotty to use python's structured logging config and
the bb.msg.setLoggingConfig() helper to setup logging. This allows the
user to specify additional logging mechanism in a config file with
BB_LOGCONFIG

(Bitbake rev: 646a68a49364b50a42168b4b16308f7217eec0dc)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
300fd2a659 bitbake: knotty: Remove dependency on format variable
Passing around the log formatter variable was unnecessary since the log
levels of interest can be accesses as class members of
bb.msg.BBLogFormatter. Switching to do this will make using the
structured python logging much easier, since it can be difficult to
extract out the formatter for a specific handler.

(Bitbake rev: c1c867df24b4ef204027d485acac7c75c63f2bc0)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
ae6e0e0a51 bitbake: knotty: Add commented logging_tree code
Adds a comment section that can be easily uncommented to enable dumping
the logging tree. This module is extremely useful for debugging issued
with logging configuration

(Bitbake rev: 30461310915f911b80f92e03df694af7c1eb1f46)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Joshua Watt
77fbe09245 bitbake: knotty: Handle logging messages with specific logger
Handles the log messages from the bitbake server with the specific
logger that the event originated from. This allows hierarchical logging
configurations to work as expected.

(Bitbake rev: 9624d42133e024fd044d0d089c7017ed53eed874)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-03-13 11:29:23 +00:00
Peter Kjellerstedt
f8d1ac6653 bitbake: knotty: Make the bb.command.CommandExit event terminate bitbake
This matches the other bb.command.Command* events and without it,
running `bitbake --revisions-changed` will hang indefinitely if there
are changed revisions.

(Bitbake rev: 40520d229c8ea51ee9784184ab5d13a82dd1eb61)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-02-19 11:26:12 +00:00
Frazer Clews
fa5524890e bitbake: lib: amend code to use proper singleton comparisons where possible
amend the code to handle singleton comparisons properly so it only checks
if they only refer to the same object or not, and not bother
comparing the values.

(Bitbake rev: b809a6812aa15a8a9af97bc382cc4b19571e6bfc)

Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19 13:31:05 +00:00
Frazer Clews
0ac5174c7d bitbake: lib: remove unused imports
removed unused imports which made the code harder to read, and slightly
but less efficient

(Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697)

Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19 13:31:05 +00:00
Peter Kjellerstedt
fbe8b3e3e9 bitbake: knotty: Be consistent when creating/updating progress bars
When creating a new progress bar (using BBProgress), a colon was
appended to the supplied message. However, when updating the message,
no colon was appended.

Change this so that the colon is instead part of the widgets that make
up the progress bar so that it does not matter when and how the
message is updated, it always displays the same.

(Bitbake rev: 08f35c04f6e1ce4c4ca5c2bef4cd8a192e12e682)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-13 13:12:23 +00:00
Peter Kjellerstedt
5c935fb29c bitbake: knotty: Hide the footer if a process progress bar is shown
With the introduction of the hash equivalence server, the progress bar
for "Checking sstate mirror object availability" is shown repeatedly
while the tasks are being executed. If the footer is not hidden then,
it will be moved up one line every time, creating a messy interface.

(Bitbake rev: 56b5ec4c2b3e658e73ca6c3a12feeb96df0977fb)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-13 13:12:23 +00:00
Richard Purdie
ccd7f12b1c bitbake: knotty/uihelper: Switch from pids to tids for Task event management
We've seen cases where a task can execute with a given pid, complete
and a new task can start using the same pid before the UI handler has
had time to adapt.

Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/knotty.py", line 484, in main
    helper.eventHandler(event)
  File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/uihelper.py", line 30, in eventHandler
    del self.running_tasks[event.pid]
KeyError: 13490

This means using pids to match up events on the UI side is a bad
idea. Change the code to use task ids instead. There is a small
amount of fuzzy matching for the progress information since there
is no task information there and we don't want the overhead of a task
ID in every event, however since pid reuse is unlikely, we can live
with a progress bar not quite working properly in a corner case like
this.

[YOCTO #13667]

(Bitbake rev: e427eafa1bb04008d12100ccc5c862122bba53e0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-06 14:39:29 +00:00
Robert Yang
e1f589f044 bitbake: knotty: Fix for the Second Keyboard Interrupt
Fixed:
$ rm -fr tmp-glibc/cache/default-glibc/qemux86/x86_64/bb_cache.dat* ; bitbake -p
Press the first Ctrl-C when the parsing process is at about 50%:

Keyboard Interrupt, closing down...

Then presss the second Ctrl-C:

  File "/path/to/bitbake/bitbake/lib/bb/ui/knotty.py", line 619, in main
    event = eventHandler.waitEvent(0.25)
  File "/path/to/bitbake/lib/bb/server/process.py", line 591, in waitEvent
    self.eventQueueNotify.wait(delay)
  File "/usr/lib/python3.5/threading.py", line 549, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/lib/python3.5/threading.py", line 297, in wait
    gotit = waiter.acquire(True, timeout)
KeyboardInterrupt

Capture the second KeyboardInterrupt during stateShutdown is running can fix
the problem. There may be still tracebacks for the third KeyboardInterrupt, but
I'm leaning to not fix it since we aimed for supporting 2 KeyboardInterrupts
only.

(Bitbake rev: 8c26b451f22193ef1c544e2017cc84515566c1b8)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-08 10:22:36 +01:00
Richard Purdie
fb8fc5e78c bitbake: event/runqueue: Drop StampUpdate event, its pointless/unused
Whilst this class has existed for years, it doesn't have any
users and has a questionable interface. Drop it to allow for further
simplification and changes.

(Bitbake rev: 3ab51764f7965d696bb2c5a872bf161473df4289)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-15 10:28:12 +01:00
Chris Laplante
8c15a93553 bitbake: knotty: allow progress rate for indeterminate bars
(Bitbake rev: 85f0b443b7ab1848abc6eb658be489fc1718004c)

Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-11 13:27:19 +01:00
Richard Purdie
cf9c0be3f6 bitbake: bitbake: Drop duplicate license boilerplace text
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.

(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
79834a7144 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:04 +01:00
Richard Purdie
fcec90de9c bitbake: knotty: Implement console 'keepalive' output
CI systems like jenkins and buildbot will timeout applications which haven't had console output
in some period of time. Add 'keepalive' output to knotty which gives output every 5000s if not
other output was made and tasks are still running. This reduces some problems encountered
with our CI testing.

(Bitbake rev: aa4f31e5741dd98acec73f16f6028e52f4c22d6f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-01 23:09:08 +01:00
Jacob Kroon
be75acf17f bitbake: knotty: Pretty print task elapsed time
A task's runtime is currently printed in seconds. Change it to
include minutes and hours for easier reading.

(Bitbake rev: c593ae5ec9fecd4bde823948024e4d56314a60ce)

Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-30 12:05:24 +01:00
Paul Eggleton
a3971620dc bitbake: tinfoil: ensure log lines get printed when tasks fail
If a task fails during build_targets(), we need to print out the log
lines as knotty does or the user will be missing information about the
failure.

(This should get some deeper refactoring, but now isn't the time for
that.)

(Bitbake rev: 24879df071d4803db3d39ae1d5cad852daa92f28)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:03 +01:00
Richard Purdie
260a22b241 bitbake: ui/knotty: Send updateConfig early
If for example you run:

bitbake -r somefile.inc
rm somefile.inc
bitbake -e

bitbake will crash with an error about not being able to find somefile.inc. This
is because it tries to reparse the base config for the early getVariable requests
before it sees the updated missing -r option.

Send the updateConfig command earlier to avoid this.

(Bitbake rev: a38164620ebdc770690c5f39ff9ed69d3f82719e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-31 23:30:03 +01:00
Paul Eggleton
e1285712fa bitbake: knotty: make it possible to use termfilter without either console
This isn't useful for knotty itself, but for use from tinfoil
in case we can't get access to either the console or errconsole, allow
either to be unspecified (None).

(Bitbake rev: 7544de437fc66b81502ecdb5db859182c45827cb)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:12 +01:00
Paul Eggleton
21bb330f46 bitbake: lib/bb/event: refactor printing events
We really ought to have just one place where the string representation
of these events is produced. This doesn't take any real control away
from the UI - if an alternative representation is desired, that can
still be made.

(Bitbake rev: cb15db2a799be6d8eab9a2a43a9a573f89229cff)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:12 +01:00
Paul Eggleton
c361cac161 bitbake: knotty: fix indentation
(Bitbake rev: 06a96d267dec5b542725defb8f01403f505c4f44)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-18 22:39:33 +01:00
Richard Purdie
f88652d284 bitbake: knotty: Drop task prefix of PLAIN log messages
To quote Paul:

Not that long ago we added a prefix to logged messages to allow us to see
where the message has been generated (recipe / task). This is undoubtedly
useful for errors and warnings, however, I'm not sure it's really appropriate
for bb.plain(). As an example, see the output for -c listtasks now:

...
NOTE: Executing RunQueue Tasks
nodejs-native-4.5.0-r0 do_listtasks: do_addto_recipe_sysroot
nodejs-native-4.5.0-r0 do_listtasks: do_build                      Default task for a recipe - depends on all other normal tasks required to 'build' a recipe
nodejs-native-4.5.0-r0 do_listtasks: do_checklicense
nodejs-native-4.5.0-r0 do_listtasks: do_checklicenseall
nodejs-native-4.5.0-r0 do_listtasks: do_checkpkg
...

This patch excludes PLAIN messages from this prefixing making the log output
neater.

[YOCTO #11457]

(Bitbake rev: 4a14b44b3e4fad3a3e5e53461aa8ba9929a515b8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08 13:29:54 +01:00
Richard Purdie
bca4396106 bitbake: bitbake: Add MultiConfigParsed event
There are some cases where the metadata needs to be aware a multiconfig build
is happening and have access to the multiconfig data stores to merge data into
the common build. This adds such an event allowing access to these datastores.

(Bitbake rev: 160e47f5df90850e64dcb857c81a5039abc9235f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-08 13:29:54 +01:00
Paul Eggleton
55c4781cde bitbake: knotty: prevent extra logger from being enabled for tinfoil
tinfoil sets up its own logger by default, but if and when we initialise
the UI (by default knotty) will also set one up, leading to duplicated
messages specifically from tasks. To avoid this, rather than adding some
kind of parameter, just check if there is already a logger outputting to
stdout/stderr and if so, skip adding our own.

Part of the fix for [YOCTO #11275].

(Bitbake rev: 66d866745f35468d1540a793d07e3a401298b84b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:07:39 +01:00
Peter Kjellerstedt
1e275980f2 bitbake: knotty: Remove extra whitespace after task progress bars
For some reason, BBProgress.setextra() always adds a space at the end.
This prevents the task progress bars from filling the entire width of
the terminal, making them stop one character short. This looks odd
when shown together with the main progress bar, which does fill the
entire terminal width.

(Bitbake rev: 6540e98090d6a88607489a23c063be338bcc3b57)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-13 09:43:40 +00:00
Paul Eggleton
7ebca83926 bitbake: knotty: fix --observe-only option
If we're in observe-only mode then we cannot run commands that would
affect the server's state, including getSetVariable, so prevent that
from being called in observe-only mode.

(Bitbake rev: 2c5a8661430edebff67ab4a108995033d182b5d6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:25:06 +00:00
Paul Eggleton
568409faa1 bitbake: knotty: make quiet option a level option
Allow you to specify -q / --quiet more than once to reduce the messages
even further. It will now operate as follows:

 Level  Option  Result
 -----  ------  ----------------------------------------
 0              Print usual output
 1      -q      Only show progress and warnings or above
 2      -qq     Only show warnings or above
 3+     -qqq    Only show errors

(Bitbake rev: 6cf2582e17c28ca04f5cfb59858c4a9778c700d4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:25:06 +00:00
Patrick Ohly
083365143e bitbake: cooker process: fire heartbeat event at regular time intervals
The intended usage is for recording current system statistics from
/proc in buildstats.bbclass during a build and for improving the
BB_DISKMON_DIRS implementation.

All other existing hooks are less suitable because they trigger at
unpredictable rates: too often can be handled by doing rate-limiting
in the event handler, but not often enough (for example, when there is
only one long-running task) cannot because the handler does not get
called at all.

The implementation of the new heartbeat event hooks into the cooker
process event queue. The process already wakes up every 0.1s, which is
often enough for the intentionally coarse 1s delay between
heartbeats. That value was chosen to keep the overhead low while still
being frequent enough for the intended usage.

If necessary, BB_HEARTBEAT_EVENT can be set to a float specifying
the delay in seconds between these heartbeat events.

(Bitbake rev: 7cf22ea057d28c54bd98dc1ab7a43402a29ff1f5)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:42:22 +00:00
Aníbal Limón
31aafe6852 bitbake: ui/knotty.py: Fix signal handling of SIGWINCH in BBProgress
Add the ability to pass default signal handler for SIGWINCH in BBProgress
because with multiple instace of BBProgress the original signal handler
set by TerminalFilter (sigwinch_handle) is lost.

This is a fix for stack trace due to multiple async calls of ProgressBar
_handle_resize (ioctl to terminal fd), see:

NOTE: Executing SetScene Tasks
Fatal Python error: Cannot recover from stack overflow.

Current thread 0x00007f70a4793700 (most recent call first):
  File
"/home/alimonb/repos/poky/bitbake/lib/progressbar/progressbar.py", line
183 in _handle_resize
  File "/home/alimonb/repos/poky/bitbake/lib/bb/ui/knotty.py", line 58
in _handle_resize
  File "/home/alimonb/repos/poky/bitbake/lib/bb/ui/knotty.py", line 60
in _handle_resize
...
  File "/home/alimonb/repos/poky/bitbake/lib/bb/ui/knotty.py", line 60
in _handle_resize
...
Aborted

(Bitbake rev: 812bd49cb569379ee90d5be28a4b6e60645f1e54)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-09 12:33:26 +01:00
Paul Eggleton
9173b11f2f bitbake: knotty: ensure progress bar output is accounted for in display
When calculating how many lines we'd printed we weren't properly taking
the progress bars into account, with the result that sometimes if the
last line printed on the terminal wrapped to the next line (which is
possible) we backed up less lines than we should have.

Additionally, we should always print a newline after updating the
progress bar - there's no need to check if there wasn't output (there
always will be courtesy of our overridden _need_update()) and we now
allow the line to wrap so we don't need to check the other condition
either.

Hopefully this will fix [YOCTO #10046].

(Bitbake rev: 326d18d96faf02675ba34ad3c3a20cd424b39b91)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-23 15:01:56 +01:00
Richard Purdie
0d76de85bd bitbake: knotty: Show task elapsed time
Its often useful to know how long a task has been running for. This patch
adds that information to the task display, updating every 5s if there
were no other updates so the user can see how long tasks have been running
for.

[YOCTO #9737]

(Bitbake rev: 6c42025e5dd7761213be3f82f3252a7892d2239d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-23 15:01:56 +01:00
Martin Jansa
9d0962491c bitbake: toasterui, knotty: don't print taskid followed by taskstring which are now in most cases identical
* unify the format how the task is described
* don't show taskid followed by taskstring as the taskstring is
  different only for setscene tasks (by _setscene suffix)
* the duplicated output was introduced by:
  2c88afb   taskdata/runqueue: Rewrite without use of ID indirection
  as reported and confirmed as a bug here:
  http://lists.openembedded.org/pipermail/openembedded-core/2016-June/123148.html
* show:
  NOTE: Running task 541 of 548 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_package)
  instead of much longer:
  NOTE: Running task 541 of 548 (ID: /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_package, /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_package)

  and similarly for failed tasks:
  ERROR: Task (virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_install) failed with exit code '1'
  instead of much longer:
  ERROR: Task virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_install (virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_install) failed with exit code '1'

(Bitbake rev: 696693d45f5eff1226866ed79dbfb67161d8cd3f)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-20 16:08:59 +01:00
Paul Eggleton
8a45291164 bitbake: knotty: don't show number of running tasks in quiet mode
There's not a whole lot of point showing how many tasks are running when
we're in quiet mode, it just looks a bit strange particularly when it's
not running any tasks.

(Bitbake rev: 5317200d9cd73c6f971bc1b0cfe8692749e27e3a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:22:58 +01:00
Paul Eggleton
ea0800049d bitbake: knotty: fix task progress bar not starting at 0%
If we have the task number here we need to subtract 1 to get the number
of tasks completed.

(Bitbake rev: 7c78a1cd3f0638ae76f7c7a469b7f667c7c58090)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-17 10:22:58 +01:00
Paul Eggleton
5f4559b2eb bitbake: knotty: don't display ETA for tasks with progress
It turns out that progress information we can extract from a task is
rarely apportioned closely enough to the time taken for the ETA to be
accurate, so showing it is going to be misleading most of the time for
anything but the most basic of examples. Let's just remove it and avoid
misleading (or worse, annoying) the user.

Fixes [YOCTO #9986].

(Bitbake rev: 235db4870b11db97250979e647b54cdb5ce4fbb6)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Paul Eggleton
23a551bea1 bitbake: knotty: fix some minor bugs in BBProgress
If you specify custom widgets then we don't want to assume where the
"extra" position is - you should have to specify it, and if it isn't
specified it shouldn't just wipe out the last widget or you can start to
see odd behaviour if you're modifying the code.

(Bitbake rev: 19e33c10feb1637589ceb05b5e8d58b1e012ccb8)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26 08:10:35 +01:00
Richard Purdie
1b5688e0f0 bitbake: knotty: Handle process indicators more gracefully
Mistakes can happen with the generation of the progress events, change
knotty to be more tolerant of this rather than crashing, reporting to the
user when something unexpected happens. I haven't debugged why multiple
finish events appear to be triggered.

(Bitbake rev: 7dd06b1016b36420a9c55a45ff29dd64ae1dbcda)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Paul Eggleton
76feac37ce bitbake: knotty: add quiet output mode
Quiet output mode disables printing most messages (below warnings) to
the console; however these messages still go to the console log file.
This is primarily for cases where bitbake is being launched
interactively from some other process, but where full console output is
not needed.

Because of the need to keep logging all normal events to the console
log, this functionality was implemented within the knotty UI rather
than in bb.msg (where verbose mode is implemented). We don't currently
have a means of registering command line options from the UI end, thus
the option actually has to be registered in main.py regardless of the
UI, however I didn't feel like it was worth setting up such a mechanism
just for this option.

(Bitbake rev: db95cdef08e339dec7462bfde3ad7d75c1c60dd8)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Paul Eggleton
465f93968f bitbake: knotty: show task progress bar
In addition to the "currently running n tasks (x of y)" message, show a
progress bar for another view on how much of the build is left. We have
to take care to reset it when moving from the scenequeue to the
runqueue, and explicitly don't include an ETA since not all tasks take
equal time and thus it isn't possible to estimate the time remaining
with the information available.

(Bitbake rev: de682015a3fefeff36ddc4197641a700f3fb558d)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Paul Eggleton
8d56d596bb bitbake: knotty: add code to support showing progress for sstate object querying
Add support code on the BitBake side to allow sstate.bbclass in
OpenEmbedded to report progress when it is checking for availability of
artifacts from shared state mirrors.

Part of the implementation for [YOCTO #5853].

(Bitbake rev: 070ae856da0715dbaf4c560c837ea796ffc29f00)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Paul Eggleton
ac5e720575 bitbake: lib: implement basic task progress support
For long-running tasks where we have some output from the task that
gives us some idea of the progress of the task (such as a percentage
complete), provide the means to scrape the output for that progress
information and show it to the user in the default knotty terminal
output in the form of a progress bar. This is implemented using a new
TaskProgress event as well as some code we can insert to do output
scanning/filtering.

Any task can fire TaskProgress events; however, if you have a shell task
whose output you wish to scan for progress information, you just need to
set the "progress" varflag on the task. This can be set to:
 * "percent" to just look for a number followed by a % sign
 * "percent:<regex>" to specify your own regex matching a percentage
   value (must have a single group which matches the percentage number)
 * "outof:<regex>" to look for the specified regex matching x out of y
   items completed (must have two groups - first group needs to be x,
   second y).
We can potentially extend this in future but this should be a good
start.

Part of the implementation for [YOCTO #5383].

(Bitbake rev: 0d275fc5b6531957a6189069b04074065bb718a0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Paul Eggleton
1cf6e14a6c bitbake: knotty: import latest python-progressbar
Since we're going to make some minor extensions to it, it makes sense to
bring in the latest version of python-progressbar. Its structure has
changed a little but the API hasn't; however we do need to ensure our
overridden _needs_update() function's signature in BBProgress() matches
properly.

(Bitbake rev: c3e51d71b36cbc9e9ed1b35fb93d0978e24bc98a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Paul Eggleton
481048cd2a bitbake: knotty: provide a symlink to the latest console log
If you're looking to find the latest console log repeatedly it can be a bit
tedious - let's just create a symlink just as we do with other logs to
make it easy to find.

(Bitbake rev: e9f41c0507a6527bf2ed86506813d4d4a89f8ebf)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Richard Purdie
0f2c59367a bitbake: bitbake: Convert to python 3
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.

(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00
Richard Purdie
07eebc6689 bitbake: knotty: Fix output buffering issues
We need to flush the footer removal, else it may not be outputted until
the buffer is flushed as part of StreamHandler and this would lead to
it removing the ERROR output just printed which is extremely confusing.

Also ensure the footer is cleared before printing a summary as in
some cases it wasn't being removed, also leading to user confusion.

(Bitbake rev: 0e030c4d074c41859608dab5f3ad26b05f56b306)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-17 14:43:31 +01:00
Richard Purdie
7d06ec193e bitbake: knotty: Ensure consolelog file handle is closed
If we don't close the console log file handle, python prints a warning
about unclosed file handles upon exit which is annoying.

(Bitbake rev: 624dd92952b2fc736fd86abe5f2390b87b3a7dd3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-13 13:41:32 +01:00
Richard Purdie
654eadfa30 bitbake: bitbake: Update logger.warn() -> logger.warning()
python deprecated logger.warn() in favour of logger.warning(). This is only
used in bitbake code so we may as well just translate everything to avoid
warnings under python 3. Its safe for python 2.7.

(Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-11 10:34:30 +01:00