Commit Graph

53 Commits

Author SHA1 Message Date
Richard Purdie
c263f31e39 bitbake: main: When retrying the connection, show the attempt number
The current bitbake output makes it hard to know which retry is being
attempted. Add this information to the output to make it clearer.

(Bitbake rev: 0774e6e03d27adb7aca6fa9c47ab6ad426c937de)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-26 11:10:33 +00:00
Richard Purdie
5806b4cbd8 bitbake: main: Don't use print() directly, use logger
Avoid failures like:

2018-12-04 03:30:56,033 - oe-selftest - INFO - ======================================================================
2018-12-04 03:30:56,033 - oe-selftest - INFO - FAIL: pkgdata.OePkgdataUtilTests.test_find_path (subunit.RemotedTestCase)
2018-12-04 03:30:56,033 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-12-04 03:30:56,033 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
[...]
AssertionError: 'Previous bitbake instance shutting down?,[77 chars]xist' != 'ERROR: Unable to find any package produci[14 chars]xist'
- Previous bitbake instance shutting down?, waiting to retry...
  ERROR: Unable to find any package producing path /not/exist

We need to use the logger so output is correctly handled in such cases.

(Bitbake rev: b5686fc07d06e1f0a55161b5ebcec6552d74ceff)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-05 13:11:27 +00:00
Richard Purdie
4806d53304 bitbake: main: Fix environment handling for UI module imports
The environment was being cleared before the UI imports occurred which
caused problems for graphical UIs like taskexp. The full environment was
intended to be available to UI clients and it was only meant to be cleared
for the server/cooker, so tweak the code order so this is the case.

This fixes problems reported for taskexp.

[YOCTO #12670]

(Bitbake rev: ac15028391fdcc3fec2dd0e64a4f763ce63e5449)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-10 12:47:48 +01:00
Richard Purdie
c75985f40a bitbake: main/runqueue: Rework runall task and add runonly option
The runall commandline option was confusing people. There are in fact two
different behaviours people may want.

a) For a given target (or set of targets) look through the task graph and
run task X only if its present and would have been built.

b) For a given target (or set of targets) look through the task graph and
run task X if any recipe in the taskgraph has such a target even if it wasn't
in the original task graph.

I've decided to interpret the existing "runall" option as b), even if right
now if behaves like a). For a), which is a valid use case, this patch adds
a "runonly" option.

With both behaviours present, I'm hoping we can then kill off the "fetchall",
"checkuriall" and other tasks from OE metadata and replace them with this
option. This would significantly speed up task graph processing.

(Deleting the checkuriall and fetchall tasks takes "bitbake core-image-sato -g"
from 22s to 8s).

(Bitbake rev: 546a662c877b2d3af35e3996950582ed2df41fe4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-14 15:26:03 +00:00
Richard Purdie
5dacc55167 bitbake: main: Give a user readable error if we can't locate topdir
Currently if you run bitbake in an invalid directory, the user experience
is poor:

birbake/lib/bb/main.py", line 427, in setup_bitbake
    topdir, lock = lockBitbake()
  File "./bitbake/lib/bb/main.py", line 494, in lockBitbake
    lockfile = topdir + "/bitbake.lock"
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

This ensures we exit straight away with a better error message.

[YOCTO #12163]

(Bitbake rev: 562f9ee674a8b392437096422b9cceab9c3cba2e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-10-31 09:09:21 +00:00
Richard Purdie
2355a9b6f0 bitbake: cookerdata/taskdata/runqueue: Drop remaining tryaltconfigs code and commandline option
I can't actually see how this was working, nothing connected the commandline option
to the data in TaskData(). Drop the remaining pieces of this option, it was a relic
from a decade ago and we want deterministic builds, not random tries until something
might work.

(Bitbake rev: 767c7ba8fc76ec667ac1567de1c971c3575f2ecd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-09-02 00:52:10 +01:00
Richard Purdie
4152c47243 bitbake: main: Attempt to gain bitbake.lock rather than just waiting
Rather than just waiting for 5s, try and get the lockfile. If we gain
the lock, we know we're ready to retry and can skip any remaining timeout.

(Bitbake rev: 8a60106c6f7d586c793b965c5e9460b6016fab15)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-24 13:48:40 +01:00
Richard Purdie
2b453483d4 bitbake: process: Clean up connection retry logic
Its possible for a connection to connect to the server as its shutting down
but before its removed the socket file. This patch:

a) Removes the socket file earlier to avoid connections.
b) Handles EOFError in initial connections gracefully. These occur if the
   socket is closed during the server shutdown.
c) Ensure duplicate events aren't shown on the console. This makes debugging
   these issues very very confusing.

With these changes the backtrace that was concerning users is hidden and the
server works as expected with a reconnect when it catches it in a bad state.

(Bitbake rev: f45196cf84669723382730944dddc7eaf50826f2)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-24 13:48:40 +01:00
Robert Yang
803b97a32e bitbake: main: Handle BB_SERVER_TIMEOUT = -1 for no server timeout
Make BB_SERVER_TIMEOUT = -1 mean no unload forever.

(Bitbake rev: 923d5567be6fd9969ca74c166f36817ec09305e3)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09 09:24:56 +01:00
Robert Yang
896243f4fe bitbake: main: Remove unneeded float()
There is already a type=float, so the float() is not needed, which also makes
the error clearer:

$ export BB_SERVER_TIMEOUT=10000AA
With float():
$ bitbake quilt-native
[snip]
ValueError: could not convert string to float: '10000AA'

Without float():
$ bitbake quilt-native
[snip]
optparse.OptionValueError: option --idle-timeout: invalid floating-point value: '10000AA'

The second one tells clearly where is wrong.

(Bitbake rev: 7a4ea1e6a10f20d80009a78f4f0aebf8f90095fd)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09 09:24:56 +01:00
Paul Eggleton
46e2251625 bitbake: Revert "tinfoil: fix duplication of log messages"
In combination with the recent server reworking, this change actually
prevents messages sent from tasks from being logged properly. This will
of course give us the duplicated messages back, and I really hate to do
that effectively a second time, but that's better than seeing no error
at all in the case of a failure - we'll have to find the proper way of
avoiding the duplication that doesn't result in some messages going
missing.

This reverts commit 8a5bae76f91f2411187c638a42fa3c762052cf11.

(Bitbake rev: 645c8dd15762516ae5ab64a1df47fadb95d072d1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-09 09:24:56 +01:00
Richard Purdie
e4c9a92cb4 bitbake: main: Always return 0 for bitbake --kill-server
If the server isn't running return 0 as to do otherwise complicates
scripts which do cleanup of bitbake servers which would potentially
be memory resident.

(Bitbake rev: 0f85c54385f325f3ebbf911b35bbac11b8863b6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-30 08:43:36 +01:00
Robert Yang
579b811233 bitbake: bb/main.py: fix logic for --observe-only
Fixed:
$ bitbake --observe-only
FATAL: '--observe-only' can only be used by UI clients connecting to a server.

And even:
$bitbake --observe-only -B localhost:-1
FATAL: '--observe-only' can only be used by UI clients connecting to a server.

This was beucase the brackets were in a wrong position.

(Bitbake rev: 4ffc91a2b3eb13e98078e6b1913f056a0c1797bc)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Robert Yang
c276e38bf8 bitbake: bb/main.py: avoid starting server when not needed
Fixed 1:
$ . ../poky/oe-init-build-env-memres .
$ bitbake -m
$ bitbake -m # shutdown it again
$ bitbake -m
Starting bitbake server...
Terminated bitbake server.

It starts the server and kill it which is redundant.

$ bitbake -m
$ bitbake --status-only
Starting bitbake server...

It starts the server which is not what we need.

(Bitbake rev: 335fd0106359dddfe1eeb0c7d0bc03f8454b895c)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Robert Yang
38af7fef51 bitbake: bb/main.py: fix infinite loop for --server-only
Fixed:
$ bitbake --server-only -B localhost:-1
Infinite loop
Reconnecting to bitbake server...
[snip]

Break the loop when server-only can fix the problem.

(Bitbake rev: c4d5c3dab5b35f3c6713efad1c8a23229ce4e56b)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Richard Purdie
4602408c69 bitbake: server: Rework the server API so process and xmlrpc servers coexist
This changes the way bitbake server works quite radically. Now, the
server is always a process based server with the option of starting
an XMLRPC listener on a specific inferface/port.

Behind the scenes this is done with a "bitbake.sock" file alongside
the bitbake.lock file. If we can obtain the lock, we know we need
to start a server. The server always listens on the socket and UIs
can then connect to this. UIs connect by sending a set of three file
descriptors over the domain socket, one for sending commands, one for
receiving command results and the other for receiving events.

These changes meant we can throw away all the horrid server abstraction
code, the plugable transport option to bitbake and the code becomes
much more readable and debuggable. It also likely removes a ton of
ways you could hang the UI/cooker in weird ways due to all the race
conditions that existed with previous processes.

Changes:

 * The foreground option for bitbake-server was dropped. Just tail
   the log if you really want this, the codepaths were complicated enough
   without adding one for this.
 * BBSERVER="autodetect" was dropped. The server will autostart and
   autoconnect in process mode. You have to specify an xmlrpc server
   address since that can't be autodetected. I can't see a use case
   for autodetect now.
 * The transport/servetype option to bitbake was dropped.
 * A BB_SERVER_TIMEOUT variable is added which allows the server
   to stay resident for a period of time after the last client
   disconnects before unloading. This is used if the -T/--idle-timeout
   option is not passed to bitbake.

This change is invasive and may well introduce new issues however I
believe the codebase is in a much better position for further
development and debugging.

(Bitbake rev: 72a3dbe13a23588e24c0baca6d58c35cdeba3f63)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:41:11 +01:00
Paul Eggleton
d44d5a2e79 bitbake: tinfoil: fix duplication of log messages
Adding an additional logger in setup_bitbake() interacts poorly with the
logger we have added by default in tinfoil's constructor, with the
result that messages may be doubled or even tripled in tinfoil-using
scripts. Disable adding this one when calling setup_bitbake() from
tinfoil to avoid this problem.

Part of the fix for [YOCTO #11275].

(Bitbake rev: 8a5bae76f91f2411187c638a42fa3c762052cf11)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-12 15:07:39 +01:00
Diana Thayer
71e48a3617 bitbake: main: Improve -v and -D option documentation
Expanded and clarify documentation for the -v, --verbose and -D, --debug
options.

[YOCTO #9962]

(Bitbake rev: 921592026c69287cdb40ffd90944d5944f28e2c3)

Signed-off-by: Diana Thayer <garbados@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-27 20:23:19 +01:00
Matthew McClintock
404f2aef94 bitbake: runqueue: add option to run all tasks in specific build target
For example:

$ bitbake core-image-minimal --runall unpack
...
NOTE: Tasks Summary: Attempted 326 tasks of which 88 didn't need to be rerun and all succeeded.

$ bitbake core-image-minimal --runall patch
NOTE: Tasks Summary: Attempted 453 tasks of which 332 didn't need to be rerun and all succeeded.

This can replace fetchall as well:
$ bitbake core-image-minimal --runall fetch
NOTE: Tasks Summary: Attempted 135 tasks of which 119 didn't need to be rerun and all succeeded.

(Bitbake rev: 7c0fa6ba66cdb956b37d94055307cde857633df9)

Signed-off-by: Matthew McClintock <msm-oss@mcclintock.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-13 09:43:40 +00:00
Aníbal Limón
8a3221e6e5 bitbake: lib/bb/main.py: Shutdown cooker when server isn't foreground
The cooker is added to the BBServer and then detached creating
a copy of the cooker in the process, if the server isn't in
foreground it cause fd leaks on inotify this can be see using
many instances of tinfoil.

Example:

from bb.tinfoil import Tinfoil
while True:
    with Tinfoil() as tinfoil:
        input("Pre\n")
        tinfoil.prepare(config_only=True)
        input("Post\n")

[YOCTO #10873]

(Bitbake rev: 2ada2efb057a291eb34c66d975eb339232137956)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-23 22:38:49 +00:00
Paul Eggleton
7d5c9860de bitbake: tinfoil: rewrite as a wrapper around the UI
Rewrite tinfoil as a wrapper around the UI, instead of the earlier
approach of starting up just enough of cooker to do what we want. This
has several advantages:

* It now works when bitbake is memory-resident instead of failing with
  "ERROR: Only one copy of bitbake should be run against a build
  directory".

* We can now connect an actual UI, thus you get things like the recipe
  parsing / cache loading progress bar and parse error handling for free

* We can now handle events generated by the server if we wish to do so

* We can potentially extend this to do more stuff, e.g. actually running
  build operations - this needs to be made more practical before we can
  use it though (since you effectively have to become the UI yourself
  for this at the moment.)

The downside is that tinfoil no longer has direct access to cooker, the
global datastore, or the cache. To mitigate this I have extended
data_smart to provide remote access capability for the datastore, and
created "fake" cooker and cooker.recipecache / cooker.collection adapter
objects in order to avoid breaking too many tinfoil-using scripts that
might be out there (we've never officially documented tinfoil or
BitBake's internal code, but we can still make accommodations where
practical). I've at least gone far enough to support all of the
utilities that use tinfoil in OE-Core with some changes, but I know
there are scripts such as Chris Larson's "bb" out there that do make
other calls into BitBake code that I'm not currently providing access to
through the adapters.

Part of the fix for [YOCTO #5470].

(Bitbake rev: 3bbf8d611c859f74d563778115677a04f5c4ab43)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:25:07 +00:00
Paul Eggleton
568409faa1 bitbake: knotty: make quiet option a level option
Allow you to specify -q / --quiet more than once to reduce the messages
even further. It will now operate as follows:

 Level  Option  Result
 -----  ------  ----------------------------------------
 0              Print usual output
 1      -q      Only show progress and warnings or above
 2      -qq     Only show warnings or above
 3+     -qqq    Only show errors

(Bitbake rev: 6cf2582e17c28ca04f5cfb59858c4a9778c700d4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-14 12:25:06 +00:00
Benjamin Esquivel
b9d6a7cc23 bitbake: main: Check bitbake server-only port is a number
Either using the memres script or the bitbake call with --server-only
if the port is a string instead of a number then the process hangs
indefinitely causing a loop that never ends.

Add a check at the beginning for the port being a number otherwise
show an error message and exit cleanly.

[YOCTO #10397]

(Bitbake rev: 35927a98daeeb854ef5782e900206af6cd96b3d7)

Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-09 12:33:26 +01:00
Ed Bartosh
317708235b bitbake: bitbake: implement idle timeout for xmlrpc server
Idle timeout can be specified either by -T/--idle-timeout option or
by sessing BBTIMEOUT environment variable. Bitbake xmlrpc server
will unload itself when timeout exprired, i.e. when server is idle
for more than <idle timeout> seconds.

[YOCTO #5534]

(Bitbake rev: 5fa0b3a19e7d0f40790f737abeddf499d53f1f6a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-21 07:48:52 +01:00
Ed Bartosh
4cba010529 bitbake: bitbake: implement --foreground command line option
This option makes bitbake xmlrpc server to run in foreground.
It should be useful for debugging purposes.

(Bitbake rev: 9d4254be5853a546a346bf0d19919dcfba12773d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-21 07:48:52 +01:00
Ed Bartosh
63ff759627 bitbake: bitbake: main: implement server autostart feature
If environment variable BBSERVER == 'autostart' bitbake will
automatically load server if it's not running yet.

If host and port are in bitbake.lock then bitbake tries to check
if server is running and responses to commands and starts new
server only if this check fails.

[YOCTO #5534]

(Bitbake rev: 89c6e625d47303b2aad8e6645762f17aee01b2d4)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:23:46 +01:00
Paul Eggleton
76feac37ce bitbake: knotty: add quiet output mode
Quiet output mode disables printing most messages (below warnings) to
the console; however these messages still go to the console log file.
This is primarily for cases where bitbake is being launched
interactively from some other process, but where full console output is
not needed.

Because of the need to keep logging all normal events to the console
log, this functionality was implemented within the knotty UI rather
than in bb.msg (where verbose mode is implemented). We don't currently
have a means of registering command line options from the UI end, thus
the option actually has to be registered in main.py regardless of the
UI, however I didn't feel like it was worth setting up such a mechanism
just for this option.

(Bitbake rev: db95cdef08e339dec7462bfde3ad7d75c1c60dd8)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:27 +01:00
Mariano Lopez
40e789d1fe bitbake: lib/bb/main.py: Fix use of BBPOSTCONF and BBPRECONF
The functionality of BBPOSTCONF and BBPRECONF was added in
commit 21b314d4d1 but there
was a typo in the variable name that raises an exception
in bitbake.

[YOCTO #9235]

(Bitbake rev: 6d1379c8818400e5cdc442e6142f08a110fd5b95)

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-09 18:00:58 +01:00
Richard Purdie
0f2c59367a bitbake: bitbake: Convert to python 3
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.

(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00
Richard Purdie
c0db739f35 bitbake: main: Ensure exceptions are correctly displayed
If the cooker fails to start, ensure a correct exception is displayed to the
user. After handling any queued events simply re-raise the original exception
else the output can be unclear.

(Bitbake rev: 9a4db1aa608c17d31bf5ea1cab5a99beb565dd83)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-13 13:41:33 +01:00
Richard Purdie
ef4a7c30ca bitbake: main: Change warn() -> warning()
This avoids a deprecation warning in python 3.

(Bitbake rev: bf1a92d0c002d73e8a34472dced1343dc4a4251a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-13 13:41:32 +01:00
Ed Bartosh
be49d0ad57 bitbake: bitake: main: fix line-too-long pytling warnings
Wrapped long lines to fix "Line too long" pylint warnings.

(Bitbake rev: e329a932e14d002a561245b5026f974897f64598)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:12:16 +01:00
Ed Bartosh
b5af50e024 bitbake: bitbake: main: fix bad-continuation warnings
Fixed pylint warning 'Wrong continued indentation'.

(Bitbake rev: 5bfc8c0dc70abd6462a6df4ce905173d9f2cdc07)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:12:16 +01:00
Ed Bartosh
3437266a08 bitbake: bitbake: main: fix bad-witespace pylint warnings
Fixed pylint warnings:
 No space allowed around keyword argument assignment
 No space allowed after bracket
 No space allowed before bracket

(Bitbake rev: c39770239f7b61217501782b9c5e9d3211355d42)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:12:16 +01:00
Ed Bartosh
24f5c32b80 bitbake: bitbake: main: set defaults from env variables
Environment variables BBSERVER, BBTOKEN and BBEVENTLOG silently
overwrite bitbake command line arguments. This is confusing and
can cause issues that are difficult to debug. It's better to use
them as default values instead.

Used environment variables BBSERVER, BBTOKEN and BBEVENTLOG to set
default values for command line arguments.

Changed setting default value of --ui command line argument from
BITBAKE_UI to look similar way.

(Bitbake rev: 87040be4ff54cd460961318224deef8f2ea4c85a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:12:16 +01:00
Ed Bartosh
21b314d4d1 bitbake: bitbake: main: add 2 environment variables
Bitbake uses set of environment variables to set command line
options, e.g. seeting BBTOKEN variable has the same effect
as using --token command line option.

Added new environment variables BBPRECONF and BBPOSTCONF that
are equivalents of --read and --postread command line options.
They can be used by high level scripts to append or prepend
configuration files to conf/local.conf

[YOCTO #9235]

(Bitbake rev: bf604ec1ca4eb4d0b22bcc72249963e6d7445f34)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:12:16 +01:00
Ed Bartosh
c7980b6089 bitbake: main: fix processing of BBEVENTLOG
Fixed typo that caused incorrect processing of BBEVENTLOG
environment variable. Even if variable is set it was ignored
by bitbake.

(Bitbake rev: 2705b5f59aef4a070e2df2752d27bd04ea747057)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-19 17:31:01 +01:00
Ed Bartosh
b58e5b1fb3 bitbake: bitbake: xmlrpc: set single use mode differently
Currently xmlrpc server implicitly sets itself into single use mode
when bitbake server is started with anonymous port (0) or no port is
provided in command line. In this mode bitbake shuts down xmlrpc server
after build is done. This assumption is incorrect in some cases.
For example Toaster uses bitbake in this mode and expects xmlrpc server
to stay in memory.

Till recent changes single use mode was always unset due to the bug.
When the bug was fixed it broke toaster builds as Toaster couldn't
communicate with bitbake server in single use mode.

Reimplemented logic of setting single use mode. The mode is explicity
set when --server-only command line parameter is not provided to bitbake.
It doesn't depend on the port number anymore.

[YOCTO #9275]
[YOCTO #9240]
[YOCTO #9252]

(Bitbake rev: afc0dd5c532684f6201b1e12bbf4c226ea19062d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-24 12:31:50 +00:00
Richard Purdie
46731daa5f bitbake: main/runqueue: Add --setscene-only option to bitbake
Its turning out that we really need a way to have bitbake just run
the setscene tasks but not any real tasks, particularly for SDK
operations.

Add an option for this since its pretty straight forward. This allows
various nasty workarounds in OE-Core to be removed.

(Bitbake rev: e4a2aafa1650a227a04d92a8a0b31efaed2c310e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11 23:26:32 +00:00
Ed Bartosh
a178c5a163 bitbake: main: kill server without queue setup
Setup of event queue includes registering of UI handler.
This operation can fail when cooker is busy. However, there is
no need in registering UI handler for terminating the server.

Moved the call of connection.terminateServer before setting up
of the event queue. This should make terminating server to work
more reliably as it doesn't depend on setting up the event queue
and registering UI handler anymore.

This should also help Toaster backend to restart bitbake server
and observer without getting "Could not register UI event handler"
errors.

[YOCTO #8776]

(Bitbake rev: 0c5a9349f797d05c282c2ada1893e187e05f0576)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh
773700dbc3 bitbake: xmplrpc: split connect method
Current code in connect method sets up event queue, which requires
registering UI handler. This functionality may not be needed for
some operations, e.g. for server termination.

Moved functionality of setting up event queue in from 'connect'
method to 'setupEventQueue' in BitBakeXMLRPCServerConnection class.

(Bitbake rev: 4429871da76d6bd29e023ff42740fe7daa6b40fa)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06 15:27:35 +00:00
Ed Bartosh
4ab7202c31 bitbake: cooker: preserve pre and post configs
Additional config files passed to bitbake server with --read and
--postread options are rewritten by client bitbake even if
it doesn't use those options.

This is a show stopper for toaster as toaster command line
builds are based on the assumption that server is aware of
toster configs, provided by --postread option.

This behaviour is fixed by preserving values of --read and
--postread options when bitbake server starts and restoring
them if client bitbake doesn't explicitly specify them.

(Bitbake rev: 02c64f7487ca8ec5d32c440f5002c4b8f64b76a6)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-10-27 07:24:30 +00:00
Paul Eggleton
bb34181b42 bitbake: lib/bb/main: avoid importing all server/UI modules on every execution
We're importing the server and UI modules in order to check they are
valid, but it turns out that that has some nasty side-effects. We don't
actually need to do this except when --help is passed or the module
doesn't exist, so rearrange the code so that we only do the module
listing in those two cases.

Additionally, let's just go ahead and catch all errors on import; we
really don't care to have them cause a failure now.

(Bitbake rev: c9dc6d9c86e8b887821a6d00346bd0b09e1da97c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-24 23:47:07 +01:00
Randy Witt
a3d228a484 bitbake: main: Handle RuntimeError exception in list_extension_modules
This exception was triggered in toaster from recent changes and is completely
breaking the whole of bitbake. Add the exception to the list so at least only
toaster is affected.

(Bitbake rev: f64def7cb6069dc1134fcd546bb59e4030c7376f)

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-20 22:04:43 +01:00
Paul Eggleton
5807ab3266 bitbake: lib/bb/main: consolidate UI/server extension listing and loading
Provide us with a means of showing the list of UIs / server choices for
the command line help, and do the processing in one place for both.

(Bitbake rev: 24035c1daad5a904c3372d21d44191ee8182338f)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:48 +01:00
Paul Eggleton
e2ddc18874 bitbake: Fix -m handling if cannot connect to server
If we can't connect to the server we should error out, because it might
not be that the server is actually dead - it might just be unable to
execute commands.

(Bitbake rev: d4b921676859d6ba4e1922fa4682ee941652f483)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:46 +01:00
Richard Purdie
4c2f28bf48 bitbake: cooker: properly fix bitbake.lock handling
If the PR server or indeed any other child process takes some time to
exit (which it sometimes does when saving its database), it can end up
holding bitbake.lock after the UI exits, which led to errors if you ran
bitbake commands successively - we saw this when running the PR server
oe-selftest tests in OE-Core. The recent attempt to fix this wasn't
quite right and ended up breaking memory resident bitbake. This time we
close the lock file when cooker shuts down (inside the UI process)
instead of unlocking it, and this is done in the cooker code rather than
the actual UI code so it doesn't matter which UI is in use. Additionally
we report that we're waiting for the lock to be released, using lsof or
fuser if available to list the processes with the lock open.

The 'magic' in the locking is due to all spawned subprocesses of bitbake
holding an open file descriptor to the bitbake.lock. It is automatically
unlocked when all those fds close the file (as all the processes terminate).
We close the UI copy of the lock explicitly, then close the server process
copy, any remaining open copy is therefore some proess exiting.

(The reproducer for the problem is to set PRSERV_HOST = "localhost:0"
and add a call to time.sleep(20) after self.server_close() in
lib/prserv/serv.py, then run "bitbake -p; bitbake -p" ).

Cleanup work done by Paul Eggleton <paul.eggleton@linux.intel.com>.

This reverts bitbake commit 69ecd15aece54753154950c55d7af42f85ad8606 and
e97a9f1528d77503b5c93e48e3de9933fbb9f3cd.

(Bitbake rev: a29780bd43f74b7326fe788dbd65177b86806fcf)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-01 07:34:09 +01:00
Ross Burton
18d71ed16e bitbake: main: add skeleton documentation for the recipe:do_task syntax
The incredibly useful recipe:do_task syntax on the command line isn't documented
at all.  This isn't much but it's better than nothing.

(Bitbake rev: 7f4c07886ecff4ac77fdd2165bedd179099fcf19)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-07-07 16:18:34 +01:00
Ed Bartosh
37e46e49dd bitbake: bitbake: Check if bitbake versions match
Bitbake program and core versions must match.

Moved __version__ from main.py back to bin/bitbake.
Implemented check for version match in bin/bitbake.

(Bitbake rev: 2fe7d8c574ddf6a30278cff1a5a5c4089dc56d6d)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>

tbs

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-04-29 10:59:53 +01:00
Ed Bartosh
47e2dbd89a bitbake: bitbake: Handle BBMainException
Some error messages were lost because BBMainException was
inherited from bb.BBHandledException. When bb.BBHandledException
is processed error messages are not printed as they suppose to
be printed before raising this exception.

Stopped to inherit BBMainException from bb.BBHandledException.
Handled BBMainException in bin/bitbake and printed error message
to the stderr.

(Bitbake rev: c8e2a40c4e9865ebef9936d23644f2602a5c90f5)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-04-29 10:59:53 +01:00