Commit Graph

86 Commits

Author SHA1 Message Date
Alexander Kanavin
941ac7d256 bitbake: parse/ast: add support for 'built-in' fragments
When reviewing proposed fragments to add settings for DISTRO and MACHINE,
RP noted that such fragments only add clutter and overhead, and there's
no need to maintain them as separate files.

Rather when bitbake sees 'fragmentvar/fragmentvalue' it can expand that into
FRAGMENTVAR = "fragmentvalue".

To achieve that, 'addfragments' directive is extended with a parameter
that sets the name of the variable that holds definitions of such
built-in fragments, for example like this:

"machine:MACHINE distro:DISTRO"

Then each enabled fragment name is matched against these definitions and the
respective variable is set, e.g. 'machine/qemuarm' would match
'machine:MACHINE' and result in MACHINE set to 'qemuarm'.

This happens before any fragment files are looked up on disk,
and no such lookup happens if there was a match, which should prevent
possible misuse of the feature. So the builtin fragment definition
is also an allowlist for them.

Please also see the patches for oe-core that show an application of the feature.

(Bitbake rev: 3b9d7bea915dc7f10e845854f1dae325743f9456)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-06-20 12:03:37 +01:00
Richard Purdie
79fa12ae4b bitbake: parse/ConfHandler: Add warning for deprecated whitespace usage
A lack of whitespace around variable assignment operators makes the
files harder to read.

There is a deeper issue in that a "+" character can sometimes be confused
between the variable name and the assignment operator.

Start showing warnings for such usage so we encourage people to use
consistent whitespace which helps with file readability in general.

(Bitbake rev: 24772dd2ae6c0cd11540a260f15065f906fb0997)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-04-16 17:41:51 +01:00
Nikolai Merinov
b65452bda3 bitbake: parse: Forbid ambiguous assignments to ${.}, ${+}, and ${:} variables
Old code that parse variable names in assignment commands behave differently for
variables that ends with special symbol for single-character variable names and
multi-character variable names. For example:

  A+="1"   # Change variable ${A}, '+' glued to '='
  A+ = "1" # Change variable ${A+}

  +="1"    # Change variable ${+}, the '+' symbol not part of assignment operator
  + = "1"  # Change variable ${+}

New code would always assume that '.=', '+=', and ':=' is assignment operator.
As result code like the following would raise parsing error

  +="value"

While code with extra spaces would work as before

  + = "value" # Change variable ${+}

This change allow to catch issues in code that generate bitbake configuration
files in a manner like "echo ${VARNAME}+=${VALUE} >> conf/local.conf"

(Bitbake rev: 93059aad13a12cd69d86368795c88e5349197d5d)

Signed-off-by: Nikolai Merinov <n.merinov@inango-systems.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-02-05 12:11:01 +00:00
Richard Purdie
5cb3d1a59c bitbake: parse: Add include_all conf file directive
In some cases it would be helpful to be able to have an include file
in a standard location which is included in all layers that are added
to the system. The intent is for these to provide configuration tweaks
of specific types so that a given file pattern can be adopted more widely
for such configuration.

The code will search for any named configuration file within BBPATH, so
a configuration directive of:

include_all conf/distro/include/myinc.conf

would include the myinc.conf file in that subpath if present in any
directory in BBPATH. Multiple files will be included if present.

(Bitbake rev: d01d5593e7829ac60f37bc23cb87dc6917026471)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-17 11:48:25 +00:00
Alexander Kanavin
e0e6124d9a bitbake: bitbake: parse: add support for flagged fragment variables
It was suggested that using flags in fragment content to
mark fragment-specific metadata (such as descriptions and
dependencies) is prone to quiet regressions when fragments
are renamed or moved, and it clutters the fragment content.

With this change allowed fragment metadata variables must
be explicitly listed in a variable that is given to addfragment
directive, and parser will add the flag containing the fragment name
to them.

(Bitbake rev: ed9a3ca9426500511feb77f41a146953dbfe9af7)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-13 10:44:35 +00:00
Alexander Kanavin
3de3bda4c4 bitbake: parse: add support for 'addfragments' directive
It takes two parameters:
- location prefix for fragments
- name of variable that holds the list of enabled fragments, each of them prefixed
by layer id

Implementation of this directive essentially expands the fragment list
obtained from the variable into absolute fragment paths and hands them to the
implementation of 'require'.

(Bitbake rev: f687746703e7b096c5480668fd4f49bd4951182b)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-08 22:36:11 +00:00
Alexander Kanavin
186b7c58d5 bitbake: parse/ConfHandler: allow / in variable flags
This is beneficial for config fragments, as their names
are specified via flags, and those names can include slashes:

BB_CONF_FRAGMENT_SUMMARY[init/systemd] = "This fragment enables systemd as an init manager"

(Bitbake rev: 80805988492b35593067230f68782a5687c8f557)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-12-08 22:36:11 +00:00
Kai Kang
a99566dac1 bitbake: bitbake: ConfHandler: Allow variable flag name with a single character
Update regex pattern to allow variable flag name with a single character.

Regression tests have also been updated in `bb.parse` and
`bin/bitbake-selftest -k ParseTest` has been successfully executed.

Eliminate a trailing space as well.

(Bitbake rev: bb9e523291a3cad6e1596ee6a1e715b5e5feba8f)

Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-13 12:01:45 +01:00
James R T
f14328c22f bitbake: ConfHandler: Allow the '@' character in variable flag names
This patch enables the usage of the '@' character in variable flag
names.

One use case of variable flags is to assign the network namespaces of
some systemd services/targets to configure other parts of the build
process of some system. The filenames of systemd services/targets might
contain the '@' character if they are template unit files that can take
in a single parameter/argument and be instanced multiple times, as
indicated by systemd's official manual page.

The '@' character is disallowed as the first character in a variable
flag name. Imposing more restrictions on the first character is a
compromise to make parsing easier and to allow for more options in the
future to extend the syntax.

This patch is successfully verified by creating a custom BitBake recipe
that sets and unsets the value of a variable flag with the '@' character
in its name and ensuring that no ParseError is being thrown. Regression
tests have also been added to `bb.parse`.

`bin/bitbake-selftest` has also been successfully executed and all tests
passed.

(Bitbake rev: 00f9ab2cacfbd2a63b6b4959cf5401babae7e32a)

Signed-off-by: James Raphael Tiovalen <jamestiotio@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-23 22:41:32 +00:00
Richard Purdie
42e8b2682f bitbake: parse: Add support for addpylib conf file directive and BB_GLOBAL_PYMODULES
For many years OE-Core has injected it's own python modules into the
python namespace using an immediate expansion of a variable in base.bbclass.
It also added all entries from BBPATH to sys.path.

We really need this to become a first class citizen of the langauge, this new
addpylib directive allows that. Usage is of the form:

addpylib <directory> <namespace>

The namespace is imported and if there is an attribute BBIMPORT, that
list of names is iterated and imported too.

This mirrors what OE-Core has done for a long time with one difference in
implmentation, sys.path is only appended to. This means later imported
namespaces can't overwrite an earlier one and can't overwrite the main
python module space. In practice we've never done that and it isn't
something we should encourage or support.

The new directive is only applied for .conf files and not other filetypes
as it only makes sense in that context. It is also only allowed in the
"base configuration" context of cookerdata since adding it at the recipe
level wouldn't work properly due to the way it changes the global namespace.

At the same time, move the list of modules to place in the global namespace
into a BB_GLOBAL_PYMODULES variable. It is intended that only the core layer
should touch this and it is meant to be a very small list, usually os and sys.
BB_GLOBAL_PYMODULES is expected to be set before the first addpylib directive.

Layers adding a lib directory will now need to use this directive as BBPATH
is not going to be added automatically by OE-Core in future. The directives are
immediate operations so it does make modules available sooner than the current
OE-Core approach.

The new code appends to sys.path rather than prepends as core did, as
overwriting python standard library modules would be a bad idea and naturally
encouraging people to collaborate around our own core modules is desireable.

(Bitbake rev: afb8478d3853f6edf3669b93588314627d617d6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-08 11:15:47 +00:00
Richard Purdie
bc6d96e696 bitbake: ConfHandler/BBHandler: Improve comment error messages and add tests
Currently if you trigger one of the comment errors, the newline characters
are stripped and the line numbers are incorrect. In one case it prints
the empty line which is also unhelpful.

Rework the code around these errors so the line numbers are correct
and the lines in question are more clearly displayed complete with newlines
so the user can more clearly see the error.

I also added a couple of simplistic test cases to ensure that errors
are raised by the two known comment format errors.

[YOCTO #11904]

(Bitbake rev: 712da71b24445c814d79a206ce26188def8fce0a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-07-04 22:52:36 +01:00
Ola x Nilsson
5aca7a8f10 bitbake: ConfHandler: Remove lingering close
The f.close() statement should have been removed in
459ad524756a3f9b50feeedf31e33502dceae8d5.

(Bitbake rev: 9fc1bab6b7e3c0fca3ddec4bc8c7763d2aff8bab)

Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-21 20:50:57 +01:00
Richard Purdie
f07a097213 bitbake: cache/ConfHandler: Drop TOPDIR/chdir manipulations
This code has been unchanged since 2006 apart from attempts to optimise
performance by minimising chdir() calls.

There is no reason the modern bitbake parser should be changing directory
all the time. We did have some path assumptions in the mists of time but
those were resovled and the code is deterministic and doesn't depend on
cwd now for parsing. We can therefore drop the changes in directory.

Also, TOPDIR is now being set by cookerdata in all cases so we don't
need the fallbacks in this code (which was used to effectively initialise
a value). We don't need to change TOPDIR when parsing a recipe, that makes
no sense. If we stop all the other messing around, we don't need to expand
TMPDIR either.

These changes have the potential to break some obscure use cases such
as an anonymous function assuming the current working directory, or some
case which depends on TOPDIR changing but I believe any such uses should
be fixed at this point.

(Bitbake rev: add5d488e1d6607a98441836075d01cb1dc9c0fa)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-15 11:03:25 +00:00
Richard Purdie
75fad23fc0 bitbake: data_smart/parse: Allow ':' characters in variable/function names
It is becomming increasingly clear we need to find a way to show what
is/is not an override in our syntax. We need to do this in a way which
is clear to users, readable and in a way we can transition to.

The most effective way I've found to this is to use the ":" charater
to directly replace "_" where an override is being specified. This
includes "append", "prepend" and "remove" which are effectively special
override directives.

This patch simply adds the character to the parser so bitbake accepts
the value but maps it back to "_" internally so there is no behaviour
change.

This change is simple enough it could potentially be backported to older
version of bitbake meaning layers using the new syntax/markup could
work with older releases. Even if other no other changes are accepted
at this time and we don't backport, it does set us on a path where at
some point in future we could
require a more explict syntax.

I've tested this patch by converting oe-core/meta-yocto to the new
syntax for overrides (9000+ changes) and then seeing that builds
continue to work with this patch.

(Bitbake rev: 0dbbb4547cb2570d2ce607e9a53459df3c0ac284)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-07-20 18:59:18 +01:00
Joshua Watt
75f87db413 bitbake: logging: Make bitbake logger compatible with python logger
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.

Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.

[RP: Small fix to ensure bb.debug calls bbdebug()]
(Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-10 23:48:16 +00:00
Ola x Nilsson
5107dce38e bitbake: bitbake: ConfHandler: Use with to manage filehandle lifetime
(Bitbake rev: 459ad524756a3f9b50feeedf31e33502dceae8d5)

Signed-off-by: Ola x Nilsson <olani@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-10-28 05:31:53 +00:00
Robert Yang
5161799993 bitbake: bitbake: lib: Cleanup /usr/bin/env python
(Bitbake rev: cc712f3257904960247a7532cfc4611f3dccd36c)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-28 13:29:04 +01:00
Richard Purdie
9501864db8 bitbake: bitbake: Strip old editor directives from file headers
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.

(Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
cf9c0be3f6 bitbake: bitbake: Drop duplicate license boilerplace text
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.

(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
79834a7144 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:04 +01:00
Robert Yang
2c15d8000d bitbake: bitbake: ConfHandler: Don't strip leading spaces
Fixed:
- Add the following lines to conf/local.conf:
  FOO = "BAR1"
  FOO_append = "\
      BAR2"

  $ bitbake -e | grep '^FOO'
  FOO="BAR1BAR2"

  The leading spaces in the second line have been removed.

- But if add the previous two lines to base.bbclass:
  $ bitbake -e | grep '^FOO'
  FOO="BAR1    BAR2"

  The leading spaces in the second line are preserved, this is inconsistent, now
  fix ConfHandler to preserve leading spaces.

[YOCTO #12380]

(Bitbake rev: 8c3bc15a7b5e0a81d7b6c9d3fe43fbff63207156)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-06 08:24:50 +00:00
Robert Yang
85fa5ee825 bitbake: bitbake: parse: fixes for resolve_file()
The resolve_file() calls mark_dependency(), so the one which calls
resolve_file() doesn't need call mark_dependency() again.

(Bitbake rev: 4682571107323a39b42cd9ec8ee67419e7f15acc)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-14 15:26:03 +00:00
Patrick Ohly
5df6796d1c bitbake: ConfHandler.py: allow require or include with multiple parameters
"inherit" already allows inheriting more than one class in a single
statement. The same also makes sense for "include" and "require",
because then one can generate a list of files to be included
dynamically also for the case that more than one file needs to be
included.

(Bitbake rev: 8d0a76f5a595dddf16b7268bae2c00ef5f568316)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-13 10:47:09 +01:00
Patrick Ohly
6dafbaeff4 bitbake: ConfHandler.py: allow require or include without parameter
Writing .bbappends that only have an effect when some configuration
variable like DISTRO_FEATURES is changed becomes easier when allowing
"include" or "require" without a parameter. The same was already
allowed for "inherit".

Then one can write in a .bbappend:

  require ${@bb.utils.contains('DISTRO_FEATURES', 'foo', 'bar.inc', '', d)}

(Bitbake rev: 8b39c6361758b96fce50a53a6dba8008cd7e6433)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-13 10:47:09 +01:00
Ola x Nilsson
ca68d71ddc bitbake: ConfHandler: Require whitespace between export and variable name
(Bitbake rev: 22bb7c9270f02ddae72e13d849375feee5f4a98b)

Signed-off-by: Ola x Nilsson <olani@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-05 09:19:51 +01:00
Peter Kjellerstedt
b1f09df0f6 bitbake: ConfHandler: Use the same regular expression for all variable names
When the regular expression for matching a variable name was amended
with allowing the ~ character as part of the variable name, this was
never done to the regular expression that matches export
lines. Similarly, the regular expression that was used for matching
unset variables also used the one without support for the ~ character.

This unifies the regular expressions. For good measures it also
corrects the regular expression used to match a variable flag name for
the unset command to match the one used when setting a variable flag.

(Bitbake rev: acd2fd74ed467dc85ec75d5d0815f43e493f29bf)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-13 09:43:40 +00:00
Joshua Lock
1fce7ecbbb bitbake: bitbake: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30 15:48:09 +00:00
Jérémy Rosen
0eb6d709b6 bitbake: ast/ConfHandler: Add a syntax to clear variable
unset VAR
will clear variable VAR
unset VAR[flag]
will clear flag "flag" from var VAR

(Bitbake rev: bedbd46ece8d1285b5cd2ea07dc64b4875b479aa)

Signed-off-by: Jérémy Rosen <jeremy.rosen@openwide.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18 10:06:26 +01:00
Richard Purdie
654eadfa30 bitbake: bitbake: Update logger.warn() -> logger.warning()
python deprecated logger.warn() in favour of logger.warning(). This is only
used in bitbake code so we may as well just translate everything to avoid
warnings under python 3. Its safe for python 2.7.

(Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-11 10:34:30 +01:00
Christopher Larson
22078d5e53 bitbake: bb.parse: properly error out on filesystem errors
We've had a long-standing bug where a legitimate error reading a file (IOError
or OSError) is always suppressed as though it was a 'file not found' case. As
a concrete example, if you do a `chmod 000 conf/local.conf`, it'll silently
not parse local.conf, rather than erroring to let the user know about the
problem.

Fix this by handling the ENOENT case specifically.

(Bitbake rev: e691312a3add222b04e7b2f52f8df6abcb9068bf)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-01 22:24:20 +01:00
Richard Purdie
69b6919341 bitbake: bitbake: Add explict getVar param for (non) expansion
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.

This patch was mostly made using the command:

sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`

(Bitbake rev: 659ef95c9b8aced3c4ded81c48bcc0fbde4d429f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-23 11:57:53 +01:00
Richard Purdie
3db9d6e02d bitbake: ConfHandler: Fix bogus dependency problems
Adding a dependency when errors occur accessing a file when calling
handle() is not the correct thing to do. THe handle() code calls
resolve_file() which can raise an exception without ever touching
"fn" itself, it has also already marked all the dependencies correctly.

This leads to bogus files being resolved to the local cwd and
hence triggers reparses for no good reason.

The solution is to simply remove the bogus dependency.

(Bitbake rev: 366af3be1cffd64e4a79c15990c1e05869022c14)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-05-29 10:17:16 +01:00
Richard Purdie
2a6f739f1d bitbake: ConfHandler: Clean up bogus imports
The import statements here are plain bizarre. Remove them, tweaking
some of the function calls to match current practices. I can't find any
reason these old imports are as they are.

(Bitbake rev: 4c2f1fe51a13ddc97e518327714292af46b9e1ab)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-16 08:25:30 +00:00
Richard Purdie
5c9e2b2f43 bitbake: ConfHandler: Rename oldfn to parentfn to be clearer
Looking at this function I had no idea what oldfn was, I doubt anyone
else would either without looking up what the caller does. "parentfn"
would seem a more appropriate name so rename it.

(Bitbake rev: fc70ed596703a1aa954223b169d4ad51193a6ec1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-16 08:25:30 +00:00
Marius Avram
888683de7a bitbake: ConfHandlerr: Use full path in variable history
When an evaluation was made for a configuration file the path to the
file was saved as a relative one. The change in this commit will save the
location as an absolute path. This way the user will have full information
regarding the location of the file where a variable was changed and the
line withing the file.

[YOCTO #5562]

(Bitbake rev: df9e22901555b06fef308f7136547f2c47ccec35)

Signed-off-by: Marius Avram <marius.avram@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-03-21 13:51:51 +00:00
Richard Purdie
f5f3ef7b09 bitbake: parse/ConfHander/BBHandler/utils: Fix cache dependency bugs
Currently bitbake only adds files to its dependency list if they exist.
If you add 'include foo.inc' to your recipe and the file doesn't exist,
then later you add the file, the cache will not be invalidated.

This leads to another bug which is that if files don't exist and then
you add them and they should be found first due to BBPATH, again the
cache won't invalidate.

This patch adds in tracking of files we check for the existence of so
that if they are added later, the cache correctly invalidates. This
necessitated a new version of bb.utils.which which returns a list of
files tested for.

The patch also adds in checks for duplicate file includes and for now
prints a warning about this. That will likely become a fatal error at
some point since its never usually desired to include a file twice.

The same issue is also fixed for class inheritance. Now when a class
is added which would be found in the usual search path, it will cause
the cache to be invalidated.

Unfortunately this is old code in bitbake and the patch isn't the
neatest since we have to work within that framework.

[YOCTO #5611]
[YOCTO #4425]

(Bitbake rev: 78d285871e4b8c54ccc4602d571e85f922e37ccd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-11-30 22:26:30 +00:00
Martin Jansa
aa0b237144 bitbake: parse: Return IOError when including file with absolute path
* resolve_file was behaving different when relative and absolute
  paths were passed to it

* include relative-path/non-existent-file.inc
  works correctly resolve_file throws IOError, BBHandler.py:handle()
  doesn't catch it, ConfHandler.py:include() catches IOError and shows:
  DEBUG: CONF file 'relative-path/non-existent-file.inc' not found
* include /absolute-path/non-existent-file.inc
  was failing, because resolve_file just returns fn,
  BBHandler.py:handle() calls bb.parse.mark_dependency(d, abs_fn)
  which throws:
  OSError: [Errno 2] No such file or directory: '/absolute-path/non-existent-file.inc'
  and parsing fails.
  Ad isfile() test for absolute fn and throw IOError to make
  resolve_file behavior consistent for both paths.

* I know we had some issues with -b relative-path-to-recipe.bb and
  absolute path, so consider this patch only as RFC and documentation of
  this problem

* Catch OSError too in ConfHandler.py:include() e.g. in case the file exists, but user
  cannot read it or something like that.

(Bitbake rev: b0bbd89a4f0b98fa1ab28b8e0526cd9ddb76fa57)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-26 11:29:46 +01:00
Richard Purdie
4a081b5a52 bitbake: lib: Clean up various file access syntax
Python 3 is stricter about how files are accessed. Specficially:

 * Use open(), not file()
 * Use binary mode for binary files (when checksumming)
 * Use with statements to ensure files get closed
 * Add missing file close statements

(Bitbake rev: 9f08b901375ba640f47596f1bcf43f98a931550f)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-05-09 22:28:04 +01:00
Olof Johansson
423dd2f7cd bitbake: ConfHandler: Use re.X to make long regexp more readable
The __config_regexp__ in ConfHandler is quite long, and using re.X to
break the expression onto several lines make it a bit easier to read.

(Bitbake rev: 54dce9e14ab0657d76f0d0ae22eef7fab8e8950d)

Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-02-15 12:12:36 +00:00
Richard Purdie
83e1015684 bitbake: ConfHandler: Improve regexp to fix mis-parsing of += and no whitespace
If you have:

FOO = "a"
FOO += "b"
FOO+= "c"

The expected result is "a b c" however we were seeing "a b" with the FOO+
variable being assigned the value "c". This isn't the expected result.

We need to make the name part of the variale non-greedy so that any + character
becomes part of the operator. This patch does that. I compared the configuration
in OE-Core before and after the change and only the test case changed.

[YOCTO #3834]

(Bitbake rev: 2cd8d7fd12a646e6516e2c985e6a54121d19eb59)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-02-14 10:46:26 +00:00
Constantin Musca
1e50274f55 bitbake: parse_py: add ~ to config_regexp
- enable the '~' character in bitbake variables

(Bitbake rev: 7c15ff1d50d7b601414f1d55c90e3c59981a0876)

Signed-off-by: Constantin Musca <constantinx.musca@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-01-22 15:35:06 +00:00
Richard Purdie
a5358511b1 bitbake: BBHandler/ConfHandler: Merge fix for multiline comments
This was meant to be squashed into the previous commit for multiline comment
handling. It fixes the case the commented multiline is followed by an empty
line which was resulting in a traceback instead of a sane error message.

(Bitbake rev: 7e7d692e244fe8dca533f842ca143b9c821e317c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-01-21 11:01:53 +00:00
Richard Purdie
73dc22b728 bitbake: bitbake: BBHandler/ConfHandler: Improve multiline comment handling
Faced with an expression like:

 # Some comment \
 FOO = "bar"

what should bitbake do? Technically, the \ character means its multiline and
currently the code treats this as a continuation of the comment. This can
surprise some people and is not intuitive.

This patch makes bitbake simply error and asks the user to be clearer
about what they mean.

(Bitbake rev: 589d31ce41e019ee6a7cb6527d67bc76c0b6382a)

(Bitbake rev: 79c00fabe08b4c210a3bd81cfaffbc47ffdc2e2b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-01-21 10:55:45 +00:00
Richard Purdie
b85c30bb7d bitbake: parse/ConfHandler: Add negative lookahead to spot some quoting problems
Syntax like:

FOO = "bar" # eek"

would result in FOO taking the value 'bar" #eek' which is clearly
not the intention. Whilst our metadata is riddled with mixtures of even
quotes like:

FOO = "d.getVar("X")"

odd numbers of quotes seem rare. This patch adds detection of one odd
quote which we don't have any of in OE-Core so it seems a valid sanity
improvement.

(Bitbake rev: 5f892d9b083550e20e37576070ec7d1a94cc88fe)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-09-28 11:15:04 +01:00
Richard Purdie
302f2cc8ce ConfHandler.py: Add a hook for config parsing
To make the UI settings take effect, we need to hook at the end of each
config file parsing and set UI specific values.

(Bitbake rev: f54e733c7863110896f43900d9e4e791602f9d65)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-04-13 12:04:18 +01:00
Richard Purdie
ff73b02a72 meta/classes: Convert to use appendVar and appendVarFlags
(From OE-Core rev: 3b57de68e70e77dbc03c0616a83a29a2e99e40b4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-03-05 10:36:53 -08:00
Richard Purdie
0cfceaa53b parse/ConfHandler: Fix enthusiatic export regexp matching
The export regexp was only meant to catch values like:

export VARIABLENAME

however after the stricter quoting patch was applied, it was also matching
variables like:

export BAR=foo

and setting the export flag on a variable called "BAR=foo". The = character
is an invalid variable name character. This patch tightens up the regexp
match so it only matches the intended character set and only matches variable
names.

(Bitbake rev: 6d1765c2eac8c1958ceb9c81d55d04a9bc961cb1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-02-29 13:12:12 +00:00
Richard Purdie
f60a5d159b bitbake/ConfHandler: Be more strict about variable quoting
Currently, bitbake will accept variables in the forms:

X = 1
X = '1 \

X = "1"
X = '1'

which will all set X=1. This patch removes the first two possibilities
and makes quoting mandatory. There is little metadata out there which
doesn't quote properly and bitbake will exit with an error about the
exact line number and file with any problem so users can easily identify
and fix issues. OE-Core has already been checked/fixed.

The motivation for this is being able to give sane errors if a user
does something like:

IMAGE_INSTALL += # tslib mtd-utils"

which currently gives a really nasty failure.

(Bitbake rev: a8ae80741fea5e0ec0fb9a52a963a4baa38d2564)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-02-28 12:33:20 +00:00
Paul Eggleton
99d326a818 bitbake: add file and line number to ParseError
Ensure that a file and line number are reported for ParseError where
possible. This helps particularly in the case of inherit and require
which previously did not report either of these upon failure.

(Bitbake rev: f588ba69622a2df35417ced184e56c79ac1b40d5)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-02-23 22:52:15 +00:00
Richard Purdie
4cd9671078 bitbake: Update users of getVar/setVar to use the data store functions directly
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-11-27 10:35:30 +00:00