Commit Graph

199 Commits

Author SHA1 Message Date
Richard Purdie
e72d641a99 bitbake: cooker/cache: Drop mc 'default' string value
The string value "default" for the default multiconfig is confusing since an
empty string is used pretty much everywhere in the code. Remove the few
remaining references to that to standarise.

This affects the default value of BB_CURRENT_MC and does have an impact
on metadata, particulalry bitbake.conf in openembedded-core. That said, the
number of bugs we'll avoid by trying to make "default" back to "" within
bitbake's code make fixing those extremely worthwhile.

(Bitbake rev: 0fa0d8d764bbeb8a44c47f79d7b849068d565199)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-01-25 11:30:51 +00:00
Richard Purdie
76d24b00ff bitbake: checksum/fetch2: Switch from persist_data to a standard cache file
The sqlite connection handling is causing problems with python 3.13. The
connection can be closed at gc time which causing warnings and those
can appear at 'random' points and break output, causing weird failures
in different tinfoil tools and other tests.

Using sqlite as an IPC was never a great idea so drop that usage entirely
and just use the standard cache mechanism we already have for other
situations.

(Bitbake rev: fdc55bb649cb77456d0ac48a9600ef289a52af18)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-10-09 13:04:30 +01:00
Richard Purdie
9a25a38ffe bitbake: codeparser: Allow code visitor expressions to be declared in metadata
Allow the metadata to define code visitor expressions which mean that
custom dependencies can be handled in function libraries.

An example is the qa.handle_error function in OE which can set something
like:

"""
def handle_error_visitorcode(name, args):
    execs = set()
    contains = {}
    warn = None
    if isinstance(args[0], ast.Constant) and isinstance(args[0].value, str):
        for i in ["ERROR_QA", "WARN_QA"]:
            if i not in contains:
                contains[i] = set()
        contains[i].add(args[0].value)
    else:
        warn = args[0]
        execs.add(name)
    return contains, execs, warn

handle_error.visitorcode = handle_error_visitorcode
"""

Meaning that it can have contains optimisations on ERROR and WARN_QA
instead of hard dependencies.

One drawback to this solution is the parsing order. Functions with
visitorcode need to be defined before anything else references them
or the visitor code will not function for the earlier references.

(Bitbake rev: 5bd0c65c217394cde4c8e382eba6cf7f4b909c97)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-08-29 21:58:19 +01:00
Richard Purdie
4eedb58a28 bitbake: cache: Drop unused function
I can't spot any users of this function and it is poking at variables
inside cooker that could and are about to change so drop it.

(Bitbake rev: 52491808706e9e58b5e6b59d2d792353d77c8b66)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-08-13 15:47:55 +01:00
Robert Yang
ac40cb5ee2 bitbake: cache: Remove invalid symlink for bb_cache.dat
The bb_cache.dat might be an invalid symlink when error happens, then
os.path.exists(symlink) would return False for it, the invalid symlink
wouldn't be removed and os.symlink can't update it any more.

Use os.path.islink(symlink) can fix the problem.

(Bitbake rev: 1387d7b9ee3f270488f89b29f36f9f240e44accc)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-06-25 11:51:45 +01:00
Peter Kjellerstedt
2a2320001e bitbake: cache: Simplify virtualfn2realfn()
Also correct the description of variant2virtual().

(Bitbake rev: d766e9bd22ec6392fbf1694eea5032b9d09f1949)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-23 12:06:06 +00:00
Richard Purdie
695998f921 bitbake: cooker: Add FILE_LAYERNAME variable containing the layername for a recipe
There are times when it would be useful for code to know which layer
(or collection in old bitbake terms) it is contained within.

Add support for FILE_LAYERNAME to be set by bitbake when parsing a recipe
so that it is possible to determine this. To do it, we need to pass data
from the cooker into the recipe endpoints, since only the top level cooker
information knows about the layer structure which makes the patch a bit
painful.

The idea is that this would make layer overrides possible:

OVERRIDES .= ":layer-${FILE_LAYERNAME}"

which then opens possibilities like:

WARN_QA:append:layer-core = " patch-fuzz"

as an example where OE-Core could enable specific QA tests only for that
specific layer.

(Bitbake rev: 7090a14b0035842112d073acf7f2ed1a01fdeccf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-05-25 13:16:24 +01:00
Robert Yang
178c6e0de8 bitbake: cache: Make EXCLUDE_FROM_WORLD boolean
Fixed:
Set EXCLUDE_FROM_WORLD = "0" in recipe_A:
$ bitbake world -g

Check pn-buildlist, the recipe_A won't be built, this patch fixes the problem:
EXCLUDE_FROM_WORLD = "1": Not build in world
EXCLUDE_FROM_WORLD = "0": Add back to world build

(Bitbake rev: 3f4ede2d67a2d75d3faf8887f90371bd7554d492)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-03-30 10:51:54 +01:00
Richard Purdie
557aab2158 bitbake: cache/codeparser: Switch to a new BB_CACHEDIR variable for cache location
Currently the codeparser cache is set from CACHE, which is typically in
bitbake.conf which means we can't read/write any cache until it is found/read.
We may well have python expressions to parse before that happens.
The net result is suboptimal functioning of the codeparser cache since it will
often be invalidated by data that is never written.

This patch changes the codeparser and filechecksum caches to use BB_CACHE as
their setting and defaults it to ${TOPDIR}/cache.

The patch doesn't change where the "persistent" data such as prserver and
hash-equiavalance resides (PERSISTENT_DIR) or where the metadata parsing
cache resists (still currently CACHE). I've left those for a later patch.

The patch does ensure data parsed by the core datastore parsing calls is
written back since this is now much more useful after this change.

(Bitbake rev: ee89ade5b5a4cf9c53f336d8b800e06fbe436628)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-26 21:50:31 +00:00
Richard Purdie
4a94cf21c5 bitbake: cache: Only write files if we have data
By writing the cache files only if there is data to write, we can save a
bit of time.

(Bitbake rev: abeff1f80bb1c690b92d535d472dff9df7a56067)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-24 21:59:44 +00:00
Richard Purdie
42487f069b bitbake: cache: Drop reciever side counting for SiggenRecipeInfo
Joshua Watt pointed out maintaining the counting on both sides of the
connection isn't needed. Remove the receiver side counting and simplify
the code, also allowing errors if the counts do go out of sync.

(Bitbake rev: aeacfd391903fe68ae600470fc2bbad0502d401e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-30 15:07:19 +00:00
Richard Purdie
6549f58cd6 bitbake: cache/siggen: Fix cache issues with signature handling
There is a bug in the current cache code where the restored data structures
were "inverted" leading to some very weird issues, not sure how anything worked
like that, this patch fixes it.

Also fix some issues with multiconfig cache ordering problems by resetting
the stream counters when appropriate.

(Bitbake rev: cd06beb948eff5eaf2d474f5b127d51a61b0b2ef)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-29 00:05:45 +00:00
Richard Purdie
7d010055e2 bitbake: cache: Allow compression of the data in SiggenRecipeInfo
The data in SiggenRecipeInfo is large and has a lot of duplication. The size
causes a few problems, impacting:

 - bitbake's overall memory usage
 - the amount of data sent over IPC between parsing processes and the server
 - the size of the cache files on disk
 - the size of "sigdata" hash information files on disk

The data consists of strings (some large) or frozenset lists of variables.
To reduce the impact we can:

a) deplicate the data
b) pass references to the object on the second usage
   (e.g. over IPC or saving into pickle).

This patch does this for SiggenRecipeInfo mostly behind the scenes
but we do need a couple of reset points so that streamed data is written
correctly on the second usage.

(Bitbake rev: 9a2b13af483c20763d6559a823310954884f6ab1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-21 14:15:26 +00:00
Richard Purdie
8fe5f307e2 bitbake: siggen/cache: Optionally allow adding siggen hash data to the bitbake cache
Being able to track siggen hash construction data can be useful for cache
debugging. For now, add an extra cache class which contains this information.
It can be enabled in the same way as the hob data cache through a feature flag
to cooker. This allows us to experiment with the data without carrying larger
patches around and ultimately may allow use to have a hash mismatch debugging
mode that is more easily enabled.

(Bitbake rev: 0736a8a03da8b774fafbd28f746bef4705378049)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-08 10:49:53 +00:00
Richard Purdie
c14d8b9b6b bitbake: cache/siggen: Simplify passing basehash data into the cache
The basehash data is really internal bitbake data and passing an object
directly is more efficient than creating and then extracting variables.

This will match the format of other data we may optionally wish to
store in the cache so more to the more efficient method. Nothing I
can see is using this data today (and nothing should be).

(Bitbake rev: e621093a1bf37cd75ede3fb77ab6845556870fc7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-08 10:49:53 +00:00
Richard Purdie
5673f731f7 bitbake: cache: Drop support for not saving the cache file
We don't test not using the cache and I'm not aware of anyone using this, it
would be hard to with modern bitbake.

Drop the conditional code and simply error if CACHE isn't set.

(Bitbake rev: 063ffe699bc5fc23174643dfedb66864cacfcff8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-11-20 08:31:28 +00:00
Richard Purdie
82b5cdad0f bitbake: cache: Drop unused function
At this point users appear to all call add_info directly. Failing
to do that means the file dependency tracking code isn't active
so would cause problems. Therefore drop the unused function.

(Bitbake rev: 6b24efc0f4d19738d96754280e70bc493005167d)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-11-20 08:31:28 +00:00
Richard Purdie
9206614d79 bitbake: cache: Drop broken/unused code
Some parts of functions in Cache() were broken and unused, there was
also a totally unused function. This was historical as a result of the
cooker parsing process needing to handle cached entries in the main
thread but parsing actions in seperate processes.

Document the way it works, update the function name to be clear about
what it now does and drop the old code which was unused.

(Bitbake rev: af83ee32df85c8e4144f022a1f58493eb72cb18e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-11-20 08:31:28 +00:00
Richard Purdie
3a4aeda0fa bitbake: cache/cookerdata: Move recipe parsing functions from cache to databuilder
When 'NoCache' was written, databuilder/cookerdata didn't exist. It does
now and the recipe parsing functionality contained in NoCache clearly
belongs there, it isn't a cache function. Move those functions, renaming
to match the style in databuilder but otherwise not changing functionality
for now. Fix up the callers to match (which make it clear this is the right
move).

(Bitbake rev: 783879319c6a4cf3639fcbf763b964e42f602eca)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-11-20 08:31:28 +00:00
Richard Purdie
bcf935cd1b bitbake: cache/siggen: Add unihash cache copy function
We see rare failures in eSDK generation with zero sized unihash cache files. This is
almost certainly due to races in the cache file being updated. Add a copy function
where the cache file can be copied with the lock held to avoid this.

(Bitbake rev: 9e72a3915e36cb843037040cb68a82077436dbef)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-02 12:28:21 +01:00
Roland Hieber
f564bf2bbd bitbake: cache: correctly handle file names containing colons
File names containing colons cause split() to return a list with more
than two elements, which will lead to a stack trace ending in:

    ValueError: too many values to unpack (expected 2)

Split only once at the last colon, thereby making sure that only two
elements are returned.

(Bitbake rev: a70a7376a8708bde07959deb5d5842d7f84ee5f8)

Signed-off-by: Roland Hieber <rhi@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-05-08 14:20:43 +01:00
Richard Purdie
f07a097213 bitbake: cache/ConfHandler: Drop TOPDIR/chdir manipulations
This code has been unchanged since 2006 apart from attempts to optimise
performance by minimising chdir() calls.

There is no reason the modern bitbake parser should be changing directory
all the time. We did have some path assumptions in the mists of time but
those were resovled and the code is deterministic and doesn't depend on
cwd now for parsing. We can therefore drop the changes in directory.

Also, TOPDIR is now being set by cookerdata in all cases so we don't
need the fallbacks in this code (which was used to effectively initialise
a value). We don't need to change TOPDIR when parsing a recipe, that makes
no sense. If we stop all the other messing around, we don't need to expand
TMPDIR either.

These changes have the potential to break some obscure use cases such
as an anonymous function assuming the current working directory, or some
case which depends on TOPDIR changing but I believe any such uses should
be fixed at this point.

(Bitbake rev: add5d488e1d6607a98441836075d01cb1dc9c0fa)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-11-15 11:03:25 +00:00
Alexander Kanavin
5c24982cc9 bitbake: bitbake: correct the collections vs collections.abc deprecation
This becomes a hard error in python 3.10.

(Bitbake rev: ae219e1f7460077f4492b31ac91cef4cf9b17277)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-09-17 07:26:23 +01:00
Richard Purdie
2abf8a699e bitbake: bitbake: Switch to using new override syntax
This change updates the datastore to use the new override syntax using
colons instead of underscores exclusively. It is expected layers would
have to be converted to work with bitbake after this change.

Supporting mixed syntax isn't possible, it is only feasible to have
one internal representation of overrides.

Whilst we can't warn for every possible override that may be set in the
old format, show errors for _append/_prepend/_remove since those should
never be present.

(Bitbake rev: 7dcf317cc141dc980634f8c18bfa84f83e57206a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-08-02 15:44:10 +01:00
Tomasz Dziendzielski
5386b3db50 bitbake: runqueue: Print pseudo.log if fakeroot task failed
Currently if pseudo fails we can only see the path to pseudo.log. If we
have no access to server and can only rely on bitbake log then debugging
becomes impossible. This printing needs to be added in runqueue level,
not inside task execution, because in some cases task fails with pseudo
abort really early and we don't even see any log.

In this change I'm adding pseudo log printing in every fakeroot task
failure that logged `mismatch`, `error` or `fatal` to logfile, because
we have no other way to communicate with pseudo if it failed or not.
Only lines from last pseudo server execution will be printed.

(Bitbake rev: e7c664a947903ed7b868abef62af2ff5f8ef0dc6)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com>
Signed-off-by: Adrian Walag
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Mikolaj Lasota <mikolaj.lasota@protonmail.com>
Signed-off-by: Wiktor Baura <wbaura@gmail.com>
Signed-off-by: Kamil Kwiek <kamil.kwiek@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11 14:04:45 +00:00
Joshua Watt
75f87db413 bitbake: logging: Make bitbake logger compatible with python logger
The bitbake logger overrode the definition of the debug() logging call
to include a debug level, but this causes problems with code that may
be using standard python logging, since the extra argument is
interpreted differently.

Instead, change the bitbake loggers debug() call to match the python
logger call and add a debug2() and debug3() API to replace calls that
were logging to a different debug level.

[RP: Small fix to ensure bb.debug calls bbdebug()]
(Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-10 23:48:16 +00:00
Tomasz Dziendzielski
1d2fe91db5 bitbake: lib/bb: Don't treat mc recipe (Midnight Commander) as a multiconfig target
When we run `devtool build mc` recipe's task dependencies are expanded
to "mc:do_populate_sysroot" where "mc" name is treated as multiconfig
and "do_package_sysroot" as multiconfigname.

| ERROR: Multiconfig dependency mc:do_populate_sysroot depends on
| nonexistent multiconfig configuration named do_populate_sysroot

(Bitbake rev: 3ce4b2caccfe608a54dff159459f3687ea610597)

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-02-06 09:12:00 +00:00
Peter Kjellerstedt
8939086648 bitbake: cache: Make CoreRecipeInfo include rprovides_pkg for skipped recipes
This will be needed by SkippedPackage in the cooker.

(Bitbake rev: 93d01614565bd540d05fbc1791dd66e46723d683)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-12-21 22:29:57 +00:00
Joshua Watt
7ae3a24079 bitbake: bitbake: cache: Remove bad keys() function
Removes the keys() function from the MulticonfigCache. This appears to
be a leftover from before the class inherited from collections.Mapping,
is now unnecessary, and was outright incorrect.

(Bitbake rev: 5f37b6d2829fcac1f16602d9697f8bfbcb65ff62)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-11-24 15:26:12 +00:00
Maxime Roussin-Bélanger
3ef540749e bitbake: cache: remove unused variables.
(Bitbake rev: 28475ae752967b0a14ed2554c1b835e96390f850)

Signed-off-by: Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-10-30 13:26:16 +00:00
Richard Purdie
1560a4b0cb bitbake: fetch2: Drop globbing supprt in file:// SRC_URIs
Globbing in file:// urls is terminally broken. Currently when its used, the
file checksum code is basically bypassed. This means changes to the source
files don't change the task checksum, the task doesn't rebuild when the
inputs change and things generally break.

To make globbing work generically, we'd have to scan the file system for
all possible matches to the glob and log whether they exist or not. We can't
simply log the files which exist, we have to also know which files could
later exist and influence the choice of file so we know when to reparse.

For a simple file://xxx/*, this could be done but for bigger patterns,
it becomes much more problemtic. We already support file://xxx/ in urls.

So, lets decide we'll not support globs in file://urls. Worse case users
can put files in a directory and reference that, moving files into place
if needed.

Remove all the glob special cases (see the comments if anyone doesn't
believe this is terminally broken) and error to the user if they have
such urls.

(Bitbake rev: 0c9302d950c6f37bfcc4256b41001d63f668bdf7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-08-26 09:05:38 +01:00
Joshua Watt
fced0b8d7c bitbake: cache: Bump cache version
The layout of the cache data hasn't changed, but the cache has now been
split into different files for multiconfig. If a user pulls in these
changes, it's possible that their base cache will still contain the
combined multiconfig cache entries, which are now unexpected and
generate errors like:

 Unexpected multiconfig: foo

Bumping the version fixes this since the old cache data won't be
considered valid anymore.

(Bitbake rev: 1082188ce633ec6891c961d726f584b3f1259941)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-19 23:18:18 +01:00
Joshua Watt
128621161c bitbake: bitbake: cache: Fix error message with bad multiconfig
The virtualfn variable is not defined, the filename is what should be
shown instead.

(Bitbake rev: 1f9d2c21db3a1ad2ab13dfebd2f8e9a7c3682ee2)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-15 14:55:25 +01:00
Joshua Watt
c3271dca21 bitbake: bitbake: cache: Fix error when cache is rebuilt
It is expected that load_cachfile() returns an integer indicating how
many entries were loaded from the cache. In the event the cache needs to
be rebuilt, 0 must be returned to prevent python from attempting to add
an None and an integer together.

(Bitbake rev: 3459d98fbc280637ecb36961bda8436818ee51e5)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-10 12:30:01 +01:00
Joshua Watt
59fb65f742 bitbake: bitbake: cache: Cache size optimization
Now that there is a cache object per multiconfig, it is not necessary
for each cache object to parse all other multiconfigs. Instead, each
cache now only parses the files for it's multiconfig.

(Bitbake rev: 3c5c7346adf4ca7ec761c08738f12401ba75b7c8)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-10 12:30:01 +01:00
Joshua Watt
3ec9d5774c bitbake: bitbake: cache: Improve logging
Improves the logging of Cache objects by prefixing the log messages with
the multiconfig name of the cache, so as to distinguish between multiple
instances of the class. Also adds a more log messages.

(Bitbake rev: 74fd10b33c66f4142d6eff6531200f7620a06ae0)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-10 12:30:01 +01:00
Joshua Watt
f8163c22f4 bitbake: bitbake: cache: Use multiconfig aware caches
Splits the parsing cache to maintain one cache per multiconfig instead
of one global cache. This is necessary now that the files and appends
can vary for each multiconfig. A bb.cache.MulticonfigCache
dictionary-like proxy object is created instead of a single
bb.cache.Cache object. This object will create and properly initialize
bb.cache.Cache object for each multiconfig, and each of these caches has
a dedicated cache file with a name based on the multiconfig.

(Bitbake rev: 5272f2489586479880ae8d046dfcdbe0963ee5bb)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-10 12:30:01 +01:00
Joshua Watt
b9fdb6a426 bitbake: bitbake: cooker: Split file collections per multiconfig
Splits the cooker to track a collection per multiconfig instead of a
single collection for all multiconfigs. Practically speaking, this
allows each multiconfigs to each have different BBMASKs that apply to it
instead of each one using the mask specified in the base configuration.

(Bitbake rev: dd6d8eca2027f8d9be8a734a493227b440075e49)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-10 12:30:01 +01:00
Richard Purdie
8b1636763d bitbake: cache: Fix performance problem with large numbers of source files
Some companies are using large numbers of patch files in SRC_URI.
Rightly or wrongly that exposes a performance problem where the code
does not handle the large string manipulations in a way which works
efficienty in python.

This is a modified version of a patch from z00539568
<zhangyifan46@huawei.com153340508@qq.com which addresses the performance
problem. I modified it to use a more advanced regex, retain the "*" check
and cache the regex.

[YOCTO #13824]

(Bitbake rev: c07f374998903359ed55f263c86466d05aa39b68)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-04-24 14:31:42 +01:00
Frazer Clews
0ac5174c7d bitbake: lib: remove unused imports
removed unused imports which made the code harder to read, and slightly
but less efficient

(Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697)

Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-19 13:31:05 +00:00
Richard Purdie
3e108b7017 bitbake: cache: Lower debug level for wold build messages
These messages spam the logs for no good reason, they were useful for debugging
a particular problem long ago but are distracting noise now. Disable them.

(Bitbake rev: 1a9247c468cf09da60e5d396ccb81e950841c99e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-02 16:43:01 +00:00
Richard Purdie
4fbb862cdc bitbake: siggen: Clean up task reference formats
Currently siggen uses the format "<filename>.<taskname>" for referencing tasks
whilst runqueue uses "<filename>:<taskname>". This converts to use ":" as the
separator everywhere.

This is an API breaking change since the cache is affected, as are siginfo files
and any custom signature handlers such as those in OE-Core.

Ultimately this will let us clean up and the accessor functions from runqueue,
removing all the ".rsplit(".", 1)[0]" type code currently all over the place.
Once a standard is used everwhere we can update the code over time to be more
optimal.

(Bitbake rev: 07e539e1c566ca3434901e1a00335cb76c69d496)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:32 +01:00
Richard Purdie
5f56244092 bitbake: cache: Add SimpleCache class
This adds a simple version of the MultiProcessCache which can be used to
save and load cache data, useful for a new usecase we have in
sigdata/runqueue.

(Bitbake rev: 19a6e35600ae6d2d1bcecca6e68ab8c37674774e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06 11:21:31 +01:00
Robert Yang
b70993cf92 bitbake: cache: Create a symlink for current cachefile
So that people or other tools can easily know which one is being used, just
like what we did for run.do_task and log.do_task, otherwise, we have no way
to know it. I usually use "ls -t", but it isn't reliable since the one which
is being used may not the latest one.

(Bitbake rev: cf286dff653eed542bf347ca46234c224944d5b0)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-19 08:45:08 +01:00
Robert Yang
afc56a43b0 bitbake: cache: Set packages for skipped recipes
The provides and rprovides had been set for skipped recipes, packages are
similar to them (all of them provide something), so also set it. This makes it
easier to figure out the RDEPENDS issues, for example, lmsensors
(lmsensors_3.5.0.bb) RRECOMMENDS lmsensors-config-fancontrol
(lmsensors-config_1.0.bb), but lmsensors-config is skipped for some reasons,
then if we run:

$ bitbake lmsensors
ERROR: Nothing RPROVIDES 'lmsensors-config-fancontrol' (but /path/to/lmsensors_3.5.0.bb RDEPENDS on or otherwise requires it)
NOTE: Runtime target 'lmsensors-config-fancontrol' is unbuildable, removing...
Missing or unbuildable dependency chain was: ['lmsensors-config-fancontrol']
ERROR: Required build target 'lmsensors' has no buildable providers.
Missing or unbuildable dependency chain was: ['lmsensors', 'lmsensors-config-fancontrol']

We had no way to know who rprovides lmsensors-config-fancontrol, we can figure
it out by bitbake/contrib/dump_cache.py after this patch.

(Bitbake rev: 9cf7a5e5a28e676427970a821893e9d930973969)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-15 09:31:48 +01:00
Robert Yang
68467cfb89 bitbake: cache: Remove duplicated lines for provides and rprovides
Whether skip or not, they are always set, so move the lines ahead to avoid
duplicated lines.

(Bitbake rev: c1a8ebb8f83e5108b667f291c924fc2fbd2ac769)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-15 09:31:48 +01:00
Richard Purdie
1f68b0bd98 bitbake: multiconfig: Switch from 'multiconfig' -> 'mc'
After real world use its clear the "multiconfig:" prefix to multiconfig tasks,
whilst clear, is also clumbersome. Switch to use the short version instead.

mcdepends will continue to work with "multiconfig:" for now as well. The commandline
will only accept mc: going forward.

[YOCTO #11168]

(Bitbake rev: 821daf093b76504067a8b77dfa4b181af6ec92b4)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-10 14:46:38 +01:00
Richard Purdie
9501864db8 bitbake: bitbake: Strip old editor directives from file headers
There are much better ways to handle this and most editors shouldn't need this
in modern times, drop the noise from the files. Its not consitently applied
anyway.

(Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
cf9c0be3f6 bitbake: bitbake: Drop duplicate license boilerplace text
With the introduction of SPDX-License-Identifier headers, we don't need a ton
of header boilerplate in every file. Simplify the files and rely on the top
level for the full licence text.

(Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:10 +01:00
Richard Purdie
79834a7144 bitbake: bitbake: Add initial pass of SPDX license headers to source code
This adds the SPDX-License-Identifier license headers to the majority of
our source files to make it clearer exactly which license files are under.

The bulk of the files are under GPL v2.0 with one found to be under V2.0
or later, some under MIT and some have dual license. There are some files
which are potentially harder to classify where we've imported upstream code
and those can be handled specifically in later commits.

The COPYING file is replaced with LICENSE.X files which contain the full
license texts.

(Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04 10:44:04 +01:00