We need to be able to support Python 3 so that we can parse master of
OE-Core with bitbake (which now requires it). This now means the
interface itself and the update script require Python 3.4+.
Part of the implementation for [YOCTO #9704].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow updating multiple branches, and if no branches are specified,
update all branches that have a new "updates_enabled" flag field set to
True. This avoids the need to have a separate shell script which runs
update.py for each branch (and thus has hardcoded knowledge of each
active branch in the index, i.e. it needs to be kept up-to-date in
addition to the database.)
The migration will default updates_enabled to True for all branches so
if you wish to take advantage of this functionality, the flag will need
to be set to False for any branches that shouldn't be updated.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to try to avoid problems with leaking memory, context bleeding
from one layer to another, and lay the groundwork for supporting
updating all branches in one operation, split the updating of a single
layer out to its own internal script. This means that the tinfoil
instantiation and metadata parsing is in a completely separate process
per layer.
Implements [YOCTO #9647].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Display blacklist information for recipes in the recipe details,
as well as the recipe search page, and layer page tables. This
information is pulled from the PNBLACKLIST variable. Includes a
hover text containing the reason for blacklist labels.
Changes to Django and Layerindex files:
- models.py, update.py
- migration file
- templates and static content (stylesheet)
[YOCTO #7855]
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
The code to populate the inherits information was using a function
designed to get the recipe name from a file path, but unlike recipe
filenames, the underscore isn't treated as special in class filenames
and in fact it's quite common to use underscores there; we were ending
up with items in the inherit list such as "populate" and "rootfs". Use
the standard python os.path functions to do the splitting instead and
avoid this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
To identify image recipes and provide inheritance data for recipes, an
inherits field was added to the recipe model, and then populated using
refactored data from __inherit_cache. Finally the field was also added
to page templates, along with style changes proposed in attachment, and
an additional style change to display the inherits field as a list in
detailed view. The field skips globally inherited data as proposed.
[YOCTO #7575]
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix up memory leak fix for bitbake in daisy and earlier which did not
have bb.codeparser.codecache.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We're kind of abusing some bitbake internals here and the result is that
there are a bunch of global lists and caches that simply grow as we
parse more layers, until available memory is exhausted. We don't care
about the contents of any of these within the layer index update script
so just clear them out between layers.
Should fix [YOCTO #7663].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When we did a full reload this list wasn't getting cleared for the next
layer, and we look at the contents of the list unconditionally later to
determine which files need to be added. Given that the list items are
full paths, this is unlikely to have caused any problems; however it
could possibly have been an issue for nested layers with the right
update order, or lead to issues if the usage of the list changes in
future.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Some layers, such as meta-intel, also contain other layers. We don't
want recipes/classes/appends in those child layers to appear in the
parent layer so ignore any that appear within subdirectories that
contain a conf/layer.conf file.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
There was a logic error here - we were checking to see if the layer
directory existed before actually checking out the branch, which means
we were looking at the state of whatever branch was previously checked
out. Replace this with some validation of the last revision check and an
additional check for the directory after checking out the branch.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
With BSPs being "retired" e.g. in meta-intel, it is possible for layers
to not exist on the master branch; since this is legitimate we need the
layer index to handle it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Recently the -r/--reload option was changed to preserve existing recipe
data, so that it could be used to load values into newly added fields;
however we still need an option to load recipe data from scratch for
testing purposes (e.g. in combination with --nocheckout) so add a
--fullreload option to do this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to test updating we need to be able to check out a previous
revision of the repository and have the update script load from that
without checking out the latest revision, so fix the --nocheckout
option to work properly.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Avoid the possibility of these two clashing especially when multiple
branches are being used.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script for doing a one-time import of OE-Classic recipe
information, so comparisons against OE-Core can be performed; this
is stored using a new ClassicRecipe model supporting additional fields
for tracking migration status. The migration status fields can be
updated as well as viewed and summarised in graph format.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The recipe detail pages use the recipe record id, so in case users
bookmark links to these pages, try to preserve the id when a recipe
upgrade occurs (which usually shows up as a delete and an add). This
also works when doing a full update with -r so that we can capture the
newly recorded items (bbclasses, bbappends) without having all the
recipe id values change.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
To allow re-use outside of the update script, split out parsing setup
code to a new recipeparse module. Also split out runcmd, get_layer,
get_branch and logger_create functions to a separate utils module.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Some layers don't use the conventional branch name (i.e. the same name
as the core branch). Add a field so we can record the actual branch name
and use this within the update script.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Collect bbappend/bbclass info during the update process and display it
on the layer detail page.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Show BUGTRACKER in the recipe detail page if set
* Track the other two variables for possible future use
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This allows disabling the checkout of the appropriate branch so that we
can test the update script with fixes to bitbake or the metadata if
needed.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The cooker module has been refactored in BitBake master and this moved
and renamed the _parse() function; change the update script to handle
that (whilst still handling older versions).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Report layer which failed to fetch in error message
* Don't retry fetching a repo if it already failed for another layer
(where more than one layer is in the same repository)
* Exit immediately if all fetches failed
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's easy to forget you need to use -l to specify the list of layers to
update and previously we just ignored unused arguments, so check for
them and error out if present.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Avoid picking up machine files in sub-layers
* Skip scanning of .git directory when doing full refresh
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fall back to NAME if DESCRIPTION is not specified and remove more
preamble from the string.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This ensures recipes that failed to parse sort properly instead of all
appearing at the start of the search results.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We don't go through the normal code path where this directory would be
created, and if it doesn't exist an error will be thrown in
insane.bbclass when any QA issue is logged; so create it manually.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Rather than letting BitBake error out, just check for existence of this
file and print a single error line if it doesn't exist.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we're creating a LayerBranch record, it's helpful to copy the
maintainer and dependency records in from master as they're usually
going to be the same; if they're not the user can always change them
afterwards.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
So it turns out that one or two layers have changed in structure between
branches, so we need to be able to specify this on a per-branch basis.
Good times...
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Don't assume the current directory contains our conf/local.conf,
explicitly use the directory above where the script is located.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Change the data structure to support multiple branches. At the top level
there is a set list of Branch objects, and then a LayerBranch object
between each layer and the maintainers, dependencies, recipes and
machines, so that the set of each can be different per branch. The
branch is a session option, and can be selected via a drop-down that is
shown for all pages.
Additionally, with this change we avoid the need to run the update
script within a build environment set up with oe-init-build-env - since
we need a specific version of BitBake per branch we now use our own copy
of BitBake which is fetched by the script itself. The update script will
need to be called multiple times however - once per branch.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If there is more than one layer in a repository, we want the latest
commit in the subdirectory not the entire repository (since we use it
to determine the last commit date which should show how "fresh" the
layer is).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we aren't doing a full refresh, when changes to an included/required
file or an inherited class occur we need to update all of the files that
depend on them. BitBake records these dependencies, so read them and
record them against each recipe.
At the moment this only handles dependencies within the same layer; that
is probably sufficient to handle most changes for the purposes of
extracting the data this application cares about. A future improved
solution will probably involve making use of BitBake's cache rather than
and parsing all layers and their dependencies rather than individual
recipes.
This change also adds -x/--nofetch and -n/--dry-run options for
debugging as well as some further debug messages.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
"Fatal" errors during parsing a single recipe (such as finding legacy
staging) shouldn't stop us from parsing the rest of the layer. To do
this we need to catch BaseException instead of Exception as fatal errors
end up calling sys.exit.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Parse layer.conf files so that BBPATH is set correctly as well as any
other custom variables. This required the repository fetching to be
split out of the update process and done first so that we're sure we
have the data for the other layers.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Look for conf/machine/*.conf and add a record for each file found. These
are displayed in the layer detail page if any are present.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Set BBPATH so that files from the layer can be found when recipes use
include/require pointing to files with a specific path, or use
inherit to inherit a class in the layer.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
meta-ti has a recipe in it that causes parsing to fail with a fatal
error due to LICENSE not being set; for the purposes of the layer index
just set a default value for LICENSE so that this doesn't break parsing.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
By default, if DESCRIPTION is not set it takes the value of SUMMARY.
However, within this tool it is useful for us to be able to tell when
DESCRIPTION has not been set so we want it to be blank if that is the
case.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The default value of HOMEPAGE ('unknown', set from bitbake.conf) is not
particularly helpful - we want it to be blank if it's not set, so just
set it to blank before parsing each recipe.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
