We're kind of abusing some bitbake internals here and the result is that
there are a bunch of global lists and caches that simply grow as we
parse more layers, until available memory is exhausted. We don't care
about the contents of any of these within the layer index update script
so just clear them out between layers.
Should fix [YOCTO #7663].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add default = datetime.now attr to updated field of Branch
model because it fails when try to insert inital data with NULL.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
We add an automatically-updateing "updated" datetime
field to Branch, LayerItem, LayerBranch, Recipe and
Machine models to allow date-based incremental updates
through the API.
The added field does not interfere with the existing
form-based pages (auto_now sets editable to false).
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
This patch enables a read-only REST API for the layerindex
application using Django REST Framework.
The objects of types Branch, LayerBranch and LayerItem are
exposed to queries so that the layerindex application can
function as a Layer Source in Toaster.
The library dependencies are documented in the requirements.txt
file.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
* Fix "sre_constants.error: nothing to repeat" error due to .* inside
group made optional with ? in regex
* Avoid error if maintainer responsibility is not specified
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When we did a full reload this list wasn't getting cleared for the next
layer, and we look at the contents of the list unconditionally later to
determine which files need to be added. Given that the list items are
full paths, this is unlikely to have caused any problems; however it
could possibly have been an issue for nested layers with the right
update order, or lead to issues if the usage of the list changes in
future.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Some layers, such as meta-intel, also contain other layers. We don't
want recipes/classes/appends in those child layers to appear in the
parent layer so ignore any that appear within subdirectories that
contain a conf/layer.conf file.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
There was a logic error here - we were checking to see if the layer
directory existed before actually checking out the branch, which means
we were looking at the state of whatever branch was previously checked
out. Replace this with some validation of the last revision check and an
additional check for the directory after checking out the branch.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If there is a % character in the path to a file / directory, we need to
properly encode it when creating the repo web URL or the link won't
work. Thanks to Khem Raj for reporting this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
With BSPs being "retired" e.g. in meta-intel, it is possible for layers
to not exist on the master branch; since this is legitimate we need the
layer index to handle it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you want to check what the same search returns on a different branch
it makes things easier if you don't have to re-type the search after
selecting the branch.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Recipes without versions in the file name (such as alsa-state.bb)
weren't having their bbappends listed. Use a regex match to fix this.
* Use the prefix from the filename instead of PN, since that's how
BitBake does it
* List version matching bbappends first, then non-matching in muted
colour
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Update d3/nvd3 js files in the process, but hold off on nv.d3.css for
now because the latest version breaks the formatting/fading on tooltips.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Adds a simple way to add a new layer to the database in an automated
manner, given only a URL and optionally a name and subdirectory. If no
subdirectory is given, the root and all first-level subdirectories that
contain conf/layer.conf will be created as layers. Guesses will be made
as to the right values for certain fields, by looking at
README/MAINTAINERS files and github repository information.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the site is configured to use a different core layer then we ought to
use it for the default layer dependency instead of hardcoding
openembedded-core.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This ensures we get a sensible error message prompting the user to set
the subdirectory, instead of a BitBake parse error.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This allows specifying the site title in the database, instead of hardcoding
it in the template. Just change it to something other than "example.com" to
apply your own title.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Recently the -r/--reload option was changed to preserve existing recipe
data, so that it could be used to load values into newly added fields;
however we still need an option to load recipe data from scratch for
testing purposes (e.g. in combination with --nocheckout) so add a
--fullreload option to do this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to test updating we need to be able to check out a previous
revision of the repository and have the update script load from that
without checking out the latest revision, so fix the --nocheckout
option to work properly.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We always need to replace %path% in the URL, not just when there is a
non-blank path to replace it with.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Change Submit layer link into a button
* Ensure tools dropdown menu doesn't show on layer submit/edit page
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's not really completely functional and puts quite a bit of load on
the server when used, so just disable it for now.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Since it's possible that the user will try to get patches while the
update script is running, ensure we show a sensible error if the script
times out waiting for a lock.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Avoid the possibility of these two clashing especially when multiple
branches are being used.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Drop the front page - this just gets in the way. Redirect to the
layers list instead. This has meant adding a touch more text to the
about page and adding the FAQ link to the footer.
* Use a separate navbar to hold the branch selector and the main
top-level pages (Layers, Recipes, Machines) instead of tabs
* All pages depending on a branch selection are now under
branch/<branchname>/ so we don't need to have the branch selection on
every page.
* Use breadcrumbs on recipe detail and layer detail pages instead of
tabs
* Add title to recipe detail page
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Don't start with all records being shown - only do this if the search
button is pressed with no keyword.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script for doing a one-time import of OE-Classic recipe
information, so comparisons against OE-Core can be performed; this
is stored using a new ClassicRecipe model supporting additional fields
for tracking migration status. The migration status fields can be
updated as well as viewed and summarised in graph format.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Having the page content dance about when switching between pages just
because one has a scrollbar and the other doesn't is annoying, so just
force the scrollbar to show all of the time to avoid this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The recipe detail pages use the recipe record id, so in case users
bookmark links to these pages, try to preserve the id when a recipe
upgrade occurs (which usually shows up as a delete and an add). This
also works when doing a full update with -r so that we can capture the
newly recorded items (bbclasses, bbappends) without having all the
recipe id values change.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This provides a way to set "meta" fields (SUMMARY, DESCRIPTION,
HOMEPAGE, BUGTRACKER, SECTION, and LICENSE) for a number of recipes at
once, and then download those changes in the form of one or more patch
files which can be submitted for merging into the layer.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
To allow re-use outside of the update script, split out parsing setup
code to a new recipeparse module. Also split out runcmd, get_layer,
get_branch and logger_create functions to a separate utils module.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Some layers don't use the conventional branch name (i.e. the same name
as the core branch). Add a field so we can record the actual branch name
and use this within the update script.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Collect bbappend/bbclass info during the update process and display it
on the layer detail page.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Show BUGTRACKER in the recipe detail page if set
* Track the other two variables for possible future use
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an index_preference field to enable control over which layer's
duplicate recipes get de-emphasised in the recipe search results.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Save individual field changes into revision comment and display this
comment on the history page. Now we're ready to add a link at the
bottom of every page so the history is easily visible.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This allows disabling the checkout of the appropriate branch so that we
can test the update script with fixes to bitbake or the metadata if
needed.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's not really appropriate to say "Hi Joe Bloggs" at the start of the
email - we only need the user's first name or the login name if the
first name hasn't been filled in.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The cooker module has been refactored in BitBake master and this moved
and renamed the _parse() function; change the update script to handle
that (whilst still handling older versions).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Report layer which failed to fetch in error message
* Don't retry fetching a repo if it already failed for another layer
(where more than one layer is in the same repository)
* Exit immediately if all fetches failed
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's easy to forget you need to use -l to specify the list of layers to
update and previously we just ignored unused arguments, so check for
them and error out if present.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This includes the layer name in layer note additions/changes in the
history page, among other places.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Somehow due to some CSS shenanigans these checkboxes had 0 width in
Chromium and thus were invisible. A style on the td had to be used
because a style on the input doesn't seem to fix it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We shouldn't be looking for this attribute unless the user has logged in
- this was a regression in the recent email case fix.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This avoids showing a 403 error when a user clicks on a review link in
the layer submission notification email but hasn't logged in yet.
Also protect the review list view with a permission check; it's not that
it's sensitive, but we should be consistent with the detail here.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This actually works for complex URLs since we normalise just the path
portion that we are adding instead of trying it on the URL after
composing it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
gitweb, ever vigilant for the slightest transgression in URL syntax,
fails to find the specified path if it is a directory and has a trailing
slash, so we need to avoid adding this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow using e.g. ../README to point to README file in the parent of the
repository subdirectory.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Avoid truncation for slightly longer fields (as happened with PN and
SUMMARY when importing some existing layers).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Avoid picking up machine files in sub-layers
* Skip scanning of .git directory when doing full refresh
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fall back to NAME if DESCRIPTION is not specified and remove more
preamble from the string.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If another recipe exists with the same name in a different layer and
that other layer is "older" (by layer ID) and is a software or base
layer then lighten the recipe entry in the search results as it may
not be the preferred version (e.g. recipes in BSP or distro layers may
have been customised specifically for the machine or distro).
This has had a performance impact on the recipes list; as a result
showing all recipes by default has been disabled. If the user really
wants to see all recipes they can just leave the search box blank and
hit the search button.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the user edits the layer and changes the repository URL,
subdirectory, or dependencies, ensure that the entire layer gets
refreshed the next time the update script runs by clearing out the last
fetched revision field.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This ensures recipes that failed to parse sort properly instead of all
appearing at the start of the search results.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We don't go through the normal code path where this directory would be
created, and if it doesn't exist an error will be thrown in
insane.bbclass when any QA issue is logged; so create it manually.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Rather than letting BitBake error out, just check for existence of this
file and print a single error line if it doesn't exist.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This should help the initial population of the database and act as a
simple example of how to populate the database programmatically.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we're creating a LayerBranch record, it's helpful to copy the
maintainer and dependency records in from master as they're usually
going to be the same; if they're not the user can always change them
afterwards.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
So it turns out that one or two layers have changed in structure between
branches, so we need to be able to specify this on a per-branch basis.
Good times...
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The recently added global redirect was also redirecting incorrect URLs
within the application to the front page rather than showing 404, which
is not desirable.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This enables sorting the branch drop-down in a configurable manner (e.g.
so that you can have them in order from newest to oldest).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Give the user a friendly introduction instead of plunging them straight
into the layers list. (Of course if people want to link directly to the
layers list they still can.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This cuts out a lot of the elements that aren't needed for the review
list, shows fields in a more suitable way for review purposes than the
standard detail (and includes some fields that don't currently get shown
on the standard detail e.g. layer type and short description).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Newly submitted layers don't have anything other than a master branch,
so they won't display properly unless we have master selected; so just
add a parameter to submission email and review URLs to ensure that is
the case.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Don't assume the current directory contains our conf/local.conf,
explicitly use the directory above where the script is located.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
