During debugging a parsing issue, we don't really want to continue if a
parsing error occurs, and in that situation I usually end up using
Ctrl+C to exit early. Add an option to exit immediately upon error to
avoid having to do that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When showing the error/warning counts for update records we need to
include any errors/warnings that are shown only in the main update log,
so we need to adjust how these are collected. Use a function rather than
pure aggregation to give a bit more control, and a {% with ... %} block
in the template to avoid the functions being called more than necessary.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If an exception occurred during the update then we were managing to save
the update record, but we did not include the exception traceback in the
log for the update. Catch the exception and log it which ensures it gets
captured in the update record and still gets printed as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make layerupdate collection slightly more reliable and make it easier
to see when updates have actually been captured:
* Split layerbranch into separate layer and branch fields, since there
may not be a layerbranch in existence but we might want to log an
error relating to the branch and layer.
* Show all layerupdates on the update detail page, not just those with
log messages
* Record before and after revisions and show these in the update detail
and layerupdate detail (with links)
* Record return code of update_layer process
* Highlight layer updates with a non-zero return code, errors or
warnings in the output on the update detail page
* Show duration on the layerupdate detail page
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When an exception occurs during the main part of update_layer, we were
catching and printing it but that's not enough - we need to do the
following as well:
* Use logger.error() to print the exception information, so that it gets
logged and highlighted as an error in the layer update
* Exit with a non-zero return code so that update.py knows it has failed
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If for some reason update_layer.py does not print out the values we
expect then we shouldn't be throwing a traceback, we should be handling
it gracefully - i.e., print an error and then move on to the next layer.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
On the recipe detail page we provide a link to the actual recipe file
for reference purposes. However, for recipes that include a common .inc
file, many of the definitions of variable values will be inside the
.inc, therefore if you just look at the recipe you won't see the full
picture. Of course you can just go up to the parent directory in the
repository web interface, but for convenience's sake add links to any
files that are included/required by the recipe that are adjacent to
the recipe itself. (We already have the data in the form of the
RecipeFileDependency records that are intended to ensure we know when
the recipe needs to be updated if one of the files it includes changes).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Specifying the covering recipe in the comparison recipe detail page was
always a bit awkward - you could only type the name, if you wanted to
actually find a recipe or look up the currently selected one's details
then you had to open another browser tab/window. To fix this, replace
the form on the comparison recipe detail page with a side-by-side
display of the covering recipe's information, along with a button that
lets you search and then select the covering recipe and at the same time
enter comments or set any of the other cover fields.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we're collecting this useful info we should really display it. Try to
make the link clickable if possible.
At the same time, add the layerbranch to the list display for Source
objects in the admin site.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script to fetch package sources from Fedora's Pagure
infrastructure site at https://src.fedoraproject.org/, based upon
github-fetch.py (I broke out the internal fetch function with the intent
to share it with that other script, however I didn't in the end as I
wanted the scripts to remain independent).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script to fetch/update all repositories within a GitHub
organisation. Note that to run this you will need an access token, which
you can create by going to GitHub and then
Settings -> Developer Settings -> Personal access tokens.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Comparison updates might involve some custom fetch process, so provide a
mechanism to register these via settings.py on a per-branch basis. If
an update command is defined for a branch and the logged-in user has the
new "update_comparison_branch" permission, an "Update" button will show
up on the recipes page for the comparison branch for authenticated
users that will trigger the command in the background (as a celery job)
and then show a page that displays the status. The status isn't shown in
real-time since that requires quite a lot of plumbing, but the page at
least auto-refreshes.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make it clearer what this field is for (it should be set for every
comparison layer, so that they don't show up in places they shouldn't).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If other distro comparison package(s) exist for a recipe being shown in
the RRS recipe detail page, link to the page for each package as well as
any extra URLs.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a structure that lets you define a template URL per layer to be
shown per comparison recipe. For example, you could use this to define a
URL template to link to the upstream summary page for the package (e.g.
Fedora's page for the acl package is at
https://apps.fedoraproject.org/packages/acl, so you would use
https://apps.fedoraproject.org/packages/%pn% as the template and then
this would be shown for every package).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to export the comparison search results to CSV format in
order to allow importing the data into external tools.
Note: I implemented this in a different way than the earlier recipe CSV
export, i.e. it uses a template to render the CSV instead of a
function-based view with the Python csv module - the reason for this is
we can reuse the same view as we use for producing the search, with all
of the flexibility that gives us.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When doing the reversed query it's often desirable to exclude recipes
by inherited class, for example those that inherit the packagegroup,
image and meta classes as they don't actually build anything and thus
aren't going to match up with anything in the other distribution.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Provide a lazy-loaded popup for selecting layers to include in the query
instead of having it as a simple drop-down, so you can select more than
one layer.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Enable "reversing" the comparison, so you can see which recipes on the
OE side match up (or don't) with the other distro. The filtering for
this is a bit awkward, since we don't have an actual foreign key for the
link, hence the hairiness of the code.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add support for importing package information extracted from a running
Debian system (i.e. from the output of apt-cache show "*").
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a flag that can be set and searched for to indicate that we need to
take care of importing a package or a patch applied by a package.
Ideally the comments would elaborate on what's needed (if it's not
obvious from the cover status).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Packages with the "Not available" status were inexplicably excluded from
the graph, include them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow searching on:
* Any "available" status (i.e. other than "Unknown" or "Not available")
* Whether the package has patches or not
* What the covering layer is (assuming there is one).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's a bit crude, but accept '' or "" as meaning search for entries
with an empty category.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an option to skip certain recipes by name (a crude workaround for
when we would otherwise get an erroneous match).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the source URL is fetching from pypi then we know it's python;
similarly cpan means perl.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we always set cover_pn if there is a match rather than only setting
it if it's different from pn then it simplifies the code elsewhere.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Only apply OE-Classic-specific logic after checking that it's actually
OE-Classic
* Ignore recipes marked deleted
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script to import "recipe" information from other distro metadata,
based on the import_classic.py script. At the moment, this assumes a
directory where each subdirectory is a package directory containing a
spec file; this would be suitable for distributions such as Fedora
assuming you have all of the package repos checked out locally. Since
you can add additional information to these records (the cover fields
pointing to matching recipes), existing records are updated rather than
deleting everything and re-importing, and we only mark records as
deleted rather than actually deleting them (in case you accidentally
point the script at an empty directory or similar).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Turn the existing OE-Classic support into something a bit more
generic so we can import data from other distributions and compare it to
what we have in layers.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were sorting the layers in the wrong order when trying to choose
which layer to pick a matching recipe from - it needs to be descending
priority order, not ascending.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script that uses the REST API to gather information from another
running layer index instance (e.g. layers.openembedded.org) and import
that into the local index. Only information for branches that are
already set up is imported, and only manually entered information -
no recipes, machines, etc.
Partially implements [YOCTO #9760].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the cover status for a comparison recipe is Unknown, Not available,
or Distro-specific, then we disable the recipe and layerbranch fields in
the form (using Javascript), however the form could still be saved and
any existing values would persist. Clear out the fields upon saving if
they are set under these circumstances.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Check if bitbake directory can be found before trying to do anything
with the repo
* Split try..finally into two so we don't try to shut down a nonexistent
tinfoil when we failed to gain a lock.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a file is moved (renamed) to a path outside of the layer, e.g.
another layer within a multi-layer repository, then we need to treat it
as a delete. Up until now we were updating the path and continuing, and
then the recipe was also picked up as an add in the other layer, leading
to duplicate recipe entries. I'd noticed these duplicates before but up
until now I'd thought that they were due to another bug we already
fixed, apparently not.
In order to remove these erroneous duplicate entries in existing
databases I have also added a layerindex/tools/fixup_duplicates.py
script. I've also made the -r/--reload option delete them as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to show a notice at the top of every page; this provides
the ability for admins to display a message to visitors in the case of
infrastructure or index data issues. Notices can have an expiry date and
can be disabled and re-enabled if needed. A subset of HTML can be used
for formatting the text, URLs will be made into clickable links, and
four "levels" are supported (info, success, warning and error).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* If a missing dependency is not required, show a warning instead of an
error
* If logger isn't specified we still need to skip to the next item, so
move the continue statement out of the conditional block. (In practice
I don't think this function is currently called anywhere in the code
without a logger specified, but let's fix it anyway).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a LAYERRECOMMENDS relationship is not satisfied, we shouldn't be
erroring out - it's a recommendation, not a hard dependency. Just show a
warning and allow processing to continue.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a file is modified and renamed it will show up in both
iter_change_type('M') and iter_change_type('R'), however naturally the
file that will exist will be the b path and not the a one, so we should
be looking at the b path or we will get errors.
FYI you can reproduce this with OE-Core (in a scratch database) using
the following procedure:
1) (in the OE-Core layer directory):
git checkout 59285b324f6d9ed270b0bef209ef5da22a620a83
2) update.py -l openembedded-core -b master -x --nofetch -r --fullreload
3) (in the OE-Core layer directory):
git checkout 086308aa2a5e332de6f00ed397c4a55d132f158f
4) update.py -l openembedded-core -b master -x --nofetch
Without this change you'll see the following error:
ERROR: Unable to read /opt/layerindex/layers/git___git_openembedded_org_openembedded-core/meta/recipes-devtools/python-numpy/python-numpy_1.13.1.bb: Traceback (most recent call last):
File "/opt/layerindex/layers/bitbake/lib/bb/command.py", line 84, in runCommand
result = command_method(self, commandline)
File "/opt/layerindex/layers/bitbake/lib/bb/command.py", line 568, in parseRecipeFile
envdata = bb.cache.parse_recipe(config_data, fn, appendfiles)['']
File "/opt/layerindex/layers/bitbake/lib/bb/cache.py", line 315, in parse_recipe
bb_data = bb.parse.handle(bbfile, bb_data)
File "/opt/layerindex/layers/bitbake/lib/bb/parse/__init__.py", line 117, in handle
return h['handle'](fn, data, include)
File "/opt/layerindex/layers/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 132, in handle
abs_fn = resolve_file(fn, d)
File "/opt/layerindex/layers/bitbake/lib/bb/parse/__init__.py", line 141, in resolve_file
raise IOError(errno.ENOENT, "file %s not found" % fn)
FileNotFoundError: [Errno 2] file /opt/layerindex/layers/git___git_openembedded_org_openembedded-core/meta/recipes-devtools/python-numpy/python-numpy_1.13.1.bb not found
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you're running a testing / internal instance then you really don't
want to be emailing maintainers on publish, so provide a setting you can
use to disable that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
Assume there is no master branch in hello layer:
$ update.py -l hello -b master
INFO: Skipping update of layer hello - branch master doesn't exist
This is correct since hello layer doesn't have master branch, but when --nocheckout:
$ update.py -l hello -b master --nocheckout
[snip]
INFO: Sorting layers for branch mater:
WARNING: Cannot find required collections on branch master:
WARNING: hello: LAYERDEPENDS: <snip>
This is incorrect, this patch fixed the problem, now it skips it since the
branch doesn't exists when --nocheckout.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When layer_a RECOMMENDS layer_b, try to add layer_b before layer_a, but if
layer_b is not found, still add layer_a.
And print summary error mesage:
$ update.py -b master
ERROR: Issues found on branch master:
openembedded-core: Added without LAYERRECOMMENDS
meta-secure-env: Failed to add since LAYERDEPENDS is not satisfied
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
$ git clone <url>
warning: remote HEAD refers to nonexistent ref, unable to checkout.
$ git rev-parse HEAD
HEAD
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Catch the error and avoid that.
And use "git reset --hard" to replace of "git reset --hard HEAD", HEAD is
default for git reset, so they are the same, but the later one reports error
when remote HEAD doesn't exist:
$ git reset --hard HEAD
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
[snip]
$ git reset --hard
No errors.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
$ update.py -b <new_branch>
[snip]
NOTE: Starting bitbake server...
Traceback (most recent call last):
File "update_layer.py", line 471, in main
utils.setup_core_layer_sys_path(settings, branch.name)
File "/buildarea1/lyang1/layerindex-web/layerindex/utils.py", line 376, in setup_core_layer_sys_path
core_layerdir = os.path.join(core_repodir, core_layerbranch.vcs_subdir)
AttributeError: 'NoneType' object has no attribute 'vcs_subdir'
[snip]
This is because core_layerbranch is not in database yet for completely new
branch, so it is None and we will get the error. Avoid calling
setup_core_layer_sys_path() when "update_layer.py --initial" will fix the
problem.
And also only add core layer's sys path when it is present, since core layer
may not be added yet for completely new branch.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to export the recipe listing for a layer to a CSV file
for importing into external tools. At the moment we include name,
version and license, but there is a parameter that lets you specify the
fields to include in the URL if desired.
Implements [YOCTO #12722].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We're about to replace this with a proper CSV export function, so we
don't need this dead code hanging around anymore.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's a bit of a pain to have to set the two python environment fields on
every record in order to have things set correctly, and it can easily
get forgotten, so try to set them automatically by default (assuming
reasonable naming).
Note that this does introduce an annoying behaviour whereby if you click
"Add another Maintenance plan layer branch" and then decide you don't
want it, the admin form will insist you fill in the fields unless you
clear out the python2/3 environment fields. I'm not sure how to fix
that, so I'm leaving it as-is for now.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Collect information about patches applied by a recipe, and record each
patch along with the upstream status, presenting them in the recipe
detail.
Implements [YOCTO #7909].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Checking out a revision in the bitbake/layer repositories is something
we are doing in a few places, so add a checkout_repo() function that
does this, ensuring that we don't get tripped up by any junk files,
and avoids checking out if the repository is already at the desired
revision (thus avoiding the clean operation and e.g. preserving any
.pyc files that aren't stale and would speed things up a little). Note
that we do the clean before checking out in case there are untracked
files that are tracked in the commit we are checking out.
In addition to adding this function, change the existing code that we
use in the update script to check out a layer use the new function.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It would be useful in some scenarios to get the complete list of
recursive dependencies for a layer, so add a function to do that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The Recipe Reporting System needs to be able to provide links to commits
in the web interface for the repository, but we can only do this if we
have a custom template URL just like we do for file/tree links, since
it's different for different git web interfaces. Add support in all the
various places for such a URL and make use of it in the RRS.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Save each remote SRC_URI so we can use these for the recipe reporting
system. This replaces an earlier implementation in the rrs branch where
we simply stored SRC_URI verbatim.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Every time we add something that links to Recipe we had to add it to the
exclusions list in the readonly_fields line for RecipeAdmin (and
ClassicRecipeAdmin), which is tedious and easily forgotten. We can avoid
this by looking at each field and excluding it by its attributes rather
than having a hardcoded list of names.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the RRS is enabled, then add a link to it in the tools menu. I don't
expect this to be used a lot, but otherwise the only way you'd get there
would be either externally or via the link from the layer detail.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The old RRS branch had its own addition of dependency support, but in the
mean time we added that to the layer index in the master branch using a
different structure. Adapt the RRS recipe detail page to that structure.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the RRS is enabled, then add a way to get from the layer detail page
to any maintenance plans in which the layer is included.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Sometimes we get back UTF-8 characters (particularly when picking up
names from git commits), and the ascii codec will error out if that
happens, so switch over to utf-8.
We can as a result remove the decode() calls from the bulk change view.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Give some space to the tabs,
Add some top margin to the nav-pills class so we
create some breathing space between the milestone
overview and the tabs.
Change 'No Maintainer' to 'No maintainer'
Just to keep the capitalisation style consistent
across the interface.
Apply the muted class to all not-sortable table headings
The class was missing from the Recipe and Maintainer
columns in the recipes table; and from the Not updated
and % done column in the maintainers table.
Remove the strong tag from the recipe status information
This is just to match the presentation of the milestone
overview information in the base_toplevel.html template.
Separate the footer from the bottom of the viewport,
It's hard to see the footer on click on its links when
they are so close to the bottom of the veiwport, so
add some margin at the bottom of the footer <div>.
Change the label of the recipes tab,
From 'Recipes status' to 'Recipes upstream status' to
match the label of the 'Upstream status' filter in the
recipes table.
Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
In e902b67bcc I missed a couple of Context
usages in the layer publish view and the result was that it broke
publishing a layer (and apparently I didn't run a final test on that,
shame on me).
Thanks to Yi Zhao <yi.zhao@windriver.com> for pointing this out.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
There were several issues with this code, including that it used
SortedDict which was removed in Django 1.9 and that it seemed not to
have been fully updated to accommodate changes in bitbake's recipe
parsing API. In the end I decided the simplest thing would be to move it
over to using oe.recipeutils.patch_recipe() which is actually a now much
more mature version of the code that originally started life here. With
that we can get the bulk change functionality working again and gain
some of the improvements in behaviour that we've developed in
oe.recipeutils.patch_recipe(), as well as avoiding effectively
duplicated code.
Implements [YOCTO #9730].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It seems that in df492b1277 the original
intention was to move setup_layer to the utils module, but that didn't
actually get done in the final patch - however the change was made here
to accommodate the move, meaning it's been broken since then. Revert
that so we're actually calling the function in the place it exists.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
With Django 1.10+, if you use get_template() to retrieve a template,
then you can't pass a context when calling .render() on it, you need to
pass a dict instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We use django-reversion to track history of admin/maintainer changes to
the site, and part of our extension on top of django-reversion involves
annotating each "revision" with a description of the changes that were
made. In django-reversion 2.0.0+ the pre_revision_commit signal that we
were using to do this annotation is gone in favour of just using
Django's standard pre-save signal. This was a little challenging to
adapt to for our purposes but not impossible.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
resolve_variable isn't available with Django 1.10+, and it's not even
used here, so just drop it from the import line.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The patterns() function is deprecated in Django 1.8 and gone in 1.10, so
we should switch over to the new list format.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Rename publish() to publish_view() and call it directly from the view
rather than using a string name (which is not possible in Django 1.10+).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The utils.setup_django() costs a lot of time, but both update.py and
update_layer.py call it, so move layer validation from update_layer.py
to update.py to avoid calling update_layer.py when possible can save a
lot of time.
Now we don't have to call update_layer.py in the following cases:
* The branch doesn't exist
* The layer is already update to date on specified branch (when no
reload)
* The layer dir or conf/layer.layer doesn't exist
We can save up to 98% time in my testing:
$ update.py -b master --nofetch [--fullreload]
Before Now Reduced
No update: 276s 3.6s 98%
Partial update: 312s 87s 72%
Full reload: 1016s 980s 3%
Note:
* All of the testing are based on --nofetch
* "No update" means all layers on the branch is up-to-date, for
example, when we run it twice, there is no update in the second run,
so we only need about 3s now, which is the most common case when we
use cron to run it per half an hour.
* "Partial update" means part of the layers have been updated.
* "Full reload" means all of the layers have been updated.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This makes it easy to see which layers failed. For example:
ERROR: Failed layers on branch master: openembedded-core meta-python
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We have an update.py running periodically in background, but we also
need to run it manually, for example, run it to update actual_branch,
the manually run usually failed because can't get lockfile, we have to
run it again and again. A timeout option helps a lot in such a case. Now
the following command can make sure we can run the command successfully:
$ update.py -b master -a actual_branch -t 2000
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The previous commit broke the layer order, e.g.:
A -> B -> C -> D
The algorithm is checking the dependencies one by one, and until we find D, add
D to layerquery_sorted, and add it "collections", the one in "collections"
means it's dependencies are OK, then C, B and A will check against collections,
so that update_layer.py will update them one by one. The previous commit added
A/B/C/D to collections directly, so that when check against it, all the
dependencies are met, thus broke the layer sorting, and then there would be
failures if we pass layer A to update_layer.py before B, C and D (suppose they
are newly to database). This commit fix the problem.
BTW., why I use collections to record the one whose dependencies are matched,
but not directly use layerquery_sorted, it is because collections contains both
the ones to be added/updated and the ones in database, but layerquery_sorted
only contains the ones to be updated/added.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a page with basic statistics for the index - number of layers,
recipes, classes, machines and distros on an overall basis (distinct
names) and per branch, since I've been asked a few times for this kind
of information. It's currently only linked from the Tools menu for
logged-in users, but the URL will work for anyone.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the user hit Ctrl+C during the initial info gathering then it didn't
break out of the loop in update.py, so you had to hit Ctrl+C for as many
layers as were involved in the update. Look for exit code 254 from
update_layer.py and stop if it is returned since that indicates Ctrl+C
has been used.
Additionally, ensure we return exit code 254 and print a message from
the main update script when it is interrupted in this way.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For reasons that are not immediately clear to me, if you don't specify a
name when adding the ViewSet it takes one from the model used in the
queryset in the ViewSet. The new layers view uses LayerBranch and so it
silently replaced the layerBranches URL in the router, even though the
URL itself was still present. Unfortunately that's broken Toaster.
Specify a name to restore the old URL.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It was a bit awkward to query layers externally - you had to do multiple
queries and you couldn't get the YP Compatible version info at all. Add
an additional LayerBranch-based view that exposes the branch name,
layer fields, YP Compatible Version and active maintainer information
with just one call.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Django REST Framework now requires a field specification for every
ModelSerializer, so specify '__all__' to retain the current behaviour.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Layer maintainer and layer note information wasn't available through the
REST API since it wasn't needed for Toaster, but for other uses it is
useful.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you query on a boolean field you can use the string "False" to match
False in the database; however if you try the same with __isnull then
the query will match every record which is obviously undesirable. If
__isnull is being used, then convert the value to a boolean so that the
query works properly.
An example of this type of query:
http://127.0.0.1:8000/layerindex/api/layerBranches/?filter=yp_compatible_version__isnull:false
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
tinfoil is not needed in cases like the layer is already up-to-date or the
layer is invalid, so only init it when needed.
This can save about 1min when run "update.py -b <branch>" (124 layers).
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This can save a lot of time, here is my testing data when PARALLEL_JOBS is 10,
this is the fetch time only, I hacked it to stop when the fetch is done to get
the data (124 layers):
$ update.py -b <branch>
Before: 2m30
Now: 16s
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
$ ./update.py -l foo -b new_branch
INFO: Fetching remote repository foo
DEBUG: run cmd 'git fetch -p' in 'foo'
[snip]
DEBUG: run cmd 'git checkout origin/new_branch' in oe-core
ERROR: error: pathspec 'origin/new_branch' did not match any file(s) known to git.
The "new_branch" is newly created, it doesn't exist in local repo since it
isn't fetched, it only fetches the layer specified by -l, so only the foo layer
is fetched. This patch fixes problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The obsolete dependency is the one which is in database but not in
conf/layer.conf anymore. The old code had a problem for newly created
layerbranch, the new layerbranch has no dependencies, so no need remove. And it
had a side effect was that when need_remove was cleaned up, it would be set
again in the next for loop, thus might wrongly remove dependencies. This patch
can fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If your classic recipe search returned a single result, then since
commit c8c25fb641 you got a standard
recipe page instead of the proper page with fields you can edit. To fix
it, just drop the redirection functionality from the classic recipe
search, since we don't really want it here.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were using the layerbranch id to search for the specified layer,
which is most likely to return either no results or results for the
wrong layer. We can also avoid specifying the id field at all here as
the filter() function can handle real objects.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Unsatisfied layer dependencies shouldn't error out of the script -
broken metadata isn't supposed to terminate the index update process.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We need to set a default for this variable or you get an
UnboundLocalError if no non-root superuser/staff accounts are set up in
the database.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you use a traditional database engine (such as MySQL/MariaDB) then
maximum character field lengths are enforced, however depending on the
configuration this may result in an exception rather than a warning and
truncation and will also break the per-layer transaction. To avoid that
ugliness, add a signal handler to do it internally, which as a bonus
lets us know if field lenghts are too short for data when using database
engines that don't enforce lengths (e.g. SQLite).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a database error occurs when we save a recipe (e.g. because a
database-level constraint is voilated) it will mess up the transaction.
Unfortunately that means we need to break out of updating the entire
layer rather than catching the error, because if we do catch it we just
get errors on every update after the initial error; failing early and
giving up on the transaction is a little better in terms of not filling
up the update logs with further useless errors.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Allow the model's __str__() function to control what is shown for each
dependency item as per other model admins
* Enable searching for PACKAGECONFIGs by recipe name
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Use the more typical handling of the return of get_or_create() and check
if the item was created before saving.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix duplicate PackageConfig records being created each time a recipe
is being updated - we need to delete the old ones first.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Since we added the "packageconfig" to Recipe we need to exclude it from
the admin or we see an error when we try to open ClassicRecipe in the
admin site.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Added a model for the PACKAGECONFIG variable, which has a one to
many relationship with the Recipe model.
Added models for static build dependencies and dynamic build
dependencies, both of which have a many to many relationship with
the Recipe model.
These objects are created in update_layer.py and are displayed on the
Recipe detail page.
Added a depends search option for recipes, allowing users to search for
recipes that depend on any particular recipe. Use "depends:recipename"
in the recipe search to activate this.
Fixes [YOCTO #12129]
Fixes [YOCTO #11415]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Use JavaScript to check if the search box for recipe search is
empty before querying the database. This will prevent the "502
Bad Gateway" error that occurs when the query takes too long due
to the large list of recipes. Since there are so many recipes
spread across the layers in the OE index, there's no point in
allowing a user to search without a keyword in order to browse
the list; it simply isn't digestible as a whole.
For the Machines, Classes, and Distros pages, the search behaviour is
unaffected, however to make it more obvious that you can browse the list
add an explicit "browse" button.
Fixes [YOCTO #11930]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add another tab to search for classes.
Fixes [YOCTO #11207]
Signed-off by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Don't assume that every model will have a "search_allowed_fields"
attribute - if it doesn't, default to all CharFields in the model. This
fixes searching via the REST API.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When publishing a layer, send an email notification to all of that
layer's maintainers. Include information on how to edit the layer, plus
contact details for the first active staff user if there are any
problems (we could make this configurable in future, but for now this is
sufficient).
Fixes [YOCTO #11208]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's really irritating to be forced to specify a value for this field
especially as it'll get auto-populated by the update script. Set
blank=True to allow that. While we're at it, touch up the description a
bit to make more sense.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you're diagnosing problems with the bitbake server when running the
update script, then you need to be able to look at
bitbake-cookerdaemon.log, but you couldn't do that after the fact
because the temporary directory it gets written out to was being
unconditionally deleted. Add a --keep-temp option which preserves it and
some debug messages to tell you where it is.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow admin to create Yocto Project versions with the version name,
description, an icon link that represents that version, and a link that
contains more information about the version.
Admins who have the "set_yp_compatibility" permission can then
decide if a layer has Yocto Project compatible certification,
and if it does, admin can choose which version of Yocto Project
the layer is compatible with. If a layer is deemed compatible,
the version's icon will appear next to the layer's name, and
the icon be a clickable link to a page with more information.
Fixes [YOCTO #11452]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a log level is set on the command line with -q/-d,
set tinfoil's log level to the appropriate log level.
Fixes [YOCTO #11931]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This patch adds asynchronous task execution using a Celery backend
and RabbitMQ task queue, so that the layer submission process to
proceed even in the event that sending the notification email fails,
and establishing an asynchronous execution mechanism that we can use
in the future e.g. for triggering parse operations from the web UI.
This pertains to bug 11197:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11197
It updates the README to reflect the installation and configuration
of a basic RabbitMQ setup, adds a 'tasks.py' file to contain task
definitions, updates the 'edit_layer_view' function to send
emails to administrators about new and updated layers asynchronously,
modifies the 'settings.py' to include a default configuration
for a RabbitMQ connection, and updates the Dockerfile to start a Celery
worker alongside the Gunicorn daemon.
Fixes [YOCTO #11197].
Signed-off-by: Diana Thayer <garbados@gmail.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Before, if a user edited a layer's name, they would be redirected
to a url utilizing the old name and then receive a 404 Page not
found error. Now, the url utilizes the new name.
Fixes [YOCTO #11932]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
At the moment the URL that is presented in the review email will have
http:// or https:// prefix depending on what the user who submitted the
layer was using, but that's irrelevant - we actually want https:// if
the server is capable of it since the reviewer may be redirected to log
in (and Django's login_required decorator will always redirect to the
login page if http is being used as far as I observed, which is a bit
annoying if you are already logged in.) Add a setting which if enabled
will substitute https:// as the prefix.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The branch is not needed any more when it has been removed from the repo, so we
also need remove its layerbranch, otherwise it still can be got from the web,
which causes confusions.
Note, we have to move the location of tinfoil's code to avoid creating tinfoil
when not needed, otherewise, if tinfoil is created (but not needed), and the
program exists earlier before "try ... finally" block, then tinfoil.shutdown()
doesn't run so that it is not shutdown. Move the code back, right before where
it is needed can fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
-p, --prune
Before fetching, remove any remote-tracking references that no longer exist on the remote.
Fixed:
$ git push origin :test_branch
$ ./update.py
The test_branch was still in fetched local repo which was incorrect, it should
be gone since it has been removed by upstream.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It doesn't need to be so ahead since we only need it when writing database, and
a following patch will remove layerbranch from database when the branch had
been removed from the repo, it's not easy to do the work in
transaction.atomic() block.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It never works since it is in the middle of transaction.atomic() block, and
update.py doesn't need it any more, so remove it.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Problems
The update.py couldn't handle new (not only new branch, in fact, but also
existing branches, see below for more info) branch well, for example, there are
3 layers: layer_A, layer_B and layer_C, and create new branch "branch_1" for
them, and they have depends:
layer_A -> layer_B -> layer_C
The "->" means depends on.
Then run "update.py -b branch_1", there would be errors like:
ERROR: Dependency layer_B of layer_A does not have branch record for branch branch_1
Though update.py runs "update_layer.py" twice, but it didn't help since
layerbranch was None when it was failed to create in the first run.
The reason is if update.py updates layer_A firstly, it would fail since it
can't find layer_B:branch_1 in database (not added to database yet), similarly,
if add layer_B before layer_C, it would also fail. Only layer_C can be added
(assume it has no dependencies). So we have to re-run update.py again and again
to make it work, here we may have to run update.py 3 times, and more runs are
needed if the dependency chain is longer.
* Solutions:
Make update.py pass layers orderly to update_layer.py according to dependencies
can fix the problem, we can get intial dependencies from tinfoil, add an option
"-i, --initial" to update_layer.py to get them.
Not only new branch, but also existing branches may have the problem, because
collections and dependencies maybe changed in the coming update, so we can't
trust database when the layer is going to be updated, for example, if there are
10 layers in database, and 3 of them will be updated (-l layer1,layer2,layer3),
then we can not use the 3 layers' collections data from database, we need get
them from tinfoil.
* Performance improvement:
It should be the same as before in theory, I have tested it with 97 layers:
- Before: 11m6.472s, but only 75 layers were added, 22 ones were failed, I
have to re-run update.py again and again (maybe 4 times to make all of
them added). So:
(11 * 60 + 6)/75*97/60 = 14m35s
- Now 12m10.350s, all the layers are added in the first run.
So about 2 minutes are saved.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we don't restore the current working directory, it may cause trouble
for the calling function.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The _add_dependency() uses:
if layerbranch.collection:
var_name = layerbranch.collection
The layerbranch.collection is none if it is newly created, thus it can't get
LAYERDEPENDS, because what defined in layer.conf is LAYERDEPENDS_<collection>,
but what it would get is LAYERDEPENDS_<layer_name>, this patch can fix the
problem.
Reproducer:
$ python3 update_layer.py -l mete-xfce -b <newbranch> --fullreload -d
It would get None LAYERDEPENDS.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
1) set LAYERDEPENDS_openembedded-layer = "core"
2) $ "update.py -l meta-oe -b master"
Check from web, its dependency is "openembedded-core"
3) Change LAYERDEPENDS_openembedded-layer = "foo"
4) Run "update.py -l meta-oe -b master"
5) Check from web, its dependency is "openembedded-core and foo", this might
be incorrect, now if set REMOVE_LAYER_DEPENDENCIES to true, the old
dependency openembedded-core will be removed, the default is False which
prints warnings to notify users.
And also the existing checking should filter(required=required), otherwise it
can't work well when a layer is in both depends and recommends, this can't
happen in a normal case, but it would surprise the user when this happens.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The layerbranch is not used in parse_layer(), so remove it.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an option "-a" to update actual branch for layer and bitbake, it is
useful when there are many layers and need update actual branches
frequently. We only can update them via website without this patch,
which is not fun and easy to make mistakes.
* It works with "-l", and "-l bitbake" means update bitbake branch.
* It requires "-b" to work, and only one branch is supported in a run.
For example:
$ update.py -b master -a branch_20170526
All the layers which have branch master and actual_branch branch_20170526
will be updated to branch_20170526.
$ update.py -b master -l meta-oe -a branch_20170526
Only meta-oe layer will be updated.
$ update.py -b master -l bitbake -a branch_20170526
The bitbake's bitbake_branch will be updated.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This change supports querying recipes that belong to a specific layer
by using the prefix "layer:" + the desired layer name, for example:
"layer: openembedded-core" and this string can be used by itself or
combined with other supported options.
A descriptive error message is displayed when the query string has an
unexpected formatting or a non-valid layer name is searched.
[YOCTO #6618]
Signed-off-by: Jose Lamego <jose.a.lamego@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a status for a layer indicating it should not be updated. I don't
expect this to be widely used (and is only settable from the admin
interface) but would be useful if you have a legacy sub-layer that you
want to prevent from being visible on certain branches - it will prevent
the update script from doing anything with the layer and thus avoid
branch records from being auto-created on branches where you've deleted
it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When reading conf/distro/*.conf to create distro records, attempt to
parse the config file and use DISTRO_NAME (if set) to populate the
description field for the record. If that's not set then fall back to
the less commonly used meta-comment that we supported previously.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we want to parse a configuration file (e.g. a distro conf file) then
we need convenient access to bitbake's conf parsing code, so create a
parse_conf() function to provide that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When the recipe search returns a single result, redirect to it
instead of showing the full result list view.
Part of the fix for [YOCTO #6618].
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The branch meta change made in b91b09a0ae
apparently requires this migration - not entirely sure why, but Django's
manage.py makemigrations wants to create it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make minor (non-structural) improvements to the model, mostly for the
benefit of the admin interface:
* Set default ordering so that branches appear in the same order they do
in the user-facing drop-down
* Include the description in the default string representation (so you
know which branch name matches with which Yocto Project version, in
the OE index).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We can't decode UTF-8 characters byte-by-byte, as soon as we hit a
character that's more than one byte long then we'll fail. Use a reader
object to do it properly. This fixes parsing current meta-angstrom on
master. At the same time, specify errors="surrogateescape" to avoid the
update process dying at this point in case of characters that aren't
valid UTF-8.
Thanks to Jiajie Hu <jiajie.hu@intel.com> who fixed this in devtool's
very similar code.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The fido branch of OE-Core expects BitBake 1.26.x. Unfortunately that
version had a commit backported which introduced a non-working
tinfoil.shutdown() method, so we can't simply rely on its presence to
determine whether or not it should be called. Use a check on the BitBake
version number instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Layers that aren't published shouldn't be visible via the API. (We don't
need to apply that filter to recipes, machines or distros though since a
layer's content won't automatically be indexed unless it has been
published).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
On the main layer index page we want openembedded-core to appear before
meta-oe - naturally openembedded-core is the layer everyone starts from.
We already have an "index_preference" column that we use for search
results, so just sort by that here before sorting by name.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Since some recent infrastructure changes, "/cgit.cgi" should no longer
be part of cgit.openembedded.org URLs, apparently.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix the transaction handling code to work with Django 1.8 in most of the
tools scripts. (Some of these are no longer used, but still serve as
examples of how to import data and update the database.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* We were passing the incorrect path (to the top of the layer repo) if
the layer had a subdirectory, so this doesn't seem to have been able
to work for such layers previously.
* Doing this update in the main update.py script meant that this could
never work across branches requiring a python version change (using
PythonEnvironment records) since the code was running within the same
environment in which update.py was launched - the entire point of the
separation of the two scripts. Move the checking to update_layer.py
and call it separately to perform these updates, splitting out some
common code in order to do so.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
A minor optimisation - it's not going to change between layers so get it
in the branch loop.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Setting of the object has to be before the try: or otherwise the
finally block can get called if that doesn't succeed with the result
that the layerconfparser object won't have a value, which will
trigger an exception
* We shouldn't be using the config_data object after shutting down
tinfoil, so avoid doing that
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix a couple of problems with shutting down tinfoil:
1) tinfoil.shutdown() wasn't being called on error because it wasn't
inside a finally: section, thus on error the script hung with bitbake
master (since the tinfoil2 changes) - this is almost certainly a bug
in bitbake but let's handle it here anyway.
2) We need to check whether the tinfoil object actually has a shutdown
attribute since we still want to support updating for older branches
where tinfoil didn't have a shutdown() method.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The new client/server API of tinfoil requires explicit call of
shutdown method to send the event for finalize cooker process.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Remove ; and spaces when setting variables in calls.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Don't assume remote name is origin, run `git remote` to get the remote
name. When checking the remote, detect the branch as well, that way the
layerindex will work if the remote branch name and local branch name do
not match.
Note that this currently only supports one remote.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
It's a little crude and certainly not optimal performance-wise, but we
can support querying for recipes that inherit a particular class without
too much trouble. This allows you to add "inherits:cmake" to the query
and have it return only recipes that inherit the cmake class. You can
use more than one inherits: item to filter down to recipes that inherit
all of the specified classes.
Note: this does not otherwise change the behaviour of specifying
multiple words - all of the words other than those that start with
"inherits:" are treated as part of a single phrase that will be searched
for - not separate keywords.
Fixes [YOCTO #9879].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
At the moment it's a bit difficult to get update logs out of the
environment in which the update script is being run. In order to make
the logs more accessible, create a LayerUpdate model to record the
output of update_layer.py separately for each layerbranch and tie the
created LayerUpdates together with a single Update model per session.
We provide two ways to look at this - a Tools->Updates page for
logged-in users, and there's also an "Updates" tab on each layer that is
accessible to anyone; which one is useful depends on whether you are
looking at the index as a whole or an individual layer.
Update records older than 30 days are deleted automatically by default.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you specified printerr=False we were referring to the output variable
that hadn't been set. Looks like I broke this back in 2013 in
93ce26f21c.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an action to duplicate a Branch object, along with all of the
LayerBranches (and LayerMaintainers and LayerDependencies) underneath
it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
update_layer may create the layerbranch record (e.g. for a new branch)
so we should be looking for it after that has been run. Also, we cannot
assume that last_rev will get set because a layer might fail to fetch,
so take that into account as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a layer is removed by its subdirectory being deleted then we want to
pick up on that and produce an appropriate error message - so let the
layer_update code do the checking out and verifying things are correct
before trying to parse layer.conf.
This also fixes --nocheckout still checking out a branch.
Additionally, drop some code that gets the layerbranch which we just
retrieved a few lines above.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the layer failed to fetch, we shouldn't be attempting to update its
contents. (This got broken when we split the update script into two in
c64e4c57a9).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Sometimes people put values that aren't URLs into the HOMEPAGE variable.
If that's the case, then we should not turn that value into a link which
will be invalid.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For some repo URLs we can automatically determine the values of all of
the other fields - e.g. for github or git.openembedded.org, and we've
been doing that for a while. However if someone submits a URL for some
other site we don't know what type of interface it uses, and usually
the submitter leaves the fields blank so it falls to the layer index
maintainer to set the values, and then you have to remember what the
correct URL format is which is awkward especially for gitweb.
In order to fix this, add a select field to the form which allows
specifying which type of interface is being used. At the moment only
cgit, gitweb, gitlab and "(custom)" (i.e. the current behaviour) are
supported. This is not a real field but activates javascript code
that sets the other fields and enables/disables the controls.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Improves slightly on 3155206e54 by doing
an exact match on name and showing that first - now when you search for
"git" you really do get the git recipe first in the list.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Usage of itertools.chain() that was introduced in
3155206e54 in order to prioritise matches
in the recipe name resulted in recipes showing up twice in the results
if they matched in both the name and the recipe name. Use a custom
chaining function that skips duplicate results in order to fix this.
Fixes [YOCTO #10177].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to keep primary keys from constantly changing, preserve the
existing keys as much as possible.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Parse layer.conf and add dependencies that are not required from
LAYERRECOMMENDS_<name>. Update the layerindex/template to support
recommends. Uses bitbake parsing code & checks versions.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Added associated migration.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Collection and version will be pulled from the layer.conf if it exists
and dependencies will be resolved by first checking for layers with the
dependency name and then checking for collections.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Added associated migration.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Read dependencies from layer.conf and try to create the LayerDependency
entry by looking up the correct database object. Dependencies are found
by layer name only - no collection support. layer.conf parsing is
handled by the bitbake code.
Once all layers are added, the dependencies have to be rechecked in case
the layers are not added in order.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
import_project will scan through a project and find any layer and add it
to the database by calling import_layer on each layer. This differs
from import_layer as it tires to figure out the remote url and uses the
subdirectory (if one exists) as the name.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Add the distros to the index. This looks a lot like the machines
and allows users to search for a particular distro.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Added associated migration.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Convert binary strings to strings and strip leading/trailing whitespace
prior to returning errors and output.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Subdirectories are scanned when adding layers. If any of the
subdirectories or root directory layers already exist in the database,
then the addition fails. This changes that behaviour to report the
failure as a warning and remove it from the list. That way, if a repo
has a new layer added it can be rescanned without issue. Layers being
rescanned are checked against the vcs_url to ensure there is not a name
collision. A name collision without the same vcs_url will still produce
a hard failure.
Note that multiple layers with the same vcs_url are supported in the
error reporting even though this should never happen.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Django will produce a cryptic error message if layers are added with
invalid names. Sanitize the layer names when trying to add them.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
We need to create an initial migration before we create any more
migrations that change the database structure, which we're about to do.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Replaced fixture with a function in the initial migration so that we
continue to get a master branch record created with Django 1.8.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The multiconfig changes broke the calls here to loadDataFull(). To avoid
this being an issue in future, make use of tinfoil's new parse_recipe_file()
function (if available) to isolate the code here from any future changes to
bitbake's internal code.
Part of the fix for [YOCTO #10192].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
With Python 3 we need to take care of decoding the output so we can
treat it as a string. At the same time, it's more useful to see the
output string rather than the exception if there is some output.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This is apparently required in Python 3.3+, although I have been unable
to verify this locally.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The updated field on the branch table was not null and this breaks the
fixture. It's too painful to specify a value for a datetime field in a
fixture json file and it wouldn't even make sense to specify a value in
this case anyway, so just allow the field to be null.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Django 1.8 warns that the default for the "permanent" parameter to
RedirectView is changing in 1.9 from True to False, but I believe we
should be specifying False for these redirections - these are not just
redirections from old URLs and may in fact change in the future if the
site structure changes.
Part of the implementation for [YOCTO #9620].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The old transaction API has been removed in Django 1.8 and was
deprecated at 1.6. There's no explicit open transaction, commit or
rollback now - we just wrap the layer operations in
"with transaction.atomic()"; if we need to roll back we just raise a
"dummy" exception.
Part of the implementation for [YOCTO #9620].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Django 1.8 does not allow DateFields that have both auto_now and default
set - since they are mutually exclusive; if you have auto_now you don't
need a default.
Part of the implementation for [YOCTO #9620].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a model to support setting a python command and virtualenv per
branch, which allows you to parse master with python3 and krogoth with
python2 for example.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Newer versions of GitPython implement rename detection, which means that
such changes don't show up as adds and deletes anymore and you get a
bunch of ENOENT errors for renamed files. Add some code to explicitly
look for renames and process them appropriately.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Stop using "d" (sometimes multiple levels!) and use proper descriptive
variable names instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We need to be able to support Python 3 so that we can parse master of
OE-Core with bitbake (which now requires it). This now means the
interface itself and the update script require Python 3.4+.
Part of the implementation for [YOCTO #9704].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We have to upgrade django-reversion to 1.8 due to upgrading Django, but
unfortunately in that same version the author has removed the type field
on Version model, without a particularly good explanation as to why. This
is really annoying as we were using it to provide a reasonable audit
including for deletes. I suspect we'll need to move away from
django-reversion and do our own thing in future, but for now at least
allow the layer index to keep working.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I'd like to be upgrading to 1.8 but that causes problems with South, and
we're not quite ready to dispense with our existing migrations yet.
Part of the implementation for [YOCTO #9620].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were getting temporary paths appearing in the tarball that gets
created to allow you to download the patches for a bulk change operation
that crosses multiple layers - we don't need those.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix "data truncated" warnings/errors when loading data for OE-Core:
* PROVIDES for recipe packagegroup-base is ~1452 characters
* LICENSE for recipe linux-firmware is ~1053 characters
(These aren't shown with SQLite, only with something like MariaDB where
column sizes are enforced.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix a greedy regex in the recipe detail view resulting in the git recipe
listing all bbappends named *_git.bbappend as its bbappends (quite a few
in the public instance).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow updating multiple branches, and if no branches are specified,
update all branches that have a new "updates_enabled" flag field set to
True. This avoids the need to have a separate shell script which runs
update.py for each branch (and thus has hardcoded knowledge of each
active branch in the index, i.e. it needs to be kept up-to-date in
addition to the database.)
The migration will default updates_enabled to True for all branches so
if you wish to take advantage of this functionality, the flag will need
to be set to False for any branches that shouldn't be updated.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to try to avoid problems with leaking memory, context bleeding
from one layer to another, and lay the groundwork for supporting
updating all branches in one operation, split the updating of a single
layer out to its own internal script. This means that the tinfoil
instantiation and metadata parsing is in a completely separate process
per layer.
Implements [YOCTO #9647].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you're moving a short description value from DESCRIPTION to SUMMARY
then part of that is setting DESCRIPTION to blank, however that wasn't
possible - the code was assuming that a null value meant "keep the
original value". Change the logic so that the value in the bulk change
object is always set and is compared to the original value to see if it
is different. This provides less safety against bulk change data going
stale in the face of the metadata being updated, but without using an
additional "magic" field value that's the price we have to pay, and it's
unlikely to bother too many people I would imagine.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Display blacklist information for recipes in the recipe details,
as well as the recipe search page, and layer page tables. This
information is pulled from the PNBLACKLIST variable. Includes a
hover text containing the reason for blacklist labels.
Changes to Django and Layerindex files:
- models.py, update.py
- migration file
- templates and static content (stylesheet)
[YOCTO #7855]
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Using the search box creates a query against the pns,
summaries, descriptions and filenames of recipes.
This results in a lot of spurious results for common terms
like "git" when performing a recipe search.
Make the results more useful by:
* Only searching against pn, description and summary (not file).
* Putting matches against pn at the top of the list, followed by
matches against description and summary.
[YOCTO #9159]
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
The code to populate the inherits information was using a function
designed to get the recipe name from a file path, but unlike recipe
filenames, the underscore isn't treated as special in class filenames
and in fact it's quite common to use underscores there; we were ending
up with items in the inherit list such as "populate" and "rootfs". Use
the standard python os.path functions to do the splitting instead and
avoid this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
To identify image recipes and provide inheritance data for recipes, an
inherits field was added to the recipe model, and then populated using
refactored data from __inherit_cache. Finally the field was also added
to page templates, along with style changes proposed in attachment, and
an additional style change to display the inherits field as a list in
detailed view. The field skips globally inherited data as proposed.
[YOCTO #7575]
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Some layers use an older-style method to append a value to BBFILES,
namely BBFILES := "${BBFILES} ..." - this normally works fine, but in
our environment we don't order parsing of layers. That's probably wrong,
but for now just ensure BBFILES has a value to begin with as a
workaround.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix up memory leak fix for bitbake in daisy and earlier which did not
have bb.codeparser.codecache.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
