This patch adds asynchronous task execution using a Celery backend
and RabbitMQ task queue, so that the layer submission process to
proceed even in the event that sending the notification email fails,
and establishing an asynchronous execution mechanism that we can use
in the future e.g. for triggering parse operations from the web UI.
This pertains to bug 11197:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11197
It updates the README to reflect the installation and configuration
of a basic RabbitMQ setup, adds a 'tasks.py' file to contain task
definitions, updates the 'edit_layer_view' function to send
emails to administrators about new and updated layers asynchronously,
modifies the 'settings.py' to include a default configuration
for a RabbitMQ connection, and updates the Dockerfile to start a Celery
worker alongside the Gunicorn daemon.
Fixes [YOCTO #11197].
Signed-off-by: Diana Thayer <garbados@gmail.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Before, if a user edited a layer's name, they would be redirected
to a url utilizing the old name and then receive a 404 Page not
found error. Now, the url utilizes the new name.
Fixes [YOCTO #11932]
Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
At the moment the URL that is presented in the review email will have
http:// or https:// prefix depending on what the user who submitted the
layer was using, but that's irrelevant - we actually want https:// if
the server is capable of it since the reviewer may be redirected to log
in (and Django's login_required decorator will always redirect to the
login page if http is being used as far as I observed, which is a bit
annoying if you are already logged in.) Add a setting which if enabled
will substitute https:// as the prefix.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The branch is not needed any more when it has been removed from the repo, so we
also need remove its layerbranch, otherwise it still can be got from the web,
which causes confusions.
Note, we have to move the location of tinfoil's code to avoid creating tinfoil
when not needed, otherewise, if tinfoil is created (but not needed), and the
program exists earlier before "try ... finally" block, then tinfoil.shutdown()
doesn't run so that it is not shutdown. Move the code back, right before where
it is needed can fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
-p, --prune
Before fetching, remove any remote-tracking references that no longer exist on the remote.
Fixed:
$ git push origin :test_branch
$ ./update.py
The test_branch was still in fetched local repo which was incorrect, it should
be gone since it has been removed by upstream.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It doesn't need to be so ahead since we only need it when writing database, and
a following patch will remove layerbranch from database when the branch had
been removed from the repo, it's not easy to do the work in
transaction.atomic() block.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It never works since it is in the middle of transaction.atomic() block, and
update.py doesn't need it any more, so remove it.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Problems
The update.py couldn't handle new (not only new branch, in fact, but also
existing branches, see below for more info) branch well, for example, there are
3 layers: layer_A, layer_B and layer_C, and create new branch "branch_1" for
them, and they have depends:
layer_A -> layer_B -> layer_C
The "->" means depends on.
Then run "update.py -b branch_1", there would be errors like:
ERROR: Dependency layer_B of layer_A does not have branch record for branch branch_1
Though update.py runs "update_layer.py" twice, but it didn't help since
layerbranch was None when it was failed to create in the first run.
The reason is if update.py updates layer_A firstly, it would fail since it
can't find layer_B:branch_1 in database (not added to database yet), similarly,
if add layer_B before layer_C, it would also fail. Only layer_C can be added
(assume it has no dependencies). So we have to re-run update.py again and again
to make it work, here we may have to run update.py 3 times, and more runs are
needed if the dependency chain is longer.
* Solutions:
Make update.py pass layers orderly to update_layer.py according to dependencies
can fix the problem, we can get intial dependencies from tinfoil, add an option
"-i, --initial" to update_layer.py to get them.
Not only new branch, but also existing branches may have the problem, because
collections and dependencies maybe changed in the coming update, so we can't
trust database when the layer is going to be updated, for example, if there are
10 layers in database, and 3 of them will be updated (-l layer1,layer2,layer3),
then we can not use the 3 layers' collections data from database, we need get
them from tinfoil.
* Performance improvement:
It should be the same as before in theory, I have tested it with 97 layers:
- Before: 11m6.472s, but only 75 layers were added, 22 ones were failed, I
have to re-run update.py again and again (maybe 4 times to make all of
them added). So:
(11 * 60 + 6)/75*97/60 = 14m35s
- Now 12m10.350s, all the layers are added in the first run.
So about 2 minutes are saved.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we don't restore the current working directory, it may cause trouble
for the calling function.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The _add_dependency() uses:
if layerbranch.collection:
var_name = layerbranch.collection
The layerbranch.collection is none if it is newly created, thus it can't get
LAYERDEPENDS, because what defined in layer.conf is LAYERDEPENDS_<collection>,
but what it would get is LAYERDEPENDS_<layer_name>, this patch can fix the
problem.
Reproducer:
$ python3 update_layer.py -l mete-xfce -b <newbranch> --fullreload -d
It would get None LAYERDEPENDS.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
1) set LAYERDEPENDS_openembedded-layer = "core"
2) $ "update.py -l meta-oe -b master"
Check from web, its dependency is "openembedded-core"
3) Change LAYERDEPENDS_openembedded-layer = "foo"
4) Run "update.py -l meta-oe -b master"
5) Check from web, its dependency is "openembedded-core and foo", this might
be incorrect, now if set REMOVE_LAYER_DEPENDENCIES to true, the old
dependency openembedded-core will be removed, the default is False which
prints warnings to notify users.
And also the existing checking should filter(required=required), otherwise it
can't work well when a layer is in both depends and recommends, this can't
happen in a normal case, but it would surprise the user when this happens.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The layerbranch is not used in parse_layer(), so remove it.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an option "-a" to update actual branch for layer and bitbake, it is
useful when there are many layers and need update actual branches
frequently. We only can update them via website without this patch,
which is not fun and easy to make mistakes.
* It works with "-l", and "-l bitbake" means update bitbake branch.
* It requires "-b" to work, and only one branch is supported in a run.
For example:
$ update.py -b master -a branch_20170526
All the layers which have branch master and actual_branch branch_20170526
will be updated to branch_20170526.
$ update.py -b master -l meta-oe -a branch_20170526
Only meta-oe layer will be updated.
$ update.py -b master -l bitbake -a branch_20170526
The bitbake's bitbake_branch will be updated.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This change supports querying recipes that belong to a specific layer
by using the prefix "layer:" + the desired layer name, for example:
"layer: openembedded-core" and this string can be used by itself or
combined with other supported options.
A descriptive error message is displayed when the query string has an
unexpected formatting or a non-valid layer name is searched.
[YOCTO #6618]
Signed-off-by: Jose Lamego <jose.a.lamego@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a status for a layer indicating it should not be updated. I don't
expect this to be widely used (and is only settable from the admin
interface) but would be useful if you have a legacy sub-layer that you
want to prevent from being visible on certain branches - it will prevent
the update script from doing anything with the layer and thus avoid
branch records from being auto-created on branches where you've deleted
it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When reading conf/distro/*.conf to create distro records, attempt to
parse the config file and use DISTRO_NAME (if set) to populate the
description field for the record. If that's not set then fall back to
the less commonly used meta-comment that we supported previously.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we want to parse a configuration file (e.g. a distro conf file) then
we need convenient access to bitbake's conf parsing code, so create a
parse_conf() function to provide that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When the recipe search returns a single result, redirect to it
instead of showing the full result list view.
Part of the fix for [YOCTO #6618].
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The branch meta change made in b91b09a0ae
apparently requires this migration - not entirely sure why, but Django's
manage.py makemigrations wants to create it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make minor (non-structural) improvements to the model, mostly for the
benefit of the admin interface:
* Set default ordering so that branches appear in the same order they do
in the user-facing drop-down
* Include the description in the default string representation (so you
know which branch name matches with which Yocto Project version, in
the OE index).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We can't decode UTF-8 characters byte-by-byte, as soon as we hit a
character that's more than one byte long then we'll fail. Use a reader
object to do it properly. This fixes parsing current meta-angstrom on
master. At the same time, specify errors="surrogateescape" to avoid the
update process dying at this point in case of characters that aren't
valid UTF-8.
Thanks to Jiajie Hu <jiajie.hu@intel.com> who fixed this in devtool's
very similar code.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The fido branch of OE-Core expects BitBake 1.26.x. Unfortunately that
version had a commit backported which introduced a non-working
tinfoil.shutdown() method, so we can't simply rely on its presence to
determine whether or not it should be called. Use a check on the BitBake
version number instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Layers that aren't published shouldn't be visible via the API. (We don't
need to apply that filter to recipes, machines or distros though since a
layer's content won't automatically be indexed unless it has been
published).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
On the main layer index page we want openembedded-core to appear before
meta-oe - naturally openembedded-core is the layer everyone starts from.
We already have an "index_preference" column that we use for search
results, so just sort by that here before sorting by name.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Since some recent infrastructure changes, "/cgit.cgi" should no longer
be part of cgit.openembedded.org URLs, apparently.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix the transaction handling code to work with Django 1.8 in most of the
tools scripts. (Some of these are no longer used, but still serve as
examples of how to import data and update the database.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* We were passing the incorrect path (to the top of the layer repo) if
the layer had a subdirectory, so this doesn't seem to have been able
to work for such layers previously.
* Doing this update in the main update.py script meant that this could
never work across branches requiring a python version change (using
PythonEnvironment records) since the code was running within the same
environment in which update.py was launched - the entire point of the
separation of the two scripts. Move the checking to update_layer.py
and call it separately to perform these updates, splitting out some
common code in order to do so.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
A minor optimisation - it's not going to change between layers so get it
in the branch loop.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Setting of the object has to be before the try: or otherwise the
finally block can get called if that doesn't succeed with the result
that the layerconfparser object won't have a value, which will
trigger an exception
* We shouldn't be using the config_data object after shutting down
tinfoil, so avoid doing that
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix a couple of problems with shutting down tinfoil:
1) tinfoil.shutdown() wasn't being called on error because it wasn't
inside a finally: section, thus on error the script hung with bitbake
master (since the tinfoil2 changes) - this is almost certainly a bug
in bitbake but let's handle it here anyway.
2) We need to check whether the tinfoil object actually has a shutdown
attribute since we still want to support updating for older branches
where tinfoil didn't have a shutdown() method.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The new client/server API of tinfoil requires explicit call of
shutdown method to send the event for finalize cooker process.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Remove ; and spaces when setting variables in calls.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Don't assume remote name is origin, run `git remote` to get the remote
name. When checking the remote, detect the branch as well, that way the
layerindex will work if the remote branch name and local branch name do
not match.
Note that this currently only supports one remote.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
It's a little crude and certainly not optimal performance-wise, but we
can support querying for recipes that inherit a particular class without
too much trouble. This allows you to add "inherits:cmake" to the query
and have it return only recipes that inherit the cmake class. You can
use more than one inherits: item to filter down to recipes that inherit
all of the specified classes.
Note: this does not otherwise change the behaviour of specifying
multiple words - all of the words other than those that start with
"inherits:" are treated as part of a single phrase that will be searched
for - not separate keywords.
Fixes [YOCTO #9879].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
At the moment it's a bit difficult to get update logs out of the
environment in which the update script is being run. In order to make
the logs more accessible, create a LayerUpdate model to record the
output of update_layer.py separately for each layerbranch and tie the
created LayerUpdates together with a single Update model per session.
We provide two ways to look at this - a Tools->Updates page for
logged-in users, and there's also an "Updates" tab on each layer that is
accessible to anyone; which one is useful depends on whether you are
looking at the index as a whole or an individual layer.
Update records older than 30 days are deleted automatically by default.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you specified printerr=False we were referring to the output variable
that hadn't been set. Looks like I broke this back in 2013 in
93ce26f21c.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an action to duplicate a Branch object, along with all of the
LayerBranches (and LayerMaintainers and LayerDependencies) underneath
it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
update_layer may create the layerbranch record (e.g. for a new branch)
so we should be looking for it after that has been run. Also, we cannot
assume that last_rev will get set because a layer might fail to fetch,
so take that into account as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a layer is removed by its subdirectory being deleted then we want to
pick up on that and produce an appropriate error message - so let the
layer_update code do the checking out and verifying things are correct
before trying to parse layer.conf.
This also fixes --nocheckout still checking out a branch.
Additionally, drop some code that gets the layerbranch which we just
retrieved a few lines above.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the layer failed to fetch, we shouldn't be attempting to update its
contents. (This got broken when we split the update script into two in
c64e4c57a9).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Sometimes people put values that aren't URLs into the HOMEPAGE variable.
If that's the case, then we should not turn that value into a link which
will be invalid.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For some repo URLs we can automatically determine the values of all of
the other fields - e.g. for github or git.openembedded.org, and we've
been doing that for a while. However if someone submits a URL for some
other site we don't know what type of interface it uses, and usually
the submitter leaves the fields blank so it falls to the layer index
maintainer to set the values, and then you have to remember what the
correct URL format is which is awkward especially for gitweb.
In order to fix this, add a select field to the form which allows
specifying which type of interface is being used. At the moment only
cgit, gitweb, gitlab and "(custom)" (i.e. the current behaviour) are
supported. This is not a real field but activates javascript code
that sets the other fields and enables/disables the controls.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Improves slightly on 3155206e54 by doing
an exact match on name and showing that first - now when you search for
"git" you really do get the git recipe first in the list.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Usage of itertools.chain() that was introduced in
3155206e54 in order to prioritise matches
in the recipe name resulted in recipes showing up twice in the results
if they matched in both the name and the recipe name. Use a custom
chaining function that skips duplicate results in order to fix this.
Fixes [YOCTO #10177].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to keep primary keys from constantly changing, preserve the
existing keys as much as possible.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Parse layer.conf and add dependencies that are not required from
LAYERRECOMMENDS_<name>. Update the layerindex/template to support
recommends. Uses bitbake parsing code & checks versions.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Added associated migration.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Collection and version will be pulled from the layer.conf if it exists
and dependencies will be resolved by first checking for layers with the
dependency name and then checking for collections.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Added associated migration.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Read dependencies from layer.conf and try to create the LayerDependency
entry by looking up the correct database object. Dependencies are found
by layer name only - no collection support. layer.conf parsing is
handled by the bitbake code.
Once all layers are added, the dependencies have to be rechecked in case
the layers are not added in order.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
import_project will scan through a project and find any layer and add it
to the database by calling import_layer on each layer. This differs
from import_layer as it tires to figure out the remote url and uses the
subdirectory (if one exists) as the name.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Add the distros to the index. This looks a lot like the machines
and allows users to search for a particular distro.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Added associated migration.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Convert binary strings to strings and strip leading/trailing whitespace
prior to returning errors and output.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Subdirectories are scanned when adding layers. If any of the
subdirectories or root directory layers already exist in the database,
then the addition fails. This changes that behaviour to report the
failure as a warning and remove it from the list. That way, if a repo
has a new layer added it can be rescanned without issue. Layers being
rescanned are checked against the vcs_url to ensure there is not a name
collision. A name collision without the same vcs_url will still produce
a hard failure.
Note that multiple layers with the same vcs_url are supported in the
error reporting even though this should never happen.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Django will produce a cryptic error message if layers are added with
invalid names. Sanitize the layer names when trying to add them.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
We need to create an initial migration before we create any more
migrations that change the database structure, which we're about to do.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Replaced fixture with a function in the initial migration so that we
continue to get a master branch record created with Django 1.8.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The multiconfig changes broke the calls here to loadDataFull(). To avoid
this being an issue in future, make use of tinfoil's new parse_recipe_file()
function (if available) to isolate the code here from any future changes to
bitbake's internal code.
Part of the fix for [YOCTO #10192].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
With Python 3 we need to take care of decoding the output so we can
treat it as a string. At the same time, it's more useful to see the
output string rather than the exception if there is some output.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This is apparently required in Python 3.3+, although I have been unable
to verify this locally.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The updated field on the branch table was not null and this breaks the
fixture. It's too painful to specify a value for a datetime field in a
fixture json file and it wouldn't even make sense to specify a value in
this case anyway, so just allow the field to be null.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Django 1.8 warns that the default for the "permanent" parameter to
RedirectView is changing in 1.9 from True to False, but I believe we
should be specifying False for these redirections - these are not just
redirections from old URLs and may in fact change in the future if the
site structure changes.
Part of the implementation for [YOCTO #9620].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The old transaction API has been removed in Django 1.8 and was
deprecated at 1.6. There's no explicit open transaction, commit or
rollback now - we just wrap the layer operations in
"with transaction.atomic()"; if we need to roll back we just raise a
"dummy" exception.
Part of the implementation for [YOCTO #9620].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Django 1.8 does not allow DateFields that have both auto_now and default
set - since they are mutually exclusive; if you have auto_now you don't
need a default.
Part of the implementation for [YOCTO #9620].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a model to support setting a python command and virtualenv per
branch, which allows you to parse master with python3 and krogoth with
python2 for example.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Newer versions of GitPython implement rename detection, which means that
such changes don't show up as adds and deletes anymore and you get a
bunch of ENOENT errors for renamed files. Add some code to explicitly
look for renames and process them appropriately.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Stop using "d" (sometimes multiple levels!) and use proper descriptive
variable names instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We need to be able to support Python 3 so that we can parse master of
OE-Core with bitbake (which now requires it). This now means the
interface itself and the update script require Python 3.4+.
Part of the implementation for [YOCTO #9704].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We have to upgrade django-reversion to 1.8 due to upgrading Django, but
unfortunately in that same version the author has removed the type field
on Version model, without a particularly good explanation as to why. This
is really annoying as we were using it to provide a reasonable audit
including for deletes. I suspect we'll need to move away from
django-reversion and do our own thing in future, but for now at least
allow the layer index to keep working.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I'd like to be upgrading to 1.8 but that causes problems with South, and
we're not quite ready to dispense with our existing migrations yet.
Part of the implementation for [YOCTO #9620].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were getting temporary paths appearing in the tarball that gets
created to allow you to download the patches for a bulk change operation
that crosses multiple layers - we don't need those.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix "data truncated" warnings/errors when loading data for OE-Core:
* PROVIDES for recipe packagegroup-base is ~1452 characters
* LICENSE for recipe linux-firmware is ~1053 characters
(These aren't shown with SQLite, only with something like MariaDB where
column sizes are enforced.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix a greedy regex in the recipe detail view resulting in the git recipe
listing all bbappends named *_git.bbappend as its bbappends (quite a few
in the public instance).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow updating multiple branches, and if no branches are specified,
update all branches that have a new "updates_enabled" flag field set to
True. This avoids the need to have a separate shell script which runs
update.py for each branch (and thus has hardcoded knowledge of each
active branch in the index, i.e. it needs to be kept up-to-date in
addition to the database.)
The migration will default updates_enabled to True for all branches so
if you wish to take advantage of this functionality, the flag will need
to be set to False for any branches that shouldn't be updated.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to try to avoid problems with leaking memory, context bleeding
from one layer to another, and lay the groundwork for supporting
updating all branches in one operation, split the updating of a single
layer out to its own internal script. This means that the tinfoil
instantiation and metadata parsing is in a completely separate process
per layer.
Implements [YOCTO #9647].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you're moving a short description value from DESCRIPTION to SUMMARY
then part of that is setting DESCRIPTION to blank, however that wasn't
possible - the code was assuming that a null value meant "keep the
original value". Change the logic so that the value in the bulk change
object is always set and is compared to the original value to see if it
is different. This provides less safety against bulk change data going
stale in the face of the metadata being updated, but without using an
additional "magic" field value that's the price we have to pay, and it's
unlikely to bother too many people I would imagine.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Display blacklist information for recipes in the recipe details,
as well as the recipe search page, and layer page tables. This
information is pulled from the PNBLACKLIST variable. Includes a
hover text containing the reason for blacklist labels.
Changes to Django and Layerindex files:
- models.py, update.py
- migration file
- templates and static content (stylesheet)
[YOCTO #7855]
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Using the search box creates a query against the pns,
summaries, descriptions and filenames of recipes.
This results in a lot of spurious results for common terms
like "git" when performing a recipe search.
Make the results more useful by:
* Only searching against pn, description and summary (not file).
* Putting matches against pn at the top of the list, followed by
matches against description and summary.
[YOCTO #9159]
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
The code to populate the inherits information was using a function
designed to get the recipe name from a file path, but unlike recipe
filenames, the underscore isn't treated as special in class filenames
and in fact it's quite common to use underscores there; we were ending
up with items in the inherit list such as "populate" and "rootfs". Use
the standard python os.path functions to do the splitting instead and
avoid this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
To identify image recipes and provide inheritance data for recipes, an
inherits field was added to the recipe model, and then populated using
refactored data from __inherit_cache. Finally the field was also added
to page templates, along with style changes proposed in attachment, and
an additional style change to display the inherits field as a list in
detailed view. The field skips globally inherited data as proposed.
[YOCTO #7575]
Signed-off-by: Alex Franco <alejandro.franco@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Some layers use an older-style method to append a value to BBFILES,
namely BBFILES := "${BBFILES} ..." - this normally works fine, but in
our environment we don't order parsing of layers. That's probably wrong,
but for now just ensure BBFILES has a value to begin with as a
workaround.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix up memory leak fix for bitbake in daisy and earlier which did not
have bb.codeparser.codecache.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We're kind of abusing some bitbake internals here and the result is that
there are a bunch of global lists and caches that simply grow as we
parse more layers, until available memory is exhausted. We don't care
about the contents of any of these within the layer index update script
so just clear them out between layers.
Should fix [YOCTO #7663].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add default = datetime.now attr to updated field of Branch
model because it fails when try to insert inital data with NULL.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
We add an automatically-updateing "updated" datetime
field to Branch, LayerItem, LayerBranch, Recipe and
Machine models to allow date-based incremental updates
through the API.
The added field does not interfere with the existing
form-based pages (auto_now sets editable to false).
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
This patch enables a read-only REST API for the layerindex
application using Django REST Framework.
The objects of types Branch, LayerBranch and LayerItem are
exposed to queries so that the layerindex application can
function as a Layer Source in Toaster.
The library dependencies are documented in the requirements.txt
file.
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
* Fix "sre_constants.error: nothing to repeat" error due to .* inside
group made optional with ? in regex
* Avoid error if maintainer responsibility is not specified
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When we did a full reload this list wasn't getting cleared for the next
layer, and we look at the contents of the list unconditionally later to
determine which files need to be added. Given that the list items are
full paths, this is unlikely to have caused any problems; however it
could possibly have been an issue for nested layers with the right
update order, or lead to issues if the usage of the list changes in
future.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Some layers, such as meta-intel, also contain other layers. We don't
want recipes/classes/appends in those child layers to appear in the
parent layer so ignore any that appear within subdirectories that
contain a conf/layer.conf file.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
There was a logic error here - we were checking to see if the layer
directory existed before actually checking out the branch, which means
we were looking at the state of whatever branch was previously checked
out. Replace this with some validation of the last revision check and an
additional check for the directory after checking out the branch.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If there is a % character in the path to a file / directory, we need to
properly encode it when creating the repo web URL or the link won't
work. Thanks to Khem Raj for reporting this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
With BSPs being "retired" e.g. in meta-intel, it is possible for layers
to not exist on the master branch; since this is legitimate we need the
layer index to handle it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you want to check what the same search returns on a different branch
it makes things easier if you don't have to re-type the search after
selecting the branch.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Recipes without versions in the file name (such as alsa-state.bb)
weren't having their bbappends listed. Use a regex match to fix this.
* Use the prefix from the filename instead of PN, since that's how
BitBake does it
* List version matching bbappends first, then non-matching in muted
colour
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Update d3/nvd3 js files in the process, but hold off on nv.d3.css for
now because the latest version breaks the formatting/fading on tooltips.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Adds a simple way to add a new layer to the database in an automated
manner, given only a URL and optionally a name and subdirectory. If no
subdirectory is given, the root and all first-level subdirectories that
contain conf/layer.conf will be created as layers. Guesses will be made
as to the right values for certain fields, by looking at
README/MAINTAINERS files and github repository information.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the site is configured to use a different core layer then we ought to
use it for the default layer dependency instead of hardcoding
openembedded-core.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This ensures we get a sensible error message prompting the user to set
the subdirectory, instead of a BitBake parse error.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This allows specifying the site title in the database, instead of hardcoding
it in the template. Just change it to something other than "example.com" to
apply your own title.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Recently the -r/--reload option was changed to preserve existing recipe
data, so that it could be used to load values into newly added fields;
however we still need an option to load recipe data from scratch for
testing purposes (e.g. in combination with --nocheckout) so add a
--fullreload option to do this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to test updating we need to be able to check out a previous
revision of the repository and have the update script load from that
without checking out the latest revision, so fix the --nocheckout
option to work properly.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We always need to replace %path% in the URL, not just when there is a
non-blank path to replace it with.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Change Submit layer link into a button
* Ensure tools dropdown menu doesn't show on layer submit/edit page
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's not really completely functional and puts quite a bit of load on
the server when used, so just disable it for now.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Since it's possible that the user will try to get patches while the
update script is running, ensure we show a sensible error if the script
times out waiting for a lock.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Avoid the possibility of these two clashing especially when multiple
branches are being used.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Drop the front page - this just gets in the way. Redirect to the
layers list instead. This has meant adding a touch more text to the
about page and adding the FAQ link to the footer.
* Use a separate navbar to hold the branch selector and the main
top-level pages (Layers, Recipes, Machines) instead of tabs
* All pages depending on a branch selection are now under
branch/<branchname>/ so we don't need to have the branch selection on
every page.
* Use breadcrumbs on recipe detail and layer detail pages instead of
tabs
* Add title to recipe detail page
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Don't start with all records being shown - only do this if the search
button is pressed with no keyword.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script for doing a one-time import of OE-Classic recipe
information, so comparisons against OE-Core can be performed; this
is stored using a new ClassicRecipe model supporting additional fields
for tracking migration status. The migration status fields can be
updated as well as viewed and summarised in graph format.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Having the page content dance about when switching between pages just
because one has a scrollbar and the other doesn't is annoying, so just
force the scrollbar to show all of the time to avoid this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The recipe detail pages use the recipe record id, so in case users
bookmark links to these pages, try to preserve the id when a recipe
upgrade occurs (which usually shows up as a delete and an add). This
also works when doing a full update with -r so that we can capture the
newly recorded items (bbclasses, bbappends) without having all the
recipe id values change.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This provides a way to set "meta" fields (SUMMARY, DESCRIPTION,
HOMEPAGE, BUGTRACKER, SECTION, and LICENSE) for a number of recipes at
once, and then download those changes in the form of one or more patch
files which can be submitted for merging into the layer.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
To allow re-use outside of the update script, split out parsing setup
code to a new recipeparse module. Also split out runcmd, get_layer,
get_branch and logger_create functions to a separate utils module.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Some layers don't use the conventional branch name (i.e. the same name
as the core branch). Add a field so we can record the actual branch name
and use this within the update script.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Collect bbappend/bbclass info during the update process and display it
on the layer detail page.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Show BUGTRACKER in the recipe detail page if set
* Track the other two variables for possible future use
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an index_preference field to enable control over which layer's
duplicate recipes get de-emphasised in the recipe search results.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Save individual field changes into revision comment and display this
comment on the history page. Now we're ready to add a link at the
bottom of every page so the history is easily visible.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This allows disabling the checkout of the appropriate branch so that we
can test the update script with fixes to bitbake or the metadata if
needed.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's not really appropriate to say "Hi Joe Bloggs" at the start of the
email - we only need the user's first name or the login name if the
first name hasn't been filled in.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The cooker module has been refactored in BitBake master and this moved
and renamed the _parse() function; change the update script to handle
that (whilst still handling older versions).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Report layer which failed to fetch in error message
* Don't retry fetching a repo if it already failed for another layer
(where more than one layer is in the same repository)
* Exit immediately if all fetches failed
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's easy to forget you need to use -l to specify the list of layers to
update and previously we just ignored unused arguments, so check for
them and error out if present.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This includes the layer name in layer note additions/changes in the
history page, among other places.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Somehow due to some CSS shenanigans these checkboxes had 0 width in
Chromium and thus were invisible. A style on the td had to be used
because a style on the input doesn't seem to fix it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We shouldn't be looking for this attribute unless the user has logged in
- this was a regression in the recent email case fix.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This avoids showing a 403 error when a user clicks on a review link in
the layer submission notification email but hasn't logged in yet.
Also protect the review list view with a permission check; it's not that
it's sensitive, but we should be consistent with the detail here.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This actually works for complex URLs since we normalise just the path
portion that we are adding instead of trying it on the URL after
composing it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
gitweb, ever vigilant for the slightest transgression in URL syntax,
fails to find the specified path if it is a directory and has a trailing
slash, so we need to avoid adding this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow using e.g. ../README to point to README file in the parent of the
repository subdirectory.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Avoid truncation for slightly longer fields (as happened with PN and
SUMMARY when importing some existing layers).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Avoid picking up machine files in sub-layers
* Skip scanning of .git directory when doing full refresh
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fall back to NAME if DESCRIPTION is not specified and remove more
preamble from the string.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If another recipe exists with the same name in a different layer and
that other layer is "older" (by layer ID) and is a software or base
layer then lighten the recipe entry in the search results as it may
not be the preferred version (e.g. recipes in BSP or distro layers may
have been customised specifically for the machine or distro).
This has had a performance impact on the recipes list; as a result
showing all recipes by default has been disabled. If the user really
wants to see all recipes they can just leave the search box blank and
hit the search button.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the user edits the layer and changes the repository URL,
subdirectory, or dependencies, ensure that the entire layer gets
refreshed the next time the update script runs by clearing out the last
fetched revision field.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This ensures recipes that failed to parse sort properly instead of all
appearing at the start of the search results.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We don't go through the normal code path where this directory would be
created, and if it doesn't exist an error will be thrown in
insane.bbclass when any QA issue is logged; so create it manually.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Rather than letting BitBake error out, just check for existence of this
file and print a single error line if it doesn't exist.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This should help the initial population of the database and act as a
simple example of how to populate the database programmatically.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we're creating a LayerBranch record, it's helpful to copy the
maintainer and dependency records in from master as they're usually
going to be the same; if they're not the user can always change them
afterwards.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
So it turns out that one or two layers have changed in structure between
branches, so we need to be able to specify this on a per-branch basis.
Good times...
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The recently added global redirect was also redirecting incorrect URLs
within the application to the front page rather than showing 404, which
is not desirable.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This enables sorting the branch drop-down in a configurable manner (e.g.
so that you can have them in order from newest to oldest).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Give the user a friendly introduction instead of plunging them straight
into the layers list. (Of course if people want to link directly to the
layers list they still can.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This cuts out a lot of the elements that aren't needed for the review
list, shows fields in a more suitable way for review purposes than the
standard detail (and includes some fields that don't currently get shown
on the standard detail e.g. layer type and short description).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Newly submitted layers don't have anything other than a master branch,
so they won't display properly unless we have master selected; so just
add a parameter to submission email and review URLs to ensure that is
the case.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Don't assume the current directory contains our conf/local.conf,
explicitly use the directory above where the script is located.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The simple views for about and the submit thanks pages don't need
special views, so the standard TemplateView can be used.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a badge next to the Review link (when shown for users with the
publish permission) on all pages showing how many layers need to be
reviewed, if any.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This means the usage URL can point to a file whose contents may be
different per branch (e.g. a README file).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Change the data structure to support multiple branches. At the top level
there is a set list of Branch objects, and then a LayerBranch object
between each layer and the maintainers, dependencies, recipes and
machines, so that the set of each can be different per branch. The
branch is a session option, and can be selected via a drop-down that is
shown for all pages.
Additionally, with this change we avoid the need to run the update
script within a build environment set up with oe-init-build-env - since
we need a specific version of BitBake per branch we now use our own copy
of BitBake which is fetched by the script itself. The update script will
need to be called multiple times however - once per branch.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
