The new client/server API of tinfoil requires explicit call of
shutdown method to send the event for finalize cooker process.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Remove ; and spaces when setting variables in calls.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Don't assume remote name is origin, run `git remote` to get the remote
name. When checking the remote, detect the branch as well, that way the
layerindex will work if the remote branch name and local branch name do
not match.
Note that this currently only supports one remote.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
It's a little crude and certainly not optimal performance-wise, but we
can support querying for recipes that inherit a particular class without
too much trouble. This allows you to add "inherits:cmake" to the query
and have it return only recipes that inherit the cmake class. You can
use more than one inherits: item to filter down to recipes that inherit
all of the specified classes.
Note: this does not otherwise change the behaviour of specifying
multiple words - all of the words other than those that start with
"inherits:" are treated as part of a single phrase that will be searched
for - not separate keywords.
Fixes [YOCTO #9879].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
At the moment it's a bit difficult to get update logs out of the
environment in which the update script is being run. In order to make
the logs more accessible, create a LayerUpdate model to record the
output of update_layer.py separately for each layerbranch and tie the
created LayerUpdates together with a single Update model per session.
We provide two ways to look at this - a Tools->Updates page for
logged-in users, and there's also an "Updates" tab on each layer that is
accessible to anyone; which one is useful depends on whether you are
looking at the index as a whole or an individual layer.
Update records older than 30 days are deleted automatically by default.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you specified printerr=False we were referring to the output variable
that hadn't been set. Looks like I broke this back in 2013 in
93ce26f21c.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an action to duplicate a Branch object, along with all of the
LayerBranches (and LayerMaintainers and LayerDependencies) underneath
it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
update_layer may create the layerbranch record (e.g. for a new branch)
so we should be looking for it after that has been run. Also, we cannot
assume that last_rev will get set because a layer might fail to fetch,
so take that into account as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a layer is removed by its subdirectory being deleted then we want to
pick up on that and produce an appropriate error message - so let the
layer_update code do the checking out and verifying things are correct
before trying to parse layer.conf.
This also fixes --nocheckout still checking out a branch.
Additionally, drop some code that gets the layerbranch which we just
retrieved a few lines above.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the layer failed to fetch, we shouldn't be attempting to update its
contents. (This got broken when we split the update script into two in
c64e4c57a9).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Sometimes people put values that aren't URLs into the HOMEPAGE variable.
If that's the case, then we should not turn that value into a link which
will be invalid.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For some repo URLs we can automatically determine the values of all of
the other fields - e.g. for github or git.openembedded.org, and we've
been doing that for a while. However if someone submits a URL for some
other site we don't know what type of interface it uses, and usually
the submitter leaves the fields blank so it falls to the layer index
maintainer to set the values, and then you have to remember what the
correct URL format is which is awkward especially for gitweb.
In order to fix this, add a select field to the form which allows
specifying which type of interface is being used. At the moment only
cgit, gitweb, gitlab and "(custom)" (i.e. the current behaviour) are
supported. This is not a real field but activates javascript code
that sets the other fields and enables/disables the controls.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Improves slightly on 3155206e54 by doing
an exact match on name and showing that first - now when you search for
"git" you really do get the git recipe first in the list.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Usage of itertools.chain() that was introduced in
3155206e54 in order to prioritise matches
in the recipe name resulted in recipes showing up twice in the results
if they matched in both the name and the recipe name. Use a custom
chaining function that skips duplicate results in order to fix this.
Fixes [YOCTO #10177].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to keep primary keys from constantly changing, preserve the
existing keys as much as possible.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Parse layer.conf and add dependencies that are not required from
LAYERRECOMMENDS_<name>. Update the layerindex/template to support
recommends. Uses bitbake parsing code & checks versions.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Added associated migration.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Collection and version will be pulled from the layer.conf if it exists
and dependencies will be resolved by first checking for layers with the
dependency name and then checking for collections.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Added associated migration.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Read dependencies from layer.conf and try to create the LayerDependency
entry by looking up the correct database object. Dependencies are found
by layer name only - no collection support. layer.conf parsing is
handled by the bitbake code.
Once all layers are added, the dependencies have to be rechecked in case
the layers are not added in order.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
import_project will scan through a project and find any layer and add it
to the database by calling import_layer on each layer. This differs
from import_layer as it tires to figure out the remote url and uses the
subdirectory (if one exists) as the name.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Add the distros to the index. This looks a lot like the machines
and allows users to search for a particular distro.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Added associated migration.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Convert binary strings to strings and strip leading/trailing whitespace
prior to returning errors and output.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Subdirectories are scanned when adding layers. If any of the
subdirectories or root directory layers already exist in the database,
then the addition fails. This changes that behaviour to report the
failure as a warning and remove it from the list. That way, if a repo
has a new layer added it can be rescanned without issue. Layers being
rescanned are checked against the vcs_url to ensure there is not a name
collision. A name collision without the same vcs_url will still produce
a hard failure.
Note that multiple layers with the same vcs_url are supported in the
error reporting even though this should never happen.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
Django will produce a cryptic error message if layers are added with
invalid names. Sanitize the layer names when trying to add them.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
We need to create an initial migration before we create any more
migrations that change the database structure, which we're about to do.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Replaced fixture with a function in the initial migration so that we
continue to get a master branch record created with Django 1.8.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The multiconfig changes broke the calls here to loadDataFull(). To avoid
this being an issue in future, make use of tinfoil's new parse_recipe_file()
function (if available) to isolate the code here from any future changes to
bitbake's internal code.
Part of the fix for [YOCTO #10192].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
With Python 3 we need to take care of decoding the output so we can
treat it as a string. At the same time, it's more useful to see the
output string rather than the exception if there is some output.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This is apparently required in Python 3.3+, although I have been unable
to verify this locally.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The updated field on the branch table was not null and this breaks the
fixture. It's too painful to specify a value for a datetime field in a
fixture json file and it wouldn't even make sense to specify a value in
this case anyway, so just allow the field to be null.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Django 1.8 warns that the default for the "permanent" parameter to
RedirectView is changing in 1.9 from True to False, but I believe we
should be specifying False for these redirections - these are not just
redirections from old URLs and may in fact change in the future if the
site structure changes.
Part of the implementation for [YOCTO #9620].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The old transaction API has been removed in Django 1.8 and was
deprecated at 1.6. There's no explicit open transaction, commit or
rollback now - we just wrap the layer operations in
"with transaction.atomic()"; if we need to roll back we just raise a
"dummy" exception.
Part of the implementation for [YOCTO #9620].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Django 1.8 does not allow DateFields that have both auto_now and default
set - since they are mutually exclusive; if you have auto_now you don't
need a default.
Part of the implementation for [YOCTO #9620].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a model to support setting a python command and virtualenv per
branch, which allows you to parse master with python3 and krogoth with
python2 for example.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Newer versions of GitPython implement rename detection, which means that
such changes don't show up as adds and deletes anymore and you get a
bunch of ENOENT errors for renamed files. Add some code to explicitly
look for renames and process them appropriately.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Stop using "d" (sometimes multiple levels!) and use proper descriptive
variable names instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We need to be able to support Python 3 so that we can parse master of
OE-Core with bitbake (which now requires it). This now means the
interface itself and the update script require Python 3.4+.
Part of the implementation for [YOCTO #9704].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We have to upgrade django-reversion to 1.8 due to upgrading Django, but
unfortunately in that same version the author has removed the type field
on Version model, without a particularly good explanation as to why. This
is really annoying as we were using it to provide a reasonable audit
including for deletes. I suspect we'll need to move away from
django-reversion and do our own thing in future, but for now at least
allow the layer index to keep working.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I'd like to be upgrading to 1.8 but that causes problems with South, and
we're not quite ready to dispense with our existing migrations yet.
Part of the implementation for [YOCTO #9620].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were getting temporary paths appearing in the tarball that gets
created to allow you to download the patches for a bulk change operation
that crosses multiple layers - we don't need those.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix "data truncated" warnings/errors when loading data for OE-Core:
* PROVIDES for recipe packagegroup-base is ~1452 characters
* LICENSE for recipe linux-firmware is ~1053 characters
(These aren't shown with SQLite, only with something like MariaDB where
column sizes are enforced.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fix a greedy regex in the recipe detail view resulting in the git recipe
listing all bbappends named *_git.bbappend as its bbappends (quite a few
in the public instance).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow updating multiple branches, and if no branches are specified,
update all branches that have a new "updates_enabled" flag field set to
True. This avoids the need to have a separate shell script which runs
update.py for each branch (and thus has hardcoded knowledge of each
active branch in the index, i.e. it needs to be kept up-to-date in
addition to the database.)
The migration will default updates_enabled to True for all branches so
if you wish to take advantage of this functionality, the flag will need
to be set to False for any branches that shouldn't be updated.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to try to avoid problems with leaking memory, context bleeding
from one layer to another, and lay the groundwork for supporting
updating all branches in one operation, split the updating of a single
layer out to its own internal script. This means that the tinfoil
instantiation and metadata parsing is in a completely separate process
per layer.
Implements [YOCTO #9647].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you're moving a short description value from DESCRIPTION to SUMMARY
then part of that is setting DESCRIPTION to blank, however that wasn't
possible - the code was assuming that a null value meant "keep the
original value". Change the logic so that the value in the bulk change
object is always set and is compared to the original value to see if it
is different. This provides less safety against bulk change data going
stale in the face of the metadata being updated, but without using an
additional "magic" field value that's the price we have to pay, and it's
unlikely to bother too many people I would imagine.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
