Make it clearer what this field is for (it should be set for every
comparison layer, so that they don't show up in places they shouldn't).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If other distro comparison package(s) exist for a recipe being shown in
the RRS recipe detail page, link to the page for each package as well as
any extra URLs.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a structure that lets you define a template URL per layer to be
shown per comparison recipe. For example, you could use this to define a
URL template to link to the upstream summary page for the package (e.g.
Fedora's page for the acl package is at
https://apps.fedoraproject.org/packages/acl, so you would use
https://apps.fedoraproject.org/packages/%pn% as the template and then
this would be shown for every package).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to export the comparison search results to CSV format in
order to allow importing the data into external tools.
Note: I implemented this in a different way than the earlier recipe CSV
export, i.e. it uses a template to render the CSV instead of a
function-based view with the Python csv module - the reason for this is
we can reuse the same view as we use for producing the search, with all
of the flexibility that gives us.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When doing the reversed query it's often desirable to exclude recipes
by inherited class, for example those that inherit the packagegroup,
image and meta classes as they don't actually build anything and thus
aren't going to match up with anything in the other distribution.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Provide a lazy-loaded popup for selecting layers to include in the query
instead of having it as a simple drop-down, so you can select more than
one layer.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Enable "reversing" the comparison, so you can see which recipes on the
OE side match up (or don't) with the other distro. The filtering for
this is a bit awkward, since we don't have an actual foreign key for the
link, hence the hairiness of the code.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add support for importing package information extracted from a running
Debian system (i.e. from the output of apt-cache show "*").
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a flag that can be set and searched for to indicate that we need to
take care of importing a package or a patch applied by a package.
Ideally the comments would elaborate on what's needed (if it's not
obvious from the cover status).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Packages with the "Not available" status were inexplicably excluded from
the graph, include them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow searching on:
* Any "available" status (i.e. other than "Unknown" or "Not available")
* Whether the package has patches or not
* What the covering layer is (assuming there is one).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's a bit crude, but accept '' or "" as meaning search for entries
with an empty category.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an option to skip certain recipes by name (a crude workaround for
when we would otherwise get an erroneous match).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the source URL is fetching from pypi then we know it's python;
similarly cpan means perl.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we always set cover_pn if there is a match rather than only setting
it if it's different from pn then it simplifies the code elsewhere.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Only apply OE-Classic-specific logic after checking that it's actually
OE-Classic
* Ignore recipes marked deleted
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script to import "recipe" information from other distro metadata,
based on the import_classic.py script. At the moment, this assumes a
directory where each subdirectory is a package directory containing a
spec file; this would be suitable for distributions such as Fedora
assuming you have all of the package repos checked out locally. Since
you can add additional information to these records (the cover fields
pointing to matching recipes), existing records are updated rather than
deleting everything and re-importing, and we only mark records as
deleted rather than actually deleting them (in case you accidentally
point the script at an empty directory or similar).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Turn the existing OE-Classic support into something a bit more
generic so we can import data from other distributions and compare it to
what we have in layers.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were sorting the layers in the wrong order when trying to choose
which layer to pick a matching recipe from - it needs to be descending
priority order, not ascending.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script that uses the REST API to gather information from another
running layer index instance (e.g. layers.openembedded.org) and import
that into the local index. Only information for branches that are
already set up is imported, and only manually entered information -
no recipes, machines, etc.
Partially implements [YOCTO #9760].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the cover status for a comparison recipe is Unknown, Not available,
or Distro-specific, then we disable the recipe and layerbranch fields in
the form (using Javascript), however the form could still be saved and
any existing values would persist. Clear out the fields upon saving if
they are set under these circumstances.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Add a few items
* Merge the top feature lines with the bottom ones
* Drop an item that has basically been satisfied by way of the REST API
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If there are errors we need to close the div or it'll expand to
encompass all of the remaining fields in the form.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Check if bitbake directory can be found before trying to do anything
with the repo
* Split try..finally into two so we don't try to shut down a nonexistent
tinfoil when we failed to gain a lock.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We weren't giving the layersapp container access to the layer
repositories, which meant that the "Bulk change" function (which lets
you generate patches on top of recipes to change certain variable
values) could not work. Enable the volume and rearrange the order so
that it does, and name the volume more appropriately.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Several distro-provided versions of Docker I have used are too old to
support the --mount option, so rather than making users find and replace
it in the instructions, just revert to the old-style -v option.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The Docker-based setup method is preferred for production, so rearrange
things a little to make it a bit easier to follow.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The home directory path in the commented line here had not been updated
when we switched to running the app as a normal user ("layers"), so do
that now. Additionally we need to copy the file *after* we create the
user, otherwise the home directory will be created by the COPY operation
first and will thus end up being owned by root which we don't want.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Docker commits changes after each RUN stage and commands that delete
files must part of the sane RUN command to actually reduce image size.
Debian stretch contains pip 9.0.3 and the upgrade to 10 caused pip
install to fail.
Signed-off-by: Konrad Scherer <Konrad.Scherer@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Put NGINX, Celery, and RabbitMQ into their own separate containers
* Use a docker network instead of the deprecated --link
* Allow for collecting the static files properly
* Create a copy of settings.py specifically for the docker setup. This
will need to be kept in sync with the main example settings.py, but
it avoids the user having to edit it too much.
* Add optional SSL configuration using letsencrypt certificate
* Create some volumes for static files / fetched repos
* Add some more helpful setup instructions
Largely based upon work by Michael Halstead <michael@yoctoproject.org>.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we're starting a new database, or one of the other applications
(i.e. dependencies of the main layerindex application) has been
upgraded, we need to be migrate all of the applications rather than just
layerindex, so have migrate.sh do that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's best practice to run as a normal account, so create one and use it.
Thanks to Michael Halstead <michael@yoctoproject.org> for this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Use plain debian instead of buildpack-deps as a base (and install just
the additional packages we need on top)
* Purge the development packages after we're done with them (they aren't
used at runtime)
* Run apt-get clean to drop package cache files
Thanks to Konrad Scherer <konrad.scherer@windriver.com> for this idea.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we split dealing with requirements.txt out before we copy the entire
source tree then docker can keep that part cached more often when we're
making changes to the code and thus rebuilds are faster. Thanks to
Konrad Scherer <konrad.scherer@windriver.com> for this idea.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Python 3 is a lot more sensitive to locale, plus we will definitely be
dealing with non-ASCII names and email addresses, so we need to get this
right.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Python 2 is only supported for indexing layers in older branches - the
web interface should be running under Python 3.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Use LABEL instead of the deprecated MAINTAINER
* Use COPY instead of ADD. There's no difference in operation here, it's
just that we don't need the extra magic that ADD provides and best
practice is to use COPY in that case.
* Fix the mkdir /opt/workdir line - it wasn't being run since it was a
CMD not RUN, and thus was overridden by a later CMD directive.
* Drop the CMD line to run celery - having more than one CMD directive
does not work. We'll launch a separate container to run it instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
A while ago I was trying to help someone new get started with a
development setup for the layer index and it occurred to me that the
virtualenv-based environment I am using for development isn't actually
covered in the documentation, so I wrote something quick for them but
didn't do anything further with it until now. Here it is in more fleshed
out form for the README.
Also clarify the python requirements for the update script a little bit.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Up to this point we haven't had any regression tests for the layer
index, but the application (in particular the update script) has become
rather complicated, and we have had a few regressions, so here are some
tests. I've implemented them using pytest and pytest-django; I chose
pytest since we are starting from scratch here and I like the idea of
pytest's fixtures among other features. Annoyingly though because of our
use of separate scripts that need to perform database operations I had
to hack around some of the behaviour of pytest-django, which is clearly
not designed with this kind of structure in mind, but that's taken care
of now. Note that I've only considered backend testing for the moment,
there's not yet a strategy for testing the UI.
To run the tests you simply run "pytest" in the root of the repository.
You will need to have a working configuration set in settings.py (a
database needs to be set, but won't actually be used), and if you're
using MariaDB/MySQL then you'll need the READ COMMITTED transaction
isolation mode selected.
At the moment there are only a few basic tests for the update script
and a bunch of comments describing some we should add. The tests use
a newly added synthetic meta-layerindex-test layer on
git.yoctoproject.org so we can have something with known and fairly
static content. I expect we will extend these tests in the near future.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a file is moved (renamed) to a path outside of the layer, e.g.
another layer within a multi-layer repository, then we need to treat it
as a delete. Up until now we were updating the path and continuing, and
then the recipe was also picked up as an add in the other layer, leading
to duplicate recipe entries. I'd noticed these duplicates before but up
until now I'd thought that they were due to another bug we already
fixed, apparently not.
In order to remove these erroneous duplicate entries in existing
databases I have also added a layerindex/tools/fixup_duplicates.py
script. I've also made the -r/--reload option delete them as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to show a notice at the top of every page; this provides
the ability for admins to display a message to visitors in the case of
infrastructure or index data issues. Notices can have an expiry date and
can be disabled and re-enabled if needed. A subset of HTML can be used
for formatting the text, URLs will be made into clickable links, and
four "levels" are supported (info, success, warning and error).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* If a missing dependency is not required, show a warning instead of an
error
* If logger isn't specified we still need to skip to the next item, so
move the continue statement out of the conditional block. (In practice
I don't think this function is currently called anywhere in the code
without a logger specified, but let's fix it anyway).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a LAYERRECOMMENDS relationship is not satisfied, we shouldn't be
erroring out - it's a recommendation, not a hard dependency. Just show a
warning and allow processing to continue.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
