If there are errors we need to close the div or it'll expand to
encompass all of the remaining fields in the form.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Check if bitbake directory can be found before trying to do anything
with the repo
* Split try..finally into two so we don't try to shut down a nonexistent
tinfoil when we failed to gain a lock.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We weren't giving the layersapp container access to the layer
repositories, which meant that the "Bulk change" function (which lets
you generate patches on top of recipes to change certain variable
values) could not work. Enable the volume and rearrange the order so
that it does, and name the volume more appropriately.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Several distro-provided versions of Docker I have used are too old to
support the --mount option, so rather than making users find and replace
it in the instructions, just revert to the old-style -v option.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The Docker-based setup method is preferred for production, so rearrange
things a little to make it a bit easier to follow.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The home directory path in the commented line here had not been updated
when we switched to running the app as a normal user ("layers"), so do
that now. Additionally we need to copy the file *after* we create the
user, otherwise the home directory will be created by the COPY operation
first and will thus end up being owned by root which we don't want.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Docker commits changes after each RUN stage and commands that delete
files must part of the sane RUN command to actually reduce image size.
Debian stretch contains pip 9.0.3 and the upgrade to 10 caused pip
install to fail.
Signed-off-by: Konrad Scherer <Konrad.Scherer@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Put NGINX, Celery, and RabbitMQ into their own separate containers
* Use a docker network instead of the deprecated --link
* Allow for collecting the static files properly
* Create a copy of settings.py specifically for the docker setup. This
will need to be kept in sync with the main example settings.py, but
it avoids the user having to edit it too much.
* Add optional SSL configuration using letsencrypt certificate
* Create some volumes for static files / fetched repos
* Add some more helpful setup instructions
Largely based upon work by Michael Halstead <michael@yoctoproject.org>.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we're starting a new database, or one of the other applications
(i.e. dependencies of the main layerindex application) has been
upgraded, we need to be migrate all of the applications rather than just
layerindex, so have migrate.sh do that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's best practice to run as a normal account, so create one and use it.
Thanks to Michael Halstead <michael@yoctoproject.org> for this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Use plain debian instead of buildpack-deps as a base (and install just
the additional packages we need on top)
* Purge the development packages after we're done with them (they aren't
used at runtime)
* Run apt-get clean to drop package cache files
Thanks to Konrad Scherer <konrad.scherer@windriver.com> for this idea.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we split dealing with requirements.txt out before we copy the entire
source tree then docker can keep that part cached more often when we're
making changes to the code and thus rebuilds are faster. Thanks to
Konrad Scherer <konrad.scherer@windriver.com> for this idea.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Python 3 is a lot more sensitive to locale, plus we will definitely be
dealing with non-ASCII names and email addresses, so we need to get this
right.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Python 2 is only supported for indexing layers in older branches - the
web interface should be running under Python 3.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Use LABEL instead of the deprecated MAINTAINER
* Use COPY instead of ADD. There's no difference in operation here, it's
just that we don't need the extra magic that ADD provides and best
practice is to use COPY in that case.
* Fix the mkdir /opt/workdir line - it wasn't being run since it was a
CMD not RUN, and thus was overridden by a later CMD directive.
* Drop the CMD line to run celery - having more than one CMD directive
does not work. We'll launch a separate container to run it instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
A while ago I was trying to help someone new get started with a
development setup for the layer index and it occurred to me that the
virtualenv-based environment I am using for development isn't actually
covered in the documentation, so I wrote something quick for them but
didn't do anything further with it until now. Here it is in more fleshed
out form for the README.
Also clarify the python requirements for the update script a little bit.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Up to this point we haven't had any regression tests for the layer
index, but the application (in particular the update script) has become
rather complicated, and we have had a few regressions, so here are some
tests. I've implemented them using pytest and pytest-django; I chose
pytest since we are starting from scratch here and I like the idea of
pytest's fixtures among other features. Annoyingly though because of our
use of separate scripts that need to perform database operations I had
to hack around some of the behaviour of pytest-django, which is clearly
not designed with this kind of structure in mind, but that's taken care
of now. Note that I've only considered backend testing for the moment,
there's not yet a strategy for testing the UI.
To run the tests you simply run "pytest" in the root of the repository.
You will need to have a working configuration set in settings.py (a
database needs to be set, but won't actually be used), and if you're
using MariaDB/MySQL then you'll need the READ COMMITTED transaction
isolation mode selected.
At the moment there are only a few basic tests for the update script
and a bunch of comments describing some we should add. The tests use
a newly added synthetic meta-layerindex-test layer on
git.yoctoproject.org so we can have something with known and fairly
static content. I expect we will extend these tests in the near future.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a file is moved (renamed) to a path outside of the layer, e.g.
another layer within a multi-layer repository, then we need to treat it
as a delete. Up until now we were updating the path and continuing, and
then the recipe was also picked up as an add in the other layer, leading
to duplicate recipe entries. I'd noticed these duplicates before but up
until now I'd thought that they were due to another bug we already
fixed, apparently not.
In order to remove these erroneous duplicate entries in existing
databases I have also added a layerindex/tools/fixup_duplicates.py
script. I've also made the -r/--reload option delete them as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to show a notice at the top of every page; this provides
the ability for admins to display a message to visitors in the case of
infrastructure or index data issues. Notices can have an expiry date and
can be disabled and re-enabled if needed. A subset of HTML can be used
for formatting the text, URLs will be made into clickable links, and
four "levels" are supported (info, success, warning and error).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* If a missing dependency is not required, show a warning instead of an
error
* If logger isn't specified we still need to skip to the next item, so
move the continue statement out of the conditional block. (In practice
I don't think this function is currently called anywhere in the code
without a logger specified, but let's fix it anyway).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a LAYERRECOMMENDS relationship is not satisfied, we shouldn't be
erroring out - it's a recommendation, not a hard dependency. Just show a
warning and allow processing to continue.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a file is modified and renamed it will show up in both
iter_change_type('M') and iter_change_type('R'), however naturally the
file that will exist will be the b path and not the a one, so we should
be looking at the b path or we will get errors.
FYI you can reproduce this with OE-Core (in a scratch database) using
the following procedure:
1) (in the OE-Core layer directory):
git checkout 59285b324f6d9ed270b0bef209ef5da22a620a83
2) update.py -l openembedded-core -b master -x --nofetch -r --fullreload
3) (in the OE-Core layer directory):
git checkout 086308aa2a5e332de6f00ed397c4a55d132f158f
4) update.py -l openembedded-core -b master -x --nofetch
Without this change you'll see the following error:
ERROR: Unable to read /opt/layerindex/layers/git___git_openembedded_org_openembedded-core/meta/recipes-devtools/python-numpy/python-numpy_1.13.1.bb: Traceback (most recent call last):
File "/opt/layerindex/layers/bitbake/lib/bb/command.py", line 84, in runCommand
result = command_method(self, commandline)
File "/opt/layerindex/layers/bitbake/lib/bb/command.py", line 568, in parseRecipeFile
envdata = bb.cache.parse_recipe(config_data, fn, appendfiles)['']
File "/opt/layerindex/layers/bitbake/lib/bb/cache.py", line 315, in parse_recipe
bb_data = bb.parse.handle(bbfile, bb_data)
File "/opt/layerindex/layers/bitbake/lib/bb/parse/__init__.py", line 117, in handle
return h['handle'](fn, data, include)
File "/opt/layerindex/layers/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 132, in handle
abs_fn = resolve_file(fn, d)
File "/opt/layerindex/layers/bitbake/lib/bb/parse/__init__.py", line 141, in resolve_file
raise IOError(errno.ENOENT, "file %s not found" % fn)
FileNotFoundError: [Errno 2] file /opt/layerindex/layers/git___git_openembedded_org_openembedded-core/meta/recipes-devtools/python-numpy/python-numpy_1.13.1.bb not found
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you're running a testing / internal instance then you really don't
want to be emailing maintainers on publish, so provide a setting you can
use to disable that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
Assume there is no master branch in hello layer:
$ update.py -l hello -b master
INFO: Skipping update of layer hello - branch master doesn't exist
This is correct since hello layer doesn't have master branch, but when --nocheckout:
$ update.py -l hello -b master --nocheckout
[snip]
INFO: Sorting layers for branch mater:
WARNING: Cannot find required collections on branch master:
WARNING: hello: LAYERDEPENDS: <snip>
This is incorrect, this patch fixed the problem, now it skips it since the
branch doesn't exists when --nocheckout.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When layer_a RECOMMENDS layer_b, try to add layer_b before layer_a, but if
layer_b is not found, still add layer_a.
And print summary error mesage:
$ update.py -b master
ERROR: Issues found on branch master:
openembedded-core: Added without LAYERRECOMMENDS
meta-secure-env: Failed to add since LAYERDEPENDS is not satisfied
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
$ git clone <url>
warning: remote HEAD refers to nonexistent ref, unable to checkout.
$ git rev-parse HEAD
HEAD
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Catch the error and avoid that.
And use "git reset --hard" to replace of "git reset --hard HEAD", HEAD is
default for git reset, so they are the same, but the later one reports error
when remote HEAD doesn't exist:
$ git reset --hard HEAD
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
[snip]
$ git reset --hard
No errors.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
$ update.py -b <new_branch>
[snip]
NOTE: Starting bitbake server...
Traceback (most recent call last):
File "update_layer.py", line 471, in main
utils.setup_core_layer_sys_path(settings, branch.name)
File "/buildarea1/lyang1/layerindex-web/layerindex/utils.py", line 376, in setup_core_layer_sys_path
core_layerdir = os.path.join(core_repodir, core_layerbranch.vcs_subdir)
AttributeError: 'NoneType' object has no attribute 'vcs_subdir'
[snip]
This is because core_layerbranch is not in database yet for completely new
branch, so it is None and we will get the error. Avoid calling
setup_core_layer_sys_path() when "update_layer.py --initial" will fix the
problem.
And also only add core layer's sys path when it is present, since core layer
may not be added yet for completely new branch.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
REGEX, REGEX_URI and GITTAGREGEX are replaced by UPSTREAM_CHECK_REGEX,
UPSTREAM_CHECK_URI and UPSTREAM_CHECK_GITTAGREGEX
Signed-off-by: Yi Zhao <yi.zhao@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add logger.setLevel to enable set log level.
Signed-off-by: Yi Zhao <yi.zhao@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to export the recipe listing for a layer to a CSV file
for importing into external tools. At the moment we include name,
version and license, but there is a parameter that lets you specify the
fields to include in the URL if desired.
Implements [YOCTO #12722].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We're about to replace this with a proper CSV export function, so we
don't need this dead code hanging around anymore.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Although it's unlikely to be an issue, technically we shouldn't be
insisting the sha1 field be unique globally, just within each
layerbranch, so adjust the constraints.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If automated emails are enabled, we need to ensure that the other email
fields are populated, so validate that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Most layers do not track maintenance on a per-recipe basis, and for
those layers we will hide some of the per-recipe maintainer features
and on the recipe detail show the layer maintainer(s) as the
maintainer(s) of the recipe.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we're in a repository containing multiple layers, we don't care about
commits that don't affect the layer we are processing, so skip those
commits rather than passing them to upgrade_history_internal.py which
will ignore them (which is significantly slower).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The processing code can't currently handle if a layerbranch is part of
more than one plan, so disallow that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I can't quite tell which Django version broke this, but in any case
based on what's in pagination.html it seems we ought to have been
using == already.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The patterns() function is deprecated in Django 1.8 and gone in 1.10, so
we should switch over to the new list format.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
RecipeUpstreamHistory was not linked to the layer it was produced from,
which meant that it wasn't easy to query for a different maintenance
plan (i.e. a different layer) and thus the maintenance plan selection
on the recipe list didn't really work. Add a link field, populate it in
a migration and then make it required.
We had added a link earlier from RecipeMaintainerHistory to LayerBranch
but it was optional; for the same reasons we now populate it and make it
required.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
These two functions aren't being used anywhere. In the interest of
having as little code directly reading the database using SQL as
possible, drop them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an option to specify which maintenance plan to operate on (largely
for debugging purposes).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Ensure the Release and Milestone names are separated by a space when
listing Milesones
* Include the maintenance plan name in the name shown for each
Release/Milestone
* Allow filtering Releases/Milestones by maintenance plan
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's a pain to have to add all the releases when adding a new
maintenance plan. Since these are likely to be the same (or similar) for
every plan, then duplicate them across from the first plan when you save
a new one.
Also add "default" milestones on the assumption that other layers
probably won't want to use the 4-milestone split per release, but there
do have to be some milestone records, so just create one milestone for
each release.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
