Import recipe data from the REST API, including sub-recipe objects
(sources, patches, PACKAGE_CONFIGs, dependencies etc.) now that they are
available from the API. Also, with the data being more or less complete,
it's also reasonable to also set the vcs_* fields so that we can tell
how up-to-date it is.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Don't assume we need to create these objects - check the updated field
value against what the API reports and update the existing object if the
latter is newer.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For LayerDependency, LayerMaintainer and LayerNote objects, properly
sync the objects in the database with the objects fetched from the API.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
These are a little less useful than the other items but if we want to be
able to create a clone of a layer index then we need them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We're tracking these items for each recipe, so they should be
accessible via the API as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The try-specfile subcommand is for testing, so it's entirely possible
the spec file we are testing with will be in the current directory;
os.path.dirname() returns '' in this case, which os.path.relpath() then
chokes on. Avoid this in the easiest way possible by just prepending ./
to the path in this case.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's not common, but if someone saves a spec file with a non-UTF8
encoding then the surrogate characters were preventing the data from
being saved to the database (at least with MariaDB). Instead of assuming
UTF8, try a few common encodings on the first 100 lines.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a SECURITY_QUESTIONS_REQUIRED setting that defaults to True, but if
set to False then a user who has not set security questions will still
be allowed to reset their password. This is convenient for the OE Layer
index because there are a number of existing accounts, none of which
will have security questions set.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you clicked on logout in the admin site then clicked on the "Login"
button and logged in, you were redirected to the logout page because we
weren't detecting that the redirection should not be applied.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Tweak message when one or more (but not all) security question answers
have been changed so that it's clear that all are required
* Ensure success message gets shown upon saving
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Require all three answers to be correct (previously only two had to
be)
* Give the user three attempts instead of just two
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We use django-axes to lock out IP addresses after a set number of
attempts at logging in, and separately we use django-reversion to
record change history. As part of the history tracking, the default
behaviour of django-reversion is to wrap all POST requests in
"with transaction.atomic()", with the result that if an exception is
raised any changes get rolled back; unfortunately when authentication
fails for the final time, axes updates the database and then raises
PermissionDenied - with the result that the database changes are rolled
back, and the user's IP is not locked out, in fact it can never be
locked out. To work around this, disable the atomic mode on
ReversionMiddleware using a subclass. (I don't like having to do this,
but this is the quickest solution for now.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Simplify things a bit. We lose the digg-style pagination but the new
behaviour is good enough and improves maintainability.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When running a task (e.g. importing other distribution data), the web
frontend polls to get task output every second. In the view handling
this request we check to see if the underlying Celery task is finished
by getting the AsyncResult and then calling result.ready().
Unfortunately that latter call seems to be failing some of the time in
the development Docker setup that I am using, throwing a
ConnectionResetError. Because the polling is regular it doesn't really
matter if this fails as there'll be another chance on the next poll, so
just allow the call to fail silently. (It pains me to put in workarounds
like this but at the moment I can't determine the real cause of the
issue, and this stops the stream of ultimately useless error report
emails that I'm getting as an admin of the development instance).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Calling communicate() blocks the process; but since we're writing the
output directly to a file and not sending any input we don't actually
need to call communicate(), just poll() (so that we can check
the returncode attribute). Subjectively this does appear to improve
performance although it has not fixed the ConnectionResetError issues.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If an update task's output contains a single quote for example (') then
this will be escaped to an HTML entity before it gets sent as a
response. However, that means that the length of the data in the response
will be greater than the length of the original data, resulting in
characters getting missed out when we read the next chunk - so we
can't use the escaped length to set the next position to read from. The
easiest thing to do is have the Django view send us the actual position
we're at and then we don't have to try to calculate it on the JS side.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
There's no particular reason these fields should be read-only (unlike
the fields picked up from Recipe, they aren't derived from the recipe
itself.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Strengthen things a little where shell=True is still being used.
(For the most part input that passes through here would already be
sanitised, but let's be careful anyway).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I have been seeing repeated emailed errors from Django reporting
"ConnectionResetError: [Errno 104] Connection reset by peer" in the call
to get task status i.e:
File "/opt/layerindex/layerindex/views.py" in task_log_view
1572. if result.ready():
Digging around this seems to be some sort of known bug:
https://github.com/celery/celery/issues/4817https://github.com/celery/celery/issues/4980
The workaround suggested is to disable the broker heartbeat, so try
that in order to avoid the errors.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Once we get to the package definitions we don't want to be gathering
field values (or we'll just get the last package's summary value for the
overall summary for example).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Record the configure script options when importing recipe / package
information so we can display them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow passing a list to run_update_command() instead of a string which
will run the command directly instead of within the shell.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
There's a little more to patch handling in spec files than "patchX:"
fields - the patch actually has to be applied in the %prep section as
well, though this can take one of several forms (%autosetup / %autopatch
which apply all patches, individual %patchX directives, or even direct
application (rare). There's also the matter of the striplevel (-p option
to the patch command). Add fields to record whether or not a patch is
applied and the striplevel.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I came across several spec files in Clear Linux that has a PNG file in
the description value. No idea how that got there, but avoid choking and
ignore the garbage description if this happens.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We don't actually need this for anything at the moment, but it would be
useful if we get to the point where we need to access imported files
within the application after the import process (e.g. to compare
patches).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a source points to a local file, get the sha256sum of it and save it
into the field we just added.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow us to record a checksum for local sources (and potentially remote
ones if it is recorded in the recipe). This just adds the field, but
doesn't populate it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make it easier to import different versions of the other distro by not
requiring you to manually create the layer and branch.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If for some reason we do not find any spec files in the specified
directory, error out instead of marking all packages as deleted. (This
can happen if the wrong directory is specified.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Make patch / user fields non-editable for existing records (patch
mostly because the list is huge for a drop-down)
* User must be a superuser or the dispositioner to edit
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to mark each patch with a disposition indicating whether
the patch is interesting or not.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Editing on this page is done by clicking through to another page, so
it should be a DetailView rather than an UpdateView and shouldn't have a
form class set.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For the purposes of creating a dump suitable for importing into a fresh
database during initial setup, add a script that creates a dump of an
existing database with the complete structure but without any data that
involves users (or anything else we don't want).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you specified only the layer index URL and not the full API URL i.e.
https://layers.openembedded.org then we put an extra slash in the URL
which resulted in the request being redirected to the main page which
breaks the request.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to have registration/password reset emails working properly we
need to have both the site domain and email host/port set correctly.
Email host/port is easy to set (environment variable read into the
configuration). However, to set the site info, create a simple script
that we can call to do so.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's best practice for security reasons to use shell=False and pass
command line arguments as a list; it also avoids some pain with
escaping, so let's use it everywhere we can (in fact we're only left
with one place in layerindex/tasks.py where we now pass shell=True).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Indentation, spacing, and wrapping of a long import line.
Signed-off-by: Amber Elliot <amber.n.elliot@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add user security questions upon registration as extra authentication
for password reset. Three unique security questions must be chosen and
answered. Answers are then stored in the database with the same hashing
algorithm as the users's password.
On password reset, users get two chances to get two out of three
security questions answered correctly. After a second failure their
account is locked and email is sent to the admin. The same template is
shown for the axes lockout. Super user cannot reset their password until
they set security questions.
Users can update their security questions or add them if they weren't
originally set (in the case of super user) in Edit Profile.
Signed-off-by: Amber Elliot <amber.n.elliot@intel.com>
This involves changing how registration templates are referenced
and how the activation email is sent on user's email address change.
Signed-off-by: Amber Elliot <amber.n.elliot@intel.com>
This allows users to see the site ID in the admin page.
Signed-off-by: Amber Elliot <amber.n.elliot@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
