* Tweak message when one or more (but not all) security question answers
have been changed so that it's clear that all are required
* Ensure success message gets shown upon saving
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When running a task (e.g. importing other distribution data), the web
frontend polls to get task output every second. In the view handling
this request we check to see if the underlying Celery task is finished
by getting the AsyncResult and then calling result.ready().
Unfortunately that latter call seems to be failing some of the time in
the development Docker setup that I am using, throwing a
ConnectionResetError. Because the polling is regular it doesn't really
matter if this fails as there'll be another chance on the next poll, so
just allow the call to fail silently. (It pains me to put in workarounds
like this but at the moment I can't determine the real cause of the
issue, and this stops the stream of ultimately useless error report
emails that I'm getting as an admin of the development instance).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If an update task's output contains a single quote for example (') then
this will be escaped to an HTML entity before it gets sent as a
response. However, that means that the length of the data in the response
will be greater than the length of the original data, resulting in
characters getting missed out when we read the next chunk - so we
can't use the escaped length to set the next position to read from. The
easiest thing to do is have the Django view send us the actual position
we're at and then we don't have to try to calculate it on the JS side.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to mark each patch with a disposition indicating whether
the patch is interesting or not.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Editing on this page is done by clicking through to another page, so
it should be a DetailView rather than an UpdateView and shouldn't have a
form class set.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's best practice for security reasons to use shell=False and pass
command line arguments as a list; it also avoids some pain with
escaping, so let's use it everywhere we can (in fact we're only left
with one place in layerindex/tasks.py where we now pass shell=True).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add user security questions upon registration as extra authentication
for password reset. Three unique security questions must be chosen and
answered. Answers are then stored in the database with the same hashing
algorithm as the users's password.
On password reset, users get two chances to get two out of three
security questions answered correctly. After a second failure their
account is locked and email is sent to the admin. The same template is
shown for the axes lockout. Super user cannot reset their password until
they set security questions.
Users can update their security questions or add them if they weren't
originally set (in the case of super user) in Edit Profile.
Signed-off-by: Amber Elliot <amber.n.elliot@intel.com>
This involves changing how registration templates are referenced
and how the activation email is sent on user's email address change.
Signed-off-by: Amber Elliot <amber.n.elliot@intel.com>
Best practices state that you should use a mechanism that ensures files
get closed in case of any error, so let's do that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add links to the same recipe in other branches in the recipe detail page
(and RRS recipe detail page) so that you can see which versions are
available in other branches and drill down to the detail if you want to.
Implements [YOCTO #13019].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If someone adds you as a maintainer of a layer as a matter of courtesy
it would be nice if you get an email.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Use a parameter to pass the current page to the Edit Profile URL so that
saving or cancelling returns you to that same page.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you're searching for items that are of status "unknown/not available"
in the other distro packages, you're usually doing so to find things
that haven't been dispositioned. Originally I had included them here
because the "available" status correctly excludes items with the
"distro-specific" status and "unknown/not available" seemed like it
should be the inverse, but usage seems to require that it not be.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a user goes to Edit Profile and changes their email address,
deactivate their account temporarily and make them go through the
registration process to confirm that the new email address is in fact
valid and theirs.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Users may want to change their usernames for a number of different
reasons, but at the moment we require them to contact an admin to do
that. Provided we validate the new username correctly and add a CAPTCHA
to make automated enumeration difficult, we can add username to the Edit
Profile form and then users can do that any time they wish.
While we're doing this, show a message when the profile is successfully
updated.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This was copy-pasted from the DistroSearchView and I clearly forgot to
replace the reference here.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
nvd3 and its python/django wrappers appear to be no longer actively
maintained, and at least the wrappers were a bit clunky to use. Looking
around for a suitable replacement, Chart.js seems capable, has no
additional dependencies and is fairly simple to use. As a bonus we get
to drop a few Python dependencies from our list.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's not too common but there are instances where people have copied
.inc files into their own layer and modified them, and if you are using
such a layer that could result in unexpected behaviour. In order to get
a handle on when this is being done, collect data about all .inc files
and show duplicates in the Duplicates screen.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Split out the code used in the recipe search views to its own function
and use that same function in three different places rather than having
a copy of largely the same code. Also take the opportunity to add some
comments.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
On layers.openembedded.org we're seeing requests from some search engine
crawlers requesting the CSV export URL with an invalid branch for the
layer. I couldn't see the referer anywhere in the logs but I suspect it
has to do with some recent cleanup work I did in the database where I
deleted some invalid LayerBranch records - they were probably following
links in a cached version of the webpage. In any event we want to return
404 in this situation rather than an internal server error.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to show bbappends on the recipe detail page we are doing a
regex query to find any whose names match up with the recipe. In the
layer index instance at layers.openembedded.org viewing the recipe
detail page for any recipe whose name contains ++ (e.g. libsigc++-2.0 in
meta-oe) results in an invalid regex and causes a database error. Escape
any + signs in the name used within the regex in order to fix this.
(I wasn't actually able to reproduce this on my own setup despite also
using MariaDB, but I did find that the unescaped query was not correctly
matching records so it needed to be fixed anyway.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When you make changes to the infrastructure it can be useful to test
that email sending is working, since for that to work that involves the
code, Celery, RabbitMQ and SMTP being functional. However, up until now
to run a test you needed to submit a fake layer which is a bit annoying.
Add an explicit "Test email" option to the Tools drop-down for staff
users to allow them to send an email to themselves.
Note: the page will come back when the Celery job has been created, it
does not check and report on the job status - you need to look on the
server side to see that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For situations where the user launches a distro comparison update
process and then shortly afterwards realises it is operating with the
wrong configuration (or is otherwise broken) and is going to take a long
time to finish, add a button to the task page to stop the task. This was
tricky to get working, since the default behaviour of Celery's revoke()
would either terminate both the Celery task process along with the update
process (leaving us with no log saved to the database) or worse not even
kill the update process, depending on the signal sent. To avoid this,
send SIGUSR2, trap it in the task process and kill the child process,
returning gracefully. To make that possible I had to rewrite runcmd() to
use subprocess.Popen() instead of subprocess.check_call() as otherwise
we can't get the child's PID.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a distro comparison update task fails (returning a non-zero value to
indicate as such) we were not able to see this easily from the frontend.
Show success/failure in the form of a label on the task page and general
update list/detail, and if the task fails while we're watching then make
the progress bar go red as well. Also make a distinction between the
process failing (retcode > 0) and being terminated (retcode < 0, e.g.
process was killed).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Provide a mechanism for distro comparison update tasks to display
progress. In practice this means the update command needs to write the
progress percentage to a file and then the log view (which is polled by
the frontend) reads this file. Originally I was going to use a FIFO for
this but that turned out to be a but unreliable; I also tried to use
Celery's state mechanism to pass it back but I simply could not get it
to work. The file-based mechanism is good enough though.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were refreshing the page constantly in order to show output while
a task was running, which basically worked but is horrible. Instead,
write the task output to a file and then use AJAX calls to request
whatever output has been written to the file since the last call
and call this roughly every second. Put the output in a scrollable <pre>
element instead of making it the length of the page, and auto-scroll
to the end (unless the user grabs the scrollbar and pulls it upwards -
it may not be immediately obvious that you can do this if there is a lot
of output since you have to pull it up when the scrolling animation is
not running, but it is possible).
An alternative would be to have used some kind of long-lived HTTP
session or a websocket, but those come with their own set of problems
so I elected to use this much simpler method.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a user doesn't have publish rights and the type of the layer isn't
already "Base" then disallow selecting the Base layer type. Some
submitters are selecting this type for their own layers, but it's pretty
much reserved for openembedded-core and meta-oe (so that they appear at
the top of the layer list).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When showing the error/warning counts for update records we need to
include any errors/warnings that are shown only in the main update log,
so we need to adjust how these are collected. Use a function rather than
pure aggregation to give a bit more control, and a {% with ... %} block
in the template to avoid the functions being called more than necessary.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make layerupdate collection slightly more reliable and make it easier
to see when updates have actually been captured:
* Split layerbranch into separate layer and branch fields, since there
may not be a layerbranch in existence but we might want to log an
error relating to the branch and layer.
* Show all layerupdates on the update detail page, not just those with
log messages
* Record before and after revisions and show these in the update detail
and layerupdate detail (with links)
* Record return code of update_layer process
* Highlight layer updates with a non-zero return code, errors or
warnings in the output on the update detail page
* Show duration on the layerupdate detail page
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
On the recipe detail page we provide a link to the actual recipe file
for reference purposes. However, for recipes that include a common .inc
file, many of the definitions of variable values will be inside the
.inc, therefore if you just look at the recipe you won't see the full
picture. Of course you can just go up to the parent directory in the
repository web interface, but for convenience's sake add links to any
files that are included/required by the recipe that are adjacent to
the recipe itself. (We already have the data in the form of the
RecipeFileDependency records that are intended to ensure we know when
the recipe needs to be updated if one of the files it includes changes).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Specifying the covering recipe in the comparison recipe detail page was
always a bit awkward - you could only type the name, if you wanted to
actually find a recipe or look up the currently selected one's details
then you had to open another browser tab/window. To fix this, replace
the form on the comparison recipe detail page with a side-by-side
display of the covering recipe's information, along with a button that
lets you search and then select the covering recipe and at the same time
enter comments or set any of the other cover fields.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Comparison updates might involve some custom fetch process, so provide a
mechanism to register these via settings.py on a per-branch basis. If
an update command is defined for a branch and the logged-in user has the
new "update_comparison_branch" permission, an "Update" button will show
up on the recipes page for the comparison branch for authenticated
users that will trigger the command in the background (as a celery job)
and then show a page that displays the status. The status isn't shown in
real-time since that requires quite a lot of plumbing, but the page at
least auto-refreshes.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make it clearer what this field is for (it should be set for every
comparison layer, so that they don't show up in places they shouldn't).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When doing the reversed query it's often desirable to exclude recipes
by inherited class, for example those that inherit the packagegroup,
image and meta classes as they don't actually build anything and thus
aren't going to match up with anything in the other distribution.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Provide a lazy-loaded popup for selecting layers to include in the query
instead of having it as a simple drop-down, so you can select more than
one layer.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Enable "reversing" the comparison, so you can see which recipes on the
OE side match up (or don't) with the other distro. The filtering for
this is a bit awkward, since we don't have an actual foreign key for the
link, hence the hairiness of the code.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a flag that can be set and searched for to indicate that we need to
take care of importing a package or a patch applied by a package.
Ideally the comments would elaborate on what's needed (if it's not
obvious from the cover status).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Packages with the "Not available" status were inexplicably excluded from
the graph, include them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow searching on:
* Any "available" status (i.e. other than "Unknown" or "Not available")
* Whether the package has patches or not
* What the covering layer is (assuming there is one).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's a bit crude, but accept '' or "" as meaning search for entries
with an empty category.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script to import "recipe" information from other distro metadata,
based on the import_classic.py script. At the moment, this assumes a
directory where each subdirectory is a package directory containing a
spec file; this would be suitable for distributions such as Fedora
assuming you have all of the package repos checked out locally. Since
you can add additional information to these records (the cover fields
pointing to matching recipes), existing records are updated rather than
deleting everything and re-importing, and we only mark records as
deleted rather than actually deleting them (in case you accidentally
point the script at an empty directory or similar).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Turn the existing OE-Classic support into something a bit more
generic so we can import data from other distributions and compare it to
what we have in layers.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you're running a testing / internal instance then you really don't
want to be emailing maintainers on publish, so provide a setting you can
use to disable that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to export the recipe listing for a layer to a CSV file
for importing into external tools. At the moment we include name,
version and license, but there is a parameter that lets you specify the
fields to include in the URL if desired.
Implements [YOCTO #12722].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We're about to replace this with a proper CSV export function, so we
don't need this dead code hanging around anymore.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the RRS is enabled, then add a way to get from the layer detail page
to any maintenance plans in which the layer is included.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
