Simplify things a bit. We lose the digg-style pagination but the new
behaviour is good enough and improves maintainability.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When running a task (e.g. importing other distribution data), the web
frontend polls to get task output every second. In the view handling
this request we check to see if the underlying Celery task is finished
by getting the AsyncResult and then calling result.ready().
Unfortunately that latter call seems to be failing some of the time in
the development Docker setup that I am using, throwing a
ConnectionResetError. Because the polling is regular it doesn't really
matter if this fails as there'll be another chance on the next poll, so
just allow the call to fail silently. (It pains me to put in workarounds
like this but at the moment I can't determine the real cause of the
issue, and this stops the stream of ultimately useless error report
emails that I'm getting as an admin of the development instance).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Calling communicate() blocks the process; but since we're writing the
output directly to a file and not sending any input we don't actually
need to call communicate(), just poll() (so that we can check
the returncode attribute). Subjectively this does appear to improve
performance although it has not fixed the ConnectionResetError issues.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If an update task's output contains a single quote for example (') then
this will be escaped to an HTML entity before it gets sent as a
response. However, that means that the length of the data in the response
will be greater than the length of the original data, resulting in
characters getting missed out when we read the next chunk - so we
can't use the escaped length to set the next position to read from. The
easiest thing to do is have the Django view send us the actual position
we're at and then we don't have to try to calculate it on the JS side.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
There's no particular reason these fields should be read-only (unlike
the fields picked up from Recipe, they aren't derived from the recipe
itself.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Strengthen things a little where shell=True is still being used.
(For the most part input that passes through here would already be
sanitised, but let's be careful anyway).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I have been seeing repeated emailed errors from Django reporting
"ConnectionResetError: [Errno 104] Connection reset by peer" in the call
to get task status i.e:
File "/opt/layerindex/layerindex/views.py" in task_log_view
1572. if result.ready():
Digging around this seems to be some sort of known bug:
https://github.com/celery/celery/issues/4817https://github.com/celery/celery/issues/4980
The workaround suggested is to disable the broker heartbeat, so try
that in order to avoid the errors.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Once we get to the package definitions we don't want to be gathering
field values (or we'll just get the last package's summary value for the
overall summary for example).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Record the configure script options when importing recipe / package
information so we can display them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow passing a list to run_update_command() instead of a string which
will run the command directly instead of within the shell.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
There's a little more to patch handling in spec files than "patchX:"
fields - the patch actually has to be applied in the %prep section as
well, though this can take one of several forms (%autosetup / %autopatch
which apply all patches, individual %patchX directives, or even direct
application (rare). There's also the matter of the striplevel (-p option
to the patch command). Add fields to record whether or not a patch is
applied and the striplevel.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I came across several spec files in Clear Linux that has a PNG file in
the description value. No idea how that got there, but avoid choking and
ignore the garbage description if this happens.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We don't actually need this for anything at the moment, but it would be
useful if we get to the point where we need to access imported files
within the application after the import process (e.g. to compare
patches).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a source points to a local file, get the sha256sum of it and save it
into the field we just added.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow us to record a checksum for local sources (and potentially remote
ones if it is recorded in the recipe). This just adds the field, but
doesn't populate it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make it easier to import different versions of the other distro by not
requiring you to manually create the layer and branch.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If for some reason we do not find any spec files in the specified
directory, error out instead of marking all packages as deleted. (This
can happen if the wrong directory is specified.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Make patch / user fields non-editable for existing records (patch
mostly because the list is huge for a drop-down)
* User must be a superuser or the dispositioner to edit
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to mark each patch with a disposition indicating whether
the patch is interesting or not.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Editing on this page is done by clicking through to another page, so
it should be a DetailView rather than an UpdateView and shouldn't have a
form class set.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For the purposes of creating a dump suitable for importing into a fresh
database during initial setup, add a script that creates a dump of an
existing database with the complete structure but without any data that
involves users (or anything else we don't want).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you specified only the layer index URL and not the full API URL i.e.
https://layers.openembedded.org then we put an extra slash in the URL
which resulted in the request being redirected to the main page which
breaks the request.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to have registration/password reset emails working properly we
need to have both the site domain and email host/port set correctly.
Email host/port is easy to set (environment variable read into the
configuration). However, to set the site info, create a simple script
that we can call to do so.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's best practice for security reasons to use shell=False and pass
command line arguments as a list; it also avoids some pain with
escaping, so let's use it everywhere we can (in fact we're only left
with one place in layerindex/tasks.py where we now pass shell=True).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Indentation, spacing, and wrapping of a long import line.
Signed-off-by: Amber Elliot <amber.n.elliot@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add user security questions upon registration as extra authentication
for password reset. Three unique security questions must be chosen and
answered. Answers are then stored in the database with the same hashing
algorithm as the users's password.
On password reset, users get two chances to get two out of three
security questions answered correctly. After a second failure their
account is locked and email is sent to the admin. The same template is
shown for the axes lockout. Super user cannot reset their password until
they set security questions.
Users can update their security questions or add them if they weren't
originally set (in the case of super user) in Edit Profile.
Signed-off-by: Amber Elliot <amber.n.elliot@intel.com>
This involves changing how registration templates are referenced
and how the activation email is sent on user's email address change.
Signed-off-by: Amber Elliot <amber.n.elliot@intel.com>
This allows users to see the site ID in the admin page.
Signed-off-by: Amber Elliot <amber.n.elliot@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Patches often need to be applied in a specific order. For OE recipes we
were always storing the Patch objects correct order as they are
refreshed every time the recipe itself is refreshed, however for other
distro comparisons, import_otherdistro.py attempts to preserve existing
records, adds new ones and then deletes whatever is left over, which may
result in the order getting messed up over time. To avoid this issue,
record the order next to the patch and set the model meta-info to use
this to sort Patch queries by default.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Best practices state that you should use a mechanism that ensures files
get closed in case of any error, so let's do that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Errors deleting bitbake.sock and bitbake.lock have been observed when
shutting down tinfoil at the end of some of these scripts. Move the code
used in the main layer index update script to a function in utils.py and
use it everywhere in order to avoid the issue.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add links to the same recipe in other branches in the recipe detail page
(and RRS recipe detail page) so that you can see which versions are
available in other branches and drill down to the detail if you want to.
Implements [YOCTO #13019].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If someone adds you as a maintainer of a layer as a matter of courtesy
it would be nice if you get an email.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Some controls on the Submit / Edit Layer page were getting the wrong
width and any "feedback" elements were being pulled to the right because
of the size of their parent divs. Instead of pulling in the size of the
form control, act on the div to fix the problem. Additionally the
help text should now be allowed to expand to the full width of the
parent div since that is now sensible.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
