Update to the latest versions, with the exception of django-reversion
which I suspect will trigger some regressions; plus django-axes which
for some reason pip did not want to fetch 5.x versions of within the
container.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the base SSL configuration has been updated, and we then run
dockersetup.py -u then we want the configuration changes to be reflected
in the web server configuration, however that was not happening because
unlike how the other configuration files are handled, nginx-ssl.conf
gets copied and then we modify the copy due to the nature of the edits
made. To fix it, when in update mode, read in the old values from the
modified configuration file and then copy and modify the base
configuration using those values.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The non-HTTPS configuration was removed some time ago from
nginx-ssl.conf, and the code here that would have updated it didn't have
access to http_port so it couldn't have worked anyway, so just remove
it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Strengthen things a little where shell=True is still being used.
(For the most part input that passes through here would already be
sanitised, but let's be careful anyway).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Strengthen things a little where shell=True is still being used.
Signed-off-by: Terri Oda <terri.oda@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Avoid using shell=True where we can easily do so so that bandit doesn't
complain. Patch courtesy of Terri Oda <terri.oda@intel.com>.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I have been seeing repeated emailed errors from Django reporting
"ConnectionResetError: [Errno 104] Connection reset by peer" in the call
to get task status i.e:
File "/opt/layerindex/layerindex/views.py" in task_log_view
1572. if result.ready():
Digging around this seems to be some sort of known bug:
https://github.com/celery/celery/issues/4817https://github.com/celery/celery/issues/4980
The workaround suggested is to disable the broker heartbeat, so try
that in order to avoid the errors.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I've seen "MySQL server has gone away" errors in the logs, and this time
it isn't the timeout; one of the suggested fixes is to increase the max
packet size (and I have also seen errors in the mariadb logs indicating
that the max has been exceeded).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Limit to TLS 1.2 and the most secure ciphers
* Set some other recommended ssl_* settings
* Disable gzip
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Once we get to the package definitions we don't want to be gathering
field values (or we'll just get the last package's summary value for the
overall summary for example).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Record the configure script options when importing recipe / package
information so we can display them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When we're running commands as part of a task that use carriage returns
(\r) to update the currently displayed line, we want to see the same
output in the web representation, so if we encounter a \r in the output
we need to look back to the last newline, truncate to that and then
start appending.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We need the task log dir to be shared between the app and celery
containers, so mount a volume and point both of them into it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow passing a list to run_update_command() instead of a string which
will run the command directly instead of within the shell.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We use rpm/rpm2cpio/cpio to deal with the derivative packages, so we
need them in the application container.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Without this, celery tasks were attempting to connect with the default
user (root) and failing because the passwords are different.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
There's a little more to patch handling in spec files than "patchX:"
fields - the patch actually has to be applied in the %prep section as
well, though this can take one of several forms (%autosetup / %autopatch
which apply all patches, individual %patchX directives, or even direct
application (rare). There's also the matter of the striplevel (-p option
to the patch command). Add fields to record whether or not a patch is
applied and the striplevel.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I came across several spec files in Clear Linux that has a PNG file in
the description value. No idea how that got there, but avoid choking and
ignore the garbage description if this happens.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We don't actually need this for anything at the moment, but it would be
useful if we get to the point where we need to access imported files
within the application after the import process (e.g. to compare
patches).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Do not allow proceeding until an email address has been entered, because
we need it later on for creating the superuser account.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a source points to a local file, get the sha256sum of it and save it
into the field we just added.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow us to record a checksum for local sources (and potentially remote
ones if it is recorded in the recipe). This just adds the field, but
doesn't populate it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make it easier to import different versions of the other distro by not
requiring you to manually create the layer and branch.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If for some reason we do not find any spec files in the specified
directory, error out instead of marking all packages as deleted. (This
can happen if the wrong directory is specified.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Make patch / user fields non-editable for existing records (patch
mostly because the list is huge for a drop-down)
* User must be a superuser or the dispositioner to edit
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to mark each patch with a disposition indicating whether
the patch is interesting or not.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Editing on this page is done by clicking through to another page, so
it should be a DetailView rather than an UpdateView and shouldn't have a
form class set.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Currently, the processing of larger images takes some time, for example
a manifest generated from core-image-sato-sdk seems to take about 45s to
load into the database which is longer than gunicorn's default worker
timeout of 30s. Increase the timeout to 60s as a workaround until we
optimise the import.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's likely an image manifest tarball will be more than 1mb in size;
increase it to 16mb to handle where a large number of patches are
included in an image containing a lot of packages from different
recipes.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Error reports get emailed to the addresses specified in the ADMINS
setting, but up until now dockersetup wasn't setting this and so the
details of any internal error were not visible anywhere. If we're not
updating, prompt the user for an email address early in the process,
and use this for the ADMINS setting, the superuser account, and Let's
Encrypt (if selected).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we've messed around with the migrations during development then they
may need to be run carefully (with some fake steps), so provide an
option to update the container and then do the migrations manually
afterwards.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For the purposes of creating a dump suitable for importing into a fresh
database during initial setup, add a script that creates a dump of an
existing database with the complete structure but without any data that
involves users (or anything else we don't want).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you specified only the layer index URL and not the full API URL i.e.
https://layers.openembedded.org then we put an extra slash in the URL
which resulted in the request being redirected to the main page which
breaks the request.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Change the focus to really be on the docker-based setup:
* Integrate docker/README into the main README
* Move all documentation of the standalone setup to README.devel
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to have registration/password reset emails working properly we
need to have both the site domain and email host/port set correctly.
Email host/port is easy to set (environment variable read into the
configuration). However, to set the site info, create a simple script
that we can call to do so.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we move the superuser account creation to the end, it's practical to
Ctrl+C it in case you already have a superuser in the data that got
imported.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the database dump is older than the application, there may be
migrations to run, so we need to run them after importing and not
beforehand.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's best practice for security reasons to use shell=False and pass
command line arguments as a list; it also avoids some pain with
escaping, so let's use it everywhere we can (in fact we're only left
with one place in layerindex/tasks.py where we now pass shell=True).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Change minimum required version of Django to 1.11.11.
Signed-off-by: Amber Elliot <amber.n.elliot@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The original TableSorter is unmaintained, move to the latest version of
the active fork:
https://mottie.github.io/tablesorter/docs/
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
