Make it possible to re-collect all the history for a given path.
(Typically this would only be used for debugging, as it saves time if
you are trying to correct an issue with upgrade data collection for a
single recipe.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we ever want to analyse the upgrade chain later on then we need to
avoid overwriting the paths when we identify a moved recipe - instead,
store a "move" upgrade record (not shown in the UI) that we can later
pick up when we are going through and deleting.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Silently catching all exceptions like this is evil. Use .first() and
check the return instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Version downgrades (or what appear to be downgrades) do occasionally
happen, and if they did then the RRS was previously simply ignoring
them, resulting in the latest version being reported incorrectly.
Allow downgrades to be recorded as an upgrade with a new 'Downgrade'
type option set, and display a label on such records in the UI.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
There are a range of commits in OE-Core which cause parsing problems;
map them to the one that fixes it in order to avoid the problem. (This
will only be done if we're dealing with OE-Core as a dependency, rather
than the actual layer we're parsing).
(The second set are some commits during the python 3 conversion time.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Now that we're using RecipeSymbols we have the complete list of recipes
that ever existed in a layer. We only want to see the ones that are
valid for the selected milestone, so when a recipe gets deleted (or
renamed or moved outside of the layer subdirectory, if any) we need to
record that - do so using a RecipeUpgrade record with a new field
upgrade_type set to 'R'. Additionally we need to store the file path so
that deletion events (where we don't parse the contents of the recipe,
thus we don't have PN) are easy to match up with RecipeUpgrade records;
naturally we need to keep the paths "up-to-date" when we notice recipe
files being moved around.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Recipes come and go over time, so when a recipe gets deleted the history
for it goes away, which means that if you look back in time you do not
see an accurate picture - you only see the subset of recipes that are
currently present. Introduce an indirection between recipes and history
that allows for old recipes to persist (mostly in name only).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Update Pillow version to incorporate a fix for a denial-of-service
vulnerability (which should not affect this application however, as it
does not use Pillow to process external images):
https://nvd.nist.gov/vuln/detail/CVE-2019-16865
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Typo, there has to be a ? in front of the id or otherwise we don't get
linked to the right commit. This would have affected recently added layers
with a cgit web frontend.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to compare available recipes and their versions between
two branches for a selection of layers (default is just OE-Core). This
was mainly intended to help us with the Yocto Project release notes
preparation (hence the "Plain text" button at the bottom of the page)
but is also useful in its own right.
Note: for readability, SRCREVs are only shown when PV has not changed.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For the purposes of the branch comparison function I'm about to add it
would be useful to track the value of SRCREV, so we can see if it has
changed even if PV hasn't.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Just because git.yoctoproject.org is in the URL, doesn't mean we can or
should force the vcs_web_url to be a specific value. If it starts with
git://git.yoctoproject.org then we can do this. git.openembedded.org
already did this.
This also changes github, gitlab and bitbucket references.
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a new BITBAKE_PATH to the settings file to specify the path within the
BITBAKE_REPO_URL where bitbake lives. This is useful when using a combined
repository, such as poky, that contains bitbake, openembedded-core and other
layers.
This change also changes the default path, in the fetch directory, for the
bitbake checkout. It no longer uses the path 'bitbake', but instead uses the
same URL processing as the layer fetching.
There is a side effect that, when using a shared fetch, the branch of the
layer will be used instead of the specified bitbake branch. Generally this
is a reasonable compromise, since in a combined repository bitbake and
openembedded-core component should already match.
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Without this change the system will fail parsing various URL components
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Where we use glyphicons to mark items in a list, ensure there is a space
between the item and the icon to make things look a bit neater.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Bump a few versions where we can
* Drop anyjson - this used to be a dependency of kombu but not anymore,
and nothing else needs it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This change is to record PE (epoch) and PR (release) values for recipes.
These values will be reflected in the REST API but will not be exposed
in the UI. The aim of this change is to enrich metadata availability to
consumers of the feed.
Signed-off-by: Pranay Mankad <pranaymankad@gmail.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
On some systems it seems the order of fields in a values() query aren't
guaranteed, so in order to avoid the contents dancing around too much,
explicitly write out the fields in the order we expect them to be in.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Drop some explicit outdated version numbers
* You do actually have to create the database with MariaDB etc. so add
instructions as to how for convenience
* Add pointer to virtualenv in python requirements item
* Add libssl-dev package to install command
* Use python3-venv instead of virtualenv package in install command
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Unfortunately the "updated" date gets changed every time the repo is
fetched (i.e. vcs_last_fetch changes), but that doesn't mean the
layerbranch or anything under it has actually changed. Use vcs_last_rev
instead although unfortunately that won't catch manual edits to the
layerbranch alone.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you enable the --letsencrypt option when you run dockersetup.py, the
script will modify the volume mount for the certificates to point to
/etc/letsencrypt instead of /opt/cert. If you then run dockersetup.py
again (with -r/--reinstall) without --letsencrypt, we want the path to
be set back to /opt/cert, so ensure that it does. Additionally, the code
wasn't actually setting the path for the layerscertbot service since
editing that section was done separately. (Admittedly, the letsencrypt
functionality has not been well-tested.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Provide a way to export and import other distro recipe coverage in
update_classic_status.py.
(Based on the Clear Linux Dissector commit
ff38e9582093453e13b90e06af125374ca4b0a16).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I'm about to add a few command line options to this script, so switch
over to argparse before that which is more modern and a bit more
flexible.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Don't confuse proxy options being defaulted through from the environment
and the user explicitly specifying them. Also look at no_proxy option.
Fixes https://github.com/intel/clear-linux-dissector-web/issues/13
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to uninstall the application for the user's convenience.
(Note that this does not undo the changes to the configuration, it only
removes the Docker containers and volumes.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* We need a SOCKS proxy to support fetching from git:// or ssh:// URLs
within the container, so add an option to specify it
* It's possible for the http and https proxy settings to be the same, so
set one from the other if only one of them is set.
* If we want to be able to fetch from internal servers inside the proxy
then we also need a "no-proxy" list, so add support for that.
* It's not unlikely that machines within networks requiring use of a
proxy for external network access will have all of the proxy settings
set in the environment, so we can try to pick up the defaults from
there.
* Ensure that we can switch from proxy to no proxy (when reinstalling)
which means we always need to edit the config files and ensure the
proxy options get commented out if we don't want them set.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's easy to get the proxy settings wrong and not realise until you've
got quite a long way into the process of setting things up. Thus, add a
check where we actually try to fetch various things within the container
environment and fail reasonably early if things aren't working.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I forgot the lesson of f5922091b4 which
was that djangorestframework has a very silly default of registering
the model name as the key for finding the ViewSet, with the result that
the recently created RecipesExtendedViewSet was also being used for the
recipes URL which we expect to point to RecipeViewSet; this broke the
bitbake-layers layerindex-* commands.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The new viewset has a different URL and uses pagination, so we need to
take that into account.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Adding these extra child items to the standard "recipes" viewset (which
we did recently in 684a06a383) means that
some current usage is impractical due to the size of the returned list
of items. Instead, create a recipesExtended viewset, move the new child
items to that and add pagination to avoid result size issues.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This is now be practical to use for updating as well as the initial
import. Additionally the user doesn't need to be using the debug
option by default so drop it from the example command.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a LayerBranch is on a branch that is in the remote layer index (and
that branch is in any branch list specified with -b/--branch) and the
layer for the LayerBranch is not found in the remote layerindex then it
should be deleted, otherwise we could end up with stale data.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an option to import only certain layers - mostly for debugging
(can make testing a lot quicker).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We only work with branches that exist in both the local database and the
remote index, but allow the user to restrict the list further if
desired.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
