If we're collecting this useful info we should really display it. Try to
make the link clickable if possible.
At the same time, add the layerbranch to the list display for Source
objects in the admin site.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script to fetch package sources from Fedora's Pagure
infrastructure site at https://src.fedoraproject.org/, based upon
github-fetch.py (I broke out the internal fetch function with the intent
to share it with that other script, however I didn't in the end as I
wanted the scripts to remain independent).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script to fetch/update all repositories within a GitHub
organisation. Note that to run this you will need an access token, which
you can create by going to GitHub and then
Settings -> Developer Settings -> Personal access tokens.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Comparison updates might involve some custom fetch process, so provide a
mechanism to register these via settings.py on a per-branch basis. If
an update command is defined for a branch and the logged-in user has the
new "update_comparison_branch" permission, an "Update" button will show
up on the recipes page for the comparison branch for authenticated
users that will trigger the command in the background (as a celery job)
and then show a page that displays the status. The status isn't shown in
real-time since that requires quite a lot of plumbing, but the page at
least auto-refreshes.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make it clearer what this field is for (it should be set for every
comparison layer, so that they don't show up in places they shouldn't).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If other distro comparison package(s) exist for a recipe being shown in
the RRS recipe detail page, link to the page for each package as well as
any extra URLs.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a structure that lets you define a template URL per layer to be
shown per comparison recipe. For example, you could use this to define a
URL template to link to the upstream summary page for the package (e.g.
Fedora's page for the acl package is at
https://apps.fedoraproject.org/packages/acl, so you would use
https://apps.fedoraproject.org/packages/%pn% as the template and then
this would be shown for every package).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to export the comparison search results to CSV format in
order to allow importing the data into external tools.
Note: I implemented this in a different way than the earlier recipe CSV
export, i.e. it uses a template to render the CSV instead of a
function-based view with the Python csv module - the reason for this is
we can reuse the same view as we use for producing the search, with all
of the flexibility that gives us.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When doing the reversed query it's often desirable to exclude recipes
by inherited class, for example those that inherit the packagegroup,
image and meta classes as they don't actually build anything and thus
aren't going to match up with anything in the other distribution.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Provide a lazy-loaded popup for selecting layers to include in the query
instead of having it as a simple drop-down, so you can select more than
one layer.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Enable "reversing" the comparison, so you can see which recipes on the
OE side match up (or don't) with the other distro. The filtering for
this is a bit awkward, since we don't have an actual foreign key for the
link, hence the hairiness of the code.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add support for importing package information extracted from a running
Debian system (i.e. from the output of apt-cache show "*").
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a flag that can be set and searched for to indicate that we need to
take care of importing a package or a patch applied by a package.
Ideally the comments would elaborate on what's needed (if it's not
obvious from the cover status).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Packages with the "Not available" status were inexplicably excluded from
the graph, include them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow searching on:
* Any "available" status (i.e. other than "Unknown" or "Not available")
* Whether the package has patches or not
* What the covering layer is (assuming there is one).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's a bit crude, but accept '' or "" as meaning search for entries
with an empty category.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an option to skip certain recipes by name (a crude workaround for
when we would otherwise get an erroneous match).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the source URL is fetching from pypi then we know it's python;
similarly cpan means perl.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we always set cover_pn if there is a match rather than only setting
it if it's different from pn then it simplifies the code elsewhere.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Only apply OE-Classic-specific logic after checking that it's actually
OE-Classic
* Ignore recipes marked deleted
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script to import "recipe" information from other distro metadata,
based on the import_classic.py script. At the moment, this assumes a
directory where each subdirectory is a package directory containing a
spec file; this would be suitable for distributions such as Fedora
assuming you have all of the package repos checked out locally. Since
you can add additional information to these records (the cover fields
pointing to matching recipes), existing records are updated rather than
deleting everything and re-importing, and we only mark records as
deleted rather than actually deleting them (in case you accidentally
point the script at an empty directory or similar).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Turn the existing OE-Classic support into something a bit more
generic so we can import data from other distributions and compare it to
what we have in layers.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were sorting the layers in the wrong order when trying to choose
which layer to pick a matching recipe from - it needs to be descending
priority order, not ascending.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script that uses the REST API to gather information from another
running layer index instance (e.g. layers.openembedded.org) and import
that into the local index. Only information for branches that are
already set up is imported, and only manually entered information -
no recipes, machines, etc.
Partially implements [YOCTO #9760].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the cover status for a comparison recipe is Unknown, Not available,
or Distro-specific, then we disable the recipe and layerbranch fields in
the form (using Javascript), however the form could still be saved and
any existing values would persist. Clear out the fields upon saving if
they are set under these circumstances.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Check if bitbake directory can be found before trying to do anything
with the repo
* Split try..finally into two so we don't try to shut down a nonexistent
tinfoil when we failed to gain a lock.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a file is moved (renamed) to a path outside of the layer, e.g.
another layer within a multi-layer repository, then we need to treat it
as a delete. Up until now we were updating the path and continuing, and
then the recipe was also picked up as an add in the other layer, leading
to duplicate recipe entries. I'd noticed these duplicates before but up
until now I'd thought that they were due to another bug we already
fixed, apparently not.
In order to remove these erroneous duplicate entries in existing
databases I have also added a layerindex/tools/fixup_duplicates.py
script. I've also made the -r/--reload option delete them as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to show a notice at the top of every page; this provides
the ability for admins to display a message to visitors in the case of
infrastructure or index data issues. Notices can have an expiry date and
can be disabled and re-enabled if needed. A subset of HTML can be used
for formatting the text, URLs will be made into clickable links, and
four "levels" are supported (info, success, warning and error).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* If a missing dependency is not required, show a warning instead of an
error
* If logger isn't specified we still need to skip to the next item, so
move the continue statement out of the conditional block. (In practice
I don't think this function is currently called anywhere in the code
without a logger specified, but let's fix it anyway).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a LAYERRECOMMENDS relationship is not satisfied, we shouldn't be
erroring out - it's a recommendation, not a hard dependency. Just show a
warning and allow processing to continue.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a file is modified and renamed it will show up in both
iter_change_type('M') and iter_change_type('R'), however naturally the
file that will exist will be the b path and not the a one, so we should
be looking at the b path or we will get errors.
FYI you can reproduce this with OE-Core (in a scratch database) using
the following procedure:
1) (in the OE-Core layer directory):
git checkout 59285b324f6d9ed270b0bef209ef5da22a620a83
2) update.py -l openembedded-core -b master -x --nofetch -r --fullreload
3) (in the OE-Core layer directory):
git checkout 086308aa2a5e332de6f00ed397c4a55d132f158f
4) update.py -l openembedded-core -b master -x --nofetch
Without this change you'll see the following error:
ERROR: Unable to read /opt/layerindex/layers/git___git_openembedded_org_openembedded-core/meta/recipes-devtools/python-numpy/python-numpy_1.13.1.bb: Traceback (most recent call last):
File "/opt/layerindex/layers/bitbake/lib/bb/command.py", line 84, in runCommand
result = command_method(self, commandline)
File "/opt/layerindex/layers/bitbake/lib/bb/command.py", line 568, in parseRecipeFile
envdata = bb.cache.parse_recipe(config_data, fn, appendfiles)['']
File "/opt/layerindex/layers/bitbake/lib/bb/cache.py", line 315, in parse_recipe
bb_data = bb.parse.handle(bbfile, bb_data)
File "/opt/layerindex/layers/bitbake/lib/bb/parse/__init__.py", line 117, in handle
return h['handle'](fn, data, include)
File "/opt/layerindex/layers/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 132, in handle
abs_fn = resolve_file(fn, d)
File "/opt/layerindex/layers/bitbake/lib/bb/parse/__init__.py", line 141, in resolve_file
raise IOError(errno.ENOENT, "file %s not found" % fn)
FileNotFoundError: [Errno 2] file /opt/layerindex/layers/git___git_openembedded_org_openembedded-core/meta/recipes-devtools/python-numpy/python-numpy_1.13.1.bb not found
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you're running a testing / internal instance then you really don't
want to be emailing maintainers on publish, so provide a setting you can
use to disable that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
Assume there is no master branch in hello layer:
$ update.py -l hello -b master
INFO: Skipping update of layer hello - branch master doesn't exist
This is correct since hello layer doesn't have master branch, but when --nocheckout:
$ update.py -l hello -b master --nocheckout
[snip]
INFO: Sorting layers for branch mater:
WARNING: Cannot find required collections on branch master:
WARNING: hello: LAYERDEPENDS: <snip>
This is incorrect, this patch fixed the problem, now it skips it since the
branch doesn't exists when --nocheckout.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When layer_a RECOMMENDS layer_b, try to add layer_b before layer_a, but if
layer_b is not found, still add layer_a.
And print summary error mesage:
$ update.py -b master
ERROR: Issues found on branch master:
openembedded-core: Added without LAYERRECOMMENDS
meta-secure-env: Failed to add since LAYERDEPENDS is not satisfied
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
$ git clone <url>
warning: remote HEAD refers to nonexistent ref, unable to checkout.
$ git rev-parse HEAD
HEAD
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Catch the error and avoid that.
And use "git reset --hard" to replace of "git reset --hard HEAD", HEAD is
default for git reset, so they are the same, but the later one reports error
when remote HEAD doesn't exist:
$ git reset --hard HEAD
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
[snip]
$ git reset --hard
No errors.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
$ update.py -b <new_branch>
[snip]
NOTE: Starting bitbake server...
Traceback (most recent call last):
File "update_layer.py", line 471, in main
utils.setup_core_layer_sys_path(settings, branch.name)
File "/buildarea1/lyang1/layerindex-web/layerindex/utils.py", line 376, in setup_core_layer_sys_path
core_layerdir = os.path.join(core_repodir, core_layerbranch.vcs_subdir)
AttributeError: 'NoneType' object has no attribute 'vcs_subdir'
[snip]
This is because core_layerbranch is not in database yet for completely new
branch, so it is None and we will get the error. Avoid calling
setup_core_layer_sys_path() when "update_layer.py --initial" will fix the
problem.
And also only add core layer's sys path when it is present, since core layer
may not be added yet for completely new branch.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to export the recipe listing for a layer to a CSV file
for importing into external tools. At the moment we include name,
version and license, but there is a parameter that lets you specify the
fields to include in the URL if desired.
Implements [YOCTO #12722].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We're about to replace this with a proper CSV export function, so we
don't need this dead code hanging around anymore.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's a bit of a pain to have to set the two python environment fields on
every record in order to have things set correctly, and it can easily
get forgotten, so try to set them automatically by default (assuming
reasonable naming).
Note that this does introduce an annoying behaviour whereby if you click
"Add another Maintenance plan layer branch" and then decide you don't
want it, the admin form will insist you fill in the fields unless you
clear out the python2/3 environment fields. I'm not sure how to fix
that, so I'm leaving it as-is for now.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Collect information about patches applied by a recipe, and record each
patch along with the upstream status, presenting them in the recipe
detail.
Implements [YOCTO #7909].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Checking out a revision in the bitbake/layer repositories is something
we are doing in a few places, so add a checkout_repo() function that
does this, ensuring that we don't get tripped up by any junk files,
and avoids checking out if the repository is already at the desired
revision (thus avoiding the clean operation and e.g. preserving any
.pyc files that aren't stale and would speed things up a little). Note
that we do the clean before checking out in case there are untracked
files that are tracked in the commit we are checking out.
In addition to adding this function, change the existing code that we
use in the update script to check out a layer use the new function.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It would be useful in some scenarios to get the complete list of
recursive dependencies for a layer, so add a function to do that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The Recipe Reporting System needs to be able to provide links to commits
in the web interface for the repository, but we can only do this if we
have a custom template URL just like we do for file/tree links, since
it's different for different git web interfaces. Add support in all the
various places for such a URL and make use of it in the RRS.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Save each remote SRC_URI so we can use these for the recipe reporting
system. This replaces an earlier implementation in the rrs branch where
we simply stored SRC_URI verbatim.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Every time we add something that links to Recipe we had to add it to the
exclusions list in the readonly_fields line for RecipeAdmin (and
ClassicRecipeAdmin), which is tedious and easily forgotten. We can avoid
this by looking at each field and excluding it by its attributes rather
than having a hardcoded list of names.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
