If a source points to a local file, get the sha256sum of it and save it
into the field we just added.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow us to record a checksum for local sources (and potentially remote
ones if it is recorded in the recipe). This just adds the field, but
doesn't populate it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make it easier to import different versions of the other distro by not
requiring you to manually create the layer and branch.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If for some reason we do not find any spec files in the specified
directory, error out instead of marking all packages as deleted. (This
can happen if the wrong directory is specified.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Make patch / user fields non-editable for existing records (patch
mostly because the list is huge for a drop-down)
* User must be a superuser or the dispositioner to edit
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to mark each patch with a disposition indicating whether
the patch is interesting or not.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Editing on this page is done by clicking through to another page, so
it should be a DetailView rather than an UpdateView and shouldn't have a
form class set.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For the purposes of creating a dump suitable for importing into a fresh
database during initial setup, add a script that creates a dump of an
existing database with the complete structure but without any data that
involves users (or anything else we don't want).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you specified only the layer index URL and not the full API URL i.e.
https://layers.openembedded.org then we put an extra slash in the URL
which resulted in the request being redirected to the main page which
breaks the request.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to have registration/password reset emails working properly we
need to have both the site domain and email host/port set correctly.
Email host/port is easy to set (environment variable read into the
configuration). However, to set the site info, create a simple script
that we can call to do so.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's best practice for security reasons to use shell=False and pass
command line arguments as a list; it also avoids some pain with
escaping, so let's use it everywhere we can (in fact we're only left
with one place in layerindex/tasks.py where we now pass shell=True).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Indentation, spacing, and wrapping of a long import line.
Signed-off-by: Amber Elliot <amber.n.elliot@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add user security questions upon registration as extra authentication
for password reset. Three unique security questions must be chosen and
answered. Answers are then stored in the database with the same hashing
algorithm as the users's password.
On password reset, users get two chances to get two out of three
security questions answered correctly. After a second failure their
account is locked and email is sent to the admin. The same template is
shown for the axes lockout. Super user cannot reset their password until
they set security questions.
Users can update their security questions or add them if they weren't
originally set (in the case of super user) in Edit Profile.
Signed-off-by: Amber Elliot <amber.n.elliot@intel.com>
This involves changing how registration templates are referenced
and how the activation email is sent on user's email address change.
Signed-off-by: Amber Elliot <amber.n.elliot@intel.com>
This allows users to see the site ID in the admin page.
Signed-off-by: Amber Elliot <amber.n.elliot@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Patches often need to be applied in a specific order. For OE recipes we
were always storing the Patch objects correct order as they are
refreshed every time the recipe itself is refreshed, however for other
distro comparisons, import_otherdistro.py attempts to preserve existing
records, adds new ones and then deletes whatever is left over, which may
result in the order getting messed up over time. To avoid this issue,
record the order next to the patch and set the model meta-info to use
this to sort Patch queries by default.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Best practices state that you should use a mechanism that ensures files
get closed in case of any error, so let's do that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Errors deleting bitbake.sock and bitbake.lock have been observed when
shutting down tinfoil at the end of some of these scripts. Move the code
used in the main layer index update script to a function in utils.py and
use it everywhere in order to avoid the issue.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add links to the same recipe in other branches in the recipe detail page
(and RRS recipe detail page) so that you can see which versions are
available in other branches and drill down to the detail if you want to.
Implements [YOCTO #13019].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If someone adds you as a maintainer of a layer as a matter of courtesy
it would be nice if you get an email.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Some controls on the Submit / Edit Layer page were getting the wrong
width and any "feedback" elements were being pulled to the right because
of the size of their parent divs. Instead of pulling in the size of the
form control, act on the div to fix the problem. Additionally the
help text should now be allowed to expand to the full width of the
parent div since that is now sensible.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Send the current URL path as a parameter to the login URL so that after
successfully logging in, you come back to where you were beforehand.
Also hide the login/user drop-down on the login page since it's
superfluous.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Use a parameter to pass the current page to the Edit Profile URL so that
saving or cancelling returns you to that same page.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were using uitablefilter.js to provide live filtering of table rows
based upon a search field value, but it turns out this module really
isn't necessary - we can accomplish the same thing using simple jQuery
code. While we're at it, enable the search field on the layers list page
to work in conjunction with with the drop-down layer type selection, fix
pasting into the search field and refreshing with a search specified.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you're searching for items that are of status "unknown/not available"
in the other distro packages, you're usually doing so to find things
that haven't been dispositioned. Originally I had included them here
because the "available" status correctly excludes items with the
"distro-specific" status and "unknown/not available" seemed like it
should be the inverse, but usage seems to require that it not be.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a user goes to Edit Profile and changes their email address,
deactivate their account temporarily and make them go through the
registration process to confirm that the new email address is in fact
valid and theirs.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
On top of the CAPTCHA, prevent more than 10 name change attempts for the
same user with the Edit Profile page so that username enumeration is
made even more difficult.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Users may want to change their usernames for a number of different
reasons, but at the moment we require them to contact an admin to do
that. Provided we validate the new username correctly and add a CAPTCHA
to make automated enumeration difficult, we can add username to the Edit
Profile form and then users can do that any time they wish.
While we're doing this, show a message when the profile is successfully
updated.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make it possible for users to delete their own account and all associated
information from the database, should they decide they no longer wish to
use it.
(I checked the implications of doing this on our model structure -
anything with a foreign key to user is safe to delete with the exception
of RRS MaintenancePlan.admin which I needed to change on_delete for so
that it doesn't get deleted with the user).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Extend and override the default views so we can extend and override the
default forms to add a CAPTCHA field. This should prevent the automated
account creation requests we've been seeing on layers.openembedded.org
(luckily failing anyway due to bad domain names), but in any case this
also improves security by making it harder to do user enumeration.
For the registration page in particular, because Django's forms logic
tries to be helpful by showing all errors at once, we need to change it
so that if there's an error for the CAPTCHA then you only see that error
and no other - in particular you won't see "that username already
exists" if that is the case.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This was copy-pasted from the DistroSearchView and I clearly forgot to
replace the reference here.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's been reported to me that it's possible in a custom setup to have a
bitbake repo that intentionally doesn't have a master branch, so pick
up the bitbake branch from the master Branch record if it exists and the
bitbake_branch field is set.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
nvd3 and its python/django wrappers appear to be no longer actively
maintained, and at least the wrappers were a bit clunky to use. Looking
around for a suitable replacement, Chart.js seems capable, has no
additional dependencies and is fairly simple to use. As a bonus we get
to drop a few Python dependencies from our list.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's not too common but there are instances where people have copied
.inc files into their own layer and modified them, and if you are using
such a layer that could result in unexpected behaviour. In order to get
a handle on when this is being done, collect data about all .inc files
and show duplicates in the Duplicates screen.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Split out the code used in the recipe search views to its own function
and use that same function in three different places rather than having
a copy of largely the same code. Also take the opportunity to add some
comments.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
On layers.openembedded.org we're seeing requests from some search engine
crawlers requesting the CSV export URL with an invalid branch for the
layer. I couldn't see the referer anywhere in the logs but I suspect it
has to do with some recent cleanup work I did in the database where I
deleted some invalid LayerBranch records - they were probably following
links in a cached version of the webpage. In any event we want to return
404 in this situation rather than an internal server error.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In order to show bbappends on the recipe detail page we are doing a
regex query to find any whose names match up with the recipe. In the
layer index instance at layers.openembedded.org viewing the recipe
detail page for any recipe whose name contains ++ (e.g. libsigc++-2.0 in
meta-oe) results in an invalid regex and causes a database error. Escape
any + signs in the name used within the regex in order to fix this.
(I wasn't actually able to reproduce this on my own setup despite also
using MariaDB, but I did find that the unescaped query was not correctly
matching records so it needed to be fixed anyway.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When you make changes to the infrastructure it can be useful to test
that email sending is working, since for that to work that involves the
code, Celery, RabbitMQ and SMTP being functional. However, up until now
to run a test you needed to submit a fake layer which is a bit annoying.
Add an explicit "Test email" option to the Tools drop-down for staff
users to allow them to send an email to themselves.
Note: the page will come back when the Celery job has been created, it
does not check and report on the job status - you need to look on the
server side to see that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Sometimes we get massively long lines from the update script
(particularly if there's an error) so ensure that long lines get
wrapped.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we want to be able to read in patch information on python2-based
branches (e.g. fido) then we need to use codecs.open() instead of open()
here since python2's open() did not support the encoding parameter.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you update a python2-based branch then the python2-compatible version
of bitbake will be checked out, but we are calling into bitbake's
bb.utils directly here from python 3 and thus you get an error about
commands.getstatusoutput being missing (since that is not available in
python 3 and the old version of bitbake refers to it). To fix this,
check out origin/master in the bitbake repo right before we call the
code in question.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When we print a warning about the value of a CharField being truncated,
print out the string representation of the object so we have a chance of
finding the offending object.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Enable searching on vcs_url on LayerItem and layer name and vcs_url on
LayerBranch. This makes it easier to find the layers/branches in a
particular repository (e.g. meta-openembedded).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
I've come across at least one layer that is now hosted on gitlab.com, so
add support in the layer submission/edit form and import_layer.py for
automatically determining the other fields for gitlab.com URLs.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We don't want to allow any other arguments to be injected into these
commands, so disable the shell and pass the parameters in the form of a
list to prevent that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Sometimes layers get created on master and then the master branch is
removed in favour of a release branch. In that case it can be useful to
switch the existing layerbranch record rather than having to create a
blank one and copy everything over.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The code in recipeparse.setup_layer() was trying to log a warning in the
case where LAYERRECOMMENDS not being satisfied, however there is no
actual logger object in this context. Pass it in via a parameter and
update all callers to pass it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In c26604146a I made a fix to change where
the bitbake code writes out bitbake.lock and other files it creates
during parsing, but didn't adequately test it and it turns out our
call to delete the temp directory races against bitbake deleting
bitbake.lock and bitbake.sock. For now the simplest way to deal with
this is to ignore the errors since we don't care about these files,
we just want the temp dir gone.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Our setup when parsing recipes is a little unusual in that we have no
bblayers.conf, thus findTopdir() which is used to find where to put
bitbake.lock (and bitbake-cookerdaemon.log as of the recent bitbake
commit 1620dbc48ffb2a882371cf9174a7b12648befc8a) defaults to the
parent's parent of where bitbake.conf can be found, which is the meta/
subdirectory of the OE-Core repo, thus that's where we now find
bitbake-cookerdaemon.log gets written out. We really don't want to be
writing anything into the metadata repositories so create a fake
conf/bblayers.conf in our temp directory to make findTipdir() pick that
instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This isn't a visual thing, this select element must remain hidden, so it
seems a bit more appropriate to me to specify the style directly on the
element rather than using a CSS class to do it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
For situations where the user launches a distro comparison update
process and then shortly afterwards realises it is operating with the
wrong configuration (or is otherwise broken) and is going to take a long
time to finish, add a button to the task page to stop the task. This was
tricky to get working, since the default behaviour of Celery's revoke()
would either terminate both the Celery task process along with the update
process (leaving us with no log saved to the database) or worse not even
kill the update process, depending on the signal sent. To avoid this,
send SIGUSR2, trap it in the task process and kill the child process,
returning gracefully. To make that possible I had to rewrite runcmd() to
use subprocess.Popen() instead of subprocess.check_call() as otherwise
we can't get the child's PID.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a distro comparison update task fails (returning a non-zero value to
indicate as such) we were not able to see this easily from the frontend.
Show success/failure in the form of a label on the task page and general
update list/detail, and if the task fails while we're watching then make
the progress bar go red as well. Also make a distinction between the
process failing (retcode > 0) and being terminated (retcode < 0, e.g.
process was killed).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Provide a mechanism for distro comparison update tasks to display
progress. In practice this means the update command needs to write the
progress percentage to a file and then the log view (which is polled by
the frontend) reads this file. Originally I was going to use a FIFO for
this but that turned out to be a but unreliable; I also tried to use
Celery's state mechanism to pass it back but I simply could not get it
to work. The file-based mechanism is good enough though.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were refreshing the page constantly in order to show output while
a task was running, which basically worked but is horrible. Instead,
write the task output to a file and then use AJAX calls to request
whatever output has been written to the file since the last call
and call this roughly every second. Put the output in a scrollable <pre>
element instead of making it the length of the page, and auto-scroll
to the end (unless the user grabs the scrollbar and pulls it upwards -
it may not be immediately obvious that you can do this if there is a lot
of output since you have to pull it up when the scrolling animation is
not running, but it is possible).
An alternative would be to have used some kind of long-lived HTTP
session or a websocket, but those come with their own set of problems
so I elected to use this much simpler method.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
In a bunch of places we needed to get the branch we were supposed to
be checking out (which is actual_branch if that is set, otherwise the
normal branch name). Add a function to do that.
Additionally, instead of showing the normal branch name next to the
"last update" date, use the result of this new function.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Use a more modern version of Bootstrap and take the opportunity to
upgrade jQuery to the latest version at the same time. This provides
better browser compatibility, moves to MIT license, allows us to make
the site more responsive for different devices in future, and provides
theming capabilities for custom installs among other improvements.
(I chose to upgrade to v3 for now rather than straight to v4 as it was
easier to do this gradually.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were missing some import statements here, so clearly I didn't test
this as I thought I had.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Now the logic is:
Use options.layer_type if specified, and guess if not. Default to 'M'.
Note choices=['A', 'B', 'S', 'D', 'M', ''], the '' is for default='', we can't
use default='M' here, otherwise we don't know whether the 'M' is specified by
user or is the default value, we don't guess if it is specified by user,
otherwise, guess.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a user doesn't have publish rights and the type of the layer isn't
already "Base" then disallow selecting the Base layer type. Some
submitters are selecting this type for their own layers, but it's pretty
much reserved for openembedded-core and meta-oe (so that they appear at
the top of the layer list).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We do not want to be prompting the user for a password during layer
updates or upstream checks, e.g. in the case where a repo requires
authentication, or on github where any fetch of a nonexistent repo
apparently triggers authentication.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
During debugging a parsing issue, we don't really want to continue if a
parsing error occurs, and in that situation I usually end up using
Ctrl+C to exit early. Add an option to exit immediately upon error to
avoid having to do that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When showing the error/warning counts for update records we need to
include any errors/warnings that are shown only in the main update log,
so we need to adjust how these are collected. Use a function rather than
pure aggregation to give a bit more control, and a {% with ... %} block
in the template to avoid the functions being called more than necessary.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If an exception occurred during the update then we were managing to save
the update record, but we did not include the exception traceback in the
log for the update. Catch the exception and log it which ensures it gets
captured in the update record and still gets printed as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make layerupdate collection slightly more reliable and make it easier
to see when updates have actually been captured:
* Split layerbranch into separate layer and branch fields, since there
may not be a layerbranch in existence but we might want to log an
error relating to the branch and layer.
* Show all layerupdates on the update detail page, not just those with
log messages
* Record before and after revisions and show these in the update detail
and layerupdate detail (with links)
* Record return code of update_layer process
* Highlight layer updates with a non-zero return code, errors or
warnings in the output on the update detail page
* Show duration on the layerupdate detail page
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When an exception occurs during the main part of update_layer, we were
catching and printing it but that's not enough - we need to do the
following as well:
* Use logger.error() to print the exception information, so that it gets
logged and highlighted as an error in the layer update
* Exit with a non-zero return code so that update.py knows it has failed
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If for some reason update_layer.py does not print out the values we
expect then we shouldn't be throwing a traceback, we should be handling
it gracefully - i.e., print an error and then move on to the next layer.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
On the recipe detail page we provide a link to the actual recipe file
for reference purposes. However, for recipes that include a common .inc
file, many of the definitions of variable values will be inside the
.inc, therefore if you just look at the recipe you won't see the full
picture. Of course you can just go up to the parent directory in the
repository web interface, but for convenience's sake add links to any
files that are included/required by the recipe that are adjacent to
the recipe itself. (We already have the data in the form of the
RecipeFileDependency records that are intended to ensure we know when
the recipe needs to be updated if one of the files it includes changes).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Specifying the covering recipe in the comparison recipe detail page was
always a bit awkward - you could only type the name, if you wanted to
actually find a recipe or look up the currently selected one's details
then you had to open another browser tab/window. To fix this, replace
the form on the comparison recipe detail page with a side-by-side
display of the covering recipe's information, along with a button that
lets you search and then select the covering recipe and at the same time
enter comments or set any of the other cover fields.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we're collecting this useful info we should really display it. Try to
make the link clickable if possible.
At the same time, add the layerbranch to the list display for Source
objects in the admin site.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script to fetch package sources from Fedora's Pagure
infrastructure site at https://src.fedoraproject.org/, based upon
github-fetch.py (I broke out the internal fetch function with the intent
to share it with that other script, however I didn't in the end as I
wanted the scripts to remain independent).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script to fetch/update all repositories within a GitHub
organisation. Note that to run this you will need an access token, which
you can create by going to GitHub and then
Settings -> Developer Settings -> Personal access tokens.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Comparison updates might involve some custom fetch process, so provide a
mechanism to register these via settings.py on a per-branch basis. If
an update command is defined for a branch and the logged-in user has the
new "update_comparison_branch" permission, an "Update" button will show
up on the recipes page for the comparison branch for authenticated
users that will trigger the command in the background (as a celery job)
and then show a page that displays the status. The status isn't shown in
real-time since that requires quite a lot of plumbing, but the page at
least auto-refreshes.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Make it clearer what this field is for (it should be set for every
comparison layer, so that they don't show up in places they shouldn't).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If other distro comparison package(s) exist for a recipe being shown in
the RRS recipe detail page, link to the page for each package as well as
any extra URLs.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a structure that lets you define a template URL per layer to be
shown per comparison recipe. For example, you could use this to define a
URL template to link to the upstream summary page for the package (e.g.
Fedora's page for the acl package is at
https://apps.fedoraproject.org/packages/acl, so you would use
https://apps.fedoraproject.org/packages/%pn% as the template and then
this would be shown for every package).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to export the comparison search results to CSV format in
order to allow importing the data into external tools.
Note: I implemented this in a different way than the earlier recipe CSV
export, i.e. it uses a template to render the CSV instead of a
function-based view with the Python csv module - the reason for this is
we can reuse the same view as we use for producing the search, with all
of the flexibility that gives us.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When doing the reversed query it's often desirable to exclude recipes
by inherited class, for example those that inherit the packagegroup,
image and meta classes as they don't actually build anything and thus
aren't going to match up with anything in the other distribution.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Provide a lazy-loaded popup for selecting layers to include in the query
instead of having it as a simple drop-down, so you can select more than
one layer.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Enable "reversing" the comparison, so you can see which recipes on the
OE side match up (or don't) with the other distro. The filtering for
this is a bit awkward, since we don't have an actual foreign key for the
link, hence the hairiness of the code.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add support for importing package information extracted from a running
Debian system (i.e. from the output of apt-cache show "*").
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a flag that can be set and searched for to indicate that we need to
take care of importing a package or a patch applied by a package.
Ideally the comments would elaborate on what's needed (if it's not
obvious from the cover status).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Packages with the "Not available" status were inexplicably excluded from
the graph, include them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Allow searching on:
* Any "available" status (i.e. other than "Unknown" or "Not available")
* Whether the package has patches or not
* What the covering layer is (assuming there is one).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's a bit crude, but accept '' or "" as meaning search for entries
with an empty category.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add an option to skip certain recipes by name (a crude workaround for
when we would otherwise get an erroneous match).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the source URL is fetching from pypi then we know it's python;
similarly cpan means perl.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If we always set cover_pn if there is a match rather than only setting
it if it's different from pn then it simplifies the code elsewhere.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Only apply OE-Classic-specific logic after checking that it's actually
OE-Classic
* Ignore recipes marked deleted
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script to import "recipe" information from other distro metadata,
based on the import_classic.py script. At the moment, this assumes a
directory where each subdirectory is a package directory containing a
spec file; this would be suitable for distributions such as Fedora
assuming you have all of the package repos checked out locally. Since
you can add additional information to these records (the cover fields
pointing to matching recipes), existing records are updated rather than
deleting everything and re-importing, and we only mark records as
deleted rather than actually deleting them (in case you accidentally
point the script at an empty directory or similar).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Turn the existing OE-Classic support into something a bit more
generic so we can import data from other distributions and compare it to
what we have in layers.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We were sorting the layers in the wrong order when trying to choose
which layer to pick a matching recipe from - it needs to be descending
priority order, not ascending.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add a script that uses the REST API to gather information from another
running layer index instance (e.g. layers.openembedded.org) and import
that into the local index. Only information for branches that are
already set up is imported, and only manually entered information -
no recipes, machines, etc.
Partially implements [YOCTO #9760].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the cover status for a comparison recipe is Unknown, Not available,
or Distro-specific, then we disable the recipe and layerbranch fields in
the form (using Javascript), however the form could still be saved and
any existing values would persist. Clear out the fields upon saving if
they are set under these circumstances.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* Check if bitbake directory can be found before trying to do anything
with the repo
* Split try..finally into two so we don't try to shut down a nonexistent
tinfoil when we failed to gain a lock.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a file is moved (renamed) to a path outside of the layer, e.g.
another layer within a multi-layer repository, then we need to treat it
as a delete. Up until now we were updating the path and continuing, and
then the recipe was also picked up as an add in the other layer, leading
to duplicate recipe entries. I'd noticed these duplicates before but up
until now I'd thought that they were due to another bug we already
fixed, apparently not.
In order to remove these erroneous duplicate entries in existing
databases I have also added a layerindex/tools/fixup_duplicates.py
script. I've also made the -r/--reload option delete them as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to show a notice at the top of every page; this provides
the ability for admins to display a message to visitors in the case of
infrastructure or index data issues. Notices can have an expiry date and
can be disabled and re-enabled if needed. A subset of HTML can be used
for formatting the text, URLs will be made into clickable links, and
four "levels" are supported (info, success, warning and error).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
* If a missing dependency is not required, show a warning instead of an
error
* If logger isn't specified we still need to skip to the next item, so
move the continue statement out of the conditional block. (In practice
I don't think this function is currently called anywhere in the code
without a logger specified, but let's fix it anyway).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a LAYERRECOMMENDS relationship is not satisfied, we shouldn't be
erroring out - it's a recommendation, not a hard dependency. Just show a
warning and allow processing to continue.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If a file is modified and renamed it will show up in both
iter_change_type('M') and iter_change_type('R'), however naturally the
file that will exist will be the b path and not the a one, so we should
be looking at the b path or we will get errors.
FYI you can reproduce this with OE-Core (in a scratch database) using
the following procedure:
1) (in the OE-Core layer directory):
git checkout 59285b324f6d9ed270b0bef209ef5da22a620a83
2) update.py -l openembedded-core -b master -x --nofetch -r --fullreload
3) (in the OE-Core layer directory):
git checkout 086308aa2a5e332de6f00ed397c4a55d132f158f
4) update.py -l openembedded-core -b master -x --nofetch
Without this change you'll see the following error:
ERROR: Unable to read /opt/layerindex/layers/git___git_openembedded_org_openembedded-core/meta/recipes-devtools/python-numpy/python-numpy_1.13.1.bb: Traceback (most recent call last):
File "/opt/layerindex/layers/bitbake/lib/bb/command.py", line 84, in runCommand
result = command_method(self, commandline)
File "/opt/layerindex/layers/bitbake/lib/bb/command.py", line 568, in parseRecipeFile
envdata = bb.cache.parse_recipe(config_data, fn, appendfiles)['']
File "/opt/layerindex/layers/bitbake/lib/bb/cache.py", line 315, in parse_recipe
bb_data = bb.parse.handle(bbfile, bb_data)
File "/opt/layerindex/layers/bitbake/lib/bb/parse/__init__.py", line 117, in handle
return h['handle'](fn, data, include)
File "/opt/layerindex/layers/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 132, in handle
abs_fn = resolve_file(fn, d)
File "/opt/layerindex/layers/bitbake/lib/bb/parse/__init__.py", line 141, in resolve_file
raise IOError(errno.ENOENT, "file %s not found" % fn)
FileNotFoundError: [Errno 2] file /opt/layerindex/layers/git___git_openembedded_org_openembedded-core/meta/recipes-devtools/python-numpy/python-numpy_1.13.1.bb not found
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If you're running a testing / internal instance then you really don't
want to be emailing maintainers on publish, so provide a setting you can
use to disable that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
Assume there is no master branch in hello layer:
$ update.py -l hello -b master
INFO: Skipping update of layer hello - branch master doesn't exist
This is correct since hello layer doesn't have master branch, but when --nocheckout:
$ update.py -l hello -b master --nocheckout
[snip]
INFO: Sorting layers for branch mater:
WARNING: Cannot find required collections on branch master:
WARNING: hello: LAYERDEPENDS: <snip>
This is incorrect, this patch fixed the problem, now it skips it since the
branch doesn't exists when --nocheckout.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
When layer_a RECOMMENDS layer_b, try to add layer_b before layer_a, but if
layer_b is not found, still add layer_a.
And print summary error mesage:
$ update.py -b master
ERROR: Issues found on branch master:
openembedded-core: Added without LAYERRECOMMENDS
meta-secure-env: Failed to add since LAYERDEPENDS is not satisfied
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
$ git clone <url>
warning: remote HEAD refers to nonexistent ref, unable to checkout.
$ git rev-parse HEAD
HEAD
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Catch the error and avoid that.
And use "git reset --hard" to replace of "git reset --hard HEAD", HEAD is
default for git reset, so they are the same, but the later one reports error
when remote HEAD doesn't exist:
$ git reset --hard HEAD
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
[snip]
$ git reset --hard
No errors.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Fixed:
$ update.py -b <new_branch>
[snip]
NOTE: Starting bitbake server...
Traceback (most recent call last):
File "update_layer.py", line 471, in main
utils.setup_core_layer_sys_path(settings, branch.name)
File "/buildarea1/lyang1/layerindex-web/layerindex/utils.py", line 376, in setup_core_layer_sys_path
core_layerdir = os.path.join(core_repodir, core_layerbranch.vcs_subdir)
AttributeError: 'NoneType' object has no attribute 'vcs_subdir'
[snip]
This is because core_layerbranch is not in database yet for completely new
branch, so it is None and we will get the error. Avoid calling
setup_core_layer_sys_path() when "update_layer.py --initial" will fix the
problem.
And also only add core layer's sys path when it is present, since core layer
may not be added yet for completely new branch.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Add the ability to export the recipe listing for a layer to a CSV file
for importing into external tools. At the moment we include name,
version and license, but there is a parameter that lets you specify the
fields to include in the URL if desired.
Implements [YOCTO #12722].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We're about to replace this with a proper CSV export function, so we
don't need this dead code hanging around anymore.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It's a bit of a pain to have to set the two python environment fields on
every record in order to have things set correctly, and it can easily
get forgotten, so try to set them automatically by default (assuming
reasonable naming).
Note that this does introduce an annoying behaviour whereby if you click
"Add another Maintenance plan layer branch" and then decide you don't
want it, the admin form will insist you fill in the fields unless you
clear out the python2/3 environment fields. I'm not sure how to fix
that, so I'm leaving it as-is for now.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Collect information about patches applied by a recipe, and record each
patch along with the upstream status, presenting them in the recipe
detail.
Implements [YOCTO #7909].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Checking out a revision in the bitbake/layer repositories is something
we are doing in a few places, so add a checkout_repo() function that
does this, ensuring that we don't get tripped up by any junk files,
and avoids checking out if the repository is already at the desired
revision (thus avoiding the clean operation and e.g. preserving any
.pyc files that aren't stale and would speed things up a little). Note
that we do the clean before checking out in case there are untracked
files that are tracked in the commit we are checking out.
In addition to adding this function, change the existing code that we
use in the update script to check out a layer use the new function.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It would be useful in some scenarios to get the complete list of
recursive dependencies for a layer, so add a function to do that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The Recipe Reporting System needs to be able to provide links to commits
in the web interface for the repository, but we can only do this if we
have a custom template URL just like we do for file/tree links, since
it's different for different git web interfaces. Add support in all the
various places for such a URL and make use of it in the RRS.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Save each remote SRC_URI so we can use these for the recipe reporting
system. This replaces an earlier implementation in the rrs branch where
we simply stored SRC_URI verbatim.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Every time we add something that links to Recipe we had to add it to the
exclusions list in the readonly_fields line for RecipeAdmin (and
ClassicRecipeAdmin), which is tedious and easily forgotten. We can avoid
this by looking at each field and excluding it by its attributes rather
than having a hardcoded list of names.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the RRS is enabled, then add a link to it in the tools menu. I don't
expect this to be used a lot, but otherwise the only way you'd get there
would be either externally or via the link from the layer detail.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
The old RRS branch had its own addition of dependency support, but in the
mean time we added that to the layer index in the master branch using a
different structure. Adapt the RRS recipe detail page to that structure.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
If the RRS is enabled, then add a way to get from the layer detail page
to any maintenance plans in which the layer is included.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Sometimes we get back UTF-8 characters (particularly when picking up
names from git commits), and the ascii codec will error out if that
happens, so switch over to utf-8.
We can as a result remove the decode() calls from the bulk change view.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Give some space to the tabs,
Add some top margin to the nav-pills class so we
create some breathing space between the milestone
overview and the tabs.
Change 'No Maintainer' to 'No maintainer'
Just to keep the capitalisation style consistent
across the interface.
Apply the muted class to all not-sortable table headings
The class was missing from the Recipe and Maintainer
columns in the recipes table; and from the Not updated
and % done column in the maintainers table.
Remove the strong tag from the recipe status information
This is just to match the presentation of the milestone
overview information in the base_toplevel.html template.
Separate the footer from the bottom of the viewport,
It's hard to see the footer on click on its links when
they are so close to the bottom of the veiwport, so
add some margin at the bottom of the footer <div>.
Change the label of the recipes tab,
From 'Recipes status' to 'Recipes upstream status' to
match the label of the 'Upstream status' filter in the
recipes table.
Signed-off-by: Belen Barros Pena <belen.barros.pena@intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
In e902b67bcc I missed a couple of Context
usages in the layer publish view and the result was that it broke
publishing a layer (and apparently I didn't run a final test on that,
shame on me).
Thanks to Yi Zhao <yi.zhao@windriver.com> for pointing this out.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
There were several issues with this code, including that it used
SortedDict which was removed in Django 1.9 and that it seemed not to
have been fully updated to accommodate changes in bitbake's recipe
parsing API. In the end I decided the simplest thing would be to move it
over to using oe.recipeutils.patch_recipe() which is actually a now much
more mature version of the code that originally started life here. With
that we can get the bulk change functionality working again and gain
some of the improvements in behaviour that we've developed in
oe.recipeutils.patch_recipe(), as well as avoiding effectively
duplicated code.
Implements [YOCTO #9730].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
It seems that in df492b1277 the original
intention was to move setup_layer to the utils module, but that didn't
actually get done in the final patch - however the change was made here
to accommodate the move, meaning it's been broken since then. Revert
that so we're actually calling the function in the place it exists.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
With Django 1.10+, if you use get_template() to retrieve a template,
then you can't pass a context when calling .render() on it, you need to
pass a dict instead.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
We use django-reversion to track history of admin/maintainer changes to
the site, and part of our extension on top of django-reversion involves
annotating each "revision" with a description of the changes that were
made. In django-reversion 2.0.0+ the pre_revision_commit signal that we
were using to do this annotation is gone in favour of just using
Django's standard pre-save signal. This was a little challenging to
adapt to for our purposes but not impossible.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
resolve_variable isn't available with Django 1.10+, and it's not even
used here, so just drop it from the import line.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
