In production, we use settings.DEFAULT_FROM_EMAIL as our primary source
of from_email, rather than settings.SUBMIT_EMAIL_FROM
Default to DEFAULT_FROM_EMAIL if it is not Null, fall back to SUBMIT_EMAIL_FROM
The real consumer (in production) is a Celery task in layerindex/tasks.py
@tasks.task
def send_email(subject, text_content, from_email=settings.DEFAULT_FROM_EMAIL, to_emails=[]):
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
There is nothing preventing a period in a layer name from the
BitBake perspective.
For example, https://github.com/webosose/meta-webosose has:
meta-webos-backports/meta-webos-backports-4.1
[YOCTO #15373]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
https://github.com/webosose/meta-webosose has layers like:
meta-webos-backports/meta-webos-backports-4.1
which is 45 characters.
Allow for up to 60 characters in the vcs_subdir field in the LayerBranch class.
[YOCTO #15374]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Exposing the actual_branch field of LayerBranch in the UI
was intending to _enable_ setting a non-default value, but
the check was accidentally left in a state that _requires_
a value.
[YOCTO #15376]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Since commits:
bitbake f33ce7e7
'BBHandler/cooker: Implement recipe and global classes'
oe-core f5c12800
'classes: Update classes to match new bitbake class scope functionality'
we now have a lot of the oe-core recipe classes in meta/classes-recipe.
We are also missing any bbclasses in meta/classes-global
[YOCTO #15238]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Add support for classes-global and classes-recipe, but use
"proxy=True" to not create new tables for the new classes.
Rather, we use the bbclass_type field.
[YOCTO #15238]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Since we are only using parse_version for comparison (typically checking
that we are greater than some minimum version for tool or package), one
would think we can use packaging.version.parse as if it was parse_version
Unfortunately, this requires conforming to PEP-440 version definitions,
which does not work for e.g. autotools (2.72d) nor older openssl (1.1.1p).
We rely in these (and to be sure other) cases on the LegacyVersion behavior.
https://packaging.python.org/en/latest/specifications/version-specifiers/#summary-of-differences-from-pkg-resources-parse-version
"This specification purposely restricts the syntax which constitutes a
valid version while pkg_resources.parse_version attempts to provide some
meaning from any arbitrary string."
In order to have the least impact to the overall code, we instead add
packaging_legacy to requirements.txt and use packaging_legacy.version.parse
as if it was parse_version.
https://pypi.org/project/packaging-legacy/https://github.com/pypa/packaging/pull/407
Since pypi.org itself is depending on packaging_legacy (in fact, a pypi dev
developed the package), we can expect it to be supported for quite some time.
https://github.com/pypi/warehouse/pull/13500
[YOCTO #15348]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Some layers now have one branch with many supported
LAYERSERIES_COMPAT. If this branch name does not match
one of the stable releases, LayerBranches might not
have been created. When actual_branch is set, it is
only set in a LayerBranch object. We previously
could not update (create) a stable branch with actual_branch
except manually in the admin interface.
Add --force-create option to be used in conjunction with
--actual-branch (which already requires --branch) in the
update.py script. This tells the script to ignore the
fact that no layerbranch exists already.
Add --actual-branch to update_layer.py so that we can create
(and more importantly checkout) an actual_branch for the
given stable --branch.
Update utils.py to allow checking out of actual_branch when
a LayerBranch does not yet exist.
While we are at it, ensure that any Branch that is marked
as no update will be skipped even with --force-create. The
main reason that a Branch has updates disabled is because the
bitbake or python syntax has changed enough to cause exceptions.
This script can now be run with:
./layerindex/update.py \
--layer meta-weird-one \
--branch kirkstone \
--actual-branch=nonstandard \
--force-create
Which will attempt to create a meta-weird-one:kirkstone layerbranch
checked out at the 'nonstandard' branch from that layer's git repo.
This allows layerindex admins to at least populate the database
without tedious creation of layerbranches in the admin interface.
Helps make the "branch mapping" actually work and be useful:
[YOCTO #8008]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
* Add an "Update Layer" button to the layer detail view.
- This allows a user that is a member of is_staff to trigger
an update of the current layer (for the current branch)
* Add an "Update Layer" button to the reviewdetail view
- This allows a user that is a member of is_staff and has
publish_layer permissions to trigger an update attempt
of the layer under review (even in the un-published state)
* The update is run as a task with Celery
NOTE:
You must have the RABBITMQ_ and DATABASE_ credentials set
correctly in the docker/settings.py file or set via
environment variables or you will get authentication errors
talking to layersdb or layersrabbit containers.
[YOCTO #12484]
layerindex/views.py: add update_layer_view
layerindex/urls.py: add update_layer_view
layerindex/urls_branch.py: add update_layer_view
templates/layerindex/reviewdetail.html: add Update Layer button
templates/layerindex/detail.html: add Update Layer button
templates/layerindex: add updatelayer.html
TODO:
While the update is happening, the AJAX rendering of the
update.log is showing the b'' characters and not adding
any new lines. If you go back to the same task view
afterwards, the log is rendered as expected.
TODO:
After the update is completed, it would be nice to have a
button to return you to the page from where you called the
"Update Layer".
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
For layers which do not follow standard branch names (including
the inclusive naming move away from "master" to "main") we have
the actual_branch field set in a LayerBranch object. Previously
this was only exposed via the admin interface.
Allow layer maintainers (including upon submitting a new layer)
to set the 'Actual branch' in the web UI.
Add a check to make sure the actual_branch is a valid branch
name using 'git check-ref-format --branch <actual_branch>'
since we are not using full refs.
[YOCTO #8008]
NOTE:
Only existing LayerBranches will be editable. A new layer
can be submitted with a different branch for "master", but
only the "master" LayerBranch will be created.
Further changes to the update.py script will be needed to
make creation of new stable branches with an actual_branch
possible.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
The Branch Comparison view would throw an error because of a parser_view instead of parse_view typo.
[YOCTO #15332]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
File "/opt/layerindex/layerindex/update.py", line 525
failed_layers[branch].append('%s: Failed to add since LAYERDEPENDS [%s ...] is not
SyntaxError: '(' was never closed
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Make errors like this more useful:
ERROR: Issues found on branch nanbield:
meta-luneos: Failed to add since LAYERDEPENDS is not satisfied
meta-luneui: Failed to add since LAYERDEPENDS is not satisfied
In this case, meta-luneos depends on meta-luneui, but we cannot create a
'nanbield' layer branch, because meta-luneui LAYERDEPENDS on qt6-layer
collection, which currently has no 'nanbield' branch.
Use next(iter(value['deps'])) to get the first element in the OrderedDict
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Currently, the error message is something like:
ERROR: conf/layer.conf not found for layer meta-doom - is subdirectory set correctly?
This is because in this case meta-doom has a 'langdale' branch without
a conf/layer.conf
Another example is:
ERROR: conf/layer.conf not found for layer meta-st-stm32mp - is subdirectory set correctly?
This is because meta-st-stm32mp has a master branch, but it only contains
a README.md telling you to use the stable branches (e.g. nanbield).
Make it more obvious what the source of the error is by also displaying
the branch that was being attempted.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
The regex for Upstream-Status had (\w+), but this
does not include hyphen/dash. Explicitly add it to
the pattern.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
For both meta-poky/meta-yocto and meta-xilinx/meta-xilinx-core
we have a situation where the "collections" stayed the same
("yocto" and "xilinx" respectively) but the layer/layerbranch
changed. Without the "branch" argument to get_dependency_layers,
we were always defaulting to the older layer which first
defined the "collection".
Instead, add an option to use "branch" to filter on the expected
LayerBranch object. Keep the old behavior just in case someone
depends upon it.
[YOCTO #15221]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
We have never been checking out dependent layers at the same
release/branch. With the introduction of 'addpylib', this
became obvious due to parsing errors.
Ensure that known LayerDependency objects are checked out at
the expected branch/release. Since openembedded-core has already
been handled elsewhere, we skip it.
[YOCTO #15236]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
When a new release is run for the first time, no LayerBranch exists
yet and the update_enabled cannot be queried. The unintended result
is that all layers are skipped. Move the update_enabled check into
the code path where the layerbranch already exists.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Add a script which can either mark one --layer --branch or
use data in a --from-file to mark multiple layer:branch objects
as Yocto Project Compatible.
The --from-file is a json file is compatible with or which can be
generated by:
yocto-autobuilder-helper/scripts/list-yp-compatible-layers.py
[YOCTO #15093]
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Especially since LTS branches have become more popular as the
only active branch, we need to be able to turn off updates on
a LayerBranch basis rather than the current Layer or Branch
heavier hammers.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
To make it easier to quickly see what version we are using, add
the version to the filenames.
Also, use minified flavor in templates/base.html
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
The url returned by the reverse lookup ends in /submit/thanks/ but
the regex was improperly searching for a string without the terminating "/".
Thank you to Pawel Zalewski for reporting the 404 error.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Bitbake's api has been changed via:
afb8478d3 parse: Add support for addpylib conf file directive and BB_GLOBAL_PYMODULES
The conf file won't be parsed without baseconfig=True:
bb.parse.ParseError: ParseError at /path/to/oe-core/meta/conf/layer.conf:132: unparsed line: 'addpylib ${LAYERDIR}/lib oe'
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
The runcmd() would print an "ERROR" on failure which causes confusion since
the failure is expected on old branches, so subprocess.getstatusoutput to fix
the problem.
Minor rewording.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
The following commit has added on_delete=models.CASCADE to the ForeignKey:
===
commit 2d526f9b0d
Author: Amber Elliot <amber.n.elliot@intel.com>
Date: Thu Jun 25 15:57:34 2020 -0700
Updating models and imports for Django 2.2 upgrade.
===
The on_delete=models.CASCADE will remove the objects which reference the
removed objects, so there will be errors when removing a LayerBranch
with admin permissions:
===
Cannot delete layer branch
Deleting the selected layer branch would result in deleting related objects, but your account doesn't have permission to delete the following types of objects:
Append
distro
===
This is because has_delete_permission() always return False, remove it to use
Django's implementation to fix the problem.
Minor rewording.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
The commit 38e6288c7d has added oe-core to
BBLAYERS by default, and then there will be multiple BBFILE_COLLECTIONS which
causes update.py failure to update layers (most of the dependencies or
recommends are missing), use get_layer_var() to get BBFILE_COLLECTIONS to fix
the problem since it can ignore the extra oe-core from BBFILE_COLLECTIONS.
Fix minor typos. Minor rewording.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Many layers append BBFILE_COLLECTIONS and therefore have 'core <layer>'
During update.py, this means we are likely not handling the collection we
expect:
WARNING: /opt/workdir/git___git_openembedded_org_meta-openembedded/meta-oe: multiple collections found, handling first one (core) only
BBFILE_COLLECTIONS = "core"
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
ERROR: Variable BB_ENV_EXTRAWHITE has been renamed to BB_ENV_PASSTHROUGH_ADDITIONS
ERROR: Variable BB_ENV_EXTRAWHITE from the shell environment has been renamed to BB_ENV_PASSTHROUGH_ADDITIONS
ERROR: Exiting to allow enviroment variables to be corrected
Replace BB_ENV_EXTRAWHITE with new variable BB_ENV_PASSTHROUGH_ADDITIONS
In order to be backward compatible with older branches, we must first check
for the presence of the bitbake commit which implemented the variable name
change, using layerindex.utils.is_commit_ancestor().
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
Add a helper function to check if a given SHA1 hash is an ancestor in
the currently checked out branch, using:
git merge-branch --is-ancestor <commit> HEAD
NOTE: This will not match commits which have been cherry-picked.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
I noticed in the branch comparison plain text view ("Plain text" button
in the Tools -> Branch Comparison page) that in current versions of
Firefox the ellipses were coming through corrupted, though they looked
fine in the HTML version, and it turns out this is because I wasn't
specifying a character set encoding. It should be UTF-8, so add a
charset to the content type stating as such for this and other similar
views.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.microsoft.com>
If a BBHandledException occurs that means some error was logged, so we
need to handle any pending events so that we can actually have the error
logged. Tinfoil should really be doing this for us but at this stage in
the release we can't really fix this there, so do it here for now.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.microsoft.com>
So with honister / current master we can no longer get away with
bypassing BBLAYERS - it now needs to point to the core layer at
minimum. This is fine, we just need to skip parsing layer.conf if we're
parsing the core layer or we get some extra warnings we don't need.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.microsoft.com>
Added SPDX identifiers to all .py files except those in migrations directory.
Fixes: [YOCTO #13527]
Signed-off-by: Meh Mbeh Ida Delphine <idadelm@gmail.com>
Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org>
Removed all obsolete references to django.core.urlresolvers. Added the
newly required on_delete fields to foreign key relationships in models.py
and in all migrations.
Signed-off-by: Amber Elliot <amber.n.elliot@intel.com>
Signed-off-by: Paul Eggleton <bluelightning@bluelightning.org>
In OE-Core commit 8c9ef587fe499c612a878a1ab42092eb79b334ef an
AVAILABLE_LICENSES variable was added in license.bbclass where its value
is assigned with immediate expansion ( := ) with the result that it
looks in LICENSE_PATH for licenses; in turn LICENSE_PATH refers to
COREBASE. You might assume that COREBASE is always set, and normally it
is (since it's set OE-Core's layer.conf) - but in the layer index
context we do not parse layer.conf until a bit later, so it immediately
fails. The quick way to fix this is just to set our own (correct) value
for COREBASE and then AVAILABLE_LICENSES can be expanded successfully.
You might ask why we don't instead just set BBLAYERS such that we *do*
parse OE-Core's layer.conf - the answer is that that can have other
effects such as BBFILE_COLLECTIONS being set, and at least at the moment
the rest of the code isn't expecting that.
Fixes [YOCTO #13723].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
