MIRRORS/layerindex-web
1
0
Fork 0
mirror of git://git.yoctoproject.org/layerindex-web.git synced 2025-07-19 20:59:01 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity

Rework README documents

Browse Source 
Change the focus to really be on the docker-based setup:

* Integrate docker/README into the main README
* Move all documentation of the standalone setup to README.devel

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
This commit is contained in:
Paul Eggleton 2019-01-25 10:23:56 +13:00
parent ee7e5a0564
commit b5892b0fdb
2 changed files with 299 additions and 122 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

300
README
View File

@ -8,113 +8,88 @@ OE-Core.
There are two main methods of setting up this application - within There are two main methods of setting up this application - within
a set of Docker containers, or standalone. The Docker-based setup a set of Docker containers, or standalone. The Docker-based setup
is more suited for production whereas standalone is a bit easier is more suited for production whereas standalone is a bit easier
for development. If you simply want to run and use the layer index, for development. This document will consider only the Docker-based
please use the docker setup. setup; for standalone please see README.devel.
Docker Setup Docker Setup
------------ ------------
To use layerindex-web hosted within Docker containers, refer to The dockersetup.py script will set up and configure a cluster of 5 or 6
docker/README keeping in mind you'll need to set up Docker properly docker containers:
as part of the setup process. Once you've run through those
instructions you'll need to return to this document and follow the - layersapp: the application
instructions in the section "Database Setup" below. - layersdb: the database
- layersweb: NGINX web server (as a proxy and for serving static content)
- layerscelery: Celery (for running background jobs)
- layersrabbit: RabbitMQ (required by Celery)
- layerscertbot: Runs certbot to keep letsencrypt certificates up-to-date
(optional, default disabled)
The script will edit all necessary configuration files, build and launch all
containers, and do the initial database setup. It is advised that you start
with a .sql database file to prepopulate your database. The following
instructions will walk you through the setup.
1) Install docker and docker-compose per instructions:
https://docs.docker.com/compose/install/
** Note: for latest docker-compose version follow the directions above,
rather than using a perhaps outdated one provided by your distribution.
2) Run the setup script (dockersetup.py). You can optionally supply your
hostname, proxy settings, a sql database file of layer mappings to import,
and a host to container port mapping. For more information, run:
./dockersetup.py -h
Example command to run containers with a proxy and with a database to
import:
./dockersetup.py -d ~/databasedump.sql -p http://<proxyserver>:<port>
NOTE: If you want email sending to work (required for user registration and
password resets), you should use the -e/--email-host option to specify your
outgoing email server.
During the setup you will be asked for a username, email and password to
set up a super user for the database.
3) Once the script completes, open a web browser and navigate to the URL
printed out by the script. By default that would be: https://localhost:8081
4) If you have chosen to not supply a prepopulated database and are instead
starting fresh, you should now follow the instructions in the
"Database Setup" section of the main README.
5) If you need to rerun this script for any reason a second time, you'll need
to choose between two modes:
A) Updating (-u/--update) - updates the code and runs any database upgrades
if applicable, or
B) Reinstalling (-r/--reinstall) - deletes the containers and reinstalls
from scratch. Be warned that this will throw away all data in the
database.
Note that updating with -u/--update will only work if the configuration
changes originally made by dockersetup.py upon installation (e.g. passwords,
hostname, etc.) are still present in the source tree.
Standalone Setup
----------------
In order to run this application standalone you will need: TROUBLESHOOTING:
* Python 3.4+ - Network issues behind a proxy when building container: On some systems
* Django 1.11.x - tested with 1.11.12; newer versions may work, but (particularly where dnsmasq is installed), /etc/resolv.conf is set to
the application has not been tested with 2.0 or newer. 127.0.0.x, rather than your local DNS server. Docker will look there for
* RabbitMQ 3.6.x - tested with 3.6.10. your DNS server, and when it fails to find it it will default to using a
* For production usage, a web server set up to host Django applications public one (frequently 8.8.8.8). Many corporate proxies blocks public DNS
(not needed for local-only testing/development) servers, so you will need to manually supply the DNS server to docker using
* A database supported by Django (SQLite, MySQL, etc.). Django takes /etc/docker/daemon.json:
care of creating the database itself, you just need to ensure that the
database server (if not using SQLite) is configured and running.
* Third-party Django/python modules as detailed by requirements.txt.
If you're using pip, you can install these by running:
pip install -r requirements.txt
* On the machine that will run the backend update script (which does not
have to be the same machine as the web server, however it does still
have to have Django installed, have the same or similar configuration
in settings.py and have access to the database used by the web
application):
* Python 2.7.6+ / Python 3.4+ to match with the version of BitBake
for the OpenEmbedded branch being parsed (for modern versions it's
Python 3.)
* Python dependencies as per requirements.txt (we still need Django
etc. here since we interact with the database through Django's ORM.)
Example virtualenv-based setup for the above: {"dns": ["xx.xx.xx.xx] }
Python's virtualenv provides an easy way to isolate the python dependencies
of applications such as the layer index. Here's an example of setting up a
virtualenv for the layer index that's particularly useful for development.
(This assumes a Debian-based distribution, adjust accordingly for other
distros).
1. Install required host distro packages (some of these are required by
pip to build the dependencies; it's also assumed you want MariaDB as
the database backend):
sudo apt-get install virtualenv libmariadb-client-lgpl-dev build-essential python3-dev libjpeg-dev libz-dev libfreetype6-dev mariadb-server rabbitmq-server
2. Work around path issues (you may not need this):
sudo ln -s /usr/bin/mariadb_config /usr/bin/mysql_config
3. Create a Python 3 virtualenv (path can be anywhere you like):
virtualenv -p python3 /path/to/desired/venv
4. Activate the virtualenv:
. /path/to/desired/venv/bin/activate
5. Install requirements:
pip install -r requirements.txt
Configuration and running:
1. Edit settings.py to specify a database, EMAIL_HOST, SECRET_KEY and
other settings specific to your installation. Ensure you set
LAYER_FETCH_DIR to an absolute path to a location with sufficient
space for fetching layer repositories. Modify RABBIT_BROKER
and RABBIT_BACKEND to reflect the settings used by your RabbitMQ
server.
2. Run the following commands within the layerindex-web directory to
initialise the database:
python3 manage.py migrate
python3 manage.py createsuperuser
3. You can test the web application locally by setting DEBUG = True in
settings.py and running the following:
python3 manage.py runserver
Then visit http://127.0.0.1:8000/layerindex/ with your browser. As
with all Django applications there is an admin interface available
at http://127.0.0.1:8000/admin/ also. The initial login and password
will be those you entered in the step above when creating an admin
account.
NOTE: This local server should only be used for testing - for
production you need to use a proper web server and have DEBUG set
to False (or use the Docker-based setup).
3.1. In order to process asynchronous tasks like sending email,
you will need to run a Celery worker:
celery -A layerindex.tasks worker --loglevel=info
@ -128,12 +103,11 @@ setup within it:
database, or some equivalent that contains conf/bitbake.conf for database, or some equivalent that contains conf/bitbake.conf for
the base system configuration. To add this, follow these steps: the base system configuration. To add this, follow these steps:
1.1. With the server running, go to the main page - 1.1. With the server running, go to the admin page -
http://127.0.0.1:8000/admin/ (or http://localhost:8080/admin/ http://localhost:8080/admin/ by default (the hostname must
for the docker-compose setup - the hostname must match what match what the web server is configured for i.e. what you
the web server is configured for). Then click on the "Log in" specified when running dockersetup.py). Use the login/password
button on the top right and use the login/password for the for the admin account specified during setup.
admin account you added previously.
1.2. Click on the "Submit Layer" button in the top right and 1.2. Click on the "Submit Layer" button in the top right and
enter the details for the core layer. To use the real enter the details for the core layer. To use the real
@ -153,7 +127,7 @@ setup within it:
unless you change CORE_LAYER_NAME in settings.py to match unless you change CORE_LAYER_NAME in settings.py to match
whatever alternative name you use here. whatever alternative name you use here.
2.3. The layer has been added but is not yet published. (For the 1.3. The layer has been added but is not yet published. (For the
public index this provides some protection against spam and public index this provides some protection against spam and
malformed entries.) To publish it, click on the orange number malformed entries.) To publish it, click on the orange number
next to your login name at the top right, click on the next to your login name at the top right, click on the
@ -181,35 +155,32 @@ setup within it:
Usage Updating OpenEmbedded data
----- --------------------------
On a regular basis you need to run the update script: You will likely want to update the OpenEmbedded layer information on a regular
basis. To do that:
path/to/layerindex/update.py Incremental update:
docker-compose run --rm layersapp /opt/layerindex/layerindex/update.py
Reload all data (should only be needed if the database structure has
changed as part of an upgrade of the application code):
docker-compose run --rm layersapp /opt/layerindex/layerindex/update.py -r
This will fetch all of the layer repositories, analyse their contents
and update the database with the results. Run the script with --help for
further information on available options.
Upgrading from an earlier version Upgrading from an earlier version
--------------------------------- ---------------------------------
This application uses Django's migrations functionality to allow changes To upgrade the Docker-based setup from a previous release, simply run:
in the database structure whilst preserving existing data. When upgrading
with an existing database, you just need to invoke the migration command.
If you're migrating from a version prior to the Django 1.8 upgrade, you ./dockersetup.py -u
need to use the following command so that the initial migration (which
takes care of creating the database structure and initial data) is
skipped:
python3 manage.py migrate --fake-initial This will update the code and run any required database migrations.
Otherwise, you can just run the plain migration command:
python3 manage.py migrate
Support for OE-Classic Support for OE-Classic
@ -260,6 +231,7 @@ doesn't have the same name, so the update_classic_status.py script
wouldn't be able to pick it up. wouldn't be able to pick it up.
Setting up other distro comparisons Setting up other distro comparisons
----------------------------------- -----------------------------------
@ -335,6 +307,90 @@ If you have the rrs application enabled the link will also be shown in the
Security Considerations
-----------------------
Some things to be aware of from a security perspective:
* By default, anyone can register themselves an account. If you wish to
disable new user registration and manage users manually through the admin
interface instead, then add the following line to docker/settings.py:
REGISTRATION_OPEN = False
Then, assuming you have already run dockersetup.py to install the
application, run the following command to update it:
./dockersetup.py -u
* By default, dockersetup.py enables connection to the web server via
HTTPS using a self-signed certificate; connections via HTTP are
re-directed to HTTPS. However, the self-signed certificate is only
intended to provide a minimum level of security, but will result in
browser warnings and is not recommended for production - instead,
obtain and use your own certificate/key pair corresponding to the
domain which will be used to access the application in production,
or alternatively if the application is accessible to the internet you
can use Let's Encrypt.
* If you provide your own certificates for HTTPS, you should probably
also enable HSTS in your configuration. Refer to the Django Security
guide for details on how to do that:
https://docs.djangoproject.com/en/1.11/topics/security/#ssl-https
* To reset a forgotten account password, you can either use the password
reset function ( /accounts/password_reset/ ) or alternatively from the
backend you can run the following command:
docker-compose exec layersapp /opt/layerindex/manage.py changepassword <username>
* The web-based password reset function will ask the user answers to
security questions they selected and answered when they created the
account. Admins can configure the selectable security questions in
the admin interface under "Security questions"; however, be cautious
about deleting or substantially changing a question if you already
have user accounts that have given answers to that question, as doing
so will invalidate the user-set answers. You can check this if you go
to delete the security question in the admin interface - any user
answers will show up as "Security question answers" listed to be
deleted along with the question.
Note: the superuser created during setup will not have answers to
security questions set, so if you think you might need to use the
password reset function later you will need to set these by logging
into the application and then going to Edit Profile on the top-right
user drop-down menu.
* Security question answers are stored using the same mechanism as
for passwords, i.e. a secure one-way hash; thus answers cannot be
retrieved from the database once set. Additionally, if a user wants
to change one of their answers via the Edit Profile function, they
will be required to re-specify all of them.
* Account lockout: this application will lock out the user in two ways:
- By IP address (using django-axes) after too many invalid login attempts.
(default 4, resets after an hour). The lockout can be removed
immediately using the following command:
docker-compose exec layersapp /opt/layerindex/manage.py axes_reset
If you wish to disable this, remove or comment out "axes" in
INSTALLED_APPS. For more information on configuring axes, see:
https://django-axes.readthedocs.io/en/latest/
- By account for too many incorrect password reset attempts. To remove
this lockout, log into the admin interface, go to Users, click on the
the user, tick the Active checkbox and then save.
* The REST API inherited from the OpenEmbedded layerindex upon which this
application is based has been disabled, since it has no access controls
on querying data (since the OE layer index operates entirely on publicly
accessible information, whereas this application may not in actual usage).
Maintenance Maintenance
----------- -----------

121
README.devel Normal file
View File

@ -0,0 +1,121 @@
Standalone Setup
----------------
NOTE: The standalone setup is most suitable for development and is not
recommended for production; for production use the Docker-based setup
covered in the main README.
In order to run this application standalone, you will need:
* Python 3.4+
* Django 1.11.x (not compatible with 2.0 or newer)
* RabbitMQ 3.7.x
* For production usage, a web server set up to host Django applications
(not needed for local-only testing/development)
* A database supported by Django (SQLite, MySQL, etc.). Django takes
care of creating the database itself, you just need to ensure that the
database server (if not using SQLite) is configured and running.
* Third-party Django/python modules as detailed by requirements.txt.
If you're using pip, you can install these by running:
pip install -r requirements.txt
* On the machine that will run the backend update script (which does not
have to be the same machine as the web server, however it does still
have to have Django installed, have the same or similar configuration
in settings.py and have access to the database used by the web
application):
* Python 2.7.6+ / Python 3.4+ to match with the version of BitBake
for the OpenEmbedded branch being parsed (for modern versions it's
Python 3.)
* Python dependencies as per requirements.txt (we still need Django
etc. here since we interact with the database through Django's ORM.)
Example virtualenv-based setup for the above:
Python's virtualenv provides an easy way to isolate the python dependencies
of applications such as the layer index. Here's an example of setting up a
virtualenv for the layer index that's particularly useful for development.
(This assumes a Debian-based distribution, adjust accordingly for other
distros).
1. Install required host distro packages (some of these are required by
pip to build the dependencies; it's also assumed you want MariaDB as
the database backend):
sudo apt-get install virtualenv libmariadb-client-lgpl-dev build-essential python3-dev libjpeg-dev libz-dev libfreetype6-dev mariadb-server rabbitmq-server
2. Work around path issues (you may not need this):
sudo ln -s /usr/bin/mariadb_config /usr/bin/mysql_config
3. Create a Python 3 virtualenv (path can be anywhere you like):
python3 -m venv /path/to/desired/venv
4. Activate the virtualenv:
. /path/to/desired/venv/bin/activate
5. Install requirements:
pip install -r requirements.txt
Configuration and running:
1. Edit settings.py to specify a database, EMAIL_HOST, SECRET_KEY and
other settings specific to your installation. Ensure you set
LAYER_FETCH_DIR to an absolute path to a location with sufficient
space for fetching layer repositories. Modify RABBIT_BROKER
and RABBIT_BACKEND to reflect the settings used by your RabbitMQ
server.
2. Run the following commands within the layerindex-web directory to
initialise the database:
./manage.py migrate
./manage.py createsuperuser
3. You can test the web application locally by setting DEBUG = True in
settings.py and running the following:
./manage.py runserver
Then visit http://127.0.0.1:8000/layerindex/ with your browser. As
with all Django applications there is an admin interface available
at http://127.0.0.1:8000/admin/ also. The initial login and password
will be those you entered in the step above when creating an admin
account.
NOTE: This local server should only be used for testing - for
production you need to use a proper web server and have DEBUG set
to False (or use the Docker-based setup).
3.1. In order to process asynchronous tasks like sending email,
you will need to run a Celery worker:
celery -A layerindex.tasks worker --loglevel=info
4. To import layer data from the public instance at layers.openembedded.org
you can run the following (defaults to the master branch only):
layerindex/tools/import_layers.py -d https://layers.openembedded.org
Alternatively you can add layers individually using the separate
layerindex/tools/import_layer.py script or through the admin interface.
Updating OpenEmbedded data
--------------------------
You will likely want to update the OpenEmbedded layer information on a regular
basis. To do that:
Incremental update:
layerindex/update.py
Reload all data (should only be needed if the database structure has
changed as part of an upgrade of the application code):
layerindex/update.py -r