Commit Graph

41 Commits

Author SHA1 Message Date
Ross Burton
e9932fca23 recipetool: use oe.license_finder
Delete the now redundant code, and import oe.license_finder instead.

(From OE-Core rev: 8bba98be5c87dd6749e5cc95e9553dffc23ada73)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2025-06-16 17:57:30 +01:00
Enguerrand de Ribaucourt
0a0caacfa5 recipetool: create_npm: reuse license utils
create_npm.py duplicated the logic for matching licenses from files and
also finding them. This patch refactors the code to reuse the license
utils. This will make the code more maintainable and also align both
behaviors. For instance, some licenses weren't matched properly because
the duplicate logic did not support the difference in format in the md5
tables for COMMON_LICENSE_DIR and licenses.csv.

This is also faster since the license files were being read twice.
The result is slightly more accurate since the utils have better
implementations, and I was able to reuse the logic for the root PN
package, as well as the base LICENSE variable.

I chose to extract generate_common_licenses_chksums into create.py
since it can be considered a general utility function to allow
other recipetool creators to refer to COMMON_LICENSE_DIR files.

I updated the wording in the code when appropriate.

v3:
 - added commit
 - this replaces the commit that added all the COMMON_LICENSE_DIR md5
   to licenses.csv

(From OE-Core rev: 7bc18bed63b94689890bcde63402d7cc1cedffa9)

Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-08-23 09:51:36 +01:00
Enguerrand de Ribaucourt
69bf37a3dd recipetool: create: split guess_license function
The npm recipetool handler redefines the license code the could be
unified. In order to do this refactoring, extract the bits we'll
need into separate functions.

guess_license() is renamed to find_licenses() and is split into
find_license_files() and match_licenses().

(From OE-Core rev: f1ec28feaea8ea6a2df894dd4ddba561c8a04ed2)

Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-08-23 09:51:36 +01:00
Enguerrand de Ribaucourt
01d17cd5d4 recipetool: create_npm: resolve licenses defined in package.json
Some npm packages do not copy the LICENSE or COPY file into their
git repository. They'll instead simply use SPDX identifiers in their
package.json. A fallback for those repositories attempted to match
the README file to a license file instead, which had a very low
probability of success.

This commit replaces this fallback with parsing the package.json and
looking for the license in COMMON_LICENSE_DIR. If the license is not
found, "Unknown" will still be produced.

This also generates "Unknown" for packages which had no README file,
which could silently not appear in the generated recipe. The user was
more likely to miss them.

Co-authored-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com>
(From OE-Core rev: 445604cfc4a5813ea635f18053cd1f673bf0b830)

Signed-off-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com>
Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-08-23 09:51:36 +01:00
BELOUARGA Mohamed
2dacac93bc recipetool: create: npm: Add support to handle peer dependencies
NPM changed its manner to handle peer dependencies over its versions.
Before NPM 3: NPM installs automatically peer dependencies
between NPM 3 and 7: NPM shows a warning about peer dependencies
After NPM 3: NPM reworked its manner how to handle peer dependencies

The shrinkwrap doesn't have the parameters of the peer dependencies, so we cannot
fetch them. in the same time peer dependencies are not direct dependencies, they should
be installed as run time dependencies.

(From OE-Core rev: f36021a749974ef3d4a6abe4d5429544a815071a)

Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-06-29 11:10:39 +01:00
BELOUARGA Mohamed
dbc1da2fb8 recipetool: create: npm: Add support for the new format of the shrinkwrap file
The shrinkwrap file changed its format, but npm does not version this file. So we can use it properly.
The actual changes make the script check if the npm package has dependencies in the actual shrinkwrap format.

(From OE-Core rev: 488d17c2af0c927ec66f0eee124bf6fc5b7f7c95)

Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-06-29 11:10:39 +01:00
BELOUARGA Mohamed
2a3888069f recipetool: create: npm: Remove duplicate function to not have future conflicts
Npm packages do not have yocto friendly names. fore instance we can have names like
"@example/npmPackage"

npm fetcher has a function that convert these names to yocto friendly names.
But in recipe tool we have an other function (duplicate).

(From OE-Core rev: 18e5438de5389b58c8b6a548d4474128d510a28d)

Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-06-29 11:10:39 +01:00
Stefan Herbrechtsmeier
36965547e3 recipetool: npm: Use README as license fallback
Use the README as license fallback if a license file is missing. Use the
linenumbers parameter of get_license_md5sums function to determine the
license text inside the README.

(From OE-Core rev: eff85c86f36673a1cb5a5dc8c66598e0dc457374)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-14 11:48:45 +01:00
Stefan Herbrechtsmeier
4ca5b3fcb0 recipetool: npm: Do not add package.json files to LIC_FILES_CHKSUM
The package.json files doesn't contain any licenses. The name of the
license doesn't comply the license requirements of most liceneses.

(From OE-Core rev: 194df9c240378b6befeb10183889093ec7bb4d5f)

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-10-14 11:48:45 +01:00
Kamel Bouhara
93543e9960 recipetool: create: only add npmsw url if required
Before adding a npmsw fetcher to a recipe we
should first check if the generated shrinkwrap file
contains dependencies.

(From OE-Core rev: ef153ad36d0299e83a03af8f207686d0d8a238b3)

Signed-off-by: Kamel Bouhara <kamel.bouhara@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-16 22:39:36 +00:00
Jean-Marie LEMETAYER
cfa5544005 recipetool/create_npm: handle the licenses of the dependencies
As usual the 'LICENSE' and the 'LIC_FILES_CHKSUM' values reflects all
the license files discovered in the source tree (including the
dependencies).

For npm recipes the 'LIC_FILES_CHKSUM' value contains also the status of
the 'package.json' file of every packages as it contains license
informations.

Finally each package has a separate 'LICENSE_${PN}-package-name' value
which describes its license.

(From OE-Core rev: 9a70d4996c84b277f423eda5aac4acbe344599f4)

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27 16:48:09 +00:00
Jean-Marie LEMETAYER
6fd9cebc98 recipetool/create_npm: refactor the npm recipe creation handler
This commit refactors the npm recipe creation handler to use the new npm
behavior. The process is kept as simple as possible and only generates
the shrinkwrap file.

To avoid naming issues the recipe name is now extracted from the npm
package name and not directly mapped.

(From OE-Core rev: 1deccb0f0c204cd02fb8606f180d8a13df9f31db)

Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-01-27 16:48:09 +00:00
Richard Purdie
ffae400179 meta/lib+scripts: Convert to SPDX license headers
This adds SPDX license headers in place of the wide assortment of things
currently in our script headers. We default to GPL-2.0-only except for the
oeqa code where it was clearly submitted and marked as MIT on the most part
or some scripts which had the "or later" GPL versioning.

The patch also drops other obsolete bits of file headers where they were
encoountered such as editor modelines, obsolete maintainer information or
the phrase "All rights reserved" which is now obsolete and not required in
copyright headers (in this case its actually confusing for licensing as all
rights were not reserved).

More work is needed for OE-Core but this takes care of the bulk of the scripts
and meta/lib directories.

The top level LICENSE files are tweaked to match the new structure and the
SPDX naming.

(From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-09 16:31:55 +01:00
Lei Yang
305f569cf2 recipetool: add missed module
When I use recipetool to add a new recipe, it says:

[snip]
sys.exit(14)
NameError: name 'sys' is not defined
[snip]

(From OE-Core rev: 051e1c27bbf6a5898f2cf13b78dd3d8d1ec31586)

Signed-off-by: Lei Yang <Lei.Yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-12 09:29:06 +01:00
Chen Qi
869e501544 logging: use warning instead warn
The warn method is deprecated. We should use the documented warning instead.

Quoting from the python's official doc:
"""
Note: There is an obsolete method warn which is functionally identical to warning.
      As warn is deprecated, please do not use it - use warning instead.
"""

(From OE-Core rev: f467fd277eb77336097cfc0f5f329bdc8d0f70cb)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-26 13:16:40 +01:00
Sarah Marsh
d845b9960b recipetool: fixed uncaught NameError exception
When packaging a node application, a `NameError` can be thrown in create_npm.py if an optional npm dependency does not
support Linux.

(From OE-Core rev: 8293201d98d368d6322eaa960fb3e7cee2ba9368)

Signed-off-by: Sarah Marsh <sarah.marsh@arm.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-04 13:28:01 +01:00
Paul Eggleton
941902d6bb recipetool: create: drop debug print
This looks like some debug printing that was left in by accident.

(From OE-Core rev: b0bfa1b1f4377270af9e7f19949cc1781a4e3b9d)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-11-11 12:14:27 +00:00
Paul Eggleton
58057d8749 recipetool: create: fix npm license code regression
OE-Core commit 1df60b09f7a60427795ec828c9c7180e4e52f98c caused a
regression in npm handling since it still expected to be able to get the
results of the license handling, but this no longer happens until after
the npm plugin is called. Thus, call the license handling function
ourselves here (which will record this as having been handled so it
doesn't get done again later).

(From OE-Core rev: 3e408aadaea85b6f192b34d37d508cbaf3cd7164)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23 13:48:50 +01:00
Paul Eggleton
433aa81f92 recipetool: create: fix broken import in npm module
With "import oe" in create_npm.py you get "AttributeError: module 'oe'
has no attribute 'package'" when it tries to call
oe.package.npm_split_package_dirs().

(From OE-Core rev: 1261900aeac725e5712e0180600753a9d4c67e60)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23 13:48:50 +01:00
Paul Eggleton
b1f237ebd0 recipetool: allow plugins to set LICENSE and LIC_FILES_CHKSUM
We were being a bit prescriptive in setting LICENSE and
LIC_FILES_CHKSUM. We can't always trust what's in the metadata
accompanying some source which plugins will almost always be pulling
from, however we do want to allow plugins to set the LICENSE and
LIC_FILES_CHKSUM values. Merge what we find in our license file scan
with what the plugin sends back.

Additionally, plugins can now add a "license" item to the handled list
in order to inhibit the normal LICENSE / LIC_FILES_CHKSUM handling if
they have already taken care of it completely.

Thanks to Mark Horn <mark.d.horn@intel.com> for prompting, testing and
fixing this patch.

(From OE-Core rev: 1df60b09f7a60427795ec828c9c7180e4e52f98c)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23 08:47:02 +01:00
Paul Eggleton
0d72748e81 recipetool: create: refactor code for ensuring npm is available
Across devtool and recipetool we had an ugly set of code for ensuring
that we can call an npm binary, and much of that ugliness was a result
of not being able to run build tasks when tinfoil was active - if
recipetool found that npm was required and we didn't know beforehand
(e.g. we're fetching from a plain git repository as opposed to an npm://
URL where it's obvious) then it had to exit and return a special result
code, so that devtool knew it needed to build nodejs-native and then
call recipetool again. Now that we are using real build tasks to fetch
and unpack, we can drop most of this and move the code to the one place
where it's still needed (i.e. create_npm where we potentially have to
deal with node.js code in a plain source repository).

(From OE-Core rev: 8450de16ddb02d863204b411a94c6d84e0f88817)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Paul Eggleton
e4346e8be5 recipetool: create: reimplement fetching with normal fetch/unpack tasks
Now that we have the ability to run the tasks in a more standard context
through tinfoil, change recipetool's fetching code to use that to fetch
files using it. This has the major advantage that any dependencies of
do_fetch and do_unpack (e.g. for subversion or npm) will be handled
automatically. This also has the beneficial side-effect of fixing a
recent regression that prevented this fetch operation from working with
memory resident bitbake.

Also fix devtool's usage of fetch_uri() at the same time so that we can
completely replace it.

Fixes [YOCTO #11710].

(From OE-Core rev: 9a47a6690052ef943c0d4760630ee630fb012153)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-21 08:44:25 +01:00
Paul Eggleton
40d1771944 recipetool: create: hide missing npm error when called from devtool
If devtool is called with a URL to a source repository containing a
node.js module, we don't know that until recipetool has fetched it, and
due to the structure of the code we have to exit with a special code in
order to let devtool know it needs to build nodejs-native. We also want
to suppress the error message that recipetool would normally print under
these circumstances; there is already a mechanism for this but it wasn't
operative in the case where we're pointed to a source repository rather
than an npm:// URL, so create some plumbing so that we know to hide the
message.

(From OE-Core rev: 0c2d0fbb1c6c5b82183799eb7ef80074f86bcfc4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:10 +01:00
Paul Eggleton
bb8f141d0c devtool: add: fix node.js/npm handling with recipe specific sysroots
The change over to recipe specific sysroots means that we can no longer
get a known location simply from configuration for the npm binary - we
need to get the recipe sysroot for nodejs-native, look there for npm if
we need to check it's present, and add that to PATH when calling out to
npm. Unfortunately this means anywhere we need to get that path we have
to have parsed all recipes, otherwise we have no reliable way of
resolving nodejs-native. Thus we have to change recipetool create to
always parse all recipes (the structure of the code does not allow us to
do this conditionally).

In the worst case, if npm hasn't already been added to its own sysroot
and we are fetching from a source repository rather than an npm
registry, this gets a bit ugly because we end up parsing recipes three
times:
1) recipetool startup, which then fetches the code and determines it's
   a node.js module, finds that npm isn't available and then exits with
   a specific error to tell devtool it needs to build npm
2) when we invoke bitbake -c addto_recipe_sysroot nodejs-native
3) when we re-invoke recipetool

This code is badly in need of refactoring, but now is unfortunately not
the time to do that, so we're going to have to live with this ugliness
for now.

Fixes [YOCTO #10992].

(From OE-Core rev: acfdbd796c99882b8586023c8c6b848716105c8d)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:10 +01:00
Paul Eggleton
0440cbccda recipetool: create: fix for regression in npm license handling
OE-Core commit c0cfd9b1d54b05ad048f444d6fe248aa0500159e added handling
for AND / OR in license strings coming from npm, but made the assumption
that an & would always be present in the license value. Check if it's
there first so we don't fail if it isn't.

(From OE-Core rev: abe2955df2dc558de6068d9373dfcb47d690704b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:09 +01:00
Anders Darander
6fa0fda116 devtool/recipetill: npm install of devDependencies
Web applications built using e.g. angular2, usually requires that the
packages in devDependencies are available.

Thus, add an option '--fetch-dev' to both devtool add and recipetool, to
add npm packages in devDependencies to DEPENDS.

(From OE-Core rev: f246f820d53b459596fde6758a09f7a0d7db7c4c)

Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-22 11:35:20 +00:00
Anders Darander
f573db010f scripts/lib/create_npm: handle Public Domain licenses
Rewrite Public Domain as PD, as that's what the place holder in
meta/files/common_licenses is called.

(From OE-Core rev: d7f0af5aa90a9ef7714c842fb4cb762017820768)

Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-04 23:18:18 +00:00
Anders Darander
1da740ef32 scripts/lib/create_npm: handle AND and OR in licenses
Handle npm packages with multiple licenses (AND and OR).
Prior to this, AND and OR were treated as licensed in their
own.

(From OE-Core rev: c0cfd9b1d54b05ad048f444d6fe248aa0500159e)

Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-04 23:18:18 +00:00
Anders Darander
837d89b47b scripts/lib/create_npm: rewrite see license in eula
Rewrite the 'SEE LICENSE IN EULA' to a single string (without
spaces), to avoid splitting the string later on.

(Otherwise, each word gets split, and assumed to be a license
on it's own.

(From OE-Core rev: 39127702cee80c972ee9a447ef4006751f47475e)

Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-04 23:18:18 +00:00
Anders Darander
ca83f35f5a create_npm.py: convert MIT/X11 to MIT
Quite a few npm packages declare MIT/X11 as their license. This is equal to
a pure MIT license.

(From OE-Core rev: 8df5e731a10cc9ade1266e9daaa26ec7c855c062)

Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-04 23:18:18 +00:00
Paul Eggleton
6ac9c605e6 recipetool: create: properly handle npm optional dependencies
npm's package.json supports two types of dependencies -
optionalDependencies and dependencies; in the code for creating a recipe
from a non-npm source (e.g. a git repository) we were not handling
optionalDependencies and thus when pointed at a node.js application
outside of npm we weren't taking care of all dependencies.

(From OE-Core rev: 2b66cb9982d10ce1744d430858eaef3e5a72c8c0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-07 14:50:09 +00:00
Joshua Lock
c0f2890c01 scripts: remove True option to getVar calls
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

(From OE-Core rev: 0a36bd96e6b29fd99a296efc358ca3e9fb5af735)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16 10:23:23 +00:00
Paul Eggleton
18caacae06 devtool: add: build nodejs-native if npm is needed and not available
If the user runs devtool add on an npm:// URL (or source tree that uses
node.js), and npm is not available, just build nodejs-native instead of
telling the user they need to do it; if that fails because there isn't
any such recipe (which would be the default, since it's not in OE-Core)
then produce a slightly more readable error message hinting at what the
user needs to do.

Note that this forces the use of nodejs-native rather than npm on the
host - this makes sense for two reasons: (1) we need it to be compatible
with nodejs for the target, and (2) we have to have a recipe for that
anyway, so allowing you to avoid having a recipe for the native version
isn't really beneficial.

There's a bit of a hack in here in order to allow this - for node.js
sources that aren't fetched via npm we don't know that they are that
until we've fetched and unpacked them, by which time we're inside
recipetool and have an active tinfoil instance that will prevent bitbake
being run. To avoid this being an issue, we allow recipetool to get to
the point where we know we need npm and then exit with a specific exit
code, at which point devtool can try to build it and then if that
succeeds, it will re-execute recipetool. This is definitely not ideal,
but it can't really be refactored and done properly until we do the
tinfoil2 refactoring; in the mean time though we still want to be
helpful to the user.

Fixes [YOCTO #10337].

(From OE-Core rev: f40662bde5aab158c4e4c3c3ff5e68665a4194a5)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-05 10:10:11 +01:00
Paul Eggleton
ff259b095d recipetool: create: support node.js code outside of npm
If you have your own node.js application you may not publish it (or at
least not immediately) in an npm registry - it might just be in a
repository on github or on your local machine. Add support to recipetool
create for creating recipes to build such applications - extract their
dependencies, fetch them, and add corresponding npm:// URLs to SRC_URI,
and ensure that LICENSE / LIC_FILES_CHKSUM are updated to match. For
example, you can now run:

  recipetool create https://github.com/diversario/node-ssdp

(I had to borrow some code from bitbake/lib/bb/fetch2/npm.py to
implement this functionality; this should be refactored out but now
isn't the time to do that refactoring.)

Part of the fix for [YOCTO #9537].

(From OE-Core rev: 4fb8b399c05a1b66986fc76e13525f6c5e0d9b58)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-08 00:36:49 +01:00
Paul Eggleton
75f1a0ed42 recipetool: create: avoid decoding errors with Python 3
We're opening source files with the default encoding (utf-8) but we
can't necessarily be sure that they are UTF-8 clean - for example,
recipetool create ftp://mama.indstate.edu/linux/tree/tree-1.7.0.tgz
prior to this patch resulted in a UnicodeDecodeError. Use the
"surrogateescape" mode to avoid this.

Fixes [YOCTO #9822].

(From OE-Core rev: 50fcd9d1b9a20d49bc873467a82a071f2f2f8b5a)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:24 +01:00
Ed Bartosh
7eab022d4b scripts: Fix deprecated dict methods for python3
Replaced iteritems -> items, itervalues -> values,
iterkeys -> keys or 'in'

(From OE-Core rev: 25d4d8274bac696a484f83d7f3ada778cf95f4d0)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:01 +01:00
Paul Eggleton
2b510f5e01 recipetool: create: support extracting SUMMARY and HOMEPAGE
Allow plugins to set any variable value through the extravalues dict,
and use this to support extracting SUMMARY and HOMEPAGE values from spec
files included with the source; additionally translate "License:" to a
comment next to the LICENSE field (we have our own logic for setting
LICENSE, but it will often be useful to see what the spec file says if
one is present).

Also use the same mechanism for setting the same variables for node.js
modules; this was already supported but wasn't inserting the settings in
the appropriate place in the file which this will now do.

(From OE-Core rev: 91fc35ff5e89aa6d4c4ad945e45406fb4f71018e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-01 12:38:41 +01:00
Paul Eggleton
8f275ff0d8 recipetool: create: fix falling back to declared license for npm packages
Fix two problems falling back to the "license" field from package.json
when no license file is present:
1) The function that was supposed to return the license field value was
   always explicitly returning None, and this was never noticed (because
   the test cases never exercised the fallback as they provided license
   files for each module).
2) Fix the main package not falling back because it had a default of an
   empty list, which evaluates to '' instead of 'Unknown'.

(From OE-Core rev: 59381a9450949ce6b4b03adb717e950b999830f3)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:31:14 +01:00
Paul Eggleton
bc0e99d2b1 recipetool: create: shrinkwrap and lockdown npm modules
"npm shrinkwrap" creates a file that ensures that the exact same
versions get fetched the next time the recipe is built. lockdown is
similar but also includes sha1sums of the modules thus validating they
haven't changed between builds. These ensure that the build is
reproducible.

Fixes [YOCTO #9225].

(From OE-Core rev: 277377f13b2b771915eb853e336ca24b84523ed1)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton
91455005b6 recipetool: create: split npm module dependencies into packages
Rather than rolling all of an npm module's dependencies into the same
package, split them into one module per package, setting the SUMMARY and
PKGV values from the package.json file for each package. Additionally,
mark each package with the appropriate license using the license
scanning we already do, falling back to the license stated in the
package.json file for the module if unknown. All of this is mostly in
aid of ensuring all modules and their licenses now show up in the
manifests for the image.

Additionally we set the main LICENSE value more concretely once we've
calculated the per-package licenses, since we have more information at
that point.

(From OE-Core rev: 8226805f83d21e7c1d2ba21969f3e8ee4b137496)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09 17:00:29 +00:00
Paul Eggleton
5c5c13d777 recipetool: create: add basic support for new npm fetcher/class
Add detection for npm modules and support for extracting the name and
version from package.json as is usually part of an npm module contents.

Note: this will likely only produce a buildable recipe if you use an
npm:// URL; simply pointing to a node.js source repository isn't going
to fetch the module's dependencies. It also doesn't set up the
shrinkwrap/lockdown automatically, so there is some room for improvement
later.

Implements [YOCTO #8690].

(From OE-Core rev: 41d0e4d75f13b53a6c1b6a8df9be4742be7534e0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02 23:08:51 +00:00