This is a followup to a537620054.
Since that commit, 'pypi-package?' would return false for most Python
packages, and thus "guix refresh python-xxx" would report that no
updaters apply to the package.
* guix/import/pypi.scm (pypi-package?)[pypi-url?]: Recognize
"files.pythonhosted.org" URLs.
Guile-JSON 3.x is incompatible with Guile-JSON 1.x, which we relied on
until now: it maps JSON dictionaries to alists (instead of hash tables),
and JSON arrays to vectors (instead of lists). This commit is about
adjusting all the existing code to this new mapping.
* m4/guix.m4 (GUIX_CHECK_GUILE_JSON): New macro.
* configure.ac: Use it.
* doc/guix.texi (Requirements): Mention the Guile-JSON version.
* guix/git-download.scm (git-fetch)[guile-json]: Use GUILE-JSON-3.
* guix/import/cpan.scm (string->license): Expect vectors instead of
lists.
(module->dist-name): Use 'json-fetch' instead of 'json-fetch-alist'.
(cpan-fetch): Likewise.
* guix/import/crate.scm (crate-fetch): Likewise, and call 'vector->list'
for DEPS.
* guix/import/gem.scm (rubygems-fetch): Likewise.
* guix/import/json.scm (json-fetch-alist): Remove.
* guix/import/pypi.scm (pypi-fetch): Use 'json-fetch' instead of
'json-fetch-alist'.
(latest-source-release, latest-wheel-release): Call 'vector->list' on
RELEASES.
* guix/import/stackage.scm (stackage-lts-info-fetch): Use 'json-fetch'
instead of 'json-fetch-alist'.
(lts-package-version): Use 'vector->list'.
* guix/import/utils.scm (hash-table->alist): Remove.
(alist->package): Pass 'vector->list' on the inputs fields, and default
to the empty vector.
* guix/scripts/import/json.scm (guix-import-json): Remove call to
'hash-table->alist'.
* guix/swh.scm (define-json-reader): Expect pair? or null? instead of
hash-table?.
[extract-field]: Use 'assoc-ref' instead of 'hash-ref'.
(json->branches): Use 'map' instead of 'hash-map->list'.
(json->checksums): Likewise.
(json->directory-entries, origin-visits): Call 'vector->list' on the
result of 'json->scm'.
* tests/import-utils.scm ("alist->package with dependencies"): New test.
* gnu/installer.scm (build-compiled-file)[builder]: Use GUILE-JSON-3.
* gnu/installer.scm (installer-program)[installer-builder]: Likewise.
* gnu/installer/locale.scm (iso639->iso639-languages): Use 'assoc-ref'
instead of 'hash-ref', and pass vectors through 'vector->list'.
(iso3166->iso3166-territories): Likewise.
* gnu/system/vm.scm (system-docker-image)[build]: Use GUILE-JSON-3.
* guix/docker.scm (manifest, config): Adjust for Guile-JSON 3.
* guix/scripts/pack.scm (docker-image)[build]: Use GUILE-JSON-3.
* guix/import/github.scm (fetch-releases-or-tags): Update docstring.
(latest-released-version): Use 'assoc-ref' instead of 'hash-ref'. Pass
the result of 'fetch-releases-or-tags' to 'vector->list'.
* guix/import/launchpad.scm (latest-released-version): Likewise.
Fixes <https://bugs.gnu.org/33046>.
* guix/build-system/python.scm (pypi-uri): Update the host URI to
"files.pythonhosted.org".
* guix/import/pypi.scm (make-pypi-sexp): Preserve the package name case when
the source URL calls for it.
* guix/import/pypi.scm (maybe-inputs): Add INPUT-TYPE argument, and use it.
(test-section?): New predicate.
(parse-requires.txt): Collect the optional test inputs, and return them as the
second element of the returned list.
(parse-wheel-metadata): Likewise.
(guess-requirements): Adapt.
(make-pypi-sexp): Likewise, and include the test inputs requirements as native
inputs in the returned package expression.
* tests/pypi.scm (test-requires.txt): Include a test section in the
test-requires.txt data.
(test-requires.txt-beaker): New variable.
("parse-requires.txt"): Adapt.
("parse-requires.txt - Beaker"): New test.
("parse-wheel-metadata, with extras"): Adapt.
("parse-wheel-metadata, with extras - Jedi"): Adapt.
("pypi->guix-package, no wheel"): Re-indent, and add the expected
native-inputs.
("pypi->guix-package, wheels"): Likewise.
With newer Wheel releases, there is no more metadata.json file; the METADATA
file should be used instead (see: https://github.com/pypa/wheel/issues/195).
This change updates our PyPI importer so that it uses the latter.
* guix/import/pypi.scm (define-module): Remove unnecessary modules and export
the PARSE-WHEEL-METADATA procedure.
(parse-wheel-metadata): Add procedure.
(guess-requirements): Use it.
* tests/pypi.scm (test-metadata): Test it.
This change enables the PyPI importer to look for requirements in a source
archive of a different type than "tar.gz" or "tar.bz2". Also, scan the source
archive to find a requires.txt file.
* guix/import/pypi.scm: (guess-requirements)[tarball-directory]: Remove procedure.
[guess-requirements-from-source]: Use COMRESSED-FILE? to determine if an
archive type is supported, and some file extension logic that chooses either
"tar" or "unzip" as the extractor. Search for the requires.txt file in the
archive instead of using a static, expected location.
(guess-requirements): Rename the TARBALL argument to ARCHIVE, to denote the
archive format is no longer bound specifically to the Tar format.
(compute-inputs): Likewise.
* tests/pypi.scm ("pypi->guix-package, no wheel"): Mock the requires.txt at a
non-standard location.
("pypi->guix-package, no usable requirement file."): New test.
The previous solution was fragile and could leave unwanted characters in a
requirement name, such as '[' or ']'.
Partially fixes <https://bugs.gnu.org/33047>.
* guix/import/pypi.scm (use-modules): Export SPECIFICATION->REQUIREMENT-NAME
(%requirement-name-regexp): New variable.
(clean-requirement): Rename to...
(specification->requirement-name): this, which now uses
%requirement-name-regexp to select the requirement name from the requirement
specification.
(parse-requires.txt): Adapt.
* guix/import/pypi.scm: Export PARSE-REQUIRES.TXT.
(clean-requirement): Move procedure to the top level.
(guess-requirements): Move the READ-REQUIREMENTS procedure to the top level,
and rename it to PARSE-REQUIRES.TXT. Move the CLEAN-REQUIREMENT procedure to
the top level. Move the COMMENT? functions inside the PARSE-REQUIRES.TXT
procedure.
(parse-requires.txt): Add a SECTION-HEADER? predicate, and use it to prevent
parsing optional requirements.
* tests/pypi.scm (test-requires-with-sections): New variable.
("parse-requires.txt, with sections"): New test.
PyPI packages are mandated to have a setup.py file, which contains a listing
of the required dependencies. The setuptools/distutils machinery embed
metadata in the archives they produce, which contains this information. There
is no need nor gain to collect the requirements from a "requirements.txt"
file, as it is not the true record of dependencies for PyPI packages and may
contain extraneous requirements or not exist at all.
* guix/import/pypi.scm (guess-requirements): Update comment.
[guess-requirements-from-source]: Do not attempt to parse the file
requirements.txt. Streamline logic.
* tests/pypi.scm (test-requires.txt): Rename from test-requirements, to hint
at the file being tested.
("pypi->guix-package"): Adapt so that the fake package contains a requires.txt
file rather than a requirements.txt file.
("pypi->guix-package, wheels"): Likewise.
* guix/import/pypi.scm (guess-requirements): Extract "requires.txt" from the
egg-info directory in addition to "requirements.txt"; strip off version
constraints; use call-with-temporary-directory.
Fixes <https://bugs.gnu.org/25635>.
Reported by Sergei Trofimovich <slyfox@inbox.ru>.
* guix/import/pypi.scm (guix-package->pypi-name)[url->pypi-name]: New
procedure.
Rewrite body to match lists in addition to strings.
* tests/pypi.scm ("guix-package->pypi-name, several URLs"): New test.
* guix/import/json.scm (json-fetch): Use http-fetch instead of url-fetch
to avoid writing to stdout and a temporary file for each invocation.
* guix/import/gem.scm (rubygems-fetch): Do not redirect json-fetch
output to /dev/null.
* guix/import/pypi.scm (pypi-fetch): Likewise.
Fixes <http://bugs.gnu.org/23997>.
* guix/import/pypi.scm (guix-package->pypi-name): Rewrite using
'basename' and 'hyphen-package-name->name+version'.
* tests/pypi.scm ("guix-package->pypi-name, old URL style")
("guix-package->pypi-name, new URL style"): New tests.
* doc/guix.tex (Invoking guix import): Mention that the pypi importer
works better with "unzip".
* guix/import/pypi.scm (latest-wheel-release,
wheel-url->extracted-directory): New procedures.
* tests/pypi.scm (("pypi->guix-package, wheels"): New test.
Fixes <http://bugs.gnu.org/23062>.
Reported by Danny Milosavljevic <dannym@scratchpost.org>.
* guix/import/pypi.scm (make-pypi-sexp): Check whether 'pypi-uri'
returns SOURCE-URL and fall back to the full URL otherwise.
* tests/pypi.scm ("pypi->guix-package"): Adjust expected URI
accordingly.
Co-authored-by: Danny Milosavljevic <dannym@scratchpost.org>
Reported by Andreas Enge <andreas@enge.fr>
at <https://lists.gnu.org/archive/html/guix-devel/2015-11/msg00764.html>.
* guix/import/pypi.scm (&missing-source-error): New error type.
(latest-source-release): Raise it instead of using 'error'.
(pypi->guix-package): Guard against it and use 'leave' to report the
error.
(latest-release): Guard against it and return #f when caught.
* guix/import/pypi.scm (make-pypi-sexp): Factorize license to symbol
conversion code.
(string->license, snake-case, guix-hash-url): Move from here...
* guix/import/utils.scm: ... to here.
(license->symbol): New procedure.
* guix/import/pypi.scm (python->package-name, maybe-inputs, compute-inputs,
guess-requirements): New procedures.
* guix/import/pypi.scm (guix-hash-url): Now takes a filename instead of an
URL as input.
* guix/import/pypi.scm (make-pypi-sexp): Now tries to generate the inputs
automagically.
* tests/pypi.scm: Update the test.
* guix/snix.scm: Delete.
* guix/import/snix.scm: New file.
* guix/import/pypi.scm: New file.
* guix/import/utils.scm: New file.
* guix/scripts/import/nix.scm: New file.
* guix/scripts/import/pypi.scm: New file.
* tests/pypi.scm: New file.
* tests/snix.scm: Import (guix import snix) module.
* guix/scripts/import.scm (%default-options, %options): Delete.
(%standard-import-options, importers): New variables.
(show-help): List importers.
(guix-import): Factor out Nix-specific logic. Delegate to correct importer
based upon first argument.
* configure.ac (HAVE_GUILE_JSON): New conditional.
* Makefile.am (MODULES): Add new files and remove 'guix/snix.scm'.
(SCM_TESTS): Add 'tests/pypi.scm' if guile-json is installed.
Ludovic Courtès