Commit Graph

86 Commits

Author SHA1 Message Date
宋文武 9bc8175cfa
download: Support 'https_proxy'.
* guix/build/download.scm (setup-http-tunnel): New procedure.
(open-connection-for-uri): Honor the 'https_proxy' environment variable.
2019-05-13 21:36:03 +08:00
Ludovic Courtès e4ee842026
download: Ask not to use TLS 1.3.
Works around <https://bugs.gnu.org/34102>.
Reported by Marius Bakke <mbakke@fastmail.com>.

* guix/build/download.scm (tls-wrap): Add "-VERS-TLS1.3" to the priority
string when (gnutls-version) is not prefixed by "3.5".
2019-01-26 23:18:14 +01:00
Ludovic Courtès 76832d3420
Remove most uses of the _IO*F constants.
These constants, for use with 'setvbuf', were deprecated in Guile 2.2
and disappeared in Guile 3.0.  Here we keep these constants in
build-side code where removing them is not feasible.

* guix/build/download-nar.scm (download-nar): Adjust 'setvbuf' calls to
the Guile 2.2+ API.
* guix/build/download.scm (open-socket-for-uri): Likewise.
(open-connection-for-uri, url-fetch): Likewise.
* guix/build/make-bootstrap.scm (make-stripped-libc): Likewise.
* guix/build/union.scm (setvbuf) [guile-2.0]: New conditional wrapper.
(union-build): Adjust to new API.
* guix/ftp-client.scm (ftp-open, ftp-list, ftp-retr): Likewise.
* guix/http-client.scm (http-fetch): Likewise.
* guix/inferior.scm (proxy): Likewise.
* guix/scripts/substitute.scm (fetch, http-multiple-get): Likewise.
* guix/self.scm (compiled-modules): Likewise.
* guix/ssh.scm (remote-daemon-channel, store-import-channel)
(store-export-channel): Likewise.
* guix/ui.scm (initialize-guix): Likewise.
* tests/publish.scm (http-get-port): Likewise.
* guix/store.scm (%newlines): Adjust comment.
2019-01-09 14:47:53 +01:00
Ludovic Courtès c3d9bca48a
download: Remove Guile 2.0 workaround.
* guix/build/download.scm: Remove Guile 2.0 workaround.
2019-01-09 14:47:53 +01:00
Ludovic Courtès a65177a657
maint: Remove 'cond-expand' forms for Guile 2.0.
Note: Leave 'cond-expand' forms used in the build-side modules that can
run on %BOOTSTRAP-GUILE, which is currently Guile 2.0.

* guix/build/compile.scm: Move 'use-modules' clause from 'cond-expand'
to 'define-module' form.
(%default-optimizations): Remove 'cond-expand'.
* guix/build/download.scm (tls-wrap): Remove 'cond-expand'.
* guix/build/syscalls.scm: Remove 'cond-expand' form around
'%set-automatic-finalization-enabled?!' and
'without-automatic-finalization'.
* guix/inferior.scm (port->inferior): Remove 'cond-expand'.
* guix/scripts/pack.scm (wrapped-package)[build]: Remove 'cond-expand'.
* guix/status.scm (build-event-output-port): Remove 'cond-expand'.
* guix/store.scm (open-inet-socket): Remove 'cond-expand'.
* guix/ui.scm (install-locale): Remove 'cond-expand'.
* tests/status.scm ("current-build-output-port, UTF-8 + garbage"):
Remove 'cond-expand'.
* tests/store.scm ("current-build-output-port, UTF-8 + garbage"):
Remove 'cond-expand'.
2019-01-09 14:47:53 +01:00
Ludovic Courtès 240a9c69a6
perform-download: Optionally report a "download-progress" trace.
* guix/scripts/perform-download.scm (perform-download): Add
 #:print-build-trace? and pass it to 'url-fetch'.
(guix-perform-download): Define 'print-build-trace?' and pass it to
'perform-download'.
* guix/build/download.scm (ftp-fetch): Add #:print-build-trace? and
honor it.
(url-fetch): Likewise.
* nix/libstore/builtins.cc (builtinDownload): Set _NIX_OPTIONS
environment variable.
2018-09-27 23:21:53 +02:00
Ludovic Courtès 1d84d7bf60
build: Require Guile >= 2.0.13.
* README, configure.ac, doc/guix.texi (Requirements): Increase minimum
Guile version from 2.0.9 to 2.0.13.
* config-daemon.ac: Remove use of 'GUIX_CHECK_UNBUFFERED_CBIP'.
* m4/guix.m4 (GUIX_CHECK_UNBUFFERED_CBIP): Remove.
* guix/build/download.scm (current-http-proxy): Remove.
* guix/build/syscalls.scm (%libc-errno-pointer, errno): Remove.
(syscall->procedure): Use #:return-errno unconditionally.
* guix/hash.scm (open-sha256-input-port)[unbuffered]: Remove outdated
comment.
* guix/http-client.scm (when-guile<=2.0.5-or-otherwise-broken): Remove.
<top level>: Remove 'when-guile<=2.0.5-or-otherwise-broken' block.
* guix/scripts/substitute.scm (fetch): Remove 'guile-version>?'
conditional.
* tests/hash.scm (supports-unbuffered-cbip?): Remove.
<top level>: Remove 'test-skip' call.
2018-02-26 18:19:34 +01:00
Maxim Cournoyer 8fbc1a2208
download: Fix return value of the url-fetch procedure.
Fixes <https://bugs.gnu.org/30270>.
Regression was introduced by commit 347fa4aebf.

* guix/build/download.scm (url-fetch): Return `file' instead of #t upon success.

Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2018-01-28 23:14:55 +01:00
Eric Bavier 7f04197fef
utils: Fix cond-expand for Guile 2.0.
* guix/build/download.scm (tls-wrap): Use 'guile-2.2' feature instead.
2017-12-10 15:58:55 -06:00
Ludovic Courtès 866f37fb7e
download: Improve efficiency of 'write-request' over TLS.
This is another instance of <https://bugs.gnu.org/22966>.
The Microsoft-IIS/7.5 server at static.nvd.nist.gov would sometimes hang
when receiving our requests byte by byte.

* guix/build/download.scm (tls-wrap) [!guile-2.0]: Add 'setvbuf' call.
2017-11-16 08:45:14 +01:00
Ludovic Courtès 9f8605958e
download: Pass the timeout to 'ftp-retr'.
This ensures the timeout applies when connecting to the port returned by
PASV.

* guix/ftp-client.scm (ftp-list): Add #:timeout parameter.  Use
'connect*' instead of 'connect' and pass TIMEOUT.
(ftp-retr): Likewise.
* guix/build/download.scm (ftp-fetch): Pass TIMEOUT to 'ftp-retr'.
2017-11-14 23:46:53 +01:00
Ludovic Courtès 65a19abf3f
download: Work around more bogus HTTP handling in Guile 2.2 <= 2.2.2.
Reported by Mark H Weaver <mhw@netris.org>
at <https://lists.gnu.org/archive/html/guix-devel/2017-11/msg00190.html>.

* guix/build/download.scm (guile-2.2) [write-request-line]: Backport
Guile commit 6ad28ae3bc6a6d9e95ab7d70510d12c97673a143.
2017-11-11 15:20:02 +01:00
Ludovic Courtès 59da6f04f4
download: Work around bogus HTTP handling in Guile 2.2 <= 2.2.2.
Reported by Konrad Hinsen <konrad.hinsen@fastmail.net>
at <https://lists.gnu.org/archive/html/guix-devel/2017-11/msg00070.html>.

* guix/build/download.scm (write-request-line) [guile-2.2]: New
procedure.
2017-11-10 23:11:52 +01:00
Ludovic Courtès 8c3488259e
Add (guix progress).
Among other things, this removes (guix utils), (guix ui), (guix config),
etc. from the closure of (guix build download), as was the case since
798648515b.

* guix/utils.scm (<progress-reporter>, call-with-progress-reporter):
Move to...
* guix/progress.scm: ... here.  New file.
* Makefile.am (MODULES): Add it.
* guix/build/download.scm (current-terminal-columns)
(nearest-exact-integer, duration->seconds, seconds->string)
(byte-count->string, progress-bar, string-pad-middle)
(rate-limited, progress-reporter/file, dump-port*)
(time-monotonic): Move to progress.scm.
* guix/scripts/download.scm: Adjust accordingly.
* guix/scripts/substitute.scm: Likewise.
2017-10-19 23:21:49 +02:00
Ludovic Courtès 347fa4aebf
download: Make 'http-fetch' public.
* guix/build/download.scm (http-fetch): Remove 'file' parameter.  Change
to return an input port and the content-length.  Make public.
(url-fetch): Adjust accordingly.
2017-10-19 23:21:49 +02:00
Ludovic Courtès b3ac341d4e
download: Remove old-Guile leftovers.
This is a followup to 36626c556e.

* guix/build/download.scm (http-fetch): Rename 'port-or-bv' to 'port'.
Assume (port? port) is always true, and remove other branch.
2017-10-19 23:21:49 +02:00
宋文武 798648515b
download: Don't report the progress too fast.
* guix/utils.scm (<progress-reporter>): New record type.
(call-with-progress-reporter): New procedure.
* guix/build/download.scm (dump-port*, rate-limited, progress-reporter/file):
New procedures.
(ftp-fetch, http-fetch): Use 'dump-port*'.
(progress-proc): Remove procedure.
* guix/scripts/substitute.scm (progress-report-port): Rewrite in terms of
<progress-reporter>.
(process-substitution): Adjust accordingly.
2017-09-20 19:49:31 +08:00
Tobias Geerinckx-Rice 57d2898772
Handle the same HTTP redirects everywhere.
* guix/build/download.scm (http-fetch): Complete the hard-coded list of HTTP
redirection status codes.
* guix/http-client.scm (http-fetch): Likewise.
* guix/scripts/lint.scm (probe-uri): Likewise.
2017-09-05 20:57:13 +02:00
Ludovic Courtès 27fd13c3c2
download: Work around GnuTLS bug with UTF-8 certificate file names.
Reported by Mark H Weaver <mhw@netris.org>
at <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=26948#17>.

* guix/build/download.scm (set-certificate-credentials-x509-trust-file!*):
New procedure.
(make-credendials-with-ca-trust-files): Use it instead of
'set-certificate-credentials-x509-trust-file!'.
2017-05-30 13:24:55 +02:00
Ludovic Courtès 7b9ac883ea
download: Continue handshake upon TLS warning alerts.
This allows us to download from site such as
<https://fusionforge.int-evry.fr> where the server does not recognize
the server name passed via the 'server_name' extension.

* guix/build/download.scm (tls-wrap): Catch 'gnutls-error' around
'handshake'.  Upon ERROR/WARNING-ALERT-RECEIVED, print a message and
call 'handshake'.
2017-05-02 23:41:13 +02:00
Ludovic Courtès 25a49294ca
cache: Work around 'time-monotonic' bug in Guile 2.2.2.
* guix/cache.scm (time-monotonic) [guile-2.2]: New variable.
* tests/cache.scm (time-monotonic) [guile-2.2]: Likewise.
* guix/build/download.scm (time-monotonic) [guile-2.2]: Adjust comment:
it's a 2.2.2 bug.
2017-04-22 14:42:15 +02:00
Ludovic Courtès 1fe958d62a
download: Work around Guile 2.2 bug with 'time-monotonic' objects.
* guix/build/download.scm (time-monotonic) [guile-2.2]: New variable.
2017-04-22 01:25:22 +02:00
Ludovic Courtès 6dfd683dc7
download: Properly parse ftp:// URLs that lack user info.
Fixes a regression introduced in
a4f5423415.

Reported by Hartmut Goebel.

* guix/build/download.scm (ftp-fetch): Account for the case where
'uri-userinfo' returns #f.  Remove the case where it returns "" since
that cannot happen.
2017-03-28 13:30:09 +02:00
Roel Janssen a4f5423415
download: Handle username and password properties for FTP URIs.
* guix/build/download.scm (ftp-fetch): Process username and password from a URI.
2017-03-21 12:15:14 +01:00
Ludovic Courtès 4fd06a4dd1
http-client: Avoid name clash with 'open-connection-for-uri' in 2.2.0.
* guix/build/download.scm (open-connection-for-uri): Add note about
same-named binding in Guile 2.2.0.
* guix/http-client.scm: Use 'guix:open-connection-for-uri' for the
procedure coming from (guix build download).
* guix/scripts/lint.scm: Likewise.
* guix/scripts/substitute.scm: Likewise.
2017-03-18 00:39:50 +01:00
Ludovic Courtès 36626c556e
build: Require Guile >= 2.0.9.
* configure.ac: Bump requirement to 2.0.9.
* doc/guix.texi (Requirements): Adjust accordingly.
* README (Requirements): Likewise.
* build-aux/download.scm: Remove workaround for <http://bugs.gnu.org/13095>.
* guix/build/download.scm: Likewise.
(http-fetch)[post-2.0.7?]: Remove.  Remove conditional code for not
POST-2.0.7?.
* guix/http-client.scm:  Remove workaround for <http://bugs.gnu.org/13095>.
(http-fetch)[post-2.0.7?]: Remove.  Remove conditional code for not
POST-2.0.7?.
* guix/serialization.scm (read-latin1-string): Remove mention of 2.0.9.
* tests/nar.scm: Use (ice-9 control).
(let/ec): Remove.
2017-03-18 00:39:49 +01:00
Ludovic Courtès b111bceecc
download: Export '%x509-certificate-directory'.
* guix/build/download.scm (%x509-certificate-directory): Export, as
expected by (guix scripts pull) since commit
7e81d699de.
2017-03-13 21:52:38 +01:00
Ludovic Courtès 580deec5b4
download: Protect against dangling symlinks in $SSL_CERT_DIR.
Reported by Christopher Baines <mail@cbaines.net>
in <https://bugs.gnu.org/25213>.

* guix/build/download.scm (make-credendials-with-ca-trust-files): Check
whether FILE exists before calling
'set-certificate-credentials-x509-trust-file!'.
2016-12-16 18:00:01 +01:00
Ludovic Courtès bc3c41ce36
download: Verify TLS certificates unless asked not to.
Fixes <http://bugs.gnu.org/24466>.
Reported by Leo Famulari <leo@famulari.name>.

* guix/build/download.scm (%x509-certificate-directory): New variable.
(make-credendials-with-ca-trust-files, peer-certificate)
(assert-valid-server-certificate, print-tls-certificate-error): New
procedures.  Add 'print-tls-certificate-error' as an exception printer
for 'tls-certificate-error'.
(tls-wrap): Add #:verify-certificate? parameter and honor it.
(open-connection-for-uri): Likewise.
(http-fetch): Likewise.
(url-fetch): Likewise.
* guix/download.scm (url-fetch)[builder]: Pass #:verify-certificate? #f.
* guix/scripts/lint.scm (probe-uri): Add case for 'tls-certificate-error'.
(validate-uri): Likewise.
* doc/guix.texi (Invoking guix download): Mention 'SSL_CERT_DIR'.
2016-11-07 23:39:01 +01:00
Ludovic Courtès 38f1cf8a8e
download: Pass the raw file name to content-addressed mirrors.
* guix/build/download.scm (url-fetch)[content-addressed-uris]: Call
'strip-store-file-name' on FILE before passing it to 'make-url'.
2016-07-31 18:27:14 +02:00
Ludovic Courtès ab84b927ef
download: Prepare to support the 'guix publish' /file URLs.
* guix/download.scm (%content-addressed-mirrors): Add 'file' parameter
to the lambda.
* guix/build/download.scm (url-fetch)[content-addressed-uris]: Adjust
accordingly.
2016-07-20 17:02:35 +02:00
David Thompson 242ad41c01
download: Use basic authentication when userinfo is present in URI.
* guix/download.scm (url-fetch): Include (guix base64) module on the
  build-side.
* guix/build/download.scm (http-fetch): Add "Authorization" header when
  userinfo is present in the URI.
2016-06-29 08:51:41 -04:00
Ludovic Courtès b18ede2704
download: Default to a 10s connection establishment timeout.
* guix/build/download.scm (ftp-fetch): Add #:timeout and pass it to
'ftp-open'.
(http-fetch): Add #:timeout and pass it to 'open-connection-for-uri' and
in recursive calls.
(url-fetch): Add #:timeout and pass it to 'http-fetch' and 'ftp-fetch'.
2016-05-27 10:37:11 +02:00
Ludovic Courtès dab2472c6a
download: Use URI objects for content-addressed mirrors.
This fixes a bug whereby 'http-fetch' would be passed a string instead
of a URI object.

* guix/build/download.scm (url-fetch): Rename 'content-addressed-urls'
to 'content-addressed-uris', and call 'string->uri'.
2016-05-27 10:37:11 +02:00
Ricardo Wurmus 8bfd602bb0 build: Accept dates with space-padded hour field.
* guix/build/download.scm: Replace "parse-rfc-822-date" from the (web
  http) module.
2016-05-16 18:26:26 +02:00
Ludovic Courtès cd436bf05a
download: Support content-addressed mirrors.
* guix/download.scm (%content-addressed-mirrors)
(%content-addressed-mirror-file): New variables.
* guix/download.scm (url-fetch)[builder]: Define
'value-from-environment.  Pass #:hashes and
 #:content-addressed-mirrors to 'url-fetch'.
Define "guix download hashes" environment variable.
* guix/build/download.scm (url-fetch): Add #:content-addressed-mirrors
and #:hashes.
[content-addressed-urls]: New variable.
Use it.
2016-05-14 17:37:47 +02:00
Ludovic Courtès cf5e58297d substitute: Better abbreviate substitute URL in progress report.
Suggested by Danny Milosavljevic <dannym@scratchpost.org>.

* guix/build/download.scm (nar-uri-abbreviation): New procedure.
* guix/scripts/substitute.scm (process-substitution): Use it instead of
'store-path-abbreviation'.
2016-04-20 23:01:41 +02:00
Ludovic Courtès 3e31ec827a download: 'uri-abbreviation' can abbreviate the URI's basename.
* guix/build/download.scm (uri-abbreviation): Use 'ellipsis' instead of
"...".  Abbreviate the basename of PATH if needed.
2016-04-20 22:52:35 +02:00
Ludovic Courtès 967ee481e8 download: Add "%COMPAT" to the priority string.
Fixes <http://bugs.gnu.org/23311>.

* guix/build/download.scm (tls-wrap): Add 'set-session-priorities!' call.
2016-04-20 13:17:52 +02:00
Ludovic Courtès 4d276c6403 download: Add 'current-terminal-columns' parameter.
* guix/build/download.scm (current-terminal-columns): New variable.
(progress-proc): Use it instead of the hard-coded "80".
2016-04-15 00:32:18 +02:00
Ludovic Courtès 8a2154fefa download: Send an ANSI erase-in-line sequence in addition to CR.
Partly fixes <http://bugs.gnu.org/22536>.
Reported by Danny Milosavljevic <dannym@scratchpost.org>.

* guix/build/download.scm (progress-proc): Send an ANSI erase-in-line
sequence.
2016-04-15 00:32:18 +02:00
Alex Kost 82fd23b81f download: Follow HTTP 307 "Temporary Redirection".
Fixes <http://bugs.gnu.org/23275>.
Reported by Albin Söderqvist <albin@fripost.org>.

* guix/build/download.scm (http-fetch): Follow redirections upon 307.
  This is what 'binaries.openttd.org' does.
2016-04-14 10:34:37 +03:00
Ludovic Courtès 097a951e96 download: Add 'close-connection'.
Partially fixes <http://bugs.gnu.org/20145>.

* guix/build/download.scm (add-weak-reference): Remove.
(%tls-ports): New variable.
(register-tls-record-port): New procedure.
(tls-wrap): Use it instead of 'add-weak-reference'.
(close-connection): New procedure.
2016-03-17 23:53:53 +01:00
Ludovic Courtès 1b9aefa394 download: Always use AI_ADDRCONFIG when resolving host names.
* guix/build/download.scm (open-socket-for-uri): Always pass
  AI_ADDRCONFIG to 'getaddrinfo' as recommended in the fine Guile
  manual.
* guix/ftp-client.scm (ftp-open): Ditto.
2015-11-12 23:47:01 +01:00
Ludovic Courtès 60fd51222f download: Add timeout parameter for connections.
* guix/build/download.scm (ensure-uri): New procedure.
(current-http-proxy): New variable.
(open-socket-for-uri): Copy from Guile commit aaea5b2, but add #:timeout
parameter and use 'connect*' instead of 'connect'.
(open-connection-for-uri): Add #:timeout parameter and pass it to
'open-socket-for-uri'.
2015-11-12 23:47:00 +01:00
Ludovic Courtès fde1783087 download: Check whether HORIZONTAL ELLIPSIS can be encoded.
* guix/build/download.scm (ellipsis): New procedure.
  (store-path-abbreviation): Use it.
2015-10-01 11:28:58 +02:00
Ludovic Courtès 75726135ce download: Don't abbreviate things that are not store items.
Fixes a regression introduced in a8be7b9a.

* guix/build/download.scm (store-path-abbreviation): Return STORE-PATH
  if it's not an actual store path.  Fixes an out-of-range exception
  when running tests/substitute.scm and tests/store.scm.
2015-09-24 21:54:37 +02:00
Steve Sprang 9462882889 download: Fix some minor progress-logging regressions.
* guix/build/download.scm
  (string-pad-middle): Allow resulting padded string to overflow.
  (store-url-abbreviation): Remove unnecessary procedure.
  (progress-proc): Use BASENAME as default for parameter 'abbreviation'.
  (url-fetch): Display extra newlines for readability.
2015-09-23 22:04:46 -04:00
Steve Sprang 47770296d2 download: Only show hours in the elapsed time if necessary.
* guix/build/download.scm
  (seconds->string): Conditionally include hours in timestamp.

Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2015-09-17 14:40:46 +02:00
Steve Sprang a8be7b9a7a substitute: Improve readability of download progress report.
* guix/build/download.scm
  (string-pad-middle, store-url-abbreviation, store-path-abbreviation):
  New procedures.
  (progress-proc): Add #:abbreviation parameter and use it.  Generate a
  better indeterminate progress string.
* guix/scripts/substitute.scm (assert-valid-narinfo): Add newlines to output.
  (process-substitution): Use byte-count->string and store-path-abbreviation.

Co-authored-by: Ludovic Courtès <ludo@gnu.org>
2015-09-16 23:50:12 +02:00