This allows substitute servers to tell 'guix substitute' how long they
can cache narinfo lookups.
* guix/scripts/substitute.scm (cache-narinfo!): Add 'ttl' parameter.
[cache-entry]: Honor it.
(fetch-narinfos)[handle-narinfo-response]: Check the 'Cache-Control'
header of RESPONSE and pass its 'max-age' value to 'cache-narinfo!'.
* guix/scripts/substitute.scm (cached-narinfo): Expect 'narinfo' sexp
version 2 with a 'ttl' field.
(cache-narinfo!)[cache-entry]: Produce 'narinfo' sexp version 2 with a
'ttl' field.
(remove-expired-cached-narinfos)[expired?]: Read 'narinfo' sexp version 2.
The connection used to fetch /nix-cache-info is now reused for the
subsequent narinfo requests.
* guix/scripts/substitute.scm (download-cache-info)[download]: Remove.
[uri, read-cache-info]: New variables.
Rewrite in terms of 'http-fetch' instead of 'fetch'. Return an open
port in addition to a <cache-info>.
* guix/scripts/substitute.scm (http-multiple-get): Add #:port parameter
and honor it.
(fetch-narinfos)[do-fetch]: Add 'port' parameter.
Adjust to new 'download-cache-info' and 'do-fetch' signatures.
This parameter became unused with the switch to HTTP pipelining in
commit d3a652037e.
* guix/scripts/substitute.scm (fetch): Remove #:quiet-404? and adjust
accordingly.
Fixes <http://bugs.gnu.org/22937>.
Reported by Chris Marusich <cmmarusich@gmail.com>.
* guix/scripts/substitute.scm (fetch): Add 'https' alongside 'http'.
Use 'open-connection-for-uri' instead of 'open-socket-for-uri'. Call
'setvbuf' only when PORT matches 'file-port?'.
(http-multiple-get): Likewise. Change 'base-url' parameter to
'base-uri'.
(fetch-narinfos)[do-fetch]: Add 'https' case alongside 'http'. Pass URI
instead of URL to 'http-multiple-get'.
* doc/guix.texi (Requirements): Move GnuTLS one level higher and mention
HTTPS substitutes.
(Substitutes): Mention HTTPS and recommend it. Explain why servers
are not authenticated. Add "On Trusting Binaries" subsection.
Reported in <http://bugs.gnu.org/22937>
by Chris Marusich <cmmarusich@gmail.com>.
* guix/scripts/substitute.scm (fetch): Add 'else' case and call 'leave'.
Suggested by Hynek Urban <hynek.urban@gmail.com>.
* guix/scripts/substitute.scm (fetch-narinfos): Move body to...
[do-fetch]: ... here. New procedure.
Emit a warning when CACHE-INFO's prefix does not match.
* guix/scripts/substitute.scm (lookup-narinfos/diverse): New procedure.
(lookup-narinfo): Use it.
(process-query): Change #:cache-url to #:cache-urls.
[valid?]: Remove 'narinfo?' check, which is no longer necessary.
Use 'lookup-narinfos/diverse' instead of 'lookup-narinfos'.
(process-substitution): Change #:cache-url to #:cache-urls.
(%cache-url): Rename to...
(%cache-urls): ... this. Turn into a list.
(guix-substitute): Remove 'getaddrinfo' test with early exit. Adjust
calls to 'process-query' and 'process-substitution'.
* tests/substitute.scm: Change '%cache-url' to '%cache-urls'.
* guix/scripts/substitute.scm (lookup-narinfos): Filter out #f values
from CACHED, such that the end result is exactly a list of narinfos,
not interspersed with #f.
* guix/scripts/challenge.scm (discrepancies): Assume REMOTE is a list of
narinfos.
* guix/build/download.scm
(string-pad-middle, store-url-abbreviation, store-path-abbreviation):
New procedures.
(progress-proc): Add #:abbreviation parameter and use it. Generate a
better indeterminate progress string.
* guix/scripts/substitute.scm (assert-valid-narinfo): Add newlines to output.
(process-substitution): Use byte-count->string and store-path-abbreviation.
Co-authored-by: Ludovic Courtès <ludo@gnu.org>
* guix/scripts/publish.scm (narinfo-string): Catch 'system-error' around
'load-derivation' call; return BASE-INFO upon ENOENT. This allows us
to return the narinfo even if DERIVER is missing. Before that, the
exception would be uncaught, leading to 500 Internal Error on the
client side.
* guix/scripts/substitute.scm (display-narinfo-data,
process-query, process-substitution): New procedures. Code moved from...
(guix-substitute): ... here. Use them.
* guix/scripts/substitute.scm (or*): New macro.
(%cache-url): Honor "untrusted-substitute-urls".
* guix/tests.scm (%test-substitute-urls): New variable.
(open-connection-for-tests): Use it.
* tests/derivations.scm ("derivation-prerequisites-to-build and substitutes",
"derivation-prerequisites-to-build and substitutes, non-substitutable
build", "derivation-prerequisites-to-build and substitutes, local build"):
Pass it to 'set-build-options'.
* tests/guix-daemon.sh: Likewise.
* tests/store.scm ("substitute query, alternating URLs"): New test.
("substitute query", "substitute", "substitute + build-things with output
path", "substitute, corrupt output hash", "substitute --fallback"): Pass
#:substitute-urls to 'set-build-options'.
This ensures that switching between different substitute servers doesn't lead
to a polluted narinfo cache.
* guix/scripts/substitute.scm (narinfo-cache-file): Add 'cache-url'
parameter. Add the base32 of CACHE-URL as a sub-directory under
%NARINFO-CACHE-DIRECTORY. Update callers.
(cached-narinfo): Likewise. Call 'mkdir-p' on the dirname of the cache
file. Update callers.
(remove-expired-cached-narinfos): Add 'directory' parameter and use it
instead of %NARINFO-CACHE-DIRECTORY.
(narinfo-cache-directories): New procedure.
(maybe-remove-expired-cached-narinfo): Call 'remove-expired-cached-narinfos'
for each item returned by 'narinfo-cache-directories'.
* guix/scripts/substitute.scm (<cache>): Rename to...
(<cache-info>): ... this.
(open-cache): Rename to...
(download-cache-info): ... this. Return a <cache-info> or #f.
(open-cache*): Remove.
(cache-narinfo!): Take a URL instead of a <cache> as the first parameter.
(fetch-narinfos): Likewise. Call 'download-cache-info'. Remove use of
'force'.
(guix-substitute): Replace calls to 'open-cache*' with %CACHE-URL.
Reported
at <http://lists.gnu.org/archive/html/guix-devel/2015-07/msg00119.html>.
* guix/scripts/substitute.scm (http-multiple-get): When RESP has "Connection:
close", consume HEAD anyway; always call PROC to read from BODY.
Co-authored-by: Ludovic Courtès <ludo@gnu.org>
Previously, code using directly (guix build download) was still affected
by <http://bugs.gnu.org/15368>. This includes source derivations, the
'guix download' command, and (guix gnu-maintenance).
'guix substitute' was unaffected since it used (guix http-client), which
already had the fix.
* guix/http-client.scm (open-socket-for-uri): Remove.
(http-fetch): Remove #:buffered? argument to 'open-socket-for-uri';
use 'setvbuf' instead.
* guix/scripts/substitute.scm (fetch): Likewise.
* guix/build/download.scm (open-socket-for-uri): New procedure, taken
from guix/http-client.scm, but without the #:buffered? parameter.
In practice we would not leak much since we reconnect after ~100
requests (with nginx running on hydra.gnu.org.)
* guix/scripts/substitute.scm (http-multiple-get): Call 'close-port'
before 'connect'.
Ludovic Courtès