Use ziglang.org community mirrors list
This new list supersedes the `mirrors.json` in this repo.
This commit is contained in:
mlugg
@@ -1,25 +0,0 @@
|
||||
{
|
||||
"name": "check-mirrors",
|
||||
"on": {
|
||||
"schedule": [
|
||||
{ "cron": "0 6,18 * * *" },
|
||||
],
|
||||
"workflow_dispatch": null,
|
||||
},
|
||||
"jobs": {
|
||||
"check-mirrors": {
|
||||
"runs-on": "ubuntu-latest",
|
||||
"steps": [
|
||||
{ "uses": "actions/checkout@v4" },
|
||||
{
|
||||
"name": "Check mirrors",
|
||||
"env": {
|
||||
"PUSHOVER_USER": "${{ secrets.PUSHOVER_USER }}",
|
||||
"PUSHOVER_TOKEN": "${{ secrets.PUSHOVER_TOKEN }}",
|
||||
},
|
||||
"run": "sh check-mirrors.sh",
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
}
|
||||
@@ -1 +0,0 @@
|
||||
check-mirrors-summary.md
|
||||
@@ -44,8 +44,8 @@ If you want to use one specific mirror, you can set it using the `mirror` option
|
||||
|
||||
Please don't do this unnecessarily; it's not nice to hammer one mirror. This mirror is not permitted to
|
||||
be https://ziglang.org/download to avoid the official website being hit with large amounts of requests.
|
||||
If you've experienced issues with a default mirror, please open an issue, and I will communicate with the
|
||||
mirror's owner or remove it from the list.
|
||||
If you've experienced issues with a default mirror, please [open an issue][report-bad-mirror] on the Zig
|
||||
website repository, which is where the list of mirrors is maintained.
|
||||
|
||||
If necessary, the caching of the global Zig cache directory can be disabled by setting the option
|
||||
`use-cache: false`. Don't do this without reason: preserving the Zig cache will typically speed things up
|
||||
@@ -63,6 +63,7 @@ setting `cache-size-limit: 0`.
|
||||
|
||||
[mach-nominated]: https://machengine.org/about/nominated-zig/
|
||||
[matrix]: https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/running-variations-of-jobs-in-a-workflow
|
||||
[report-bad-mirror]: https://github.com/ziglang/www.ziglang.org/issues/new
|
||||
|
||||
## Details
|
||||
|
||||
@@ -75,65 +76,10 @@ local caches are redirected to the global cache directory to make optimal use of
|
||||
|
||||
## Adding a mirror
|
||||
|
||||
Anyone is welcome to host a Zig download mirror; thanks to the tarball signatures, the mirror provider need
|
||||
not be trusted. Naturally, if a mirror is found to be a bad actor, it will be removed, and likewise if a
|
||||
mirror repeatedly encounters reliability problems.
|
||||
The list of tarball mirrors is not in this repository; rather, the [community mirror list][mirrors] from
|
||||
ziglang.org is used. If you are interested in hosting a mirror of your own, check out the
|
||||
[documentation][host-mirror] on the Zig website repository. That way, your mirror can benefit not just
|
||||
setup-zig, but also any other tooling which wants to fetch Zig!
|
||||
|
||||
The rules for adding a mirror are listed below. Note that I (@mlugg) reserve the right to, for any or no
|
||||
reason, exclude mirrors which obey these rules, or include mirrors which violate them.
|
||||
|
||||
> [!NOTE]
|
||||
> While there are a lot of rules listed here, most of them should be obvious. They are stated explicitly here
|
||||
> to ensure complete clarity on what is expected of a mirror. Please do read these requirements through before
|
||||
> attempting to add a mirror.
|
||||
|
||||
* A mirror provides a single base URL, which we will call `X`.
|
||||
* `X` **may** include a path component, but is not required to. For instance, `https://foo.bar/zig/` is okay,
|
||||
as is `https://zig.baz.qux/`.
|
||||
* The mirror **must** have working HTTPS support. `X` **must** start with `https://`.
|
||||
* The mirror **must** cache tarballs locally. For instance, it may not simply forward all requests to another
|
||||
mirror.
|
||||
* The mirror **may** routinely evict its local tarball caches based on any reasonable factor, such as age,
|
||||
access frequency, or the existence of newer versions. This does not affect whether the mirror may return 404
|
||||
for requests to these files (see below).
|
||||
* The mirror **must** download its tarballs from either `https://ziglang.org/`, or another mirror which
|
||||
follows these rules.
|
||||
* Tarballs **must** be accessible by sending GET requests for files under `X`, where the filename matches that
|
||||
of the files provided by `https://ziglang.org/`, not including the directory part. For instance,
|
||||
`X/zig-linux-x86_64-0.13.0.tar.xz` is a valid access, and should return the same file as
|
||||
`https://ziglang.org/download/0.13.0/zig-linux-x86_64-0.13.0.tar.xz`.
|
||||
* Files provided by the mirror **must** be bit-for-bit identical to their `https://ziglang.org/` counterparts.
|
||||
* If a file is accessed whose Zig version is a master branch build (i.e. a `-dev` version), and the version is
|
||||
ordered before the latest major release of Zig, the mirror **may** respond with 404 Not Found, but is not
|
||||
required to. For instance, at the time of writing, `0.13.0` is the latest major release of Zig, so a mirror
|
||||
may respond with 404 for `0.13.0-dev...` builds, but *not* for `0.14.0-dev...` builds.
|
||||
* If a file is accessed whose Zig version is `0.5.0` or below, the mirror **may** respond with 404 Not Found,
|
||||
but is not required to.
|
||||
* If a file is acccessed which represents a *source* tarball, such as `X/zig-0.13.0.tar.xz`, the mirror
|
||||
**may** respond with 404 Not Found, but is not required to. The same applies to "bootstrap source tarballs",
|
||||
such as `X/zig-bootstrap-0.13.0.tar.xz`.
|
||||
* For all other accesses of valid Zig tarballs, the mirror **must** respond with status code 200 OK and the
|
||||
file in question. If the mirror has not yet cached the file locally, it should immediately download it from
|
||||
a permitted source (as covered above), and respond with the downloaded file.
|
||||
* If a tarball `X/foo.ext` is available by the above rules, requesting the minisign signature file
|
||||
`X/foo.ext.minisig` **must** also respond with status code 200 OK and the signature file in question, like
|
||||
the tarball itself.
|
||||
* The mirror **may** rate-limit accesses. If an access failed due to rate-limiting, the mirror **should**
|
||||
return HTTP status code 429 Too Many Requests.
|
||||
* The mirror **may** undergo maintenance, upgrades, and other scheduled downtime. If an access fails for this
|
||||
reason, where possible, the mirror **should** return HTTP status code 503 Unavailable. The mirror **should**
|
||||
try to minimize such downtime.
|
||||
* The mirror **may** undergo occasional unintended and unscheduled downtime. The mirror **must** go to all
|
||||
efforts to minimize such outages, and **must** resolve such outages within a reasonable time upon being
|
||||
notified of them.
|
||||
* The mirror **may** observe the `?source=github-actions` query parameter to track how many requests originate
|
||||
from this Action. This Action will provide this query parameter to all mirror requests.
|
||||
|
||||
The easiest way to set up a mirror right now is using Mach's [Wrench][wrench]. For instructions, please see
|
||||
[the relevant section of their README][setup-wrench].
|
||||
|
||||
[wrench]: https://github.com/hexops/wrench
|
||||
[setup-wrench]: https://github.com/hexops/wrench?tab=readme-ov-file#run-your-own-ziglangorgdownload-mirror
|
||||
|
||||
After setting up a mirror, you can add it to this GitHub Action by opening a PR which adds it to the list in
|
||||
[mirrors.json](https://github.com/mlugg/setup-zig/blob/main/mirrors.json).
|
||||
[mirrors]: https://ziglang.org/download/community-mirrors/
|
||||
[host-mirror]: https://github.com/ziglang/www.ziglang.org/blob/main/MIRRORS.md
|
||||
|
||||
@@ -1,47 +0,0 @@
|
||||
#!/bin/sh
|
||||
|
||||
summary_path="${GITHUB_STEP_SUMMARY:-check-mirrors-summary.md}"
|
||||
temp_path="$(mktemp)"
|
||||
|
||||
zig_version="$(curl -sSL 'https://ziglang.org/download/index.json' | jq -r '.master.version')"
|
||||
filename="zig-x86_64-linux-$zig_version.tar.xz"
|
||||
|
||||
curl -L "https://ziglang.org/builds/$filename" >"$temp_path"
|
||||
correct_sum="$(sha512sum "$temp_path")"
|
||||
|
||||
rm "$summary_path"
|
||||
|
||||
echo "| URL | Owner | Result | Time (s) |" >>"$summary_path"
|
||||
echo "|:----|:------|:-------|---------:|" >>"$summary_path"
|
||||
|
||||
jq '.[] | .[0] + " " + .[1]' -r <mirrors.json | while read -r mirror_url mirror_name; do
|
||||
time_secs="$(curl -w "%{time_total}" -L "$mirror_url/$filename" -o "$temp_path")"
|
||||
download_status="$?"
|
||||
|
||||
if [ "$download_status" -eq 0 ]; then
|
||||
mirror_sum="$(sha512sum "$temp_path")"
|
||||
if [ "$mirror_sum" = "$correct_sum" ]; then
|
||||
err="" # success
|
||||
else
|
||||
err="checksum mismatch: $mirror_sum (expected $correct_sum)"
|
||||
fi
|
||||
else
|
||||
err="curl exit status: $download_status"
|
||||
fi
|
||||
|
||||
md_time_secs="$(printf '`%0.2f`' "$time_secs")" # 2dp, backticks
|
||||
md_mirror_name="$(echo "$mirror_name" | sed -e 's/[<>]/\\\0/g')" # backslash-prefix angle brackets
|
||||
md_status="$(if [ "$err" = "" ]; then echo ":white_check_mark:"; else echo ":warning: \`$err\`"; fi)" # tick or cross for success
|
||||
echo "| $mirror_url | $md_mirror_name | $md_status | $md_time_secs |" >>"$summary_path"
|
||||
|
||||
if [ "$err" = "" ]; then
|
||||
continue # success
|
||||
fi
|
||||
|
||||
curl -s \
|
||||
-F user="$PUSHOVER_USER" -F token="$PUSHOVER_TOKEN" \
|
||||
-F message="Zig download mirror '$mirror_name' is down" \
|
||||
https://api.pushover.net/1/messages.json
|
||||
done
|
||||
|
||||
rm "$temp_path"
|
||||
@@ -13,10 +13,8 @@ const MINISIGN_KEY = 'RWSGOq2NVecA2UPNdBUZykf1CCb147pkmdtYxgb3Ti+JO/wCYvhbAb/U';
|
||||
// The base URL of the official builds of Zig. This is only used as a fallback, if all mirrors fail.
|
||||
const CANONICAL = 'https://ziglang.org/builds';
|
||||
|
||||
// The list of mirrors we attempt to fetch from. These need not be trusted, as
|
||||
// we always verify the minisign signature.
|
||||
// This is an array of URLs.
|
||||
const MIRRORS = require('./mirrors.json').map((x) => x[0]);
|
||||
// The URL of the mirror list. This should be an ASCII-encoded text file, with one mirror per LF-separated line.
|
||||
const MIRRORS_URL = 'https://ziglang.org/download/community-mirrors.txt';
|
||||
|
||||
async function downloadFromMirror(mirror, tarball_filename) {
|
||||
const tarball_path = await tc.downloadTool(`${mirror}/${tarball_filename}?source=github-actions`);
|
||||
@@ -52,9 +50,21 @@ async function downloadTarball(tarball_filename) {
|
||||
return await downloadFromMirror(preferred_mirror, tarball_filename);
|
||||
}
|
||||
|
||||
// Fetch the list of mirrors from ziglang.org. Caching the mirror list is awkward in this context,
|
||||
// so if the list is inaccessible, we just fetch from ziglang.org as a fallback.
|
||||
let mirrors = [];
|
||||
try {
|
||||
const mirrors_response = await fetch(MIRRORS_URL);
|
||||
mirrors = (await mirrors_response.text()).split('\n').filter((url) => url.length != 0);
|
||||
} catch {
|
||||
// For some reason the mirrors are inaccessible. That's okay; allow ourselves to fall back to ziglang.org below.
|
||||
}
|
||||
|
||||
core.info(`Available mirrors: ${mirrors.join(", ")}`);
|
||||
|
||||
// We will attempt all mirrors before making a last-ditch attempt to the official download.
|
||||
// To avoid hammering a single mirror, we first randomize the array.
|
||||
const shuffled_mirrors = MIRRORS.map((m) => [m, Math.random()]).sort((a, b) => a[1] - b[1]).map((a) => a[0]);
|
||||
const shuffled_mirrors = mirrors.map((m) => [m, Math.random()]).sort((a, b) => a[1] - b[1]).map((a) => a[0]);
|
||||
for (const mirror of shuffled_mirrors) {
|
||||
core.info(`Attempting mirror: ${mirror}`);
|
||||
try {
|
||||
|
||||
@@ -1,5 +0,0 @@
|
||||
[
|
||||
[ "https://pkg.machengine.org/zig", "emidoots <[email protected]>" ],
|
||||
[ "https://zigmirror.hryx.net/zig", "hryx <[email protected]>" ],
|
||||
[ "https://zig.linus.dev/zig", "linusg <[email protected]>" ]
|
||||
]
|
||||
Reference in New Issue
Block a user