Initial commit
This commit is contained in:
@@ -0,0 +1,80 @@
|
||||
# Code of Conduct
|
||||
|
||||
## Our Pledge
|
||||
|
||||
In the interest of fostering an open and welcoming environment, we as
|
||||
contributors and maintainers pledge to make participation in our project and
|
||||
our community a harassment-free experience for everyone, regardless of age, body
|
||||
size, disability, ethnicity, sex characteristics, gender identity and expression,
|
||||
level of experience, education, socio-economic status, nationality, personal
|
||||
appearance, race, religion, or sexual identity and orientation.
|
||||
|
||||
## Our Standards
|
||||
|
||||
Examples of behavior that contributes to creating a positive environment
|
||||
include:
|
||||
|
||||
* Using welcoming and inclusive language
|
||||
* Being respectful of differing viewpoints and experiences
|
||||
* Gracefully accepting constructive criticism
|
||||
* Focusing on what is best for the community
|
||||
* Showing empathy towards other community members
|
||||
|
||||
Examples of unacceptable behavior by participants include:
|
||||
|
||||
* The use of sexualized language or imagery and unwelcome sexual attention or
|
||||
advances
|
||||
* Trolling, insulting/derogatory comments, and personal or political attacks
|
||||
* Public or private harassment
|
||||
* Publishing others' private information, such as a physical or electronic
|
||||
address, without explicit permission
|
||||
* Other conduct which could reasonably be considered inappropriate in a
|
||||
professional setting
|
||||
|
||||
## Our Responsibilities
|
||||
|
||||
Project maintainers are responsible for clarifying the standards of acceptable
|
||||
behavior and are expected to take appropriate and fair corrective action in
|
||||
response to any instances of unacceptable behavior.
|
||||
|
||||
Project maintainers have the right and responsibility to remove, edit, or
|
||||
reject comments, commits, code, wiki edits, issues, and other contributions
|
||||
that are not aligned to this Code of Conduct, or to ban temporarily or
|
||||
permanently any contributor for other behaviors that they deem inappropriate,
|
||||
threatening, offensive, or harmful.
|
||||
|
||||
## Scope
|
||||
|
||||
This Code of Conduct applies within all project spaces, and it also applies when
|
||||
an individual is representing the project or its community in public spaces.
|
||||
Examples of representing a project or community include using an official
|
||||
project e-mail address, posting via an official social media account, or acting
|
||||
as an appointed representative at an online or offline event. Representation of
|
||||
a project may be further defined and clarified by project maintainers.
|
||||
|
||||
This Code of Conduct also applies outside the project spaces when there is a
|
||||
reasonable belief that an individual's behavior may have a negative impact on
|
||||
the project or its community.
|
||||
|
||||
## Enforcement
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
||||
reported by contacting the project team at <opensource-conduct@meta.com>. All
|
||||
complaints will be reviewed and investigated and will result in a response that
|
||||
is deemed necessary and appropriate to the circumstances. The project team is
|
||||
obligated to maintain confidentiality with regard to the reporter of an incident.
|
||||
Further details of specific enforcement policies may be posted separately.
|
||||
|
||||
Project maintainers who do not follow or enforce the Code of Conduct in good
|
||||
faith may face temporary or permanent repercussions as determined by other
|
||||
members of the project's leadership.
|
||||
|
||||
## Attribution
|
||||
|
||||
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
|
||||
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
|
||||
|
||||
[homepage]: https://www.contributor-covenant.org
|
||||
|
||||
For answers to common questions about this code of conduct, see
|
||||
https://www.contributor-covenant.org/faq
|
||||
@@ -0,0 +1,32 @@
|
||||
# Contributing to Meta Open Source Projects
|
||||
|
||||
We want to make contributing to this project as easy and transparent as
|
||||
possible.
|
||||
|
||||
## Pull Requests
|
||||
We actively welcome your pull requests.
|
||||
|
||||
1. Fork the repo and create your branch from `main`.
|
||||
2. If you've added code that should be tested, add tests.
|
||||
3. If you've changed APIs, update the documentation.
|
||||
4. Ensure the test suite passes.
|
||||
5. Make sure your code lints.
|
||||
6. If you haven't already, complete the Contributor License Agreement ("CLA").
|
||||
|
||||
## Contributor License Agreement ("CLA")
|
||||
In order to accept your pull request, we need you to submit a CLA. You only need
|
||||
to do this once to work on any of Meta's open source projects.
|
||||
|
||||
Complete your CLA here: <https://code.facebook.com/cla>
|
||||
|
||||
## Issues
|
||||
We use GitHub issues to track public bugs. Please ensure your description is
|
||||
clear and has sufficient instructions to be able to reproduce the issue.
|
||||
|
||||
Meta has a [bounty program](https://www.facebook.com/whitehat/) for the safe
|
||||
disclosure of security bugs. In those cases, please go through the process
|
||||
outlined on that page and do not file a public issue.
|
||||
|
||||
## License
|
||||
By contributing to this project, you agree that your contributions will be licensed
|
||||
under the LICENSE file in the root directory of this source tree.
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
FROM ubuntu:22.04
|
||||
|
||||
# Consider?
|
||||
# FROM python:3.12-slim
|
||||
|
||||
# https://serverfault.com/a/1016972 to ensure installing tzdata does not
|
||||
# result in a prompt that hangs forever.
|
||||
ARG DEBIAN_FRONTEND=noninteractive
|
||||
ENV TZ=Etc/UTC
|
||||
|
||||
# Update and install some basic packages to register a PPA.
|
||||
RUN apt update -y
|
||||
|
||||
RUN apt install -y curl python3 python3-pip
|
||||
|
||||
# Install GitHub CLI.
|
||||
RUN curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg \
|
||||
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
|
||||
&& apt update \
|
||||
&& apt install -y gh
|
||||
|
||||
# Install necessary crypto algos.
|
||||
RUN pip3 install blake3
|
||||
|
||||
COPY process_config.py /process_config.py
|
||||
|
||||
ENTRYPOINT ["/process_config.py"]
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) Meta Platforms, Inc. and its affiliates.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,266 @@
|
||||
# dotslash-publish-release
|
||||
|
||||
This GitHub action can create [DotSlash](https://dotslash-cli.com/) files for
|
||||
executables that you have published as part of a [GitHub release](
|
||||
https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases).
|
||||
The newly generated DotSlash files will be added to the existing release.
|
||||
|
||||
This action is designed to run after the [GitHub Actions workflows](
|
||||
https://docs.github.com/en/actions/using-workflows) that are responsible for
|
||||
uploading your primary release artifacts via `gh release upload` or equivalent.
|
||||
|
||||
## Example
|
||||
|
||||
If you had separate workflows for each platform such as `linux-release`,
|
||||
`macos-release`, and `windows-release`, then you could define a new GitHub
|
||||
action under `.github/workflows/dotslash.yml` as follows:
|
||||
|
||||
|
||||
```yaml
|
||||
name: Generate DotSlash files
|
||||
|
||||
on:
|
||||
workflow_run:
|
||||
# These must match the names of the workflows that publish
|
||||
# artifacts to your GitHub release.
|
||||
workflows: [linux-release, macos-release, windows-release]
|
||||
types:
|
||||
- completed
|
||||
|
||||
jobs:
|
||||
generate-dotslash-files:
|
||||
name: Generating and uploading DotSlash files
|
||||
runs-on: ubuntu-latest
|
||||
if: ${{ github.event.workflow_run.conclusion == 'success' }}
|
||||
steps:
|
||||
- uses: facebook/dotslash-publish-release@v1
|
||||
# This is necessary because the action uses
|
||||
# `gh release upload` to publish the generated DotSlash file(s)
|
||||
# as part of the release.
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
with:
|
||||
# Additional file that lives in your repo that defines
|
||||
# how your DotSlash file(s) should be generated.
|
||||
config: .github/workflows/dotslash-config.json
|
||||
# Tag for the release to to target.
|
||||
tag: ${{ github.event.workflow_run.head_branch }}
|
||||
```
|
||||
|
||||
Note the `config` line that specifies a path to a JSON file in your repo that
|
||||
determines what DotSlash files to generate. For example, if this GitHub action
|
||||
were defined in the [facebook/hermes](https://github.com/facebook/hermes)
|
||||
repository on GitHub, and the contents of
|
||||
`.github/workflows/dotslash-config.json` were as follows:
|
||||
|
||||
```json
|
||||
{
|
||||
"outputs": {
|
||||
"hermes": {
|
||||
"platforms": {
|
||||
"macos-x86_64": {
|
||||
"regex": "^hermes-cli-darwin-",
|
||||
"path": "hermes"
|
||||
},
|
||||
"macos-aarch64": {
|
||||
"regex": "^hermes-cli-darwin-",
|
||||
"path": "hermes"
|
||||
},
|
||||
"linux-x86_64": {
|
||||
"regex": "^hermes-cli-linux-",
|
||||
"path": "hermes"
|
||||
},
|
||||
"windows-x86_64": {
|
||||
"regex": "^hermes-cli-windows-",
|
||||
"path": "hermes.exe"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Then this action would have added the following DotSlash file named `hermes` to
|
||||
the [v0.12.0 release](https://github.com/facebook/hermes/releases/tag/v0.12.0):
|
||||
|
||||
```json
|
||||
#!/usr/bin/env dotslash
|
||||
|
||||
{
|
||||
"name": "hermes",
|
||||
"platforms": {
|
||||
"macos-x86_64": {
|
||||
"size": 10600817,
|
||||
"hash": "blake3",
|
||||
"digest": "25f984911f199f9229ca0327c52700fa9a8db9aefe95e84f91ba6be69902436a",
|
||||
"format": "tar.gz",
|
||||
"path": "hermes",
|
||||
"providers": [
|
||||
{
|
||||
"url": "https://github.com/facebook/hermes/releases/download/v0.12.0/hermes-cli-darwin-v0.12.0.tar.gz"
|
||||
},
|
||||
{
|
||||
"type": "github-release",
|
||||
"repo": "https://github.com/facebook/hermes",
|
||||
"tag": "v0.12.0",
|
||||
"name": "hermes-cli-darwin-v0.12.0.tar.gz"
|
||||
}
|
||||
]
|
||||
},
|
||||
"macos-aarch64": {
|
||||
"size": 10600817,
|
||||
"hash": "blake3",
|
||||
"digest": "25f984911f199f9229ca0327c52700fa9a8db9aefe95e84f91ba6be69902436a",
|
||||
"format": "tar.gz",
|
||||
"path": "hermes",
|
||||
"providers": [
|
||||
{
|
||||
"url": "https://github.com/facebook/hermes/releases/download/v0.12.0/hermes-cli-darwin-v0.12.0.tar.gz"
|
||||
},
|
||||
{
|
||||
"type": "github-release",
|
||||
"repo": "https://github.com/facebook/hermes",
|
||||
"tag": "v0.12.0",
|
||||
"name": "hermes-cli-darwin-v0.12.0.tar.gz"
|
||||
}
|
||||
]
|
||||
},
|
||||
"linux-x86_64": {
|
||||
"size": 47099598,
|
||||
"hash": "blake3",
|
||||
"digest": "8d2c1bcefc2ce6e278167495810c2437e8050780ebb4da567811f1d754ad198c",
|
||||
"format": "tar.gz",
|
||||
"path": "hermes",
|
||||
"providers": [
|
||||
{
|
||||
"url": "https://github.com/facebook/hermes/releases/download/v0.12.0/hermes-cli-linux-v0.12.0.tar.gz"
|
||||
},
|
||||
{
|
||||
"type": "github-release",
|
||||
"repo": "https://github.com/facebook/hermes",
|
||||
"tag": "v0.12.0",
|
||||
"name": "hermes-cli-linux-v0.12.0.tar.gz"
|
||||
}
|
||||
]
|
||||
},
|
||||
"windows-x86_64": {
|
||||
"size": 17456100,
|
||||
"hash": "blake3",
|
||||
"digest": "7efee4f92a05e34ccfa7c21c7a05f939d8b724bc802423d618db22efb83bfe1b",
|
||||
"format": "tar.gz",
|
||||
"path": "hermes.exe",
|
||||
"providers": [
|
||||
{
|
||||
"url": "https://github.com/facebook/hermes/releases/download/v0.12.0/hermes-cli-windows-v0.12.0.tar.gz"
|
||||
},
|
||||
{
|
||||
"type": "github-release",
|
||||
"repo": "https://github.com/facebook/hermes",
|
||||
"tag": "v0.12.0",
|
||||
"name": "hermes-cli-windows-v0.12.0.tar.gz"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Note that each entry in `platforms` in the `dotslash-config.json` is reflected
|
||||
in the `platforms` section of the generated DotSlash file. Each config entry
|
||||
takes a `"name"` or a `"regex"` to use to identify the appropriate artifact in
|
||||
the release and the `"path"` indicates the `"path"` that should be used for the
|
||||
artifact in the generated DotSlash file.
|
||||
|
||||
The `dotslash-publish-release` action defaults to using BLAKE3 as the hash
|
||||
function, so it takes responsibility for computing the `size` and `digest`
|
||||
values. It also tries to "guess" the appropriate value of `"format"` based on
|
||||
the suffix of the URL, though this can also be specified explicitly, which is a
|
||||
bit safer:
|
||||
|
||||
```json
|
||||
{
|
||||
"outputs": {
|
||||
"hermes": {
|
||||
"platforms": {
|
||||
"macos-x86_64": {
|
||||
"regex": "^hermes-cli-darwin-",
|
||||
"format": "tar.gz",
|
||||
"path": "hermes"
|
||||
},
|
||||
...
|
||||
```
|
||||
|
||||
By default, `dotslash-publish-release` generates both the HTTP provider as well
|
||||
as the `github-release` provider for each entry in the DotSlash file. Either of
|
||||
these can be disabled via top-level `"exclude-http-provider"` and
|
||||
`"exclude-github-release-provider"` properties, respectively. For example, if
|
||||
you are using this action in a private GitHub repo, then you probably want to
|
||||
disable the HTTP provider:
|
||||
|
||||
```json
|
||||
{
|
||||
"exclude-http-provider": false,
|
||||
"outputs": {
|
||||
"hermes": {
|
||||
"platforms": {
|
||||
"macos-x86_64": {
|
||||
"regex": "^hermes-cli-darwin-",
|
||||
"format": "tar.gz",
|
||||
"path": "hermes"
|
||||
},
|
||||
...
|
||||
```
|
||||
|
||||
The generated DotSlash file would reflect this change:
|
||||
|
||||
```json
|
||||
#!/usr/bin/env dotslash
|
||||
|
||||
{
|
||||
"name": "hermes",
|
||||
"platforms": {
|
||||
"macos-x86_64": {
|
||||
"size": 10600817,
|
||||
"hash": "blake3",
|
||||
"digest": "25f984911f199f9229ca0327c52700fa9a8db9aefe95e84f91ba6be69902436a",
|
||||
"format": "zst",
|
||||
"path": "hermes",
|
||||
"providers": [
|
||||
{
|
||||
"type": "github-release",
|
||||
"repo": "https://github.com/facebook/hermes",
|
||||
"tag": "v0.12.0",
|
||||
"name": "hermes-cli-darwin-v0.12.0.tar.gz"
|
||||
}
|
||||
]
|
||||
},
|
||||
...
|
||||
```
|
||||
|
||||
## Config File Details
|
||||
|
||||
The most important part of the config file is the top-level `"outputs"` entry.
|
||||
Each key in this entry will be the name of the generated DotSlash file that is
|
||||
added to the release.
|
||||
|
||||
The `"platforms"` map for each entry requires that the keys are [platforms that
|
||||
are recognized by DotSlash](
|
||||
https://dotslash-cli.com/docs/dotslash-file/).
|
||||
|
||||
Each platform entry recognizes the following properties:
|
||||
|
||||
* One of `regex` or `name` is required to identify the file in the release that
|
||||
should be used as the DotSlash artifact for the platform.
|
||||
* `path` is required and is used as the corresponding `path` value in the
|
||||
DotSlash file.
|
||||
* `format` is optional, but recommended. It must be a valid [DotSlash artifact
|
||||
format](
|
||||
https://dotslash-cli.com/docs/dotslash-file/#artifact-format), such as
|
||||
`tar.gz`. If the artifact is not compressed, then `"format": null` must be
|
||||
specified explicitly in the config JSON.
|
||||
* `hash` must be one of `"blake3"` or `"sha256"`, but it defaults to `"blake3"`,
|
||||
so it is optional.
|
||||
|
||||
## License
|
||||
|
||||
dotslash-publish-release is [MIT licensed](./LICENSE).
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
name: DotSlash files for GitHub releases
|
||||
description: Adds DotSlash files to a GitHub release once artifacts are present
|
||||
inputs:
|
||||
config:
|
||||
description: path to .json file in the repo
|
||||
required: true
|
||||
tag:
|
||||
description: tag identifying the release whose assets should be used
|
||||
required: true
|
||||
runs:
|
||||
using: docker
|
||||
image: Dockerfile
|
||||
args:
|
||||
- '--config'
|
||||
- ${{ inputs.config }}
|
||||
- '--tag'
|
||||
- ${{ inputs.tag }}
|
||||
- '--upload'
|
||||
Executable
+452
@@ -0,0 +1,452 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright (c) Meta Platforms, Inc. and affiliates.
|
||||
#
|
||||
# This source code is licensed under the MIT license found in the
|
||||
# LICENSE file in the root directory of this source tree.
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
from functools import cache
|
||||
from typing import Any, Dict, Literal, Optional, Tuple, Union
|
||||
|
||||
|
||||
HashAlgorithm = Literal["blake3", "sha256"]
|
||||
ArtifactFormat = Literal["gz", "tar", "tar.gz", "tar.zst", "zst"]
|
||||
|
||||
# Recognized properties in the JSON config.
|
||||
OUTPUTS_PARAM = "outputs"
|
||||
EXCLUDE_HTTP_PROVIDER_PARAM = "exclude-http-provider"
|
||||
EXCLUDE_GITHUB_PROVIDER_PARAM = "exclude-github-release-provider"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
exit_code = _main()
|
||||
sys.exit(exit_code)
|
||||
|
||||
|
||||
def _main() -> int:
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s [%(levelname)s] %(message)s",
|
||||
handlers=[logging.StreamHandler()],
|
||||
)
|
||||
|
||||
args = parse_args()
|
||||
repo: str = args.repo
|
||||
if not repo:
|
||||
raise ValueError(
|
||||
"no repo specified: must specify --repo or set the GITHUB_REPOSITORY environment variable"
|
||||
)
|
||||
|
||||
output_folder = args.output
|
||||
if not output_folder:
|
||||
output_folder = tempfile.mkdtemp(prefix=f"{repo.replace('/', '_')}_dotslash")
|
||||
logging.info(f"DotSlash files will be written to `{output_folder}")
|
||||
|
||||
tag = args.tag
|
||||
github_server_url = args.server
|
||||
api_server_url = args.api_server
|
||||
gh_repo_arg = f"{github_server_url}/{repo}"
|
||||
|
||||
if args.local_config:
|
||||
print(args.config)
|
||||
with open(args.config, "r") as f:
|
||||
config = json.load(f)
|
||||
else:
|
||||
config = get_config(
|
||||
path_to_config=args.config,
|
||||
config_ref=args.config_ref,
|
||||
github_repository=repo,
|
||||
api_url=api_server_url,
|
||||
)
|
||||
if not isinstance(config, dict):
|
||||
logging.error(f"config should be a dict, but was:")
|
||||
logging.error(json.dumps(config, indent=2))
|
||||
return 1
|
||||
|
||||
outputs = config.get(OUTPUTS_PARAM)
|
||||
if not outputs:
|
||||
logging.error(f"no {OUTPUTS_PARAM} specified in config:")
|
||||
logging.error(json.dumps(config, indent=2))
|
||||
return 1
|
||||
|
||||
exclude_http_provider = config.get(EXCLUDE_HTTP_PROVIDER_PARAM, False)
|
||||
if not isinstance(exclude_http_provider, bool):
|
||||
logging.error(
|
||||
f'"{EXCLUDE_HTTP_PROVIDER_PARAM}" field must be a boolean, but was `{exclude_http_provider}`'
|
||||
)
|
||||
return 1
|
||||
exclude_github_release_provider = config.get(EXCLUDE_GITHUB_PROVIDER_PARAM, False)
|
||||
if not isinstance(exclude_github_release_provider, bool):
|
||||
logging.error(
|
||||
f'"{EXCLUDE_GITHUB_PROVIDER_PARAM}" field must be a boolean, but was `{exclude_github_release_provider}`'
|
||||
)
|
||||
return 1
|
||||
|
||||
logging.info("using config:")
|
||||
logging.info(json.dumps(config, indent=2))
|
||||
|
||||
name_to_asset = get_release_assets(tag=tag, github_repository=repo)
|
||||
logging.info(json.dumps(name_to_asset, indent=2))
|
||||
|
||||
for output_filename, output_config in outputs.items():
|
||||
platform_entries = map_platforms(output_config, name_to_asset)
|
||||
if not isinstance(platform_entries, dict):
|
||||
logging.error(f"failed with error type {platform_entries}")
|
||||
return 1
|
||||
|
||||
logging.info(json.dumps(platform_entries, indent=2))
|
||||
|
||||
manifest_file_contents = generate_manifest_file(
|
||||
output_filename,
|
||||
gh_repo_arg,
|
||||
tag,
|
||||
platform_entries,
|
||||
include_http_provider=not exclude_http_provider,
|
||||
include_github_release_provider=not exclude_github_release_provider,
|
||||
)
|
||||
logging.info(manifest_file_contents)
|
||||
|
||||
output_file = os.path.join(output_folder, output_filename)
|
||||
with open(output_file, "w") as f:
|
||||
f.write(manifest_file_contents)
|
||||
logging.info(f"wrote manifest to {output_file}")
|
||||
|
||||
if args.upload:
|
||||
# Upload manifest to release, but do not clobber. Note that this may
|
||||
# fail if this action has been called more than once for the same config.
|
||||
subprocess.run(
|
||||
[
|
||||
"gh",
|
||||
"release",
|
||||
"upload",
|
||||
tag,
|
||||
output_file,
|
||||
"--repo",
|
||||
gh_repo_arg,
|
||||
]
|
||||
)
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
def generate_manifest_file(
|
||||
name: str,
|
||||
gh_repo_arg: str,
|
||||
tag: str,
|
||||
platform_entries,
|
||||
include_http_provider: bool,
|
||||
include_github_release_provider: bool,
|
||||
) -> str:
|
||||
platforms = {}
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
for platform_name, platform_entry in platform_entries.items():
|
||||
asset, platform_config = platform_entry
|
||||
hash_algo = platform_config.get("hash", "blake3")
|
||||
size = asset.get("size")
|
||||
if size is None:
|
||||
logging.error(f"missing 'size' field in asset: {asset}")
|
||||
return 1
|
||||
|
||||
asset_name = asset.get("name")
|
||||
if asset_name is None:
|
||||
logging.error(f"missing 'name' field in asset: {asset}")
|
||||
return 1
|
||||
|
||||
path = platform_config.get("path")
|
||||
if not path:
|
||||
logging.error(f"missing `path` field in asset: {asset}")
|
||||
return 1
|
||||
|
||||
if "format" in platform_config:
|
||||
# If the user is knowingly not using any sort of compression,
|
||||
# then `"format": null` must be explicitly specified in the JSON.
|
||||
asset_format = platform_config["format"]
|
||||
else:
|
||||
asset_format = guess_artifact_format_from_asset_name(asset_name)
|
||||
if not asset_format:
|
||||
logging.error(
|
||||
f'"format" could not be inferred from asset name: {asset_name} in {asset}, must specify explicitly'
|
||||
)
|
||||
return 1
|
||||
|
||||
hash_hex = compute_hash(
|
||||
gh_repo_arg, temp_dir, tag, asset_name, hash_algo, size
|
||||
)
|
||||
|
||||
providers = []
|
||||
if include_http_provider:
|
||||
providers.append(
|
||||
{
|
||||
"url": asset["url"],
|
||||
}
|
||||
)
|
||||
if include_github_release_provider:
|
||||
providers.append(
|
||||
{
|
||||
"type": "github-release",
|
||||
"repo": gh_repo_arg,
|
||||
"tag": tag,
|
||||
"name": asset_name,
|
||||
}
|
||||
)
|
||||
|
||||
artifact_entry = {
|
||||
"size": size,
|
||||
"hash": hash_algo,
|
||||
"digest": hash_hex,
|
||||
"format": asset_format,
|
||||
"path": path,
|
||||
"providers": providers,
|
||||
}
|
||||
|
||||
# If `"format": null` was specified, there should not be a "format"
|
||||
# field in the arifact entry.
|
||||
if not asset_format:
|
||||
del artifact_entry["format"]
|
||||
|
||||
platforms[platform_name] = artifact_entry
|
||||
|
||||
manifest = {
|
||||
"name": name,
|
||||
"platforms": platforms,
|
||||
}
|
||||
|
||||
return f"""#!/usr/bin/env dotslash
|
||||
|
||||
{json.dumps(manifest, indent=2)}
|
||||
"""
|
||||
|
||||
|
||||
def map_platforms(
|
||||
config, name_to_asset: Dict[str, Any]
|
||||
) -> Union[
|
||||
Dict[str, Tuple[Any, Any]],
|
||||
Literal["BothNameAndRegex", "NeitherNameNorRegex", "NoMatchForAsset", "ParseError"],
|
||||
]:
|
||||
"""Attempts to take every platform specified in the config and return a map
|
||||
of platform names to their corresponding asset information. If successful,
|
||||
each value in the dict will be a tuple of (asset, platform_config).
|
||||
|
||||
Note that it is possible that not all assets have been uploaded yet, in
|
||||
which case "NoMatchForAsset" will be returned.
|
||||
"""
|
||||
platforms = config.get("platforms")
|
||||
if platforms is None:
|
||||
logging.error("'platforms' field missing from config: {config}")
|
||||
return "ParseError"
|
||||
|
||||
platform_entries = {}
|
||||
for platform, platform_config in platforms.items():
|
||||
name = platform_config.get("name")
|
||||
name_regex = platform_config.get("regex")
|
||||
if name and name_regex:
|
||||
logging.error(
|
||||
f"only one of 'name' and 'regex' should be specified for {platform}"
|
||||
)
|
||||
return "BothNameAndRegex"
|
||||
elif not name and not name_regex:
|
||||
logging.error(
|
||||
f"exactly one of 'name' and 'regex' should be specified for {platform}"
|
||||
)
|
||||
return "NeitherNameNorRegex"
|
||||
|
||||
if name:
|
||||
# Try to match the name exactly:
|
||||
for asset_name, asset in name_to_asset.items():
|
||||
if asset_name == name:
|
||||
platform_entries[platform] = (asset, platform_config)
|
||||
break
|
||||
if platform in platform_entries:
|
||||
continue
|
||||
else:
|
||||
logging.error(f"could not find asset with name '{name}'")
|
||||
return "NoMatchForAsset"
|
||||
else:
|
||||
# Try to match the name using a regular expression.
|
||||
regex = re.compile(name_regex)
|
||||
for asset_name, asset in name_to_asset.items():
|
||||
if regex.match(asset_name):
|
||||
platform_entries[platform] = (asset, platform_config)
|
||||
break
|
||||
if platform in platform_entries:
|
||||
continue
|
||||
else:
|
||||
logging.error(f"could not find asset matching regex '{name_regex}'")
|
||||
return "NoMatchForAsset"
|
||||
|
||||
return platform_entries
|
||||
|
||||
|
||||
@cache
|
||||
def compute_hash(
|
||||
gh_repo_arg: str,
|
||||
temp_dir: str,
|
||||
tag: str,
|
||||
name: str,
|
||||
hash_algo: HashAlgorithm,
|
||||
size: int,
|
||||
) -> str:
|
||||
"""Fetches the release entry corresponding to the specified (tag, name) tuple,
|
||||
fetches the contents, verifies the size matches, and computes the hash.
|
||||
|
||||
Return value is a hex string representing the hash.
|
||||
"""
|
||||
output_filename = os.path.join(temp_dir, name)
|
||||
|
||||
# Fetch the url using the gh CLI to ensure authentication is handled correctly.
|
||||
args = [
|
||||
"gh",
|
||||
"release",
|
||||
"download",
|
||||
tag,
|
||||
"--repo",
|
||||
gh_repo_arg,
|
||||
# --pattern takes a "glob pattern", though we want to match an exact
|
||||
# filename. Using re.escape() seems to do the right thing, though adding
|
||||
# ^ and $ appears to break things.
|
||||
"--pattern",
|
||||
re.escape(name),
|
||||
"--output",
|
||||
output_filename,
|
||||
]
|
||||
subprocess.run(args, check=True)
|
||||
stats = os.stat(output_filename)
|
||||
if stats.st_size != size:
|
||||
raise Exception(f"expected size {size} for {name} but got {stats.st_size}")
|
||||
|
||||
if hash_algo == "blake3":
|
||||
import blake3
|
||||
|
||||
hasher = blake3.blake3()
|
||||
with open(output_filename, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(4096), b""):
|
||||
hasher.update(chunk)
|
||||
digest = hasher.digest()
|
||||
return digest.hex()
|
||||
elif hash_algo == "sha256":
|
||||
import hashlib
|
||||
|
||||
hasher = hashlib.sha256()
|
||||
with open(output_filename, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(4096), b""):
|
||||
hasher.update(chunk)
|
||||
return hasher.hexdigest()
|
||||
|
||||
|
||||
def get_config(
|
||||
*, path_to_config: str, config_ref: str, github_repository: str, api_url: str
|
||||
) -> Any:
|
||||
args = [
|
||||
"gh",
|
||||
"api",
|
||||
"-X",
|
||||
"GET",
|
||||
f"{api_url}/repos/{github_repository}/contents/{path_to_config}",
|
||||
"-H",
|
||||
"Accept: application/vnd.github.raw",
|
||||
"-f",
|
||||
f"ref={config_ref}",
|
||||
]
|
||||
output = subprocess.check_output(args)
|
||||
return json.loads(output.decode("utf-8"))
|
||||
|
||||
|
||||
def get_release_assets(*, tag: str, github_repository) -> Dict[str, Any]:
|
||||
args = [
|
||||
"gh",
|
||||
"release",
|
||||
"view",
|
||||
tag,
|
||||
"--repo",
|
||||
github_repository,
|
||||
"--json",
|
||||
"assets",
|
||||
]
|
||||
output = subprocess.check_output(args)
|
||||
release_data = json.loads(output.decode("utf-8"))
|
||||
assets = release_data.get("assets")
|
||||
if not assets:
|
||||
raise Exception(f"no assets found for release '{tag}'")
|
||||
return {asset["name"]: asset for asset in assets if asset["state"] == "uploaded"}
|
||||
|
||||
|
||||
def guess_artifact_format_from_asset_name(asset_name: str) -> Optional[ArtifactFormat]:
|
||||
if asset_name.endswith(".tar.gz") or asset_name.endswith(".tgz"):
|
||||
return "tar.gz"
|
||||
if asset_name.endswith(".tar.zst") or asset_name.endswith(".tzst"):
|
||||
return "tar.zst"
|
||||
elif asset_name.endswith(".tar"):
|
||||
return "tar"
|
||||
elif asset_name.endswith(".gz"):
|
||||
return "gz"
|
||||
elif asset_name.endswith(".zst"):
|
||||
return "zst"
|
||||
else:
|
||||
return None
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Generate DotSlash files for a GitHub release"
|
||||
)
|
||||
|
||||
parser.add_argument("--tag", required=True, help="tag identifying the release")
|
||||
parser.add_argument("--config", required=True, help="path to JSON config file")
|
||||
parser.add_argument(
|
||||
"--local-config",
|
||||
action="store_true",
|
||||
help="if specified, --config is treated as a local path and --config-ref is ignored",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--repo",
|
||||
help="github repo specified in `ORG/REPO` format",
|
||||
default=os.getenv("GITHUB_REPOSITORY"),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--upload",
|
||||
action="store_true",
|
||||
help="if specified, upload the generated DotSlash files to the release",
|
||||
)
|
||||
|
||||
# It would make things slightly easier for the user to default to the
|
||||
# default branch of the repo, which might not be main.
|
||||
default_config_ref = "main"
|
||||
parser.add_argument(
|
||||
"--config-ref",
|
||||
help=f"SHA of Git commit to look up the config, defaults to {default_config_ref}",
|
||||
default=os.getenv("GITHUB_SHA", default_config_ref),
|
||||
)
|
||||
|
||||
default_server = "https://github.com"
|
||||
parser.add_argument(
|
||||
"--server",
|
||||
help=f"URL for the GitHub server, defaults to {default_server}",
|
||||
default=os.getenv("GITHUB_SERVER_URL", default_server),
|
||||
)
|
||||
|
||||
default_api_server = "https://api.github.com"
|
||||
parser.add_argument(
|
||||
"--api-server",
|
||||
help=f"URL for the GitHub API server, defaults to {default_api_server}",
|
||||
default=os.getenv("GITHUB_API_URL", default_api_server),
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
help=f"folder where DotSlash files should be written, defaults to $GITHUB_WORKSPACE",
|
||||
default=os.getenv("GITHUB_WORKSPACE"),
|
||||
)
|
||||
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user