feat(ext): prefer Cloudflare R2 CDN and allow separate ext pack versions
Download order is CDN (ext.marqdo.com) → GitHub → proxy; pack SemVer from ext/VERSION / CDN latest; add upload-ext-r2.py and release-skill notes. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
1 parent
6324036e41
commit
2984d7d4f8
9 files changed
+442
-38
No files matched your search
@@ -27,6 +27,8 @@ Canonical release playbook for **cflmy/marqdo**. Read this skill **before** tagg
|
||||
6. Prefer **tag push → GitHub Actions** (`.github/workflows/release.yml`) for **both** Windows and **Linux** install assets. Local packaging is fallback / verification.
|
||||
7. After network errors: apply [reference.md § Proxy](reference.md). **First try** `https://proxy.cflmy.top/github.com/cflmy/marqdo.git` (this clone’s usual `origin`). Retry; do not silently skip uploads. If the reverse-proxy still drops long pushes, use **HK jump** (`hk.cflmy.de`) via `scripts/push-via-hk-jump.py` — see reference § **Proxy** / **HK SSH jump**.
|
||||
8. **Do not ship a GitHub Release without the Linux extension/native zip.** Job `linux` must attach `marqdo-VER-native-x86_64-unknown-linux-gnu.zip` (and the Linux CLI bundle). Missing Linux packages = release incomplete.
|
||||
9. **After publishing GitHub assets that include ext/native zips, upload the same packs to Cloudflare R2** (`https://ext.marqdo.com`) via `scripts/upload-ext-r2.py` so `marqdo ext add` prefers CDN. Credentials live only in `~/.marqdo/r2.env` / CI secrets — **never** commit. See [ext-cdn.md](../../../doc/design/ext-cdn.md).
|
||||
10. **CLI and extension packs may version independently** (`ext/VERSION` / CDN `latest/VERSION` vs `Cargo.toml`). Ext-only releases use tag `ext-vX.Y.Z` (or attach assets without bumping CLI).
|
||||
|
||||
## Defaults (after version is confirmed)
|
||||
|
||||
@@ -160,10 +162,29 @@ Local Windows fallback: `scripts/release-full.ps1 -Tag TAG -Upload` (see script
|
||||
## Phase 6 — Post-release
|
||||
|
||||
1. `gh release view TAG` — confirm assets + notes.
|
||||
2. `git status` clean; `main` synced.
|
||||
3. Tell user: release URL `https://github.com/cflmy/marqdo/releases/tag/TAG`.
|
||||
4. Optional: `./scripts/deploy-public.ps1` / pages workflow if user wants user-site refresh.
|
||||
5. Open fresh `## Unreleased` already done in Phase 2.
|
||||
2. **Upload extension packs to R2 CDN** (required when shipping ext/native zips):
|
||||
```bash
|
||||
# credentials: ~/.marqdo/r2.env (never commit) — see doc/design/ext-cdn.md
|
||||
python3 -m pip install --user boto3
|
||||
python3 scripts/upload-ext-r2.py --version VER --from-dir dist/
|
||||
# verify: curl -fsS https://ext.marqdo.com/latest/VERSION
|
||||
```
|
||||
Prefer downloading CI artifacts into `dist/` if local build skipped Linux/Windows natives.
|
||||
3. `git status` clean; `main` synced.
|
||||
4. Tell user: release URL `https://github.com/cflmy/marqdo/releases/tag/TAG` **and** CDN `https://ext.marqdo.com/vVER/`.
|
||||
5. Optional: `./scripts/deploy-public.ps1` / pages workflow if user wants user-site refresh.
|
||||
6. Open fresh `## Unreleased` already done in Phase 2.
|
||||
|
||||
## Extension-only release (no CLI bump)
|
||||
|
||||
When only `ext/` / native plugins change:
|
||||
|
||||
1. Ask user for **ext pack** SemVer (may differ from `Cargo.toml`).
|
||||
2. Bump `ext/VERSION`; update CHANGELOG Unreleased → `## Ext pack vVER`.
|
||||
3. Build/package `marqdo-VER-ext.zip` + `marqdo-VER-native-*.zip` (CI or local).
|
||||
4. Upload to **R2** (`upload-ext-r2.py`) **and** GitHub (tag `ext-vVER` or attach to latest CLI release).
|
||||
5. Do **not** retag CLI `vX.Y.Z` unless Cargo also bumps.
|
||||
6. Detail: [ext-cdn.md](../../../doc/design/ext-cdn.md).
|
||||
|
||||
## Failure matrix (short)
|
||||
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
# Ext CDN / R2 (Cloudflare) — copy to ~/.marqdo/r2.env and chmod 600
|
||||
# Never commit real keys. Public downloads: https://ext.marqdo.com
|
||||
|
||||
R2_ACCOUNT_ID=
|
||||
R2_BUCKET=marqdo
|
||||
R2_ENDPOINT=https://<accountid>.r2.cloudflarestorage.com
|
||||
R2_ACCESS_KEY_ID=
|
||||
R2_SECRET_ACCESS_KEY=
|
||||
R2_PUBLIC_BASE=https://ext.marqdo.com
|
||||
@@ -25,6 +25,9 @@
|
||||
**/.marqdo/
|
||||
.marqdo-ext/
|
||||
.env
|
||||
.env.r2
|
||||
*.pem
|
||||
**/r2.env
|
||||
|
||||
# Generated HTML beside tracked public/*.mq.md (sources stay in git)
|
||||
/public/index.html
|
||||
|
||||
@@ -3,12 +3,14 @@
|
||||
## Unreleased
|
||||
|
||||
### Added
|
||||
- **Extension CDN** (`https://ext.marqdo.com` / Cloudflare R2):`ext add` 下载顺序改为 CDN → GitHub → proxy;独立包版本见 `ext/VERSION` 与 CDN `latest/VERSION`;上传脚本 `scripts/upload-ext-r2.py`(凭据 `~/.marqdo/r2.env`)。设计见 [ext-cdn.md](doc/design/ext-cdn.md)。
|
||||
- `ext add` 下载:GitHub 失败后自动回退 `https://proxy.cflmy.top/github.com/…`;可用 `MARQDO_EXT_DOWNLOAD_BASE` 指定首选镜像;加长读写超时。
|
||||
|
||||
### Fixed
|
||||
- `ext/` 导入默认搜索 `~/.marqdo/ext`(与 `marqdo ext add` 安装根一致),无需再手动 `export MARQDO_EXT`。
|
||||
|
||||
### Changed
|
||||
- 允许 **CLI 与扩展包分开发版**:扩展包 SemVer 可与 `Cargo.toml` 不同(`MARQDO_EXT_VERSION` / `ext/VERSION` / CDN latest)。
|
||||
|
||||
## v1.0.2 — 2026-09-16
|
||||
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
# Extension CDN (Cloudflare R2 → https://ext.marqdo.com)
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | **Accepted** — download prefer CDN; upload via `scripts/upload-ext-r2.py` |
|
||||
| Date | 2026-09-17 |
|
||||
| Related | [ext-cli.md](ext-cli.md) · [marqdo-release](../../.cursor/skills/marqdo-release/SKILL.md) |
|
||||
|
||||
## Why
|
||||
|
||||
CLI (语法解释器) and official **extension packs** (`ext/` L1 + native plugins) update on different cadences. Users behind unstable GitHub links should get packs from a stable object store first.
|
||||
|
||||
## Public URL layout
|
||||
|
||||
Bucket: `marqdo` · Custom domain: **https://ext.marqdo.com**
|
||||
|
||||
| Key | Meaning |
|
||||
|-----|---------|
|
||||
| `v{VER}/marqdo-{VER}-ext.zip` | L1 sources |
|
||||
| `v{VER}/marqdo-{VER}-native-{triple}.zip` | Prebuilt plugins |
|
||||
| `v{VER}/VERSION` | Pack SemVer echo |
|
||||
| `latest/VERSION` | Recommended pack SemVer (plain text) |
|
||||
| `latest/marqdo-…zip` | Same assets mirrored for convenience |
|
||||
|
||||
`{VER}` is the **extension pack** version (`ext/VERSION`), which **may differ** from CLI `Cargo.toml` version.
|
||||
|
||||
## Client download order (`marqdo ext add`)
|
||||
|
||||
1. `MARQDO_EXT_DOWNLOAD_BASE` (optional override)
|
||||
2. **CDN** `https://ext.marqdo.com/v{VER}/…` (override host with `MARQDO_EXT_CDN`)
|
||||
3. GitHub Releases
|
||||
4. `https://proxy.cflmy.top/github.com/…`
|
||||
|
||||
Pack SemVer resolution (`release_version()`):
|
||||
|
||||
1. `MARQDO_EXT_VERSION`
|
||||
2. CDN `GET …/latest/VERSION` (short timeout)
|
||||
3. Embedded `ext/VERSION` at build time
|
||||
4. CLI `CARGO_PKG_VERSION`
|
||||
|
||||
## Separate ext release (no CLI bump)
|
||||
|
||||
1. Bump `ext/VERSION` (e.g. `1.0.3`) — leave root `Cargo.toml` alone if CLI unchanged.
|
||||
2. Build native zips (CI Linux/Windows jobs or local `scripts/build-web-plugin.sh` + cargo plugins).
|
||||
3. Package `marqdo-{VER}-ext.zip` and `marqdo-{VER}-native-*.zip`.
|
||||
4. Upload **both**:
|
||||
- R2: `python3 scripts/upload-ext-r2.py --version VER --from-dir dist/`
|
||||
- GitHub: attach to a GitHub Release (tag `ext-vVER` **or** reuse latest CLI release assets)
|
||||
5. Confirm `https://ext.marqdo.com/latest/VERSION` and a sample zip URL return 200.
|
||||
|
||||
## Full product release (CLI + ext)
|
||||
|
||||
Keep `ext/VERSION` equal to (or compatible with) CLI SemVer unless documenting a deliberate skew. CI still publishes GitHub assets; **after** CI green, run `upload-ext-r2.py` so CDN is populated (or add a CI step with R2 secrets).
|
||||
|
||||
## Credentials
|
||||
|
||||
| Where | What |
|
||||
|-------|------|
|
||||
| Local | `~/.marqdo/r2.env` (`chmod 600`) — see `.env.r2.example` |
|
||||
| CI (optional) | GitHub Actions secrets: `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `R2_ENDPOINT`, `R2_BUCKET` |
|
||||
|
||||
**Never** commit Account API tokens or S3 secret keys. If a token was pasted into chat, **rotate** it in the Cloudflare dashboard.
|
||||
|
||||
## Object ACL
|
||||
|
||||
Custom domain public reads require the bucket/objects to be publicly readable (R2 public bucket / r2.dev / custom domain). Uploads use S3 API credentials; downloads by `ext add` are anonymous HTTPS GETs.
|
||||
@@ -51,12 +51,15 @@ L1 sources:
|
||||
|
||||
| Variable | Meaning |
|
||||
|----------|---------|
|
||||
| `MARQDO_EXT_VERSION` | Override SemVer used for Release asset names (default: CLI `CARGO_PKG_VERSION`) |
|
||||
| `MARQDO_EXT_VERSION` | Override **extension pack** SemVer (default: CDN `latest/VERSION` → embedded `ext/VERSION` → CLI version) |
|
||||
| `MARQDO_EXT_CDN` | Public CDN base (default `https://ext.marqdo.com`) |
|
||||
| `MARQDO_EXT_NO_DOWNLOAD=1` | Never hit the network (CI / air-gap); local/cargo only |
|
||||
| `MARQDO_EXT_DOWNLOAD_BASE` | Optional mirror base tried **first** (e.g. `https://proxy.cflmy.top/github.com` or a flat asset dir) |
|
||||
| `MARQDO_EXT_DOWNLOAD_BASE` | Optional extra base tried **before** CDN |
|
||||
| `MARQDO_WEB_PLUGIN` / `MARQDO_AGENT_PLUGIN` / `MARQDO_QUANTUM_PLUGIN` | Explicit native path |
|
||||
|
||||
Download order for each Release asset: `MARQDO_EXT_DOWNLOAD_BASE` (if set) → `github.com` → **`https://proxy.cflmy.top/github.com/…`** fallback. Connect timeout 20s; read timeout 10min so large native zips can finish on slow links.
|
||||
Download order for each Release asset: `MARQDO_EXT_DOWNLOAD_BASE` → **CDN `ext.marqdo.com`** → `github.com` → **`proxy.cflmy.top`**. Connect timeout 20s; read timeout 10min.
|
||||
|
||||
See [ext-cdn.md](ext-cdn.md) for R2 layout, separate ext releases, and `scripts/upload-ext-r2.py`.
|
||||
|
||||
Windows Release **zip** also ships `ext/` + `ext/native/*.dll` next to `marqdo.exe` so a portable unzip works offline after one download.
|
||||
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
1.0.2
|
||||
Executable
+222
@@ -0,0 +1,222 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Upload Marqdo extension packs to Cloudflare R2 (CDN: https://ext.marqdo.com).
|
||||
|
||||
No third-party deps (stdlib only) — AWS Signature V4 PutObject.
|
||||
|
||||
Credentials: ~/.marqdo/r2.env (chmod 600; never commit). Required:
|
||||
R2_ENDPOINT, R2_BUCKET, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY
|
||||
Optional: R2_PUBLIC_BASE, R2_REGION (default auto)
|
||||
|
||||
Usage:
|
||||
python3 scripts/upload-ext-r2.py --version 1.0.2 dist/marqdo-1.0.2-ext.zip \\
|
||||
dist/marqdo-1.0.2-native-x86_64-unknown-linux-gnu.zip
|
||||
python3 scripts/upload-ext-r2.py --version 1.0.2 --from-dir dist/
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import hashlib
|
||||
import hmac
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from urllib.parse import quote, urlparse
|
||||
|
||||
|
||||
def load_env_file(path: Path) -> None:
|
||||
if not path.is_file():
|
||||
return
|
||||
for line in path.read_text(encoding="utf-8").splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
k, _, v = line.partition("=")
|
||||
os.environ.setdefault(k.strip(), v.strip().strip("'").strip('"'))
|
||||
|
||||
|
||||
def sha256_hex(data: bytes) -> str:
|
||||
return hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def hmac_sha256(key: bytes, msg: str) -> bytes:
|
||||
return hmac.new(key, msg.encode("utf-8"), hashlib.sha256).digest()
|
||||
|
||||
|
||||
def signing_key(secret: str, datestamp: str, region: str, service: str) -> bytes:
|
||||
k_date = hmac_sha256(("AWS4" + secret).encode("utf-8"), datestamp)
|
||||
k_region = hmac.new(k_date, region.encode("utf-8"), hashlib.sha256).digest()
|
||||
k_service = hmac.new(k_region, service.encode("utf-8"), hashlib.sha256).digest()
|
||||
return hmac.new(k_service, b"aws4_request", hashlib.sha256).digest()
|
||||
|
||||
|
||||
def put_object(
|
||||
*,
|
||||
endpoint: str,
|
||||
bucket: str,
|
||||
key: str,
|
||||
body: bytes,
|
||||
content_type: str,
|
||||
access_key: str,
|
||||
secret_key: str,
|
||||
region: str,
|
||||
) -> None:
|
||||
endpoint = endpoint.rstrip("/")
|
||||
host = urlparse(endpoint).netloc
|
||||
# Path-style: https://endpoint/bucket/key
|
||||
canonical_uri = "/" + bucket + "/" + quote(key, safe="/~")
|
||||
amz_date = dt.datetime.now(dt.timezone.utc).strftime("%Y%m%dT%H%M%SZ")
|
||||
datestamp = amz_date[:8]
|
||||
payload_hash = sha256_hex(body)
|
||||
canonical_headers = (
|
||||
f"content-type:{content_type}\n"
|
||||
f"host:{host}\n"
|
||||
f"x-amz-content-sha256:{payload_hash}\n"
|
||||
f"x-amz-date:{amz_date}\n"
|
||||
)
|
||||
signed_headers = "content-type;host;x-amz-content-sha256;x-amz-date"
|
||||
canonical_request = "\n".join(
|
||||
[
|
||||
"PUT",
|
||||
canonical_uri,
|
||||
"",
|
||||
canonical_headers,
|
||||
signed_headers,
|
||||
payload_hash,
|
||||
]
|
||||
)
|
||||
credential_scope = f"{datestamp}/{region}/s3/aws4_request"
|
||||
string_to_sign = "\n".join(
|
||||
[
|
||||
"AWS4-HMAC-SHA256",
|
||||
amz_date,
|
||||
credential_scope,
|
||||
sha256_hex(canonical_request.encode("utf-8")),
|
||||
]
|
||||
)
|
||||
sig = hmac.new(
|
||||
signing_key(secret_key, datestamp, region, "s3"),
|
||||
string_to_sign.encode("utf-8"),
|
||||
hashlib.sha256,
|
||||
).hexdigest()
|
||||
auth = (
|
||||
f"AWS4-HMAC-SHA256 Credential={access_key}/{credential_scope}, "
|
||||
f"SignedHeaders={signed_headers}, Signature={sig}"
|
||||
)
|
||||
url = endpoint + canonical_uri
|
||||
req = urllib.request.Request(url, data=body, method="PUT")
|
||||
req.add_header("Content-Type", content_type)
|
||||
req.add_header("x-amz-content-sha256", payload_hash)
|
||||
req.add_header("x-amz-date", amz_date)
|
||||
req.add_header("Authorization", auth)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=300) as resp:
|
||||
if resp.status not in (200, 201):
|
||||
raise RuntimeError(f"PUT {key} HTTP {resp.status}")
|
||||
except urllib.error.HTTPError as e:
|
||||
detail = e.read().decode("utf-8", errors="replace")
|
||||
raise RuntimeError(f"PUT {key} HTTP {e.code}: {detail}") from e
|
||||
|
||||
|
||||
def content_type_for(path: Path) -> str:
|
||||
if path.name == "VERSION" or path.suffix.lower() == ".txt":
|
||||
return "text/plain; charset=utf-8"
|
||||
if path.suffix.lower() == ".json":
|
||||
return "application/json"
|
||||
return "application/zip"
|
||||
|
||||
|
||||
def upload_file(
|
||||
path: Path,
|
||||
key: str,
|
||||
*,
|
||||
endpoint: str,
|
||||
bucket: str,
|
||||
access_key: str,
|
||||
secret_key: str,
|
||||
region: str,
|
||||
public_base: str,
|
||||
) -> None:
|
||||
body = path.read_bytes()
|
||||
print(f"upload {path.name} → s3://{bucket}/{key} ({len(body)} bytes)")
|
||||
put_object(
|
||||
endpoint=endpoint,
|
||||
bucket=bucket,
|
||||
key=key,
|
||||
body=body,
|
||||
content_type=content_type_for(path),
|
||||
access_key=access_key,
|
||||
secret_key=secret_key,
|
||||
region=region,
|
||||
)
|
||||
print(f" public: {public_base.rstrip('/')}/{key}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
load_env_file(Path.home() / ".marqdo" / "r2.env")
|
||||
load_env_file(Path(".env"))
|
||||
load_env_file(Path(".env.r2"))
|
||||
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("--version", required=True, help="Extension pack SemVer (no v prefix)")
|
||||
ap.add_argument("--from-dir", type=Path, help="Upload all marqdo-{VER}-*.zip from dir")
|
||||
ap.add_argument("files", nargs="*", type=Path, help="Files to upload")
|
||||
ap.add_argument("--no-latest", action="store_true", help="Do not update latest/ pointers")
|
||||
args = ap.parse_args()
|
||||
|
||||
ver = args.version.lstrip("v")
|
||||
for key in ("R2_ENDPOINT", "R2_BUCKET", "R2_ACCESS_KEY_ID", "R2_SECRET_ACCESS_KEY"):
|
||||
if not os.environ.get(key):
|
||||
print(
|
||||
f"missing {key}; write ~/.marqdo/r2.env (see .env.r2.example)",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
endpoint = os.environ["R2_ENDPOINT"]
|
||||
bucket = os.environ["R2_BUCKET"]
|
||||
access_key = os.environ["R2_ACCESS_KEY_ID"]
|
||||
secret_key = os.environ["R2_SECRET_ACCESS_KEY"]
|
||||
region = os.environ.get("R2_REGION", "auto")
|
||||
public_base = os.environ.get("R2_PUBLIC_BASE", "https://ext.marqdo.com")
|
||||
|
||||
files: list[Path] = [p for p in args.files if p.is_file()]
|
||||
if args.from_dir:
|
||||
files.extend(sorted(args.from_dir.glob(f"marqdo-{ver}-*.zip")))
|
||||
if not files:
|
||||
print("no files to upload", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
kw = dict(
|
||||
endpoint=endpoint,
|
||||
bucket=bucket,
|
||||
access_key=access_key,
|
||||
secret_key=secret_key,
|
||||
region=region,
|
||||
public_base=public_base,
|
||||
)
|
||||
for path in files:
|
||||
upload_file(path, f"v{ver}/{path.name}", **kw)
|
||||
if not args.no_latest:
|
||||
upload_file(path, f"latest/{path.name}", **kw)
|
||||
|
||||
if not args.no_latest:
|
||||
with tempfile.NamedTemporaryFile("w", encoding="utf-8", delete=False) as tf:
|
||||
tf.write(ver + "\n")
|
||||
tmp = Path(tf.name)
|
||||
try:
|
||||
upload_file(tmp, "latest/VERSION", **kw)
|
||||
upload_file(tmp, f"v{ver}/VERSION", **kw)
|
||||
finally:
|
||||
tmp.unlink(missing_ok=True)
|
||||
|
||||
print("done.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
+108
-31
@@ -1,23 +1,32 @@
|
||||
//! Download official `ext/` L1 sources and native plugins from GitHub Releases.
|
||||
//! Download official `ext/` L1 sources and native plugins.
|
||||
//!
|
||||
//! Enables `marqdo ext add web` without a local Rust toolchain or repo checkout.
|
||||
//! Disable with `MARQDO_EXT_NO_DOWNLOAD=1`. Override tag with `MARQDO_EXT_VERSION=0.3.4`.
|
||||
//!
|
||||
//! When GitHub is unreachable, downloads automatically fall back to the project
|
||||
//! reverse proxy (`https://proxy.cflmy.top/github.com/…`). Override the candidate
|
||||
//! list with `MARQDO_EXT_DOWNLOAD_BASE` (tried first).
|
||||
//! Disable with `MARQDO_EXT_NO_DOWNLOAD=1`. Override pack SemVer with
|
||||
//! `MARQDO_EXT_VERSION` (independent of CLI `CARGO_PKG_VERSION`).
|
||||
//!
|
||||
//! Download order (first success wins):
|
||||
//! 1. `MARQDO_EXT_DOWNLOAD_BASE` (optional override)
|
||||
//! 2. **CDN** `https://ext.marqdo.com` (Cloudflare R2 custom domain)
|
||||
//! 3. GitHub Releases `github.com/cflmy/marqdo`
|
||||
//! 4. GitHub via reverse proxy `proxy.cflmy.top`
|
||||
|
||||
use std::env;
|
||||
use std::fs::{self, File};
|
||||
use std::io::{self, Write};
|
||||
use std::io::{self, Read, Write};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::OnceLock;
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{bail, Context, Result};
|
||||
|
||||
const REPO: &str = "cflmy/marqdo";
|
||||
/// HTTPS reverse proxy used when github.com is slow or blocked (same as git origin).
|
||||
/// Public CDN for extension packs (R2 bucket `marqdo`, custom domain).
|
||||
pub const EXT_CDN_BASE: &str = "https://ext.marqdo.com";
|
||||
/// HTTPS reverse proxy used when github.com is slow or blocked.
|
||||
const GITHUB_MIRROR_PREFIX: &str = "https://proxy.cflmy.top/github.com";
|
||||
/// Packaged with the repo; may diverge from CLI SemVer for ext-only releases.
|
||||
const EMBEDDED_EXT_VERSION: &str = include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/ext/VERSION"));
|
||||
|
||||
pub fn downloads_enabled() -> bool {
|
||||
match env::var("MARQDO_EXT_NO_DOWNLOAD") {
|
||||
@@ -29,17 +38,69 @@ pub fn downloads_enabled() -> bool {
|
||||
}
|
||||
}
|
||||
|
||||
/// SemVer without leading `v` (matches release asset names).
|
||||
fn normalize_semver(raw: &str) -> Option<String> {
|
||||
let v = raw.trim().trim_start_matches('v').trim();
|
||||
if v.is_empty() {
|
||||
return None;
|
||||
}
|
||||
// one line only
|
||||
let v = v.lines().next()?.trim().trim_start_matches('v').trim();
|
||||
if v.is_empty() {
|
||||
return None;
|
||||
}
|
||||
Some(v.to_string())
|
||||
}
|
||||
|
||||
/// SemVer of the **extension pack** (may differ from CLI version).
|
||||
///
|
||||
/// Resolution order:
|
||||
/// 1. `MARQDO_EXT_VERSION`
|
||||
/// 2. CDN `GET {EXT_CDN}/latest/VERSION` (short timeout; skipped if downloads disabled)
|
||||
/// 3. Embedded `ext/VERSION` at build time
|
||||
/// 4. CLI `CARGO_PKG_VERSION`
|
||||
pub fn release_version() -> String {
|
||||
if let Ok(v) = env::var("MARQDO_EXT_VERSION") {
|
||||
let v = v.trim().trim_start_matches('v');
|
||||
if !v.is_empty() {
|
||||
return v.to_string();
|
||||
if let Some(n) = normalize_semver(&v) {
|
||||
return n;
|
||||
}
|
||||
}
|
||||
if downloads_enabled() {
|
||||
if let Some(n) = fetch_cdn_latest_version() {
|
||||
return n;
|
||||
}
|
||||
}
|
||||
if let Some(n) = normalize_semver(EMBEDDED_EXT_VERSION) {
|
||||
return n;
|
||||
}
|
||||
env!("CARGO_PKG_VERSION").to_string()
|
||||
}
|
||||
|
||||
fn fetch_cdn_latest_version() -> Option<String> {
|
||||
static CACHED: OnceLock<Option<String>> = OnceLock::new();
|
||||
CACHED
|
||||
.get_or_init(|| {
|
||||
let base = env::var("MARQDO_EXT_CDN")
|
||||
.ok()
|
||||
.map(|s| s.trim().trim_end_matches('/').to_string())
|
||||
.filter(|s| !s.is_empty())
|
||||
.unwrap_or_else(|| EXT_CDN_BASE.to_string());
|
||||
let url = format!("{base}/latest/VERSION");
|
||||
let agent = ureq::AgentBuilder::new()
|
||||
.timeout_connect(Duration::from_secs(5))
|
||||
.timeout_read(Duration::from_secs(10))
|
||||
.user_agent(&user_agent())
|
||||
.build();
|
||||
let resp = agent.get(&url).call().ok()?;
|
||||
if !(200..300).contains(&resp.status()) {
|
||||
return None;
|
||||
}
|
||||
let mut body = String::new();
|
||||
resp.into_reader().read_to_string(&mut body).ok()?;
|
||||
normalize_semver(&body)
|
||||
})
|
||||
.clone()
|
||||
}
|
||||
|
||||
pub fn host_target_triple() -> Option<&'static str> {
|
||||
if cfg!(all(target_os = "windows", target_arch = "x86_64")) {
|
||||
Some("x86_64-pc-windows-msvc")
|
||||
@@ -76,21 +137,33 @@ fn http_agent() -> ureq::Agent {
|
||||
.build()
|
||||
}
|
||||
|
||||
/// Candidate download URLs: optional `MARQDO_EXT_DOWNLOAD_BASE`, GitHub, then proxy mirror.
|
||||
fn cdn_base() -> String {
|
||||
env::var("MARQDO_EXT_CDN")
|
||||
.ok()
|
||||
.map(|s| s.trim().trim_end_matches('/').to_string())
|
||||
.filter(|s| !s.is_empty())
|
||||
.unwrap_or_else(|| EXT_CDN_BASE.to_string())
|
||||
}
|
||||
|
||||
/// Candidate download URLs for a release asset.
|
||||
///
|
||||
/// Order: optional `MARQDO_EXT_DOWNLOAD_BASE` → CDN → GitHub → GitHub proxy.
|
||||
pub fn release_asset_urls(ver: &str, filename: &str) -> Vec<String> {
|
||||
let mut urls = Vec::new();
|
||||
if let Ok(base) = env::var("MARQDO_EXT_DOWNLOAD_BASE") {
|
||||
let base = base.trim().trim_end_matches('/');
|
||||
if !base.is_empty() {
|
||||
// Flexible bases: full github path prefix, repo root, or flat asset dir.
|
||||
urls.push(format!(
|
||||
"{base}/{REPO}/releases/download/v{ver}/{filename}"
|
||||
));
|
||||
urls.push(format!("{base}/{REPO}/releases/download/v{ver}/{filename}"));
|
||||
urls.push(format!("{base}/releases/download/v{ver}/{filename}"));
|
||||
urls.push(format!("{base}/v{ver}/{filename}"));
|
||||
urls.push(format!("{base}/{filename}"));
|
||||
}
|
||||
}
|
||||
let cdn = cdn_base();
|
||||
// Preferred public layout on ext.marqdo.com / R2
|
||||
urls.push(format!("{cdn}/v{ver}/{filename}"));
|
||||
urls.push(format!("{cdn}/releases/v{ver}/{filename}"));
|
||||
urls.push(format!("{cdn}/{filename}"));
|
||||
urls.push(format!(
|
||||
"https://github.com/{REPO}/releases/download/v{ver}/{filename}"
|
||||
));
|
||||
@@ -132,7 +205,7 @@ fn download_first_ok(urls: &[String], dest: &Path) -> Result<()> {
|
||||
if i == 0 {
|
||||
println!("downloading ({url})…");
|
||||
} else {
|
||||
println!("retry via mirror ({url})…");
|
||||
println!("retry ({url})…");
|
||||
}
|
||||
match download_to_file(url, dest) {
|
||||
Ok(()) => return Ok(()),
|
||||
@@ -194,7 +267,6 @@ pub fn ensure_ext_source_tree() -> Result<PathBuf> {
|
||||
if marker.is_file() {
|
||||
return Ok(cache);
|
||||
}
|
||||
// Zip layout from CI: top-level `ext/…`
|
||||
let nested = cache.join("ext").join("web").join("web.mq.md");
|
||||
if nested.is_file() {
|
||||
return Ok(cache.join("ext"));
|
||||
@@ -211,7 +283,6 @@ pub fn ensure_ext_source_tree() -> Result<PathBuf> {
|
||||
let _ = fs::remove_dir_all(&stage);
|
||||
}
|
||||
extract_zip(&zip_path, &stage)?;
|
||||
// Prefer `ext/` child if present
|
||||
let src = if stage.join("ext").join("web").join("web.mq.md").is_file() {
|
||||
stage.join("ext")
|
||||
} else if stage.join("web").join("web.mq.md").is_file() {
|
||||
@@ -224,7 +295,6 @@ pub fn ensure_ext_source_tree() -> Result<PathBuf> {
|
||||
if cache.exists() {
|
||||
let _ = fs::remove_dir_all(&cache);
|
||||
}
|
||||
// Move/copy tree into stable cache path
|
||||
copy_dir_recursive(&src, &cache)?;
|
||||
let _ = fs::remove_dir_all(&stage);
|
||||
if !cache.join("web").join("web.mq.md").is_file() {
|
||||
@@ -271,7 +341,6 @@ pub fn download_native_plugin(short: &str, lib_name: &str) -> Result<PathBuf> {
|
||||
if lib_in_cache.is_file() {
|
||||
return Ok(lib_in_cache);
|
||||
}
|
||||
// Also accept flat layout
|
||||
let flat = cache.join(lib_name);
|
||||
if flat.is_file() {
|
||||
return Ok(flat);
|
||||
@@ -280,9 +349,8 @@ pub fn download_native_plugin(short: &str, lib_name: &str) -> Result<PathBuf> {
|
||||
download_first_ok(&urls, &zip_path).with_context(|| {
|
||||
format!(
|
||||
"failed to download prebuilt plugins for {triple}. \
|
||||
Set MARQDO_EXT_DOWNLOAD_BASE to a reachable mirror, build locally \
|
||||
(`bash ./scripts/build-web-plugin.sh` for web), \
|
||||
or check https://github.com/{REPO}/releases/tag/v{ver}"
|
||||
Set MARQDO_EXT_CDN / MARQDO_EXT_DOWNLOAD_BASE, build locally, \
|
||||
or check https://github.com/{REPO}/releases and https://ext.marqdo.com/v{ver}/"
|
||||
)
|
||||
})?;
|
||||
if cache.exists() {
|
||||
@@ -297,7 +365,6 @@ pub fn download_native_plugin(short: &str, lib_name: &str) -> Result<PathBuf> {
|
||||
if flat.is_file() {
|
||||
return Ok(flat);
|
||||
}
|
||||
// Search recursively for the lib name
|
||||
if let Some(found) = find_named_file(&cache, lib_name)? {
|
||||
return Ok(found);
|
||||
}
|
||||
@@ -342,21 +409,31 @@ mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn asset_urls_include_github_and_mirror() {
|
||||
fn asset_urls_prefer_cdn_then_github_then_proxy() {
|
||||
let u = release_asset_urls("1.0.2", "marqdo-1.0.2-ext.zip");
|
||||
assert!(u[0].starts_with("https://ext.marqdo.com/"));
|
||||
assert!(u.iter().any(|x| x.contains("github.com/cflmy/marqdo")));
|
||||
assert!(u
|
||||
.iter()
|
||||
.any(|x| x.contains("proxy.cflmy.top/github.com/cflmy/marqdo")));
|
||||
assert_eq!(
|
||||
u[0],
|
||||
"https://github.com/cflmy/marqdo/releases/download/v1.0.2/marqdo-1.0.2-ext.zip"
|
||||
);
|
||||
let gh = u
|
||||
.iter()
|
||||
.position(|x| x.starts_with("https://github.com/"))
|
||||
.unwrap();
|
||||
let proxy = u
|
||||
.iter()
|
||||
.position(|x| x.contains("proxy.cflmy.top"))
|
||||
.unwrap();
|
||||
assert!(gh < proxy);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn embedded_ext_version_parses() {
|
||||
assert!(normalize_semver(EMBEDDED_EXT_VERSION).is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn triple_known_on_ci_hosts() {
|
||||
// At least one of these builds is what we ship.
|
||||
let t = host_target_triple();
|
||||
if cfg!(any(
|
||||
all(target_os = "linux", target_arch = "x86_64"),
|
||||
|
||||
Reference in new issue
Block a user