-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
All extensions show as unsigned by the Extension Marketplace preventing auto installation in VSCode 1.9.4+ #65
Comments
We've experienced the very same issue. The signature seems not to be contained in the actual VSXI package. Instead, the extensionquery-API provides it as a separate asset for a given version. Our solution is to download it separately (we have a mirroring mechanism which uses the extensionquery-API to fetch the version information of the extensions and passes new version assets to This way, VS Code can download the signature and stops complaining. |
Bringing over my notes from #67:
This should also allow adding your own signatures since it will only generate if one does not already exist. |
duplicated by https://github.com/coder/customers/issues/702 |
Hello @janLo Could you give me more information about have you download the signature ? I'm experiencing the same issue with code-server 1.91.1 Thanks. |
@p1r4t3-s4il0r we have a downloader that does it all for us and a bit of infrastructure to use it on the other side. This is the code that downloads a list of extensions and places them into artifactory: DownloaderThe method import asyncio
import os
import tempfile
import orjson
import typing
import aiohttp
import logging
import urllib.parse
import concurrent.futures
import pprint
import subprocess
MARKETPLACE_URL = "https://marketplace.visualstudio.com/_apis/public/gallery/extensionquery?api-version=3.0-preview.1"
_log = logging.getLogger(__name__)
PLATFORMS = {"linux-x64", "web", "alpine-x64", "universal", "unkknown", "undefined"}
class ExtensionInfo(typing.NamedTuple):
extension_id: str
extension_name: str
extension_publisher: str
extension_version: str
targetPlatform: str | None
vsxi_url: str
signature_url: str
def make_filter(extensions: list[str]) -> bytes:
return orjson.dumps(
{
"filters": [
{
"criteria": [{"filterType": 7, "value": ext} for ext in extensions],
"pageNumber": 1,
"pageSize": len(extensions),
"sortBy": 0,
"sortOrder": 0,
}
],
"flags": 147,
}
)
def _filter_pre_release(version: dict) -> bool:
return next(
iter(
prop["value"] != "true"
for prop in version["properties"]
if prop["key"] == "Microsoft.VisualStudio.Code.PreRelease"
),
True,
)
class ManifestMetadata(typing.NamedTuple):
extension_id: str
extension_publisher: str
extension_version: str
def _get_asset_file_url(files: list[dict[str, str]], asset_type: str) -> str | None:
return next(
iter((file["source"] for file in files if file["assetType"] == asset_type)),
None,
)
class PluginDownloader(object):
def __init__(
self,
artifactory_url: str,
artifactory_repo: str,
artifactory_token: str,
code_marketplace_bin: str,
):
self._artifactory_url = artifactory_url
self._artifactory_repo = artifactory_repo
self._artifactory_token = artifactory_token
self._marketplace_url = MARKETPLACE_URL
self._code_marketplace_bin = code_marketplace_bin
self._session: aiohttp.ClientSession | None = None
self._pool = concurrent.futures.ThreadPoolExecutor(max_workers=4)
@property
def session(self) -> aiohttp.ClientSession:
if self._session is None:
self._session = aiohttp.ClientSession(trust_env=True)
return self._session
async def _fech_manifest_metadata(self, manifest_url) -> ManifestMetadata | None:
async with self.session.get(manifest_url) as resp:
if not resp.ok:
_log.warning("Could not fetch manifest metadata: %s", manifest_url)
return None
data = orjson.loads(await resp.text())
return ManifestMetadata(
extension_id=data["name"],
extension_publisher=data["publisher"],
extension_version=data["version"],
)
async def _fetch_marketplace_info(
self, extensions: list[str]
) -> typing.AsyncGenerator[ExtensionInfo, None]:
version_data = []
async with self.session.post(
self._marketplace_url,
headers={"Content-Type": "application/json"},
data=make_filter(extensions),
) as resp:
if not resp.ok:
_log.error("Could not fetch marketplace info for %s", extensions)
data = orjson.loads(await resp.text())
for result in data.get("results", []):
for extension in result.get("extensions", []):
publisher = extension["publisher"]
ext_id = (
f"{publisher["publisherName"]}.{extension['extensionName']}"
)
all_versions = extension.get("versions", [])
first_version = next(
iter(
version
for version in all_versions
if _filter_pre_release(version)
),
None,
)
if first_version is None:
_log.warning("No version found for extension %s", ext_id)
continue
if "targetPlatform" in first_version:
try:
versions = [
version
for version in all_versions
if version["version"] == first_version["version"]
and (
"targetPlatform" not in version
or version["targetPlatform"] in PLATFORMS
)
and _filter_pre_release(version)
]
except KeyError:
_log.exception(
"Version broken for %s?\n%s\n",
ext_id,
pprint.pformat(
[
v
for v in all_versions
if v["version"] == first_version["version"]
],
indent=4,
),
)
raise
else:
versions = [first_version]
_log.debug(
"Found %d versions for extension %s", len(versions), ext_id
)
version_data.extend(versions)
_log.debug("Processing %d version information requests", len(version_data))
pending: set[asyncio.Future[ExtensionInfo | None]] = set()
for version in version_data:
if len(pending) > 6:
finished, pending = await asyncio.wait(
pending, return_when=asyncio.FIRST_COMPLETED
)
while finished:
res = await finished.pop()
if res is not None:
yield res
pending.add(asyncio.ensure_future(self._fetch_version_info(version)))
while pending:
finished, pending = await asyncio.wait(
pending, return_when=asyncio.FIRST_COMPLETED
)
while finished:
res = await finished.pop()
if res is not None:
yield res
async def _fetch_version_info(
self, version: dict[str, typing.Any]
) -> ExtensionInfo | None:
files = version["files"]
manifest = _get_asset_file_url(files, "Microsoft.VisualStudio.Code.Manifest")
if manifest is None:
_log.warning("No manifest url for extension")
return None
metadata = await self._fech_manifest_metadata(manifest)
if metadata is None:
return None
ext_id = f"{metadata.extension_publisher}.{metadata.extension_id}"
vsxi = _get_asset_file_url(files, "Microsoft.VisualStudio.Services.VSIXPackage")
if vsxi is None:
_log.warning("No vsxi package found for extension %s", ext_id)
return None
sig = _get_asset_file_url(
files, "Microsoft.VisualStudio.Services.VsixSignature"
)
if sig is None:
_log.warning("No signature found for extension %s", ext_id)
return None
return ExtensionInfo(
extension_id=ext_id,
extension_publisher=metadata.extension_publisher,
extension_name=metadata.extension_id,
extension_version=metadata.extension_version,
targetPlatform=version.get("targetPlatform"),
vsxi_url=vsxi,
signature_url=sig,
)
def _artifactory_signature_url(self, extension: ExtensionInfo) -> str:
version = (
extension.extension_version
if extension.targetPlatform in (None, "universal", "unknown", "undefined")
else f"{extension.extension_version}@{extension.targetPlatform}"
)
return urllib.parse.urljoin(
self._artifactory_url,
"/".join(
[
self._artifactory_repo,
extension.extension_publisher,
extension.extension_name,
version,
"signature",
]
),
)
def _artifactory_auth_header(self) -> dict[str, str]:
return {"Authorization": f"Bearer {self._artifactory_token}"}
async def _plugin_already_present(
self,
extension: ExtensionInfo,
) -> bool:
async with self.session.head(
url=self._artifactory_signature_url(extension),
headers=self._artifactory_auth_header(),
) as resp:
return 100 < resp.status < 400
def _do_fetch_plugin(
self, extension_url: str, description: str | None = None
) -> bool:
proc = subprocess.Popen(
[
self._code_marketplace_bin,
"add",
extension_url,
"--artifactory",
self._artifactory_url,
"--repo",
self._artifactory_repo,
],
env={"ARTIFACTORY_TOKEN": self._artifactory_token} | os.environ,
)
try:
proc.wait(1800)
except: # noqa
_log.warning(
"Failed to fetch plugin %s (download did not finish)",
f"{extension_url} ({description})"
if description is not None
else extension_url,
exc_info=True,
)
return False
if proc.returncode != 0:
_log.warning("Failed to add extension %s", extension_url)
return False
return True
def _fetch_manual_download(self, extension_url: str) -> bool:
with tempfile.NamedTemporaryFile() as fn:
curl_proc = subprocess.Popen(
["/usr/bin/curl", "-s", "-f", "-o", fn.name, extension_url],
env=os.environ,
)
try:
curl_proc.wait(240)
except: # noqa
_log.warning(
"Failed to fetch plugin %s (download did not finish)",
extension_url,
exc_info=True,
)
return False
if curl_proc.returncode != 0:
_log.warning(
"Failed to fetch plugin %s (download failed)", extension_url
)
return False
return self._do_fetch_plugin(fn.name, extension_url)
async def _fetch_plugin(
self,
extension: ExtensionInfo,
) -> bool:
_log.debug(
"Fetch vsxi package for %s from %s",
extension.extension_id,
extension.vsxi_url,
)
loop = asyncio.get_running_loop()
async with self.session.head(extension.vsxi_url) as resp:
if not resp.ok:
_log.warning(
"Failed to fetch extension info for %s", extension.extension_id
)
return False
if (
"Content-Length" in resp.headers
and int(resp.headers["Content-Length"]) > 90 * 1024 * 1024
):
return await loop.run_in_executor(
self._pool, self._fetch_manual_download, extension.vsxi_url
)
return await loop.run_in_executor(
self._pool, self._do_fetch_plugin, extension.vsxi_url
)
async def _fetch_signature(self, extension: ExtensionInfo) -> bool:
async with self.session.get(extension.signature_url) as sig_resp:
if not sig_resp.ok:
_log.warning(
"Failed to fetch signature for extension %s", extension.extension_id
)
return False
data = await sig_resp.read()
async with self.session.put(
self._artifactory_signature_url(extension),
data=data,
headers=self._artifactory_auth_header(),
) as upload_resp:
if not upload_resp.ok:
_log.warning(
"Failed to upload signature for extension %s",
extension.extension_id,
)
return False
return True
async def _process_extension(self, extension: ExtensionInfo):
if await self._plugin_already_present(extension):
_log.debug("Extension %s already present", extension.extension_id)
return False
if await self._fetch_plugin(extension):
if await self._fetch_signature(extension):
_log.info("Extension %s downloaded", extension.extension_id)
return True
return True
async def _process_batch(
self, extensions: list[str]
) -> typing.AsyncGenerator[asyncio.Task, None]:
_log.info("Fetching info for %d extensions", len(extensions))
dl_cont = 0
async for info in self._fetch_marketplace_info(extensions):
if await self._plugin_already_present(info):
_log.debug(
"Extension %s (%s, %s) already present",
info.extension_id,
info.extension_version,
info.targetPlatform,
)
continue
dl_cont += 1
yield asyncio.ensure_future(self._process_extension(info))
_log.info(
"Finished processing information for %d extensions, %d artifacts will be downloaded",
len(extensions),
dl_cont,
)
async def process_extensions(self, extensions: list[str]):
chunk_size = 10
chunks = [
extensions[i : i + chunk_size]
for i in range(0, len(extensions), chunk_size)
]
_log.info("Processing %d extensions in %d chunks", len(extensions), len(chunks))
tasks = []
for chunk in chunks:
async for task in self._process_batch(chunk):
tasks.append(task)
res = await asyncio.gather(*tasks)
_log.info(
"Downloaded %d extensions, %d failed",
len(res),
len([x for x in res if not x]),
)
await self.session.close()
self._session = None And then I have a bit of LUA magic in our reverse proxy in front of the code-marketplace, that adds the signature: Nginx configinit_by_lua_block { require "cjson" }
server {
listen 443 ssl;
server_name marketplace.visualstudio.com;
ssl_certificate /etc/nginx/certs/marketplace.visualstudio.com.crt;
ssl_certificate_key /etc/nginx/certs/marketplace.visualstudio.com.key;
ssl_dhparam /etc/nginx/certs/dhparam.pem;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
# ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_prefer_server_ciphers on;
# proxy settings
# proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_redirect off;
set $config_name "marketplace.visualstudio.com";
location /_apis/public/gallery/ {
location ~ ^/.*/extensionquery$ {
if ($request_method = POST ) {
content_by_lua_block {
local cjson = require "cjson"
local dest = ngx.re.sub(ngx.var.uri, "^/_apis/public/gallery/?(.*)$", "/api/$1", "o")
ngx.req.read_body()
local capt = ngx.location.capture(dest, {method = ngx.HTTP_POST, body =ngx.req.get_body_data()})
local content = cjson.decode(capt.body)
for r_idx, res in ipairs(content.results) do
for e_idx, ext in ipairs(res.extensions) do
for v_idx, ver in ipairs(ext.versions) do
table.insert(ver.files, {assetType = "Microsoft.VisualStudio.Services.VsixSignature", source = "https://artifactory.example.com/artifactory/code-marketplace-generic/" .. ext.publisher.publisherId .. "/".. ext.extensionName .."/" .. ver.version .. "/signature"})
end
end
end
local final = cjson.encode(content)
for k, v in pairs(capt.header) do
ngx.header[k] = v
end
ngx.header.content_length = #final
ngx.print(final)
}
}
rewrite ^/_apis/public/gallery/?(.*)$ /api/$1 break;
proxy_pass https://code-marketplace.example.com;
}
rewrite ^/_apis/public/gallery/?(.*)$ /api/$1 break;
proxy_pass https://code-marketplace.example.com;
}
location /api/ {
proxy_pass https://code-marketplace.example.com;
}
location /items {
rewrite ^/items/?(.*)$ /item/$1 break;
proxy_pass https://code-marketplace.example.com;
}
location / {
location ~ ^/assets/(.*)/Microsoft.VisualStudio.Services.VsixSignature$ {
rewrite ^/assets/(.*)/Microsoft.VisualStudio.Services.VsixSignature$ https://artifactory.example.com/artifactory/code-marketplace-generic/$1/signature break;
proxy_pass https://artifactory.example.com;
}
proxy_pass https://code-marketplace.example.com;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
|
Just finished setting up the code-marketplace with Artifactory and getting the same error.
|
That might be because vscode does not trust the server certificate of the marketplace service |
I am using coder with code-server, the pod running the coder workspace is trusting the domain, |
With vscode desktop I had to put the ca cert of the ca that issued the Code-Server certificate to the chrome trust store. (Separate trust stores for different software instances are certainly an invention from hell. They're just there to ruin your day 😉) |
That sounds terrible! |
If you mean extension signing, there is no way to do that currently as far as I know aside from the workarounds above. It needs to be implemented here, and disabling signature verification in code-server appears to have no effect from what I read (I have not tried it myself though, so maybe it does work). If you mean trust as in a TLS certificate, then likely you need to add your CA to both the local machine (some requests are made from the browser) and the remote machine (other requests are made from the server). Edit: oh I missed the conversation above, you definitely mean the TLS cert. Yeah you have to trust your CA on both machines. |
That's exactly what I was thinking, both the client machine and the code-server pod trust the CA (curl works just fine with HTTPS) Edit 1 When opening the dev tools (F12) on the code-server browser it looks like all requests to the code-marketplace domain are HTTPS and work well (200 OK), for example, fetching the
Edit 2 These are the logs in the code-server log file:
Tried compiling my own extension and pushing to the registry and got the same error. |
You might try setting the |
That WORKED! Now we're at a point where 2 things are missing for making this the ultimate solution:
I'm definitely going to stick with code-marketplace as it's working well with Artifactory and integrates well with code-server! |
Reopened, I want to do a follow up PR. The current signing is not complete |
Here is where we are at: #84 Essentially we are unable to sign the packages ourselves. VSCode uses this to verify signatures: https://www.npmjs.com/package/@vscode/vsce-sign I have a branch that allows specifying certs and keys. It attempts to recreate the same signature payloads from the Microsoft marketplace, but using using a different signing key. I have a feeling the certificate that is trusted is baked into Current solutionThe Without the utility, the signature checking code in the editor only checks if the signature files exist. It does not actually enforce the signature to be valid. And for some reason VSCode on linux also does not enforce the validity https://github.com/microsoft/vscode/blob/65edc4939843c90c34d61f4ce11704f09d3e5cb6/src/vs/platform/extensionManagement/node/extensionManagementService.ts#L353 (!isLinux). If using This will still fail downloading on windows (and mac?) VSCode if you turn on Unfortunately, there is no way to get around this at the marketplace level. If VSCode begins to support other keys, I can revisit actually signing payloads with some your own certs. NotesThe branch |
Those PRs look awesome! We're using I'll wait for a release and will update with feedback ASAP, I'll gladly use a pre-release version to test this. Thanks! |
@DataOps7 Glad to hear this will work for someone! @angrycub @janLo Unfortunately Given OpenVSX made their own verification binary, https://www.npmjs.com/package/node-ovsx-sign. This allows OpenVSX to sign packages, but these signatures are formatted differently than the official, and use a different key. So this is not a solution for VSCode users. We will keep an eye on things, if VSCode ever supports supplying a key, then we should be able to revisit this. |
@Emyrk @code-asher The new version works well on my end, all of the extensions seem to be signed when using |
@DataOps7 🥳 awesome! |
Problem Statement
It seems that VS Code 1.94+ is not compatible with extensions hosted in the code-marketplace. They all have this signature warning (Screenshot 1) which prevents them from being installed in the standard way i.e. blue Install button, or automatically if you’ve enabled auto updates. You can still install via the cog wheel if you proceed passed the warning (Screenshots 2 & 3).
🖥️ Screenshots
Screenshot 1
Screenshot 2
Screenshot 3
Potentially related issues
The text was updated successfully, but these errors were encountered: