[Bug] `dlx` on packages with a single binary doesn't execute it if the name doesn't match the package #2013

IT-CASADO · 2020-10-20T20:40:08Z

I'd be willing to implement a fix

Describe the bug

I cannot run yarn dlx @yarnpkg/pnpify --sdk vscode with a private registry on Azure Artifacts.

Internal Error: Binary not found (pnpify) for root-workspace-0b6124@workspace:.

To Reproduce (Sherlock)

const installPromise = packageJsonAndInstall({
  dependencies: {
  },
});

// arrange private registry
await yarn(`config`, `set`, `npmRegistryServer`, `https://pkgs.dev.azure.com/it-casado/yarn-bugs/_packaging/yarn-bugs/npm/registry`);

await yarn(`config`, `set`, `npmRegistries["//pkgs.dev.azure.com/it-casado/yarn-bugs/_packaging/yarn-bugs/npm/registry"].npmAlwaysAuth`, `true`);
await yarn(`config`, `set`, `npmRegistries["//pkgs.dev.azure.com/it-casado/yarn-bugs/_packaging/yarn-bugs/npm/registry"].npmAuthIdent`, `eWFybi1idWdzOnM2Z2o3b3IzZGNicTY1cHVyZHdoZ282bnNoNnljZHRzMm43ZXZ3dG55YW8zc2lmb2F0bWE=`);

// act
const output = await yarn(`dlx`, `@yarnpkg/pnpify`, `--sdk`, `vscode`);

// assert
expect(output).not.stringContaining(`Internal Error: Binary not found (pnpify) for root-workspace`);

To Reproduce (Manual)

.yarn.yml

npmRegistries:
  //pkgs.dev.azure.com/it-casado/yarn-bugs/_packaging/yarn-bugs/npm/registry:
    npmAlwaysAuth: true
    npmAuthIdent: eWFybi1idWdzOnM2Z2o3b3IzZGNicTY1cHVyZHdoZ282bnNoNnljZHRzMm43ZXZ3dG55YW8zc2lmb2F0bWE=

npmRegistryServer: "https:////pkgs.dev.azure.com/it-casado/yarn-bugs/_packaging/yarn-bugs/npm/registry"

package.json

{
  "name": "SomeProject",
  "description": "Some project",
  "license": "MIT",
  "author": "Author",
  "version": "0.0.1",
  "devDependencies": {
    "typescript": "^4.0.3"
  }
}

Or use this GITHUB repository: https://github.com/IT-CASADO/yarn_issue_2013_repro

Side notes
My authentication must be fine, because I can run yarn npm publish without any errors and I can see the published package on my private repository.

The used registry on Azure and the token (read-only) exists only for this bug. So there is no security issue here!

Using the default NPM repository works fine for me.

Environment if relevant (please complete the following information):

OS: [Windows]
Node version [10.18.3]
Yarn version [2.3.3]

The text was updated successfully, but these errors were encountered:

darthtrevino · 2020-10-20T22:16:39Z

This occurs on any project where the monorepo root has private: true

IT-CASADO · 2020-10-21T08:07:59Z

This occurs on any project where the monorepo root has private: true

I'm using the wrong word "repository"! My repository is public, but I'm using a private registry (Azure Artifacts).
Sorry for the confusion!

darthtrevino · 2020-10-21T15:41:18Z

I'm just saying that I ran into this on a project that was using a public registry. I don't think the Azure piece was triggering this bug.

Removing 'private: true' from the top level package before installing the SDKs was a usable workaround

arcanis · 2020-10-21T16:02:23Z

I'll need a repro, because it works here (tested on this very repository).

darthtrevino · 2020-10-21T16:37:24Z

https://github.com/darthtrevino/yarn_issue_2013_repro

darthtrevino · 2020-10-21T16:39:50Z

So it looks like this repro works as expected if you add 'typescript' as a devDependency. Yesterday the behavior I was seeing was that it would behave like this even with TypeScript as a devDependency.

I could be wrong though, maybe I'm gaslighting myself.

I remember for sure removing "private: true", and then the SDKs rolled in.

darthtrevino · 2020-10-21T16:42:18Z

It would be nice if you could tool SDKs through the CLI, maybe something like

yarn dlx @yarnpkg/pnpify --sdk vscode --toolchain typescript prettier

In my projects I'm driving prettier through a build package, not as a direct dep. In order to get the Yarn SDK I need to add prettier as a devDep temporarily.

IT-CASADO · 2020-10-21T16:43:47Z

Hi all, thanks for quick response.

I'm currently preparing some stuff in my Azure Devops (private registry for a dedicated user so that I can create a npmAuthIdent that I can make public.

I will use your repository for this by changing the configuration...

IT-CASADO · 2020-10-21T17:19:32Z

I cloned your repository and added following to .yarn.yml:

npmRegistries:
  //pkgs.dev.azure.com/it-casado/yarn-bugs/_packaging/yarn-bugs/npm/registry:
    npmAlwaysAuth: true
    npmAuthIdent: eWFybi1idWdzOnM2Z2o3b3IzZGNicTY1cHVyZHdoZ282bnNoNnljZHRzMm43ZXZ3dG55YW8zc2lmb2F0bWE=

npmRegistryServer: "https://pkgs.dev.azure.com/it-casado/yarn-bugs/_packaging/yarn-bugs/npm/registry"

Then I see immediately my error!

IT-CASADO · 2020-10-21T17:27:01Z

darthtrevino · 2020-10-21T17:30:31Z

You'll want to edit your comment so that other viewers won't get access to your private NPM.

Is your azure artifact registry set up to mirror NPM for other packages? This issue may be with Azure.

You may also consider using Azure private registries only for specific scopes (e.g.. @my-super-secret-project/library).

IT-CASADO · 2020-10-21T17:47:43Z

The token in npmAuthIdent is for a specific user with restricted scope to a specific project only in my Azure DevOps for the purpose of this bug. So its fine for now, but I will revoke my token in the next days...

Yes, NPM is mirrored as an upstream source for my artifact repository.

IT-CASADO · 2020-10-21T17:48:54Z

@darthtrevino Did you see the same error after adding my registry configuration?

darthtrevino · 2020-10-21T18:01:29Z

I do, but I think this is a different issue than the issue filed. This looks like an issue with dlx. As a workaround you may want to do the following:

Comment out your private azure repo config
Add typescript and prettier as dev dependencies to the top level workspace
pnpify the repository, verifying that the typescript, prettier SDKs are loaded in and that the VSCode settings are changed
commit the sdk files and uncomment out your private registry settings

IT-CASADO · 2020-10-21T18:20:59Z

I used your workaround already and it is working so far.

I doesn't know if the issue is related to private azure registry, but this is the only relevant setting for my issue.

Let me know, if I can provide you with more details.
And please let me know if you still need my npmAuthIdent!

arcanis · 2020-10-21T18:24:49Z

Ahh I know what happens. When using dlx we copy the configuration from your project, but not from any parent folders. As a result, the token isn't available anymore when Yarn runs inside the temporary dlx directory.

As a workaround, I believe you can move your token into the ~/.yarnrc.yml file (from the home folder), which I think will work as you would expect.

IT-CASADO · 2020-10-21T18:31:09Z

@arcanis
I think you are talking here to a different issue (#1998).
In this scenario I see:

➤ YN0027: @yarnpkg/pnpify@unknown can't be resolved to a satisfying range:
Internal Error: Invalid authentication (as an anonymous user)

If I copy my token to the project configuration or ~/.yarnrc.yml then I see the error:
Internal Error: Binary not found (pnpify) for root-workspace-0b6124@workspace:.

yarnbot · 2020-10-25T19:30:45Z

The reproduction case in your issue seems broken (ie it neither pass nor fail due to throwing an unmanaged exception):

Error: Command failed: /usr/bin/node /github/workspace/scripts/actions/../run-yarn.js install

➤ YN0000: ┌ Resolution step
::group::Resolution step
➤ YN0041: │ typescript@*: Invalid authentication (as an unknown user)
::endgroup::
➤ YN0000: └ Completed in 1s 708ms
➤ YN0000: Failed with errors in 1s 712ms

    at ChildProcess.exithandler (child_process.js:303:12)
    at ChildProcess.emit (events.js:315:20)
    at maybeClose (internal/child_process.js:1021:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)

Remember: any non-Jest exceptions will cause the test to be reported as broken. If you expect something to pass without throwing, you must wrap it into something like await expect(...).resolves.toBeTruthy(). If you instead expect something to throw, you need to wrap it into await expect(...).rejects.toThrow().

IT-CASADO · 2020-10-25T19:35:23Z

Quote reply from darthtrevino:

I do, but I think this is a different issue than the issue filed. This looks like an issue with dlx. As a workaround you may want to do the following:

Comment out your private azure repo config

Add typescript and prettier as dev dependencies to the top level workspace

pnpify the repository, verifying that the typescript, prettier SDKs are loaded in and that the VSCode settings are changed

commit the sdk files and uncomment out your private registry settings

@arcanis Can you also confirm this bug with my repository

I tried to reproduce it with Sherlock but had no success.
You can see the code for Sherlock in the issue details.

yarnbot · 2020-11-24T21:12:06Z

Hi! 👋

This issue looks stale, and doesn't feature the reproducible label - which implies that you didn't provide a working reproduction using Sherlock. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it or you edit your first post to include a formal reproduction (you can use the playground for that).

Note that we require Sherlock reproductions for long-lived issues (rather than standalone git repositories or similar) because we're a small team. Sherlock gives us the ability to check which bugs are still affecting the master branch at any given point, and decreases the amount of code we need to run on our own machines (thus leading to faster bug resolutions). It helps us help you! 😃

If you absolutely cannot reproduce a bug on Sherlock (for example because it's a Windows-only issue), a maintainer will have to manually add the upholded label. Thanks for helping us triaging our repository! 🌟

Jessica-Koch · 2021-03-23T23:02:30Z

FWIW, working on my team's private repo, I was able to see this same issue, I initially saw it with when running yarn dlx @yarnpkg/doctor. When I comment out the azure registry items in my yarnrc.yml file, I am able to run the commands as expected

IT-CASADO · 2021-03-24T07:25:17Z

Unfortunately I didn't get it work (last year) and that's the reason why I gave up to try to use YARN 2.

The one big reason for us using YARN 2 is the concept of PnP (to get rid of node_modules folder).
But because of this issue and some others with Angular (libs), NX, .... I cannot use PnP. Could be that some of them are solved nowaday.

IT-CASADO · 2021-03-24T22:15:32Z

Is this related to / a duplicate of #1484

merceyz · 2021-03-24T22:32:10Z

Based on the repro you've provided no, I tried your repro but can't reproduce as even yarn npm whoami fails

IT-CASADO · 2021-03-24T22:48:30Z

hm... i get the same error with yarn npm whoami

But configuration seems to be fine, because I can successfully run yarn
But after this yarn dlx [@yarnpkg/pnpify --sdk vscode] is failing

BTW: I update my repro to the latest version of YARN 2.4.1

IT-CASADO · 2021-03-24T22:50:19Z

ok, it's a different error now... I need to check it...

merceyz · 2021-03-24T23:52:58Z

The command you're running is invalid

-yarn dlx [@yarnpkg/pnpify --sdk vscode]
+yarn dlx @yarnpkg/pnpify --sdk vscode

IT-CASADO · 2021-03-27T22:15:07Z

ok, for a few weeks I revoked my token in npmAuthIdent. Now there is a fresh and valid token configured.
(the token can be public, because it is for a non-productive environment for a demo-only user with limited rights only)

To proof that the token is valid I used it to publish my package:

yarn npm publish

As you can see everything is working fine!

Then I tried yarn npm whoami, but got an error:

Then I tried to install the SDK with yarn dlx @yarnpkg/pnpify --sdk vscode.
I'm getting the same error as already reported in this ticket:

You can check it with my latest code in my repository.

merceyz · 2021-03-27T22:41:24Z

Cloned and tested again, everything returns a 401

$ yarn
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0041: │ typescript@npm:4.0.3: Invalid authentication (as an unknown user)
➤ YN0013: │ No packages were cached - 2 packages had to be fetched
➤ YN0000: └ Completed in 0s 414ms
➤ YN0000: Failed with errors in 0s 432ms

IT-CASADO · 2021-03-27T22:46:47Z

ahhh, sorry!! I forgot to push my changes to GITHUB... Please can you try again

merceyz · 2021-03-28T17:53:25Z

Your registry is returning the wrong value for the bin field, comparing https://registry.yarnpkg.com/@yarnpkg/pnpify/2.4.0 vs your registry and you see this diff:

- "bin": "./lib/cli.js",
+ "bin": {"@yarnpkg/pnpify": "./lib/cli.js"}

IT-CASADO · 2021-03-28T18:25:37Z

I don't see any difference. What are you comparing?

I downloaded the .tar file (manually from the website of my Azure registry).

Inside that I found this package.json

{
  "name": "@yarnpkg/pnpify",
  "version": "2.4.0",
  "license": "BSD-2-Clause",
  "main": "./lib/index.js",
  "bin": "./lib/cli.js",
  "types": "./lib/index.d.ts",
  "sideEffects": false,
  "dependencies": {
    "@yarnpkg/core": "^2.4.0",
    "@yarnpkg/fslib": "^2.4.0",
    "@yarnpkg/parsers": "^2.3.0",
    "chalk": "^3.0.0",
    "clipanion": "^2.6.2",
    "comment-json": "^2.2.0",
    "lodash": "^4.17.15",
    "tslib": "^1.13.0"
  },
  "devDependencies": {
    "@types/comment-json": "^1.1.1",
    "@types/lodash": "^4.14.136",
    "@yarnpkg/monorepo": "0.0.0",
    "@yarnpkg/pnp": "^2.3.2",
    "eslint": "^6.8.0",
    "typescript": "4.1.0-beta"
  },
  "peerDependencies": {
    "eslint": "*",
    "typescript": "*"
  },
  "peerDependenciesMeta": {
    "eslint": {
      "optional": true
    },
    "typescript": {
      "optional": true
    }
  },
  "scripts": {
    "postpack": "rm -rf lib",
    "prepack": "run build:compile \"$(pwd)\"",
    "release": "yarn npm publish",
    "test": "run test:unit packages/yarnpkg-pnpify"
  },
  "publishConfig": {
    "main": "./lib/index.js",
    "bin": "./lib/cli.js",
    "types": "./lib/index.d.ts"
  },
  "files": [
    "/lib/**/*",
    "/bin.js"
  ],
  "repository": {
    "type": "git",
    "url": "ssh://[email protected]/yarnpkg/berry.git"
  },
  "engines": {
    "node": ">=10.19.0"
  }
}

It's the same value for the bin field.

I also added the @yarnpkg/pnpify to my projects package.json and installed it via YARN.
There is also the same value.

merceyz · 2021-03-28T19:38:09Z

Indeed the tar is identical but the response from the registry isn't, compare the response from https://pkgs.dev.azure.com/it-casado/yarn-bugs/_packaging/yarn-bugs/npm/registry/@yarnpkg%2fpnpify vs https://registry.yarnpkg.com/@yarnpkg/pnpify

IT-CASADO · 2021-03-28T20:06:57Z

You are right. I tried it only the NPM WAY and got the same wrong result.
So it seems to be Azure DevOps related.

IT-CASADO · 2021-03-28T20:48:25Z

@merceyz Many thanks for your help!

I created a bug ticket here:
https://developercommunity.visualstudio.com/t/azure-artifacts-packages-feed-as-upstream-source-r/1383545

IT-CASADO · 2021-03-31T19:11:05Z

@merceyz Are you sure that this is not related to YARN!?

I can successfully execute following command: npx @yarnpkg/pnpify --sdk vscode

merceyz · 2021-03-31T19:15:28Z

Don't need to involve npm nor Yarn to reproduce, just compare the output from the two registries.

IT-CASADO · 2021-03-31T19:21:54Z

Yes, I see the difference.
But even with this difference it is working perfectly fine with NPM/NPX on the private registry, but NOT with YARN.

To be honest: I don't know what should be the correct value in a private (upsstream) repository.

arcanis · 2021-03-31T19:29:12Z

There are two problems. One is the registry (it returns @yarnpkg/pnpify as binary name, which isn't correct - the binary is called pnpify), one is Yarn (if there's a single binary we should use it regardless of the package name, but we don't).

merceyz · 2021-03-31T19:40:07Z

That will indeed fix it for dlx and is something I've thought about doing but it wont help with

yarn add @yarnpkg/pnpify && yarn pnpify --sdk vscode

but that is a problem with the registry changing the published data

IT-CASADO · 2021-03-31T19:54:37Z

I'm not a technical expert on NPM/YARN/REGISTRIES, and I don't know exactly what information Microsoft needs to fix this issue (or at least confirm it as an issue with Azure DevOps Artifacts Feeds).

If you can, you can help me by providing more details to my ticket @ Visual Studio Developer Community:
https://developercommunity.visualstudio.com/t/azure-artifacts-packages-feed-as-upstream-source-r/1383545

Many thanks so far!

acton-bell · 2021-04-11T22:17:51Z

A quick workaround* is to set a custom scope just for @yarnpkg in your .yarnrc.yml:

npmScopes:
  yarnpkg:
    npmRegistryServer: "https://registry.yarnpkg.com"

*context: we had the same issue attempting to execute yarn dlx @yarnpkgdoctor in a repository exclusively using an Azure Artifacts feed with npm as the upstream.

IT-CASADO added the bug Something isn't working label Oct 20, 2020

IT-CASADO changed the title ~~[Bug] yarn dlx [@yarnpkg/pnpify --sdk vscode] doesn't work with private repository (Azure)~~ [Bug] yarn dlx [@yarnpkg/pnpify --sdk vscode] doesn't work with private registry (Azure) Oct 21, 2020

yarnbot added the broken-repro The reproduction in this issue is broken label Oct 25, 2020

yarnbot added the stale Issues that didn't get attention label Nov 24, 2020

merceyz added the upholded Real issues without formal reproduction label Nov 24, 2020

yarnbot removed the stale Issues that didn't get attention label Jan 22, 2021

merceyz removed the broken-repro The reproduction in this issue is broken label Mar 24, 2021

merceyz closed this as completed Mar 28, 2021

arcanis reopened this Mar 31, 2021

merceyz changed the title ~~[Bug] yarn dlx [@yarnpkg/pnpify --sdk vscode] doesn't work with private registry (Azure)~~ [Bug] dlx on packages with a single binary doesn't execute it if the name doesn't match the package Mar 31, 2021

merceyz mentioned this issue Apr 10, 2021

fix(dlx): use binary name if package name doesn't match #2728

Merged

3 tasks

arcanis closed this as completed in #2728 Apr 10, 2021

merceyz mentioned this issue Nov 24, 2021

[Bug?]: doctor and sdks don't work when running against private NPM registry #3793

Closed

1 task

gabidobo mentioned this issue Apr 28, 2023

yarn global install & exec in yarn temp env (dlx) fail; local install OK sandworm-hq/sandworm-audit#88

Closed

Uh oh!

[Bug] dlx on packages with a single binary doesn't execute it if the name doesn't match the package #2013

[Bug] dlx on packages with a single binary doesn't execute it if the name doesn't match the package #2013

Comments

IT-CASADO commented Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

darthtrevino commented Oct 20, 2020

Uh oh!

IT-CASADO commented Oct 21, 2020

Uh oh!

darthtrevino commented Oct 21, 2020

Uh oh!

arcanis commented Oct 21, 2020

Uh oh!

darthtrevino commented Oct 21, 2020

Uh oh!

darthtrevino commented Oct 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

darthtrevino commented Oct 21, 2020

Uh oh!

IT-CASADO commented Oct 21, 2020

Uh oh!

IT-CASADO commented Oct 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IT-CASADO commented Oct 21, 2020

Uh oh!

darthtrevino commented Oct 21, 2020

Uh oh!

IT-CASADO commented Oct 21, 2020

Uh oh!

IT-CASADO commented Oct 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

darthtrevino commented Oct 21, 2020

Uh oh!

IT-CASADO commented Oct 21, 2020

Uh oh!

arcanis commented Oct 21, 2020

Uh oh!

IT-CASADO commented Oct 21, 2020

Uh oh!

yarnbot commented Oct 25, 2020

Uh oh!

IT-CASADO commented Oct 25, 2020

Uh oh!

yarnbot commented Nov 24, 2020

Uh oh!

Jessica-Koch commented Mar 23, 2021

Uh oh!

IT-CASADO commented Mar 24, 2021

Uh oh!

IT-CASADO commented Mar 24, 2021

Uh oh!

merceyz commented Mar 24, 2021

Uh oh!

IT-CASADO commented Mar 24, 2021

Uh oh!

IT-CASADO commented Mar 24, 2021

Uh oh!

merceyz commented Mar 24, 2021

Uh oh!

IT-CASADO commented Mar 27, 2021

Uh oh!

merceyz commented Mar 27, 2021

Uh oh!

IT-CASADO commented Mar 27, 2021

Uh oh!

merceyz commented Mar 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IT-CASADO commented Mar 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

merceyz commented Mar 28, 2021

Uh oh!

IT-CASADO commented Mar 28, 2021

Uh oh!

IT-CASADO commented Mar 28, 2021

[Bug] `dlx` on packages with a single binary doesn't execute it if the name doesn't match the package #2013

[Bug] `dlx` on packages with a single binary doesn't execute it if the name doesn't match the package #2013

IT-CASADO commented Oct 20, 2020 •

edited

Loading

darthtrevino commented Oct 21, 2020 •

edited

Loading

IT-CASADO commented Oct 21, 2020 •

edited

Loading

IT-CASADO commented Oct 21, 2020 •

edited

Loading

merceyz commented Mar 28, 2021 •

edited

Loading

IT-CASADO commented Mar 28, 2021 •

edited

Loading