diff --git a/CHANGELOG.md b/CHANGELOG.md index 52dfc913acc8..6dff7c795224 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,23 +6,37 @@ - Packages will now only be built when one of their dependencies is changed in some way. Note that this includes both direct dependencies and transitive dependencies, which might trigger unintuitive rebuilds in some case (for example, since `node-sass` depends on `lodash.assign`, upgrading `lodash.assign` will trigger a rebuild). This will be improved in a later release by introducing a new `runtime` field for the `dependenciesMeta` object that will exclude the package from the build key computation (feel free to start setting this flag as of now, even if it won't have any effect until then). - - Registry hostnames finally aren't part of the lockfile anymore. It means that you can switch the registry at any time by changing the `npmRegistryServer` settings. One unfortunate side effect is that NPM Enterprise cannot be supported by default anymore (they use their own weird conventions), and as such you will need to enable `npmRegistryEnterprise: true` in your settings if you're in this case. Send complaints to npm, as we don't have the power to fix this without making the experience worse for everyone else. + - Registry hostnames finally aren't part of the lockfile anymore. It means that you can switch the registry at any time by changing the `npmRegistryServer` settings. One unfortunate limitation is that this doesn't apply to registries that use non-standard paths for their archives (ie `/@scope/name/-/name-version.tgz`). One such example is NPM Enterprise, which will see the full path being stored in the lockfile. - - The `--frozen-lockfile` option will now properly report when the lockfile would be changed because of entry removals (it would previously only reject new entries, not removals) + - The `--immutable` option (new name for `--frozen-lockfile`) will now properly report when the lockfile would be changed because of entry removals (it would previously only reject new entries, not removals). ### Notable changes - The meaning of `devDependencies` is slightly altered. Until then dev dependencies were described as "dependencies we only use in development". Given that we now advocate for all your packages to be stored within the repository (in order to guarantee reproducible builds), this doesn't really make sense anymore. As a result, our description of dev dependencies is now "dependencies that aren't installed by the package consumers". It doesn't really change anything else than the name, but the more you know. + - One particular note is that you cannot install production dependencies only at the moment. We plan to add back this feature at a later time, but given that Zero-Installs means that your repository contains all your packages (prod & dev), the priority isn't as high as it used to be. + - Running `yarn link ` now has a semi-permanent effect in that `` will be added as a dependency of your active workspace (using the new `portal:` protocol). Apart from that the workflow stays the same, meaning that running `yarn link` somewhere will add the local path to the local registry, and `yarn link ` will add a dependency to the previously linked package. - - The lockfiles now generated should be compatible with Yaml, while staying compatible with old-style lockfiles. If a lockfile is generated that cannot be parsed by a YAML parser, please open an issue and we'll look to fix it. + - To disable such a link, just remove its `resolution` entry and run `yarn install` again. + + - The Yarn configuration has been revamped and *will not read the `.npmrc` files anymore.* This used to cause a lot of confusion as to where the configuration was coming from, so the logic is now very simple: Yarn will look in the current directory and all its ancestors for `.yarnrc.yml` files. + + - Note that the configuration files are now called `.yarnrc.yml` and thus are expected to be valid YAML. The available settings are listed [here](https://next.yarnpkg.com/configuration/yarnrc). + + - The lockfiles now generated should be compatible with Yaml, while staying compatible with old-style lockfiles. Old-style lockfiles will be automatically migrated, but that will require some round-trips to the registry to obtain more information that wasn't stored previously, so the first install will be slightly slower. + + - The cache files are now zip instead of tgz. This has an impact on cold install performances, because the currently available registries don't support it, which requires us to convert it on our side. Zero-Install is one way to offset this cost, and we're hoping that registries will consider offering zip as an option in the future. - - The Yarn configuration has been revamped and *will not read the `.npmrc` files anymore.* This caused a lot of confusion as to where the configuration was coming from, so the logic is now very simple: Yarn will look in the current directory and all its ancestors for `.yarnrc` files. + - We chose zip because of its perfect combination in terms of tooling ubiquity and random access performances (tgz would require to decompress the whole archive to access a single file). ### Package manifests (`package.json`) - - Two new fields are now supported in the `publishConfig` key of your manifests: the `main` and `module` fields will be used to replace the value of their respective top-level counterparts in the manifest shipped along with the generated file. +To see a comprehensive documentation about each possible field, please check our [documentation](https://next.yarnpkg.com/configuration/manifest). + + - Two new fields are now supported in the `publishConfig` key of your manifests: the `main`, `bin`, and `module` fields will be used to replace the value of their respective top-level counterparts in the manifest shipped along with the generated file. + + - The `typings` and `types` fields will also be replaced if you use the [TypeScript plugin](https://github.com/yarnpkg/berry/tree/master/packages/plugin-typescript). - Two new fields are now supported at the root of the manifest: `dependenciesMeta` and `peerDependenciesMeta` (`peerDependenciesMeta` actually was supported in Yarn 1 as well, but `dependenciesMeta` is a new addition). These fields are meant to store dependency settings unique to each package. @@ -41,6 +55,7 @@ - The `peerDependenciesMeta[].optional` field is a boolean flag; setting it to `true` will stop the package manager from emitting a warning when the specified peer dependency is missing (you typically want to use it if you provide optional integrations with specific third-party packages and don't want to pollute your users' installs with a bunch of irrelevant warnings). This settings is package-specific. - The `resolutions` field no longer support the glob syntax within its patterns, as it was redundant with its own glob-less syntax and caused unnecessary confusion. + ```diff { "resolutions": { @@ -79,7 +94,7 @@ - Running `yarn up ` will upgrade `` in all of your workspaces at once (only if they already use the specified package - those that don't won't see it being added). Adding the `-i` flag will also cause Yarn to ask you to confirm for each workspace. - Running `yarn config --why` will tell you the source for each value in your configuration. We recommend using it when you're not sure to understand why Yarn would have a particular settings. - + - Running `yarn pack` will no longer always include *nested* README, CHANGELOG, LICENSE or LICENCE files (note that those files will still be included if found at the root of the workspace being packed, as is usually the case). If you rely on this ([somewhat unintended](https://github.com/npm/npm-packlist/blob/270f534bc70dfb1d316682226332fd05e75e1b14/index.js#L162-L168)) behavior you can add those files manually to the `files` field of your `package.json`. ### Miscellaneous diff --git a/packages/acceptance-tests/pkg-tests-core/sources/utils/tests.ts b/packages/acceptance-tests/pkg-tests-core/sources/utils/tests.ts index 1336d878ea5e..47344eece9cf 100644 --- a/packages/acceptance-tests/pkg-tests-core/sources/utils/tests.ts +++ b/packages/acceptance-tests/pkg-tests-core/sources/utils/tests.ts @@ -139,8 +139,10 @@ exports.getPackageHttpArchivePath = async function getPackageHttpArchivePath( if (!packageVersionEntry) throw new Error(`Unknown version "${version}" for package "${name}"`); + const localName = name.replace(/^@[^\/]+\//, ``); + const serverUrl = await exports.startPackageServer(); - const archiveUrl = `${serverUrl}/${name}/-/${name}-${version}.tgz`; + const archiveUrl = `${serverUrl}/${name}/-/${localName}-${version}.tgz`; return archiveUrl; }; @@ -217,7 +219,9 @@ exports.startPackageServer = function startPackageServer(): Promise { [version as string]: Object.assign({}, packageVersionEntry.packageJson, { dist: { shasum: await exports.getPackageArchiveHash(name, version), - tarball: await exports.getPackageHttpArchivePath(name, version), + tarball: localName === `unconventional-tarball` + ? (await exports.getPackageHttpArchivePath(name, version)).replace(`/-/`, `/tralala/`) + : await exports.getPackageHttpArchivePath(name, version), }, }), }; @@ -344,8 +348,11 @@ exports.startPackageServer = function startPackageServer(): Promise { scope, localName, }; - } else if (match = url.match(/^\/(?:(@[^\/]+)\/)?([^@\/][^\/]*)\/-\/\2-(.*)\.tgz$/)) { - const [_, scope, localName, version] = match; + } else if (match = url.match(/^\/(?:(@[^\/]+)\/)?([^@\/][^\/]*)\/(-|tralala)\/\2-(.*)\.tgz$/)) { + const [_, scope, localName, split, version] = match; + + if (localName === `unconventional-tarball` && split === `-`) + return null; return { type: RequestType.PackageTarball, diff --git a/packages/acceptance-tests/pkg-tests-fixtures/packages/unconventional-tarball-1.0.0/index.js b/packages/acceptance-tests/pkg-tests-fixtures/packages/unconventional-tarball-1.0.0/index.js new file mode 100644 index 000000000000..a6bf8f586524 --- /dev/null +++ b/packages/acceptance-tests/pkg-tests-fixtures/packages/unconventional-tarball-1.0.0/index.js @@ -0,0 +1,10 @@ +/* @flow */ + +module.exports = require(`./package.json`); + +for (const key of [`dependencies`, `devDependencies`, `peerDependencies`]) { + for (const dep of Object.keys(module.exports[key] || {})) { + // $FlowFixMe The whole point of this file is to be dynamic + module.exports[key][dep] = require(dep); + } +} diff --git a/packages/acceptance-tests/pkg-tests-fixtures/packages/unconventional-tarball-1.0.0/package.json b/packages/acceptance-tests/pkg-tests-fixtures/packages/unconventional-tarball-1.0.0/package.json new file mode 100644 index 000000000000..aecbe835296a --- /dev/null +++ b/packages/acceptance-tests/pkg-tests-fixtures/packages/unconventional-tarball-1.0.0/package.json @@ -0,0 +1,4 @@ +{ + "name": "unconventional-tarball", + "version": "1.0.0" +} diff --git a/packages/acceptance-tests/pkg-tests-specs/sources/auth.test.js b/packages/acceptance-tests/pkg-tests-specs/sources/auth.test.js index fcbc6ec9bdb4..0d314db1d625 100644 --- a/packages/acceptance-tests/pkg-tests-specs/sources/auth.test.js +++ b/packages/acceptance-tests/pkg-tests-specs/sources/auth.test.js @@ -1,6 +1,5 @@ const { fs: {writeFile}, - tests: {getPackageArchivePath, getPackageHttpArchivePath, getPackageDirectoryPath}, } = require('pkg-tests-core'); const AUTH_TOKEN = `686159dc-64b3-413e-a244-2de2b8d1c36f`; diff --git a/packages/acceptance-tests/pkg-tests-specs/sources/protocols/npm.test.js b/packages/acceptance-tests/pkg-tests-specs/sources/protocols/npm.test.js index 598bd3e92866..a17de5a1c37a 100644 --- a/packages/acceptance-tests/pkg-tests-specs/sources/protocols/npm.test.js +++ b/packages/acceptance-tests/pkg-tests-specs/sources/protocols/npm.test.js @@ -1,3 +1,5 @@ +import {xfs} from '@yarnpkg/fslib'; + describe(`Protocols`, () => { describe(`npm:`, () => { test( @@ -33,5 +35,45 @@ describe(`Protocols`, () => { }, ), ); + + test( + `it should allow fetching packages that have an unconventional url (semver)`, + makeTemporaryEnv( + { + dependencies: {[`unconventional-tarball`]: `1.0.0`}, + }, + async ({path, run, source}) => { + await run(`install`); + + await expect(source(`require('unconventional-tarball')`)).resolves.toMatchObject({ + name: `unconventional-tarball`, + version: `1.0.0`, + }); + + await xfs.removePromise(`${path}/.yarn`); + await run(`install`); + }, + ), + ); + + test( + `it should allow fetching packages that have an unconventional url (tag)`, + makeTemporaryEnv( + { + dependencies: {[`unconventional-tarball`]: `latest`}, + }, + async ({path, run, source}) => { + await run(`install`); + + await expect(source(`require('unconventional-tarball')`)).resolves.toMatchObject({ + name: `unconventional-tarball`, + version: `1.0.0`, + }); + + await xfs.removePromise(`${path}/.yarn`); + await run(`install`); + }, + ), + ); }); }); diff --git a/packages/plugin-npm/package.json b/packages/plugin-npm/package.json index 910493425c7a..cc626079fbcc 100644 --- a/packages/plugin-npm/package.json +++ b/packages/plugin-npm/package.json @@ -9,6 +9,10 @@ "semver": "^5.6.0" }, "version": "2.0.0-rc.2", + "nextVersion": { + "semver": "2.0.0-rc.3", + "nonce": "119175428621803" + }, "repository": { "type": "git", "url": "ssh://git@github.com/yarnpkg/berry.git" diff --git a/packages/plugin-npm/sources/NpmFetcher.ts b/packages/plugin-npm/sources/NpmFetcher.ts deleted file mode 100644 index b09d93684b78..000000000000 --- a/packages/plugin-npm/sources/NpmFetcher.ts +++ /dev/null @@ -1,78 +0,0 @@ -import {Fetcher, FetchOptions, MinimalFetchOptions} from '@yarnpkg/core'; -import {structUtils, tgzUtils} from '@yarnpkg/core'; -import {Locator, MessageName, ReportError} from '@yarnpkg/core'; -import {PortablePath} from '@yarnpkg/fslib'; -import semver from 'semver'; - -import {PROTOCOL} from './constants'; -import * as npmHttpUtils from './npmHttpUtils'; - -export class NpmFetcher implements Fetcher { - supports(locator: Locator, opts: MinimalFetchOptions) { - if (!locator.reference.startsWith(PROTOCOL)) - return false; - - if (!semver.valid(locator.reference.slice(PROTOCOL.length))) - return false; - - return true; - } - - getLocalPath(locator: Locator, opts: FetchOptions) { - return null; - } - - async fetch(locator: Locator, opts: FetchOptions) { - const expectedChecksum = opts.checksums.get(locator.locatorHash) || null; - - const [packageFs, releaseFs, checksum] = await opts.cache.fetchPackageFromCache( - locator, - expectedChecksum, - async () => { - opts.report.reportInfoOnce(MessageName.FETCH_NOT_CACHED, `${structUtils.prettyLocator(opts.project.configuration, locator)} can't be found in the cache and will be fetched from the remote registry`); - return await this.fetchFromNetwork(locator, opts); - }, - ); - - return { - packageFs, - releaseFs, - prefixPath: this.getPrefixPath(locator), - checksum, - }; - } - - private async fetchFromNetwork(locator: Locator, opts: FetchOptions) { - let sourceBuffer; - try { - sourceBuffer = await npmHttpUtils.get(this.getLocatorUrl(locator, opts), { - configuration: opts.project.configuration, - ident: locator, - }); - } catch (error) { - // The npm registry doesn't always support %2f when fetching the package tarballs 🤡 - // Ex: https://registry.yarnpkg.com/@emotion%2fbabel-preset-css-prop/-/babel-preset-css-prop-10.0.7.tgz - sourceBuffer = await npmHttpUtils.get(this.getLocatorUrl(locator, opts).replace(/%2f/g, `/`), { - configuration: opts.project.configuration, - ident: locator, - }); - } - - return await tgzUtils.makeArchive(sourceBuffer, { - stripComponents: 1, - prefixPath: this.getPrefixPath(locator), - }); - } - - private getLocatorUrl(locator: Locator, opts: FetchOptions) { - const version = semver.clean(locator.reference.slice(PROTOCOL.length)); - if (version === null) - throw new ReportError(MessageName.RESOLVER_NOT_FOUND, `The npm semver resolver got selected, but the version isn't semver`); - - return `${npmHttpUtils.getIdentUrl(locator)}/-/${locator.name}-${version}.tgz`; - } - - private getPrefixPath(locator: Locator) { - return `/node_modules/${structUtils.requirableIdent(locator)}` as PortablePath; - } -} diff --git a/packages/plugin-npm/sources/NpmHttpFetcher.ts b/packages/plugin-npm/sources/NpmHttpFetcher.ts new file mode 100644 index 000000000000..31d631456438 --- /dev/null +++ b/packages/plugin-npm/sources/NpmHttpFetcher.ts @@ -0,0 +1,63 @@ +import {Fetcher, FetchOptions, MinimalFetchOptions} from '@yarnpkg/core'; +import {Locator, MessageName} from '@yarnpkg/core'; +import {httpUtils, structUtils, tgzUtils} from '@yarnpkg/core'; +import semver from 'semver'; +import {URL} from 'url'; + +import {PROTOCOL} from './constants'; +import * as npmConfigUtils from './npmConfigUtils'; + +export class NpmHttpFetcher implements Fetcher { + supports(locator: Locator, opts: MinimalFetchOptions) { + if (!locator.reference.startsWith(PROTOCOL)) + return false; + + const url = new URL(locator.reference); + + if (!semver.valid(url.pathname)) + return false; + if (!url.searchParams.has(`archiveUrl`)) + return false; + + return true; + } + + getLocalPath(locator: Locator, opts: FetchOptions) { + return null; + } + + async fetch(locator: Locator, opts: FetchOptions) { + const expectedChecksum = opts.checksums.get(locator.locatorHash) || null; + + const [packageFs, releaseFs, checksum] = await opts.cache.fetchPackageFromCache( + locator, + expectedChecksum, + async () => { + opts.report.reportInfoOnce(MessageName.FETCH_NOT_CACHED, `${structUtils.prettyLocator(opts.project.configuration, locator)} can't be found in the cache and will be fetched from the remote server`); + return await this.fetchFromNetwork(locator, opts); + }, + ); + + return { + packageFs, + releaseFs, + prefixPath: npmConfigUtils.getVendorPath(locator), + checksum, + }; + } + + async fetchFromNetwork(locator: Locator, opts: FetchOptions) { + const archiveUrl = new URL(locator.reference).searchParams.get(`archiveUrl`); + if (archiveUrl === null) + throw new Error(`Assertion failed: The archiveUrl querystring parameter should have been available`); + + const sourceBuffer = await httpUtils.get(archiveUrl, { + configuration: opts.project.configuration, + }); + + return await tgzUtils.makeArchive(sourceBuffer, { + stripComponents: 1, + prefixPath: npmConfigUtils.getVendorPath(locator), + }); + } +} diff --git a/packages/plugin-npm/sources/NpmSemverFetcher.ts b/packages/plugin-npm/sources/NpmSemverFetcher.ts new file mode 100644 index 000000000000..122ebed3de75 --- /dev/null +++ b/packages/plugin-npm/sources/NpmSemverFetcher.ts @@ -0,0 +1,99 @@ +import {Configuration, Fetcher, FetchOptions, MinimalFetchOptions} from '@yarnpkg/core'; +import {structUtils, tgzUtils} from '@yarnpkg/core'; +import {Locator, MessageName, ReportError} from '@yarnpkg/core'; +import semver from 'semver'; +import {URL} from 'url'; + +import {PROTOCOL} from './constants'; +import * as npmConfigUtils from './npmConfigUtils'; +import * as npmHttpUtils from './npmHttpUtils'; + +export class NpmSemverFetcher implements Fetcher { + supports(locator: Locator, opts: MinimalFetchOptions) { + if (!locator.reference.startsWith(PROTOCOL)) + return false; + + const url = new URL(locator.reference); + + if (!semver.valid(url.pathname)) + return false; + if (url.searchParams.has(`archiveUrl`)) + return false; + + return true; + } + + getLocalPath(locator: Locator, opts: FetchOptions) { + return null; + } + + async fetch(locator: Locator, opts: FetchOptions) { + const expectedChecksum = opts.checksums.get(locator.locatorHash) || null; + + const [packageFs, releaseFs, checksum] = await opts.cache.fetchPackageFromCache( + locator, + expectedChecksum, + async () => { + opts.report.reportInfoOnce(MessageName.FETCH_NOT_CACHED, `${structUtils.prettyLocator(opts.project.configuration, locator)} can't be found in the cache and will be fetched from the remote registry`); + return await this.fetchFromNetwork(locator, opts); + }, + ); + + return { + packageFs, + releaseFs, + prefixPath: npmConfigUtils.getVendorPath(locator), + checksum, + }; + } + + private async fetchFromNetwork(locator: Locator, opts: FetchOptions) { + let sourceBuffer; + try { + sourceBuffer = await npmHttpUtils.get(NpmSemverFetcher.getLocatorUrl(locator), { + configuration: opts.project.configuration, + ident: locator, + }); + } catch (error) { + // The npm registry doesn't always support %2f when fetching the package tarballs 🤡 + // OK: https://registry.yarnpkg.com/@emotion%2fbabel-preset-css-prop/-/babel-preset-css-prop-10.0.7.tgz + // KO: https://registry.yarnpkg.com/@xtuc%2fieee754/-/ieee754-1.2.0.tgz + sourceBuffer = await npmHttpUtils.get(NpmSemverFetcher.getLocatorUrl(locator).replace(/%2f/g, `/`), { + configuration: opts.project.configuration, + ident: locator, + }); + } + + return await tgzUtils.makeArchive(sourceBuffer, { + stripComponents: 1, + prefixPath: npmConfigUtils.getVendorPath(locator), + }); + } + + static isConventionalTarballUrl(locator: Locator, url: string, {configuration}: {configuration: Configuration}) { + let registry = npmConfigUtils.getScopeRegistry(locator.scope, {configuration}); + const path = NpmSemverFetcher.getLocatorUrl(locator); + + // From time to time the npm registry returns http urls instead of https 🤡 + url = url.replace(/^https?:(\/\/(?:[^\/]+\.)?npmjs.org(?:$|\/))/, `https:$1`); + + // The yarnpkg and npmjs registries are interchangeable for that matter, so we uniformize them + registry = registry.replace(/^https:\/\/registry\.npmjs\.org($|\/)/, `https://registry.yarnpkg.com$1`); + url = url.replace(/^https:\/\/registry\.npmjs\.org($|\/)/, `https://registry.yarnpkg.com$1`); + + if (url === registry + path) + return true; + if (url === registry + path.replace(/%2f/g, `/`)) + return true; + + return false; + } + + static getLocatorUrl(locator: Locator) { + const version = semver.clean(locator.reference.slice(PROTOCOL.length)); + if (version === null) + throw new ReportError(MessageName.RESOLVER_NOT_FOUND, `The npm semver resolver got selected, but the version isn't semver`); + + return `${npmHttpUtils.getIdentUrl(locator)}/-/${locator.name}-${version}.tgz`; + } +} diff --git a/packages/plugin-npm/sources/NpmSemverResolver.ts b/packages/plugin-npm/sources/NpmSemverResolver.ts index 5b4fb40654d4..4db5cf0550c7 100644 --- a/packages/plugin-npm/sources/NpmSemverResolver.ts +++ b/packages/plugin-npm/sources/NpmSemverResolver.ts @@ -2,8 +2,11 @@ import {ReportError, MessageName, Resolver, ResolveOptions, MinimalResolveOption import {Descriptor, Locator} from '@yarnpkg/core'; import {LinkType} from '@yarnpkg/core'; import {structUtils} from '@yarnpkg/core'; +import querystring from 'querystring'; import semver from 'semver'; +import {URL} from 'url'; +import {NpmSemverFetcher} from './NpmSemverFetcher'; import {PROTOCOL} from './constants'; import * as npmHttpUtils from './npmHttpUtils'; @@ -25,7 +28,9 @@ export class NpmSemverResolver implements Resolver { if (!locator.reference.startsWith(PROTOCOL)) return false; - if (!semver.valid(locator.reference.slice(PROTOCOL.length))) + const url = new URL(locator.reference); + + if (!semver.valid(url.pathname)) return false; return true; @@ -42,9 +47,6 @@ export class NpmSemverResolver implements Resolver { async getCandidates(descriptor: Descriptor, opts: ResolveOptions) { const range = descriptor.range.slice(PROTOCOL.length); - if (semver.valid(range)) - return [structUtils.convertDescriptorToLocator(descriptor)]; - const registryData = await npmHttpUtils.get(npmHttpUtils.getIdentUrl(descriptor), { configuration: opts.project.configuration, ident: descriptor, @@ -59,12 +61,19 @@ export class NpmSemverResolver implements Resolver { }); return candidates.map(version => { - return structUtils.makeLocator(descriptor, `${PROTOCOL}${version}`); + const versionLocator = structUtils.makeLocator(descriptor, `${PROTOCOL}${version}`); + const archiveUrl = registryData.versions[version].dist.tarball; + + if (NpmSemverFetcher.isConventionalTarballUrl(versionLocator, archiveUrl, {configuration: opts.project.configuration})) { + return versionLocator; + } else { + return structUtils.makeLocator(versionLocator, `${versionLocator.reference}?${querystring.stringify({archiveUrl})}`); + } }); } async resolve(locator: Locator, opts: ResolveOptions) { - const version = semver.clean(locator.reference.slice(PROTOCOL.length)); + const version = semver.clean(new URL(locator.reference).pathname); if (version === null) throw new ReportError(MessageName.RESOLVER_NOT_FOUND, `The npm semver resolver got selected, but the version isn't semver`); diff --git a/packages/plugin-npm/sources/NpmTagResolver.ts b/packages/plugin-npm/sources/NpmTagResolver.ts index d6d08e3f27d2..0bc2130420d1 100644 --- a/packages/plugin-npm/sources/NpmTagResolver.ts +++ b/packages/plugin-npm/sources/NpmTagResolver.ts @@ -1,7 +1,9 @@ import {ReportError, MessageName, Resolver, ResolveOptions, MinimalResolveOptions} from '@yarnpkg/core'; import {structUtils} from '@yarnpkg/core'; import {Descriptor, Locator, Package} from '@yarnpkg/core'; +import querystring from 'querystring'; +import {NpmSemverFetcher} from './NpmSemverFetcher'; import {PROTOCOL} from './constants'; import * as npmHttpUtils from './npmHttpUtils'; @@ -49,11 +51,20 @@ export class NpmTagResolver implements Resolver { if (!Object.prototype.hasOwnProperty.call(distTags, tag)) throw new ReportError(MessageName.REMOTE_NOT_FOUND, `Registry failed to return tag "${tag}"`); - return [structUtils.makeLocator(descriptor, `${PROTOCOL}${distTags[tag]}`)]; + const version = distTags[tag]; + const versionLocator = structUtils.makeLocator(descriptor, `${PROTOCOL}${version}`); + + const archiveUrl = registryData.versions[version].dist.tarball; + + if (NpmSemverFetcher.isConventionalTarballUrl(versionLocator, archiveUrl, {configuration: opts.project.configuration})) { + return [versionLocator]; + } else { + return [structUtils.makeLocator(versionLocator, `${versionLocator.reference}?${querystring.stringify({archiveUrl})}`)]; + } } async resolve(locator: Locator, opts: ResolveOptions): Promise { - // Once transformed into locators, the tags are resolved by the NpmSemverResolver + // Once transformed into locators (through getCandidates), the tags are resolved by the NpmSemverResolver throw new Error(`Unreachable`); } } diff --git a/packages/plugin-npm/sources/index.ts b/packages/plugin-npm/sources/index.ts index 8152876327ee..cb080b162009 100644 --- a/packages/plugin-npm/sources/index.ts +++ b/packages/plugin-npm/sources/index.ts @@ -1,8 +1,9 @@ import {Plugin, SettingsType} from '@yarnpkg/core'; import {SettingsDefinition} from '@yarnpkg/core'; -import {NpmFetcher} from './NpmFetcher'; +import {NpmHttpFetcher} from './NpmHttpFetcher'; import {NpmRemapResolver} from './NpmRemapResolver'; +import {NpmSemverFetcher} from './NpmSemverFetcher'; import {NpmSemverResolver} from './NpmSemverResolver'; import {NpmTagResolver} from './NpmTagResolver'; import * as npmConfigUtils from './npmConfigUtils'; @@ -72,7 +73,8 @@ const plugin: Plugin = { }, }, fetchers: [ - NpmFetcher, + NpmHttpFetcher, + NpmSemverFetcher, ], resolvers: [ NpmRemapResolver, diff --git a/packages/plugin-npm/sources/npmConfigUtils.ts b/packages/plugin-npm/sources/npmConfigUtils.ts index 4eabf3c6411a..64e004409702 100644 --- a/packages/plugin-npm/sources/npmConfigUtils.ts +++ b/packages/plugin-npm/sources/npmConfigUtils.ts @@ -1,4 +1,5 @@ -import {Configuration, Manifest} from '@yarnpkg/core'; +import {Configuration, Ident, Manifest, structUtils} from '@yarnpkg/core'; +import {PortablePath} from '@yarnpkg/fslib'; export enum RegistryType { FETCH_REGISTRY = 'npmRegistryServer', @@ -9,9 +10,13 @@ export interface MapLike { get(key: string): any; } +function normalizeRegistry(registry: string) { + return registry.replace(/\/$/, ``); +} + export function getPublishRegistry(manifest: Manifest, {configuration}: {configuration: Configuration}) { if (manifest.publishConfig && manifest.publishConfig.registry) - return manifest.publishConfig.registry; + return normalizeRegistry(manifest.publishConfig.registry); if (manifest.name) return getScopeRegistry(manifest.name.scope, {configuration, type: RegistryType.PUBLISH_REGISTRY}); @@ -28,15 +33,15 @@ export function getScopeRegistry(scope: string | null, {configuration, type = Re if (scopeRegistry === null) return getDefaultRegistry({configuration, type}); - return scopeRegistry; + return normalizeRegistry(scopeRegistry); } export function getDefaultRegistry({configuration, type = RegistryType.FETCH_REGISTRY}: {configuration: Configuration, type?: RegistryType}): string { const defaultRegistry = configuration.get(type); if (defaultRegistry !== null) - return defaultRegistry; + return normalizeRegistry(defaultRegistry); - return configuration.get(RegistryType.FETCH_REGISTRY); + return normalizeRegistry(configuration.get(RegistryType.FETCH_REGISTRY)); } export function getRegistryConfiguration(registry: string, {configuration}: {configuration: Configuration}): MapLike | null { @@ -65,3 +70,7 @@ export function getScopeConfiguration(scope: string | null, {configuration}: {co return scopeConfiguration; } + +export function getVendorPath(ident: Ident) { + return `/node_modules/${structUtils.requirableIdent(ident)}` as PortablePath; +} diff --git a/packages/plugin-npm/sources/npmHttpUtils.ts b/packages/plugin-npm/sources/npmHttpUtils.ts index 618ea62addde..157f93c157fc 100644 --- a/packages/plugin-npm/sources/npmHttpUtils.ts +++ b/packages/plugin-npm/sources/npmHttpUtils.ts @@ -47,7 +47,7 @@ export async function get(path: string, {configuration, headers, ident, authType if (auth) headers = {...headers, authorization: auth}; - return await httpUtils.get(resolveUrl(registry, path), {configuration, headers, ...rest}); + return await httpUtils.get(registry + path, {configuration, headers, ...rest}); } export async function put(path: string, body: httpUtils.Body, {configuration, headers, ident, authType = AuthType.ALWAYS_AUTH, registry, ...rest}: Options) { @@ -62,7 +62,7 @@ export async function put(path: string, body: httpUtils.Body, {configuration, he headers = {...headers, authorization: auth}; try { - return await httpUtils.put(resolveUrl(registry, path), body, {configuration, headers, ...rest}); + return await httpUtils.put(registry + path, body, {configuration, headers, ...rest}); } catch (error) { if (!isOtpError(error)) throw error; @@ -75,10 +75,6 @@ export async function put(path: string, body: httpUtils.Body, {configuration, he } } -function resolveUrl(registry: string, path: string) { - return registry.replace(/\/+$/, ``) + path; -} - function getAuthenticationHeader(registry: string, {authType = AuthType.CONFIGURATION, configuration}: {authType?: AuthType, configuration: Configuration}) { const registryConfiguration = npmConfigUtils.getRegistryConfiguration(registry, {configuration}); const effectiveConfiguration = registryConfiguration || configuration; diff --git a/packages/plugin-npm/tests/NpmSemverFetcher.test.js b/packages/plugin-npm/tests/NpmSemverFetcher.test.js new file mode 100644 index 000000000000..35d8bf6afaca --- /dev/null +++ b/packages/plugin-npm/tests/NpmSemverFetcher.test.js @@ -0,0 +1,55 @@ +import {structUtils} from '@yarnpkg/core'; + +import {NpmSemverFetcher} from '../sources/NpmSemverFetcher'; + +import {makeConfiguration} from './_makeConfiguration'; + +describe(`NpmSemverFetcher`, () => { + describe(`isConventionalTarballUrl`, () => { + it(`it should detect a conventional path (foo)`, async () => { + const configuration = await makeConfiguration(); + + const locator = structUtils.makeLocator(structUtils.makeIdent(null, `foo`), `npm:1.0.0`); + const url = `${configuration.get(`npmRegistryServer`)}/foo/-/foo-1.0.0.tgz`; + + expect(NpmSemverFetcher.isConventionalTarballUrl(locator, url, {configuration})).toEqual(true); + }); + + it(`it should detect a conventional path (@scope/foo)`, async () => { + const configuration = await makeConfiguration(); + + const locator = structUtils.makeLocator(structUtils.makeIdent(`scope`, `foo`), `npm:1.0.0`); + const url = `${configuration.get(`npmRegistryServer`)}/@scope/foo/-/foo-1.0.0.tgz`; + + expect(NpmSemverFetcher.isConventionalTarballUrl(locator, url, {configuration})).toEqual(true); + }); + + it(`it should detect a conventional path (@scope%2ffoo)`, async () => { + const configuration = await makeConfiguration(); + + const locator = structUtils.makeLocator(structUtils.makeIdent(`scope`, `foo`), `npm:1.0.0`); + const url = `${configuration.get(`npmRegistryServer`)}/@scope%2ffoo/-/foo-1.0.0.tgz`; + + expect(NpmSemverFetcher.isConventionalTarballUrl(locator, url, {configuration})).toEqual(true); + }); + + it(`it should detect non-conventional path (different registry)`, async () => { + const configuration = await makeConfiguration(); + + const locator = structUtils.makeLocator(structUtils.makeIdent(null, `foo`), `npm:1.0.0`); + const url = `https://not-the-right-registry/foo/-/foo-1.0.0.tgz`; + + expect(NpmSemverFetcher.isConventionalTarballUrl(locator, url, {configuration})).toEqual(false); + }); + + it(`it should detect non-conventional path (different path)`, async () => { + const configuration = await makeConfiguration(); + + const locator = structUtils.makeLocator(structUtils.makeIdent(null, `foo`), `npm:1.0.0`); + const url = `${configuration.get(`npmRegistryServer`)}/archives/foo/foo-1.0.0.tgz`; + + expect(NpmSemverFetcher.isConventionalTarballUrl(locator, url, {configuration})).toEqual(false); + }); + }); +}); + diff --git a/packages/plugin-npm/tests/_makeConfiguration.js b/packages/plugin-npm/tests/_makeConfiguration.js new file mode 100644 index 000000000000..012535a09024 --- /dev/null +++ b/packages/plugin-npm/tests/_makeConfiguration.js @@ -0,0 +1,15 @@ +import {Configuration} from '@yarnpkg/core'; + +export const makeConfiguration = () => Configuration.find(__dirname, { + modules: new Map([ + [`@yarnpkg/core`, require(`@yarnpkg/core`)], + [`@yarnpkg/fslib`, require(`@yarnpkg/core`)], + [`@yarnpkg/plugin-npm`, require(`@yarnpkg/plugin-npm`)], + ]), + plugins: new Set([ + `@yarnpkg/plugin-npm`, + ]), +}, { + useRc: false, + strict: false, +}); diff --git a/packages/plugin-npm/tests/npmHttpUtils.test.js b/packages/plugin-npm/tests/npmHttpUtils.test.js index c6d01db3a0b3..7623cdffea85 100644 --- a/packages/plugin-npm/tests/npmHttpUtils.test.js +++ b/packages/plugin-npm/tests/npmHttpUtils.test.js @@ -1,28 +1,16 @@ -import {Configuration, httpUtils} from '@yarnpkg/core'; -import {get} from '@yarnpkg/plugin-npm/sources/npmHttpUtils'; +import {httpUtils} from '@yarnpkg/core'; +import {get} from '@yarnpkg/plugin-npm/sources/npmHttpUtils'; + +import {makeConfiguration} from './_makeConfiguration'; jest.mock(`@yarnpkg/core`, () => ({ - ... require.requireActual(`@yarnpkg/core`), + ...require.requireActual(`@yarnpkg/core`), httpUtils: { - ... require.requireActual(`@yarnpkg/core`).httpUtils, + ...require.requireActual(`@yarnpkg/core`).httpUtils, get: jest.fn(() => Promise.resolve()), }, })); -const makeConfiguration = () => Configuration.find(__dirname, { - modules: new Map([ - [`@yarnpkg/core`, require(`@yarnpkg/core`)], - [`@yarnpkg/fslib`, require(`@yarnpkg/core`)], - [`@yarnpkg/plugin-npm`, require(`@yarnpkg/plugin-npm`)], - ]), - plugins: new Set([ - `@yarnpkg/plugin-npm`, - ]), -}, { - useRc: false, - strict: false, -}); - describe(`npmHttpUtils.get`, () => { for (const registry of [`https://example.org`, `https://example.org/`, `https://example.org/foo`, `https://example.org/foo/`]) { for (const path of [`/bar`]) { diff --git a/packages/yarnpkg-builder/package.json b/packages/yarnpkg-builder/package.json index 8e6d9f3d9b69..b2fda8bad250 100644 --- a/packages/yarnpkg-builder/package.json +++ b/packages/yarnpkg-builder/package.json @@ -2,7 +2,8 @@ "name": "@yarnpkg/builder", "version": "2.0.0-rc.5", "nextVersion": { - "nonce": "4691680159291785" + "semver": "2.0.0-rc.6", + "nonce": "6702230860696063" }, "bin": "./sources/boot-dev.js", "dependencies": { diff --git a/packages/yarnpkg-cli/package.json b/packages/yarnpkg-cli/package.json index 976b8ebcb89f..81198b328e89 100644 --- a/packages/yarnpkg-cli/package.json +++ b/packages/yarnpkg-cli/package.json @@ -3,7 +3,7 @@ "version": "2.0.0-rc.5", "nextVersion": { "semver": "2.0.0-rc.6", - "nonce": "5784033715071654" + "nonce": "2434853068055069" }, "main": "./sources/index.ts", "bin": { diff --git a/packages/yarnpkg-core/package.json b/packages/yarnpkg-core/package.json index 8707bc37c925..f95c0fd0ec11 100644 --- a/packages/yarnpkg-core/package.json +++ b/packages/yarnpkg-core/package.json @@ -2,7 +2,8 @@ "name": "@yarnpkg/core", "version": "2.0.0-rc.5", "nextVersion": { - "nonce": "7589284121938487" + "semver": "2.0.0-rc.6", + "nonce": "3219402341724809" }, "main": "./sources/index.ts", "sideEffects": false, diff --git a/packages/yarnpkg-core/sources/Project.ts b/packages/yarnpkg-core/sources/Project.ts index 46f95113f819..de36efa89c5f 100644 --- a/packages/yarnpkg-core/sources/Project.ts +++ b/packages/yarnpkg-core/sources/Project.ts @@ -285,11 +285,8 @@ export class Project { bestWorkspace = workspace; } - if (!bestWorkspace) { - for (const workspace of this.workspaces) - console.log(workspace.cwd); + if (!bestWorkspace) throw new Error(`Workspace not found (${filePath})`); - } return bestWorkspace; }