Adds support for unconventional tarball paths (#457)

* Adds support for unconventional tarball paths

* Fixes test server

* Fixes detection

* Adds a few extra tests

* Bumps the versions

* Adds explicit imports for URL

Author: arcanis

CHANGELOG.md

@@ -6,23 +6,37 @@
 
   - Packages will now only be built when one of their dependencies is changed in some way. Note that this includes both direct dependencies and transitive dependencies, which might trigger unintuitive rebuilds in some case (for example, since `node-sass` depends on `lodash.assign`, upgrading `lodash.assign` will trigger a rebuild). This will be improved in a later release by introducing a new `runtime` field for the `dependenciesMeta` object that will exclude the package from the build key computation (feel free to start setting this flag as of now, even if it won't have any effect until then).
 
-  - Registry hostnames finally aren't part of the lockfile anymore. It means that you can switch the registry at any time by changing the `npmRegistryServer` settings. One unfortunate side effect is that NPM Enterprise cannot be supported by default anymore (they use their own weird conventions), and as such you will need to enable `npmRegistryEnterprise: true` in your settings if you're in this case. Send complaints to npm, as we don't have the power to fix this without making the experience worse for everyone else.
+  - Registry hostnames finally aren't part of the lockfile anymore. It means that you can switch the registry at any time by changing the `npmRegistryServer` settings. One unfortunate limitation is that this doesn't apply to registries that use non-standard paths for their archives (ie `/@scope/name/-/name-version.tgz`). One such example is NPM Enterprise, which will see the full path being stored in the lockfile.
 
-  - The `--frozen-lockfile` option will now properly report when the lockfile would be changed because of entry removals (it would previously only reject new entries, not removals)
+  - The `--immutable` option (new name for `--frozen-lockfile`) will now properly report when the lockfile would be changed because of entry removals (it would previously only reject new entries, not removals).
 
 ### Notable changes
 
   - The meaning of `devDependencies` is slightly altered. Until then dev dependencies were described as "dependencies we only use in development". Given that we now advocate for all your packages to be stored within the repository (in order to guarantee reproducible builds), this doesn't really make sense anymore. As a result, our description of dev dependencies is now "dependencies that aren't installed by the package consumers". It doesn't really change anything else than the name, but the more you know.
 
+      - One particular note is that you cannot install production dependencies only at the moment. We plan to add back this feature at a later time, but given that Zero-Installs means that your repository contains all your packages (prod & dev), the priority isn't as high as it used to be.
+
   - Running `yarn link <package>` now has a semi-permanent effect in that `<package>` will be added as a dependency of your active workspace (using the new `portal:` protocol). Apart from that the workflow stays the same, meaning that running `yarn link` somewhere will add the local path to the local registry, and `yarn link <package>` will add a dependency to the previously linked package.
 
-  - The lockfiles now generated should be compatible with Yaml, while staying compatible with old-style lockfiles. If a lockfile is generated that cannot be parsed by a YAML parser, please open an issue and we'll look to fix it.
+      - To disable such a link, just remove its `resolution` entry and run `yarn install` again.
+
+  - The Yarn configuration has been revamped and *will not read the `.npmrc` files anymore.* This used to cause a lot of confusion as to where the configuration was coming from, so the logic is now very simple: Yarn will look in the current directory and all its ancestors for `.yarnrc.yml` files.
+
+      - Note that the configuration files are now called `.yarnrc.yml` and thus are expected to be valid YAML. The available settings are listed [here](https://next.yarnpkg.com/configuration/yarnrc).
+
+  - The lockfiles now generated should be compatible with Yaml, while staying compatible with old-style lockfiles. Old-style lockfiles will be automatically migrated, but that will require some round-trips to the registry to obtain more information that wasn't stored previously, so the first install will be slightly slower.
+
+  - The cache files are now zip instead of tgz. This has an impact on cold install performances, because the currently available registries don't support it, which requires us to convert it on our side. Zero-Install is one way to offset this cost, and we're hoping that registries will consider offering zip as an option in the future.
 
-  - The Yarn configuration has been revamped and *will not read the `.npmrc` files anymore.* This caused a lot of confusion as to where the configuration was coming from, so the logic is now very simple: Yarn will look in the current directory and all its ancestors for `.yarnrc` files.
+      - We chose zip because of its perfect combination in terms of tooling ubiquity and random access performances (tgz would require to decompress the whole archive to access a single file).
 
 ### Package manifests (`package.json`)
 
-  - Two new fields are now supported in the `publishConfig` key of your manifests: the `main` and `module` fields will be used to replace the value of their respective top-level counterparts in the manifest shipped along with the generated file.
+To see a comprehensive documentation about each possible field, please check our [documentation](https://next.yarnpkg.com/configuration/manifest).
+
+  - Two new fields are now supported in the `publishConfig` key of your manifests: the `main`, `bin`, and `module` fields will be used to replace the value of their respective top-level counterparts in the manifest shipped along with the generated file.
+
+    - The `typings` and `types` fields will also be replaced if you use the [TypeScript plugin](https://github.com/yarnpkg/berry/tree/master/packages/plugin-typescript).
 
   - Two new fields are now supported at the root of the manifest: `dependenciesMeta` and `peerDependenciesMeta` (`peerDependenciesMeta` actually was supported in Yarn 1 as well, but `dependenciesMeta` is a new addition). These fields are meant to store dependency settings unique to each package.
 
@@ -41,6 +55,7 @@
   - The `peerDependenciesMeta[].optional` field is a boolean flag; setting it to `true` will stop the package manager from emitting a warning when the specified peer dependency is missing (you typically want to use it if you provide optional integrations with specific third-party packages and don't want to pollute your users' installs with a bunch of irrelevant warnings). This settings is package-specific.
 
   - The `resolutions` field no longer support the glob syntax within its patterns, as it was redundant with its own glob-less syntax and caused unnecessary confusion.
+
     ```diff
     {
       "resolutions": {
@@ -79,7 +94,7 @@
   - Running `yarn up <package>` will upgrade `<package>` in all of your workspaces at once (only if they already use the specified package - those that don't won't see it being added). Adding the `-i` flag will also cause Yarn to ask you to confirm for each workspace.
 
   - Running `yarn config --why` will tell you the source for each value in your configuration. We recommend using it when you're not sure to understand why Yarn would have a particular settings.
-  
+
   - Running `yarn pack` will no longer always include *nested* README, CHANGELOG, LICENSE or LICENCE files (note that those files will still be included if found at the root of the workspace being packed, as is usually the case). If you rely on this ([somewhat unintended](https://github.com/npm/npm-packlist/blob/270f534bc70dfb1d316682226332fd05e75e1b14/index.js#L162-L168)) behavior you can add those files manually to the `files` field of your `package.json`.
 
 ### Miscellaneous

packages/acceptance-tests/pkg-tests-core/sources/utils/tests.ts

@@ -139,8 +139,10 @@ exports.getPackageHttpArchivePath = async function getPackageHttpArchivePath(
   if (!packageVersionEntry)
     throw new Error(`Unknown version "${version}" for package "${name}"`);
 
+  const localName = name.replace(/^@[^\/]+\//, ``);
+
   const serverUrl = await exports.startPackageServer();
-  const archiveUrl = `${serverUrl}/${name}/-/${name}-${version}.tgz`;
+  const archiveUrl = `${serverUrl}/${name}/-/${localName}-${version}.tgz`;
 
   return archiveUrl;
 };
@@ -217,7 +219,9 @@ exports.startPackageServer = function startPackageServer(): Promise<string> {
                 [version as string]: Object.assign({}, packageVersionEntry.packageJson, {
                   dist: {
                     shasum: await exports.getPackageArchiveHash(name, version),
-                    tarball: await exports.getPackageHttpArchivePath(name, version),
+                    tarball: localName === `unconventional-tarball`
+                      ? (await exports.getPackageHttpArchivePath(name, version)).replace(`/-/`, `/tralala/`)
+                      : await exports.getPackageHttpArchivePath(name, version),
                   },
                 }),
               };
@@ -344,8 +348,11 @@ exports.startPackageServer = function startPackageServer(): Promise<string> {
         scope,
         localName,
       };
-    } else if (match = url.match(/^\/(?:(@[^\/]+)\/)?([^@\/][^\/]*)\/-\/\2-(.*)\.tgz$/)) {
-      const [_, scope, localName, version] = match;
+    } else if (match = url.match(/^\/(?:(@[^\/]+)\/)?([^@\/][^\/]*)\/(-|tralala)\/\2-(.*)\.tgz$/)) {
+      const [_, scope, localName, split, version] = match;
+
+      if (localName === `unconventional-tarball` && split === `-`)
+        return null;
 
       return {
         type: RequestType.PackageTarball,

packages/acceptance-tests/pkg-tests-fixtures/packages/unconventional-tarball-1.0.0/index.js

@@ -0,0 +1,10 @@
+/* @flow */
+
+module.exports = require(`./package.json`);
+
+for (const key of [`dependencies`, `devDependencies`, `peerDependencies`]) {
+  for (const dep of Object.keys(module.exports[key] || {})) {
+    // $FlowFixMe The whole point of this file is to be dynamic
+    module.exports[key][dep] = require(dep);
+  }
+}

packages/acceptance-tests/pkg-tests-fixtures/packages/unconventional-tarball-1.0.0/package.json

@@ -0,0 +1,4 @@
+{
+    "name": "unconventional-tarball",
+    "version": "1.0.0"
+}

packages/acceptance-tests/pkg-tests-specs/sources/auth.test.js

@@ -1,6 +1,5 @@
 const {
   fs: {writeFile},
-  tests: {getPackageArchivePath, getPackageHttpArchivePath, getPackageDirectoryPath},
 } = require('pkg-tests-core');
 
 const AUTH_TOKEN = `686159dc-64b3-413e-a244-2de2b8d1c36f`;

packages/acceptance-tests/pkg-tests-specs/sources/protocols/npm.test.js

@@ -1,3 +1,5 @@
+import {xfs} from '@yarnpkg/fslib';
+
 describe(`Protocols`, () => {
   describe(`npm:`, () => {
     test(
@@ -33,5 +35,45 @@ describe(`Protocols`, () => {
         },
       ),
     );
+
+    test(
+      `it should allow fetching packages that have an unconventional url (semver)`,
+      makeTemporaryEnv(
+        {
+          dependencies: {[`unconventional-tarball`]: `1.0.0`},
+        },
+        async ({path, run, source}) => {
+          await run(`install`);
+
+          await expect(source(`require('unconventional-tarball')`)).resolves.toMatchObject({
+            name: `unconventional-tarball`,
+            version: `1.0.0`,
+          });
+
+          await xfs.removePromise(`${path}/.yarn`);
+          await run(`install`);
+        },
+      ),
+    );
+
+    test(
+      `it should allow fetching packages that have an unconventional url (tag)`,
+      makeTemporaryEnv(
+        {
+          dependencies: {[`unconventional-tarball`]: `latest`},
+        },
+        async ({path, run, source}) => {
+          await run(`install`);
+
+          await expect(source(`require('unconventional-tarball')`)).resolves.toMatchObject({
+            name: `unconventional-tarball`,
+            version: `1.0.0`,
+          });
+
+          await xfs.removePromise(`${path}/.yarn`);
+          await run(`install`);
+        },
+      ),
+    );
   });
 });

packages/plugin-npm/package.json

@@ -9,6 +9,10 @@
     "semver": "^5.6.0"
   },
   "version": "2.0.0-rc.2",
+  "nextVersion": {
+    "semver": "2.0.0-rc.3",
+    "nonce": "119175428621803"
+  },
   "repository": {
     "type": "git",
     "url": "ssh://[email protected]/yarnpkg/berry.git"

packages/plugin-npm/sources/NpmFetcher.ts

@@ -0,0 +1,63 @@
+import {Fetcher, FetchOptions, MinimalFetchOptions} from '@yarnpkg/core';
+import {Locator, MessageName}                       from '@yarnpkg/core';
+import {httpUtils, structUtils, tgzUtils}           from '@yarnpkg/core';
+import semver                                       from 'semver';
+import {URL}                                        from 'url';
+
+import {PROTOCOL}                                   from './constants';
+import * as npmConfigUtils                          from './npmConfigUtils';
+
+export class NpmHttpFetcher implements Fetcher {
+  supports(locator: Locator, opts: MinimalFetchOptions) {
+    if (!locator.reference.startsWith(PROTOCOL))
+      return false;
+
+    const url = new URL(locator.reference);
+
+    if (!semver.valid(url.pathname))
+      return false;
+    if (!url.searchParams.has(`archiveUrl`))
+      return false;
+
+    return true;
+  }
+
+  getLocalPath(locator: Locator, opts: FetchOptions) {
+    return null;
+  }
+
+  async fetch(locator: Locator, opts: FetchOptions) {
+    const expectedChecksum = opts.checksums.get(locator.locatorHash) || null;
+
+    const [packageFs, releaseFs, checksum] = await opts.cache.fetchPackageFromCache(
+      locator,
+      expectedChecksum,
+      async () => {
+        opts.report.reportInfoOnce(MessageName.FETCH_NOT_CACHED, `${structUtils.prettyLocator(opts.project.configuration, locator)} can't be found in the cache and will be fetched from the remote server`);
+        return await this.fetchFromNetwork(locator, opts);
+      },
+    );
+
+    return {
+      packageFs,
+      releaseFs,
+      prefixPath: npmConfigUtils.getVendorPath(locator),
+      checksum,
+    };
+  }
+
+  async fetchFromNetwork(locator: Locator, opts: FetchOptions) {
+    const archiveUrl = new URL(locator.reference).searchParams.get(`archiveUrl`);
+    if (archiveUrl === null)
+      throw new Error(`Assertion failed: The archiveUrl querystring parameter should have been available`);
+
+    const sourceBuffer = await httpUtils.get(archiveUrl, {
+      configuration: opts.project.configuration,
+    });
+
+    return await tgzUtils.makeArchive(sourceBuffer, {
+      stripComponents: 1,
+      prefixPath: npmConfigUtils.getVendorPath(locator),
+    });
+  }
+}