Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] certain combinations of $ref overrides cause "ERR! Unable to resolve reference" from arborist #4395

Closed
2 tasks done
dbjorge opened this issue Feb 10, 2022 · 11 comments · Fixed by #5227 or #5241
Closed
2 tasks done
Assignees
Labels
Bug thing that needs fixing config:overrides Issues dealing with the overrides feature Priority 2 secondary priority issue Release 8.x work is associated with a specific npm 8 release

Comments

@dbjorge
Copy link

dbjorge commented Feb 10, 2022

Is there an existing issue for this?

  • I have searched the existing issues

This issue exists in the latest npm version

  • I am using the latest npm

Current Behavior

If a project has a dependency on "@types/node": "^16.0.0" and attempts to use an overrides entry for "@types/node": "$@types/node" to pin all references of @types/node to v16, and then also attempt to create a dependency on either the jest or jest-cli packages, arborist becomes confused and produces the following error message:

npm ERR! Unable to resolve reference $@types/node

(see repro steps below for stack trace from debug logs)

I was unable to work out why this specific combination of packages caused problems. Some of the findings from the unsuccessful theories I investigated include:

  • Repros with any package that depends on jest-config (including a minimal synthetic package, see repro steps), but does not repro with jest-config itself
  • No repro with other parent packages of similar complexity with many nested @types/node dependencies (eg, @jest/core)
  • No repro if pinning other jest-config dependencies with similar names instead of @types/node (eg, @jest/environment)
  • No repro if the override pins to a version number or version range, only if pinning to a $ref
  • Repros on both Windows and Linux

Expected Behavior

npm shouldn't error out; it should install the jest dependency correctly, with transitive dependencies on @types/node overriden to match the direct dependency entry for @types/node.

Steps To Reproduce

> mkdir npm-cli-overrides-ref-parse-repro && cd npm-cli-overrides-ref-parse-repro && npm init -y
> npm install @types/node@16

# Edit package.json adding: "overrides": { "@types/node": "$@types/node" }

# Motivating case:
> npm install jest # or jest-cli
npm ERR! Unable to resolve reference $@types/node

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\danielbj\AppData\Local\npm-cache\_logs\2022-02-09T23_06_13_180Z-debug-0.log

# More minimal repro (any package wrapping jest-config triggers this, but installing jest-config itself works fine)
> cat ./repro-dependency/package.json
{
    "name": "repro-dependency",
    "description": "repro of Unable to resolve reference npm/cli issue.",
    "version": "27.5.1",
    "main": "./build/index.js",
    "dependencies": {
        "jest-config": "^27.5.1"
    }
}
> cd ./repro-dependency && tar -czvf ./repro-dependency.tar.gz ./package.json && cd ..
> npm install ./repro-dependency/repro-dependency.tar.gz
npm ERR! Unable to resolve reference $@types/node

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\danielbj\AppData\Local\npm-cache\_logs\2022-02-09T23_46_53_856Z-debug-0.log

Relevant snippet of the debug log:

688 verbose stack Error: Unable to resolve reference $@types/node
688 verbose stack     at Edge.get spec [as spec] (C:\Users\danielbj\AppData\Local\Volta\tools\image\npm\8.4.1\node_modules\@npmcli\arborist\lib\edge.js:177:15)
688 verbose stack     at Arborist.[nodeFromEdge] (C:\Users\danielbj\AppData\Local\Volta\tools\image\npm\8.4.1\node_modules\@npmcli\arborist\lib\arborist\build-ideal-tree.js:1081:46)
688 verbose stack     at Arborist.[loadPeerSet] (C:\Users\danielbj\AppData\Local\Volta\tools\image\npm\8.4.1\node_modules\@npmcli\arborist\lib\arborist\build-ideal-tree.js:1317:36)
688 verbose stack     at async Arborist.[loadPeerSet] (C:\Users\danielbj\AppData\Local\Volta\tools\image\npm\8.4.1\node_modules\@npmcli\arborist\lib\arborist\build-ideal-tree.js:1317:11)
688 verbose stack     at async Arborist.[buildDepStep] (C:\Users\danielbj\AppData\Local\Volta\tools\image\npm\8.4.1\node_modules\@npmcli\arborist\lib\arborist\build-ideal-tree.js:951:11)
688 verbose stack     at async Arborist.buildIdealTree (C:\Users\danielbj\AppData\Local\Volta\tools\image\npm\8.4.1\node_modules\@npmcli\arborist\lib\arborist\build-ideal-tree.js:215:7)
688 verbose stack     at async Promise.all (index 1)
688 verbose stack     at async Arborist.reify (C:\Users\danielbj\AppData\Local\Volta\tools\image\npm\8.4.1\node_modules\@npmcli\arborist\lib\arborist\reify.js:152:5)
688 verbose stack     at async Install.exec (C:\Users\danielbj\AppData\Local\Volta\tools\image\npm\8.4.1\lib\commands\install.js:157:5)
688 verbose stack     at async module.exports (C:\Users\danielbj\AppData\Local\Volta\tools\image\npm\8.4.1\lib\cli.js:66:5)
689 verbose cwd C:\repos\npm-cli-overrides-ref-parse-repro
690 verbose Windows_NT 10.0.22000
691 verbose argv "C:\\Users\\danielbj\\AppData\\Local\\Volta\\tools\\image\\node\\16.14.0\\node.exe" "C:\\Users\\danielbj\\AppData\\Local\\Volta\\tools\\image\\npm\\8.4.1\\bin\\npm-cli.js" "install" "jest"
692 verbose node v16.14.0
693 verbose npm  v8.4.1
694 error Unable to resolve reference $@types/node
695 verbose exit 1

Environment

  • npm: 8.4.1
  • Node.js: 16.14.0
  • OS Name: Verified on both Windows 11 21H2 22000.493 and also an Ubuntu 20.04 WSL environment (same node/npm versions)
  • System Model Name: Surface Laptop 3
  • npm config:
; node bin location = C:\Users\danielbj\AppData\Local\Volta\tools\image\node\16.14.0\node.exe
; cwd = C:\repos\npm-cli-overrides-ref-parse-repro
; HOME = C:\Users\danielbj
; Run `npm config ls -l` to show all defaults.
@dbjorge dbjorge added Bug thing that needs fixing Needs Triage needs review for next steps Release 8.x work is associated with a specific npm 8 release labels Feb 10, 2022
@nlf
Copy link
Contributor

nlf commented Feb 10, 2022

that's a fun one.. thanks for all the work you put into figuring out the reproduction case, that's super helpful!

@nlf nlf added Priority 2 secondary priority issue and removed Needs Triage needs review for next steps labels Feb 10, 2022
@fritzy
Copy link
Contributor

fritzy commented Feb 14, 2022

closing #4409 as a dupe, but it references a shrinkwrap case specifically. It'll be worth referencing for this bug, and reopening if it turns out to be an independent issue.

@mycroes
Copy link

mycroes commented Apr 5, 2022

Experiencing the same issue, but with a private package source. I have the following overrides:

  "overrides": {
    "luxon": "^1.21.3",
    "@vlc/components": "$@vlc/components",
    "@vlc/dialog": "$@vlc/dialog",
    "@vlc/form": "$@vlc/form",
    "@vlc/snackbar": "$@vlc/snackbar",
    "@vlc/table": "$@vlc/table",
    "formik-material-ui": "$formik-material-ui"
  },

These work for installing the first package that depends on older versions of all these packages. When installing the next package from the same repository as previous package (which actually only has the previous package as dependency) it errors on the first reference, if that's replaced it errors on the next reference and so on. Changing the references to version numbers works around the problem. I'm actually on npm 8.3.0, but my coworker just upgraded to 8.5.0 yesterday and notified me of the problem. I'm available to perform further diagnosis / share more information.

@mycroes
Copy link

mycroes commented Apr 5, 2022

And also, uninstalling package A and just installing package B (that depends on A only) leads to the same issue.

jpveooys added a commit to Royal-Navy/design-system that referenced this issue Jun 15, 2022
The CRA template and smoke test starting failing again since the release of React 18.2.0, because the overrides are for react@^18.1.0 while the main React dependency injected by CRA is now react@^18.2.0.

There is a proper way to avoid this using:

```
    "overrides": {
      "react": "$react",
      "react-dom": "$react-dom"
    },
```

However, this is currently not working and results in this error:

```
npm ERR! Unable to resolve reference $react
```

At first I thought this was due to how CRA installs packages, but that doesn't actually seem to be the case; it may be due to npm/cli#4395

For now the only solution seems to be to manually keep it up-to-date, unfortunately (until either `$react` is working or all peer dependency ranges in our tree are good).
jpveooys added a commit to Royal-Navy/design-system that referenced this issue Jun 15, 2022
The CRA template and smoke test starting failing again since the release of React 18.2.0, because the overrides are for react@^18.1.0 while the main React dependency injected by CRA is now react@^18.2.0.

There is a proper way to avoid this using:

```
    "overrides": {
      "react": "$react",
      "react-dom": "$react-dom"
    },
```

However, this is currently not working and results in this error:

```
npm ERR! Unable to resolve reference $react
```

At first I thought this was due to how CRA installs packages, but that doesn't actually seem to be the case; it may be due to npm/cli#4395

For now the only solution seems to be to manually keep it up-to-date, unfortunately (until either `$react` is working or all peer dependency ranges in our tree are good).
@darcyclarke darcyclarke added the config:overrides Issues dealing with the overrides feature label Jul 11, 2022
nlf added a commit that referenced this issue Jul 28, 2022
… for overrides

when we examine override references, if we look at only `this.from.root.package` the root could actually be a virtual one. in order to ensure we resolve references from the real root, we instead need to look at `this.from.sourceReference.root.package` to get the correct value.

closes #4395
@nlf nlf self-assigned this Jul 28, 2022
@nlf nlf closed this as completed in #5227 Aug 1, 2022
nlf added a commit that referenced this issue Aug 1, 2022
… for overrides (#5227)

when we examine override references, if we look at only `this.from.root.package` the root could actually be a virtual one. in order to ensure we resolve references from the real root, we instead need to look at `this.from.sourceReference.root.package` to get the correct value.

closes #4395
@TheSlimvReal
Copy link

I am having the same issue with npm version 8.17.0.
I try to override webpack:

  "overrides": {
    "webpack": "$webpack"
  },

which gives me

Unable to resolve reference $webpack

even though I have webpack in the dependencies

@heyalexchoi
Copy link

same.
Unable to resolve reference $react

problem started after deleting package-lock.json, deleting node_modules, and reinstalling everything with npm install

@itsjohncs
Copy link

I'm encountering this on version 8.19.2. "Unable to resolve reference $react" despite React being present in my dependencies.

72 verbose stack Error: Unable to resolve reference $react
72 verbose stack     at Edge.get spec [as spec] (/Users/johnsullivan/.nvm/versions/node/v14.17.6/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/edge.js:186:15)
72 verbose stack     at Arborist.[nodeFromEdge] (/Users/johnsullivan/.nvm/versions/node/v14.17.6/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:1060:46)
72 verbose stack     at Arborist.[loadPeerSet] (/Users/johnsullivan/.nvm/versions/node/v14.17.6/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:1315:36)
72 verbose stack     at Arborist.[nodeFromEdge] (/Users/johnsullivan/.nvm/versions/node/v14.17.6/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:1103:30)
72 verbose stack     at async Arborist.[buildDepStep] (/Users/johnsullivan/.nvm/versions/node/v14.17.6/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:930:11)
72 verbose stack     at async Arborist.buildIdealTree (/Users/johnsullivan/.nvm/versions/node/v14.17.6/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:211:7)
72 verbose stack     at async Promise.all (index 1)
72 verbose stack     at async Arborist.reify (/Users/johnsullivan/.nvm/versions/node/v14.17.6/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:153:5)
72 verbose stack     at async Install.exec (/Users/johnsullivan/.nvm/versions/node/v14.17.6/lib/node_modules/npm/lib/commands/install.js:145:5)
72 verbose stack     at async module.exports (/Users/johnsullivan/.nvm/versions/node/v14.17.6/lib/node_modules/npm/lib/cli.js:78:5)

@Twipped
Copy link

Twipped commented Nov 30, 2022

Node v16.18.1, npm v8.19.2.

npm ERR! Unable to resolve reference $react-dom

This is definitely something to do with private packages. Here's my reproduction package.json

{
  "name": "npm-error-repro",
  "engines": {
    "node": ">=16.13.0"
  },
  "scripts": {
    "start": "rm -rf package-lock.json node_modules && npm i"
  },
  "dependencies": {
    "@zenbusiness/react-testing": "^2.5.2",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-router-dom": "^6.4.3"
  },
  "overrides": {
    "react": "$react",
    "react-dom": "$react-dom"
  }
}

@zenbusiness/react-testing has the following config:

  "dependencies": {
    "@testing-library/dom": "^8.17.1",
    "@testing-library/jest-dom": "^5.16.4",
    "@testing-library/react": "^13.0.1",
    "@testing-library/user-event": "^14.4.3",
    "jest-extended": "^3.1.0",
    "match-media-mock": "^0.1.1",
    "waait": "^1.0.5",
    "whatwg-fetch": "^3.6.2"
  },
  "peerDependencies": {
    "prop-types": ">=15",
    "react": ">=17",
    "react-router-dom": ">=6"
  },

@testing-library/react also has a dependency against react-dom. If I copy @zenbusiness/react-testing into a folder local to the project and change my dependency to point to it, $react-dom resolves just fine. If I change the overrides to use explicit react versions instead of a $ref, it resolves fine.

I'd provide a production repo, but you'd need access to our private github registry to see it.

Could somebody at NPM re-open this issue, or do I need to make a new one?

Vtec234 added a commit to leanprover-community/lean4web that referenced this issue Dec 1, 2022
@justin0mcateer
Copy link

I was having the same basic problem referencing a dependency which points to a local path.

  "dependencies": {
    "orbit-db-io": "file:./vendored/orbit-db-io",
    ...
  },
  "overrides": {
    "orbit-db-io": "$orbit-db-io"
    ...
  }

As a work around I had to do the following:

npm install --legacy-peer-deps
npm install

To be honest, I am not really sure why this set of operations works, but I thought it might shed some light onto the root cause and/or help someone else get around the issue.

@lll000111
Copy link

lll000111 commented Sep 4, 2023

EDIT: Last comment (THETCR commented on Jul 9) in related issue #5730 sounds like it could be the actual solution.


I have npm 9.8.1 and have

    "overrides": {
        "typescript": "$typescript"
    },

and in "dependencies" I have

"typescript": "^5.1.6",

It worked fine until now, but when I just tried (1st step for migration from CRA to vite)

npm install --save-dev vite @vitejs/plugin-react vite-tsconfig-paths vite-plugin-svgr

I got this same error for "$typescript". I had to replace the reference with the concrete version copied from the "dependencies"` entry.

Log file section:

199 verbose stack Error: Unable to resolve reference $typescript
199 verbose stack     at get spec [as spec] (C:\Users\user1\AppData\Roaming\npm\node_modules\npm\node_modules\@npmcli\arborist\lib\edge.js:202:15)
199 verbose stack     at #nodeFromEdge (C:\Users\user1\AppData\Roaming\npm\node_modules\npm\node_modules\@npmcli\arborist\lib\arborist\build-ideal-tree.js:1048:46)
199 verbose stack     at #loadPeerSet (C:\Users\user1\AppData\Roaming\npm\node_modules\npm\node_modules\@npmcli\arborist\lib\arborist\build-ideal-tree.js:1310:35)
199 verbose stack     at #nodeFromEdge (C:\Users\user1\AppData\Roaming\npm\node_modules\npm\node_modules\@npmcli\arborist\lib\arborist\build-ideal-tree.js:1091:29)
199 verbose stack     at async #buildDepStep (C:\Users\user1\AppData\Roaming\npm\node_modules\npm\node_modules\@npmcli\arborist\lib\arborist\build-ideal-tree.js:917:11)
199 verbose stack     at async Arborist.buildIdealTree (C:\Users\user1\AppData\Roaming\npm\node_modules\npm\node_modules\@npmcli\arborist\lib\arborist\build-ideal-tree.js:196:7)
199 verbose stack     at async Promise.all (index 1)
199 verbose stack     at async Arborist.reify (C:\Users\user1\AppData\Roaming\npm\node_modules\npm\node_modules\@npmcli\arborist\lib\arborist\reify.js:159:5)
199 verbose stack     at async Install.exec (C:\Users\user1\AppData\Roaming\npm\node_modules\npm\lib\commands\install.js:149:5)
199 verbose stack     at async module.exports (C:\Users\user1\AppData\Roaming\npm\node_modules\npm\lib\cli-entry.js:61:5)

@tobias-edwards
Copy link

I have the same error: npm ERR! Unable to resolve reference $typescript

"devDependencies": {
  "typescript": "^5.0.2"
},
"overrides": {
  "typescript": "^typescript"
}

Workaround for me is to use the "GOOD" approach for npm overrides by inlining the version:

"devDependencies": {
  "typescript": "^5.0.2"
},
"overrides": {
  "typescript": "^5.0.2"
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug thing that needs fixing config:overrides Issues dealing with the overrides feature Priority 2 secondary priority issue Release 8.x work is associated with a specific npm 8 release
Projects
None yet