Understanding dependency management with Node Modules
How version control in your package.json works and how the lock file can prevent breaking changes to your application
The rise of JavaScript modules has led to much more complex tooling to manage script files. Most modern web applications now use module bundlers like webpack to leverage the benefits of JavaScript modules and more complex libraries such as React, instead of using simple task runners like Gulp to transpile and concatenate assets.
This has created an entirely new set of challenges for JavaScript developers, including dependency management. There is a common term in the tech community known as dependency hell — referring to the challenges and difficulties that arise from having packages with shared dependencies of different or incompatible versions.
Some common issues I have seen developers face with node modules are often related to a lack of understanding of how version control and package resolution works with Node Package Manager. This could be accidentally upgrading a package which introduces a breaking change, not understanding how to resolve duplicate or conflicting dependencies, not knowing how to resolve merge conflicts in the lock file, and the ultimate offence — deleting the lock file.
What are JavaScript Modules?
JavaScript modules are simply script files which can be loaded into each other to share functionality. One of the main benefits of modules are that they allow us to break down large scripts into smaller, reusable self-contained pieces of code.
A module or a collection of modules can be published as a package to a public or private registry for consumption. The main public registry for JS modules is Node Package Manager (NPM). This has facilitated the growth of the JS community by enabling developers to share code globally.
Packages can be installed in projects on your local machine using a command line package manager such as npm or yarn and are added to a node_modules folder in your project root.
What is the package.json and lock file?
The package.json and lock file (yarn.lock if you are using yarn, package-lock.json if you are using npm) are configuration files that tell the installer which versions of the packages to install.
Think of the package.json as the specification — it lists all the packages and their version requirements (e.g. minimum version, maximum version, etc). When you install your packages for the first time, a lock file is created. Think of this as the contract — it has a list of the exact versions of packages that met the specification in the package.json at the time they were first installed.
Typically a project does not exist solely on a single developer’s machine, the repository is cloned and installed by many developers or even in a CI/CD pipeline, therefore it is important to ensure packages are properly version controlled — which is the role of the lock file. Any time modules are subsequently installed, the installer will use the version specified in the lock file, not the package.json.
If you’re not familiar with how module resolution with package managers works, if you run into an installation problem you might be tempted try deleting everything (including the lock file) and installing again, but you should never delete the lock file as this can introduce breaking changes by accidentally upgrading dependencies. To better understand this you need to know how semantic versioning (semver) is used in the package.json.
Semver 101
Version numbers in semver consist of 3 sets of numbers separated by a ., for example 1.2.3 which represent major.minor.patch respectively. The semver documentation explains each of these:
1. MAJOR version when you make incompatible API changes,
2. MINOR version when you add functionality in a backwards compatible manner, and
3. PATCH version when you make backwards compatible bug fixes.In the package.json, for each dependency you will see a corresponding semver number. Sometimes this is prefixed with a carat ^ or tilde ~.
^1.2.3— install the latest version of1.X.X(don’t go past specified major, only a higher minor and patch number).~1.2.3— install the latest version of1.2.X(don’t go past specified major or minor, only a higher patch number).
You can use range operators (e.g. >=) if you need more fine tuned control of the version range.
How deleting the lock file can affect version control
Imagine our project has a dependency, example-package at version ^1.0.0. This is the latest available version at the time of first install and is added to the lock file as 1.0.0. A month later, a new release of example-package comes out 1.1.0. Let’s pretend a new developer joins your team and clones the project repository, and installs the project.
The version in the package.json is ^1.0.0 (meaning the latest 1.X.X), so you would think that the 1.1.0 would be installed on the new developer’s machine, but this is not the case. When the dependency was first added to the project, 1.1.0 did not exist, so the version listed in the lock file is 1.0.0. Therefore 1.0.0 will always be installed, despite there being a newer available version that meets the semver specification in the package.json.
If you do not commit the lock file to your repository, or delete the lock file and reinstall your node modules, then version 1.1.0 of example-package will be installed, since it will be treated as if this is the first time installing. New bugs could have been introduced into 1.1.0, or the maybe the package dependencies changed and are incompatible with other dependencies in your project.
This is how deleting the lock file can unintentionally cause dependencies to be upgraded which can break your application.
Version Pinning
You might think that simply removing the ^ or ~ from your version number to specify the exact version of the dependency will fix this problem. Unfortunately, even if you explicitly put the exact version of each dependency in your package.json, without a lock file you can still accidentally upgrade dependencies of dependencies.
Remember that packages are simply JavaScript modules and can interchange functionality, meaning they can import and utilise modules from other packages. Therefore, the lock file does not only handle the version control of our dependencies listed in our package.json, but all packages in the dependency tree. In the example below, example-package has a dependency foobar in its own package.json, at version ^1.0.0, and foobar has its own dependency, fizzbuzz at ~1.0.0.
// Hierarchal representation of dependency treeroot
└─┬ example-package@1.0.0
└─┬ foobar@^1.0.0
└── fizzbuzz@~1.0.0
Even if we pin the version of example-package, we’ll still run into the same accidental upgrade problem with foobar and fizzbuzz since a newer version could be released which has a bug in it, and their semver numbers are not version pinned.
Without a lock file, the package manager will update any dependencies of dependencies and their dependencies and so forth, which could introduce breaking changes.
Handling duplicate or conflicting package versions
Some packages like React require that only one version is installed and will throw an error if multiple versions are present. If you have two different packages in your project and each has a different version of React as a dependency, you could end up with two different versions of React installed.
You can inspect the dependency tree by using the commands yarn why or npm ls followed by the package name (e.g. yarn why example-package). I personally prefer npm ls as it provides a visual representation of the dependency hierarchy.
If there is a common version that meets the semver specified in the package.json, yarn will automatically dedupe on install and resolve to the common version. If you are using npm, you need to run npm dedupe.
If the installer cannot find a common version, then you will need to specify which version should be used. In your package.json add a resolutions field to specify the dependency and the version that should used. The version in each of the two packages will be overridden by the version in your resolutions field when you yarn install.
// Example of resolutions field in the package.json{
...
resolutions: {
react: "17.0.1"
}
}
Although npm does not have an equivalent of resolutions in yarn, you can specify the version by using a pre-install hook with this package, npm-force-resolutions.
Resolutions are incredibly useful, and can be used not only to dedupe packages, but to fix any issues where the version of a sub-dependency could be causing a problem. For example, a package might have a dependency which has a security vulnerability, you can use the resolutions field in the package.json to force it to update to a version where this is fixed.
Conflict resolution in the lock file
Another issue I’ve seen is confusion around how to resolve conflicts in the lock file when merging branches, especially if not all developers are on your project are JavaScript developers or used to working with yarn, npm or node modules. This can be frustrating for them as it can block their workflow if they need to sync branches.
When multiple git branches modify the dependencies (e.g. adding new dependencies required for a new feature), it is likely to introduce a conflict in the lock file. This is perfectly normal and both yarn and npm actually make resolving conflicts quite simple.
First, always resolve the conflicts in the package.json and do not manually edit the lock file. In fact, there is a warning in the yarn.lock stating not to manually edit it. If you have not made any changes to the dependencies in your branch (e.g. added a new dependency, or explicitly upgraded one), then you should take the changes from the destination branch. Then, simply run yarn to run the package installer, this will automatically fix all conflicts in the yarn.lock. If using npm, run npm install --package-lock-only to do the same thing. Then, stage the package.json and lock file, and continue merging your branch with git merge --continue.
Updating and removing unused dependencies
Good housekeeping is a key part of dependency management. You should update your dependencies regularly in order to keep up to date with the latest security fixes, but this needs to be done carefully and safely to prevent breaking your application.
I’ve seen projects where dependencies have not been updated in years, and the upgrade process to address all the security vulnerabilities was incredibly painful and introduced a lot of breaking changes.
You can check which packages are out of date with the command yarn outdated or npm outdated, however do not install the latest version of these all at once. Here are some tips on how to safely upgrade your dependencies.
- Only upgrade one package or a group of related packages which rely on each other (e.g. React and ReactDOM) at a time on a feature branch. If your application breaks, it will be much easier to identify which package is the offender and the pull request can easily be reverted.
- Ensure your application has sufficient test coverage. If you run your test suite against your pull requests in your CI/CD pipeline you can automatically catch breaking changes on the feature branch where the dependency was upgraded.
- Use an automated dependency manager. Services like Snyk, Dependabot, and Greenkeeper can be added to your CI/CD pipeline. The bot will automatically checkout a new branch and raise a pull request to upgrade out of date dependencies or dependencies with known security vulnerabilities.
You should also remove unused dependencies as these can free up space and improve install times, which will be more efficient for your application build process. Depcheck is a great tool which will analyse your project and identify the dependencies in your package.json which are missing or unused.
Conclusion
Version control with NPM packages is important in maintaining the stability of your application.
Upgrading dependencies should be done intentionally and systematically to keep up to date with the latest security fixes, not accidentally by deleting the lock file. I’ve heard horror stories about applications breaking from developer friends that work at companies where the lock file is deleted in the CI/CD pipeline, or is not committed to the repository.
Understanding semantic versioning and how the package manager resolves versions using the package.json and lock file is crucial to good dependency management and will help prevent you from making these common mistakes which can introduce breaking changes into your application.
