Why You Should (Often) Wrap Your Dependencies
How Wrapping Your Dependencies Will Save You From Future Suffering
A few months back, I was talking with one of the other developers on our team and somehow we got onto the topic of what I thought my biggest learning in the past couple years was at our company. I’d been working as a software developer for the past 6 years and at Resource for just over 2 years.
“Wrap your dependencies,” I told my coworker with the sort of gaunt stare you expect to see from a grizzled war veteran.
Despite the fact that I’ve been writing software for over half a decade, I didn’t learn this lesson until more recently. Prior to working at Resource, I worked at a company doing full stack work with Vue.js on the frontend and a Django backend. During my time there, I primarily dealt with dependencies from PyPI and occasionally dealt with the npm ecosystem for adding a component library or a small utility library. This all changed once I started working at Resource. When I initially joined, the code base was written using MeteorJS. Shortly after I began, we migrated to a React/Node stack. It didn’t take long for me to realize that a lot of what I’d been told by friends and colleagues about the quality of npm packages was true: a lot of packages are just shit.
Historically I’ve tended to be the sort of developer who has leaned into using existing packages when possible. I have an aversion to reinventing wheels and abhor NIH syndrome. Yet this attitude definitely caused me some pain once I was working in the npm ecosystem day in and day out.
After spending far more time coping with this pain than I’m proud to admit, we began to adopt a strategy that would save us a lot of heartache: we decided to start wrapping the third party dependencies in our code.
Note: I use the word wrapper in this post to describe a group of design patterns that involve adding some abstraction between your code and a third party dependency. People sometimes use words like “adapter” or “facade” to indicate wrapping code with a certain intent. Does this code need to adapt its arguments to interact with some other code, or am I creating a facade in front of this other module to simplify its interface? When I say “wrapper” for the rest of this article, I use it as an all encompassing term which includes these patterns and doesn’t worry too much about intent.
The Setup
The first time you’ll regret not wrapping your dependencies is probably gonna play out like the following. One day, you need some common functionality that you suspect exists in a third party library. You dutifully search npm until you find a library, such as request, which provides a simple API to make HTTP requests. You add it to your project and don’t really think about it.
Months or maybe years later, something unthinkable happens; request has become deprecated, with no plans to add any new features and no guarantees that bugfixes or security patches will be merged.
Uh-oh. That’s not good. Guess you better find a replacement for it. You select what seems to be a popular HTTP library, cross-fetch.
Unfortunately, the API for request and cross-fetch is fairly different. Request has convenience methods for each HTTP method, whereas cross-fetch expects a url and a configuration object with a method key. Request takes a json parameter, whereas cross-fetch expects you to call .json() on the response. And so the list goes on.
If you have good tooling and aren’t doing anything too crazy with your build system, this might not be too difficult of a change. You might start by doing some form of a crude find and replace, and after that start manually inspecting all the places where you previously used request. In a large codebase, this can be a nightmare and can be quite risky.
This is one of the best reasons to wrap your dependencies: it makes replacing a dependency a much easier task.
A Tale of Two Imports
Now imagine that when you had first decided you needed an HTTP library, you had instead wrapped the request library with a module of its own. You can envision it looks something like the following.
At first, this looks like a fairly useless module. Functionally, it does the same thing, but now instead of writing an import like import request from 'request' all over our codebase, you instead write import myRequest from 'myrequest.js'.
Eventually, when that day arrives where you need to replace request with cross-fetch, you now only need to make changes in this one file. You just need to adapt the arguments that the request API takes and transform them into the shape that cross-fetch expects. The following gist shows how this might look.
It’s definitely a bit painful to write that code. But it’s a lot easier than trying to find all the places in the codebase where you’re making HTTP requests and updating them to conform to the API of cross-fetch. It’s also a lot safer.
Not only are you now able to replace this dependency when you have to, such as for security reasons, but it also gives you the flexibility to replace it for other reasons. Perhaps you discover that one of these code paths is really critical to your product and is too slow. You go check npm and discover there’s a fetching library that performs better which can be swapped in.
Or maybe you run into a bug while the maintainer of the library is on vacation. Ideally, you’d fix the bug in the source of the library itself and submit a pull request to the project. But if you need the fix immediately, you can easily patch it at the source inside of your wrapper.
However, being able to easily replace a third party dependency isn’t the only benefit using a wrapper confers.
When Good Tests Go Bad
In the scenario without the wrapper, you probably noticed something else while replacing all the call sites to now use the cross-fetch API. You made your changes and then saw a bunch of tests break. This is because a bunch of tests were mocking the request dependency, but are now trying to use cross-fetch. Now you’re stuck updating a bunch of test code, even though you haven’t changed the actual business logic that makes HTTP requests.
There’s an adage often stated when it comes to testing: Don’t mock what you don’t own. The idea is that if you don’t maintain the source code to something, you should not be mocking it.
By mocking the request library in tests, this adage was broken, and you’re paying the price. Now you must go and update all the calls to mock request and instead mock cross-fetch. Ugh.
In the scenario with the wrapper, you can avoid this pain by simply mocking the wrapper. Instead of application code like the following:
You would just reference the wrapper in your application code:
In the tests, you would now mock out the wrapper with a statement like jest.mock('./wrappers/request.js'). If you went with this approach, when swapping the request dependency for cross-fetch you wouldn’t break any tests.
This should raise some eyebrows though… If you’re mocking the wrapper, won’t the tests fail to detect if the wrapper incorrectly adapted the request API to the cross-fetch API?
Yep.
This is why you MUST write integration tests for the wrapper. Ideally, you want to write tests for the wrapper that don’t mock anything and verify that swapping one third party implementation for another doesn’t break anything. In some cases this is more feasible than others. When writing an HTTP client wrapper, as in this example, instead of mocking the third party dependencies, you can use something like pollyjs to mock everything at the HTTP layer.
There’s also another, more insidious, scenario that you can run into when mocking third party dependencies. Imagine a scenario where there wasn’t a wrapper, you had mocked out request, and instead of trying to replace request with cross-fetch you were just looking to upgrade request from version 2.81.0 to version 2.82.0.
You bump the library, run the tests, and everything passes. Great! You merge the code and push to production only to watch all hell break loose.
That’s because 2.82.0 introduced a breaking change, but since you were mocking the request library and never executed any of it’s code during the tests, they pass with flying colors.
Again, this is a scenario that could be avoided by wrapping the dependency and writing integration tests for the wrapper. In that case, the tests that mocked the wrapper would happily pass, and the integration test would fail hard due to the breaking change.
In Order to Form a More Perfect API
One of the harder to quantify benefits of using a wrapper is that you can more accurately design an API based on your application’s actual needs. Although third party dependencies save time on implementation, they aren’t a substitute for design. In many cases, a third party dependency has discovered its API through lots of battle tested use. But often you’ll find that how your application specifically needs to use it is a little different than how the library has been designed.
Let’s go back to looking at request. For this example, we’ll be looking at how you would set up a scraping application that needs to specify some different sets of proxy configuration.
Let’s assume that you have a number of different scraping “profiles” that you wish to use. Each of these profiles will configure a couple things, such as a proxy server to use, some specific HTTP headers, and some specific authentication credentials. For example, you’ll have a US-EAST profile that requires one specific Authorization header for that proxy, and a US-WEST profile that requires a different Authorization header.
Requiring the calling code to specify profile specific Authorization headers everywhere you want to make an HTTP request is not ideal. Instead, you can just define each of these profiles with a name and its configuration inside the wrapper file. You’ll then modify the wrapper API to accept a profile parameter. This way, each of the calling sites doesn’t need to worry about the nitty-gritty details of HTTP headers but can instead just specify a specific profile to use.
The result is an API that better suits the use case which means that testing will be easier, and making modifications to calling code should be less coupled. Better yet, there are more clearly defined boundaries for the system such that any implementation concerns about how profiles are handled don’t leak into tests.
Only Sith Engineers Deal in Absolutes
Of course, like any choice in software design, there’s always trade-offs to using wrappers. For each case, you’ll want to ask yourself a few questions to determine if building a wrapper for a dependency is really worth its drawbacks.
At a minimum, adding a wrapper always adds extra work. There’s extra indirection, and now a developer can’t just go read the docs for the dependency to understand how it works. They might need to look at the source for the wrapper to really understand what contract the wrapper API provides. Good tooling can help you out a lot here. Using a fully featured IDE will provide things like tooltips with parameters and docstrings.
Adding a wrapper usually requires adding more documentation as well. Since you’re not dealing with the third party package directly, looking at those docs may be misleading, depending on what has been implemented in your wrapper. With modern tooling, I don’t tend to think this is that big of a deal. Using a static analysis tool like Typescript or Flow will go a long way towards telling you if you try to do anything too stupid.
As I mentioned earlier, you’ll always want to make sure you test this wrapper, which is another burden added to your codebase.
So when is this burden not worth it? I typically ask myself a few questions.
How close is this code to my core business logic? Is the code that uses this dependency or wrapper likely to change, or are my needs likely to change?
The less I understand the API contract I need, the less likely I am to commit to a wrapper right away. Otherwise, it can be a lot of wasted work that will just be thrown out when I discover that I actually need something totally different.
Is this a “commodity” library? Are there other libraries that exist which provide a similar service at the same level of abstraction?
Generally, the more of a “commodity” a library is, the more likely I am to wrap it. If a lot of libraries work at the same level of abstraction, there’s a good chance its roughly correct, and wrapping will allow me to replace or extend the abstraction to suit my application.
Is this an implementation detail? Is this just data manipulation calls to a functional library like Lodash?
Probably not worth wrapping.
Is this just too coupled/difficult/infeasible to wrap?
For something like an ORM, I would probably end up effectively implementing an in-house ORM just by trying to wrap an existing one. In practice, codebases rarely decide to actually swap out one ORM in favor of another, so little benefit is gained.
Is this an API client library, like Github’s octokit?
Choosing to write a wrapper here depends on a few things. Using the example of octokit, if you plan on supporting multiple git hosting providers, you’ll almost certainly be best served by writing an abstraction that sits above the specific library which could handle multiple providers. But if you’re just using a library to fetch data without a better abstraction, it might be overkill to wrap it. However, wrapping it will still prevent you from accidentally mocking it and dealing with false positive tests.
Some examples that come to mind which are good candidates for wrapping are: logging, HTTP requests, ui elements/components, generic string utils, or transactional email sending.
You might have noticed that I suggest wrapping “commodity” libraries, like making HTTP calls, and I also suggest designing more application specific APIs as a benefit of wrapping. How can we wrap something that should be a fairly generic operation and yet also make its API more specific to our application’s needs? Just add another layer of wrappers! Sort of. If you find yourself with a wrapper API that isn’t quite at the right abstraction, feel free to move the wrapper down. For example, if your application had an HTTP wrapper that always added a specific header, and then you discovered that in one place, you need to make HTTP calls without that header, it’s perfectly okay to factor out a lower level wrapper from your initial wrapper which can be reused in places that don’t need the extra header. The important part is to make sure that you’re still only importing a third party dependency in one place.
“Implementation Details”
In our case, we’ve found that the best way to implement this strategy is to take advantage of yarn workspaces and build internal wrapper packages that provide clear boundaries. This makes them easy to package and distribute between multiple projects that we maintain.
If you’re not familiar, yarn workspaces provide a way to easily manage multiple packages within the same repository. Even if you have no intention of publishing these packages to a package registry, enforcing some isolation via workspaces can help ensure there isn’t too much coupling between discrete components.
To start using an HTTP wrapper, all we had to do was to add a new package to our monorepo: @resource/http . Now, we only import @resource/http wherever we want to make HTTP calls. If we ever find ourselves again needing to swap implementations for a different HTTP library, rather than having to update call sites everywhere, we only have to update @resource/http to coerce our wrapper API to the newly swapped implementation. This also means we can write tests mocking out @resource/http, and we don’t have to be afraid that the API signature will change on us, since we control it.
The important thing here is that we have the freedom and control to patch, fix, replace, and otherwise alter how we make HTTP calls without having to modify all the call sites in our codebase.
And That’s a Wrap(per)
Although you shouldn’t always wrap your dependencies, you should use your judgement to determine if it’s the right move while writing code. When determining if I should wrap something, I always ask myself:
- Is this a “commodity” library?
- Do I have a good understanding for what abstraction I need in my application?
- Am I sure this dependency isn’t too coupled or too much of an implementation detail?
If I can confidently answer yes to those three questions, it’s a good indicator that the extra work of a wrapper will be more than worth it.
