Structuring complex projects

“Simplicity is a prerequisite for reliability.”
Edsger W. Dijkstra

Photo by Hasan Almasi on Unsplash

We are all observing a paradigm shift to use single repositories (monorepos) instead of managing many independent git repositories individually.

I think this shift is a sort of compromise between fully monolithic product design (when all connected parts have tight linkage to the root project) and fully distributed product design (when connected parts represent architectural quantum, so can coexist independently from design and deployment perspective). Both these extremes have significant downsides that slow-down product development.

Monorepos resolves the most critical of them:

• connected components are still a self-sufficient architectural quantum and can be re-used outside of the root project
• connected components are designed as part of the project that uses its, but not in a vacuum, so final adoption is much smoother

I have shared my past experience with migrating the existing project to be monorepo with all gained benefits in this post.

As part of the migration, I found another interesting use case that I want to share in this post — supporting an efficient hierarchical project structure.

Respecting the common closure principle

Focusing more on JAVA/Spring/Gradle projects than Node/JS/TypeScript — I have already pictured in my mind how a big project needs to be structured to preserve high supportability and manageability architecture characteristics:

• The root project is just a façade that is heavily serviced by sub-modules

Hierarchy of modules

• The child module encapsulates all required dependencies, that are not exposed outside

Encapsulated dependencies

• The child is a self-sufficient module and exposes high-level API with hiding its internal implementation

Hide complexity

• The child project is managed independently and has a full set of required tasks and targets to produce a buildable or deployable artifact

Sandboxed development and promotion

JAVA and Gradle

Gradle build automation tool has basic support for multi-projects configurations, so it automatically constructs a complete build chain based on dependencies config.

So running gradle build command will produce the next result — the build engine will analyze the dependencies tree and will execute the build task in each dependency based on references order.

Project configuration is quite simple:

settings.gradle for root project should describe all linked dependencies

rootProject.name = 'gradle-hierarchy-example'
include ':packages:core'

build.gradle for root project should describe linkage type, e.g.

dependencies {
    implementation project(":packages:core")
    // other dependencies
}

JAVA and Maven

Maven build automation tool supports a similar multi-project paradigm, the only opposite parent-child recognition model — the child project is aware of its parent reference (while Gradle defines links to all child projects in parent config).

Child POM settings file need to declare parent reference:

<project>
  <parent>
    <artifactId>project-hierarchy-maven</artifactId>
    <groupId>com.example</groupId>
    <version>0.0.1-SNAPSHOT</version>
  </parent>

  ...
</project>

While packaging the parent project — all required dependencies will be recognized and built appropriately

[INFO] Reactor Summary:
[INFO] project-hierarchy-maven 0.0.1-SNAPSHOT .. SUCCESS [ 13.302 s]
[INFO] core 1.0-SNAPSHOT ....................... SUCCESS [ 43.760 s]
[INFO] -------------------------------------------------------------
[INFO] BUILD SUCCESS

NodeJS and NPM

In the JavaScript ecosystem, multi-project solutions are quite a new paradigm and classic build tools were not ready to support it.

But fortunately starting from version 7 of node package manager (NPM) this support was added by introducing an additional section in package.json project configuration file — workspaces

Workspaces is a generic term that refers to the set of features in the npm cli that provides support to managing multiple packages from your local files system from within a singular top-level, root package.

As it states in the description, this new config allows describing project dependencies that the root application should be aware of (similar to how Gradle defines its dependencies).

As an example, lets review the project with 2 child sub-modules: app and core, where the app is dependent on the core project.

To add support for multi-project behaviors all we need is to extend package configuration with links to child projects.

{
  "name": "npm-hierarchy",
  "version": "0.0.1",
  "workspaces": [
    "packages/**",
    "app"
  ],
  "dependencies": {
    . . .
  }
}

All child packages might have defined tasks that are appropriate to their scope and then run all of them at once from the root project.

// this command will run "test" command in all sub-projects
npm run test --workspaces

The correct task execution order will be obtained if one child object will have a reference to another in their project.json file.

NodeJS and LernaJS

LernaJS is the next step in managing modules in a multi-repo solution. It provides a smooth mechanism to manage and coordinate the life-cycle for sub-modules. It is an improved version of NPM workspaces with additional features like caching execution results, ordering, and task configuration.

So running all sets of child tasks in proper order does not require referring to workspaces, root tasks can be configured to do it, e.g.

lerna run compile

As a result, LernaJS build tool will analyze dependency tree and also will call “compile” tasks for each project that has this task defined.

Execution task order

What I was doing wrong — dependency management and monorepo

What I was doing wrong — managing micro-services common dependencies

Level Up Coding

Thanks for being a part of our community! More content in the Level Up Coding publication.
Follow: Twitter, LinkedIn, Newsletter
Level Up is transforming tech recruiting ➡️ Join our talent collective