"How do I structure my Node.js/Express project?". I see this question come up quite regularly, especially on the Node.js and Javascript subreddits. I also remember thinking the exact same thing years ago when I first started using Node. Today, Im going to outline how I typically start structuring projects so that they're modular and easy to extend as the project grows.
The Majestic Monolith
At the end of February 2016, David Heinemeier Hansson (or you may know him simply as DHH, creator of Rails, founder of Basecamp, etc) wrote an article on Medium called "The Majestic Monolith". For the longest time, web apps existed as giagantic, monolithic, codebases where it was only one codebase that contained nearly everything for the app to run. In fact, only within the last 5 or so years have the terms "microservice" and "service oriented architecture" become really mainstream; So mainstream that I see people on discussion forums trying to pre-optimize the ever loving crap out of their platforms before they even exist!
Stop. Hold on. Back up. Let's first talk about the reasons microservices and service oriented architectures exist. These modular patterns exist generally to solve a problem of scale; this could be one of a number of things. Maybe you have a very large team and you want to break things up into smaller pieces so that smaller teams can own specific things. Take Google for example. They have hundreds of public facing services and god knows how many internal services. Splitting things into a SOA makes sense for them. At the same time though, all of their code is in a single monolithic repository.
What about scaling the actual application. Once you hit a certain size maybe you need to split things up and have an authorization server, a billing service, a logging service - this way you can scale each service independently without (hopefully) bringing down the entire platform.
Embrace the Majestic Monolith - at least to start
As a single developer, or even a team of 5, starting on a project for the first time, you dont have any of the problems above. Rather than trying to worry about dependency management of Node modules and spending time trying to write, deploy, and monitor tons of services, my suggestion is to start with a monolith.
Just because you are starting with a monolith, doesnt mean it cant be modular.
Starting with a monolith gives you a few advantages:
All of your code is in one place
This makes managing things easy. Rather than writing Node modules that are installed via npm, you can require them out of a directory of your project. Because of this...
Everyone on your team can find things easily
There's only one repository to look at which means you dont have to go digging through tons of repos on Github to find what you're looking for. Git exists for a reason, so the excuse of "there are too many people doing too many things at once" is really a poor one. Instead, learn how to properly use branches and merge features properly. Feature-flags are also your friend in this case.
No npm dependency management hell
From personal experience, prematurely creating npm modules is just shooting yourself in the foot. If you end up with 3 runnable services that depend on the same module, that is now three things that can easily break, especially if this shared module does something important like interact with your database. If you make a schema change, you now need to go through the tedious process of updating the version of your DB module in each service, re-test each service, deploy it, etc. This gets incredibly annoying, especially when your schema is still being hashed out and is prone to change.
Building your monolith majestic
Let's say for instance that we're writing a RESTful API service built on top of PostgreSQL. I tend to have three different layers to provide the best combination of separation of concerns and modularity. The example Im going to walk you through is fairly simple: we're going to have the notion of a "company" in our database and each "company" can have n many "users" associated with it. Think of this as the start of a multi-seat SaaS app where users are grouped/scoped by the company they work for, but can only belong to one company.
Here's the directory structure we'll be working with:
.
├── index.js
├── lib
│ ├── company
│ │ └── index.js
│ └── user
│ └── index.js
├── models
│ ├── company.js
│ └── user.js
├── routes
│ └── account
│ └── index.js
└── services
└── account
└── index.jsThe schema of our models is going to look something like this:
user
----------- id
- name
- email
- password
- company_id
company
----------- id
- name
user (1) --> (1) company
company (1) --> (n) userLet's start with the foundation of our platform:
Models
These are the core of everything. I really like sequelize as its a very featureful and powerfuly ORM that can also get out of your way if you need to write raw SQL queries.
Your models (and thus data in your database) are the very foundation of everything in you application. Your models can express relationships and are used to build sets of data to eventually send to the end user.
Core library/model buisness logic/CRUD layer
This is a small step up from the model level, but still pretty low level. This is where we start to actually interact with our models. Typically I'll create a corresponding file for each model that will wrap basic CRUD operations of a model so that we're not repeating the same operations all over the place. The reason I do this here and not the model is so we can start to handle some higher level features.
Given our example use-case, if you wanted to list all users in a company, your model shouldnt be concerned with interpreting query data, it is only concerned with actually querying the database. For example:
lib/user/index.js
let models = require('../../models');
const listUsersForCompany = exports.listUsersForCompany = (companyId, options = { limit: 10, offset: 0 }){
let { limit, offset } = options;
return models.Users.findAll({
where: {
company_id: companyId
},
limit: limit,
offset: offset
})
.then((users) => {
let cursor = null;
if(users.length === limit){
cursor = {
limit: limit,
offset: offset + limit
};
}
return Promise.all([users, cursor]);
});
}In this example, we've created a very basic function to list users given a companyId and some limit/offset parameters.
Each of these modules should correspond to a particular model. At this level, we dont want to be introducing other model module dependencies to allow for the greatest level of composability. Thats where the next level up comes in:
Services
I refer to these modules as services because they take different model-level modules and perform some combination of actions. Say we want to write a registration system for our application. When a user registers, you take their name, email, and password, but you also need to create a company profile which could potentially have more users down the road.
One company per user, many users per company. Being that a user depends on the existence of a company, we're going to transactionally create the two together to illustrate how a service would work.
We have our user module:
lib/user/index.js
let models = require('../../models');
exports.createUser = (userData = {}, transaction) => {
// do some stuff here like hash their password, etc
let txn = (!!transaction ? { transaction: transaction } : {});
return models.User.create(userData, transaction);
};
And our company module:
lib/company/index.js
let models = require('../../models');
exports.createCompany = (companyData = {}, transaction) => {
// do some other prep stuff herelet txn = (!!transaction ? { transaction: transaction } : {});
return models.Company.create(companyData, txn);
};And now we have our service that combines the two:
services/account/index.js
const User = require('../../lib/user');
const Company = require('../../lib/company');
let models = require('../../models');
exports.registerUserAndCompany = (data = { user: {}, company: {} }) => {
return models.sequelize.transaction()
.then((t) => {
return Company.createCompany(data.company, t)
.then((company) => {
let user = data.user;
user.company_id = company.get('id');
return User.createUser(user, t);
})
.then((user) => {
return t.commit()
.then(() => user);
})
.catch((err) => {
t.rollback();
throw err;
});
});
};By doing things this way, it doesnt matter if the user or company are created in the same transaction, or even the same request, or even if a company is created at all. For example, what if we wanted to create a user, adding them to an existing company? We can either add another function to our account service, or our route handler could call our user module directly since it would already have the company_id in the request payload.
My app has grown; I NEED microservices!
Thats great! However, you can still build microservices without breaking apart your monolithic repository (at least until you absolutely need to due to team sizes, iteration speed, etc). Our goal from the beginning was to structure our application in a way that was modular and composeable. This means that there is nothing wrong with creating new executables that simply use your monolith as a central library. This way, everything remains in the same repository and services all share the same identical modules. You've essentially created a core library that you can build things on top of.
The only overhead when deploying things is the potential duplication of your repository across services. If you're using something like Docker, which has file system layering, or rkt containers which also do some file magic to cache things, then you can actually share the single repository and simple execute whichever service you need and that overhead potentially decreases.