One of the most fundamental aspects to consider when starting a new project (or refactoring an old one) is how to organize your files and folders. Having a well-organised project makes it easier to scale, find bugs, write tests and refactor the project with minimal conflicts.
A disorganised project is a ticking time-bomb. Everything may be fine now, but once the project hits a certain size (~30k LOC, or so I'm told) all your bad decisions start to catch up with you. You'll struggle to onboard new team members onto the codebase, blunder your way through numerous refactors, and, because dependencies are tightly-coupled, you'll face 10x the conflicts when rebasing/merging with git.
It’s when you decide to write tests and realise it’s basically impossible that you finally realise the problem runs deep. You're doomed. No developer likes to hear it, but at this point, you've accumulated so much technical debt there's no option but to start over from scratch.
The last thing you want to have is a big ol' mess of loosely-related files littered all over your project. Before it gets messy, figure out how you want to organize your files
Note: A lot of the ideas in this section are borrowed from this excellent blog post by Egon Elbre.
Let's start backwards. When would you say something is disorganised?
Disorganisation
The quality of being badly planned and without order
– Cambridge Dictionary
So, for something to be “disorganised”, it should meet at least two criteria:
It’s fair to say that if your jumped right into writing code without thinking about what goes where (i.e., you didn’t plan), it’s probably not going to have any order – it’s disorganised.
If that’s the case, organisation is the exact opposite. It’s “the quality of being well planned; orderly.”
Okay, but how do we recognise such a quality? How do we know something is well-planned? How do we know it’s orderly?
In order to understand what “orderly” means, we have to understand why we organise things to begin with. Organisation/order serves three main purposes:
Put more succinctly, organisation is based on three main principles:
Let’s make those phrases more relatable in a software engineering context. Organising a project is important because:
Before we get to the meat of things, I think it’s important to see the folder structure I suggest to use:
.
├── docker-compose.yml
├── Dockerfile
├── docs
├── migrations
├── package.json
├── package-lock.json
├── README.md
├── scripts // build scripts
├── src // contains source code files
│ ├── bin // files related to the server
│ ├── clients // code used to access third-party services
│ ├── config // files related to the project configuration (eg. environment variables)
│ ├── constants // files containing static values
│ ├── controllers // route controllers.
│ ├── helpers // helper files
│ ├── listeners // event listeners
│ ├── models // or (/DAO)
│ ├── modules //
│ ├── plugins // (or /middlewares)
│ ├── routes
│ ├── services // project services
│ ├── static // static files
│ ├── tests
│ │ ├── factories
│ │ ├── helpers
│ │ └── src
│ │ ├── integration
│ │ └── unit
│ │ ├── config
│ │ ├── controllers
│ │ ├── helpers
│ │ ├── models
│ │ ├── schemas
│ │ └── services
│ ├── types // typescript types
│ └── validators // route validaton schema
└── tsconfig.json
This folder structure is borrowed deeply from the world of Java, Hibernate and Spring Boot. It was the first framework to introduce me to the concept of services and controllers, but I didn’t really understand why they were so important until I had to implement it by myself from scratch in NodeJS.
Many other frameworks - notably Django and Ruby on Rails - rely on the same concepts to varying degrees.
The specific folder names don’t matter much. There isn’t much of a difference between lib
and library
, or if you decide to name it libre
. Again,the concepts are what matter the most. If the word “module” is confusing to you, swap it out for something that makes sense.
An important aspect of how I structure my projects is the file naming convention. Depending on what the file does, it is suffixed with .role
before the file extension.
Therefore, an account route handler would be named accounts.route.ts
, the controller becomes accounts.controller.ts
, the service accounts.sevice.ts
, and related helpers as accounts.helper.ts
.
It’s entirely a matter of preference, however. AccountRoute.ts
, AccountService.ts
and AccountHelper
are just as readable. The most important thing to ensure is consistency. All your files should be named using a consistent formula.
The gist of it is this: once a request is sent from the client to the server, it passes through the following two-way flow before a response is passed back to the client:
route handler <=> controller <=> service <=> database access object (DAO)
This grouping is based on the role that the individual things serve in the bigger picture. It’s referred to as grouping by classification.
The route handler is responsible passing the request object and related metadata to a controller. It doesn't contain any logic.
Here is how we’d implement a route handler in Hapi:
server.route({
path: '/accounts/login',
method: 'POST',
auth: false,
handler: async request => await accountsController.loginUser(request),
});
Notice how “bare bones” this route handler is.
All it does is to pass the request object to accountsController
. It doesn’t handle any logic. This way, we can always be sure the request object is only accessed and manipulated at the controller layer.
Route handlers also work hand-in-hand with middleware. Any request or response validation is done at the route level. This always happens before the request is passed to the controller so that the controller doesn’t have to worry about receiving "dirty" data.
Here is an example of a route that can only be accessed by logged-in users:
server.route({
path: '/accounts/me',
method: ['GET'],
options: {
plugins: {
rbac: permissionsHelper.getAdminOrUserRBAC(),
}
},
handler: async request => await userController.getUserInfo(request),
});
The plugins
key has an rbac
object for handling permissions. With a library like Express, this would probably be handled using middleware.
With that done, we can now define a set of responsibilities for route handlers:
Passes request metadata to the controller
This is a route handler's primary role.
Validates request metadata
The route handler is responsible for validating the payload, query params and path parameters through middleware. This should happen before any data reaches the controller.
Validates route permissions
Just like ensuring the request metadata is valid, the route handler is also responsible for ensuring the client has the right permissions before the request is processed.
Provides documentation
Through middleware like Swagger, the route handler provides documentation regarding the request payload, response and query params.
A controller acts as a middleman between route handlers and services . It “controls” the flow of data from the client to the service layer. If possible, they shouldn't contain any business logic.
Here is an example of a controller for our login route:
class AccountsController {
//...
async loginUser(request: Request) {
const loginUserResult = await authService.loginUser(request.payload)
switch (loginUserResult.status){
case "success":
return loginUserResult.data;
case "fail":
return Boom.badRequest()
case "error":
default:
return Boom.internalError()
}
}
// ...
}
The controller calls a service function, and, depending on the status of the result it receives, returns an appropriate response back to the client.
However, we’d be so lucky if our algorithm were always so simple. What if we want to track every successful login? To track all the failed login attempts? Maybe to modify the object returned by the service, so we don’t return sensitive data to the client?
Well, we can modify the controller to suit our needs.
class AccountsController {
async loginUser(request: Request) {
const payload = request.payload as LoginUserPayload
// run the authentication logic
const loginUserResult = await authService.loginUser(payload)
switch (loginUserResult.status){
// if the login was successful
case 'success':
const {tokens, user} = loginUserResult.data;
// emit an "ON_LOGIN_SUCCESSFUL" event
request.server.events.emit(
events.accounts.ON_LOGIN_SUCCESSFUL,
tokens
);
// track the successful login using our analytics client
if (env.isProduction()){
await analyticsClient.capture({
event: Events.ANALYTICS.USER_LOGGED_IN,
distinctId: user.uuid,
properties: {
email: user.email,
},
});
}
// modify the response object to only return an access token and refresh token
return {
tokens: {
accessToken: tokens.accessToken,
refreshToken: tokens.refreshToken,
}
}
}
// most error messages are the same. Our response handler will manage them effectively.
return responseHelper.returnGenericResponses(loginUserResult);
}
}
This line may be easy to miss, but you should pay particular attention to it:
const loginUserResult = await authService.loginUser(payload)
Notice that we pass just the payload, not the whole request object. Every layer should receive the minimal amount of data it requires to function.
Facilitates communication between route handlers and services
This is a controller's primary role. All the data passed to the service layer should be deconstructed to its most minimal form.
Carries any "miscellaneous" logic
Some logic doesn't fit neatly into any service. Code such as analytics tracking and event emits should be included in the controller
Formats response messages.
The service is where all the complex stuff happens, so called the “business logic”. Let’s expand on the above example of a service that handles login functionality for an app:
class AccountsService {
//...
async loginUser(payload: LoginUserPayload): Promise<ServiceReturnType<LoginUserResult>> {
try {
// find the user by their email/username/phone number
const user = await userDAO.findUserByLogin(payload.login);
// if the user exists
if (user) {
// check if the password they provided is correct
const isPasswordCorrect = await passwordHelper.isPasswordCorrect({
passwordToVerify: payload.password,
bcryptPasswordHash: user.account.password,
});
// if the password is correct
if (isPasswordCorrect) {
// generate a set of tokens (access & refresh)
let tokens = jwtHelper.generateTokens(user);
return {
status: 'success',
message: messageHelper.auth.login.SUCCESSFUL_LOGIN,
data: {
tokens,
user
},
};
} else {
// if the password is wrong
return {
status: 'fail',
message: messageHelper.generic.GENERIC_INPUT_ERROR,
errors: [
{
message: messageHelper.passwords.INCORRECT_PASSWORD,
path: 'password'
}
],
};
}
} else {
// if the account doesn't exist
return {
status: 'fail',
message: messageHelper.generic.GENERIC_INPUT_ERROR,
errors: [
{
message: messageHelper.auth.login.MISSING_ACCOUNT,
path: 'login'
}
]
};
}
}catch (e){
return {
status: "error",
message: "An error occurred while trying to log a in.",
cause: e
}
}
}
}
A very important bit of functionality that ties everything together is return type of this function – the ServiceReturnType<T>
type,
A ServiceReturnType<T>
can be in one of three states: success
, fail
or error
.
The success
state is reached when a service completes running a specific task successfully. This state is usually accompanied by related metadata and an optional message.
It is represented by the following object:
interface ServiceProcessSuccess<T> {
message?: string;
data: T;
status: 'success';
}
The fail
state implies that the processes being run by the service did not complete successfully. However, this was likely due to a client-side error, not a server-side one. Maybe the client provided the wrong password or their account doesn’t exist, for example.
It is represented by the following object:
interface ServiceProcessFail {
// a friendly, sensible, message about why the error failed
// this message should be able to be displayed in a toast notification
message: string;
// the errors, formatted nicely
errors: Array<ResponseError>;
status: 'fail';
}
Notice the errors
key. This is an array of all the errors that occurred while processing the client’s request.
This is what the ResponseError
type looks like:
type ResponseError = {
path?: any; // the key that errored. Useful for accessing the error value on the client
message: string; // a friendly, sensible error message
value?: any;
}
Why are there two message
fields (one in the main object and other in ResponseError
), you may ask? Good question.
A common pattern lots of web applications follow is to display an error message in form of a toast when a request fails:
Alternatively (or in addition), apps also show an error message on the form itself when a request fails:
The processResult.errors[i].message
field is used to map multiple errors to a single form field.
The error
state implies the service resulted in a server-side error. This could be a database error, for example.
This state will always result in a 500 Internal Server Error
.
It is represented by the following error object:
interface ServiceProcessError {
status: 'error';
// a friendly, sensible message
message: string;
// The error that caused this process to fail. Useful for debugging.
cause?: Error;
}
All of this comes together inside the ServiceReturnType<T>
, a generic object that’s shaped like so:
type ServiceReturnType<T> =
| ServiceProcessSuccess<T>
| ServiceProcessFail
| ServiceProcessError;
This allows us to use a simple switch-case
expression for every result returned by services inside our controllers:
class SomeController<T> {
async doStuff() {
const serviceResult = await someService.doStuff();
switch (serviceResult.status) {
case "success":
return serviceResult.data;
case "fail":
// Boom is a hapi library that helps to format error responses
return Boom.badRequest();
case "error":
return Boom.internalError();
}
}
}
When using Typescript, this comes with all the added benefits of string literal types.
Your error messages should be sensible, friendly and actionable.
An error message such as “bad request” doesn’t make much sense to a user, let alone a developer that’s trying to debug your application. It’s neither friendly nor sensible. Additionally, while the user knows something went wrong, it doesn’t let them know if there’s anything they can do to fix it. It’s not actionable.
“Incorrect password” has the exact number of words, but a lot easier to wrap your head around. However, it feels like the kind of response you’d give a bot, not a human. Its sensible, but not friendly. And, It doesn’t direct the user on what to do.
“The password you provided is incorrect. Please try again.” is even better. It tells the user what they did wrong and informs the user how to fix it.
DAOs are commonly referred to as models. They are responsible for retrieving and sending data to the database.
Here is an example of a user DAO (or user model, if you prefer) using Primsa. (Side note: I love Prisma :star_struck:)
class UserDAO {
// ...
async findOneWhere(
where: Prisma.UserWhereUniqueInput,
select?: Prisma.UserSelect,
include?: Prisma.UserInclude
): Promise<UserType> {
// you can't use both select and include together, so...
if (select) {
return this.prismaClient.user.findUnique({
where,
select,
});
}
return this.prismaClient.user.findUnique({
where,
include,
});
}
// ...
}
This is where all queries related to a certain model are written. findUnique
, findMany
, deleteMany
and so on.
Since NodeJS is single-threaded, it relies a lot on the pub/sub pattern to run asynchronous actions. A listener “listens” for event propagation and performs certain actions in response.
For instance, when a user logs in, if we want to add their token to a specific Redis bucket, first we emit the event after a user logs in successfully:
class AccountsController {
async loginUser(request: Request) {
const payload = request.payload as LoginUserPayload;
const loginUserResult = await authService.loginUser(padyload)
switch (loginUserResult.status){
case 'success':
// emit a successful login event
server.events.emit(
events.accounts.ON_LOGIN_SUCCESSFUL,
loginUserResult.data.tokens
);
return loginUserResult.data
}
return responseHelper.returnGenericResponses(loginUserResult);
}
}
Then, inside our listener:
class AccountsListener {
async onLoginSuccessful({tokens}: {tokens: FreshTokens}){
try {
await jwtHelper.addTokensToBucket(tokens);
} catch (e) {
console.error(e)
}
}
}
It’s pretty useful to have all project configuration files in one folder. The two most notable configuration files I use in my projects are the env.config.ts
file, which is responsible for accessing environments variables, and the api.config.ts
file which contains information about the API to be accessed.
I cover these in detail in another blog post
One of my favourite aspects of using Hapi is the in-built ability to validate request metadata using Joi. It allows us to define specific, re-usable schema for validating request and response objects.
For example, to validate the login payload, we can define the following schema:
export const LoginUserSchema = Joi.object({
login: Joi.string().required(),
password: Joi.string().min(8).required()
})
And, to validate the response:
export const LoginUserResponseSchema = Joi.object({
tokens: {
accessToken: Joi.string().required(),
refreshToken: Joi.string().required(),
}
});
Then, let’s export them in a way that ensures it’s easy to use:
export * as user from './schema/UserSchema';
And, in use:
server.route({
path: '/accounts/login',
method: 'POST',
options: {
validate: {
payload: validator.schemas.user.LoginUserSchema,
},
response: {
schema: validator.schemas.user.LoginUserResponseSchema
},
},
handler: async request =>
await userController.getCurrentUserInfo(request),
});
Constants are self-explanatory. These are “constant” strings that can be re-used throughout a project.
It gets tiring hard-coding strings in a large project. Having to CTRL + SHIFT + F
to find out where you defined certain strings in your project even moreso.
Instead, it’s a good idea to define constant strings in their own dedicated file(s). My personal preference is grouping related strings in a single file.
For instance, here is a truncated example of my events.constants.ts
file. This contains the names of all events used throughout the project:
export default {
accounts: {
ON_REGISTERED_SUCCESSFULLY: 'user_registered_successfully',
ON_PASSWORD_CHANGED: 'user_password_changed',
ON_RESET_PASSWORD_REQUESTED: 'password_reset_requested',
ON_LOGIN_SUCCESSFUL: 'logged_in_successfully',
ON_LOGIN_FAILED: 'login_failed',
ON_REFRESHED_ACCESS_TOKEN: 'refreshed_access_token',
},
COMMENTS: {
ON_COMMENT_CREATED: 'on_comment_created'
},
ANALYTICS: {
USER_CREATED_ACCOUNT: "user created account",
USER_LOGGED_IN: "user logged in",
USER_CREATED_NEW_CART: "user created new cart",
USER_COMPLETED_CHECKOUT_FLOW: "user completed checkout flow",
USER_PASSED_CART_INSPECTION: "user cart passed inspection",
}
};
And, here is a truncated example of my messages.constants.ts
:
const alreadyExists = (tag, label) => `A ${tag} with that ${label} already exists`;
const doesNotExist = (tag)=> `That ${tag} does not exist.`
export default class Messages {
generic = {
TOO_SHORT: 'This field is too short',
TOO_LONG: 'This field is too long',
REQUIRED: 'This field is required',
INVALID_PHONE_NUMBER: 'Please provide a valid Kenyan phone number.',
ALPHANUMERIC: 'This field must be an alphanumeric string',
GENERIC_ERROR: 'An error occurred. Please try again later.',
GENERIC_INPUT_ERROR: 'Please fix the errors and try again.',
NOT_IMPLEMENTED: "This feature has not been implemented."
};
passwords = {
COMMON_PASSWORD: 'That password is too easy to guess. Try something stronger.',
TOO_SHORT: 'Your password should be at least 8 characters long',
INCORRECT_PASSWORD: 'Invalid username/email or password. Please try again.',
TOO_LONG: 'Your password should not be longer than 256 characters',
NO_DIGITS: 'Your password should have at least one number.',
LOWERCASE_CHARS: 'Your password should have at least one lowercase character.',
UPPERCASE_CHARS: 'Your password should have at least one uppercase character.',
SYMBOLS: 'Your password should have at least one symbol.',
};
http = {
BAD_REQUEST: 'Bad Request',
INTERNAL_ERROR: 'An internal error occurred - our engineers have already been notified.',
NOT_FOUND: "Sorry. We couldn't find what you're looking for.",
UNAUTHORIZED: "You're not authorized to do that.",
};
}
A client abstracts away code that accesses third-party modules.
For instance, in order to access Redis, we can create a client the covers all the functionality we frequently use.
import IORedis from 'ioredis';
import Env from './env.config';
const env = new Env();
class RedisClient {
getClient(options?: IORedis.RedisOptions): IORedis.Redis {
return new IORedis(6379, this.getRedisHost(), options);
}
async getItem(key: string): Promise<any> {
const data = await this.getClient().get(key);
return JSON.parse(data);
}
setItem(
key: string,
value: any,
expiryMode?: 'EX' | 'PX' | 'NX' | 'XX' | 'KEEPTTL' | 'GET',
expiryTime?: number
) {
return this.getClient().set(key, JSON.stringify(value), expiryMode, expiryTime);
}
public getRedisHost() {
if (env.isDevelopment() || env.isTest()) {
return 'localhost';
}
if (env.isStaging() || env.isProduction()) {
// the name of the Redis docker container
return 'redis';
}
return env.get('REDIS_HOST');
}
/**
*
* @param {?object} options
*/
connect(options?: IORedis.RedisOptions) {
const password = env.get('REDIS_PASSWORD');
const client = this.getClient({
password: !env.isDevelopment() ? password : '',
});
const host = this.getRedisHost();
client.once('ready', function () {
console.info('Connected to Redis cache at', host);
});
client.on('error', function (error) {
console.error(error);
});
return client;
}
}
export default RedisClient;
This is very useful because it then becomes trivial to swap out IORedis
for any other module in the future, if IORedis
stops being maintained, for example. This is referred to as grouping by dependency.
Here is an even simpler example:
import PostHog, { EventMessage, IdentifyMessage } from 'posthog-node'
import {env} from '@config';
export default class AnalyticsClient {
private client = new PostHog(
env.getVariable('ANALYTICS_KEY'),
{host: 'https://app.posthog.com'}
);
getClient() {
return this.client;
}
async capture(payload: AnalyticsPayload){
const data: EventMessage = {
distinctId: payload.distinctId,
event: payload.event,
properties: payload.properties
}
return this.client.capture(data)
}
}
Posthog is my favourite analytics service. But if I ever decided to swap it out for Mixpanel, for instance, this is the only file I’d ever need to modify! 😁
A module is a piece of code that abstracts away certain functionality.
It’s similar to a client in a lot of ways. However, it’s not an abstraction on top of another library. It provides its own functionality, and can be easily recognised due to the possibility of refactoring its code into a separate library.
For instance, here is the username generator I use:
import Wordy from './words';
import {randomnessHelper, utils} from '@helpers';
const wordy = new Wordy();
export default class UsernameGenerator {
async generate() {
const adverb = await wordy.getRandomWord(
'Adverb', // the type of word
'random' // the number of syllables
);
const adjective = await wordy.getRandomWord(
'Adjective', // the type of word
'positive' // positive adjectives only!
);
const food = await wordy.getRandomWord(
'Food', // the type of word
null // any number of syllables
);
const animal = await wordy.getRandomWord(
'Animal', // the type of word
null // any number of syllables
);
// randomnessHelper.coinToss() always returns 0 or 1
const isHeads = randomnessHelper.coinToss();
// if heads
if (isHeads) {
// randomly pick either the name of a random food or animal
const word = randomnessHelper.coinToss() ? food : utils.titleCase(animal);
// and concatenate the chosen adjective and chosen word
return utils.capitalize(adjective) + utils.capitalize(word);
}
// if tails
// use an adverb + adjective + animal name
return utils.capitalize(adverb) + utils.capitalize(adjective) + utils.titleCase(animal);
}
}
This could as well be a separate library, but I choose to include it together with my code.
Helpers are like smaller versions of modules. They encapsulate repetitive, complicated tasks that only makes sense within the scope of your project, but are used widely enough to require a separate definition to prevent repetition.
It might be a bit tricky to figure out what to split into a helper function, and whether to create a whole new file for specific helpers, but I believe that’s knowledge that comes with experience. A good rule of thumb is that if you find yourself using the same piece of code in two different unrelated files, that code could probably benefit from being split into its own helper function.
For instance, I generate access tokens when a user logs in and when refreshing an expired token. This is enough for me to create a jwt.helper.ts
file. However, this wouldn’t make much sense as a library because the way I generate tokens is probably very different from how you’d do it.
Also, consider the randomnessHelper.coinToss()
function we’ve used above. It’s a pretty simple function:
function coinToss() {
return Math.random() < 0.5;
}
However, considering the sheer number of times I use it in my project, it definitely warrants creating a separate randomness.helper.ts
file encapsulating the behaviour. Not to mention randomnessHelper.coinToss()
is a lot easier to make sense of than Math.random() < 0.5
thrown all over the place.
Care should be taken to prevent arbitrary code from ending up in an unrelated file. On one hand, it’s tiring creating a separate file for every new helper function you come up with, and on the other, you don’t want to crowd a single file with unrelated code.
A helpful tip would be to create a “miscellaneous” file for all helper functions that don’t fit in with other code in your file. As more helper functions make their way into your codebase, this file can eventually be split into smaller files.
That concludes the foundation of how I organise my NodeJS Applications.
It’s important to understand, however, that there are several ways to organise an application. This might not necessarily be the right approach for your project for your project.
Take for example the dictionary. Why is it in alphabetic order? Because it provides value for someone with a specific problem - when looking up the meaning of a certain word, looking it up according to the order of the letters is pretty natural. What if I wanted to look up words that rhyme with each other? A traditional dictionary suddenly won’t be of much use.
Have thoughts? Questions? Recommendations?
Feel free to hit me up
Copyright © 2022 Bradley K.