Breaking the monolith

Lots of articles have been written about the topic of microservices and ways of transforming legacy large enterprise platforms into scalable, performant, manageable, granular applications and services. Microservices is not a new concept, it's just a new term applied to proven software architecture and design patterns. From components to SOA to microservices, they all have plenty in common, specifically ability to separate concerns, reuse, scale, manage, deploy without resetting and changing the world.

With that in mind here is what I found important when the decision has been made to "break the monolith". And believe me, sooner or later this decision will definitely be made if you want your business to grow, evolve and stay agile. There is a lot more than technology decisions that have to be considered, but from purely technical point of view, everything you do will have to revolve around IaaS/PaaS (e.g. AWS), containers (e.g. docker), events/messages (e.g. Kafka, ActiveQM, RabbitMQ, SQS), horizontally scalable databases (e.g. Cassandra, MongoDB), Single Page Web Applications (e.g. Angular, React), CI/CD (e.g. Bamboo, Jenkens, Ansible, Chef, Puppet) and an open, agile, lean, DevOps culture.

Here is a high level process:

Discovery

  • Document all of the existing business processes, web, backend and messaging components. Doesn't have to be a 100 page document. High level overview is enough at this stage in order to assess the complexity of the existing platform.
  • Document goals of digital transformation. Why and what are the important questions to answer.

Assess your development team's strengths (programming languages, databases, tools, frameworks)

  • Are you a RoR/Postgres shop? Java/Spring/Oracle? PHP/MySQL? Python/Django/MongoDB? The list goes on, but the goal here is to understand that if you have a good development team with solid knowledge in specific stacks, it makes sense to base your target architecture on those skillsets. Introduction of new technologies will be inevitable, but should be minimized to backend databases, tools and frameworks, and not core programming languages.

Assess your ops/infrastructure team's skill set and experience

  • Most legacy platforms are managed by traditional operations/infrastructure teams, separated by a brick wall from development teams. This is just the way things are. Companies are slowly moving to DevOps culture, which means different things to different people, but the goal here is to make sure to identify the gaps within current it/ops and to gradually fill them via mentoring/training and new hires.

Research microservice frameworks that are most widely used with your team's existing technology stack.

  • If you're a Java shop then go with Java/Spring Boot. For JavaScript go with Node.js. Python, .Net, you name it, each stack has good microservices frameworks. Do some research, build a POC and get your team excited.

Research JavaScript frameworks.

  • It's very important to understand that the traditional monolith, server driven MVC style web applications do not scale in the new world and are anti-patterns for microservices based architecture. SPA (Single Page Application) is the way to go, with separate deployments of web applications from all microservices. Angular, React, Amber are a few popular frameworks to choose from.

Decide on on-premise vs cloud vs hybrid target infrastructure.

  • It's not easy to just move to AWS or Azure if you have existing large on-premise infrastructure. Decision on what goes to the cloud vs what stays on premise has to be given serious thought. Take following into consideration: security, scalability, fault tolerance, cost, team skill set, ease of deployments.

Design high level architecture.

  • Ok, here is where the fun part begins. How do you want your super-duper microservices based platform to look when it's all done (let's say 12 to 24 months from now) and why? How can you demonstrate the benefit of a large digital transformation initiative in 10 to 20 powerpoint slides and architectural diagrams and get full support and buy-in from the C-level executives? Well, make it balanced with the right level of business benefits, required investment, research/comparisons and of course sleek diagrams. You will have to go through multiple iterations, with multiple stakeholders until everyone (or most) are happy with the direction. Word of caution - there would often be resistance to change and even when the initiative is on its way there will be those who would enjoy to see your hard work go up in flames. But you're not going to let it happen!

Create POC for any technology not currently used by existing technology team.

  • Let's assume that your current development team is using Java/Spring MVC, Tomcat and Oracle RDBMS and your target architecture consists of following technologies (AWS, Java, Spring Boot, SQS, SNS, ECS,Angular, Cassandra, Solr, Docker, etc). Here is where you need to make sure that development team understands all the moving parts, how they are built and how they interact with each other. Clone some repos from GitHub, create sample services, build docker images, deploy to AWS, hit some /hello endpoints, get the developers excited!

Partition architecture into "framework candidates" - parts of the architecture that can be developed as frameworks to simplify and standardize future development effort.

  • Some of the obvious candidates are: Core (common db config/access, utilities, rest clients, security, logging, monitoring), ETL Framework (generic way to move data from legacy database to new database), S3 Storage Service, etc.

Select first application to develop using new architecture.

  • This should be medium complexity new application. Not too simple and not too complex. Should preferably include most components of the target architecture (web application, microservices, events, etl, database). The goal of this first project is to show without any doubt that the new architecture is solid and make it an example and a reference implementation for all future development.

Break the monolith. This deserves a detailed separate writeup, but let's stick with high level for now.

  • Document future detailed architecture ( web applications, microservices, interactions, security, etc).
  • Prioritize refactoring effort.
  • Prioritize new services/applications effort.
  • Adopt agile process.
  • Involve DevOps from day one.

Have fun! Really!

---- "Life is a series of natural and spontaneous changes. Don't resist them - that only creates sorrow. Let reality be reality. Let things flow naturally forward in whatever way they like." Lao Tzu ----