Forem: Studyportals

CI/CD as a Service

Dan Dobrescu — Fri, 24 Nov 2023 13:37:49 +0000

Abstract

Continuous Deployment has transformed how developers introduce changes, placing emphasis on automation, reliability, and speed. As applications grow, the pipelines that transition code from development to production must also evolve. The objective is to offer quicker feedback through shortened deployment cycles. These pipelines should be adaptable, reflecting the needs of the software they handle. It's vital to incorporate rigorous quality gates at key intervals, making certain that agility doesn't sacrifice quality.

This article dives deep into the synergy of AWS and its Cloud Development Kit (CDK), illustrating how these tools can help DevOps engineers craft CI/CD pipelines that are not only efficient but also agile and responsive to the ever-evolving demands of the software.

About Us

At Studyportals, we started using serverless automation to create environments and CI/CD pipelines from AWS Lambda’s early stages. As we started transitioning towards a microservice architecture, the complexity of a deployment also increased and, without constant maintenance, the deployment tools that we were using proved to be increasingly difficult to extend and maintain. Our engineering teams relied mostly on pipelines that were manually created for each service. This approach, while initially serving our needs, started to manifest some challenges. One notable struggle was devising a robust and uniform notification system to centralize all the deployment notifications in the same place and use a similar format. Additionally, pushing updates to the pipelines and to the resources backing up the pipelines across multiple accounts and regions posed another hurdle.

Our architecture is distinctly categorized into three primary types of services:

Docker (Monoliths): These are our traditional monolithic applications that encapsulate various functionalities within a single service boundary, containerized using Docker for consistent deployment and scaling.
Frontend Microservices: These microservices focus on the user interface and experience. Recently, we've standardized them by transitioning to Vue 3, ensuring a cohesive development approach. Each of these web applications is proxied via CloudFront, enhancing content delivery speed and security.
Backend Microservices: These handle business logic, data processing, and backend operations. Developed with a serverless architecture, they offer a modular and scalable approach.

Charting new territories

Our objective was to devise a CI/CD service that could proficiently manage each of our three distinct service types. The goal extended beyond merely ensuring smooth continuous integration and deployment; we also aimed to oversee the comprehensive creation of all requisite infrastructure for deploying the services. Additionally, we prioritized deploying production and non-production environments on distinct AWS accounts for the purpose of security and compliance audits, ensuring they remained isolated from one another. This approach not only bolstered security, but also offered enhanced separation and the ability to fully manage the service’s containment.

Building robust infrastructure can be complex, and dynamically generating CloudFormation scripts in JSON/YAML format can be a real headache, even when using frameworks like Serverless or SAM. But with AWS CDK (Cloud Development Kit) we tap into the flexibility of TypeScript, which is a game-changer. The basic building blocks of AWS CDK are called constructs, which map to one or more AWS resources, and can be composed of other constructs. This resembles the Composite Design Pattern, and just like in a Lego structure where individual pieces can be snapped together to form larger sections, in the AWS CDK, smaller constructs can be grouped together to create larger, more complex constructs. These groupings can further be connected to shape an even larger structure

Why AWS CDK stands out for us:

Reusable Components: Crafting infrastructure chunks that can be used again and again, making our setup tasks way more efficient.
Team Collaboration: Sharing these setups is a breeze, ensuring our teams can build consistently without reinventing the wheel.
High-Level Constructs: We can set up anything from security groups to ElasticBeanstalk environments. They’re designed to complete common tasks in AWS, often involving multiple types of resources.
Agility and Adaptability: The flexibility AWS CDK offers ensures we adapt swiftly to changes, keeping our infrastructure and applications in harmony.

First things first was to determine how are we going to trigger the creation of the infrastructure. Two solid paths lie ahead: GitHub Webhooks and GitHub Workflows.

Webhooks are straightforward - they ping our system with a notification when specific GitHub events happen. It's efficient but requires us to have a centralized system ready and waiting to handle those pings and translate them into action.

On the other hand, GitHub Workflows sit right in the GitHub space, allowing us to script responses to those same events without leaving the environment. This means that when a change is detected, GitHub Actions can prepare and execute the CloudFormation templates we’ve brewed up with AWS CDK, all in one smooth motion.

Why GitHub Actions caught our eye:

Integrated: It's in GitHub, keeping things tidy and contained.
Control: It lets us dictate the what, when, and how of action triggers.
Efficient: Less platform-hopping for the team equals more streamlined workflows.

Setting up infrastructure and CI/CD pipelines for various services and environments often involves dealing with overlapping components. The challenge here is finding a way to keep things DRY (Don't Repeat Yourself) and efficient, without tying our stacks up in knots with dependencies they don’t need. Each stack needs to be its own island - able to function alone, but also be part of a bigger archipelago of services, sharing resources smartly where it makes sense, and avoiding unnecessary duplication.

This means we need a setup that’s lean, avoiding unnecessary duplication, and mindful of AWS quotas, while also ensuring our stacks can go solo on any AWS account without tripping over each other.

The obvious choice was to split things in two main stacks for each service - create an environment stack holding the communal resources (e.g. CodeBuild instances, S3 buckets, CloudFront policies, etc. ) and a service stack with the particular resources that an individual service needs to operate. This way, everything common is in one place, and everything specific has its spot, which interestingly, is a model AWS Proton also follows with its environment and service stacks. It was nice seeing AWS Proton, a fully managed application deployment service, coming out with a similar approach after we'd adopted this strategy. While we’ve set a solid course with our current setup months before AWS Proton was released, exploring an integration with AWS Proton is something we’re keeping an eye on for the road ahead.

This is how it would look like in big lines:

As we dug deeper into the CI/CD architecture, it became clear that we needed to pay close attention to our branching strategies. This wasn’t just a side note - it was crucial to shaping our entire CI/CD setup. We needed to be nimble and support two main branching strategies:

Git Flow variation, used mostly by projects that require a permanent testing environment
Trunk Based Development, used by small projects and microservices with a lower complexity level

Based on the branching strategy, we managed to identify the points where the CI/CD should run and spin up environments:

PR environment, when creating a PR from the feature branch to the default branch
Testing environment, when pushing to the default branch
RC environment, when creating a PR from the develop branch to the release branch
Production environment, when pushing to the main branch.

* for trunk-based development strategies, the Testing environment wouldn’t exist and the PR environment would coincide with the RC environment

Putting it all together

Piecing all of this together was not a walk in the park, to say the least. While on paper everything might seem linear, our real-world implementation brought us face to face with challenges we hadn't quite anticipated. The most notable among these was the intricate relationship between the environment stack and the service stack.

Using AWS CDK proved powerful, but it did have its challenges. One issue we bumped into was the occasional intent of CDK to recreate a resource in the parent (environment) stack that was still in use by the child (service) stack. This isn't a new issue, as highlighted by Dependency stacks cause update failure. There are strategies to get past these cross stack references, usually requiring manual steps. However, we aimed for full automation. Our workaround? We leaned on sharing data between stacks using the SSM Parameter Store. This approach was particularly helpful for those situations where we foresaw potential issues, enabling us to update a resource in the parent stack without disturbing its relationship in the child stack.

Another struggle was with specific account limits, such as the number of CloudFront Custom Policies or the number IAM roles, which meant that we had to introduce account level support stacks in addition to our environment and service stacks. These account-level stacks housed resources that could be utilized by multiple services in the same account, and were designed to be long-lasting stacks.

With the groundwork laid out, let's pull the curtain on our service blueprint:

For every service, we established a service descriptor file (cicd.json) that’s versioned in GitHub along with the rest of the project’s source code. This helped us tailor the stacks we deploy for each specific service’s requirements.
The environments' branches will match the branches on which the GitHub workflow listens for events, and based on the event type (pull_request or push) we choose whether to create a PR/RC or a Testing/Production environment.

The environments from the descriptor extend an out of the box feature of CDK, also called Environments. It gave us a ton of flexibility: we could deploy our setup to any AWS account or region we wanted, making sure our deployment was as flexible and scalable as the services we were working on.

With every GitHub event, our system responds promptly. Based on the configurations we've established with the CDK's Environments feature, our entire setup – including the environment and its associated pipelines – either undergoes creation or receives updates. This ensures we're always in sync with the latest code changes. After these changes are completed, the AWS CodePipeline springs into action. It retrieves the updated code from git, then builds, tests, and deploys it to the intended environment, likely followed by additional testing. By doing so, we ensure a high standard of quality while maintaining the adaptability and efficiency of our deployment method.

The road ahead

Continuous Deployment, with its emphasis on speed, reliability, and automation, has revolutionized the way developers operate today. While setting up our infrastructure and improving our CI/CD pipeline, we realized the benefits of the AWS ecosystem and the versatility of the Cloud Development Kit (CDK). The journey with Studyportals, adapting serverless automation, refining branching strategies, and accommodating the dynamic nature of microservice architecture, shed light on the complexities and challenges inherent in the transition process. Yet, it was this very journey that equipped us with the tools and strategies to overcome these challenges and streamline our operations.

AWS CDK stood out as an invaluable tool in this process, offering modularity, reusability, and agility, which ultimately enabled us to keep our infrastructure and applications seamlessly aligned. However, it's crucial to remember that, like any tool, CDK has its own set of challenges, necessitating innovation and strategic workarounds to fully harness its potential.

Going forward, we remain committed to evolving and adapting. Technologies will continue to change, and new challenges will emerge. But with a robust framework in place, a commitment to continuous learning, and an emphasis on adaptability, we are confident in our ability to navigate the future landscape of CI/CD.

A migration to Vue 3: part 2

Joos Huys — Tue, 28 Mar 2023 09:47:23 +0000

Introduction

Last summer, I wrote a blogpost about the migration of most of the microservices we have at Studyportals to Vue 3. The blogpost painted a picture of our approach and the progress made up till that point, but there was still a lot of work left to do. At the time, we found ourselves right in the middle of the migration process, with five microservices moved over and eleven more to go.

A couple of weeks ago, the last migration-related steps were wrapped up. With the fourteen-month-long process coming to an end, it was time for another blogpost. This blogpost is a continuation of the previous one: A Migration to Vue 3: part 1. It highlights the steps that we have taken since then and the things we have learnt from going through this entire process.

Time for a disclaimer: If you have not read the first blogpost yet, you might feel like you are missing key details to understanding the contents of this one. Questions might pop up such as: What is the microservice architecture that Studyportals uses? Why did they decide to migrate to Vue 3? And what approach did they define at the beginning? If you are looking for the answers to those questions, please check out the previous blogpost.

Remaining steps

When the last blogpost was written, there was still plenty of work left to do. The steps that lay ahead of us can be labelled as follows:

Migration of remaining microservices: With about one third of the microservices moved over, quite a bit of the actual migration work was still left to do.
Clean-up: The final bits of work could be carried out only once all microservices were migrated.
Retrospective: No project is complete without a retrospective, in which those involved reflect on the whole process to see what can be learnt from it.

The following sections will describe each of these steps in more detail.

Migration of remaining microservices

First off, it is important to note that the overall plan, which had been defined at the beginning of the migration project, did not really change. Details were filled in along the way, such as the exact order in which microservices were migrated, who would take care of what, and additional smaller challenges that needed to be tackled, which we had initially not foreseen.

There were a few things that did change.

Sharing the load

Of the five microservices which were migrated first, four were migrated by the same Scrum Team. This fact underlines the biggest difference between the way the migration started and the way it continued afterwards. By the end of the project, four Scrum Teams had been involved, each of them responsible for the migration of at least three microservices. Aside from the front-end and full-stack engineers (the people who work most with the microservices on a day-to-day basis) back-end engineers also chipped in.

As an example, the work of migrating our search page microservices was distributed among three Scrum Teams.

It was important to involve several Scrum Teams and several disciplines, not only because that would make the workload more manageable, but also so that the knowledge of Vue 3 could be shared by a bigger group of people across the Engineering department. Moving forward, the migrated microservices would have to be maintained by those same people, after all.

The group of people included in the migration grew substantially in a short amount of time. This was possible by ensuring that the knowledge gained from the first migrated microservices was sufficiently shared. Those involved in the earlier migrations were included whenever the migration of a new application was kicked off, and they were also available for any questions or issues that came up along the way. Not much of the knowledge was put in writing, but a lot of tips and tricks were shared via word-of-mouth.

Tooling

As the migration process progressed further, we looked for ways that newly migrated applications could benefit from the learning points of those microservices which had already been moved over. This resulted in some new tooling. One example of that was a Github repository set up with the barebone structure of a “Hello World” Vue 3 application. It contained configuration for Webpack, NPM, linting, and additional generic code that all microservices would need.

Furthermore, some NPM packages were built in-house, which brought added benefits such as ensuring that a microservice would have all the right dependencies installed and making the setup & unit testing of composition-API-based components easier.

Clean-up

Once the last microservice was moved over to Vue 3, it was time for the final clean-up. Other migrations might have depended on more “bridge” code, which provides functionality that serves the sole purpose of having two versions run side by side. In our case, the clean-up was a short and smooth process. The migration of the individual microservices was tackled in a proper way, and not many leftovers remained.

Throughout the migration process, we tried to have as few as possible Vue 2 and Vue 3 microservices running on the same pages, because the extra included kilobytes would negatively influence page speed. To some extent, a temporary negative impact was unavoidable. Once the Vue 2 framework / configuration was removed from our pages, we were happy to see the efforts of the clean-up back in our page performance metrics. Our pages were once again as fast as they had been before the migration.

Retrospective

With the clean-up taken care of, the migration reached its completion, but the project was not fully wrapped up yet. When working with Scrum, it’s good to take the time to reflect on every sprint; but the same is the case for bigger projects such as a year-long Vue migration. After all the time and work that was put in, it was worth it to organise a retrospective and see what could be learnt from the process.

For our retrospective, at least one representative of each Scrum Team was invited. Our goal was to extract useful takeaways that would help us for potential future migrations. We were happy with the way the migration to Vue 3 played out, but there is always room for improvement. The retrospective was structured as follows:

The timeline that was used for the Vue 3 migration retrospective.

A timeline was prepared, which created an overview of the steps that were taken from start to finish. Under the timeline, four tracks were added: Keep, Add, More, and Less.
We observed the timeline and discussed together all the details that we could remember from the migration process. Any thoughts that popped up about what parts to keep, add, do more of, or do less of, were written on post-its and added to the respective tracks.
By the time the timeline was enriched with any details we could think of, each of us voted for the three post-its which we thought to be most valuable (by dot voting).
The emphasis that the votes created helped us to formulate three main takeaways.

Takeaways

The biggest points of improvement, which we identified during the retrospective, are the following:

Do more elaborate planning / refinement up front. In our migration, some additional work was taken along, such as the upgrade from Webpack 4 to Webpack 5. Other aspects, such as automated testing and the bottlenecks some of the microservices would face, were not discussed in enough detail when planning the migration at the start. It could have helped us to allocate more time for planning at the start and to create a more detailed overview of what to include / exclude.
Aim for a reasonable story / pull request size. The timeline visualised very well the different approaches that were taken with the migration of each microservice. The changes of some bigger microservices were combined into one big pull request, whereas the changes of other microservices were spread across many smaller pull requests. We concluded that, when choosing how to split up code changes, one should aim for the sweet spot of creating chunks of work that are manageable to be completed in a sprint and optimal for reviewing, while limiting overhead as much as possible.
Approach migrations as a team effort. Especially in the second half of the migration, we did a better job at spreading the workload across the Engineering department. More people could have been involved sooner, to further accelerate the migration. It’s important to be pragmatic, yet critical with capacity planning for a project such as this one.

Adding notes under the timeline during the Vue 3 migration retrospective.

Conclusion

Overall, our migration to Vue 3 can be considered successful. Like any process, there were things that went well and things that could have gone better. We took the time to reflect on the whole project and defined takeaways which can help us with future migrations.

I hope that, after reading this blogpost and the previous vue-migration-related one, you have gotten useful insights that can help you tackle your migration in an effective way, whether it is to Vue 3 or something else entirely.

ViteJS: Why you should use it and why we chose not to

Michael Aplas — Tue, 13 Dec 2022 15:46:24 +0000

Studyportals is a company which has been built, fundamentally, from web technology. As such, it’s very important to us that we keep up on the changes and assess when to adapt; though we strive to keep our stack up-to-date, occasionally we need to balance that out with pragmatism. You may have read about our recent journey in upgrading to Vue3. As part of that, our engineers had to assess a lot of the tooling we use surrounding our Front-End stack. That is where we met Vite and delved into a comparison with our current bundler, Webpack.

Effective JavaScript bundling is a huge part of our ability to successfully build, maintain, and run our portals and ~20 microservices. The challenge we have is achieving a modern and effective bundling setup that adheres to our quality guidelines, whilst also ensuring our engineers are empowered by a smooth development experience.

In this blog, we will explore what Vite is, how it aims to improve developer experience, and dive into some of the decisions we made when assessing it as a viable alternative to our current setup.

What is Vite?

Vite (pronounced \ ˈvēt \, French for fast) is the “next generation of Front-End tooling” developed by Evan You, the founder and creator of VueJS. It is a build tool which comes with a pre-configured dev server and also bundles your code for production.

The first stable version, Vite 2, was released in February of 2020 and did well to gain traction. With the development of a successful ecosystem, community adoption has grown exceptionally well to its current 1 million+ average weekly downloads. Not too long ago, Vite 3 was released with a generous payload of new features and support. The Vite team now aims to release a new major version annually, to align with Node.js’s EOL.

It's no coincidence either, that the Vite 3 release has aligned neatly with the re-write of VueJS to Vue3. The Vue team hasn’t made it a secret that Vite, with its latest features, has been built as a replacement for vue-cli, with the ultimate goal of providing all of the spanners a modern Front-End build kit needs, with drastically improved developer experience.

Webpack vs Vite

The reason Vite claims to be a “new” type of build tool is that, as a dev server, it functions fundamentally differently to a traditional bundling setup. Vite natively handles ES modules and uses a pre-configured dev server, meaning things are very snappy compared to a Webpack with CommonJs setup, for example.

But, how does this actually work?

Well, Vite allows us to author our code as native ES modules (ESM). The browser will then take these ES modules, parse the files looking for import statements. It then makes HTTP requests for each required module, to the dev server. The dev server simply serves up each of these modules as is. Think of it as introducing a level of asynchronous behaviour that was not previously possible with CommonJs and AMD.

Esbuild + Rollup + TypeScript

Vite uses esbuild under the hood for ESM and bundles your code for production using Rollup. So, the speed benefits are really around developer experience here; Vite pre-bundles all your node modules with esbuild and serves them to the browser on page load. The rest of the application modules are served up as needed, meaning it's really fast to get going.

A great feature of Vite is its out-of-the-box support for Typescript. Vite makes it easy to scaffold your project with Typescript, or simply add Typescript files to an existing project without any extra dependencies or configuration changes.

It is worth highlighting an important caveat of esbuild and Typescript here:

Since esbuild is written in Go, it is naturally transpile-only. This means that for projects built in Typescript, there will be limited type checking in the compiler with Vite.

Evan You has defended this as “a small price to pay” for the significant speed improvements; when you think of the extensive support we have with modern IDEs and extensions, he’s probably right. It is good to remember, though, that issues that slip through will be caught by the Rollup build, anyway.

Vite-cli

Vite is very streamlined to install. It adds a single dependency to your project, which installs a few necessary packages. By running an npm init vite command, you can get a hello-world project setup and scaffolded with a dev server running in less than 20 seconds.

Although likely to have been developed to be used with Vue, Vite is framework agnostic, so the CLI offers us a few options when setting up a project. The list includes Vanilla, Vue, React, Preact, Lit, and Svelte. As mentioned above, Typescript is out-of-the-box supported, so can be preset along with JavaScript (and Nuxt for Vue projects!).

Though it doesn’t offer loads of features, Vite’s focus on being streamlined and minimal seems logical, when considering the goal here is speed and efficiency.

Vite CLI quick setup

Sounds great, sign me up!

… Not so fast.

So far, Vite sounds like exactly the tool any team would want to use. It has all the modern features you need, is framework agnostic, and blisteringly fast. So, this begs the question, why did we choose not to use it? Well, that really boils down to how we work at Studyportals and the maturity of our stack and tooling.

At Studyportals, we run 5 platforms (or, portals, as we call them) to help people all around the world find education options at various levels. A few years ago, when endeavoring on an initiative to modernize our stack, we made the decision to implement a microservice architecture and chose Vue to be the primary JavaScript framework to enable this.

An example of a page using Studyportals microservices

Today, we are running a suite of roughly 20 Front-End microservices across our portals. Keeping all of these services consistent and maintainable requires a strong set of tooling and quality criteria. One of the tools which is paramount in achieving this is a package we developed called WebpackHelper.

WebpackHelper, as the name suggests, helps us quickly set up Webpack across all of our services. It comes with a few flavours, includes our core dependencies, and outputs Webpack configuration that allows us to bundle our code in a way which is standardised and highly optimised for production.

Studyportals WebpackHelper package

For a number of reasons, we decided that this was not the right moment to migrate over to Vite, even though we knew it would improve our developer efficiency (and probably happiness too!). As you can see with WebpackHelper above, we have a mature set of tooling surrounding our Webpack setup which means our services are standardised, optimised, and easy to deploy. To add to this, our teams have built up solid knowledge of Webpack over the years and are well-versed in maintaining and evolving the current setup.

Ultimately, we realised that the amount of resources it would take to migrate our existing solution to a Vite based setup with a similar level of maturity, was too high to offset the value it would provide us. Not to mention that we are already starting to see ESM support in Webpack-cli, so it’s likely just a matter of time before we see improvements in developer experience too.

Takeaways

Clearly, there are some changes coming to the way modern web applications are developed. With the rise of ESM, it seems likely that we will see a wave of support in all the long-standing tools like Webpack, Rollup, and Parcel. So, the question now beckons, should you use Vite?

Vite is undoubtedly a great tool with a great set of features. But, the true moral of the story here is that it depends. It depends on the tooling you have already setup, it depends on the project you are setting up/migrating, and it depends on your team. Our decision came down to an assessment of how much value this change would bring, vis-à-vis our success criteria; ultimately we decided that on this occasion it would not be sufficient. Regardless of that, Vite is an exciting project that I will be keeping my eye on and I suggest you do too!

A migration to Vue 3: part 1

Joos Huys — Thu, 30 Jun 2022 08:12:48 +0000

Introduction

At Studyportals, we have around 20 microservices running on Vue.js (commonly known as Vue). Ever since we started working with Vue, we have been using version 2. Vue 3 has been out for a while now. This year, in 2022, we decided that we would pick up the migration of most of our microservices to Vue 3.

Up till this moment, all our Vue 2 applications ran without problems. So, why did we decide to migrate any of them? Although we are not in dire need of any of the new features Vue 3 offers at this moment, we still decided to migrate most of our applications.

The reason for that is that we want to ensure our codebase stays maintainable and, with that in mind, it's important to not lag behind on older versions. This way we can make sure that we can optimally keep benefiting from our good page speed, and overall development efficiency. Vue 3 is the latest stable version, fully built with Typescript at its core, allowing us to build even more performant and high-quality applications.

In this blogpost, I would like to share what our approach is to this challenge of upgrading tens of thousands lines of code across hundreds of components spread over about twenty microservices.

Our infrastructure in a nutshell

Several years back we decided to utilise new reactive JS frameworks for new applications that we build. There are plenty of good frameworks out there, but we chose to build our applications in Vue. Vue came across as a fast, light-weight, reliable, and easy-to-use framework that would be around for many years to come.

We opted for a microservice infrastructure, as migrating all of our old code to Vue would not be worth the effort. This means that the foundation of our websites is still in place, running on the older architecture we have had for many years. The newer applications inside the websites that we build are able to run stand-alone and can be injected onto any page.

An example of one of our pages, on which multiple microservices can be identified.

We use a service called Bob, which is our page compositing and lay-out service. It helps to integrate microservices on our websites in a structured way. Additionally, Bob ensures that, when multiple microservices run on one page, shared dependencies are only loaded in once. The same goes for the core Vue code that is the same for every microservice.

Making a plan of approach

In the last months of 2021, we started to look ahead to the next year and consider which technical challenges would fit nicely if picked up as part of our strategy for 2022. After all, we strongly believe that, at all times, a bit of time should be dedicated to constant maintenance and innovation. It is necessary to make sure we stay in control of the applications we build.

Vue 3 was one of the topics that was on our radar. Since it had first come out, we had kept an eye on developments surrounding this new version. The core packages necessary to use Vue 3 had been published and were out of beta at this point. This meant that it was a good moment to dedicate time for the transition to Vue 3.

We knew that a migration in some shape or form was coming up. And so, we used Q4 of 2021 to do the necessary research that would allow us to formulate a plan for this big change.

Like any plan, our Vue 3 migration plan was soon defined as a couple of high-level steps:

Research
Deciding on strategy
Take care of migration
Clean-up

Research first

We read more about Vue 3 so that we could discover any opportunities or limitations that we had to be aware of.

Additionally, we took a closer look at each of our applications and pinpointed details such as the pages on which they run, the Vue syntax used for components, the Webpack setup used for deployment, and additional dependencies we had to be aware of. We made an overview that allowed us to compare all of our microservices.

The overview that allowed us to compare our microservices to support decision making.

Deciding on strategy

We took all of the information we had gathered and discussed it. We made sure that at least one representative of each scrum team was included in the discussion.

This way, we could easily agree on some important decisions, such as which products should be migrated, and in what order to ensure that the impact on page performance would be minimal.

The main decisions made were the following:

One-page and internal microservices first

Microservices running on just one page would be migrated first, along with some microservices that only our internal users and clients use. But, why did it make sense to tackle those first?

Our Personality Test is a simple microservice and runs only on one page; therefore it was picked up first.

There are no overlapping dependencies that can be shared between Vue 2 and Vue 3 applications. So, regardless of Bob’s efforts, more kilobytes will be loaded in on a page with Vue 2 and Vue 3 applications running side-by-side.

So, by last migrating those microservices that run on many pages, including core landing pages, the timeframe during which the performance of those pages is negatively impacted is kept to a minimum.

For about half of our microservices, a specific migration order was already defined at this point.

Some microservices not migrated

Microservices that are not actively developed on and are not loaded in initially do not impact page performance much. Hence, we decided not to migrate those applications, as the needed efforts would not be worth it.

Component syntax

When migrating to Vue 3, we would configure our components using the Composition API and the so-called Vue Class Component setup. Later on, this decision changed to: only use the Composition API in Vue 3. A bit further down this change of plan is clarified a bit more.

A better defined plan moving forward

With all our scrum teams on the same page and the most important decisions made, the broad migration plan mentioned earlier could now be made more specific. The taking-care-of-migration step could be further specified as follows:

Prepare tooling around microservices: Resources, such as NPM packages developed by Studyportals, should be ready to be used in Vue 3. More details are mentioned on this further down.
Kick off with a first application: It should be a relatively small and simple microservice, so that additional time needed to get familiar with Vue 3 syntax does not result in a huge scope and a migration that takes many weeks to complete.
Continue with a second application: Unlike the first one, this one should be big and challenging. After having gotten acquainted with Vue 3 by migrating the first application, it is useful to now pinpoint the biggest remaining bottlenecks that need to be taken into account for the migration of other microservices.
Diverge: Get more people involved, so that it is possible to migrate multiple microservices alongside each other. In our case, the first two microservices would be tackled by the same scrum team. So at this state of the migration, all the insights gained could be shared with other scrum teams. It is important that all of the front-enders in the company get involved and get acquainted with Vue 3. After all, they should all be able to work with it and can help to make the overall migration go faster.

The Enrolment Review Tool is our biggest, most complex, internal microservice. So, it was migrated second.

Takeaways

With any big project, you can have a well-thought-out plan and still encounter some unforeseen issues and challenges. You can also stumble upon new opportunities that are worth exploring. While attempting to migrate the first few microservices, we also had a few unexpected findings. Some of them are worth mentioning here as takeaways.

Migrating in-house-built NPM packages

We use Webpack to build and bundle our microservices. Before we started the migration to Vue 3, we were using Webpack 4. We figured that, while we are upgrading Vue, we might as well upgrade to the latest version of Webpack as well: version 5.

What we did not foresee however is that this Webpack-upgrade also impacts pieces of Webpack-dependent functionality we develop in-house and make available for our microservices as NPM packages. This meant that extra work was needed to upgrade additional dependencies that we initially did not think of.

Some NPM packages that needed to be upgraded, which are used in microservices for various purposes.

Vue Class Component not working

In Vue 3, there are two main ways to configure your components: the Options API and the Composition API. In Vue 2, we used another way which is known as Vue Class Component. This setup depends on a package that is not developed by the Vue core team. Initially, it seemed that the maintainer had also got his package working in Vue 3.

So, we thought we could limit our migration efforts somewhat by continuing to use Vue Class Component for some microservices in Vue 3. We had a working proof of concept, but when we tried to migrate the first microservice in this way, it did not work. The reason for this, as we found out, was that the latest minor version of Vue 3 was not supported anymore by the package.

We concluded that it would be best not to make use of this unstable setup any more and move over fully to the Composition API. This is always the risk when you rely on packages that are not well-maintained. For our migration, it meant that more time would be needed than we initially expected.

Replacing non-essential dependencies

One of the opportunities we recognized was that we could fairly easily build the functionality of dependencies such as Vuex and Vue-Router ourselves with the features that are available in Vue 3. We figured that the code needed for that would be small and simple enough, and that it would fit nicely with the code changes we needed to make anyway to move to Vue 3.

The advantage of it is that we have less dependencies to worry about and that we can reduce the bundle size of our applications.

Conclusion

At the time of writing this article, five microservices are migrated with another two in progress, and the rest still waiting to be picked up later in the year. So far, we are quite happy with how it is going, and we are still able to follow the strategy we had initially defined.

The aforementioned issues, challenges, and opportunities caused some delays for the migration of the first microservices, but none major enough to put the planning that we had in mind for the year in jeopardy. The insights gained will surely help speed up the migration of microservices that are up next to be migrated.

We are optimistic and excited to see how this big migration project will progress. Hopefully, at the end of the year the core parts of our codebase will be up-to-date with the latest technology, making it therefore maintainable and sustainable for many years to come.

Update (March 2023): I wrote a second blogpost, A migration to Vue 3: part 2. If you're interested how the rest of the migration process turned out, go ahead and check it out!