How Haven Technologies moved to Kubernetes with GitLab

Haven Technologies is an insurance technology software as a service (SaaS) provider based in New York, New York. Prior to becoming Haven Technologies, the company helped transform the life insurance industry in 2015 with the launch of Haven Life, a digital insurance technology company that allowed customers to get a quote, apply for a policy, get approved, and manage that policy, almost entirely online. Now, Haven Technologies makes its unique cloud-native insurance platform available to other businesses in the life, annuities, and disability industries. An example was HealthBridge, a free online life insurance policy offered by MassMutual to eligible frontline healthcare workers and volunteers in the wake of the COVID pandemic, that was built and launched in just a few weeks using the Haven Technologies solution.

As an insurance technology company, Haven Technologies is strongly aware of the need to protect sensitive customer data. The company’s code is valuable intellectual property, says platform engineering manager Evan O’Connor.

Haven Technologies started using GitLab for version control and source code management (SCM). By 2019, the company was heavily utilizing GitLab CI/CD and container registry. Developers were continuously adding new projects, each of which had pipelines to build Docker images, run units, regression and integration tests, and security scanning.

An early project was to move all Docker builds to GitLab and build images there, which was a big time-saver as images were pre-built and tested come deploy time, says O’Connor.

In late 2019, Haven Technologies began its journey to migrate to a self-managed GitLab instance on Kubernetes using Helm charts to allow for greater scalability, customization, and security. Prior to that time, Haven Technologies used the Omnibus deployment, a more commonly used installation that requires minimal configuration to get up and running.

Before Kubernetes, the team used Convox, which is built on Docker and ECS (Elastic Container Service) and aims to make it easy to deploy and manage applications in the cloud. This setup worked well as a smaller organization, but O’Connor says that eventually, they reached a breaking point where Convox did not provide enough control over their cloud resources. During peak usage days (especially around release time), GitLab would become painfully slow and users would receive frequent 500 errors in the UI and failed pipelines, he recalls.

As an innovator in the insurance tech space, the Haven Technologies engineering team felt they were a natural fit to become an early adopter of GitLab’s Helm Chart. Haven Technologies leadership was given fair warning that because the product was new, they might encounter roadblocks that would require help from GitLab’s support and engineering teams.

But Haven Technologies was undaunted. “Our team was somewhat new to Kubernetes and Helm, but GitLab became an opportunity for us to learn and gain experience that we could apply when we moved the rest of our apps to Kubernetes,” says O’Connor.

And GitLab made the communication seamless. “GitLab’s commitment to an open source community meant that we could engage directly with engineers to work through difficult technical problems,’’ says O’Connor. “Overall, I would recommend adopting the Helm Charts if your team is experienced with Kubernetes or if, like us, you are willing to brave the unknown with the goal of using the latest technology and a desire to contribute to improving the Helm Charts for everyone.”

The decision to adopt GitLab’s Helm Chart was in part influenced by a larger move to Kubernetes for all of Haven Technologies’ applications. At the same time Haven Technologies decided to become an early adopter of the GitLab Helm Chart, the engineering team was also planning to move the entire production and user acceptance testing (UAT) environments to run on Kubernetes. It made a lot of sense to deploy GitLab in the same way as their other applications, so all of the configurations would be in one place, says O’Connor.

The other advantage of Kubernetes was that autoscaling and high availability are native to the software and are easy to enable via Helm Charts. “If we went in the direction of multiple EC2 instances, we would have to manage many different configurations that each would require their own solutions for autoscaling and high availability,’’ says O’Connor.

The first prerequisite for running GitLab on Kubernetes is to build a cluster. Haven Technologies chose Amazon EKS, since all of the company’s applications reside in AWS. Engineers decided to build out GitLab in both UAT and production environments to allow them to test upgrades and other configuration changes before they went live.

Haven Technologies uses Jira and Terraform, and uses GitLab pipelines to automate Terraform tasks.

O’Connor highly recommends maintaining a staging environment for GitLab to maintain stability and reduce the risk of data loss during difficult updates. “GitLab makes this easy, and only one license is needed to support both environments,’’ he says. “Using Terraform and Helm, our UAT and production environments are configured almost identically, aside from more conservative scaling in UAT to save money.”

In the future, O’Connor says engineers will aim to build a way to easily shut down or spin up their UAT GitLab system so they do not have idle machines running when they are not needed.

Another struggle was finding a way to load test their UAT setup before cutting it into production. Volunteers from the development and QA teams were asked to help load test GitLab and search for abnormalities. O’Connor’s team created a spreadsheet and had users enter any abnormalities they found. This helped the team fix some misconfigurations before fully cutting over, he says.

Engineers had built out the monitoring portion of the Helm Charts after they had cut over their production GitLab system to Kubernetes. This made diagnosing issues that were caused by load especially difficult. At the time, all they had were pod-level CPU and memory metrics reported by the Kubernetes cluster. These dashboards helped team members resolve issues where their pod limits were set too conservatively.

Haven Technologies also utilized AWS CloudWatch metrics and log queries to look for bottlenecks and correlate issues with AWS resources.

Going forward, as issues come up when testing GitLab on Kubernetes, O’Connor says they will create dashboards to identify them more quickly.

By using the GitLab package registry, Haven Technologies has been able to stop using the public npm registry — at a cost of $7 per user for more than 150 users (as of December 23, 2022). By implementing autoscaling Docker-machine and Kubernetes runners, O’Connor’s team was able to increase the number of regression and integration tests performed on each commit to developer branches.

The team has also greatly increased the number of security pipelines they are running on each merge request, which has helped them to increase developer involvement in securing applications earlier in the development lifecycle.

Overall, using GitLab CI has added value to Haven Technologies’ software development lifecycle by automating a number of processes — especially around branching — and freeing up developer time.