Why GitOps isn’t ready for the mainstream (yet)

Since being coined in 2017, GitOps has emerged as a natural evolution of modern software development practices like devops, infrastructure as code, and CI/CD principles, specifically for organisations that are building microservices deployed across distributed containers and orchestrated by Kubernetes, as is the fashion these days.

However, there are still several significant cultural and technological hurdles for the industry to overcome to allow GitOps to truly hit mainstream adoption at the scale that agile and devops practices have so far.

What is GitOps?

GitOps extends devops primarily by treating infrastructure as code, so that both the application and its underlying infrastructure can be treated as code and stored in a version control system, most likely Git, providing a single source of truth for both dev and ops. Done right, this allows for all changes to be pushed through declarative code, with a set of automated steps that corrects any deviations from the desired state.

While that all sounds great in theory, among the enterprises known to be dabbling with GitOps practices—companies like Peloton, Volvo, Ticketmaster, and Just Eat Takeaway.com—none were willing to talk to InfoWorld at this stage.

“I haven’t talked to any organisations that are rolling out a GitOps initiative, and most of the organisations I speak to probably haven’t even heard of it,” said Jim Mercer, a research director in IDC’s devops solutions practice.

“[GitOps] is still in the early stages of maturity,” said Mukulika Kapas, director of product management for the internal developer platform at fintech firm Intuit, an early GitOps adopter after it acquired the Argo CD creator Applatix in 2018.

Instead, smaller cloud-native organisations are starting to investigate the potential for GitOps to improve their software delivery processes, and larger organisations are most likely looking at GitOps in the pockets where cloud-native practices are more prevalent, like greenfield digital initiatives or research and development centres.

“Smart organisations are asking themselves how to make it so developers aren’t SSHing into servers, creating instances, and making changes in an uncontrolled fashion. That is a problem that GitOps solves,” said James Governor, co-founder of developer-focused analyst firm RedMonk.

Still, despite its promise, GitOps remains on the fringes today. Here are four reasons why GitOps practices have yet to see mainstream adoption, and some of the efforts being made to change that.

GitOps lacks established patterns

The GitOps ecosystem remains small but passionate. As a result, there remain very few resources for organisations that want established best practices, case studies, and patterns to base their decisions on.

“The biggest challenge with GitOps right now is that there are no established patterns to guide you in your choices,” wrote Ian Miell, a cloud-native consultant at Container Solutions, in his 2020 blog post on the topic. “Until we have genuine standards in this space, getting your GitOps architecture right will always be an art rather than a science.”

A GitOps working group was established as a CNCF open community project in November 2020 to start to tackle some of these challenges and ease the adoption curve for newcomers. Led by vendors Amazon, Codefresh, GitHub, and Weaveworks, the group has initially been tasked with clearly defining the core principles of GitOps in a vendor-neutral way and growing adoption of the practice.

“Right now, we are in the accessibility phase, where we are getting the tribal knowledge out into the open and easy for people to consume,” said Dan Garfield, chief open source officer for Codefresh and co-founder of the GitOps working group, in an interview with InfoWorld.

“We are formalising the GitOps principles to be more mature and leaning on practitioners to identify things we haven’t seen and shave off the rough edges and gather community patterns and reference implementations for people to look at.”

Garfield said the working group received an “overwhelming amount of community support” when it was established, with 80 company representatives dialing into the first call.

GitOps tools need to mature

The typical GitOps deployment process involves a developer making a pull request for a new feature, most commonly through Git (hence the name), which once approved triggers a CI/CD pipeline, tests the code, and deploys to a registry. Then a software agent, typically Argo or Flux, automatically detects if the state of a cluster matches the config in Git, pulls the changes, and deploys the new feature.

“Years ago, people built what looks a lot like Git operators, to take and sync infrastructure as code using version-controlled storage. The issue isn’t that it met the definitions [of GitOps], [but that] it was a dark art with a team using custom tooling and that it was difficult. Now, with cloud-native tools like Argo or Flux, we can really simplify the process,” Garfield said.

While those tools have come on by leaps and bounds in recent years, there are still gaps that the community needs to fill to simplify adoption. “Even though GitOps the methodology has some interesting characteristics and selling points, the current GitOps tools focus only on the deployment part of an application and nothing else,” wrote Kostis Kapelonis, developer advocate at Codefresh in a 2020 blog post titled “The Pains of Gitops 1.0.”

He points to the ability to enact promotions between environments, secret handling, smoke testing, and audit as currently missing from the consumable GitOps tool stack. This means that teams currently “need to create their own best practices for all aspects of software delivery,” he wrote.

For Christopher Condo, a principal analyst at Forrester, the next phase of GitOps tools will probably be embedded with cloud platforms where developers already work, for “something like GitHub Actions that integrates directly with Terraform, to allow developers to more easily do infrastructure as code, so they are doing GitOps without even realising they are doing it. That is when it will be mainstream,” he told InfoWorld.

Running GitOps at scale poses significant challenges

GitOps still has some clearly defined limitations when run at scale, wrote Ádám Sándor, a cloud-native architect at professional services firm Container Solutions, in a 2020 blog post. Those limitations including audit, remediation, and observability challenges when running across multiple Git repositories.

“For small teams with 10 to 15 experts, GitOps is the best possible thing you can do,” said Kaspar von Grunberg, CEO of Humanitec, a start-up that helps organisations build their own internal developer platforms. “That is great at a certain level, but the issue I have started to see with larger organisations is the implementation of GitOps at scale being extremely frustrating.”

Take the process of promoting a change across various environments. “This is probably one of the most well-known issues with GitOps and one of the first topics discussed when it comes to how GitOps can work in big organisations,” Kapelonis wrote.

“Every time somebody declares that adopting GitOps is an easy process, I always ask how promotion between different environments works in their case. And I always get different answers,” he wrote.

“I am really disappointed that even the page specifically created for addressing GitOps questions says, ‘GitOps doesn’t provide a solution to propagating changes from one stage to the next one. We recommend using only a single environment and avoid stage propagation altogether.’”

Then there is the observability problem plaguing large GitOps deployments. “At their current state, GitOps tools are great for observing the content of a cluster on a technical level but fail miserably on monitoring the business metrics of each deployment,” Kapelonis wrote.

“If you adopt GitOps in a large company with a big number of environments and applications, the number of Git repositories quickly skyrockets. This makes it very hard to keep track of what is going on in each environment and can quickly lead to configuration duplication or people making commits to specific environments.”

For example, if you have 20 Git repositories with Kubernetes manifests and you need to make a central change, you currently need to manually make 20 Git commits or create some homegrown glue code that does it for you.

“We are building some cool tooling to see all deployments to overcome that observability challenge,” Codefresh’s Garfield said.

“Getting to the scale side is important as you have the reconcilers running around and suddenly [don’t know] which one of the many changes today caused the regression and you need a way to deal with the scale. … That is the frontier right now,” he said.

Getting GitOps buy-in is challenging

You’ve only just convinced your boss that devops is the way to go to get more features to your users, and now you have to go back and convince them to do that all over again with GitOps. It’s a daunting task for anyone, and is certainly another drag on GitOps reaching mainstream adoption.

“We are starting to see organisations where the practitioners who are on the platform teams or developer enablement teams, who are starting to understand the benefits that GitOps can bring, are having a heck of a time helping the decision makers understand the value that GitOps brings, because often the way we describe it is either overly simplistic or doesn’t address the business value,” said Cornelia Davis, CTO at Weaveworks.

One mistake Davis often comes across is in seeing GitOps as a replacement for devops practices. “It’s not a switch, it is a revolution,” she said. “We have matured a great deal with agile development, the tooling that supports that, there is all sorts of optimisations happening. GitOps is saying we have done a lot on the dev side and now we need to do more on the ops side.”

“The problem is the technology is sophisticated, and not many people are familiar with it,” Forrester’s Condo said. “We will see greater refinement in the coming years as enterprises without that combination of developer and cloud engineer skills find better ways to bring together [dev and ops]. If there is any problem with GitOps, it is jumping in with both feet and not getting everyone involved in that process aligned.”

For RedMonk’s Governor, focusing on the control aspect that GitOps brings can be a compelling reason to make the shift. “The business case rests on concerns around developers making system changes that can cause problems. At the moment, it is the Wild West, and GitOps is about regaining some control,” he said.

Properly investing in your people and giving them the time and space to understand what GitOps can bring is vital if the practice is to take hold in a meaningful way.

“Don’t expect a new set of working practices to just happen. The whole organisation will not suddenly do GitOps, but when you are planning a new project and looking at cloud-native infrastructure, maybe try some GitOps practices there to build up organisational confidence,” Governor said.

While all signs point to the industry still being early on the adoption curve for GitOps, IDC’s Mercer believes it will probably take hold “faster [than devops], as the cultural barriers have been broken down a bit already. If you are doing devops and continuous delivery, you are in the minority, but you will be pretty well-attuned to adopt GitOps.”