Credit: Dreamstime
With the growing popularity of infrastructure as code, devops, and internal platforms, back-end developers are better equipped than ever to build highly resilient, performant, and scalable server-side applications and services. But they are also drowning.
The complexity of modern applications requires back-end developers to master a growing array of tools, technologies, and techniques, from the fundamentals of Linux, scripting languages, logging, and monitoring, to cloud-based networking, service meshes, observability, Kubernetes clusters, and the dreaded YAML files.
Back-end developers could use a break — or, more specifically, a better developer experience. Fortunately, tool makers are rushing to provide it.
From lowering the bar to infrastructure as code, to smoothing Kubernetes workflows and distributed app deployments, to spinning up developer workspaces in the cloud on demand, a new wave of projects promises to make life easier for the developers who toil on the server side.
Back-end engineers have feelings too
In today’s cloud-native world, developers of all kinds will naturally gravitate towards tools that are more intuitive and pleasurable to use, even if they are working in a domain that has typically not optimised for simplicity and ease of use.
While companies like Vercel and Netlify have found plenty of success by focusing on the front-end developer experience and abstracting away the back-end, many organisations will still want some control over their server infrastructure. The engineers responsible for that back-end may want a better experience too.
“It is natural to see providers making it easier for developers to do those things and that is where we get into infrastructure meeting software development,” RedMonk analyst James Governor told InfoWorld. “At the end of the day, you need platforms to enable you to be more productive without manually dealing with Helm charts, operators, or YAML.”
Improving the back-end developer experience can do more than improve the lives of back-end developers. Providing better, more intuitive tools can enable back-end developers to get more done, while also bringing down barriers to allow a wider cohort of developers to manage their own infrastructure through thoughtful abstractions.
“Developer control over infrastructure isn’t an all-or-nothing proposition,” Gartner analyst Lydia Leong wrote.
“Responsibility can be divided across the application lifecycle, so that you can get benefits from “you build it, you run it” without necessarily parachuting your developers into an untamed and unknown wilderness and wishing them luck in surviving because it’s not an ‘infrastructure and operations team problem’ anymore.”
In other words, “it’s perfectly okay to allow your developers full self-service access to development and testing environments, and the ability to build infrastructure-as-code templates for production, without making them fully responsible for production,” Leong writes.
Pulumi: Bringing developer experience to a forgotten land
Take Pulumi, which aims to make the job of configuring infrastructure something any developer can do by working in their preferred programming language, instead of something proprietary like CloudFormation for Amazon Web Services (AWS), Bicep for Microsoft Azure, or YAML for Kubernetes. The open source Pulumi engine then configures the virtual infrastructure and endpoints for code to be dropped in.
Pulumi is, essentially, another option for provisioning infrastructure as code, but promises a shallower learning curve than tools like CloudFormation or HashiCorp’s Terraform.
While working on developer tools for over a decade at Microsoft, Pulumi founder and CEO Joe Duffy was struck by the realisation that “no one had applied the level of care and love to the infrastructure space” as they had to front-end development. “The cloud was a boring and messy afterthought,” he told InfoWorld.
“The best technology we had was Terraform, which is a proprietary technology, whereas I just wanted an IDE and to use my favourite language, but still use the cloud to the fullest,” Duffy said.
Developer experience is “life itself” for Pulumi, according to Duffy. “We want to bring a great developer experience to a space that didn’t have one. That’s why we use general purpose languages,” he said. “We want that experience to be pleasurable and not tedious.”
For infrastructure, platform, and devops engineers working in legacy-strewn environments, Pulumi enables reusable infrastructure to be put in place for developers, complete with policies and testing as code. Pulumi recently announced a Business Critical edition for enterprise customers of this kind.
For developers working in more greenfield, cloud-native environments, Pulumi allows them to be truly full-stack, but without having to learn new languages or domain-specific back-end skills.
“I wanted to find a solution that enables us to model infrastructure, not by writing config code and running a script, but as a software development process, through code and a Git-based workflow,” Andy Dang, former senior AWS engineer and cofounder of observability start-up WhyLabs, told InfoWorld.
WhyLabs turned to Pulumi after Dang found existing infrastructure-as-code options like CloudFormation and Terraform too complex, noting that CloudFormation in particular is “not written for human consumption.”
Instead, Dang wanted the lean devops and engineering teams at WhyLabs to be able to “reduce the amount of infrastructure-specific knowledge the team requires, so that they can focus best on their specific piece of the stack,” he said.
HashiCorp Waypoint: Building a consistent developer workflow
Another company entering the fray here is HashiCorp, creators of the popular Terraform infrastructure-as-code tool. With the goal of smoothing developers’ path to building and deploying their applications on Kubernetes and Amazon’s Elastic Container Service (ECS), HashiCorp launched Waypoint in 2020.
“We believe developers just want to deploy,” Chang Li, director of product marketing at HashiCorp, told InfoWorld.
To help them do this, Waypoint offers a user-friendly abstraction layer and interface for both developers and operators. For developers, this means simple execution of build and release processes, while operators gain some much-needed consistency and standardisation across different environments, all using a single command: waypoint up.
“Our main audience is operators,” Li said. “At enterprise scale, the operations teams feel pressure to standardise a workflow to scale across multiple platforms. Traditionally they use CI/CD tools like Jenkins or Spinnaker and they find them too heavyweight.”
For HashiCorp, “Waypoint could be a replacement of those manuscripts or customised CI/CD workflows,” Li said. “It should be a simplified experience, without the complex interaction with the underlying infrastructure.”
It’s this same frustration with existing CI/CD tools that has driven the Docker founder Solomon Hykes to try and make CI/CD more developer-friendly with his open source project, Dagger.
Render: A Goldilocks mix of hosting options
Founded by the former head of risk at Stripe, Anurag Goel, Render aims to give developers a Goldilocks mix of hosting options that land between highly opinionated platforms like Heroku and the unbound complexity of AWS.
After leaving Stripe, Goel was dabbling with machine learning in his spare time. While learning to build machine learning models, he saw data scientists spending days just getting their Jupyter Notebook environments ready. Instead, Goel wanted “one-click provisioning in the cloud” and an experience that mirrored that of integrating payments using the Stripe API.
“I kept getting drawn to infrastructure and developer productivity,” he told InfoWorld. “At its core, Render is automating devops.”
In practice, developers can connect their GitHub or GitLab repository to Render, which then starts to suggest commands to build and start your application. Render is a proprietary service and runs on top of AWS and Google Cloud for now, with bare metal support coming soon.
“People are coming to us because they think Kubernetes is too complex, but the features they need are on Render because we have built on top of Kubernetes in a way that exposes only what they want,” Goel said.
Where Render differs from other backend-as-a-service (BaaS) hosting options is that it doesn’t bring an opinion on how to build and run your applications, whether it is a static site, Cron job, or anything in a Docker container. “We aren’t pushing a specific ideology of how to build applications,” Goel said.
Render isn’t alone in its developer-centric approach to BaaS either. The open source project Appwrite is working on a self-hosted platform for developers to easily run web, mobile, and Flutter applications through a set of curated APIs.
Encore: Bringing the Spotify back-end approach to the masses
Another, more recent open source project looking to improve the developer experience for back-end engineering tasks is Encore. Built in Go, Encore is a back-end framework that abstracts manual configuration tasks for cloud environments.
Encore aims to help developers stay in a state of flow by automating manual configuration steps for provisioning cloud infrastructure, generating boilerplate code, instrumenting the application, and creating documentation. Developers can then deploy to AWS, Azure, or Google Cloud.
The project was founded in 2021 by ex-Spotify engineers André Eriksson, Marcus Kohlberg, and Horia Jurcu, along with ex-Monzo engineer Dominic Black and ex-Google engineer Stefan Ekerfelt. The founders say that Encore was born out of their shared frustrations composing complex distributed back-end infrastructure and the constant repetition of certain back-end tasks.
Encore strives to “keep the good parts of the cloud and change what developers are working on, allowing them to build products and not be mired in the swamps of YAML and plumbing,” Eriksson told InfoWorld.
The start-up raised a $3 million seed round led by Crane Venture Partners and made its open source back-end development engine generally available in April 2022.
Gitpod: On-demand developer environments
Then there is Gitpod, an open source project contending with the very specific problem of providing developers with their well-worn local development environment, but ready to go in the cloud in a matter of seconds.
While the industry has largely moved to automated infrastructure creation and deployment pipelines, cofounder Johannes Landgraf thought it was “kind of crazy” that developers were still “replicating developer environments that sit on individual computers and experience configuration drift.”
Gitpod starts with a YAML file that describes how the developer environment should look, from the IDE to the database, complete with compilers, language servers, and the operating system. “Everything that sits on your local computer we are porting to the cloud with the same experience developers are used to,” Landgraf said.
Gitpod integrates with GitHub, GitLab, and Bitbucket and pre-builds all of your projects whenever code is pushed to your repo, like a continuous integration server. When developers spin up an environment, everything is already initialised and ready to run. They don’t have to wait for dependencies to load and scripts to build.
Developer experience is “part of our core DNA,” Landgraf said. “Our goal is to remove as much friction as possible for developers to do what they are paid for and what they love, which is being creative, writing code, and creating value, not messing around with things that are slowing them down.”
The open source project has already seen good traction, with GitHub soon following suit by launching its own cloud-based development environment called Codespaces in 2021.
All GitHub engineers now work in Codespaces, where “development environments are provisioned on demand for the task at hand” and developers can “create reliable, preconfigured Codespaces, primed and ready for GitHub.com development in 10 seconds,” according to GitHub senior director of engineering Cory Wilkerson.
How AWS, Azure, and Google Cloud are simplifying infrastructure builds
The big three cloud providers still tend to offer primitives rather than opinionated tooling. However, last year AWS moved to enable enterprises to configure their own set of repeatable environment templates with the introduction of AWS Proton. This self-service tool allows developers to deploy to production-ready infrastructure that has already been created and approved by a specialist devops or platform team.
Proton differs from fully managed back-end solutions like AWS App Runner or AWS Amplify, which take things a step further by managing all of the configuration, networking, load balancing, and deployment pipelines for web, mobile, or containerised applications to run in the cloud.
Google and Microsoft offer similar opinionated back-end services, such as Google Firebase and Azure App Service, but not the capability to create enterprise-specific environment templates that Proton provides.
Enabling true self-service development
Even as developers gain more and more good options to abstract away all of their back-end development tasks, there will still be plenty of demand for tools that enable enterprises to provision and manage their own infrastructure and neatly catalog it for developers to use.
“It is still really early days for these tools, but I see quick adoption because they solve two common problems we hear a lot: corporate IT wanting to lock down environments, and stopping developers from pulling unregulated stuff into the organisation,” former Gartner analyst Fintan Ryan told InfoWorld.
If these tools can also abstract away the toil involved with various back-end engineering and development tasks, then they will take their place not only in the winners circle of this emerging category, but also in the hearts and minds of the builders of back-end apps and services.
