Experimenting with Pulumi for DevOps: An Insider's Perspective

As a software developer, I recently embarked on an exciting journey: experimenting with Pulumi for our DevOps processes. Here's my tale, distilled into a blog post to share the insights and understandings of the setup.

Structured Infrastructure Layers

First, to make sense of the multiple components, I broke down the Pulumi infrastructure into different layers: L0, L1, and L2.

1. Project Structure & Repository: Each of these layers has its distinct folder and serves as an independent Pulumi project. However, for coherence and ease of access, they all reside within a single infrastructural repository.

2. L0 - The Base Infrastructure: At the core, L0 shapes the foundational infrastructure. A prime example would be the creation of a Kubernetes cluster. This cluster can be instantiated from various cloud hosts, like AWS, Azure, Vultr, and more. Alternatively, if you prefer a hands-on approach, you can manually establish it via k3s or k8s through SSH. Post setup, the generated kubeconfig is relayed as a Pulumi stack output.

3. Dependent Layers Concept: A foundational principle here is the cascading dependency rule. Each layer is only permitted to have dependencies on the one below it. Thus, while L1 can lean on L0 resources, it can't do the same with L2, and so forth.

4. L1 - Next Tier Infrastructure: After establishing the basic cluster through L0, L1 takes over to lay down the secondary layer of infrastructure. This includes core components like databases, storage drivers such as Longhorn, and monitoring tools.

5. L2 - Advanced Cluster Applications: This layer introduces sophisticated cluster applications. A good example is a CMS system that taps into the PostgreSQL database from L1. Other applications in this layer include Umami for website analytics and Vaultwarden for in-house password management.

6. Stack Dependencies & Outputs: The real magic is in the interchange of stack outputs. For instance, L1 can transmit the database user and secret to L2, empowering L2 to delineate specific users and passwords for its applications.

7. Addressing Challenges: Not everything was smooth. A notable challenge arose with the PostgreSQL provider's inability to access the non-public database. A direct port-forwarding solution proved problematic within the GitHub Action. However, I found a workaround by employing the Pulumi flux operator for Kubernetes. This allowed the Pulumi program to operate directly within the cluster upon a repository push.

8. L3 Strategy Change: Initially, there was a plan for an L3. However, I opted to shift this to the respective service repositories. For instance, my personal website repository now includes a 'deploy' folder housing the Pulumi project. This layout promotes individual service releases, eliminating the need to trigger the entire infrastructure pipeline incessantly.

On The Horizon: Future Endeavors

Though I've made significant strides, there's still a roadmap ahead:

• Service Secrets: Currently, services do not tap into the outputs from the infrastructure repo. As a workaround, they are added manually as pipeline secrets. An assessment is underway to potentially read the secrets directly from the repository.

• External Secrets Provider: Incorporating an external secrets provider like AWS is on the agenda.

• Cluster Setup Integration: Although separate scripts manage the cluster setup, the goal is to embed them within the L0 layer.

• User-Friendly Resource Creation: To foster user autonomy, I'm contemplating a solution where users don't need direct access to the infra repository for tasks like database creation. Envisioning a YAML-based system, users can list their requirements. The infra repo pipeline, in response, would create the necessary resources and relay the credentials to the user's repo. This would be achieved using GitHub provider secrets.

In summary, Pulumi has ushered in a realm of possibilities and has refined our DevOps operations. It's a tool I'd recommend to anyone looking to scale their infrastructure management processes, and I'm eager to see where it takes us next.