Pulumi and the mystery of the integer ID

In Pulumi resources have inputs and outputs. Inputs are fairly obvious but Outputs have some interesting characteristics that are worth taking a closer look at. Outputs are effectively promises to resolve to a value once a resource is created. Outputs can in turn be used as inputs to a different resource which has the nice side-effect of establishing ordering/dependencies.

When using Pulumi with Go you’ll see that resource structs have a $TypeOutput for most of their fields, like StringOutput. These will eventually resolve to a value, after the resource has been created.A $TypeOutput type satisfies the requirements of the equivalent $TypeInput. This means you can chuck an output into an equivalent input and not worry about it. This is really lovely and is also more or less what you expect. It makes the input and outputs feel like normal types.

An ID is usually assigned by your cloud provider’s backend and is returned upon resource creation. It’s typically an opaque thing you should not attach any meaning to. For example, if you want to attach an IP to a server you’ll need to provide the IP resource’s ID to your server and not the IP address. Resources returned by a provider embed pulumi.CustomResourceState which gives us access to the ID() method. This returns a pulumi.IDOutput which you can use anywhere a pulumi.ID(Ptr)Input is expected or by virtue of it having ToString(Ptr)Output methods anywhere a pulumi.String(Ptr)Input is desired.

Unfortunately, things get complicated when the ID is actually an integer. The cloud provider API expects the ID to be passed in as such and won’t accept its stringified counterpart. Easy, just call ToInt(Ptr)Output on IDOutput. You could, except it doesn’t exist. Ohno.

A concrete example is when creating servers on Hetzner Cloud. Though you don’t have to do this, you can first create the IPv4 and IPv6 addresses and then pass their IDs into the hcloud.ServerArgs when calling hcloud.NewServer(). Though the Hetzner Cloud API states that it only accepts the ID of an already created PrimaryIp resource, Pulumi wants a pulumi.IntPtrInput there and not a pulumi.IDInput. This makes sense in that IDOutput can’t be converted to an IntPtrOutput currently, but it means we need to work around that type dissonance.

The way to do this is using Apply. The result of a call to ApplyT() in Go is a new output of some new type. This means we end up writing code like this:

ip, err := hcloud.NewPrimaryIp(...)

server, err := hcloud.NewServer(ctx, name, &hcloud.ServerArgs{
    PublicNets: hcloud.ServerPublicNetArray{
			&hcloud.ServerPublicNetArgs{
				Ipv4Enabled: pulumi.Bool(true),
				Ipv4: ip.ID().ToStringOutput().ApplyT(func(val string) (*int, error) {
					i, err := strconv.Atoi(val)
					return &i, err
				}).(pulumi.IntPtrOutput),
			},
		},
    })

This code is a lot and hard to understand. Lets first factor out the callback in ApplyT() to its own function to make this a bit more manageable:

func IDToIntPtr(val string) (*int, error) {
	i, err := strconv.Atoi(val)
    return &i, err
}

Then pass it into ApplyT by name, but don’t call it (don’t add the ()):

ip.ID().ToStringOutput().ApplyT(IDToIntPtr).(pulumi.IntPtrOutput),

The benefit of this is that you can reuse the function. You’ll need it a lot when working with providers that need the ID as an integer. You can now also write tests for this thing which isn’t possible for inline callbacks.

So what wizardry is it that we’re performing here? We leverage the fact that IDOutput has a ToStringOutput. This gets us close to a native Go string which we can then convert to its non-stringified integer using the standard library. The second trick is going through ApplyT(). Its signature accepts an interface{} callback but its documented to be one of func (v U) T or func (v U) (T, error). The duality and opaqueness is a bit annoying here but remember that this existed well before generics were a thing in Go. The type of v is the ElementType() of the Output we call ApplyT() on. Unsurprisingly for a StringOutput that’s a string.

At this point we now have a native Go string inside our callback which we can transform to an integer. Since we know we’ll need to get to an IntPtrOutput we have our callback return a *int. However, the return type of ApplyT() is Output. For native types that have an equivalent Pulumi type we can use a type assertion to get to our final destination, the .(pulumi.IntPtrOutput). In cases where this isn’t available you’ll need to coerce to AnyOutput.

We’ve now come full circle and finally have an Output type that satisfies the IntPtrInput requirement of hcloud.ServerPublicNetArgs.Ipv4. If you think this is a bit too much effort to have to go through you’re not alone. The ID being an integer and needing to be treated as such is an issue affecting a few providers like Hetzner and Digital Ocean. I’ve filed issue #11750 to see how to resolve that. Though this post has looked at the issue from a Go perspective this problem applies to other languages you can use with Pulumi too.