customdecode

README

HCL Custom Static Decoding Extension

This HCL extension provides a mechanism for defining arguments in an HCL-based language whose values are derived using custom decoding rules against the HCL expression syntax, overriding the usual behavior of normal expression evaluation.

"Arguments", for the purpose of this extension, currently includes the following two contexts:

• For applications using hcldec for dynamic decoding, a hcldec.AttrSpec or hcldec.BlockAttrsSpec can be given a special type constraint that opts in to custom decoding behavior for the attribute(s) that are selected by that specification.

• When working with the HCL native expression syntax, a function given in the hcl.EvalContext during evaluation can have parameters with special type constraints that opt in to custom decoding behavior for the argument expression associated with that parameter in any call.

The above use-cases are rather abstract, so we'll consider a motivating real-world example: sometimes we (language designers) need to allow users to specify type constraints directly in the language itself, such as in Terraform's Input Variables. Terraform's variable blocks include an argument called type which takes a type constraint given using HCL expression building-blocks as defined by the HCL typeexpr extension.

A "type constraint expression" of that sort is not an expression intended to be evaluated in the usual way. Instead, the physical expression is deconstructed using the static analysis operations to produce a cty.Type as the result, rather than a cty.Value.

The purpose of this Custom Static Decoding Extension, then, is to provide a bridge to allow that sort of custom decoding to be used via mechanisms that normally deal in cty.Value, such as hcldec and native syntax function calls as listed above.

(Note: gohcl has its own mechanism to support this use case, exploiting the fact that it is working directly with "normal" Go types. Decoding into a struct field of type hcl.Expression obtains the expression directly without evaluating it first. The Custom Static Decoding Extension is not necessary for that gohcl technique. You can also implement custom decoding by working directly with the lowest-level HCL API, which separates extraction of and evaluation of expressions into two steps.)

Custom Decoding Types

This extension relies on a convention implemented in terms of Capsule Types in the underlying cty type system. cty allows a capsule type to carry arbitrary extension metadata values as an aid to creating higher-level abstractions like this extension.

A custom argument decoding mode, then, is implemented by creating a new cty capsule type that implements the ExtensionData custom operation to return a decoding function when requested. For example:

var keywordType cty.Type
keywordType = cty.CapsuleWithOps("keyword", reflect.TypeOf(""), &cty.CapsuleOps{
    ExtensionData: func(key interface{}) interface{} {
        switch key {
        case customdecode.CustomExpressionDecoder:
            return customdecode.CustomExpressionDecoderFunc(
                func(expr hcl.Expression, ctx *hcl.EvalContext) (cty.Value, hcl.Diagnostics) {
                    var diags hcl.Diagnostics
                    kw := hcl.ExprAsKeyword(expr)
                    if kw == "" {
                        diags = append(diags, &hcl.Diagnostic{
                            Severity: hcl.DiagError,
                            Summary:  "Invalid keyword",
                            Detail:   "A keyword is required",
                            Subject:  expr.Range().Ptr(),
                        })
                        return cty.UnkownVal(keywordType), diags
                    }
                    return cty.CapsuleVal(keywordType, &kw)
                },
            )
        default:
            return nil
        }
    },
})

The boilerplate here is a bit fussy, but the important part for our purposes is the case customdecode.CustomExpressionDecoder: clause, which uses a custom extension key type defined in this package to recognize when a component implementing this extension is checking to see if a target type has a custom decode implementation.

In the above case we've defined a type that decodes expressions as static keywords, so a keyword like foo would decode as an encapsulated "foo" string, while any other sort of expression like "baz" or 1 + 1 would return an error.

We could then use keywordType as a type constraint either for a function parameter or a hcldec attribute specification, which would require the argument for that function parameter or the expression for the matching attributes to be a static keyword, rather than an arbitrary expression. For example, in a hcldec.AttrSpec:

keywordSpec := &hcldec.AttrSpec{
    Name: "keyword",
    Type: keywordType,
}

The above would accept input like the following and would set its result to a cty.Value of keywordType, after decoding:

keyword = foo

The Expression and Expression Closure cty types

Building on the above, this package also includes two capsule types that use the above mechanism to allow calling applications to capture expressions directly and thus defer analysis to a later step, after initial decoding.

The customdecode.ExpressionType type encapsulates an hcl.Expression alone, for situations like our type constraint expression example above where it's the static structure of the expression we want to inspect, and thus any variables and functions defined in the evaluation context are irrelevant.

The customdecode.ExpressionClosureType type encapsulates a *customdecode.ExpressionClosure value, which binds the given expression to the hcl.EvalContext it was asked to evaluate against and thus allows the receiver of that result to later perform normal evaluation of the expression with all the same variables and functions that would've been available to it naturally.

Both of these types can be used as type constraints either for hcldec attribute specifications or for function arguments. Here's an example of ExpressionClosureType to implement a function that can evaluate an expression with some additional variables defined locally, which we'll call the with(...) function:

var WithFunc = function.New(&function.Spec{
    Params: []function.Parameter{
        {
            Name: "variables",
            Type: cty.DynamicPseudoType,
        },
        {
            Name: "expression",
            Type: customdecode.ExpressionClosureType,
        },
    },
    Type: func(args []cty.Value) (cty.Type, error) {
        varsVal := args[0]
        exprVal := args[1]
        if !varsVal.Type().IsObjectType() {
            return cty.NilVal, function.NewArgErrorf(0, "must be an object defining local variables")
        }
        if !varsVal.IsKnown() {
            // We can't predict our result type until the variables object
            // is known.
            return cty.DynamicPseudoType, nil
        }
        vars := varsVal.AsValueMap()
        closure := customdecode.ExpressionClosureFromVal(exprVal)
        result, err := evalWithLocals(vars, closure)
        if err != nil {
            return cty.NilVal, err
        }
        return result.Type(), nil
    },
    Impl: func(args []cty.Value, retType cty.Type) (cty.Value, error) {
        varsVal := args[0]
        exprVal := args[1]
        vars := varsVal.AsValueMap()
        closure := customdecode.ExpressionClosureFromVal(exprVal)
        return evalWithLocals(vars, closure)
    },
})

func evalWithLocals(locals map[string]cty.Value, closure *customdecode.ExpressionClosure) (cty.Value, error) {
    childCtx := closure.EvalContext.NewChild()
    childCtx.Variables = locals
    val, diags := closure.Expression.Value(childCtx)
    if diags.HasErrors() {
        return cty.NilVal, function.NewArgErrorf(1, "couldn't evaluate expression: %s", diags.Error())
    }
    return val, nil
}

If the above function were placed into an hcl.EvalContext as with, it could be used in a native syntax call to that function as follows:

  foo = with({name = "Cory"}, "${greeting}, ${name}!")

The above assumes a variable in the main context called greeting, to which the with function adds name before evaluating the expression given in its second argument. This makes that second argument context-sensitive -- it would behave differently if the user wrote the same thing somewhere else -- so this capability should be used with care to make sure it doesn't cause confusion for the end-users of your language.

There are some other examples of this capability to evaluate expressions in unusual ways in the tryfunc directory that is a sibling of this one.

Documentation

Overview

Package customdecode contains a HCL extension that allows, in certain contexts, expression evaluation to be overridden by custom static analysis.

This mechanism is only supported in certain specific contexts where expressions are decoded with a specific target type in mind. For more information, see the documentation on CustomExpressionDecoder.

Index

• Constants
• Variables
• func ExpressionClosureVal(closure *ExpressionClosure) cty.Value
• func ExpressionFromVal(v cty.Value) hcl.Expression
• func ExpressionVal(expr hcl.Expression) cty.Value
• type CustomExpressionDecoderFunc
  • func CustomExpressionDecoderForType(ty cty.Type) CustomExpressionDecoderFunc
• type ExpressionClosure
  • func ExpressionClosureFromVal(v cty.Value) *ExpressionClosure
  • func (c *ExpressionClosure) Value() (cty.Value, hcl.Diagnostics)

Constants

const CustomExpressionDecoder = customDecoderImpl(1)

CustomExpressionDecoder is a value intended to be used as a cty capsule type ExtensionData key for capsule types whose values are to be obtained by static analysis of an expression rather than normal evaluation of that expression.

When a cooperating capsule type is asked for ExtensionData with this key, it must return a non-nil CustomExpressionDecoderFunc value.

This mechanism is not universally supported; instead, it's handled in a few specific places where expressions are evaluated with the intent of producing a cty.Value of a type given by the calling application.

Specifically, this currently works for type constraints given in hcldec.AttrSpec and hcldec.BlockAttrsSpec, and it works for arguments to function calls in the HCL native syntax. HCL extensions implemented outside of the main HCL module may also implement this; consult their own documentation for details.

Variables

var ExpressionClosureType cty.Type

ExpressionClosureType is a cty capsule type that carries hcl.Expression values along with their original evaluation contexts.

This is similar to ExpressionType except that during custom decoding it also captures the hcl.EvalContext that was provided, allowing callers to evaluate the expression later in the same context where it would originally have been evaluated, or a context derived from that one.

var ExpressionType cty.Type

ExpressionType is a cty capsule type that carries hcl.Expression values.

This type implements custom decoding in the most general way possible: it just captures whatever expression is given to it, with no further processing whatsoever. It could therefore be useful in situations where an application must defer processing of the expression content until a later step.

ExpressionType only captures the expression, not the evaluation context it was destined to be evaluated in. That means this type can be fine for situations where the recipient of the value only intends to do static analysis, but ExpressionClosureType is more appropriate in situations where the recipient will eventually evaluate the given expression.

Functions

func ExpressionClosureVal

func ExpressionClosureVal(closure *ExpressionClosure) cty.Value

ExpressionClosureVal returns a new cty value of type ExpressionClosureType, wrapping the given expression closure.

func ExpressionFromVal

func ExpressionFromVal(v cty.Value) hcl.Expression

ExpressionFromVal returns the expression encapsulated in the given value, or panics if the value is not a known value of ExpressionType.

func ExpressionVal

func ExpressionVal(expr hcl.Expression) cty.Value

ExpressionVal returns a new cty value of type ExpressionType, wrapping the given expression.

Types

type CustomExpressionDecoderFunc

type CustomExpressionDecoderFunc func(expr hcl.Expression, ctx *hcl.EvalContext) (cty.Value, hcl.Diagnostics)

CustomExpressionDecoderFunc is the type of value that must be returned by a capsule type handling the key CustomExpressionDecoder in its ExtensionData implementation.

If no error diagnostics are returned, the result value MUST be of the capsule type that the decoder function was derived from. If the returned error diagnostics prevent producing a value at all, return cty.NilVal.

func CustomExpressionDecoderForType

func CustomExpressionDecoderForType(ty cty.Type) CustomExpressionDecoderFunc

CustomExpressionDecoderForType takes any cty type and returns its custom expression decoder implementation if it has one. If it is not a capsule type or it does not implement a custom expression decoder, this function returns nil.

type ExpressionClosure

type ExpressionClosure struct {
	Expression  hcl.Expression
	EvalContext *hcl.EvalContext
}

ExpressionClosure is the type encapsulated in ExpressionClosureType

func ExpressionClosureFromVal

func ExpressionClosureFromVal(v cty.Value) *ExpressionClosure

ExpressionClosureFromVal returns the expression closure encapsulated in the given value, or panics if the value is not a known value of ExpressionClosureType.

The caller MUST NOT modify the returned closure or the EvalContext inside it. To derive a new EvalContext, either create a child context or make a copy.

func (*ExpressionClosure) Value

func (c *ExpressionClosure) Value() (cty.Value, hcl.Diagnostics)

Value evaluates the closure's expression using the closure's EvalContext, returning the result.