how to create an opensearch index in CDK?

Question

am trying to create an opensearch index in CDK, I've tried the below

const indexName = "my-index";

    // Load the settings from a JSON file
    const settingsFilePath = "./settings.json";
    const settings = JSON.parse(fs.readFileSync(settingsFilePath, "utf8"));

    // Trial 1
    // const index = new opensearch.OpenSearchIndex(this, "my-index", {
    //   domain,
    //   indexName,
    //   settings
    // });


    // Trial 2
    const openSearchIndexProps: opensearch.CfnDomain.IndexProperty = {
      indexName,
      indexSettings: settings,
      inputRecordFields: {
        source: "my-source-field"
      }
    };

    const openSearchIndex = new opensearch.CfnDomain(
      this,
      "my-index",
      {
        domainName,
        index: openSearchIndexProps
      }
    );

but none are working, I also couldn't find a documentation for that, am using "aws-cdk-lib": "2.41.0"

Answer 1

It has to be created as custom resource

lambda code

const AWS = require("aws-sdk");
const elasticsearch = require("@elastic/elasticsearch");
const { createConnector } = require("aws-elasticsearch-js");

const region = process.env.AWS_REGION || "us-east-1";

exports.handler = async function onEvent(event, context) {
  console.log("[ES Index Event]", event);
  // create a new instance of the SDK
  const credentials = new AWS.CredentialProviderChain();

  // retrieve the credentials using a Promise
  const promise = credentials.resolvePromise();

  // use the credentials
  let creds = await promise;

  const domain = event.ResourceProperties.domain;
  const indexName = event.ResourceProperties.IndexName;
  const indexSettings = event.ResourceProperties.IndexSettings;
  try {
    const client = new elasticsearch.Client({
      node: domain,
      Connection: createConnector({ region, credentials })
    });
    var indexCreationResponse = await client.indices.create({
      index: indexName,
      body: indexSettings
    });

    console.log(indexCreationResponse);

    const response = {
      StackId: event.StackId,
      RequestId: event.RequestId,
      LogicalResourceId: event.LogicalResourceId,
      PhysicalResourceId: indexName
    };

    return response;
  } catch (error) {
    console.error(error);
    return {
      status: "FAILED",
      physicalResourceId: indexName,
      data: {},
      reason: error.message
    };
  }
};

CDK code

private createElasticSearch() {
    const indexName = "my-index";

    // Load the settings from a JSON file
    const settingsFilePath = "./my-index-settings.json";
    const indexSettings = JSON.parse(fs.readFileSync(settingsFilePath, "utf8"));

    const esDomain = new elasticsearch.Domain(this, "myEsDomain", {
      domainName: `my-${config.env}-app-es`,
      version: elasticsearch.ElasticsearchVersion.V7_10,
      capacity: {
        dataNodes: 1,
        dataNodeInstanceType: "t3.small.elasticsearch",
      },
      ebs: {
        enabled: true,
        volumeSize: 15,
        volumeType: ec2.EbsDeviceVolumeType.GENERAL_PURPOSE_SSD,
      },
    });

    const createEsIndexLambda = new lambda.Function(
      this,
      `esIndexCustomResourceLambda`,
      {
        code: lambda.Code.fromAsset(
          path.join(RESOURCE_PATH, "es-index-assets")
        ),
        functionName: "es-index-custom-resource-creation-lambda",
        handler: "index.handler",
        timeout: Duration.seconds(90),
        runtime: lambda.Runtime.NODEJS_16_X,
      }
    );

    createEsIndexLambda.addToRolePolicy(
      new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        actions: ["es:*"],
        resources: [`*`],
      })
    );

    if (createEsIndexLambda && createEsIndexLambda.role) {
      esDomain.addAccessPolicies(
        new iam.PolicyStatement({
          effect: iam.Effect.ALLOW,
          actions: ["es:*"],
          resources: [esDomain.domainArn, `${esDomain.domainArn}/*`],
          principals: [new iam.ArnPrincipal(createEsIndexLambda.role.roleArn)],
        })
      );
    }

    const customResourceProvider = new customResources.Provider(
      this,
      `esIndexCustomResourceProvider`,
      {
        onEventHandler: createEsIndexLambda,
        providerFunctionName: "es-index-custom-provider-lambda",
      }
    );

    new CustomResource(this, `customResource`, {
      serviceToken: customResourceProvider.serviceToken,
      properties: {
        domain: `https://${esDomain.domainEndpoint}`,
        IndexName: indexName,
        IndexSettings: indexSettings,
      },
    });

    return esDomain;
  }

Answer 2

The short answer is you are going to have to use a Lambda to use HTTP requests against your cluster do to this after the cluster has been created.

The long answer is you can use can use a CDK Custom Resource as part of your CDK implementation. A Custom Resource needs a Provider to execute. In this case our Provider will be the Lambda I mentioned before.

From this Lambda we can do anything we want in an async manner including do any HTTP requests.

All of this is tied to your CDK deployment by the Custom Resource & Provider. This means that if you HTTP request fails (or whatever you choose to do) You can signal that CDK should rollback all of your changes.

Basically it's a nice way to tie distributed transactions into your stack creation.

What I use this for is to run any number of configurations against the cluster just after it is created. If any of them fail the whole thing rolls back.

I have a terribly named construct to orchestrate the requests:
(take special note of requests: props.requests)

export class Configurator extends Construct {
  constructor(scope: Construct, id: string, props: ConfiguratorProps) {
    super(scope, id);

    const configuratorLambda = new NodejsFunction(this, 'ConfiguratorLambda', {
      securityGroups: [props.securityGroup],
      vpc: props.vpc,
      runtime: Runtime.NODEJS_18_X,
      handler: 'handler',
      role: props.role,
      entry: path.join(__dirname, '../../src/configurator-lambda.ts'),
      timeout: Duration.seconds(30),
      environment: {
        DOMAIN: props.domain.domainEndpoint,
      },
    });

    const configuratorProvider = new Provider(this, 'ConfiguratorProvider', {
      onEventHandler: configuratorLambda,
    });

    const customResource = new CustomResource(
      this,
      'ConfiguratorCustomResource',
      {
        serviceToken: configuratorProvider.serviceToken,
        properties: {
          requests: props.requests,
        },
      },
    );

  }
}

Then from my stack I use it like this - as you noted previously the requests are passed on to the Lambda as an argument in the Configurator code.

    new Configurator(this, 'Configurator', {
      stage,
      vpc,
      securityGroup: inboundSecurityGroup,
      role: adminFnRole,
      domain,
      requests: [
        {
          method: 'PUT',
          path: '/sample-index1',
          body: {
          },
        },
        {
          method: 'PUT',
          path: '/_plugins/_security/api/rolesmapping/all_access',
          body: {
            users: [
              'username',
              'other-username'
            ].filter(Boolean),
          },
        },
      ],
    });

I won't put the lambda code in for brevity but you just use fetch or something to execute the request against the cluster. Make sure you handle errors and deal with appropriately.

From the Lambda you also need to explicitly respond correctly so that CDK can correctly interpret success or failure.

Here is a good example of the Lambda implementation which you can modify.