Cloudformation should create documentdb replicas in different availability zones-CodePudding

I'm creating a DocumentDB (MongoDB) cluster in AWS using Cloudformation. The documentation says that the DocumentDB instances WILL be spread across the subnet groups, in different AvailabilityZones (AZ), as defined here:

What's going on here? Why did AWS create my two READ REPLICAS in the same AZ? Shouldn't the last one have gone to us-east-1b ?!?!

is there a race condition in Cloudformation? Cloudformation seems to create all three instances in parallel so maybe it doesn't have the smarts to wait until the other instances have been assigned to an AZ before determining which AZ each instance should go to? For example: maybe all three make a request to see what instances already exist and in which AZ they are located and then two notice that there's nothing in us-east-1a and both decide to create the instance there?
should I create a "dependsOn" to avoid the race condition? I.e. instance2 depends on instance1, and instance3 depends on instance2? Ha, I'll try that and report back :)

See my Cloudformation template below. Note that the VPC, public and private subnets are defined in another Cloudformation template/stack but it's the usual VPC with 3 public and 3 private subnets, one NAT EIP per AZ, and all the routes that go with it (outside the scope of this question). I just import the values exported from that other stack (VPC id, VPC CIDR, and subnets):

AWSTemplateFormatVersion: "2010-09-09"

Description: DocumentDB/MongoDB database server in the VPC, in all private subnets. And the database master username and password in secrets manager.

Parameters:
  MongoDBInstanceIdentifier:
    Description: The mongodb database identifier (ex dev, prod, etc).
    Type: String

Resources:
  # Create the MongoDB Master username password Secret with a randomly generated password
  MongoDBSecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Sub 'docdb/masterpwd/${MongoDBInstanceIdentifier}'
      Description: !Sub 'MongoDB ${MongoDBInstanceIdentifier} master user credentials'
      GenerateSecretString:
        SecretStringTemplate: '{"username": "admin"}'
        GenerateStringKey: "password"
        PasswordLength: 16
        ExcludePunctuation: true
        ExcludeCharacters: "\"@/\\"
      Tags:
        - Key: Name
          Value: !Sub 'MongoDB ${MongoDBInstanceIdentifier} master user password secret'

  MongoDBSubnetGroup:
    Type: AWS::DocDB::DBSubnetGroup
    Properties:
      DBSubnetGroupDescription: MongoDB subnet group
      DBSubnetGroupName: !Sub "mongo-subnet-group-${MongoDBInstanceIdentifier}"
      SubnetIds:
        - !ImportValue PrivateSubnet1
        - !ImportValue PrivateSubnet2
        - !ImportValue PrivateSubnet3

  MongoDBSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: DocumentDB Security Group
      GroupName: !Sub docdb-sg-${MongoDBInstanceIdentifier}
      VpcId: !ImportValue VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          CidrIp: !ImportValue VpcCIDR
          FromPort: 27017
          ToPort: 27017
      Tags:
        - Key: Name
          Value: !Sub "MongoDB security group ${MongoDBInstanceIdentifier}"

  MongoDBCluster:
    Type: AWS::DocDB::DBCluster
    Properties:
      DBClusterIdentifier: !Sub "mongo-cluster-${MongoDBInstanceIdentifier}"
      DBClusterParameterGroupName: !Ref MongoDBClusterParameterGroup
      DeletionProtection: true
      BackupRetentionPeriod : 1
      DBSubnetGroupName : !Ref MongoDBSubnetGroup
      AvailabilityZones:
        - us-east-1a
        - us-east-1b
        - us-east-1c
      MasterUsername:
        Fn::Sub: "{{resolve:secretsmanager:${MongoDBSecret}::username}}"
      MasterUserPassword:
        Fn::Sub: "{{resolve:secretsmanager:${MongoDBSecret}::password}}"
      Port : 27017
      EngineVersion: 4.0.0
      PreferredBackupWindow : "23:00-23:59"
      PreferredMaintenanceWindow : "sun:00:00-sun:05:00"
      VpcSecurityGroupIds:
        - !Ref MongoDBSecurityGroup
      StorageEncrypted : true
      Tags:
        - Key: Name
          Value: !Sub "MongoDB cluster ${MongoDBInstanceIdentifier}"

  MongoDBClusterParameterGroup:
    Type: AWS::DocDB::DBClusterParameterGroup
    Properties:
      Description: !Sub "MongoDB cluster ${MongoDBInstanceIdentifier}"
      Family: "docdb4.0"
      Name: !Sub "mongo-cluster-params-${MongoDBInstanceIdentifier}"
      Parameters:
        audit_logs: "disabled"
        tls: "disabled"
        ttl_monitor: "enabled"
      Tags:
        - Key: Name
          Value: !Sub "MongoDB cluster parameters ${MongoDBInstanceIdentifier}"

  MongoDBInstance1:
    Type: AWS::DocDB::DBInstance
    Properties:
      DBInstanceClass: "db.r6g.large"
      DBClusterIdentifier: !Ref MongoDBCluster
      DBInstanceIdentifier: !Sub "mongodb-instance1-${MongoDBInstanceIdentifier}"
      PreferredMaintenanceWindow: "sun:00:00-sun:05:00"
      Tags:
        - Key: Name
          Value: !Sub "MongoDB Instance1 ${MongoDBInstanceIdentifier}"

  MongoDBInstance2:
    Type: AWS::DocDB::DBInstance
    Properties:
      DBInstanceClass: "db.r6g.large"
      DBClusterIdentifier: !Ref MongoDBCluster
      DBInstanceIdentifier: !Sub "mongodb-instance2-${MongoDBInstanceIdentifier}"
      PreferredMaintenanceWindow: "sun:00:00-sun:05:00"
      Tags:
        - Key: Name
          Value: !Sub "MongoDB Instance2 ${MongoDBInstanceIdentifier}"

  MongoDBInstance3:
    Type: AWS::DocDB::DBInstance
    Properties:
      DBInstanceClass: "db.r6g.large"
      DBClusterIdentifier: !Ref MongoDBCluster
      DBInstanceIdentifier: !Sub "mongodb-instance3-${MongoDBInstanceIdentifier}"
      PreferredMaintenanceWindow: "sun:00:00-sun:05:00"
      Tags:
        - Key: Name
          Value: !Sub "MongoDB Instance3 ${MongoDBInstanceIdentifier}"

Outputs:
  MongoDBClusterId:
    Description: DocumentDB/MongoDB cluster ID
    Value: !Ref MongoDBCluster
    Export:
      Name: !Sub "documentdb-cluster-id-${MongoDBInstanceIdentifier}"
  MongoDBClusterEndpoint:
    Description: DocumentDB/MongoDB cluster endpoint
    Value: !GetAtt MongoDBCluster.Endpoint
    Export:
      Name: !Sub "documentdb-cluster-endpoint-${MongoDBInstanceIdentifier}"
  MongoDBClusterReadEndpoint:
    Description: DocumentDB/MongoDB cluster READ endpoint
    Value: !GetAtt MongoDBCluster.ReadEndpoint
    Export:
      Name: !Sub "documentdb-cluster-read-endpoint-${MongoDBInstanceIdentifier}"
  MongoDBClusterPort:
    Description: DocumentDB/MongoDB cluster port
    Value: !GetAtt MongoDBCluster.Port
    Export:
      Name: !Sub "documentdb-cluster-port-${MongoDBInstanceIdentifier}"
  MongoDBSecret:
    Description: DocumentDB/MongoDB credentials secret ARN
    Value: !Ref MongoDBSecret
    Export:
      Name: !Sub "documentdb-${MongoDBInstanceIdentifier}-SecretARN"

CodePudding user response：

Sometimes you must ask the question to think of other potential answers. And, indeed, I was missing a "dependson".

This is mentioned in the Cloudformation documentation:

BUT:

the Cloudformation template now takes three times as long to run. Which is not "super terrible" because I only create my DocumentDB once per ENV. But if you're trying to recreate a new ENV in the middle of a panic, you will jump in place for around 20 minutes waiting for your cluster to be created, one turtle cluster at a time (you WILL lose hair).
Clouformation should be smarter and, when it detects a bunch of DocumentDB instances to create, it should invoke "aws sdk" calls to check in which AZs the instances can be created and pre-decide where each instance will get created, so that the parallel creation can run AND with insurance that all cluster instances will be in different AZs.
the AWS DocumentDB Cloudformation documentation should be updated. This page only shows a sample DocumentDB cluster with only ONE instance. I had to dig far and wide to find someone (public project on github) who was showing the best practice example of THREE instances per cluster. If AWS says DocumentDB best practice is to have THREE instances, there should be a sample Cloudformation template which, in turn, would contain the "dependson" fix/requirement above (unless they make Cloudformation smarter; see my previous bullet). So, AWS should update the templates linked at this page: https://docs.aws.amazon.com/documentdb/latest/developerguide/quick_start_cfn.html

CodePudding user response：

You can specify the AZ in the DBInstance resource, I see is supported. This way you can avoid the serialization of database instances creation.