How to configure IAM role to enable SSM for a new EC2 instance?-CodePudding

I'm running the following:

KEY=test
QUERY=ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20210430

aws ec2 create-key-pair --key-name $KEY --query 'KeyMaterial' --output text > $KEY.pem
chmod 600 $KEY.pem

aws ec2 create-security-group --group-name "$KEY" --description "$KEY" --output text > $KEY.sg.txt
SGID=$(cat $KEY.sg.txt)
aws ec2 authorize-security-group-ingress --group-id $SGID --protocol tcp --port 22 --cidr 0.0.0.0/0 > $KEY.sg.json

AMIID=$(aws ec2 describe-images --filters "Name=name,Values=$QUERY" --query "reverse(sort_by(Images, &CreationDate))[0].[ImageId]" --output text)
INSTANCEID=$(aws ec2 run-instances --count 1 --instance-type t2.micro --key-name "$KEY" --security-group-ids "$KEY" --image-id $AMIID --query 'Instances[*].InstanceId' --output text)

# after a wait, instance appears running

aws ssm describe-instance-information --output text
# prints nothing

aws ssm send-command --instance-ids "$INSTANCEID" --document-name "AWS-RunShellScript" --comment "IP config" --parameters commands=ifconfig --output text
# fails with invalid InstanceId

I've tried ssh'ing, ssm agent appears to be running, it seems from https://aws.amazon.com/premiumsupport/knowledge-center/systems-manager-ec2-instance-not-appear/ that IAM roles are misconfigured.

How can I configure the IAM roles / IAM instance profiles properly to use SSM agent and run commands with it? Or maybe something else is the problem considering the logs?

Thank you!

ssh works:

$ sudo cat /var/log/amazon/ssm/amazon-ssm-agent.log

ubuntu@ip-172-31-28-150:~$ sudo cat /var/log/amazon/ssm/amazon-ssm-agent.log
2021-10-21 14:43:21 WARN Error adding the directory '/etc/amazon/ssm' to watcher: no such file or directory
2021-10-21 14:43:21 INFO [amazon-ssm-agent] amazon-ssm-agent - v3.0.529.0
2021-10-21 14:43:21 INFO [amazon-ssm-agent] OS: linux, Arch: amd64
2021-10-21 14:43:22 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] [WorkerProvider] Worker ssm-agent-worker is not running, starting worker process
2021-10-21 14:43:22 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] [WorkerProvider] Worker ssm-agent-worker (pid:1185) started
2021-10-21 14:43:22 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] Monitor long running worker health every 60 seconds
2021-10-21 14:43:23 WARN Error adding the directory '/etc/amazon/ssm' to watcher: no such file or directory
2021-10-21 14:43:23 INFO [ssm-agent-worker] Dial to Core Agent broadcast channel
2021-10-21 14:43:23 INFO [ssm-agent-worker] Start to listen to Core Agent termination channel
2021-10-21 14:43:23 INFO [ssm-agent-worker] Dial to Core Agent broadcast channel
2021-10-21 14:43:23 INFO [ssm-agent-worker] Start to listen to Core Agent health channel
2021-10-21 14:43:23 INFO [ssm-agent-worker] Create new startup processor
2021-10-21 14:43:23 INFO [ssm-agent-worker] [StartupProcessor] Executing startup processor tasks
2021-10-21 14:43:23 INFO [ssm-agent-worker] [StartupProcessor] Write to serial port: Amazon SSM Agent v3.0.529.0 is running
2021-10-21 14:43:23 INFO [ssm-agent-worker] [StartupProcessor] Write to serial port: OsProductName: Ubuntu
2021-10-21 14:43:23 INFO [ssm-agent-worker] [StartupProcessor] Write to serial port: OsVersion: 20.04
2021-10-21 14:43:23 INFO [ssm-agent-worker] Entering SSM Agent hibernate - EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
        status code: 404, request id:
caused by: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>

$ sudo snap services amazon-ssm-agent

Service                            Startup  Current  Notes
amazon-ssm-agent.amazon-ssm-agent  enabled  active   -

$ sudo systemctl status snap.amazon-ssm-agent.amazon-ssm-agent.service

● snap.amazon-ssm-agent.amazon-ssm-agent.service - Service for snap application amazon-ssm-agent.amazon-ssm-agent
     Loaded: loaded (/etc/systemd/system/snap.amazon-ssm-agent.amazon-ssm-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2021-10-21 14:43:20 UTC; 4min 32s ago
   Main PID: 1153 (amazon-ssm-agen)
      Tasks: 17 (limit: 1160)
     Memory: 94.9M
     CGroup: /system.slice/snap.amazon-ssm-agent.amazon-ssm-agent.service
             ├─1153 /snap/amazon-ssm-agent/3552/amazon-ssm-agent
             └─1185 /snap/amazon-ssm-agent/3552/ssm-agent-worker

Oct 21 14:43:20 ip-172-31-28-150 systemd[1]: Started Service for snap application amazon-ssm-agent.amazon-ssm-agent.
Oct 21 14:43:21 ip-172-31-28-150 amazon-ssm-agent.amazon-ssm-agent[1153]: Error occurred fetching the seelog config file path:  open /etc/amazon/ssm/seelog.xml: no such file or directory
Oct 21 14:43:21 ip-172-31-28-150 amazon-ssm-agent.amazon-ssm-agent[1153]: Initializing new seelog logger
Oct 21 14:43:21 ip-172-31-28-150 amazon-ssm-agent.amazon-ssm-agent[1153]: New Seelog Logger Creation Complete
Oct 21 14:43:21 ip-172-31-28-150 amazon-ssm-agent.amazon-ssm-agent[1153]: 2021-10-21 14:43:21 WARN Error adding the directory '/etc/amazon/ssm' to watcher: no such file or directory
Oct 21 14:43:21 ip-172-31-28-150 amazon-ssm-agent.amazon-ssm-agent[1153]: 2021-10-21 14:43:21 INFO [amazon-ssm-agent] amazon-ssm-agent - v3.0.529.0
Oct 21 14:43:21 ip-172-31-28-150 amazon-ssm-agent.amazon-ssm-agent[1153]: 2021-10-21 14:43:21 INFO [amazon-ssm-agent] OS: linux, Arch: amd64
Oct 21 14:43:22 ip-172-31-28-150 amazon-ssm-agent.amazon-ssm-agent[1153]: 2021-10-21 14:43:22 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] [WorkerProvider] Worker ssm-agent-worker is not running, starting worker process
Oct 21 14:43:22 ip-172-31-28-150 amazon-ssm-agent.amazon-ssm-agent[1153]: 2021-10-21 14:43:22 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] [WorkerProvider] Worker ssm-agent-worker (pid:1185) started
Oct 21 14:43:22 ip-172-31-28-150 amazon-ssm-agent.amazon-ssm-agent[1153]: 2021-10-21 14:43:22 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] Monitor long running worker health every 60 seconds

CodePudding user response：

I'm assuming that your EC2 instance has internet access via an internet gateway. If not you'd have to setup a VPC Endpoint for SSM (see https://aws.amazon.com/blogs/mt/automated-configuration-of-session-manager-without-an-internet-gateway/ for that).

You then need to attach an instance profile with the appropriate permissions to your instance. For that you can e.g. use the existing managed policy AmazonSSMManagedInstanceCore. To attach the profile you use --iam-instance-profile from your aws ec2 run-instancescommand.

You can find a hands-on-lab at https://acloudguru.com/hands-on-labs/creating-an-ssm-iam-role-and-configuring-an-ec2-instance-with-aws-systems-manager-via-the-cli which seems to describe all the necessary steps on how to create an instance profile and attach it to the instance via the cli. Note that this lab does not use the AmazonSSMManagedInstanceCore managed policy. But the steps stay the same.