Using Custom Ansible Playbooks

This guide explains how custom Ansible playbooks can be integrated with Turbot for automated software configuration management and provisioning of applications in accounts.

Overview

Custom Ansible playbooks can be added to a Turbot account, and are then automatically executed on matching hosts during Turbot’s automated task runs (approx every 10 mins).

Creating a playbook

Turbot supports any Ansible playbook, and we recommend you follow Ansible best practices for the file and project structure. Typically, that will look like this:

main.yml
roles/
  common/
    files/
    templates/
    tasks/
    handlers/
    vars/
    defaults/
    meta/
  webservers/
    files/
    templates/
    tasks/
    handlers/
    vars/
    defaults/
    meta/

The main.yml file, which indicates the roles to be executed on each host can leverage predefined groups and hosts that are passed to the playbook by Turbot. The simplest example:

- hosts: all
  roles:
     - common
     - webservers

Hosts and playbook matching

Turbot generates the hosts file automatically based on the current AWS EC2 instances and their metadata. Your Ansible playbook should use this host information to match appropriate hosts for the playbook run.

Below is an example hosts file generated by Turbot. You will see tags for all EC2 servers (ec2), the account ID (turbot_account_abb), the network, the vpc, the hostname, the AWS EC2 instance id, the SSH key, the AMI, the instance type, the AWS VPC, the AWS subnet, the hostname, the IP address, the security group, and tags (tag_{Key}_{Value}, e.g. tag_Name_RHEL_Test).

{
  "ec2": [
    "10.1.2.154",
    "10.1.2.155"
  ],
  "_meta": {
    "hostvars": {
      "10.1.2.154": {
        "turbot": {
          "hostname": "dev120abbauva02154"
        },
        "ansible_ssh_user": "ec2-user"
      },
      "10.1.2.155": {
        "turbot": {
          "hostname": "dev120abbauva02155"
        },
        "ansible_ssh_user": "ubuntu"
      }
    }
  },
  "turbot_account_abb": [
    "10.1.2.154",
    "10.1.2.155"
  ],
  "turbot_network_tbtusea1": [
    "10.1.2.154",
    "10.1.2.155"
  ],
  "turbot_vpc_abbusea1": [
    "10.1.2.154",
    "10.1.2.155"
  ],
  "dev120abbauva02154": [
    "10.1.2.154"
  ],
  "us-east-1a": [
    "10.1.2.154",
    "10.1.2.155"
  ],
  "us-east-1": [
    "10.1.2.154",
    "10.1.2.155"
  ],
  "i-96fed144": [
    "10.1.2.154"
  ],
  "10.1.2.154": [
    "10.1.2.154"
  ],
  "52.22.1.154": [
    "10.1.2.154"
  ],
  "ami-12663b7a": [
    "10.1.2.154"
  ],
  "key_turbot": [
    "10.1.2.154",
    "10.1.2.155"
  ],
  "t2.micro": [
    "10.1.2.154"
  ],
  "security_group_default": [
    "10.1.2.154",
    "10.1.2.155"
  ],
  "sg-24967b43": [
    "10.1.2.154",
    "10.1.2.155"
  ],
  "tag_Name_RHEL_Test": [
    "10.1.2.154"
  ],
  "vpc-96b0bdf3": [
    "10.1.2.154",
    "10.1.2.155"
  ],
  "subnet-14a8f463": [
    "10.1.2.154",
    "10.1.2.155"
  ],
  "platform_linux": [
    "10.1.2.154",
    "10.1.2.155"
  ],
  "dev120abbauva02155": [
    "10.1.2.155"
  ],
  "i-f0a5f722": [
    "10.1.2.155"
  ],
  "10.1.2.155": [
    "10.1.2.155"
  ],
  "52.21.194.18": [
    "10.1.2.155"
  ],
  "ami-00615068": [
    "10.1.2.155"
  ],
  "t1.micro": [
    "10.1.2.155"
  ],
  "tag_Name_Ubuntu_12_04_Test_DMZ": [
    "10.1.2.155"
  ]
}

An example playbook to match all hosts is:

- hosts: all
  roles:
     - common

An example playbook to match all servers with a tag of AppTier=Web is:

- hosts: tag_AppTier_Web
  roles:
    - webservers

An example playbook to match a specific hostname is:

- hosts: dev120abbauva02154
  roles:
    - specific_host_role

Storing a playbook in S3

Turbot downloads and executes playbooks stored in an S3 bucket for the account where they are to be run. Simply use a tool like the AWS command line aws s3 sync to upload your playbook to an S3 bucket.

Multiple playbooks can be stored in the same S3 bucket by placing them into folders.

Registering a playbook with Turbot

To register a custom playbook with a Turbot account:

  1. Login to the Turbot console as a user with Turbot/Admin permissions or higher for the account.
  2. Go to the account page.
  3. Go to the Advanced tab, then Playbooks.
  4. Click Add Playbook.
  5. Choose a unique playbook ID, considering that playbooks are run in alphabetical order.
  6. Enter details of the S3 bucket and (optional) key prefix.
  7. Click Save.
  8. Your playbook will start automatically executing within 10 mins.

Running multiple playbooks

Turbot runs playbooks in the following order:

  • Custom playbooks, alphabetical by playbook ID.
  • Turbot core playbooks

Turbot core playbooks, such as Linux security hardening and user management, are run last to ensure their settings take precedence over those in custom playbooks.

To control the order of execution for custom playbooks, simply define playbook IDs in alphabetical order.

What happens when a playbook fails?

Ansible is designed to fail fast, so if any host fails for any playbook that playbook execution will stop in the account. Turbot runs each playbook separately, so execution of each playbook in the account can succeed or fail independently.

A single job is used to run all playbooks (in order) against all hosts. As per Ansible design, each playbook is run for all hosts before the next playbook is started. For example:

  1. Playbook A, Host 1
  2. Playbook A, Host 2
  3. Playbook B, Host 1
  4. Playbook B, Host 2
  5. Playbook turbot-config, Host 1
  6. Playbook turbot-config, Host 2

If any host for any playbook fails, the overall playbook run will raise an alarm. For example, if Playbook B, Host 1 fails then Playbook A will have executed successfully, Playbook turbot-config will have executed successfully, but Playbook B failed and Playbook B, Host 2 will have been skipped since the failure happened on Playbook B, Host 1. In this case, Turbot will raise an alarm (since Playbook B failed) but will report success within the alarm for Playbook A and Playbook turbot-config.

Checking playbook results

Playbook results are summarized in Turbot alarms and also logged to S3 with every run. This allows you to examine the full history of runs while providing a simple interface to the current status.

Alarm status can be viewed in the Account page of the Turbot console. Failing playbooks are prominantly displayed in the alarms at the top. Successful playbooks are visible in the Configuration Status section.

All playbook runs are logged to the Turbot S3 logs bucket for the region (e.g. aaa-usea1-asdfjan42), in the key prefix TurbotLogs/Ansible/{playbookId}/. The alarm shows the full path to the log file.

Turbot logs both a history of all playbook runs, but also updates status files for each playbook (_ok, _failed, etc) in S3. These status files are updated in place, so seeing multiple states is common, and the timestamp can be used to see what’s current and when each state last occured.

Development Tips

  1. Always develop and test Ansible playbooks manually before integration with Turbot. The cycle is faster and easier. It’s best to test them on a server in a Turbot environment however, making it as close as possible.

  2. Check the results of your playbooks by examining S3 logs. Check the timestamp to ensure you are viewing the latest log file.

  3. Always write your playbooks to be idempotent. They will be run every 10 mins so need to handle initial setup, fixes and ongoing runs.

Was this article helpful?
0 out of 0 found this helpful