Throttling AWS Resource Discovery

Overview

Turbot has continually improved in real-time discovery of resources, while also rapidly expanding coverage of AWS regions and resource types. As a result, jobs to discover AWS resources by fanning have grown in workload and reduced in importance.

This guide provides an overview of how these jobs can be throttled to explicitly control and reduce this work. Setting the throttle for a region, service or account will bring a corresponding reduction in running of tasks.

Setting Throttling

For each AWS account, AWS > Resource Discovery Throttle Rules can be set with different configurations to control resource discovery job rates across regions and services.

For instance, to throttle 50% of resource discovery jobs in all regions and services, the policy can be set to:

- { throttle: 50 }

To throttle us-east-1 at 30%, but continue throttling other regions at 50%:

- { throttle: 50 }
- { region: us-east-1, throttle: 50 }

Specific services can also be throttled:

- { throttle: 50 }
- { service: ec2, throttle: 70 }

The order of the items in the policy does not matter; however, each item in the policy MUST have the throttle property as an integer between 0 and 100. If an item does not meet these conditions, then it will not be included when evaluating throttling rates.

Order of Precendence

Turbot uses the following order of precendence when evaluating which throttling rate to use:

  • Region + service throttling rate
  • Region or service throttling rate (whichever is greater)
  • Account throttling rate

For instance, for the following policy value:

- { throttle: 50 }
- { region: us-east-1, throttle: 70 }
- { service: iam, throttle: 20 }
- { service: ec2, throttle: 90 }
- { region: ap-south-1, service: ec2, throttle: 10 }
  • EC2 in ap-south-1 is throttled at 10%
  • EC2 in us-east-1 is throttled at 90% (as the EC2 service item’s throttling rate is greater than the us-east-1 item’s throttling rate)
  • EC2 in all other regions is throttled at 90%
  • IAM is throttled at 90%
  • All services except EC2 in us-east-1 are throttled at 70%
  • All services beside EC2 in regions other than us-east-1 are throttled at 50%

Conflicting Items

If there are items that conflict with one another, such as:

- { service: ec2, throttle: 70 }
- { service: ec2, throttle: 80 }
- { service: ec2, throttle: 90 }

Turbot will take the higher of the throttling values (no matter the order of the items), so in this case, EC2 discovery will be throttled at 90%.

Was this article helpful?
0 out of 0 found this helpful