Amazon Kinesis support for use of regional STS endpoints when assuming IAM role

[dpiekacz]

Description

Druid allows to provide IAM role that can be assumed to ingest Kinesis data from the same or another AWS account.
However, it doesn't allow to specify AWS region and the default behaviour of AWS SDK 1.x for Java is to use the legacy/global STS endpoint.

This proposal is to add "awsRegion" option to supervisor ioConfig to allow users to specify it, additionally to the IAM role ARN "awsAssumedRoleArn ", and then when assuming the IAM role send the request to the regional STS endpoint sts.[region_name].amazonaws.com instead of the legacy/global endpoint sts.amazonaws.com.

This feature request may potentially relate to an existing "AWS SDK 1.x EOL - Migrate AWS SDK for Java from 1.x to 2.x #16903 because SDK v2.x by default will use regional STS endpoint if AWS region is provided.

Motivation

The proposed feature will allow to:

• Reduce latency by making STS calls going to an endpoint that is geographically closer to where Druid cluster is setup.
• Reduce cross-region dependency. The legacy/global STS endpoint is highly available, but it is hosted in a single AWS US-East region only.
• Use generated session token to access Kinesis is all regions. Session tokens from regional STS endpoints are valid in all AWS regions where these from the global STS endpoint are valid only in AWS regions that are enabled by default.

Source Docs:

• https://docs.aws.amazon.com/general/latest/gr/sts.html
• https://docs.aws.amazon.com/sdkref/latest/guide/feature-sts-regionalized-endpoints.html
• https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html