Memoizing and caching CredentialsProviders return credentials that are just about to expire

[ho-obviux]

My PHP application uses aws-sdk-php v. 3.61.5 to publish messages to an sns topic. The application is running on an ec2 instance with an instance profile that allows it to publish messages to the sns topic.

The application is supposed to publish around 600,000 messages to the SNS topic as fast as it can. Everything works well for the first several hundreds of thousands messages. After several hours, however, it fails with this message:

Aws\Sns\Exception\SnsException

Error executing "Publish" on "https://sns.eu-west-1.amazonaws.com"; AWS HTTP error: Client error: `POST https://sns.eu-west-1.amazonaws.com` resulted in a `403 Forbidden` response:
<ErrorResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/">
  <Error>
    <Type>Sender</Type>
    <Code>ExpiredToke (truncated...)
 ExpiredToken (client): The security token included in the request is expired - <ErrorResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/">
  <Error>
    <Type>Sender</Type>
    <Code>ExpiredToken</Code>
    <Message>The security token included in the request is expired</Message>
  </Error>
  <RequestId>e6c59e4f-b998-5d99-b926-cafe6ac2711a</RequestId>
</ErrorResponse>

So clearly the security token expired.

I am using the following code to create the sns client:

        $provider = CredentialProvider::defaultProvider();
        $provider = CredentialProvider::cache(
            $provider,
            new DoctrineCacheAdapter(new FilesystemCache('/tmp/cache'))
        );

        $this->client = new SnsClient([
            'region'      => 'eu-west-1',
            'version'     => 'latest',
            'credentials' => $provider
        ]);

The caching CredentialsProvider will ask the underlying (default) CredentialsProvider for credentials if the currently cached credentials are expired. And the default CredentialsProvider will refresh the credentials if they are expired.

The problem, as I see it, is that both the caching CredentialsProvider and the default CredentialsProvider continue to return the existing credentials up to the very last millisecond where they are not expired. This means that just before they expire, the sns client will fetch credentials and use them to call the sns service. By the time the request reaches the sns service, the credentials have expired, and the service consequently rejects the request.

According to AWS docs (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials) refreshed credentials are available from the instance metadata at least five minutes before they expire, so perhaps it would be better for the CredentialsProvider implementations to refresh the token when it expires in less than five minutes?

If I am not using the sdk correctly, some guidance would be highly appreciated :-)

[veihmeyer]

nice

[bgaillard]

WARNING : Our assumptions was bad, the problem is not the same as the one described by @ho-obviux. Our problem came from a throttling problem on the ECS endpoint which is used to get AWS credentials. In consequence the credentials provider chain fall back to an instance profile attach to an IAM role which did not have the required permissions.

Hi, we recently update our PHP application to use credentials provided by ECS (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html). The principle is the same as credentials associated to Instance Profile.

After our update everything works as expected during several hours. But since few days we encounter random errors we did not have before.

Those errors are Access Denied errors, we do not understand why 99.99% of our API calls work and only very few did not randomly.

For example we get errors like this one with Firehose (less than 10 errors for several hundred thousand calls).

Error executing "PutRecord" on "https://firehose.eu-west-1.amazonaws.com"; AWS HTTP error: Client error: `POST https://firehose.eu-west-1.amazonaws.com` resulted in a `400 Bad Request` response:
{"__type":"AccessDeniedException","Message":"User: arn:aws:sts::XXXXXXXX:assumed-role/gdc-website/i-xxxxxxxx (truncated...)
 AccessDeniedException (client): User: arn:aws:sts::771787368257:assumed-role/gdc-website/i-xxxxxxxx is not authorized to perform: firehose:PutRecord on resource: arn:aws:firehose:eu-west-1:XXXXXXXX:deliverystream/analytics-events - {"__type":"AccessDeniedException","Message":"User: arn:aws:sts::771787368257:assumed-role/gdc-website/i-xxxxxxxx is not authorized to perform: firehose:PutRecord on resource: arn:aws:firehose:eu-west-1:XXXXXXXX:deliverystream/analytics-events"}

We also suspect expiration management problems in the PHP SDK and suspect that we encounter our random errors just before / after the temporary credentials are renewed.

In our case we do not use CredentialProvider::cache(...).

Looking inside the source code I believe their are potential problems inside the isExpired() method (see https://github.com/aws/aws-sdk-php/blob/master/src/Credentials/Credentials.php#L62 and https://github.com/aws/aws-sdk-php/blob/master/src/Credentials/CredentialProvider.php#L199).

• The time() method is dependent on server time, even if we use Amazon Linux 2 machine correctly configured with NTP we could encounter milliseconds / seconds deltas
• No additional security delta is taken into account for expiration

Looking inside the Botocore implementation it appears that this is better managed and that a refresh_in parameter has been added to manage a time delta.

See https://github.com/boto/botocore/blob/develop/botocore/credentials.py#L339 and especially https://github.com/boto/botocore/blob/develop/botocore/credentials.py#L431.

I'm not 100% sure (we can do additional tests / debugging) but I think this case should be considered to be a bug and not only a feature request ? In fact we're surprised to not see much more people which encounter this problem.

Thanks in for your help.

Baptiste

[kaktusas2598]

We are having same issue, except that in our case it is mainly s3:GetObject operation. We use memoize and chain method of CredentialProvider class. I don't why this is marked as feature request, it looks like a bug to me, because we took this snippet from official AWS SDK Documentation: https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/guide_credentials_provider.html#chaining-providers