Problem trying to refresh assume-role credentials automatically with KCL on EC2 #51

jazzTheJackRabbit · 2017-05-09T01:40:41Z

I have a Kinesis Stream Reader setup on an EC2 instance with the an instance profile with assume-role permissions to assume a reader-role to a Kinesis Stream in a separate AWS account. I can access the stream perfectly fine if I configure my credentials by using the STS service through the AWS-CLI, retrieve temporary credentials and setup the ~/.aws/credentials file accordingly. But, I'm running into problems if I try to use the STSAssumeRoleSessionCredentialsProvider in kcl.properties to automatically refresh the temporary credentials.

The credentials in kcl.properties is as follows:

AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::<account-id>:role/<role-name>|session-name

But I seem to be running into the following issue.

No credential providers specifiedjava com.amazonaws.services.kinesis.multilang.MultiLangDaemon <properties file>

I'm using the Amazon KCL Version : 1.4.4 with the following JARS:

amazon-kinesis-client-1.7.5.jar
aws-java-sdk-cloudwatch-1.11.115.jar
aws-java-sdk-core-1.11.115.jar
aws-java-sdk-dynamodb-1.11.115.jar
aws-java-sdk-kinesis-1.11.115.jar
aws-java-sdk-kms-1.11.115.jar
aws-java-sdk-s3-1.11.115.jar
commons-codec-1.9.jar
commons-lang-2.6.jar
commons-logging-1.1.3.jar
guava-18.0.jar
httpclient-4.5.2.jar
httpcore-4.4.4.jar
ion-java-1.0.2.jar
jackson-annotations-2.6.0.jar
jackson-core-2.6.6.jar
jackson-databind-2.6.6.jar
jackson-dataformat-cbor-2.6.6.jar
jmespath-java-1.11.115.jar
joda-time-2.8.1.jar
protobuf-java-2.6.1.jar

Any help with respect to missing JARs or configuration specifics would be helpful.

The text was updated successfully, but these errors were encountered:

antgus · 2017-05-18T15:16:37Z

Hi - I'm also trying to assume a role and so far have been unable to. I get the same message as you if I use the same format for AWSCredentialsProvider. I'm curious to know where you found the format in the first place? I haven't been able to find any kind of documentation for it.

Edit: Found the source code: https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/config/AWSCredentialsProviderPropertyValueDecoder.java#L72

pfifer · 2017-05-18T16:21:07Z

The STSAssumeRoleSessionCredentialsProvider isn't in the aws-java-sdk-core jar, so you would need to add the aws-java-sdk-sts jar to the class path.

If you don't mind using Apache Maven this is a pom.xml that will create a launch script that will setup the class path for you.

To create the launcher script:

Create an empty directory
Create a new file in the directory called pom.xml
Install Apache Maven if you don't already have it
In the directory created in 1, run the command mvn package
If everything works you should have a script in target/appassembler/bin/multilang-daemon, and all the jars are in target/appassembler/repo.
Copy the directory target/appassembler to wherever you need it.
To start your application run the mutlilang-daemon script with the location of your property file.

ykhrustalev · 2017-07-09T20:20:33Z

@pfifer , after following your suggestion and building multilang-daemo I seem still have an issue

./bin/multilang-daemon config/kcl.local.properties
/usr/bin/java -classpath /home/worker/app/etc:/home/worker/app/repo/com/amazonaws/amazon-kinesis-client/1.7.5/amazon-kinesis-client-1.7.5.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-dynamodb/1.11.115/aws-java-sdk-dynamodb-1.11.115.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-s3/1.11.115/aws-java-sdk-s3-1.11.115.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-kms/1.11.115/aws-java-sdk-kms-1.11.115.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-kinesis/1.11.115/aws-java-sdk-kinesis-1.11.115.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-cloudwatch/1.11.115/aws-java-sdk-cloudwatch-1.11.115.jar:/home/worker/app/repo/com/google/guava/guava/18.0/guava-18.0.jar:/home/worker/app/repo/com/google/protobuf/protobuf-java/2.6.1/protobuf-java-2.6.1.jar:/home/worker/app/repo/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/home/worker/app/repo/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-sts/1.11.129/aws-java-sdk-sts-1.11.129.jar:/home/worker/app/repo/com/amazonaws/aws-java-sdk-core/1.11.129/aws-java-sdk-core-1.11.129.jar:/home/worker/app/repo/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.jar:/home/worker/app/repo/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.jar:/home/worker/app/repo/commons-codec/commons-codec/1.9/commons-codec-1.9.jar:/home/worker/app/repo/software/amazon/ion/ion-java/1.0.2/ion-java-1.0.2.jar:/home/worker/app/repo/com/fasterxml/jackson/core/jackson-databind/2.6.6/jackson-databind-2.6.6.jar:/home/worker/app/repo/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.jar:/home/worker/app/repo/com/fasterxml/jackson/core/jackson-core/2.6.6/jackson-core-2.6.6.jar:/home/worker/app/repo/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.6.6/jackson-dataformat-cbor-2.6.6.jar:/home/worker/app/repo/joda-time/joda-time/2.8.1/joda-time-2.8.1.jar:/home/worker/app/repo/com/amazonaws/jmespath-java/1.11.129/jmespath-java-1.11.129.jar:/home/worker/app/repo/com/amazonaws/amazon-kinesis-multilang-launcher/1.0-SNAPSHOT/amazon-kinesis-multilang-launcher-1.0-SNAPSHOT.jar -Dapp.name=multilang-daemon -Dapp.pid=15086 -Dapp.repo=/home/worker/app/repo -Dapp.home=/home/worker/app -Dbasedir=/home/worker/app com.amazonaws.services.kinesis.multilang.MultiLangDaemon config/kcl.local.properties
No credential providers specifiedjava com.amazonaws.services.kinesis.multilang.MultiLangDaemon <properties file>

output contains extra echo call to dump classpath before calling exec.

ykhrustalev · 2017-07-09T20:45:41Z

@antgus here is an example https://github.com/awslabs/amazon-kinesis-client/blob/master/src/test/java/com/amazonaws/services/kinesis/clientlibrary/config/AWSCredentialsProviderPropertyValueDecoderTest.java#L63

AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::1111:role/name|session1

but seems does not work

ykhrustalev · 2017-07-09T21:09:11Z

it seems works with AWSCredentialsProvider = InstanceProfileCredentialsProvider

pfifer · 2017-07-19T17:58:03Z

I've just tested it using the line below (with redactions), and was able to access a stream in another account:

AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::<account id>:role/pfifer-cross-account|pfifer-session

I've tested this with version 1.11.151, and 1.11.129 of the STS SDK.

To get more information I've updated the pom.xml to include some logging libraries. You can use this without a logging configuration file, but it will default to DEBUG. You can configure the logging using these steps:

In you application directory create a directory called logback (you can call it something else if you want to).
Download this logback.xml file, and save it to logback/logback.xml
Start the script again like: CLASSPATH_PREFIX=./logback ./bin/multilang-daemon config/kcl.local.properties

It defaults to logging output to the console, but if you switch the logger to MAIN it will output to a file. This also enable debug logging, which can help us understand what is going on.

holmesjr · 2017-07-29T08:25:56Z

I have a slightly different issue with this. I have the jars loaded and it's picking up the STS provider, but it can no longer find the env vars containing my long lived creds:

Exception in thread "main" com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131)
at com.amazonaws.services.kinesis.multilang.MultiLangDaemonConfig.prepare(MultiLangDaemonConfig.java:116)
at com.amazonaws.services.kinesis.multilang.MultiLangDaemonConfig.(MultiLangDaemonConfig.java:107)
at com.amazonaws.services.kinesis.multilang.MultiLangDaemonConfig.(MultiLangDaemonConfig.java:77)
at com.amazonaws.services.kinesis.multilang.MultiLangDaemonConfig.(MultiLangDaemonConfig.java:64)
at com.amazonaws.services.kinesis.multilang.MultiLangDaemon.main(MultiLangDaemon.java:131)

The DefaultAWSCredentialsProviderChain works in the same config.

holmesjr · 2017-07-29T10:09:58Z

Oh, right, I jumped the gun. That's probably the highly obscure exception raised when the AssumeRole api call gets AccessDenied, which happened when I used the CLI with the same roles.

holmesjr · 2017-07-29T12:17:04Z

So, once I fixed the trust relationship, all worked well.

Belkhojayev · 2017-11-15T22:44:11Z

Hello @pfifer!

I'm struggling with the same problem as jazzTheJackRabbit and ykhrustalev. I'm using python.
What I tried:

Your approach adding aws-java-sdk-sts jar to the class path but receive an error:
Can't find any credentials provider matching com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::1111111111111:role/RoleName
Specifying
AWSCredentialsProvider = ProfileCredentialsProvider
with env var AWS_PROFILE=my_profile_name.
But received an error:
No credential providers specifiedjava com.amazonaws.services.kinesis.multilang.MultiLangDaemon <properties file>

Do I understand it correctly that KCL does not read configuration from ~/.aws/config and it is useless to create profiles with (assume)roles there? This approach works for me with boto3 library on python.

Any help will be highly appreciated.

Belkhojayev · 2017-11-16T22:42:02Z

Found solution myself. For those experiencing the same problem and using python:

I did what @pfifer suggested (it was very helpful). Afterwards, I took aws-java-sdk-sts-1.11.129.jar from
/target/appassembler/repo/com/amazonaws/aws-java-sdk-sts/1.11.129/ and put it into
lib/python2.7/site-packages/amazon_kclpy (the folder with python KCL library.)

Of course, it would have been much better if the file was added to the library by default.

In the sample.parameters file I wrote:

AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::<account>:role/<role>|<session_name>
AWSCredentialsProviderDynamoDB = DefaultAWSCredentialsProviderChain
AWSCredentialsProviderCloudWatch = DefaultAWSCredentialsProviderChain

Previously, I didn't specify the session_name, cause boto3 didn't need it. That was a mistake. session_name can be anything, up to you.

In my case I need different credentials for DynamoDB and for Kinesis. In this commit I found how to do it.

pierredilauro · 2017-12-07T17:46:45Z

Hello @pfifer!

This thread has been really helpful to setup the Python KCL with STSAssumeRoleSessionCredentialsProvider. However, we enforce MFA to Assume Roles here.

I am jumping in to know how I can configure this in the properties file. Is it even supported?

Thanks,
Pierre

jiawen-earnin · 2018-03-28T06:00:55Z

Hello @holmesjr . I have met the same error "Unable to load AWS credentials from any provider in the chain" Can I know how did u fix the trust relationship? Thanks

jiawen-earnin · 2018-03-28T07:35:05Z

Hello @pfifer !
I have created an IAM role which defines all the permissions of AWS resources I will use in my application. And in my local environment, I created config file under ~/.aws path as following:
[default]
region = us-west-2
[profile my_profile_name]
role_arn = my_role_arn
region = us-west-2
source_profile = default

then I export the AWS_PROFILE variable in the following way:
export AWS_PROFILE = my_profile_name
so that the app can run according to the IAM role.

As you mentioned above, STSAssumeRoleSessionCredentialsProvider isn't in the aws-java-sdk-core jar. So I manually downloaded aws-java-sdk-sts-1.11.300.jar and added it into the same directory where other jars stored. Then I changed AWSCredentialsProvider property into STSAssumeRoleSessionCredentialsProvider in property file. However, I always had this error "Unable to load AWS credentials from any provider in the chain"

I have two questions:

Is it possible to run the consumer code locally (in my laptop environment) with IAM role defined in /.aws/config file and what causes that error?
There is a PR Added aws-java-sdk-sts to jar-dependencies #70 which let setup.py downloads the jar including sts.jar. Why this PR hasn't be merged? I found the corresponding STSAssumeRoleSessionCredentialsProvider class is deprecated https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleSessionCredentialsProvider.html . Is this the reason why setup.py won't download sts jar for developers?

pfifer · 2018-04-06T15:04:56Z

The script uses a defined classpath so just adding the jar won't include it. I would recommend following the instructions in my earlier post.

To answer your second question the STS jar is only required for those using the STS credentials. We don't want to increase the required dependencies unless absolutely necessary.

ShelRoman · 2020-03-25T09:47:38Z

Hello @pfifer, is there a way to pass external_Id when using STSAssumeRoleSessionCredentialsProvider?

bandand · 2020-08-05T10:49:58Z

Found solution myself. For those experiencing the same problem and using python:

I did what @pfifer suggested (it was very helpful). Afterwards, I took aws-java-sdk-sts-1.11.129.jar from
/target/appassembler/repo/com/amazonaws/aws-java-sdk-sts/1.11.129/ and put it into
lib/python2.7/site-packages/amazon_kclpy (the folder with python KCL library.)

Of course, it would have been much better if the file was added to the library by default.

In the sample.parameters file I wrote:
AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::<account>:role/<role>|<session_name>
AWSCredentialsProviderDynamoDB = DefaultAWSCredentialsProviderChain
AWSCredentialsProviderCloudWatch = DefaultAWSCredentialsProviderChain
Previously, I didn't specify the session_name, cause boto3 didn't need it. That was a mistake. session_name can be anything, up to you.

In my case I need different credentials for DynamoDB and for Kinesis. In this commit I found how to do it.

I have followed the same , however not sure why still getting this below error - any idea please

Caused by: java.lang.IllegalArgumentException: No credential providers specified
at com.amazonaws.auth.AWSCredentialsProviderChain.(AWSCredentialsProviderChain.java:59)
at software.amazon.kinesis.multilang.config.AWSCredentialsProviderPropertyValueDecoder.decodeValue(AWSCredentialsProviderPropertyValueDecoder.java:57)
at software.amazon.kinesis.multilang.config.MultiLangDaemonConfiguration.lambda$new$0(MultiLangDaemonConfiguration.java:237)
at software.amazon.kinesis.multilang.config.BuilderDynaBean.set(BuilderDynaBean.java:258)
at software.amazon.kinesis.multilang.config.MultiLangDaemonConfiguration.setAWSCredentialsProvider(MultiLangDaemonConfiguration.java:172)
... 18 more
java.lang.reflect.InvocationTargetException

manuelgeek · 2022-03-15T08:28:45Z

I've just tested it using the line below (with redactions), and was able to access a stream in another account:
AWSCredentialsProvider = STSAssumeRoleSessionCredentialsProvider|arn:aws:iam::<account id>:role/pfifer-cross-account|pfifer-session
I've tested this with version 1.11.151, and 1.11.129 of the STS SDK.

To get more information I've updated the pom.xml to include some logging libraries. You can use this without a logging configuration file, but it will default to DEBUG. You can configure the logging using these steps:

In you application directory create a directory called logback (you can call it something else if you want to).

Download this logback.xml file, and save it to logback/logback.xml

Start the script again like: CLASSPATH_PREFIX=./logback ./bin/multilang-daemon config/kcl.local.properties

It defaults to logging output to the console, but if you switch the logger to MAIN it will output to a file. This also enable debug logging, which can help us understand what is going on.

this worked for me after hours of errors

alexevan13 · 2023-03-17T13:55:07Z

If this doesnt work, add sts-sdk manually using this command
wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-sts/1.12.428/aws-java-sdk-sts-1.12.428.jar -O "$APP_ROOT"/kcl/aws-sdk-sts.jar

and append java path of the location
KCL_COMMAND="$(python "$APP_ROOT"/kcl/helper.py --print_command --java /usr/bin/java --properties "$APP_ROOT"/kcl/consumer.properties --classpath "$APP_ROOT"/kcl/*.jar)"

Works perfectly 💯

enriquecorp · 2023-09-25T22:26:48Z

If this doesnt work, add sts-sdk manually using this command wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-sts/1.12.428/aws-java-sdk-sts-1.12.428.jar -O "$APP_ROOT"/kcl/aws-sdk-sts.jar

and append java path of the location KCL_COMMAND="$(python "$APP_ROOT"/kcl/helper.py --print_command --java /usr/bin/java --properties "$APP_ROOT"/kcl/consumer.properties --classpath "$APP_ROOT"/kcl/*.jar)"

Works perfectly 💯
Thank you @alexevan13 , one variant I did was to include this reference in the pom.xml I have

pfifer mentioned this issue Mar 7, 2018

Records not being processed with sample app - simple print(data) not working #74

Open

pfifer mentioned this issue Aug 8, 2018

How to assume role while using KCL-Net awslabs/amazon-kinesis-client-net#22

Closed

pfifer mentioned this issue Aug 27, 2018

Check/Monitor the kcl awslabs/amazon-kinesis-client-net#25

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem trying to refresh assume-role credentials automatically with KCL on EC2 #51

Problem trying to refresh assume-role credentials automatically with KCL on EC2 #51

jazzTheJackRabbit commented May 9, 2017 •

edited

Loading

antgus commented May 18, 2017 •

edited

Loading

pfifer commented May 18, 2017

ykhrustalev commented Jul 9, 2017 •

edited

Loading

ykhrustalev commented Jul 9, 2017 •

edited

Loading

ykhrustalev commented Jul 9, 2017

pfifer commented Jul 19, 2017

holmesjr commented Jul 29, 2017

holmesjr commented Jul 29, 2017

holmesjr commented Jul 29, 2017

Belkhojayev commented Nov 15, 2017 •

edited

Loading

Belkhojayev commented Nov 16, 2017 •

edited

Loading

pierredilauro commented Dec 7, 2017

jiawen-earnin commented Mar 28, 2018

jiawen-earnin commented Mar 28, 2018

pfifer commented Apr 6, 2018

ShelRoman commented Mar 25, 2020

bandand commented Aug 5, 2020

manuelgeek commented Mar 15, 2022

alexevan13 commented Mar 17, 2023

enriquecorp commented Sep 25, 2023

Problem trying to refresh assume-role credentials automatically with KCL on EC2 #51

Problem trying to refresh assume-role credentials automatically with KCL on EC2 #51

Comments

jazzTheJackRabbit commented May 9, 2017 • edited Loading

antgus commented May 18, 2017 • edited Loading

pfifer commented May 18, 2017

ykhrustalev commented Jul 9, 2017 • edited Loading

ykhrustalev commented Jul 9, 2017 • edited Loading

ykhrustalev commented Jul 9, 2017

pfifer commented Jul 19, 2017

holmesjr commented Jul 29, 2017

holmesjr commented Jul 29, 2017

holmesjr commented Jul 29, 2017

Belkhojayev commented Nov 15, 2017 • edited Loading

Belkhojayev commented Nov 16, 2017 • edited Loading

pierredilauro commented Dec 7, 2017

jiawen-earnin commented Mar 28, 2018

jiawen-earnin commented Mar 28, 2018

pfifer commented Apr 6, 2018

ShelRoman commented Mar 25, 2020

bandand commented Aug 5, 2020

manuelgeek commented Mar 15, 2022

alexevan13 commented Mar 17, 2023

enriquecorp commented Sep 25, 2023

jazzTheJackRabbit commented May 9, 2017 •

edited

Loading

antgus commented May 18, 2017 •

edited

Loading

ykhrustalev commented Jul 9, 2017 •

edited

Loading

ykhrustalev commented Jul 9, 2017 •

edited

Loading

Belkhojayev commented Nov 15, 2017 •

edited

Loading

Belkhojayev commented Nov 16, 2017 •

edited

Loading