AWSSDK Logger cannot fetch credentials from instance profile after upgrading to AWSSDK.Core 3.7.x #235
Comments
kirillowen
added
bug
ashishdhingra
added
module/logging
needs-reproduction
|
@kirillowen Good morning. Thanks for opening the issue. The error Could you please share the below information:
Thanks, |
ashishdhingra
added
the
response-requested
|
Hi Ashish, please see responses below:
The previous versions of the packages were:
The use of configuration keys in Removing In the link you provided "In Configuration Files Reference for AWS SDK for .NET " the last option in "Credential search order" is "8. Amazon EC2 instance metadata." which is what I assume the previous version of package was using, as profile was never set on the instance, however this now appears to fail or never reach IMDS to retrieve the credentials.
No, this has not been required in the past as the credentials resolved from the EC2 Instance profile. Does the profile now need to be set for the service to be able to read the credentials from the instance profile? I've also just tried to add the aws configSection to
Yes, the account runs as local system account and has full access to the folder and file. |
|
@kirillowen Thanks for your inputs. The link https://aws.amazon.com/blogs/developer/access-key-management-for-net-applications-part-1/ you shared is dated back in 2013, which is quite old and could be referencing v1 SDK. I do not see the mentioned keys supported in v3.5 or current version of SDK.
Thanks, |
ashishdhingra
added
response-requested
|
Understood, the keys have been removed from Added the options you suggested, and got the following output from IMDS v2 is set to "Optional" in the EC2 console, $TOKEN= (Invoke-WebRequest -Method PUT -Uri "http://169.254.169.254/latest/api/token" -Headers @{"X-aws-ec2-metadata-token-ttl-seconds" = "21600"} -UseBasicParsing) | Select-Object -ExpandProperty Content
$instanceId= Invoke-WebRequest -Uri "http://169.254.169.254/latest/meta-data/instance-id" -Headers @{"X-aws-ec2-metadata-token" = $TOKEN} -UseBasicParsing | Select-Object -ExpandProperty Content
$instanceId
The "default" profile is not set, confirmed with AWS CLI on the instance: |
|
@kirillowen In install-log.txt, I could see bunch of While starting the service, you mentioned that it freezes. Do you see any error in System Diagnostics, i.e., in event viewer? Thanks, |
github-actions
bot
removed
the
response-requested
|
@ashishdhingra I've created the directory, however the start still freezes. After uninstalling the app and removing from the instance, and trying to install from a fresh build the install now freezes again and there is no log output to |
|
To update the thread, I have further tested the repro repository on a fresh instance of the custom AMI, another custom AMI for another service and a new instance using @ashishdhingra - do you have any other suggestions to get around this issue; Anything further I can assist with to verify that that the issue exists within the package? |
@kirillowen After you created the said directory, what error did you see in the logs? If same error, did you grant account which is configured for Windows service necessary read/write permissions to directory. Do you see any error in System Diagnostics, i.e., in event viewer Also are you able to capture detailed verbose logs after enabling the same via below code: Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.SystemDiagnostics;
Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());Verbose logs will capture everything, including how credentials are resolved. I can try to reproduce the issue at my end. However, kindly note that we cannot use 3rd party tool Thanks, |
|
@ashishdhingra thank you for your suggestions. After creating the directory for aws logs, no errors were being written to the log file and nothing is being captured in System Diagnostics/ Event Viewer. I have added the I've also tried to replicate the issue without using We are currently reviewing options to remove the Thanks for your support and suggestions! |
|
Describe the bug
We have a windows service that is running on a Windows EC2 instance that is failing to start after upgrading the AWSSDK.Core and AWSSDK.Logger packages. The service uses Topshelf to install into Windows services, Log4Net to write logs and AWSSDK.CloudWatchLogs to push the logs to CloudWatch. After upgrading the packages the service is no longer able to install or start without the required credentials to sign into CloudWatch. If the
<appender-ref ref="AWS" />is removed fromLog4Net.configfile then the service installs and runs fine, logging to file/ console only.I've upload a repo here to reproduce the issue: https://github.com/kirillowen/awssdk-log4net-repro
The log4net appender config for AWS:
Package versions:
*Note: when updating packages to the latest versions, the issue can still be reproduced.
EC2 instance IAM role permissions for accessing logs:
{ "Version": "2012-10-17", "Statement": [ { "Action": [ "logs:Create*", "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "logs:DescribeLogGroups", "logs:PutRetentionPolicy", "s3:GetObject" ], "Resource": [ "arn:aws:logs:*:*:*", "arn:aws:s3:::*" ], "Effect": "Allow" } ] }Expected Behavior
The service, installs, starts and runs as before the package update.
Current Behavior
When installing the service through Topshelf command
C:\Service\Service.exe installthe install freezes and then times out.Reproduction Steps
See repository here with minimal service to test: https://github.com/kirillowen/awssdk-log4net-repro
.\AWSSDK.Log4Net.Repro.Service.exe install- the install will freeze<appender-ref ref="AWS" />fromLog4Net.configand run the install command again - the service will install.\AWSSDK.Log4Net.Repro.Service.exe start- service will start and log tolog-file.txt..\AWSSDK.Log4Net.Repro.Service.exe stopLog4Net.config,AWSAccessKey,AWSSecretKey,AWSSessionTokeninAppSettings.config. The service will start and run, but throws exception toawslog.txt:Possible Solution
No response
Additional Information/Context
No response
AWS .NET SDK and/or Package version used
Targeted .NET Platform
.NET Framework 4.6.2
Operating System and version
Windows Server 2019 Datacenter; Version 1809; OS build 17763.5206
The text was updated successfully, but these errors were encountered: