-
Notifications
You must be signed in to change notification settings - Fork 266
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ASGHandler - High Memory needs (Runtime.ExitError) #552
Comments
Thanks for providing such a detailed report! I really appreciate it. Do you know if this error is consistent or if it comes and goes randomly? I am currently still investigating possible sources of the memory usage but the thing that stands out to me first is that "cold_start" was false so it's possible that memory usage is coming from previous executions. If this error occurs consistently for the same account/region then warm-starts likely aren't the culprit, but if it occurs randomly then they would seem like a much more likely candidate |
consistent, but not for all accounts I have switch back to 128MB for the ASGHandler for cost purposes. The solution is running every 60min but the errors are constantly appearing over the whole hour. between 6:11 and 6:13 i have 150 streams in the ASGHandler0F6D6751 Log group (Errors+Sucessfull). a successful run is this (also cold_start false):
also call with no schedule_names fail.
Do you need additional info ? DebugLog (Trace) is now turned on |
Yes any trace information you have from failed runs would be very much appreciated, and if you have a failure that includes If you would rather not post full traces to this thread you can send them to my work email: calepea@amazon.com For reference, there are 2 things that should trigger the ASG handler to take action:
If the errors are consistently occuring in specific accounts, then I wonder if we are receiving far more data than expected from one of the describe calls. I'll take another look at the control flow to try to confirm how long the results of those calls stay in memory. |
Where can i find the trace log entries? Can you also check that an deactivated region in an account is not a problem. |
on the AWS Lambda console, go to your ASG Handler function and click The expected behavior for a deactivated region in certain accounts would be an STS assume role error when the lambda first initializes. This is technically an error and will be reported as such, but the error will not cascade into other scheduling targets. We do not currently have a mechanism to suppress these errors in the logs or dashboard metrics, however. |
We could also setup a time to get onto a call with a screenshare to help troubleshoot exactly what's going on. I am on EDT time (UTC-4) and can be fairly flexible. |
Lambda memory allocation has been made configurable in v3.0.1 along with some minor updates to the control flow for the ASG scheduling function, but I have been unable to reproduce any high memory usage scenarios during load testing. If anybody continues to see high memory usage from the ASG handler, please let us know and kindly provide us with as much information as you have available to assist in identifying any underlying root causes. |
Describe the bug
Stack region: eu-west-1
Stack-Name: cs-instance-scheduler
TagName: scheduler_period
UsingAWSOrganizations: Yes
regions: ap-southeast-1,ap-southeast-2,eu-central-1,eu-west-1,eu-west-3,eu-north-1,me-south-1,me-central-1,us-east-1,us-east-2,us-west-1
InstanceScheduler Memory of 256MB needed (template setting MemorySize)
+100 spoke accounts
ASG scheduling works for the sandbox account using it, No other account use the tag as its a new deployment
Some accounts/regions need >700MB of memory for the ASGHandler execution or Runtime.ExitError happens.
Even in an account/region which doesn't use EC2,RDS services
Also the scheduler:run event contains schedule_names which are not used in this account.
High Memory usage:
REPORT RequestId: 531e9a6b-f6e1-4c02-9a27-32272ea6688b Duration: 702.22 ms Billed Duration: 703 ms Memory Size: 1024 MB Max Memory Used: 706 MB
The account 777777777777 in us-east-1 has 3 VPCs, no EC2 instances, no RDS databases, no ASG groups and uses Remote stack 3.0.0
The schedules "debug1" and "debug2" are not used in this account. These schedule names are only used in the Sandbox account eu-west-1 region and nowhere else in the AWS Organization spoke accounts.
A lot of accounts run below 128MB usage
To Reproduce
It's not happening in development environment but we only have 2 supported regions and 2 spoke accounts deployed there.
Expected behavior
Less memory need.
The default Memory usage 128MB should suffice or it needs a configuration parameter in Cloudformation template.
Please complete the following information about the solution:
the services this solution uses? no
Troubleshooting
Screenshots
no
Additional context
Information: the memory size of instance_scheduler.handler.asg.lambda_handler in fixed with 128 MB in the cloud formation stack template. For testing the value was manually changed on the already deployed Lambda function.
All accounts are managed by Controltower,
Control Tower setting prevent the default VPC creation, so supported scheduling regions can have 0 VPC
The text was updated successfully, but these errors were encountered: