-
Notifications
You must be signed in to change notification settings - Fork 18
ProdTest Stress Testing
The purpose of this documentation is to help serve as an SOP for testing new API endpoints within Caseflow by replicating load on the system in a way that is only seen in Production. The stress test below can help:
- Validate new API endpoints
- Benchmark new code being introduced to Caseflow
- Look for weaknesses in Caseflow's current architecture
Due to the limited traffic in our testing environments, this stress test is the closest way to replicate production traffic in a safe and controlled way. With Caseflow being an enterprise level software, new code needs to be tested for Big O Notation concerns to prevent issues upon release to our production environment.
To stress test in prod test, you can write a script using Locust: https://locust.io/. Locust allows you to write tasks that generate API calls to simulate user traffic. These tasks will randomly call out one after another as long as the script is running (you can weight different tasks to run more often if needed though).
To set up this script, you first need Python 3.11 on your GFE: https://www.python.org/downloads/
After python is setup on your GFE, you can install locust by running pip install locust locust-plugins
in either your terminal or PowerShell.
An example script is attached below. To build the script you will need:
- The name of the file must be
locustfile.py
- CSV file(s) with prod test veteran identifiers (used in API call headers). These files can be generated from a Metabase query on your GFE.
- Methods (notated with
@task
) that generate API calls - API key(s) for your headers.
API keys will need to be generated after a successful Prod Test deployment and added to the script. API keys can be generated in the Prod Test Rails console using the instructions on the Caseflow Wiki
To generate an IDT token, use this in the Prod Test Rails console:
key, token = Idt::Token.generate_one_time_key_and_proposed_token
Idt::Token.activate_proposed_token(key, "CSS_ID_OF_ELIBIGLE_USER")
# Token for IDT API
token
To get an eligible user to generate your token, run: CSS_ID_OF_ELIGIBLE_USER = User.all.filter(&:intake_user?).first.css_id
To execute your script on your GFE:
- Open up a terminal or PowerShell window.
- Navigate to the folder with your locustfile.py file
- To launch the script, run
locust -H https://appeals.cf.prodtest.ds.va.gov/
to point your API calls to the prod test instance
While the script is running, Locust will generate a UI that is accessible at http://localhost:8089/
. You can use the UI to set the number of users, the spawn rate of requests, and the host URL if needed. While the tasks are running in the background hitting the Prod Test endpoint, the Locust UI will also give you metrics about the number of requests and overall health of the prod test endpoint. Under the charts tab, you can have HTML charts that you can download to show the results of your test.
- Home
- Acronyms and Glossary
- Caseflow products
- Caseflow Intake
- Caseflow Queue
- Appeals Consumer
- Caseflow Reader
- Caseflow eFolder
- Caseflow Hearings
- Caseflow Certification
- Caseflow APIs
- Appeal Status API
- Caseflow Dispatch
-
CSUM Roles
- System Admin
- VHA Team Management
- Active Record Queries Resource
- External Integrations
- Caseflow Demo
- Caseflow ProdTest
- Background
- Stuck Jobs
- VA Notify
- Caseflow-Team
- Frontend Best Practices
- Accessibility
- How-To
- Debugging Tips
- Adding a Feature Flag with FeatureToggle
- Editing AMA issues
- Editing a decision review
- Fixing task trees
- Investigating and diagnosing issues
- Data and Metric Request Workflow
- Exporting and Importing Appeals
- Explain page for Appeals
- Record associations and Foreign Keys
- Upgrading Ruby
- Stuck Appeals
- Testing Action Mailer Messages Locally
- Re-running Seed Files
- Rake Generator for Legacy Appeals
- Manually running Scheduled Jobs
- System Admin UI
- Caseflow Makefile
- Upgrading Postgresql from v11.7 to v14.8 Locally
- VACOLS VM Trigger Fix M1
- Using SlackService to Send a Job Alert
- Technical Talks