-
Notifications
You must be signed in to change notification settings - Fork 18
Quality Review
A small number of appeals need to be routed to quality review before dispatching appeals each month in order to monitor decision-writing quality and consistency.
- They have a quality review team queue, with individuals on the QR team having their own queues
- QR admin will manually assign cases selected for Quality Review to individuals on the team
- Quality Review has the ability to send a case:
- Back to their judge (if there are required corrections to the decision)
- Straight to BVA dispatch (if there are no issues with the decision and it can go straight out)
When submitting a decision using the "Ready for Dispatch" task action on a JudgeDecisionReviewTask, the request is sent to the CaseReviewsController which uses the CompleteCaseReview workflow to process the request. For an AMA appeal, this workflow is where the QR selection occurs; the create_next_task
method will use the QualityReviewCaseSelector class to randomly select an appeal using the QUALITY_REVIEW_SELECTION_PROBABILITY
constant defined in the class. There also is a limit on the number of cases which can be selected for QR per month, which is checked by comparing the defined constant to the number of QualityReviewTask already created in the current month. If enough cases have already been created, the selector will return false for the rest of that month.
def select_case_for_quality_review?
return false if reached_monthly_limit_in_quality_reviews?
rand < QUALITY_REVIEW_SELECTION_PROBABILITY
end
If the appeal is selected, a QualityReviewTask will be created on the appeal and assigned to the QualityReview team. If it is not selected, a BvaDispatchTask is created and assigned to BvaDispatch instead.
if QualityReviewCaseSelector.select_case_for_quality_review?
QualityReviewTask.create_from_root_task(root_task)
elsif open_qr_tasks?
BvaDispatchTask.create_from_root_task(root_task)
end
For Legacy cases, the selection of an appeal to go to Quality Review is done as an after_create
hook on the JudgeCaseReview model. Functionally it does the same check as the QualityReviewCaseSelector class, but with constants defined in the JudgeCaseReview class itself. Like the QualityReviewCaseSelector, it also has a monthly limit of cases which can be sent to the QR team.
def select_case_for_legacy_quality_review
return if !legacy? || self.class.reached_monthly_limit_in_quality_reviews?
update(location: :quality_review) if bva_dispatch? && rand < QUALITY_REVIEW_SELECTION_PROBABILITY
end
The constants are periodically updated at the request of the Board to maintain the correct number of AMA and Legacy cases being sent to QR. There are automated tests in the quality_review_case_selector_spec.rb
file which verify the function of creating the QR tasks, as well as verifying that the random selection is selecting approximately the desired number of appeals.
- Github issue with history
- QR_SELECTION_PROBABILITY = 0.04
- Probability that a given case should be selected for Quality Review (p)
- We want the number of cases selected in a given month (X) to equal or exceed the target (x), as any excess selected cases will be ignored. The target during the pilot is 34 cases.
- We want the probability that X is less than x (P(X<x)) to be less than 0.000833, an event that would happen once per century.
- In December 2017, the Board wrote 5,549 decisions. Taking 1/4 of that figure, reflecting that Caseflow is piloted with 1/4 of the Board's judges, we have a number of trials (n) equal to 1,387.
- When n = 1,387, x = 34, and p = 0.04, we find that P(X<x) = 0.000464, which is less than our maximum probability 0.000833.
- This probability should be 0.032 when Caseflow expands to include all judges.
- When n = 5,549, x = 136, and p = 0.032, we find that P(X<x) = 0.00043, which is less than our maximum probability 0.000833.
- An easy way to avoid doing the math is to use this calculator: https://stattrek.com/online-calculator/binomial.aspx
- Home
- Acronyms and Glossary
- Caseflow products
- Caseflow Intake
- Caseflow Queue
- Appeals Consumer
- Caseflow Reader
- Caseflow eFolder
- Caseflow Hearings
- Caseflow Certification
- Caseflow APIs
- Appeal Status API
- Caseflow Dispatch
-
CSUM Roles
- System Admin
- VHA Team Management
- Active Record Queries Resource
- External Integrations
- Caseflow Demo
- Caseflow ProdTest
- Background
- Stuck Jobs
- VA Notify
- Caseflow-Team
- Frontend Best Practices
- Accessibility
- How-To
- Debugging Tips
- Adding a Feature Flag with FeatureToggle
- Editing AMA issues
- Editing a decision review
- Fixing task trees
- Investigating and diagnosing issues
- Data and Metric Request Workflow
- Exporting and Importing Appeals
- Explain page for Appeals
- Record associations and Foreign Keys
- Upgrading Ruby
- Stuck Appeals
- Testing Action Mailer Messages Locally
- Re-running Seed Files
- Rake Generator for Legacy Appeals
- Manually running Scheduled Jobs
- System Admin UI
- Caseflow Makefile
- Upgrading Postgresql from v11.7 to v14.8 Locally
- VACOLS VM Trigger Fix M1
- Using SlackService to Send a Job Alert
- Technical Talks