Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(scrape_pacer_free_opinions): apply task recap_document_into_opinions #4638

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

grossir
Copy link
Contributor

@grossir grossir commented Oct 31, 2024

Features:

  • add corpus_importer.tasks.recap_document_into_opinions into the scrape_pacer_free_opinions chain: this will make the recap documents be ingested in real time into the case law database
  • tweaked the task so it can be used by both the scraper_pacer_free_opinions and recap_into_opinions command

Refactors:

  • renamed task ingest_recap_document to recap_document_into_opinions
  • renamed task extract_recap_document to extract_recap_document_for_opinions
  • deleted duplicated function extract_recap_document in recap_into_opinions command file; delete unused imports and code

Copy link

sentry-io bot commented Oct 31, 2024

🔍 Existing Issues For Review

Your pull request is modifying functions with the following pre-existing issues:

📄 File: cl/corpus_importer/tasks.py

Function Unhandled Issue
ingest_recap_document HTTPStatusError: Server error '500 Internal Server Error' for url 'http://cl-doctor:5050/extract/recap/text/?strip... ...
Event Count: 3

Did you find this useful? React with a 👍 or 👎

@grossir grossir force-pushed the regular_ingestion_pacerfree_to_caselaw branch 2 times, most recently from 3d4ab38 to 79957bc Compare October 31, 2024 01:13
@grossir grossir requested a review from flooie October 31, 2024 01:17
…ions

Features:
- add corpus_importer.tasks.recap_document_into_opinions into the scrape_pacer_free_opinions chain: this will make the recap documents be ingested in real time into the case law database
- tweaked the task so it can be used by both the scraper_pacer_free_opinions and recap_into_opinions command

Refactors:
- renamed task `ingest_recap_document` to `recap_document_into_opinions`
- renamed task `extract_recap_document` to `extract_recap_document_for_opinions`
- deleted duplicated function `extract_recap_document` in recap_into_opinions command file; delete unused imports and code
@grossir grossir force-pushed the regular_ingestion_pacerfree_to_caselaw branch from 79957bc to 7ca17a0 Compare October 31, 2024 03:31
@@ -89,6 +49,8 @@ def import_opinions_from_recap(

# Manually select the replica db which has an addt'l index added to
# improve this query.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a one-time script, right? Let's make sure we remove that index when we're done. Can we make a sub-issue to remember to do that? We need to be really careful not to permanently add indexes we won't use in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants