feat(scrapers.update_from_text): new command #4520
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Helps solve: freelawproject/juriscraper#858 - New command to re-run Site.extract_from_text over downloaded opinions - Able to filter by Docket.court_id , OpinionCluster.date_filed, OpinionCluster.precedential_status - Updates tasks.update_from_document_text to return information for logging purposes - Updates test_opinion_scraper to get a Site.extract_from_text method
🔍 Existing Issues For ReviewYour pull request is modifying functions with the following pre-existing issues: 📄 File: cl/scrapers/tasks.py
Did you find this useful? React with a 👍 or 👎 |
flooie
reviewed
Oct 3, 2024
flooie
reviewed
Oct 18, 2024
flooie
reviewed
Oct 19, 2024
flooie
reviewed
Oct 19, 2024
flooie
reviewed
Oct 19, 2024
Comment on lines
127
to
163
| stats = {"Docket": 0, "OpinionCluster": 0, "Opinion": 0, "Citation": 0} | ||
|
|
||
| if options["opinion_ids"]: | ||
| opinions = Opinion.objects.filter(id__in=options["opinion_ids"]) | ||
| for op in opinions: | ||
| rerun_extract_from_text(op, juriscraper_module, stats) | ||
|
|
||
| logger.info("Modified objects counts: %s", stats) | ||
| return | ||
|
|
||
| if not (options["date_filed_gte"] and options["date_filed_lte"]): | ||
| raise ValueError( | ||
| "Both `date-filed-gte` and `date-filed-lte` arguments should have values" | ||
| ) | ||
|
|
||
| court_id = juriscraper_module.split(".")[-1].split("_")[0] | ||
| gte_date = datetime.strptime(options["date_filed_gte"], "%Y/%m/%d") | ||
| lte_date = datetime.strptime(options["date_filed_lte"], "%Y/%m/%d") | ||
| query = { | ||
| "docket__court_id": court_id, | ||
| "date_filed__gte": gte_date, | ||
| "date_filed__lte": lte_date, | ||
| } | ||
|
|
||
| if options["cluster_status"]: | ||
| query["precedential_status"] = options["cluster_status"] | ||
|
|
||
| qs = OpinionCluster.objects.filter(**query).prefetch_related( | ||
| "sub_opinions" | ||
| ) | ||
| for cluster in qs: | ||
| opinions = cluster.sub_opinions.all() | ||
| for op in opinions: | ||
| rerun_extract_from_text(op, juriscraper_module, stats) | ||
|
|
||
| logger.info("Modified objects counts: %s", stats) | ||
| self.stats = stats |
There was a problem hiding this comment.
maybe its just me but I prefer to have a clean handle method that doesnt contain large portions of code to run and passes that off. do you think we could move this out of the handle into its own method.
There was a problem hiding this comment.
I think a Command's handle should have the code to manipulate the input arguments, otherwise it would be just boilerplate taking up space. I actually tried to abstract it out, and the handle would look like this, which I don't think is good:
def handle(self, *args, **options):
super().handle(*args, **options)
some_other_function(options)
flooie
reviewed
Oct 19, 2024
flooie
reviewed
Oct 19, 2024
Comment on lines
157
to
160
| for cluster in qs: | ||
| opinions = cluster.sub_opinions.all() | ||
| for op in opinions: | ||
| rerun_extract_from_text(op, juriscraper_module, stats) |
There was a problem hiding this comment.
It's possible to have opinions that are merged with Harvard. If thats the case this could bring back empty plain_text and html fields. which will crash
There was a problem hiding this comment.
flooie
reviewed
Oct 19, 2024
flooie
reviewed
Oct 19, 2024
flooie
reviewed
Oct 19, 2024
flooie
reviewed
Oct 20, 2024
grossir
force-pushed
the
scrapers_update_from_text_command
branch
3 times, most recently
from
789ae66
to
2838202
Compare
- validate citation objects from `Site.extract_from_text`. Add tests for this - abstract --courts required argument for scrapers into ScraperCommand class also, made it more flexible - refactor cl_scrape_opinions; cl_scrape_oral_arguments to account for this - delete cl.scrapers.utils.extract_recap_documents which was generating a circular import. This function was not used anywhere
grossir
force-pushed
the
scrapers_update_from_text_command
branch
from
2838202
to
6a1fadb
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Helps solve: freelawproject/juriscraper#858