Remove metrics (#6983)

* Remove metrics subpackage * Update Makefile * Delete tests of metric module factories * Delete metric tests * Delete metric warning tests * Delete inspect_metric tests * Delete inspect_metric and list_metrics * Delete load_metric * Update import_main_class * Delete Metric * Delete MetricInfo * Update CI * Delete metrics-tests extras require and update CI * Update .gitignore * Update docs * Delete config.HF_METRICS_CACHE * Update setup keywords * Update increase_load_count * Update hf_github_url * Update cache docs * Delete metric card template * Delete metric_loading_script_dir test fixture * Update comments and docstrings * Delete config METRIC_INFO_FILENAME * Update main classes docs * Delete MetricModule * Update docstring * Delete metrics additional tests requirements
huggingface · Jun 28, 2024 · 70e7355 · 70e7355
1 parent dd63143
commit 70e7355
Show file tree

Hide file tree

Showing 118 changed files with 35 additions and 15,542 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -28,8 +28,8 @@ jobs:
           pip install .[quality]
       - name: Check quality
         run: |
-          ruff check tests src benchmarks metrics utils setup.py # linter
-          ruff format --check tests src benchmarks metrics utils setup.py # formatter
+          ruff check tests src benchmarks utils setup.py # linter
+          ruff format --check tests src benchmarks utils setup.py # formatter
 
   test:
     needs: check_code_quality
@@ -56,7 +56,7 @@ jobs:
       - name: Install uv
         run: pip install --upgrade uv
       - name: Install dependencies
-        run: uv pip install --system "datasets[tests,metrics-tests] @ ."
+        run: uv pip install --system "datasets[tests] @ ."
       - name: Install dependencies (latest versions)
         if: ${{ matrix.os == 'ubuntu-latest' }}
         run: uv pip install --system -r additional-tests-requirements.txt --no-deps

diff --git a/.gitignore b/.gitignore
@@ -42,13 +42,6 @@ venv.bak/
 .idea
 .vscode
 
-# keep only the empty datasets and metrics directory with it's __init__.py file
-/src/*/datasets/*
-!/src/*/datasets/__init__.py
-
-/src/*/metrics/*
-!/src/*/metrics/__init__.py
-
 # Vim
 .*.swp
 

diff --git a/Makefile b/Makefile
@@ -1,6 +1,6 @@
 .PHONY: quality style test
 
-check_dirs := tests src benchmarks metrics utils
+check_dirs := tests src benchmarks utils
 
 # Check that source code meets quality standards
 

diff --git a/additional-tests-requirements.txt b/additional-tests-requirements.txt
@@ -1,5 +1 @@
-unbabel-comet>=1.0.0
 git+https://github.com/pytorch/data.git
-git+https://github.com/google-research/bleurt.git
-git+https://github.com/ns-moosavi/coval.git
-git+https://github.com/hendrycks/math.git
diff --git a/docs/source/_redirects.yml b/docs/source/_redirects.yml
@@ -8,7 +8,6 @@ splits: loading#slice-splits
 processing: process
 faiss_and_ea: faiss_es
 features: about_dataset_features
-using_metrics: how_to_metrics
 exploring: access
 package_reference/logging_methods: package_reference/utilities
 # end of first_section
diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml
@@ -15,8 +15,6 @@
     title: Know your dataset
   - local: use_dataset
     title: Preprocess
-  - local: metrics
-    title: Evaluate predictions
   - local: create_dataset
     title: Create a dataset
   - local: upload_dataset
@@ -48,8 +46,6 @@
       title: Search index
     - local: cli
       title: CLI
-    - local: how_to_metrics
-      title: Metrics
     - local: troubleshoot
       title: Troubleshooting
     title: "General usage"
@@ -111,8 +107,6 @@
     title: Build and load
   - local: about_map_batch
     title: Batch mapping
-  - local: about_metrics
-    title: All about metrics
   title: "Conceptual guides"
 - sections:
   - local: package_reference/main_classes

diff --git a/docs/source/about_metrics.mdx b/docs/source/about_metrics.mdx
diff --git a/docs/source/cache.mdx b/docs/source/cache.mdx
@@ -24,13 +24,6 @@ When you load a dataset, you also have the option to change where the data is ca
 >>> dataset = load_dataset('LOADING_SCRIPT', cache_dir="PATH/TO/MY/CACHE/DIR")
 ```
 
-Similarly, you can change where a metric is cached with the `cache_dir` parameter:
-
-```py
->>> from datasets import load_metric
->>> metric = load_metric('glue', 'mrpc', cache_dir="MY/CACHE/DIRECTORY")
-```
-
 ## Download mode
 
 After you download a dataset, control how it is loaded by [`load_dataset`] with the `download_mode` parameter. By default, 🤗 Datasets will reuse a dataset if it exists. But if you need the original dataset without any processing functions applied, re-download the files as shown below:
@@ -77,19 +70,6 @@ If you want to reuse a dataset from scratch, try setting the `download_mode` par
 
 </Tip>
 
-You can also avoid caching your metric entirely, and keep it in CPU memory instead:
-
-```py
->>> from datasets import load_metric
->>> metric = load_metric('glue', 'mrpc', keep_in_memory=True)
-```
-
-<Tip warning={true}>
-
-Keeping the predictions in-memory is not possible in a distributed setting since the CPU memory spaces of the various processes are not shared.
-
-</Tip>
-
 <a id='load_dataset_enhancing_performance'></a>
 
 ## Improve performance

diff --git a/docs/source/how_to_metrics.mdx b/docs/source/how_to_metrics.mdx