Skip to content

Commit

Permalink
deploy: e3c736f
Browse files Browse the repository at this point in the history
  • Loading branch information
Kaihui-intel committed Apr 29, 2024
1 parent d8fe5b7 commit 4b527d3
Show file tree
Hide file tree
Showing 750 changed files with 800 additions and 866 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,40 @@
Module Contents
---------------

Classes
~~~~~~~

.. autoapisummary::

neural_compressor.torch.algorithms.weight_only.autoround.AutoRoundQuantizer



Functions
~~~~~~~~~

.. autoapisummary::

neural_compressor.torch.algorithms.weight_only.autoround.get_autoround_default_run_fn
neural_compressor.torch.algorithms.weight_only.autoround.autoround_quantize



.. py:class:: AutoRoundQuantizer(weight_config: dict = {}, enable_full_range: bool = False, batch_size: int = 8, amp: bool = True, device=None, lr_scheduler=None, use_quant_input: bool = True, enable_minmax_tuning: bool = True, lr: float = None, minmax_lr: float = None, low_gpu_mem_usage: bool = True, iters: int = 200, seqlen: int = 2048, n_samples: int = 512, sampler: str = 'rand', seed: int = 42, n_blocks: int = 1, gradient_accumulate_steps: int = 1, not_use_best_mse: bool = False, dynamic_max_gap: int = -1, scale_dtype='fp32')
The base quantizer for all algorithm quantizers.

The `Quantizer` unifies the interfaces across various quantization algorithms, including GPTQ, RTN, etc.
Given a float model, `Quantizer` apply the quantization algorithm to the model according to the `quant_config`.

To implement a new quantization algorithm,, inherit from `Quantizer` and implement the following methods:
- `prepare`: prepare a given model for convert.
- `convert`: convert a prepared model to a quantized model.
Note: `quantize` and `execute` are optional for new quantization algorithms.


.. py:function:: get_autoround_default_run_fn(model, tokenizer, dataset_name='NeelNanda/pile-10k', n_samples=512, seqlen=2048, seed=42, bs=8, dataset_split: str = 'train', dataloader=None)
Perform calibration for quantization.
Expand All @@ -34,51 +57,3 @@ Functions
:type n_samples: int


.. py:function:: autoround_quantize(model, weight_config: dict = {}, enable_full_range: bool = False, batch_size: int = 8, amp: bool = True, device=None, lr_scheduler=None, use_quant_input: bool = True, enable_minmax_tuning: bool = True, lr: float = None, minmax_lr: float = None, low_gpu_mem_usage: bool = True, iters: int = 200, seqlen: int = 2048, n_samples: int = 512, sampler: str = 'rand', seed: int = 42, n_blocks: int = 1, gradient_accumulate_steps: int = 1, not_use_best_mse: bool = False, dynamic_max_gap: int = -1, scale_dtype='fp16', run_fn=None, run_args=None)
The entry point of the autoround weight-only quantization.
Args:
model: The PyTorch model to be quantized.
weight_config (dict): Configuration for weight quantization (default is an empty dictionary).
weight_config={
'layer1':##layer_name
{
'data_type': 'int',
'bits': 4,
'group_size': 32,
'sym': False,
}
...
}
keys:
data_type (str): The data type to be used (default is "int").
bits (int): Number of bits for quantization (default is 4).
group_size (int): Size of the quantization group (default is 128).
sym (bool): Whether to use symmetric quantization. (default is None).
enable_full_range (bool): Whether to enable full range quantization (default is False).
batch_size (int): Batch size for training (default is 8).
amp (bool): Whether to use automatic mixed precision (default is True). Automatically detect and set.
device: The device to be used for tuning (default is None). Automatically detect and set.
lr_scheduler: The learning rate scheduler to be used.
use_quant_input (bool): Whether to use quantized input data (default is True).
enable_minmax_tuning (bool): Whether to enable min-max tuning (default is True).
lr (float): The learning rate (default is 0.005).
minmax_lr (float): The learning rate for min-max tuning (default is None).
low_gpu_mem_usage (bool): Whether to use low GPU memory (default is True).
iters (int): Number of iterations (default is 200).
seqlen (int): Length of the sequence.
n_samples (int): Number of samples (default is 512).
sampler (str): The sampling method (default is "rand").
seed (int): The random seed (default is 42).
n_blocks (int): Number of blocks (default is 1).
gradient_accumulate_steps (int): Number of gradient accumulation steps (default is 1).
not_use_best_mse (bool): Whether to use mean squared error (default is False).
dynamic_max_gap (int): The dynamic maximum gap (default is -1).
scale_dtype (str): The data type of quantization scale to be used (default is "float32"), different kernels
have different choices.
run_fn: a calibration function for calibrating the model. Defaults to None.
run_args: positional arguments for `run_fn`. Defaults to None.

:returns: The quantized model.


2 changes: 1 addition & 1 deletion latest/autoapi/block_mask/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703fdda50>
<jinja2.runtime.BlockReference object at 0x7f89189cae90>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ <h3>Functions<a class="headerlink" href="#functions" title="Permalink to this he
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703cf94e0>
<jinja2.runtime.BlockReference object at 0x7f89188c2d10>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
2 changes: 1 addition & 1 deletion latest/autoapi/neural_compressor/adaptor/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -217,7 +217,7 @@ <h2>Package Contents<a class="headerlink" href="#package-contents" title="Permal
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703dd20e0>
<jinja2.runtime.BlockReference object at 0x7f89190e6230>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
2 changes: 1 addition & 1 deletion latest/autoapi/neural_compressor/adaptor/keras/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ <h3>Classes<a class="headerlink" href="#classes" title="Permalink to this headin
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703dd2ec0>
<jinja2.runtime.BlockReference object at 0x7f89190e65c0>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703cf9c90>
<jinja2.runtime.BlockReference object at 0x7f89188c25f0>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703dd0070>
<jinja2.runtime.BlockReference object at 0x7f8918eb3340>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703dd21d0>
<jinja2.runtime.BlockReference object at 0x7f89187b26e0>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703921e40>
<jinja2.runtime.BlockReference object at 0x7f8918e58370>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703dd2860>
<jinja2.runtime.BlockReference object at 0x7f89187b3ee0>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703fdb2b0>
<jinja2.runtime.BlockReference object at 0x7f891884e3e0>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703fd9b70>
<jinja2.runtime.BlockReference object at 0x7f891884f5b0>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
2 changes: 1 addition & 1 deletion latest/autoapi/neural_compressor/adaptor/mxnet/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ <h3>Classes<a class="headerlink" href="#classes" title="Permalink to this headin
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0709f187c0>
<jinja2.runtime.BlockReference object at 0x7f891884e1a0>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ <h2>Submodules<a class="headerlink" href="#submodules" title="Permalink to this
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703dd18d0>
<jinja2.runtime.BlockReference object at 0x7f89188c2bf0>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -679,7 +679,7 @@ <h3>Functions<a class="headerlink" href="#functions" title="Permalink to this he
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703e77400>
<jinja2.runtime.BlockReference object at 0x7f89190e5210>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
2 changes: 1 addition & 1 deletion latest/autoapi/neural_compressor/adaptor/onnxrt/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ <h3>Classes<a class="headerlink" href="#classes" title="Permalink to this headin
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703ff3400>
<jinja2.runtime.BlockReference object at 0x7f8918761630>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ <h3>Classes<a class="headerlink" href="#classes" title="Permalink to this headin
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703eac370>
<jinja2.runtime.BlockReference object at 0x7f89187602e0>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,7 @@ <h3>Functions<a class="headerlink" href="#functions" title="Permalink to this he
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f070d60cf70>
<jinja2.runtime.BlockReference object at 0x7f891888f1c0>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ <h2>Submodules<a class="headerlink" href="#submodules" title="Permalink to this
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703dd2440>
<jinja2.runtime.BlockReference object at 0x7f8921c5a8c0>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ <h3>Classes<a class="headerlink" href="#classes" title="Permalink to this headin
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703cfbe20>
<jinja2.runtime.BlockReference object at 0x7f89219be9e0>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ <h3>Classes<a class="headerlink" href="#classes" title="Permalink to this headin
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703e77910>
<jinja2.runtime.BlockReference object at 0x7f89187617e0>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ <h3>Classes<a class="headerlink" href="#classes" title="Permalink to this headin
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703e77370>
<jinja2.runtime.BlockReference object at 0x7f891884fbe0>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ <h3>Classes<a class="headerlink" href="#classes" title="Permalink to this headin
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f070d60f9d0>
<jinja2.runtime.BlockReference object at 0x7f8918924f40>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ <h3>Classes<a class="headerlink" href="#classes" title="Permalink to this headin
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
<jinja2.runtime.BlockReference object at 0x7f0703eaf100>
<jinja2.runtime.BlockReference object at 0x7f89218afd30>
<p></p><div><a href='https://www.intel.com/content/www/us/en/privacy/intel-cookie-notice.html' data-cookie-notice='true'>Cookies</a> <a href='https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html'>| Privacy</a></div>


Expand Down
Loading

0 comments on commit 4b527d3

Please sign in to comment.