From b62b320f1346f14d43529098d8ac9b522709d389 Mon Sep 17 00:00:00 2001 From: tfx-team Date: Sun, 10 Dec 2023 22:46:11 -0800 Subject: [PATCH] TFX LLM tutorial PiperOrigin-RevId: 589713199 --- docs/tutorials/_toc.yaml | 2 + .../tfx/gpt2_finetuning_and_conversion.ipynb | 1527 +++++++++++++++++ .../perplexity.png | Bin 0 -> 64896 bytes 3 files changed, 1529 insertions(+) create mode 100644 docs/tutorials/tfx/gpt2_finetuning_and_conversion.ipynb create mode 100644 docs/tutorials/tfx/images/gpt2_fine_tuning_and_conversion/perplexity.png diff --git a/docs/tutorials/_toc.yaml b/docs/tutorials/_toc.yaml index 184235c3888..91df2347a74 100644 --- a/docs/tutorials/_toc.yaml +++ b/docs/tutorials/_toc.yaml @@ -29,6 +29,8 @@ toc: path: /tfx/tutorials/tfx/cloud-ai-platform-pipelines - heading: "TFX: Advanced tutorials" +- title: "LLM finetuning and conversion" + path: /tfx/tutorials/tfx/gpt2_finetuning_and_conversion - title: "Custom component tutorial" path: /tfx/tutorials/tfx/python_function_component - title: "Recommenders with TFX" diff --git a/docs/tutorials/tfx/gpt2_finetuning_and_conversion.ipynb b/docs/tutorials/tfx/gpt2_finetuning_and_conversion.ipynb new file mode 100644 index 00000000000..35f8af7b4e8 --- /dev/null +++ b/docs/tutorials/tfx/gpt2_finetuning_and_conversion.ipynb @@ -0,0 +1,1527 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [], + "collapsed_sections": [ + "iwgnKVaUuozP" + ], + "gpuType": "T4", + "toc_visible": true + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + }, + "language_info": { + "name": "python" + }, + "accelerator": "GPU" + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "YtDTm6wbIbpy" + }, + "source": [ + "##### Copyright 2024 The TensorFlow Authors." + ] + }, + { + "cell_type": "markdown", + "source": [ + "# Licensed under the Apache License, Version 2.0 (the \"License\");" + ], + "metadata": { + "id": "iwgnKVaUuozP" + } + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "kBFkQLk1In7I" + }, + "outputs": [], + "source": [ + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "uf3QpfdiIl7O" + }, + "source": [ + "# TFX Pipeline for Fine-Tuning a Large Language Model (LLM)\n", + "\n", + "\n", + "This codelab demonstrates how to leverage the power of Keras 3, KerasNLP and TFX pipelines to fine-tune a pre-trained GPT-2 model on the IMDb movie reviews dataset. The dataset that is used in this demo is [IMDB Reviews dataset](https://www.tensorflow.org/datasets/catalog/imdb_reviews).\n", + "\n", + "Note: We recommend running this tutorial in a Colab notebook, with no setup required! Just click \"Run in Google Colab\".\n", + "\n", + "\u003cdiv class=\"devsite-table-wrapper\"\u003e\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n", + "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://www.tensorflow.org/tfx/tutorials/tfx/gpt2_finetuning_and_conversion\"\u003e\n", + "\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\"/\u003eView on TensorFlow.org\u003c/a\u003e\u003c/td\u003e\n", + "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tfx/blob/master/docs/tutorials/tfx/gpt2_finetuning_and_conversion.ipynb\"\u003e\n", + "\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\"\u003eRun in Google Colab\u003c/a\u003e\u003c/td\u003e\n", + "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://github.com/tensorflow/tfx/tree/master/docs/tutorials/tfx/gpt2_finetuning_and_conversion.ipynb\"\u003e\n", + "\u003cimg width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\"\u003eView source on GitHub\u003c/a\u003e\u003c/td\u003e\n", + "\u003ctd\u003e\u003ca href=\"https://storage.googleapis.com/tensorflow_docs/tfx/docs/tutorials/tfx/gpt2_finetuning_and_conversion.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\u003c/td\u003e\n", + "\u003c/table\u003e\u003c/div\u003e\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HU9YYythm0dx" + }, + "source": [ + "### Why is this pipeline useful?\n", + "\n", + "TFX pipelines provide a powerful and structured approach to building and managing machine learning workflows, particularly those involving large language models. They offer significant advantages over traditional Python code, including:\n", + "\n", + "1. Enhanced Reproducibility: TFX pipelines ensure consistent results by capturing all steps and dependencies, eliminating the inconsistencies often associated with manual workflows.\n", + "\n", + "2. Scalability and Modularity: TFX allows for breaking down complex workflows into manageable, reusable components, promoting code organization.\n", + "\n", + "3. Streamlined Fine-Tuning and Conversion: The pipeline structure streamlines the fine-tuning and conversion processes of large language models, significantly reducing manual effort and time.\n", + "\n", + "4. Comprehensive Lineage Tracking: Through metadata tracking, TFX pipelines provide a clear understanding of data and model provenance, making debugging, auditing, and performance analysis much easier and more efficient.\n", + "\n", + "By leveraging the benefits of TFX pipelines, organizations can effectively manage the complexity of large language model development and deployment, achieving greater efficiency and control over their machine learning processes.\n", + "\n", + "### Note\n", + "*GPT-2 is used here only to demonstrate the end-to-end process; the techniques and tooling introduced in this codelab are potentially transferrable to other generative language models such as Google T5.*" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2WgJ8Z8gJB0s" + }, + "source": [ + "## Before You Begin\n", + "\n", + "Colab offers different kinds of runtimes. Make sure to go to **Runtime -\u003e Change runtime type** and choose the GPU Hardware Accelerator runtime since you will finetune the GPT-2 model.\n", + "\n", + "**This tutorial's interactive pipeline is designed to function seamlessly with free Colab GPUs. However, for users opting to run the pipeline using the LocalDagRunner orchestrator (code provided at the end of this tutorial), a more substantial amount of GPU memory is required. Therefore, Colab Pro or a local machine equipped with a higher-capacity GPU is recommended for this approach.**" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-sj3HvNcJEgC" + }, + "source": [ + "## Set Up\n", + "\n", + "We first install required python packages." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "73c9sPckJFSi" + }, + "source": [ + "### Upgrade Pip\n", + "To avoid upgrading Pip in a system when running locally, check to make sure that we are running in Colab. Local systems can of course be upgraded separately." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "45pIxa6afWOf", + "tags": [] + }, + "outputs": [], + "source": [ + "try:\n", + " import colab\n", + " !pip install --upgrade pip\n", + "\n", + "except:\n", + " pass" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yIf40NdqJLAH" + }, + "source": [ + "### Install TFX, Keras 3, KerasNLP and required Libraries" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "A6mBN4dzfct7", + "tags": [] + }, + "outputs": [], + "source": [ + "!pip install -q tfx tensorflow-text more_itertools tensorflow_datasets\n", + "!pip install -q --upgrade keras-nlp\n", + "!pip install -q --upgrade keras" + ] + }, + { + "cell_type": "markdown", + "source": [ + "*Note: pip's dependency resolver errors can be ignored. The required packages for this tutorial works as expected.*" + ], + "metadata": { + "id": "KnyILJ-k3NAy" + } + }, + { + "cell_type": "markdown", + "metadata": { + "id": "V0tnFDm6JRq_", + "tags": [] + }, + "source": [ + "### Did you restart the runtime?\n", + "\n", + "If you are using Google Colab, the first time that you run the cell above, you must restart the runtime by clicking above \"RESTART SESSION\" button or using `\"Runtime \u003e Restart session\"` menu. This is because of the way that Colab loads packages.\n", + "\n", + "Let's check the TensorFlow, Keras, Keras-nlp and TFX library versions." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Hf5FbRzcfpMg", + "tags": [] + }, + "outputs": [], + "source": [ + "import os\n", + "os.environ[\"KERAS_BACKEND\"] = \"tensorflow\"\n", + "\n", + "import tensorflow as tf\n", + "print('TensorFlow version: {}'.format(tf.__version__))\n", + "from tfx import v1 as tfx\n", + "print('TFX version: {}'.format(tfx.__version__))\n", + "import keras\n", + "print('Keras version: {}'.format(keras.__version__))\n", + "import keras_nlp\n", + "print('Keras NLP version: {}'.format(keras_nlp.__version__))\n", + "\n", + "keras.mixed_precision.set_global_policy(\"mixed_float16\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ng1a9cCAtepl" + }, + "source": [ + "### Using TFX Interactive Context" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "k7ikXCc7v7Rh" + }, + "source": [ + "An interactive context is used to provide global context when running a TFX pipeline in a notebook without using a runner or orchestrator such as Apache Airflow or Kubeflow. This style of development is only useful when developing the code for a pipeline, and cannot currently be used to deploy a working pipeline to production." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "TEge2nYDfwaM", + "tags": [] + }, + "outputs": [], + "source": [ + "from tfx.orchestration.experimental.interactive.interactive_context import InteractiveContext\n", + "context = InteractiveContext()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "GF6Kk3MLxxCC" + }, + "source": [ + "## Pipeline Overview\n", + "\n", + "Below are the components that this pipeline follows.\n", + "\n", + "* Custom Artifacts are artifacts that we have created for this pipeline. **Artifacts** are data that is produced by a component or consumed by a component. Artifacts are stored in a system for managing the storage and versioning of artifacts called MLMD.\n", + "\n", + "* **Components** are defined as the implementation of an ML task that you can use as a step in your pipeline\n", + "* Aside from artifacts, **Parameters** are passed into the components to specify an argument.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BIBO-ueGVVHa" + }, + "source": [ + "## ExampleGen\n", + "We create a custom ExampleGen component which we use to load a TensorFlow Datasets (TFDS) dataset. This uses a custom executor in a FileBasedExampleGen.\n", + "\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "pgvIaoAmXFVp", + "tags": [] + }, + "outputs": [], + "source": [ + "from typing import Any, Dict, List, Text\n", + "import tensorflow_datasets as tfds\n", + "import apache_beam as beam\n", + "import json\n", + "from tfx.components.example_gen.base_example_gen_executor import BaseExampleGenExecutor\n", + "from tfx.components.example_gen.component import FileBasedExampleGen\n", + "from tfx.components.example_gen import utils\n", + "from tfx.dsl.components.base import executor_spec\n", + "import os\n", + "import pprint\n", + "pp = pprint.PrettyPrinter()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Cjd9Z6SpVRCE", + "tags": [] + }, + "outputs": [], + "source": [ + "@beam.ptransform_fn\n", + "@beam.typehints.with_input_types(beam.Pipeline)\n", + "@beam.typehints.with_output_types(tf.train.Example)\n", + "def _TFDatasetToExample(\n", + " pipeline: beam.Pipeline,\n", + " exec_properties: Dict[str, Any],\n", + " split_pattern: str\n", + " ) -\u003e beam.pvalue.PCollection:\n", + " \"\"\"Read a TensorFlow Dataset and create tf.Examples\"\"\"\n", + " custom_config = json.loads(exec_properties['custom_config'])\n", + " dataset_name = custom_config['dataset']\n", + " split_name = custom_config['split']\n", + "\n", + " builder = tfds.builder(dataset_name)\n", + " builder.download_and_prepare()\n", + "\n", + " return (pipeline\n", + " | 'MakeExamples' \u003e\u003e tfds.beam.ReadFromTFDS(builder, split=split_name)\n", + " | 'AsNumpy' \u003e\u003e beam.Map(tfds.as_numpy)\n", + " | 'ToDict' \u003e\u003e beam.Map(dict)\n", + " | 'ToTFExample' \u003e\u003e beam.Map(utils.dict_to_example)\n", + " )\n", + "\n", + "class TFDSExecutor(BaseExampleGenExecutor):\n", + " def GetInputSourceToExamplePTransform(self) -\u003e beam.PTransform:\n", + " \"\"\"Returns PTransform for TF Dataset to TF examples.\"\"\"\n", + " return _TFDatasetToExample" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2D159hAzJgK2" + }, + "source": [ + "For this demonstration, we're using a subset of the IMDb reviews dataset, representing 20% of the total data. This allows for a more manageable training process. You can modify the \"custom_config\" settings to experiment with larger amounts of data, up to the full dataset, depending on your computational resources." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "nNDu1ECBXuvI", + "tags": [] + }, + "outputs": [], + "source": [ + "example_gen = FileBasedExampleGen(\n", + " input_base='dummy',\n", + " custom_config={'dataset':'imdb_reviews', 'split':'train[:20%]'},\n", + " custom_executor_spec=executor_spec.BeamExecutorSpec(TFDSExecutor))\n", + "context.run(example_gen, enable_cache=False)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "74JGpvIgJgK2" + }, + "source": [ + "We've developed a handy utility for examining datasets composed of TFExamples. When used with the reviews dataset, this tool returns a clear dictionary containing both the text and the corresponding label." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "GA8VMXKogXxB", + "tags": [] + }, + "outputs": [], + "source": [ + "def inspect_examples(component,\n", + " channel_name='examples',\n", + " split_name='train',\n", + " num_examples=1):\n", + " # Get the URI of the output artifact, which is a directory\n", + " full_split_name = 'Split-{}'.format(split_name)\n", + " print('channel_name: {}, split_name: {} (\\\"{}\\\"), num_examples: {}\\n'.format(\n", + " channel_name, split_name, full_split_name, num_examples))\n", + " train_uri = os.path.join(\n", + " component.outputs[channel_name].get()[0].uri, full_split_name)\n", + " print('train_uri: {}'.format(train_uri))\n", + "\n", + " # Get the list of files in this directory (all compressed TFRecord files)\n", + " tfrecord_filenames = [os.path.join(train_uri, name)\n", + " for name in os.listdir(train_uri)]\n", + "\n", + " # Create a `TFRecordDataset` to read these files\n", + " dataset = tf.data.TFRecordDataset(tfrecord_filenames, compression_type=\"GZIP\")\n", + "\n", + " # Iterate over the records and print them\n", + " print()\n", + " for tfrecord in dataset.take(num_examples):\n", + " serialized_example = tfrecord.numpy()\n", + " example = tf.train.Example()\n", + " example.ParseFromString(serialized_example)\n", + " pp.pprint(example)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "rcUvtz5egaIy", + "tags": [] + }, + "outputs": [], + "source": [ + "inspect_examples(example_gen, num_examples=1, split_name='eval')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gVmx7JHK8RkO" + }, + "source": [ + "## StatisticsGen\n", + "\n", + "`StatisticsGen` component computes statistics over your dataset for data analysis, such as the number of examples, the number of features, and the data types of the features. It uses the [TensorFlow Data Validation](https://www.tensorflow.org/tfx/data_validation/get_started) library. `StatisticsGen` takes as input the dataset we just ingested using `ExampleGen`.\n", + "\n", + "*Note that the statistics generator is appropriate for tabular data, and therefore, text dataset for this LLM tutorial may not be the optimal dataset for the analysis with statistics generator.*" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "TzeNGNEnyq_d", + "tags": [] + }, + "outputs": [], + "source": [ + "from tfx.components import StatisticsGen" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "xWWl7LeRKsXA", + "tags": [] + }, + "outputs": [], + "source": [ + "statistics_gen = tfx.components.StatisticsGen(\n", + " examples=example_gen.outputs['examples'], exclude_splits=['eval']\n", + ")\n", + "context.run(statistics_gen, enable_cache=False)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "LnWKjMyIVVB7" + }, + "outputs": [], + "source": [ + "context.show(statistics_gen.outputs['statistics'])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "oqXFJyoO9O8-" + }, + "source": [ + "## SchemaGen\n", + "\n", + "The `SchemaGen` component generates a schema based on your data statistics. (A schema defines the expected bounds, types, and properties of the features in your dataset.) It also uses the [TensorFlow Data Validation](https://www.tensorflow.org/tfx/data_validation/get_started) library.\n", + "\n", + "Note: The generated schema is best-effort and only tries to infer basic properties of the data. It is expected that you review and modify it as needed.\n", + "\n", + "`SchemaGen` will take as input the statistics that we generated with `StatisticsGen`, looking at the training split by default.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "PpPFaV6tX5wQ", + "tags": [] + }, + "outputs": [], + "source": [ + "schema_gen = tfx.components.SchemaGen(\n", + " statistics=statistics_gen.outputs['statistics'],\n", + " infer_feature_shape=False,\n", + " exclude_splits=['eval'],\n", + ")\n", + "context.run(schema_gen, enable_cache=False)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "H6DNNUi3YAmo", + "tags": [] + }, + "outputs": [], + "source": [ + "context.show(schema_gen.outputs['schema'])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "GDdpADUb9VJR" + }, + "source": [ + "## ExampleValidator\n", + "\n", + "The `ExampleValidator` component detects anomalies in your data, based on the expectations defined by the schema. It also uses the [TensorFlow Data Validation](https://www.tensorflow.org/tfx/data_validation/get_started) library.\n", + "\n", + "`ExampleValidator` will take as input the statistics from `StatisticsGen`, and the schema from `SchemaGen`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "S_F5pLZ7YdZg" + }, + "outputs": [], + "source": [ + "example_validator = tfx.components.ExampleValidator(\n", + " statistics=statistics_gen.outputs['statistics'],\n", + " schema=schema_gen.outputs['schema'],\n", + " exclude_splits=['eval'],\n", + ")\n", + "context.run(example_validator, enable_cache=False)" + ] + }, + { + "cell_type": "markdown", + "source": [ + "After `ExampleValidator` finishes running, we can visualize the anomalies as a table." + ], + "metadata": { + "id": "DgiXSTRawolF" + } + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "3eAHpc2UYfk_" + }, + "outputs": [], + "source": [ + "context.show(example_validator.outputs['anomalies'])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7H6fecGTiFmN" + }, + "source": [ + "## Transform\n", + "\n", + "For a structured and repeatable design of a TFX pipeline we will need a scalable approach to feature engineering. The `Transform` component performs feature engineering for both training and serving. It uses the [TensorFlow Transform](https://www.tensorflow.org/tfx/transform/get_started) library.\n", + "\n", + "\n", + "The Transform component uses a module file to supply user code for the feature engineering what we want to do, so our first step is to create that module file. We will only be working with the summary field.\n", + "\n", + "**Note:**\n", + "*The %%writefile {_movies_transform_module_file} cell magic below creates and writes the contents of that cell to a file on the notebook server where this notebook is running (for example, the Colab VM). When doing this outside of a notebook you would just create a Python file.*" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "22TBUtG9ME9N" + }, + "outputs": [], + "source": [ + "import os\n", + "if not os.path.exists(\"modules\"):\n", + " os.mkdir(\"modules\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "teaCGLgfnjw_" + }, + "outputs": [], + "source": [ + "_transform_module_file = 'modules/_transform_module.py'" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "rN6nRx3KnkpM" + }, + "outputs": [], + "source": [ + "%%writefile {_transform_module_file}\n", + "\n", + "import tensorflow as tf\n", + "\n", + "def _fill_in_missing(x, default_value):\n", + " \"\"\"Replace missing values in a SparseTensor.\n", + "\n", + " Fills in missing values of `x` with the default_value.\n", + "\n", + " Args:\n", + " x: A `SparseTensor` of rank 2. Its dense shape should have size at most 1\n", + " in the second dimension.\n", + " default_value: the value with which to replace the missing values.\n", + "\n", + " Returns:\n", + " A rank 1 tensor where missing values of `x` have been filled in.\n", + " \"\"\"\n", + " if not isinstance(x, tf.sparse.SparseTensor):\n", + " return x\n", + " return tf.squeeze(\n", + " tf.sparse.to_dense(\n", + " tf.SparseTensor(x.indices, x.values, [x.dense_shape[0], 1]),\n", + " default_value),\n", + " axis=1)\n", + "\n", + "def preprocessing_fn(inputs):\n", + " outputs = {}\n", + " # outputs[\"summary\"] = _fill_in_missing(inputs[\"summary\"],\"\")\n", + " outputs[\"summary\"] = _fill_in_missing(inputs[\"text\"],\"\")\n", + " return outputs" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "v-f5NaLTiFmO" + }, + "outputs": [], + "source": [ + "preprocessor = tfx.components.Transform(\n", + " examples=example_gen.outputs['examples'],\n", + " schema=schema_gen.outputs['schema'],\n", + " module_file=os.path.abspath(_transform_module_file))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "MkjIuwHeiFmO" + }, + "outputs": [], + "source": [ + "context.run(preprocessor, enable_cache=False)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OH8OkaCwJgLF" + }, + "source": [ + "Let's take a look at some of the transformed examples and check that they are indeed processed as intended." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "bt70Z16zJHy7" + }, + "outputs": [], + "source": [ + "def pprint_examples(artifact, n_examples=2):\n", + " print(\"artifact:\", artifact, \"\\n\")\n", + " uri = os.path.join(artifact.uri, \"Split-eval\")\n", + " print(\"uri:\", uri, \"\\n\")\n", + " tfrecord_filenames = [os.path.join(uri, name) for name in os.listdir(uri)]\n", + " print(\"tfrecord_filenames:\", tfrecord_filenames, \"\\n\")\n", + " dataset = tf.data.TFRecordDataset(tfrecord_filenames, compression_type=\"GZIP\")\n", + " for tfrecord in dataset.take(n_examples):\n", + " serialized_example = tfrecord.numpy()\n", + " example = tf.train.Example.FromString(serialized_example)\n", + " pp.pprint(example)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Tg4I-TvXJIuO" + }, + "outputs": [], + "source": [ + "pprint_examples(preprocessor.outputs['transformed_examples'].get()[0])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mJll-vDn_eJP" + }, + "source": [ + "## Trainer\n", + "\n", + "Trainer component trains an ML model, and it requires a model definition code from users.\n", + "\n", + "The `run_fn` function in TFX's Trainer component is the entry point for training a machine learning model. It is a user-supplied function that takes in a set of arguments and returns a model artifact.\n", + "\n", + "The `run_fn` function is responsible for:\n", + "\n", + "* Building the machine learning model.\n", + "* Training the model on the training data.\n", + "* Saving the trained model to the serving model directory.\n", + "\n", + "\n", + "### Write model training code\n", + "We will create a very simple fine-tuned model, with the preprocessing GPT-2 model. First, we need to create a module that contains the `run_fn` function for TFX Trainer because TFX Trainer expects the `run_fn` function to be defined in a module. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "OQPtqKG5pmpn" + }, + "outputs": [], + "source": [ + "model_file = \"modules/model.py\"\n", + "model_fn = \"modules.model.run_fn\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6drMNHJMAk7g" + }, + "source": [ + "Now, we write the run_fn function:\n", + "\n", + "This run_fn function first gets the training data from the `fn_args.examples` argument. It then gets the schema of the training data from the `fn_args.schema` argument. Next, it loads finetuned GPT-2 model along with its preprocessor. The model is then trained on the training data using the model.train() method.\n", + "Finally, the trained model weights are saved to the `fn_args.serving_model_dir` argument.\n", + "\n", + "\n", + "Now, we are going to work with Keras NLP's GPT-2 Model! You can learn about the full GPT-2 model implementation in KerasNLP on [GitHub](https://github.com/keras-team/keras-nlp/tree/r0.5/keras_nlp/models/gpt2) or can read and interactively test the model on [Google IO2023 colab notebook](https://colab.research.google.com/github/tensorflow/codelabs/blob/main/KerasNLP/io2023_workshop.ipynb#scrollTo=81EZQ0D1R8LL ).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "B-ME_d8i2sTB" + }, + "outputs": [], + "source": [ + "import keras_nlp\n", + "import keras\n", + "import tensorflow as tf" + ] + }, + { + "cell_type": "markdown", + "source": [ + "*Note: To accommodate the limited resources of a free Colab GPU, we've adjusted the GPT-2 model's `sequence_length` parameter to `128` from its default `256`. This optimization enables efficient model training on the T4 GPU, facilitating faster fine-tuning while adhering to resource constraints.*" + ], + "metadata": { + "id": "NnvkSqd6AB0q" + } + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "N9yjLDqHoFb-" + }, + "outputs": [], + "source": [ + "%%writefile {model_file}\n", + "\n", + "import os\n", + "import time\n", + "from absl import logging\n", + "import keras_nlp\n", + "import more_itertools\n", + "import pandas as pd\n", + "import tensorflow as tf\n", + "import keras\n", + "import tfx\n", + "import tfx.components.trainer.fn_args_utils\n", + "import gc\n", + "\n", + "\n", + "_EPOCH = 1\n", + "_BATCH_SIZE = 20\n", + "_INITIAL_LEARNING_RATE = 5e-5\n", + "_END_LEARNING_RATE = 0.0\n", + "_SEQUENCE_LENGTH = 128 # default value is 256\n", + "\n", + "def _input_fn(file_pattern: str) -\u003e list:\n", + " \"\"\"Retrieves training data and returns a list of articles for training.\n", + "\n", + " For each row in the TFRecordDataset, generated in the previous ExampleGen\n", + " component, create a new tf.train.Example object and parse the TFRecord into\n", + " the example object. Articles, which are initially in bytes objects, are\n", + " decoded into a string.\n", + "\n", + " Args:\n", + " file_pattern: Path to the TFRecord file of the training dataset.\n", + "\n", + " Returns:\n", + " A list of training articles.\n", + "\n", + " Raises:\n", + " FileNotFoundError: If TFRecord dataset is not found in the file_pattern\n", + " directory.\n", + " \"\"\"\n", + "\n", + " if os.path.basename(file_pattern) == '*':\n", + " file_loc = os.path.dirname(file_pattern)\n", + "\n", + " else:\n", + " raise FileNotFoundError(\n", + " f\"There is no file in the current directory: '{file_pattern}.\"\n", + " )\n", + "\n", + " file_paths = [os.path.join(file_loc, name) for name in os.listdir(file_loc)]\n", + " train_articles = []\n", + " parsed_dataset = tf.data.TFRecordDataset(file_paths, compression_type=\"GZIP\")\n", + " for raw_record in parsed_dataset:\n", + " example = tf.train.Example()\n", + " example.ParseFromString(raw_record.numpy())\n", + " train_articles.append(\n", + " example.features.feature[\"summary\"].bytes_list.value[0].decode('utf-8')\n", + " )\n", + " return train_articles\n", + "\n", + "def run_fn(fn_args: tfx.components.trainer.fn_args_utils.FnArgs) -\u003e None:\n", + " \"\"\"Trains the model and outputs the trained model to a the desired location given by FnArgs.\n", + "\n", + " Args:\n", + " FnArgs : Args to pass to user defined training/tuning function(s)\n", + " \"\"\"\n", + "\n", + " train_articles = pd.Series(_input_fn(\n", + " fn_args.train_files[0],\n", + " ))\n", + " tf_train_ds = tf.data.Dataset.from_tensor_slices(train_articles)\n", + "\n", + " gpt2_preprocessor = keras_nlp.models.GPT2CausalLMPreprocessor.from_preset(\n", + " 'gpt2_base_en',\n", + " sequence_length=_SEQUENCE_LENGTH,\n", + " add_end_token=True,\n", + " )\n", + " gpt2_lm = keras_nlp.models.GPT2CausalLM.from_preset(\n", + " 'gpt2_base_en', preprocessor=gpt2_preprocessor\n", + " )\n", + "\n", + " processed_ds = (\n", + " tf_train_ds\n", + " .batch(_BATCH_SIZE)\n", + " .cache()\n", + " .prefetch(tf.data.AUTOTUNE)\n", + " )\n", + "\n", + " gpt2_lm.include_preprocessing = False\n", + "\n", + " lr = tf.keras.optimizers.schedules.PolynomialDecay(\n", + " 5e-5,\n", + " decay_steps=processed_ds.cardinality() * _EPOCH,\n", + " end_learning_rate=0.0,\n", + " )\n", + " loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n", + "\n", + " gpt2_lm.compile(\n", + " optimizer=keras.optimizers.Adam(lr),\n", + " loss=loss,\n", + " weighted_metrics=['accuracy'],\n", + " )\n", + "\n", + " gpt2_lm.fit(processed_ds, epochs=_EPOCH)\n", + " if os.path.exists(fn_args.serving_model_dir):\n", + " os.rmdir(fn_args.serving_model_dir)\n", + " os.mkdir(fn_args.serving_model_dir)\n", + " gpt2_lm.save_weights(\n", + " filepath=os.path.join(fn_args.serving_model_dir, \"model_weights.weights.h5\")\n", + " )\n", + " del gpt2_lm, gpt2_preprocessor, processed_ds, tf_train_ds\n", + " gc.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "bnbMFKqc5gfK" + }, + "outputs": [], + "source": [ + "trainer = tfx.components.Trainer(\n", + " run_fn=model_fn,\n", + " examples=preprocessor.outputs['transformed_examples'],\n", + " train_args=tfx.proto.TrainArgs(splits=['train']),\n", + " eval_args=tfx.proto.EvalArgs(splits=['train']),\n", + " schema=schema_gen.outputs['schema'],\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "COCqeu-8CyHN" + }, + "outputs": [], + "source": [ + "context.run(trainer, enable_cache=False)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "btljwhMwWeQ9" + }, + "source": [ + "## Inference and Evaluation\n", + "\n", + "With our model fine-tuned, let's evaluate its performance by generating inferences. To capture and preserve these results, we'll create an EvaluationMetric artifact.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "S79afpeeVkwc" + }, + "outputs": [], + "source": [ + "from tfx.types import artifact\n", + "from tfx import types\n", + "\n", + "Property = artifact.Property\n", + "PropertyType = artifact.PropertyType\n", + "\n", + "DURATION_PROPERTY = Property(type=PropertyType.FLOAT)\n", + "EVAL_OUTPUT_PROPERTY = Property(type=PropertyType.STRING)\n", + "\n", + "class EvaluationMetric(types.Artifact):\n", + " \"\"\"Artifact that contains metrics for a model.\n", + "\n", + " * Properties:\n", + "\n", + " - 'model_prediction_time' : time it took for the model to make predictions\n", + " based on the input text.\n", + " - 'model_evaluation_output_path' : saves the path to the CSV file that\n", + " contains the model's prediction based on the testing inputs.\n", + " \"\"\"\n", + " TYPE_NAME = 'Evaluation_Metric'\n", + " PROPERTIES = {\n", + " 'model_prediction_time': DURATION_PROPERTY,\n", + " 'model_evaluation_output_path': EVAL_OUTPUT_PROPERTY,\n", + " }" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "GQ3Wq2Ylb6JF" + }, + "source": [ + "These helper functions contribute to the evaluation of a language model (LLM) by providing tools for calculating perplexity, a key metric reflecting the model's ability to predict the next word in a sequence, and by facilitating the extraction, preparation, and processing of evaluation data. The `input_fn` function retrieves training data from a specified TFRecord file, while the `trim_sentence` function ensures consistency by limiting sentence length. A lower perplexity score indicates higher prediction confidence and generally better model performance, making these functions essential for comprehensive evaluation within the LLM pipeline.\n", + "\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "tkXaZlsg38jI" + }, + "outputs": [], + "source": [ + "\"\"\"This is an evaluation component for the LLM pipeline takes in a\n", + "standard trainer artifact and outputs a custom evaluation artifact.\n", + "It displays the evaluation output in the colab notebook.\n", + "\"\"\"\n", + "import os\n", + "import time\n", + "import keras_nlp\n", + "import numpy as np\n", + "import pandas as pd\n", + "import tensorflow as tf\n", + "import tfx.v1 as tfx\n", + "\n", + "def input_fn(file_pattern: str) -\u003e list:\n", + " \"\"\"Retrieves training data and returns a list of articles for training.\n", + "\n", + " Args:\n", + " file_pattern: Path to the TFRecord file of the training dataset.\n", + "\n", + " Returns:\n", + " A list of test articles\n", + "\n", + " Raises:\n", + " FileNotFoundError: If the file path does not exist.\n", + " \"\"\"\n", + " if os.path.exists(file_pattern):\n", + " file_paths = [os.path.join(file_pattern, name) for name in os.listdir(file_pattern)]\n", + " test_articles = []\n", + " parsed_dataset = tf.data.TFRecordDataset(file_paths, compression_type=\"GZIP\")\n", + " for raw_record in parsed_dataset:\n", + " example = tf.train.Example()\n", + " example.ParseFromString(raw_record.numpy())\n", + " test_articles.append(\n", + " example.features.feature[\"summary\"].bytes_list.value[0].decode('utf-8')\n", + " )\n", + " return test_articles\n", + " else:\n", + " raise FileNotFoundError(f'File path \"{file_pattern}\" does not exist.')\n", + "\n", + "def trim_sentence(sentence: str, max_words: int = 20):\n", + " \"\"\"Trims the sentence to include up to the given number of words.\n", + "\n", + " Args:\n", + " sentence: The sentence to trim.\n", + " max_words: The maximum number of words to include in the trimmed sentence.\n", + "\n", + " Returns:\n", + " The trimmed sentence.\n", + " \"\"\"\n", + " words = sentence.split(' ')\n", + " if len(words) \u003c= max_words:\n", + " return sentence\n", + " return ' '.join(words[:max_words])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ypRrAQMpfEFd" + }, + "source": [ + "![perplexity.png](images/gpt2_fine_tuning_and_conversion/perplexity.png)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yo5fvOa9GmzL" + }, + "source": [ + "One of the useful metrics for evaluating a Large Language Model is **Perplexity**. Perplexity is a measure of how well a language model predicts the next token in a sequence. It is calculated by taking the exponentiation of the average negative log-likelihood of the next token. A lower perplexity score indicates that the language model is better at predicting the next token.\n", + "\n", + "This is the *formula* for calculating perplexity.\n", + "\n", + " $\\text{Perplexity} = \\exp(-1 * $ Average Negative Log Likelihood $) =\n", + " \\exp\\left(-\\frac{1}{T} \\sum_{t=1}^T \\log p(w_t | w_{\u003ct})\\right)$.\n", + "\n", + "\n", + "In this colab notebook, we calculate perplexity using [keras_nlp's perplexity](https://keras.io/api/keras_nlp/metrics/perplexity/)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "kNfs9ZplgPAH" + }, + "source": [ + "**Computing Perplexity for Base GPT-2 Model and Finetuned Model**\n", + "\n", + "The code below is the function which will be used later in the notebook for computing perplexity for the base GPT-2 model and the finetuned model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "27iA8w6-GlSz" + }, + "outputs": [], + "source": [ + "def calculate_perplexity(gpt2_model, gpt2_tokenizer, sentence) -\u003e int:\n", + " \"\"\"Calculates perplexity of a model given a sentence.\n", + "\n", + " Args:\n", + " gpt2_model: GPT-2 Language Model\n", + " gpt2_tokenizer: A GPT-2 tokenizer using Byte-Pair Encoding subword segmentation.\n", + " sentence: Sentence that the model's perplexity is calculated upon.\n", + "\n", + " Returns:\n", + " A perplexity score.\n", + " \"\"\"\n", + " # gpt2_tokenizer([sentence])[0] produces a tensor containing an array of tokens that form the sentence.\n", + " tokens = gpt2_tokenizer([sentence])[0].numpy()\n", + " # decoded_sentences is an array containing sentences that increase by one token in size.\n", + " # e.g. if tokens for a sentence \"I love dogs\" are [\"I\", \"love\", \"dogs\"], then decoded_sentences = [\"I love\", \"I love dogs\"]\n", + " decoded_sentences = [gpt2_tokenizer.detokenize([tokens[:i]])[0].numpy() for i in range(1, len(tokens))]\n", + " predictions = gpt2_model.predict(decoded_sentences)\n", + " logits = [predictions[i - 1][i] for i in range(1, len(tokens))]\n", + " target = tokens[1:].reshape(len(tokens) - 1, 1)\n", + " perplexity = keras_nlp.metrics.Perplexity(from_logits=True)\n", + " perplexity.update_state(target, logits)\n", + " result = perplexity.result()\n", + " return result.numpy()\n", + "\n", + "def average_perplexity(gpt2_model, gpt2_tokenizer, sentences):\n", + " perplexity_lst = [calculate_perplexity(gpt2_model, gpt2_tokenizer, sent) for sent in sentences]\n", + " return np.mean(perplexity_lst)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ELmkaY-ygbog" + }, + "source": [ + "## Evaluator\n", + "\n", + "Having established the necessary helper functions for evaluation, we proceed to define the Evaluator component. This component facilitates model inference using both base and fine-tuned models, computes perplexity scores for all models, and measures inference time. The Evaluator's output provides comprehensive insights for a thorough comparison and assessment of each model's performance." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Eb5fD5vzEQJ0" + }, + "outputs": [], + "source": [ + "@tfx.dsl.components.component\n", + "def Evaluator(\n", + " examples: tfx.dsl.components.InputArtifact[\n", + " tfx.types.standard_artifacts.Examples\n", + " ],\n", + " trained_model: tfx.dsl.components.InputArtifact[\n", + " tfx.types.standard_artifacts.Model\n", + " ],\n", + " max_length: tfx.dsl.components.Parameter[int],\n", + " evaluation: tfx.dsl.components.OutputArtifact[EvaluationMetric],\n", + ") -\u003e None:\n", + " \"\"\"Makes inferences with base model, finetuned model, TFlite model, and quantized model.\n", + "\n", + " Args:\n", + " examples: Standard TFX examples artifacts for retreiving test dataset.\n", + " trained_model: Standard TFX trained model artifact finetuned with imdb-reviews\n", + " dataset.\n", + " tflite_model: Unquantized TFLite model.\n", + " quantized_model: Quantized TFLite model.\n", + " max_length: Length of the text that the model generates given custom input\n", + " statements.\n", + " evaluation: An evaluation artifact that saves predicted outcomes of custom\n", + " inputs in a csv document and inference speed of the model.\n", + " \"\"\"\n", + " _TEST_SIZE = 10\n", + " _INPUT_LENGTH = 10\n", + " _SEQUENCE_LENGTH = 128\n", + "\n", + " path = os.path.join(examples.uri, 'Split-eval')\n", + " test_data = input_fn(path)\n", + " evaluation_inputs = [\n", + " trim_sentence(article, max_words=_INPUT_LENGTH)\n", + " for article in test_data[:_TEST_SIZE]\n", + " ]\n", + " true_test = [\n", + " trim_sentence(article, max_words=max_length)\n", + " for article in test_data[:_TEST_SIZE]\n", + " ]\n", + "\n", + " # Loading base model, making inference, and calculating perplexity on the base model.\n", + " gpt2_preprocessor = keras_nlp.models.GPT2CausalLMPreprocessor.from_preset(\n", + " 'gpt2_base_en',\n", + " sequence_length=_SEQUENCE_LENGTH,\n", + " add_end_token=True,\n", + " )\n", + " gpt2_lm = keras_nlp.models.GPT2CausalLM.from_preset(\n", + " 'gpt2_base_en', preprocessor=gpt2_preprocessor\n", + " )\n", + " gpt2_tokenizer = keras_nlp.models.GPT2Tokenizer.from_preset('gpt2_base_en')\n", + "\n", + " base_average_perplexity = average_perplexity(\n", + " gpt2_lm, gpt2_tokenizer, true_test\n", + " )\n", + "\n", + " start_base_model = time.time()\n", + " base_evaluation = [\n", + " gpt2_lm.generate(input, max_length)\n", + " for input in evaluation_inputs\n", + " ]\n", + " end_base_model = time.time()\n", + "\n", + " # Loading finetuned model and making inferences with the finetuned model.\n", + " model_weights_path = os.path.join(\n", + " trained_model.uri, \"Format-Serving\", \"model_weights.weights.h5\"\n", + " )\n", + " gpt2_lm.load_weights(model_weights_path)\n", + "\n", + " trained_model_average_perplexity = average_perplexity(\n", + " gpt2_lm, gpt2_tokenizer, true_test\n", + " )\n", + "\n", + " start_trained = time.time()\n", + " trained_evaluation = [\n", + " gpt2_lm.generate(input, max_length)\n", + " for input in evaluation_inputs\n", + " ]\n", + " end_trained = time.time()\n", + "\n", + " # Building an inference table.\n", + " inference_data = {\n", + " 'input': evaluation_inputs,\n", + " 'actual_test_output': true_test,\n", + " 'base_model_prediction': base_evaluation,\n", + " 'trained_model_prediction': trained_evaluation,\n", + " }\n", + "\n", + " models = [\n", + " 'Base Model',\n", + " 'Finetuned Model',\n", + " ]\n", + " inference_time = [\n", + " (end_base_model - start_base_model),\n", + " (end_trained - start_trained),\n", + " ]\n", + " average_inference_time = [time / _TEST_SIZE for time in inference_time]\n", + " average_perplexity_lst = [\n", + " base_average_perplexity,\n", + " trained_model_average_perplexity,\n", + " ]\n", + " evaluation_data = {\n", + " 'Model': models,\n", + " 'Average Inference Time (sec)': average_inference_time,\n", + " 'Average Perplexity': average_perplexity_lst,\n", + " }\n", + "\n", + " # creating directory in examples artifact to save metric dataframes\n", + " metrics_path = os.path.join(evaluation.uri, 'metrics')\n", + " if not os.path.exists(metrics_path):\n", + " os.mkdir(metrics_path)\n", + "\n", + " evaluation_df = pd.DataFrame(evaluation_data).set_index('Model').transpose()\n", + " evaluation_path = os.path.join(metrics_path, 'evaluation_output.csv')\n", + " evaluation_df.to_csv(evaluation_path)\n", + "\n", + " inference_df = pd.DataFrame(inference_data)\n", + " inference_path = os.path.join(metrics_path, 'inference_output.csv')\n", + " inference_df.to_csv(inference_path)\n", + " evaluation.model_evaluation_output_path = inference_path" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "UkC0RrleWP9O" + }, + "outputs": [], + "source": [ + "evaluator = Evaluator(examples = preprocessor.outputs['transformed_examples'],\n", + " trained_model = trainer.outputs['model'],\n", + " max_length = 50)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "KQQvbT96XXDT" + }, + "outputs": [], + "source": [ + "context.run(evaluator, enable_cache = False)" + ] + }, + { + "cell_type": "markdown", + "source": [ + "### Evaluator Results" + ], + "metadata": { + "id": "xVUIimCogdjZ" + } + }, + { + "cell_type": "markdown", + "source": [ + "Once our evaluation component execution is completed, we will load the evaluation metrics from evaluator URI and display them.\n", + "\n", + "\n", + "*Note:*\n", + "\n", + "**Perplexity Calculation:**\n", + "*Perplexity is only one of many ways to evaluate LLMs. LLM evaluation is an [active research topic](https://arxiv.org/abs/2307.03109) and a comprehensive treatment is beyond the scope of this notebook.*" + ], + "metadata": { + "id": "EPKArU8f3FpD" + } + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "NVv5F_Ok7Jss" + }, + "outputs": [], + "source": [ + "evaluation_path = os.path.join(evaluator.outputs['evaluation']._artifacts[0].uri, 'metrics')\n", + "inference_df = pd.read_csv(os.path.join(evaluation_path, 'inference_output.csv'), index_col=0)\n", + "evaluation_df = pd.read_csv(os.path.join(evaluation_path, 'evaluation_output.csv'), index_col=0)" + ] + }, + { + "metadata": { + "id": "qndIFspM9ELf" + }, + "cell_type": "markdown", + "source": [ + "The fine-tuned GPT-2 model exhibits a slight improvement in perplexity compared to the baseline model. Further training with more epochs or a larger dataset may yield more substantial perplexity reductions." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "XvtAnvrm6H-a" + }, + "outputs": [], + "source": [ + "from IPython import display\n", + "display.display(display.HTML(inference_df.to_html()))\n", + "display.display(display.HTML(evaluation_df.to_html()))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "RiCy6OQ7J3C5" + }, + "source": [ + "# Running the Entire Pipeline" + ] + }, + { + "cell_type": "markdown", + "source": [ + "*Note: For running below section, a more substantial amount of GPU memory is required. Therefore, Colab Pro or a local machine equipped with a higher-capacity GPU is recommended for running below pipeline.*" + ], + "metadata": { + "id": "AJmAdbO9AWpx" + } + }, + { + "cell_type": "markdown", + "metadata": { + "id": "kvYtjmkFHSxu" + }, + "source": [ + "TFX supports multiple orchestrators to run pipelines. In this tutorial we will use LocalDagRunner which is included in the TFX Python package and runs pipelines on local environment. We often call TFX pipelines \"DAGs\" which stands for directed acyclic graph.\n", + "\n", + "LocalDagRunner provides fast iterations for development and debugging. TFX also supports other orchestrators including Kubeflow Pipelines and Apache Airflow which are suitable for production use cases. See [TFX on Cloud AI Platform Pipelines](https://www.tensorflow.org/tfx/tutorials/tfx/cloud-ai-platform-pipelines) or [TFX Airflow](https://www.tensorflow.org/tfx/tutorials/tfx/airflow_workshop) Tutorial to learn more about other orchestration systems.\n", + "\n", + "Now we create a LocalDagRunner and pass a Pipeline object created from the function we already defined. The pipeline runs directly and you can see logs for the progress of the pipeline including ML model training." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4FQgyxOQLn22" + }, + "outputs": [], + "source": [ + "import urllib.request\n", + "import tempfile\n", + "import os\n", + "\n", + "PIPELINE_NAME = \"tfx-llm-imdb-reviews\"\n", + "model_fn = \"modules.model.run_fn\"\n", + "_transform_module_file = \"modules/_transform_module.py\"\n", + "\n", + "# Output directory to store artifacts generated from the pipeline.\n", + "PIPELINE_ROOT = os.path.join('pipelines', PIPELINE_NAME)\n", + "# Path to a SQLite DB file to use as an MLMD storage.\n", + "METADATA_PATH = os.path.join('metadata', PIPELINE_NAME, 'metadata.db')\n", + "# Output directory where created models from the pipeline will be exported.\n", + "SERVING_MODEL_DIR = os.path.join('serving_model', PIPELINE_NAME)\n", + "\n", + "from absl import logging\n", + "logging.set_verbosity(logging.INFO) # Set default logging level." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "tgTwBpN-pe3_" + }, + "outputs": [], + "source": [ + "def _create_pipeline(\n", + " pipeline_name: str,\n", + " pipeline_root: str,\n", + " model_fn: str,\n", + " serving_model_dir: str,\n", + " metadata_path: str,\n", + ") -\u003e tfx.dsl.Pipeline:\n", + " \"\"\"Creates a Pipeline for Fine-Tuning and Converting an Large Language Model with TFX.\"\"\"\n", + "\n", + " example_gen = FileBasedExampleGen(\n", + " input_base='dummy',\n", + " custom_config={'dataset':'imdb_reviews', 'split':'train[:5%]'},\n", + " custom_executor_spec=executor_spec.BeamExecutorSpec(TFDSExecutor))\n", + "\n", + " statistics_gen = tfx.components.StatisticsGen(\n", + " examples=example_gen.outputs['examples'], exclude_splits=['eval']\n", + " )\n", + "\n", + " schema_gen = tfx.components.SchemaGen(\n", + " statistics=statistics_gen.outputs['statistics'],\n", + " infer_feature_shape=False,\n", + " exclude_splits=['eval'],\n", + " )\n", + "\n", + " example_validator = tfx.components.ExampleValidator(\n", + " statistics=statistics_gen.outputs['statistics'],\n", + " schema=schema_gen.outputs['schema'],\n", + " exclude_splits=['eval'],\n", + " )\n", + "\n", + " preprocessor = tfx.components.Transform(\n", + " examples=example_gen.outputs['examples'],\n", + " schema=schema_gen.outputs['schema'],\n", + " module_file= _transform_module_file,\n", + " )\n", + "\n", + " trainer = tfx.components.Trainer(\n", + " run_fn=model_fn,\n", + " examples=preprocessor.outputs['transformed_examples'],\n", + " train_args=tfx.proto.TrainArgs(splits=['train']),\n", + " eval_args=tfx.proto.EvalArgs(splits=['train']),\n", + " schema=schema_gen.outputs['schema'],\n", + " )\n", + "\n", + "\n", + " evaluator = Evaluator(\n", + " examples=preprocessor.outputs['transformed_examples'],\n", + " trained_model=trainer.outputs['model'],\n", + " max_length=50,\n", + " )\n", + "\n", + " # Following 7 components will be included in the pipeline.\n", + " components = [\n", + " example_gen,\n", + " statistics_gen,\n", + " schema_gen,\n", + " example_validator,\n", + " preprocessor,\n", + " trainer,\n", + " evaluator,\n", + " ]\n", + "\n", + " return tfx.dsl.Pipeline(\n", + " pipeline_name=pipeline_name,\n", + " pipeline_root=pipeline_root,\n", + " metadata_connection_config=tfx.orchestration.metadata.sqlite_metadata_connection_config(\n", + " metadata_path\n", + " ),\n", + " components=components,\n", + " )" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "DkgLXyZGJ9CO" + }, + "outputs": [], + "source": [ + "tfx.orchestration.LocalDagRunner().run(\n", + " _create_pipeline(\n", + " pipeline_name=PIPELINE_NAME,\n", + " pipeline_root=PIPELINE_ROOT,\n", + " model_fn=model_fn,\n", + " serving_model_dir=SERVING_MODEL_DIR,\n", + " metadata_path=METADATA_PATH,\n", + " )\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Mo3Z08xzHa4G" + }, + "source": [ + "You should see INFO:absl:Component Evaluator is finished.\" at the end of the logs if the pipeline finished successfully because evaluator component is the last component of the pipeline." + ] + } + ] +} diff --git a/docs/tutorials/tfx/images/gpt2_fine_tuning_and_conversion/perplexity.png b/docs/tutorials/tfx/images/gpt2_fine_tuning_and_conversion/perplexity.png new file mode 100644 index 0000000000000000000000000000000000000000..6944bb9ac9db1aa3d715b4976b94ea39cd000fc0 GIT binary patch literal 64896 zcmeFZWmH?+y7*lQ5?qRv7T4nLP>M^TxI?f&ai@53m*P%Kp@l*r5Zr?nr$vKH@gRlb z_O70D_u2d0Z~rmgk9UkU1|i8>nQP9+pWm9%uhbN9u^waHyLS&)Nl{kw-o1zH_wGHo zhlvin1IgI30{+~0*Hn0MuVUoM_Pu*F_mpI%wcnVc^3WPgdL|?E=F=F$gztx_xOlVd zPgxYd2_}tv(33~Up;^l=wHdS+aI3Vj4?h#RI&V7*JQf%4kT8O_ zTkhrMHNyisntvPw1W6DLIfDCy|2z~q?t{}O zNaJ~#{_)K}XCVqj|9LX-I?3<>5ql}eV{28A?DKyfA#$Go^*>0KrWVAYuW7_TkojLj zi+T%+-TCj)Mm@I!E5eHEY*TsuYg`2P!K9P_Jye3=38Ey3lH+$4%m2DB8emvo9{#Vl zN*QSo$+LwTz5hK_U|8B1|LZMHmFFRhwQ?WhqW^nnim(#A|2?BqjL>%|*`6aCa+(=&xSek@YnW>v z5|sS(A+WYhCJoXe!d4nT(?{^?)B>`^jmti^(sK=&Uc874bd5YtOq$ zG1zST?)h_?2WrFl?q+UZFUQfNW+(0MdEv(b{6#qFu^~q0o}bYN?`>a9eqF@T)-PzK zX_g5_`oaL}+veJC1Sobk{_nH&*B-G`-t{=tO2A_aUVhh-&%zDpL49l7xzK8$i9E+MfOjh>hZCY(pXvGiwiM5t_CVKTiJRrQ&0r16OP|%>uLK zu*Khvpf>V`d;QY_CZ9ED-E4o5*U=a_s*2Zv3BR#y9|;rKpPwr~FoeF${@MG5B)wW& zOhWdrHF+OMoCN7-HEa`Vb`XFYjPICEwxl5$+p`4*tFoE`r(*|xwpC%LzbI9PJyKSk zj{VEY@jnOV;R5RHL{~ zRYysmXCeV2&o|4HdcvhsT?O_?x80-muX%kQ$^ZyOrM^^CAL>-f%FOe896IPs86@Oq zzSuH?TJsz34WbTsy7F3Yfu8sm*iy`_Q!$8}PISU+(^?j^J;*$4+BOuIGD$XM_FI-N zC-1jL?|bj65DWoKA7(L%&N;{ zSUz$VgY%NjbEVN8UTJhdz(+UQ$iK}DC*w5!eW%rz`Kwjq^-UN2u4(c$)Z{XYLmzTF z1ODy@n`xhP5BF!R-SQ?U=I^LfxgN-p6*Dt!!BoqCXu|qe05QtRs!IrW!>#;oJV@QYEb4NyQ$i{LO+Edb1x-O&cl5kqksar z>G<;w)KP76klDHN5ehVR6PeQ*V0j9kl9+KQ$Vq8myAd1Ok0EfL2$+5Qk3dyB69}Uu zJBHp!dHSB}XuRvp^t>D0hc1a{9F%@Bgb{9+^;u~d&Su|k78yd;OEUO4 zT?)is%?Qynad#L@Myd2*k;<+;y~;+^kAYM&d%s5XP@F#5cysLs9X7OTckPaZ>Qz_@ z2AQ5Zn-g}D)Do~`w#4cUmDinvsWz{3T~$<*ZyP^QusN>$-Xt4FcYOB~+$8^(T0I9T z13_>yHub(FdB9o()Xeic16(kV>?TjEzjAH>bv3Q>jKIhh11>|A=w%&4%)U!aDv?7q zqC|uU5gSiDcyxx_5WVuO?}ARFlpJxiP7JL_K_kSBtD;F#KAI)q$;D$fIkt{$ph#AIlm@zO?M2r!scSO$bmd4}&;jM}lo9GI9yE4lJ{@P1=*$aNR09XuS6U8Xf zQ|mBXx41>Vpl`BgM_NRPPO1nFH9^Z_(D%b#TR4wdPQa}qR5vgFmQ!xe{MV9#RTYQ2 z0A))(#fMy_D3bXe(-xMsokwX?Gt&-tV(=hVZJ2Sh&j!J$i^Wo(>2yw&8AC1s=h<~X zDmLnd#OQX|0mE6VIfyG~6&XgYrc!R@wHld4Fhv~r4J8MrW{H6z;?9Bt*K(o!f^ttGkTTL=^i&rqK466@JVizjNP(D@4UI zeQb~U%UDM}H+>MV1$*1`)$2UU<%~AfL6|N*2zQ^GImja7-Odl428~uDGC^~%#4EGg zDAjT-NM@uShgSm&J?V9-I&ArIU7u%v*X0HgdsOPuU0=EN+?KMe>AuuZ+)Lz2w=xdj zxp{9mCaH0j%WEv=Fgm{L#=;=N^W{%@Lv6m#tQ@@iQA<&?^xc7}L5`Uw3E~KFUXb(C zR(0E}V?Jr(RZUFX>>XrKW&(D=#pag8>H{cZap1snkM=XW2+T7z^ISO z<3GW6vEYBb*#k~z6yg@!m~1L!-%%kzpLV2Z#d^^V-{UIr)1wVxw7Zj=zMs`f9A-O` zp>H7STsov2TyXmvhJ4m6zcjpqer3>!5l+^{;XQh$~J8{(>JaX4GF-lgox7gcRuvuv37AJ(WBM+|bR&0nx zdUW!8iO;;Q3p|Z@%GG~+RWdrnX(DiSP`3B_wC^hAcslJLYJ?7WdL^1u6MFX%Oh3`i z?HJFYnVP3+lvHUy`?Ksxhxq(V% zn6Hf5SZ9W=ab@XxtIKI3j!yWXLJk=>y=mTpyA#Y8-qmEUe=2xZn%P;xElme+a7GrQ z#bDptk2|h-BcU5AwWRC~>3QQzaf)7ncJ-M^Bue46$=LxuRhr0e{@lVBKATrwH=KX@ zK$-`PK(61923si>syUg>U(hIF;k~Qb2#l?WBW9O%urO2jz^V*GxrHM? z^^u$w303sVYGX4OhF?SqGLf)znL$s_`@`31w-#!!`Eam4Jb#fCYAQ?Nw;mE*^O3zBUnu0^=;fN;OA;qtf_&#Um z?L4J2*t)r>ut6`#^3PZWsc6eq`Hya#u^>7TiLReJ`y`Hmm_O=0iJC&YojmuwjAX2I zg#sL!nEp**ANbq+!_6 zGCT@lx|#$&Hg~Duz4Culo^+Uim>tXI#R}CyxRBTA&g_yySCl(1R>1VL=1kYR2$e|! z_7y&8^3RK)yW!jbBDGwzRqCcpZCJM>(b9G`Lt}ry%SfsVJuH{VPR2MH-nAd-3fX%U zRipGfmo=EAr|U2&=3EzWmq6T&swYgmDx?yB7e46+lt`QSZ0J-O7?t=!c!=K{5 zT&2b*^33g-6-$a--P&QMTO8f~JN(aEa!G|3g)r`;lCXPosx(@DusXFXoVQ-!nR!fq zmV*&QT!vWhQmH9XAw5Pnu0E^r`H9t##psae772%GEB$RC?4CalDc0UXxkK7R3q(21 zy-uTkn#ZSXe0Us!Chf9c@^fG6$r~SCYWVB+iJO?yh2atW8;(-Q#dZLkZEECGXAC+U zWtyt%m22Umk_d4gM?uuHdWbX*J$tNDrJ3XDp;`3-!#v*$9{;!7hF3XzlvM`pncllp zf0xi$LVys9aBOG(=(2r?DnXdi>aLmhdr$Hwn^8d{^D8+&aN&Mg1Z1UE%s5kIK|;yG zI-Dfk#qj0~iZO3;R;h%B&2~TN;I(`Wrxalug%G^Rk5zc$_)2I|0t43A-$7sEn{!-2_ z{og810%U}m&vZu`guUM)=M6jyx<7<42RsQ%ZU{PwCCibxxO)(cdmwxB80O?P@`X`9 zTTsaI_@>%u{SzOG18y%kD!CNej2Ac&=dN{~stt3&rz7AYS9|iL|9FwSe4(;m@^EZD zd3a6ez(UYRcENwG$@3>RG=Q%2suvA!Yhn=PY?f8KZ$(Om`_)qco+8CZKl4KSnJ(oOCyJtxyhfQ{e_h4VE`3tjc>Y4oyiYOzP{sofFJRuG=jraP?n}h# zV%*l_mU@FkY3LNg8tF;@!$x}Br>9oEtsJ>ngzWQI^NrBaT$16)SIx5Zj~hM!NjUU) zGy8_O8x`9(BpJXmOMXyahpck8IpFAg%lu6L945Qy|IHH0wpYV~e@2x}Xul$RzVXpC zK@w8)2tib3_Z@mc$V!~wBb&7J!;JP&U z#aJjN_n&}^h6k9Y?h+!?Z}Pvz@T0s>V*GG=Ir)A!c24Tbz)%5}Oq$4Hu}!b5qxOfc zwZF`Uz1rC8u{C%?!p~sp5%H_e#+DKH%@iNQOrdEEc0v6!mBY7HzP)+Ra-&@2i{K3& zKpuIpkbT2ei1~e0ffpFvA$nnP)sIJ#!EM?IBC-{yR+-@vjDILq(gQ%L(pbWf<9xjW zIor{6>1^gR6?Kuwy3GlRNy(jjV8)K5Q>!UbhNwhmQ6K%>VnKAejzQ8;E~Pf@a)|4w zwZPj+YF*(O+k7@pqXC@hrds?MNqh`%INNkA>3z0+h}RZzW5M0r&p3Pi3FV0mX6px} z4ccEIDtLZyO| zv$maQhsp!+;frNi9PKZelAhNRql*=;Zzl5jDG3{ejnBHS?KBo5H7Nz=;WGN>kB`HV z=hQ<4y+8`)xt(*d|KdmP($L7hjbMQPLUTHD9#xWo<&#`N@}JULa10PCm9i|RN{K7O zV3E*7d(C`T>q@H!&L+;M9q*8dU~2@gY+w%Ga#kYJ$fD+(NOQXMP&VfZ!wao zkMu@=f#q*)Nc&T%1AjSL&mv+!DJ~kw86BJ+z!Mz%(ux$3vN_fJXH_o-6KDXHaDT}s z0jGbVue`u+_nSmz4_Ctp@#WnifTgKaDLi4KzL$gr;-@tN-uJN?x13uyHq|oTr3DJgh@)e>cd+gaPg4 zA8{M?uLUms?B)uO&u{Y+%2^Vn1ShF;HfH}_`yX#`^2dO-vVOs`(hezp@Jj1a5${I) za{*95^5pv#oRaI1X!OsfTu~D6bu4T-1ERl1_U}f~w?Eyum;cu^`oC`6|5KZ@3iRn^ z4wER|E5ZkRbM=Y-`$MD7Y`a&6mHrTeg3pov(O2yPKoSJFksYE3cTj`;;>iDWJSP%>E9}jDz8wa-zFdXA zS<>KiUjv7Mn*S-@PQ(G9508Dm>mg#_=K6S7jhaj$l|!T^s^-!7s}x?Mk{SemR-Qtf}@?g2Hxs-YBBf|keCr5wE%eLmMZEzEMZ z9Cki$n4l9u@cNyiLJGrc9p|R9`Cn_svMaQ?&%n&5s;yOa7n;58jf3MX|Es6FNKI_7W`e((%=8VyH1J}b=AixV=D;ycM_1|kDeg+= z2bsv(pu4Lj$URe=JWq!V6R*+>n{{G#tzJAhb|eZizFAw7|Ml}s>7cmfO9yGhh1y4; zKz=;Bw;IcZH()u)QC}lb7U zZSc+M+~~$=0d>vG*%HldQ0AK(;1VI1FD1B`1U|X>Z032RcFHR8xEEaeo=%k;z%7yl z8vGda?evQ<)9fo6d-d2SI^nXZ58Jzur+#Qx$%7*j|}xAKD7-Q~0)l=+RfB;sgKop^7Dx=ufH6kG4j zw^@N46PLDe7JVw;E!s6|&84=0y<`Oo^ta%9;j4>s#^?S~_y|zDvdCnr1gS0$Jmygk0bNE_&*Cf85yMj~;oZhLZ z0+T&XhOm}XrS0wLnFUNbHSY=L2-xc$4#6NR&O%H6?l94@RzL6N)fl!f71doK;l=$A z@q}r(LK)Nb3wM0t2OsT|o-KHrEfaZ-Nn=p>*sFr4>36mjW0T{b)rFupOMVBfqRvr7>}!3A^=k=YBOAjxHj&hU4UX0Q zOe6z3zb(5$irqIy5bq6_hz74vrAp&%*>-$Ic*WKRA_Wjv=7mwCv59PfEHnNu{8*%C zJg=W67TyGmv3m z;*ihIZ5JD-2n_0(%J$pGjh&qW=RBG|83s@avCUlfXGvw}3WARinmV?=2Br^`&{P&N z)*EStLy9My$8XjdUqwM!4|3e4UWOEB*YF(x>n7z6EMlfIOc7{TSw3?SkVG&S_`4Bs ztc}Uw*VaxM4kk!mZqZo0n!v&~I6q%YeE3eL39uYxJ3qAX5Fz-2S1yOseXke|C$A3L zoh^Igz``DTvjUjxNuyFZHp`0wR^O63@;>R5=?2gBE+axiA!y~s@#H*o`f>)qJ&~p7 zUx@|j+m#lR$_LOsz>vmOG#(A0S4gN)m%LygNPU?jSW6}03*yvn__?kutFhKb9dr#) zF@;4d?rpPnW$UdOo22eiO9S=4sE#5B$&8~dx|Ol6JkfE;31FXb6dSq-=Mlw*X)lQo zybkh0IXsnVo=%brzV%tT;dgx$3#f6{0Ec;0f}ppw*6t6B2##^5xUWB67YW?m>16qb zgfEjifIL0B&{{Fu9RSZFz2NQ{esn@y@dAfC^)8NrFXZwY=%MvkTfmj*v?kRA3DLgP zL{P@LSi6?G5Bg0fTV;!dsz;LE&RVv81%)ixVA`8rKw`)nh^mz50PnVJf(qUJQLMr( zeROlSgyESlYG_+{(|WOyBkkwvSY_58ShwG_D=)u_9YnUbA1}0sd3kTAbJ6d>B=X@! zpH9qGZ#*Sg%?aS-&>UTRNc4uT_8b!ah)d`u{PBxCYJS^S%eo|-V!|Dv%Z~u@w>?Bj zU9aF{w}93Jvq)%?ZX{{rI(cJ22L$24fpgFieV7e+PE8USi#YP_Y&^q>m6dx zx7GAC${z^(ON1s0ddSSg$+1g!o!f7`#`Z6o%bF>jTU^I5WQ$+SF^#f($_11yvt@%g zx#u^fWnY2->d7qL`t!@ByIV3<2EKGHNlFJ;Fh~8o8}_fpz_-3ic|}wjqe62I@bcTm z1#Mf2bV#%la<)rT@xUO$o+65yJMTlXI*j|<3SLC;O`p{5_qs*jUEZH`NcsU=d_b2_w$C#uGykkiS8J?G{@a^8jy*&Gk{MeLdYM z3UfGy&x)%*4$>8OnyYjt2D81^ATNz1zj=A}AF210hdfc9eCobL-$`@BP^ zd&m1PYTz67u(=?^(YDOTPYZpk(#KJWH;dJQj!cG)tAO{qJPH-Z%jsXE#(o`~ux z3M%105q4Rzmi6(y{?$jn%YQ{wau(QuGhlglbB4b!wHif_W7)|lM03m;SJb@U#71zk z(tS^vCZVJ*gw|s{O%FZPMVJRxjS(;SmR5D{!!X0k8p|Cd->5io7T3;I-JFv>V1=EZ zo~hh5XSzHm=iX|Ty>d^%A-{*3c8_=?>>L$onJ6LXM)%ga9guR11G#c;jWE}(WFthB z@2)qcLJyAg{8sbjVCb^< z2r~%aDD{s4eTFp2Z~W@ODCd%1i0d|2Frk4n}J3(hfowL^GLnD4yM zHreoQ3;xAtR~YQKtTY#|9M@hGz%Oh`h*ehPZYr#ePLgJZ5y24XW?Ztk~J5%Nd zm<(M%nXR8wVlsDQ)yZft=f9YRvaX|!nzldXps#7N`oza9>N>Pbhe=+#fG~(3pqM|l zbVZ9Fo5~L9=$TkA`5(WW62SDB)qO*4$a6A7#y%sd5fC69Um9%8j?i2S8DiG|W-5}d z6?-uRpapT#&?atO1CkG+pHj>tBC}o)Jv4k%fEO`$GHpgpzJ~LAC}fOw>}^)R)Xm6w#lFhKq>n1lqnNAB`Q((@E> z9cH%QmXHWWY_c#K?wPoe7_MmFk(e0!z9HpmI@wMZ5frS8bYqx8E($(v79?>l*7;Dk#vBI? zgiTD;(0cS_R_H?Z$q~oT0t5A<#NZegtJ87DOR(Uu<*UQiiJ-Q;P1p-LpE3D2q{L)u zxfa6T@m6-3p_pKK55&Jt7)q+eDAGfa?|tn5m}OxsLhTTrH4%St{(}cm?=|e zIkWiHE3265n)<9mY7-|#c9@7we@04_DLwqya4Fjw3SP4C4_YW1asXmKco*O=5b#ZD+8CDq}-{=*fjfR-(9#{2IVW)R4up0EJ(IzJ>lo#HI|==SM}iE*gWU?{#Mxkp$S zdry*k=>#CCu1il6?nStw$1;Qz&suAhL$dfk=jGkJ>{VPU-DIwlJB!OnSNUk;C8!H+ zs>yY_`W?8GPEq3^hlWjCNWsoCrY8QYThuf1$T~?rm7dcW$xO)Zj~_uUpM0Phu@dLOeds-jlbnt2XY_hmd9(gt3Duy62h2%;%W<{PdJq!tr@p`b}O z2L3kp+_(y}TN-tq%|^_1tNrHh777V-#-}cds_I8caT=5R2VKoAk)}%oHJ{GfhdPUI z7o}A1_q^#%daoY4qmY6!k7jEI!tX_=)|09C?3o|p-GV|y*aZB!?mo5C%7IDS)i&q4 zRx59LWT*w(eA4fn=*D@Bav2!SJFzeyylwJ0L|{xC@cexGJ1_U)dYIi0f*a#eSrV`` zkr%~=cNmEU==(}0{g%4K$wbFD*|8t;uT=t9g+R1rb5XQAdDJAF%CJ<;Ud|_iduX%= zm0yy%W(C;t4>!cl4`6JswS7bgJDlT;8hD-BZ%#cqL@g0)oydf$3 zeRnY$l*~UZ?-~zMiH{YDhO8XQhz5xA#D&-uTa1{Sk6G1qGBWFvj}eFsfg*_EGESuT zsYco#2RYAwRrJst@|9%_L+hnPmjep7S;~u)0%Fl+ao6bfK{Z(8Un(J3nK!! z$~&}Y8FB7oo8h3%qjyD9h{`q`2n z=PQupDCchBv%#4=NnM<+#8R%I68b`u7$Wwbbe=QXANt>=xA;`3)AZvR@h52K1ydSE zFX-O#yp4WW5|;V-<#*}}&IIF}BFFTPxr14HX0X&*=kt*8N#-|U#AORLltg8@zn`w$ z$a5yMt;b`ZQ~99?jm;b+psNv=v2T0oGzNeYV5#d%MLK=|B^#V8yYNkp&k*xo+6wq5 zdTXY4vNn5yTB}YnT|F$(e96YYR$|rr4C5f?o}I&QBFY|?F_2R-fMO|8JQd_tg7Yq2 zs=M}Ce!LwwW}iP7Ask6)K$~3+H7`Bv@+9c`zd>`Fu?|H_$VWHMKx#BaC1HBf2|+d zZQES2<#wgTe(kg=!b(-!f-Xw13yX;@tH@%v%H9vt%|_~H8GihVH#q%qh8w^c;Yh(X zaoTDhu}9s9whn~=7MEBK zoPPKv>5dANGWWS-!!yn?juGb}QD}H9CsCRdl+476hB;2TwcrduGL|>|Q6CozWMSIP z4nWJ&f64MA#@yfp1Bl$!DkH zhtAh~ADKv#G1D6ieA}`p#D&N$avO1fmYdQgtQF#3dagBY=3@)aF_W;D?_10>63S?J zj#0>|>T?f1Nh2U|a`&9K`_Pbr%D zh&F#!a_pJVu=|m;8C+FF!}++n^rdT!MCBC3#vp;C;YFNT()~sa!tVqn`*X)=q{|Dp zRDx|iaznAi>^-SG`{G<}=N5;h7drxe87wkPAWW)vTq1<`GW;uMQ&T9LU0ty%12^n) z)>$^)HYUl~i%%)lU^J&&n}tMdKEG>=jDrQ{g)Ym+dL}FvWlSJw-2GP1+kAf{1p<8z z*VjO7HWHL>OU=2eBT#KJ^70JFF3^KY>drMJJh*`IPnIS@Uhc-YDXBYzg`F2*ytT@L zctCbK^(cGBqp63~2o$QxdUk9YG#)TNkj7ViBnK|@LZ;H`lj;-uSzo)It~RVYUd}i1 z#i-QeJ5X{6PE73{wF)>^zhC7vKeYDBQ3`WmppV0gQ^4^E8r3C25RstuP1XV;AyGrG zOMTs{so^=%n8--NuKI*#1QY#vkdEyDSwiY&)n(MK0WG*|a?-Kk$0y^WXWjQ5v%zRM z24n;s67BOL;hPo=@KL5G9vxo;Tqh*rAjyszI3{fr@G7giFq$LVE^df{YgCK#3{gAVU${oLx z%C%b-+fo1TzzUE{zmO~h1EJ}_g$RHUm0NJ948$2vKsE1-@^*CvK-lm6j2}_UDpv5} zUW;2iui9>yxblI1MN?t5){23+if@iM+ZDXrvp0LWXD#!I!%1s{UDO7Woak)<2R0;5$DE;}R}sh@ ztL2Vxp}INhU!6PeV1|TAW$#w&@tmt$ADf%z$=Z> z_ofZUa#1zwh01PU+qt^E#k$thfAQ)Ss@ig4l1X*H`JaiFk<3*wmQ+(3Rd7SEJ_vbq z<}lFTsKadV@~A*7!L35uRAo=-3$6Z>!J?DQ-Mn3{U@aE`t-oE*JUfNF*uKi7GBCU6 zgM64(yu_PAJNgYTQ2>}jj^R{ruQxarGr7k2k13ik1Y#n=bpm$tDXE*ONz*6wApxR{ zmPG{84rc;$>yOyUQ^+qFf(Y=XC<#}M?{#g$WYa{P;sDV1WI`+WL(EmU7-flO2%1KB z#3jQA%LR}kFta|~{wfo*CE7* z&h_BDXr?#ByichwU^$RR5cbkep%O1oCqRe>g)N@$nk65(P5&fjFLxq*&DI}Cn91k0 z5<#-qr=^&0D;37yc6WP4Ms>YA%Q^gdehPSjjV%4u>hRtC)@V0|3&1KJ( z?%unDdBU}}$tB^@Kn-JEC8b%Y$Uvi}4@d1%$tTFKK_)$k4bfTb6#N-J*T_K9A5M=4 zYX8*i`+@3Mu?xD?GzH&#J+?_hm@r{Tuu5u5L(1DM9_w-DP;)?HuU(~r)rIF>wUH3= zNSVBJ-=M$5HH)u3>t)__Z&O_+DH6x&+%*LNP@|0%W~mALlZ22Ga)Q3|7P2 zAg-=gYq~R0+?$2-E1zV-N|eE*8488mRqAw{M2`;o7(p5US9t#JjsM+Q$pc8}vn$-| z$#@3&cYLU1*>@JT-Al16qp9w;*1|PqgUn?xCmDZE#0xVbollJeBdwtXdtB|;N1Zp4 zPD}FhRKWnC#z%dfvj1_0cs9@e8}nVS9L`5(&Ux21*KRU)4OIf%Cl9h2E;FggX}0YW zcSk=8ei`>7WGsJj&W5APK!}@g!*X$I^h5CV#|DRLo9@~y|C8Dz-cx)OjzTT=u}$u> zQ3{A`8K@%=ATTF{>!B5%K6v$&{!0U`fZ0}FuT2dMbp~)H)=yNM_Brt<|8l+ll89CB zH{^7#v|?<(DE^3V0xw&%`8O)k(`s2pD4|L2ogJ98zP|twa#+V%Z6F+!tMR<#rT^Fd zcFIR8t?g4G#i@T{-4;GqRzC~Wm#K@06hqK{Fw_71%3CYX3(3b{0$av>BJvY&`aM@I8qU=0^x z#OR#xu@Mt4c76Rki?orFjNDFEsQYmlMsCyt=Y&3Ydt`c9$%A`PLfS$#no*`DGn zpG2B*^pqvbe*Rlu7~GM-%4H?V5U(#@IadipJgcl(5A7!XzCGi&rYqew>J=$+84`ZO z$%*5@`JorxcZa9UX+?tOV#l7=8i#V{u{@i#;zB$TgR0^Y<=v;-jX5|NTS}XoEp}qA zg!nt5OIQKaBJm#Et&hb;TiRLb$J~Ee^mt|LbRauEiSQa_&^yN|RG#tuSN~stv(!JN z=c0oN*HeFF6pT#@4N4B{`RRZ;&DzXj-s6DUFq&7Dyc@Pdmrt6*e_Ox|&*FE8d!YLf zK)$xFv(mc0zV)_XS!vq98@8h9cA=pome&{da49{T{+BiC!Ac(u+H(}bUlJ}^eK;2n zzqxF&kF7PX`N4Zs&v|-4Ed+nB(Wes3)2QiRF>~UKnfOV{>xX` z#!AZ__hR%9M;kh#WI6~JBI-tK)l?aHV%}i98?@zOh%b#QvBd@A7ggR z&;c_&xwc_0tqf1CqMl86eL}lix%?e!D^^3>i+@SJS)CU9j`+UIdNaA@OxV7zA-QD~ zQJ?K)iFs=|HH4eJFQ(img&R^l^GAn44>z|a_a=W0qGLvbrs{`fcW17J za(F*~jn70*j*(p7xvEGw$KpT9)q}l4Z%BaTzT-~fVLA_j9%DhqSr`i=z@;!od8 z(a}b5XJ-jRLWQCadfqY=W~wN~)4icZVLn9V_r&1sp@&Y(U3NZ=LwW{j3CN4Lonp#m z2mzG+918ya3G)5T09z#}gqdBxp)XHp<*bT4c*r;fcx=GiDc%py#tu>on zDP_{pXwjPm3F;L&pG`w#^pI*d^)XNIF3=8rR2~6xtbRTI-gxV7`yneCk&pDx4eb2Y z>W;G?Q??2WJNI5hUdBcJL^5 z26#rT99EyJ2>p^aLo|d_Qv0~Y%`E40T~}EGDU>C(MPv{`UDQ*cx`uaB5T>|d)J1!q z;?H)LVK~BBY2@7M>0>HA2%tqIc2V8Ut>uK*r!Wh`S>C6*kfE$Q?{7 zqz|zoc+6MoY~Lu!iW|}sfL)>Uq;tLdevr*4a+xFwekdu=C-U6u(Bn)U&kyf;R4Et~ zo#Zp`IQh$~M#(fnR33-K7}!hkJm)=Mb|cX9>RBv62Y&8Pi2Y`BjyR+4b8m&i69x)3 zHZOAP=Cey~eV=}Byx8W9rqDwd`T8p{_&Ls2d2Rz-y)$Q@!awV)^*uc1IEJuiHVO^e zN&xp5%+mvPzVovsBae;ZqT?WRBB4(r`SgP5``FC<9*)wTDc*>cI>A0`pt9W~k*+A* zTL*2ghjo6W=y({`M+qgzvfS5Oe0d*s_TAaK!X9}?s>y>%=d04OSpH%GdRbI-Kjg9; z%Mb$TmpFzugDh9stK7J!8Ao8!eaqT=)4Vp7KVu($*LKHePU+r6D5?T{nLogf*?DCrlX=_|1pu(VSpo zVq0-0P%^dt@%Eu=Ru}`fQsYs~`)8k)N@^g%20SO3lP)062YF!Z#&B!9uOJoLkDerz zJv1GMUf*?+M^w}JV%1=&$63YV*rX1s-AU5ru!6oluzCBvJp3R_`R3VZ15e|qZ)LHbnzwMlDOv^sfB64%mtufTY!bsq6Kb)X0`o`_K-=os=JrH*7Q8QMXpE6)R`s z!$88`g8h)hRfR$`5NE%EMwG1G|3VrDiQe-+9ep|F8=fK#;g|WDF{K7ILwxi`HB4w{ze ze=v?(pp+sOUOWq{)>cL;0xSV1_m!%dbWV>%x7-$1d;bz~aE?(j@xWk#kwg^{$TmgR zz3b_K)Nl*Y!4E!rb)d|CYW@C+b_PHkdHOI1Qhs-v;TL^#xZHh~desR$zX6TQ!tFf? zqgdOko8vr`IJHT1?V=SW3KgJyi%cZ)Mwe^6;NRFx(-Z_S?n;qVjCZjvlkWqTBNJ_Z zov9HnrnZVkQOG+3WxjOPvFQ{xdC3)Kf@TZS1G; zw>HjT$A<+zvwYF5?6SIUE52+gloGYspqaTBH1|)le~OC3vweOvKLd87D6aND`Td&^ z*zCeAtB6VAH354EjVdEE5wS*Jc(_N>Ojz_vj|f-b_1(mChh+-ZPk(+%k9DR*sYX2H zm254^X}7DcOry}M?tbj=pA4X=FTGj#Q5txquM;w=240XGknN{_qjRWMClX=dSQ>kEtYM2h$Edwfg$A5i84%VQ`bJhu zR~5J=Qx3u4mj72o34DG=azJMj*j|KpJBb^S`#l-Ec1$G4F? z*#G9}cLV^W*w@5At-l4NV%@M)I!O;aAsh!F!@O!u%Kw58-`4?@wB4Xo8bC=aSy!=W zp#e31C3KY#MhfQtd0OC=5ipS2^@4$V6Ad=D;fexItAk?^ow}&GwU;`Ty!ZBXR!# zQsJ5KP|yBzWj0)Jx>c^nzjr`*?g)U+6X`*#^=1Gec)AivZFqNkv)6Jw0tdh?=C=AV z>;MF>`-WAy#kYT-3-qx(0F~9e|@zw3s!NcI`M)ys-7GBQ-azTg8S_kA0 z-?RO^rj{P-tqilkDpsveGe1+p-2=Gj_dcuajp*S*zTJUaCa~dh4{xLQAw_Ay=d}`5htR% z<(5yyyoLrQ0iJRX;DCfc-TTc4RM%RiKe5vRfaW{5ZwdqY06&~fv zBaVL0r0q4D>0-5QcCdbFOT=v|GmW{6-rI z;>(I>UFmdN3OGZP?~}%OYNf(nCA%XPwAvl*bv!II2LPJ?3V0^<%v%3NV*2yc)-T+S zUrv2N8jBU)u}w7(|150#nk%FP%M1W8C$Ehx8+83O=#w$0 zdFa&%kqaq?J=Q{$Ej;gFI|Zl;2(d)jv!XxHv5V!ufUs++RR&b=F(7)gp3^2|=ci@> zEJs3!sR58G@2;onyK$^7PvthQO4)k$(MolE=toSNS(xx&KVtXqyGDCi9?b!xtKWDO zO1BB%%KadfS6(ExDvdjhAQfbp-w1{>Ogo|;SQ=wSeSjM+;~{Ce1kzmo^h=>xTY9mh zZ}jlR?zq`#=6rIL{hzjUXe(Ku#oIM!VkT7>K#L(yft#z|tfK{|S9g7XxQiBCgOy%` zNP#GE>VGluB4VFp7ck7rZ32l*>ip9+b&;yU|d7 zXyQLU_5ZQ=)^BkH-@0#b_u#=D2Au>5?oJ2^fg$MN!QCamV8PuXXasi%76uqxgS$&` zcetJJzWdqdp8F@9!w>L0L-$Nib=B%swLb4#dJZ&=&T6?5;>HSVcMq7{bv&8EJV|I? zOsjB&$906Rz5Ze-E+ugiDVJg`)8BZyR*qZ%@^A`@&YFGYd5%q^fL21Q`Dtz6UC>+G z(#HDC?gqc0h9{k`xz+DwlhJYM@>X3XFln0oT)%F6G2B^ZP&4aRo-5Ii{BW(uN;UD- z-LzExkNntGTgF_Z8~NU{MN6H7!~pBaCz~U+7ghxp?)6PL8S^IG-)1>!6lcC(nz}S3 z-wYiD_o>buQySRyX9-T_-KgpPdf=`1{lyt|?GY`_nNel>6ynUab-kx?Su7 zm<#^g3m?hvM-$waN&tV7XZN;Pzw@XwjG#ei$&+aZ#fna?ag?ADrV1MQCjPE1iOIeg zKo+H!)JJP2Z^`HQACS0QBf0C-%34nQrSTKw}*(Ku|YLuXll!C>$20{ z%G|kP$G8>z`(#~GmU6b}%e4IWHt@T%GlAn~NC}l)lHsv$g{OH~=FA z`vaW|jJ_Gd<{gIzy(Pz|uvE@wEpyTQsFzi8rF40KCG^*Hvf}Xffo;f_Nv(USKL|7d zxq8w^#G!aC(yi6Cz~$S=b3IR<=QRxi3*|d|Sd}vV>E|1>VyS+bnzZQpNBya{DD$zB zkHS`xwW2`VmELy=9WMc4`=!-WQY>|6x_%L@B0z-TOI)w~E^8TWVI}$pf0mbx5ma?# z%VpoJ1QP_c;f0=y`p>)LeoviHZcF!Wo3+S9yHwO-JY8r$YNI~?R4d<~WA|*YyxG1m z6NslkkEBg0Y_%k<=;V%SC8rol)pu5&%Jj>%@!$UCTnsGa-B<5U%t&^usEmI8PT1#~ z)|e^pR%#pS2QW9}hX*I^iGguDI{LvxwSxi!sEVQhm({Kf>>&Sblm6GWB#`~3$A9bBolF@hz>>(wv} z8paFQaI7o9i%v%*Xzx`=1j{@KuSbD{a(|FK(vqRy0uMJPisJ%elpcw@eG&@z4e9^} zQ|7FlWEq{<)@kx#!W_euO<4w{KJB5X6;Q)V-;i<9fIKz@gMXrhSgIf)oLG>(m?wRIsk z3;SSDa~50gkn2(QDbY6YVz$EX;E>2>-r=kUFN}xVVkYomseYvRyehSa*K(ka&Q6i% z#@Yl3M#F~aQ5oN1y*>2M&sT|HZL0tZB}|m|@*KA*-TM9N0rUp?myZ_}0cgSA!qB(S zdqBOA7avN5j_Q-kSD#ND%ER@?-^lBvn;Zk-sDG-{*nYi+(|ioa30D0_6rn4KDv0}Z zS?x_5`q;hQUP}PW@C7yI9zh9k#IC8A)lUsV3vxm_g#sm_D24y1r*czoE#Hpu%a6$fW zI^!SwJwWcsJikzma^pi6!z4^^bql{6qxg+RvxZ(LF@a(Vq%pL^!#^DBNCtRk7;@H5fCT;%762 zGS`G);)ys}*yzV0q1UPl&K3?g2rz{}7$i&a`M?MeCD@ZE0YV z9Au1C*NGC;yy5n*IjNbS4@<;A405LpeB%-=mPm&~2hz*k#kp zzc9;uRng=pTJ{Yfti^?&%%$eBI-bZGA!&IQE3Pu}vr2P};E+;~fR9q9*)s*u9pfS; z*Zl#26p)WtZ*$w*BXl)!{co}6 zp2-vdkbg&x6fU7s{6G~&^4Tz|pSb^(yK1m#e&Y<5%42>u`sa7py2<1)$g zBSbw9H-BkJ9QKE81qX{MitqOX67k^f@-9YtU;rRg3VMv|!j6|re}3ivh|hbguif|E z6c)ENd$>TwCa_z9g?Df?Vw!+WfOhT3>lP>x_!vA1X!T=$NlE_2nE{li)|7uNjJq1> zk=eO|KyjAZt!YEw9^>d3$5_N;d+NAsVgG2~-^!V0n-P|0+!H=Ms}2Ab3IBjve$hTz zr@l;rbcZ6F*#%MkO8=L4r1+?ahH)YQ$o*ER*{SC?h!kHCiGGq!^KOuAm}vBPqmD^!X!$_=G%NVz3 zH)1s-6nJxQ2b429x_+oOEc-W$-|g3gvMZu;aR2kcs-9aBoJ`~vNpIY=!NrVvnF2|l z1z7^5+{VLv%>98;kJYYVf;raX^M|+}Sz} zY`gIb_D7K_rkywdflHBrkgZGR%Wu&KGiPAE22}(N8^BindbDGkTSYPavHmVpbat{+ zuwe`=syA;@vwy!yS7q_k-XHX}$!w=y(8vF5MXF!+w%tBk&kX;!%Wfja!+s3W-rNFO z`ZGGfEaM{LgMO48j1e6xxGtCPZ(}GH>KuPf+^{&DQd>XPhb;6JI(39)v+ODa$&B8z zKJf{^S*J3{8<v?m$8`9deyh1P=F%U9GvGokR3j*MzA zK=!Y)YzVD4vIQe>Dnz>kRVuG<`WwQ~*6)Gz5f7K`6lvMf`KXuD6EXR4Y z74u_d>}V3`ZlWdCLrhHK@HEmxUdzSt5m6fM{$8;xiv2YL8A;1!zB%|3A*HR!=lB5Q zSJ&H^EnON_SnSGtDqBF|)h^Xvl>7X)yb(-vkI_kEnt647IJq$3&=Tf^7vC#%XE~Du zu4n^*FqK`QEaO#btITtnM(ST(W}JyAkNp&GS;~|~%+2epjUxm}a+%-C1@JWy{<#Q> zXuQCguI#~FrA7b@+6QJ2n6|{1V6Ojk(fm27&ac761wHCt5EQBUY*cmp?GqZDTQA5& z{5*D2eA>aKyVq%WQ6hFZl)Me+YGJ!?%q%%*gXvJhW0;2*gV2vGm?McmP55jueiLju z?WYB_{91m|HH)(XM#m09a;kDuJ}Jj0t-lX3T-hKlM<)d(0GBPPjwmSrXs}6;b`b?3 z(VpfYn|em==t0>y540zss$mF_x49qQRM{|w@^mKDPj6X;NAbL z7P7&!1z8<-8_=K(KI$sUt!cAkMSptu{r%O!5Y_;y%$7rNPF2iMKxEh5+_m)O@y_7q zECB5Y6+u9=BALK_ji*hQ8(sNe?e8TNVVZ}@_MeN(*0-KFYROSzL zA5eE*UmNz|&boaSC*|J)f7&$;+vT5CP6d~)09H>1A~ zV)HDx2JLU9m;mLU~zs+|JD-q z+-ZR-V`=@GztrTXe? z$LE%mZ;^oPFNKv`f13|d+G=8$g`NdA<8ql4I_^9*ih=zFs^q=sv5CT(-c9^e&NAkx zk4mpJF^`*cYk1<|(I|zRp1WLXC9I?9H>(IdHn)B()p{Mj8&JYK2fqnsvE6lmb-uH^zOxSHh8c@9rv26akwthKm1oZuW+ajt00O= z6hA&g;eYNhwlxY5=^%VnXz+8Vnta=LFo><%OH4WJ!wwrzf8DkwxdJT^!)r2!D-jK% z_7*ATIt>D{Fxr%E-M{AUL2`<;h6)7RfuvNpb24bE69A=ai;JY~xYMie`DsCQM@^|x zF{+EDEtn-BfkdHbX{6@G^mji=e?y>H3FK}sY8pw&9)|}T&+g3mV3OduMvAJV4vl0o zQ0~**y#39mb>k6C>Xp<_LQ^S6eEsNkPu<>i8PC0jRRF%T!KAaHD*bVS*MmUf?5n@S zoOQMWYW(lvdyCW+hXBg!eC|me5&A$=mw{uv=%E{B9SGk`eqj$_`O)}EKCh3GOAWFQ z6y1@Hqoy#OC%IFvc-*Xt_KH;n#a|x&fj34fG)_D@;d}vg4bMB^ zK3Rd%k6!h3+7>H>}i{;kdt8*aO?VwP<)2}kk96C z72Z*;5G5ScGKdP@KV@H;2l%K21W4;P2f- zo`A9h(r6A5d0=0*ebNIIh%36^jiD?!>ykXczbrDbI1H?uN?I$ zVgIIln5Is+dzU|{qyV)EWpGkQkciaJTH{q37^A=IxiPh9iD~vo2a8R1d8cjUp^;u@ zaik12Tj)w#@qe=bL?A%Hy(|a;kw^$-0qc?B%fIF%+XhOd9{pz7O7#YIdX)}!31U9} zSxT!^GwBH8`)wEKM>j}ImT{Wh@%bySg(-(Ud81JrUlJtV4Z#pfY0oo(qy2K0Yx|os z4%w}vw(*Zd3S0YjYB$PMZvWW1BqMx4K~YsL^AzT331S%2?!|essAnMN{P&BV&Ph(lXNU!_rPjxLQg@z>lF#0z^YFS7Ik4rA2GZX%%3F|9N_c@mKSy zM#3tYltp5*gkIhdf0HsaoNQET5#>SqMk_<7z;e}HcAo-Ws3z_d(ACcPTGWEBNGJ6F} z{*>?A^gZD|)XJ$G0Sh#j?+fu8zS{pzk9`w$Ei;g&%y}A{2}W2ZYz$ZQd6e1cV45@j z-Bj5}bA#3-ma;Zk-jw?)$7neE1Xa@AKiDVt+PuDv2HLHMcoqJaH;2Af$D;+M1hoGi ziZ;d~CxYC+OpsV1@iJu@FyYV|2azypI~s+5c^l8A=MPyg1nY3B*B4O$g zOw-*lvHS!C!dnSeoP@?QzMRVOE)@ue0)-z69L+xbZGwyVsZe=MJL~jn+QRmwUZsgF zg&AVSH>hY-z}@_u7+MU2k#0ksVRMX;-@CiMT1DMFtYvLu39Feky<<3OgyDk0nzi{0y8lbIKkh}i=2E=`#B&B zwD=abvwz?o#!Jr+9@MA&; zugTqZ1V+9-GW;bZ@n4B#UhaQTdtu_h93ilMX%lkg{DHHg=$zVD55T?`3T{c3G#(0Q z69~_nY$JRx@~ZBE62@i%;wyYPURM>}-2!7*30wJa!{okpcHrG%5_-p4ZZNUL$$}p* zn5a>_G5P>N@>}NZPQRHV*dwr~2i+QfHbr}8rLMpGFBrDKcI~fimN5f}n`nL}iH2Oh zCnWC;Bg8+QuY34TaJ{%{o|tZqburY;d^hq(>p?M{5dPy~uItl!lzj(7G|5(NLt?5w zg3KYA`=dnm`1Ir1`>oHd0M*;T#$$f<(Bghv4zRVRHg0{W-p79yZqyIv=!6>RiaiJ( zf%!c+9V6g=TfO3m-{4C>U>aOTq9PqQ2NH0Fmy!Dd0E=Z#v)ud zgnZM|n`BG$w3`lpxyE zuQaFjJ)Dpz;RGIGTz8u3nEfaJilN}CQuiCn!0W5#2cG+(3a~N|FZ(Huur5E3y2Haz zZf`nRUXVNB_Y@lTP&Ph_KhCl~`j_*uk%K7b%#Wg}n$aiG6y|COw+WBb ze|n2uq3av~8)cn#WGGw?$GMA6#oMH@*u&3d{i!GH>S4xYBNvcl{KFaAT;ysv0D2Lu zQr^We>i-f89a9e9@(nXyAY_LPA&(UqdA^x5@>Sv6-aFz>5$iwVSBV(t^ti;hluJVv z+CuB0O3>r4HDLtQ;1U*yaja7mQdg*WKN8ANHn~(-C&JeQ+1}IC=f@S>Dt4!e?Jr)P z*{;bxK3%mQ@-mmQg*^UrC#@rpi(~O3=k+1pawv;(2M4-o5R@jklMDazFz!6Zj`0D!*J--k+ z7LhmHO!VS0M<}0nmEesQ!y!uu8cs7MRq<6w4n+R$L+~H~yXZ&EnK3bN5SGcKQeC!LI54#xb@B0U0qYz(kHb?&AuA`QR zs-gdy#OO~48)6p2TUX!C|Lb1^6bvy?bs7DRF$YoG@Gzng^E%iAMER2XW}A8p9inNH z<&;ICbEB=@f33~xt*_*Gny{f~V5E^_rRdo85Y?Mdezdu`KSX~^an$vj75>f0Fu5N< zuF;pZl{=7KY{iB6qca8IOOx_28j_Jl)>EMcrQP=9=U>bI%uy2AWIc-25T%j@%rvKOG#F?s`_`;Sl9*jJ1^WHzrC5hK=?B-^zO z+UnbL7K?3FmQ0tGz5+!RixMWm#<|#7(|m%{BU#JvP<11aux3u;&TBXS5d4B58%`i^ z6kLsVxOO4|1YoHlR`Wu+q~V;X*b%;QJ=3fpy*T4MHAkHM=y*b6HMO_EGCr;xap)^> zKgARe5Nz~SGod44HhUCeVqtBVtDmcsO;EsoQ4WOAbN&?Pe;=gHFqZQC{X6pJ@1c+ysd$IBGd-HUl+bk4r^; z>W!mXA^iwKYX3RQTtC}y+3whns19??B@E2M?=337saOA8`-=naQ#LT%nb&1t_>LYS zcqo(lp4?Lqi$7Fzr&)GvdzCRniiH7xhi&KEuBd*1x;Xq{bU8R}BHY80-u}skcOZn7!kKCXEHTl^LF^p)M+P3SJ^`KrpI6SW zD9=7l^$~CX++W(0rIu_CUKH~!=vBX3)&PoLF&~+Pr3A84ti)LyUr|p;nCi!KazKJk z)0PJTMO%I=KXiA6UO5%BX7CHn+FliZ@IPU}+!q@E`cCu~`vXC-KXdl3Z}NFyN@^5} z!-N+Oj9nn`+#vBS#}DIG+BqQ~3%={@yA)v6SHwIrkj(9pC?8FSQJePxR@J}HdeL!# z1e7SQ_+JO%;G(1T5m4s-Tg)?V!;HB>#qlzpg#AIJV=^#IV0uv{FMPp&x-59f!nqJ0 z2>`$X5Dq5}PBAbb*uJl9T&2aoGV2B{>wa9F{sR=z{pSd52Hlb*m#`*6*!g=m&@RIL z#CfRx!0Sh=ObOXPPlg|N)F?*-54p4}0l~L1QKWAcn(yg(90NDrA{wz$zMr}rV0$pJ z9zcpt1|ocRFizB@*sn{*?o-b77MnRx^j}blE;cdx&%1KjYIfl38-5BnCnyC~R`h8U zZPd;97XU)?Bhw-n7W(Zs#~@NJ?Fq%y?_8yC$Ce2X(;gf&eFP)z{#-mr01%y45a?u$ ztYc*;N^wS*efrxqka>=PE?!d*4Jz<+1yUFb_TUNg<)N6dGV-Xr{c$wnCaFLSW*RJv zV*^SkyZvs&%}*`wjw&`9SiIcp2GC8>rzy|_(0wiN(*rLa_5q1bo5SV(mY}OQL8frC zfg`3_?i5K`!2IB#Wdh6?YHo^HAe*Ycd9nA$JApf%t;I26o6KjZ1x-Ep;njPKhI)K% z3*D=fcJvbQLQ#cM2RgMwhdqynrOU$yL+PN9yItzEZiKLT0CF%7SLUYv?q*Xl z&>w~pYBTZU>SigvBA_7F^Kj+8nv; z)PWKyBC4%Q*0Rh)ihk{Hf{JpI=<%}D%Umg%1ZR`iUB($9JA^lf`ZX>aq>>s@I@&J< z7hV52(~p8r36*?ipEE+JWT5Log1!pVS!|0n)0o&yp4TBcV2j5_3A?q%d4`VwNWjADzqa<0fClJ(k;&eI^g~#$_?=GbrV+S^x{ePYfQI(Kew_EwHcf4*p# zIGd`UwZ%$Fsx5j4cpsa&@Ni78rWds-UVRG#KGC~h=JgL+p#A?=l?yqjL>ePe_&^}? z48A79Rs1QKTrc{Q_fQYLQQC6MUk;i7nL)+*5ay!&7~6XGIi{rX_WDB=OH;q|kzhH6 z=`W=BG7d<%uu&{R87l@X79^Q4Ua6hDkOV?Z7+FT(s87}IB%+o2iczoHT<<`%eRk0M z{Cw$9?~~3Q^RmyIH&K_#=}&{1O;7Xay#4oI5^k4O-qb}MzSN7^xru(hp^XU8Ay5V1 z?f7O4cyV}Uo=CY&MW_rYN@1xE(3u`U{ZNxCcS^s9J%e@U*btF=eov4^d)~qsM`E4v z{x$m`gP!KDtg4;j^PbSZZ=@$Zrdk|rRcWyQRD_hsTT~1tpW-o=omv+aM)#&H%446J zCd2jnj>N=Vb=S%8K7tQle8_{;{P0gM!-fn|v{}r$>aFFrj#dnM)a)k-ek4)~`b3L%EqqJ1Z^*NIKS2U*4Uv4(LORdcg1Uiyc?OHtdwP?V; zwfi5pm${R(?%_i2UtO60hb#(D-Ax_y6ah|i6rxycWf}2jhE7yXL zzf*X3mxpO<2)zAl3qsV@A-NJ3s=}*hbF*>d0mp9j9A{|aNlqZjOE0$j>Sz4ohmP|?-_ zoK&cbCD8nFN08lz9 zMXNIU4)phL^w0$)1)^Z6NJ~qrQeyv!{T>}goO1lf<|H(x`d775XuTfe>vzsxk#!UWZixV&t@+D!_g%$@B#O4velGLF zg(d-$F3cu?Qxo|4`6K)k0MrNc96~CdpMR5l;1Hmj1;{Lpqn5Lnj#GfcN!2V+Fn6lW zeEer2oB9vGwm?||81&6>rWmVRg~Dq@oy~N=UBe>l-c%WQvcw-S-Zgj4`8Ly{U_je_ z0>mkMrEe)$aqUA(s@5!u?p*0SJ?D>%3(;8R-sXK%r?=zrTVLMa`h(`lM{Lf)onpW( z;ZtGDc1@U=nCRsVxKW)2$a?tiAJfdz&rtTdB1p^uiCXg6uxN#}r`?-%0Vo``31A^{ z-vH*I-#0clUt!jycqS63fTFD3-0J>W_nZRu#Z>0wq%93}B=z#u1J)_w zLT6vN$_ZQCxux@kX;mO5;JxNazd{A)sBZ->d#B!i4klTcHtUM?GTW~}pk&q1mbo1c zud97Ob8j!9vn+8@jl|rG+v;JkW!V9z%}?y2_a$!Q?L%J_+D9blaiJag;-AKuT?Pc_ zZT}b#c+OER$`QtU@*nU%+aV z-es#V>$OXiIpr)gjAkPUhv&Pc!6RVtW8Z=f2xnfU?5I^a@IpIY-9#GQGjgDd`&`ZaM=B2AnU8>6SShk~e)C0^=Wfp&UFsDS#$InCYm0L)PhNz3$69Lv_}> zy(`--mEv%_;#+tl5$h~5i^_T)j76|Bfrw0$pv1<7nU_uchD)rbSg>cK^SbDP1N!N< z!U(u9ejRuMRj~>rsj^VeRKWA%TdqInKfMM62f#|3ipys>@}9mwu8$iDO6}(A5)$|k zhD*sft?#0)@NxbfrmDj4u`o_#Zd4$1_(7CX{KB0Dq#ohEZ=eDhk8L7Onc({X&xf(p zQ6{dIFtEUO3Guhs2*l6hlwFUO+uH#DI)@^Lcf@dMhtxSA~I`RP9=V`Jy+;y2mW85+X+Jux)v27Q+^%uzW&y@#L-v+=Gc{GRlkZB)ncfO&b7BGNsP>zZn3PyhKkWAf@52!h^ z)Bi+Cu7(Y&V~cr4TFNrttztb#dJ7 z@;%VzWXO1tTe>SRglRz%I!T{yq-i2Xx?zuN-Tcy@PWd2l>O6(6-@E@yv|W*#VO>(~ zM=rg(r!lQ<73Kxey3%Gfo+Fk4!P>;XbNZih+YQm{4S41~vbf~E083`0wyJE(YiJbnPh^tbd_Q2#*vK}|-tFn*<(gyW1my(SM|Mhh{x&`+i zw%_}NC>T-ZL5YH1)F^o!k>J@DQALpk&ct{fP<)o}leId`q+GW~ft~OKj>DT~nd1n@ zI^m4;wb42{!`@^4Lbi;(jCA%Bpf_1%J|C6j$%sPc$+$n7wYg1cx19kBH4fVJl`Gq} zS5uq+#7$aTou+#4-kD%e5nA_q{ADwQ!zFr~iQ0DMa;)dB0Q2Weo86kLI}B<@5ain@ zx^@B@kqP4@Yk7ewo+&@<#LA_WIvPNQp&NPKO!H0rYT^BtFJFA`uqA4P>k|Ev+^&GI z^89wt+Y@;5QVr@e^u)@2_foH5+ldBEO>Lg?>`fOC(ahJplwe>HOcqiA0dE ztHZ!X_oxrgvXPj0!tWMS7~=z@`Jo@6470Gk ze&b1SM+JS-vu`qQE#`n?8FIf2E6U^ggnrZvSl|wZ^@N2sbaR2CV8MA4*?|7Y`8N_q zVe6}1deGB#kK`b-)C^VE=fQ`r?Ehu~9EHfenoAWSkCU0~n=m zYi1+>35^8`;6S{$mU?18_xb)LHXsQQ1EmKP=r094J!~!O7`P?30IuW~(kIvq%3`@_ z0~#}*X#?(^WYb#U`Eu2NH5F+{8!2C%`;+S^aeIl_(rRBW2Lu3*FTX)ng?RK0z#G{s z`$dXvta?6{4=_g3hZ2WE-Sq;+#BMecLtAdKGPx#KW~36Z&f=ntWD zOD(ZVyzM8+lIzI6pV>z*kJ*Mjf@=9p($k8YW-9afeul;NM3m<4gl3%&_3;6Ni%jL= zKFebbRz>&K%Zl&De?4B+d-`E`*>+H*8xh?hT9I`buFU3RGC7@E>ug~ywVs*3e~hy_ z1TXNdy*?AoGbUFP8@|8mYRm24S<@BpF(7X;-Fn<|X#e{ZXuG)D^V`y|?pM2*fJ0Qr zR&^e)>hzSkKv_%J0@%mFW#mJr;md~%y)k+FOd=nPIDLT)QegN>muHAfZ7jZ+vCDrk zE?&%j*VpyRZ=<8B3YmKw-&YZ#4`z$2T)9xJ!X{3708F{BPI5J8uF>g*1|2GnX}%6t zwSCJ^GR15Jl8R{Jzie6)_XiFET>k`Vt+~M7!d7j%Q3C#O5vSqT+Z3m3q!lJnGCsiG z1epWC9v3nNX6O+gUhaZJ<$TVckxKNjz;LgzzVO=&^u||xae{UwJj3pjTACT5moW@1 z{mOJrx>M!HGLh*o5Nc0Ef2m;eOm2FGCXQ4x@EI*2z;gz69hkhEt#_+vy%e~6XFB&+ zrxlM_N`lv{UxR*TP`3h!$TTPVE!1}u{I9ophs4pl?hsHg{g}G|DkKv=c)yTE(#b8i z812@d06z3-$ie@8kLLj~+1$iC*@Di&Gw(g+vIeyevN`!zg?LD0_!5kvoBy<#q@DN7 zH1<<+(QnwG9bY~>A4KlCtJ*q#K_-`E9iCty)^rpF z9{z5bZZr6hVmoTcSDT_JXfmwd@v#v{CRT zfgGifEM2gxFK4d=Q!4qM9xof((_aTA6FJantM_ub#5YA&2GTh9GHfkFF;(BkDpJ6M zw%0md`}7u8r-kJeerU4Q5i^?G)=~Oou6JYMCdpA2rR(Ie`%*RL=HT`7j&0^g|sgVKN#!Qwm}r!A@r8#S2{{f!)tuC zizcrCk|lRr z`z(nA-YO&TN#t4OB-dDhzPVa6WdjGY6knJdJ{koyHc0EMF~i`Kf%|MXr2b?a*wBRB zNgmHcy1Tt{n=|j&RBlbrQhdGH*v`VBTNMdQD)4^5=+UHwM;qc3p(53>WjLjDiFerj zRz9;a)os1BW;kKdeZCCe!av3Lxpyyt8QM58v%7`@R#8iPy3ph#g3lxxREt(3Zr9_2 z;$BKw8@4b}8KSx0Y(=t%he{xe2i@>gwBi-6qGLE8D3`5&sIMSYX~D*{;S80X62fGF zp~X^)M6k$t4nvM0SA6W_7&hU|ywlCG^YsETgJp=?NmU4+*|;Ef##eHwr8Ju;@g*5V zCjam6z&4jx0d6gZJkX8-`9_H)Mm}?1Y2HdhPe*fqk{fuD+@aviBz#Mtnqt}c^FzttT7Uj; z3^-D}nk(5yvi`4TPNjd>5z*j&$w;4H-4&%(MiS9?hGG=;pPvOZe0lX`L1bR#R5J8+ z{~zw}#6|UL?%?QNp*+S-qo$EKVI8|YuPcaQoDdH8tEWnid=&{ckEg@|IsMxX<9^bX z55#^iSsozh=&)-53LX(Vi<+Gv5IdIY_+?g9b=BNBZs~x=N2d2ks~|l*HN%`y;SV^{ zN+pq?a^*D4z*rgvrH_qGm6)ShK>uV)gDvKNOiVp`a+~h%&qk1luS@I+_wya*~#3yP3~wEo}MuHndI;w0*C!yDfc>H!_ru z*L}M8c!ZNfq4WLN?I|m_Hvx~PP)|Dj#L<&QcifKw*DQ!A(gC}}^e?w!Y(n@n ziphD2(L+B)Ga;v~p>rP#0u$lzyiA(XHDnoZV!=0+l8v}O%YD+f)Hr1VK_-6aWB&O3 z^@dXRZ5Dr0<5sdv2fu3Bx=>cgPuNe_+IGNz*0%y*%6So7TZT zyUM6(d0kebCv49CUfXr6F`-e! zdb1$|m_pN$(8c)J%m4cIC(Z@Z!T>Hep>cI&>KBc~k3=D(-+RYGmgdL>lkc}zS$dQP zRl3f1znYBWTexgAdoW4FBJK1JYM$q||IHe;mxhLhqcVz)a>I=KtQsNnX?bks&Ol!? zG+1T|PMl&JrJnEgR}q3qR1?|mkuz&4Z4Ei^)PmW7pim};Hh*A8CTBpgJ+X%mls}w3 z?19Nc0#VAVnx#}ORyq!9n^QB_Z?n!Oxf;GzIymM@sI^(vAdn$;5?<*Qs|_2JI0 z6})D#!+-V`V=w4pG^KKck-gruyL6t&%@HRSU_yY+J$+%Kh23YWNmwkIZi6Mhou(r{r{LJK@>oGkDcKsyvN6?MSmw z*2v{#?BI;$Dv|W80`UhH^Cv{Jlt1@IU;*{Mf9i1kFBlq1#VeCB#g+j7wXOR#zna1! z7SJHSE5^}M#|)uFxReM(lD}8-efCu)-Hh`JMG`-#22pK!cc4mW;P9OaHR@)-Fx0-F z8Vi^a>*nD3!_aiNR0#QxrPmMxYl651g+=3&uEb|ufyA9JGyI(*a)>9@2IlOk% z)vuIbCO*}HK&aQC9aIiw&?OIJX-g&um+kBaY)tV_p|)hRwc8se}kbY6i+0E}9TX`$evKGBbhbe8uNI zHIBdQgMG7Hdn+(sh_olPr&QJs81YWy2XqMWKUCc`1I~T9`5#WhIx1N1r+6P$*>!ac zj86@F15QZhs_K7@!ammx%@bas8@$}5_)*%I2uAm%XPTdmzH(rPu09C9jLb(4pt@Za zQ4i`h0ADzAl-V+2xMi|qh&HhTlM%_J3xOr3jr@*^$rH8Jn_t%b5RMYgq=)7QV=+@D z5zNRs+JdAtEH-`Cv6dd6;f#oe+1Qolt9&zA!fQj!5$Rn57f%o^>2oRP>t0n==BbtK ziwsm257XzqN3B;;Lg?KKq+&lkVKbe*x?;CG1w$hgb(LSw@Sl&!<&cP7!jP+C;zSeF zKv*kxfSOe1pmc5UV|N{y>bwr`Czs`+kAslJxAr&>sC9VjtAN@Y;|z=%wavI|TcQft z!9ka9p%tX#|8D2mK>u_9N|1M)1$2L?Al*1pb@eOHf9Wbt?*`b2ETe@*>|HqC!P;^Q zi3#PFC(_;`tas+V&+H89a$|s{nxaTmf{g>7q5##adFypx%P;Qv*9~=_#*(5Ck3NT? z2}rH!GC~<#C6!7b7`k}}HI+(4Baz>JhmNfDBfeOW^pdou{$+=pP{}}oN-tUn;AXP4 zn45w0l{>3MGBco{gt{A=lEgA9(+Ew|qwOX#9YG;#jl(bv3bfaHrioT!!OoD_iU9OoQ8Ol#czz?zm3X6BDqP-Tfm=z7eoyy6e0CAZs=? z$5Sjrfjq`2XS;WavvIJ=gc^%^<Z)Y5*e@qn}lfiOYW#s&akGWhN*Gxu@DU2 zM0~=CxX>c6Nx+B8v^%8Hb~&_5`)1p2Ll$T6nqu!>|0ZYB6ku5U+`52Qq4MuJ=$!sq<3}9TfT=nBMPO zN`mRU7^_xI@fbiGo2bcjmrNjA&M4)dG6IZfPQkK7UuNmi-;D`A2I6A-CqRo|f} zXWAB>)!i0d`dIHj6ji}OE^8v7t+mt%2P9@+)FVdU=rn!#PYAhcYu9Wu`lIL0p2O9? z)BK~>e&&Ww&u(Kc@6|U6 zx$n8B?OqSV0vJ3-mUN0!J;2PzNo{mJC+Zl`D&-S7B*J%@YTu>0Z%m<``qxtoAd38! zWR^baLIKw0)Q73E1VvGJ((U$*r}vD50gJHajea=cYa+;6)i1rO*R)H8zLA;&whPOh zRP@|fnCn%Hiev-Ma9f*<@Q1vK9)6T;+r(SCi-6C)tLKpvt%6VSEy(#Kc-Yi}MOfMEujy zt-0G4nU#g6w!}k|^l+0S>Aufu)vHoum<&~L#$3=-q|F~`$i65853`C%2n)kZiif#` z47=UUgc?x_B1%yJ?)-#7T8usO9sg|e{E@Dxr3=OY3drzHVj#=nt=5?4$XgoYacDhv zAQ?z>8X$2g$ARh`HZj9Y{i?0w`j{BxxJY0jUraTg8WVxCc;A!VO^c&Db{<4W{WV;R zLeyu)N1(-^R3tswm5A)rvOLEbb|OZzv`+n=0=kR9Ki&RO#cb=W*XvXT4C$cw2u`gC zc*H{6+zWzfWEzXFnx{(FTX(=#JvgO~yvwAt&t%*d9homX%`DrGxf6Jiot&5`i8)Sy zRlRy$pG7{9v+PWP7cF#z;&UnCu~w1`+E-IfV)Oo9^8R*p+lXVm_k?8g=yRV&`tnR(0PadH zH|iVOt7y(=(1g{tru%Xj3l09(5kazJI=AzrUwt5xpA1l7|+1nAgT_yQ&}XAth^uh1Wy0@WMZ`idlDR za@hHP@3c^_E>L7c+e>FaMb<>u(P=nGA8ljW= zHwi7fcuuQ2`*NYvtM;cXB>~R#d#{uxhF*F*mX)`TyqX2Gucb`<^AkGEW)k3C4g{C? zSR;=f&mO)@Rt{jkB)L!@&B=mYOc9r9Bgf!*Vbvl1Cj1Uh?9&o99Rg^g2i#2vP)A_x zRpF25?HgF<;9YTqq*7e((e%`rS7I$?$os(USyor3%rRbc?eVvGk3MYpptAhT`k$T! zX4I53$oijqS_eWQMb;$9u(Y2|X!~ovC~5PbICe+D0utf;;C@(G*0ODR`Ok!U7R4ve z;rRkoTDP7T-!>XkyUBNB+#>Qrd{%2y=PcCB0z@P zIixjDG{&7K`e+woZRI&oDng$5`ynS+@{Jn+oIL4lKL2|hH(ebxQY(n7X47<~jw z4~D6f#}u=)J>{wG;ObIqOqcv#H0(mte6CXYoXyLwpYSiN1|K3kmi(rCO*y=L^ZdKH(taNhlOLnmQ$eNW zv$}A~IQLkqex#v-_P`$8MTf1zL2mi}4-78nQYvI!SfpV?pvv7ZvEg`)i4~;ezjllD zTrH`xPhwuDos`ln;)+Rx!8w7o8?mK9tvUH(@uT(wouKl-bVhvHKO~b1;Mwgs=$mz6 z$xm&TFG4+!KL)*y^ba_~^iBP;PG@kOVT+Xej?Og@Ye=R0XO2$w7cn`4ZIHGj{cM+f z3Exri9bhs#KSmPU4J-;!sXVN7Z`Y1dUS!Ilb6JyFc=htkdzv@|Z#y>`f$*9}jU|>E z$$T;_3?s8^{p_>vQ8MQ#7F0*>c5J6?6Pc~9xW1H0&p9>UO@;t!eS2#bk_ z6qU(DN$ts>JDK}f91w)t6+kYwDZ%z_Z#1F~osh#qB~^LKJg6reFCdZx1*R6x=1q_! zPh&AW&y+C8MLPJ7i%ejIQofUz{3Z?CFG5-sh1^1PhJGF^nXx--Y39(@jXrv97I+o5R0yx>K`eUrh}KTK1E|2e$z(>{!ub)&widF5_rj|5KNA!|%6Y zERvJ{F3;DE9Pd0uA5$!ESICZS?*ndw^_FEt$spe!1GM}s)mKT61?f}Ns_^(sL2lyE z6{1so9Ny*q-iGE9=OSKW3Y6((mPgor@tH0dB3OGud;>&jiltKvx2wE6o})o3f3fA0 z0zbK40y8-^oFFssyIr+8iOl^~@l{(f92Sx>u(f1^p-+DC5w2JK5fjIc#bDggQ<6k- zI>O$aklehm2Juyd43G@BpQ_^=+|MuU5ogr%mY*`IYwY;5so# zghbc_8b&0Wimiq72gw<(SA!a9#dk+&^3b14!Q4UpZs9eeDm%(xXI;^jO(7h?4nMAE z|C4IQ6=r~?1}%Jja<@N^6)fMKsmI)mqn6K5tHVIsgu(bQqJ!0|mF!tut5x@XHkYNE z61N>e<*h*Ny~DZne`Cy(O|=Ht6s#QPPNNZOprovb3{%+jZ_mUj~Ibh{@wF#cI3z z4{AKQpLsXAJ)wf-=pJc|6A&X@HUiq6RBv-NRDSO}CEpwh8N67yc$b;s!ZtX}4a3Aa zn-+RJf2kc&zG>KQ(|bqQZ*}vdWTh*r_kjxKE>DJxp4+gn;I7oj@o(*xZ@kFgsOkdZ$ z*W;Mh{+W#(9j79Td)|@B9^3OZvb(8Zc8CKkbYC^;u;>}Owk=_mG3iqU#^FWqi;J+I zso#@(Yu{&W{>fnfE$+2}1%(o+p@<$PnZQ&jLbl>OyZAw*w){af3e0vF?6*gU z#_g-rbfIqZ*x6I~$2m8b_lGV9BHKKE4)c^M<%q4v$PYgF*hP56wov&k=1urn4>t#% zZ%?T^Bp#lhvqqGJIK0}nYaAE|SL#c5PirFk)4cGqEH>-x2FcRWNSuUlCh+3$BwaKX zQJ>T$oAAOnd4TUD*)X-GGhgV>Io`D+(YuCo_}RYV>iC}O72WKb#7v20(6l`JH73rK zj?wZdEAtv_Qur~~MmNz&%3C-q1ZOfEjK$ner!x8KJc7k0Q`uF3U?CD(muS88303D>2qiJdR7ebWk7Xf!PiUjd}6;bu$eIpc%9mzH*b( zGA}m|95Pn6ZuBHyq$>`sip2XE7>meP_(IU=V6%I@y{}m~V)v{3zQtkCZY7qPESi%v zI*YogR<|b|IbY_GWWZOr@B@B6Z3V5Ulz87BsCdUBeL>_mqxgQ$&vH((QMaUBUgH|@bF#U58DG01wG%w}E zA=^w=CZNzzCyIVRAQQFtfTPbJ+n&qwSgPXS$UZ?DJ7swYldEavN!pIU%kpQ01p^)$ zy)Sb~ww~J>%9nOQ6-}W+m;@+UmaHk)i)$W>SeFFr-^smU8l7v2&tMKYuPIMHoJKWA z{rPf1CrJ+{J;s%^+O{dm;9g(gxAx*?;U`1Y7cY0s3UrCPF{k{fU|>AL7o2+eH;4fu z+iG2yq5ZCuIO%Kj=bcSwzOJLzF@VtJ=>J}?iKX_YTh4Io z%^k8UUQBb%RtO`kRJ(F40MBpe;R1A;_mqAK}j6L&Q%%@mwuxnLKU9RCC0&o@1S_5%QJe5zVzr*iirRPs{0TNFMZMrpY^ zwT6@86CDhc=QQG`lyfCPeeYjC0yUX&FTSE-{sdRY`;p~8R|l77jGC*+n30rk zdeu;oVfpB#Ix7TER_x^&*gTr_w6Xj20Gusod~~by!ATU!W#O>U4KH6Lmuq9-h7He! z=t#dbH+*BCD_r<3D%jK(3dD?WAM+*z93uHpXo}3OlHJ@Cu#Z2=x9<+o`u}NKU#H;u zG(>qn>`;p=Ton#0(iU0Ue+?+t{~z;M^IY#KiWh#s^y&`6*01b7WqEgm&iIdDn1K3W~HP zkgm;1CF+=deg-B;Zi}9%A)TGm>VG%aZ5hxZ#kGD||64(^MEd|3HZ?Z&6tm2QS|`)< zJF4Rb5XAO(IQi;9(|Vkm<&Ur0o2_N_o7Ab|?uF-PfMRB*F!gm&oGl>GLj&Fc@Cm5F zAovJx_=5c3-~R!FjLj67Y`OIG^zKe)d+%dBG|SX6Hs%4Wcg9_iJYlL`S+hPiHuj(s zCL zC2kBPtEs4j_Pub)FD@P=OnSRF%USsEA3XFB3<*QN_o~nTTZffVQv{bnjm~&4?Y~Jq zqk}-k85_jVN-O(s_D+>0_?cDf-zQrByQcF$AM<~`XbzT3uV25WK1XXdzH8!l%|;82 zIzYzJvGl;9dgq1kx{J4_kSxO&#B(8A+i{WSSZd>|3&o521_qos6ukEO4wKz?GRQx7 zYl!wkZZ6aIWJ#Kns4D%;Q!Wbs+4;t=duOuQfg7u%I5Aq|6IY_O(!O2g`{l_I$4$_ zA|4UT{IBnBa}T^d^}Uxkbs#B%IEkD;GOYy%C@yqvetG}-uxz;1NjmBsLSB`U+G_Ki z=`JF8Bf0e|zv}e_SKhS_GEUp`bmhNVCLef9k&)0G_){#k@RV~JQOAU(>wS+3@6*?J z&IGP)-{(5G)!uA%jwyQVYaoOC+!6AF85pEq0k1>Qd0B>&gQNHN4%7M7Or>?2!1h9; z2g}^n)Yr}FO~i7|dP`tecD+gRetA)0VO*+^wJLaK&#$Yit4EHR&l-cixieKJdi~)M zS|a)In_xp~7&zKysIF#0_tFDrOc{c{6V+lBE#d_)NH-i3=F?{rvc5lbM8K^&g75-{ zlarI#K?1uizQ%|rjqvu9HNr1{P==q#)+^3}am5Lh29Uz@AaQ^5Y=E!TNHKykb{0pc zZk81SiW%Gflu(MM<58mXvIM`#i&0^xX3jC*eP@y?@E$TbK0dzhh2s7UVBHJA_#jbW z3?!VU2x_wHxX_{UAs9vr>lqK^!O&@gdc*=Mfh0o`+BGN?k^C&fBe|&6=;5vydXZJOf_y`*Vu1ry=T3DIX-%;dKMte zDV--aZ-zA>eRD+Mb-7n-^>Z}G6%e1I!3EY$kwApPcX`c63C3DU5OBX)@DX2J+5afq z^oLK|-OzVeQ{(sU%yY+vZMKd!Ax54Lh-}PkLTLSJ7`KRoUDtoq6hcr!gl8=r&#B?CeXJ2C^_Z9mJtmOz0bSEk)*_ zAK(buPb1jQjDp))!YpP|M1aJA>)n3F1;Yw10%}P*Sbe3*Yq(Baou6z}4Dgc3p%D`$ zSq&uh1yY#UlwJZHi3g7X>}hcm3NPIo7=MXb|4Qt z@ut_EUo5%w&{AQ!0kR6jM~_&6+cEN(ZrlHU&fA%=r_Vf0sgETuG&zjiu!HTQ9PlMX z{&fjw-C>C*>Ji4H7oXwz+t#7PNrPskbV<&2cg6{^V}a#SApoxR|C)jcWPNLE061T$ z@<4erby_)wrV7lyxW`M{{|0z}FeCnJW0y|L+RM^Ux9mLJO2x;0E54{8&@+ z*qF{ZJ2u;B4L+^7@|{HomL~Bv4s&d@H2mivn;TC%pH>$qR|6CftY$>`3%dW(b>neZ zr;V8gB{=$*RkQFPFW(!t#wN)A%kc35URZKR6mlBvKFN<)K2qeiNVfuwv{&6>M+am1 z;hb`^!gYv>S3}eWbxOlYMI&q7(GURFVT+#R|8Q-OX10nmk+$2n@>0n|~-r zk#3bGGVUeyi_mVW6&c-Po4FvzF5SQukz&6CUi_QZoPtDOgIUd2rF(2vdVvq(c}oVY zf>*^ps9d;BxhEa}lG`3@7&NLZXVYXWriP3UG^#i_M7`lNVzy9ym9hn0`81-2LA^WZ zb38dP`c}=dXH_gN`T3ee8Q1HV15*VK#zU8PPES^@&P?f=(YC%8c`IGU z+s*fN?j4zZyy(;PXR^-pdX*jmj&ss`j@RQgKMf7&3#(Y~jLAahyKX_=d>p8JG!uhg zhKS9s+HKe=1-=IJXYmmXdS61#XYPD*Dv$q6TWow+q8EnWpKlxM2lJ3MDUmV}ql5m} z^e(rVCQZpBh4Zu4TiO@93tM>*?W=nzE9~@szhj$jE_ROQna`+uyFmxx^VzTea?s3C zJ0gAC#7ONQAg8kQfPl{EzZnQ`rS#QnbKqAvEcq;-5CkT+Pq>Wjj$`NXS;bx%G4s; z8=ti-kS~~~hAk*YS9Ra&8W?Y($*NdgH{7BW1Yva2LN>6qGa)X^?Z#`5N zQe1Zufp~HJP3v{Z;ocqD8+9w%zp)lA?Wp|zh1Ov0%qtqPUtjZl$;`S}K!taq{+6ul z;^PtvP$neJeldGVDwy8i)7SWzMj`}%NxC!Sbyz4asqapf-=9=%opbVS0x^hhVGRKwX5JJwzs?JMX%0!ZTZtdqCNPj@6P@GV@5u*JR|pizyzRcT&wFx31d)E-j| zu<EG208!KKGfKI+?D&YT#PrKFY(kxx1^mRYAIa;0jfcNt4wQP-jE0oR@5nenG zt&MOtc~qxJkiZd4cU4%MV2 zDPaOWB=H+Dc~KK$CbzA8d#;k`(2S7~L+-FCO)?BSjY=-fhQ{Hu!*3iX43rPK#-*cM zcNLAg)q+Ny7Dptsy0r_+pXFhe$HD^Lx`@d`Qo4dQqQ&5%_qrZIV11L*XhTyB`E&s? zsjo%e^`d&tm4875=pAb*4uc9HpT#8Z=yD(>;Tn{aJN62#bv13y3^I4eyH;~JC{rTG zMvu(svOmijl~1qbpwdh7Hc^=4I1UOUR09$GUv7cjK{bHLD2tKbDPJ+P^le-HnHL?k z)liR}ol)d_nbaxp;)tJ_Fb5P28>D2@DsI0YjXjcoy^7#e+CdX;b z!SA^^7R)YSmQv71&pEX&G>!NdRdt+>f|%u&?&s8x-Y&bYMT&kB#?2dWY~0(FDB11+ zwG*7WQ4Y31EhLB16J|&RV_m(2>s(+m0n<~{Z`6J6o+n=gH`xc}7l*t9v_pJIRJc-> z{n{AhS}A=tNlym+F=DtDN8SnX840F+#iKk*aa!*G)K4k%i=l#$N-YYENd$Li0KX)$ zQBg0s{_;zw;&}E?kEh7j98Hh88(=-lr76uqF6T{9A|& z2`~nu^V~MGLVlsh3v4%TFgMhXO?avuTfO$uT$_-2 zP-DQm<#3c_n@6Swt$E$5RZz0W`_LQl{0OrBQj2#$q$YXf1>PeQibKi#B~PzkJ>PY- z7gRyV>h;u{)T@pQwQ9!G$6U;_@P0@4?xrb>xZZE`e!*wA;4AEz@^K?hTRq2+iCVjh z&V#ux1Ehew-gyjzIptttx}9mhyLgT~aCbY`1`fT_C%lx4Sx7H*aymU0oKfHr?r@w{ z-oNx>ryiG*Mh2{Egy_Z~#GqXfcJp0Xw>60k^XpDcObIwv9n7D%z;qKOy{N~b{r2%v z7hbHne83~ZcaE%jK*=y+9e9Fc$Dyx{rnY!j(e52Vk{Kq3L$)c$eaF89G=%y?pbSQ* zWLB6XAtv_^Xiot0!Jo>k>tYiDRFO^au+!=3$DqEI_0As<-D)6qHzlta${Hggz-RL> z7?tEvbiizc9R7!=z&Lwo!Rp?{wacO5Io`$-`zQ!O7)1xZz4tkDNR>%eeT%j#=eR69eXMDUpkAfUP|*k8ni7QPxPyi%Rg8($s^N`v-a{rh?X_Bvor#zPzPPh z6fpB9=(;fPziTxjOb{mJt<6ZKCKz@OcqmVN_!Am99WDTq^>tE|#iAnKtP#`BmqE0G zI~}1%NPyz!F?FvSJuvg=1bb3})Up?Uagd?n92B@qaTZCCp49S?T4`KOlb<;ukt<`s zoF8a!EM<85n`csHRMs zuNDqLJbDvxv@5$#)Ej1s|FCt%xKk!jW|YBD#JX?pW=!dAY=2-@80&pc$>H7G%<_v% ztaD0^O^#iqvkRR+KOf@sy}?pb*{OEzi+QnX+3h!Mr>4|jq*f!cqDz72ZR|~2`iFk& z7fQYDg=|G(MP$UUI+j<_vgGoDe;_9b!_g-^Iu*OREqoeo3|n@9-4 z8$JFk3RX7-FSnnczSjuSyL{jONph{aVy}h~RzpY}#MNOPkEoQdh2d5*%-gHEK zBP17jt4u)deUto5MHJTFA7M4FX*3`PbF>%y9j<}j9#5LU(s$r#yWeu@=;k1giSzh@ z0jL&4^zJJd^2t65@9K?7+`tz_lQmMKDS``%5QLxi+rM0;|yen2cfJM zp4OPDmF8G#g6Iqp0qG^>Sijt~(iroXKg7_W&DagLe*^qHR9>4@+Y@t&Sy5WsNa_%#fo4xcse9OI_Al01{aco=dKU1b5(`?TDhF2~wJx1`n?n$M2x zYs#7o>H51Opr)_;K6#vc(yAL(2UAR)#or{J0j!e*YBHJRYT`LF*S{Yb?}J10ZadN> z#4?qv$}^UCX_d9=oI5b$-+Yu;1@%8zAuLn3p9c!&1a8v0FjyOxHZsd%JYT>N34(?b36VLHQGqai-)f1V$FSY{4%;ahu)BO>)m-VPi zp+30~*!GXt8AihP=~x~o{de$-5ROGGYb*V4{S1V-56|LP9Cli}o0IjHwLpHU_4=n0 zy_W;Q)D>p~{`9G<4?+Rp_2xknM5I=1MI)^*`WLJZDMNA+<=T$&^F`caGGu^H7T&jS zm^hXU-^C#IyrpBCyf?D?HbHF~bX~`N-QkT{XvT81(BAlrJadXmU99k_qO}a-w`-l= zAsf^#WX8R!TC9AfU8t`{rph~if}O1x5ZCcTa)9RLnxAxMWI8!y(-Bg{3h`f2-WU3p z8k9Z4_jQP)16t@XAq4rmthq>}Y8GQ9^-~Ho9?9{Sm*=7;sKxCf>GL0c{f7l02qUDD z?(6jA?_toyT%5I>D1>0)t3fd*aw{lJ;&r9!VUzdb)*V)*c9desKHZc%68i^whKeYX z@(J#OttC4`f-30WA(ATMyib zq;m+^6z`q!7>(g?8AYQ!3ruvH_Zs)sLj9!4TmuMue*larf9qe*7HW6Ql9y0EyrQITM!M@@G`{hQ|ApY02Be#b(PtcPuO!i0)cl{MG zMHZ?<-9NJYGo+|g4;hV!!cjtA443SqRXqU=3#a}qe2L|_6Ft%@#f>LqMM+FU@2-9@BPSh< zmZh_YE12+s+-M2~$8VPO#yCghs`vb1&5VMdE2q|v^M#QPkoY3kis%;1!qNsu-LOGV zQJ6*FB$o6p2@&}eYG$vzL)JiTKrodKSq+b8!D0X5P+fq_y%9mQ+0%$QXQeV1b{ms= zSjJmC^+%>Og57`SRID5;qln1KlMv~Px6IEQ9>)Tsr1r|svG^rx=rt_-_{il_fBu)|8%5Hda8&qXh5kr{VxHR{Z zDzlx7g5C-M{DA;%QkK6$4^DjA6_oG@9abJ%TGcZ&qY@-M;r25c@2T>J|A7I{3=3qN zI*p-;*q~xhih}^EtWv<1LzX|Nn%l&1mH{46{}>RmV{#;3+DOn5aC6W@6H2y#y9C1psd_Q^7dsz?T?Izf1@N==%CMn6nH6G?Jj#IY{zG-WoI(bLb?rinZCuU~lI zrd^?heJgEKrtHRE^BxQG9-jpiDVHmmSWesr?6TBO=k?8!Op(wd80Nt0DXqGW#NDbQ z-x_9%Jd1hnztU-;(Gsy^i|F?ipKI{+Q>uq~7t77JFR-gFc_->|tI(O&`!F2Z3+kPnV;6*d_5XnN9tf z%&1$X3Uo|ZMAi?`wevny9&v`+H(Khk{4jQ2*wN;rx{<9WI*3Uh5@SzFS3zz@-r%Vb zN+K2#dDRmap54ZZBUP?44f@?V%+rOK>!_g1db=|;hR|Jq50pLs8(~yyh!AUmh)r#u@;;%vPEuEEBs7#Jz#8j3?kRV>(F}A zdg4_Z4-w{Hm(!{+-s;5Svo0AY+uOM0TF*66i|#(2@zrpT*rWV>vG-Yr71!!I+p(O# zkHO1tTiFXFqjguBtC0mWLTFweZ$!tNZzX3;9$4f+T$l`+a`NW+c@&xE*WJ{hV>9}3 z6MUJZ&o?#q&dTkRKRfvEIjSO8K~RdFY2u&c7fDfqWW$Fwz(jev?Xzhex8#rzO4r66 zAu6CJFVD5~+^s=h3ZfzTlBJjpSm^P^P!Cb8G!tY%QTjsy^NnwV@v8Q&DULwts7tJx zwkI+8GJo6wl!Fi7 z2>j-KplJ>Yf{lpsX?NM1k~6>3fRAGxUdsOFWWIc`6&jdtKw+HJ)nP~X&gv$wWdLII z)m(RWp=yn(BZ?;!Y0mBWL2M;;vAVYl7w0h@@y>#B!M$0Z67ww{hka2xxM%%_Pk9r` z6IjCMr7N~kc?&#rBMJrR6<5fN2%K@Oc*0D*j#!l|)3ga&*2^5+Z6BqWu7#SBP3L7V z25nk-inX?R{QhsMRTNMpF+5Z%K~()zAgyODd5z@91LbF0S;o=C!8@)k;WQ}8_9M#~ zyJ?gZ^9Kxrz8Y2$d+Dh1n?|`t>K-Ovemevnc1)^^mbS7{S@@tG4dTy6HFoFihEDCu>$3_wIVmOWkeQsv=B=U66` zo80|W*g}jkFjARJtDC6@~oj+dO zPsb86eNJT&Catv>=hSl$qZW=jwZh1(6W}#eOCrJ&VO{4%Wn#~HsdjF<9&tn8-;0kk z*xEsy1WqmbElxZH&a}p+d^kS)EzO4wyKW6$tKalzW_;-z@%E!~3iH8N6Q8?i56d~; zMbfIfmzkcP@u{gZ-({~z)|*uwAs;Sn3E5t^R@w-vKslg?hgSk@9vko{S^dMqGd$!e z?i1*)+XUsg4|rka!J2N5ol@MwsgM0cYkdF<=-dffn8-bREct2AnS?POe>khUoDfa* zbsG|LQzgOC8D?~kR7->%;+7!s)Zl5VnOhKM4CzQuBr-&968IU!pQ`W#WwW$bLnLZ4 zd1vA7k%#D=X*R!-e&z=doi);s=4jZrz~3xux4G^HT?57sXp;e(_A%x^U-ok5_otFL z>WQl!$Pqpdjjts`XgO>78a+)FSoPwl|MSCMuBXW@>F6U%9Ub)<|6@{Kn&5O!0-U|P z0BS|tO@<{d)&K_%)+MH=5QO^4^`T!d*cqAPn!HN8F>4P>VQ5NoqGW2X#m4tU`^(l* zdf#^>hBo2N{+dL$JS>!9Nko;6CH9QykK$jN04Cfc1?{Z`-3gcY@;*27hets!WyYcl zTrge%R_mD8HW3ondf3hZ({oP-! z1{!*Oz^}I!(9&Ud$0Dpf)aD+dlO$A0%;4N96Q|yi_s~d{t+9)H zUf%0G<51#qPxdV|+2EZ3Tlv16S&1yBdY@IU)`Nqiq(B1DzGy8}f5`O`;c^c0oXy?0 z?%z5wLSnXuNU=g-IdJ}S!pk=q=EQa1QUaZ)SJKpK6Ka#3te!P~37a%PT?{Ai01t`H z2~RgAW9QRflK9uWNVe5EZ0#Tdxr7@76R7K{^tGY~>*~qU`2N+;gn7W^u%N%_cR6?+ z=%&?+F4oi|3RYyPn@h@i4w+50z9~&jo(V^fyyAC+RG@8plc^YeS7FbKfd}+q3 zZYdaH;}(SjUqZlxse#}Y5E*j=eH|8@*1;q|TFyr#oc;N!8+5E>?aPK2Pz91GMc7pL z%+J+59#hiV_KHE?%GQui^M}7hdT=BeI#EumCn~5p+W5ekFD3p2)?rTdA`>+#g0G)V4n+uE#i)xSKl+!C z5l*|c$*4yQb7s$9#52OhAWtdC@955VqAbZf^QSDhuoXHVN4(otH;1L!kAEw`11}vu zpq&}2JaRO%RB=zKln!W-y`g(Rf{EgW&;03{Z6=u#HB5H;msfElDTfN_&99n3m-p}G z1&Jn~?gyg5a1kS?5+0a3Q7?Wh4oGHe_@a6W?$)+RBGa~lc7ig>zC{S^kOxvg?RH2dK+d}3gHhsAmVd;xEe=2x}LVtVWdy^%pM*ZkVo_2w73eGP+f8KlSJS?V6 zLX6fN7s`2=4QB>WbVay1CjN$S2wQn!c_Gu@J7@M$T`y5AZh_5y3m<=GC&&UhLo`!cIB&KdArlOm7kwXDm zs#09t*>N*md4ys{i%U$(GrPi-`sbyHO>sCJ!;=*oo?mW30#{+`it!4KQ$iL4(%(9V zWyMQYK!a5~c9gs~xb`>CL?))GY^);HAGHQ{((pf?1JvVm$0~J8r%je|h{I>SSoIjo z7AjBP1uM&>pNpiF5hhul^ZKvJv$AE5h;EaS+CpX>-*s@k|6@>NFWlK%a@BnhJ>E+$W> zPe?`7GI|vmOY_J^+Iz&|?Kl0kV>{-xbWD={731O&xJrZDa1>QH*eBZ2!$j*iL8TLF zBJpQk_97?S_`7(GMl7?4>m@}l>O*_xO7Eq(=tcUrr9iN|X(~yEJ$*3FXO1qyIfC<~ zbMsZc928S@Hg`|Z%e;M9(9K~=+}U{_+d4poRC)I;D;d(fyIw@o#8?ni;duzLE3#l| z!|t@Y#CNl_A92dh7_mJ8+fXZahp?HZRUfakCWRav0u!35Y-kvL9V|3V7HT$h2f)lZ zpy}hweJnF~ni3;UEQg<ATdj z7|F~LBLc1`qlCDgya#5!?G(gl1)dOMtZT*8&kk2KWvpj`M>R8B0aUkZ66$XYG~_QP zV|*A~p=(s~7odQM={IAg(t;1fzr$K1+d(Sppo(OtZ`z;>E*9nIc_v~>W-{bzrbmU1pVG>1!s$fG z=jS`sf`}n(a1g<0P^oj#T*0Qfsrx22k!J3bZ(^LjUH-eYM9$XW@vWn2XI0;Z*@n`F z>Otcv1F%&n9Q^dHWAkK8J@PiNEdR&%N2W$E4J?m?`ipQh47N*4rh4Ct4D6)un2*3< zkydyX3phpcpX=*mebR}Odht|N9}xLmwxJSGqT}H-rol7zOBDhUh?Q(f7Hn3tBla0m zv7O>dUn%14Zlx^t?xJi9l8n^%`80t7-&M}+;hhS3Z7bh;6M{v=zuG_Hf!Sl(b?m4; z!J@-V+zgeG$5K1lEvcDhGQ$dI(xJa&du{a1?z1HkJw-hePt*|P)MkLd9gd`nHFeFg zD>COEJC!#6T@4qC8;@5hPFt&V2mJr*_2);lW}#oibqL!%xFrI`awK0i|CnTvYa>5y zyOcY(t^N9*X{zZ?f@tdIHZIXa5=VRni|C;WHg9}Jc3o8_ZT@jbBv8d_Rq-5s%_|mO zQ^6UDBs7fpiFH5Kga%nxdPhcSR0X9WsJJqZGzjq(Y*@rt^${)d`n^ZXbI%emh)ObT z?T}1nBdb{Ap9Oc_emCqg0~YntA|{fY<1Pz1(rapWi3%Nw5|SXmN{Y`&8a0nE$;t7m zlkMadb4yTb^AQQz_@`^LS6@-ZMW|3}K12s9=tTCC7i3dY-hM2OVYZ6*JEut;nE$1W z=bOJsPK3rmDpSWF_w#Tt_%1@8`ieRN%Hci#z)_)7%0caQwVr5l(_OfjJc9p!u>ZZn!4 z5!_={G%~c&EH3h$hO>pF(CP?J{@GWN4k}1rq5ab`_Ssy1%$PC*j5+v{KXJAHgChT%NGAQz3y5LX3i&*|=l}g*;F^hJ{O5K50+_*H3>9}zZM&m3 z9sj?7$zX*K|JUpOA^Y$C3V{I5mr3H$EdRe~`TzYD$XvJodfk6f(oEqX-o6(J^qM%jAdmPLB(htlemL-e_vELberkv%dVx5zr#|AUyJrm!_cg$8PMl=RD1r z_wo(iub#nFq5P_ZKb8>OTB|U=zjM8ep&gGTDU5w=RSiXN|xHt6WU$|55z5%krbwibtlhZc0 zR%BnerbHVX>kjH3%#6MKfXN;$+uoIy0JS3? zq1Y$Th}eHU`IMgi5TNSH7r*X691b+=4ELnp>7Jp*88?OX?0jDokKJDXnm090W7exK zue+M8p8!CkyZ@GGDb|IQtp&BGth+-g;2*me)1I?-^;~WtAO;l*!_v3DZVS~@Cfpw} z2ujZsF$Vx~`njT_f-bWJ6kIUF5%QS(HBn^M9)K=r)^^W!uEt^0>mD2+7^NRLB*K?u7wvzjhwLrWL>#s9y=9`E1>pn zdgnCdl9@hxuqJ*MVOW)Mmz8rqSKch0EWj%C>&ky9q2n&xR^GCd{wTewZ!V?f8Jp*r z&p>eB3DDbL+~wi@mOe*c`g{p!|KW78<2$r@kw!ZPl+56i*sF@SFYhAP7^_v_o2hfj zll;54U6eODbv2BCh@gi=>YOd>g1Eqh@?;z(`=ckbe`D@UELHhH$ZAp=aM}I z>+DpcH{YKIeLmZpdvW7zV=Z>uQidik(|C1yyv37MHomK3)x7aR%;!_qy@V*T@Gd{vq;YT_0J&i=p14!7AjS>4;vW00;^aH~ z=ODy%=#DjIFNvSSw2W z*Vn%zx`E!yA20^rc^oVnKNH{n%=WQ42Ob!MxC>DyR9y!&QxfnF>iT9g`ok%cL4^%v zS2wo}fXv444`e3q(EIAWD%pEYck(6YHgO-_ZZ+E1yxr#9a)RMx354R`UDKj9t9r*@ zunJ#`L1&i^{Sg&y$6rC=3`ju$A=ZPebtd6x`$Ya7Je1^}verdUDN#{vWJP^s-JchC zgYpMbLT4B01XOnSY`EF?7T^yIwY~!-NBV{(4(wf_j(!#y z>U;;UZv1CPlk*K_i{U?WPiZ&0w}U(XzxKW}uBm0+TM(p|00$L92_2<3Lq~cMqzHs2 z0YsW0QF;r#SAoz4=_n;M5u_MOK#E-Dy{puLO(}GlXrz0p=a!%jct-Otl{70L_FeVI%M+ zWun1jS;HPH@8uojHzzHdA|H#FjupRpfRoY0-S^tem9L3f*_EdF0quaYdvES@b(7z| zh&K8b7Gn8wNS6>qzby4jLUXSNKwKWYM4p+|{;YXhQ|aOYWd7HTh>fbhNJhM@b2KRS zc&H+PsZBqg$wgi^CYD&!CD!u`SSf?t=`_mMb-p_eziO{7<`cel4a}XD0&Zy4>uOg3%2@LvH}d z9$Rh0cXhpHg22X}Q@+7vS~^-+yMj1_tWmO_%Xqo_pbI-S0vMl2oruMZ-R^z=O+p{{ z-W-6fBpn%(C$G>y=i(smvH@~a!jUb28s5FugS=YaI*+gEzWS1QHQ&P4ck3lS!LXq| zi4$96M%(;^<4Cdg`NW~*IYzEmnxd!76?xP;fN756IqQB za{4`ia@*_ABn@%{PRpu)vGa5vu18;k@P6@rdUZS>(VGZS>dXDr%xy64)>n46~mo2|pPDG80 z5}e`j?3*4mZ?)M0OEzv1>}_QLe50INYh*)>-HO9*?NiIlt8+(pr1!I@eXiBl3d>#z zFMqd)_OSp*P`kl6*@X5w-c^PpFbc@`J-_&O{1+Q{JDP>JqEA~iVfk{3lX8DO)N3LY z*xG}4!}7zZ-zeIXYH|sW@r;E=fds-8V8#B@PhGOS6=YSGP49asMT z#r*>5eiZGQpLJ!6)#+o~ol{ES)!}_$hN+TXu6UnZ%MChDjMWssN>=&ke0gi44KN7s zOyfn2qL*8no)rQQ4pugg=@2K?5pBQLJE$Xu<{v&*IBT!OP{A# zgXsmWAl!H=1P@Q90UBtiIqUnoNyd1PPDVIYgY1fQYpOpq(h^PhM8PaOmEbE?ffuCNgvB1)sh+wNK7HVdVBt=2rQ^gc!t*je(v1xEL3HHVF9sH+4-(L%Cxh}t2889D=^to!bvxMn%rw;U)-UGW|bm+mWftMr6TLD~5La&sX0#ubcueMW7 z)^_SC-klg51SCCQKl_SRky)JiU{^HzVpVfd&807kv7$T&4*4bm`%_B0>321LqkWDs zi=qf_((Hw3+QBmJiI1A5W-;aXjaB+o1ltLY_#v9Q^0{J*~hVX1{y47lZTe@@hR z{?t%0*G)XTj3~g>YYk`|&l?;Dh+bTbs_oTNm7n&4=^gP-W(*sHz*g@m zXHH zhmU^&^210ptlf)pIh%*x=GW;p(Jkh4WVr?M;y}`w3dAt`*=RA~cUj5qUV|tB-I0SW z1a^liHvfaE_eK==+w>8X&^WF%2+=_%9HL@oL;8mo-M!p*tPyXo>i=!tw@RBEysa2t zZf&gX3uO{H%*mF}Jmm>~{!{qKtt1v@bNmAU2M$(bWpl-w=2l9da`WdF<-mi6;pGJ% z^URuI1HN~i?-PzuN0O z_T-{;6vk(nPhgX1_a~=EhTJQ&gR4FQIg)EXBpHIgD0H&=4M-lRt)Z)(Ml!V*ak5q5 zw5PMv>e@|y-)tOCTdO?D=r6->K)MP+{S(|wV#_mNMAT&FY%W?RM+9Ng`d|&NjA*}Q zhIXA**H(m6a}syc*-;HsQwslBXm)+7tV~ADT}U?{%1)>5(7uDEjt5xIee!CgqtXAm zkMlng&tytDU{F?YRj(E~9nwT!GvuQV`^EuQf0oL>21FeUtCh@E4WvMP#9X1Ys-dCk zBm=^fGWe0z^?$-uNNL2G?!?OuGLnzi7bksTPZ0af9C=8+6h$YKy^*XV#hdERsr+!f zi|gRlJ2f>yf?Iom249h9nJmiwqsr8GB>_GAU2rnxA>aw(Sykbxuk_>%zA6(fev~_= z9?gZ*X9jD`%Q^*~a0icNg`w9Dg5{!*l+fD&%fS?+V*-O%r*r9`;g;CU62lKhOSv`z z)N?Rml*sqp;EqdfCW+_HnGMau&^afI!D3o$Tly+_`j7TPJKR@VRFPAz`~#$5!t_eq zzNgw{d3V9~k+VZWDBPsxat@q9f>AvCRg>&4jS22yigh227p+psmm^xF2?_J(BD<@(rQv^ZBBU0Ltbp{nH#v^Bj@fSdwaXeiuR?gM?m|}qMH-ADm%5+ zafDz=+wr3UO1AY`BVc4+z~B^NaEG`2;8&Bos!oeI2-n6fHBLS@0;mi$p+Bd~bpXT; z__95^roVZ-;kk!JsFBZe(0=_g-xW0Q`qg~c1TQv|kSvL6AY@K1`$EqneDSn~nE=x@ z?a2mnrAK^el(!~FJd!d+4hHh7Fa$@?Wv1RYes1-!={A1Yjt$_8_boR>0YfcDoN`BS zA&YHCpEZ^6RGHR6ao8mwQgt{ecF-z-I=(1%L6CsV!MHGgzI->wdNWciwam2$4)F=$ zX&>idRUG?PDo*xpVi&)aaV00spx4xd7q;A&F~VfEV0sys=PoRLHSBR4^Q{z0iABOmV%SHN0^lJBB@!ZBOmQsTt+&KeQ z512&dZ;xK}#*{oyQqnvBKEgO)>d;UQm2J?Dqwgt#N9`wnx(uVRSj#oeMj=~{(cL46 zv#Q^Ss+u8B9s)AywQUYfLbL2AxaM}vh;*@%8uUe~@ly?e6=nsrxLR zHsLs`uJn^tEgYg9&E`n1_b&%MgzN(R^8Dn)m_aPjdNc|?@4s}s&{{5-{eX>cBev-_ zHg)PK)r9}%ld6;VicbczUyJge_C>)+b4HhO!y--aUB8=~<^9J^C5yLn+9;XMJmid{w#4jer=g3j&36%7 zA1ohT?JVmxhV8KEn2)s}b{?q7p^7*5d=iqVis5%xr5rgKP69xjHa4s{UWb&SGB~6k zwIw^OMoGb>160td-KfoMt2Ob2eCo$^$h!gq7$=Fpt|7|F~ zFX=`#OVx>@&w5-&V^2UQ^HfdVyLw)Jo$5<6RQ+1CL&Cl~$=P9_FKyU~Rph&^O%4pM zl8YT{OiKS-02}WtM)pNB!V7?-&#b}w_u8F6{uqSyk6b^md?<8yc&x!Blw28|4Xjd6 zE?Y?G9SIk}Nj0l)tCJa2Q-bR9Yc_)|d@7yoWEgei@r4{MdC*kVN9cGk!`+qZDsnKT>u-#nPFZ5wk8-wT-5*3Ml&`bGNKq`!=SBO-EpXBUi2781 z`1%9>KK8h($9D=t)0m|QTqFY*L0el>&m+2`k z7Q;Q>R9UiXKh{223@%x&R9#w|?vLqt-##=OVbLUtyX4@E6|i}oc7d3j)Y632!q@I| z#)_rb{dA%uqVfo}%?l1|x;^k26XfD9*rY$-BjGuAZzSG}Nez|b%SNzNzoFkwEd`w~ zs@EimvA*O9P54vhJ?~Ra#}|o@Wzv(*mehX>izs9Xr=6Y>DGrZL$r56tT;Epv!RaUf ztu3lsjmFMGYr_V)zo}4YOh`1vM%HPPIMMS>CDUNgH*rKuXzXy zWEn~-)J~; zVO6^t=yS8C5+j)Ew=(yC7%)`inN29fY!=*{RrmqQPnNXlS8jMP#(E&Dymj1~I+mO7 z>zH93uQ9{X{~DSRYzyuGICu|OK<6pp=aAC73|b-9$!l%22E>g>={*c5EGQQFQeNs< z3|&Yca2pHCaW}|PoMA`RMnJv~Cfg5rnphg;a(sKLaPMr!ihoGN_1YEd`XzL|du@9i z@cAY)OPPw#t8atPDQ^xD?X`<3Eat83PKguW5`uUg*s0J&2OumZ?ok@4|0iwCTqQC z>N>u&5bb1zY3@FXX#?%7^{<9w4>n>MvVNK_p~U$h%aO&@OYHm_ESB1TFgpVz9PWq+ye+04V4wp{VGwf)petdFB@2q)}TQ+%Y5 z3N69d{eF?ilzk(cQ5KB8hb;%rxOm}X&iKqP6~bN@ktDfr0uI)ocdTgH;4mp7sf}&q zXq{=-T|UCzrV1OJyd2Ww$aU>pg$RxN)`MLInc+O+8@Wp&!w$ZW9T$?cX&|K<5tR=u zgTwIW7$zASGTU0g)*k3Gi@ZEAnVsB#4W{wg1JukZuGAmGfMV*BJ?gXbN+HtTJ=$<~ zdPFt_T}2xzhxjfwzo85X1k(CmObCmOow@CqIatoIc=GO=2z14&A{1t_9)Ha0NSjo2 zSttJ7PMVF}zn^OOgU%#FT1APCOE8{{J%K_>SgwYd;@pnt;B>Q-eEJpheo2KeuP@zr z1nBMpwMlF#wKlsZrhT@p<%Nfjvq8ZJ%Z$UjM|tzuiz*39m#*<~cu2tmMm~(lkh^l> zEeuV2`awKQZ7wY$5c_(c6LzdKG=9TMee&mY<7=YVKAus&yfBX_hgXR-iGc(jltCLH zKB$AT$+5hXM~Vgr*it-JBpZPi?tF8Q?+VE9{|Ag0|nYH~;?!}>~y)VEFSGHlacHx5LPjkbqmD$=oK z`-hT=b3l*#6+@3w*0h%%#&FK+P{ngZcVLqGDerEWB6_S}K*yjLb#z>r`o*CjLe|vq zx>BJ|f&u2wqb(b{X^Zi70+(2;;(qC2I{6N=j*|i=HwR*Zfewh{6IE+30j%zrH}ID~ z3C=>KKI*NrsM3+*sVWl!gON?s&6&M|-)l#zYqO6t7U5I2HxF0EN{vRS?w&wkkrH}p zO8Xp+q166jJgjdFi-&{**aRYt^rOaH_v zh48(3HQz$Q8FnMCgaGc_J1$iqXNaNgc_~Gi(|*(Andj5F{OSe8eoF>O!7!#0 zpOJ?Z6Y2-^=b7(BJlxCdzVNm`Tz@G&x4}fE0|ProiYACoW$6tDb?6kX08oS%_hnq& zKrJ#<_=K2;ro-_PouoFZ%G)pK6sS5EG9rEICv63{;%a;CM8l8QBj4%~J=7*$psZGy z_jg-l>^YunX?R4k$l=vV`sc22ROD~|CIe=UnDNhZK?qk9=734FoEs6p&T%dMwO-||rz42J{TKL2;-+6P6QlP~5LQ{zSs*9>| z*}IR=F}SZQD%_%zaOC2LvtsSGPx;9Wo^Bfp?#UxZPk)>ADmn7-=1Xg#J6G5`CHqoL z1=*ez!5^N}KICQhDP$69DAd8RG0#8xu)2IW&V)A^CTN6SZ%g=%+gOybu*=%+ObHIk@$g;EVQr!?4)kT$GqheDg{o8=1BND!d0|GmZ#Db24QrDkx12sv)SO7`d^#z;A01y#mFO%4c&I7hGtrFC z5NaV^d0?kXU!6|bbeX{uKk7pcP8A5WB*T~U&oB&@d;Y7tM~BPR4(m_NM#e!|zRh4> ztNyKtq~M~N*J>?fyg7OuB2ThCiq7uAJc=e>mH(AF^za#(XJwlvV`cUW#eQ#Ntvqr} z>jF0R{hta3y}5CqA>C9Qp-yn1cc;u^%83}2@KH=^a)WQmX`xT-boFf<{T;2i*~j^I zL`?e{W-B>e<>7;d=usiE_=ms@0Cu22Y_B&U_8JP9h4%j_06xCw%!8MCID*svyVmvh zd2j^9b&LP0`<+G?{X|-v^3i`50%f0W{r{`~RX50yga`IMhk=Oe|NZ2@FTevczxf{* z1Kz3B6VUN}*NMBC{6F0T>eBI5{;i#Qo+&;l%CQ6+r17`jkkbG7TRRS2emp|0dV|F; zf9nk%VT{w?+R>Dv0)vYt%MkFt^(HX5OZ?w4`gez)XenUyH%6gCe+BN}h3Y`~82nFG AL;wH) literal 0 HcmV?d00001