From 95351a446a8abef79288d5c333e5841b1f32e5bc Mon Sep 17 00:00:00 2001 From: Peter Jun Park Date: Mon, 17 Jun 2024 04:38:30 -0400 Subject: [PATCH] clean up installation guides Signed-off-by: Peter Jun Park --- docs/data/{ => install}/datasource_config.jpg | Bin .../{ => install}/datasource_settings.jpg | Bin docs/data/{ => install}/grafana_welcome.png | Bin .../grafana_workload_selection.png | Bin docs/data/{ => install}/import_dashboard.png | Bin .../install-decision-tree.png} | Bin docs/data/{ => install}/opening_dashboard.png | Bin docs/index.rst | 18 +- docs/install/client-side.rst | 0 docs/install/core-install.rst | 191 +++++++++++++++++ docs/install/grafana-setup.rst | 200 ++++++++++++++++++ docs/install/omniperf.rst | 189 ----------------- docs/install/quick-start.rst | 0 docs/install/server-side.rst | 0 docs/sphinx/_toc.yml.in | 6 +- 15 files changed, 404 insertions(+), 200 deletions(-) rename docs/data/{ => install}/datasource_config.jpg (100%) rename docs/data/{ => install}/datasource_settings.jpg (100%) rename docs/data/{ => install}/grafana_welcome.png (100%) rename docs/data/{ => install}/grafana_workload_selection.png (100%) rename docs/data/{ => install}/import_dashboard.png (100%) rename docs/data/{install_decision_tree.png => install/install-decision-tree.png} (100%) rename docs/data/{ => install}/opening_dashboard.png (100%) delete mode 100644 docs/install/client-side.rst create mode 100644 docs/install/core-install.rst create mode 100644 docs/install/grafana-setup.rst delete mode 100644 docs/install/omniperf.rst delete mode 100644 docs/install/quick-start.rst delete mode 100644 docs/install/server-side.rst diff --git a/docs/data/datasource_config.jpg b/docs/data/install/datasource_config.jpg similarity index 100% rename from docs/data/datasource_config.jpg rename to docs/data/install/datasource_config.jpg diff --git a/docs/data/datasource_settings.jpg b/docs/data/install/datasource_settings.jpg similarity index 100% rename from docs/data/datasource_settings.jpg rename to docs/data/install/datasource_settings.jpg diff --git a/docs/data/grafana_welcome.png b/docs/data/install/grafana_welcome.png similarity index 100% rename from docs/data/grafana_welcome.png rename to docs/data/install/grafana_welcome.png diff --git a/docs/data/grafana_workload_selection.png b/docs/data/install/grafana_workload_selection.png similarity index 100% rename from docs/data/grafana_workload_selection.png rename to docs/data/install/grafana_workload_selection.png diff --git a/docs/data/import_dashboard.png b/docs/data/install/import_dashboard.png similarity index 100% rename from docs/data/import_dashboard.png rename to docs/data/install/import_dashboard.png diff --git a/docs/data/install_decision_tree.png b/docs/data/install/install-decision-tree.png similarity index 100% rename from docs/data/install_decision_tree.png rename to docs/data/install/install-decision-tree.png diff --git a/docs/data/opening_dashboard.png b/docs/data/install/opening_dashboard.png similarity index 100% rename from docs/data/opening_dashboard.png rename to docs/data/install/opening_dashboard.png diff --git a/docs/index.rst b/docs/index.rst index e5298122c..b6f31a6d3 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -11,19 +11,19 @@ In addition to a full deployment guide with installation instructions, this documentation also explains the ideas motivating the design behind the tool and its components. +See Omniperf source code at ``__. + If you're new to Omniperf, acquaint yourself with the tool by reviewing the chapters that follow and gradually learn its more advanced features. To get started, see :doc:`What is Omniperf? `. -See Omniperf source code at ``__. - .. grid:: 2 :gutter: 3 .. grid-item-card:: Install - * :doc:`install/quick-start` - * :doc:`install/omniperf` + * :doc:`install/core-install` + * :doc:`install/grafana-setup` .. grid-item:: @@ -34,11 +34,6 @@ in practice. .. grid:: 2 :gutter: 3 - .. grid-item-card:: Conceptual - - * :doc:`conceptual/overview` - * :doc:`conceptual/performance-model` - .. grid-item-card:: How to * :doc:`how-to/profiling` @@ -48,6 +43,11 @@ in practice. * `Omniperf example exercises `_ + .. grid-item-card:: Conceptual + + * :doc:`conceptual/overview` + * :doc:`conceptual/performance-model` + .. grid-item-card:: Reference * :doc:`reference/compatible-accelerators` diff --git a/docs/install/client-side.rst b/docs/install/client-side.rst deleted file mode 100644 index e69de29bb..000000000 diff --git a/docs/install/core-install.rst b/docs/install/core-install.rst new file mode 100644 index 000000000..574584443 --- /dev/null +++ b/docs/install/core-install.rst @@ -0,0 +1,191 @@ +.. meta:: + :description: Omniperf installation and deployment + :keywords: Omniperf, ROCm, profiler, tool, Instinct, accelerator, AMD, + install, deploy, Grafana, client, configuration, modulefiles + +********************************* +Installing and deploying Omniperf +********************************* + +Omniperf consists of two installation components. + +* :ref:`Omniperf core installation ` + + * Provides the core application profiling capability. + * Allows the collection of performance counters, filtering by hardware + block, dispatch, kernel, and more. + * Provides a CLI-based analysis mode. + * Provides a standalone web interface for importing analysis metrics. + +* :doc:`Grafana server for Omniperf ` (optional) + + * Hosts the MongoDB backend and Grafana instance. + * Is packaged in a Docker container for easy setup. + +Determine what you need to install based on how you would like to interact with +Omniperf. See the following decision tree to help determine what installation is +right for you. + +.. image:: ../data/install/install-decision-tree.png + :align: center + :alt: Decision tree for installing and deploying Omniperf + +.. _core-install: + +Core installation +================= + +The core Omniperf application requires the following basic software +dependencies. + +* Python ``>= 3.8`` +* CMake ``>= 3.19`` +* ROCm ``>= 5.7.1`` + +Omniperf depends on a number of Python packages documented in the top-level +``requirements.txt`` file. Install these *before* configuring Omniperf. + +.. admonition:: Optional packages for developers + + For those looking to build Omniperf as a developer, consider these additional + requirements. + + .. list-table:: + :header-rows: 1 + + * - Requirements file + - Description + + * - ``docs/sphinx/requirements.txt`` + - Python packages required to build this documentation from source. + + * - ``requirements-test.txt`` + - Python packages required to run Omniperf's CI suite using PyTest. + + The recommended procedure for Omniperf usage is to install into a shared file + system so that multiple users can access the final installation. The + following steps illustrate how to install the necessary Python dependencies + using `pip `_ and Omniperf into a + shared location controlled by the ``INSTALL_DIR`` environment variable. + +.. _core-install-cmake-vars: + +Configuration variables +----------------------- +The following installation example leverages several +`CMake `_ project variables defined as +follows. + +.. list-table:: + :header-rows: 1 + + * - CMake variable + - Description + + * - ``CMAKE_INSTALL_PREFIX`` + - Controls the install path for Omniperf files. + + * - ``PYTHON_DEPS`` + - Specifies an optional path to resolve Python package dependencies. + + * - ``MOD_INSTALL_PATH`` + - Specifies an optional path for separate Omniperf modulefile installation. + +.. _core-install-steps: + +Installation steps +------------------ + +#. A typical install begins by downloading the latest release tarball available + from ``__. From there, untar and + navigate into the top-level directory. + + .. code-block:: shell + + tar xfz omniperf-v{__VERSION__}.tar.gz + cd omniperf-v{__VERSION__} + +#. Next, install Python dependencies and complete the Omniperf configuration and + install process. + + .. code-block:: shell + + # define top-level install path + export INSTALL_DIR= + + # install python deps + python3 -m pip install -t ${INSTALL_DIR}/python-libs -r requirements.txt + + # configure Omniperf for shared install + mkdir build + cd build + cmake -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR}/{__VERSION__} \ + -DPYTHON_DEPS=${INSTALL_DIR}/python-libs \ + -DMOD_INSTALL_PATH=${INSTALL_DIR}/modulefiles .. + + # install + make install + + .. tip:: + + You might need to ``sudo`` the final installation step if you don't have + write access for the chosen installation path. + +#. Upon successful installation, your top-level installation directory should + look like this. + + .. code-block:: shell + + $ ls $INSTALL_DIR + modulefiles {__VERSION__} python-libs + +.. _core-install-modulefiles: + +Execution using modulefiles +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The installation process includes the creation of an environment modulefile for +use with `Lmod `_. On systems that support Lmod, +you can register the Omniperf modulefile directory and setup your environment +for execution of Omniperf as follows. + +.. code-block:: shell + + $ module use $INSTALL_DIR/modulefiles + $ module load omniperf + $ which omniperf + /opt/apps/omniperf/{__VERSION__}/bin/omniperf + + $ omniperf --version + ROC Profiler: /opt/rocm-5.1.0/bin/rocprof + + omniperf (v{__VERSION__}) + +.. tip:: + + If you're relying on an Lmod Python module locally, you may wish to customize + the resulting Omniperf modulefile post-installation to include extra + module dependencies. + +Execution without modulefiles +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To use Omniperf without the companion modulefile, update your ``PATH`` +settings to enable access to the command line binary. If you installed Python +dependencies in a shared location, also update your `PYTHONPATH` configuration. + +.. code-block:: shell + + export PATH=$INSTALL_DIR/{__VERSION__}/bin:$PATH + export PYTHONPATH=$INSTALL_DIR/python-libs + +.. _core-install-rocprof-var: + +ROCProfiler +----------- + +Omniperf relies on ROCProfiler's ``rocprof`` binary during the profiling +process. Normally, the path to this binary is detected automatically, but you +can override the path by the setting the optional `ROCPROF` environment +variable. + diff --git a/docs/install/grafana-setup.rst b/docs/install/grafana-setup.rst new file mode 100644 index 000000000..f1fc38441 --- /dev/null +++ b/docs/install/grafana-setup.rst @@ -0,0 +1,200 @@ +.. meta:: + :description: Omniperf client-side installation and deployment + :keywords: Omniperf, ROCm, profiler, tool, Instinct, accelerator, AMD, + install, deploy, Grafana, server, configuration, + +************************************** +Setting up Grafana server for Omniperf +************************************** + +A Grafana instance is *not required* to profile or analyze performance data +from the CLI. It's a supplementary mechanism to help you import performance +data and examine it in a detailed +`Grafana `_ dashboard GUI. + +Setting up a Grafana instance for Omniperf requires the following basic software +dependencies. + +* `Docker Engine `_ + +The recommended process for enabling the server-side of Omniperf is to use the +provided ``Dockerfile`` to build the Grafana and MongoDB instance. + +.. _grafana-mongodb-setup: + +Set up Grafana and MongoDB +========================== + +Once you've decided where to host the Grafana and MongoDB instance, complete the +the following setup instructions. + +Install MongoDB utilities +------------------------- + +Omniperf uses the +`mongoimport `_ +utility to upload data to your Grafana instance's backend database. + +Use the following commands to install MongoDB utilities for Ubuntu 20.04. + +.. code-block:: bash + + $ wget https://fastdl.mongodb.org/tools/db/mongodb-database-tools-ubuntu2004-x86_64-100.6.1.deb + $ sudo apt install ./mongodb-database-tools-ubuntu2004-x86_64-100.6.1.deb + +.. note:: + + Find installation instructions for other distributions in + `MongoDB Database Tools Downloads `_. + +.. _grafana-persistent-storage-setup: + +Set up persistent storage +------------------------- + +Bind MongoDB to a directory on the host OS to create a local backup in case of a +crash or reset. This is called "creating a persistent volume" in the Docker +world. + +.. code-block:: bash + + $ sudo mkdir -p /usr/local/persist && cd /usr/local/persist/ + $ sudo mkdir -p grafana-storage mongodb + $ sudo docker volume create --driver local --opt type=none --opt device=/usr/local/persist/grafana-storage --opt o=bind grafana-storage + $ sudo docker volume create --driver local --opt type=none --opt device=/usr/local/persist/mongodb --opt o=bind grafana-mongo-db + +.. _grafana-docker-container: + +Build and launch the Docker container +------------------------------------- + +You're now ready to build your ``Dockerfile``. Navigate to your Omniperf install +directory to begin. + +.. code-block:: bash + + $ sudo docker-compose build + $ sudo docker-compose up -d + +The TCP ports for Grafana (``4000``) and MongoDB (``27017``) in the Docker +container are mapped to ``14000`` and ``27018``, respectively, on the host side. + +.. tip:: + + In the event that either your Grafana or MongoDB instance crashes fatally, + just restart the server. Navigate to your install directory and run: + + .. code-block:: + + $ sudo docker-compose down + $ sudo docker-compose up -d + +.. _grafana-dashboard-setup: + +Set up the Grafana dashboard +---------------------------- + +Once you've launched your Docker container you should be able to reach Grafana +at ``http://:14000``. The default login credentials for your first-time +Grafana setup are: + +* **Username**: ``admin`` +* **Password**: ``admin`` + +.. figure:: ../data/install/grafana_welcome.png + :align: center + :alt: Grafana dashboard welcome screen + + Grafana's welcome screen. + +.. _grafana-datasource-setup: + +Configure the MongoDB data source +--------------------------------- + +You must configure your MongoDB data source in Grafana before first-time use. +Navigate to Grafana's **Configuration** page to add the "Omniperf Data" +connection. + +.. figure:: ../data/install/datasource_config.jpg + :align: center + :alt: Grafana data source configuration + + Grafana's Configuration page. + +Configure the following fields in the data source settings. + +.. list-table:: + :stub-columns: 1 + + * - HTTP URL + - ``http://localhost:3333`` + + * - MongoDB URL + - ``mongodb://temp:temp123@\:27018/admin?authSource=admin`` + + * - Database Name + - ``admin`` + +After configuring these fields, click **Save & test** to make sure your +connection is successful. + +.. note:: + + To avoid potential DNS issues, you might need to use the actual IP address + for the host node in the MongoDB URL. + +.. figure:: ../data/install/datasource_settings.jpg + :align: center + :alt: Grafana data source settings + + Grafana data source settings. + +.. _grafana-import-dashboard-file: + +Import the Omniperf dashboard file +---------------------------------- + +From the **Create** → **Import** page, upload the dashboard file, +``/dashboards/Omniperf_v{__VERSION__}_pub.json`` from the +:doc:`Omniperf tarball `. + +Edit both the dashboard **Name** and the **Unique identifier (UID)** fields to +uniquely identify the dashboard. Click **Import** to complete the process. + +.. figure:: ../data/install/import_dashboard.png + :align: center + :alt: Grafana's import dashboard + + Grafana's Import dashboard. + +.. _grafana-select-workload: + +Select and load the Omniperf workload +------------------------------------- + +Once you have imported a dashboard you're ready to begin. Start by browsing +available dashboards and selecting the dashboard you have just imported. + +.. figure:: ../data/install/opening_dashboard.png + :align: center + :alt: Opening your Omniperf dashboard in Grafana + + Opening your Omniperf profiling dashboard in Grafana. + +Remember that you need to upload workload data to the MongoDB backend before +analyzing in your Grafana interface. See a detailed example of this in +:ref:`grafana-gui-import`. + +After a workload has been successfully uploaded, you should be able to select it +from the workload dropdown located at the top of your Grafana dashboard. + +.. figure:: ../data/install/grafana_workload_selection.png + :align: center + :alt: Omniperf workload selection in Grafana + + Selecting your Omniperf workload in Grafana. + +For more information on how to use the Grafana interface for analysis see +:ref:`grafana-analysis`. + diff --git a/docs/install/omniperf.rst b/docs/install/omniperf.rst deleted file mode 100644 index ddb94e9b7..000000000 --- a/docs/install/omniperf.rst +++ /dev/null @@ -1,189 +0,0 @@ -**************** -Install Omniperf -**************** - -Omniperf consists of two installation components: - -* :ref:`Omniperf client-side ` - - * Provides core application profiling capability. - * Allows collection of performance counters, filtering by hardware block, - dispatch, kernel, and more. - * Provides CLI-based analysis mode. - * Provides standalone web interface for importing analysis metrics. - -* :ref:`Omniperf server-side ` - - * Hosts the MongoDB backend and Grafana instance. - * Packaged in a Docker container for easy setup. - -Determine what you need to install based on how you would like to interact with -Omniperf. See the following decision tree to help determine what installation is -right for you. - -.. _install-client-side: - -Client-side installation -======================== - -Omniperf client-side requires the following basic software dependencies prior to usage: - -* Python ``>= 3.8`` -* CMake ``>= 3.19`` -* ROCm ``>= 5.7.1`` - -In addition, Omniperf leverages a number of Python packages that are -documented in the top-level ``requirements.txt`` file. These must be -installed *before* configuring Omniperf. - -.. admonition:: Optional packages for developers - - If you would like to build Omniperf as a developer, consider these additional - requirements: - - .. list-table:: - :header-rows: 1 - - * - Requirement file - - Description - - * - requirements-doc.txt **CHANGE ME** - - Python packages required to build docs from source. - - * - requirements-test.txt - - Python packages required to run Omniperf's CI suite via PyTest. - - The recommended procedure for Omniperf usage is to install into a shared file - system so that multiple users can access the final installation. The - following steps illustrate how to install the necessary Python dependencies - using `pip `_ and Omniperf into a - shared location controlled by the ``INSTALL_DIR`` environment variable. - -Configuration variables ------------------------ -The following installation example leverages several -[CMake](https://cmake.org/cmake/help/latest/) project variables -defined as follows: - -.. list-table:: - :header-rows: 1 - - * - Variable name - - Description - - * - ``CMAKE_INSTALL_PREFIX`` - - Controls the install path for Omniperf files. - - * - ``PYTHON_DEPS`` - - Specifies an optional path to resolve Python package dependencies. - - * - ``MOD_INSTALL_PATH`` - - Specifies an optional path for separate Omniperf modulefile installation. - -A typical install will begin by downloading the latest release tarball -available from the -[Releases](https://github.com/ROCm/omniperf/releases) section -of the Omniperf development site. From there, untar and descend into -the top-level directory as follows: - -```shell-session -$ tar xfz omniperf-v{__VERSION__}.tar.gz -$ cd omniperf-v{__VERSION__} -``` - -Next, install Python dependencies and complete the Omniperf configuration/install process as follows: - -```shell-session -# define top-level install path -$ export INSTALL_DIR= - -# install python deps -$ python3 -m pip install -t ${INSTALL_DIR}/python-libs -r requirements.txt - -# configure Omniperf for shared install -$ mkdir build -$ cd build -$ cmake -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR}/{__VERSION__} \ - -DPYTHON_DEPS=${INSTALL_DIR}/python-libs \ - -DMOD_INSTALL_PATH=${INSTALL_DIR}/modulefiles .. - -# install -$ make install -``` - -```{tip} -You may require `sudo` during the final install step if you -do not have write access to the chosen install path. -``` - - -After completing these steps, a successful top-level installation directory looks as follows: -```shell-session -$ ls $INSTALL_DIR -modulefiles {__VERSION__} python-libs -``` - -### Execution using modulefiles - -The installation process includes creation of an environment -modulefile for use with [Lmod](https://lmod.readthedocs.io). On -systems that support Lmod, a user can register the Omniperf modulefile -directory and setup their environment for execution of Omniperf as -follows: - - - -```shell-session -$ module use $INSTALL_DIR/modulefiles -$ module load omniperf -$ which omniperf -/opt/apps/omniperf/{__VERSION__}/bin/omniperf - -$ omniperf --version -ROC Profiler: /opt/rocm-5.1.0/bin/rocprof - -omniperf (v{__VERSION__}) -``` - -```{tip} Users relying on an Lmod Python module locally may wish to -customize the resulting Omniperf modulefile post-installation to -include additional module dependencies. -``` - -### Execution without modulefiles - -To use Omniperf without the companion modulefile, update your `PATH` -settings to enable access to the command-line binary. If you installed Python -dependencies in a shared location, update your `PYTHONPATH` config as well: - -```shell-session -export PATH=$INSTALL_DIR/{__VERSION__}/bin:$PATH -export PYTHONPATH=$INSTALL_DIR/python-libs -``` - -### rocProf - -Omniperf relies on a rocProf binary during the profiling -process. Normally the path to this binary will be detected -automatically, but it can also be overridden via the setting the -optional `ROCPROF` environment variable to the path of the binary the user -wishes to use instead. - - - - - -%%% ### Generate Packaging -%%% ```console -%%% cd build -%%% cpack -G STGZ -%%% cpack -G DEB -D CPACK_PACKAGING_INSTALL_PREFIX=/opt/omniperf -%%% cpack -G RPM -D CPACK_PACKAGING_INSTALL_PREFIX=/opt/omniperf -%%% ``` - -.. _install-server-side: - -Server-side installation -======================== - -Hello this abc def ghi jkl mnop qrst uvw xyz. diff --git a/docs/install/quick-start.rst b/docs/install/quick-start.rst deleted file mode 100644 index e69de29bb..000000000 diff --git a/docs/install/server-side.rst b/docs/install/server-side.rst deleted file mode 100644 index e69de29bb..000000000 diff --git a/docs/sphinx/_toc.yml.in b/docs/sphinx/_toc.yml.in index f453900ab..b7dc890c9 100644 --- a/docs/sphinx/_toc.yml.in +++ b/docs/sphinx/_toc.yml.in @@ -10,11 +10,13 @@ subtrees: - caption: Install entries: - - file: install/quick-start.rst - - file: install/omniperf.rst + - file: install/core-install.rst + - file: install/grafana-setup.rst + - caption: How to entries: + - file: install/quick-start.rst - file: how-to/profiling.rst - file: how-to/analysis.rst