H2O-GPT on AMD GPUs (ROCm)

[rohitnanda1443]

Hi, How can we run H20-GPT on AMD-GPUs using the AMD ROCm libraries.

One can easily run an inference server on Ollama using ROCm thereby H2O-GPT needs to use this Ollama server for inferencing.

Problem: H2o-GPT install fails as it keeps finding CUDA during install. Some guidance here on editing the install script for ROCm would be helpful,

Method:

1. LLM runs on an inference server using ROCm
2. H2o-GPT sends LLM requests to the inference server

[pseudotensor]

Can you share what you mean by it finds CUDA during install and fails? Maybe logs etc.?

I adjusted one block In docs/linux_install.sh CUDA is mentioned.

[rohitnanda1443]

It should not be uninstalling ROCm-Torch

`` Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
/tmp/pip-install-jav98t1i/flash-attn_c0c8ed92b3c147bfa04d7e6ab7c98f49/setup.py:95: UserWarning: flash_attn was requested, but nvcc was not found. Are you sure your environment has nvcc available? If you're installing within a container from https://hub.docker.com/r/pytorch/pytorch, only images whose names contain 'devel' will provide nvcc.
warnings.warn(
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/tmp/pip-install-jav98t1i/flash-attn_c0c8ed92b3c147bfa04d7e6ab7c98f49/setup.py", line 179, in
CUDAExtension(
File "/home/rohit/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1074, in CUDAExtension
library_dirs += library_paths(cuda=True)
File "/home/rohit/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1201, in library_paths
if (not os.path.exists(_join_cuda_home(lib_dir)) and
File "/home/rohit/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 2407, in _join_cuda_home
raise OSError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.

  torch.__version__  = 2.2.1+cu121
  
  
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Attempting uninstall: torch
Found existing installation: torch 2.5.0.dev20240822+rocm6.1
Uninstalling torch-2.5.0.dev20240822+rocm6.1:
Successfully uninstalled torch-2.5.0.dev20240822+rocm6.1
Attempting uninstall: sse_starlette
Found existing installation: sse-starlette 0.10.3
Uninstalling sse-starlette-0.10.3:
Successfully uninstalled sse-starlette-0.10.3
Attempting uninstall: torchvision
Found existing installation: torchvision 0.20.0.dev20240823+rocm6.1
Uninstalling torchvision-0.20.0.dev20240823+rocm6.1:
Successfully uninstalled torchvision-0.20.0.dev20240823+rocm6.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tts 0.22.0 requires numpy==1.22.0; python_version <= "3.10", but you have numpy 1.26.4 which is incompatible.
tts 0.22.0 requires pandas<2.0,>=1.4, but you have pandas 2.2.2 which is incompatible.
awscli 1.34.5 requires docutils<0.17,>=0.10, but you have docutils 0.21.2 which is incompatible.
fiftyone 0.25.0 requires sse-starlette<1,>=0.10.3, but you have sse-starlette 2.1.3 which is incompatible.
torchaudio 2.4.0.dev20240823+rocm6.1 requires torch==2.5.0.dev20240822, but you have torch 2.2.1 which is incompatible.
vllm 0.5.5+rocm614 requires pydantic>=2.8, but you have pydantic 2.7.0 which is incompatible.
Successfully installed docutils-0.21.2 pandas-2.2.2 pydantic-2.7.0 pydantic-core-2.18.1 pypandoc_binary-1.13 sse_starlette-2.1.3 torch-2.2.1 torchvision-0.17.1 ``

[rohitnanda1443]

Do we have an ROCm Docker image?

[pseudotensor]

We don't build one, but you can build one.