Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add libcufile recipe #21908

Merged
merged 72 commits into from
Apr 12, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
b5e65d6
Add libcufile recipe
mroeschke Jan 31, 2023
c9f1fc1
Add Licence metadata
mroeschke Jan 31, 2023
372f26e
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
488dbda
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
4279763
Add summary
mroeschke Jan 31, 2023
1a4969e
Change description to summary
mroeschke Jan 31, 2023
0c20a35
Add file tests
mroeschke Jan 31, 2023
fbac483
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
e41a0ac
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
5d18921
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
cdbcdba
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
9e6d832
Update recipes/libcufile/meta.yaml
mroeschke Jan 31, 2023
ee146b6
Add build.sh instead of build.bat
mroeschke Feb 1, 2023
9ad326f
Update recipes/libcufile/Makefile.patch
mroeschke Feb 3, 2023
64b3393
some formatting
mroeschke Feb 3, 2023
a90df31
Try patchfile fix
mroeschke Feb 4, 2023
702a290
Add directory
mroeschke Feb 4, 2023
dbbd113
Create and move tools and samples to gds dir
mroeschke Feb 7, 2023
8e21b79
Merge branch 'main' into libcufile
mroeschke Feb 7, 2023
1970a53
typo -p
mroeschke Feb 7, 2023
6158a96
just -d
mroeschke Feb 7, 2023
7497aed
Move man
mroeschke Feb 7, 2023
c8e9e42
include new gds dir
mroeschke Feb 7, 2023
a17812a
Update recipes/libcufile/meta.yaml
mroeschke Feb 8, 2023
5317c44
Move to difference section
mroeschke Feb 8, 2023
f264627
Move to outputs build
mroeschke Feb 8, 2023
f93c0b2
New build section
mroeschke Feb 8, 2023
11ee0dc
Update recipes/libcufile/meta.yaml
mroeschke Feb 9, 2023
79c750a
Update recipes/libcufile/meta.yaml
mroeschke Feb 9, 2023
c8de857
Update recipes/libcufile/meta.yaml
mroeschke Feb 9, 2023
54a340a
Update recipes/libcufile/meta.yaml
mroeschke Feb 9, 2023
178099f
Update recipes/libcufile/meta.yaml
mroeschke Feb 9, 2023
1ba2afe
Update recipes/libcufile/meta.yaml
mroeschke Feb 11, 2023
76cd37c
Merge branch 'main' into libcufile
mroeschke Feb 11, 2023
61ab85d
Update recipes/libcufile/meta.yaml
mroeschke Feb 13, 2023
93593ce
Update recipes/libcufile/meta.yaml
mroeschke Feb 13, 2023
498ac3a
Update recipes/libcufile/meta.yaml
mroeschke Feb 13, 2023
89875b4
wildcards missing?
mroeschke Feb 14, 2023
86b9763
Revert "wildcards missing?"
mroeschke Feb 16, 2023
7939393
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
9cd3345
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
2556d27
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
9d4601f
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
0452766
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
28723e1
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
f03b6c1
Update recipes/libcufile/meta.yaml
mroeschke Mar 22, 2023
be9e78c
Merge branch 'main' into libcufile
mroeschke Mar 22, 2023
900a8aa
Merge branch 'main' into libcufile
jakirkham Mar 27, 2023
1ed22aa
Merge branch 'main' into libcufile
mroeschke Apr 5, 2023
25ee3e5
Make changes to meta and build files
mroeschke Apr 5, 2023
c671519
Keep top level about section
mroeschke Apr 5, 2023
afa2ad6
Merge branch 'main' into libcufile
mroeschke Apr 7, 2023
ba1e05d
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
ca53dd3
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
21f096d
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
694ac62
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
32b88d0
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
3cda9c8
Update recipes/libcufile/meta.yaml
mroeschke Apr 7, 2023
23d673c
Refactor -dev tests
mroeschke Apr 7, 2023
ebfbf07
pin_compatible?
mroeschke Apr 7, 2023
f32c306
Update recipes/libcufile/meta.yaml
mroeschke Apr 10, 2023
7456ba7
Update recipes/libcufile/meta.yaml
mroeschke Apr 10, 2023
296b975
Update recipes/libcufile/meta.yaml
mroeschke Apr 10, 2023
08a0182
Update recipes/libcufile/meta.yaml
mroeschke Apr 10, 2023
4b402b1
Update recipes/libcufile/meta.yaml
mroeschke Apr 10, 2023
9e35a54
Fix libcufile tests and packaging.
bdice Apr 12, 2023
c1f3c4b
Attempt to fix RDMA issue.
bdice Apr 12, 2023
5b28f1b
Make tests pass.
bdice Apr 12, 2023
08a70e1
Fix RDMA cdt dependency.
bdice Apr 12, 2023
e1d5f02
Don't ship etc/cufile.json.
bdice Apr 12, 2023
16cfcdd
Rerun CI.
bdice Apr 12, 2023
21053dd
Merge pull request #1 from bdice/fix-libcufile-tests
mroeschke Apr 12, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions recipes/libcufile/Makefile.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
--- samples/Makefile 2022-06-24 00:23:23.000000000 +0000
+++ samples/Makefile.new 2022-06-27 15:04:32.235125647 +0000
@@ -35,8 +35,8 @@

# Common includes and paths for CUDA
-CUDA_PATH := /usr/local/cuda
-CUFILE_PATH ?= /usr/local/cuda/targets/x86_64-linux/lib/
-CUFILE_INCLUDE_PATH ?= /usr/local/cuda/targets/x86_64-linux/include/
+CUDA_PATH := ${PREFIX}
+CUFILE_PATH ?= $(CUDA_PATH)/targets/x86_64-linux/lib/
+CUFILE_INCLUDE_PATH ?= $(CUDA_PATH)/targets/x86_64-linux/include/
CXXFLAGS := -Wall
CXXFLAGS += -I $(CUDA_PATH)/include/
CXXFLAGS += -I $(CUFILE_INCLUDE_PATH)

36 changes: 36 additions & 0 deletions recipes/libcufile/build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/bin/bash

# Install to conda style directories
[[ -d lib64 ]] && mv lib64 lib
mkdir -p ${PREFIX}/lib
mkdir -p ${PREFIX}/gds
rm -rv etc
mv -v man ${PREFIX}/man
mv -v samples ${PREFIX}/gds/samples
mv -v tools ${PREFIX}/gds/tools
[[ -d pkg-config ]] && mv pkg-config ${PREFIX}/lib/pkgconfig
[[ -d "$PREFIX/lib/pkgconfig" ]] && sed -E -i "s|cudaroot=.+|cudaroot=$PREFIX|g" $PREFIX/lib/pkgconfig/cufile*.pc

[[ ${target_platform} == "linux-64" ]] && targetsDir="targets/x86_64-linux"
[[ ${target_platform} == "linux-aarch64" ]] && targetsDir="targets/sbsa-linux"

for i in `ls`; do
[[ $i == "build_env_setup.sh" ]] && continue
[[ $i == "conda_build.sh" ]] && continue
[[ $i == "metadata_conda_debug.yaml" ]] && continue
if [[ $i == "lib" ]] || [[ $i == "gds" ]] || [[ $i == "include" ]]; then
mkdir -p ${PREFIX}/${targetsDir}
mkdir -p ${PREFIX}/$i
cp -rv $i ${PREFIX}/${targetsDir}
if [[ $i == "lib" ]]; then
for j in "$i"/*.so*; do
# Shared libraries are symlinked in $PREFIX/lib
ln -s ${PREFIX}/${targetsDir}/$j ${PREFIX}/$j
done
fi
else
# Put all other files in targetsDir
mkdir -p ${PREFIX}/${targetsDir}/${PKG_NAME}
cp -rv $i ${PREFIX}/${targetsDir}/${PKG_NAME}
fi
done
2 changes: 2 additions & 0 deletions recipes/libcufile/conda_build_config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
arm_variant_type: # [aarch64]
- sbsa # [aarch64]
189 changes: 189 additions & 0 deletions recipes/libcufile/meta.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,189 @@
{% set name = "libcufile" %}
{% set version = "1.5.0.59" %}
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
{% set cuda_version = "12.0" %}
{% set platform = "linux-x86_64" %} # [linux64]
{% set platform = "linux-sbsa" %} # [aarch64]
{% set target_name = "x86_64-linux" %} # [linux64]
{% set target_name = "sbsa-linux" %} # [aarch64]
{% set extension = "tar.xz" %} # [not win]

package:
name: {{ name|lower }}
version: {{ version }}

source:
url: https://developer.download.nvidia.com/compute/cuda/redist/{{ name }}/{{ platform }}/{{ name }}-{{ platform }}-{{ version }}-archive.{{ extension }}
sha256: 222c49b29afb544c69542c4e658b0a1bd01fd2306f2136ad6990ccfff482a097 # [linux64]
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
sha256: 4e8d2b525322be9ab6ae574bd1d0325a4bf9c6c3bbdaa1611b5b969b7b34b0c4 # [aarch64]
patches:
- Makefile.patch

build:
number: 0
skip: true # [not (linux64 or aarch64)]

test:
commands:
# Note that the version of libcufile does not match
# {{ version.split(".")[0] }} but the major version of libcufile_rdma does
# match that. Also we must drop the last component of the version when
# finding the versioned .so file.
{% set full_version = version.split(".")[:-1] | join(".") %}
- test -L $PREFIX/lib/libcufile.so.0
- test -L $PREFIX/lib/libcufile_rdma.so.{{ version.split(".")[0] }} # [linux64]
- test -L $PREFIX/lib/libcufile.so.{{ full_version }}
- test -L $PREFIX/lib/libcufile_rdma.so.{{ full_version }} # [linux64]
- test -L $PREFIX/targets/{{ target_name }}/lib/libcufile.so.0
- test -L $PREFIX/targets/{{ target_name }}/lib/libcufile_rdma.so.{{ version.split(".")[0] }} # [linux64]
- test -f $PREFIX/targets/{{ target_name }}/lib/libcufile.so.{{ full_version }}
- test -f $PREFIX/targets/{{ target_name }}/lib/libcufile_rdma.so.{{ full_version }} # [linux64]

outputs:
- name: libcufile
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
build:
missing_dso_whitelist:
- "*/libcuda.so.*"
files:
- lib/libcufile*.so.*
- targets/{{ target_name }}/lib/libcufile*.so.*
requirements:
build:
- {{ compiler("c") }}
- {{ compiler("cxx") }}
- arm-variant * {{ arm_variant_type }} # [aarch64]
- sysroot_{{ target_platform }} 2.17 # [linux]
- {{ cdt("rdma-core-devel") }} # [linux64]
host:
- cuda-version {{ cuda_version }}
run:
- {{ pin_compatible("cuda-version", max_pin="x.x") }}
run_constrained:
- arm-variant * {{ arm_variant_type }} # [aarch64]
# Tests are defined at the top level, due to package/output name conflicts.
about:
home: https://developer.nvidia.com/cuda-toolkit
license: LicenseRef-NVIDIA-End-User-License-Agreement
license_file: LICENSE
license_url: https://docs.nvidia.com/cuda/eula/index.html
summary: Library for NVIDIA GPUDirect Storage
description: |
The cuFile library provides a direct data path between GPU memory and storage.
doc_url: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

- name: libcufile-dev
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
build:
run_exports:
- {{ pin_subpackage("libcufile", max_pin="x") }}
files:
- lib/libcufile*.so
- lib/pkgconfig
- targets/{{ target_name }}/include
- targets/{{ target_name }}/lib/libcufile*.so
- man/man3
requirements:
build:
- {{ compiler("c") }}
- {{ compiler("cxx") }}
- arm-variant * {{ arm_variant_type }} # [aarch64]
- sysroot_{{ target_platform }} 2.17 # [linux]
host:
- cuda-version {{ cuda_version }}
run:
- {{ pin_compatible("cuda-version", max_pin="x.x") }}
- {{ pin_subpackage("libcufile", exact=True) }}
run_constrained:
- arm-variant * {{ arm_variant_type }} # [aarch64]
- libcufile-static >={{ version }}
test:
commands:
jakirkham marked this conversation as resolved.
Show resolved Hide resolved
- test -L $PREFIX/lib/libcufile.so
- test -L $PREFIX/lib/libcufile_rdma.so
- test -f $PREFIX/targets/{{ target_name }}/include/cufile.h
- test -L $PREFIX/targets/{{ target_name }}/lib/libcufile.so
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
- test -L $PREFIX/targets/{{ target_name }}/lib/libcufile_rdma.so
- test -d $PREFIX/man/man3
about:
home: https://developer.nvidia.com/cuda-toolkit
license: LicenseRef-NVIDIA-End-User-License-Agreement
license_file: LICENSE
license_url: https://docs.nvidia.com/cuda/eula/index.html
summary: Library for NVIDIA GPUDirect Storage
description: |
The cuFile library provides a direct data path between GPU memory and storage.
doc_url: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

- name: libcufile-static
files:
- targets/{{ target_name }}/lib/libcufile*_static.a
requirements:
build:
- {{ compiler("c") }}
- {{ compiler("cxx") }}
- arm-variant * {{ arm_variant_type }} # [aarch64]
- sysroot_{{ target_platform }} 2.17 # [linux]
host:
- cuda-version {{ cuda_version }}
run:
- {{ pin_compatible("cuda-version", max_pin="x.x") }}
run_constrained:
- arm-variant * {{ arm_variant_type }} # [aarch64]
test:
commands:
- test -f $PREFIX/targets/{{ target_name }}/lib/libcufile_static.a
- test -f $PREFIX/targets/{{ target_name }}/lib/libcufile_rdma_static.a
about:
home: https://developer.nvidia.com/cuda-toolkit
license: LicenseRef-NVIDIA-End-User-License-Agreement
license_file: LICENSE
license_url: https://docs.nvidia.com/cuda/eula/index.html
summary: Library for NVIDIA GPUDirect Storage
description: |
The cuFile library provides a direct data path between GPU memory and storage.
doc_url: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

- name: gds-tools
files:
- gds/samples
- gds/tools
jakirkham marked this conversation as resolved.
Show resolved Hide resolved
requirements:
build:
- {{ compiler("c") }}
- {{ compiler("cxx") }}
- arm-variant * {{ arm_variant_type }} # [aarch64]
- sysroot_{{ target_platform }} 2.17 # [linux]
host:
- cuda-version {{ cuda_version }}
- libcufile-dev {{ version }}
- libnuma # [linux]
run:
- {{ pin_compatible("cuda-version", max_pin="x.x") }}
- {{ pin_subpackage("libcufile", max_pin="x") }}
run_constrained:
- arm-variant * {{ arm_variant_type }} # [aarch64]
test:
commands:
- test -d $PREFIX/gds/samples
- test -d $PREFIX/gds/tools
about:
home: https://developer.nvidia.com/cuda-toolkit
license: LicenseRef-NVIDIA-End-User-License-Agreement
license_file: LICENSE
license_url: https://docs.nvidia.com/cuda/eula/index.html
summary: Library for NVIDIA GPUDirect Storage
description: |
The cuFile library provides a direct data path between GPU memory and storage.
doc_url: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

about:
home: https://developer.nvidia.com/cuda-toolkit
license: LicenseRef-NVIDIA-End-User-License-Agreement
license_file: LICENSE
license_url: https://docs.nvidia.com/cuda/eula/index.html
summary: Library for NVIDIA GPUDirect Storage
description: |
The cuFile library provides a direct data path between GPU memory and storage.
doc_url: https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html

extra:
recipe-maintainers:
- adibbley