HDF-AI Tools and Information Center

This is a central hub for exchanging AI solutions in HDF. It also hosts sample AI data / model files in HDF.

Ideas

Query HDF in Natural Langauge

• Upload HDF file(s) as source for LLM.
• Query HDF contents in Natural Langauge.
• Manipulate data. (e.g., Create PNG image from /g1/dset1 using rainbow palette)

Make Bigdata AI-ready using HDF

• Save metadata with data.
  • algorithms and their versions used, model parameters, authors, etc.
• Save training / model / testing data in hiearchy with groups.
• Save knowledge graph (semantic network) in HDF.

Reduce Bigdata Storage using HDF AI Filter

Filtering Bigdata with AI is a solution to reduce the burden of managing a large amount of training and testing data.

HDF AI filter can automatically sanitize your data in a scalable manner when you archive data in HDF. It can save a lot of space by storing only models, not real data.

HDF AI Filter can

• store everything in hierarchy including algorithms to use and learned models.
• link to the raw data for provenance.
• set a time to remove raw data and a desired accuracy threshold to prune models.
• run several ML algorithms in parallel according to the HDF's group hierarchy.

FAQ

• What is HAI API? This is a high level API that can run I/O-efficient AI tasks for HDF data.

User Guide

• HAI Reference Manual

Data

• Cat vs. Non-Cat
• Core ML Specification
• Hypersim
• Megatron-LM
• Joke Generator (Jump to 14:40 in the video.)
  • textgenrnn_weights.hdf5
• Face Emotion Recognition
• A Deep Learning-Based Hybrid Model of Global Terrestrial Evaporation
• JFT-3B

Code

• AnnData
• PIMFlow
• John Snow Labs
• GeoWatch
• safetensors
• Croissant
• GraphCast
• Deep Learning for Climiate Modeling Data: specifically, data_helpers.py
• Kubeflow HOW-TO
• Keras Spark Rossmann Run Example
• Deep Learning IO Benchmark
• FlexFlow
• parallelformers
• Keras/TensorFlow - Save Model in HDF5
• flowEQ = MATLAB + Python (Keras)

Reference

• GraphCast
• Shrink floating point format to accelerate DNN training
• h5cpp
• bfloat16
• Switch Transformer
• https://docs.nersc.gov/machinelearning/benchmarks/
• https://analyticsindiamag.com/face-emotion-recognizer-in-6-lines-of-code/
• https://semiengineering.com/the-best-ai-edge-inference-benchmark/
• REMOTE PATHOLOGICAL GAIT CLASSIFICATION SYSTEM (@mfolk)
• ai.gov
• SpaceML
• Mathematics for Machine Learning
• Neural Networks and Deep Learning
• Interpretable Machine Learning
• Applied ML
• A Review of Earth AI
• AI Builder in Power Platform
• Intel oneAPI AI Analytics Toolkit
• Open Catalyst 2020 (OC20) Dataset in LMDB format for Caffe
• SambaNova AI
• Horovod
• DeepHyper
• sits
• Drake
• ZSTD in training mode