demo_f0.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9f1dfc44",
   "metadata": {},
   "source": [
    "# Sonification of f0 Annotations\n",
    "In this notebook, we illustrate the sonification of fundamental frequency annotations using the ```libsoni.core.f0``` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e00b7485",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "import librosa\n",
    "import json\n",
    "\n",
    "from IPython import display as ipd\n",
    "\n",
    "from libsoni.core.f0 import sonify_f0\n",
    "from libsoni.util.utils import mix_sonification_and_original\n",
    "\n",
    "Fs = 22050\n",
    "\n",
    "AUDIO_DIR = 'data_audio'\n",
    "CSV_DIR = 'data_csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f01b69e",
   "metadata": {},
   "source": [
    "## Simple Scenario: C Major Triad\n",
    "\n",
    "To start with a simple example, let's look at the fundamental frequencies of a **C major triad**.\n",
    "\n",
    "<img src=\"figures/demo_f0/C-Dur-DM.png\" alt=\"C-Major-Triad\" width=\"250\">\n",
    "\n",
    "The frequencies corresponding to the notes are:\n",
    "\n",
    "| Note | Frequency (Hz) |\n",
    "|------|----------------|\n",
    "| C4   |     261.63     |\n",
    "| E4   |     329.63     |\n",
    "| G4   |     392.00     |\n",
    "| C5   |     523.25     |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "104c019c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define list of frequencies for C Major triad\n",
    "C_Major_Triad_f0s = [261.63, 329.63, 392.00, 523.25, 0.0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4924c6d",
   "metadata": {},
   "source": [
    "To sonify these notes, or more precisely their frequencies, we need an array of time positions at which the notes are to be played. Let's play the first note at 0.25 seconds for 0.5 seconds and all other notes consecutively for 0.5 seconds as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "402584eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define starting time position in seconds\n",
    "start_sec = 0.25\n",
    "\n",
    "# Define duration for each frequency\n",
    "duration_note_sec = 0.5\n",
    "\n",
    "# Create array from time information\n",
    "time_positions = np.arange(start_sec, len(C_Major_Triad_f0s) * duration_note_sec, duration_note_sec)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aea3d5e9",
   "metadata": {},
   "source": [
    "The function ```sonify_f0``` from the module ```f0``` takes a Nx2-dimensional ```numpy.ndarray``` containing the time positions in the first column and the f0s in the second column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "096717e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create Nx2-dimenstioinal numpy.ndarray time_f0\n",
    "time_f0 = np.column_stack((time_positions, C_Major_Triad_f0s))\n",
    "\n",
    "# Display time_f0 as Pandas DataFrame\n",
    "time_f0_df = pd.DataFrame(np.column_stack((time_positions, C_Major_Triad_f0s)), columns =['start','f0'])\n",
    "ipd.display(time_f0_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3f15b41",
   "metadata": {},
   "source": [
    "### Sonified C Major Triad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34e518c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sonification using libsoni\n",
    "sonified_C_Major_Triad_f0 = sonify_f0(time_f0=time_f0, fs=Fs)\n",
    "\n",
    "print('Sonified C Major triad:')\n",
    "ipd.display(ipd.Audio(sonified_C_Major_Triad_f0, rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b45e94d",
   "metadata": {},
   "source": [
    "### Customizing the Sonification\n",
    "To adjust the sonification, the function ```sonify_f0``` offers the possibility to set the so-called partial frequencies as well as their amplitudes to create a certain timbre. Let's say we want to use the fundamental frequency f0 as well as two times, three times and four times the fundamental frequency for sonification. For the amplitudes we want to set 1, 1/2, 1/3, 1/4.\n",
    "\n",
    "| Custom frequencies | Custom amplitudes |\n",
    "|--------------------|------------------|\n",
    "| f0                 | 1                |\n",
    "| 2*f0               | 1/2              |\n",
    "| 3*f0               | 1/3              |\n",
    "| 4*f0               | 1/4              |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0422961f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set custom_partials\n",
    "custom_partials = np.array([1,2,3,4])\n",
    "\n",
    "# Set amplitudes for custom_partials\n",
    "custom_partials_amplitudes = np.array([1,1/2,1/3,1/4])\n",
    "\n",
    "# Sonification with custom parital settings using libsoni\n",
    "sonified_C_Major_Triad_f0 = sonify_f0(time_f0=time_f0,\n",
    "                                      partials=custom_partials,\n",
    "                                      partials_amplitudes=custom_partials_amplitudes,\n",
    "                                      fs=Fs)\n",
    "\n",
    "\n",
    "print('Sonified C Major Triad with custom parials:')\n",
    "ipd.display(ipd.Audio(sonified_C_Major_Triad_f0, rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f46e0416",
   "metadata": {},
   "source": [
    "## Scenario 1: *Ach Gott und Herr* by *J.S. Bach*\n",
    "\"Ach Gott und Herr\" is a sacred choral composition by the Baroque composer Johann Sebastian Bach, taken from. The piece typically features a four-part choir, supported by an instrumental ensemble. Bach often employed various instrumental combinations to enhance the choral texture and evoke different emotional layers. The following excerpt comprises a violin, a clarinet, a saxophone and a bassoon. This example is taken from the <a href=\"https://ieeexplore.ieee.org/document/5404324\">Bach10</a> dataset.\n",
    "\n",
    "\n",
    "\n",
    "<img src=\"figures/demo_f0/01-AchGottundHerr_score.png\" alt=\"Locus Iste\" width=\"500\" height=\"800\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9ca0f71",
   "metadata": {},
   "outputs": [],
   "source": [
    "bach_audio, _ = librosa.load(os.path.join(AUDIO_DIR,'demo_f0','01-AchGottundHerr.wav'), sr=Fs)\n",
    "\n",
    "print('\"Ach Gott und Herr\", by J. S. Bach:')\n",
    "ipd.display(ipd.Audio(bach_audio, rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "724e2f07",
   "metadata": {},
   "source": [
    "### Preparing the Data\n",
    "For the excerpt above, we load the corresponding .csv tables containing the time and f0 information for each instrument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "348a0021",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load .csv-data for each instrument\n",
    "violin_df = pd.read_csv(os.path.join(CSV_DIR,'demo_f0','01-AchGottundHerr_violin.csv'), sep=';')\n",
    "clarinet_df = pd.read_csv(os.path.join(CSV_DIR,'demo_f0','01-AchGottundHerr_clarinet.csv'), sep=';')\n",
    "saxophone_df = pd.read_csv(os.path.join(CSV_DIR,'demo_f0','01-AchGottundHerr_saxophone.csv'), sep=';')\n",
    "bassoon_df = pd.read_csv(os.path.join(CSV_DIR,'demo_f0','01-AchGottundHerr_bassoon.csv'), sep=';')\n",
    "\n",
    "print('Extract from violin_df:')\n",
    "ipd.display(violin_df.iloc[100:105])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "973d3842",
   "metadata": {},
   "source": [
    "### Sonification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40a17979",
   "metadata": {},
   "outputs": [],
   "source": [
    "bach_sonified = sonify_f0(violin_df.to_numpy(),\n",
    "                          fs=Fs)\n",
    "bach_sonified += sonify_f0(clarinet_df.to_numpy(),\n",
    "                           fs=Fs)\n",
    "bach_sonified += sonify_f0(saxophone_df.to_numpy(),\n",
    "                           fs=Fs)\n",
    "bach_sonified += sonify_f0(bassoon_df.to_numpy(),\n",
    "                           fs=Fs)\n",
    "                            \n",
    "\n",
    "bach_sonified_w_original = mix_sonification_and_original(sonification=bach_sonified,\n",
    "                                                         original_audio=bach_audio,\n",
    "                                                         gain_lin_sonification=1.0,\n",
    "                                                         gain_lin_original_audio=0.5)\n",
    "\n",
    "print('Original audio:')\n",
    "ipd.display(ipd.Audio(bach_audio, rate=Fs))\n",
    "\n",
    "print('Sonified with libsoni:')\n",
    "ipd.display(ipd.Audio(bach_sonified, rate=Fs))\n",
    "\n",
    "print('Original audio with sonification (stereo):')\n",
    "ipd.display(ipd.Audio(bach_sonified_w_original, rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "336f5b8c",
   "metadata": {},
   "source": [
    "## Scenario 2: 'SATB'  *Locus Iste* by *Anton Bruckner* \n",
    "\"Locus Iste\" is a sacred motet composed by Anton Bruckner, a renowned Austrian composer of the Romantic era. This composition, often performed in choral settings, showcases Bruckner's mastery of harmonies and expressive depth.\n",
    "\"SATB\" is an abbreviation used to describe the voicing and arrangement of a choir in choral music. It stands for Soprano, Alto, Tenor, and Bass, representing the four main vocal ranges in a choir. When applied to \"Locus Iste\" by Anton Bruckner, which is a choral composition, SATB signifies how the voices are organized and distributed within the piece. This example is taken from the <a href=\"https://www.audiolabs-erlangen.de/resources/MIR/2020-DagstuhlChoirSet\">Dagstuhl Choirset</a>.\n",
    "\n",
    "<img src=\"figures/demo_f0/Locus_iste_score.png\" alt=\"Locus Iste\" width=\"500\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0facf3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "satb_audio, _ = librosa.load(os.path.join(AUDIO_DIR,\n",
    "                                          'demo_f0',\n",
    "                                          'DCS_LI_QuartetA_Take04_StereoReverb_STM.wav'), sr=Fs)\n",
    "\n",
    "print('\"Locus Iste\" by Anton Bruckner:')\n",
    "ipd.display(ipd.Audio(satb_audio, rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c935a883",
   "metadata": {},
   "source": [
    "### Preparing the Data\n",
    "For the excerpt above, we load the corresponding .csv tables containing the time and f0 information for each voice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c90785ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load .csv-data for each instrument\n",
    "soprano_df = pd.read_csv(os.path.join(CSV_DIR,'demo_f0','DCS_LI_QuartetA_Take04_S2_LRX.csv'), sep=';')\n",
    "alto_df = pd.read_csv(os.path.join(CSV_DIR,'demo_f0','DCS_LI_QuartetA_Take04_A1_LRX.csv'), sep=';')\n",
    "tenor_df = pd.read_csv(os.path.join(CSV_DIR,'demo_f0','DCS_LI_QuartetA_Take04_T1_LRX.csv'), sep=';')\n",
    "bass_df = pd.read_csv(os.path.join(CSV_DIR,'demo_f0','DCS_LI_QuartetA_Take04_B1_LRX.csv'), sep=';')\n",
    "\n",
    "print('Extract from soprano_df:')\n",
    "ipd.display(soprano_df.iloc[100:105])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6e0f625",
   "metadata": {},
   "source": [
    "As in the previous example, we again arrange the data as a dictionary. This time we use the presets **soprano**, **alto**, **tenor** and **bass**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d85c52d",
   "metadata": {},
   "outputs": [],
   "source": [
    "satb_sonified = sonify_f0(soprano_df.to_numpy(),\n",
    "                          fs=Fs)\n",
    "satb_sonified += sonify_f0(alto_df.to_numpy(),\n",
    "                           fs=Fs)\n",
    "satb_sonified += sonify_f0(tenor_df.to_numpy(),\n",
    "                           fs=Fs)\n",
    "satb_sonified += sonify_f0(bass_df.to_numpy(),\n",
    "                           fs=Fs)\n",
    "\n",
    "satb_sonified_w_original = mix_sonification_and_original(sonification=satb_sonified,\n",
    "                                                         original_audio=satb_audio,\n",
    "                                                         gain_lin_original_audio=0.05)\n",
    "\n",
    "print('Original audio:')\n",
    "ipd.display(ipd.Audio(satb_audio, rate=Fs))\n",
    "\n",
    "print('Sonified with libsoni')\n",
    "ipd.display(ipd.Audio(satb_sonified, rate=Fs))\n",
    "\n",
    "print('Original audio with sonification (stereo)')\n",
    "ipd.display(ipd.Audio(satb_sonified_w_original, rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35270f45",
   "metadata": {},
   "source": [
    "### Incorporating confidence\n",
    "If f0 annotations with confidence values are available, these can be used to give corresponding f0 sections a gain corresponding to the confidence.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5f80302",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load .csv-data for each instrument\n",
    "soprano_df = pd.read_csv(os.path.join(CSV_DIR,'demo_f0','DCS_LI_QuartetA_Take01_S2_LRX_with_confidence.csv'), sep=';')\n",
    "alto_df = pd.read_csv(os.path.join(CSV_DIR,'demo_f0','DCS_LI_QuartetA_Take01_A1_LRX_with_confidence.csv'), sep=';')\n",
    "tenor_df = pd.read_csv(os.path.join(CSV_DIR,'demo_f0','DCS_LI_QuartetA_Take01_T1_LRX_with_confidence.csv'), sep=';')\n",
    "bass_df = pd.read_csv(os.path.join(CSV_DIR,'demo_f0','DCS_LI_QuartetA_Take01_B1_LRX_with_confidence.csv'), sep=';')\n",
    "\n",
    "\n",
    "soprano_confidences = soprano_df['confidence'].to_numpy()\n",
    "soprano_df = soprano_df[['start','f0']]\n",
    "\n",
    "alto_confidences = alto_df['confidence'].to_numpy()\n",
    "alto_df = alto_df[['start','f0']]\n",
    "\n",
    "tenor_confidences = tenor_df['confidence'].to_numpy()\n",
    "tenor_df = tenor_df[['start','f0']]\n",
    "\n",
    "bass_confidences = bass_df['confidence'].to_numpy()\n",
    "bass_df = bass_df[['start','f0']]\n",
    "\n",
    "print('Extract from soprano_df:')\n",
    "ipd.display(soprano_df[['start','f0']].iloc[100:105])\n",
    "\n",
    "\n",
    "\n",
    "satb_sonified_w_original = mix_sonification_and_original(sonification=satb_sonified,\n",
    "                                                         original_audio=satb_audio,\n",
    "                                                         gain_lin_original_audio=0.05)\n",
    "\n",
    "print('Original audio:')\n",
    "ipd.display(ipd.Audio(satb_audio, rate=Fs))\n",
    "\n",
    "print('Sonified with libsoni')\n",
    "ipd.display(ipd.Audio(satb_sonified, rate=Fs))\n",
    "\n",
    "print('Original audio with sonification (stereo)')\n",
    "ipd.display(ipd.Audio(satb_sonified_w_original, rate=Fs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef164772",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "**[1]** Zhiyao Duan, Bryan Pardo and Changshui Zhang, “Multiple fundamental frequency estimation by modeling spectral peaks and non-peak regions,” IEEE Transactions of Audio Speech Language Process., vol. 18, no. 8, pp. 2121–2133, 2010.\n",
    "\n",
    "**[2]** S. Rosenzweig, H. Cuesta, C. Weiß, F. Scherbaum, E. Gómez, and M. Müller, “Dagstuhl ChoirSet: A multitrack dataset for MIR research on choral singing,” Transactions of the International Society for Music Information Retrieval (TISMIR), vol. 3, no. 1, pp. 98–110, 2020."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}