01_switch-to-BIDS.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import mne\n",
    "import mne_bids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The assumed folder structure is a single project folder that contains\n",
    "the repo folder and the data folders 'BIOMAG2020_comp_data' with\n",
    "the original data and 'data_bids' with the BIDS data.\n",
    "\n",
    "TODO: switch to using environment variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_root = Path() / '..'\n",
    "original_data_dir = project_root / 'BIOMAG2020_comp_data'\n",
    "bids_root = project_root / 'data_bids'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convert to BIDS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "72"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The ctf data is organized in folders whose name end with \".ds\"\n",
    "folders = list(original_data_dir.glob('*.ds'))\n",
    "len(folders)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We encountered a couple of files that mne couldn't read.\n",
    "\n",
    "We just skipped over them manually by running the loop over all,\n",
    "letting it break with an error, then getting the index of the \n",
    "offender (say, `n`) and then continuing processing the folders\n",
    "after it (looping over `folders[n + 1:]`).\n",
    "\n",
    "TODO: make a list of the offending folders and skip them like adults.\n",
    "Otherwise, we can't re-run this script if we need to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ds directory : ..\\BIOMAG2020_comp_data\\WWIZIWSJ_rest_20120816_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "       3.39   74.50    0.00 mm <->    3.39   74.50    0.00 mm (orig :  -39.18   65.58 -284.93 mm) diff =    0.000 mm\n",
      "      -3.39  -74.50    0.00 mm <->   -3.39  -74.50    0.00 mm (orig :   59.41  -46.30 -287.69 mm) diff =    0.000 mm\n",
      "      87.38    0.00    0.00 mm <->   87.38    0.00    0.00 mm (orig :   71.76   68.81 -268.03 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\WWIZIWSJ_rest_20120816_01.ds\\WWIZIWSJ_rest_20120816_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 332 chs\n",
      "Current compensation grade : 3\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\WWIZIWSJ_rest_20120816_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "       3.39   74.50    0.00 mm <->    3.39   74.50    0.00 mm (orig :  -39.18   65.58 -284.93 mm) diff =    0.000 mm\n",
      "      -3.39  -74.50    0.00 mm <->   -3.39  -74.50    0.00 mm (orig :   59.41  -46.30 -287.69 mm) diff =    0.000 mm\n",
      "      87.38    0.00    0.00 mm <->   87.38    0.00    0.00 mm (orig :   71.76   68.81 -268.03 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\WWIZIWSJ_rest_20120816_01.ds\\WWIZIWSJ_rest_20120816_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 332 chs\n",
      "Current compensation grade : 3\n",
      "Effective window size : 0.213 (s)\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YDCCVVKE_rest_20160114_03.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "       2.11   71.57   -0.00 mm <->    2.11   71.57   -0.00 mm (orig :  -43.91   54.66 -279.13 mm) diff =    0.000 mm\n",
      "      -2.11  -71.57    0.00 mm <->   -2.11  -71.57    0.00 mm (orig :   66.35  -36.40 -286.80 mm) diff =    0.000 mm\n",
      "     100.66   -0.00    0.00 mm <->  100.65   -0.00    0.00 mm (orig :   71.32   84.03 -252.79 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YDCCVVKE_rest_20160114_03.ds\\YDCCVVKE_rest_20160114_03.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 301 chs\n",
      "Current compensation grade : 3\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YDCCVVKE_rest_20160114_03.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "       2.11   71.57   -0.00 mm <->    2.11   71.57   -0.00 mm (orig :  -43.91   54.66 -279.13 mm) diff =    0.000 mm\n",
      "      -2.11  -71.57    0.00 mm <->   -2.11  -71.57    0.00 mm (orig :   66.35  -36.40 -286.80 mm) diff =    0.000 mm\n",
      "     100.66   -0.00    0.00 mm <->  100.65   -0.00    0.00 mm (orig :   71.32   84.03 -252.79 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YDCCVVKE_rest_20160114_03.ds\\YDCCVVKE_rest_20160114_03.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 301 chs\n",
      "Current compensation grade : 3\n",
      "Effective window size : 0.213 (s)\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YDCCVVKE_rest_20160128_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "      -0.51   76.73    0.00 mm <->   -0.51   76.73   -0.00 mm (orig :  -33.06   71.33 -306.04 mm) diff =    0.000 mm\n",
      "       0.51  -76.73    0.00 mm <->    0.51  -76.73   -0.00 mm (orig :   74.17  -38.39 -302.10 mm) diff =    0.000 mm\n",
      "      96.07    0.00    0.00 mm <->   96.07   -0.00    0.00 mm (orig :   83.35   78.32 -265.86 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YDCCVVKE_rest_20160128_01.ds\\YDCCVVKE_rest_20160128_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 301 chs\n",
      "Current compensation grade : 3\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YDCCVVKE_rest_20160128_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "      -0.51   76.73    0.00 mm <->   -0.51   76.73   -0.00 mm (orig :  -33.06   71.33 -306.04 mm) diff =    0.000 mm\n",
      "       0.51  -76.73    0.00 mm <->    0.51  -76.73   -0.00 mm (orig :   74.17  -38.39 -302.10 mm) diff =    0.000 mm\n",
      "      96.07    0.00    0.00 mm <->   96.07   -0.00    0.00 mm (orig :   83.35   78.32 -265.86 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YDCCVVKE_rest_20160128_01.ds\\YDCCVVKE_rest_20160128_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 301 chs\n",
      "Current compensation grade : 3\n",
      "Effective window size : 0.213 (s)\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YFLOQXEZ_rest_20131107_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "      -2.33   69.43    0.00 mm <->   -2.33   69.43    0.00 mm (orig :  -36.94   53.91 -273.72 mm) diff =    0.000 mm\n",
      "       2.33  -69.43   -0.00 mm <->    2.33  -69.43   -0.00 mm (orig :   56.68  -48.63 -268.95 mm) diff =    0.000 mm\n",
      "      96.43   -0.00    0.00 mm <->   96.43   -0.00    0.00 mm (orig :   76.43   60.82 -232.83 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YFLOQXEZ_rest_20131107_01.ds\\YFLOQXEZ_rest_20131107_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 300 chs\n",
      "Current compensation grade : 3\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YFLOQXEZ_rest_20131107_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "      -2.33   69.43    0.00 mm <->   -2.33   69.43    0.00 mm (orig :  -36.94   53.91 -273.72 mm) diff =    0.000 mm\n",
      "       2.33  -69.43   -0.00 mm <->    2.33  -69.43   -0.00 mm (orig :   56.68  -48.63 -268.95 mm) diff =    0.000 mm\n",
      "      96.43   -0.00    0.00 mm <->   96.43   -0.00    0.00 mm (orig :   76.43   60.82 -232.83 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YFLOQXEZ_rest_20131107_01.ds\\YFLOQXEZ_rest_20131107_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 300 chs\n",
      "Current compensation grade : 3\n",
      "Effective window size : 0.213 (s)\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YFLOQXEZ_rest_20131121_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "       0.48   68.89    0.00 mm <->    0.48   68.89   -0.00 mm (orig :  -37.91   60.86 -266.78 mm) diff =    0.000 mm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      -0.48  -68.89    0.00 mm <->   -0.48  -68.89   -0.00 mm (orig :   55.38  -40.34 -273.35 mm) diff =    0.000 mm\n",
      "      94.78    0.00    0.00 mm <->   94.78    0.00    0.00 mm (orig :   71.56   66.23 -226.44 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YFLOQXEZ_rest_20131121_01.ds\\YFLOQXEZ_rest_20131121_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 300 chs\n",
      "Current compensation grade : 3\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YFLOQXEZ_rest_20131121_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "       0.48   68.89    0.00 mm <->    0.48   68.89   -0.00 mm (orig :  -37.91   60.86 -266.78 mm) diff =    0.000 mm\n",
      "      -0.48  -68.89    0.00 mm <->   -0.48  -68.89   -0.00 mm (orig :   55.38  -40.34 -273.35 mm) diff =    0.000 mm\n",
      "      94.78    0.00    0.00 mm <->   94.78    0.00    0.00 mm (orig :   71.56   66.23 -226.44 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YFLOQXEZ_rest_20131121_01.ds\\YFLOQXEZ_rest_20131121_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 300 chs\n",
      "Current compensation grade : 3\n",
      "Effective window size : 0.213 (s)\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YGSUCZFD_rest_20111027_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "      -0.98   76.40    0.00 mm <->   -0.98   76.40    0.00 mm (orig :  -46.74   67.28 -265.90 mm) diff =    0.000 mm\n",
      "       0.98  -76.40   -0.00 mm <->    0.98  -76.40   -0.00 mm (orig :   59.05  -42.98 -265.24 mm) diff =    0.000 mm\n",
      "      88.85    0.00    0.00 mm <->   88.85    0.00   -0.00 mm (orig :   69.34   71.31 -245.53 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YGSUCZFD_rest_20111027_01.ds\\YGSUCZFD_rest_20111027_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 332 chs\n",
      "Current compensation grade : 3\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YGSUCZFD_rest_20111027_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "      -0.98   76.40    0.00 mm <->   -0.98   76.40    0.00 mm (orig :  -46.74   67.28 -265.90 mm) diff =    0.000 mm\n",
      "       0.98  -76.40   -0.00 mm <->    0.98  -76.40   -0.00 mm (orig :   59.05  -42.98 -265.24 mm) diff =    0.000 mm\n",
      "      88.85    0.00    0.00 mm <->   88.85    0.00   -0.00 mm (orig :   69.34   71.31 -245.53 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YGSUCZFD_rest_20111027_01.ds\\YGSUCZFD_rest_20111027_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 332 chs\n",
      "Current compensation grade : 3\n",
      "Effective window size : 0.213 (s)\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YGSUCZFD_rest_20111110_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "     -18.59   55.03   -0.00 mm <->  -18.59   55.03   -0.00 mm (orig :   59.51  -42.41 -262.60 mm) diff =    0.000 mm\n",
      "      18.59  -55.03    0.00 mm <->   18.59  -55.03    0.00 mm (orig :   69.00   72.43 -247.84 mm) diff =    0.000 mm\n",
      "     123.43    0.00    0.00 mm <->  123.43    0.00    0.00 mm (orig :  -48.22   65.27 -263.03 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YGSUCZFD_rest_20111110_01.ds\\YGSUCZFD_rest_20111110_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 332 chs\n",
      "Current compensation grade : 3\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YGSUCZFD_rest_20111110_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "     -18.59   55.03   -0.00 mm <->  -18.59   55.03   -0.00 mm (orig :   59.51  -42.41 -262.60 mm) diff =    0.000 mm\n",
      "      18.59  -55.03    0.00 mm <->   18.59  -55.03    0.00 mm (orig :   69.00   72.43 -247.84 mm) diff =    0.000 mm\n",
      "     123.43    0.00    0.00 mm <->  123.43    0.00    0.00 mm (orig :  -48.22   65.27 -263.03 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YGSUCZFD_rest_20111110_01.ds\\YGSUCZFD_rest_20111110_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 332 chs\n",
      "Current compensation grade : 3\n",
      "Effective window size : 0.213 (s)\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YIWMSUPM_rest_20140424_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "      -0.53   74.18   -0.00 mm <->   -0.53   74.18   -0.00 mm (orig :  -43.13   62.60 -261.58 mm) diff =    0.000 mm\n",
      "       0.53  -74.18    0.00 mm <->    0.53  -74.18    0.00 mm (orig :   61.60  -42.40 -265.49 mm) diff =    0.000 mm\n",
      "      84.69   -0.00    0.00 mm <->   84.69    0.00    0.00 mm (orig :   67.94   66.98 -241.35 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YIWMSUPM_rest_20140424_01.ds\\YIWMSUPM_rest_20140424_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 300 chs\n",
      "Current compensation grade : 3\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YIWMSUPM_rest_20140424_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "      -0.53   74.18   -0.00 mm <->   -0.53   74.18   -0.00 mm (orig :  -43.13   62.60 -261.58 mm) diff =    0.000 mm\n",
      "       0.53  -74.18    0.00 mm <->    0.53  -74.18    0.00 mm (orig :   61.60  -42.40 -265.49 mm) diff =    0.000 mm\n",
      "      84.69   -0.00    0.00 mm <->   84.69    0.00    0.00 mm (orig :   67.94   66.98 -241.35 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YIWMSUPM_rest_20140424_01.ds\\YIWMSUPM_rest_20140424_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 300 chs\n",
      "Current compensation grade : 3\n",
      "Effective window size : 0.213 (s)\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YIWMSUPM_rest_20140508_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "       1.94   73.07    0.00 mm <->    1.94   73.07    0.00 mm (orig :  -41.58   62.04 -268.02 mm) diff =    0.000 mm\n",
      "      -1.94  -73.07   -0.00 mm <->   -1.94  -73.07   -0.00 mm (orig :   52.42  -49.92 -268.23 mm) diff =    0.000 mm\n",
      "      91.27    0.00    0.00 mm <->   91.27    0.00    0.00 mm (orig :   69.06   62.59 -235.19 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YIWMSUPM_rest_20140508_01.ds\\YIWMSUPM_rest_20140508_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 300 chs\n",
      "Current compensation grade : 3\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YIWMSUPM_rest_20140508_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "       1.94   73.07    0.00 mm <->    1.94   73.07    0.00 mm (orig :  -41.58   62.04 -268.02 mm) diff =    0.000 mm\n",
      "      -1.94  -73.07   -0.00 mm <->   -1.94  -73.07   -0.00 mm (orig :   52.42  -49.92 -268.23 mm) diff =    0.000 mm\n",
      "      91.27    0.00    0.00 mm <->   91.27    0.00    0.00 mm (orig :   69.06   62.59 -235.19 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YIWMSUPM_rest_20140508_01.ds\\YIWMSUPM_rest_20140508_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 300 chs\n",
      "Current compensation grade : 3\n",
      "Effective window size : 0.213 (s)\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YWRNZYIR_rest_20160420_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "      -2.55   70.62    0.00 mm <->   -2.55   70.62    0.00 mm (orig :  -41.01   61.36 -279.05 mm) diff =    0.000 mm\n",
      "       2.55  -70.62   -0.00 mm <->    2.55  -70.62   -0.00 mm (orig :   59.21  -38.27 -276.62 mm) diff =    0.000 mm\n",
      "      97.68   -0.00    0.00 mm <->   97.68   -0.00    0.00 mm (orig :   76.10   74.73 -245.30 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YWRNZYIR_rest_20160420_01.ds\\YWRNZYIR_rest_20160420_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 301 chs\n",
      "Current compensation grade : 3\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YWRNZYIR_rest_20160420_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "      -2.55   70.62    0.00 mm <->   -2.55   70.62    0.00 mm (orig :  -41.01   61.36 -279.05 mm) diff =    0.000 mm\n",
      "       2.55  -70.62   -0.00 mm <->    2.55  -70.62   -0.00 mm (orig :   59.21  -38.27 -276.62 mm) diff =    0.000 mm\n",
      "      97.68   -0.00    0.00 mm <->   97.68   -0.00    0.00 mm (orig :   76.10   74.73 -245.30 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YWRNZYIR_rest_20160420_01.ds\\YWRNZYIR_rest_20160420_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 301 chs\n",
      "Current compensation grade : 3\n",
      "Effective window size : 0.213 (s)\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YWRNZYIR_rest_20160505_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "      -2.23   71.59    0.00 mm <->   -2.23   71.59   -0.00 mm (orig :  -40.58   64.10 -269.75 mm) diff =    0.000 mm\n",
      "       2.23  -71.59   -0.00 mm <->    2.23  -71.59   -0.00 mm (orig :   61.08  -36.78 -272.93 mm) diff =    0.000 mm\n",
      "      98.51    0.00   -0.00 mm <->   98.51   -0.00   -0.00 mm (orig :   75.68   73.91 -228.97 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YWRNZYIR_rest_20160505_01.ds\\YWRNZYIR_rest_20160505_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 301 chs\n",
      "Current compensation grade : 3\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\YWRNZYIR_rest_20160505_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "      -2.23   71.59    0.00 mm <->   -2.23   71.59   -0.00 mm (orig :  -40.58   64.10 -269.75 mm) diff =    0.000 mm\n",
      "       2.23  -71.59   -0.00 mm <->    2.23  -71.59   -0.00 mm (orig :   61.08  -36.78 -272.93 mm) diff =    0.000 mm\n",
      "      98.51    0.00   -0.00 mm <->   98.51   -0.00   -0.00 mm (orig :   75.68   73.91 -228.97 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\YWRNZYIR_rest_20160505_01.ds\\YWRNZYIR_rest_20160505_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 301 chs\n",
      "Current compensation grade : 3\n",
      "Effective window size : 0.213 (s)\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\ZDIAXRUW_rest_20150115_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "      -1.59   66.89   -0.00 mm <->   -1.59   66.89   -0.00 mm (orig :  -32.88   62.95 -287.60 mm) diff =    0.000 mm\n",
      "       1.59  -66.89    0.00 mm <->    1.59  -66.89    0.00 mm (orig :   59.12  -34.08 -282.34 mm) diff =    0.000 mm\n",
      "      86.19   -0.00    0.00 mm <->   86.19   -0.00    0.00 mm (orig :   73.77   70.46 -260.22 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\ZDIAXRUW_rest_20150115_01.ds\\ZDIAXRUW_rest_20150115_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 300 chs\n",
      "Current compensation grade : 3\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\ZDIAXRUW_rest_20150115_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "      -1.59   66.89   -0.00 mm <->   -1.59   66.89   -0.00 mm (orig :  -32.88   62.95 -287.60 mm) diff =    0.000 mm\n",
      "       1.59  -66.89    0.00 mm <->    1.59  -66.89    0.00 mm (orig :   59.12  -34.08 -282.34 mm) diff =    0.000 mm\n",
      "      86.19   -0.00    0.00 mm <->   86.19   -0.00    0.00 mm (orig :   73.77   70.46 -260.22 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\ZDIAXRUW_rest_20150115_01.ds\\ZDIAXRUW_rest_20150115_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 300 chs\n",
      "Current compensation grade : 3\n",
      "Effective window size : 0.213 (s)\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\ZDIAXRUW_rest_20150129_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "       1.35   66.41    0.00 mm <->    1.35   66.41    0.00 mm (orig :  -35.10   60.45 -274.10 mm) diff =    0.000 mm\n",
      "      -1.35  -66.41    0.00 mm <->   -1.35  -66.41   -0.00 mm (orig :   53.28  -38.55 -268.39 mm) diff =    0.000 mm\n",
      "      86.82    0.00    0.00 mm <->   86.82    0.00    0.00 mm (orig :   69.29   68.47 -246.64 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\ZDIAXRUW_rest_20150129_01.ds\\ZDIAXRUW_rest_20150129_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    1 x 300000 = 300000 samples from 300 chs\n",
      "Current compensation grade : 3\n",
      "ds directory : ..\\BIOMAG2020_comp_data\\ZDIAXRUW_rest_20150129_01.ds\n",
      "    res4 data read.\n",
      "    hc data read.\n",
      "    Separate EEG position data file read.\n",
      "    Quaternion matching (desired vs. transformed):\n",
      "       1.35   66.41    0.00 mm <->    1.35   66.41    0.00 mm (orig :  -35.10   60.45 -274.10 mm) diff =    0.000 mm\n",
      "      -1.35  -66.41    0.00 mm <->   -1.35  -66.41   -0.00 mm (orig :   53.28  -38.55 -268.39 mm) diff =    0.000 mm\n",
      "      86.82    0.00    0.00 mm <->   86.82    0.00    0.00 mm (orig :   69.29   68.47 -246.64 mm) diff =    0.000 mm\n",
      "    Coordinate transformations established.\n",
      "    Polhemus data for 3 HPI coils added\n",
      "    Device coordinate locations for 3 HPI coils added\n",
      "    Measurement info composed.\n",
      "Finding samples for ..\\BIOMAG2020_comp_data\\ZDIAXRUW_rest_20150129_01.ds\\ZDIAXRUW_rest_20150129_01.meg4: \n",
      "    System clock channel is available, checking which samples are valid.\n",
      "    1 x 300000 = 300000 samples from 300 chs\n",
      "Current compensation grade : 3\n",
      "Effective window size : 0.213 (s)\n"
     ]
    }
   ],
   "source": [
    "for folder in folders[59:]:\n",
    "    raw = mne.io.read_raw_ctf(str(folder))\n",
    "    bids_basename = mne_bids.make_bids_basename(\n",
    "        subject=raw.info['subject_info']['his_id'],\n",
    "        # There is a session number (1 or 2) in the folder name.\n",
    "        # We didn't want to use\n",
    "        session=str(raw.info['meas_id']['secs']),\n",
    "        task='restingstate'\n",
    "    )\n",
    "\n",
    "    mne_bids.write_raw_bids(raw=raw, bids_basename=bids_basename, bids_root=bids_root, verbose=False, overwrite=True) #overwrite doesnt work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the loop above breaks, `folder` points to the recording we couldn't read.\n",
    "Below, we find its index and then restart the loop starting with the folder\n",
    "after the offender."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "58"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "folders.index(folder)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}