05-io-streams.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IO and streams\n",
    "\n",
    ".NET provides classes for working with files and streaming data. This notebook will go through some of these classes and some examples how to make best use of them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with files and directories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "using System.IO;\n",
    "\n",
    "// DirectoryInfo class can be used to get information about the file system\n",
    "var directoryInfo = new DirectoryInfo(\".\");\n",
    "\n",
    "foreach (DirectoryInfo directory in directoryInfo.GetDirectories())\n",
    "{\n",
    "    Console.WriteLine(directory.Name);\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Similarly FileInfo can be used to get information about files\n",
    "foreach (FileInfo file in directoryInfo.GetFiles())\n",
    "{\n",
    "    Console.WriteLine(file.Name);\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Both classes can be leverage together to get the full information about the available files and directories\n",
    "\n",
    "void PrintContentsRecursively(DirectoryInfo root, int depth = 0)\n",
    "{\n",
    "    foreach (var directory in root.GetDirectories())\n",
    "    {\n",
    "        Console.WriteLine($\"{new String('-', depth)} {directory.Name}\");\n",
    "        PrintContentsRecursively(directory, depth + 1);\n",
    "    }\n",
    "\n",
    "    foreach (var file in root.GetFiles())\n",
    "    {\n",
    "        Console.WriteLine($\"{new String('-', depth)} {file.Name}\");\n",
    "    }\n",
    "\n",
    "    return;\n",
    "}\n",
    "\n",
    "PrintContentsRecursively(new DirectoryInfo(\".\"), 0);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Any errors will be provded as exceptions\n",
    "\n",
    "var directory = new DirectoryInfo(\"K://\");\n",
    "\n",
    "try\n",
    "{\n",
    "    directory.GetDirectories();\n",
    "}\n",
    "catch (DirectoryNotFoundException)\n",
    "{\n",
    "    Console.WriteLine(\"Directory does not exist\");\n",
    "}\n",
    "catch (UnauthorizedAccessException)\n",
    "{\n",
    "    Console.WriteLine(\"Cannot access\");\n",
    "}\n",
    "\n",
    "var file = new FileInfo(\"fileeee\");\n",
    "\n",
    "try\n",
    "{\n",
    "    file.OpenRead();\n",
    "}\n",
    "catch (FileNotFoundException)\n",
    "{\n",
    "    Console.WriteLine(\"File does not exist\");\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading from files\n",
    "\n",
    "In simplest cases files can be read and written to using the `File` class. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Write to File\n",
    "\n",
    "File.WriteAllText(\"test.txt\", \"Hello World\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Read from the same file\n",
    "\n",
    "Console.WriteLine(File.ReadAllText(\"test.txt\"));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// File can be written line by line\n",
    "\n",
    "for (int i = 0; i < 10; i++)\n",
    "{\n",
    "    File.AppendAllText(\"test.txt\", $\"Line {i}\\n\");\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// File can also be read line by line\n",
    "\n",
    "foreach (var line in File.ReadLines(\"test.txt\"))\n",
    "{\n",
    "    Console.WriteLine(line);\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// File can be cleared\n",
    "// WriteLine appends to the end of file, while WriteAllText clears the file and writes the content\n",
    "\n",
    "File.WriteAllText(\"test.txt\", \"\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Streams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// File can be written to using the stream\n",
    "\n",
    "// Passing the string argument to the StreamWriter constructor will create the file to write to\n",
    "// However there are constructor overload for other cases like writing to other streams and such\n",
    "using (var streamWriter = new StreamWriter(\"test.txt\"))\n",
    "{\n",
    "    streamWriter.WriteLine(\"Hello World\");\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Stream allows to continually write to the file with the same handle\n",
    "\n",
    "// Writing byte by byte\n",
    "using (var fileStream = new FileStream(\"test.txt\", FileMode.Append))\n",
    "{\n",
    "    for (int i = 0; i < 10; i++)\n",
    "    {\n",
    "        fileStream.WriteByte((byte)i);\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Using the stream to write to file you don't need to have all the data in memory at once,\n",
    "// but it can be created and written in chunks\n",
    "\n",
    "using (var fileStream = new FileStream(\"test.txt\", FileMode.Create))\n",
    "for (var i = 0; i < 1_000_000; i++)\n",
    "{\n",
    "    // format i to 7 digit string and convert it to byte array\n",
    "    var byteString = Encoding.UTF8.GetBytes(i.ToString(\"D7\") + \"\\n\");\n",
    "\n",
    "    fileStream.Write(byteString);\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Using a stream reader, bytes at an offset can be read\n",
    "\n",
    "using (var fileStream = new FileStream(\"test.txt\", FileMode.Open))\n",
    "{\n",
    "    var buffer = new byte[8];\n",
    "    fileStream.Seek(8 * 123556, SeekOrigin.Begin);\n",
    "    fileStream.Read(buffer, 0, 8);\n",
    "    Console.WriteLine(Encoding.UTF8.GetString(buffer));\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example of processing large file\n",
    "\n",
    "This example will focus on how to read large from the disk while still maintaining small memory footprint in the process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Seed a file with lots of random numbers\n",
    "using System.IO;\n",
    "\n",
    "using (var fileStream = new FileStream(\"test.txt\", FileMode.Create))\n",
    "{\n",
    "    var random = new Random();\n",
    "    // Will effectively write 10000000 * 8 bytes to file (~800MB)\n",
    "\n",
    "    for (var i = 0; i < 100_000_000; i++)\n",
    "    {\n",
    "        var number = random.Next(0, 100_000_00);\n",
    "        var byteString = Encoding.UTF8.GetBytes(number.ToString(\"D7\") + \"\\n\");\n",
    "        fileStream.Write(byteString);\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "void PrintMemoryUsage()\n",
    "{\n",
    "    // Not the best or most accurate way to measure memory usage\n",
    "    GC.Collect();\n",
    "    long memory = GC.GetTotalMemory(true);\n",
    "    Console.WriteLine(memory);\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Find the biggest number inside the file\n",
    "using System.IO;\n",
    "\n",
    "Console.WriteLine(\"Before reading to memory:\");\n",
    "PrintMemoryUsage();\n",
    "\n",
    "var lines = File.ReadAllLines(\"test.txt\");\n",
    "int max = int.MinValue;\n",
    "foreach (var line in lines)\n",
    "{\n",
    "    var number = int.Parse(line);\n",
    "\n",
    "    if (number > max)\n",
    "    {\n",
    "        max = number;\n",
    "    }\n",
    "}\n",
    "\n",
    "Console.WriteLine($\"Biggest number: {max}\");\n",
    "\n",
    "Console.WriteLine(\"After reading to memory:\");\n",
    "PrintMemoryUsage();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Find the biggest number while stream reading from the file\n",
    "\n",
    "using System.IO;\n",
    "\n",
    "Console.WriteLine(\"Before reading to memory:\");\n",
    "PrintMemoryUsage();\n",
    "\n",
    "int max = int.MinValue;\n",
    "using (var fileStream = new FileStream(\"test.txt\", FileMode.Open))\n",
    "{\n",
    "    var buffer = new byte[8];\n",
    "    for (var i = 0; i < 100_000_000; i++)\n",
    "    {\n",
    "        fileStream.Read(buffer, 0, 8);\n",
    "        var str = Encoding.UTF8.GetString(buffer).Trim();\n",
    "\n",
    "        var number = int.Parse(str);\n",
    "\n",
    "        if (number > max)\n",
    "        {\n",
    "            max = number;\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "Console.WriteLine($\"Biggest number: {max}\");\n",
    "\n",
    "Console.WriteLine(\"After reading to memory:\");\n",
    "PrintMemoryUsage();"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".NET (C#)",
   "language": "C#",
   "name": ".net-csharp"
  },
  "language_info": {
   "name": "polyglot-notebook"
  },
  "polyglot_notebook": {
   "kernelInfo": {
    "defaultKernelName": "csharp",
    "items": [
     {
      "aliases": [],
      "languageName": "csharp",
      "name": "csharp"
     }
    ]
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}