diff --git a/README.md b/README.md index 389faa7..1f2c521 100644 --- a/README.md +++ b/README.md @@ -1,30 +1,40 @@ ![img](https://raw.githubusercontent.com/koculu/ZoneTree/main/src/ZoneTree/docs/ZoneTree/images/logo2.png) # ZoneTree + ZoneTree is a persistent, high-performance, transactional, and ACID-compliant [ordered key-value database](https://en.wikipedia.org/wiki/Ordered_Key-Value_Store) for .NET. It can operate in memory or on local/cloud storage. +![License](https://img.shields.io/badge/license-MIT-blue.svg) [![Downloads](https://img.shields.io/nuget/dt/ZoneTree)](https://www.nuget.org/packages/ZoneTree/) +![Platform](https://img.shields.io/badge/platform-.NET-blue.svg) +![Build](https://img.shields.io/badge/build-passing-brightgreen.svg) [![](https://dcbadge.vercel.app/api/server/d9aDtzVNNv?logoColor=f1c400&theme=discord&style=flat)](https://discord.gg/d9aDtzVNNv) ZoneTree is a lightweight, transactional and high-performance LSM Tree for .NET. It is several times faster than Facebook's RocksDB and hundreds of times faster than SQLite. It is faster than any other alternative that I have tested so far. -100 Million integer key-value pair inserts in 20 seconds. You may get longer durations based on the durability level. +100 Million integer key-value pair inserts in 20 seconds. You may get longer durations based on the durability level. For example, with async-compressed WAL mode, you can insert 100M integer key-value pairs in 28 seconds. Background merge operation that might take a bit longer is excluded from the insert duration because your inserted data is immediately queryable. Loading 100M integer key-value pair database is in 812 ms. The iteration on 100M key-value pairs takes 24 seconds. There are so many tuning options wait you to discover. - ## [INTRODUCTION](https://tenray.io/docs/ZoneTree/guide/introduction.html) + ## [QUICK START GUIDE](https://tenray.io/docs/ZoneTree/guide/quick-start.html) + ## [API DOCS](https://tenray.io/docs/ZoneTree/api/Tenray.ZoneTree.html) + ## [TUNING ZONETREE](https://tenray.io/docs/ZoneTree/guide/tuning-disk-segment.html) + ## [FEATURES](https://tenray.io/docs/ZoneTree/guide/features.html) + ## [TERMINOLOGY](https://tenray.io/docs/ZoneTree/guide/terminology.html) + ## [PERFORMANCE](https://tenray.io/docs/ZoneTree/guide/performance.html) ## Why ZoneTree? + 1. It is pure C#. 2. It is fast. See benchmark below. 3. Your data is protected against crashes / power cuts (optional). @@ -33,6 +43,7 @@ There are so many tuning options wait you to discover. 6. You can create scalable and non-scalable databases using ZoneTree as core database engine. ## How fast is it? + It is possible with ZoneTree to insert 100 Million integer key-value pairs in 20 seconds using WAL mode = NONE. Benchmark for all modes: [benchmark](https://raw.githubusercontent.com/koculu/ZoneTree/main/src/Playground/BenchmarkForAllModes.txt) @@ -53,6 +64,7 @@ Benchmark for all modes: [benchmark](https://raw.githubusercontent.com/koculu/Zo | | Benchmark Configuration: + ```c# DiskCompressionBlockSize = 1024 * 1024 * 10; WALCompressionBlockSize = 1024 * 32 * 8; @@ -67,21 +79,22 @@ Tested up to 1 billion records in desktop computers till now. ### ZoneTree offers 4 WAL modes to let you make a flexible tradeoff. -* The sync mode provides maximum durability but slower write speed. - In case of a crash/power cut, the sync mode ensures that the inserted data is not lost. +- The sync mode provides maximum durability but slower write speed. + In case of a crash/power cut, the sync mode ensures that the inserted data is not lost. -* The sync-compressed mode provides faster write speed but less durability. +- The sync-compressed mode provides faster write speed but less durability. Compression requires chunks to be filled before appending them into the WAL file. It is possible to enable a periodic job to persist decompressed tail records into a separate location in a specified interval. See IWriteAheadLogProvider options for more details. -* The async-compressed mode provides faster write speed but less durability. +- The async-compressed mode provides faster write speed but less durability. Log entries are queued to be written in a separate thread. async-compressed mode uses compression in WAL files and provides immediate tail record persistence. -* None WAL mode disables WAL completely to get maximum performance. Data still can be saved to disk by tree maintainer automatically or manually. +- None WAL mode disables WAL completely to get maximum performance. Data still can be saved to disk by tree maintainer automatically or manually. ### Environment: + ``` BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000 Intel Core i7-6850K CPU 3.60GHz (Skylake), 1 CPU, 12 logical and 6 physical cores @@ -101,6 +114,7 @@ zoneTree.Upsert(39, "Hello Zone Tree"); ``` The following sample demonstrates creating a database. + ```c# var dataPath = "data/mydatabase"; using var zoneTree = new ZoneTreeFactory() @@ -109,23 +123,26 @@ The following sample demonstrates creating a database. .SetKeySerializer(new Int32Serializer()) .SetValueSerializer(new Utf8StringSerializer()) .OpenOrCreate(); - + // atomic (thread-safe) on single mutable-segment. zoneTree.Upsert(39, "Hello Zone Tree!"); - + // atomic across all segments - zoneTree.TryAtomicAddOrUpdate(39, "a", - bool (ref string x) => + zoneTree.TryAtomicAddOrUpdate(39, "a", + bool (ref string x) => { x += "b"; return true; }); ``` + ## How to maintain LSM Tree? + Big LSM Trees require maintenance tasks. ZoneTree provides the IZoneTreeMaintenance interface to give you full power on maintenance tasks. It also comes with a default maintainer to let you focus on your business logic without wasting time with LSM details. You can start using the default maintainer like in the following sample code. Note: For small data you don't need a maintainer. + ```c# var dataPath = "data/mydatabase"; @@ -136,7 +153,7 @@ Note: For small data you don't need a maintainer. .SetKeySerializer(new Int32Serializer()) .SetValueSerializer(new Utf8StringSerializer()) .OpenOrCreate(); - + using var maintainer = zoneTree.CreateMaintainer(); maintainer.EnableJobForCleaningInactiveCaches = true; @@ -148,43 +165,57 @@ Note: For small data you don't need a maintainer. ``` ## How to delete keys? -In LSM trees, the deletions are handled by upserting key/value with deleted flag. -Later on, during the compaction stage, the actual deletion happens. -ZoneTree does not implement this flag format by default. It lets the user to define the suitable deletion flag themselves. -For example, the deletion flag might be defined by user as -1 for int values. -If user wants to use any int value as a valid record, then the value-type should be changed. -For example, one can define the following struct and use this type as a value-type. + +In Log-Structured Merge (LSM) trees, deletions are managed by upserting a key/value pair with a deletion marker. The actual removal of the data occurs during the compaction stage. In ZoneTree, by default, the system assumes that the default values indicate deletion. However, you can customize this behavior by defining a specific deletion flag, such as using -1 for integer values or completely disable deletion by calling DisableDeletion method. + +### Custom Deletion Flag + +If you need more control over how deletions are handled, you can define a custom structure to represent your values and their deletion status. For example: + ```c# [StructLayout(LayoutKind.Sequential)] struct MyDeletableValueType { - int Number; - bool IsDeleted; + int Number; + bool IsDeleted; } ``` -You can micro-manage the tree size with ZoneTree. -The following sample shows how to configure the deletion markers for your database. + +This struct allows you to include a boolean flag indicating whether a value is deleted. You can then use this custom type as the value type in your ZoneTree. + +### Configuring Deletion Markers + +ZoneTree provides flexibility in managing the tree size by allowing you to configure how deletion markers are set and identified. Below are examples of how you can configure these markers for your database: + +#### Example 1: Using an Integer Deletion Flag + +In this example, -1 is used as the deletion marker for integer values: + ```c# using var zoneTree = new ZoneTreeFactory() // Additional stuff goes here .SetIsValueDeletedDelegate((in int x) => x == -1) .SetMarkValueDeletedDelegate((ref int x) => x = -1) - .OpenOrCreate(); + .OpenOrCreate(); ``` -or + +#### Example 2: Using a Custom Struct for Deletion + +Alternatively, if you're using a custom struct to manage deletions, you can configure ZoneTree to recognize and mark deletions as follows: + ```c# using var zoneTree = new ZoneTreeFactory() // Additional stuff goes here .SetIsValueDeletedDelegate((in MyDeletableValueType x) => x.IsDeleted) .SetMarkValueDeletedDelegate((ref MyDeletableValueType x) => x.IsDeleted = true) - .OpenOrCreate(); + .OpenOrCreate(); ``` -If you forget to provide the deletion marker delegates, you can never delete the record from your database. ## How to iterate over data? Iteration is possible in both directions, forward and backward. Unlike other LSM tree implementations, iteration performance is equal in both directions. The following sample shows how to do the iteration. + ```c# using var zoneTree = new ZoneTreeFactory() // Additional stuff goes here @@ -193,8 +224,8 @@ The following sample shows how to do the iteration. while(iterator.Next()) { var key = iterator.CurrentKey; var value = iterator.CurrentValue; - } - + } + using var reverseIterator = zoneTree.CreateReverseIterator(); while(reverseIterator.Next()) { var key = reverseIterator.CurrentKey; @@ -206,26 +237,28 @@ The following sample shows how to do the iteration. ZoneTreeIterator provides Seek() method to jump into any record with in O(log(n)) complexity. That is useful for doing prefix search with forward-iterator or with backward-iterator. + ```c# using var zoneTree = new ZoneTreeFactory() // Additional stuff goes here .OpenOrCreate(); using var iterator = zoneTree.CreateIterator(); - // iterator jumps into the first record starting with "SomePrefix" in O(log(n)) complexity. + // iterator jumps into the first record starting with "SomePrefix" in O(log(n)) complexity. iterator.Seek("SomePrefix"); - + //iterator.Next() complexity is O(1) while(iterator.Next()) { var key = iterator.CurrentKey; var value = iterator.CurrentValue; - } + } ``` - ## Transaction Support + ZoneTree supports Optimistic Transactions. It is proud to announce that the ZoneTree is ACID-compliant. Of course, you can use non-transactional API for the scenarios where eventual consistency is sufficient. ZoneTree supports 3 way of doing transactions. + 1. Fluent Transactions with ready to use retry capability. 2. Classical Transaction API. 3. Exceptionless Transaction API. @@ -254,11 +287,12 @@ using var transaction = ``` The following sample shows traditional way of doing transactions with ZoneTree. + ```c# using var zoneTree = new ZoneTreeFactory() // Additional stuff goes here .OpenOrCreateTransactional(); - try + try { var txId = zoneTree.BeginTransaction(); zoneTree.TryGet(txId, 3, out var value); @@ -277,6 +311,7 @@ The following sample shows traditional way of doing transactions with ZoneTree. ``` ## Features + | ZoneTree Features | | --------------------------------------------------------------------------------------------- | | Works with .NET primitives, structs and classes. | @@ -318,12 +353,15 @@ The following sample shows traditional way of doing transactions with ZoneTree. | Snapshot iterators. | ## I want to contribute. What can I do? + I appreciate any contribution to the project. These are the things I do think we need at the moment: + 1. Write tests / benchmarks. 2. Write documentation. 3. Feature requests & bug fixes. 4. Performance improvements. ## Contributing + This project welcomes contributions and suggestions. Please follow [CONTRIBUTING.md](.github/CONTRIBUTING.md) instructions. diff --git a/src/Playground/Benchmark/OldTests.cs b/src/Playground/Benchmark/OldTests.cs index 4b87a7b..faa25db 100644 --- a/src/Playground/Benchmark/OldTests.cs +++ b/src/Playground/Benchmark/OldTests.cs @@ -268,7 +268,7 @@ public static void MultipleIterate(WriteAheadLogMode mode, int count, int iterat private static IZoneTree OpenOrCreateZoneTree(WriteAheadLogMode mode, string dataPath) { return new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetMutableSegmentMaxItemCount(TestConfig.MutableSegmentMaxItemCount) .SetDiskSegmentCompressionBlockSize(TestConfig.DiskCompressionBlockSize) .SetDataDirectory(dataPath) diff --git a/src/Playground/Benchmark/ZoneTreeTestBase.cs b/src/Playground/Benchmark/ZoneTreeTestBase.cs index 0856039..0e36e4b 100644 --- a/src/Playground/Benchmark/ZoneTreeTestBase.cs +++ b/src/Playground/Benchmark/ZoneTreeTestBase.cs @@ -32,7 +32,7 @@ protected string GetLabel(string label) protected ZoneTreeFactory GetFactory() { return new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetMutableSegmentMaxItemCount(TestConfig.MutableSegmentMaxItemCount) .SetDiskSegmentMaxItemCount(TestConfig.DiskSegmentMaxItemCount) .SetDiskSegmentCompressionBlockSize(TestConfig.DiskCompressionBlockSize) diff --git a/src/ZoneTree.UnitTests/AtomicUpdateTests.cs b/src/ZoneTree.UnitTests/AtomicUpdateTests.cs index 8985f92..c664c83 100644 --- a/src/ZoneTree.UnitTests/AtomicUpdateTests.cs +++ b/src/ZoneTree.UnitTests/AtomicUpdateTests.cs @@ -16,7 +16,7 @@ public void IntIntAtomicIncrement(WriteAheadLogMode walMode) Directory.Delete(dataPath, true); var counterKey = -3999; using var data = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetComparer(new Int32ComparerDescending()) .SetMutableSegmentMaxItemCount(500) .SetDataDirectory(dataPath) @@ -87,7 +87,7 @@ public void IntIntAtomicIncrementForBTree(WriteAheadLogMode walMode) Directory.Delete(dataPath, true); using var data = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetComparer(new Int32ComparerDescending()) .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) @@ -153,7 +153,7 @@ public void IntIntMutableSegmentOnlyAtomicIncrement(WriteAheadLogMode walMode) Directory.Delete(dataPath, true); var counterKey = -3999; using var data = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetComparer(new Int32ComparerDescending()) .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) @@ -218,7 +218,7 @@ public void IntIntMutableSegmentSeveralUpserts(WriteAheadLogMode walMode) if (Directory.Exists(dataPath)) Directory.Delete(dataPath, true); using var data = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetComparer(new Int32ComparerDescending()) .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) diff --git a/src/ZoneTree.UnitTests/BottomSegmentMergeTests.cs b/src/ZoneTree.UnitTests/BottomSegmentMergeTests.cs index fe4714f..863fc51 100644 --- a/src/ZoneTree.UnitTests/BottomSegmentMergeTests.cs +++ b/src/ZoneTree.UnitTests/BottomSegmentMergeTests.cs @@ -13,7 +13,7 @@ public void IntIntBottomMerge() Directory.Delete(dataPath, true); var zoneTree = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetDiskSegmentMaxItemCount(10) .SetDataDirectory(dataPath) .ConfigureDiskSegmentOptions( @@ -57,7 +57,7 @@ public void IntIntBottomMerge() zoneTree.Dispose(); zoneTree = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetDiskSegmentMaxItemCount(10) .SetDataDirectory(dataPath) .OpenOrCreate(); diff --git a/src/ZoneTree.UnitTests/ExceptionlessTransactionTests.cs b/src/ZoneTree.UnitTests/ExceptionlessTransactionTests.cs index 5968aae..05d86d9 100644 --- a/src/ZoneTree.UnitTests/ExceptionlessTransactionTests.cs +++ b/src/ZoneTree.UnitTests/ExceptionlessTransactionTests.cs @@ -17,7 +17,7 @@ public void TransactionWithNoThrowAPI(int compactionThreshold) Directory.Delete(dataPath, true); using var zoneTree = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) .OpenOrCreateTransactional(); @@ -64,7 +64,7 @@ public async Task TransactionWithFluentAPI(int compactionThreshold) Directory.Delete(dataPath, true); using var zoneTree = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) .ConfigureWriteAheadLogOptions(x => diff --git a/src/ZoneTree.UnitTests/FixedSizeKeyAndValueTests.cs b/src/ZoneTree.UnitTests/FixedSizeKeyAndValueTests.cs index 751da03..84aac46 100644 --- a/src/ZoneTree.UnitTests/FixedSizeKeyAndValueTests.cs +++ b/src/ZoneTree.UnitTests/FixedSizeKeyAndValueTests.cs @@ -12,7 +12,7 @@ public void IntIntTreeTest() if (Directory.Exists(dataPath)) Directory.Delete(dataPath, true); using var data = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetMutableSegmentMaxItemCount(5) .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) @@ -243,7 +243,7 @@ public void StringIntTreeTest(bool useSparseArray) Directory.Delete(dataPath, true); using var data = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetMutableSegmentMaxItemCount(5) .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) diff --git a/src/ZoneTree.UnitTests/IteratorTests.cs b/src/ZoneTree.UnitTests/IteratorTests.cs index 649636b..f6c2b11 100644 --- a/src/ZoneTree.UnitTests/IteratorTests.cs +++ b/src/ZoneTree.UnitTests/IteratorTests.cs @@ -245,7 +245,7 @@ public void IntIntIteratorParallelInserts() var iteratorCount = 1000; using var zoneTree = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetMutableSegmentMaxItemCount(insertCount * 2) .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) @@ -294,7 +294,7 @@ public void IntIntReverseIteratorParallelInserts(bool reverse) var iteratorCount = 1000; using var zoneTree = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetMutableSegmentMaxItemCount(insertCount * 2) .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) @@ -357,7 +357,7 @@ public void IntIntSnapshotIteratorParallelInserts() var iteratorCount = 1000; using var zoneTree = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetMutableSegmentMaxItemCount(insertCount * 2) .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) @@ -400,7 +400,7 @@ public void ReversePrefixSearch() Directory.Delete(dataPath, true); using var zoneTree = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) .OpenOrCreate(); @@ -442,7 +442,7 @@ public void PrefixSearch() Directory.Delete(dataPath, true); using var zoneTree = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) .OpenOrCreate(); @@ -491,7 +491,7 @@ public void SeekIteratorsAfterMerge( Directory.Delete(dataPath, true); using var zoneTree = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) .SetComparer(new StringCurrentCultureComparerAscending()) @@ -566,7 +566,7 @@ public void SeekIteratorsAfterMergeReload( Directory.Delete(dataPath, true); var zoneTree = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) .SetComparer(new StringCurrentCultureComparerAscending()) @@ -589,7 +589,7 @@ public void SeekIteratorsAfterMergeReload( zoneTree.Dispose(); zoneTree = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) .SetComparer(new StringCurrentCultureComparerAscending()) diff --git a/src/ZoneTree.UnitTests/OptimisticTransactionTests.cs b/src/ZoneTree.UnitTests/OptimisticTransactionTests.cs index 1e0bb77..79b877a 100644 --- a/src/ZoneTree.UnitTests/OptimisticTransactionTests.cs +++ b/src/ZoneTree.UnitTests/OptimisticTransactionTests.cs @@ -17,7 +17,7 @@ public void FirstTransaction(int compactionThreshold) Directory.Delete(dataPath, true); using var zoneTree = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) .OpenOrCreateTransactional(); @@ -65,7 +65,7 @@ public void SeveralParallelTransactions(WriteAheadLogMode walMode) int n = 10000; using var zoneTree = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) .ConfigureWriteAheadLogOptions(x => x.WriteAheadLogMode = walMode) @@ -93,7 +93,7 @@ public void SeveralParallelUpserts(WriteAheadLogMode walMode) int n = 10000; using var zoneTree = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetDataDirectory(dataPath) .SetWriteAheadLogDirectory(dataPath) .ConfigureWriteAheadLogOptions(x => x.WriteAheadLogMode = walMode) diff --git a/src/ZoneTree.UnitTests/StringTreeTests.cs b/src/ZoneTree.UnitTests/StringTreeTests.cs index 498542f..34a8e22 100644 --- a/src/ZoneTree.UnitTests/StringTreeTests.cs +++ b/src/ZoneTree.UnitTests/StringTreeTests.cs @@ -67,11 +67,10 @@ public void TestSingleCharacter() for (var i = 0; i < 2; ++i) { using var db = new ZoneTreeFactory() - .DisableDeleteValueConfigurationValidation(false) + .DisableDeletion() .SetDataDirectory(dataPath) .OpenOrCreate(); db.Upsert("0", 123); - } } @@ -83,6 +82,7 @@ public void HelloWorldTest() Directory.Delete(dataPath, true); using var zoneTree = new ZoneTreeFactory() + .SetDataDirectory(dataPath) .OpenOrCreate(); zoneTree.Upsert(39, "Hello Zone Tree"); zoneTree.TryGet(39, out var value); @@ -122,11 +122,17 @@ public void HelloWorldTest2() [Test] public void HelloWorldTest3() { - var dataPath = "data/HelloWorldTest"; + var dataPath = "data/HelloWorldTest3"; if (Directory.Exists(dataPath)) Directory.Delete(dataPath, true); - Assert.Throws(() => new ZoneTreeFactory() - .OpenOrCreate()); + var tree = new ZoneTreeFactory() + .SetDataDirectory(dataPath) + .OpenOrCreate(); + tree.Upsert(1, 0); + // The value 0 represents deleted record. + Assert.That(tree.Count(), Is.EqualTo(0)); + tree.Upsert(1, 2); + Assert.That(tree.Count(), Is.EqualTo(1)); } } \ No newline at end of file diff --git a/src/ZoneTree/Options/ZoneTreeOptions.cs b/src/ZoneTree/Options/ZoneTreeOptions.cs index a3ae461..d696afe 100644 --- a/src/ZoneTree/Options/ZoneTreeOptions.cs +++ b/src/ZoneTree/Options/ZoneTreeOptions.cs @@ -5,6 +5,7 @@ using Tenray.ZoneTree.Serializers; using Tenray.ZoneTree.Comparers; using Tenray.ZoneTree.Segments.RandomAccess; +using Tenray.ZoneTree.PresetTypes; namespace Tenray.ZoneTree.Options; @@ -30,14 +31,6 @@ namespace Tenray.ZoneTree.Options; /// The value type public sealed class ZoneTreeOptions { - static bool DefaultIsValueDeleted(in TValue _) => false; - - static void DefaultMarkValueDeleted(ref TValue value) { value = default; } - - static bool ReferenceTypeIsValueDeleted(in TValue value) => object.ReferenceEquals(value, default(TValue)); - - static void ReferenceTypeMarkValueDeleted(ref TValue value) { value = default; } - /// /// Mutable segment maximumum key-value pair count. /// When the maximum count is reached @@ -71,12 +64,12 @@ public sealed class ZoneTreeOptions /// /// Delegate to query value deletion state. /// - public IsValueDeletedDelegate IsValueDeleted { get; set; } = DefaultIsValueDeleted; + public IsValueDeletedDelegate IsValueDeleted { get; set; } /// /// Delegate to mark value deleted. /// - public MarkValueDeletedDelegate MarkValueDeleted { get; set; } = DefaultMarkValueDeleted; + public MarkValueDeletedDelegate MarkValueDeleted { get; set; } /// /// Write Ahead Log Options. The options are used @@ -150,39 +143,6 @@ public bool TryValidate(out Exception exception) return false; } - switch (DeleteValueConfigurationValidation) - { - case DeleteValueConfigurationValidation.Required: - { - if (IsValueDeleted == DefaultIsValueDeleted) - { - exception = new MissingOptionException(nameof(IsValueDeleted)); - return false; - } - - if (MarkValueDeleted == DefaultMarkValueDeleted) - { - exception = new MissingOptionException(nameof(MarkValueDeleted)); - return false; - } - } - break; - case DeleteValueConfigurationValidation.Warning: - { - if (IsValueDeleted == DefaultIsValueDeleted) - { - Logger.LogWarning(new MissingOptionException(nameof(IsValueDeleted))); - } - - if (MarkValueDeleted == DefaultMarkValueDeleted) - { - Logger.LogWarning(new MissingOptionException(nameof(MarkValueDeleted))); - } - } - break; - default: break; - } - exception = ValidateCompressionLevel( "disk segment", DiskSegmentOptions.CompressionMethod, @@ -250,12 +210,6 @@ public void Validate() /// public IRandomAccessDeviceManager RandomAccessDeviceManager { get; set; } - /// - /// Defines the validation behavior - /// of not providing the delete value delegates. - /// - public DeleteValueConfigurationValidation DeleteValueConfigurationValidation { get; set; } - /// /// If the ZoneTree contains only a single segment (which is the mutable segment), /// there is an opportunity to perform a hard delete of the soft deleted values. @@ -264,22 +218,14 @@ public void Validate() public bool EnableSingleSegmentGarbageCollection { get; set; } /// - /// Creates default delete delegates for nullable types. + /// Creates default delete delegates for nullable types if they are not already set. /// public void CreateDefaultDeleteDelegates() { - if (!IsAssignableToNull(typeof(TValue))) - return; - - if (MarkValueDeleted == DefaultMarkValueDeleted) - { - MarkValueDeleted = ReferenceTypeMarkValueDeleted; - } - - if (IsValueDeleted == DefaultIsValueDeleted) - { - IsValueDeleted = ReferenceTypeIsValueDeleted; - } + if (IsValueDeleted == null) + IsValueDeleted = ComponentsForKnownTypes.GetIsValueDeleted(); + if (MarkValueDeleted == null) + MarkValueDeleted = ComponentsForKnownTypes.GetMarkValueDeleted(); } static bool IsAssignableToNull(Type type) @@ -291,4 +237,14 @@ static bool IsAssignableToNull(Type type) return false; } + + /// + /// Disables deletion to be able to insert default values of the value type. + /// Databases created with this option are not able to delete records. + /// + public void DisableDeletion() + { + IsValueDeleted = (in TValue _) => false; + MarkValueDeleted = (ref TValue _) => { }; + } } diff --git a/src/ZoneTree/PresetTypes/ComponentsForKnownTypes.cs b/src/ZoneTree/PresetTypes/ComponentsForKnownTypes.cs new file mode 100644 index 0000000..7d9ac29 --- /dev/null +++ b/src/ZoneTree/PresetTypes/ComponentsForKnownTypes.cs @@ -0,0 +1,158 @@ +using System.Runtime.CompilerServices; +using Tenray.ZoneTree.Comparers; +using Tenray.ZoneTree.Exceptions; +using Tenray.ZoneTree.Options; +using Tenray.ZoneTree.Serializers; + +namespace Tenray.ZoneTree.PresetTypes; + +/// +/// Provides utility methods for handling known types, including obtaining comparers, serializers, +/// and determining if values are deleted. +/// +public static class ComponentsForKnownTypes +{ + /// + /// Returns a comparer for a given type key. The comparer is used to compare instances + /// of the type in ascending order. + /// + /// The type of the key to compare. + /// An implementation of appropriate for the type, or null if unsupported. + public static IRefComparer GetComparer() + { + TKey key = default; + var comparer = key switch + { + byte => new ByteComparerAscending() as IRefComparer, + char => new CharComparerAscending() as IRefComparer, + DateTime => new DateTimeComparerAscending() as IRefComparer, + decimal => new DecimalComparerAscending() as IRefComparer, + double => new DoubleComparerAscending() as IRefComparer, + short => new Int16ComparerAscending() as IRefComparer, + ushort => new UInt16ComparerAscending() as IRefComparer, + int => new Int32ComparerAscending() as IRefComparer, + uint => new UInt32ComparerAscending() as IRefComparer, + long => new Int64ComparerAscending() as IRefComparer, + ulong => new UInt64ComparerAscending() as IRefComparer, + Guid => new GuidComparerAscending() as IRefComparer, + _ => null + }; + if (typeof(TKey) == typeof(string)) + comparer = + new StringOrdinalComparerAscending() as IRefComparer; + + else if (typeof(TKey) == typeof(Memory)) + comparer = + new ByteArrayComparerAscending() as IRefComparer; + else if (typeof(TKey) == typeof(byte[])) + { + throw new ZoneTreeException("ZoneTree is not supported. Use ZoneTree, ...> instead."); + } + return comparer; + } + + /// + /// Returns a serializer for a given type. The serializer is used to serialize and deserialize + /// instances of the type. + /// + /// The type to be serialized. + /// An implementation of appropriate for the type, or null if unsupported. + public static ISerializer GetSerializer() + { + T key = default; + var serializer = key switch + { + byte => new ByteSerializer() as ISerializer, + bool => new BooleanSerializer() as ISerializer, + char => new CharSerializer() as ISerializer, + DateTime => new DateTimeSerializer() as ISerializer, + decimal => new DecimalSerializer() as ISerializer, + double => new DoubleSerializer() as ISerializer, + short => new Int16Serializer() as ISerializer, + ushort => new UInt16Serializer() as ISerializer, + int => new Int32Serializer() as ISerializer, + uint => new UInt32Serializer() as ISerializer, + long => new Int64Serializer() as ISerializer, + ulong => new UInt64Serializer() as ISerializer, + Guid => new StructSerializer() as ISerializer, + _ => null + }; + + if (typeof(T) == typeof(string)) + serializer = + new Utf8StringSerializer() as ISerializer; + else if (typeof(T) == typeof(Memory)) + { + serializer = + new ByteArraySerializer() as ISerializer; + } + else if (typeof(T) == typeof(byte[])) + { + throw new ZoneTreeException("ZoneTree is not supported. Use ZoneTree, ...> instead."); + } + return serializer; + } + + // Specific methods for checking if certain primitive types are considered deleted + static bool IsValueDeletedByte(in byte value) => value == default; + static bool IsValueDeletedChar(in char value) => value == default; + static bool IsValueDeletedDateTime(in DateTime value) => value == default; + static bool IsValueDeletedDecimal(in decimal value) => value == default; + static bool IsValueDeletedDouble(in double value) => value == default; + static bool IsValueDeletedShort(in short value) => value == default; + static bool IsValueDeletedUShort(in ushort value) => value == default; + static bool IsValueDeletedInt(in int value) => value == default; + static bool IsValueDeletedUInt(in uint value) => value == default; + static bool IsValueDeletedLong(in long value) => value == default; + static bool IsValueDeletedULong(in ulong value) => value == default; + static bool IsValueDeletedGuid(in Guid value) => value == default; + static bool IsValueDeletedMemoryByte(in Memory value) => value.Length == 0; + static bool IsValueDeletedReferenceType(in TValue value) => ReferenceEquals(value, default(TValue)); + static bool IsValueDeletedDefault(in TValue value) => EqualityComparer.Default.Equals(value, default); + + static void MarkValueDeletedDefault(ref TValue value) { value = default; } + + /// + /// Returns a delegate that checks if a value of a specific type is considered deleted. + /// + /// The type of the value. + /// A delegate that checks if the value is considered deleted. + public static IsValueDeletedDelegate GetIsValueDeleted() + { + static IsValueDeletedDelegate Cast(object method) => + (IsValueDeletedDelegate)method; + TValue value = default; + var result = value switch + { + byte => Cast(new IsValueDeletedDelegate(IsValueDeletedByte)), + char => Cast(new IsValueDeletedDelegate(IsValueDeletedChar)), + DateTime => Cast(new IsValueDeletedDelegate(IsValueDeletedDateTime)), + decimal => Cast(new IsValueDeletedDelegate(IsValueDeletedDecimal)), + double => Cast(new IsValueDeletedDelegate(IsValueDeletedDouble)), + short => Cast(new IsValueDeletedDelegate(IsValueDeletedShort)), + ushort => Cast(new IsValueDeletedDelegate(IsValueDeletedUShort)), + int => Cast(new IsValueDeletedDelegate(IsValueDeletedInt)), + uint => Cast(new IsValueDeletedDelegate(IsValueDeletedUInt)), + long => Cast(new IsValueDeletedDelegate(IsValueDeletedLong)), + ulong => Cast(new IsValueDeletedDelegate(IsValueDeletedULong)), + Guid => Cast(new IsValueDeletedDelegate(IsValueDeletedGuid)), + _ => IsValueDeletedDefault + }; + + if (typeof(TValue) == typeof(Memory)) + result = Cast(new IsValueDeletedDelegate>(IsValueDeletedMemoryByte)); + if (RuntimeHelpers.IsReferenceOrContainsReferences()) + return IsValueDeletedReferenceType; + return result; + } + + /// + /// Returns a delegate that marks a value of a specific type as deleted by setting it to its default value. + /// + /// The type of the value. + /// A delegate that marks the value as deleted. + public static MarkValueDeletedDelegate GetMarkValueDeleted() + { + return MarkValueDeletedDefault; + } +} \ No newline at end of file diff --git a/src/ZoneTree/ZoneTreeFactory.cs b/src/ZoneTree/ZoneTreeFactory.cs index 085bab7..4854ed7 100644 --- a/src/ZoneTree/ZoneTreeFactory.cs +++ b/src/ZoneTree/ZoneTreeFactory.cs @@ -9,6 +9,7 @@ using Tenray.ZoneTree.Options; using Tenray.ZoneTree.Logger; using Tenray.ZoneTree.Segments.RandomAccess; +using Tenray.ZoneTree.PresetTypes; namespace Tenray.ZoneTree; @@ -151,19 +152,6 @@ public ZoneTreeFactory SetOptions(ZoneTreeOptions op return this; } - /// - /// Disables the delete value configuration validation. - /// - /// If true, validation logs a warning when value deletion is not configured. - /// ZoneTree Factory - public ZoneTreeFactory DisableDeleteValueConfigurationValidation(bool keepWarning = true) - { - Options.DeleteValueConfigurationValidation = keepWarning ? - DeleteValueConfigurationValidation.Warning : - DeleteValueConfigurationValidation.NotRequired; - return this; - } - /// /// Sets random access device manager. /// @@ -355,105 +343,21 @@ void FillComparer() { if (Options.Comparer != null) return; - TKey key = default; - Options.Comparer = key switch - { - byte => new ByteComparerAscending() as IRefComparer, - char => new CharComparerAscending() as IRefComparer, - DateTime => new DateTimeComparerAscending() as IRefComparer, - decimal => new DecimalComparerAscending() as IRefComparer, - double => new DoubleComparerAscending() as IRefComparer, - short => new Int16ComparerAscending() as IRefComparer, - ushort => new UInt16ComparerAscending() as IRefComparer, - int => new Int32ComparerAscending() as IRefComparer, - uint => new UInt32ComparerAscending() as IRefComparer, - long => new Int64ComparerAscending() as IRefComparer, - ulong => new UInt64ComparerAscending() as IRefComparer, - Guid => new GuidComparerAscending() as IRefComparer, - _ => null - }; - if (typeof(TKey) == typeof(string)) - Options.Comparer = - new StringOrdinalComparerAscending() as IRefComparer; - - else if (typeof(TKey) == typeof(Memory)) - Options.Comparer = - new ByteArrayComparerAscending() as IRefComparer; + Options.Comparer = ComponentsForKnownTypes.GetComparer(); } void FillKeySerializer() { if (Options.KeySerializer != null) return; - TKey key = default; - Options.KeySerializer = key switch - { - byte => new ByteSerializer() as ISerializer, - char => new CharSerializer() as ISerializer, - DateTime => new DateTimeSerializer() as ISerializer, - decimal => new DecimalSerializer() as ISerializer, - double => new DoubleSerializer() as ISerializer, - short => new Int16Serializer() as ISerializer, - ushort => new UInt16Serializer() as ISerializer, - int => new Int32Serializer() as ISerializer, - uint => new UInt32Serializer() as ISerializer, - long => new Int64Serializer() as ISerializer, - ulong => new UInt64Serializer() as ISerializer, - Guid => new StructSerializer() as ISerializer, - _ => null - }; - - if (typeof(TKey) == typeof(string)) - Options.KeySerializer = - new Utf8StringSerializer() as ISerializer; - else if (typeof(TKey) == typeof(Memory)) - { - Options.KeySerializer = - new ByteArraySerializer() as ISerializer; - } - else if (typeof(TKey) == typeof(byte[])) - { - throw new ZoneTreeException("ZoneTree is not supported. Use ZoneTree, ...> instead."); - } + Options.KeySerializer = ComponentsForKnownTypes.GetSerializer(); } void FillValueSerializer() { if (Options.ValueSerializer != null) return; - TValue value = default; - Options.ValueSerializer = value switch - { - byte => new ByteSerializer() as ISerializer, - bool => new BooleanSerializer() as ISerializer, - char => new CharSerializer() as ISerializer, - DateTime => new DateTimeSerializer() as ISerializer, - decimal => new DecimalSerializer() as ISerializer, - double => new DoubleSerializer() as ISerializer, - short => new Int16Serializer() as ISerializer, - ushort => new UInt16Serializer() as ISerializer, - int => new Int32Serializer() as ISerializer, - uint => new UInt32Serializer() as ISerializer, - long => new Int64Serializer() as ISerializer, - ulong => new UInt64Serializer() as ISerializer, - Guid => new StructSerializer() as ISerializer, - _ => null - }; - - if (typeof(TValue) == typeof(string)) - Options.ValueSerializer = - new Utf8StringSerializer() as ISerializer; - - else if (typeof(TValue) == typeof(Memory)) - { - Options.ValueSerializer = - new ByteArraySerializer() as ISerializer; - } - else if (typeof(TValue) == typeof(byte[])) - { - throw new ZoneTreeException("ZoneTree<..., byte[]> is not supported. Use ZoneTree<..., Memory> instead."); - } - + Options.ValueSerializer = ComponentsForKnownTypes.GetSerializer(); } /// @@ -538,4 +442,15 @@ public ITransactionalZoneTree OpenTransactional() InitTransactionLog(); return new OptimisticZoneTree(Options, TransactionLog, zoneTree); } + + /// + /// Disables deletion to be able to insert default values of the value type. + /// Databases created with this option are not able to delete records. + /// + /// ZoneTree Factory + public ZoneTreeFactory DisableDeletion() + { + Options.DisableDeletion(); + return this; + } } \ No newline at end of file diff --git a/src/ZoneTree/docs/ZoneTree/README-NUGET.md b/src/ZoneTree/docs/ZoneTree/README-NUGET.md index d747e15..499ce26 100644 --- a/src/ZoneTree/docs/ZoneTree/README-NUGET.md +++ b/src/ZoneTree/docs/ZoneTree/README-NUGET.md @@ -4,7 +4,10 @@ ZoneTree is a persistent, high-performance, transactional, and ACID-compliant [ordered key-value database](https://en.wikipedia.org/wiki/Ordered_Key-Value_Store) for .NET. It can operate in memory or on local/cloud storage. +![License](https://img.shields.io/badge/license-MIT-blue.svg) [![Downloads](https://img.shields.io/nuget/dt/ZoneTree)](https://www.nuget.org/packages/ZoneTree/) +![Platform](https://img.shields.io/badge/platform-.NET-blue.svg) +![Build](https://img.shields.io/badge/build-passing-brightgreen.svg) ZoneTree is a lightweight, transactional and high-performance LSM Tree for .NET. @@ -146,12 +149,11 @@ Note: For small data you don't need a maintainer. ``` ## How to delete keys? -In LSM trees, the deletions are handled by upserting key/value with deleted flag. -Later on, during the compaction stage, the actual deletion happens. -ZoneTree does not implement this flag format by default. It lets the user to define the suitable deletion flag themselves. -For example, the deletion flag might be defined by user as -1 for int values. -If user wants to use any int value as a valid record, then the value-type should be changed. -For example, one can define the following struct and use this type as a value-type. +In Log-Structured Merge (LSM) trees, deletions are managed by upserting a key/value pair with a deletion marker. The actual removal of the data occurs during the compaction stage. In ZoneTree, by default, the system assumes that the default values indicate deletion. However, you can customize this behavior by defining a specific deletion flag, such as using -1 for integer values or completely disable deletion by calling DisableDeletion method. + +### Custom Deletion Flag +If you need more control over how deletions are handled, you can define a custom structure to represent your values and their deletion status. For example: + ```c# [StructLayout(LayoutKind.Sequential)] struct MyDeletableValueType { @@ -159,8 +161,15 @@ struct MyDeletableValueType { bool IsDeleted; } ``` -You can micro-manage the tree size with ZoneTree. -The following sample shows how to configure the deletion markers for your database. + +This struct allows you to include a boolean flag indicating whether a value is deleted. You can then use this custom type as the value type in your ZoneTree. + +### Configuring Deletion Markers +ZoneTree provides flexibility in managing the tree size by allowing you to configure how deletion markers are set and identified. Below are examples of how you can configure these markers for your database: + +#### Example 1: Using an Integer Deletion Flag +In this example, -1 is used as the deletion marker for integer values: + ```c# using var zoneTree = new ZoneTreeFactory() // Additional stuff goes here @@ -168,7 +177,10 @@ using var zoneTree = new ZoneTreeFactory() .SetMarkValueDeletedDelegate((ref int x) => x = -1) .OpenOrCreate(); ``` -or + +#### Example 2: Using a Custom Struct for Deletion +Alternatively, if you're using a custom struct to manage deletions, you can configure ZoneTree to recognize and mark deletions as follows: + ```c# using var zoneTree = new ZoneTreeFactory() // Additional stuff goes here @@ -176,7 +188,8 @@ using var zoneTree = new ZoneTreeFactory() .SetMarkValueDeletedDelegate((ref MyDeletableValueType x) => x.IsDeleted = true) .OpenOrCreate(); ``` -If you forget to provide the deletion marker delegates, you can never delete the record from your database. + +You can also use built in generic [Deletable<TValue>](/docs/ZoneTree/api/Tenray.ZoneTree.PresetTypes.Deletable-1.html) for deletion. ## How to iterate over data? diff --git a/src/ZoneTree/docs/ZoneTree/guide/quick-start.md b/src/ZoneTree/docs/ZoneTree/guide/quick-start.md index a2797ef..771b89a 100644 --- a/src/ZoneTree/docs/ZoneTree/guide/quick-start.md +++ b/src/ZoneTree/docs/ZoneTree/guide/quick-start.md @@ -81,12 +81,11 @@ zoneTree.Maintenance.StartMergeOperation()?.Join(); ``` ## How to delete keys? -In LSM trees, the deletions are handled by upserting key/value with deleted flag. -Later on, during the compaction stage, the actual deletion happens. -ZoneTree does not implement this flag format by default. It lets the user to define the suitable deletion flag themselves. -For example, the deletion flag might be defined by user as -1 for int values. -If user wants to use any int value as a valid record, then the value-type should be changed. -For example, one can define the following struct and use this type as a value-type. +In Log-Structured Merge (LSM) trees, deletions are managed by upserting a key/value pair with a deletion marker. The actual removal of the data occurs during the compaction stage. In ZoneTree, by default, the system assumes that the default values indicate deletion. However, you can customize this behavior by defining a specific deletion flag, such as using -1 for integer values or completely disable deletion by calling DisableDeletion method. + +### Custom Deletion Flag +If you need more control over how deletions are handled, you can define a custom structure to represent your values and their deletion status. For example: + ```c# [StructLayout(LayoutKind.Sequential)] struct MyDeletableValueType { @@ -94,8 +93,15 @@ struct MyDeletableValueType { bool IsDeleted; } ``` -You can micro-manage the tree size with ZoneTree. -The following sample shows how to configure the deletion markers for your database. + +This struct allows you to include a boolean flag indicating whether a value is deleted. You can then use this custom type as the value type in your ZoneTree. + +### Configuring Deletion Markers +ZoneTree provides flexibility in managing the tree size by allowing you to configure how deletion markers are set and identified. Below are examples of how you can configure these markers for your database: + +#### Example 1: Using an Integer Deletion Flag +In this example, -1 is used as the deletion marker for integer values: + ```c# using var zoneTree = new ZoneTreeFactory() // Additional stuff goes here @@ -103,7 +109,10 @@ using var zoneTree = new ZoneTreeFactory() .SetMarkValueDeletedDelegate((ref int x) => x = -1) .OpenOrCreate(); ``` -or + +#### Example 2: Using a Custom Struct for Deletion +Alternatively, if you're using a custom struct to manage deletions, you can configure ZoneTree to recognize and mark deletions as follows: + ```c# using var zoneTree = new ZoneTreeFactory() // Additional stuff goes here @@ -111,9 +120,8 @@ using var zoneTree = new ZoneTreeFactory() .SetMarkValueDeletedDelegate((ref MyDeletableValueType x) => x.IsDeleted = true) .OpenOrCreate(); ``` -If you forget to provide the deletion marker delegates, you can never delete the record from your database. -You can use built in generic [Deletable<TValue>](/docs/ZoneTree/api/Tenray.ZoneTree.PresetTypes.Deletable-1.html) for deletion. +You can also use built in generic [Deletable<TValue>](/docs/ZoneTree/api/Tenray.ZoneTree.PresetTypes.Deletable-1.html) for deletion. ## How to iterate over data?