add sync missing sidecars functionality

Makefile

@@ -1,12 +1,9 @@
-version = 0.4.1-beta
+version = 0.5-beta
 zip_file = releases/KOReader Sync v$(version).zip
 zip_contents = about.txt LICENSE plugin-import-name-koreader.txt *.py *.md  images/*.png
 
 all: zip
 
-dependencies:
-	@ wget -N https://github.com/SirAnthony/slpp/raw/master/slpp.py
-
 zip:
 	@ echo "creating new $(zip_file)" && zip "$(zip_file)" $(zip_contents) && echo "created new $(zip_file)"
 

README.md

@@ -10,7 +10,9 @@ A calibre plugin to synchronize metadata from KOReader to calibre.
 
 [KOReader](https://koreader.rocks/) creates sidecar files that hold read progress and annotations. This plugin reads the data from those sidecar files and updates calibre's metadata based on them. It is inspired by [the Kobo Utilities plugin](https://www.mobileread.com/forums/showthread.php?t=215339), that synchronizes reading progress between the original Kobo firmware (“Nickel”) and custom columns in calibre.
 
-Note that at the moment the sync is one-way—from the KOReader device to calibre—and only works for USB and [wireless](https://github.com/koreader/koreader/wiki/Calibre-wireless-connection) devices. For the latter, you'll need [KOReader 2021.04 or newer](https://github.com/koreader/koreader/releases).
+Note that at the moment the sync is primarily one-way—from the KOReader device to calibre — and only works for USB and [wireless](https://github.com/koreader/koreader/wiki/Calibre-wireless-connection) devices. For the latter, you'll need [KOReader 2021.04 or newer](https://github.com/koreader/koreader/releases).
+
+Pushing metadata from Calibre to KOReader currently works only for books which do not have KOReader sidecar files, and of course requires the raw metadata column to be mapped. The use-case is for e.g. setting up a new device, or if a book was removed from your device and you've now added it back. This has been tested for Calibre's Connect to Folder and Custom USB Device modes. It does not seem to work for the Kobo Touch device driver nor with wireless connections, but I (@charlesangus) find those don't communicate perfectly with Calibre/KOReader in any case... I haven't disabled it for other devices - it may be a quirk in my setup which is causing it to fail, and it may work fine for you.
 
 Releases will also be uploaded to [this plugin thread on the MobileRead Forums](https://www.mobileread.com/forums/showthread.php?p=4060141). If you are on there as well, please let me know what you think of the plugin in that thread.
 
@@ -41,7 +43,7 @@ Releases will also be uploaded to [this plugin thread on the MobileRead Forums](
     - A “Date” column to store **the date on which the first highlight or bookmark was made**. (This is probably around the time you started reading.)
     - A “Date” column to store **the date on which the last highlight or bookmark was made**. (This is probably around the time you finished reading.)
     - A regular “Text” column to store the **MD5 hash** KOReader uses to sync progress to a [**KOReader Sync Server**](https://github.com/koreader/koreader-sync-server#koreader-sync-server). (“Progress sync” in the KOReader app.) This might allow for syncing progress to calibre without having to connect your KOReader device, in the future.
-    - A “Long text” column to store the **raw contents of the metadata sidecar**, with “Interpret this column as” set to “Plain text”.
+    - A “Long text” column to store the **raw contents of the metadata sidecar**, with “Interpret this column as” set to “Plain text”. This is required to sync metadata back to KOReader sidecars.
 10. Add “KOReader Sync” to “main toolbar when a device is connected”, if it isn't there already.
 11. Right-click the “KOReader Sync” icon and “Configure”.
 12. Map the metadata you want to sync to the newly created calibre columns.
@@ -50,7 +52,9 @@ Releases will also be uploaded to [this plugin thread on the MobileRead Forums](
 
 ### Things to consider
 
-- The plugin overwrites existing metadata without asking. That usually isn’t a problem, because you will probably only add to KOReader’s metadata. But be aware that you might lose data in calibre if you’re not careful.
+- The plugin overwrites existing metadata in Calibre without asking. That usually isn’t a problem, because you will probably only add to KOReader’s metadata. But be aware that you might lose data in calibre if you’re not careful.
+- Pushing sidecars back to KOReader currently only happens for sidecars which are missing. For now, manually delete the `<bookname>.sdr` folder from the device before attempting to push the sidecars back to KOReader for any books you would like to overwrite the current metadata with Calibre's metadata.
+- When pushing missing sidecars to the device, no attempt is made to convert Calibre's metadata to account for changes in KOReader's sidecar format. Old metadata may work unpredictably if it's from a different version of KOReader.
 
 ### Supported devices
 
@@ -76,6 +80,7 @@ If you encounter any issues with the plugin, please submit them [here](https://g
 ## Acknowledgements
 
 - Multiple tweaks and bug fixes by [Glen Sawyer](https://git.sr.ht/~snelg)
+- Additional functionality by [Charles Taylor](https://github.com/charlesangus/)
 - Contains [SirAnthony's SLPP](https://github.com/SirAnthony/slpp) to parse Lua in Python.
 - Some code borrowed from--and heavily inspired by--the great [Kobo Utilities](https://www.mobileread.com/forums/showthread.php?t=215339) calibre plugin.
 
@@ -87,17 +92,11 @@ If you encounter any issues with the plugin, please submit them [here](https://g
 - calibre allows you to auto-connect to a folder device on boot, which greatly speeds up your workflow when testing. You can find this under “Preferences” > “Tweaks”, search for `auto_connect_to_folder`. Point that to the `dummy_device` folder in this repository. (I have included royalty free EPUBs for your and my convenience.)
 - If you're testing and don't actually want to update any metadata, set `DRY_RUN` to `True` in `__init__.py`.
 - I work in PyCharm, which offers a remote debugging server. To enable that in this plugin, set `PYDEVD` to `True` in `__init__.py`.You might need to change `sys.path.append` in `action.py`.
-- The supported device drivers can be found in [the `supported_devices` list at line 387 in `action.py`](https://github.com/harmtemolder/koreader-calibre-plugin/blob/main/action.py#L387). Adding a new type here is the first step to adding support, but make sure all features are tested thoroughly before releasing a version with an added device
-
-### Downloading dependencies
-
-```shell
-make dependencies
-```
+- The supported device drivers can be found in [the `SUPPORTED_DEVICES` list in `config.py`](https://github.com/harmtemolder/koreader-calibre-plugin/blob/main/config.py#L30). Adding a new type here is the first step to adding support, but make sure all features are tested thoroughly before releasing a version with an added device
 
 ### Testing in calibre
 
-Make sure you have the dependencies. Then:
+Use make to load the plugin into calibre and launch it:
 
 ```shell
 make dev
@@ -137,6 +136,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 ## [Unreleased]
 
+## [0.5] - 2022-12-27
+
+### Changed
+
+- Add "Sync Missing Sidecars to KOReader" functionality
+- Vendor in slpp.py instead of adding it as a separate dependency to reduce fragility
+
 ## [0.4.1-beta] - 2022-11-08
 
 ### Changed

__init__.py

@@ -31,7 +31,7 @@ class KoreaderSync(InterfaceActionBase):
     name = 'KOReader Sync'
     description = 'Get metadata from a locally connected KOReader device '
     author = 'harmtemolder'
-    version = (0, 4, 1)
+    version = (0, 5, 0)
     minimum_calibre_version = (5, 0, 1)  # Because Python 3
     config = JSONConfig(os.path.join('plugins', 'KOReader Sync.json'))
     actual_plugin = 'calibre_plugins.koreader.action:KoreaderAction'

action.py

@@ -4,13 +4,16 @@
 from functools import partial
 import io
 import json
+import os
 import re
 import sys
 from calibre.utils.iso8601 import utc_tz
 
 from PyQt5.Qt import QUrl  # pylint: disable=no-name-in-module
 from calibre_plugins.koreader.slpp import slpp as lua  # pylint: disable=import-error
 from calibre_plugins.koreader.config import (
+    SUPPORTED_DEVICES,
+    UNSUPPORTED_DEVICES,
     COLUMNS,
     CONFIG,  # pylint: disable=import-error
 )
@@ -88,6 +91,17 @@ def genesis(self):
         self.qaction.triggered.connect(self.sync_to_calibre)
 
         # Right-click menu (already includes left-click action)
+        self.create_menu_action(
+            self.qaction.menu(),
+            'Sync Missing Sidecars to KOReader',
+            'Sync Missing Sidecars to KOReader',
+            icon='config.png',
+            description='Where Calibre has a raw metadata entry but KOReader '
+                'does not have a sidecar file, push the metadata from Calibre '
+                'to a new sidecar file.',
+            triggered=self.sync_missing_sidecars_to_koreader
+        )
+
         self.qaction.menu().addSeparator()
 
         self.create_menu_action(
@@ -373,41 +387,20 @@ def update_metadata(self, uuid, keys_values_to_update):
             'book_id': book_id,
         }
 
-    def sync_to_calibre(self):
-        """This plugin’s main purpose. It syncs the contents of
-        KOReader’s metadata sidecar files into calibre’s metadata.
+    def check_device(self, device):
+        """Return .
 
-        :return:
+        :param device: The connected device.
+        :return: False if device is specifically not supported,
+        otherwise True
         """
-        debug_print = partial(
-            module_debug_print,
-            'KoreaderAction:sync_to_calibre:'
-        )
-
-        supported_devices = [
-            'FOLDER_DEVICE',
-            'KINDLE2',
-            'KOBO',
-            'KOBOTOUCH',
-            'KOBOTOUCHEXTENDED',
-            'POCKETBOOK622',
-            'POCKETBOOK626',
-            'SMART_DEVICE_APP',
-            'TOLINO',
-            'USER_DEFINED',
-            'POCKETBOOK632',
-        ]
-        unsupported_devices = [
-            'MTP_DEVICE',
-        ]
-        device = self.get_connected_device()
 
         if not device:
-            return None
+            return False
 
         device_class = device.__class__.__name__
 
-        if device_class in unsupported_devices:
+        if device_class in UNSUPPORTED_DEVICES:
             debug_print('unsupported device, device_class = ', device_class)
             error_dialog(
                 self.gui,
@@ -420,8 +413,10 @@ def sync_to_calibre(self):
                 show=True,
                 show_copy_button=False
             )
-            return None
-        elif device_class not in supported_devices:
+            return False
+        elif device_class in SUPPORTED_DEVICES:
+            return True
+        else:
             debug_print(
                 'not yet supported device, device_class = ',
                 device_class
@@ -438,6 +433,180 @@ def sync_to_calibre(self):
                 show=True,
                 show_copy_button=False
             )
+            return True
+
+    def push_metadata_to_koreader_sidecar(self, book_uuid, path):
+        """Create a sidecar file for the given book.
+
+        :param book_uuid: Calibre's uuid for the book
+        :param path: path to sidecar file to create
+        :return: tuple of bool and result dict
+        """
+
+        debug_print = partial(
+            module_debug_print,
+            'KoreaderAction:push_metadata_to_koreader_sidecar:'
+        )
+
+        try:
+            db = self.gui.current_db.new_api
+            book_id = db.lookup_by_uuid(book_uuid)
+            debug_print(f"Book id is {book_id}")
+        except:
+            book_id = None
+
+        if not book_id:
+            debug_print('could not find {} in calibre’s library'.format(book_uuid))
+            return "failure", {
+                'result': f"Could not find uuid {book_uuid} in Calibre's library."
+            }
+
+        # Get the current metadata for the book from the library
+        metadata = db.get_metadata(book_id)
+        sidecar_metadata = metadata.get(CONFIG["column_sidecar"])
+        if not sidecar_metadata:
+            return "no_metadata", {
+                'result': f'No KOReader metadata for book_id {book_id}, no need to push.'
+            }
+        sidecar_dict = json.loads(sidecar_metadata)
+        sidecar_lua = lua.encode(sidecar_dict)
+        # not certain if tabs need to be replaced with spaces but it can't hurt
+        sidecar_lua = sidecar_lua.replace("\t", "    ")
+        # something is happening in the decoding/encoding which is replacing [1] with ["1"]
+        # which ofc breaks the settings file; this regex strips the "" marks
+        sidecar_lua = re.sub(r'\["([0-9])+"\]', r'[\1]', sidecar_lua)
+        sidecar_lua_formatted = f"-- we can read Lua syntax here!\nreturn {sidecar_lua}\n"
+        try:
+            os.makedirs(os.path.dirname(path))
+        except FileExistsError:
+            # dir exists, so we're fine
+            pass
+
+        with open(path, "w") as f:
+            debug_print(f"Writing to {path}")
+            f.write(sidecar_lua_formatted)
+
+        return "success", {
+            'result': 'success',
+            'book_id': book_id,
+        }
+
+    def sync_missing_sidecars_to_koreader(self):
+        """Push the content of Calibre's raw metadata column to KOReader
+        for any files which are missing in KOReader. Does not touch existing
+        metadata sidecars on KOReader.
+
+        Intended for e.g. setting up a new device and syncing to it for the first
+        time.
+
+        :return:
+        """
+        debug_print = partial(
+            module_debug_print,
+            'KoreaderAction:sync_missing_sidecars_to_koreader:'
+        )
+
+        if CONFIG["column_sidecar"] is '':
+            error_dialog(
+                self.gui,
+                'Failure',
+                'Raw metadata column not mapped, impossible to push metadata to sidecars',
+                show=True,
+                show_copy_button=False
+            )
+            return None
+
+        device = self.get_connected_device()
+
+        if not self.check_device(device):
+            return None
+
+        sidecar_paths = self.get_paths(device)
+        sidecar_paths_exist = {}
+        sidecar_paths_not_exist = {}
+        for book_uuid, path in sidecar_paths.items():
+            if os.path.exists(path):
+                sidecar_paths_exist[book_uuid] = path
+            else:
+                sidecar_paths_not_exist[book_uuid] = path
+        debug_print(
+            "Sidecars not present on device:",
+            "\n".join(sidecar_paths_not_exist.values())
+        )
+
+        results = []
+        num_success = 0
+        num_no_metadata = 0
+        num_fail = 0
+        for book_uuid, path in sidecar_paths_not_exist.items():
+            result, details = self.push_metadata_to_koreader_sidecar(book_uuid, path)
+            if result is "success":        
+                results.append(
+                    {
+                        **details,
+                        'book_uuid': book_uuid,
+                        'sidecar_path': path,
+                    }
+                )
+                num_success += 1
+            elif result is "failure":        
+                results.append(
+                    {
+                        **details,
+                        'book_uuid': book_uuid,
+                        'sidecar_path': path,
+                    }
+                )
+                num_fail += 1  
+            elif result is "no_metadata":
+                num_no_metadata += 1
+
+        if num_success > 0 and num_fail > 0:
+            warning_dialog(
+                self.gui,
+                'Results',
+                f'Metadata pushed to sidecars successfully for {num_success}.\n'
+                f'Metadata push to sidecars failed for {num_fail}.\n'
+                f'No metadata, and therefore no attempt made for {num_no_metadata}.',
+                det_msg=json.dumps(results, indent=2),
+                show=True,
+                show_copy_button=False
+            )
+        elif num_success > 0:  # and num_fail == 0
+            info_dialog(
+                self.gui,
+                'Success',
+                f'Metadata pushed successfully for all books ({num_success}). See below for details.\n'
+                f'No metadata, and therefore no attempt made for {num_no_metadata}.',
+                det_msg=json.dumps(results, indent=2),
+                show=True,
+                show_copy_button=False
+            )
+        else:  # not num_success
+            error_dialog(
+                self.gui,
+                'Failure',
+                'No metadata could be pushed to KOReader.',
+                det_msg=json.dumps(results, indent=2),
+                show=True,
+                show_copy_button=False
+            )
+
+    def sync_to_calibre(self):
+        """This plugin’s main purpose. It syncs the contents of
+        KOReader’s metadata sidecar files into calibre’s metadata.
+
+        :return:
+        """
+        debug_print = partial(
+            module_debug_print,
+            'KoreaderAction:sync_to_calibre:'
+        )
+
+        device = self.get_connected_device()
+
+        if not self.check_device(device):
+            return None
 
         sidecar_paths = self.get_paths(device)