Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance optimization: make PREFERRED_TRIM_SIZE configurable #36

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

frontend-sensei
Copy link

@frontend-sensei frontend-sensei commented Oct 2, 2023

Why

I faced issue that if my initial YDoc starts from 3-5Mb this can lead to serious memory usage by IndexedDB on each update.

First steps

I investigated a lot of discussions, issues, forks etc. Understood each situation in detail.

What a problem?

People wait that this just should works good. But everyone has their own expectations. I think each tool need more detailed documentation about such processes.

After bunch hours of learning code and other situations I realized that inside y-indexeddb exists logic for optimization memory usage. It triggers under certain condition. You just need to reach it. But not so simple... 🫤

PREFERRED_TRIM_SIZE

On each document update this constant used for checking whether IndexedDB exceed to this limit. If yes - do trim.
Trimming - reduce data usage of IndexedDB.

  • PREFERRED_TRIM_SIZE equals 500
  • _dbsize equals records count in IndexedDB.

By simple words:

if (idbPersistence._dbsize >= PREFERRED_TRIM_SIZE) {
  // do trim
}

So, for triggering data trimming I need to do 500 YDoc updates to update IndexedDb 500 times respectively.

Okay, whats next?

If your update size is 12b it really doesn't matter - 12b * 500 = 6000b - 6kb
But if your update is 4Mb here is interesting - 4Mb * 500 = 2000Mb - 2Gb 🥲

And yeah, this is expected behavior.

So what do you want man? 🤔

I think the best approach would be to make PREFERRED_TRIM_SIZE configurable. For example decrease it value from 500 to 100. (4Mb*100 = 400Mb)
Main idea that each can configure PREFERRED_TRIM_SIZE by their data size.
If you know that your updates big - you decrease PREFERRED_TRIM_SIZE for triggering trimming earlier.

So this PR:

  • Makes PREFERRED_TRIM_SIZE configurable from IndexeddbPersistence instance via passing options object

Example:

new IndexeddbPersistence("my-document-id", doc, {
    PREFERRED_TRIM_SIZE: 100
})

@dmonad, I would be glad if you can provide feedback for my contribution!

Peace ✌️

Copy link
Member

@dmonad dmonad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. But please make the suggested changes.

src/y-indexeddb.js Outdated Show resolved Hide resolved
src/y-indexeddb.js Outdated Show resolved Hide resolved
@frontend-sensei
Copy link
Author

Feel free to ping me, if something can be improved 😉

@frontend-sensei
Copy link
Author

@dmonad It's ready for your review

@frontend-sensei
Copy link
Author

@dmonad Any updates?

@frontend-sensei
Copy link
Author

@dmonad By any chance, do you have an update?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants