Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix an N^2 in batch hydration #172

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Fix an N^2 in batch hydration #172

wants to merge 1 commit into from

Commits on Oct 15, 2024

  1. Perform batch hydration in O(N) 

    This hadn't been noticed previously since N is generally small, eg. 50
or 60 at most. Working with a Metabase dashboard that loaded 10000
fields (100 each from 100 tables) in a single `select` + `hydrate`, I
found this was consuming about 3 seconds.

The slow path was effectively a very slow `conj`: taking the `acc`
vector, and then for each `annotated-instances` doing:

```clojure
(recur (vec (concat acc nil [(first hydrated-instances)]))
       ...)
```

which is pouring the whole vector into a lazy sequence and back into a
vector, once for each instance in the list.

The new version first creates an "index" of instances that need hydration, and
later uses that index to map the instances in the hydrated instances batch to
their places in the initial collection.

Co-authored-by: Braden Shepherdson <braden@metabase.com>
    @alexander-yakushev @bshepherdson
    alexander-yakushev and bshepherdson committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    4748350 View commit details
    Browse the repository at this point in the history