Auto dereferencing pipe function

[boyeln]

It would be really handy to have a pip function for automatic dereferencing. I'm not quite sure what the best approach would be. One possible solution is that it just dereference every reference it comes across with a predefined depth, like *[ _id == "some-document" ] | deref(1) (where 1 is the depth). This would look a lot like the "raw"-prefixed property provided when you're using graphql.

One scenario where this would be super useful is when you're having an array of several different objects, and some of those objects have references you'll want to dereference. This is a little cumbersome when using the dereferencing operator ->.

[djfarly]

This feature would be extremely helpful.
We're developing a Sanity Studio that is meant to be extended by 3rd parties. We're hoping to have the queries (from a next app) mostly tucked away in library code. This means that using the dereferencing operator is not only cumbersome but borderline impossible. :D

We're currently looking into writing a recursive dereferencing algorithm using multiple queries (as described in #25). This is also what the gatsby source plugin does if I understand correctly. Afaik there is no generic js implementation for GROQ-queries? Also it would be great to not need to fire multiple queries.

A quick search on the sanity slack reveals many many people asking how to auto-resolve references so think it is a quite common issue.

Is there any way to help pushing this forward?

[judofyr]

Afaik there is no generic js implementation for GROQ-queries?

There is https://github.com/sanity-io/groq-js if you already have the data locally and just want to be able to filter/query it.

[wardsky]

+1 for this. Would be extremely useful.

[klaasman]

Until it becomes part of the groq query language we use a utility that recursively loads references. I'll share it here, might help/inspire others.

import { AsyncWalkBuilder } from 'walkjs';

const data = await sanityClient.fetch(`your query`);

// usage:
await replaceReferences(data, sanityClient)


/**
 * This function will mutate reference-objects:
 * The keys of a reference-object will be deleted and the keys of the reference-
 * document will be added.
 * eg:
 * { _type: 'reference', _ref: 'abc' }
 * becomes:
 * { _type: 'document', _id: 'abc', ...allOtherDocumentProps }
 */
async function replaceReferences(
  input: unknown,
  client: SanityClient,
  resolvedIds: string[] = []
) {
  await new AsyncWalkBuilder()
    .withGlobalFilter((x) => x.val?._type === 'reference')
    .withSimpleCallback(async (node) => {
      const refId = node.val._ref;

      if(typeof refId !== 'string') {
        throw new Error('node.val._ref is not set');
      }

      if (resolvedIds.includes(refId)) {
        const ids = `[${resolvedIds.concat(refId).join(',')}]`;
        throw new Error(
          `Ran into an infinite loop of references, please investigate the following sanity document order: ${ids}`
        );
      }

      const doc = await client.fetch(`*[_id == '${refId}']{...}[0]`);

      // recursively replace references
      await replaceReferences(doc, client, resolvedIds.concat(refId));

      /**
       * Here we'll mutate the original reference object by clearing the
       * existing keys and adding all keys of the reference itself.
       */
      Object.keys(node.val).forEach((key) => delete node.val[key]);
      Object.keys(doc).forEach((key) => (node.val[key] = doc[key]));
    })
    .walk(input);
}

[refactorized]

Really, any way to pass functionality down to arbitrarily, deeply nested data is needed to avoid really expensive n+1 type situations when the data is inherently nested.

What about an array of resolvers for references, which would just be groq projections (of the object reference).

More control over which references would be resolved could be expressed with similar syntax to filters and path operations

[refactorized]

Actually what I am trying to articulate is pretty exactly issue #25. Implementing it would be so helpful and open up an entire dimension of querying power.

In the meantime, we are resorting to the recurse-and-fetch strategy, using the handy code from @klaasman above.

[RavenHursT]

+1 for this as well (Not sure how much this'll help considering no on from sanity.io has bothered to even comment on this request)

I'm using the recurse/traverse & fetch strategy as well, but making all the requests results in page load times 2+s.. this isn't acceptable 😕

[EECOLOR]

@klaasman Interesting solution, however could potentially be a performance and quota hit.

I'm thinking, this could probably be solved at the other end by adjusting or generating the query with information from the schema. Did you investigate or experiment with that?

[kmelve]

@RavenHursT: We're absolutely watching these issues and are always listening to the community. Of course, we would like to be better about following up with updates too, but you'll have to bear with us as we scale and get processes in place.

In our experience, a lot of use cases can be solved by dereferencing in projections, even though it can require some specification. If it's recurring patterns, you can also use string concatenation to construct the GROQ query.

We're also super interested in learning about concrete use cases, which often helps when we make prioritizations for our roadmap.

[EECOLOR]

@kmelve A small example of a route map from one of our sites:

         home: [
            groq`*[_type == 'voorpagina' && language == $language] | order(_updatedAt desc)[0] {
              ...,
              hero {
                ...,
                image{..., asset->}
              },
              callToAction {
                ...,
                internalLink { ..., ref-> }
              },
              solutionCardsSlider {
                ...,
                cards[] {
                  ...,
                  image{..., asset->},
                  internalLink { ..., ref-> }
                },
              },
              content[] {
                ...,
                _type == 'productCards' => {
                  cards[] {
                    ...,
                    internalLink {
                      ...,
                      ref->{
                        ...,
                        hero {
                          ...,
                          image{..., asset->}
                        }
                      }
                    }
                  },
                },
                _type == 'partnerverhaalSlider' => {
                  items[] {
                    ...,
                    internalLink {
                      ...,
                      ref->{
                        ...,
                        hero {
                          ...,
                          image{..., asset->}
                        }
                      }
                    }
                  }
                }
              }
            }`,
            { language }
          ],
        }),
        translate: (params) => {
          return {
            _type: 'voorpagina',
            language: params.language
          }
        },
        extractParams: { voorpagina: extract(language) },
        derived: ({ data }) => ({
          doc: data.home,
          dataLayer: dataLayer({
            title: 'home',
            id: data.home._id,
            language: data.language,
            category: 'home',
          })
        })
      }
    },
    partnerverhalen: {
      path: { nl: 'partnerverhalen', en: 'partner-stories' },
      index: {
        path: '',
        data: {
          groq: ({ params: { language } }) => ({
            partnerverhalen: [
              groq`*[_type == 'partnerverhalen' && language == $language] | order(_updatedAt desc)[0] {
                ...,
                factCardPartnerverhaalGrid1 {
                  ...,
                  internalLink {
                    ...,
                    ref->{
                      ...,
                      hero {
                        ...,
                        image{..., asset->}
                      }
                    }
                  }
                },
                factCardPartnerverhaalGrid2 {
                  ...,
                  internalLink {
                    ...,
                    ref->{
                      ...,
                      hero {
                        ...,
                        image{..., asset->}
                      }
                    }
                  }
                },
                quote {
                  ...,
                  image{..., asset->}
                }
              }`,
              { language }
            ],
          }),
          translate: (params) => {
            return {
              _type: 'partnerverhalen',
              language: params.language
            }
          },
          extractParams: { partnerverhalen: extract(language) },
          derived: ({ data }) => (data.partnerverhalen && {
            doc: data.partnerverhalen,
            dataLayer: dataLayer({
              title: data.partnerverhalen.title,
              id: data.partnerverhalen._id,
              language: data.language,
              category: 'partnerverhalen',
            })
          })
        }
      },

The real route map now is more than 600 lines of which 300 are used for dereferencing. This one alone is 85 lines, just because we need to show images and links:

           *[_type == 'solution' && language == $language && slug.current == $slug] | order(_updatedAt desc)[0] {
              ...,
              hero {
                ...,
                image{..., asset->}
              },
              content[] {
                ...,
                _type == 'productCards' => {
                  cards[] {
                    ...,
                    asset->,
                    internalLink {
                      ...,
                      ref->{
                        ...,
                        hero {
                          ...,
                          image{..., asset->}
                        }
                      }
                    }
                  },
                },
                _type == 'partnerverhaalSlider' => {
                  items[] {
                    ...,
                    internalLink {
                      ...,
                      ref->{
                        ...,
                        hero {
                          ...,
                          image{..., asset->}
                        }
                      }
                    }
                  }
                },
                _type == 'productBanner' => {
                  ...,
                  internalLink {
                    ...,
                    ref->{
                      ...,
                      hero {
                        ...,
                        image{..., asset->}
                      }
                    }
                  }
                },
                _type == 'factCards' => {
                  ...,
                  items {
                    ...,
                    slotNarrow1 {
                      ...,
                      internalAndExternalLink[] {
                        ...,
                        _type == 'internalLink' => { ..., ref -> }
                      }
                    },
                    slotNarrow2 {
                      ...,
                      internalAndExternalLink[] {
                        ...,
                        _type == 'internalLink' => { ..., ref -> }
                      }
                    },
                    slotWide1 {
                      ...,
                      internalAndExternalLink[] {
                        ...,
                        _type == 'internalLink' => { ..., ref -> }
                      }
                    },
                  }
                }
              },
              contactCallToAction {
                ...,
                image{..., asset->}
              }
            }

As you can see we only use projection for dereferencing. Main reason for this is to try and keep down the noise.

[klaasman]

@klaasman Interesting solution, however could potentially be a performance and quota hit.

I'm thinking, this could probably be solved at the other end by adjusting or generating the query with information from the schema. Did you investigate or experiment with that?

True - in our case it only ran build time so performance + quota limits were less of a concern. I don't recall we've thought about mapping sanity schemas to groq queries but I don't see why that wouldn't work (assuming the schemas are structured nicely). It has been a while though, haven't touched sanity for like 6 months.

[refactorized]

@kmelve I think a lot of this just comes down to DX. If we want a (even partially) generalized solution for auto-dereferencing we have to write one ourselves, and that means either n+1 type issues or a fairly complex system for using the schema and/or api to create fully expressive queries - which as shown above can be really long if done manually. All this work certainly will beg the question 'what is sanity really doing for me here?'.

I understand that auto-dereferencing is a complicated issue that also invites a lot of misuse, but I don't think it's something that can be left to the developer indefinitely. My team, as I guess many others, are building a platform on top of sanity, and this is one of those 'just works' features we either need to create or (preferably) utilize.

One of my suggestions ( here: sanity-io/sanity#2771 ) is to leverage what seems to be the already existing functionality for selecting and projecting results used to describe studio preview rendering in the schema. I was super excited when I saw how simply it seemed to work for previews.

[refactorized]

True - in our case it only ran build time so performance + quota limits were less of a concern. I don't recall we've thought about mapping sanity schema's to groq queries but I don't see why that wouldn't work (assuming the schema's are structured nicely). It has been a while though, haven't touched sanity for like 6 months.

I think the part that's difficult is for list and nested structures of data that can be many different types. Imagine a page-building CMS (our actual use case) in which each page can contain one of dozens of components and their associated data, each of which could likely include images and other assets presented as references. The query for a page alone becomes very long and full of large select:case-like statements that pick out the right projection. Default projections, or even projections passed into a query as a list of delegates could really clean things up.

[brehen]

Until it becomes part of the groq query language we use a utility that recursively loads references. I'll share it here, might help/inspire others.

We recently migrated a website with a lot of page building from Gatsby to Next, where Gatsby solved this issue for us with the resolveReference parameter.

Each page in Sanity can link to other pages through a plethora of different link-like-components, and we had to rely on your script to resolve these references. We hit some performance issues, however. Our sanity quota maxed out quite quickly and we ended up with some insane build times.

I adjusted your utility function to allow for a more performant dereference journey. Sharing our changes to help other people who might also be hitting performance issues until (hopefully) GROQ supports this by itself. 🙏

In our case, we might have to re-evaluate our choice of CMS, seeing as linking between pages in our page builders turned out to be a real headache outside of Gatsby. 😓

import { AsyncWalkBuilder, Break } from 'walkjs'

// Script taken from https://github.com/sanity-io/GROQ/issues/21

// What reference types should be dereferenced in order to get a page slug.
const linkTypes = ['toPage', 'topicLink']

// Use map to cache references, to limit the number of fetches
const refCache = new Map<string, any>()

// Our specific groq query for resolving link references, adjust as needed
const linkQuery = (refId: string) =>
  `*[_id == '${refId}']{
            pageMetadata {
                 localeSlug
            }
     }[0]`
/**
 * This function will mutate reference-objects:
 * The keys of a reference-object will be deleted and the keys of the reference-
 * document will be added.
 * eg:
 * { _type: 'reference', _ref: 'abc' }
 * becomes:
 * { _type: 'document', _id: 'abc', ...allOtherDocumentProps }
 */
export const replaceReferences = async (
  input: unknown,
  client: any,
  maxLevel?: number,
) => {
  await replaceRefs({ input, client, maxLevel })
}

type ReplaceRefs = {
  input: unknown
  client: any
  maxLevel?: number
  resolvedIds?: string[]
  currentLevel?: number
}

const replaceRefs = async ({
  input,
  client,
  maxLevel = 2,
  resolvedIds = [],
  currentLevel = 0,
}: ReplaceRefs) => {
  await new AsyncWalkBuilder()
    .withGlobalFilter((x) => x.val?._type === 'reference')
    .withSimpleCallback(async (node) => {
      const refId = node.val._ref

      if (typeof refId !== 'string') {
        throw new Error('node.val._ref is not set')
      }

      if (resolvedIds.includes(refId) || currentLevel === maxLevel) {
        throw new Break()
      }

      let doc: Record<string, any>
      // If we have already resolved this reference, we can fetch it from memory
      if (refCache.has(refId)) {
        doc = refCache.get(refId)
      } else {
        // Fetching page slugs, which is usually one level deeper.
        if (linkTypes.includes(String(node.key))) {
          const query = linkQuery(refId)
          doc = await client.fetch(query)
       // If document is asset type, just dereference everything without going deeper.
        } else if (node.key === 'asset') {
          const query = `*[_id == '${refId}'][0]{
					...
				}`
          doc = await client.fetch(query)
        // For other references, go further down the rabbit hole.
        } else {
          doc = await client.fetch(`*[_id == '${refId}']{...}[0]`)

          // recursively replace references
          await replaceRefs({
            input: doc,
            client,
            maxLevel,
            resolvedIds: [...resolvedIds, refId],
            currentLevel: currentLevel + 1,
          })
        }
        refCache.set(refId, doc)
      }

      /**
       * Here we'll mutate the original reference object by clearing the
       * existing keys and adding all keys of the reference itself.
       */
      Object.keys(node.val).forEach((key) => delete node.val[key])
      Object.keys(doc).forEach((key) => (node.val[key] = doc[key]))
    })
    .walk(input)
}

[intensr]

its been a while, is it possible to give us an update here @kmelve ?

[kmelve]

It has – I'll bring it up with the team this week!

[matijagrcic]

Is there a possibility for a new option in Sanity to do a deep create as this would solve lot of reference managing engineers need to do now. Sanity can charge such operations differently if needed.

• Get the existing document
• Change the id to drafts. or whatever
• Pass the flag deepCopy: true

Sanity creates a new top level document and all the references are new pieces of content.

[raulfdm]

+1 for that. I also encountered this situation where I needed to resolve all references from items that may or may not reference another item

[leo-cheron]

It has – I'll bring it up with the team this week!

Can't wait to have some news about this.

90% of our groq requests need all references to be resolved. Auto dereferencing would massively reduce code specificity / time spent writing groq queries.

[danielskogstad]

+1 for this. The GROQ queries for dereferencing inside structures like pagebuilders are painful when we need to deal with a wide array of blocks containing references and/or assets. Although we could develop custom solutions to avoid having giant queries in the code that takes a lot away from readability and maintainability. For us this is a major lack with GROQ at the moment and we hope it gets looked into

[prostoleo]

Any updates?

[pm0u]

+1 Anything here?

[karlomedallo]

My use case is this:

I just want all _type: image to show just the asset->url. You can see that navigationItems is recursive with no predefined depth.
Either get this feature added or just expand all image types by default (or by a flag)

[ShayNehmad-RecoLabs]

Would like this as well

[choutkamartin]

Any updates please? I don't mean to sound rude, I think Sanity is a great CMS, however this is really a bottleneck of it. Using the scripts for "walking" through the object, finding references and then initiating the query to replace the reference with a populated one is killer for the Content Lake and the API quota. Run the script on several pages when static building the page, and you will be in very high number of API calls. With auto dereferencing, it would all happen inside one call / query, and we would not need to hit the API endpoint several times, which puts an extra pressure on everything sitting before the database.

[devarshiqm]

Hey @kmelve just wanted to know, if there are any updates on this.

[JoeMatkin]

@kmelve Hi, any update on this?

[sergeyzenchenko]

Same for us, huge pain, that was solvable with Contentful by using very deep selects

[refactorized]

I have given a lot of thought to this, and worked out a few solutions, mostly around resolving types post query, and using complex (recursive) typescript generics to reflect the as-yet-unknown type of the returned element.

I think a better solution would:

• allow us to specify 0..n dereferencing strategies
  • 0: don't dereference server-side
  • 1: default dereferencing strategies
  • 2-n: alternate dereferencing strategies in some query addressable index
• leverage existing patterns like groq with query params
• given above, well-defined references could be used to deterministically generate simpler / more performant types for the returned objects
• allow each combination of reference (_id) and deref. strategy to become a signature and boundary for caching/memoization - which could be implemented at several points between the client making the request and rendering the results
• be simple to understand and ideally based off of known behavior.

I once again point to the current (as of V2, I need to get on V3) behavior of select and prepare for previews. Here groq queries to get/project the preview data in studio are defined right inside the schema.

This works just fine in Studio which I am guessing executes the queries as needed, but it could be made to work in a broader context. My initial idea was to base our next work-around on the above by having a few extra properties in our schema defs which would then be used to dynamically rewrite (or statically generate) the actual queries that would be used. This is still a viable implementation, and maybe a library I would like to work on.

But Ideally this sub-query execution would happen server-side, allowing for much more optimization, and taking the burden off of out-of-band tooling.

[JoeMatkin]

any updates?

[JKarlavige]

Asked about this in Sanity's Slack and was sent to this issue. Would be great to have a way to automatically pull full data from all references within the query.

If we're able to expand reference data using ->, the functionality seems to be there. It would be ideal if we can add a flag to the query, such as the author's suggestion using | deref, and have that expand all reference fields within the query.

In the meantime we are manually expanding references where needed, which is quickly getting messy:

*[_type == "pageBuilder" && slug.current == $slug][0]{
  ...,
  pageBuilder[] {
    ...,
    refUser->,
    refUsers[]->,
    differentObj {
      ...,
      refUser->
    },
    cards[] {
      ...,
      refUser->
    },
  }
}

[komarovartem]

I believe I do have a good solution. It's a projections resolver which is building request string based on all of your schemas. In result, you will get all of you data in one request with all deferences and mutations for the requested document. I have been using this solution for quite some time now. I've tried to publish my solution in Sanity snippets but two months passed and it's still not published. So I'll try here.

One of the page builder sections - section-text-media.ts

import { defineType, defineField } from '@sanity/types';

export default defineType(
  {
    name: 'sectionTextMedia',
    title: 'Text + Media',
    type: 'object',
    fields: [
      defineField({
        name: 'title',
        type: 'string',
      }),
      defineField({
        name: 'image',
        type: 'image',
      }),
      defineField({
        name: 'link',
        type: 'reference',
      }),
    ],
    preview: {
      select: {
        subtitle: 'title',
      },
      prepare: ({ subtitle }) => ({ title: 'Text + Media', subtitle }),
    },
  },
);

Group all page builder sections in one index file - sections.ts

import sectionTextMedia from '@/sanity/schemas/sections/section-text-media';
import sectionCta from '@/sanity/schemas/sections/sectionCta';

const sections = [sectionTextMedia, sectionCta];

export default sections;

Custom field for the page builder - page-builder.ts

import { defineType } from '@sanity/types';
import sections from '@/sanity/schemas/sections';

export default defineType(
  {
    name: 'pageBuilder',
    type: 'array',
    of: [...sections.map((section) => ({ type: section.name }))],
  }
);

A document with a title, slug, image, and page builder fields - page.ts

import { defineType, defineField } from '@sanity/types';

export default defineType(
  {
    name: 'page',
    title: 'Pages',
    type: 'document',
    fields: [
      defineField({
        name: 'title',
        type: 'string',
      }),
      defineField({
        name: 'slug',
        type: 'slug',
        options: {
          source: 'title',
        },
      }),
      defineField({
        name: 'image',
        type: 'image',
      }),
      defineField({
        name: 'sections',
        type: 'pageBuilder',
      }),
  }
)

Combine all documents in index file - documents.ts

import page from '@/sanity/schemas/documents/page';
import posts from '@/sanity/schemas/documents/posts';

const documents = [page, posts];

export default documents;

A function to parse all fields within given schema type - utils.ts

import documents from '@/sanity/schemas/documents';
import sections from '@/sanity/schemas/sections';

export const projectionsResolver = (type, toDeep = false) => {
  const schema = [...documents, ...sections].find((d) => d.name === type);

  let query = '...,';

  const getFieldQuery = (field) => {
    switch (field.type) {
      case 'slug':
        return `"${field.name}": ${field.name}.current,`;
      case 'pageBuilder':
        return `${field.name}[] {
          ${sections.map(
            (section) => `_type == "${section.name}" => {
              ${toDeep ? '...,' : projectionsResolver(section.name)}
            }`
          )}
          },`;
      case 'image':
        return `${field.name} { ..., asset-> },`;
      case 'object':
        return `${field.name} {
            ${field.fields.map((f) => getFieldQuery(f)).join('')}
        },`;
      case 'array':
        // in case array of references
        if (field.of[0].type === 'reference') {
          return `${field.name}[]-> {
            ${field.of[0].to.map(
              (to) => `_type=="${to.type}" => { ${toDeep ? '...,' : projectionsResolver(getDoc(to.type), true)} }`
            )}
          },`;
        }

        // in case array of objects
        if (field.of[0].type === 'object') {
          return `${field.name}[] {
            ...,
            ${
              field.of[0].name
                ? field.of.map(
                    // if object has name attribute filter by type
                    (of) => `_type == "${of.name}" => {
                      ${of.fields.map((f) => getFieldQuery(f)).join('')}
                    }`
                  )
                : // if object has no name reference all its types
                  field.of.map((of) => `${of.fields.map((f) => getFieldQuery(f)).join('')}`)
            }
          },`;
        }

        // in case array of images
        if (field.of[0].type === 'image') {
          return `${field.name}[] {
            ..., asset->
          },`;
        }

        return `${field.name}[],`;
      case 'reference':
        if (toDeep) {
          return `${field.name}->,`;
        }

        return `${field.name}-> {
              ${field.to.map((to) => `_type=="${to.type}" => { ${projectionsResolver(to.type, true)} }`)}
        },`;
      default:
        return `${field.name},`;
    }
  };

  if (schema.fields) {
    schema.fields.forEach((field) => {
      query += getFieldQuery(field);
    });
  }

  return query;
};

When request is made, we generate the query string based on the document type - queries.ts

import { client, projectionsResolver } from '@/sanity/utils';

export async function getDocument(type, slug) {
    return client.fetch(
        `*[_type=="${type}" && slug.current=="${slug}"][0] {
            ${projectionsResolver(type)}
        }`,
    );
}

As an example, the next request returns all dereferenced data for a homepage - page.tsx

import { getDocument } from '@/sanity/queries';

export default async function Home() {
  const page = await getDocument({
    type: 'page',
    slug: 'homepage',
  });

  return <pre>{JSON.stringify(page, null, 4)}</pre>;
}