The Tale of Listings

[thislooksfun]

There has been some (understandable) confusion about how snoots handles Listings, namely why they are not arrays. The goal of this post is to explain why I made the decisions I did, as well as some of the issues I ran into along the way for those who want to dig into the depths of Reddit's terrible API.

A Short Story

I started work on this project as many of you likely are now (and should remain, if you value your sanity): ignorant of Reddit's API's inner workings. Alas by the very nature of this project, that innocence was not to last. Much of the API is straightforward: most of the responses include an object that kindly tells you what type it is:

{
  kind: "t1",
  data: { ... }
}

t1 is a comment, t2 is a user, t3 is a post, and so on. But little did I know that a monster lurked in the depths of the API: kind: "Listing". For you see, Listings are no mere object. Their very existence defies the laws of sanity themselves.

The Details

Ok dramatics aside, Listing responses really are a nightmare to work with. Sometimes they are kind and give back a complete list of all the data you requested, neatly bundled up for you. But most of the time they do not. Here's some examples:

1. Requesting a listing of some object (posts, comments, users, etc) paginates the response. Always. Yes you can request a large number, but Reddit's api has a hard cap of 100 items per request, so if you want more than that (and you often do), you have to perform a second request to get the next page.

   // An example response
   {
     kind: "Listing",
     data: {
       modhash: null,
       dist: 100,
       children: [
         {
           kind: "t3",
           data: { ... },
         },
         ...
       ],
       // This line tells us how to fetch more data. It means "to get the
       // next page request this same listing again but add ?after=t3_mpn4uh"
       after: "t3_mpn4uh",
       before: null,
     },
   }

2. When interacting with a Listing of comments like replies to a post or comment Reddit still uses the same Listing response structure, but instead of telling us to fetch the next page via the after key, like above, it instead adds something called a More object to the end of the children array. To make matters worse, that More object comes in one of two flavors:
   1. An array of the IDs of the comments to fetch
   2. A single ID of a comment that you then need to fetch and get the replies to
3. Empty listings on some nested / explicitly fetched objects that aren't actually empty. Namely when fetching comments by ID their replies key is actually an empty array, whether or not it has replies.

Why does this matter?

After all, snoowrap has to deal with these issues as well but it still exposes Listings as an array. Well there are some good reasons for treating Listings as a black box. Namely API calls. The vast majority of the time when you are working with Listings it will, at some point, require another API call to fetch more data. This means that if you treat a listing as an array and simply iterate over it you'll only get the first page. This can be solved in one of two ways:

1. Always fetch every page of a Listing before iterating over it, or fetch pages as you go. In the former case this can eat up large amounts of your ratelimit even when you are not using the results. For example, if you call client.posts.fetch("m1kw8i") (which has 1.9k comments) it would fetch all 1.9k comments before you can interact with the post at all, which as insane if you only wanted to know the title of the post.
2. Make you fetch the next page yourself. This is more-or-less what snoowrap does, and, as I explain here that approach quickly leads to easy mistakes that are next-to-impossible to catch.

I feel that both of these approaches are deeply flawed in their own way, and both go against the core design guidelines of snoots: "keep it simple" and "fetch only what you need". Unfortunately the only way to not have either of those problems is to make Listings a black box, not an array. This does introduce a slight bit of cognitive overhead, I'll admit, since operations like map and the index operator [] don't work, but the alternative is requiring you to remember to add the fetch in just the right way every single time. I think the results speak for themselves:

/// Print out all the comments until one of them was made by AutoModerator, making as few API calls as possible.

// snoowrap
const post: Submission = await client.getSubmission("m1kw8i").fetch();
let comments = post.comments;

outer:
while (!comments.isFinished) {
  for (const cmt of comments) {
    console.log(cmt.body);
    if (cmt.author.name === "AutoModerator) break outer;
  }

  comments = await comments.fetchMore({ amount: 50, append: false });
}

// snoots
const post = await client.posts.fetch("m1kw8i");
await post.comments.each(c => {
  console.log(c.body);
  return cmt.author.name !== "AutoModerator";
});

Conclusion

The takeaway here is that Listings are not arrays, and any attempt to treat them as such necessarily introduces some problem. There is no one-size-fits-all solution to this; some projects will always fetch everything anyway, so these details don't matter, and some only look at the first item of the list. I feel like the Listing API that snoots has is a decent middle ground. It only fetches what is needed when it's needed and it makes it impossible to forget to fetch the next page, but also makes it easy to loop over every entry in the list if that's what you need to do.

The future

The API for Listings is open for expansion. When I first wrote it it only had .each(), but in my own testing I discovered a need for .eachPage() (which gives you the array for each fetched page at a time) and .some() (which is essentially a promisified version of the default Array.some()). If there's some interaction that you are trying to perform that is made difficult because of this black-box approach please create a new discussion and share some example code of what you're trying to do, and we'll see if it can be made easier!