Proposal: Standardize content fallback text by putting it in codecs

[jhaaaa]

Currently, when clients send a content type, they are encouraged to include a contentFallback value that provides an alt text-like description of the original content for other clients that do not support that content type.

For example, let's take a look at the contentFallback for the standards-track Reaction content type:

await conversation.messages.send(reaction, {
  contentType: ContentTypeReaction,
  contentFallback: `Reacted ${reaction.content} to ${someMessage.content}`,
});

If an app is unable to handle the Reaction content type, it can display this descriptive text instead. For example:

Reacted 👍 to “Let’s meet for coffee at 10:30am”

In this way, providing a contentFallback value helps all apps built with XMTP maintain a basic level of usability.

Proposal

To reduce the friction for developers and provide a consistent experience as users navigate across different apps, we propose that a content type provide its contentFallback text in its codec.

As we consider using codecs (based on this PR - xmtp/xmtp-js#426) to standardize fallback text for core features in the XMTP ecosystem, we propose fallback text with the following goals:

• Provides helpful user context
• Be app-neutral (no app-specific references in fallback text)
• Be sender-receiver neutral (no sender-specific references in fallback text because the fallback text is constant for all chat participants)

Attachments

Can’t display "sunday-with-doggo.png"

Reactions

Based on the standards-track reactions content type:

Reacted {reaction} to an earlier message

Removed {reaction} from an earlier message

Alternatively, while not currently supported, being able to reference the message in fallback text would provide valuable context:

Reacted ${reaction} to “${message/filename referenced in reaction}”

Removed ${reaction} from “${message/filename referenced in reaction}”

Quote replies

Based on the standards-track reply content type:

Replied with “${message}” to an earlier message

Alternatively, while not currently supported, being able to reference the message in fallback text would provide valuable context:

Replied with “${message}” to “${message/filename referenced in reaction}”

Transaction receipts

[Onchain transaction] Use a blockchain explorer to learn more using the chainId: ${chainId} and transaction hash: ${txnhash}

Read receipts

Undefined (indicating that this fallback should not be visible to the user)

Enable user to find apps that support specific features

By appending to the end of “fallback text” that is visible to the user:

Here's a list of apps that support this feature: https://github.com/xmtp/app-feature-support

The value of adding a link such as above would be much higher if there was no fallback text available from the sending client. If context was available, the applicability of such a link would differ based on the content type.

Applicable:

• Attachments
• Transaction receipts

Maybe applicable:

• Reactions
• Quote replies

Not applicable:

• Read receipts

[cboscolo]

Thanks for writing up this proposal!
Our team has written a couple of codecs that we use to provide a richer app experience. With that context, we have two bits of feedback.

1. Some codecs are not ever intended for human consumption. It would be great of we could indicate this in the codec so that other apps that don't have logic to process the codec can refrain from cluttering the conversation with useless text.
2. For those codecs where it does make sense to provide fallback text, how do you propose we handle i18n and L10n in an interoperable way (and not expect everyone to understand english)?

[yash-luna]

Great points @cboscolo! What codecs have you been working on? Would love to learn more.

1. Are you suggesting that the content type itself should have a field that distinguishes it for human vs non-human consumption? Or are you referring to the fallback text? An example for the latter is read receipts: @daria-github recently started a discussion for a standards-track receipt content type where we recommend the fallback to undefined since the fallback not meant to be human readable and would clutter the UI.
2. I appreciate you raising this concern — we are building for a global audience where language considerations are critical. @neekolas has some ideas on handling i18n and l10n where the fallback text is translated to the sender's native language.

[neekolas]

We are not the only protocol to run into these kinds of issues around i18n.

High quality fallback text can only be created at the time of sending, because the sender is the one with a codec that knows how to handle the custom content type. They have the ability to inspect the content, and convert the relevant information to text.

Given those constraints, we probably need to make our SDK aware of the sender's native language (which is accessible from OS and browser APIs), and use that to translate the static components of the fallback text at the time of sending. While imperfect (maybe the sender is multi-lingual), it's the best we can do without degrading the quality of the fallback (generating text on the receiver side that is missing key context from the full message)

iMessage for example, falls back to SMS and has to make similar decisions for iMessage specific content (replies, reactions)

[cboscolo]

Whoops, I dropped the ball responding to this.

Are you suggesting that the content type itself should have a field that distinguishes it for human vs non-human consumption? Or are you referring to the fallback text? An example for the latter is read receipts: @daria-github recently started a discussion for a standards-track receipt content type where we recommend the fallback to undefined since the fallback not meant to be human readable and would clutter the UI.

Yes I was, but thinking a bit more about this, and I think the suggestion to set the fallback text to undefined is the right thing to do for a particular message.

In our case, we were seeking to mark an entire conversation as not intended for humans. I submitted a separate proposal to capture this idea. https://github.com/orgs/xmtp/discussions/50

[nmalzieu]

Thanks a lot for the proposal!
I like it and I think it makes sense.
Only thing is that we loose a bit of the experience for Converse since we use this format for reactions right now:

Replied with “${message}” to “${message/filename referenced in reaction}”

But I'm still in to have a more consistent experience across clients.

[yash-luna]

Appreciate your comments @nmalzieu!

Looping in @nakajima who had an idea for how we could tweak the reaction content type to support the fallback text where the reaction and the reference message are both included.