Storiny API

Note

This repository was branched out of storiny/web which was originally a monolith. If you're looking for the original git history before the split, you can find it over on the original repository.

Storiny API

This is the primary backend service that powers Storiny, handling the core site functionality such as user authentication, CRUD endpoints and the real-time story collaboration.

Tech stack

Framework: Actix Web serves our HTTP endpoints, web sockets for real-time story collaboration, and middlewares
Database: PostgreSQL paired with SQLX
State & caching: Redis (using deadpool-redis) handles our session storage, rate limiting and pub/sub
Real-time collaboration: Custom web socket based engine built around yrs (the Rust port of Yjs CRDTs)
Internal RPC: gRPC via Tonic for fast & strictly typed internal service communication between micro services
Background jobs: Apalis for cron jobs and Lapin for RabbitMQ message processing
Media storage: AWS S3 for storing stories and user uploaded media files
Email service: AWS SES for dispatching emails
Telemetry: Sentry for tracing, monitoring, and capturing errors in production

Realms (collaborative editing)

The heart of this repository is Realms (src/realms/): our real-time multi-peer collaborative story editing engine (similar to Google Docs or Notion, this is the system that makes cursor tracking and live story syncing possible across multiple peers).

We built Realms on top of Yjs using the Rust yrs port. It relies on conflict free replicated data types (CRDTs) to guarantee that all connected peers eventually end up with the exact same story state despite facing unavoidable network latency.

Under the hood

Connection & authentication: When a user opens a story/document, their client establishes a web socket connection to the API server. Inside src/realms/server.rs, we intercept the handshake, parse the SESSION_COOKIE_NAME, validate it against the user session present in Redis, and then query Postgres to ensure the user has the correct PeerRole (Viewer or Editor) for that specific document.
Realm manager (realm.rs): Once authenticated, the user is placed into a Realm. A Realm is essentially a document loaded into the memory.
- If the document is just opened, the Realm fetches the latest compressed binary CRDT state from AWS S3 (persistent long term document storage).
- The Realm maintains a BroadcastGroup (managed in broadcast.rs) that tracks all currently connected peers (tokio::sync::RwLock<HashMap<Uuid, Peer>>).
Sync Protocol (protocol.rs): We implemented an optimized binary protocol over web sockets. It consists of a few core message types:
- SyncStep1 & SyncStep2: The standard Yjs handshake. When a client joins, they send their StateVector (a summary of their document version). The server replies with any missing updates.
- Update: As users type, incremental CRDT updates are broadcasted via a tokio::sync::broadcast::channel to all peers in the BroadcastGroup.
- Awareness: This handles ephemeral data: cursor positions, selection ranges, and "who is currently online". This is kept out of the core CRDT document to keep the document size small (awareness data is never stored in long term document storage).
Persistence loop: The live document lives in memory (and is constantly being mutated) until there is no peer accessing it. Every PERSISTENCE_LOOP_DURATION (currently 60s), the Realm checks if the StateVector has changed. If it has, it takes a snapshot of the document, compresses it using gzip, and flushes it to S3 for long term persistent storage. When the last peer disconnects, the Realm does one final sync to S3 and then gracefully destroys itself to free up memory.

Structure

We organize the codebase as a Cargo workspace:

src/: The main API service
- realms/: The collaborative editing engine (as detailed above)
- routes/: HTTP handlers and controllers
- models/: Database models and data access layer
- middlewares/: Actix middlewares (Identity, Rate Limiting, Session)
- oauth/: Handlers for our social logins (Google, GitHub, Discord, Spotify, Dribbble, ...)
- grpc/: Tonic servers for internal network communication
- amqp/: RabbitMQ consumers
- cron/: Apalis scheduled workers
session/: Workspace crate handling our custom Redis session store logic
macros/: Workspace crate containing custom procedural macros
migrations/: SQLx migration files for setting up PostgreSQL schemas
geo/: Contains the MaxMind GeoLite2 databases used for IP-based geolocation features

Launching locally

1. Prerequisites

Rust
PostgreSQL
Redis
Just: a handy command runner (install via cargo install just)

2. Environment setup

Copy the mapping file and fill in your own environment variables:

cp .env.mapping .env

3. Database & GeoIP

Fetch the GeoIP database:

just update_geo

Apply the database migrations using SQLX:

just migrate

4. Running the service

Fire up the development server with host reload (requires cargo-watch and optionally bunyan for pretty logs):

just dev

5. Testing

We use cargo-nextest for running our test suites:

just test

API endpoints

Auth

/v1/auth
- POST /v1/auth/login: Authenticates a user with email, password, and optional MFA or recovery code to establish a session.
- POST /v1/auth/mfa-preflight: Verifies user credentials and returns whether multi-factor authentication (MFA) is enabled for the account.
- POST /v1/auth/recovery: Initiates account recovery by generating a password reset token and sending a reset link via email.
- POST /v1/auth/resend-verification-email: Generates a new verification token and resends the email verification link to an unverified user.
- POST /v1/auth/reset-password: Resets a user's password using a valid reset token with an option to log out of all active sessions.
- POST /v1/auth/signup: Registers a new user account with provided credentials and sends an email verification link.
- /v1/auth/external
  - /v1/auth/external/google
    - GET /v1/auth/external/google: Initiates Google OAuth2 authentication flow by redirecting the user to Google's authorization URL.
    - GET /v1/auth/external/google/callback: Handles the Google OAuth2 callback to authenticate or register the user and establish a session.

Blogs

/v1/blogs
- GET /v1/blogs/{blog_id}/archive: Retrieves a paginated archive of published stories for a blog, optionally filtered by year and month.
- GET /v1/blogs/{blog_id}/editors: Retrieves a paginated list of active editors for a blog, including follow and mute status for authenticated users.
- GET /v1/blogs/{blog_id}/feed: Retrieves a paginated feed of published stories for a blog, supporting keyword search and sorting by date.
- POST /v1/blogs/{blog_id}/subscribe: Subscribes an email address to a blog's newsletter and sends a confirmation email.
- GET /v1/blogs/{blog_id}/writers: Retrieves a paginated list of active writers for a blog, including follow and mute status for authenticated users.

Me

/v1/me:
- GET: Retrieves the authenticated user's profile details.
- /account-activity:
  - GET: Retrieves paginated account activity logs for the authenticated user.
- /assets:
  - GET: Retrieves uploaded media assets for the authenticated user.
  - POST: Uploads a new media asset.
  - /{asset_id}:
    - DELETE: Deletes a media asset.
    - /alt:
      - PATCH: Updates the alt text of a media asset.
    - /favourite:
      - POST: Favorites a media asset.
      - DELETE: Removes a media asset from favorites.
    - /rating:
      - PATCH: Updates the content rating of a media asset.
- /blocked-users:
  - GET: Retrieves blocked users for the authenticated user.
  - /{blocked_id}:
    - POST: Blocks a user.
    - DELETE: Unblocks a user.
- /blog-requests:
  - GET: Retrieves incoming blog writer and editor invitation requests.
  - /{id}:
    - POST: Accepts an incoming blog invitation request.
    - DELETE: Declines or removes an incoming blog invitation request.
- /blogs:
  - GET: Retrieves blogs owned by or contributed to by the authenticated user.
  - POST: Creates a new blog.
  - /{blog_id}:
    - /content:
      - /pending-stories:
        
        GET: Retrieves pending stories submitted to a blog.
      - /published-stories:
        
        GET: Retrieves published stories of a blog.
    - /editor-requests:
      - GET: Retrieves outgoing editor invitation requests for a blog.
      - /{request_id}/cancel:
        
        POST: Cancels a pending outgoing editor invitation request.
    - /editors:
      - POST: Invites a user to be an editor for a blog.
      - /{user_id}:
        
        DELETE: Removes an editor from a blog.
    - /settings:
      - /appearance:
        
        /branding:
        
        PATCH: Updates blog branding display settings.
        
        /favicon:
        
        PATCH: Updates the favicon of a blog.
        
        /fonts:
        
        /upload:
        
        POST: Uploads a custom font for a blog.
        
        /{variant}:
        
        DELETE: Deletes a custom font variant of a blog.
        
        /mark:
        
        PATCH: Updates the mark or watermark image of a blog.
        
        /page-layout:
        
        PATCH: Updates page layout settings of a blog.
        
        /story-layout:
        
        PATCH: Updates story layout settings of a blog.
        
        /theme:
        
        PATCH: Updates theme settings of a blog.
      - /banner:
        
        PATCH: Updates the banner image of a blog.
      - /connections:
        
        PATCH: Updates social media connection links for a blog.
      - /delete-blog:
        
        POST: Deletes a blog.
      - /domain:
        
        POST: Verifies and attaches a custom domain to a blog.
        
        DELETE: Removes a custom domain from a blog.
        
        /code-request:
        
        POST: Requests a domain verification code for a custom domain.
      - /general:
        
        PATCH: Updates general settings of a blog.
      - /logo:
        
        PATCH: Updates the logo of a blog.
      - /newsletter_splash:
        
        PATCH: Updates newsletter splash page settings for a blog.
      - /seo:
        
        PATCH: Updates SEO settings for a blog.
      - /sidebars:
        
        /lsb:
        
        PATCH: Updates left sidebar settings for a blog.
        
        /rsb:
        
        PATCH: Updates right sidebar settings for a blog.
      - /slug:
        
        PATCH: Updates the URL slug of a blog.
      - /visibility:
        
        PATCH: Updates visibility settings of a blog.
    - /stats/stories:
      - GET: Retrieves story statistics for a blog.
    - /stories/{story_id}:
      - POST: Publishes a submitted story or draft to a blog.
      - PUT: Updates an already published story on a blog.
      - DELETE: Removes a story from a blog.
    - /subscribers:
      - GET: Retrieves subscribers of a blog.
      - /import:
        
        POST: Imports subscribers to a blog.
      - /{subscriber_id}:
        
        DELETE: Removes a subscriber from a blog.
    - /writer-requests:
      - GET: Retrieves outgoing writer invitation requests for a blog.
      - /{request_id}/cancel:
        
        POST: Cancels a pending outgoing writer invitation request.
    - /writers:
      - POST: Invites a user to be a writer for a blog.
      - /{user_id}:
        
        DELETE: Removes a writer from a blog.
- /bookmarks:
  - GET: Retrieves bookmarked stories for the authenticated user.
  - /{story_id}:
    - POST: Bookmarks a story.
    - DELETE: Removes a story from bookmarks.
- /collaboration-requests:
  - GET: Retrieves story collaboration requests for the authenticated user.
  - /{id}:
    - POST: Accepts a story collaboration request.
    - DELETE: Declines or removes a story collaboration request.
    - /cancel:
      - POST: Cancels a sent story collaboration request.
- /comments:
  - GET: Retrieves comments authored by the authenticated user.
  - POST: Posts a new comment on a story.
  - /{comment_id}:
    - PATCH: Updates an existing comment.
    - DELETE: Deletes a comment.
- /contributions:
  - GET: Retrieves stories contributed to by the authenticated user.
- /drafts:
  - GET: Retrieves draft stories of the authenticated user.
  - /{draft_id}:
    - DELETE: Soft-deletes a draft story.
    - /recover:
      - POST: Recovers a soft-deleted draft story.
- /flow/onboarding:
  - /tags:
    - GET: Retrieves recommended tags during onboarding.
  - /writers:
    - GET: Retrieves recommended writers during onboarding.
- /followed-blogs/{blog_id}:
  - POST: Follows a blog.
  - DELETE: Unfollows a blog.
- /followed-tags:
  - GET: Retrieves tags followed by the authenticated user.
  - /{tag_id}:
    - POST: Follows a tag.
    - DELETE: Unfollows a tag.
- /followers:
  - GET: Retrieves followers of the authenticated user.
  - /{follower_id}:
    - DELETE: Removes a follower.
- /following:
  - GET: Retrieves users followed by the authenticated user.
  - /{followed_id}:
    - POST: Follows a user.
    - DELETE: Unfollows a user.
- /friend-requests:
  - GET: Retrieves incoming friend requests for the authenticated user.
  - /{transmitter_id}:
    - POST: Accepts a friend request from a user.
    - DELETE: Declines a friend request from a user.
  - /{receiver_id}/cancel:
    - POST: Cancels a sent friend request.
- /friends:
  - GET: Retrieves friends of the authenticated user.
  - /{receiver_id}:
    - POST: Sends a friend request to a user.
  - /{transmitter_or_receiver_id}:
    - DELETE: Removes a friend.
- /gallery:
  - GET: Retrieves gallery images of the authenticated user.
  - POST: Uploads a new image to the user's gallery.
- /history:
  - GET: Retrieves the reading history of the authenticated user.
- /leave-blog/{blog_id}:
  - POST: Leaves a blog as a writer or editor.
- /leave-story/{story_id}:
  - POST: Leaves a story as a contributor.
- /liked-comments/{comment_id}:
  - POST: Likes a comment.
  - DELETE: Removes a like from a comment.
- /liked-replies/{reply_id}:
  - POST: Likes a comment reply.
  - DELETE: Removes a like from a comment reply.
- /liked-stories:
  - GET: Retrieves stories liked by the authenticated user.
  - /{story_id}:
    - POST: Likes a story.
    - DELETE: Removes a like from a story.
- /lookup/username:
  - GET: Searches for users by username query.
- /muted-users:
  - GET: Retrieves muted users for the authenticated user.
  - /{muted_id}:
    - POST: Mutes a user.
    - DELETE: Unmutes a user.
- /newsletters/{blog_id}:
  - POST: Subscribes to a blog newsletter.
  - DELETE: Unsubscribes from a blog newsletter.
- /notifications:
  - GET: Retrieves notifications for the authenticated user.
  - /read-all:
    - POST: Marks all notifications as read.
  - /{notification_id}/read:
    - POST: Marks a specific notification as read.
- /replies:
  - GET: Retrieves comment replies authored by the authenticated user.
  - POST: Posts a reply to a comment.
  - /{reply_id}:
    - PATCH: Updates a comment reply.
    - DELETE: Deletes a comment reply.
- /settings:
  - /accounts:
    - /add/google:
      - POST: Initiates Google OAuth account linking.
      - /callback:
        
        GET: Handles Google OAuth callback to complete account linking.
    - /remove:
      - POST: Unlinks a connected OAuth provider account.
  - /avatar:
    - PATCH: Updates the profile avatar image of the authenticated user.
  - /banner:
    - PATCH: Updates the profile banner image of the authenticated user.
  - /connections:
    - /{connection_id}:
      - DELETE: Removes a social connection.
      - /visibility:
        
        PATCH: Updates the visibility of a social connection.
  - /email:
    - PATCH: Updates the email address of the authenticated user.
  - /mfa:
    - /generate-codes:
      - POST: Generates new multi-factor authentication recovery codes.
    - /recovery-codes:
      - GET: Retrieves multi-factor authentication recovery codes.
    - /remove:
      - POST: Disables multi-factor authentication.
    - /request:
      - POST: Requests multi-factor authentication setup details.
    - /verify:
      - POST: Verifies code and enables multi-factor authentication.
  - /notifications:
    - /mail:
      - PATCH: Updates email notification preferences.
    - /site:
      - PATCH: Updates site notification preferences.
    - /unsubscribe:
      - PATCH: Unsubscribes from specific notification channels.
  - /password:
    - /add:
      - POST: Adds a password to an account lacking one.
      - /request-verification:
        
        POST: Requests verification code to set up a new password.
    - /update:
      - PATCH: Updates the user password.
  - /privacy:
    - /delete-account:
      - POST: Initiates permanent account deletion.
    - /disable-account:
      - POST: Temporarily disables the user account.
    - /following-list:
      - PATCH: Updates privacy settings for following list visibility.
    - /friend-list:
      - PATCH: Updates privacy settings for friend list visibility.
    - /incoming-blog-requests:
      - PATCH: Updates privacy settings for incoming blog requests.
    - /incoming-collaboration-requests:
      - PATCH: Updates privacy settings for incoming collaboration requests.
    - /incoming-friend-requests:
      - PATCH: Updates privacy settings for incoming friend requests.
    - /private-account:
      - PATCH: Toggles private account mode.
    - /read-history:
      - PATCH: Toggles saving reading history.
    - /sensitive-content:
      - PATCH: Toggles display of sensitive content.
  - /profile:
    - PATCH: Updates user profile information (bio, name, etc.).
  - /sessions:
    - /acknowledge:
      - POST: Acknowledges an active session security notice.
    - /destroy:
      - POST: Revokes all active user sessions.
    - /logout:
      - POST: Logs out a specific session by ID.
  - /username:
    - PATCH: Updates the username of the authenticated user.
- /stats:
  - /account:
    - GET: Retrieves account-wide analytics and statistics.
  - /stories:
    - GET: Retrieves story analytics and performance statistics.
- /status:
  - POST: Sets custom user status.
  - DELETE: Clears custom user status.
- /stories:
  - GET: Retrieves published stories authored by the authenticated user.
  - /{story_id}:
    - DELETE: Soft-deletes a story.
    - /contributors:
      - GET: Retrieves contributors of a story.
      - POST: Invites a contributor to a story.
      - /{user_id}:
        
        PATCH: Updates contributor role/permissions for a story.
        
        DELETE: Removes a contributor from a story.
    - /metadata:
      - PATCH: Updates metadata (title, tags, cover image, etc.) of a story.
    - /publish:
      - POST: Publishes a draft story.
      - PUT: Updates a published story.
    - /recover:
      - POST: Recovers a soft-deleted story.
    - /stats:
      - GET: Retrieves analytics and statistics for a specific story.
    - /unpublish:
      - POST: Unpublishes a published story back to draft.
- /subscriptions/{followed_id}:
  - POST: Subscribes to a creator/user.
  - DELETE: Unsubscribes from a creator/user.
- /unread-notifications:
  - GET: Retrieves unread notification counts and items for the authenticated user.

Public

/v1/public
- /blogs
  - GET /v1/public/blogs/{blog_id}/stories/{story_id}/recommendations: Fetches recommended stories for a given story within a specific blog.
- /cards
  - GET /v1/public/cards/user/{identifier}: Retrieves public profile card data for a user by username or ID.
- /comments
  - GET /v1/public/comments/{comment_id}/replies: Retrieves paginated replies for a specific comment.
  - POST /v1/public/comments/{comment_id}/visibility: Toggles the visibility (hidden status) of a comment on a story owned by the authenticated user.
- /explore
  - GET /v1/public/explore/stories: Explores or searches published stories filtered by category and optional query text.
  - GET /v1/public/explore/tags: Explores or searches popular tags for published stories in a given category.
  - GET /v1/public/explore/writers: Explores or searches active writers with published stories in a given category.
- /preview
  - GET /v1/public/preview/{story_id}: Retrieves basic preview metadata for a public story.
- /replies
  - POST /v1/public/replies/{reply_id}/visibility: Toggles the visibility (hidden status) of a reply on a comment owned by the authenticated user.
- /reports
  - POST /v1/public/reports: Submits a new content report for a specified entity.
- /stories
  - GET /v1/public/stories/{story_id}/comments: Retrieves paginated comments for a specific story with optional sorting, filtering, and text search.
  - POST /v1/public/stories/{story_id}/read: Records a story read event and updates reading analytics after verifying an active reading session token.
  - GET /v1/public/stories/{story_id}/recommendations: Fetches recommended stories related to a specific story.
- /tags
  - GET /v1/public/tags: Searches tag names by text query for authenticated users.
- /validation
  - POST /v1/public/validation/username: Validates username availability and checks against reserved keywords.

Misc

/v1/newsletters
- unsubscribe
  - POST /v1/newsletters/unsubscribe/{digest}/{encoded}: Unsubscribes a user from a newsletter via a one-click HTTP POST request (used by email clients).
  - GET /v1/newsletters/unsubscribe/{digest}/{encoded}: Handles newsletter unsubscription via GET for manual links in email footers.
/v1/tags
- stories
  - GET /v1/tags/{tag_name}/stories: Retrieves stories associated with a specific tag, supporting pagination, sorting, and search filtering.
- writers
  - GET /v1/tags/{tag_name}/writers: Retrieves top writers associated with a specific tag.
/v1/users
- followers
  - GET /v1/users/{user_id}/followers: Retrieves a paginated list of followers for a specified user.
- following
  - GET /v1/users/{user_id}/following: Retrieves a paginated list of users followed by a specified user.
- friends
  - GET /v1/users/{user_id}/friends: Retrieves a paginated list of mutual friends for a specified user.
- stories
  - GET /v1/users/{user_id}/stories: Retrieves stories published by a specified user.
/v1/feed
- GET /v1/feed: Retrieves the user's main story feed (suggested or following stories) with pagination support.
/v1/rsb-content
- GET /v1/rsb-content: Retrieves right-sidebar (RSB) content containing recommended stories, users, and tags.

Name		Name	Last commit message	Last commit date
.config		.config
.github/workflows		.github/workflows
.sqlx		.sqlx
data/ua_parser		data/ua_parser
geo		geo
lua		lua
macros		macros
migrations		migrations
queries		queries
session		session
src		src
static		static
templates		templates
tests		tests
.dockerignore		.dockerignore
.env.mapping		.env.mapping
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
build.rs		build.rs
clippy.toml		clippy.toml
justfile		justfile
redis_namespaces.md		redis_namespaces.md
rustfmt.toml		rustfmt.toml
sailfish.toml		sailfish.toml
sqlstates.md		sqlstates.md

Navigation Menu

Uh oh!

Repository files navigation

Storiny API

Tech stack

Realms (collaborative editing)

Under the hood

Structure

Launching locally

1. Prerequisites

2. Environment setup

3. Database & GeoIP

4. Running the service

5. Testing

API endpoints

Auth

Blogs

Me

Public

Misc

About

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Storiny API

Tech stack

Realms (collaborative editing)

Under the hood

Structure

Launching locally

1. Prerequisites

2. Environment setup

3. Database & GeoIP

4. Running the service

5. Testing

API endpoints

Auth

Blogs

Me

Public

Misc

About

Topics

Resources

Stars

Watchers

Forks

Contributors

Languages