Skip to content

Latest commit

 

History

History
113 lines (87 loc) · 4.13 KB

DEVELOPER_GUIDE.md

File metadata and controls

113 lines (87 loc) · 4.13 KB
Raw

Development Guide

Table of Contents

  1. Prerequisites
  2. Project Structure
  3. Building the Project
  4. Running Tests
  5. Code Style
  6. Pre-commit Hooks
  7. Publishing

Prerequisites

  • Java Development Kit (JDK) 11
  • Gradle 8
  • Python3
  • Docker and Docker Compose (for local deployment)
  • AWS CLI (for AWS deployment)
  • AWS Cloud Development Kit (CDK) (for AWS deployment)
  • Node.js v18+ (for AWS deployment)

Project Structure

  • CreateSnapshot: Tools for creating cluster snapshots.
  • DocumentsFromSnapshotMigration: Utilities for migrating documents from snapshots.
  • MetadataMigration: Core functionality for migrating cluster metadata.
  • RFS (Reindex-From-Snapshot):
    • Migration utilities for document reindexing and metadata migration.
    • Includes tracing contexts for both document and metadata migrations.
  • TrafficCapture (Capture-and-Replay): Projects for proxying, capturing, and replaying HTTP traffic.
  • migrationConsole: A comprehensive CLI tool for executing the migration workflow.
    • console_api (experimental): Django-based API for orchestrating migration tasks.
    • lib/console_link: Core library for migration operations.
  • deployment: AWS deployment scripts and configurations.
  • dev-tools: Development utilities and API request templates.
  • docs: Project documentation and architecture diagrams.
  • libraries: Shared libraries used across the project.
  • test: End-to-end testing scripts and configurations.
  • transformation: Data transformation utilities for migration processes.
  • dashboardsSanitizer: CLI tool for sanitizing dashboard configurations.
  • testHelperFixtures: Test utilities including HTTP client for testing.

The migration console CLI provides users with a centralized interface to execute and manage the entire migration workflow, including:

  • Configuring source and target clusters
  • Managing backfill operations
  • Controlling traffic replay
  • Monitoring migration progress through metrics
  • Handling snapshots and metadata
  • Integrating with various deployment environments (Docker locally and AWS ECS)

Users can interact with the migration process through the CLI, which orchestrates the different components of the migration toolkit to perform a seamless migration between Elasticsearch and OpenSearch clusters.

Building the Project

./gradlew build

Running Tests

./gradlew test

Code Style

We use Spotless for code formatting. To check and apply the code style:

./gradlew spotlessCheck
./gradlew spotlessApply

Pre-Commit Hooks

Install the pre-commit hooks:

./install_githooks.sh

Publishing

This project can be published to a local Maven repository with:

./gradlew publishToMavenLocal

And subsequently imported into a separate Gradle project with (replacing name with any subProject name):

repositories {
    mavenCentral()
    mavenLocal()
}

dependencies {
    implementation group: "org.opensearch.migrations.trafficcapture", name: "captureKafkaOffloader", version: "0.1.0-SNAPSHOT"
    //... other dependencies
}

The entire list of published subprojects can be viewed as follows:

./gradlew listPublishedArtifacts

To include a test Fixture dependency, define the import similar to the following:

testImplementation testFixtures('org.opensearch.migrations.trafficcapture:trafficReplayer:0.1.0-SNAPSHOT')