Skip to content

@AWS-Big-Data-Projects Big Data Journal Projects

Change the repository type filter

All

    Repositories list

    80 repositories

    • aws-glue-test-data-generator

      Public
      AWS Glue Configurable Test Data Generator
      Python
      •
      MIT No Attribution
      •7•0•0•0•Updated Jun 2, 2023Jun 2, 2023

    • saas-analytics-infrastructure-on-aws

      Public
      JavaScript
      •
      MIT No Attribution
      •3•1•0•0•Updated May 22, 2023May 22, 2023

    • aws-glue-job-tracker

      Public
      Python
      •
      Other
      •4•1•0•0•Updated May 22, 2023May 22, 2023

    • host-openemr-on-aws-fargate

      Public
      Python
      •
      MIT No Attribution
      •3•0•0•0•Updated May 12, 2023May 12, 2023

    • data-engineering-zoomcamp

      Public template
      Free Data Engineering course!
      Jupyter Notebook
      •5.3k•0•0•0•Updated Apr 21, 2023Apr 21, 2023

    • dbt-glue

      Public
      This repository contains de dbt-glue adapter
      Python
      •
      Apache License 2.0
      •68•1•0•0•Updated Apr 7, 2023Apr 7, 2023

    • ClickHouse

      Public
      ClickHouse® is a free analytics DBMS for big data
      C++
      •
      Apache License 2.0
      •6.8k•0•0•0•Updated Mar 31, 2023Mar 31, 2023

    • emr-trino-autoscale

      Public
      Scala
      •
      MIT No Attribution
      •4•0•0•0•Updated Mar 20, 2023Mar 20, 2023

    • HeartRate-Monitoring-using-AWS-IOT-and-AWS-KINESIS

      Public
      you run a script to mimic multiple sensors publishing messages on an IoT MQTT topic, with one message published every second. The events get sent to AWS IoT, where an IoT rule is configured. The IoT rule captures all messages and sends them to Firehose. From there, Firehose writes the messages in batches to objects stored in S3. In S3, you set u…
      aws-kinesisaws-iotamazon-athenaaws-quicksight
      Python
      •
      Apache License 2.0
      •1•10•3•1•Updated Mar 19, 2023Mar 19, 2023

    • amazon-kinesis-data-analytics-blueprints

      Public
      Kinesis Data Analytics Blueprints are a curated collection of Apache Flink applications. Each blueprint will walk you through how to solve a practical problem related to stream processing using Apache Flink. These blueprints can be leveraged to create more complex applications to solve your business challenges in Apache Flink.
      TypeScript
      •
      MIT No Attribution
      •7•0•0•0•Updated Mar 14, 2023Mar 14, 2023

    • nextflow

      Public
      A DSL for data-driven computational pipelines
      Groovy
      •
      Apache License 2.0
      •623•0•0•0•Updated Mar 11, 2023Mar 11, 2023

    • awesome-opensource-data-engineering

      Public
      An Awesome List of Open-Source Data Engineering Projects
      Other
      •335•2•0•0•Updated Mar 7, 2023Mar 7, 2023

    • data-science-on-aws

      Public
      AI and Machine Learning with Kubeflow, Amazon EKS, and SageMaker
      Jupyter Notebook
      •
      Apache License 2.0
      •1.1k•0•0•0•Updated Feb 20, 2023Feb 20, 2023

    • spark-on-aws-lambda

      Public
      Python
      •
      MIT No Attribution
      •38•0•0•0•Updated Feb 19, 2023Feb 19, 2023

    • serverless-datalake

      Public
      Python
      •
      MIT No Attribution
      •6•0•0•0•Updated Dec 31, 2022Dec 31, 2022

    • msk-serverless-data-pipeline

      Public
      Java
      •
      MIT No Attribution
      •3•0•0•0•Updated Nov 10, 2022Nov 10, 2022

    • monitor-serverless-datalake

      Public
      Alerting and notification in a serverless data lake during failures
      Python
      •
      MIT No Attribution
      •3•0•0•0•Updated Nov 4, 2022Nov 4, 2022

    • aws-glue-cdk-cicd

      Public
      Build, Test and Deploy ETL solutions using AWS Glue and AWS CDK based CI/CD pipelines
      Python
      •
      MIT No Attribution
      •20•0•0•0•Updated Oct 4, 2022Oct 4, 2022

    • amazon-opensearch-batch-indexing-with-aws-lambda

      Public
      Python
      •
      MIT No Attribution
      •2•0•0•0•Updated Oct 3, 2022Oct 3, 2022

    • arvados

      Public
      An open source platform for managing and analyzing biomedical big data
      Go
      •
      Other
      •116•0•0•0•Updated Sep 19, 2022Sep 19, 2022

    • .github

      Public
      0•0•0•0•Updated Sep 19, 2022Sep 19, 2022

    • data-engineering

      Public
      Construct a modern data stack and orchestration the workflows to create high quality data for analytics and ML applications.
      Jupyter Notebook
      •33•0•0•0•Updated Sep 12, 2022Sep 12, 2022

    • aws-emr-serverless-using-terraform

      Public
      HCL
      •
      MIT No Attribution
      •12•0•0•0•Updated Sep 9, 2022Sep 9, 2022

    • AWS_File_Trans_Lamda_S3_SNS

      Public
      AWS Data Engineering Project using Lambda, S3 and SNS
      Python
      •4•2•0•0•Updated Aug 29, 2022Aug 29, 2022

    • querypal

      Public
      Web UI for Amazon Athena
      Vue
      •
      Apache License 2.0
      •26•0•0•0•Updated Aug 29, 2022Aug 29, 2022

    • aws-security-hub-glue-aggregator-terraform

      Public
      These Terraform modules aggregate Security Hub findings to centralized account using Amazon Kinesis Firehose and AWS Glue
      HCL
      •
      Apache License 2.0
      •5•0•0•0•Updated Aug 23, 2022Aug 23, 2022

    • bigdata-file-viewer

      Public
      A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.
      Java
      •
      GNU General Public License v2.0
      •54•0•0•0•Updated Aug 12, 2022Aug 12, 2022

    • Airline_Data_Analysis

      Public
      Process to gather streaming data from Airline API using NiFi & batch data using AWS redshift using Sqoop and build a data pipeline to analyse the data using Apache Hive and Druid and compare the performances ,to discuss the hive optimization techniques and visualise the data using AWS Quicksight
      GNU General Public License v3.0
      •1•11•0•0•Updated Jul 20, 2022Jul 20, 2022

    • emr-studio-notebook-examples

      Public
      This repository contains ready-to-use notebook examples for a wide variety of use cases in Amazon EMR Studio.
      MIT No Attribution
      •40•0•0•0•Updated Jul 18, 2022Jul 18, 2022

    • serverless-etl-cdk

      Public
      Python
      •
      MIT License
      •0•0•0•0•Updated Jul 18, 2022Jul 18, 2022