Consider splitting geostore into separate buckets

[billgeo]

User Story

So that we can apply policies to entire groups of datasets more easily, as a data manager, I want my dataset group/type in a separate bucket.

Instead of having all the dataset groups/types in one bucket

Acceptance Criteria

• Given there is more than one dataset group / bucket, when I create a dataset, then I can specify which dataset group / bucket I want to add it to
• Given there is more than one dataset group / bucket, when I specify an existing dataset group/bucket while create my dataset, then my dataset is created and 'assigned' to the corret dataset group/bucket
• Given there is more than one dataset group / bucket, when I want to find out which dataset groups/buckets there are already exisitng, then I can view a list of dataset groups / buckets
• Given there is not a suitable dataset group / bucket for my dataset, when I want to craete a dataset, then I can also create a dataset group / bucket

Additional context

S3 bucket naming restrictions https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html

See confluence page here and slack discussion here, it has become apparent that LINZ will have quite a static structure of dataset groups/types (whether we use one bucket or more. This allows us the opportunity to split the dataset types into separate buckets and apply many useful policies at the bucket level.

Tasks

• [ ]
• ...

Definition of Ready

• This story is ready to work on
  • Independent (story is independent of all other tasks)
  • Negotiable (team can decide how to design and implement)
  • Valuable (from a user perspective)
  • Estimate value applied (agreed by team)
  • Small (so as to fit within an iteration)
  • Testable (in principle, even if there isn't a test for it yet)
  • Environments are ready to meet definition of done
  • Resources required to implement will be ready
  • Everyone understands and agrees with the tasks to complete the story
  • Release value (e.g. Iteration 3) applied
  • Sprint value (e.g. Aug 1 - Aug 15) applied

Definition of Done

• This story is done:
  • Acceptance criteria completed
  • Automated tests are passing
  • Code is peer reviewed and pushed to master
  • Deployed successfully to test environment
  • Checked against
    CODING guidelines
  • Relevant new tasks are added to backlog and communicated to the team
  • Important decisions recorded in the issue ticket
  • Readme/Changelog/Diagrams are updated
  • Product Owner has approved acceptance criteria as complete
  • Meets non-functional requirements:
    • Scalability (data): Can scale to 300TB of data and 100,000,000 files and ability to
      increase 10% every year
    • Scability (users): Can scale to 100 concurrent users
    • Cost: Data can be stored at < 0.5 NZD per GB per year
    • Performance: A large dataset (500 GB and 50,000 files - e.g. Akl aerial imagery) can be
      validated, imported and stored within 24 hours
    • Accessibility: Can be used from LINZ networks and the public internet
    • Availability: System available 24 hours a day and 7 days a week, this does not include
      maintenance windows < 4 hours and does not include operational support
    • Recoverability: RPO of fully imported datasets < 4 hours, RTO of a single 3 TB dataset
      < 12 hours