diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..0d5c4b2 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,138 @@ +# Contributor's Code of Conduct + +If you contribute to this repo, you agree to abide by this code of conduct for +this community. + +We abide by the +[Contributor Covenant Code of Conduct, 2.0](https://www.contributor-covenant.org/version/2/0/code_of_conduct/). +It is reproduced below for ease of use. + +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, religion, or sexual identity +and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, +diverse, inclusive, and healthy community. + +## Our Standards + +Examples of behavior that contributes to a positive environment for our +community include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +* Focusing on what is best not just for us as individuals, but for the + overall community + +Examples of unacceptable behavior include: + +* The use of sexualized language or imagery, and sexual attention or + advances of any kind +* Trolling, insulting or derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or email + address, without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Enforcement Responsibilities + +Community leaders are responsible for clarifying and enforcing our standards of +acceptable behavior and will take appropriate and fair corrective action in +response to any behavior that they deem inappropriate, threatening, offensive, +or harmful. + +Community leaders have the right and responsibility to remove, edit, or reject +comments, commits, code, wiki edits, issues, and other contributions that are +not aligned to this Code of Conduct, and will communicate reasons for moderation +decisions when appropriate. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when +an individual is officially representing the community in public spaces. +Examples of representing our community include using an official e-mail address, +posting via an official social media account, or acting as an appointed +representative at an online or offline event. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the community leaders responsible for enforcement at +[opensource@logdna.com](mailto:opensource@logdna.com). +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the +reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining +the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed +unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing +clarity around the nature of the violation and an explanation of why the +behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series +of actions. + +**Consequence**: A warning with consequences for continued behavior. No +interaction with the people involved, including unsolicited interaction with +those enforcing the Code of Conduct, for a specified period of time. This +includes avoiding interactions in community spaces as well as external channels +like social media. Violating these terms may lead to a temporary or +permanent ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including +sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public +communication with the community for a specified period of time. No public or +private interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, is allowed during this period. +Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behavior, harassment of an +individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within +the community. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 2.0, available at +https://www.contributor-covenant.org/version/2/0/code_of_conduct.html. + +Community Impact Guidelines were inspired by [Mozilla's code of conduct +enforcement ladder](https://github.com/mozilla/diversity). + +[homepage]: https://www.contributor-covenant.org + +For answers to common questions about this code of conduct, see the FAQ at +https://www.contributor-covenant.org/faq. Translations are available at +https://www.contributor-covenant.org/translations. + diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d27c902..422fd6a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,15 +1,44 @@ # Contributing -## Github Workflow - -Contributions are always welcome! Be sure to follow the [github workflow](https://guides.github.com/introduction/flow/) when contributing to this project: - -* Create an issue, or comment on an issue to indicate what you are working on. This avoids work duplication. -* Fork the repository and clone to your local machine -* You should already be on the default branch `master` - if not, check it out (`git checkout master`) -* Create a new branch for your feature/fix `git checkout -b my-new-feature`) -* Write your feature/fix -* Stage the changed files for a commit (`git add .`) -* Commit your files with a *useful* commit message ([example](https://github.com/Azure/azure-quickstart-templates/commit/53699fed9983d4adead63d9182566dec4b8430d4)) (`git commit`) -* Push your new branch to your GitHub Fork (`git push origin my-new-feature`) -* Visit this repository in GitHub and create a Pull Request. +## Process + +We use a fork-and-PR process, also known as a triangular workflow. This process +is fairly common in open-source projects. Here's the basic workflow: + +1. Fork the upstream repo to create your own repo. This repo is called the origin repo. +2. Clone the origin repo to create a working directory on your local machine. +3. Work your changes on a branch in your working directory, then add, commit, and push your work to your origin repo. +4. Submit your changes as a PR against the upstream repo. You can use the upstream repo UI to do this. +5. Maintainers review your changes. If they ask for changes, you work on your + origin repo's branch and then submit another PR. Otherwise, if no changes are made, + then the branch with your PR is merged to upstream's main trunk, the main branch. + +When you work in a triangular workflow, you have the upstream repo, the origin +repo, and then your working directory (the clone of the origin repo). You do +a `git fetch` from upstream to local, push from local to origin, and then do a PR from origin to +upstream—a triangle. + +If this workflow is too much to understand to start, that's ok! You can use +GitHub's UI to make a change, which is autoset to do most of this process for +you. We just want you to be aware of how the entire process works before +proposing a change. + +Thank you for your contributions; we appreciate you! + +## License + +Note that we use a standard [MIT](./LICENSE) license on this repo. + +## Coding style + +Code style is enforced by [eslint][]. Linting is applied CI builds when a pull request +is made. The rule set being enforced is provided by [eslint-config-logdna][] + +## Questions? + +The easiest way to get our attention is to comment on an existing, or open a new +[issue][]. + +[eslint]: https://eslint.org +[eslint-config-logdna]: https://github.com/logdna/eslint-config-logdna +[issue]: https://github.com/logdna/eslint-config-logdna/issues diff --git a/README.md b/README.md index 1fac533..43898c9 100644 --- a/README.md +++ b/README.md @@ -2,62 +2,75 @@ The LogDNA Amazon S3 integration relies on [AWS Lambda](https://docs.aws.amazon.com/lambda/index.html) to route your logs from [S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html) to LogDNA. -## Configure the LogDNA AWS Lambda function +## How to Use +### Deploy the Code 1. Create a [new AWS Lambda function](https://console.aws.amazon.com/lambda/home) and select `Author from scratch` -2. For the basic information: - * Function Name: `logdna_s3` (you can choose what to name it) - * Runtime: `Node.js.10.x` -3. Click on the Lambda function to edit the details: +2. Click on the Lambda function to edit the details: * Code entry type: `Upload a .ZIP file` * Upload our LogDNA Lambda function [.ZIP File](https://github.com/logdna/logdna-s3/releases/latest/download/logdna-s3.zip) * Handler: `index.handler` - * Runtime: `Node.js.10.x` - * Environment variables: - * `LOGDNA_KEY`: `YOUR_INGESTION_KEY_HERE` *(Required)* - * `LOGDNA_HOSTNAME`: Alternative Host Name *(Optional)* - * `LOGDNA_TAGS`: Comma-separated Tags *(Optional)* - * `LOGDNA_URL`: Custom Ingestion URL *(Optional)* -4. For Execution role, assign a role that has the following policies: + * Runtime: `Node.js.14.x` + +### Configuration +#### General Configuration +If the lambda is being used to stream from gzipped files: +1. Set `Timeout` to, at least, `30 seconds` +2. Set `Memory` limit to, at least, `512 MB` + +**Notes**: + * The number of retries is recommended to be set to 0 since retrying lambda execution can result in duplicate logs. It can be modified in `Configuration > Asynchronous invoication` + * `Timeout` and `Memory` limit may need to be increased if the file size is too big + +#### Triggers +Add `S3` as a trigger with the following configuration: + * Specify the `bucket` you want to stream from + * Specify the event types you want to capture + * Optional prefix and suffix options are related to object paths within the specified bucket + +**Notes**: + * This S3 Lambda function and S3 Bucket must be in the same availability zone + * You can specify a bucket only in one trigger and/or S3 Lambda function since a bucket accepts only one subscription but one S3 Lambda function can stream from multiple buckets at the same time + +#### Permissions +For Execution role, assign a role that has the following policies: * [`AmazonS3ReadOnlyAccess`](https://gist.github.com/bernadinm/6f68bfdd015b3f3e0a17b2f00c9ea3f8#file-all_aws_managed_policies-json-L4392-L4417) * [`AWSLambdaBasicExecutionRole`](https://gist.github.com/bernadinm/6f68bfdd015b3f3e0a17b2f00c9ea3f8#file-all_aws_managed_policies-json-L1447-L1473) -![Policies](https://raw.githubusercontent.com/logdna/artwork/master/logdna-s3/permissions.png) -5. It is recommended to set the `Timeout` to 30 seconds if the function is going to be used to stream the logs from `gzipped` files. You may change `Memory (MB)` limit as well depending on how heavy those files are going to be - -### Configure your AWS S3 Bucket -You have the option of connecting your Amazon S3 Bucket within the S3 Lambda function console or in your S3 console. - -### In the S3 Lambda Function -1. Add S3 as a Trigger and click it to Configure: -![Configure](https://raw.githubusercontent.com/logdna/artwork/master/logdna-s3/designer.png) -2. Select the `Bucket` and `Event type` to stream the logs from to LogDNA - a. *Optional* You can also specify `Prefix` and `Suffix` for the files to capture -3. Make sure you check `Enable trigger` box: -![Trigger](https://raw.githubusercontent.com/logdna/artwork/master/logdna-s3/trigger.png) -4. Repeat steps 1-3 to add multiple buckets - -### Optional Environment Variables -The following variables can be used to tune this S3 Lambda function for specific use cases. - -* **LOGDNA_BATCH_INTERVAL**: How frequently (in `milliseconds`) to flush the batch of logs, *Optional* - * **Default**: 50 -* **LOGDNA_BATCH_LIMIT**: The maximum number of logs in one batch, *Optional* - * **Default**: 25 -* **LOGDNA_FREE_SOCKET_TIMEOUT**: How long (in `milliseconds`) to wait for inactivity before timing out on the free socket, *Optional* - * **Default**: 300000 - * **Source**: [agentkeepalive#agentoptions](https://github.com/node-modules/agentkeepalive/blob/master/README.md#new-agentoptions) -* **LOGDNA_MAX_LINE_LENGTH**: The maximum character length for each line, *Optional* - * **Default**: 32000 -* **LOGDNA_MAX_REQUEST_RETRIES**: The maximum number of retries for sending a line when there are network failures, *Optional* - * **Default**: 5 -* **LOGDNA_MAX_REQUEST_TIMEOUT**: Time limit (in `seconds`) for requests made by this HTTP Client, *Optional* - * **Default**: 300 -* **LOGDNA_REQUEST_RETRY_INTERVAL**: How frequently (in `milliseconds`) to retry for sending a line when there are network failures, *Optional* - * **Default**: 100 - -### Notes -* This S3 Lambda function and S3 Bucket should be in the same availability zone -* You can specify a bucket only in one trigger and/or S3 Lambda function since a bucket accepts only one subscription -* `Node.js.10.x` is the minimum runtime requirement for successfully running this S3 Lambda function + +#### Environment Variables +Set `INGESTION_KEY` variable to your LogDNA ingestion key. + +**Notes**: + * For more information about required and optional environment variables, please, check out [this](./doc/env.md) + +#### Monitoring +Enabling monitoring means forwarding the metrics and logs about the execution of the S3 Lambda function to `CloudWatch`. You can also use [`logdna-cloudwatch`](github.com/logdna/logdna-cloudwatch) to monitor the performance of this S3 Lambda function. + +### Test +You can test the configuration and code package using the following test input containing the bucket name and the path to the file you want to test the lambda on: +```json +{ + "Records": [ + { + "s3": { + "bucket": { + "name": "" + }, + "object": { + "key": "" + } + } + } + ] +} +``` + +## Migration Guide +If you have been using `v1` and want to switch to using the newer versions, check out [this guide](./doc/v2.md) + +## License +Copyright © [LogDNA](https://logdna.com), released under an MIT license. See the [LICENSE](./LICENSE) file and https://opensource.org/licenses/MIT ## Contributing -Contributions are always welcome. See the [contributing guide](/CONTRIBUTING.md) to learn how you can help. Build instructions for the agent are also in the guide. +Contributions are always welcome. See the [contributing guide](/CONTRIBUTING.md) to learn how you can help. + +*Happy Logging!* diff --git a/doc/v2.md b/doc/v2.md new file mode 100644 index 0000000..f7f7fc3 --- /dev/null +++ b/doc/v2.md @@ -0,0 +1,22 @@ +## Guide for Migration from v1 to v2 + +`v2` contains lots of refactoral changes to improve the performance and configuration by introducing [`@logdna/logger`](https://www.npmjs.com/package/@logdna/logger), [`@logdna/env-config`](https://www.npmjs.com/package/@logdna/env-config), and [`eslint-config-logdna`](https://www.npmjs.com/package/eslint-config-logdna). There are several changes in naming and handling environment variables as well. All environment variables are documented [here](./env.md) + +### Renamed Variables +* `LOGDNA_KEY` has been replaced with [`INGESTION_KEY`](./env.md#ingestion_key) +* `LOGDNA_HOSTNAME` has been replaced with [`HOSTNAME`](./env.md#hostname) +* `LOGDNA_TAGS` has been replaced with [`TAGS`](./env.md#tags) +* `LOGDNA_URL` has been split into multiple new variables: [`INGESTION_ENDPOINT`](./env.md#ingestion_endpoint), [`INGESTION_HOST`](./env.md#ingestion_host), [`INGESTION_PORT`](./env.md#ingestion_port), and [`SSL`](./env.md#ssl). +* `LOGDNA_BATCH_INTERVAL` has been replaced with [`FLUSH_INTERVAL`](./env.md#flush_interval) +* `LOGDNA_BATCH_LIMIT` has been replaced with [`FLUSH_LIMIT`](./env.md#flush_limit) + +### Removed Variables +* `LOGDNA_FREE_SOCKET_TIMEOUT` has been removed since it gets handled within the logger client +* `LOGDNA_MAX_LINE_LENGTH` has been removed since a line length gets handled in the server-side +* `LOGDNA_MAX_REQUEST_RETRIES` has been removed since it gets handled within the logger client +* `LOGDNA_MAX_REQUEST_TIMEOUT` has been removed since it gets handled within the logger client +* `LOGDNA_REQUEST_RETRY_INTERVAL` has been removed since it gets handled within the logger client + +### New Variables +* [`PROXY`](./env.md#proxy), [`HTTPS_PROXY`](./env.md#https_proxy), and [`HTTP_PROXY`](./env.md#http_proxy) to provide proxy support +* [`USER_AGENT`](./env.md#user_agent) to support custom `user-agent` overrides diff --git a/index.js b/index.js index 595254e..9d3375d 100644 --- a/index.js +++ b/index.js @@ -50,5 +50,5 @@ async function handler(event, context) { // Ensure logs have been flushed to LogDNA before finishing await once(logger, 'cleared') - return + return 'Done!' }