Skip to content

Latest commit

 

History

History
192 lines (138 loc) · 11 KB

README.md

File metadata and controls

192 lines (138 loc) · 11 KB
Raw

process-postman-test-results

This action works in conjunction with another step that runs Postman tests. That step should include a json reporter like the following npm script: "postman": "newman run postman_collection.json -r cli,json -reporter-json-export postman-results.json".

This action takes the json file and creates a Status Check or PR Comment depending on the flags set with the test outcome. This action does not execute the tests itself, it relies on a previous step to run the Postman tests. It can only process one result file.

Index

Failures

The status check can be an item on the workflow run, a PR comment or on the PR Status Check section. If the test results contain failures, the status check will be marked as failed. Having the status check marked as failed will prevent PRs from being merged. If this status check behavior is not desired, the ignore-test-failures input can be set and the outcome will be marked as neutral if test failures are detected. The status badge shown in the comment or status check body will still indicate a failure.

Limitations

GitHub does have a size limitation of 65535 characters for a Status Check body or a PR Comment. This action will fail if the test results exceed the GitHub limit. To mitigate this size issue only failed tests are included in the output.

If you have multiple workflows triggered by the same pull_request or push event, GitHub creates one checksuite for that commit. The checksuite gets assigned to one of the workflows randomly and all status checks for that commit are reported to that checksuite. That means if there are multiple workflows with the same trigger, your status checks may show on a different workflow run than the run that created them.

Action Outputs

Pull Request Comment

This is shown on the pull request when the create-pr-comment is set to true and there is a PR associated with the commit.

Pull Request Status Check

This is shown on the pull request when the create-status-check is set to true and there is a PR associated with the commit.

Workflow Run

This is shown on the workflow run when the create-status-check is set to true.

Failed Test Details

For failed test runs you can expand each failure to view more details.

Inputs

Parameter Is Required Default Description
github-token true N/A Used for the GitHub Checks API. Value is generally: secrets.GITHUB_TOKEN.
results-file true N/A The json test results file generated by the postman/newman json reporter.
report-name false Postman Test Results The desired name of the report that is shown on the PR Comment and inside the Status Check.
create-status-check false true Flag indicating whether a status check with postman test results should be generated.
create-pr-comment false true Flag indicating whether a PR comment with postman test results should be generated. When true the default behavior is to update an existing comment if one exists.
update-comment-if-one-exists false true When create-pr-comment is true, this flag determines whether a new comment is created or if the action updates an existing comment if one is found which is the default behavior.
ignore-test-failures false false When set to true the check status is set to Neutral when there are test failures and it will not block pull requests.
timezone false UTC IANA time zone name (e.g. America/Denver) to display dates in.

Outputs

Output Description
test-outcome Test outcome based on presence of failing tests: Failed,Passed
If exceptions are thrown or if it exits early because of argument errors, this is set to Failed.

Usage Examples

npm setup

  1. Some npm scripts have also been added to the project

    "scripts": {
  "postman": "newman run postman_collection.json -r cli,json -reporter-json-export postman-results.json"
}

Workflow

env:
  PACKAGE_JSON_DIR: 'src'                       # The directory containing package.json
  POSTMAN_NPM_SCRIPT_NAME: 'postman'            # The name of the postman script in package.json
  POSTMAN_RESULTS_NAME: 'postman-results.json'  # The name of the results file set in npm script in package.json

jobs:
  run-postman:
    runs-on: ubuntu-20.04

    defaults:
      run:
        shell: bash
        working-directory: ${{ env.PACKAGE_JSON_DIR }}

    steps:
      - uses: actions/checkout@v3

      - name: Restore npm packages
        run: npm install

      - name: Run Postman Tests
        continue-on-error: true
        run: npm run ${{ env.POSTMAN_NPM_SCRIPT_NAME }}

      - name: Create Status check based on postman results
        id: process-postman
        # You may also reference the major or major.minor version
        uses: im-open/process-postman-test-results@v2.1.5
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          results-file: ${{env.PACKAGE_JSON_DIR }}/${{ env.POSTMAN_RESULTS_NAME }}
          report-name: 'Postman ${{ github.run_number }}'     # Default: Postman Test Results
          create-status-check: true                           # Default: true
          create-pr-comment: false                            # Default: true
          update-comment-if-one-exists: false                 # Default: true
          ignore-test-failures: true                          # Default: false
          timezone: 'america/denver'                          # Default: UTC
      
      - name: Fail if there were errors in the postman tests
        if: steps.process-postman.outputs.test-outcome == 'Failed'
        run: |
          echo "There were postman failures."
          exit 1

Contributing

When creating new PRs please ensure:

  1. For major or minor changes, at least one of the commit messages contains the appropriate +semver: keywords listed under Incrementing the Version.
  2. The action code does not contain sensitive information.

When a pull request is created and there are changes to code-specific files and folders, the build workflow will run and it will recompile the action and push a commit to the branch if the PR author has not done so. The usage examples in the README.md will also be updated with the next version if they have not been updated manually. The following files and folders contain action code and will trigger the automatic updates:

  • action.yml
  • package.json
  • package-lock.json
  • src/**
  • dist/**

There may be some instances where the bot does not have permission to push changes back to the branch though so these steps should be done manually for those branches. See Recompiling Manually and Incrementing the Version for more details.

Recompiling Manually

If changes are made to the action's code in this repository, or its dependencies, the action can be re-compiled by running the following command:

# Installs dependencies and bundles the code
npm run build

# Bundle the code (if dependencies are already installed)
npm run bundle

These commands utilize esbuild to bundle the action and its dependencies into a single file located in the dist folder.

Incrementing the Version

Both the build and PR merge workflows will use the strategies below to determine what the next version will be. If the build workflow was not able to automatically update the README.md action examples with the next version, the README.md should be updated manually as part of the PR using that calculated version.

This action uses git-version-lite to examine commit messages to determine whether to perform a major, minor or patch increment on merge. The following table provides the fragment that should be included in a commit message to active different increment strategies.

Increment Type Commit Message Fragment
major +semver:breaking
major +semver:major
minor +semver:feature
minor +semver:minor
patch default increment type, no comment needed

Code of Conduct

This project has adopted the im-open's Code of Conduct.

License

Copyright © 2021, Extend Health, LLC. Code released under the MIT license.