[Fix] update bash scripts for E2E tests run to propagate failures

[jstourac]

We haven't been propagating the test failures in our E2E CI jobs because we weren't checking for the return codes of the commands run in the bash scripts.

This depends on #432!

I bumped into this during my work on https://issues.redhat.com/browse/RHOAIENG-14687.

This fix is part of https://issues.redhat.com/browse/RHOAIENG-15141.

Description

How Has This Been Tested?

Try to run in components/odh-notebook-controller:

• make run-ci-e2e-tests - 15 pass, 2 fail - you need to be logged into some cluster to make this work
• since 2 tests are failing, see that now the execution itself fails with return code 2 - before this change the return code was just 0 as a success, which was wrong of course

Merge criteria:

• The commits are squashed in a cohesive manner and have meaningful messages.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
• The developer has manually tested the changes and verified that the changes work

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from jstourac. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jiridanek]

Here's my version of the logic in bash, with made-up failing commands

# https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/
set -Eeuxo pipefail

echo "do setup, fail fast"
echo "done setup"

errorcnt=0
function error {
  echo "handling error"
  errorcnt=$(( ++errorcnt ))
}
trap error ERR

function cleanup {
  set +Ee

  echo "doing cleanup"
  nosuchcommand1
  nosuchcommand2
  echo "done cleanup"

  exit ${errorcnt}
}
trap cleanup EXIT

echo "running tests"
nosuchtestcommand1
nosuchtestcommand2

[openshift-ci]

@jstourac: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/odh-notebook-controller-e2e	23fb875	link	unknown	/test odh-notebook-controller-e2e

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.