Skip to content

Commit

Permalink
Make get_targets return list if names is None (#606)
Browse files Browse the repository at this point in the history
* return get_targets returns list on names=None

* chore: update README.md to configure cert auth

* add more scrubbing in test recordings
  • Loading branch information
ArthurKamalov authored May 2, 2024
1 parent 0699fa5 commit 45329fd
Show file tree
Hide file tree
Showing 109 changed files with 29,993 additions and 14,336 deletions.
11 changes: 8 additions & 3 deletions azure-quantum/azure/quantum/target/target_factory.py
Original file line number Diff line number Diff line change
Expand Up @@ -111,8 +111,8 @@ def from_target_status(

def get_targets(
self,
name: str,
provider_id: str,
name: str = None,
provider_id: str = None,
**kwargs
) -> Union[Target, List[Target]]:
"""Create targets that are available to this workspace
Expand All @@ -129,8 +129,13 @@ def get_targets(
"""
target_statuses = self._workspace._get_target_status(name, provider_id)

# TODO: Make this function always return a list in the next major release.
if len(target_statuses) == 1:
return self.from_target_status(*target_statuses[0], **kwargs)
result = self.from_target_status(*target_statuses[0], **kwargs)
if name is None:
return [result]
else:
return result

else:
# Don't return redundant targets
Expand Down
42 changes: 26 additions & 16 deletions azure-quantum/tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,22 +10,27 @@ The 'recordings' directory is used to replay network connections.
To manually **create new recordings**, remove the 'recordings' subdirectory and run the tests.

To **force the tests to run live**, even with existing recordings, set the environment variable:

```plaintext
AZURE_TEST_RUN_LIVE="True"
```

This will force the recording files to be deleted before running the tests.

To be able to run the tests in recording or live mode, make sure to set the following environment variables:
To be able to run the tests in recording or live mode, make sure:

- You have a client app registered in Microsoft Entra ID (formerly Azure Active Directory)
- The client app is configured with [certificate-based authentication](https://learn.microsoft.com/en-us/entra/identity/authentication/how-to-certificate-based-authentication)
- The client app has "Contributor" permissions to your Azure Quantum Workspace
- The following environment variables are set:
- `AZURE_CLIENT_ID` - application (client) ID from Microsoft Entra ID
- `AZURE_TENANT_ID` - directory (tenant) ID from Microsoft Entra ID
- `AZURE_CLIENT_CERTIFICATE_PATH` - path to PEM or PKCS12 certificate file (including the private key) that is configured for the client app
- `AZURE_QUANTUM_SUBSCRIPTION_ID` - ID of the Subscription where Azure Quantum Workspace is deployed
- `AZURE_QUANTUM_WORKSPACE_RG` - name of the Resource Group where Azure Quantum Workspace is deployed
- `AZURE_QUANTUM_WORKSPACE_NAME` - name of the Azure Quantum Workspace
- `AZURE_QUANTUM_WORKSPACE_LOCATION` - Azure region where the Azure Quantum Workspace is deployed

```plaintext
AZURE_CLIENT_ID
AZURE_CLIENT_SECRET
AZURE_TENANT_ID
AZURE_QUANTUM_SUBSCRIPTION_ID
AZURE_QUANTUM_WORKSPACE_RG
AZURE_QUANTUM_WORKSPACE_NAME
AZURE_QUANTUM_WORKSPACE_LOCATION
```

## Recordings

Expand All @@ -37,24 +42,27 @@ the recordings (aka "cassetes") to playback the responses, essentially creating
When the intention is to simply playback the recordings without recording it again, sometimes the Python VCR framework may give an error "Cannot Overwrite Existing Cassette".

#### Cause

The VCR works like a HTTP proxy. It attempts to find the request by matching the full URI and HTTP headers in the recorded file. If found, it will playback the corresponding response. Otherwise, it will attempt to do a live call to the web API and will try to record the results at the end. When it tries to do a recording, if there is already a recording file, it will give the error `CannotOverwriteExistingCassetteException`.

This error could also be caused if the recorded files are manually updated and do not really match the requests that the SDK will actually request.

#### Potential solutions
1) One way to remove the error is to delete the existing recording file and let it do all the live calls and create a new recording file that contains all the requests/responses that the tests need. After that, you should be able to simple playback the recordings with no errors.

2) If the error still persist after trying (1), then probably there is something unique in the URL or HTTP headers of the HTTP request that changes every time you run the tests. In this case, we need to either make that thing constant in the tests, or if they are genuinely unique, we need to replace that unique value in the request recording pipeline such that, at least in the recording file, it will be unique.
1. One way to remove the error is to delete the existing recording file and let it do all the live calls and create a new recording file that contains all the requests/responses that the tests need. After that, you should be able to simple playback the recordings with no errors.

2. If the error still persist after trying (1), then probably there is something unique in the URL or HTTP headers of the HTTP request that changes every time you run the tests. In this case, we need to either make that thing constant in the tests, or if they are genuinely unique, we need to replace that unique value in the request recording pipeline such that, at least in the recording file, it will be unique.

For example, see the [process_request](https://github.com/microsoft/qdk-python/blob/main/azure-quantum/tests/unit/common.py) method in the [tests/unit/common.py](https://github.com/microsoft/qdk-python/blob/main/azure-quantum/tests/unit/common.py) file.
In there, we replace several Guids and resource identifiers in the URL and in the HTTP Headers to make sure that the request that will be searched for (during a playback) or recorded (during recording) will have no unique values that could cause VCR to not find the request recording and attempt to do a new live call and rewrite the recording. Another reason we replace the identifiers is to remove potentially sensitive information from the recordings (like authentication tokens, access keys, etc).

See [Sequence ids](#Sequence-ids) and [Non-deterministic Job ids and Session ids](#Non-deterministic-Job-ids-and-Session-ids) for more ideas.

### Recording sanitization

To prevent potentially sensitive information to be checked-in in the repository (like authentication tokens, access keys, etc) inside the recordings, we do several text replacements in the HTTP requests and responses in the VCR pipeline before they end-up persisted in the recorded files.

The [QuantumTestBase __init__ method](https://github.com/microsoft/qdk-python/blob/main/azure-quantum/tests/unit/common.py) contains several rules (mostly regular expressions) that are applied in the HTTP requests and reponses via several recording and playback processors that are injected in the VCR HTTP pipeline.
The [QuantumTestBase.\_\_init\_\_ method](https://github.com/microsoft/azure-quantum-python/blob/main/azure-quantum/tests/unit/common.py#L73) contains several rules (mostly regular expressions) that are applied in the HTTP requests and reponses via several recording and playback processors that are injected in the VCR HTTP pipeline.
We use some common processors provided by the Azure SDK framework (including AccessTokenReplacer, InteractiveAccessTokenReplacer, RequestUrlNormalizer) but we also apply custom text replacement logic in URLs and HTTP Headers via the `process_request` and `process_response` methods and some other processors/filters found at the end of the file.

### Ability to Pause Recordings
Expand All @@ -76,16 +84,18 @@ if self.in_recording:
```

### Sequence ids

By default, VCR does not allow the same request (identified by URI and HTTP Headers) to have multiple responses associated with it.
For example, when a job is submitted and we want to fetch the job status, the HTTP request to get the job status is the same, but the response can be different, initially returning Status="In-Progress" and later returning Status="Completed".

This limitation is solved by the test base class automatically injecting a `test-sequence-id` in the query string via the [CustomRecordingProcessor _append_sequence_id method](https://github.com/microsoft/qdk-python/blob/main/azure-quantum/tests/unit/common.py).
This limitation is solved by the test base class automatically injecting a `test-sequence-id` in the query string via the [CustomRecordingProcessor.\_append_sequence_id method](https://github.com/microsoft/azure-quantum-python/blob/main/azure-quantum/tests/unit/common.py#L458).

### Non-deterministic Job ids and Session ids

By default, the Azure Quantum Python SDK automatically creates a random guid when creating a new job or session if an id is not specified.
This non-deterministic behavior can cause problems when the test recordings are played-back and the ids in the recordings won't match the random ids created the by SDK when the tests run again.

To mitigate this problem, the `CustomRecordingProcessor` automatically looks for guids (registered with the `register_guid_regex` method) in the HTTP requests and replace them with a deterministic and sequential guid via the [CustomRecordingProcessor _search_for_guids method](https://github.com/microsoft/qdk-python/blob/main/azure-quantum/tests/unit/common.py) and the same mechanism to sanitize the recordings (regex replacement of URI, Headers and Body).
To mitigate this problem, the `CustomRecordingProcessor` automatically looks for guids (registered with the `register_guid_regex` method) in the HTTP requests and replace them with a deterministic and sequential guid via the [CustomRecordingProcessor.\_search_for_guids method](https://github.com/microsoft/azure-quantum-python/blob/main/azure-quantum/tests/unit/common.py#L438) and the same mechanism to sanitize the recordings (regex replacement of URI, Headers and Body).

## Tests

Expand All @@ -102,7 +112,7 @@ Example:
pytest ./tests/unit/test_job.py
```

To run the a specific test case, run `pytest -k [test_method_name]`.
To run the a specific test case, run `pytest -k [test_method_name]`.
Example:

```bash
Expand Down
11 changes: 11 additions & 0 deletions azure-quantum/tests/unit/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ class RegexScrubbingPatterns:
URL_QUERY_SAS_KEY_EXPIRATION = r"se=[^&]+\&"
URL_QUERY_AUTH_CLIENT_ID = r"client_id=[^&]+\&"
URL_QUERY_AUTH_CLIENT_SECRET = r"client_secret=[^&]+\&"
URL_QUERY_AUTH_CLIENT_ASSERTION = r"client_assertion=[^&]+\&"
URL_QUERY_AUTH_CLIENT_ASSERTION_TYPE = r"client_assertion_type=[^&]+\&"
URL_QUERY_AUTH_CLAIMS = r"claims=[^&]+\&"
URL_QUERY_AUTH_CODE_VERIFIER = r"code_verifier=[^&]+\&"
URL_QUERY_AUTH_CODE = r"code=[^&]+\&"
Expand Down Expand Up @@ -167,6 +169,15 @@ def __init__(self, method_name):
"code=PLACEHOLDER&")
self._regex_replacer.register_scrubbing(RegexScrubbingPatterns.URL_HTTP, "https://")

self._regex_replacer.register_scrubbing(
RegexScrubbingPatterns.URL_QUERY_AUTH_CLIENT_ASSERTION,
"client_assertion=PLACEHOLDER&"
)
self._regex_replacer.register_scrubbing(
RegexScrubbingPatterns.URL_QUERY_AUTH_CLIENT_ASSERTION_TYPE,
"client_assertion_type=PLACEHOLDER&"
)

def disable_scrubbing(self, pattern: str) -> None:
"""
Disable scrubbing of a pattern.
Expand Down
Loading

0 comments on commit 45329fd

Please sign in to comment.