generated from ApeWorX/project-template
-
-
Notifications
You must be signed in to change notification settings - Fork 15
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: initial version of Hardhat network provider (#1)
What I did: * Updated the project template boilerplate, README, etc. * Fairly straightforward subclassing of the existing HTTP provider (`EthereumProvider`). All the inherited web3 calls worked without any major changes. * Added subprocess management code to run `npx hardhat node` in the background. * Added steps in the CI job to install Node, NPM, and Hardhat. * Added more subprocess management code, configurable retry strategy, and signal handling to run multiple hardhats under one parent process with randomized port numbers and protect against orphaned subprocesses. This was needed to get pytest+xdist working across multiple processors for faster test suite runs. * Wrote a bunch of tests. * Released Ape 0.1.0-alpha.22 with a workaround for an issue I ran into when subclassing `NetworkConfig`. * Added support & tests for Hardhat's custom RPC methods (sleep, mine, snapshot, revert, impersonate, etc.). Test plan: See `tests/` and successful test runs in the PR: #1 Squashed commits: * chore: update project template boilerplate * feat: initial working version of hardhat network provider * test: add steps in CI to set up node, npm, and hardhat * chore: remove python 3.6 support to match eth-ape * feat: add support for custom Hardhat RPC methods * chore: linting and type checking * chore: bump eth-ape required version to pick up a change we need * chore: replace web3 make_request with request_blocking * test: remove second connect() call in test_connect * feat: support for running multiple hardhat processes under a single Ape process 1. Randomize the hardhat port number when a port isn't specified. 2. Use a per-instance web3 client instead of a shared class variable. 3. Improved and configurable retry strategy for starting up subprocesses and network communication. 4. Use the configurable retry strategy to bump up timeouts & attempts in the test suite to increase test suite reliability. 5. Add a test for running two hardhat processes under a single parent process. 6. Improved reliability on some other flaky tests (e.g. hh.sleep). 7. Added extra atexit, signal handler, and process management code to ensure that subprocesses are cleaned up when the parent process exits or is killed. * chore: ignore typing of pytest * chore: move most of the implementation to providers module * chore: remove support for fork mode (we want to make an official API in core Ape for this) * chore: remove duplicate plugin code and configure Hardhat base fee to be 0. * chore: use Ape exception hierachy
- Loading branch information
1 parent
eccd943
commit 7f10b8d
Showing
10 changed files
with
487 additions
and
22 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,12 +1,18 @@ | ||
try: | ||
from importlib.metadata import PackageNotFoundError as _PackageNotFoundError # type: ignore | ||
from importlib.metadata import version as _version # type: ignore | ||
except ModuleNotFoundError: | ||
from importlib_metadata import PackageNotFoundError as _PackageNotFoundError # type: ignore | ||
from importlib_metadata import version as _version # type: ignore | ||
|
||
try: | ||
__version__ = _version(__name__) | ||
except _PackageNotFoundError: | ||
# package is not installed | ||
__version__ = "<unknown>" | ||
""" | ||
Ape network provider plugin for Hardhat (Ethereum development framework and network | ||
implementation written in Node.js). | ||
""" | ||
|
||
from ape import plugins | ||
|
||
from .providers import HardhatNetworkConfig, HardhatProvider | ||
|
||
|
||
@plugins.register(plugins.Config) | ||
def config_class(): | ||
return HardhatNetworkConfig | ||
|
||
|
||
@plugins.register(plugins.ProviderPlugin) | ||
def providers(): | ||
yield "ethereum", "development", HardhatProvider |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,223 @@ | ||
""" | ||
Implementation for HardhatProvider. | ||
""" | ||
|
||
import atexit | ||
import ctypes | ||
import platform | ||
import random | ||
import signal | ||
import sys | ||
import time | ||
from subprocess import PIPE, Popen, call | ||
from typing import Any, List, Optional | ||
|
||
from ape.exceptions import ProviderError | ||
from ape_http.providers import DEFAULT_SETTINGS, EthereumProvider, NetworkConfig | ||
|
||
EPHEMERAL_PORTS_START = 49152 | ||
EPHEMERAL_PORTS_END = 60999 | ||
HARDHAT_CHAIN_ID = 31337 | ||
HARDHAT_CONFIG = """ | ||
// See https://hardhat.org/config/ for config options. | ||
module.exports = { | ||
networks: { | ||
hardhat: { | ||
hardfork: "london", | ||
// base fee of 0 allows use of 0 gas price when testing | ||
initialBaseFeePerGas: 0, | ||
}, | ||
}, | ||
}; | ||
""" | ||
HARDHAT_START_NETWORK_RETRIES = [0.1, 0.2, 0.3, 0.5, 1.0] # seconds between network retries | ||
HARDHAT_START_PROCESS_ATTEMPTS = 3 # number of attempts to start subprocess before giving up | ||
|
||
|
||
def _signal_handler(signum, frame): | ||
"""Runs on SIGTERM and SIGINT to force ``atexit`` handlers to run.""" | ||
atexit._run_exitfuncs() | ||
sys.exit(143 if signum == signal.SIGTERM else 130) | ||
|
||
|
||
def _set_death_signal(): | ||
"""Automatically sends SIGTERM to child subprocesses when parent process dies.""" | ||
if platform.uname().system == "Linux": | ||
# from: https://stackoverflow.com/a/43152455/75956 | ||
# the first argument "1" is PR_SET_PDEATHSIG | ||
# the second argument is what signal to send the child | ||
libc = ctypes.CDLL("libc.so.6") | ||
return libc.prctl(1, signal.SIGTERM) | ||
|
||
|
||
class HardhatSubprocessError(ProviderError): | ||
pass | ||
|
||
|
||
class HardhatNetworkConfig(NetworkConfig): | ||
# --port <INT, default from Hardhat is 8545, but our default is to assign a random port number> | ||
port: Optional[int] = None | ||
|
||
# retry strategy configs, try increasing these if you're getting HardhatSubprocessError | ||
network_retries: List[float] = HARDHAT_START_NETWORK_RETRIES | ||
process_attempts: int = HARDHAT_START_PROCESS_ATTEMPTS | ||
|
||
|
||
class HardhatProvider(EthereumProvider): | ||
def __post_init__(self): | ||
self._hardhat_web3 = ( | ||
None # we need to maintain a separate per-instance web3 client for Hardhat | ||
) | ||
self.port = None | ||
self.process = None | ||
|
||
hardhat_config_file = self.network.config_manager.PROJECT_FOLDER / "hardhat.config.js" | ||
|
||
if not hardhat_config_file.is_file(): | ||
hardhat_config_file.write_text(HARDHAT_CONFIG) | ||
|
||
if call(["npx", "--version"], stderr=PIPE, stdout=PIPE, stdin=PIPE) != 0: | ||
raise HardhatSubprocessError( | ||
"Missing npx binary. See ape-hardhat README for install steps." | ||
) | ||
if call(["npx", "hardhat", "--version"], stderr=PIPE, stdout=PIPE, stdin=PIPE) != 0: | ||
raise HardhatSubprocessError( | ||
"Missing hardhat NPM package. See ape-hardhat README for install steps." | ||
) | ||
|
||
# register atexit handler to make sure disconnect is called for normal object lifecycle | ||
atexit.register(self.disconnect) | ||
|
||
# register signal handlers to make sure atexit handlers are called when the parent python | ||
# process is killed | ||
signal.signal(signal.SIGINT, _signal_handler) | ||
signal.signal(signal.SIGTERM, _signal_handler) | ||
|
||
def _start_process(self): | ||
"""Start the hardhat process and wait for it to respond over the network.""" | ||
# handle configs | ||
cmd = ["npx", "hardhat", "node"] | ||
|
||
# pick a random port if one isn't configured | ||
self.port = self.config.port | ||
if not self.port: | ||
self.port = random.randint(EPHEMERAL_PORTS_START, EPHEMERAL_PORTS_END) | ||
cmd.extend(["--port", str(self.port)]) | ||
|
||
# TODO: Add configs to send stdout to logger / redirect to a file in plugin data dir? | ||
process = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE, preexec_fn=_set_death_signal) | ||
connected = False | ||
for retry_time in self.config.network_retries: | ||
time.sleep(retry_time) | ||
super().connect() | ||
connected = self._verify_connection() | ||
if connected: | ||
break | ||
if not connected: | ||
if process.poll() is None: | ||
raise HardhatSubprocessError( | ||
"Hardhat process is running, but could not connect to RPC server. " | ||
"Run `npx hardhat node` or adjust retry strategy configs to troubleshoot." | ||
) | ||
raise HardhatSubprocessError( | ||
"Hardhat command exited prematurely. Run `npx hardhat node` to troubleshoot or " | ||
"adjust retry strategy configs to troubleshoot." | ||
) | ||
elif process.poll() is not None: | ||
raise HardhatSubprocessError( | ||
"Hardhat process exited prematurely, but connection succeeded. " | ||
"Is something else listening on the port?" | ||
) | ||
self.process = process | ||
|
||
def _verify_connection(self): | ||
"""Make a network call for chain_id and verify the result.""" | ||
try: | ||
chain_id = self.chain_id | ||
if chain_id != HARDHAT_CHAIN_ID: | ||
raise AssertionError(f"Unexpected chain ID: {chain_id}") | ||
return True | ||
except Exception as exc: | ||
print("Hardhat connection failed:", exc) | ||
return False | ||
|
||
def connect(self): | ||
"""Start the hardhat process and verify it's up and accepting connections.""" | ||
|
||
if self.process: | ||
raise RuntimeError("Cannot connect twice. Call disconnect before connecting again.") | ||
|
||
if self.config.port: | ||
# if a port is configured, only make one start up attempt | ||
self._start_process() | ||
else: | ||
for _ in range(self.config.process_attempts): | ||
try: | ||
self._start_process() | ||
break | ||
except HardhatSubprocessError as exc: | ||
print("Retrying hardhat subprocess startup:", exc) | ||
|
||
# subprocess should be running and receiving network requests at this point | ||
if not (self.process and self.process.poll() is None and self.port): | ||
raise HardhatSubprocessError( | ||
"Could not start hardhat subprocess on a random port. " | ||
"See logs or run `npx hardhat node` or adjust retry strategy configs to " | ||
"troubleshoot." | ||
) | ||
|
||
@property | ||
def uri(self) -> str: | ||
uri = getattr(self.config, self.network.ecosystem.name)[self.network.name]["uri"] | ||
if uri != DEFAULT_SETTINGS["uri"]: | ||
# the user configured their own URI in the project configs, let's use that | ||
return uri | ||
elif self.port: | ||
# the user did not override the default URI, and we have a port | ||
# number, so let's build the URI using that port number | ||
return f"http://localhost:{self.port}" | ||
else: | ||
raise RuntimeError("Can't build URI before `connect` is called.") | ||
|
||
@property # type: ignore | ||
def _web3(self): | ||
""" | ||
This property overrides the ``EthereumProvider._web3`` class variable to return our | ||
instance variable. | ||
""" | ||
return self._hardhat_web3 | ||
|
||
@_web3.setter | ||
def _web3(self, value): | ||
""" | ||
Redirect the base class's assignments of self._web3 class variable to our instance variable. | ||
""" | ||
self._hardhat_web3 = value | ||
|
||
def disconnect(self): | ||
super().disconnect() | ||
if self.process: | ||
self.process.kill() | ||
self.process = None | ||
self.port = None | ||
|
||
def _make_request(self, rpc: str, args: list) -> Any: | ||
return self._web3.manager.request_blocking(rpc, args) # type: ignore | ||
|
||
def set_block_gas_limit(self, gas_limit: int) -> bool: | ||
return self._make_request("evm_setBlockGasLimit", [hex(gas_limit)]) | ||
|
||
def sleep(self, seconds: int) -> int: | ||
return int(self._make_request("evm_increaseTime", [seconds])) | ||
|
||
def mine(self, timestamp: Optional[int] = None) -> str: | ||
return self._make_request("evm_mine", [timestamp] if timestamp else []) | ||
|
||
def snapshot(self) -> int: | ||
return self._make_request("evm_snapshot", []) | ||
|
||
def revert(self, snapshot_id: int) -> bool: | ||
return self._make_request("evm_revert", [snapshot_id]) | ||
|
||
def unlock_account(self, address: str) -> bool: | ||
return self._make_request("hardhat_impersonateAccount", [address]) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
|
||
// See https://hardhat.org/config/ for config options. | ||
module.exports = { | ||
networks: { | ||
hardhat: { | ||
hardfork: "london", | ||
// base fee of 0 allows use of 0 gas price when testing | ||
initialBaseFeePerGas: 0, | ||
}, | ||
}, | ||
}; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
[pytest] | ||
# if you're experiencing problems with running multiple hardhat processes in | ||
# parallel, change this line to "-n1" to restrict the test suite to a single process | ||
addopts = -nauto |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.