feat: initial version of Hardhat network provider (#1)

What I did:

* Updated the project template boilerplate, README, etc.
* Fairly straightforward subclassing of the existing HTTP provider
  (`EthereumProvider`). All the inherited web3 calls worked without any
  major changes.
* Added subprocess management code to run `npx hardhat node` in the
  background.
* Added steps in the CI job to install Node, NPM, and Hardhat.
* Added more subprocess management code, configurable retry strategy,
  and signal handling to run multiple hardhats under one parent process
  with randomized port numbers and protect against orphaned
  subprocesses. This was needed to get pytest+xdist working across
  multiple processors for faster test suite runs.
* Wrote a bunch of tests.
* Released Ape 0.1.0-alpha.22 with a workaround for an issue I ran into
  when subclassing `NetworkConfig`.
* Added support & tests for Hardhat's custom RPC methods (sleep, mine,
  snapshot, revert, impersonate, etc.).

Test plan:

See `tests/` and successful test runs in the PR: #1

Squashed commits:

* chore: update project template boilerplate
* feat: initial working version of hardhat network provider
* test: add steps in CI to set up node, npm, and hardhat
* chore: remove python 3.6 support to match eth-ape
* feat: add support for custom Hardhat RPC methods
* chore: linting and type checking
* chore: bump eth-ape required version to pick up a change we need
* chore: replace web3 make_request with request_blocking
* test: remove second connect() call in test_connect
* feat: support for running multiple hardhat processes under a single Ape process
    1. Randomize the hardhat port number when a port isn't specified.
    2. Use a per-instance web3 client instead of a shared class variable.
    3. Improved and configurable retry strategy for starting up subprocesses
       and network communication.
    4. Use the configurable retry strategy to bump up timeouts & attempts in
       the test suite to increase test suite reliability.
    5. Add a test for running two hardhat processes under a single parent
       process.
    6. Improved reliability on some other flaky tests (e.g. hh.sleep).
    7. Added extra atexit, signal handler, and process management code to
       ensure that subprocesses are cleaned up when the parent process exits
       or is killed.
* chore: ignore typing of pytest
* chore: move most of the implementation to providers module
* chore: remove support for fork mode (we want to make an official API in core Ape for this)
* chore: remove duplicate plugin code and configure Hardhat base fee to be 0.
* chore: use Ape exception hierachy

.github/workflows/test.yaml

@@ -21,10 +21,10 @@ jobs:
           run: black --check .
 
         - name: Run flake8
-          run: flake8 ./ape_hardhat ./tests ./setup.py
+          run: flake8 .
 
         - name: Run isort
-          run: isort --check-only --diff ./ape_hardhat ./tests
+          run: isort --check-only .
 
     type-check:
         runs-on: ubuntu-latest
@@ -41,14 +41,15 @@ jobs:
           run: pip install .[lint]
 
         - name: Run MyPy
-          run: mypy -p ape_hardhat
+          run: mypy .
 
     functional:
-        runs-on: ubuntu-latest
+        runs-on: ${{ matrix.os }}
 
         strategy:
             matrix:
-                python-version: [3.6, 3.7, 3.8]
+                python-version: [3.7, 3.8, 3.9]
+                os: [ubuntu-latest, macos-latest]   # eventually add `windows-latest`
 
         steps:
         - uses: actions/checkout@v2
@@ -61,6 +62,14 @@ jobs:
         - name: Install Dependencies
           run: pip install .[test]
 
+        - name: Setup Node
+          uses: actions/setup-node@v2
+          with:
+              node-version: '14'
+
+        - name: Install Node Dependencies
+          run: npm install --save-dev hardhat
+
         - name: Run Tests
           run: pytest -m "not fuzzing"
 

.gitignore

@@ -8,6 +8,7 @@ __pycache__/
 
 # Distribution / packaging
 .Python
+.build/
 build/
 develop-eggs/
 dist/
@@ -116,3 +117,11 @@ dmypy.json
 
 # setuptools-scm
 version.py
+
+# Ape stuff
+tests/hardhat.config.js
+
+# NPM
+package.json
+package-lock.json
+node_modules/

README.rst

@@ -6,7 +6,7 @@ Hardhat network provider plugin for Ape. Hardhat is a development framework writ
 Dependencies
 ************
 
-* `python3 <https://www.python.org/downloads>`_ version 3.6 or greater, python3-dev
+* `python3 <https://www.python.org/downloads>`_ version 3.7 or greater, python3-dev
 * Node.js, NPM, and Hardhat. See Hardhat's `Installation <https://hardhat.org/getting-started/#installation>`_ documentation for steps.
 
 Installation
@@ -37,13 +37,22 @@ Quick Usage
 
 To use the plugin, first install Hardhat locally into your Ape project directory:
 
-
 .. code-block:: bash
 
     cd your-ape-project
     npm install --save-dev hardhat
 
-After that, you can use the ``--network`` command line flag, or add a ``network`` key in your ``ape-config.yaml`` file to tell Ape to use the Hardhat provider.
+After that, you can use the ``--network ethereum:development:hardhat`` command line flag to use the hardhat network (if it's not already configured as the default).
+
+This network provider takes additional Hardhat-specific configuration options. To use them, add these configs in your project's ``ape-config.yaml``:
+
+.. code-block:: yaml
+
+    hardhat:
+      port: 8555
+      ethereum:
+        development:
+          uri: http://localhost:8555
 
 Development
 ***********

ape_hardhat/__init__.py

@@ -1,12 +1,18 @@
-try:
-    from importlib.metadata import PackageNotFoundError as _PackageNotFoundError  # type: ignore
-    from importlib.metadata import version as _version  # type: ignore
-except ModuleNotFoundError:
-    from importlib_metadata import PackageNotFoundError as _PackageNotFoundError  # type: ignore
-    from importlib_metadata import version as _version  # type: ignore
-
-try:
-    __version__ = _version(__name__)
-except _PackageNotFoundError:
-    # package is not installed
-    __version__ = "<unknown>"
+"""
+Ape network provider plugin for Hardhat (Ethereum development framework and network
+implementation written in Node.js).
+"""
+
+from ape import plugins
+
+from .providers import HardhatNetworkConfig, HardhatProvider
+
+
+@plugins.register(plugins.Config)
+def config_class():
+    return HardhatNetworkConfig
+
+
+@plugins.register(plugins.ProviderPlugin)
+def providers():
+    yield "ethereum", "development", HardhatProvider

ape_hardhat/providers.py

@@ -0,0 +1,223 @@
+"""
+Implementation for HardhatProvider.
+"""
+
+import atexit
+import ctypes
+import platform
+import random
+import signal
+import sys
+import time
+from subprocess import PIPE, Popen, call
+from typing import Any, List, Optional
+
+from ape.exceptions import ProviderError
+from ape_http.providers import DEFAULT_SETTINGS, EthereumProvider, NetworkConfig
+
+EPHEMERAL_PORTS_START = 49152
+EPHEMERAL_PORTS_END = 60999
+HARDHAT_CHAIN_ID = 31337
+HARDHAT_CONFIG = """
+// See https://hardhat.org/config/ for config options.
+module.exports = {
+  networks: {
+    hardhat: {
+      hardfork: "london",
+      // base fee of 0 allows use of 0 gas price when testing
+      initialBaseFeePerGas: 0,
+    },
+  },
+};
+"""
+HARDHAT_START_NETWORK_RETRIES = [0.1, 0.2, 0.3, 0.5, 1.0]  # seconds between network retries
+HARDHAT_START_PROCESS_ATTEMPTS = 3  # number of attempts to start subprocess before giving up
+
+
+def _signal_handler(signum, frame):
+    """Runs on SIGTERM and SIGINT to force ``atexit`` handlers to run."""
+    atexit._run_exitfuncs()
+    sys.exit(143 if signum == signal.SIGTERM else 130)
+
+
+def _set_death_signal():
+    """Automatically sends SIGTERM to child subprocesses when parent process dies."""
+    if platform.uname().system == "Linux":
+        # from: https://stackoverflow.com/a/43152455/75956
+        # the first argument "1" is PR_SET_PDEATHSIG
+        # the second argument is what signal to send the child
+        libc = ctypes.CDLL("libc.so.6")
+        return libc.prctl(1, signal.SIGTERM)
+
+
+class HardhatSubprocessError(ProviderError):
+    pass
+
+
+class HardhatNetworkConfig(NetworkConfig):
+    # --port <INT, default from Hardhat is 8545, but our default is to assign a random port number>
+    port: Optional[int] = None
+
+    # retry strategy configs, try increasing these if you're getting HardhatSubprocessError
+    network_retries: List[float] = HARDHAT_START_NETWORK_RETRIES
+    process_attempts: int = HARDHAT_START_PROCESS_ATTEMPTS
+
+
+class HardhatProvider(EthereumProvider):
+    def __post_init__(self):
+        self._hardhat_web3 = (
+            None  # we need to maintain a separate per-instance web3 client for Hardhat
+        )
+        self.port = None
+        self.process = None
+
+        hardhat_config_file = self.network.config_manager.PROJECT_FOLDER / "hardhat.config.js"
+
+        if not hardhat_config_file.is_file():
+            hardhat_config_file.write_text(HARDHAT_CONFIG)
+
+        if call(["npx", "--version"], stderr=PIPE, stdout=PIPE, stdin=PIPE) != 0:
+            raise HardhatSubprocessError(
+                "Missing npx binary. See ape-hardhat README for install steps."
+            )
+        if call(["npx", "hardhat", "--version"], stderr=PIPE, stdout=PIPE, stdin=PIPE) != 0:
+            raise HardhatSubprocessError(
+                "Missing hardhat NPM package. See ape-hardhat README for install steps."
+            )
+
+        # register atexit handler to make sure disconnect is called for normal object lifecycle
+        atexit.register(self.disconnect)
+
+        # register signal handlers to make sure atexit handlers are called when the parent python
+        # process is killed
+        signal.signal(signal.SIGINT, _signal_handler)
+        signal.signal(signal.SIGTERM, _signal_handler)
+
+    def _start_process(self):
+        """Start the hardhat process and wait for it to respond over the network."""
+        # handle configs
+        cmd = ["npx", "hardhat", "node"]
+
+        # pick a random port if one isn't configured
+        self.port = self.config.port
+        if not self.port:
+            self.port = random.randint(EPHEMERAL_PORTS_START, EPHEMERAL_PORTS_END)
+        cmd.extend(["--port", str(self.port)])
+
+        # TODO: Add configs to send stdout to logger / redirect to a file in plugin data dir?
+        process = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE, preexec_fn=_set_death_signal)
+        connected = False
+        for retry_time in self.config.network_retries:
+            time.sleep(retry_time)
+            super().connect()
+            connected = self._verify_connection()
+            if connected:
+                break
+        if not connected:
+            if process.poll() is None:
+                raise HardhatSubprocessError(
+                    "Hardhat process is running, but could not connect to RPC server. "
+                    "Run `npx hardhat node` or adjust retry strategy configs to troubleshoot."
+                )
+            raise HardhatSubprocessError(
+                "Hardhat command exited prematurely. Run `npx hardhat node` to troubleshoot or "
+                "adjust retry strategy configs to troubleshoot."
+            )
+        elif process.poll() is not None:
+            raise HardhatSubprocessError(
+                "Hardhat process exited prematurely, but connection succeeded. "
+                "Is something else listening on the port?"
+            )
+        self.process = process
+
+    def _verify_connection(self):
+        """Make a network call for chain_id and verify the result."""
+        try:
+            chain_id = self.chain_id
+            if chain_id != HARDHAT_CHAIN_ID:
+                raise AssertionError(f"Unexpected chain ID: {chain_id}")
+            return True
+        except Exception as exc:
+            print("Hardhat connection failed:", exc)
+        return False
+
+    def connect(self):
+        """Start the hardhat process and verify it's up and accepting connections."""
+
+        if self.process:
+            raise RuntimeError("Cannot connect twice. Call disconnect before connecting again.")
+
+        if self.config.port:
+            # if a port is configured, only make one start up attempt
+            self._start_process()
+        else:
+            for _ in range(self.config.process_attempts):
+                try:
+                    self._start_process()
+                    break
+                except HardhatSubprocessError as exc:
+                    print("Retrying hardhat subprocess startup:", exc)
+
+        # subprocess should be running and receiving network requests at this point
+        if not (self.process and self.process.poll() is None and self.port):
+            raise HardhatSubprocessError(
+                "Could not start hardhat subprocess on a random port. "
+                "See logs or run `npx hardhat node` or adjust retry strategy configs to "
+                "troubleshoot."
+            )
+
+    @property
+    def uri(self) -> str:
+        uri = getattr(self.config, self.network.ecosystem.name)[self.network.name]["uri"]
+        if uri != DEFAULT_SETTINGS["uri"]:
+            # the user configured their own URI in the project configs, let's use that
+            return uri
+        elif self.port:
+            # the user did not override the default URI, and we have a port
+            # number, so let's build the URI using that port number
+            return f"http://localhost:{self.port}"
+        else:
+            raise RuntimeError("Can't build URI before `connect` is called.")
+
+    @property  # type: ignore
+    def _web3(self):
+        """
+        This property overrides the ``EthereumProvider._web3`` class variable to return our
+        instance variable.
+        """
+        return self._hardhat_web3
+
+    @_web3.setter
+    def _web3(self, value):
+        """
+        Redirect the base class's assignments of self._web3 class variable to our instance variable.
+        """
+        self._hardhat_web3 = value
+
+    def disconnect(self):
+        super().disconnect()
+        if self.process:
+            self.process.kill()
+        self.process = None
+        self.port = None
+
+    def _make_request(self, rpc: str, args: list) -> Any:
+        return self._web3.manager.request_blocking(rpc, args)  # type: ignore
+
+    def set_block_gas_limit(self, gas_limit: int) -> bool:
+        return self._make_request("evm_setBlockGasLimit", [hex(gas_limit)])
+
+    def sleep(self, seconds: int) -> int:
+        return int(self._make_request("evm_increaseTime", [seconds]))
+
+    def mine(self, timestamp: Optional[int] = None) -> str:
+        return self._make_request("evm_mine", [timestamp] if timestamp else [])
+
+    def snapshot(self) -> int:
+        return self._make_request("evm_snapshot", [])
+
+    def revert(self, snapshot_id: int) -> bool:
+        return self._make_request("evm_revert", [snapshot_id])
+
+    def unlock_account(self, address: str) -> bool:
+        return self._make_request("hardhat_impersonateAccount", [address])

hardhat.config.js

@@ -0,0 +1,11 @@
+
+// See https://hardhat.org/config/ for config options.
+module.exports = {
+  networks: {
+    hardhat: {
+      hardfork: "london",
+      // base fee of 0 allows use of 0 gas price when testing
+      initialBaseFeePerGas: 0,
+    },
+  },
+};

pytest.ini

@@ -0,0 +1,4 @@
+[pytest]
+# if you're experiencing problems with running multiple hardhat processes in
+# parallel, change this line to "-n1" to restrict the test suite to a single process
+addopts = -nauto

setup.py

@@ -66,9 +66,10 @@
     url="https://github.com/ApeWorX/ape-hardhat",
     include_package_data=True,
     install_requires=[
+        "eth-ape==0.1.0a23",
         "importlib-metadata ; python_version<'3.8'",
     ],  # NOTE: Add 3rd party libraries here
-    python_requires=">=3.6,<4",
+    python_requires=">=3.7,<4",
     extras_require=extras_require,
     py_modules=["ape_hardhat"],
     license="Apache-2.0",
@@ -84,7 +85,6 @@
         "Operating System :: MacOS",
         "Operating System :: POSIX",
         "Programming Language :: Python :: 3",
-        "Programming Language :: Python :: 3.6",
         "Programming Language :: Python :: 3.7",
         "Programming Language :: Python :: 3.8",
         "Programming Language :: Python :: 3.9",