ape-hardhat with local fork gives INTERNALERROR with newer pydantic version [APE-1583]

[Doc-Pixel]

Environment information

• OS: Ubuntu 22.* (new laptop, fresh install, updated to the latest versions)

• Python Version: 3.10.12

• ape and plugin versions:
  ape 0.6.26

  plugins:

  • name: infura
  • name: alchemy
  • name: vyper
  • name: ens
  • name: etherscan
  • name: hardhat

Additional related software
vyper 0.3.10
hardhat 2.19.1
ape-hardhat latest version

What went wrong?

I am trying to run my tests using my own node's RPC and fork to a local hardhat instance
ape test --network ethereum:mainnet-fork:hardhat

I'm getting pydantic errors, and it seems it is misinterpreting the package-lock.json.

================================================================================== test session starts ===================================================================================
platform linux -- Python 3.10.12, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/user/Projects/vyper/shibber
plugins: eth-ape-0.6.26, forked-1.4.0, xdist-1.34.0, hypothesis-6.27.3, web3-6.11.4
collected 6 items                                                                                                                                                                        
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/_pytest/main.py", line 271, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/_pytest/main.py", line 324, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/_pytest/main.py", line 335, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/_pytest/main.py", line 682, in perform_collect
INTERNALERROR>     hook.pytest_collection_finish(session=self)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/pluggy/_callers.py", line 130, in _multicall
INTERNALERROR>     teardown[0].send(outcome)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/ape/pytest/runners.py", line 209, in pytest_collection_finish
INTERNALERROR>     self._provider_context.push_provider()
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/ape/pytest/runners.py", line 39, in _provider_context
INTERNALERROR>     return self.network_manager.parse_network_choice(self.config_wrapper.network)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/ape/managers/networks.py", line 446, in parse_network_choice
INTERNALERROR>     provider = self.get_provider_from_choice(
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/ape/managers/networks.py", line 408, in get_provider_from_choice
INTERNALERROR>     return network.get_provider(
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/ape/api/networks.py", line 936, in get_provider
INTERNALERROR>     if provider.connection_id in ProviderContextManager.connected_providers:
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/ape/api/providers.py", line 1765, in connection_id
INTERNALERROR>     cmd_id = ",".join(self.build_command())
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/ape_hardhat/provider.py", line 1027, in build_command
INTERNALERROR>     if not self.enable_hardhat_deployments and self._has_hardhat_plugin("hardhat-deploy"):
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/ape_hardhat/provider.py", line 484, in _has_hardhat_plugin
INTERNALERROR>     return next((True for plugin in self._hardhat_plugins if plugin == plugin_name), False)
INTERNALERROR>   File "/usr/lib/python3.10/functools.py", line 981, in __get__
INTERNALERROR>     val = self.func(instance)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/ape_hardhat/provider.py", line 475, in _hardhat_plugins
INTERNALERROR>     if self._package_json.dependencies:
INTERNALERROR>   File "/usr/lib/python3.10/functools.py", line 981, in __get__
INTERNALERROR>     val = self.func(instance)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/ape_hardhat/provider.py", line 466, in _package_json
INTERNALERROR>     return PackageJson.parse_file(json_path)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/typing_extensions.py", line 2360, in wrapper
INTERNALERROR>     return arg(*args, **kwargs)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/pydantic/main.py", line 1111, in parse_file
INTERNALERROR>     return cls.parse_obj(obj)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/typing_extensions.py", line 2360, in wrapper
INTERNALERROR>     return arg(*args, **kwargs)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/pydantic/main.py", line 1027, in parse_obj
INTERNALERROR>     return cls.model_validate(obj)
INTERNALERROR>   File "/home/user/.local/lib/python3.10/site-packages/pydantic/main.py", line 503, in model_validate
INTERNALERROR>     return cls.__pydantic_validator__.validate_python(
INTERNALERROR> pydantic_core._pydantic_core.ValidationError: 4 validation errors for PackageJson
INTERNALERROR> name
INTERNALERROR>   Field required [type=missing, input_value={'devDependencies': {'hardhat': '^2.19.1'}}, input_type=dict]
INTERNALERROR>     For further information visit https://errors.pydantic.dev/2.5/v/missing
INTERNALERROR> version
INTERNALERROR>   Field required [type=missing, input_value={'devDependencies': {'hardhat': '^2.19.1'}}, input_type=dict]
INTERNALERROR>     For further information visit https://errors.pydantic.dev/2.5/v/missing
INTERNALERROR> description
INTERNALERROR>   Field required [type=missing, input_value={'devDependencies': {'hardhat': '^2.19.1'}}, input_type=dict]
INTERNALERROR>     For further information visit https://errors.pydantic.dev/2.5/v/missing
INTERNALERROR> dependencies
INTERNALERROR>   Field required [type=missing, input_value={'devDependencies': {'hardhat': '^2.19.1'}}, input_type=dict]
INTERNALERROR>     For further information visit https://errors.pydantic.dev/2.5/v/missing

================================================================================== 8 warnings in 1.79s ===================================================================================

How can it be fixed?

Downgrade pydantic. On my dev VM it is version 1.10.9

Seems like pydantic 2.5.* changed the way it handles dictionaries. Didn't downgrade to any sub versions of pydantic 2.* since my focus was on getting things running again

Fill this in if you have ideas on how the bug could be fixed.
pip3 uninstall pydantic pydantic-core
pip3 install pydantic==1.10.9 fixed it.

I think ape-hardhat needs to be updated to correctly use pydantic 2.*