Providers of Large Size: Import is very slow in Python

[brent-at-aam]

Expected Behavior

When using Python, the time taken for a synth is only correlated to the volume of resources within a stack.

Actual Behavior

When using Python, running any operations with larger providers takes a long time due to lengthy module import load times. You could have a single resource and it would still take ~30s to even get started with the synth.

Steps to Reproduce

1. Use Python
2. Add the cdktf-cdktf-provider-aws library to your project
3. Make a small stack that uses any resource from the aws provider module
4. Run a synth

Versions

Running with the latest versions

Providers

No response

Gist

No response

Possible Solutions

This is really just a general issue for all providers, but only becomes a big problem for the large ones like AWS. Looking through the issues, this is related to #2792, which was ostensibly fixed by #3030.

Perhaps this is a regression due to some newer behavior in upstream packages but importing AWS is back to taking around 30 seconds and has been for quite a while. I've switched a few projects from python to typescript to get away from it but really it would be nice to not have to do that.

From what I can tell, the bulk of the time is spent loading the very large gzipped assembly in _jsii.

The root of the module loads it:

/init.py

from ._jsii import *

And worth noting that resource modules also load it:

/foo/init.py

from .._jsii import *

I know much of this is really just behavior of other libraries, and thus this might not be something you can control here. I also realize this is something you have probably already considered, but there is no discussion about it in issues I could find. Is it possible to instruct the package generation to build separate assemblies for each of the submodules of a provider package?

And you would then remove from ._jsii import * from the root. And resources would just import their specific jsii assembly.

Workarounds

None that are feasible

Anything Else?

No response

References

• azurerm provider: Python package cdktf-cdktf-provider-azurerm extremely slow to import  #2792
• feat: remove root-level import of all modules in python #3030

Help Wanted

• I'm interested in contributing a fix myself

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[DanielMSchmidt]

I think the changes we made in 0.18 might help here already: https://developer.hashicorp.com/terraform/cdktf/release/upgrade-guide-v0-18#python-performance-improvements-disable-root-level-provider-imports

I think any other improvements would need to be made on the JSII side, so I would suggest checking if there is a similar issue here: https://github.com/aws/jsii

[brent-at-aam]

@DanielMSchmidt Yes I referenced those changes above actually. It seems to have zero effect since the jsii assembly is imported at the root level, which is what slows down the runtime.

My question here really is if the package you are building could be structured differently so that it doesn't generate one large JSII assembly but many.