[Question] Accessing and Modifying Actor Scales During Simulation

[hesic73]

I am looking to adjust the scales of actors when an environment is reset. I have identified where the properties for entities in subscenes are specified, which is in sapien.wrapper.actor_builder.ActorBuilder.build_physx_component:

def build_physx_component(self, link_parent=None):
        for r in self.collision_records:
            assert isinstance(r.material, sapien.physx.PhysxMaterial)

        if self.physx_body_type == "dynamic":
            component = sapien.physx.PhysxRigidDynamicComponent()
        elif self.physx_body_type == "kinematic":
            component = sapien.physx.PhysxRigidDynamicComponent()
            component.kinematic = True
        elif self.physx_body_type == "static":
            component = sapien.physx.PhysxRigidStaticComponent()
        elif self.physx_body_type == "link":
            component = sapien.physx.PhysxArticulationLinkComponent(link_parent)
        else:
            raise Exception(f"invalid physx body type [{self.physx_body_type}]")

        for r in self.collision_records:
            try:
                if r.type == "plane":
                    shape = sapien.physx.PhysxCollisionShapePlane(
                        material=r.material,
                    )
                    shapes = [shape]
                elif r.type == "box":
                    shape = sapien.physx.PhysxCollisionShapeBox(
                        half_size=r.scale, material=r.material
                    )
                    shapes = [shape]
                elif r.type == "capsule":
                    shape = sapien.physx.PhysxCollisionShapeCapsule(
                        radius=r.radius,
                        half_length=r.length,
                        material=r.material,
                    )
                    shapes = [shape]
                elif r.type == "cylinder":
                    shape = sapien.physx.PhysxCollisionShapeCylinder(
                        radius=r.radius,
                        half_length=r.length,
                        material=r.material,
                    )
                    shapes = [shape]
                elif r.type == "sphere":
                    shape = sapien.physx.PhysxCollisionShapeSphere(
                        radius=r.radius,
                        material=r.material,
                    )
                    shapes = [shape]
                elif r.type == "convex_mesh":
                    shape = sapien.physx.PhysxCollisionShapeConvexMesh(
                        filename=preprocess_mesh_file(r.filename),
                        scale=r.scale,
                        material=r.material,
                    )
                    shapes = [shape]
                elif r.type == "nonconvex_mesh":
                    shape = sapien.physx.PhysxCollisionShapeTriangleMesh(
                        filename=preprocess_mesh_file(r.filename),
                        scale=r.scale,
                        material=r.material,
                    )
                    shapes = [shape]
                elif r.type == "multiple_convex_meshes":
                    if r.decomposition == "coacd":
                        params = r.decomposition_params
                        if params is None:
                            params = dict()

                        filename = do_coacd(preprocess_mesh_file(r.filename), **params)
                    else:
                        filename = preprocess_mesh_file(r.filename)

                    shapes = sapien.physx.PhysxCollisionShapeConvexMesh.load_multiple(
                        filename=filename,
                        scale=r.scale,
                        material=r.material,
                    )
                else:
                    raise RuntimeError(f"invalid collision shape type [{r.type}]")
            except RuntimeError as e:
                # ignore runtime error (e.g., failed to cooke mesh)
                continue

            for shape in shapes:
                shape.local_pose = r.pose
                shape.set_collision_groups(self.collision_groups)
                shape.set_density(r.density)
                shape.set_patch_radius(r.patch_radius)
                shape.set_min_patch_radius(r.min_patch_radius)
                component.attach(shape)

My question is: How can I access these PhysxCollisionShape instances and modify their properties, such as scale, during the simulation? Thank you very much!

[StoneT2000]

Modifying collision shapes at runtime (after loading the objects and starting the gpu simulation) is not possible at the moment. Therefore they cannot be changed during an environment reset. You can however with GPU simulation generate a ton of parallel environments each with different scales for the actors and maybe that will be sufficient randomization. If that is not enough, you can turn on the reconfiguration frequency (set reconfiguration_freq >= 1), which essentially destroys all objects and allows you to recreate them but with new actor scales

With GPU sim you will need to iterate over each the actor._objs list and modify the elements inside there, no API for batch modifying shape sizes/topology at the moment.

[hesic73]

Is it possible to extend the SAPIEN API to support batched properties? This would be useful for domain randomization.

For example, consider the current sapien.wrapper.actor_builder.CollisionShapeRecord:

@dataclass
class CollisionShapeRecord:
    type: Literal[
        "convex_mesh",
        "multiple_convex_meshes",
        "nonconvex_mesh",
        "plane",
        "box",
        "capsule",
        "sphere",
        "cylinder",
    ]

    # for mesh type
    filename: str = ""

    # mesh & box
    scale: Tuple = (1, 1, 1)

    # circle & capsule
    radius: float = 1
    # capsule
    length: float = 1

    material: Union[sapien.physx.PhysxMaterial, None] = None
    pose: sapien.Pose = sapien.Pose()

    density: float = 1000
    patch_radius: float = 0
    min_patch_radius: float = 0
    is_trigger: bool = False

    decomposition: str = "none"
    decomposition_params: Union[Dict[str, Any], None] = None

And the current sapien.physx.PhysxMaterial:

class PhysxMaterial:
    dynamic_friction: float
    restitution: float
    static_friction: float

Would it be possible to modify PhysxMaterial and CollisionShapeRecord to support batched inputs using torch.Tensor?

class PhysxMaterial:
    dynamic_friction: float | torch.Tensor
    restitution: float | torch.Tensor
    static_friction: float | torch.Tensor

@dataclass
class CollisionShapeRecord:
    type: Literal[
        "convex_mesh",
        "multiple_convex_meshes",
        "nonconvex_mesh",
        "plane",
        "box",
        "capsule",
        "sphere",
        "cylinder",
    ]

    # for mesh type
    filename: str = ""

    # mesh & box
    scale: Tuple | torch.Tensor = (1, 1, 1)

    # circle & capsule
    radius: float | torch.Tensor = 1
    # capsule
    length: float | torch.Tensor = 1

    material: Union[sapien.physx.PhysxMaterial, None] = None
    pose: sapien.Pose = sapien.Pose()

    density: float | torch.Tensor = 1000
    patch_radius: float = 0
    min_patch_radius: float = 0
    is_trigger: bool = False

    decomposition: str = "none"
    decomposition_params: Union[Dict[str, Any], None] = None

Additionally, could mani_skill.utils.building.actor_builder.ActorBuilder be updated so that build_entity can index these batched properties and create entities with different parameters across subscenes?

ManiSkill/mani_skill/utils/building/actor_builder.py

Line 206 in 318d511

entity = self.build_entity()

[StoneT2000]

I see the use case and it would be really useful to do more batch randomizations (textures can also be given the same treatment, currently the way it is done is not as convenient).

What would you prefer most as an API for this? When building a task (I imagine the randomization must occur during the _load_scene function).

We can change the API in maniskill to support more batched like convenience operations to avoid users writing for loops and simply supplying a batched array which may be easier to understand/read.

SAPIEN's API will very likely remain the same without batched APIs. Between maintainers this is what we decided was best (SAPIEN is the most low-level of the python APIs for very intricate control over the simulation setup and ManiSkill is prebuilt / simplified APIs for fast iteration and ML workflows)

[hesic73]

Thank you! Below is my idea on how to change the API:

Modify ActorBuilder Inheritance

Let mani_skill.utils.building.actor_builder.ActorBuilder not inherit from sapien.wrapper.actor_builder.ActorBuilder. Instead, make the SAPIEN builder a member variable within ManiSkill's ActorBuilder:

class ActorBuilder:
    def __init__(self, ...):
        self._sapien_builder = sapien.ActorBuilder(...)

Create Batched Material Classes

Avoid using sapien.physx.PhysxMaterial and sapien.render.RenderMaterial directly. Instead, create batched versions of these materials to handle varying properties across subscenes:

class PhysxMaterial:
    dynamic_friction: torch.Tensor
    restitution: torch.Tensor
    static_friction: torch.Tensor

Implement Batched Collision and Visual Methods

Implement methods like add_box_collision to handle batched properties:

def add_box_collision(
    self,
    pose: Pose = Pose(),  # ManiSkill pose
    half_size: Union[Vec3, torch.Tensor] = (1, 1, 1),
    material: Union[PhysxMaterial, sapien.physx.PhysxMaterial, None] = None,
    density: float = 1000,
    patch_radius: float = 0,
    min_patch_radius: float = 0,
    is_trigger: bool = False,
):
    if material is None:
        material = sapien.physx.get_default_material()
    if isinstance(material, sapien.physx.PhysxMaterial):
        # Convert to batched version if necessary

    # Use placeholders for pose, material, or other parameters as needed
    # Save the information somewhere for later use
    self._sapien_builder.add_box_collision(...)
    return self

Inject Batched Properties During Build:

During the build phase, inject the batched properties into the SAPIEN builder:

for i, scene_idx in enumerate(self.scene_idxs):
    if self.scene.parallel_in_single_scene:
        sub_scene = self.scene.sub_scenes[0]
    else:
        sub_scene = self.scene.sub_scenes[scene_idx]

    # Inject the properties for the i-th subscene into the SAPIEN builder
    # Modify the placeholders with the i-th batched property values
    # This may involve setting the material, pose, scale, etc., for each collision shape

    entity = self._sapien_builder.build_entity()
    # Prepend scene index to entity name to indicate which subscene it belongs to
    entity.name = f"scene-{scene_idx}_{self.name}"
    # Set the pose before adding to the scene
    entity.pose = to_sapien_pose(initial_pose_np[i])
    sub_scene.add_entity(entity)
    entities.append(entity)

This way we maintain compatibility with existing APIs.