Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Proposal] Checklist of environments to add to the framework #748

Open
6 of 26 tasks
Mayankm96 opened this issue Jul 29, 2024 · 5 comments
Open
6 of 26 tasks

[Proposal] Checklist of environments to add to the framework #748

Mayankm96 opened this issue Jul 29, 2024 · 5 comments
Assignees
@kellyguo11 @Dhoeller19 @Mayankm96 @Toni-SM
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed

Comments

@Mayankm96
Copy link
Contributor

Mayankm96 commented Jul 29, 2024 •
edited
Loading

Proposal

This issue is a tracking page for our current plan to add environments to the Isaac Lab framework.

Motivation

The initial Orbit work mentioned a suite of rigid and deformable body simulation environments. However, many of these environments have been held back due to breaking changes in the framework APIs, poor simulation stability and scaling (particularly for deformable bodies and particle-based cloths), and a lack of resources to transfer all the implementations. Since this is a repeated topic of discussion on GitHub, I am creating this issue to keep track of the environments and provide a (very rough) outline/roadmap.

We are actively working on adding new features and improvements. It is hard to dedicate time to everything at once, but slowly, we are adding new environments as well. If you're interested in contributing, feel free to contact us!

Checklist

Completed ✅

For a complete list, please check here: https://isaac-sim.github.io/IsaacLab/source/features/environments.html

  • Reach - Tracks a pose command for an end effector
  • Lift - Lifts the object to the desired position
  • In-Hand Repose - Re-orient a cube with a dexterous hand
  • Cabinet - Open or close a cabinet (hinged object)
  • Quadrupedal Locomotion - Track velocity command over a variety of terrain with a legged robot
  • Quadrupedal Navigation - Track desired position command with a legged robot

In Progress 🏗️

  • Nut-bolt - Tighten the Nut on the specified Bolt
  • Soft Lifting - Lift a soft object to the target position: Adds environment for lifting a deformable cube #842
  • Throwing and catching - Throw and catch a ball with two floating dexterous hands
  • Soft Stacking - Stacking soft objects together

Help Needed 🆘

  • Beat the Buzz - Move a key around a pole without touching it
  • Pyramid Stack - Build blocks into a pyramid
  • Hockey - Shoot the puck into the net with the stick
  • Peg In Hole - Insert the block into the hole
  • Jenga - Take out blocks and stack them into towers
  • Soft Placing - Placing soft objects on a shelf
  • Soft Dropping - Dropping soft objects into a container
  • Tower of Hanoi - Stacks of soft objects around pillars
  • Rope Reshaping - Reshape the rope on the table

Stand-by 🛑

  • Cloth Lifting - Lift the cloth to the desired position
  • Cloth Folding - Fold the cloth into the desired state
  • Cloth Spreading - Spread cloth across the table
  • Cloth Dropping - Place the cloth in the container
  • Flag Hoisting - Raise a flag on a table
  • Fluid Pouring - Pour the liquid into another container
  • Fluid Transport - Move a container full of liquid without causing any spills
@Mayankm96 Mayankm96 added enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed labels Jul 29, 2024
@Mayankm96 Mayankm96 pinned this issue Jul 29, 2024
This was referenced Jul 29, 2024
Closed
Closed
Closed
@hanschanhs
Copy link

Thanks you for the great work - Wanted to ask about the cloth/tshirt prims in IsaacLab. What is the hold up/why is it listed as standby? Separately, If I just want to spawn a tshirt, what is the best way to do this currently in Isaac Lab? Novice question. Thanks for this!

@Mayankm96
Copy link
Contributor Author

Hi @hanschanhs

We initially experimented with particle-based cloth simulation. However, we had difficulty tuning these cloth properties for smooth gripper and cloth interactions. On further discussions with the PhysX team, we have been informed that an FEM implementation for cloths is under works. Thus, the environments for cloths are in standby as we wait for the nicer solver implementation to be available.

If you wish to spawn a shirt, the easiest is make a USD file with the right properties and then spawn it into the scene through AssetBase class or drive a ParticleBasedCloth class from it yourself with the desired APIs.

@robegi
Copy link

robegi commented Aug 29, 2024

Good morning, and thank you for the update! If I may ask, I would like to know something more about the issues with the particle based liquid tasks.
I am currently working on a liquid pouring task, and until now I managed to spawn and set up the fluid correctly, but I am a bit worried about the "poor simulation stability and scaling" you mentioned above. Is it possible to train an agent using particle based fluids?
Thanks in advance!

@allenzren
Copy link

allenzren commented Sep 5, 2024 •
edited
Loading

Hi @hanschanhs

We initially experimented with particle-based cloth simulation. However, we had difficulty tuning these cloth properties for smooth gripper and cloth interactions. On further discussions with the PhysX team, we have been informed that an FEM implementation for cloths is under works. Thus, the environments for cloths are in standby as we wait for the nicer solver implementation to be available.

If you wish to spawn a shirt, the easiest is make a USD file with the right properties and then spawn it into the scene through AssetBase class or drive a ParticleBasedCloth class from it yourself with the desired APIs.

Hi @Mayankm96, do you think then it is possible to simulate manipulating a cloth with a Franka arm right now? or the gripper and cloth interaction issue is not possible to solve with the current solver? Thanks!

I am happy to spend time setting this up and tuning the parameters, but not sure if cloth manipulation is fundamentally possible right now.

@wzx16
Copy link

wzx16 commented Sep 6, 2024

Hi @hanschanhs
We initially experimented with particle-based cloth simulation. However, we had difficulty tuning these cloth properties for smooth gripper and cloth interactions. On further discussions with the PhysX team, we have been informed that an FEM implementation for cloths is under works. Thus, the environments for cloths are in standby as we wait for the nicer solver implementation to be available.
If you wish to spawn a shirt, the easiest is make a USD file with the right properties and then spawn it into the scene through AssetBase class or drive a ParticleBasedCloth class from it yourself with the desired APIs.

Hi @Mayankm96, do you think then it is possible to simulate manipulating a cloth with a Franka arm right now? or the gripper and cloth interaction issue is not possible to solve with the current solver? Thanks!

I am happy to spend time setting this up and tuning the parameters, but not sure if cloth manipulation is fundamentally possible right now.

I have the same question. In addition, I'd like to know what would be the recommended way to spawn a rope prim. Thank you!

@yuezhezhang yuezhezhang mentioned this issue Sep 11, 2024
Open
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kellyguo11 kellyguo11

@Dhoeller19 Dhoeller19

@Mayankm96 Mayankm96

@Toni-SM Toni-SM

Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@kellyguo11 @Dhoeller19 @Mayankm96 @allenzren @Toni-SM @wzx16 @hanschanhs @robegi