Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

semantic segmentation is not enabled in apartment_0000 and kujiale_0000 scenes #234

Open
RachithP opened this issue Jul 26, 2023 · 3 comments

Comments

@RachithP
Copy link
Collaborator

RachithP commented Jul 26, 2023

The actors and static mesh components in apartment_0000 and kujiale_0000 scenes do not have semantic tags as part of their tags. These are required to enable semantic segmentation in these scenes. Also, the programmatic way to process these semantic tags, and convert them to custom depth stencil value is missing.

How to reproduce:

  • run generate_images.py with render pass value set to segmentation. The output images will be blank.
@mikeroberts3000
Copy link
Collaborator

mikeroberts3000 commented Jul 27, 2023

We are already requiring the artist to group actors into top-level folders in the World Outliner view, based on their semantic categories. Using folders in the World Outliner view is convenient for editing and debugging, but the folder structure is not available at runtime, so we need to somehow convert the folder structure to tags, and ultimately to custom depth-stencil values.

We could require the artist to do this manually, but that seems error-prone. Instead, we should automate this tagging procedure via a Python script that can be invoked via the command-line, the Unreal console, or via a custom editor button. In principle, this Python script could also be used to set custom depth-stencil values, so we wouldn't need to do it at runtime.

@RachithP
Copy link
Collaborator Author

RachithP commented Aug 1, 2023

I agree. Since we cannot obtain the World Outliner view folder structure in runtime, and allowing this process to be manual is error-prone, we can use Python APIs to perform this task. This way we can execute this script on a need-basis, whenever new actors/components are added, and are re-arranged.

Some references:

I've a pseudo code here. One drawback in this code is that we cannot specify different semantic tags for the components under the same actor. This could be a problem for actors that have components merged, yet each component needs to have different semantic tags.

import unreal

editor_subsytem = unreal.get_editor_subsystem(unreal.EditorActorSubsystem)
level_actors = editor_subsystem.get_all_level_actors()
for actor in level_actors:
    # get semantic tags 
    actor_path = actor.get_folder_path()
    semantic_tag = actor_path.split('_')[-1]
    
    # get all staticmeshcomponents
    static_mesh_components = actor.get_components_by_class(unreal.StaticMeshComponent)
    
    # apply semantic tags to all components
    for component in static_mesh_components:
        component.set_editor_property("component_tags", [semantic_tag])

@mikeroberts3000
Copy link
Collaborator

mikeroberts3000 commented Aug 1, 2023

This sketch is useful, thank you for that.

  • I don't think that pseudocode will correctly handle the case where the semantic category name has underscores in it, e.g., pillow_case or something.
  • I think our design philosophy should be to provide sensible default behavior that doesn't require any additional artist steps most of the time, but allows for full artist control in cases when the artist wants it. For example, in the sketch above, we are unconditionally setting component tags. I think it would be better to only set the component tag if the component doesn't have a spear:semantic:whatever tag already set. This would make the most common case (i.e., an actor with a single semantic class) very easy for the artist, because they could just place the actor in the 00_whatever folder in the World Outliner. But in rare cases when an artist wants additional control, they can always express exactly what they want by manually tagging individual components.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants