Model YAML structure #20

gregcube · 2024-09-27T15:01:03Z

gregcube
Sep 27, 2024
Maintainer

I'd like to discuss the YAML model structure. Currently, it follows the JSON format, but represented in YAML. For example, using the Amber model, it looks like this:

framework:
  name: 'Model Openness Framework'
  version: '1.0'
  date: '2024-12-15'
release:
  name: Amber
  version: 7B
  type: 'Language Model'
  architecture: 'Transformer (Decoder-only)'
  origin: Amber
  producer: LLM360
  components:
    - 
      name: 'Model architecture'
      description: "Well commented code for the model's architecture"
      location: ''
      license_name: Apache-2.0
      license_path: ''
    - 
      name: 'Data preprocessing code'
      description: 'Code for data cleansing, normalization, and augmentation'
      location: ''
      license_name: Apache-2.0
      license_path: ''
    - 
      name: 'Training code'
      description: 'Code used for training the model'
      location: ''
      license_name: Apache-2.0
      license_path: ''
    - 
      name: 'Inference code'
      description: 'Code used for running the model to make predictions'
      location: ''
      license_name: Apache-2.0
      license_path: ''
    - 
      name: 'Evaluation code'
      description: 'Code used for evaluating the model'
      location: ''
      license_name: Apache-2.0
      license_path: ''
    - 
      name: 'Supporting libraries and tools'
      description: "Libraries and tools used in the model's development"
      location: ''
      license_name: Apache-2.0
      license_path: ''
    - 
      name: 'Model parameters (Final)'
      description: 'Trained model parameters, weights and biases'
      location: ''
      license_name: Apache-2.0
      license_path: ''
    - 
      name: 'Model parameters (Intermediate)'
      description: 'Trained model parameters, weights and biases'
      location: ''
      license_name: Apache-2.0
      license_path: ''
    - 
      name: Datasets
      description: 'Training, validation and testing datasets used for the model'
      location: ''
      license_name: 'License not specified'
      license_path: ''
    - 
      name: 'Evaluation data'
      description: 'Data used for evaluating the model'
      location: ''
      license_name: 'License not specified'
      license_path: ''
    - 
      name: 'Model metadata'
      description: 'Any model metadata including training configuration and optimizer states'
      location: ''
      license_name: 'Component not included'
      license_path: ''
    - 
      name: 'Sample model outputs'
      description: 'Examples of outputs generated by the model'
      location: ''
      license_name: 'License not specified'
      license_path: ''
    - 
      name: 'Model card'
      description: 'Model details including performance metrics, intended use, and limitations'
      location: ''
      license_name: 'License not specified'
      license_path: ''
    - 
      name: 'Data card'
      description: 'Documentation for datasets including source, characteristics, and preprocessing details'
      location: ''
      license_name: 'License not specified'
      license_path: ''
    - 
      name: 'Technical report'
      description: 'Technical report detailing capabilities and usage instructions for the model'
      location: ''
      license_name: 'License not specified'
      license_path: ''
    - 
      name: 'Research paper'
      description: 'Research paper detailing the development and capabilities of the model'
      location: ''
      license_name: 'License not specified'
      license_path: ''
    - 
      name: 'Evaluation results'
      description: 'The results from evaluating the model'
      location: ''
      license_name: 'License not specified'
      license_path: ''

Questions:

Do we need the framework key and its key-value pairs?
Is the description key-value pair for each component necessary? It's included in the JSON format, but we may not need it in YAML.
I'm thinking perhaps we should rename location to component_path for each component?

Let me know your thoughts.

lehors · 2024-09-29T16:16:02Z

lehors
Sep 29, 2024
Maintainer

I wasn't involved in the development of this format so @matthew-d-white @ccosborne should chime in but here is my take on this:

1. Do we need the `framework` key and its key-value pairs?

My understanding is that it is meant to define which version of the MOF the file is based on. I think a single version identifier could do but it's always good to have some form of identifier no matter what so I wouldn't just drop the whole thing.

2. Is the `description` key-value pair for each component necessary? It's included in the JSON format, but we may not need it in YAML.

This is meant to allow providing a description for non standard components, but such components are exceptional so I think we could make the description optional so that for all of the standard ones we can skip it.

3. I'm thinking perhaps we should rename `location` to `component_path` for each component?

I agree that this would be more consistent with license_path.

2 replies

matthew-d-white Sep 29, 2024
Collaborator

Yes it is needed.
Yes it is needed. These are descriptive fields for freeform text.
seems logical to me. Any changes to the MOF.JSON format require that the version number be incremented.

matthew-d-white Sep 29, 2024
Collaborator

I would also suggest keeping the key names in the file even when the value is undefined: as here: -> https://github.com/isitopenai/MOF/blob/main/MOF.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Model YAML structure #20

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 2 replies

{{title}}

{{title}}

{{title}}

Select a reply

Model YAML structure #20

gregcube Sep 27, 2024 Maintainer

Replies: 1 comment · 2 replies

lehors Sep 29, 2024 Maintainer

matthew-d-white Sep 29, 2024 Collaborator

matthew-d-white Sep 29, 2024 Collaborator

gregcube
Sep 27, 2024
Maintainer

Replies: 1 comment 2 replies

lehors
Sep 29, 2024
Maintainer

matthew-d-white Sep 29, 2024
Collaborator

matthew-d-white Sep 29, 2024
Collaborator