[BFCL] Introducing custom handlers #756
Conversation
| available_models = set(HANDLER_MAP.keys()) | ||
|
|
||
| # If a custom handler setting exists, add it to the | ||
| handler_config_path = os.getenv("BFCL_HANDLER_CONFIG") | ||
| if handler_config_path and os.path.exists(handler_config_path): | ||
| try: | ||
| with open(handler_config_path) as f: | ||
| handler_config = json.load(f) | ||
| available_models.update(handler_config.keys()) | ||
| except Exception as e: | ||
| print(f"Warning: Error loading custom handler config: {str(e)}") | ||
|
|
There was a problem hiding this comment.
Previously, you could simply use dictionary mapping to get handler.
For now, HandlerLoader will controll.
You can set environment variable for BFCL_HANDLER_CONFIG.
This environ var. point to the path of json file.
and if this exists, Loader can find the path of json file.
There was a problem hiding this comment.
This is example of json files
{
"my-custom-model": {
"module_path": "/path/to/my_handlers.py",
"class_name": "MyCustomHandler"
}
}and example of setting env var.
# you can enter this on command"
export BFCL_HANDLER_CONFIG=/path/to/handler_config.json
python -m bfcl generate --model my-custom-model --test-category all
There was a problem hiding this comment.
At /path/to/my_handlers.py you can make custom handlers clas
from bfcl.model_handler.base_handler import BaseHandler
from bfcl.model_handler.model_style import ModelStyle
class MyCustomHandler(BaseHandler):
def __init__(self, model_name, temperature):
super().__init__(model_name, temperature)
self.model_style = ...
...There was a problem hiding this comment.
For detailed example, I've introduced my case.
berkeley-function-call-leaderboard/custom_handler.py
class MyCustomHandler(OpenAIHandler):
def __init__(self, model_name, temperature):
BaseHandler.__init__(self, model_name, temperature)
self.model_style = ModelStyle.OpenAI
self.client = OpenAI(
api_key="EMPTY",
base_url="..."
)berkeley-function-call-leaderboard/handler_config.json
{
"allganize/Alpha-Ko-32B-GPTQ-Int8-202411": {
"module_path": "custom_handler.py",
"class_name": "MyCustomHandler"
}
}export BFCL_HANDLER_CONFIG=handler_config.json
There was a problem hiding this comment.
The result works well when I enter below command
python3 -m bfcl generate --model allganize/Alpha-Ko-32B-GPTQ-Int8-202411 --test-category multi_turn_base --num-threads 40
There was a problem hiding this comment.
Thanks for the PR @sangmandu ! Could you provide an example BFCL_HANDLER_CONFIG?
Wow, rapid reply! Thanks. |
| # Load metadata at the beginning of the function | ||
| model_metadata, _, _ = metadata_loader.load_metadata() | ||
|
|
There was a problem hiding this comment.
For dynamic load, we'd used model_metadata rather than MODEL_METADATA_MAPPING
| no_cost = list(NO_COST_MODELS) | ||
|
|
||
| # Check for additional metadata config file path in environment variables | ||
| metadata_config_path = os.getenv("BFCL_MODEL_METADATA") |
There was a problem hiding this comment.
I've implemented the same config-based handler loading functionality in the evaluate module that was previously available in generation.
export BFCL_MODEL_METADATA=model_metadata.json
python3 -m bfcl evaluate --model allganize/Alpha-Ko-32B-GPTQ-Int8-202411 --test-category multi_turn_baseThere was a problem hiding this comment.
This is example of model_metadata.json
{
"metadata": {
"allganize/Alpha-Ko-32B-GPTQ-Int8-202411": [
"alpha-ko-32b-gptq-int8",
"https://huggingface.co/allganize/Alpha-Ko-32B-GPTQ-Int8-202411",
"Allganize",
"Custom License"
]
},
"prices": {
"allganize/Alpha-Ko-32B-GPTQ-Int8-202411": 0.0
},
"no_cost_models": [
"allganize/Alpha-Ko-32B-GPTQ-Int8-202411"
]
} 
sangmandu
changed the title
sangmandu
force-pushed
the
feature-customhandler
branch
from
99c9f09 to
3579377
Compare
Hi there,
I am a current user of bfcl and would like to submit this PR to address an improvement opportunity I've identified while working with the module.
Currently, model handlers are managed through the
handler_map.py, which presents some challenges. This managed approach requires code updates from maintainers whenever new models are released. Additionally, users must wait for updates if their desired model isn't yet supported.While technically proficient users might implement their own solutions for unsupported models, this especially becomes problematic when using bfcl as a submodule. Modifying the original code is not ideal from a maintenance perspective, as it requires additional resources for ongoing maintenance.
Therefore, I'm submitting this PR to enhance the main codebase.
The current
build_handlerfunction only allows fetching predefined model handlers. I propose implementing aHandlerLoaderthat enables users to input custom handlers. This approach minimizes changes to the main codebase while allowing users to manage their custom handlers through just two files: a Python file containing the handler definition and a JSON configuration file.