Implemented generic multimodal chat handler.

[alcoftTAO]

• Implemented a generic/global multimodal chat handler.

What does it do?

It automatically uses the model's chat template and replaces all of the model's multimodal tags with the media_marker tag.

This allows a much easier implementation for multimodal models, since the chat template doesn't need to be hard-coded for each model.

How to use it?

It is as simple as passing the clip_model_path parameter to the Llama class when created.

Note

Using the previous implementation (e.g. Qwen35ChatHandler) still works.

I'm also looking forward to implement more model architectures. Please, reply if you want me to implement any.

[JamePeng]

You can take a look at how to improve the injection process. #110

[JamePeng]

It seems there's no work on how to perform URL injection for multimedia; simply replacing it with a media marker isn't enough.

This code also needs to be removed:

if hasattr(llama, 'input_ids'):
    llama.input_ids.fill(0)

Architecture-based tag guessing should not default unknown models to Qwen-style tags. Prefer detecting media tags from the actual chat template, or better, avoid tag guessing by normalizing OpenAI content parts into placeholders before rendering.

KNOWN_MEDIA_TAGS = [
    "<|image_pad|>",
    "<|audio_pad|>",
    "<|video_pad|>",
    "<|image|>",
    "<|audio|>",
    "<|video|>",
    "[IMG]",
]

and

self._chat_format_parser_tags = [
    tag for tag in KNOWN_MEDIA_TAGS
    if tag in self.chat_format
]

In addition, a check is needed to ensure that the number of replacement markers matches the number of incoming media.

[alcoftTAO]

@JamePeng What do you think of this code?

[JamePeng]

You can test the multimodal usage of qwen3vl, qwen3.5/3.6, and gemma4.
In particular, check if the omni function of gemma4 is affected.