? 多图、视频首上端！3 SOTA 面壁小钢炮，创 GPT-4V 端侧全面对标新时代！
? https://github.com/OpenBMB/MiniCPM-V
? https://huggingface.co/openbmb/MiniCPM-V-2_6
? https://www.bilibili.com/video/BV1sM4m1172r/?spm_id_from=333.999.0.0&vd_source=1534be4f756204643265d5f6aaa38c7b
vLLM推理
使用Python进行推理
-
首先前往huggingface下载pytorch权重：
git clone https://huggingface.co/openbmb/MiniCPM-V-2_6
-
使用git下载安装vllm
git clone https://github.com/vllm-project/vllm.gitcd vllmpip install e .
-
创建一个python代码调用vllm
from PIL import Imagefrom transformers import AutoTokenizerfrom vllm import LLM, SamplingParams# 图像文件路径列表IMAGES = ["/root/ld/ld_project/MiniCPM-V/assets/airplane.jpeg",# 本地图片路径]# 模型名称或路径MODEL_NAME = "/root/ld/ld_model_pretrained/Minicpmv2_6"# 本地模型路径或Hugging Face模型名称# 打开并转换图像image = Image.open(IMAGES[0]).convert("RGB")# 初始化分词器tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)# 初始化语言模型llm = LLM(model=MODEL_NAME,gpu_memory_utilization=1,# 使用全部GPU内存trust_remote_code=True,max_model_len=2048)# 根据内存状况可调整此值# 构建对话消息messages = [{'role': 'user', 'content': '(<image>./</image>)n' + '请描述这张图片'}]# 应用对话模板到消息prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)# 设置停止符ID# 2.0# stop_token_ids = [tokenizer.eos_id]# 2.5#stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id]# 2.6stop_tokens = ['<|im_end|>', '<|endoftext|>']stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]# 设置生成参数sampling_params = SamplingParams(stop_token_ids=stop_token_ids,# temperature=0.7,# top_p=0.8,# top_k=100,# seed=3472,max_tokens=1024,# min_tokens=150,temperature=0,use_beam_search=True,# length_penalty=1.2,best_of=3)# 获取模型输出outputs = llm.generate({"prompt": prompt,"multi_modal_data": {"image": image}}, sampling_params=sampling_params)print(outputs[0].outputs[0].text)
4. 使用视频推理
from transformers import AutoTokenizerfrom decord import VideoReader, cpufrom PIL import Imagefrom vllm import LLM, SamplingParams# 进行图片推理MAX_NUM_FRAMES = 64def encode_video(filepath):def uniform_sample(l, n):gap = len(l) / nidxs = [int(i * gap + gap / 2) for i in range(n)]return [l[i] for i in idxs]vr = VideoReader(filepath, ctx=cpu(0))sample_fps = round(vr.get_avg_fps() / 1)# FPSframe_idx = [i for i in range(0, len(vr), sample_fps)]if len(frame_idx)>MAX_NUM_FRAMES:frame_idx = uniform_sample(frame_idx, MAX_NUM_FRAMES)video = vr.get_batch(frame_idx).asnumpy()video = [Image.fromarray(v.astype('uint8')) for v in video]return videoMODEL_NAME = "openbmb/MiniCPM-V-2_6" # or local model pathvideo = encode_video("xxx.mp4")messages = [{"role":"user","content":"".join(["(<image>./</image>)"] * len(video)) +"nPlease describe this video."}]prompt = tokenizer.apply_chat_template(messages,tokenize=False,add_generation_prompt=True)tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)llm = LLM(model=MODEL_NAME,gpu_memory_utilization=1,max_model_len=4096)stop_tokens = ['<|im_end|>', '<|endoftext|>']stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]sampling_params = SamplingParams(stop_token_ids=stop_token_ids,use_beam_search=Truetemperature=0,max_tokens=64)outputs = llm.generate({"prompt": prompt,"multi_modal_data": {"image": {"images": video,"use_image_id": False,"max_slice_nums": 1 if len(video) > 16 else 2}}}, sampling_params=sampling_params_beam)
vLLM API Server
-
使用git下载安装vllm
git clone https://github.com/vllm-project/vllm.gitcd vllmpip install e .
-
命令行启动vllm服务
vllm serve /root/ld/ld_model_pretrained/Minicpmv2_6 --dtype auto --max-model-len 2048 --api-key token-abc123 --gpu_memory_utilization 1 --trust-remote-code
-
使用python代码调用vllm的http服务
3.1 传入网络图片
from openai import OpenAIopenai_api_key = "token-abc123" # your api key set in launch serveropenai_api_base = "http://localhost:8000/v1" # http idclient = OpenAI(api_key=openai_api_key,base_url=openai_api_base,)chat_response = client.chat.completions.create(model="/root/ld/ld_model_pretrained/Minicpmv2_6", # model_local_path or huggingface idmessages=[{"role": "user","content": [# NOTE: 使用图像令牌 <image> 的提示格式是不必要的，因为提示将由API服务器自动处理。# 由于提示将由API服务器自动处理，因此不需要使用包含 <image> 图像令牌的提示格式。{"type": "text", "text": "请描述这张图片"},{"type": "image_url","image_url": {"url": "https://air-example-data-2.s3.us-west-2.amazonaws.com/vllm_opensource_llava/stop_sign.jpg",},},],}],extra_body={"stop_token_ids": [151645, 151643]})print("Chat response:", chat_response)print("Chat response content:", chat_response.choices[0].message.content)
3.2 传入本地图片
from openai import OpenAIopenai_api_key = "token-abc123" # your api key set in launch serveropenai_api_base = "http://localhost:8000/v1" # http idclient = OpenAI(api_key=openai_api_key,base_url=openai_api_base,)# 用于传本地图片with open('your/local/pic/path','rb') as file:image = "data:image/jpeg;base64,"+ base64.b64encode(file.read()).decode('utf-8')chat_response = client.chat.completions.create(model="/root/ld/ld_model_pretrained/Minicpmv2_6", # model_local_path or huggingface idmessages=[{"role": "user","content": [# NOTE: 使用图像令牌 <image> 的提示格式是不必要的，因为提示将由API服务器自动处理。# 由于提示将由API服务器自动处理，因此不需要使用包含 <image> 图像令牌的提示格式。{"type": "text", "text": "请描述这张图片"},{"type": "image_url","image_url": {"url": image,},},],}],extra_body={"stop_token_ids": [151645, 151643]})print("Chat response:", chat_response)print("Chat response content:", chat_response.choices[0].message.content)
llama.cpp推理
设备要求：运行非量化版内存超过19g，运行量化版超过8g内存
1. 下载依赖包
brew install ffmpegbrew install pkg-config
2. 获取openbmb的llama.cpp分支
git clone -b minicpm-v2.5 https://github.com/OpenBMB/llama.cpp.git
3. 编译llama.cpp
cd llama.cppmake
4. 获取MiniCPM-V 2.6的gguf权重
方法一：
a. 首先前往huggingface或者modelscope下载pytorch权重：
git clone https://huggingface.co/openbmb/MiniCPM-V-2_6
# 第一行为获得模型中间输出，为转换为gguf作准备python ./examples/llava/minicpmv-convert/minicpmv2_6-surgery.py -m ../MiniCPM-V-2_6# 将siglip模型转换为ggufpython ./examples/llava/minicpmv-convert/minicpmv2_6-convert-image-encoder-to-gguf.py -m ../MiniCPM-V-2_6 --minicpmv-projector ../MiniCPM-V-2_6/minicpmv.projector --output-dir ../MiniCPM-V-2_6/ --image-mean 0.5 0.5 0.5 --image-std 0.5 0.5 0.5# 将语言模型转换为ggufpython ./convert-hf-to-gguf.py ../MiniCPM-V-2_6/model
# quantize int4 version./llama-quantize ../MiniCPM-V-2_6/model/ggml-model-f16.gguf ../MiniCPM-V-2_6/model/ggml-model-Q4_K_M.gguf Q4_K_M
方法二：
5. 开始推理：
5.1 图片推理指令
5.2 视频推理指令
Ollama推理
ollma官方github即将合并我们的ollama分支，请关注我们的 GitHub 设备要求：运行非量化版本需要19g以上内存，运行量化版本需要8g以上内存 1. 按照上述llama.cpp教程获取gguf模型
语言模型最好是量化模型。
2. 获取openbmb官方ollma分支
3. 环境需求
-
cmake version 3.24 or above -
go version 1.22 or above -
gcc version 11.4.0 or above
brew install go cmake gcc
4. 安装大型模型依赖项:
go generate ./...
5. 编译ollama
go build .
6. 编译成功，在ollama主路径启动ollama：
./ollama serve
7. 创建一个Modelfile:
vim minicpmv2_6.Modelfile
FROM ./MiniCPM-V-2_6/model/ggml-model-Q4_K_M.ggufFROM ./MiniCPM-V-2_6/mmproj-model-f16.ggufTEMPLATE """{{ if .System }}<|im_start|>system{{ .System }}<|im_end|>{{ end }}{{ if .Prompt }}<|im_start|>user{{ .Prompt }}<|im_end|>{{ end }}<|im_start|>assistant<|im_end|>{{ .Response }}<|im_end|>"""PARAMETER stop "<|endoftext|>"PARAMETER stop "<|im_end|>"PARAMETER num_ctx 20488. 创建ollama模型实例：
ollama create minicpm2.6 -f minicpmv2_6.Modelfile9. 另起一个命令行窗口，运行ollama模型实例：
ollama run minicpm2.610. 输入问题和图片 URL，以空格分隔
