探索基于Qwen2.5实现DeepSeek推理的奇妙之旅

%%capture
# 该魔法命令用于在 Jupyter Notebook中隐藏输出
# 避免输出干扰执行结果，例如 `pip install` 过程中的信息


# 安装 Unsloth 和 vLLM 这两个库
!pip install unsloth vllm


# 升级 Pillow（Python Imaging Library），确保使用最新版本
!pip install --upgrade pillow

from unsloth import FastLanguageModel, PatchFastRL  
# 从 unsloth 库中导入 FastLanguageModel（加速的语言模型）和 PatchFastRL（加速补丁）  


PatchFastRL("GRPO", FastLanguageModel)  
# 对 FastLanguageModel 应用 "GRPO" 方案的补丁优化，以提高推理效率

from unsloth import is_bfloat16_supported  # 检测是否支持 bfloat16（更节省显存的数值格式）
import torch  # 导入 PyTorch 进行深度学习计算


# 设置模型参数
max_seq_length = 1024  # 最大序列长度，可以增大以支持更长的推理
lora_rank = 64  # LoRA 低秩适配矩阵的秩，较大值提高智能性但降低推理速度


# 加载 FastLanguageModel 预训练模型
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="Qwen/Qwen2.5-3B-Instruct",  # 使用 Qwen2.5-3B-Instruct 作为基础模型
    max_seq_length=max_seq_length,  # 设置最大序列长度
    load_in_4bit=True,  # 使用 4-bit 量化（减少显存消耗），若为 False 则使用 16-bit LoRA
    fast_inference=True,  # 启用 vLLM 加速推理（优化并行计算）
    max_lora_rank=lora_rank,  # 设置 LoRA 适配的秩
    gpu_memory_utilization=0.5,  # 限制 GPU 内存占用，降低该值可减少显存溢出
)


# 为模型应用 PEFT（参数高效微调）和 LoRA（低秩适配）
model = FastLanguageModel.get_peft_model(
    model,
    r=lora_rank,  # LoRA 低秩矩阵的秩，可选 8, 16, 32, 64, 128
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",  # 量化关键模块：QKV 投影和输出投影
        "gate_proj", "up_proj", "down_proj",  # MLP 相关模块
    ],  # 若显存不足，可以移除 QKVO 相关模块
    lora_alpha=lora_rank,  # LoRA 适配器的 alpha 参数，通常设为与 r 相同
    use_gradient_checkpointing="unsloth",  # 启用梯度检查点，减少显存占用，适用于长上下文微调
    random_state=3407,  # 设置随机种子，保证实验可复现
)

# 输出 
==((====))==  Unsloth 2025.2.12: Fast Qwen2 patching. Transformers: 4.48.3.
   \\   /|    GPU: Tesla T4. Max memory: 14.741 GB. Platform: Linux.
O^O/ \_/ \    Torch: 2.5.1+cu124. CUDA: 7.5. CUDA Toolkit: 12.4. Triton: 3.1.0
\        /    Bfloat16 = FALSE. FA [Xformers = 0.0.28.post3. FA2 = False]
 "-____-"     Free Apache license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
Unsloth: vLLM loading unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit with actual GPU utilization = 49.66%
Unsloth: Your GPU has CUDA compute capability 7.5 with VRAM = 14.74 GB.
Unsloth: Using conservativeness = 1.0. Chunked prefill tokens = 1024. Num Sequences = 192.
Unsloth: vLLM's KV Cache can use up to 4.9 GB. Also swap space = 2 GB.
WARNING 02-17 07:17:06 config.py:2386] Casting torch.bfloat16 to torch.float16.
INFO 02-17 07:17:19 config.py:542] This model supports multiple tasks: {'classify', 'generate', 'embed', 'reward', 'score'}. Defaulting to 'generate'.
Unsloth: vLLM Bitsandbytes config using kwargs = {'load_in_8bit': False, 'load_in_4bit': True, 'bnb_4bit_compute_dtype': 'float16', 'bnb_4bit_quant_storage': 'uint8', 'bnb_4bit_quant_type': 'nf4', 'bnb_4bit_use_double_quant': True, 'llm_int8_enable_fp32_cpu_offload': False, 'llm_int8_has_fp16_weight': False, 'llm_int8_skip_modules': ['lm_head', 'multi_modal_projector', 'merger', 'modality_projection', 'model.layers.2.mlp', 'model.layers.3.mlp', 'model.layers.30.mlp'], 'llm_int8_threshold': 6.0}
INFO 02-17 07:17:19 llm_engine.py:234] Initializing a V0 LLM engine (v0.7.2) with config: model='unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit', speculative_cnotallow=None, tokenizer='unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revisinotallow=None, override_neuron_cnotallow=None, tokenizer_revisinotallow=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=1024, download_dir=None, load_format=LoadFormat.BITSANDBYTES, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantizatinotallow=bitsandbytes, enforce_eager=False, kv_cache_dtype=auto,  device_cnotallow=cuda:0, decoding_cnotallow=DecodingConfig(guided_decoding_backend='xgrammar'), observability_cnotallow=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_cnotallow=None, compilation_cnotallow={"level":0,"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":192}, use_cached_outputs=False, 
tokenizer_config.json: 100%
 7.36k/7.36k [00:00<00:00, 451kB/s]
vocab.json: 100%
 2.78M/2.78M [00:00<00:00, 8.77MB/s]
merges.txt: 100%
 1.67M/1.67M [00:00<00:00, 6.92MB/s]
tokenizer.json: 100%
 11.4M/11.4M [00:00<00:00, 42.1MB/s]
added_tokens.json: 100%
 605/605 [00:00<00:00, 37.9kB/s]
special_tokens_map.json: 100%
 614/614 [00:00<00:00, 54.5kB/s]
generation_config.json: 100%
 271/271 [00:00<00:00, 17.2kB/s]
INFO 02-17 07:17:23 cuda.py:179] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 02-17 07:17:23 cuda.py:227] Using XFormers backend.
INFO 02-17 07:17:24 model_runner.py:1110] Starting to load model unsloth/qwen2.5-3b-instruct-unsloth-bnb-4bit...
INFO 02-17 07:17:24 loader.py:1102] Loading weights with BitsAndBytes quantization.  May take a while ...
INFO 02-17 07:17:25 weight_utils.py:252] Using model weights format ['*.safetensors']
model.safetensors: 100%
 2.36G/2.36G [00:15<00:00, 165MB/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:04<00:00,  4.67s/it]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.17s/it]
INFO 02-17 07:17:48 model_runner.py:1115] Loading model weights took 2.2160 GB
INFO 02-17 07:17:48 punica_selector.py:18] Using PunicaWrapperGPU.
INFO 02-17 07:17:58 worker.py:267] Memory profiling takes 9.77 seconds
INFO 02-17 07:17:58 worker.py:267] the current vLLM instance can use total_gpu_memory (14.74GiB) x gpu_memory_utilization (0.50) = 7.32GiB
INFO 02-17 07:17:58 worker.py:267] model weights take 2.22GiB; non_torch_memory takes 0.05GiB; PyTorch activation peak memory takes 1.05GiB; the rest of the memory reserved for KV Cache is 4.01GiB.
INFO 02-17 07:17:58 executor_base.py:110] # CUDA blocks: 7300, # CPU blocks: 3640
INFO 02-17 07:17:58 executor_base.py:115] Maximum concurrency for 1024 tokens per request: 114.06x
INFO 02-17 07:18:01 model_runner.py:1434] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
Capturing CUDA graph shapes: 100%|██████████| 27/27 [00:43<00:00,  1.60s/it]
INFO 02-17 07:18:44 model_runner.py:1562] Graph capturing finished in 43 secs, took 0.62 GiB
INFO 02-17 07:18:44 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 56.33 seconds
tokenizer_config.json: 100%
 7.36k/7.36k [00:00<00:00, 650kB/s]
vocab.json: 100%
 2.78M/2.78M [00:00<00:00, 14.2MB/s]
merges.txt: 100%
 1.67M/1.67M [00:00<00:00, 26.5MB/s]
added_tokens.json: 100%
 605/605 [00:00<00:00, 45.0kB/s]
special_tokens_map.json: 100%
 614/614 [00:00<00:00, 32.3kB/s]
tokenizer.json: 100%
 11.4M/11.4M [00:00<00:00, 40.2MB/s]
Unsloth 2025.2.12 patched 36 layers with 36 QKV layers, 36 O layers and 36 MLP layers.

import re
from datasets import load_dataset, Dataset
# Load and prep dataset
SYSTEM_PROMPT = """
Respond in the following format:
<reasoning>
...
</reasoning>
<answer>
...
</answer>
"""
def extract_xml_answer(text: str) -> str:
    """提取 <answer> 标签内的内容"""
    match = re.search(r"<answer>(.*?)</answer>", text, re.DOTALL)
    return match.group(1).strip() if match else text.strip()
def get_medical_questions(split="train") -> Dataset:
    """加载 FreedomIntelligence/medical-o1-reasoning-SFT 数据集并格式化"""
    data = load_dataset("FreedomIntelligence/medical-o1-reasoning-SFT", 'zh')[split]  
    def format_example(x):
        xml_answer = f"""\
<reasoning>
{x['Complex_CoT'].strip()}
</reasoning>
<answer>
{x['Response'].strip()}
</answer>"""
        
        return {
            'prompt': [
                {'role': 'system', 'content': SYSTEM_PROMPT},
                {'role': 'user', 'content': x['Question']}
            ],
            'answer': extract_xml_answer(xml_answer)  # 确保解析正确答案
        }
    data = data.map(format_example)
    return data
# 加载数据集
dataset = get_medical_questions()
# 奖励函数
def correctness_reward_func(prompts, completions, answer, **kwargs) -> list[float]:
    """检查模型输出的答案是否正确"""
    responses = [completion[0]['content'] for completion in completions]
    extracted_responses = [extract_xml_answer(r) for r in responses]
    return [2.0 if r == a else 0.0 for r, a in zip(extracted_responses, answer)]
def int_reward_func(completions, **kwargs) -> list[float]:
    """检查答案是否为整数"""
    responses = [completion[0]['content'] for completion in completions]
    extracted_responses = [extract_xml_answer(r) for r in responses]
    return [0.5 if r.isdigit() else 0.0 for r in extracted_responses]
def strict_format_reward_func(completions, **kwargs) -> list[float]:
    """严格格式检查：必须有换行符"""
    pattern = r"^<reasoning>\n.*?\n</reasoning>\n<answer>\n.*?\n</answer>\n$"
    responses = [completion[0]["content"] for completion in completions]
    return [0.5 if re.match(pattern, r) else 0.0 for r in responses]
def soft_format_reward_func(completions, **kwargs) -> list[float]:
    """宽松格式检查：允许不严格换行"""
    pattern = r"<reasoning>.*?</reasoning>\s*<answer>.*?</answer>"
    responses = [completion[0]["content"] for completion in completions]
    return [0.5 if re.match(pattern, r) else 0.0 for r in responses]
def count_xml(text) -> float:
    """检查 XML 结构完整性"""
    count = 0.0
    if text.count("<reasoning>\n") == 1:
        count += 0.125
    if text.count("\n</reasoning>\n") == 1:
        count += 0.125
    if text.count("\n<answer>\n") == 1:
        count += 0.125
        count -= len(text.split("\n</answer>\n")[-1]) * 0.001
    if text.count("\n</answer>") == 1:
        count += 0.125
        count -= (len(text.split("\n</answer>")[-1]) - 1) * 0.001
    return count
def xmlcount_reward_func(completions, **kwargs) -> list[float]:
    """计算 XML 结构完整性分数"""
    contents = [completion[0]["content"] for completion in completions]
    return [count_xml(c) for c in contents]

#输出
medical_o1_sft_Chinese.json: 100%
 64.8M/64.8M [00:00<00:00, 120MB/s]
Generating train split: 100%
 24772/24772 [00:01<00:00, 12662.63 examples/s]
Map: 100%
 24772/24772 [00:04<00:00, 5735.65 examples/s]

from trl import GRPOConfig, GRPOTrainer  # 导入 GRPO 训练配置和训练器


# 设置 GRPO（General Reinforcement Preference Optimization）的训练参数
training_args = GRPOConfig(
    use_vllm=True,  # 启用 vLLM 进行推理加速，提高训练效率
    learning_rate=5e-6,  # 设置学习率，较小值防止梯度爆炸
    adam_beta1=0.9,  # AdamW 优化器的 beta1 参数（动量项）
    adam_beta2=0.99,  # AdamW 优化器的 beta2 参数（平方梯度平滑项）
    weight_decay=0.1,  # 权重衰减，防止过拟合
    warmup_ratio=0.1,  # 预热步数比例，避免初始学习率过高导致的不稳定
    lr_scheduler_type="cosine",  # 采用余弦退火学习率调度
    optim="adamw_8bit",  # 使用 `8-bit AdamW` 优化器，节省显存
    logging_steps=1,  # 每 1 步记录日志
    bf16=is_bfloat16_supported(),  # 如果支持 bfloat16，则使用 bfloat16 精度
    fp16=not is_bfloat16_supported(),  # 如果不支持 bfloat16，则使用 fp16 精度
    per_device_train_batch_size=1,  # 每个设备的批量大小，显存不足时设为 1
    gradient_accumulation_steps=1,  # 梯度累积步数，增加到 4 可以平滑训练
    num_generations=8,  # 生成的样本数，减少该值可降低显存占用
    max_prompt_length=256,  # 最大输入提示长度
    max_completion_length=200,  # 最大模型输出长度
    # num_train_epochs=1,  # 训练 1 轮，实际使用时可以调整
    max_steps=10,  # 训练的最大步数（短时间测试用）正常为250
    save_steps=10,  # 每 10 步保存一次模型 正常为250
    max_grad_norm=0.1,  # 梯度裁剪，防止梯度爆炸
    report_to="none",  # 训练日志不上传到 W&B（Weights & Biases）
    output_dir="outputs",  # 训练结果的保存目录
)

trainer = GRPOTrainer(
    model=model,  # 使用 FastLanguageModel 加载的 Qwen2.5-3B-Instruct
    processing_class=tokenizer,  # 令牌化工具（Tokenizer）
    reward_funcs=[  # 定义奖励函数
        xmlcount_reward_func,  # XML 结构完整性评分
        soft_format_reward_func,  # 宽松格式检查
        strict_format_reward_func,  # 严格格式检查
        int_reward_func,  # 评估答案是否为整数
        correctness_reward_func,  # 检查答案是否正确
    ],
    args=training_args,  # 训练参数配置
    train_dataset=dataset,  # 训练数据集（医学问答数据）
)


trainer.train()  # 运行 GRPO 强化学习训练

==((====))==  Unsloth - 2x faster free finetuning | Num GPUs = 1
   \\   /|    Num examples = 24,772 | Num Epochs = 1
O^O/ \_/ \    Batch size per device = 8 | Gradient Accumulation steps = 1
\        /    Total batch size = 8 | Total steps = 10
 "-____-"     Number of trainable parameters = 119,734,272
 [10/10 03:46, Epoch 0/1]
Step
Training Loss
reward
reward_std
completion_length
kl
rewards / xmlcount_reward_func
rewards / soft_format_reward_func
rewards / strict_format_reward_func
rewards / int_reward_func
rewards / correctness_reward_func
....因输出太长，此处省略
TrainOutput(global_step=10, training_loss=2.103237093251664e-05, metrics={'train_runtime': 254.9268, 'train_samples_per_second': 0.314, 'train_steps_per_second': 0.039, 'total_flos': 0.0, 'train_loss': 2.103237093251664e-05})

text = tokenizer.apply_chat_template([
    {"role" : "user", "content" : "一个10岁男孩在患脓皮症后出现眼睑水肿、尿少和高血压等症状，作为医生，你认为首选的治疗药物是什么?"
},
], tokenize = False, add_generation_prompt = True)
from vllm import SamplingParams
sampling_params = SamplingParams(
    temperature = 0.8,
    top_p = 0.95,
    max_tokens = 1024,
)
output = model.fast_generate(
    [text],
    sampling_params = sampling_params,
    lora_request = None,
)[0].outputs[0].text
output

# 回答
Processed prompts: 100%|██████████| 1/1 [00:08<00:00,  8.94s/it, est. speed input: 6.94 toks/s, output: 37.73 toks/s]
'
脓皮症（也称为脓疱疮或脓疱病）是一种由细菌感染引起的皮肤感染，虽然脓皮症与眼睑水肿、尿少和高血压等症状联系不大，但是这些症状可能提示存在其他问题，如脓皮症并发了其他并发症，或者出现了脓皮症以外的其他疾病。因此，首先需要通过详细的病史询问、体检、实验室检查（如血液检查、尿液检查等）和影像学检查（如必要时进行）来明确诊断。\n\n如果确诊为脓皮症，治疗通常依赖抗生素。对于10岁男孩的脓皮症，首选的治疗药物通常是：\n\n1. **青霉素类**：对于大多数脓皮症患者，首选青霉素类抗生素治疗（如阿莫西林），尤其是对于儿童而言。\n2. **大环内酯类**：如果患者对青霉素过敏，可以考虑使用大环内酯类抗生素（如红霉素）。\n\n如果出现眼睑水肿、尿少和高血压等症状，可能需要更进一步的评估和治疗，这可能包括：\n\n- **利尿剂**：用于减少体内液体积聚，减轻水肿。\n- **血管紧张素转换酶抑制剂（ACEI）或血管紧张素受体拮抗剂（ARB）**：用于降低血压，尤其是如果是因为肾脏受损导致的高血压时。\n\n需要注意的是，这些症状可能提示存在并发症或其它系统性疾病，因此，必须首先由专业的医疗人员进行评估和治疗。请务必咨询专业医生，获得针对具体情况的专业治疗建议。在没有得到专业医生指导的情况下，不要自行给孩子使用任何药物。

model.save_lora("grpo_saved_lora")

text = tokenizer.apply_chat_template([
    {"role" : "system", "content" : SYSTEM_PROMPT},
    {"role" : "user", "content" : "一个10岁男孩在患脓皮症后出现眼睑水肿、尿少和高血压等症状，作为医生，你认为首选的治疗药物是什么?"},
], tokenize = False, add_generation_prompt = True)
from vllm import SamplingParams
sampling_params = SamplingParams(
    temperature = 0.8,
    top_p = 0.95,
    max_tokens = 1024,
)
output = model.fast_generate(
    text,
    sampling_params = sampling_params,
    lora_request = model.load_lora("grpo_saved_lora"),
)[0].outputs[0].text
output

#回答
Processed prompts: 100%|██████████| 1/1 [00:09<00:00,  9.17s/it, est. speed input: 7.42 toks/s, output: 34.04 toks/s]
'
<reasoning>\n根据病史描述，这个10男孩出现眼睑水肿、
尿少和高血压的症状，而脓皮症是引起这些症状的一个可能的原因，
但是最应该考虑的是水肿和高血压可能与感染相关的系统性炎症反应有关。
在这种情况下，首选的治疗药物需要考虑控制感染（脓皮症）、
控制炎症反应以及处理高血压。考虑到患儿的年龄，
首选的药物应该是安全的并且可以兼顾上述三个问题。
\n\n1. 抗生素：脓皮症是一种细菌感染，需要使用抗生素进行治疗。对于10岁的小男孩，青霉素类药物通常是首选，因为这类抗生素对儿童来说副作用较小。
\n\n2. 非甾体抗炎药（NSAIDs）：这类药物可以减轻炎症反应，帮助控制脓皮症。例如布洛芬，它相对安全，副作用较少，适用于儿童。
\n\n3. 抗高血压药物：如果伴有高血压，可能需要使用抗高血压药物，例如ACE抑制剂或者钙通道阻滞剂。然而，考虑到10岁儿童的特殊性，这些药物的使用需要谨慎，并且需要儿科医生的指导。
\n\n综上所述，治疗首选药物应该是青霉素类抗生素来控制脓皮症，并且根据病情可能还需要使用NSAIDs以及儿科医生指导下可能需要的抗高血压药物。
\n...\n<answer>\n首选治疗药物是青霉素类抗生素来控制脓皮症。其他可能需要的药物包括NSAIDs和抗高血压药物，但是需要儿科医生指导。