Hugging Face NLP Course 第 2 模块中英对照¶
课程: Hugging Face NLP Course
原链接: https://huggingface.co/learn/nlp-course/chapter2/1
版本: transformers >= 4.40.0
翻译时间: 2026-06-01
状态: 中英对照编译版
前续: 第 1 模块
📚 目录 (Table of Contents)¶
Module 2: Using Pretrained Models¶
模块 2: 使用预训练模型¶
- Chapter 2/1: Introduction to Transformers (Transformer 简介)
- Chapter 2/2: Loading Model and Tokenizer (加载模型和 Tokenizer)
- Chapter ⅔: Model Duplex (模型双工)
- Chapter 2/4: Fine-tuning a Model (微调模型)
- Chapter ⅖: Training a Model from Scratch (从头训练模型)
Chapter 2/1: Introduction to Transformers¶
第 2/1 章：Transformer 简介¶
Transformer 架构回顾 (Transformer Architecture Review)¶
English:
The Transformer architecture was introduced in the paper "Attention Is All You Need" (Vaswani et al., 2017). It has become the foundation for modern NLP models like BERT, GPT, T5, and many others.
中文:
Transformer 架构在论文《Attention Is All You Need》(Vaswani et al., 2017) 中引入。它已成为现代 NLP 模型（如 BERT、GPT、T5 等）的基础。
核心组件 (Core Components):
Transformer 架构
├── Encoder (编码器)
│ ├── Self-Attention Layer (自注意力层)
│ ├── Feed-Forward Layer (前馈层)
│ ├── Layer Normalization (层归一化)
│ └── Residual Connection (残差连接)
│
├── Decoder (解码器)
│ ├── Masked Self-Attention (掩码自注意力)
│ ├── Cross-Attention (交叉注意力)
│ └── Feed-Forward Layer (前馈层)
│
└── Encoder-Decoder (编码器 - 解码器)
├── Encoder 处理输入
└── Decoder 生成输出
三种架构类型 (Three Architecture Types):
| 类型 | 英文 | 用途 | 代表模型 |
|---|---|---|---|
| 编码器 | Encoder | 理解任务 | BERT, RoBERTa, XLM |
| 解码器 | Decoder | 生成任务 | GPT, CausalLM |
| 编码器 - 解码器 | Encoder-Decoder | 序列到序列 | T5, BART, Seq2Seq |
Chapter 2/2: Loading Model and Tokenizer¶
第 2/2 章：加载模型和 Tokenizer¶
Auto 类 (Auto Classes)¶
English:
Hugging Face provides Auto classes that automatically infer the correct model architecture from a pretrained model name.
中文:
Hugging Face 提供 Auto 类，可以从预训练模型名称自动推断正确的模型架构。
常用 Auto 类 (Common Auto Classes):
from transformers import (
AutoModel,
AutoModelForSequenceClassification,
AutoModelForQuestionAnswering,
AutoModelForTokenClassification,
AutoModelForSeq2SeqLM,
AutoTokenizer
)
# 加载 tokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
# 加载基础模型
model = AutoModel.from_pretrained("bert-base-uncased")
# 加载特定任务模型
model_cls = AutoModelForSequenceClassification.from_pretrained(
"bert-base-uncased",
num_labels=2
)
Auto 类映射 (Auto Class Mapping):
| Auto 类 | 用途 | 返回模型类型 |
|---|---|---|
| AutoModel | 基础特征提取 | BaseModel |
| AutoModelForSequenceClassification | 序列分类 | ClassificationHead |
| AutoModelForQuestionAnswering | 问答 | QAHead |
| AutoModelForTokenClassification | Token 分类 | TokenClassificationHead |
| AutoModelForSeq2SeqLM | 序列到序列语言模型 | Seq2SeqLMHead |
检查点与模型名称 (Checkpoints and Model Names)¶
English:
A checkpoint is a pretrained model that can be loaded by its name. The Hugging Face Hub contains thousands of checkpoints for various architectures and tasks.
中文:
检查点是可以由其名称加载的预训练模型。Hugging Face Hub 包含数千个各种架构和任务的检查点。
模型命名约定 (Model Naming Convention):
{model-architecture}-{training-dataset}-{language}-{size}
示例:
- bert-base-uncased: BERT 基础版，未分大小写，英语
- bert-large-cased: BERT 大版，分大小写，英语
- distilbert-base-multilingual-cased: DistilBERT，多语言，分大小写
- xlm-roberta-base: XLM-RoBERTa 基础版
查找模型 (Finding Models):
from transformers import pipeline
# 使用任务查找模型
# 访问 https://huggingface.co/models
# 示例：情感分析模型
classifier = pipeline(
"sentiment-analysis",
model="distilbert-base-uncased-finetuned-sst-2-english"
)
Chapter ⅔: Model Duplex¶
第 ⅔ 章：模型双工¶
编码器模型 (Encoder Models)¶
English:
Encoder models process the entire input sequence at once and are bidirectional. They are best suited for understanding tasks like classification and named entity recognition.
中文:
编码器模型一次性处理整个输入序列，并且是双向的。它们最适合理解任务，如分类和命名实体识别。
编码器模型特点 (Encoder Model Characteristics):
| 特点 | 说明 |
|---|---|
| 双向注意力 | 可以看到序列中所有位置的 token |
| 掩码语言模型 | 训练时随机掩码部分 token |
| 适用任务 | 分类、NER、问答 |
| 代表模型 | BERT, RoBERTa, XLM-R |
使用示例 (Usage Example):
from transformers import AutoModel, AutoTokenizer
# 加载编码器模型
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
# 编码文本
text = "Hello, I'm a BERT model"
inputs = tokenizer(text, return_tensors="pt")
# 获取隐藏状态
outputs = model(**inputs)
last_hidden_states = outputs.last_hidden_state
print(f"Input shape: {inputs['input_ids'].shape}")
print(f"Output shape: {last_hidden_states.shape}")
# 输出：[batch_size, sequence_length, hidden_size]
解码器模型 (Decoder Models)¶
English:
Decoder models process the sequence from left to right and use causal (masked) attention. They are best suited for generation tasks.
中文:
解码器模型从左到右处理序列，使用因果（掩码）注意力。它们最适合生成任务。
解码器模型特点 (Decoder Model Characteristics):
| 特点 | 说明 |
|---|---|
| 因果注意力 | 只能看到之前的 token，不能看到未来 |
| 自回归 | 逐个 token 生成 |
| 适用任务 | 文本生成、代码生成 |
| 代表模型 | GPT-2, GPT-3, CausalLM |
使用示例 (Usage Example):
from transformers import AutoModelForCausalLM, AutoTokenizer
# 加载解码器模型
model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
# 设置 padding token
tokenizer.pad_token = tokenizer.eos_token
# 生成文本
prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt")
# 自回归生成
outputs = model.generate(
**inputs,
max_length=50,
num_return_sequences=1,
do_sample=True,
temperature=0.7
)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)
编码器 - 解码器模型 (Encoder-Decoder Models)¶
English:
Encoder-decoder models combine both architectures. The encoder processes the input, and the decoder generates the output. They are used for sequence-to-sequence tasks.
中文:
编码器 - 解码器模型结合了两者的架构。编码器处理输入，解码器生成输出。它们用于序列到序列任务。
适用任务 (Applicable Tasks):
| 任务 | 输入 | 输出 | 示例模型 |
|---|---|---|---|
| 翻译 | 源语言句子 | 目标语言句子 | T5, BART |
| 摘要 | 长文本 | 摘要 | BART, T5 |
| 问答 | 问题 + 上下文 | 答案 | T5 |
| 文本到文本 | 任意文本 | 任意文本 | T5 |
使用示例 (Usage Example):
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
# 加载编码器 - 解码器模型
model_name = "t5-small"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
# 翻译任务
prompt = "translate English to French: Hello, how are you?"
inputs = tokenizer(prompt, return_tensors="pt")
# 生成翻译
outputs = model.generate(
**inputs,
max_length=50,
num_return_sequences=1
)
translation = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(translation) # Bonjour, comment allez-vous?
# 摘要任务
prompt = "summarize: " + long_text
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
summary = tokenizer.decode(outputs[0], skip_special_tokens=True)
Chapter 2/4: Fine-tuning a Model¶
第 2/4 章：微调模型¶
什么是微调？(What is Fine-tuning?)¶
English:
Fine-tuning is the process of taking a pretrained model and training it further on a specific task or dataset. This allows the model to adapt to domain-specific language and tasks.
中文:
微调是在特定任务或数据集上进一步训练预训练模型的过程。这使模型能够适应特定领域的语言和任务。
微调流程 (Fine-tuning Process):
为什么需要微调？(Why Fine-tune?):
| 原因 | 说明 |
|---|---|
| 领域适配 | 通用模型可能不理解专业术语 |
| 任务特定 | 预训练模型未针对特定任务优化 |
| 性能提升 | 微调后通常有更好的性能 |
| 数据效率 | 比从头训练需要更少数据 |
使用 Trainer API 微调 (Fine-tuning with Trainer API)¶
English:
The Trainer API provides a high-level interface for training and fine-tuning models with minimal code.
中文:
Trainer API 提供高级接口，用最少代码训练和微调模型。
完整微调示例 (Complete Fine-tuning Example):
from transformers import (
AutoTokenizer,
AutoModelForSequenceClassification,
TrainingArguments,
Trainer,
DataCollatorWithPadding
)
from datasets import load_dataset
import numpy as np
from sklearn.metrics import accuracy_score, f1_score
# 1. 加载数据集
dataset = load_dataset("imdb")
# 2. 加载 tokenizer 和模型
model_name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(
model_name,
num_labels=2
)
# 3. 预处理数据
def tokenize_function(examples):
return tokenizer(
examples["text"],
padding="max_length",
truncation=True,
max_length=128
)
tokenized_datasets = dataset.map(
tokenize_function,
batched=True,
remove_columns=["text"]
)
# 4. 数据整理器
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
# 5. 评估指标
def compute_metrics(eval_pred):
logits, labels = eval_pred
predictions = np.argmax(logits, axis=-1)
return {
"accuracy": accuracy_score(labels, predictions),
"f1": f1_score(labels, predictions, average="weighted")
}
# 6. 训练参数
training_args = TrainingArguments(
output_dir="./results",
learning_rate=2e-5,
per_device_train_batch_size=16,
per_device_eval_batch_size=64,
num_train_epochs=3,
weight_decay=0.01,
evaluation_strategy="epoch",
save_strategy="epoch",
load_best_model_at_end=True,
logging_dir="./logs",
logging_steps=100,
)
# 7. 创建 Trainer
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_datasets["train"],
eval_dataset=tokenized_datasets["test"],
tokenizer=tokenizer,
data_collator=data_collator,
compute_metrics=compute_metrics,
)
# 8. 开始微调
trainer.train()
# 9. 评估
results = trainer.evaluate()
print(f"Evaluation results: {results}")
# 10. 保存模型
trainer.save_model("./fine-tuned-model")
tokenizer.save_pretrained("./fine-tuned-model")
训练参数说明 (Training Arguments Explanation)¶
关键参数 (Key Parameters):
| 参数 | 默认值 | 说明 | 推荐设置 |
|---|---|---|---|
| learning_rate | 5e-5 | 学习率 | 2e-5 到 5e-5 |
| per_device_train_batch_size | 8 | 训练批次大小 | 16-32 (GPU 允许) |
| per_device_eval_batch_size | 8 | 评估批次大小 | 64 (可更大) |
| num_train_epochs | 3 | 训练轮数 | 3-5 |
| weight_decay | 0.0 | 权重衰减 | 0.01 (防止过拟合) |
| warmup_ratio | 0.0 | 预热比例 | 0.1 |
| evaluation_strategy | "no" | 评估策略 | "epoch" 或 "steps" |
| save_strategy | "epoch" | 保存策略 | "epoch" |
| load_best_model_at_end | False | 加载最佳模型 | True |
学习率调度器 (Learning Rate Schedulers):
training_args = TrainingArguments(
output_dir="./results",
learning_rate=2e-5,
lr_scheduler_type="linear", # 线性衰减
warmup_ratio=0.1, # 10% 预热
# 其他调度器: "cosine", "polynomial", "constant"
)
防止过拟合 (Preventing Overfitting)¶
English:
Fine-tuning can lead to overfitting, especially with small datasets. Several techniques can help prevent this:
中文:
微调可能导致过拟合，特别是在小数据集上。几种技术可以帮助防止这种情况：
正则化技术 (Regularization Techniques):
| 技术 | 实现方法 | 效果 |
|---|---|---|
| 权重衰减 | weight_decay=0.01 |
L2 正则化 |
| Dropout | 模型内置 | 随机失活神经元 |
| 早停 | load_best_model_at_end=True |
在验证集最佳时停止 |
| 学习率衰减 | lr_scheduler_type |
逐渐降低学习率 |
| 数据增强 | 回译、同义词替换 | 增加训练数据多样性 |
早停示例 (Early Stopping Example):
from transformers import EarlyStoppingCallback
training_args = TrainingArguments(
output_dir="./results",
evaluation_strategy="epoch",
save_strategy="epoch",
load_best_model_at_end=True,
num_train_epochs=10, # 设置较大的轮数
metric_for_best_model="f1", # 使用 F1 选择最佳模型
greater_is_better=True,
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
callbacks=[EarlyStoppingCallback(early_stopping_patience=3)], # 3 轮无改进则停止
)
Chapter ⅖: Training a Model from Scratch¶
第 ⅖ 章：从头训练模型¶
何时从头训练？(When to Train from Scratch?)¶
English:
Training a model from scratch is rarely necessary for NLP tasks. However, there are some scenarios where it might be appropriate:
中文:
对于 NLP 任务，从头训练模型很少是必要的。然而，在某些场景下可能是合适的：
适用场景 (Applicable Scenarios):
| 场景 | 说明 | 推荐度 |
|---|---|---|
| 新语言 | 预训练模型不支持该语言 | ⭐⭐⭐ |
| 特殊领域 | 医学、法律等高度专业领域 | ⭐⭐ |
| 研究目的 | 探索新架构或训练方法 | ⭐⭐⭐ |
| 数据充足 | 有数百万级别的训练数据 | ⭐⭐ |
| 一般任务 | 有现成预训练模型 | ❌ 不推荐 |
从头训练步骤 (Steps for Training from Scratch)¶
完整示例 (Complete Example):
from transformers import (
AutoTokenizer,
AutoModelForSequenceClassification,
TrainingArguments,
Trainer
)
from datasets import load_dataset
import torch
# 1. 准备数据
dataset = load_dataset("imdb")
# 2. 创建 tokenizer (从头训练需要新建)
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
# 或使用自定义词汇表
# 3. 定义模型配置
from transformers import BertConfig
config = BertConfig(
vocab_size=30522,
hidden_size=768,
num_hidden_layers=12,
num_attention_heads=12,
intermediate_size=3072,
num_labels=2,
)
# 4. 初始化随机模型 (非预训练)
model = AutoModelForSequenceClassification.from_config(config)
# 5. 预处理数据
def tokenize_function(examples):
return tokenizer(
examples["text"],
padding="max_length",
truncation=True,
max_length=128
)
tokenized_datasets = dataset.map(
tokenize_function,
batched=True,
remove_columns=["text"]
)
# 6. 训练参数 (从头训练需要更多轮数)
training_args = TrainingArguments(
output_dir="./scratch-training",
learning_rate=5e-5, # 可能需要更高学习率
per_device_train_batch_size=32,
per_device_eval_batch_size=64,
num_train_epochs=10, # 更多轮数
weight_decay=0.01,
evaluation_strategy="epoch",
save_strategy="epoch",
logging_steps=100,
)
# 7. 创建 Trainer
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_datasets["train"],
eval_dataset=tokenized_datasets["test"],
tokenizer=tokenizer,
)
# 8. 开始训练
trainer.train()
# 9. 评估
results = trainer.evaluate()
print(f"Results: {results}")
从头训练 vs 微调 (From Scratch vs Fine-tuning)¶
对比表 (Comparison):
| 方面 | 从头训练 | 微调 |
|---|---|---|
| 数据需求 | 数百万样本 | 数千样本 |
| 计算资源 | 需要 GPU 集群 | 单 GPU 即可 |
| 训练时间 | 数天到数周 | 数小时到数天 |
| 性能 | 可能较差 | 通常更好 |
| 成本 | 高 | 低 |
| 推荐场景 | 新语言、研究 | 大多数应用 |
成本估算 (Cost Estimation):
从头训练 BERT 大小模型:
- GPU 时间：~1000 GPU 小时
- 云服务成本：~$1000-5000
- 碳排放：~650 kg CO2
微调 BERT 大小模型:
- GPU 时间：~10 GPU 小时
- 云服务成本：~$10-50
- 碳排放：~6.5 kg CO2
结论：微调比从头训练高效 100 倍
🛠️ 实践练习 (Hands-on Exercises)¶
练习 1: 比较不同架构 (Exercise 1: Compare Architectures)¶
from transformers import pipeline
# 编码器模型
encoder_cls = pipeline(
"text-classification",
model="bert-base-uncased-finetuned-sst-2-english"
)
# 解码器模型
decoder_gen = pipeline(
"text-generation",
model="gpt2"
)
# 编码器 - 解码器模型
encdec_sum = pipeline(
"summarization",
model="t5-small"
)
# 测试相同输入
text = "The movie was fantastic! The acting was superb and the plot was engaging."
print("Encoder (Classification):")
print(encoder_cls(text))
print("\nDecoder (Generation):")
print(decoder_gen(text[:20], max_length=50)[0])
print("\nEncoder-Decoder (Summarization):")
print(encdec_sum(text)[0])
练习 2: 微调实践 (Exercise 2: Fine-tuning Practice)¶
# TODO: 使用以下数据集微调 DistilBERT
# 数据集：emotion (Hugging Face datasets)
# 任务：情感分类 (6 类)
# 指标：accuracy, macro_f1
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification
# 加载数据
dataset = load_dataset("emotion")
# TODO: 完成微调整个流程
# 1. 加载 tokenizer 和模型
# 2. 预处理数据
# 3. 设置训练参数
# 4. 创建 Trainer
# 5. 训练并评估
练习 3: 模型导出 (Exercise 3: Model Export)¶
# 将微调后的模型导出为 ONNX 格式
from transformers import AutoTokenizer, AutoModel
import torch
model_name = "./fine-tuned-model"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
# 创建示例输入
dummy_input = tokenizer(
"Example input for ONNX export",
return_tensors="pt"
)
# 导出为 ONNX
torch.onnx.export(
model,
(dummy_input['input_ids'], dummy_input['attention_mask']),
"model.onnx",
opset_version=11,
input_names=['input_ids', 'attention_mask'],
output_names=['last_hidden_state']
)
print("Model exported to model.onnx")
📖 关键术语表 (Glossary)¶
| English | 中文 | 定义 |
|---|---|---|
| Transformer | Transformer | 基于注意力的神经网络架构 |
| Encoder | 编码器 | 处理输入序列的部分 |
| Decoder | 解码器 | 生成输出序列的部分 |
| Self-attention | 自注意力 | 序列内元素间的注意力机制 |
| Fine-tuning | 微调 | 在预训练模型上进一步训练 |
| Trainer API | Trainer API | Hugging Face 高级训练接口 |
| TrainingArguments | 训练参数 | 训练配置参数集合 |
| Data collator | 数据整理器 | 批量数据预处理 |
| Early stopping | 早停 | 验证集性能下降时停止训练 |
| Overfitting | 过拟合 | 在训练集上表现好，测试集差 |
| Checkpoint | 检查点 | 预训练模型文件 |
| Hub | Hub | Hugging Face 模型共享平台 |
翻译完成时间: 2026-06-01
原课程模块: Module 2 (Chapter 2/1 - ⅖)
前续: 第 1 模块
后续: 第 3 模块 (待创建)