Fine-tuning BERT for Target and Frame Identification

This repository provides code for fine-tuning BERT-based models on Frame-Semantic Parsing tasks using the FrameNet 1.7 dataset.

The project focuses on the first stages of Frame-Semantic Parsing, and includes implementations for the following tasks:

1. Target Identification (TI) Detection of text spans (targets) in a sentence that evoke semantic frames.

2. Frame Classification (FC) Given a sentence with a marked target span, predict the semantic frame evoked by the target.

3. Frame Identification (FI) Two approaches are implemented:

    a. End-to-end detection: Direct prediction of frame labels in an input sentence without explicit target marking, treated as a sequence labeling task.

    b. Pipeline-based detection: Sequential application of the TI and FC models. The predicted target spans from the TI model are fed into the FC model to determine the corresponding semantic frame, producing the final Frame Identification output.

The repository includes the following components:

• Data preprocessing and parsing utilities for FrameNet 1.7.
• Tokenization, training and evaluation scripts for the three tasks (TI, FC, FI).
• Inference utilities for both standalone tasks and joint pipelines.

Usage

1. Preprocessing

Download and prepare the FrameNet 1.7 data from nltk.corpora.framenet_v17.

python scripts/run_preprocessing.py [--include_exemplars] [--output_dir <path_to_output_directory>]

2. Tokenization

Tokenize the preprocessed data for a specific task and model:

python scripts/run_tokenization.py --task target-id|frame-id|frame-cls --model_name bert-base-cased|roberta-base|distilbert-base-cased

3. Training

Train a model for a specific task. Different training strategies can be applied:

• Baseline: Train on the base train split only.
• Joint: Train on the base train split plus exemplars sentences from FrameNet.
• Sequential: Pre-train on the exemplars sentences, then fine-tune on the base train split.
• Oversampling: Same as joint, but oversample the base train split.

Additionally, two loss functions are available during training:

• Cross-entropy (CE): Standard cross-entropy loss.
• Weighted cross-entropy (WCE): Cross-entropy loss with class weights to address class imbalance.

python scripts/run_train.py --task target-id|frame-id|frame-cls \
                            [--model_name bert-base-cased|roberta-base|distilbert-base-cased] \
                            [--strategy baseline|joint|sequential|oversampling] \
                            [--loss weighted_cross_entropy|cross_entropy] \
                            [--oversample_factor <int>] \
                            [--seed <int>] \
                            [--batch_size <int>] \
                            [--learning_rate <float>] \
                            [--epochs <int>] \
                            [--warmup_ratio <float>] \
                            [--weight_decay <float>] \
                            [--load_best_model_at_end] \
                            [--metric_for_best_model f1|precision|recall]
                            # check the script for more options and default values

Or using the API:

from src.config import TrainingConfig
from src.training import train

config = TrainingConfig(
    pretrained_model="bert-base-cased",
    task="target-id",
    strategy="joint",
    dataset="../data/fn17/tokenized/target-id/",
    output_dir="../models/target-id/bert-base-cased/joint/weighted_cross_entropy_loss",
    seed=42,
    batch_size=32,
    learning_rate=2e-5,
    epochs=4,
    warmup_ratio=0.05,
    weight_decay=0.01,
    loss="weighted_cross_entropy",
    oversample_factor=10,
    load_best_model_at_end=True,
    metric_for_best_model="f1",
)

trainer, tokenized_dataset = train(config)

if tokenized_dataset["test"] is not None:
    test_metrics = trainer.predict(tokenized_dataset["test"])
    print(f"Results on test set: {test_metrics.metrics}")

4. Evaluation

Assess model performance:

python scripts/run_eval.py --task target-id|frame-id|frame-cls --model_path <path_to_trained_model>

5. Inference

from src.inference import FrameIdentifier, FrameClassifier, FrameIdentificationPipeline, TargetIdentifier
from src.visualization import render_predictions

sentences = [
    "Hoover Dam played a major role in preventing Las Vegas from drying up.",
    "The hallway smelt of boiled cabbage and old rag mats.",
    "The price of bananas rose 5%",
]

# Target identification
target_identifier = TargetIdentifier(model_path="<path_to_trained_target_id_model>")
preds = target_identifier.predict(sentences)
render_predictions(preds)

# Frame Classification
frame_classifier = FrameClassifier(model_path="<path_to_trained_frame_cls_model>")
sentence = "Hoover Dam played a major role in preventing Las Vegas from [TGT] drying up [/TGT]."
preds = frame_classifier.predict(sentence)
preds.to_dict()

# End-to-end Frame Identification
frame_identifier = FrameIdentifier(model_path="<path_to_trained_frame_id_model>")
preds = frame_identifier.predict(sentence, top_k=3, lu_filter=True)
render_predictions(preds)

# Pipeline-based Frame Identification
pipeline = FrameIdentificationPipeline(target_identifier, frame_identifier)
preds = pipeline.run(sentences)
render_predictions(preds)

Performance

The experiments follow the same train/dev/test document splits and evaluation methodology as Open-Sesame (Swayamdipta et al., 2017). The results for TI and FC are compared with Open-Sesame and Chanin (2023).

The results reported here correspond to models trained with the best-performing loss for each task.

Target Identification

Strategy	Model	dev F1	test F1
baseline_wce	bert-base-cased	0.81	0.76
joint_wce	bert-base-cased	0.73	0.62
sequential_wce	bert-base-cased	0.81	0.76
oversampling_wce	bert-base-cased	0.73	0.62
Swayamdipta (2017)	Open-Sesame	0.80	0.73
Chanin (2023)	t5-base	0.78	0.74

Frame Classification

Strategy	Model	dev F1	test F1
baseline_wce	bert-base-cased	0.88	0.85
joint_wce	bert-base-cased	0.86	0.84
sequential_wce	bert-base-cased	0.90	0.89
oversampling_wce	bert-base-cased	0.86	0.84
Swayamdipta (2017)	Open-Sesame	0.90	0.87
Chanin (2023)	t5-base	0.91	0.89

Frame Identification (End-to-end)

Strategy	Model	dev F1	test F1
baseline_ce	bert-base-cased	0.71	0.63
joint_ce	bert-base-cased	0.47	0.47
sequential_ce	bert-base-cased	0.76	0.70
oversampling_wce	bert-base-cased	0.52	0.52

Frame Identification (Pipeline)

Using the best performing models for TI and FC (i.e. bert-base-cased_baseline_wce and bert-base-cased_sequential_wce, respectively):

Model	test F1
bert-base-cased	0.70