Information
The problem arises in chapter:
Making Transformers Efficient in Production Describe the bug while training I am getting proper F1 score of 0.755940 image
while finding best fit value of alpha and temperature value for NER task f1 score is 0.096029 which is less than 0.1 image
To Reproduce
Steps to reproduce the behavior:
- compute metric is same as in huggingface NER tutorial.
- Hyperparameter are for alpha and temperature.
def hp_space(trial):
return {"alpha": trial.suggest_float("alpha", 0, 1),
"temperature": trial.suggest_int("temperature", 2, 20)}
best_run = distil_roberta_trainer.hyperparameter_search(
n_trials=12, direction="maximize",backend="optuna", hp_space=hp_space)
Expected behavior
After the hyperparameter search the F1 score should be higher than baseline.
More details:https://github.com/nlp-with-transformers/notebooks/issues/115#issue-1860567807
Hyperparameter optimization giving less F1 score less than 0.1 while default value giving 0.75.
Want good parameter value which gives better result than baseline.
