Zero-Shot Text Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "mathematical-biography",
   "metadata": {},
   "source": [
    "# Adventures in Zero-Shot Text Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "subtle-metropolitan",
   "metadata": {},
   "source": [
    "Transfer learning has had an enormous impact in Natural Language Processing. Thanks to models like BERT, it is now possible to train more accurate NLP models than before, and typically do so with less labeled data. Now that finetuning language models has become the standard procedure in NLP, it’s only natural to get curious and ask: do we need any task-specific labeled training items at all?\n",
    "\n",
    "In this notebook, we investigate two available models for zero-shot text classification and evaluate how they perform. The code for this article is available in [our repository of NLP notebooks](https://nlptown.github.io/nlp-notebooks/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "extreme-costs",
   "metadata": {},
   "source": [
    "## Zero-shot text classification\n",
    "\n",
    "Zero-shot and few-shot NLP models take transfer learning to the extreme: their goal is to make predictions for an NLP task without having seen one single labeled item (for zero-shot learning), or very few such items (for few-shot learning) specific to that task. The most well-known example is doubtlessly [OpenAI’s GPT-3](https://arxiv.org/abs/2005.14165), which has proved to be a very successful few-shot learner for a wide range of applications. While running GPT-3 lies beyond the means of most developers, luckily several smaller alternatives are available.\n",
    "\n",
    "In 2020, [Flair](https://github.com/flairNLP/flair) and [Transformers](https://huggingface.co/transformers/), two of the most popular NLP libraries, both added zero-shot classification to their offering. Flair, on the one hand, makes use of a so-called TARS classifier, short for [Text-Aware Representation of Sentences](https://kishaloyhalder.github.io/pdfs/tars_coling2020.pdf), which can be run with just a few lines of code:\n",
    "\n",
    "```\n",
    "classifier = TARSClassifier.load('tars-base')\n",
    "sentence = Sentence('Your example text')\n",
    "classifier.predict_zero_shot(sentence, [label1, label2, …])\n",
    "```\n",
    "\n",
    "Transformers, on the other hand, makes it possible to use a range of models from the [Hugging Face model hub](https://huggingface.co/models) in their `zero-shot-classification` pipeline:\n",
    "\n",
    "```\n",
    "classifier = pipeline('zero-shot-classification', model=\"your-nli-model\", device=0)\n",
    "classifier('Your example text', [label1, label2, …])\n",
    "```\n",
    "\n",
    "Despite the obvious similarities, the two implemented classifiers approach zero-shot text classification quite differently.\n",
    "\n",
    "The zero-shot pipeline in the Transformers library treats text classification as natural language inference (NLI). This approach was pioneered by [Yin et al. in 2019](https://arxiv.org/abs/1909.00161). In NLI, a model takes two sentences as input &mdash; a premise and a hypothesis &mdash; and decides whether the hypothesis follows from the premise (`entailment`), contradicts it (`contradiction`), or neither (`neutral`). For example, the premise _David killed Goliath_ entails the hypothesis _Goliath is dead_, is contradicted by _Goliath is alive_ and doesn’t allow us to draw any conclusions about _Goliath is a giant_. This NLI template can be reused for text classification by taking the text we’d like to label as the premise, and rephrasing every candidate class as a hypothesis. For a task such as polarity classification, the premise could be an opinion like _I loved this movie_, with the hypotheses _This sentence is positive_, _This sentence is negative_ or _This sentence is neutral_. The classifier will then determine the relationship between the premise and every hypothesis. In single-label classification, all resulting `entailment` scores are softmaxed to identify the single most probable class; in multi-label classification, the scores for `entailment` and `contradiction` are softmaxed for every label independently, so that several relevant labels can be identified.\n",
    "\n",
    "The TARS classifier in the Flair library takes a different course. Similar to the previous approach, it abstracts away from the specificities of individual classification tasks by feeding both the label and the text as input to a BERT classifier, separated by the `[SEP]` token. The main difference lies in the fact that this BERT model is not finetuned for NLI, but for a generic version of text classification. This is done by training the model to label every input pair as either true or false. To make sure it can handle a variety of classification tasks, Flair’s TARS classifier is finetuned on nine different datasets, with applications ranging from polarity to topic classification. For single-label classification, only the class with the highest score for `True` is selected as the final prediction; for multi-label classification, all classes with the prediction `True` are returned.\n",
    "\n",
    "Although both approaches to zero-shot classification sound very attractive, they share one disadvantage: in contrast to traditional text classification, each input text requires several forward passes through the model &mdash; one for each candidate label. The models are therefore less computationally efficient than more traditional text classifiers. Still, if they can bypass the need for expensive data labeling, for many applications this may be a small price to pay."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "regional-climb",
   "metadata": {},
   "source": [
    "## Data\n",
    "\n",
    "The more pressing question is therefore how well zero-shot text classification works exactly. To find out, we evaluated the classifiers above on five different tasks, from topic classification to sentiment analysis. We used four datasets that are all available from the [Hugging Face datasets hub](https://huggingface.co/datasets),\n",
    "making sure that none of these datasets was used to finetune Flair’s TARS classifier. From each we selected 1,000 random test items:\n",
    "\n",
    "- [yahoo_answers_topics](https://huggingface.co/datasets/yahoo_answers_topics): questions and answers from Yahoo Answers, classified into 10 topics, such as `Society & Culture` and `Science & Mathematics`. As the input to the model we use the best answer only (without the question).\n",
    "- [banking 77](https://huggingface.co/datasets/banking77): a set of online user queries from the banking domain, each labeled with one of 77 intents. This is a challenging dataset, as the intents (such as `card_about_to_expire` and `card_not_working`) are very fine-grained.\n",
    "- [tweet_eval](https://huggingface.co/datasets/tweet_eval): English tweets labeled for a variety of tasks. We tested if the models can predict the emotion &mdash; `anger`, `joy`, `optimism` or `sadness` &mdash; and the sentiment polarity of the tweets &mdash; `positive`, `negative` or `neutral`.\n",
    "- [financial_phrasebank](https://huggingface.co/datasets/financial_phrasebank): financial news sentences (such as _Sales have risen in other export markets_) with a polarity label &mdash; `positive`, `negative` or `neutral`. We only selected sentences for which all annotators agreed on the label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cheap-chocolate",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Reusing dataset yahoo_answers_topics (/home/yves/.cache/huggingface/datasets/yahoo_answers_topics/yahoo_answers_topics/1.0.0/a1bc78ef81f1057593777a45c199fada1fd6decb8a3d9f5aea63b47dc884b0dd)\n",
      "Couldn't find file locally at banking77/banking77.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.4.1/datasets/banking77/banking77.py.\n",
      "The file was picked from the master branch on github instead at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/banking77/banking77.py.\n",
      "Using custom data configuration default\n",
      "Reusing dataset banking77 (/home/yves/.cache/huggingface/datasets/banking77/default/1.1.0/17ffc2ed47c2ed928bee64127ff1dbc97204cb974c2f980becae7c864007aed9)\n",
      "No config specified, defaulting to: amazon_reviews_multi/all_languages\n",
      "Reusing dataset amazon_reviews_multi (/home/yves/.cache/huggingface/datasets/amazon_reviews_multi/all_languages/1.0.0/f3357bd271e187385a38574fe31b8fb10055303f67fa9fce55e84d08c4870efd)\n",
      "Reusing dataset financial_phrasebank (/home/yves/.cache/huggingface/datasets/financial_phrasebank/sentences_allagree/1.0.0/8573a5b5922d152c7b77924429a18b5546458c179db2685eb266b227d51d1b6b)\n",
      "Reusing dataset tweet_eval (/home/yves/.cache/huggingface/datasets/tweet_eval/sentiment/1.1.0/79e21f7659e902ea14f624232219492d972fe5e0f9d8c94363acc7f916a6be48)\n",
      "Reusing dataset tweet_eval (/home/yves/.cache/huggingface/datasets/tweet_eval/emotion/1.1.0/79e21f7659e902ea14f624232219492d972fe5e0f9d8c94363acc7f916a6be48)\n"
     ]
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "yahoo = load_dataset('yahoo_answers_topics')\n",
    "banking = load_dataset('banking77')\n",
    "amazon = load_dataset('amazon_reviews_multi')\n",
    "financial = load_dataset('financial_phrasebank', 'sentences_allagree')\n",
    "tweets = load_dataset('tweet_eval', 'sentiment')\n",
    "tweets_emotion = load_dataset('tweet_eval', 'emotion')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "planned-alcohol",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def clean(label):\n",
    "    label = re.sub(\"([a-z])([A-Z])\", \"\\\\1 \\\\2\", label)\n",
    "    label = label.replace(\"_\", \" \")\n",
    "    return label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "activated-centre",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "random.seed(1)\n",
    "\n",
    "def sample_test_data(texts,  labels, size):\n",
    "    \n",
    "    data = list(zip(texts, labels))\n",
    "    data = [item for item in data if len(item[0].strip()) > 0]\n",
    "    \n",
    "    random.shuffle(data)\n",
    "\n",
    "    texts, labels = zip(*data)\n",
    "    \n",
    "    return texts[:size], labels[:size], texts[size:], labels[size:]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "animated-clearing",
   "metadata": {},
   "outputs": [],
   "source": [
    "yahoo_test_texts, yahoo_test_labels, _, _ = sample_test_data(yahoo['test']['best_answer'], yahoo['test']['topic'], 1000)\n",
    "banking_test_texts, banking_test_labels, _, _ = sample_test_data(banking['test']['text'], banking['test']['label'], 1000)\n",
    "financial_test_texts, financial_test_labels, financial_train_texts, financial_train_labels = sample_test_data(financial['train']['sentence'], financial['train']['label'], 1000)\n",
    "tweets_test_texts, tweets_test_labels, _, _ = sample_test_data(tweets['test']['text'], tweets['test']['label'], 1000)\n",
    "tweets_emotion_test_texts, tweets_emotion_test_labels, _, _ = sample_test_data(tweets_emotion['test']['text'], tweets_emotion['test']['label'], 1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "competent-renaissance",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [\n",
    "    {\n",
    "        \"name\": \"yahoo_answers_topics\", \n",
    "        \"test_texts\": yahoo_test_texts, \n",
    "        \"test_labels\": yahoo_test_labels, \n",
    "        \"train_texts\": yahoo['train']['best_answer'], \n",
    "        \"train_labels\": yahoo['train']['topic'], \n",
    "        \"class_names\": [clean(label) for label in yahoo['test'].features['topic'].names]\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"banking77\", \n",
    "        \"test_texts\": banking_test_texts,\n",
    "        \"test_labels\": banking_test_labels, \n",
    "        \"train_texts\": banking['train']['text'], \n",
    "        \"train_labels\": banking['train']['label'], \n",
    "        \"class_names\": [clean(label) for label in banking['test'].features['label'].names]\n",
    "    },    \n",
    "    {\n",
    "        \"name\": \"tweet_eval: emotion\", \n",
    "        \"test_texts\": tweets_emotion_test_texts, \n",
    "        \"test_labels\": tweets_emotion_test_labels, \n",
    "        \"train_texts\": tweets_emotion['train']['text'], \n",
    "        \"train_labels\": tweets_emotion['train']['label'], \n",
    "        \"class_names\": [clean(label) for label in tweets_emotion['test'].features['label'].names]\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"tweet_eval: sentiment\", \n",
    "        \"test_texts\": tweets_test_texts, \n",
    "        \"test_labels\": tweets_test_labels,\n",
    "        \"train_texts\": tweets['train']['text'], \n",
    "        \"train_labels\": tweets['train']['label'],\n",
    "        \"class_names\": [clean(label) for label in tweets['test'].features['label'].names]\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"financial_phrasebank\", \n",
    "        \"test_texts\": financial_test_texts, \n",
    "        \"test_labels\": financial_test_labels, \n",
    "        \"train_texts\": financial_train_texts, \n",
    "        \"train_labels\": financial_train_labels, \n",
    "        \"class_names\":  [clean(label) for label in financial['train'].features['label'].names]\n",
    "    }    \n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "liable-apollo",
   "metadata": {},
   "source": [
    "## The models\n",
    "\n",
    "We used three different zero-shot text classifiers in our tests: Flair’s `TARSClassifier`, and two Transformers models finetuned for NLI: `bart-large-mnli` and `roberta-large-mnli`. The graph below shows their accuracies on the five tasks. The results are as varied as the datasets, but one pattern is immediately clear: the best model always takes the NLI approach. For the Yahoo Answers topics, Bart gives the best accuracy (39.2%), followed by TARS and Roberta, which both obtain 27.5%. Although the banking task appears much more challenging at first sight, the NLI models perform even better here: they both classify over 41% of the test items correctly, leaving the TARS classifier far behind. On four-way tweet emotion classification, both Bart (73.6%) and Roberta (71.9%) perform surprisingly well and easily beat TARS (32.3%).\n",
    "\n",
    "The two polarity tasks deserve some additional explanation. Because our first evaluation run showed very low scores for TARS (accuracies below the random baseline of 33%), we took a closer look at the results and found that in most cases, TARS failed to predict a single label for the news sentences and tweets. To fix this, we performed a second run, where we made TARS return `neutral` for every sentence without a label. It is those scores you see below.\n",
    "For both polarity tasks, Roberta gives the best results, with 54.0% accuracy for the tweets and 58.8% for the\n",
    "financial news sentences. TARS and Bart obtain a similar result on the tweets, with 48.2% and 49.0% accuracy, respectively. On the financial news sentences, TARS is better, with 51.7% against 38.8%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "governmental-worst",
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.models.text_classification_model import TARSClassifier\n",
    "from flair.data import Sentence\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "def evaluate_flair(dataset, default_name='neutral'):\n",
    "\n",
    "    classifier = TARSClassifier.load('tars-base')\n",
    "        \n",
    "    total, correct = 0, 0\n",
    "    for item, gold_label_idx in tqdm(zip(dataset[\"test_texts\"], dataset[\"test_labels\"]), total = len(dataset[\"test_texts\"])):\n",
    "        sentence = Sentence(item)\n",
    "        classifier.predict_zero_shot(sentence, dataset[\"class_names\"])\n",
    "        sorted_labels = sorted(sentence.to_dict()['labels'], key=lambda k: k['confidence'], reverse=True)\n",
    "        gold_label = dataset[\"class_names\"][gold_label_idx]\n",
    "        if len(sorted_labels) > 0:\n",
    "            predicted_label = sorted_labels[0]['value']\n",
    "        else:\n",
    "            predicted_label = default_name\n",
    "        if predicted_label == gold_label:\n",
    "            correct += 1\n",
    "        total += 1\n",
    "        \n",
    "    return correct/total"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "married-wesley",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from transformers import pipeline\n",
    "\n",
    "def evaluate_huggingface(dataset, template=None, model='base'):\n",
    "\n",
    "    if model == 'base':\n",
    "        classifier = pipeline(\"zero-shot-classification\", device=0)\n",
    "    else:\n",
    "        classifier = pipeline(\"zero-shot-classification\", model=\"roberta-large-mnli\", device=0)\n",
    "    \n",
    "    correct = 0\n",
    "    predictions, gold_labels = [], []\n",
    "    for text, gold_label_idx in tqdm(zip(dataset[\"test_texts\"], dataset[\"test_labels\"]), total=len(dataset[\"test_texts\"])):\n",
    "\n",
    "        if template is not None:\n",
    "            result = classifier(text, dataset[\"class_names\"], multi_label=False, template=template)\n",
    "        else:\n",
    "            result = classifier(text, dataset[\"class_names\"], multi_label=False)\n",
    "        predicted_label = result['labels'][0]\n",
    "        \n",
    "        gold_label = dataset[\"class_names\"][gold_label_idx]\n",
    "        \n",
    "        predictions.append(predicted_label)\n",
    "        gold_labels.append(gold_label)\n",
    "        \n",
    "        if predicted_label == gold_label:\n",
    "            correct += 1\n",
    "            \n",
    "    accuracy = correct/len(predictions)\n",
    "    return accuracy\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "italian-murder",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "desirable-friendly",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "yahoo_answers_topics\n",
      "2021-05-25 10:35:55,743 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "19b803bf10694d49974cab3253799e41",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Flair: 0.275\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6a2eb5c4668b4b6a8f120ac7181f0c72",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Huggingface Bart 0.392\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at roberta-large-mnli were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
      "- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "99096d6dcd7b430ba1c652b83c65f2aa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Huggingface Roberta 0.275\n",
      "banking77\n",
      "2021-05-25 10:41:14,247 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "27cae30822d6401ebadfbad915fb71ff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Flair: 0.109\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e528febb245840529e90e079202747bb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Huggingface Bart 0.41\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at roberta-large-mnli were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
      "- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0e40d3cfcd22412ca1992b1bb9172774",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Huggingface Roberta 0.413\n",
      "tweet_eval: emotion\n",
      "2021-05-25 10:52:26,358 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2694f0d258df475b8d25a8af08b9dab9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Flair: 0.323\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7e1a14895c734b74bb497619cb51a1d1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Huggingface Bart 0.736\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at roberta-large-mnli were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
      "- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9b7715aecf9e4f68b25990ca53b2e3b6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Huggingface Roberta 0.719\n",
      "tweet_eval: sentiment\n",
      "2021-05-25 10:53:59,809 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ee8bfe8bee1446ee80a83e7661a9eea8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Flair: 0.482\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9bfb8f18c8724fb9890acabb2aaa3abe",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Huggingface Bart 0.49\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at roberta-large-mnli were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
      "- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c8162d7da0ed4f0d9e5fc55474bfc193",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Huggingface Roberta 0.54\n",
      "financial_phrasebank\n",
      "2021-05-25 10:55:23,729 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1421e3a88197445e9084633c5949b919",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Flair: 0.517\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8a074fb9c93e4b58a7d9d8b71f247546",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Huggingface Bart 0.388\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at roberta-large-mnli were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
      "- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "480ea599ca5842d488f22469fe56e750",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Huggingface Roberta 0.588\n"
     ]
    }
   ],
   "source": [
    "results = {}\n",
    "\n",
    "for dataset in data:\n",
    "    print(dataset['name'])\n",
    "    results[dataset['name']] = {}\n",
    "    \n",
    "    flair_acc = evaluate_flair(dataset)\n",
    "    results[dataset['name']]['Flair TARS'] = flair_acc\n",
    "    print(\"Flair:\", flair_acc)\n",
    "    \n",
    "    huggingface_acc = evaluate_huggingface(dataset)\n",
    "    results[dataset['name']]['Transformers Bart'] = huggingface_acc\n",
    "    print(\"Huggingface Bart\", huggingface_acc)\n",
    "\n",
    "    huggingface_acc_roberta = evaluate_huggingface(dataset, model='roberta')\n",
    "    results[dataset['name']]['Transformers Roberta'] = huggingface_acc_roberta\n",
    "    print(\"Huggingface Roberta\", huggingface_acc_roberta)\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "suspended-coffee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsIAAAGcCAYAAADTfsbDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAy0ElEQVR4nO3de5wU1Z3//9eHQS6KUVA23wgqLIt4ARZ0QNCoIN6yRNSgkfxcw6jourtizM01GiNr1Fwwxm+yJq7mi+iul0SNBF2jxtuKUcQREPHCQhAUYlYkEa8ol/P7o2rGdphheqDHAer1fDzmMdXVp6pOdVdXvev06apIKSFJkiQVTbu2roAkSZLUFgzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkNq31YJ33XXX1KtXr7ZavCRJkgrimWeeeSOl1L3h+DYLwr169aK2tratFi9JkqSCiIiljY23a4QkSZIKySAsSZKkQjIIS5IkqZDarI+wJElSU9asWcOyZctYvXp1W1dFW5FOnTrRs2dPtttuu7LKG4QlSdIWZ9myZey444706tWLiGjr6mgrkFJi5cqVLFu2jN69e5c1jV0jJEnSFmf16tXssssuhmCVLSLYZZddWvQtgkFYkiRtkQzBaqmWbjMGYUmSJBWSQViSJG3xIqKif+Woqqpi0KBB9X9Llizh0Ucf5fOf//xGp6utreXcc8/daJnnnnuufr7dunWjd+/eDBo0iCOOOAKAuXPnEhHcd999jdapf//+HHvssbz55psArF+/nnPPPZf+/fszYMAAhgwZwssvv1zWehaZP5aTJElqROfOnZk7d+7Hxi1ZsqTZ6aqrq6murt5g/Nq1a2nfPoteAwYMqJ93TU0Nn//85znxxBPry95666189rOf5dZbb+WYY45ptE7jx4/nmmuu4aKLLuKXv/wlf/zjH5k3bx7t2rVj2bJl7LDDDi1b4QKyRViSJGkTzJo1i+HDhzN48GAOOuggFixYAPCxVuNJkyZx6qmncvDBB3PqqaeWNd+UErfffjtTp07ld7/7XZM//ho+fDjLly8H4LXXXuMzn/kM7dpl0a5nz5507dp1c1dxm2cQliRJasT7779f333hhBNO2OD5vffemxkzZjBnzhwuvfRSLrzwwkbn88ILL/Dggw9y6623lrXcJ554gt69e9OnTx9GjBjBf/3Xf21QZt26dTz00EOMGTMGgC9+8YvcfffdDBo0iK9//evMmTOnBWtaXHaNkCRJakRjXSNKrVq1ivHjx7Nw4UIigjVr1jRabsyYMXTu3Lns5d56662MGzcOgHHjxnHTTTcxduxY4KNwvnz5cvbZZx+OPPJIIGsBXrBgAQ8//DAPP/wwo0aN4vbbb2fUqFFlL7eIbBGWJEnaBBdffDEjR45k/vz53H333U12YWhJX91169Zx5513cumll9KrVy8mTpzIfffdx9tvvw18FM6XLl1KSolrrrmmftqOHTvyuc99jsmTJ3PhhRcybdq0zVq/IjAIS5IkbYJVq1bRo0cPAKZOnVqReT700EMMHDiQV199lSVLlrB06VLGjh3LXXfd9bFy22+/PT/5yU/40Y9+xNq1a5k9ezZ//OMfgewKEvPmzWPPPfesSJ22ZQZhSZK0xUspVfSvEs4//3y+9a1vMXjwYNauXVuRed56660b9EceO3Zso/2LBw8ezMCBA7n11lt5/fXXOfbYY+nfvz8DBw6kffv2nHPOORWp07YsKrUxtFR1dXWqra1tk2VL0pbkhjNHtsp8T7v+kVaZr/RJePHFF9lnn33auhraCjW27UTEMymlDa5pZ4uwJEmSCskgLEmSpEIyCEuSJKmQDMKSJEkqJIOwJEmSCskgLEmSpELyFsuSJGmLV+nLDG7s8oIrV66svzXxn/70J6qqqujevTsAs2bNokOHDhWrx0svvcS4ceOICO644w769OlTsXlX0ogRI3jttdfo3LkzH3zwAV/96lc566yzWjSPqVOnctRRR7Hbbru1Ui1bziAsSZJUYpdddmHu3LkATJo0iS5duvCNb3yj/vm1a9fSvn1lItS0adM48cQT+fa3v11W+bobgrRrV5kv9VuyLjfffDPV1dX8+c9/pk+fPtTU1JR9UrBu3TqmTp1K//79DcKSpE/AzEmtM99hrTRfaQtWU1NDp06dmDNnDgcffDDjxo3jK1/5CqtXr6Zz587ccMMN9OvXj6lTpzJ9+nTee+89/vCHP3DCCSfwwx/+kHXr1nHGGWdQW1tLRHD66afTr18/rr76aqqqqnjooYd45JFHuOqqq5gyZQoAEyZM4LzzzmPJkiUcffTRHHjggTzzzDP87Gc/4x/+4R8YNmwYTzzxBEOGDOG0007jkksu4fXXX+fmm29m6NChvPvuu0ycOJH58+ezZs0aJk2axHHHHcfUqVP59a9/zTvvvMO6deu47bbbOPnkk3nrrbdYu3YtP//5zznkkEOafC3eeecddthhB6qqqgD4x3/8R55++mnef/99TjzxRP71X/8VgF69enHyySfzu9/9jq997WvU1tZyyimn0LlzZ5588kk6d+7c+m9cMwzCkiRJZVi2bBlPPPEEVVVVvPXWW8yYMYP27dvz4IMPcuGFF3LnnXcCMHfuXObMmUPHjh3p168fEydO5PXXX2f58uXMnz8fgDfffJOdd96Zs88+u77F+ZlnnuGGG27gqaeeIqXEgQceyGGHHUbXrl1ZuHAhN954I8OGDWPJkiUsWrSI22+/nSlTpjBkyBBuueUWHn/8caZPn84VV1zBtGnTuPzyyzn88MOZMmUKb775JkOHDuWII44AYPbs2cybN49u3brxox/9iKOPPpqLLrqIdevW8d577zW6/qeccgodO3Zk4cKF9QEe4PLLL6dbt26sW7eOUaNGMW/ePAYOHAhkreuzZ88G4Be/+AVXXnkl1dUb3OCtzRiEJUmSynDSSSfVh79Vq1Yxfvx4Fi5cSESwZs2a+nKjRo1ip512AmDfffdl6dKl7LfffixevJiJEycyevRojjrqqA3m//jjj3PCCSewww47APCFL3yBGTNmMGbMGPbcc0+GDRtWX7Z3794MGDAAgP32249Ro0YREQwYMIAlS5YA8MADDzB9+nSuvPJKAFavXs0rr7wCwJFHHkm3bt0AGDJkCKeffjpr1qzh+OOPZ9CgQY2uf13XiBUrVnDQQQdxzDHHsOeee/KrX/2K6667jrVr1/Laa6/xwgsv1Afhk08+eZNe60+KV42QJEkqQ11ABbj44osZOXIk8+fP5+6772b16tX1z3Xs2LF+uKqqirVr19K1a1eeffZZRowYwbXXXsuECRM2edkNl9GuXbv6x+3atWPt2rVA1p/4zjvvZO7cucydO5dXXnmFffbZZ4P5HXrooTz22GP06NGDmpoabrrppo3WpXv37uy///489dRTvPzyy1x55ZU89NBDzJs3j9GjR3/stWhY7y2NQViSJKmFVq1aRY8ePYDsagjNeeONN1i/fj1jx47lsssuq+8uUOqQQw5h2rRpvPfee7z77rvcddddG+2r25yjjz6an/70p6SUAJgzZ06j5ZYuXcqnP/1pzjzzTCZMmNBo3Uq99957zJkzhz59+vDWW2+xww47sNNOO/G///u//Pa3v21yuh133JG33357k9enNdg1QpIkbfE2drmztnD++eczfvx4LrvsMkaPHt1s+eXLl3Paaaexfv16AL73ve9tUGb//fenpqaGoUOHAtmP5QYPHlzf1aGlLr74Ys477zwGDhzI+vXr6d27N/fcc88G5R599FEmT57MdtttR5cuXZpsEa77odsHH3xATU0NBxxwAACDBw9m7733Zvfdd+fggw9usj41NTWcffbZW9SP5aLuLGGjhSKOAf4vUAX8IqX0/QbP/xiou8Df9sBfpZR23tg8q6urU21t7abUWZK2KZW+Pmqd0844rFXm61Uj9El48cUX67/Gl1qisW0nIp5JKW3wK71mW4Qjogq4BjgSWAY8HRHTU0ov1JVJKX21pPxEYPCmV1+SJElqfeX0ER4KLEopLU4pfQjcBhy3kfJfAm6tROUkSZKk1lJOEO4BvFryeFk+bgMRsSfQG3i4iefPiojaiKhdsWJFS+sqSZIkVUylrxoxDrgjpbSusSdTStellKpTStV19+yWJEmS2kI5QXg5sHvJ4575uMaMw24RkiRJ2gqUE4SfBvpGRO+I6EAWdqc3LBQRewNdgScrW0VJkiSp8pq9akRKaW1EnAPcT3b5tCkppecj4lKgNqVUF4rHAbelcq7HJkmS1BIzJ1V2fhu5DODKlSsZNWoUAH/605+oqqqirkvnrFmz6NChQ8Wq8dJLLzFu3DgigjvuuIM+ffpUbN6VNGLECF577TU6depEhw4duP7665u8FTPApEmT6NKlC9/4xjc2eZlXXHEFF1544SZPX46y+ginlO5NKe2VUuqTUro8H/edkhBMSmlSSumC1qqoJEnSJ2GXXXapvy3x2WefzVe/+tX6xx06dKi/hXElTJs2jRNPPLH+Tm3NSSnV35SjElqyLjfffDPPPvss//RP/8Q3v/nNitWhobp1vOKKK1ptGXW8xbIkSVIz6u6KduCBB3L++ecza9Yshg8fzuDBgznooINYsGABkN1u+Qtf+ALHHHMMffv25fzzzwdg3bp11NTU0L9/fwYMGMCPf/xj7r33Xq6++mp+/vOfM3JkdmOdq666iv79+9O/f3+uvvpqAJYsWUK/fv348pe/TP/+/ZkxYwZ77703NTU17LXXXpxyyik8+OCDHHzwwfTt25dZs2YB8O6773L66aczdOhQBg8ezG9+85v6Oo4ZM4bDDz+cUaNG8dprr3HooYcyaNCg+vlvzPDhw1m+PPu52J///GeOP/54Bg4cyLBhw5g3b159uWeffZbhw4fTt29frr/++vrxkydPZsiQIQwcOJBLLrmk0XU844wzeP/99xk0aBCnnHIKAMcffzwHHHAA++23H9ddd91mvZ91vMWyJElSGZYtW8YTTzxBVVUVb731FjNmzKB9+/Y8+OCDXHjhhdx5550AzJ07lzlz5tCxY0f69evHxIkTef3111m+fDnz588H4M0332TnnXfm7LPPru9C8Mwzz3DDDTfw1FNPkVLiwAMP5LDDDqNr164sXLiQG2+8kWHDhrFkyRIWLVrE7bffzpQpUxgyZAi33HILjz/+ONOnT+eKK65g2rRpXH755Rx++OFMmTKFN998k6FDh3LEEUcAMHv2bObNm0e3bt340Y9+xNFHH81FF13EunXreO+99zb6Otx3330cf/zxAFxyySUMHjyYadOm8fDDD/PlL3+ZuXPnAjBv3jxmzpzJu+++y+DBgxk9ejTz589n4cKFzJo1i5QSY8aM4bHHHmOPPfb42DoC3H777fXzApgyZQrdunXj/fffZ8iQIYwdO5Zddtlls95Tg7AkSVIZTjrpJKqqqgBYtWoV48ePZ+HChUQEa9asqS83atQodtppJwD23Xdfli5dyn777cfixYuZOHEio0eP5qijjtpg/o8//jgnnHACO+ywAwBf+MIXmDFjBmPGjGHPPfesD4gAvXv3ZsCAAQDst99+jBo1iohgwIABLFmyBIAHHniA6dOnc+WVVwKwevVqXnnlFQCOPPJIunXrBsCQIUM4/fTTWbNmDccff3yTfX9POeUUPvzwQ9555536gPr444/XnwAcfvjhrFy5krfeeguA4447js6dO9O5c2dGjhzJrFmzePzxx3nggQcYPDi7CfE777zDwoUL2WOPPTZYx4Z+8pOfcNdddwHw6quvsnDhws0OwnaNkCRJKkNdQAW4+OKLGTlyJPPnz+fuu+9m9erV9c917Nixfriqqoq1a9fStWtXnn32WUaMGMG1117LhAkTNnnZDZfRrl27+sft2rWr7/ebUuLOO++s79/8yiuvsM8++2wwv0MPPZTHHnuMHj16UFNTw0033dRoHW6++WYWL17M+PHjmThxYrN1jogNHqeU+Na3vlVfp0WLFnHGGWc0uo6lHn30UR588EGefPJJnn32WQYPHvyx13xTGYQlSZJaaNWqVfTokd1od+rUqc2Wf+ONN1i/fj1jx47lsssuY/bs2RuUOeSQQ5g2bRrvvfce7777LnfddReHHHLIJtfx6KOP5qc//Sl1F/SaM2dOo+WWLl3Kpz/9ac4880wmTJjQaN3qRATf/e53mTlzJi+99BKHHHIIN998M5CF1V133ZVPfepTAPzmN79h9erVrFy5kkcffZQhQ4Zw9NFHM2XKFN555x0Ali9fzuuvv97osrbbbrv6lvZVq1bRtWtXtt9+e1566SVmzpy5aS9KA3aNkCRJW76NXO6sLZx//vmMHz+eyy67jNGjRzdbfvny5Zx22mn1V3z43ve+t0GZ/fffn5qaGoYOHQrAhAkTGDx4cH1Xh5a6+OKLOe+88xg4cCDr16+nd+/e3HPPPRuUe/TRR5k8eTLbbbcdXbp0abJFuE7nzp35+te/zuTJk5k8eTKnn346AwcOZPvtt+fGG2+sLzdw4EBGjhzJG2+8wcUXX8xuu+3Gbrvtxosvvsjw4cMB6NKlC//5n/9Z3+Wk1FlnncXAgQPZf//9mTJlCtdeey377LMP/fr122gXipaItrrsb3V1daqtrW2TZUvSluSGM0e2ynxPO+OwVpnvlhZItG168cUX67/Gl1qisW0nIp5JKVU3LGvXCEmSJBWSQViSJEmFZBCWJElbpLbqvqmtV0u3GYOwJEna4nTq1ImVK1cahlW2lBIrV66kU6dOZU/jVSMkSdIWp2fPnixbtowVK1a0dVW0FenUqRM9e/Ysu7xBWJIkbXG22247evfu3dbV0DbOrhGSJEkqJIOwJEmSCskgLEmSpEIyCEuSJKmQDMKSJEkqJIOwJEmSCskgLEmSpEIyCEuSJKmQDMKSJEkqJIOwJEmSCskgLEmSpEIyCEuSJKmQDMKSJEkqJIOwJEmSCskgLEmSpEIyCEuSJKmQDMKSJEkqJIOwJEmSCskgLEmSpEIyCEuSJKmQDMKSJEkqJIOwJEmSCskgLEmSpEIyCEuSJKmQDMKSJEkqJIOwJEmSCskgLEmSpEIqKwhHxDERsSAiFkXEBU2U+WJEvBARz0fELZWtpiRJklRZ7ZsrEBFVwDXAkcAy4OmImJ5SeqGkTF/gW8DBKaW/RMRftVaFJUmSpEoop0V4KLAopbQ4pfQhcBtwXIMyZwLXpJT+ApBSer2y1ZQkSZIqq5wg3AN4teTxsnxcqb2AvSLi9xExMyKOaWxGEXFWRNRGRO2KFSs2rcaSJElSBVTqx3Ltgb7ACOBLwPURsXPDQiml61JK1Sml6u7du1do0ZIkSVLLlROElwO7lzzumY8rtQyYnlJak1J6GfgfsmAsSZIkbZHKCcJPA30jondEdADGAdMblJlG1hpMROxK1lViceWqKUmSJFVWs0E4pbQWOAe4H3gR+FVK6fmIuDQixuTF7gdWRsQLwCPAN1NKK1ur0pIkSdLmavbyaQAppXuBexuM+07JcAK+lv9JkiRJW7yygrAkSZK2YTMntc58h7XSfCvEWyxLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgqpfVtXQJIkfYJmTmqd+Q5rpflKrcgWYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmF5C2WJUmSKigiWm3eKaVWm3cR2SIsSZKkQjIIS5IkqZAMwpIkSSokg7AkSZIKySAsSZKkQiorCEfEMRGxICIWRcQFjTxfExErImJu/jeh8lWVJEmSKqfZy6dFRBVwDXAksAx4OiKmp5ReaFD0lymlc1qhjpIkSVLFldMiPBRYlFJanFL6ELgNOK51qyVJkiS1rnKCcA/g1ZLHy/JxDY2NiHkRcUdE7N7YjCLirIiojYjaFStWbEJ1JUmSpMqo1I/l7gZ6pZQGAr8DbmysUErpupRSdUqpunv37hVatCRJktRy5QTh5UBpC2/PfFy9lNLKlNIH+cNfAAdUpnqSJElS6ygnCD8N9I2I3hHRARgHTC8tEBGfKXk4BnixclWUJEmSKq/Zq0aklNZGxDnA/UAVMCWl9HxEXArUppSmA+dGxBhgLfBnoKYV6yxJkiRttmaDMEBK6V7g3gbjvlMy/C3gW5WtmiRJxXXDmSNbZb6nnXFYq8xX2hp5ZzlJkiQVkkFYkiRJhWQQliRJUiEZhCVJ2kQR0Wp/klqfQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmF1L6tKyCpAmZOap35Dmul+UqStAUwCEufoBvOHNkq8z3tjMNaZb6SJG3L7BohSZKkQiorCEfEMRGxICIWRcQFGyk3NiJSRFRXroqSJElS5TUbhCOiCrgG+BywL/CliNi3kXI7Al8Bnqp0JSVJkqRKK6dFeCiwKKW0OKX0IXAbcFwj5b4L/ABYXcH6SZIkSa2inCDcA3i15PGyfFy9iNgf2D2l9F8bm1FEnBURtRFRu2LFihZXVpIkSaqUzf6xXES0A64Cvt5c2ZTSdSml6pRSdffu3Td30ZIkSdImKycILwd2L3ncMx9XZ0egP/BoRCwBhgHT/cGcJEmStmTlBOGngb4R0TsiOgDjgOl1T6aUVqWUdk0p9Uop9QJmAmNSSrWtUmNJkiSpApq9oUZKaW1EnAPcD1QBU1JKz0fEpUBtSmn6xudQTK1244TrH2mV+UqSJBVNWXeWSyndC9zbYNx3mig7YvOrJUmSJLUub7EsSZK0lWi1b5zPOKxV5rul8xbLkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLUhkiotX+JEltwyAsSZKkQjIIS5IkqZAMwpIkSSokg7AkSZIKySAsSZKkQjIIS5IkqZAMwpIkSSqksoJwRBwTEQsiYlFEXNDI82dHxHMRMTciHo+IfStfVUmSJKlymg3CEVEFXAN8DtgX+FIjQfeWlNKAlNIg4IfAVZWuqCRJklRJ5bQIDwUWpZQWp5Q+BG4DjistkFJ6q+ThDkCqXBUlSZKkymtfRpkewKslj5cBBzYsFBH/DHwN6AAc3tiMIuIs4CyAPfbYo6V1lSRJkiqmYj+WSyldk1LqA/wL8O0mylyXUqpOKVV37969UouWJEmSWqycILwc2L3kcc98XFNuA47fjDpJkiRJra6cIPw00DciekdEB2AcML20QET0LXk4GlhYuSpKkiRJlddsH+GU0tqIOAe4H6gCpqSUno+IS4HalNJ04JyIOAJYA/wFGN+alZYkSZI2Vzk/liOldC9wb4Nx3ykZ/kqF6yVJkiS1Ku8sJ0mSpEIyCEuSJKmQDMKSJEkqJIOwJEmSCskgLEmSpEIyCEuSJKmQyrp82rYsIlplvlMmjGiV+TJzUuvMF2BYK85bkiRpC2OLsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgrJICxJkqRCMghLkiSpkAzCkiRJKiSDsCRJkgqprCAcEcdExIKIWBQRFzTy/Nci4oWImBcRD0XEnpWvqiRJklQ5zQbhiKgCrgE+B+wLfCki9m1QbA5QnVIaCNwB/LDSFZUkSZIqqZwW4aHAopTS4pTSh8BtwHGlBVJKj6SU3ssfzgR6VraakiRJUmWVE4R7AK+WPF6Wj2vKGcBvG3siIs6KiNqIqF2xYkX5tZQkSZIqrKI/louIvweqgcmNPZ9Sui6lVJ1Squ7evXslFy1JkiS1SPsyyiwHdi953DMf9zERcQRwEXBYSumDylRPkiRJah3ltAg/DfSNiN4R0QEYB0wvLRARg4F/B8aklF6vfDUlSZKkymo2CKeU1gLnAPcDLwK/Sik9HxGXRsSYvNhkoAtwe0TMjYjpTcxOkiRJ2iKU0zWClNK9wL0Nxn2nZPiICtdLkiRJalXeWU6SJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYUmSJBWSQViSJEmFZBCWJElSIRmEJUmSVEgGYamBiGi1P0mStOUoKwhHxDERsSAiFkXEBY08f2hEzI6ItRFxYuWrKUmSJFVWs0E4IqqAa4DPAfsCX4qIfRsUewWoAW6pdAUlSZKk1tC+jDJDgUUppcUAEXEbcBzwQl2BlNKS/Ln1rVBHSZIkqeLK6RrRA3i15PGyfJwkSZK01fpEfywXEWdFRG1E1K5YseKTXLQkSZL0MeUE4eXA7iWPe+bjWiyldF1KqTqlVN29e/dNmYUkSZJUEeUE4aeBvhHROyI6AOOA6a1bLUmSJKl1NRuEU0prgXOA+4EXgV+llJ6PiEsjYgxARAyJiGXAScC/R8TzrVlpSZIkaXOVc9UIUkr3Avc2GPedkuGnybpMSJIkSVsF7ywnSZKkQjIIS5IkqZAMwpIkSSokg7AkSZIKySAsSZKkQjIIS5IkqZAMwpIkSSokg7AkSZIKySAsSZKkQjIIS5IkqZAMwpIkSSokg7AkSZIKySAsSZKkQjIIS5IkqZAMwpIkSSokg7AkSZIKySAsSZKkQjIIS5IkqZAMwpIkSSokg7AkSZIKySAsSZKkQjIIS5IkqZAMwpIkSSokg7AkSZIKySAsSZKkQjIIS5IkqZAMwpIkSSokg7AkSZIKySAsSZKkQjIIS5IkqZAMwpIkSSokg7AkSZIKySAsSZKkQjIIS5IkqZAMwpIkSSokg7AkSZIKySAsSZKkQjIIS5IkqZAMwpIkSSokg7AkSZIKySAsSZKkQiorCEfEMRGxICIWRcQFjTzfMSJ+mT//VET0qnhNJUmSpApqNghHRBVwDfA5YF/gSxGxb4NiZwB/SSn9DfBj4AeVrqgkSZJUSeW0CA8FFqWUFqeUPgRuA45rUOY44MZ8+A5gVERE5aopSZIkVVaklDZeIOJE4JiU0oT88anAgSmlc0rKzM/LLMsf/yEv80aDeZ0FnJU/7AcsqNSKbOV2Bd5otpSKxu1CjXG7UGPcLtQYt4uP7JlS6t5wZPtPsgYppeuA6z7JZW4NIqI2pVTd1vXQlsXtQo1xu1Bj3C7UGLeL5pXTNWI5sHvJ4575uEbLRER7YCdgZSUqKEmSJLWGcoLw00DfiOgdER2AccD0BmWmA+Pz4ROBh1NzfS4kSZKkNtRs14iU0tqIOAe4H6gCpqSUno+IS4HalNJ04P8B/xERi4A/k4Vllc/uImqM24Ua43ahxrhdqDFuF81o9sdykiRJ0rbIO8tJkiSpkAzCkiRJKiSDsNRCEdErv3b25s6nJiL+rZHxZ0fElzdxnjtGxNySvzci4ur8uR+XjP+fiHhz89ZAkqSt2zYZhCsVVLY1ETEoIv5uM6bfLSLuqGSdtKGU0rUppZs2cdq3U0qD6v6ApcCv8+e+WjL+p3XjiyAido6If2rlZZwXEdu35jJKlvVoRLTJtUEbrmdE3BsRO7dFXVrCbWCzlvWJv+cRcWFrzn8jyz03Il6MiL9ExAWf0DKbPbZuaq75pPNQJZfXVGNRpW2TQbhI8us2l2sQsMlBOKX0x5TSiZs6/TamfUTcnO8w74iI7SPiOxHxdETMj4jr6m4znh+wfhARs/KW2EMaziwiRkfEkxGxa0RMiohvbGzafHm/iogXIuKuiHiq4UExIvYC/gqY0Uj9vwTcWukXZQu2M9CqIQg4D/hEQlAbO4+S9Uwp/V1K6c02q035dsZtYFOdxyf/nrdJECbbRo5MKXVNKX3/k1jglnBsbWGW2KZsNUE4Ii6NiPNKHl8eEV+JiIciYnZEPBcRx5VMUhUR10fE8xHxQER0zqcbFBEzI2JeHiC6bmx8E3U5Mw88z0bEnXVnyhExNSJ+EhFPRMTiyG5PTUR8JiIey7+Snh8Rh0TESRFxVf78VyJicT781xHx+3z4gIj474h4JiLuj4jP5OMfjYirI6IW+Eo+r/l5fR5ros4dgEuBk/N6nBwR3SJiWr7OMyNiYF52UkT8Rx7MFkbEmfn4+jO9iKiKiCvz5c6LiIn5+O/n4WxeRFzZwrd5a9IP+FlKaR/gLbKd57+llIaklPoDnYHPl5Rvn1IaSnZAuaR0RhFxAnAB8HcNb0u+kWn/CfhLSmlf4GLggEamGwf8suE1vSNiT6A38HD5q7vV+z7QJ9/2J0fENRExBiD/vE/Jh0+PiMvz4b/PT0DmRsS/R0RVPv6o/LMxOyJuj4guEXEusBvwSEQ80lQlmpj2mIi4vaTMiIi4Jx/+eUTU5vuxf23JCkdE93z/9HT+d3A+flJE3BgRMyJiaUR8ISJ+mO9D74uI7fJyoyJiTj5+SkR0bGw9I2JJROyaD38t3yfMj3x/ne83XoxG9sefsCJuAxscG/J99+R8m5gXEf9QssxHIzuxfymyE/3Y2Huev7cvRXbs+598miMi4veRHTuG5uV3yLehWfk2dVw+viYifp1vdwsj4of5+O8DnfPX/eaWrPPmiIhrgb8GfhsRX428NTKaPrZ3iUYyyMa2+Yj4m4h4MH9PZkdEn/j4sbVXZJ/N2fnfQWXWvSYifpO/hwsjovQ401Qeapgljo2sUWVOXsdP5+UOi4+61c2JiB3z8d8s2Y5Kt80NGory8hVpLCr7DS1XSmmr+AN6AbPz4XbAH4BPA5/Kx+0KLAIiL7sWGJQ/9yvg7/PhecBh+fClwNUbG99EXXYpGb4MmJgPTwVuz+u3L7AoH/914KJ8uArYEfg/wNP5uDvIblzSg+zGJN8DtgOeALrnZU4mu4YzwKNkIayuDs8BPfLhnTdS7xqysFb3+KfAJfnw4cDcfHgS8CxZmNsVeJVsR9gLmJ+X+ce83u3zx92AXYAFfHRZvibrsjX/5a/DKyWPDwemAWOBp/L3YzlwQcn7dXA+/OmS7aIGeAGYWbcdl7z+32hm2mnAyJJpZgPVDer5AnBAI/X/F+Cnbf06tsF7Nr/k8Thgcj48C5iZD98AHA3sA9wNbJeP/xnw5fzz8BiwQ8lr+Z18eAmw60bq0Oi0ZNdzf6Vk/M/5aH/VLf9flW8LA0u2i+p8+BcN3/t8/C3AZ/PhPYAXS7avx8n2MX8LvAd8Ln/uLuB4oBPZ536vfPxNwHmNrWfdY7KTseeAHYAuwPPAYDayP3YbaPVtYINjA3AW8O18uCNQS3ZiPAJYRXb32HbAkyXbT1Pved17OyCf5hlgCtlx+DhgWl7+ipL12Rn4n3w7qQEWk92NthNZV67d83LvtNG+om7dasiPlzR9bG9PyzPIU8AJ+XAnspb2Xnx0bN0e6JQP9yW7XwM02H4bqXcN8BrZcbgzMB+obqYuj/LxLNGVj47fE4Af5cN389FxqEu+3keRXaM48tflHuDQfHmppPwUPjqedStZ1n8Ax5bUo25Zfwc8WLJO/wacQPbNZtfWeM+3mqbwlNKSiFgZEYPJAsEcspt3/DgiDgXWkwXJT+eTvJxSmpsPPwP0ioidyHYG/52PvxG4vanxG6lO/4i4jOwD3YXsZiN1pqWU1gMv1J1NkYXcKZG1tEzL6/V2fja5I9ntqW8h24gOIeu72Q/oD/wuP2mqItvI6/yyZPj3wNSI+BUt6/f5WbLwRkrp4YjYJSI+lT/3m5TS+8D7eSvAUGBuybRHANemlNbm0/85sq9WVgP/L7LWjHtaUJetTWrk8c/IDkavRsQksp1cnQ/y/+v4+I1s/kDWArEX2QGpMU1N26SI+Fuyk5RnGnl6HPDP5cxnGzYDOC8i9iU7Yega2Tcuw4FzyU5IDwCezj9/nYHXgWFkB8Lf5+M7kAWGcjQ6bcpuWnQfcGxk/QRHA+fn03wxIs4ie98/k08/r3SmKaUJTSzvCGDffFkAn4qILvnwb1NKayLiObJ9y335+OfIDmT9yPah/5OPv5Fsm7l6I+v3WeCulNK7ABHxa7L92XQa2R9vZD6flCJsA40dG44CBta1apKF0L7Ah8CslNIygIiYS/Y+Pd7MOr2cUnoun+Z54KGUUsq3rV4lyxwTeZcvsn3jHvnwQymlVfn0LwB7kp2EbWkaO7YHcEULMsiOZCcmdwGklFYDlHxGITtB/beIGES2z9+rBXX8XUppZT7PX5N9Jqc1VpeSaUqzRE/gl/nnoAPwcj7+98BVkbXO/zqltCwijiJ7X+fkZbqQbUevAK+mlH6fj/9Pss/TlcDIiDifLOx3IztZvjsvV7d9Nqzf4WSB/qiU0lsteC3KttUE4dwvyM4Q/g/ZWcYpQHeyVq81EbGEj8LHByXTrSPbiVXKVOD4lNKzEVFDdiZdp3S5AZBSeiz/oIwm2yldlbIfQz0BnEbWijoDOJ1sJ/x1sp3E8yml4U3U4d26gZTS2RFxYD7/ZyLigLoPw2ZoLOhtfIJsZz4UGEV2q+1zyDbibdEeETE8pfQk8P+RHSwOAt7Iw8aJZC3mzVkKfBP4dUSclFJ6vszl/x74ItnXlfuStciUarQPcETsTXbWX+6Be5uUUloe2Y99jiFroetG9nq+k1J6O//K7saU0rdKp4uIY8kONl/ahMXGRqa9jezz8meyFqC3I6I38A1gSErpLxExlY+fXDWnHTCs7mBbsg6Q76dSSusjYk3Km1/IDuatcVxozf3xJinCNtDYsSGvw8SUUmkDDhExgg3fp3K2hdJp1pc8Lt2WAhibUlrQYJkHbuIy28IGx3ZaJ4N8Ffhfsm9r2pE1LpWrqeP2xurybsnwT4GrUkrT8+1hEkBK6fsR8V9krbW/j4ijyV6D76WU/r10gRHRq7F6REQnWq+xaLNsNX2Ec3eR7bSGkLXC7gS8nm+AI8nOJJuUn3X+paT/yanAfzc1fiOz2hF4LW/hPaW5SkfWJ/N/U0rXk4X5/fOnZpDt5B4jO6saCXyQ12cB0D0ihufz2C4i9mti/n1SSk+llL4DrCBrYW7M23nd68yoq3++0b9RcsZ1XER0iohdyIL+0w3m9TvgH/JWYCLrb9wF2CmldC/Zh/lvN/rCbN0WAP8cES+SBcufA9eTfR11Pxu+Xk1KKb1E9j7cHhF9ypzsZ2Tbxwtk3XOeJ/tas84XafzHcOOA20qCT1E03PYh65JyHtnnr+6zWPfDwoeAEyPir6B++94zn+bgiPibfPwOkf0osallNFxeU9P+N9l+4UyyQATwKbKD1Kq8BepzLVznB4CJdQ/yFqZyLSBrwfqb/HHpPrGp9ZwBHB/ZDzl34KOvM7cUhdsGmjg23A/8Y3zUF3yv/P3amObWqzn3AxPzkwsi+2a3OWvq6rgFa2kGeRtYFhHHA0TW777hjyt3Al7LW59PJfvGplxH5ttpZ7IuTr9vpnxDO5F164PsGxHyevZJKT2XUvoB2bFtb7L39PS6b5kiokfdZ4W8oSgfrmsoqgu9pY1F5VhK9s31TU1loM21pZ55NSql9GFkX9O/mVJalzfT351/BVMLvFTGbMYD1+Yb32KyFtmNjW/MxWT9fFbk/5vbQYwAvhkRa4B3yPqZQbbD3R14LF+fV+vWIV/XE4GfRNZ1oz3Z15KNtRhOjoi+ZGdoD5H1723MI8AFkX3l9T2ys70pETGPrJ/g+JKy8/LyuwLfTSn9MT/Tq/MLsjO0efl6XQ/cCfwmP/ML4GvNvC5bpZTSErIdQUPfzv8alh9RMvwG+dc+KaWpZN8ukFKaQ/aVJ+Rn4RublqyV4O9TSqvz8Pwg2Q6jruxfN1H3SY2N39allFZG9gOe+WTdAr5J9vk7KqW0KCKWkrUIzsjLvxAR3wYeiIh2wBrgn1NKM/NvgW6NiI757L9N1ufxOuC+iPhjSmlkI3VY0dS0+ef/HrJvvMbn5Z+NiDlk+4RXaeKgFhG/IOum1LC15Fzgmvzz3Z4s7J1d5uu1OiJOIzs5a0928Ls2f7rR9Uwpzc5bLGflo36RUprTYL/RZgq6DTR2bJhH/pubPJiuIAtNG7PR9SrDd8mOX/Py1/JlPv5j4qaWOS8iZqeUmm1waiObkkFOBf49Ii4l26ZOIms9r/Mz4M7IriV/Hx9vsW3OLLLjcE/gP1NKtS38/E0i+8z/hezH1L3z8eflQX89WQb5bUrpg4jYB3gyP795B/h7shbduoaiKWTdjn6eUnovIuoai/5ECxuLIqKusejYlNIfWrBOzarrFL1VyD9As4GTUkoL27o+26rIvrJ4J6W0LV/1YasWWV+zR8j6kwXwLyml37ZtrSRJbSE/wapOKZ3T1nXZ2mw1LcKR9YO8h+zHGIZgFVr+FVub3FBBkqRtxVbVIvxJi4hrgIMbjP6/KaUb2qI+5YqsI/sPGox+OaV0QlvURyqaiHiK7NJUpU5N+a/rte1zG1Br8PheeQZhSZIkFdLWdtUISZIkqSIMwpIkSSokg7AkSZIKySAsSZKkQvr/AejjtjwgyleuAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x504 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame.from_dict(results)\n",
    "df = df.transpose()\n",
    "df.plot(kind='bar', figsize=(12,7), colormap='copper', rot=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "blind-japanese",
   "metadata": {},
   "source": [
    "## The impact of class names\n",
    "\n",
    "In traditional supervised learning, the actual names of the labels do not have any impact on the performance of the model. You’re free to call your classes whatever you want &mdash; `positive`, `politics` or `aardvark`, it makes no difference at all. In zero-shot text classification, these names suddenly become important. Both the NLI and TARS classifiers add the label itself to the input of the model, so that the class names have the power to change the predictions. In general, the more semantic information about the class they contain, and the more similar they are to the type of data that the model was finetuned on, the better we can expect the classifier to perform.\n",
    "\n",
    "Let’s take polarity classification as an example. While `positive`, `neutral` and `negative` are the traditional class names for this task, they may not be optimal for a zero-shot approach. To test this out, we experimented\n",
    "with two alternative sets of names for the financial news data: `good news`, `neutral news` and `bad news` on the one hand, and `happy news`, `neutral news` and `unhappy news` on the other. As you can see in the figure below, this has a very positive effect on the accuracy of the classifiers. Both TARS (62.0%) and Bart (61.9%) now perform better than the original Roberta, although Bart only does so with the happy/unhappy class names. Roberta itself jumps another 13%, to an accuracy of over 73% with both alternative sets of names. Zero-shot classifiers may reduce the need for labeling, but they do introduce the necessary task of searching for good class names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "bound-bouquet",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['negative', 'neutral', 'positive']\n"
     ]
    }
   ],
   "source": [
    "print(data[3][\"class_names\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "normal-distance",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 11:49:41,900 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fb8b183295124681bca2742223daf44f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Flair: 0.619\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "60b5c8d7e132404f8a483288a7d9b432",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Huggingface Bart 0.458\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at roberta-large-mnli were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
      "- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3af48bf4c0ca4704bc5305298c94ee8c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Huggingface Roberta 0.738\n"
     ]
    }
   ],
   "source": [
    "alternative_results = {}\n",
    "alternative_results['positive/negative'] = results['financial_phrasebank']\n",
    "\n",
    "alternative_financial_news = {\n",
    "    \"name\": \"good/bad\", \n",
    "    \"test_texts\": financial_test_texts, \n",
    "    \"test_labels\": financial_test_labels, \n",
    "    \"class_names\":  [\"bad news\", \"neutral news\", \"good news\"]\n",
    "    }\n",
    "\n",
    "alternative_results[alternative_financial_news['name']] = {}\n",
    "flair_acc = evaluate_flair(alternative_financial_news, default_name='neutral news')\n",
    "alternative_results[alternative_financial_news['name']]['Flair TARS'] = flair_acc\n",
    "print(\"Flair:\", flair_acc)\n",
    "\n",
    "huggingface_acc = evaluate_huggingface(alternative_financial_news)\n",
    "alternative_results[alternative_financial_news['name']]['Transformers Bart'] = huggingface_acc\n",
    "print(\"Huggingface Bart\", huggingface_acc)\n",
    "\n",
    "huggingface_acc_roberta = evaluate_huggingface(alternative_financial_news, model='roberta')\n",
    "alternative_results[alternative_financial_news['name']]['Transformers Roberta'] = huggingface_acc_roberta\n",
    "print(\"Huggingface Roberta\", huggingface_acc_roberta)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "after-property",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 11:51:31,776 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "66d7ff3c928241ef8a700e4533258a24",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Flair: 0.62\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "868515dfd16f44d2aada688282648619",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Huggingface Bart 0.619\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at roberta-large-mnli were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']\n",
      "- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "441718dbef6a4d38a7f21d967b273b60",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Huggingface Roberta 0.735\n"
     ]
    }
   ],
   "source": [
    "alternative_financial_news = {\n",
    "    \"name\": \"happy/unhappy\", \n",
    "    \"test_texts\": financial_test_texts, \n",
    "    \"test_labels\": financial_test_labels, \n",
    "    \"class_names\":  [\"unhappy news\", \"neutral news\", \"happy news\"]\n",
    "    }\n",
    "\n",
    "alternative_results[alternative_financial_news['name']] = {}\n",
    "flair_acc = evaluate_flair(alternative_financial_news, default_name='neutral news')\n",
    "alternative_results[alternative_financial_news['name']]['Flair TARS'] = flair_acc\n",
    "print(\"Flair:\", flair_acc)\n",
    "\n",
    "huggingface_acc = evaluate_huggingface(alternative_financial_news)\n",
    "alternative_results[alternative_financial_news['name']]['Transformers Bart'] = huggingface_acc\n",
    "print(\"Huggingface Bart\", huggingface_acc)\n",
    "\n",
    "huggingface_acc_roberta = evaluate_huggingface(alternative_financial_news, model='roberta')\n",
    "alternative_results[alternative_financial_news['name']]['Transformers Roberta'] = huggingface_acc_roberta\n",
    "print(\"Huggingface Roberta\", huggingface_acc_roberta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "indoor-polls",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsIAAAGbCAYAAADOe/Z7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAoYElEQVR4nO3de5xVdb3/8ddHUDEhKRx7mFADhQrFRR1AQ4WOl/AcgxTtwNETiGl08tTpHP397JfH8HIupqeLxSOPpI4ahorFQVO8UDxMipjBABnwgoQ11inESyCgIN/fH3szbYaBGWQPe+T7ej4e+8Fa3/Vda31mzyz2e6/9XXtFSglJkiQpN/tVugBJkiSpEgzCkiRJypJBWJIkSVkyCEuSJClLBmFJkiRlqXOldnzooYem6urqSu1ekiRJmVi0aNFLKaWq5u0VC8LV1dXU19dXaveSJEnKRES80FK7QyMkSZKUJYOwJEmSsmQQliRJUpYqNka4JZs3b6axsZFNmzZVuhS1ky5dutCzZ0/233//SpciSZIy16GCcGNjI926daO6upqIqHQ5KrOUEmvXrqWxsZHevXtXuhxJkpS5DjU0YtOmTfTo0cMQvI+KCHr06OEZf0mS1CF0qCAMGIL3cf5+JUlSR9HhgrAkSZK0N3ToIBwRZX3sLTfddBN33HEHALW1tfz+979vWvbZz36W5cuXv+1t/+d//ifTp0/f4xrbYtasWdvVeuWVV/LYY4/tlX1LkiS1tw51sdy+YvLkyU3TtbW1fPSjH+X9738/AN///vf3aNsPP/ww99xzzx5to61mzZrFmWeeSf/+/QG4+uqr98p+JUmS9oYOfUa4ElavXs3RRx/NeeedR79+/TjnnHPYsGEDc+fO5ZhjjmHAgAFMmjSJN954A4DLL7+c/v37M3DgQC699FIApkyZwg033MDMmTOpr6/nvPPOY/DgwWzcuJGRI0dSX1/PTTfdxGWXXda039raWi655BIAfvCDHzB06FAGDx7M5z73Od566y0A/vznP/Pmm29SVVXFxIkT+eIXv8jHPvYx+vTpw8yZM5u2df311zNkyBAGDhzI1772tab2a665hqOOOooTTzyR8ePHc8MNNwAwbdo0hgwZwqBBgxg7diwbNmzgF7/4BbNnz+ayyy5j8ODBPP/880ycOJGZM2cyZ84czj333Kbtzps3jzPPPBOARx55hBNOOIFjjz2Wc889l/Xr17fHr0mSJGmPGYRb8Mwzz/AP//APrFixgne/+9184xvfYOLEidx999089dRTbNmyhe9973usXbuWH//4xzQ0NLB06VKuuOKK7bZzzjnnUFNTw/Tp01m8eDEHHXRQ07KxY8fy4x//uGn+7rvvZty4caxYsYK7776b+fPns3jxYjp16tQ0FOKxxx7jlFNOaVrnD3/4A0888QQPPPAAl19+OVAIos899xwLFy5k8eLFLFq0iMcff5y6ujruu+8+lixZwkMPPUR9fX3Tds4++2zq6upYsmQJ/fr145ZbbuFjH/sYo0eP5vrrr2fx4sV86EMfaup/6qmn8qtf/YrXX399u9pfeuklrr32Wh577DGefPJJampq+MY3vlHG34wkSVL5GIRb0KtXL4YPHw7A+eefz9y5c+nduzdHHnkkABMmTODxxx/nkEMOoUuXLlx44YX86Ec/4l3veleb91FVVUWfPn1YsGABa9eu5emnn2b48OHMnTuXRYsWMWTIEAYPHszcuXNZtWoVAHPmzOGMM85o2sanPvUp9ttvP/r3788f//hHoBCEH3nkEY455hiOPfZYnn76aZ577jnmz5/PmDFj6NKlC926deOTn/xk03aWLVvGSSedxIABA5g+fToNDQ27rL1z586MGjWK+++/ny1btvCTn/yEMWPGsGDBApYvX87w4cMZPHgwt99+Oy+88EKbnxNJkqS9yTHCLWh+YV337t1Zu3btDv06d+7MwoULmTt3LjNnzuS73/0uP/3pT9u8n3HjxnHPPfdw9NFHc9ZZZxERpJSYMGEC//Ef/7FD/4ULF/K9732vaf7AAw9smk4pNf37la98hc997nPbrfutb31rp3VMnDiRWbNmMWjQIGpra5k3b16bav/ud7/Le9/7XmpqaujWrRspJU477TR++MMftrq+JElSpXlGuAW//e1v+eUvfwnAXXfdRU1NDatXr2blypUA3HnnnYwYMYL169fz2muv8dd//dd885vfZMmSJTtsq1u3bqxbt67F/Zx11ln8z//8Dz/84Q8ZN24cAKeccgozZ87kT3/6EwAvv/wyL7zwAg0NDRx99NF06tRpl7V/4hOf4NZbb20am/viiy/ypz/9ieHDh3P//fezadMm1q9fzwMPPNC0zrp16zj88MPZvHnzdt9IsavaR4wYwZNPPsm0adOaaj/++OOZP39+0/P0+uuv8+yzz+6yXkmSpErp0GeEt53l3NuOOuoopk6dyqRJk+jfvz833ngjxx9/POeeey5btmxhyJAhTJ48mZdffpkxY8awadMmUkotjoedOHEikydP5qCDDmoK19u85z3voV+/fixfvpyhQ4cC0L9/f6699lpOP/10tm7dyv7778/UqVN54oknGDVqVKu1n3766axYsYITTjgBgK5du/KDH/yAIUOGMHr0aAYOHMj73vc+BgwYwCGHHAIULqIbNmwYVVVVDBs2rCn8jhs3josuuogbb7xxu4vxADp16sSZZ55JbW0tt99+O1AY7lFbW8v48eObLia89tprm4aUSJIkdSRRqbBZU1OTSi/YAlixYgX9+vWrSD3brF69mjPPPJNly5ZVtI7mTjvtNO644w4OP/zwt72N9evX07VrVzZs2MDJJ5/MzTffzLHHHlvGKtumI/yeJUn7vtsu+nilS2jRBdN+VukSshMRi1JKNc3bO/QZYf3Fo48+usfbuPjii1m+fDmbNm1iwoQJFQnBkiRJHYVBuJnq6uoOdza4XO66665KlyBJkhZMqXQFOzp+SqUrqAgvlpMkSVKWDMKSJEnKkkFYkiRJWTIIS5IkKUsd+mK5cn/tSaW+rqSlr2Q77rjj+MUvfkGPHj2abn7xdnTt2nWP1pckScqVZ4Qr4De/+Q1HHHHEdrdIliRJ0t5lEG7BNddcw1FHHcWJJ57I+PHjueGGG1i8eDHHH388AwcO5KyzzuKVV14B2Gn7okWLGDRoEIMGDWLq1KnbbX/OnDnb3SXuy1/+Mh/5yEc45ZRTWLNmDQDTpk1jyJAhDBo0iLFjx7JhwwagEKJPOOEEBgwYwBVXXLE3ng5JkqR9kkG4mbq6Ou677z6WLFnCQw89xLa7333mM5/huuuuY+nSpQwYMICrrrpql+0XXHAB3/nOd1iyZMkO+ygNwq+//jo1NTU0NDQwYsSIpvXPPvts6urqWLJkCf369eOWW24B4Etf+hKf//zneeqpp/boLnOSJEm5Mwg3M3/+fMaMGUOXLl3o1q0bn/zkJ3n99dd59dVXGTFiBAATJkzg8ccf57XXXmux/dVXX+XVV1/l5JNPBuDv//7vm7b/5ptv0tjYSJ8+fQDYb7/9+Nu//VsAzj//fJ544gkAli1bxkknncSAAQOYPn06DQ0NTfWNHz9+h+1KkiRp93Toi+X2RT//+c858cQTd7o8IgCYOHEis2bNYtCgQdTW1jJv3rwd+kiSJOnt84xwM8OHD+f+++9n06ZNrF+/ngceeICDDz6Y97znPfz85z8H4M4772TEiBEccsghLbZ3796d7t27N53dnT59etP258yZwxlnnNE0v3XrVmbOnAkUboG8LSSvW7eOww8/nM2bN2+3/vDhw5kxY8YO25UkSdLu6dBnhCvxdWdDhgxh9OjRDBw4kPe9730MGDCAQw45hNtvv53JkyezYcMG+vTpw2233Qaw0/bbbruNSZMmERGcfvrpTdufN28eV199ddP8wQcfzMKFC7n22ms57LDDuPvuu4HCBXvDhg2jqqqKYcOGsW7dOgC+/e1v83d/93dcd911jBkzZm89LZIkSfucSClVZMc1NTVp24Vo26xYsYJ+/fpVpJ5S69evp2vXrmzYsIGTTz6Zm2++mWOPPXaPt9vY2MhFF13EQw89VIYq37k6yu9ZkrRvK/f9CMrlggtHVLqEHR0/pdIVtKuIWJRSqmne3qYzwhExCvg20An4fkrpP5st/yaw7a/tXcBhKaXue1RxBV188cUsX76cTZs2MWHChLKEYICePXtmH4IlSZI6ilaDcER0AqYCpwGNQF1EzE4pLd/WJ6X05ZL+/wgc0w617jV33XVXpUuQJElSO2vLxXJDgZUppVUppTeBGcCuBqeOB35YjuIkSZKk9tKWIHwE8LuS+cZi2w4i4oNAb+CnO1l+cUTUR0T9tjuoSZIkSZVQ7q9PGwfMTCm91dLClNLNKaWalFJNVVVVmXctSZIktV1bgvCLQK+S+Z7FtpaMw2ERkiRJegdoy7dG1AF9I6I3hQA8Dvi75p0i4mjgPcAvy1bdgill2xTQpq8GWb16NWeeeSbLli0r777b6IwzzmDatGn07Nlzt9ardN2SJEnvNK2eEU4pbQEuAR4GVgD3pJQaIuLqiBhd0nUcMCNV6ouJ9wEbN25k7dq1ux2CJUmStPvaNEY4pfRgSunIlNKHUkr/Vmy7MqU0u6TPlJTS5e1V6N701ltvcdFFF/GRj3yE008/nY0bNzJt2jSGDBnCoEGDGDt2LBs2bABg4sSJTJ48mZqaGo488kgeeOABAGpraxkzZgwjR46kb9++XHXVVQBceeWVfOtb32ra11e/+lW+/e1vA4W7zo0cORKA6upqXnrpJQDq6+ub2qdMmcKkSZMYOXIkffr04cYbb9xl3cBeqV2SJOmdptwXy+0TnnvuOb7whS/Q0NBA9+7due+++zj77LOpq6tjyZIl9OvXj1tuuaWp/+rVq1m4cCE/+clPmDx5Mps2bQJg4cKF3HfffSxdupR7772X+vp6Jk2axB133AHA1q1bmTFjBueffz4ADz30EKNGjWq1vqeffpqHH36YhQsXctVVV7F58+ad1g3sldolSZLeaQzCLejduzeDBw8G4LjjjmP16tUsW7aMk046iQEDBjB9+nQaGhqa+n/6059mv/32o2/fvvTp04enn34agNNOO40ePXpw0EEHcfbZZ/PEE09QXV1Njx49+PWvf80jjzzCMcccQ48ePQCYP38+J554Yqv1/c3f/A0HHngghx56KIcddhh//OMfd1o3sFdqlyRJeqdp0y2Wc3PggQc2TXfq1ImNGzcyceJEZs2axaBBg6itrWXevHlNfSJiu/W3ze+s/bOf/Sy1tbX87//+L5MmTQJg1apV9OrViwMOOACAzp07s3XrVoCms7Q7q2/Lli07rRto99olSZLeiTwj3Ebr1q3j8MMPZ/PmzUyfPn27Zffeey9bt27l+eefZ9WqVRx11FEAPProo7z88sts3LiRWbNmMXz4cADOOuss5syZQ11dHZ/4xCeAHYdFVFdXs2jRIoCmIQ4dtXZJkqR3oo59RrgNX3e2t1xzzTUMGzaMqqoqhg0bxrp165qWfeADH2Do0KH8+c9/5qabbqJLly4ADB06lLFjx9LY2Mj5559PTU0NAAcccAAf//jH6d69O506dQJgzpw5fOc732na5te+9jUuvPBC/vVf/7XpQrmOWrskSdI7UccOwhVQXV293XfxXnrppU3Tn//851tc59RTT+Wmm27aob1nz57MmjVrh/atW7eyYMEC7r33XgDeeOMN/vCHP1BdXd3U56STTuLZZ5/dYd0pU6ZsN19a667qbq/aJUmS3qkcGrGXLV++nA9/+MOccsop9O3bFyiM7a2vr69wZa1rqXZJkqR3qqjU/S9qampS8/C3YsUK+vXrV5F6tPf4e5Yk7Q23XfTxSpfQogsuHFHpEnbUgYajtoeIWJRSqmne3uHOCHtjun2bv19JktRRdKgg3KVLF9auXWtY2kellFi7dm3TBXmSJEmV1KEuluvZsyeNjY2sWbOm0qWonXTp0oWePXtWugxJkqSOFYT3339/evfuXekyJEmSlIEONTRCkiRJ2lsMwpIkScqSQViSJElZMghLkiQpSwZhSZIkZckgLEmSpCwZhCVJkpQlg7AkSZKyZBCWJElSlgzCkiRJypJBWJIkSVkyCEuSJClLBmFJkiRlySAsSZKkLBmEJUmSlCWDsCRJkrJkEJYkSVKWDMKSJEnKkkFYkiRJWTIIS5IkKUsGYUmSJGWpc6ULUOYWTKl0BTs6fkqlK5AkSXuBQTgjt1308UqXsIMLLhxR6RKkPHXEN6HgG1FJe5VDIyRJkpQlg7AkSZKyZBCWJElSlgzCkiRJypJBWJIkSVlqUxCOiFER8UxErIyIy3fS59MRsTwiGiLirvKWKUmSJJVXq1+fFhGdgKnAaUAjUBcRs1NKy0v69AW+AgxPKb0SEYe1V8GSJElSObTljPBQYGVKaVVK6U1gBjCmWZ+LgKkppVcAUkp/Km+ZkiRJUnm1JQgfAfyuZL6x2FbqSODIiJgfEQsiYlRLG4qIiyOiPiLq16xZ8/YqliRJksqgXBfLdQb6AiOB8cC0iOjevFNK6eaUUk1KqaaqqqpMu5YkSZJ2X1uC8ItAr5L5nsW2Uo3A7JTS5pTSb4BnKQRjSZIkqUNqSxCuA/pGRO+IOAAYB8xu1mcWhbPBRMShFIZKrCpfmZIkSVJ5tRqEU0pbgEuAh4EVwD0ppYaIuDoiRhe7PQysjYjlwM+Ay1JKa9uraEmSJGlPtfr1aQAppQeBB5u1XVkynYB/Lj4kSZKkDs87y0mSpD0WER3uIbXGICxJkqQsGYQlSZKUJYOwJEmSsmQQliRJUpYMwpIkScqSQViSJElZMghLkiQpSwZhSZIkZckgLEmSpCwZhCVJkpQlg7AkSZKyZBCWJElSlgzCkiRJypJBWJIkSVkyCEuSJClLBmFJkiRlySAsSZKkLBmEJUmSlCWDsCRJkrJkEJYkSVKWDMKSJEnKkkFYkiRJWTIIS5IkKUsGYUmSJGXJICxJkqQsGYQlSZKUJYOwJEmSsmQQliRJUpYMwpIkScqSQViSJElZMghLkiQpSwZhSZIkZckgLEmSpCwZhCVJkpQlg3A7iYgO95AkSdJfGIQlSZKUJYOwJEmSsmQQliRJUpYMwpIkScpSm4JwRIyKiGciYmVEXN7C8okRsSYiFhcfny1/qZIkSVL5dG6tQ0R0AqYCpwGNQF1EzE4pLW/W9e6U0iXtUKMkSZJUdm05IzwUWJlSWpVSehOYAYxp37IkSZKk9tWWIHwE8LuS+cZiW3NjI2JpRMyMiF4tbSgiLo6I+oioX7NmzdsoV5IkSSqPcl0sdz9QnVIaCDwK3N5Sp5TSzSmlmpRSTVVVVZl2LUmSJO2+tgThF4HSM7w9i21NUkprU0pvFGe/DxxXnvIkSZKk9tGWIFwH9I2I3hFxADAOmF3aISIOL5kdDawoX4mSJElS+bX6rREppS0RcQnwMNAJuDWl1BARVwP1KaXZwBcjYjSwBXgZmNiONUuSJEl7rNUgDJBSehB4sFnblSXTXwG+Ut7SJEmSpPbjneUkSZKUJYOwJEmSsmQQliRJUpYMwpIkScqSQViSJElZMghLkiQpSwZhSZIkZckgLEmSpCwZhCVJkpQlg7AkSZKyZBCWJElSlgzCkiRJypJBWJIkSVkyCEuSJClLBmFJkiRlqXOlC5Ckfd1tF3280iXs4IILR1S6BEmqOM8IS5IkKUsGYUmSJGXJICxJkqQsGYQlSZKUJYOwJEmSsmQQliRJUpYMwpIkScqSQViSJElZMghLkiQpSwZhSZIkZckgLEmSpCwZhCVJkpQlg7AkSZKyZBCWJElSlgzCkiRJypJBWJIkSVkyCEuSJClLBmFJkiRlySAsSZKkLBmEJUmSlCWDsCRJkrJkEJYkSVKWDMKSJEnKkkFYkiRJWWpTEI6IURHxTESsjIjLd9FvbESkiKgpX4mSJElS+bUahCOiEzAVOAPoD4yPiP4t9OsGfAn4VbmLlCRJksqtLWeEhwIrU0qrUkpvAjOAMS30uwa4DthUxvokSZKkdtGWIHwE8LuS+cZiW5OIOBbolVL6ya42FBEXR0R9RNSvWbNmt4uVJEmSymWPL5aLiP2AbwD/0lrflNLNKaWalFJNVVXVnu5akiRJetvaEoRfBHqVzPcstm3TDfgoMC8iVgPHA7O9YE6SJEkdWVuCcB3QNyJ6R8QBwDhg9raFKaXXUkqHppSqU0rVwAJgdEqpvl0qliRJksqg1SCcUtoCXAI8DKwA7kkpNUTE1RExur0LlCRJktpD57Z0Sik9CDzYrO3KnfQduedlSZIkSe3LO8tJ2qdERId7SJI6JoOwJEmSsmQQliRJUpYMwpIkScqSQViSJElZMghLkiQpSwZhSZIkZckgLEmSpCwZhCVJkpQlg7AkSZKyZBCWJElSlgzCkiRJypJBWJIkSVkyCEuSJClLBmFJkiRlySAsSZKkLBmEJUmSlCWDsCRJkrJkEJYkSVKWDMKSJEnKkkFYkiRJWTIIS5IkKUsGYUmSJGXJICxJkqQsGYQlSZKUJYOwJEmSsmQQliRJUpYMwpIkScqSQViSJElZMghLkiQpSwZhSZIkZckgLEmSpCwZhCVJkpQlg7AkSZKyZBCWJElSlgzCkiRJypJBWJIkSVkyCEuSJClLBmFJkiRlySAsSZKkLLUpCEfEqIh4JiJWRsTlLSyfHBFPRcTiiHgiIvqXv1RJkiSpfFoNwhHRCZgKnAH0B8a3EHTvSikNSCkNBr4OfKPchUqSJEnl1JYzwkOBlSmlVSmlN4EZwJjSDimlP5fMHgyk8pUoSZIklV/nNvQ5AvhdyXwjMKx5p4j4AvDPwAHAX7W0oYi4GLgY4AMf+MDu1ipJkiSVTdkulkspTU0pfQj4v8AVO+lzc0qpJqVUU1VVVa5dS5IkSbutLUH4RaBXyXzPYtvOzAA+tQc1SZIkSe2uLUG4DugbEb0j4gBgHDC7tENE9C2Z/RvgufKVKEmSJJVfq2OEU0pbIuIS4GGgE3BrSqkhIq4G6lNKs4FLIuJUYDPwCjChPYuWJEmS9lRbLpYjpfQg8GCztitLpr9U5rokSZKkduWd5SRJkpQlg7AkSZKyZBCWJElSlgzCkiRJypJBWJIkSVkyCEuSJClLBmFJkiRlySAsSZKkLBmEJUmSlCWDsCRJkrJkEJYkSVKWDMKSJEnKkkFYkiRJWTIIS5IkKUsGYUmSJGXJICxJkqQsGYQlSZKUJYOwJEmSsmQQliRJUpYMwpIkScqSQViSJElZMghLkiQpSwZhSZIkZckgLEmSpCwZhCVJkpQlg7AkSZKyZBCWJElSlgzCkiRJypJBWJIkSVkyCEuSJClLBmFJkiRlySAsSZKkLBmEJUmSlCWDsCRJkrJkEJYkSVKWDMKSJEnKkkFYkiRJWTIIS5IkKUsGYUmSJGXJICxJkqQsGYQlSZKUpTYF4YgYFRHPRMTKiLi8heX/HBHLI2JpRMyNiA+Wv1RJkiSpfFoNwhHRCZgKnAH0B8ZHRP9m3X4N1KSUBgIzga+Xu1BJkiSpnNpyRngosDKltCql9CYwAxhT2iGl9LOU0obi7AKgZ3nLlCRJksqrLUH4COB3JfONxbaduRB4qKUFEXFxRNRHRP2aNWvaXqUkSZJUZmW9WC4izgdqgOtbWp5SujmlVJNSqqmqqirnriVJkqTd0rkNfV4EepXM9yy2bSciTgW+CoxIKb1RnvIkSZKk9tGWM8J1QN+I6B0RBwDjgNmlHSLiGOC/gdEppT+Vv0xJkiSpvFoNwimlLcAlwMPACuCelFJDRFwdEaOL3a4HugL3RsTiiJi9k81JkiRJHUJbhkaQUnoQeLBZ25Ul06eWuS5JkiSpXXlnOUmSJGXJICxJkqQsGYQlSZKUJYOwJEmSsmQQliRJUpYMwpIkScqSQViSJElZMghLkiQpSwZhSZIkZckgLEmSpCwZhCVJkpQlg7AkSZKyZBCWJElSlgzCkiRJypJBWJIkSVkyCEuSJClLBmFJkiRlySAsSZKkLBmEJUmSlCWDsCRJkrJkEJYkSVKWDMKSJEnKkkFYkiRJWTIIS5IkKUsGYUmSJGXJICxJkqQsGYQlSZKUJYOwJEmSsmQQliRJUpYMwpIkScqSQViSJElZMghLkiQpSwZhSZIkZckgLEmSpCwZhCVJkpQlg7AkSZKyZBCWJElSlgzCkiRJypJBWJIkSVkyCEuSJClLbQrCETEqIp6JiJURcXkLy0+OiCcjYktEnFP+MiVJkqTyajUIR0QnYCpwBtAfGB8R/Zt1+y0wEbir3AVKkiRJ7aFzG/oMBVamlFYBRMQMYAywfFuHlNLq4rKt7VCjJEmSVHZtGRpxBPC7kvnGYttui4iLI6I+IurXrFnzdjYhSZIklcVevVgupXRzSqkmpVRTVVW1N3ctSZIkbactQfhFoFfJfM9imyRJkvSO1ZYgXAf0jYjeEXEAMA6Y3b5lSZIkSe2r1SCcUtoCXAI8DKwA7kkpNUTE1RExGiAihkREI3Au8N8R0dCeRUuSJEl7qi3fGkFK6UHgwWZtV5ZM11EYMiFJkiS9I3hnOUmSJGXJICxJkqQsGYQlSZKUJYOwJEmSsmQQliRJUpYMwpIkScqSQViSJElZMghLkiQpSwZhSZIkZckgLEmSpCwZhCVJkpQlg7AkSZKyZBCWJElSlgzCkiRJypJBWJIkSVkyCEuSJClLBmFJkiRlySAsSZKkLBmEJUmSlCWDsCRJkrJkEJYkSVKWDMKSJEnKkkFYkiRJWTIIS5IkKUsGYUmSJGXJICxJkqQsGYQlSZKUJYOwJEmSsmQQliRJUpYMwpIkScqSQViSJElZMghLkiQpSwZhSZIkZckgLEmSpCwZhCVJkpQlg7AkSZKyZBCWJElSlgzCkiRJypJBWJIkSVkyCEuSJClLbQrCETEqIp6JiJURcXkLyw+MiLuLy38VEdVlr1SSJEkqo1aDcER0AqYCZwD9gfER0b9ZtwuBV1JKHwa+CVxX7kIlSZKkcmrLGeGhwMqU0qqU0pvADGBMsz5jgNuL0zOBUyIiylemJEmSVF6RUtp1h4hzgFEppc8W5/8eGJZSuqSkz7Jin8bi/PPFPi8129bFwMXF2aOAZ8r1g2ivOxR4qdVeksrNY0+qDI+9d7YPppSqmjd23psVpJRuBm7em/tU+4iI+pRSTaXrkHLjsSdVhsfevqktQyNeBHqVzPcstrXYJyI6A4cAa8tRoCRJktQe2hKE64C+EdE7Ig4AxgGzm/WZDUwoTp8D/DS1NuZCkiRJqqBWh0aklLZExCXAw0An4NaUUkNEXA3Up5RmA7cAd0bESuBlCmFZ+zaHuEiV4bEnVYbH3j6o1YvlJEmSpH2Rd5aTJElSlgzCkiRJypJBeB8VEW9FxOKSR3VEjIyIB1pZryYibmylz4CS7b4cEb8pTj9WXD44IlJEjNpJTcsi4v6I6F5s3y8ibiy2PxURdRHRew+fAqlVEdGj5G/5fyPixZL5A8q8r6OL2/11RHyonNsup4iYFxHPFGtdUfz+993dxsSIeH971KeOz+NqRyXH1ZLia9zgVvpPiYhL93Cf/29P1s/FXv0eYe1VG1NKg0sbIqK6tZVSSvVAffP2iOicUtpS7PMUMLjYXgs8kFKaWdJ9PPBE8d85LdUUEbcDXwD+Dfhb4P3AwJTS1ojoCbzehp9R2iMppbX85W95CrA+pXTDtuWlf/dl8ClgZkrp2rZ0Lt6dM1JKW8ux8938Wc5LKdVHxHuB5yOitnhn0bbspxMwEVgG/P7tVat3Mo+rndp2XF0AXA+cVo4aWqgpgAD+H/Dv7bGPfYlnhDMVEUMj4pfFd9G/iIijiu1NZ42L70jvjIj5wJ1t3G4A51J4ITwtIrrspOsvgSOK04cDf9j2H1NKqTGl9Mrb/uGkPRARtRFxU0T8Cvj6Lo6ViRHxo4iYExHPRcTXi+2ditvY9gnHlyPir4F/Aj4fET8r9vvnYp9lEfFPxbbq4lmjOygEyZMi4uni9p6NiOkRcWpEzC/uc2hxvYMj4taIWFisc0xJjbMj4qfA3Ig4PCIej798MnNSK09HVwpvSt8qbu97EVEfEQ0RcVXJc7Y6Iq6LiCcpvAGuAaYX93NQOX4vemfzuNpO0+tfRLw3ImZFxNKIWBARA0v6DSo+R89FxEUlz+VlUTirvHTbcdjCz3gLcFCxpunFPrMiYlHx+N3tT3r2WSklH/vgg8IL1+Li48fFtpEUzt4CvBvoXJw+FbivhT5TgEXAQbvYTy1wTsn8cGBucfouYGzJsvXFfzsB91K4LTcUbtKyuljrfwHHVPr585Hfo/j3fmnxb/oBoFOxfWfHykRgFYUbCHUBXqBwY6HjgEdLttu9dPvF6eOAp4CDKYTNBuAYoBrYChxf7FcNbAEGUDhxsQi4lcLZnjHArGK/fwfO37Y/4NniticCjcB7i8v+BfhqcboT0K2F52Ee8AywFNgIfK5k2XtL1p1H4VMcisfv/2m2jZpK/059VP7hcdVUb9MxQSG8/3tx+jvA14rTfwUsLvm5lgAHUbi18+8ofHJ6OoWvcYti7Q8AJzf/GYvbWN+shm31HkQhLPeo9N9HR3g4NGLftcPQiGYOAW6PiL5AAvbfSb/ZKaWNu7Hf8cCM4vQM4DPAfcX5gyJiMYV3wiuAR6FwBrh4NuCvio+5EXFuSmnubuxXKqd7U0pvFad3dazMTSm9BhARy4EPUnjx7RMR3wF+AjzSwvZPpPAG9fXiuj8CTqJwc6IXUkoLSvr+JhWGIxERDcV9poh4isKLHxReHEfHX8YUdgE+UJx+NKX0cnG6Drg1Ivan8GK/eCc//7aPcKuAX0TEnJTSC8Cni2eSOlP4JKc/hcAMcPdOtiVtk/txNT0KY6S7Uhw6Uqx5LEBK6adRGF/97uKy/ym+/m4snvEeWux/OvDrYp+uQF/gty38jM19MSLOKk73Kq6X/V2AHRqRr2uAn6WUPgp8ksIB3pI2j9WNwtjAscCVEbGawjvdURHRrdhlWzj/IIV3s1/Ytm5K6Y2U0kMppcsovAv/1G79NFJ5lf7d7+pYeaNk+i0KZ7heAQZROAM0Gfj+Huy7+T62lsxv5S/XeQSFT18GFx8fSCmtaL69lNLjFM4evQjURsRndlVISmkN8CQwLAoXsF4KnJJSGkghjJQ+F47rV2tyP67OA/oAt1N4fWxN8xs9pGJN/1FS04dTSrfs5GdsEhEjKZx5PyGlNIhCkN7Z635WDML5OoTCQQuFj3nK4RRgaUqpV0qpOqX0QQpng88q7ZRS2gB8EfiXiOgcEcdG8QrziNgPGEjh4zCpI9itYyUiDgX2SyndB1wBHNtCt58Dn4qId0XEwRSOkZ/vQY0PA/8YEVGs4Zid1PZB4I8ppWkUgkRLtZX2fxeFj5afp/BR9uvAaxHxPuCMXay6Dui2i+VSlsdVSikB/wocHxFHF+s7r7idkcBLKaU/F7uPiYguEdGDwrDFumJNkyKia3GdIyLisJ3sbnPxLDUUnu9XUkobivs9vg0/fxYcGpGvr1P4WOoKCmd2ymE88ONmbfcBnwfuKG1MKf06IpYW11kDTIuIA4uLFwLfLVNN0p7a3WPlCOC24ps6gK8075BSejIK37iysNj0/eIxUf02a7wG+BawtLjf3wBnttBvJHBZRGwG1lMYutSS6RGxETgQqE0pLQKIiF8DT1MYrzh/F/XUAjcVt3HCbg6vUh5yPK621bkxIv4LuKz4uLX4ergBmFDSdSnwMwpjhK9JKf0e+H1E9AN+Wczn64HzKV7Q2szNxdqfBCYBkyNiBYVrAHY1hCIr3mJZkiRJWXJohCRJkrJkEJYkSVKWDMKSJEnKkkFYkiRJWTIIS5IkKUsGYUmSJGXJICxJkqQs/X/oSmFhS97dWQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x504 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = pd.DataFrame.from_dict(alternative_results)\n",
    "df.plot(kind='bar', figsize=(12,7), colormap='copper', rot=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "fleet-beast",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>positive/negative</th>\n",
       "      <th>good/bad</th>\n",
       "      <th>happy/unhappy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Flair TARS</th>\n",
       "      <td>0.517</td>\n",
       "      <td>0.619</td>\n",
       "      <td>0.178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Transformers Bart</th>\n",
       "      <td>0.388</td>\n",
       "      <td>0.458</td>\n",
       "      <td>0.619</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Transformers Roberta</th>\n",
       "      <td>0.588</td>\n",
       "      <td>0.588</td>\n",
       "      <td>0.588</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      positive/negative  good/bad  happy/unhappy\n",
       "Flair TARS                        0.517     0.619          0.178\n",
       "Transformers Bart                 0.388     0.458          0.619\n",
       "Transformers Roberta              0.588     0.588          0.588"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "champion-olive",
   "metadata": {},
   "source": [
    "## Few-shot learning\n",
    "\n",
    "As we saw with polarity classification, the TARS classifier tends to suffer from low recall. In tasks with a default class, such as `neutral`, this label can serve as a fallback option, but in other cases we need a different solution.\n",
    "Luckily, Flairs has made it easy to finetune TARS on a handful of training examples. Let’s see what happens if we give the model one example of what we mean by each class, and finetune it on this small training set. Because the performance of the final model will depend on what training instances we pick, we repeat this process ten times and always select random examples from the training corpus.\n",
    "\n",
    "The figure below shows that the TARS classifier benefits greatly from this few-shot learning procedure. The impact is clearest for the three tasks without a default class: TARS’s accuracy jumps significantly, and in two out of three cases it becomes competitive with the best NLI model. Interestingly, this even happens for the emotion dataset, where we’ve used just four examples as our training set. For the polarity tasks, the benefit is less clear, as we already fixed the recall problem by introducing a default class, and only worked with three labeled examples for few-shot learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "boring-framing",
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.data import Corpus\n",
    "from flair.datasets import SentenceDataset\n",
    "\n",
    "def sample_training_corpus(dataset, seed):\n",
    "\n",
    "    random.seed(seed)\n",
    "    \n",
    "    seen_labels = set()\n",
    "    sentences = []\n",
    "\n",
    "    train_set = list(zip(dataset['train_texts'], dataset['train_labels']))\n",
    "    random.shuffle(train_set)\n",
    "    \n",
    "    for text, label in train_set:\n",
    "        topic = dataset['class_names'][label]\n",
    "        if topic not in seen_labels:\n",
    "            sentences.append(Sentence(text).add_label(\"_or_\".join(dataset['class_names']), topic))\n",
    "            seen_labels.add(topic)\n",
    "        if len(seen_labels) == len(dataset['class_names']):\n",
    "            break\n",
    "\n",
    "    train = SentenceDataset(sentences)\n",
    "    corpus = Corpus(train=train)\n",
    "    return corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "perfect-juvenile",
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.trainers import ModelTrainer\n",
    "\n",
    "def train(corpus, dataset):\n",
    "\n",
    "    tars = TARSClassifier.load('tars-base')\n",
    "    tars.add_and_switch_to_new_task(dataset['name'], label_dictionary=corpus.make_label_dictionary())\n",
    "\n",
    "    trainer = ModelTrainer(tars, corpus)\n",
    "\n",
    "    trainer.train(base_path='/tmp/' + dataset['name'], # path to store the model artifacts\n",
    "              learning_rate=0.02, # use very small learning rate\n",
    "              mini_batch_size=1, # small mini-batch size since corpus is tiny\n",
    "              max_epochs=10, # terminate after 10 epochs\n",
    "              train_with_dev=False,\n",
    "              )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "inside-underground",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(dataset, default_name='neutral'):\n",
    "    tars = TARSClassifier.load(f'/tmp/{dataset[\"name\"]}/final-model.pt')\n",
    "\n",
    "    total, correct = 0, 0\n",
    "    for item, gold_label_idx in tqdm(zip(dataset[\"test_texts\"], dataset[\"test_labels\"]), total = len(dataset[\"test_texts\"])):\n",
    "        sentence = Sentence(item)\n",
    "        tars.predict(sentence)\n",
    "        sorted_labels = sorted(sentence.to_dict()['labels'], key=lambda k: k['confidence'], reverse=True)\n",
    "        \n",
    "        gold_label = dataset[\"class_names\"][gold_label_idx]\n",
    "        if len(sorted_labels) > 0:\n",
    "            predicted_label = sorted_labels[0]['value']\n",
    "        else:\n",
    "            predicted_label = default_name                \n",
    "\n",
    "        if predicted_label == gold_label:\n",
    "            correct += 1\n",
    "\n",
    "        total += 1\n",
    "\n",
    "    return correct/total"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "sufficient-substance",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:10:45,968 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:10:49,908 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 9656.88it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:10:49,911 [b'negative', b'neutral', b'positive']\n",
      "2021-05-25 12:10:49,912 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:49,914 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:10:49,915 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:49,915 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:10:49,915 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:49,915 Parameters:\n",
      "2021-05-25 12:10:49,915  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:10:49,916  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:10:49,916  - patience: \"3\"\n",
      "2021-05-25 12:10:49,916  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:10:49,916  - max_epochs: \"10\"\n",
      "2021-05-25 12:10:49,917  - shuffle: \"True\"\n",
      "2021-05-25 12:10:49,917  - train_with_dev: \"False\"\n",
      "2021-05-25 12:10:49,917  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:10:49,917 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:49,918 Model training base path: \"/tmp/tweet_eval: sentiment\"\n",
      "2021-05-25 12:10:49,918 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:49,918 Device: cuda:0\n",
      "2021-05-25 12:10:49,918 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:49,918 Embeddings storage mode: cpu\n",
      "2021-05-25 12:10:49,921 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:49,970 epoch 1 - iter 1/3 - loss 0.30214882 - samples/sec: 26.49 - lr: 0.020000\n",
      "2021-05-25 12:10:50,013 epoch 1 - iter 2/3 - loss 0.67939276 - samples/sec: 23.44 - lr: 0.020000\n",
      "2021-05-25 12:10:50,059 epoch 1 - iter 3/3 - loss 0.84909026 - samples/sec: 21.94 - lr: 0.020000\n",
      "2021-05-25 12:10:50,060 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:50,061 EPOCH 1 done: loss 0.8491 - lr 0.0200000\n",
      "2021-05-25 12:10:50,061 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:10:50,742 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:50,799 epoch 2 - iter 1/3 - loss 0.43752971 - samples/sec: 22.85 - lr: 0.020000\n",
      "2021-05-25 12:10:50,845 epoch 2 - iter 2/3 - loss 0.24862534 - samples/sec: 22.04 - lr: 0.020000\n",
      "2021-05-25 12:10:50,888 epoch 2 - iter 3/3 - loss 0.33462723 - samples/sec: 23.54 - lr: 0.020000\n",
      "2021-05-25 12:10:50,889 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:50,889 EPOCH 2 done: loss 0.3346 - lr 0.0200000\n",
      "2021-05-25 12:10:50,890 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:10:50,890 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:50,946 epoch 3 - iter 1/3 - loss 0.16561399 - samples/sec: 22.18 - lr: 0.020000\n",
      "2021-05-25 12:10:50,986 epoch 3 - iter 2/3 - loss 0.09413094 - samples/sec: 24.92 - lr: 0.020000\n",
      "2021-05-25 12:10:51,026 epoch 3 - iter 3/3 - loss 0.06390932 - samples/sec: 25.41 - lr: 0.020000\n",
      "2021-05-25 12:10:51,027 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:51,027 EPOCH 3 done: loss 0.0639 - lr 0.0200000\n",
      "2021-05-25 12:10:51,027 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:10:51,028 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:51,078 epoch 4 - iter 1/3 - loss 0.00305355 - samples/sec: 25.30 - lr: 0.020000\n",
      "2021-05-25 12:10:51,120 epoch 4 - iter 2/3 - loss 0.08544377 - samples/sec: 24.06 - lr: 0.020000\n",
      "2021-05-25 12:10:51,158 epoch 4 - iter 3/3 - loss 0.06022818 - samples/sec: 26.13 - lr: 0.020000\n",
      "2021-05-25 12:10:51,159 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:51,160 EPOCH 4 done: loss 0.0602 - lr 0.0200000\n",
      "2021-05-25 12:10:51,160 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:10:51,160 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:51,211 epoch 5 - iter 1/3 - loss 0.01271538 - samples/sec: 25.04 - lr: 0.020000\n",
      "2021-05-25 12:10:51,250 epoch 5 - iter 2/3 - loss 0.00946203 - samples/sec: 25.97 - lr: 0.020000\n",
      "2021-05-25 12:10:51,292 epoch 5 - iter 3/3 - loss 0.08555255 - samples/sec: 23.92 - lr: 0.020000\n",
      "2021-05-25 12:10:51,293 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:51,293 EPOCH 5 done: loss 0.0856 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:10:51,293 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:10:51,294 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:51,346 epoch 6 - iter 1/3 - loss 0.00306069 - samples/sec: 23.74 - lr: 0.010000\n",
      "2021-05-25 12:10:51,388 epoch 6 - iter 2/3 - loss 0.00551691 - samples/sec: 24.12 - lr: 0.010000\n",
      "2021-05-25 12:10:51,428 epoch 6 - iter 3/3 - loss 0.01290372 - samples/sec: 25.35 - lr: 0.010000\n",
      "2021-05-25 12:10:51,429 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:51,429 EPOCH 6 done: loss 0.0129 - lr 0.0100000\n",
      "2021-05-25 12:10:51,429 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:10:51,430 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:51,485 epoch 7 - iter 1/3 - loss 0.01023954 - samples/sec: 22.62 - lr: 0.010000\n",
      "2021-05-25 12:10:51,527 epoch 7 - iter 2/3 - loss 0.00547479 - samples/sec: 23.77 - lr: 0.010000\n",
      "2021-05-25 12:10:51,568 epoch 7 - iter 3/3 - loss 0.00688729 - samples/sec: 24.92 - lr: 0.010000\n",
      "2021-05-25 12:10:51,569 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:51,569 EPOCH 7 done: loss 0.0069 - lr 0.0100000\n",
      "2021-05-25 12:10:51,569 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:10:51,570 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:51,620 epoch 8 - iter 1/3 - loss 0.00112509 - samples/sec: 24.90 - lr: 0.010000\n",
      "2021-05-25 12:10:51,662 epoch 8 - iter 2/3 - loss 0.00545284 - samples/sec: 24.38 - lr: 0.010000\n",
      "2021-05-25 12:10:51,702 epoch 8 - iter 3/3 - loss 0.00543684 - samples/sec: 25.16 - lr: 0.010000\n",
      "2021-05-25 12:10:51,703 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:51,703 EPOCH 8 done: loss 0.0054 - lr 0.0100000\n",
      "2021-05-25 12:10:51,704 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:10:51,704 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:51,758 epoch 9 - iter 1/3 - loss 0.00396186 - samples/sec: 23.35 - lr: 0.010000\n",
      "2021-05-25 12:10:51,800 epoch 9 - iter 2/3 - loss 0.00614024 - samples/sec: 23.63 - lr: 0.010000\n",
      "2021-05-25 12:10:51,841 epoch 9 - iter 3/3 - loss 0.00562329 - samples/sec: 24.72 - lr: 0.010000\n",
      "2021-05-25 12:10:51,842 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:51,842 EPOCH 9 done: loss 0.0056 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:10:51,843 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:10:51,843 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:51,894 epoch 10 - iter 1/3 - loss 0.00361438 - samples/sec: 24.89 - lr: 0.005000\n",
      "2021-05-25 12:10:51,933 epoch 10 - iter 2/3 - loss 0.00516002 - samples/sec: 25.63 - lr: 0.005000\n",
      "2021-05-25 12:10:51,973 epoch 10 - iter 3/3 - loss 0.00368310 - samples/sec: 25.24 - lr: 0.005000\n",
      "2021-05-25 12:10:51,974 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:10:51,975 EPOCH 10 done: loss 0.0037 - lr 0.0050000\n",
      "2021-05-25 12:10:51,975 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:10:52,643 Test data not provided setting final score to 0\n",
      "2021-05-25 12:10:52,654 loading file /tmp/tweet_eval: sentiment/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "830e3a85a86749329a6c452dd57643ee",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.522\n",
      "2021-05-25 12:11:06,271 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:11:10,001 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 11607.85it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:11:10,003 [b'positive', b'negative', b'neutral']\n",
      "2021-05-25 12:11:10,004 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:10,005 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:11:10,006 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:10,006 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:11:10,007 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:10,007 Parameters:\n",
      "2021-05-25 12:11:10,007  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:11:10,008  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:11:10,008  - patience: \"3\"\n",
      "2021-05-25 12:11:10,009  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:11:10,009  - max_epochs: \"10\"\n",
      "2021-05-25 12:11:10,009  - shuffle: \"True\"\n",
      "2021-05-25 12:11:10,010  - train_with_dev: \"False\"\n",
      "2021-05-25 12:11:10,010  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:11:10,010 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:10,010 Model training base path: \"/tmp/tweet_eval: sentiment\"\n",
      "2021-05-25 12:11:10,011 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:10,011 Device: cuda:0\n",
      "2021-05-25 12:11:10,011 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:10,011 Embeddings storage mode: cpu\n",
      "2021-05-25 12:11:10,014 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:10,065 epoch 1 - iter 1/3 - loss 1.99796259 - samples/sec: 25.10 - lr: 0.020000\n",
      "2021-05-25 12:11:10,111 epoch 1 - iter 2/3 - loss 2.05324358 - samples/sec: 22.14 - lr: 0.020000\n",
      "2021-05-25 12:11:10,159 epoch 1 - iter 3/3 - loss 1.54521847 - samples/sec: 20.96 - lr: 0.020000\n",
      "2021-05-25 12:11:10,160 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:10,160 EPOCH 1 done: loss 1.5452 - lr 0.0200000\n",
      "2021-05-25 12:11:10,160 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:11:10,850 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:10,907 epoch 2 - iter 1/3 - loss 0.48396707 - samples/sec: 22.52 - lr: 0.020000\n",
      "2021-05-25 12:11:10,953 epoch 2 - iter 2/3 - loss 0.60163066 - samples/sec: 21.87 - lr: 0.020000\n",
      "2021-05-25 12:11:10,996 epoch 2 - iter 3/3 - loss 0.50394394 - samples/sec: 23.62 - lr: 0.020000\n",
      "2021-05-25 12:11:10,997 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:10,997 EPOCH 2 done: loss 0.5039 - lr 0.0200000\n",
      "2021-05-25 12:11:10,998 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:11:10,998 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:11,054 epoch 3 - iter 1/3 - loss 0.20033677 - samples/sec: 22.30 - lr: 0.020000\n",
      "2021-05-25 12:11:11,098 epoch 3 - iter 2/3 - loss 0.29501208 - samples/sec: 22.81 - lr: 0.020000\n",
      "2021-05-25 12:11:11,143 epoch 3 - iter 3/3 - loss 0.23272274 - samples/sec: 22.37 - lr: 0.020000\n",
      "2021-05-25 12:11:11,144 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:11,144 EPOCH 3 done: loss 0.2327 - lr 0.0200000\n",
      "2021-05-25 12:11:11,144 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:11:11,145 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:11,195 epoch 4 - iter 1/3 - loss 0.02337860 - samples/sec: 25.11 - lr: 0.020000\n",
      "2021-05-25 12:11:11,238 epoch 4 - iter 2/3 - loss 0.05990312 - samples/sec: 23.76 - lr: 0.020000\n",
      "2021-05-25 12:11:11,282 epoch 4 - iter 3/3 - loss 0.09250412 - samples/sec: 22.99 - lr: 0.020000\n",
      "2021-05-25 12:11:11,283 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:11,283 EPOCH 4 done: loss 0.0925 - lr 0.0200000\n",
      "2021-05-25 12:11:11,283 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:11:11,284 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:11,338 epoch 5 - iter 1/3 - loss 0.12213853 - samples/sec: 22.90 - lr: 0.020000\n",
      "2021-05-25 12:11:11,377 epoch 5 - iter 2/3 - loss 0.07287907 - samples/sec: 25.65 - lr: 0.020000\n",
      "2021-05-25 12:11:11,418 epoch 5 - iter 3/3 - loss 0.05181458 - samples/sec: 24.85 - lr: 0.020000\n",
      "2021-05-25 12:11:11,419 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:11,419 EPOCH 5 done: loss 0.0518 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:11:11,419 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:11:11,420 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:11,470 epoch 6 - iter 1/3 - loss 0.00072876 - samples/sec: 25.30 - lr: 0.010000\n",
      "2021-05-25 12:11:11,514 epoch 6 - iter 2/3 - loss 0.00981074 - samples/sec: 22.71 - lr: 0.010000\n",
      "2021-05-25 12:11:11,555 epoch 6 - iter 3/3 - loss 0.01135186 - samples/sec: 24.97 - lr: 0.010000\n",
      "2021-05-25 12:11:11,556 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:11,556 EPOCH 6 done: loss 0.0114 - lr 0.0100000\n",
      "2021-05-25 12:11:11,556 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:11:11,557 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:11,608 epoch 7 - iter 1/3 - loss 0.00833548 - samples/sec: 24.43 - lr: 0.010000\n",
      "2021-05-25 12:11:11,648 epoch 7 - iter 2/3 - loss 0.00534345 - samples/sec: 25.67 - lr: 0.010000\n",
      "2021-05-25 12:11:11,687 epoch 7 - iter 3/3 - loss 0.00895981 - samples/sec: 25.53 - lr: 0.010000\n",
      "2021-05-25 12:11:11,688 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:11,688 EPOCH 7 done: loss 0.0090 - lr 0.0100000\n",
      "2021-05-25 12:11:11,689 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:11:11,689 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:11,739 epoch 8 - iter 1/3 - loss 0.00188707 - samples/sec: 25.26 - lr: 0.010000\n",
      "2021-05-25 12:11:11,781 epoch 8 - iter 2/3 - loss 0.00510533 - samples/sec: 24.55 - lr: 0.010000\n",
      "2021-05-25 12:11:11,820 epoch 8 - iter 3/3 - loss 0.00867850 - samples/sec: 25.80 - lr: 0.010000\n",
      "2021-05-25 12:11:11,821 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:11,821 EPOCH 8 done: loss 0.0087 - lr 0.0100000\n",
      "2021-05-25 12:11:11,821 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:11:11,822 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:11,877 epoch 9 - iter 1/3 - loss 0.00648296 - samples/sec: 22.42 - lr: 0.010000\n",
      "2021-05-25 12:11:11,919 epoch 9 - iter 2/3 - loss 0.00480361 - samples/sec: 24.16 - lr: 0.010000\n",
      "2021-05-25 12:11:11,959 epoch 9 - iter 3/3 - loss 0.00544241 - samples/sec: 24.88 - lr: 0.010000\n",
      "2021-05-25 12:11:11,960 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:11,960 EPOCH 9 done: loss 0.0054 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:11:11,961 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:11:11,961 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:12,012 epoch 10 - iter 1/3 - loss 0.01060913 - samples/sec: 24.91 - lr: 0.005000\n",
      "2021-05-25 12:11:12,053 epoch 10 - iter 2/3 - loss 0.00687536 - samples/sec: 24.51 - lr: 0.005000\n",
      "2021-05-25 12:11:12,092 epoch 10 - iter 3/3 - loss 0.00805358 - samples/sec: 25.97 - lr: 0.005000\n",
      "2021-05-25 12:11:12,093 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:12,093 EPOCH 10 done: loss 0.0081 - lr 0.0050000\n",
      "2021-05-25 12:11:12,094 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:11:12,773 Test data not provided setting final score to 0\n",
      "2021-05-25 12:11:12,780 loading file /tmp/tweet_eval: sentiment/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fa8181084aad42fda9b2723ef414372c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.584\n",
      "2021-05-25 12:11:26,301 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:11:29,978 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 9754.20it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:11:29,981 [b'positive', b'negative', b'neutral']\n",
      "2021-05-25 12:11:29,982 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:29,984 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:11:29,985 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:29,985 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:11:29,986 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:29,986 Parameters:\n",
      "2021-05-25 12:11:29,987  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:11:29,987  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:11:29,987  - patience: \"3\"\n",
      "2021-05-25 12:11:29,988  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:11:29,988  - max_epochs: \"10\"\n",
      "2021-05-25 12:11:29,988  - shuffle: \"True\"\n",
      "2021-05-25 12:11:29,988  - train_with_dev: \"False\"\n",
      "2021-05-25 12:11:29,988  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:11:29,989 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:29,989 Model training base path: \"/tmp/tweet_eval: sentiment\"\n",
      "2021-05-25 12:11:29,989 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:29,989 Device: cuda:0\n",
      "2021-05-25 12:11:29,990 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:29,990 Embeddings storage mode: cpu\n",
      "2021-05-25 12:11:30,002 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:30,056 epoch 1 - iter 1/3 - loss 0.77249020 - samples/sec: 24.70 - lr: 0.020000\n",
      "2021-05-25 12:11:30,104 epoch 1 - iter 2/3 - loss 0.93706074 - samples/sec: 21.49 - lr: 0.020000\n",
      "2021-05-25 12:11:30,149 epoch 1 - iter 3/3 - loss 0.78394372 - samples/sec: 22.33 - lr: 0.020000\n",
      "2021-05-25 12:11:30,150 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:30,151 EPOCH 1 done: loss 0.7839 - lr 0.0200000\n",
      "2021-05-25 12:11:30,152 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:11:30,827 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:30,884 epoch 2 - iter 1/3 - loss 0.22854643 - samples/sec: 22.67 - lr: 0.020000\n",
      "2021-05-25 12:11:30,929 epoch 2 - iter 2/3 - loss 0.12994959 - samples/sec: 22.64 - lr: 0.020000\n",
      "2021-05-25 12:11:30,972 epoch 2 - iter 3/3 - loss 0.10878482 - samples/sec: 23.57 - lr: 0.020000\n",
      "2021-05-25 12:11:30,973 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:30,973 EPOCH 2 done: loss 0.1088 - lr 0.0200000\n",
      "2021-05-25 12:11:30,974 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:11:30,974 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:31,027 epoch 3 - iter 1/3 - loss 0.02111939 - samples/sec: 24.59 - lr: 0.020000\n",
      "2021-05-25 12:11:31,071 epoch 3 - iter 2/3 - loss 0.02268536 - samples/sec: 23.16 - lr: 0.020000\n",
      "2021-05-25 12:11:31,114 epoch 3 - iter 3/3 - loss 0.11907534 - samples/sec: 23.42 - lr: 0.020000\n",
      "2021-05-25 12:11:31,115 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:31,116 EPOCH 3 done: loss 0.1191 - lr 0.0200000\n",
      "2021-05-25 12:11:31,116 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:11:31,117 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:31,171 epoch 4 - iter 1/3 - loss 0.00126142 - samples/sec: 23.27 - lr: 0.020000\n",
      "2021-05-25 12:11:31,212 epoch 4 - iter 2/3 - loss 0.00288134 - samples/sec: 24.88 - lr: 0.020000\n",
      "2021-05-25 12:11:31,256 epoch 4 - iter 3/3 - loss 0.02353404 - samples/sec: 23.22 - lr: 0.020000\n",
      "2021-05-25 12:11:31,257 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:31,258 EPOCH 4 done: loss 0.0235 - lr 0.0200000\n",
      "2021-05-25 12:11:31,258 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:11:31,259 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:31,311 epoch 5 - iter 1/3 - loss 0.02827183 - samples/sec: 24.71 - lr: 0.020000\n",
      "2021-05-25 12:11:31,354 epoch 5 - iter 2/3 - loss 0.01504430 - samples/sec: 23.80 - lr: 0.020000\n",
      "2021-05-25 12:11:31,394 epoch 5 - iter 3/3 - loss 0.01169710 - samples/sec: 25.16 - lr: 0.020000\n",
      "2021-05-25 12:11:31,395 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:31,396 EPOCH 5 done: loss 0.0117 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:11:31,397 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:11:31,398 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:31,451 epoch 6 - iter 1/3 - loss 0.00203042 - samples/sec: 23.49 - lr: 0.010000\n",
      "2021-05-25 12:11:31,492 epoch 6 - iter 2/3 - loss 0.00523727 - samples/sec: 25.11 - lr: 0.010000\n",
      "2021-05-25 12:11:31,532 epoch 6 - iter 3/3 - loss 0.00389241 - samples/sec: 24.92 - lr: 0.010000\n",
      "2021-05-25 12:11:31,534 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:31,534 EPOCH 6 done: loss 0.0039 - lr 0.0100000\n",
      "2021-05-25 12:11:31,535 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:11:31,535 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:31,590 epoch 7 - iter 1/3 - loss 0.00099605 - samples/sec: 23.05 - lr: 0.010000\n",
      "2021-05-25 12:11:31,632 epoch 7 - iter 2/3 - loss 0.00276554 - samples/sec: 24.73 - lr: 0.010000\n",
      "2021-05-25 12:11:31,673 epoch 7 - iter 3/3 - loss 0.00320495 - samples/sec: 24.39 - lr: 0.010000\n",
      "2021-05-25 12:11:31,674 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:31,675 EPOCH 7 done: loss 0.0032 - lr 0.0100000\n",
      "2021-05-25 12:11:31,676 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:11:31,676 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:31,727 epoch 8 - iter 1/3 - loss 0.00152601 - samples/sec: 25.75 - lr: 0.010000\n",
      "2021-05-25 12:11:31,766 epoch 8 - iter 2/3 - loss 0.00250854 - samples/sec: 25.51 - lr: 0.010000\n",
      "2021-05-25 12:11:31,807 epoch 8 - iter 3/3 - loss 0.00363007 - samples/sec: 25.19 - lr: 0.010000\n",
      "2021-05-25 12:11:31,808 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:31,808 EPOCH 8 done: loss 0.0036 - lr 0.0100000\n",
      "2021-05-25 12:11:31,809 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:11:31,810 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:31,865 epoch 9 - iter 1/3 - loss 0.00324687 - samples/sec: 22.42 - lr: 0.010000\n",
      "2021-05-25 12:11:31,906 epoch 9 - iter 2/3 - loss 0.00382959 - samples/sec: 24.59 - lr: 0.010000\n",
      "2021-05-25 12:11:31,948 epoch 9 - iter 3/3 - loss 0.00275257 - samples/sec: 24.83 - lr: 0.010000\n",
      "2021-05-25 12:11:31,949 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:31,949 EPOCH 9 done: loss 0.0028 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:11:31,950 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:11:31,951 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:32,003 epoch 10 - iter 1/3 - loss 0.00591271 - samples/sec: 24.12 - lr: 0.005000\n",
      "2021-05-25 12:11:32,045 epoch 10 - iter 2/3 - loss 0.00374360 - samples/sec: 24.31 - lr: 0.005000\n",
      "2021-05-25 12:11:32,085 epoch 10 - iter 3/3 - loss 0.00286210 - samples/sec: 24.79 - lr: 0.005000\n",
      "2021-05-25 12:11:32,087 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:32,087 EPOCH 10 done: loss 0.0029 - lr 0.0050000\n",
      "2021-05-25 12:11:32,088 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:11:32,765 Test data not provided setting final score to 0\n",
      "2021-05-25 12:11:32,772 loading file /tmp/tweet_eval: sentiment/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e493d0990c16421fa3dc7217ab0e1a6d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.501\n",
      "2021-05-25 12:11:46,559 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:11:50,328 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 9198.04it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:11:50,331 [b'neutral', b'positive', b'negative']\n",
      "2021-05-25 12:11:50,332 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:50,333 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:11:50,334 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:50,334 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:11:50,335 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:50,335 Parameters:\n",
      "2021-05-25 12:11:50,335  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:11:50,335  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:11:50,336  - patience: \"3\"\n",
      "2021-05-25 12:11:50,336  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:11:50,336  - max_epochs: \"10\"\n",
      "2021-05-25 12:11:50,336  - shuffle: \"True\"\n",
      "2021-05-25 12:11:50,336  - train_with_dev: \"False\"\n",
      "2021-05-25 12:11:50,337  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:11:50,337 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:50,337 Model training base path: \"/tmp/tweet_eval: sentiment\"\n",
      "2021-05-25 12:11:50,337 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:50,338 Device: cuda:0\n",
      "2021-05-25 12:11:50,338 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:50,338 Embeddings storage mode: cpu\n",
      "2021-05-25 12:11:50,341 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:50,394 epoch 1 - iter 1/3 - loss 0.71757287 - samples/sec: 24.45 - lr: 0.020000\n",
      "2021-05-25 12:11:50,439 epoch 1 - iter 2/3 - loss 1.17316869 - samples/sec: 22.42 - lr: 0.020000\n",
      "2021-05-25 12:11:50,484 epoch 1 - iter 3/3 - loss 1.38578580 - samples/sec: 22.51 - lr: 0.020000\n",
      "2021-05-25 12:11:50,485 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:50,485 EPOCH 1 done: loss 1.3858 - lr 0.0200000\n",
      "2021-05-25 12:11:50,486 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:11:51,132 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:51,200 epoch 2 - iter 1/3 - loss 0.41373849 - samples/sec: 18.78 - lr: 0.020000\n",
      "2021-05-25 12:11:51,248 epoch 2 - iter 2/3 - loss 0.29808506 - samples/sec: 21.21 - lr: 0.020000\n",
      "2021-05-25 12:11:51,291 epoch 2 - iter 3/3 - loss 0.21795232 - samples/sec: 23.30 - lr: 0.020000\n",
      "2021-05-25 12:11:51,292 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:51,293 EPOCH 2 done: loss 0.2180 - lr 0.0200000\n",
      "2021-05-25 12:11:51,293 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:11:51,294 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:51,349 epoch 3 - iter 1/3 - loss 0.00415832 - samples/sec: 23.33 - lr: 0.020000\n",
      "2021-05-25 12:11:51,392 epoch 3 - iter 2/3 - loss 0.28565705 - samples/sec: 23.34 - lr: 0.020000\n",
      "2021-05-25 12:11:51,437 epoch 3 - iter 3/3 - loss 0.22408211 - samples/sec: 22.81 - lr: 0.020000\n",
      "2021-05-25 12:11:51,438 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:51,439 EPOCH 3 done: loss 0.2241 - lr 0.0200000\n",
      "2021-05-25 12:11:51,439 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:11:51,440 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:51,497 epoch 4 - iter 1/3 - loss 0.00066509 - samples/sec: 22.51 - lr: 0.020000\n",
      "2021-05-25 12:11:51,541 epoch 4 - iter 2/3 - loss 0.10140468 - samples/sec: 22.84 - lr: 0.020000\n",
      "2021-05-25 12:11:51,587 epoch 4 - iter 3/3 - loss 0.07631788 - samples/sec: 22.28 - lr: 0.020000\n",
      "2021-05-25 12:11:51,588 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:51,589 EPOCH 4 done: loss 0.0763 - lr 0.0200000\n",
      "2021-05-25 12:11:51,589 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:11:51,590 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:51,646 epoch 5 - iter 1/3 - loss 0.02632957 - samples/sec: 22.55 - lr: 0.020000\n",
      "2021-05-25 12:11:51,691 epoch 5 - iter 2/3 - loss 0.01826577 - samples/sec: 22.46 - lr: 0.020000\n",
      "2021-05-25 12:11:51,731 epoch 5 - iter 3/3 - loss 0.01266621 - samples/sec: 25.49 - lr: 0.020000\n",
      "2021-05-25 12:11:51,732 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:51,733 EPOCH 5 done: loss 0.0127 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:11:51,733 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:11:51,734 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:51,786 epoch 6 - iter 1/3 - loss 0.00496318 - samples/sec: 24.40 - lr: 0.010000\n",
      "2021-05-25 12:11:51,828 epoch 6 - iter 2/3 - loss 0.00711012 - samples/sec: 24.57 - lr: 0.010000\n",
      "2021-05-25 12:11:51,869 epoch 6 - iter 3/3 - loss 0.00524835 - samples/sec: 24.72 - lr: 0.010000\n",
      "2021-05-25 12:11:51,870 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:51,870 EPOCH 6 done: loss 0.0052 - lr 0.0100000\n",
      "2021-05-25 12:11:51,871 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:11:51,872 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:51,928 epoch 7 - iter 1/3 - loss 0.00094777 - samples/sec: 22.48 - lr: 0.010000\n",
      "2021-05-25 12:11:51,969 epoch 7 - iter 2/3 - loss 0.00406983 - samples/sec: 24.59 - lr: 0.010000\n",
      "2021-05-25 12:11:52,011 epoch 7 - iter 3/3 - loss 0.00662163 - samples/sec: 24.18 - lr: 0.010000\n",
      "2021-05-25 12:11:52,012 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:52,013 EPOCH 7 done: loss 0.0066 - lr 0.0100000\n",
      "2021-05-25 12:11:52,013 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:11:52,013 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:52,063 epoch 8 - iter 1/3 - loss 0.00066739 - samples/sec: 25.76 - lr: 0.010000\n",
      "2021-05-25 12:11:52,104 epoch 8 - iter 2/3 - loss 0.00100421 - samples/sec: 24.99 - lr: 0.010000\n",
      "2021-05-25 12:11:52,144 epoch 8 - iter 3/3 - loss 0.00360951 - samples/sec: 25.40 - lr: 0.010000\n",
      "2021-05-25 12:11:52,145 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:52,145 EPOCH 8 done: loss 0.0036 - lr 0.0100000\n",
      "2021-05-25 12:11:52,146 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:11:52,146 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:52,201 epoch 9 - iter 1/3 - loss 0.01099538 - samples/sec: 23.19 - lr: 0.010000\n",
      "2021-05-25 12:11:52,242 epoch 9 - iter 2/3 - loss 0.00717583 - samples/sec: 24.61 - lr: 0.010000\n",
      "2021-05-25 12:11:52,282 epoch 9 - iter 3/3 - loss 0.00515989 - samples/sec: 25.03 - lr: 0.010000\n",
      "2021-05-25 12:11:52,283 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:52,284 EPOCH 9 done: loss 0.0052 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:11:52,285 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:11:52,285 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:52,336 epoch 10 - iter 1/3 - loss 0.00265953 - samples/sec: 25.62 - lr: 0.005000\n",
      "2021-05-25 12:11:52,375 epoch 10 - iter 2/3 - loss 0.00289059 - samples/sec: 25.86 - lr: 0.005000\n",
      "2021-05-25 12:11:52,416 epoch 10 - iter 3/3 - loss 0.00458921 - samples/sec: 25.25 - lr: 0.005000\n",
      "2021-05-25 12:11:52,417 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:11:52,417 EPOCH 10 done: loss 0.0046 - lr 0.0050000\n",
      "2021-05-25 12:11:52,418 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:11:53,065 Test data not provided setting final score to 0\n",
      "2021-05-25 12:11:53,072 loading file /tmp/tweet_eval: sentiment/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d2e8e5b23b344332acec788e55cbabb6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.559\n",
      "2021-05-25 12:12:06,661 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:12:10,454 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 16469.78it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:12:10,456 [b'positive', b'negative', b'neutral']\n",
      "2021-05-25 12:12:10,456 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:10,458 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:12:10,458 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:10,459 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:12:10,459 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:10,459 Parameters:\n",
      "2021-05-25 12:12:10,459  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:12:10,460  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:12:10,460  - patience: \"3\"\n",
      "2021-05-25 12:12:10,460  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:12:10,460  - max_epochs: \"10\"\n",
      "2021-05-25 12:12:10,460  - shuffle: \"True\"\n",
      "2021-05-25 12:12:10,461  - train_with_dev: \"False\"\n",
      "2021-05-25 12:12:10,461  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:12:10,461 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:10,461 Model training base path: \"/tmp/tweet_eval: sentiment\"\n",
      "2021-05-25 12:12:10,462 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:10,462 Device: cuda:0\n",
      "2021-05-25 12:12:10,462 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:10,462 Embeddings storage mode: cpu\n",
      "2021-05-25 12:12:10,465 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:10,516 epoch 1 - iter 1/3 - loss 1.76851177 - samples/sec: 25.11 - lr: 0.020000\n",
      "2021-05-25 12:12:10,564 epoch 1 - iter 2/3 - loss 1.56889945 - samples/sec: 21.23 - lr: 0.020000\n",
      "2021-05-25 12:12:10,608 epoch 1 - iter 3/3 - loss 1.49334033 - samples/sec: 23.40 - lr: 0.020000\n",
      "2021-05-25 12:12:10,609 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:10,609 EPOCH 1 done: loss 1.4933 - lr 0.0200000\n",
      "2021-05-25 12:12:10,610 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:12:11,404 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:11,466 epoch 2 - iter 1/3 - loss 0.60256243 - samples/sec: 21.08 - lr: 0.020000\n",
      "2021-05-25 12:12:11,509 epoch 2 - iter 2/3 - loss 0.35580805 - samples/sec: 23.44 - lr: 0.020000\n",
      "2021-05-25 12:12:11,551 epoch 2 - iter 3/3 - loss 0.36727833 - samples/sec: 24.42 - lr: 0.020000\n",
      "2021-05-25 12:12:11,552 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:11,552 EPOCH 2 done: loss 0.3673 - lr 0.0200000\n",
      "2021-05-25 12:12:11,553 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:12:11,554 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:11,607 epoch 3 - iter 1/3 - loss 0.43775463 - samples/sec: 24.22 - lr: 0.020000\n",
      "2021-05-25 12:12:11,649 epoch 3 - iter 2/3 - loss 0.28453019 - samples/sec: 23.70 - lr: 0.020000\n",
      "2021-05-25 12:12:11,692 epoch 3 - iter 3/3 - loss 0.31358772 - samples/sec: 23.91 - lr: 0.020000\n",
      "2021-05-25 12:12:11,693 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:11,693 EPOCH 3 done: loss 0.3136 - lr 0.0200000\n",
      "2021-05-25 12:12:11,694 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:12:11,695 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:11,744 epoch 4 - iter 1/3 - loss 0.01349229 - samples/sec: 26.31 - lr: 0.020000\n",
      "2021-05-25 12:12:11,786 epoch 4 - iter 2/3 - loss 0.09163160 - samples/sec: 24.14 - lr: 0.020000\n",
      "2021-05-25 12:12:11,828 epoch 4 - iter 3/3 - loss 0.18165199 - samples/sec: 24.57 - lr: 0.020000\n",
      "2021-05-25 12:12:11,829 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:11,829 EPOCH 4 done: loss 0.1817 - lr 0.0200000\n",
      "2021-05-25 12:12:11,830 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:12:11,831 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:11,883 epoch 5 - iter 1/3 - loss 0.07510511 - samples/sec: 24.52 - lr: 0.020000\n",
      "2021-05-25 12:12:11,923 epoch 5 - iter 2/3 - loss 0.04669210 - samples/sec: 25.45 - lr: 0.020000\n",
      "2021-05-25 12:12:11,964 epoch 5 - iter 3/3 - loss 0.03183926 - samples/sec: 25.05 - lr: 0.020000\n",
      "2021-05-25 12:12:11,965 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:11,966 EPOCH 5 done: loss 0.0318 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:12:11,966 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:12:11,967 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:12,017 epoch 6 - iter 1/3 - loss 0.00493332 - samples/sec: 25.46 - lr: 0.010000\n",
      "2021-05-25 12:12:12,059 epoch 6 - iter 2/3 - loss 0.18272564 - samples/sec: 24.24 - lr: 0.010000\n",
      "2021-05-25 12:12:12,099 epoch 6 - iter 3/3 - loss 0.12218547 - samples/sec: 25.46 - lr: 0.010000\n",
      "2021-05-25 12:12:12,100 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:12,101 EPOCH 6 done: loss 0.1222 - lr 0.0100000\n",
      "2021-05-25 12:12:12,101 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:12:12,102 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:12,152 epoch 7 - iter 1/3 - loss 0.00441928 - samples/sec: 26.35 - lr: 0.010000\n",
      "2021-05-25 12:12:12,191 epoch 7 - iter 2/3 - loss 0.01625960 - samples/sec: 25.60 - lr: 0.010000\n",
      "2021-05-25 12:12:12,231 epoch 7 - iter 3/3 - loss 0.01149969 - samples/sec: 25.97 - lr: 0.010000\n",
      "2021-05-25 12:12:12,232 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:12,232 EPOCH 7 done: loss 0.0115 - lr 0.0100000\n",
      "2021-05-25 12:12:12,233 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:12:12,234 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:12,283 epoch 8 - iter 1/3 - loss 0.01566232 - samples/sec: 26.36 - lr: 0.010000\n",
      "2021-05-25 12:12:12,322 epoch 8 - iter 2/3 - loss 0.01240609 - samples/sec: 25.89 - lr: 0.010000\n",
      "2021-05-25 12:12:12,362 epoch 8 - iter 3/3 - loss 0.00865678 - samples/sec: 25.88 - lr: 0.010000\n",
      "2021-05-25 12:12:12,363 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:12,363 EPOCH 8 done: loss 0.0087 - lr 0.0100000\n",
      "2021-05-25 12:12:12,364 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:12:12,365 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:12,414 epoch 9 - iter 1/3 - loss 0.01888768 - samples/sec: 26.41 - lr: 0.010000\n",
      "2021-05-25 12:12:12,455 epoch 9 - iter 2/3 - loss 0.01034536 - samples/sec: 25.18 - lr: 0.010000\n",
      "2021-05-25 12:12:12,494 epoch 9 - iter 3/3 - loss 0.00752520 - samples/sec: 25.67 - lr: 0.010000\n",
      "2021-05-25 12:12:12,495 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:12,496 EPOCH 9 done: loss 0.0075 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:12:12,497 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:12:12,497 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:12,547 epoch 10 - iter 1/3 - loss 0.04088605 - samples/sec: 26.42 - lr: 0.005000\n",
      "2021-05-25 12:12:12,586 epoch 10 - iter 2/3 - loss 0.02090851 - samples/sec: 25.84 - lr: 0.005000\n",
      "2021-05-25 12:12:12,626 epoch 10 - iter 3/3 - loss 0.01444165 - samples/sec: 25.87 - lr: 0.005000\n",
      "2021-05-25 12:12:12,627 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:12,627 EPOCH 10 done: loss 0.0144 - lr 0.0050000\n",
      "2021-05-25 12:12:12,628 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:12:13,436 Test data not provided setting final score to 0\n",
      "2021-05-25 12:12:13,446 loading file /tmp/tweet_eval: sentiment/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "760136d065c8423ea5db0775320251b5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.536\n",
      "2021-05-25 12:12:27,373 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:12:31,210 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 8911.41it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:12:31,213 [b'neutral', b'positive', b'negative']\n",
      "2021-05-25 12:12:31,215 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:31,217 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:12:31,217 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:31,217 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:12:31,218 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:31,218 Parameters:\n",
      "2021-05-25 12:12:31,218  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:12:31,219  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:12:31,219  - patience: \"3\"\n",
      "2021-05-25 12:12:31,219  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:12:31,220  - max_epochs: \"10\"\n",
      "2021-05-25 12:12:31,220  - shuffle: \"True\"\n",
      "2021-05-25 12:12:31,220  - train_with_dev: \"False\"\n",
      "2021-05-25 12:12:31,221  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:12:31,221 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:31,221 Model training base path: \"/tmp/tweet_eval: sentiment\"\n",
      "2021-05-25 12:12:31,222 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:31,222 Device: cuda:0\n",
      "2021-05-25 12:12:31,222 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:31,223 Embeddings storage mode: cpu\n",
      "2021-05-25 12:12:31,227 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:31,284 epoch 1 - iter 1/3 - loss 0.21083419 - samples/sec: 23.31 - lr: 0.020000\n",
      "2021-05-25 12:12:31,335 epoch 1 - iter 2/3 - loss 0.16467892 - samples/sec: 19.78 - lr: 0.020000\n",
      "2021-05-25 12:12:31,382 epoch 1 - iter 3/3 - loss 0.19941952 - samples/sec: 21.61 - lr: 0.020000\n",
      "2021-05-25 12:12:31,383 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:31,384 EPOCH 1 done: loss 0.1994 - lr 0.0200000\n",
      "2021-05-25 12:12:31,385 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:12:32,066 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:32,128 epoch 2 - iter 1/3 - loss 0.01217946 - samples/sec: 20.95 - lr: 0.020000\n",
      "2021-05-25 12:12:32,171 epoch 2 - iter 2/3 - loss 0.16262701 - samples/sec: 23.39 - lr: 0.020000\n",
      "2021-05-25 12:12:32,215 epoch 2 - iter 3/3 - loss 0.11167678 - samples/sec: 23.07 - lr: 0.020000\n",
      "2021-05-25 12:12:32,216 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:32,217 EPOCH 2 done: loss 0.1117 - lr 0.0200000\n",
      "2021-05-25 12:12:32,218 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:12:32,220 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:32,276 epoch 3 - iter 1/3 - loss 0.01772460 - samples/sec: 22.20 - lr: 0.020000\n",
      "2021-05-25 12:12:32,317 epoch 3 - iter 2/3 - loss 0.00998707 - samples/sec: 24.79 - lr: 0.020000\n",
      "2021-05-25 12:12:32,358 epoch 3 - iter 3/3 - loss 0.02714849 - samples/sec: 24.86 - lr: 0.020000\n",
      "2021-05-25 12:12:32,359 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:32,360 EPOCH 3 done: loss 0.0271 - lr 0.0200000\n",
      "2021-05-25 12:12:32,360 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:12:32,362 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:32,418 epoch 4 - iter 1/3 - loss 0.00078276 - samples/sec: 22.17 - lr: 0.020000\n",
      "2021-05-25 12:12:32,460 epoch 4 - iter 2/3 - loss 0.00752629 - samples/sec: 24.52 - lr: 0.020000\n",
      "2021-05-25 12:12:32,500 epoch 4 - iter 3/3 - loss 0.01252910 - samples/sec: 25.11 - lr: 0.020000\n",
      "2021-05-25 12:12:32,501 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:32,502 EPOCH 4 done: loss 0.0125 - lr 0.0200000\n",
      "2021-05-25 12:12:32,503 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:12:32,504 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:32,557 epoch 5 - iter 1/3 - loss 0.01960991 - samples/sec: 23.52 - lr: 0.020000\n",
      "2021-05-25 12:12:32,600 epoch 5 - iter 2/3 - loss 0.01034022 - samples/sec: 24.06 - lr: 0.020000\n",
      "2021-05-25 12:12:32,644 epoch 5 - iter 3/3 - loss 0.00815531 - samples/sec: 22.84 - lr: 0.020000\n",
      "2021-05-25 12:12:32,646 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:32,647 EPOCH 5 done: loss 0.0082 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:12:32,647 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:12:32,648 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:32,703 epoch 6 - iter 1/3 - loss 0.00464235 - samples/sec: 23.05 - lr: 0.010000\n",
      "2021-05-25 12:12:32,748 epoch 6 - iter 2/3 - loss 0.04812716 - samples/sec: 22.52 - lr: 0.010000\n",
      "2021-05-25 12:12:32,791 epoch 6 - iter 3/3 - loss 0.03879268 - samples/sec: 23.71 - lr: 0.010000\n",
      "2021-05-25 12:12:32,793 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:32,793 EPOCH 6 done: loss 0.0388 - lr 0.0100000\n",
      "2021-05-25 12:12:32,794 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:12:32,795 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:32,849 epoch 7 - iter 1/3 - loss 0.00050020 - samples/sec: 23.72 - lr: 0.010000\n",
      "2021-05-25 12:12:32,891 epoch 7 - iter 2/3 - loss 0.00120849 - samples/sec: 24.31 - lr: 0.010000\n",
      "2021-05-25 12:12:32,932 epoch 7 - iter 3/3 - loss 0.00856927 - samples/sec: 24.76 - lr: 0.010000\n",
      "2021-05-25 12:12:32,933 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:32,934 EPOCH 7 done: loss 0.0086 - lr 0.0100000\n",
      "2021-05-25 12:12:32,935 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:12:32,936 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:32,987 epoch 8 - iter 1/3 - loss 0.00418174 - samples/sec: 25.42 - lr: 0.010000\n",
      "2021-05-25 12:12:33,027 epoch 8 - iter 2/3 - loss 0.00981170 - samples/sec: 25.26 - lr: 0.010000\n",
      "2021-05-25 12:12:33,067 epoch 8 - iter 3/3 - loss 0.00707412 - samples/sec: 25.09 - lr: 0.010000\n",
      "2021-05-25 12:12:33,069 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:33,069 EPOCH 8 done: loss 0.0071 - lr 0.0100000\n",
      "2021-05-25 12:12:33,070 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:12:33,071 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:33,123 epoch 9 - iter 1/3 - loss 0.00202044 - samples/sec: 24.59 - lr: 0.010000\n",
      "2021-05-25 12:12:33,164 epoch 9 - iter 2/3 - loss 0.00769199 - samples/sec: 24.92 - lr: 0.010000\n",
      "2021-05-25 12:12:33,206 epoch 9 - iter 3/3 - loss 0.00629694 - samples/sec: 24.35 - lr: 0.010000\n",
      "2021-05-25 12:12:33,207 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:33,208 EPOCH 9 done: loss 0.0063 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:12:33,209 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:12:33,210 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:33,262 epoch 10 - iter 1/3 - loss 0.00310997 - samples/sec: 24.33 - lr: 0.005000\n",
      "2021-05-25 12:12:33,303 epoch 10 - iter 2/3 - loss 0.00223761 - samples/sec: 24.75 - lr: 0.005000\n",
      "2021-05-25 12:12:33,344 epoch 10 - iter 3/3 - loss 0.00487353 - samples/sec: 24.94 - lr: 0.005000\n",
      "2021-05-25 12:12:33,345 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:33,346 EPOCH 10 done: loss 0.0049 - lr 0.0050000\n",
      "2021-05-25 12:12:33,347 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:12:34,020 Test data not provided setting final score to 0\n",
      "2021-05-25 12:12:34,027 loading file /tmp/tweet_eval: sentiment/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "04b637cac79f4f589a3077e42b5111a1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.566\n",
      "2021-05-25 12:12:47,672 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:12:51,466 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 14250.18it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:12:51,468 [b'negative', b'neutral', b'positive']\n",
      "2021-05-25 12:12:51,469 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:51,471 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:12:51,471 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:51,471 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:12:51,472 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:51,472 Parameters:\n",
      "2021-05-25 12:12:51,472  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:12:51,472  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:12:51,473  - patience: \"3\"\n",
      "2021-05-25 12:12:51,473  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:12:51,473  - max_epochs: \"10\"\n",
      "2021-05-25 12:12:51,473  - shuffle: \"True\"\n",
      "2021-05-25 12:12:51,473  - train_with_dev: \"False\"\n",
      "2021-05-25 12:12:51,474  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:12:51,474 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:51,474 Model training base path: \"/tmp/tweet_eval: sentiment\"\n",
      "2021-05-25 12:12:51,474 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:51,475 Device: cuda:0\n",
      "2021-05-25 12:12:51,475 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:51,475 Embeddings storage mode: cpu\n",
      "2021-05-25 12:12:51,478 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:51,529 epoch 1 - iter 1/3 - loss 1.22729826 - samples/sec: 25.10 - lr: 0.020000\n",
      "2021-05-25 12:12:51,572 epoch 1 - iter 2/3 - loss 1.00587967 - samples/sec: 23.52 - lr: 0.020000\n",
      "2021-05-25 12:12:51,616 epoch 1 - iter 3/3 - loss 1.31387168 - samples/sec: 22.92 - lr: 0.020000\n",
      "2021-05-25 12:12:51,617 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:51,618 EPOCH 1 done: loss 1.3139 - lr 0.0200000\n",
      "2021-05-25 12:12:51,619 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:12:52,319 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:52,379 epoch 2 - iter 1/3 - loss 0.63191372 - samples/sec: 21.82 - lr: 0.020000\n",
      "2021-05-25 12:12:52,424 epoch 2 - iter 2/3 - loss 0.90958640 - samples/sec: 22.47 - lr: 0.020000\n",
      "2021-05-25 12:12:52,469 epoch 2 - iter 3/3 - loss 0.79482782 - samples/sec: 22.70 - lr: 0.020000\n",
      "2021-05-25 12:12:52,470 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:52,471 EPOCH 2 done: loss 0.7948 - lr 0.0200000\n",
      "2021-05-25 12:12:52,471 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:12:52,472 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:52,528 epoch 3 - iter 1/3 - loss 1.06006277 - samples/sec: 22.35 - lr: 0.020000\n",
      "2021-05-25 12:12:52,569 epoch 3 - iter 2/3 - loss 0.71456975 - samples/sec: 24.47 - lr: 0.020000\n",
      "2021-05-25 12:12:52,614 epoch 3 - iter 3/3 - loss 0.54762906 - samples/sec: 22.84 - lr: 0.020000\n",
      "2021-05-25 12:12:52,615 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:52,615 EPOCH 3 done: loss 0.5476 - lr 0.0200000\n",
      "2021-05-25 12:12:52,616 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:12:52,617 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:52,672 epoch 4 - iter 1/3 - loss 0.09333497 - samples/sec: 22.82 - lr: 0.020000\n",
      "2021-05-25 12:12:52,721 epoch 4 - iter 2/3 - loss 0.10154697 - samples/sec: 20.83 - lr: 0.020000\n",
      "2021-05-25 12:12:52,771 epoch 4 - iter 3/3 - loss 0.12854407 - samples/sec: 20.26 - lr: 0.020000\n",
      "2021-05-25 12:12:52,772 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:52,773 EPOCH 4 done: loss 0.1285 - lr 0.0200000\n",
      "2021-05-25 12:12:52,774 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:12:52,774 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:52,839 epoch 5 - iter 1/3 - loss 0.32931378 - samples/sec: 19.34 - lr: 0.020000\n",
      "2021-05-25 12:12:52,885 epoch 5 - iter 2/3 - loss 0.17009996 - samples/sec: 22.32 - lr: 0.020000\n",
      "2021-05-25 12:12:52,927 epoch 5 - iter 3/3 - loss 0.12871865 - samples/sec: 24.13 - lr: 0.020000\n",
      "2021-05-25 12:12:52,928 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:52,928 EPOCH 5 done: loss 0.1287 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:12:52,929 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:12:52,929 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:52,982 epoch 6 - iter 1/3 - loss 0.00855241 - samples/sec: 24.71 - lr: 0.010000\n",
      "2021-05-25 12:12:53,029 epoch 6 - iter 2/3 - loss 0.00637405 - samples/sec: 21.65 - lr: 0.010000\n",
      "2021-05-25 12:12:53,075 epoch 6 - iter 3/3 - loss 0.01481664 - samples/sec: 22.14 - lr: 0.010000\n",
      "2021-05-25 12:12:53,076 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:53,076 EPOCH 6 done: loss 0.0148 - lr 0.0100000\n",
      "2021-05-25 12:12:53,077 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:12:53,077 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:53,129 epoch 7 - iter 1/3 - loss 0.00860815 - samples/sec: 25.27 - lr: 0.010000\n",
      "2021-05-25 12:12:53,169 epoch 7 - iter 2/3 - loss 0.01414531 - samples/sec: 25.05 - lr: 0.010000\n",
      "2021-05-25 12:12:53,218 epoch 7 - iter 3/3 - loss 0.01141326 - samples/sec: 20.54 - lr: 0.010000\n",
      "2021-05-25 12:12:53,220 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:53,220 EPOCH 7 done: loss 0.0114 - lr 0.0100000\n",
      "2021-05-25 12:12:53,221 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:12:53,221 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:53,280 epoch 8 - iter 1/3 - loss 0.01084151 - samples/sec: 21.93 - lr: 0.010000\n",
      "2021-05-25 12:12:53,330 epoch 8 - iter 2/3 - loss 0.01168904 - samples/sec: 20.30 - lr: 0.010000\n",
      "2021-05-25 12:12:53,375 epoch 8 - iter 3/3 - loss 0.00845704 - samples/sec: 22.67 - lr: 0.010000\n",
      "2021-05-25 12:12:53,376 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:53,377 EPOCH 8 done: loss 0.0085 - lr 0.0100000\n",
      "2021-05-25 12:12:53,377 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:12:53,378 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:53,435 epoch 9 - iter 1/3 - loss 0.01968222 - samples/sec: 22.41 - lr: 0.010000\n",
      "2021-05-25 12:12:53,478 epoch 9 - iter 2/3 - loss 0.01126368 - samples/sec: 23.60 - lr: 0.010000\n",
      "2021-05-25 12:12:53,520 epoch 9 - iter 3/3 - loss 0.00817752 - samples/sec: 24.47 - lr: 0.010000\n",
      "2021-05-25 12:12:53,521 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:53,522 EPOCH 9 done: loss 0.0082 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:12:53,522 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:12:53,523 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:53,574 epoch 10 - iter 1/3 - loss 0.00663567 - samples/sec: 25.21 - lr: 0.005000\n",
      "2021-05-25 12:12:53,615 epoch 10 - iter 2/3 - loss 0.00739985 - samples/sec: 24.94 - lr: 0.005000\n",
      "2021-05-25 12:12:53,656 epoch 10 - iter 3/3 - loss 0.00800309 - samples/sec: 24.65 - lr: 0.005000\n",
      "2021-05-25 12:12:53,657 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:12:53,658 EPOCH 10 done: loss 0.0080 - lr 0.0050000\n",
      "2021-05-25 12:12:53,658 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:12:54,376 Test data not provided setting final score to 0\n",
      "2021-05-25 12:12:54,383 loading file /tmp/tweet_eval: sentiment/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c8f97223aa2848d18229786460644b36",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.498\n",
      "2021-05-25 12:13:07,874 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:13:11,602 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 12052.60it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:13:11,605 [b'positive', b'neutral', b'negative']\n",
      "2021-05-25 12:13:11,606 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:11,607 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:13:11,608 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:11,608 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:13:11,608 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:11,609 Parameters:\n",
      "2021-05-25 12:13:11,609  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:13:11,609  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:13:11,609  - patience: \"3\"\n",
      "2021-05-25 12:13:11,610  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:13:11,610  - max_epochs: \"10\"\n",
      "2021-05-25 12:13:11,610  - shuffle: \"True\"\n",
      "2021-05-25 12:13:11,610  - train_with_dev: \"False\"\n",
      "2021-05-25 12:13:11,611  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:13:11,611 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:11,611 Model training base path: \"/tmp/tweet_eval: sentiment\"\n",
      "2021-05-25 12:13:11,611 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:11,611 Device: cuda:0\n",
      "2021-05-25 12:13:11,612 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:11,612 Embeddings storage mode: cpu\n",
      "2021-05-25 12:13:11,615 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:11,667 epoch 1 - iter 1/3 - loss 0.95876074 - samples/sec: 24.88 - lr: 0.020000\n",
      "2021-05-25 12:13:11,713 epoch 1 - iter 2/3 - loss 0.55546974 - samples/sec: 21.69 - lr: 0.020000\n",
      "2021-05-25 12:13:11,756 epoch 1 - iter 3/3 - loss 0.94780600 - samples/sec: 23.63 - lr: 0.020000\n",
      "2021-05-25 12:13:11,758 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:11,758 EPOCH 1 done: loss 0.9478 - lr 0.0200000\n",
      "2021-05-25 12:13:11,759 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:13:12,493 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:12,553 epoch 2 - iter 1/3 - loss 0.15030257 - samples/sec: 21.54 - lr: 0.020000\n",
      "2021-05-25 12:13:12,596 epoch 2 - iter 2/3 - loss 0.08704242 - samples/sec: 23.21 - lr: 0.020000\n",
      "2021-05-25 12:13:12,640 epoch 2 - iter 3/3 - loss 0.07470692 - samples/sec: 23.27 - lr: 0.020000\n",
      "2021-05-25 12:13:12,641 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:12,642 EPOCH 2 done: loss 0.0747 - lr 0.0200000\n",
      "2021-05-25 12:13:12,642 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:13:12,643 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:12,702 epoch 3 - iter 1/3 - loss 0.14640899 - samples/sec: 20.85 - lr: 0.020000\n",
      "2021-05-25 12:13:12,749 epoch 3 - iter 2/3 - loss 0.13288752 - samples/sec: 21.83 - lr: 0.020000\n",
      "2021-05-25 12:13:12,791 epoch 3 - iter 3/3 - loss 0.08977919 - samples/sec: 24.13 - lr: 0.020000\n",
      "2021-05-25 12:13:12,792 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:12,793 EPOCH 3 done: loss 0.0898 - lr 0.0200000\n",
      "2021-05-25 12:13:12,793 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:13:12,794 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:12,849 epoch 4 - iter 1/3 - loss 0.00530702 - samples/sec: 22.61 - lr: 0.020000\n",
      "2021-05-25 12:13:12,891 epoch 4 - iter 2/3 - loss 0.00818352 - samples/sec: 24.42 - lr: 0.020000\n",
      "2021-05-25 12:13:12,932 epoch 4 - iter 3/3 - loss 0.00564110 - samples/sec: 24.91 - lr: 0.020000\n",
      "2021-05-25 12:13:12,933 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:12,933 EPOCH 4 done: loss 0.0056 - lr 0.0200000\n",
      "2021-05-25 12:13:12,934 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:13:12,935 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:12,990 epoch 5 - iter 1/3 - loss 0.00609893 - samples/sec: 22.45 - lr: 0.020000\n",
      "2021-05-25 12:13:13,031 epoch 5 - iter 2/3 - loss 0.00454144 - samples/sec: 24.78 - lr: 0.020000\n",
      "2021-05-25 12:13:13,072 epoch 5 - iter 3/3 - loss 0.00365200 - samples/sec: 25.06 - lr: 0.020000\n",
      "2021-05-25 12:13:13,073 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:13,073 EPOCH 5 done: loss 0.0037 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:13:13,074 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:13:13,075 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:13,132 epoch 6 - iter 1/3 - loss 0.00613458 - samples/sec: 21.83 - lr: 0.010000\n",
      "2021-05-25 12:13:13,173 epoch 6 - iter 2/3 - loss 0.00683412 - samples/sec: 24.78 - lr: 0.010000\n",
      "2021-05-25 12:13:13,214 epoch 6 - iter 3/3 - loss 0.00514579 - samples/sec: 24.61 - lr: 0.010000\n",
      "2021-05-25 12:13:13,215 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:13,216 EPOCH 6 done: loss 0.0051 - lr 0.0100000\n",
      "2021-05-25 12:13:13,216 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:13:13,217 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:13,272 epoch 7 - iter 1/3 - loss 0.00920404 - samples/sec: 22.70 - lr: 0.010000\n",
      "2021-05-25 12:13:13,314 epoch 7 - iter 2/3 - loss 0.00791096 - samples/sec: 24.26 - lr: 0.010000\n",
      "2021-05-25 12:13:13,354 epoch 7 - iter 3/3 - loss 0.00759809 - samples/sec: 25.17 - lr: 0.010000\n",
      "2021-05-25 12:13:13,355 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:13,356 EPOCH 7 done: loss 0.0076 - lr 0.0100000\n",
      "2021-05-25 12:13:13,356 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:13:13,357 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:13,413 epoch 8 - iter 1/3 - loss 0.00250326 - samples/sec: 22.28 - lr: 0.010000\n",
      "2021-05-25 12:13:13,458 epoch 8 - iter 2/3 - loss 0.00300162 - samples/sec: 22.66 - lr: 0.010000\n",
      "2021-05-25 12:13:13,497 epoch 8 - iter 3/3 - loss 0.00304196 - samples/sec: 25.95 - lr: 0.010000\n",
      "2021-05-25 12:13:13,498 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:13,499 EPOCH 8 done: loss 0.0030 - lr 0.0100000\n",
      "2021-05-25 12:13:13,499 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:13:13,500 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:13,555 epoch 9 - iter 1/3 - loss 0.00081164 - samples/sec: 22.33 - lr: 0.010000\n",
      "2021-05-25 12:13:13,596 epoch 9 - iter 2/3 - loss 0.00189261 - samples/sec: 24.75 - lr: 0.010000\n",
      "2021-05-25 12:13:13,637 epoch 9 - iter 3/3 - loss 0.00442286 - samples/sec: 24.77 - lr: 0.010000\n",
      "2021-05-25 12:13:13,638 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:13,639 EPOCH 9 done: loss 0.0044 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:13:13,639 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:13:13,640 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:13,696 epoch 10 - iter 1/3 - loss 0.00216347 - samples/sec: 22.43 - lr: 0.005000\n",
      "2021-05-25 12:13:13,745 epoch 10 - iter 2/3 - loss 0.00274056 - samples/sec: 20.71 - lr: 0.005000\n",
      "2021-05-25 12:13:13,791 epoch 10 - iter 3/3 - loss 0.00340585 - samples/sec: 21.89 - lr: 0.005000\n",
      "2021-05-25 12:13:13,793 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:13,794 EPOCH 10 done: loss 0.0034 - lr 0.0050000\n",
      "2021-05-25 12:13:13,794 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:13:14,517 Test data not provided setting final score to 0\n",
      "2021-05-25 12:13:14,525 loading file /tmp/tweet_eval: sentiment/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e4964515230e4d5795cac722100b6496",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.483\n",
      "2021-05-25 12:13:28,073 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:13:31,835 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 11715.93it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:13:31,838 [b'negative', b'neutral', b'positive']\n",
      "2021-05-25 12:13:31,839 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:31,841 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:13:31,842 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:31,842 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:13:31,843 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:31,843 Parameters:\n",
      "2021-05-25 12:13:31,844  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:13:31,845  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:13:31,845  - patience: \"3\"\n",
      "2021-05-25 12:13:31,846  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:13:31,846  - max_epochs: \"10\"\n",
      "2021-05-25 12:13:31,846  - shuffle: \"True\"\n",
      "2021-05-25 12:13:31,846  - train_with_dev: \"False\"\n",
      "2021-05-25 12:13:31,847  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:13:31,847 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:31,847 Model training base path: \"/tmp/tweet_eval: sentiment\"\n",
      "2021-05-25 12:13:31,847 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:31,848 Device: cuda:0\n",
      "2021-05-25 12:13:31,848 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:31,848 Embeddings storage mode: cpu\n",
      "2021-05-25 12:13:31,852 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:31,902 epoch 1 - iter 1/3 - loss 1.28062928 - samples/sec: 25.32 - lr: 0.020000\n",
      "2021-05-25 12:13:31,945 epoch 1 - iter 2/3 - loss 1.32770026 - samples/sec: 23.37 - lr: 0.020000\n",
      "2021-05-25 12:13:31,992 epoch 1 - iter 3/3 - loss 1.22681328 - samples/sec: 21.74 - lr: 0.020000\n",
      "2021-05-25 12:13:31,993 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:32,006 EPOCH 1 done: loss 1.2268 - lr 0.0200000\n",
      "2021-05-25 12:13:32,007 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:13:32,780 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:32,843 epoch 2 - iter 1/3 - loss 0.02350384 - samples/sec: 20.83 - lr: 0.020000\n",
      "2021-05-25 12:13:32,888 epoch 2 - iter 2/3 - loss 0.39860916 - samples/sec: 22.35 - lr: 0.020000\n",
      "2021-05-25 12:13:32,933 epoch 2 - iter 3/3 - loss 0.43792073 - samples/sec: 22.66 - lr: 0.020000\n",
      "2021-05-25 12:13:32,934 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:32,934 EPOCH 2 done: loss 0.4379 - lr 0.0200000\n",
      "2021-05-25 12:13:32,934 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:13:32,935 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:32,991 epoch 3 - iter 1/3 - loss 0.10933778 - samples/sec: 21.95 - lr: 0.020000\n",
      "2021-05-25 12:13:33,034 epoch 3 - iter 2/3 - loss 0.09499148 - samples/sec: 23.65 - lr: 0.020000\n",
      "2021-05-25 12:13:33,078 epoch 3 - iter 3/3 - loss 0.06518649 - samples/sec: 22.86 - lr: 0.020000\n",
      "2021-05-25 12:13:33,079 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:33,079 EPOCH 3 done: loss 0.0652 - lr 0.0200000\n",
      "2021-05-25 12:13:33,080 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:13:33,080 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:33,133 epoch 4 - iter 1/3 - loss 0.00123733 - samples/sec: 23.83 - lr: 0.020000\n",
      "2021-05-25 12:13:33,172 epoch 4 - iter 2/3 - loss 0.00171964 - samples/sec: 25.70 - lr: 0.020000\n",
      "2021-05-25 12:13:33,214 epoch 4 - iter 3/3 - loss 0.00850603 - samples/sec: 23.89 - lr: 0.020000\n",
      "2021-05-25 12:13:33,216 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:33,216 EPOCH 4 done: loss 0.0085 - lr 0.0200000\n",
      "2021-05-25 12:13:33,217 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:13:33,217 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:33,271 epoch 5 - iter 1/3 - loss 0.01896674 - samples/sec: 24.00 - lr: 0.020000\n",
      "2021-05-25 12:13:33,312 epoch 5 - iter 2/3 - loss 0.01017350 - samples/sec: 24.87 - lr: 0.020000\n",
      "2021-05-25 12:13:33,355 epoch 5 - iter 3/3 - loss 0.05522493 - samples/sec: 23.52 - lr: 0.020000\n",
      "2021-05-25 12:13:33,357 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:33,357 EPOCH 5 done: loss 0.0552 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:13:33,358 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:13:33,359 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:33,414 epoch 6 - iter 1/3 - loss 0.05949636 - samples/sec: 23.13 - lr: 0.010000\n",
      "2021-05-25 12:13:33,458 epoch 6 - iter 2/3 - loss 0.03034261 - samples/sec: 23.04 - lr: 0.010000\n",
      "2021-05-25 12:13:33,504 epoch 6 - iter 3/3 - loss 0.04933730 - samples/sec: 22.08 - lr: 0.010000\n",
      "2021-05-25 12:13:33,505 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:33,505 EPOCH 6 done: loss 0.0493 - lr 0.0100000\n",
      "2021-05-25 12:13:33,506 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:13:33,507 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:33,562 epoch 7 - iter 1/3 - loss 0.01436932 - samples/sec: 22.75 - lr: 0.010000\n",
      "2021-05-25 12:13:33,603 epoch 7 - iter 2/3 - loss 0.00760414 - samples/sec: 24.79 - lr: 0.010000\n",
      "2021-05-25 12:13:33,644 epoch 7 - iter 3/3 - loss 0.00534108 - samples/sec: 24.65 - lr: 0.010000\n",
      "2021-05-25 12:13:33,645 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:33,646 EPOCH 7 done: loss 0.0053 - lr 0.0100000\n",
      "2021-05-25 12:13:33,646 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:13:33,647 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:33,700 epoch 8 - iter 1/3 - loss 0.00259549 - samples/sec: 23.95 - lr: 0.010000\n",
      "2021-05-25 12:13:33,742 epoch 8 - iter 2/3 - loss 0.00379831 - samples/sec: 24.25 - lr: 0.010000\n",
      "2021-05-25 12:13:33,783 epoch 8 - iter 3/3 - loss 0.00299876 - samples/sec: 24.70 - lr: 0.010000\n",
      "2021-05-25 12:13:33,785 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:33,785 EPOCH 8 done: loss 0.0030 - lr 0.0100000\n",
      "2021-05-25 12:13:33,786 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:13:33,786 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:33,838 epoch 9 - iter 1/3 - loss 0.01058276 - samples/sec: 24.90 - lr: 0.010000\n",
      "2021-05-25 12:13:33,880 epoch 9 - iter 2/3 - loss 0.00572113 - samples/sec: 24.23 - lr: 0.010000\n",
      "2021-05-25 12:13:33,925 epoch 9 - iter 3/3 - loss 0.00409427 - samples/sec: 22.99 - lr: 0.010000\n",
      "2021-05-25 12:13:33,926 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:33,926 EPOCH 9 done: loss 0.0041 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:13:33,927 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:13:33,928 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:33,979 epoch 10 - iter 1/3 - loss 0.00412738 - samples/sec: 25.12 - lr: 0.005000\n",
      "2021-05-25 12:13:34,022 epoch 10 - iter 2/3 - loss 0.00247102 - samples/sec: 23.79 - lr: 0.005000\n",
      "2021-05-25 12:13:34,064 epoch 10 - iter 3/3 - loss 0.00189826 - samples/sec: 24.16 - lr: 0.005000\n",
      "2021-05-25 12:13:34,065 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:34,066 EPOCH 10 done: loss 0.0019 - lr 0.0050000\n",
      "2021-05-25 12:13:34,066 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:13:34,803 Test data not provided setting final score to 0\n",
      "2021-05-25 12:13:34,812 loading file /tmp/tweet_eval: sentiment/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "867b859d1320478b8cd6f832658575c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.512\n",
      "2021-05-25 12:13:48,532 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:13:52,349 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 10951.19it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:13:52,351 [b'positive', b'negative', b'neutral']\n",
      "2021-05-25 12:13:52,352 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:52,354 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:13:52,354 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:52,355 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:13:52,355 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:52,355 Parameters:\n",
      "2021-05-25 12:13:52,355  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:13:52,356  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:13:52,356  - patience: \"3\"\n",
      "2021-05-25 12:13:52,356  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:13:52,356  - max_epochs: \"10\"\n",
      "2021-05-25 12:13:52,356  - shuffle: \"True\"\n",
      "2021-05-25 12:13:52,357  - train_with_dev: \"False\"\n",
      "2021-05-25 12:13:52,357  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:13:52,357 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:52,357 Model training base path: \"/tmp/tweet_eval: sentiment\"\n",
      "2021-05-25 12:13:52,358 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:52,358 Device: cuda:0\n",
      "2021-05-25 12:13:52,358 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:52,358 Embeddings storage mode: cpu\n",
      "2021-05-25 12:13:52,362 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:52,411 epoch 1 - iter 1/3 - loss 2.05034852 - samples/sec: 26.13 - lr: 0.020000\n",
      "2021-05-25 12:13:52,456 epoch 1 - iter 2/3 - loss 1.61631113 - samples/sec: 22.47 - lr: 0.020000\n",
      "2021-05-25 12:13:52,502 epoch 1 - iter 3/3 - loss 1.62299124 - samples/sec: 22.10 - lr: 0.020000\n",
      "2021-05-25 12:13:52,503 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:52,503 EPOCH 1 done: loss 1.6230 - lr 0.0200000\n",
      "2021-05-25 12:13:52,503 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:13:53,304 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:53,361 epoch 2 - iter 1/3 - loss 0.09006252 - samples/sec: 22.54 - lr: 0.020000\n",
      "2021-05-25 12:13:53,406 epoch 2 - iter 2/3 - loss 0.14907194 - samples/sec: 22.52 - lr: 0.020000\n",
      "2021-05-25 12:13:53,450 epoch 2 - iter 3/3 - loss 0.71551200 - samples/sec: 23.17 - lr: 0.020000\n",
      "2021-05-25 12:13:53,451 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:53,451 EPOCH 2 done: loss 0.7155 - lr 0.0200000\n",
      "2021-05-25 12:13:53,451 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:13:53,452 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:53,505 epoch 3 - iter 1/3 - loss 0.09656962 - samples/sec: 23.31 - lr: 0.020000\n",
      "2021-05-25 12:13:53,550 epoch 3 - iter 2/3 - loss 0.25679008 - samples/sec: 22.70 - lr: 0.020000\n",
      "2021-05-25 12:13:53,590 epoch 3 - iter 3/3 - loss 0.18233582 - samples/sec: 25.18 - lr: 0.020000\n",
      "2021-05-25 12:13:53,591 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:53,591 EPOCH 3 done: loss 0.1823 - lr 0.0200000\n",
      "2021-05-25 12:13:53,591 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:13:53,592 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:53,641 epoch 4 - iter 1/3 - loss 0.05560404 - samples/sec: 25.69 - lr: 0.020000\n",
      "2021-05-25 12:13:53,680 epoch 4 - iter 2/3 - loss 0.04122017 - samples/sec: 25.75 - lr: 0.020000\n",
      "2021-05-25 12:13:53,722 epoch 4 - iter 3/3 - loss 0.05332797 - samples/sec: 24.12 - lr: 0.020000\n",
      "2021-05-25 12:13:53,723 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:53,723 EPOCH 4 done: loss 0.0533 - lr 0.0200000\n",
      "2021-05-25 12:13:53,724 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:13:53,724 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:53,779 epoch 5 - iter 1/3 - loss 0.01386975 - samples/sec: 22.31 - lr: 0.020000\n",
      "2021-05-25 12:13:53,821 epoch 5 - iter 2/3 - loss 0.01260065 - samples/sec: 24.59 - lr: 0.020000\n",
      "2021-05-25 12:13:53,860 epoch 5 - iter 3/3 - loss 0.01163678 - samples/sec: 25.62 - lr: 0.020000\n",
      "2021-05-25 12:13:53,861 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:53,861 EPOCH 5 done: loss 0.0116 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:13:53,862 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:13:53,862 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:53,912 epoch 6 - iter 1/3 - loss 0.00688780 - samples/sec: 25.04 - lr: 0.010000\n",
      "2021-05-25 12:13:53,951 epoch 6 - iter 2/3 - loss 0.00496109 - samples/sec: 25.88 - lr: 0.010000\n",
      "2021-05-25 12:13:53,991 epoch 6 - iter 3/3 - loss 0.00697508 - samples/sec: 25.35 - lr: 0.010000\n",
      "2021-05-25 12:13:53,992 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:53,992 EPOCH 6 done: loss 0.0070 - lr 0.0100000\n",
      "2021-05-25 12:13:53,993 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:13:53,993 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:54,045 epoch 7 - iter 1/3 - loss 0.00337578 - samples/sec: 24.23 - lr: 0.010000\n",
      "2021-05-25 12:13:54,084 epoch 7 - iter 2/3 - loss 0.00469474 - samples/sec: 25.64 - lr: 0.010000\n",
      "2021-05-25 12:13:54,123 epoch 7 - iter 3/3 - loss 0.00847276 - samples/sec: 25.87 - lr: 0.010000\n",
      "2021-05-25 12:13:54,124 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:54,125 EPOCH 7 done: loss 0.0085 - lr 0.0100000\n",
      "2021-05-25 12:13:54,125 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:13:54,126 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:54,180 epoch 8 - iter 1/3 - loss 0.00645942 - samples/sec: 23.31 - lr: 0.010000\n",
      "2021-05-25 12:13:54,222 epoch 8 - iter 2/3 - loss 0.00424099 - samples/sec: 24.44 - lr: 0.010000\n",
      "2021-05-25 12:13:54,263 epoch 8 - iter 3/3 - loss 0.00477387 - samples/sec: 24.78 - lr: 0.010000\n",
      "2021-05-25 12:13:54,264 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:54,265 EPOCH 8 done: loss 0.0048 - lr 0.0100000\n",
      "2021-05-25 12:13:54,265 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:13:54,266 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:54,317 epoch 9 - iter 1/3 - loss 0.00589262 - samples/sec: 25.48 - lr: 0.010000\n",
      "2021-05-25 12:13:54,356 epoch 9 - iter 2/3 - loss 0.00501115 - samples/sec: 25.84 - lr: 0.010000\n",
      "2021-05-25 12:13:54,396 epoch 9 - iter 3/3 - loss 0.00582763 - samples/sec: 25.36 - lr: 0.010000\n",
      "2021-05-25 12:13:54,397 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:54,398 EPOCH 9 done: loss 0.0058 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:13:54,399 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:13:54,399 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:54,454 epoch 10 - iter 1/3 - loss 0.00674336 - samples/sec: 23.01 - lr: 0.005000\n",
      "2021-05-25 12:13:54,495 epoch 10 - iter 2/3 - loss 0.00529077 - samples/sec: 24.53 - lr: 0.005000\n",
      "2021-05-25 12:13:54,535 epoch 10 - iter 3/3 - loss 0.00438426 - samples/sec: 25.18 - lr: 0.005000\n",
      "2021-05-25 12:13:54,536 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:13:54,537 EPOCH 10 done: loss 0.0044 - lr 0.0050000\n",
      "2021-05-25 12:13:54,537 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:13:55,317 Test data not provided setting final score to 0\n",
      "2021-05-25 12:13:55,327 loading file /tmp/tweet_eval: sentiment/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fd389ee1705445528a4802755a44e52f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.477\n",
      "2021-05-25 12:14:09,283 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:14:13,204 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 7467.60it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:14:13,208 [b'positive', b'neutral', b'negative']\n",
      "2021-05-25 12:14:13,210 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:13,212 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:14:13,212 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:13,213 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:14:13,213 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:13,213 Parameters:\n",
      "2021-05-25 12:14:13,213  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:14:13,214  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:14:13,214  - patience: \"3\"\n",
      "2021-05-25 12:14:13,214  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:14:13,214  - max_epochs: \"10\"\n",
      "2021-05-25 12:14:13,215  - shuffle: \"True\"\n",
      "2021-05-25 12:14:13,215  - train_with_dev: \"False\"\n",
      "2021-05-25 12:14:13,215  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:14:13,215 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:13,216 Model training base path: \"/tmp/financial_phrasebank\"\n",
      "2021-05-25 12:14:13,216 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:13,216 Device: cuda:0\n",
      "2021-05-25 12:14:13,216 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:13,217 Embeddings storage mode: cpu\n",
      "2021-05-25 12:14:13,220 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:13,270 epoch 1 - iter 1/3 - loss 0.24268800 - samples/sec: 25.33 - lr: 0.020000\n",
      "2021-05-25 12:14:13,317 epoch 1 - iter 2/3 - loss 0.68550548 - samples/sec: 21.79 - lr: 0.020000\n",
      "2021-05-25 12:14:13,366 epoch 1 - iter 3/3 - loss 0.68111523 - samples/sec: 20.41 - lr: 0.020000\n",
      "2021-05-25 12:14:13,368 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:13,368 EPOCH 1 done: loss 0.6811 - lr 0.0200000\n",
      "2021-05-25 12:14:13,369 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:14:14,026 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:14,086 epoch 2 - iter 1/3 - loss 0.05499863 - samples/sec: 21.64 - lr: 0.020000\n",
      "2021-05-25 12:14:14,131 epoch 2 - iter 2/3 - loss 0.06659599 - samples/sec: 22.54 - lr: 0.020000\n",
      "2021-05-25 12:14:14,172 epoch 2 - iter 3/3 - loss 0.05526551 - samples/sec: 25.25 - lr: 0.020000\n",
      "2021-05-25 12:14:14,173 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:14,174 EPOCH 2 done: loss 0.0553 - lr 0.0200000\n",
      "2021-05-25 12:14:14,174 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:14:14,175 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:14,228 epoch 3 - iter 1/3 - loss 0.04710569 - samples/sec: 23.82 - lr: 0.020000\n",
      "2021-05-25 12:14:14,269 epoch 3 - iter 2/3 - loss 0.02881581 - samples/sec: 24.78 - lr: 0.020000\n",
      "2021-05-25 12:14:14,314 epoch 3 - iter 3/3 - loss 0.02781606 - samples/sec: 23.05 - lr: 0.020000\n",
      "2021-05-25 12:14:14,315 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:14,315 EPOCH 3 done: loss 0.0278 - lr 0.0200000\n",
      "2021-05-25 12:14:14,316 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:14:14,317 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:14,368 epoch 4 - iter 1/3 - loss 0.01222528 - samples/sec: 24.87 - lr: 0.020000\n",
      "2021-05-25 12:14:14,411 epoch 4 - iter 2/3 - loss 0.00706270 - samples/sec: 23.64 - lr: 0.020000\n",
      "2021-05-25 12:14:14,453 epoch 4 - iter 3/3 - loss 0.01623520 - samples/sec: 24.26 - lr: 0.020000\n",
      "2021-05-25 12:14:14,454 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:14,454 EPOCH 4 done: loss 0.0162 - lr 0.0200000\n",
      "2021-05-25 12:14:14,455 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:14:14,455 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:14,507 epoch 5 - iter 1/3 - loss 0.00368239 - samples/sec: 23.91 - lr: 0.020000\n",
      "2021-05-25 12:14:14,546 epoch 5 - iter 2/3 - loss 0.01064588 - samples/sec: 26.10 - lr: 0.020000\n",
      "2021-05-25 12:14:14,587 epoch 5 - iter 3/3 - loss 0.00965975 - samples/sec: 25.06 - lr: 0.020000\n",
      "2021-05-25 12:14:14,588 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:14,589 EPOCH 5 done: loss 0.0097 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:14:14,590 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:14:14,591 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:14,640 epoch 6 - iter 1/3 - loss 0.01254164 - samples/sec: 25.57 - lr: 0.010000\n",
      "2021-05-25 12:14:14,683 epoch 6 - iter 2/3 - loss 0.00829709 - samples/sec: 23.84 - lr: 0.010000\n",
      "2021-05-25 12:14:14,724 epoch 6 - iter 3/3 - loss 0.00868009 - samples/sec: 24.84 - lr: 0.010000\n",
      "2021-05-25 12:14:14,725 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:14,725 EPOCH 6 done: loss 0.0087 - lr 0.0100000\n",
      "2021-05-25 12:14:14,726 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:14:14,727 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:14,779 epoch 7 - iter 1/3 - loss 0.00154669 - samples/sec: 24.85 - lr: 0.010000\n",
      "2021-05-25 12:14:14,819 epoch 7 - iter 2/3 - loss 0.00423310 - samples/sec: 25.12 - lr: 0.010000\n",
      "2021-05-25 12:14:14,861 epoch 7 - iter 3/3 - loss 0.00389326 - samples/sec: 24.61 - lr: 0.010000\n",
      "2021-05-25 12:14:14,862 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:14,862 EPOCH 7 done: loss 0.0039 - lr 0.0100000\n",
      "2021-05-25 12:14:14,863 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:14:14,864 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:14,915 epoch 8 - iter 1/3 - loss 0.00234796 - samples/sec: 24.97 - lr: 0.010000\n",
      "2021-05-25 12:14:14,957 epoch 8 - iter 2/3 - loss 0.00374417 - samples/sec: 24.50 - lr: 0.010000\n",
      "2021-05-25 12:14:14,998 epoch 8 - iter 3/3 - loss 0.00547761 - samples/sec: 24.62 - lr: 0.010000\n",
      "2021-05-25 12:14:14,999 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:15,000 EPOCH 8 done: loss 0.0055 - lr 0.0100000\n",
      "2021-05-25 12:14:15,000 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:14:15,001 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:15,052 epoch 9 - iter 1/3 - loss 0.00178844 - samples/sec: 25.03 - lr: 0.010000\n",
      "2021-05-25 12:14:15,097 epoch 9 - iter 2/3 - loss 0.00196005 - samples/sec: 23.00 - lr: 0.010000\n",
      "2021-05-25 12:14:15,139 epoch 9 - iter 3/3 - loss 0.00505367 - samples/sec: 24.07 - lr: 0.010000\n",
      "2021-05-25 12:14:15,140 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:15,141 EPOCH 9 done: loss 0.0051 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:14:15,141 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:14:15,142 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:15,193 epoch 10 - iter 1/3 - loss 0.00235813 - samples/sec: 25.14 - lr: 0.005000\n",
      "2021-05-25 12:14:15,236 epoch 10 - iter 2/3 - loss 0.00250490 - samples/sec: 23.88 - lr: 0.005000\n",
      "2021-05-25 12:14:15,277 epoch 10 - iter 3/3 - loss 0.01044397 - samples/sec: 24.79 - lr: 0.005000\n",
      "2021-05-25 12:14:15,278 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:15,279 EPOCH 10 done: loss 0.0104 - lr 0.0050000\n",
      "2021-05-25 12:14:15,279 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:14:15,910 Test data not provided setting final score to 0\n",
      "2021-05-25 12:14:15,917 loading file /tmp/financial_phrasebank/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "67f99b9686f840e4b42e1f9506939ca0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.484\n",
      "2021-05-25 12:14:29,460 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:14:33,184 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 10147.51it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:14:33,187 [b'positive', b'neutral', b'negative']\n",
      "2021-05-25 12:14:33,188 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:33,190 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:14:33,191 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:33,191 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:14:33,192 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:33,193 Parameters:\n",
      "2021-05-25 12:14:33,193  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:14:33,193  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:14:33,193  - patience: \"3\"\n",
      "2021-05-25 12:14:33,194  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:14:33,194  - max_epochs: \"10\"\n",
      "2021-05-25 12:14:33,194  - shuffle: \"True\"\n",
      "2021-05-25 12:14:33,194  - train_with_dev: \"False\"\n",
      "2021-05-25 12:14:33,195  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:14:33,195 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:33,195 Model training base path: \"/tmp/financial_phrasebank\"\n",
      "2021-05-25 12:14:33,195 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:33,195 Device: cuda:0\n",
      "2021-05-25 12:14:33,196 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:33,196 Embeddings storage mode: cpu\n",
      "2021-05-25 12:14:33,199 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:33,253 epoch 1 - iter 1/3 - loss 0.64369500 - samples/sec: 23.60 - lr: 0.020000\n",
      "2021-05-25 12:14:33,300 epoch 1 - iter 2/3 - loss 0.79545847 - samples/sec: 21.61 - lr: 0.020000\n",
      "2021-05-25 12:14:33,350 epoch 1 - iter 3/3 - loss 0.95340767 - samples/sec: 20.19 - lr: 0.020000\n",
      "2021-05-25 12:14:33,351 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:33,352 EPOCH 1 done: loss 0.9534 - lr 0.0200000\n",
      "2021-05-25 12:14:33,352 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:14:34,028 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:34,092 epoch 2 - iter 1/3 - loss 0.16530804 - samples/sec: 19.83 - lr: 0.020000\n",
      "2021-05-25 12:14:34,137 epoch 2 - iter 2/3 - loss 0.37757813 - samples/sec: 22.60 - lr: 0.020000\n",
      "2021-05-25 12:14:34,181 epoch 2 - iter 3/3 - loss 0.75083724 - samples/sec: 23.15 - lr: 0.020000\n",
      "2021-05-25 12:14:34,183 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:34,183 EPOCH 2 done: loss 0.7508 - lr 0.0200000\n",
      "2021-05-25 12:14:34,184 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:14:34,185 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:34,246 epoch 3 - iter 1/3 - loss 0.05767964 - samples/sec: 20.14 - lr: 0.020000\n",
      "2021-05-25 12:14:34,289 epoch 3 - iter 2/3 - loss 0.04969387 - samples/sec: 23.81 - lr: 0.020000\n",
      "2021-05-25 12:14:34,332 epoch 3 - iter 3/3 - loss 0.19642801 - samples/sec: 23.84 - lr: 0.020000\n",
      "2021-05-25 12:14:34,333 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:34,333 EPOCH 3 done: loss 0.1964 - lr 0.0200000\n",
      "2021-05-25 12:14:34,334 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:14:34,335 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:34,394 epoch 4 - iter 1/3 - loss 0.09363491 - samples/sec: 21.02 - lr: 0.020000\n",
      "2021-05-25 12:14:34,439 epoch 4 - iter 2/3 - loss 0.06909623 - samples/sec: 22.58 - lr: 0.020000\n",
      "2021-05-25 12:14:34,482 epoch 4 - iter 3/3 - loss 0.09508085 - samples/sec: 23.64 - lr: 0.020000\n",
      "2021-05-25 12:14:34,483 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:34,484 EPOCH 4 done: loss 0.0951 - lr 0.0200000\n",
      "2021-05-25 12:14:34,484 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:14:34,485 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:34,546 epoch 5 - iter 1/3 - loss 0.06426954 - samples/sec: 20.52 - lr: 0.020000\n",
      "2021-05-25 12:14:34,588 epoch 5 - iter 2/3 - loss 0.04150187 - samples/sec: 24.52 - lr: 0.020000\n",
      "2021-05-25 12:14:34,628 epoch 5 - iter 3/3 - loss 0.03841863 - samples/sec: 25.20 - lr: 0.020000\n",
      "2021-05-25 12:14:34,629 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:34,630 EPOCH 5 done: loss 0.0384 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:14:34,630 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:14:34,631 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:34,687 epoch 6 - iter 1/3 - loss 0.00316196 - samples/sec: 22.30 - lr: 0.010000\n",
      "2021-05-25 12:14:34,729 epoch 6 - iter 2/3 - loss 0.00622125 - samples/sec: 24.51 - lr: 0.010000\n",
      "2021-05-25 12:14:34,772 epoch 6 - iter 3/3 - loss 0.02069327 - samples/sec: 23.77 - lr: 0.010000\n",
      "2021-05-25 12:14:34,773 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:34,773 EPOCH 6 done: loss 0.0207 - lr 0.0100000\n",
      "2021-05-25 12:14:34,774 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:14:34,775 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:34,831 epoch 7 - iter 1/3 - loss 0.00328941 - samples/sec: 22.20 - lr: 0.010000\n",
      "2021-05-25 12:14:34,872 epoch 7 - iter 2/3 - loss 0.00405965 - samples/sec: 24.40 - lr: 0.010000\n",
      "2021-05-25 12:14:34,913 epoch 7 - iter 3/3 - loss 0.00631292 - samples/sec: 25.21 - lr: 0.010000\n",
      "2021-05-25 12:14:34,914 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:34,915 EPOCH 7 done: loss 0.0063 - lr 0.0100000\n",
      "2021-05-25 12:14:34,915 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:14:34,916 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:34,973 epoch 8 - iter 1/3 - loss 0.00255473 - samples/sec: 21.52 - lr: 0.010000\n",
      "2021-05-25 12:14:35,020 epoch 8 - iter 2/3 - loss 0.00205134 - samples/sec: 21.96 - lr: 0.010000\n",
      "2021-05-25 12:14:35,062 epoch 8 - iter 3/3 - loss 0.00213865 - samples/sec: 23.87 - lr: 0.010000\n",
      "2021-05-25 12:14:35,063 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:35,064 EPOCH 8 done: loss 0.0021 - lr 0.0100000\n",
      "2021-05-25 12:14:35,064 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:14:35,065 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:35,119 epoch 9 - iter 1/3 - loss 0.00357556 - samples/sec: 23.54 - lr: 0.010000\n",
      "2021-05-25 12:14:35,165 epoch 9 - iter 2/3 - loss 0.00223788 - samples/sec: 22.16 - lr: 0.010000\n",
      "2021-05-25 12:14:35,204 epoch 9 - iter 3/3 - loss 0.00192908 - samples/sec: 26.08 - lr: 0.010000\n",
      "2021-05-25 12:14:35,205 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:35,206 EPOCH 9 done: loss 0.0019 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:14:35,206 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:14:35,207 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:35,262 epoch 10 - iter 1/3 - loss 0.00461158 - samples/sec: 22.75 - lr: 0.005000\n",
      "2021-05-25 12:14:35,308 epoch 10 - iter 2/3 - loss 0.00317469 - samples/sec: 22.00 - lr: 0.005000\n",
      "2021-05-25 12:14:35,351 epoch 10 - iter 3/3 - loss 0.00318605 - samples/sec: 23.90 - lr: 0.005000\n",
      "2021-05-25 12:14:35,352 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:35,353 EPOCH 10 done: loss 0.0032 - lr 0.0050000\n",
      "2021-05-25 12:14:35,353 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:14:36,012 Test data not provided setting final score to 0\n",
      "2021-05-25 12:14:36,020 loading file /tmp/financial_phrasebank/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "863d50d9354b4a08aa01df9d8bd8b493",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.393\n",
      "2021-05-25 12:14:49,599 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:14:53,342 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 16822.07it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:14:53,345 [b'neutral', b'positive', b'negative']\n",
      "2021-05-25 12:14:53,346 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:53,347 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:14:53,348 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:53,348 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:14:53,348 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:53,349 Parameters:\n",
      "2021-05-25 12:14:53,349  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:14:53,349  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:14:53,350  - patience: \"3\"\n",
      "2021-05-25 12:14:53,350  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:14:53,350  - max_epochs: \"10\"\n",
      "2021-05-25 12:14:53,350  - shuffle: \"True\"\n",
      "2021-05-25 12:14:53,351  - train_with_dev: \"False\"\n",
      "2021-05-25 12:14:53,351  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:14:53,351 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:53,351 Model training base path: \"/tmp/financial_phrasebank\"\n",
      "2021-05-25 12:14:53,352 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:53,352 Device: cuda:0\n",
      "2021-05-25 12:14:53,352 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:53,353 Embeddings storage mode: cpu\n",
      "2021-05-25 12:14:53,356 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:53,405 epoch 1 - iter 1/3 - loss 0.54374725 - samples/sec: 26.69 - lr: 0.020000\n",
      "2021-05-25 12:14:53,452 epoch 1 - iter 2/3 - loss 1.47993329 - samples/sec: 21.25 - lr: 0.020000\n",
      "2021-05-25 12:14:53,496 epoch 1 - iter 3/3 - loss 1.52187667 - samples/sec: 23.42 - lr: 0.020000\n",
      "2021-05-25 12:14:53,496 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:53,497 EPOCH 1 done: loss 1.5219 - lr 0.0200000\n",
      "2021-05-25 12:14:53,497 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:14:54,169 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:54,230 epoch 2 - iter 1/3 - loss 0.64381558 - samples/sec: 21.05 - lr: 0.020000\n",
      "2021-05-25 12:14:54,276 epoch 2 - iter 2/3 - loss 0.82270262 - samples/sec: 21.67 - lr: 0.020000\n",
      "2021-05-25 12:14:54,319 epoch 2 - iter 3/3 - loss 0.95644826 - samples/sec: 23.61 - lr: 0.020000\n",
      "2021-05-25 12:14:54,320 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:54,320 EPOCH 2 done: loss 0.9564 - lr 0.0200000\n",
      "2021-05-25 12:14:54,321 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:14:54,321 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:54,375 epoch 3 - iter 1/3 - loss 0.02133310 - samples/sec: 23.40 - lr: 0.020000\n",
      "2021-05-25 12:14:54,418 epoch 3 - iter 2/3 - loss 0.15643220 - samples/sec: 23.33 - lr: 0.020000\n",
      "2021-05-25 12:14:54,461 epoch 3 - iter 3/3 - loss 0.14917379 - samples/sec: 23.27 - lr: 0.020000\n",
      "2021-05-25 12:14:54,462 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:54,463 EPOCH 3 done: loss 0.1492 - lr 0.0200000\n",
      "2021-05-25 12:14:54,463 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:14:54,463 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:54,514 epoch 4 - iter 1/3 - loss 0.03389860 - samples/sec: 24.80 - lr: 0.020000\n",
      "2021-05-25 12:14:54,555 epoch 4 - iter 2/3 - loss 0.03087335 - samples/sec: 24.68 - lr: 0.020000\n",
      "2021-05-25 12:14:54,599 epoch 4 - iter 3/3 - loss 0.02337128 - samples/sec: 23.16 - lr: 0.020000\n",
      "2021-05-25 12:14:54,600 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:54,600 EPOCH 4 done: loss 0.0234 - lr 0.0200000\n",
      "2021-05-25 12:14:54,600 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:14:54,601 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:54,651 epoch 5 - iter 1/3 - loss 0.01982139 - samples/sec: 24.91 - lr: 0.020000\n",
      "2021-05-25 12:14:54,693 epoch 5 - iter 2/3 - loss 0.09361595 - samples/sec: 24.08 - lr: 0.020000\n",
      "2021-05-25 12:14:54,733 epoch 5 - iter 3/3 - loss 0.06430432 - samples/sec: 25.77 - lr: 0.020000\n",
      "2021-05-25 12:14:54,734 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:54,734 EPOCH 5 done: loss 0.0643 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:14:54,735 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:14:54,735 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:54,790 epoch 6 - iter 1/3 - loss 0.00945649 - samples/sec: 23.32 - lr: 0.010000\n",
      "2021-05-25 12:14:54,831 epoch 6 - iter 2/3 - loss 0.00578156 - samples/sec: 24.67 - lr: 0.010000\n",
      "2021-05-25 12:14:54,873 epoch 6 - iter 3/3 - loss 0.00700548 - samples/sec: 24.10 - lr: 0.010000\n",
      "2021-05-25 12:14:54,874 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:54,875 EPOCH 6 done: loss 0.0070 - lr 0.0100000\n",
      "2021-05-25 12:14:54,875 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:14:54,876 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:54,934 epoch 7 - iter 1/3 - loss 0.01980152 - samples/sec: 21.10 - lr: 0.010000\n",
      "2021-05-25 12:14:54,975 epoch 7 - iter 2/3 - loss 0.01454996 - samples/sec: 25.01 - lr: 0.010000\n",
      "2021-05-25 12:14:55,016 epoch 7 - iter 3/3 - loss 0.01254015 - samples/sec: 24.71 - lr: 0.010000\n",
      "2021-05-25 12:14:55,017 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:55,018 EPOCH 7 done: loss 0.0125 - lr 0.0100000\n",
      "2021-05-25 12:14:55,018 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:14:55,019 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:55,074 epoch 8 - iter 1/3 - loss 0.00537940 - samples/sec: 23.00 - lr: 0.010000\n",
      "2021-05-25 12:14:55,115 epoch 8 - iter 2/3 - loss 0.00727704 - samples/sec: 24.83 - lr: 0.010000\n",
      "2021-05-25 12:14:55,155 epoch 8 - iter 3/3 - loss 0.00632454 - samples/sec: 25.12 - lr: 0.010000\n",
      "2021-05-25 12:14:55,156 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:55,157 EPOCH 8 done: loss 0.0063 - lr 0.0100000\n",
      "2021-05-25 12:14:55,157 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:14:55,158 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:55,212 epoch 9 - iter 1/3 - loss 0.00597507 - samples/sec: 23.22 - lr: 0.010000\n",
      "2021-05-25 12:14:55,254 epoch 9 - iter 2/3 - loss 0.00378632 - samples/sec: 24.11 - lr: 0.010000\n",
      "2021-05-25 12:14:55,297 epoch 9 - iter 3/3 - loss 0.00513258 - samples/sec: 23.89 - lr: 0.010000\n",
      "2021-05-25 12:14:55,298 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:55,299 EPOCH 9 done: loss 0.0051 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:14:55,299 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:14:55,300 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:55,356 epoch 10 - iter 1/3 - loss 0.00710914 - samples/sec: 22.51 - lr: 0.005000\n",
      "2021-05-25 12:14:55,397 epoch 10 - iter 2/3 - loss 0.00473534 - samples/sec: 24.63 - lr: 0.005000\n",
      "2021-05-25 12:14:55,438 epoch 10 - iter 3/3 - loss 0.00500569 - samples/sec: 24.58 - lr: 0.005000\n",
      "2021-05-25 12:14:55,439 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:14:55,440 EPOCH 10 done: loss 0.0050 - lr 0.0050000\n",
      "2021-05-25 12:14:55,440 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:14:56,104 Test data not provided setting final score to 0\n",
      "2021-05-25 12:14:56,112 loading file /tmp/financial_phrasebank/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8605f810ef3c4f6985594c31546de730",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.476\n",
      "2021-05-25 12:15:09,630 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:15:13,327 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 13259.13it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:15:13,330 [b'neutral', b'positive', b'negative']\n",
      "2021-05-25 12:15:13,331 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:13,333 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:15:13,333 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:13,334 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:15:13,334 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:13,334 Parameters:\n",
      "2021-05-25 12:15:13,335  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:15:13,335  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:15:13,335  - patience: \"3\"\n",
      "2021-05-25 12:15:13,335  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:15:13,336  - max_epochs: \"10\"\n",
      "2021-05-25 12:15:13,336  - shuffle: \"True\"\n",
      "2021-05-25 12:15:13,336  - train_with_dev: \"False\"\n",
      "2021-05-25 12:15:13,336  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:15:13,337 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:13,337 Model training base path: \"/tmp/financial_phrasebank\"\n",
      "2021-05-25 12:15:13,337 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:13,338 Device: cuda:0\n",
      "2021-05-25 12:15:13,338 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:13,338 Embeddings storage mode: cpu\n",
      "2021-05-25 12:15:13,342 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:13,391 epoch 1 - iter 1/3 - loss 0.68143779 - samples/sec: 26.33 - lr: 0.020000\n",
      "2021-05-25 12:15:13,437 epoch 1 - iter 2/3 - loss 1.19122323 - samples/sec: 22.00 - lr: 0.020000\n",
      "2021-05-25 12:15:13,482 epoch 1 - iter 3/3 - loss 1.29859565 - samples/sec: 22.60 - lr: 0.020000\n",
      "2021-05-25 12:15:13,483 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:13,483 EPOCH 1 done: loss 1.2986 - lr 0.0200000\n",
      "2021-05-25 12:15:13,484 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:15:14,139 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:14,198 epoch 2 - iter 1/3 - loss 0.77962440 - samples/sec: 22.43 - lr: 0.020000\n",
      "2021-05-25 12:15:14,245 epoch 2 - iter 2/3 - loss 0.83153510 - samples/sec: 21.55 - lr: 0.020000\n",
      "2021-05-25 12:15:14,288 epoch 2 - iter 3/3 - loss 0.83282683 - samples/sec: 23.79 - lr: 0.020000\n",
      "2021-05-25 12:15:14,289 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:14,289 EPOCH 2 done: loss 0.8328 - lr 0.0200000\n",
      "2021-05-25 12:15:14,290 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:15:14,291 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:14,346 epoch 3 - iter 1/3 - loss 0.29645780 - samples/sec: 22.90 - lr: 0.020000\n",
      "2021-05-25 12:15:14,393 epoch 3 - iter 2/3 - loss 0.26796313 - samples/sec: 21.53 - lr: 0.020000\n",
      "2021-05-25 12:15:14,436 epoch 3 - iter 3/3 - loss 0.55655131 - samples/sec: 23.52 - lr: 0.020000\n",
      "2021-05-25 12:15:14,437 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:14,438 EPOCH 3 done: loss 0.5566 - lr 0.0200000\n",
      "2021-05-25 12:15:14,439 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:15:14,439 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:14,498 epoch 4 - iter 1/3 - loss 0.15992029 - samples/sec: 21.77 - lr: 0.020000\n",
      "2021-05-25 12:15:14,543 epoch 4 - iter 2/3 - loss 0.19739735 - samples/sec: 22.67 - lr: 0.020000\n",
      "2021-05-25 12:15:14,587 epoch 4 - iter 3/3 - loss 0.22344581 - samples/sec: 22.84 - lr: 0.020000\n",
      "2021-05-25 12:15:14,589 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:14,589 EPOCH 4 done: loss 0.2234 - lr 0.0200000\n",
      "2021-05-25 12:15:14,590 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:15:14,592 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:14,645 epoch 5 - iter 1/3 - loss 0.05712646 - samples/sec: 23.73 - lr: 0.020000\n",
      "2021-05-25 12:15:14,688 epoch 5 - iter 2/3 - loss 0.12580956 - samples/sec: 23.69 - lr: 0.020000\n",
      "2021-05-25 12:15:14,728 epoch 5 - iter 3/3 - loss 0.09644712 - samples/sec: 25.40 - lr: 0.020000\n",
      "2021-05-25 12:15:14,729 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:14,730 EPOCH 5 done: loss 0.0964 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:15:14,730 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:15:14,732 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:14,786 epoch 6 - iter 1/3 - loss 0.04316843 - samples/sec: 23.21 - lr: 0.010000\n",
      "2021-05-25 12:15:14,826 epoch 6 - iter 2/3 - loss 0.02294230 - samples/sec: 25.25 - lr: 0.010000\n",
      "2021-05-25 12:15:14,869 epoch 6 - iter 3/3 - loss 0.03340929 - samples/sec: 23.50 - lr: 0.010000\n",
      "2021-05-25 12:15:14,870 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:14,871 EPOCH 6 done: loss 0.0334 - lr 0.0100000\n",
      "2021-05-25 12:15:14,871 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:15:14,872 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:14,927 epoch 7 - iter 1/3 - loss 0.08975155 - samples/sec: 22.90 - lr: 0.010000\n",
      "2021-05-25 12:15:14,968 epoch 7 - iter 2/3 - loss 0.05423468 - samples/sec: 24.85 - lr: 0.010000\n",
      "2021-05-25 12:15:15,009 epoch 7 - iter 3/3 - loss 0.04123289 - samples/sec: 24.84 - lr: 0.010000\n",
      "2021-05-25 12:15:15,010 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:15,011 EPOCH 7 done: loss 0.0412 - lr 0.0100000\n",
      "2021-05-25 12:15:15,012 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:15:15,013 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:15,064 epoch 8 - iter 1/3 - loss 0.00448428 - samples/sec: 25.03 - lr: 0.010000\n",
      "2021-05-25 12:15:15,105 epoch 8 - iter 2/3 - loss 0.00906486 - samples/sec: 24.55 - lr: 0.010000\n",
      "2021-05-25 12:15:15,147 epoch 8 - iter 3/3 - loss 0.01637859 - samples/sec: 24.35 - lr: 0.010000\n",
      "2021-05-25 12:15:15,148 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:15,148 EPOCH 8 done: loss 0.0164 - lr 0.0100000\n",
      "2021-05-25 12:15:15,149 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:15:15,150 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:15,201 epoch 9 - iter 1/3 - loss 0.00565309 - samples/sec: 25.28 - lr: 0.010000\n",
      "2021-05-25 12:15:15,240 epoch 9 - iter 2/3 - loss 0.00846385 - samples/sec: 25.61 - lr: 0.010000\n",
      "2021-05-25 12:15:15,282 epoch 9 - iter 3/3 - loss 0.00999819 - samples/sec: 24.59 - lr: 0.010000\n",
      "2021-05-25 12:15:15,283 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:15,283 EPOCH 9 done: loss 0.0100 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:15:15,285 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:15:15,286 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:15,340 epoch 10 - iter 1/3 - loss 0.00869280 - samples/sec: 22.98 - lr: 0.005000\n",
      "2021-05-25 12:15:15,382 epoch 10 - iter 2/3 - loss 0.01123182 - samples/sec: 24.45 - lr: 0.005000\n",
      "2021-05-25 12:15:15,422 epoch 10 - iter 3/3 - loss 0.00823033 - samples/sec: 24.73 - lr: 0.005000\n",
      "2021-05-25 12:15:15,423 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:15,424 EPOCH 10 done: loss 0.0082 - lr 0.0050000\n",
      "2021-05-25 12:15:15,424 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:15:16,092 Test data not provided setting final score to 0\n",
      "2021-05-25 12:15:16,100 loading file /tmp/financial_phrasebank/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2ddfbd55880d4958957186597cace8d4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.476\n",
      "2021-05-25 12:15:29,756 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:15:33,482 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 12520.31it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:15:33,484 [b'positive', b'neutral', b'negative']\n",
      "2021-05-25 12:15:33,485 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:33,487 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:15:33,487 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:33,488 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:15:33,488 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:33,488 Parameters:\n",
      "2021-05-25 12:15:33,489  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:15:33,489  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:15:33,489  - patience: \"3\"\n",
      "2021-05-25 12:15:33,489  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:15:33,490  - max_epochs: \"10\"\n",
      "2021-05-25 12:15:33,490  - shuffle: \"True\"\n",
      "2021-05-25 12:15:33,490  - train_with_dev: \"False\"\n",
      "2021-05-25 12:15:33,490  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:15:33,491 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:33,491 Model training base path: \"/tmp/financial_phrasebank\"\n",
      "2021-05-25 12:15:33,491 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:33,491 Device: cuda:0\n",
      "2021-05-25 12:15:33,491 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:33,492 Embeddings storage mode: cpu\n",
      "2021-05-25 12:15:33,495 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:33,546 epoch 1 - iter 1/3 - loss 0.35632864 - samples/sec: 25.15 - lr: 0.020000\n",
      "2021-05-25 12:15:33,591 epoch 1 - iter 2/3 - loss 0.83979906 - samples/sec: 22.19 - lr: 0.020000\n",
      "2021-05-25 12:15:33,637 epoch 1 - iter 3/3 - loss 0.77075480 - samples/sec: 22.25 - lr: 0.020000\n",
      "2021-05-25 12:15:33,638 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:33,638 EPOCH 1 done: loss 0.7708 - lr 0.0200000\n",
      "2021-05-25 12:15:33,638 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:15:34,311 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:34,371 epoch 2 - iter 1/3 - loss 0.01027861 - samples/sec: 21.37 - lr: 0.020000\n",
      "2021-05-25 12:15:34,417 epoch 2 - iter 2/3 - loss 0.34968774 - samples/sec: 21.89 - lr: 0.020000\n",
      "2021-05-25 12:15:34,462 epoch 2 - iter 3/3 - loss 0.29164301 - samples/sec: 22.57 - lr: 0.020000\n",
      "2021-05-25 12:15:34,463 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:34,463 EPOCH 2 done: loss 0.2916 - lr 0.0200000\n",
      "2021-05-25 12:15:34,464 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:15:34,464 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:34,519 epoch 3 - iter 1/3 - loss 0.15262568 - samples/sec: 22.22 - lr: 0.020000\n",
      "2021-05-25 12:15:34,561 epoch 3 - iter 2/3 - loss 0.08597644 - samples/sec: 24.20 - lr: 0.020000\n",
      "2021-05-25 12:15:34,602 epoch 3 - iter 3/3 - loss 0.06013140 - samples/sec: 24.63 - lr: 0.020000\n",
      "2021-05-25 12:15:34,603 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:34,603 EPOCH 3 done: loss 0.0601 - lr 0.0200000\n",
      "2021-05-25 12:15:34,604 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:15:34,604 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:34,654 epoch 4 - iter 1/3 - loss 0.01385172 - samples/sec: 25.34 - lr: 0.020000\n",
      "2021-05-25 12:15:34,697 epoch 4 - iter 2/3 - loss 0.13858191 - samples/sec: 23.40 - lr: 0.020000\n",
      "2021-05-25 12:15:34,741 epoch 4 - iter 3/3 - loss 0.09610971 - samples/sec: 23.21 - lr: 0.020000\n",
      "2021-05-25 12:15:34,742 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:34,742 EPOCH 4 done: loss 0.0961 - lr 0.0200000\n",
      "2021-05-25 12:15:34,742 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:15:34,743 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:34,796 epoch 5 - iter 1/3 - loss 0.09802064 - samples/sec: 23.22 - lr: 0.020000\n",
      "2021-05-25 12:15:34,836 epoch 5 - iter 2/3 - loss 0.05345176 - samples/sec: 25.74 - lr: 0.020000\n",
      "2021-05-25 12:15:34,875 epoch 5 - iter 3/3 - loss 0.04028938 - samples/sec: 25.36 - lr: 0.020000\n",
      "2021-05-25 12:15:34,876 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:34,877 EPOCH 5 done: loss 0.0403 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:15:34,877 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:15:34,877 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:34,928 epoch 6 - iter 1/3 - loss 0.00936926 - samples/sec: 24.84 - lr: 0.010000\n",
      "2021-05-25 12:15:34,968 epoch 6 - iter 2/3 - loss 0.00627219 - samples/sec: 25.52 - lr: 0.010000\n",
      "2021-05-25 12:15:35,007 epoch 6 - iter 3/3 - loss 0.01142161 - samples/sec: 25.68 - lr: 0.010000\n",
      "2021-05-25 12:15:35,008 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:35,008 EPOCH 6 done: loss 0.0114 - lr 0.0100000\n",
      "2021-05-25 12:15:35,009 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:15:35,009 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:35,062 epoch 7 - iter 1/3 - loss 0.01528707 - samples/sec: 23.68 - lr: 0.010000\n",
      "2021-05-25 12:15:35,105 epoch 7 - iter 2/3 - loss 0.00905441 - samples/sec: 23.42 - lr: 0.010000\n",
      "2021-05-25 12:15:35,146 epoch 7 - iter 3/3 - loss 0.00657691 - samples/sec: 25.00 - lr: 0.010000\n",
      "2021-05-25 12:15:35,147 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:35,147 EPOCH 7 done: loss 0.0066 - lr 0.0100000\n",
      "2021-05-25 12:15:35,147 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:15:35,148 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:35,198 epoch 8 - iter 1/3 - loss 0.00194198 - samples/sec: 25.01 - lr: 0.010000\n",
      "2021-05-25 12:15:35,239 epoch 8 - iter 2/3 - loss 0.00605878 - samples/sec: 24.84 - lr: 0.010000\n",
      "2021-05-25 12:15:35,280 epoch 8 - iter 3/3 - loss 0.00488139 - samples/sec: 24.60 - lr: 0.010000\n",
      "2021-05-25 12:15:35,281 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:35,281 EPOCH 8 done: loss 0.0049 - lr 0.0100000\n",
      "2021-05-25 12:15:35,281 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:15:35,282 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:35,339 epoch 9 - iter 1/3 - loss 0.01549992 - samples/sec: 21.34 - lr: 0.010000\n",
      "2021-05-25 12:15:35,380 epoch 9 - iter 2/3 - loss 0.00846810 - samples/sec: 24.56 - lr: 0.010000\n",
      "2021-05-25 12:15:35,421 epoch 9 - iter 3/3 - loss 0.00620999 - samples/sec: 24.79 - lr: 0.010000\n",
      "2021-05-25 12:15:35,422 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:35,422 EPOCH 9 done: loss 0.0062 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:15:35,423 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:15:35,423 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:35,476 epoch 10 - iter 1/3 - loss 0.00136010 - samples/sec: 23.50 - lr: 0.005000\n",
      "2021-05-25 12:15:35,520 epoch 10 - iter 2/3 - loss 0.00624158 - samples/sec: 23.50 - lr: 0.005000\n",
      "2021-05-25 12:15:35,560 epoch 10 - iter 3/3 - loss 0.00475398 - samples/sec: 25.02 - lr: 0.005000\n",
      "2021-05-25 12:15:35,561 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:35,561 EPOCH 10 done: loss 0.0048 - lr 0.0050000\n",
      "2021-05-25 12:15:35,561 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:15:36,209 Test data not provided setting final score to 0\n",
      "2021-05-25 12:15:36,218 loading file /tmp/financial_phrasebank/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5b83b3ce14cc4ef9a25e83743a24f20e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.427\n",
      "2021-05-25 12:15:49,940 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:15:53,780 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 11870.67it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:15:53,783 [b'neutral', b'positive', b'negative']\n",
      "2021-05-25 12:15:53,784 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:53,785 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:15:53,786 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:53,786 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:15:53,786 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:53,786 Parameters:\n",
      "2021-05-25 12:15:53,787  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:15:53,787  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:15:53,787  - patience: \"3\"\n",
      "2021-05-25 12:15:53,787  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:15:53,788  - max_epochs: \"10\"\n",
      "2021-05-25 12:15:53,788  - shuffle: \"True\"\n",
      "2021-05-25 12:15:53,788  - train_with_dev: \"False\"\n",
      "2021-05-25 12:15:53,788  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:15:53,788 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:53,789 Model training base path: \"/tmp/financial_phrasebank\"\n",
      "2021-05-25 12:15:53,789 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:53,789 Device: cuda:0\n",
      "2021-05-25 12:15:53,789 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:53,790 Embeddings storage mode: cpu\n",
      "2021-05-25 12:15:53,793 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:53,844 epoch 1 - iter 1/3 - loss 1.02563608 - samples/sec: 25.15 - lr: 0.020000\n",
      "2021-05-25 12:15:53,892 epoch 1 - iter 2/3 - loss 1.10570645 - samples/sec: 20.97 - lr: 0.020000\n",
      "2021-05-25 12:15:53,936 epoch 1 - iter 3/3 - loss 1.16077832 - samples/sec: 23.04 - lr: 0.020000\n",
      "2021-05-25 12:15:53,937 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:53,938 EPOCH 1 done: loss 1.1608 - lr 0.0200000\n",
      "2021-05-25 12:15:53,938 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:15:54,612 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:54,671 epoch 2 - iter 1/3 - loss 0.05040987 - samples/sec: 21.49 - lr: 0.020000\n",
      "2021-05-25 12:15:54,716 epoch 2 - iter 2/3 - loss 0.10424401 - samples/sec: 22.51 - lr: 0.020000\n",
      "2021-05-25 12:15:54,759 epoch 2 - iter 3/3 - loss 0.26191931 - samples/sec: 23.55 - lr: 0.020000\n",
      "2021-05-25 12:15:54,760 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:54,761 EPOCH 2 done: loss 0.2619 - lr 0.0200000\n",
      "2021-05-25 12:15:54,761 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:15:54,762 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:54,818 epoch 3 - iter 1/3 - loss 0.02477176 - samples/sec: 22.09 - lr: 0.020000\n",
      "2021-05-25 12:15:54,859 epoch 3 - iter 2/3 - loss 0.01981251 - samples/sec: 24.99 - lr: 0.020000\n",
      "2021-05-25 12:15:54,902 epoch 3 - iter 3/3 - loss 0.23545666 - samples/sec: 23.64 - lr: 0.020000\n",
      "2021-05-25 12:15:54,903 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:54,904 EPOCH 3 done: loss 0.2355 - lr 0.0200000\n",
      "2021-05-25 12:15:54,904 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:15:54,905 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:54,960 epoch 4 - iter 1/3 - loss 0.00400593 - samples/sec: 23.02 - lr: 0.020000\n",
      "2021-05-25 12:15:55,001 epoch 4 - iter 2/3 - loss 0.01302922 - samples/sec: 24.97 - lr: 0.020000\n",
      "2021-05-25 12:15:55,044 epoch 4 - iter 3/3 - loss 0.30326090 - samples/sec: 23.30 - lr: 0.020000\n",
      "2021-05-25 12:15:55,046 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:55,046 EPOCH 4 done: loss 0.3033 - lr 0.0200000\n",
      "2021-05-25 12:15:55,047 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:15:55,047 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:55,101 epoch 5 - iter 1/3 - loss 0.00402595 - samples/sec: 23.97 - lr: 0.020000\n",
      "2021-05-25 12:15:55,143 epoch 5 - iter 2/3 - loss 0.00983456 - samples/sec: 24.18 - lr: 0.020000\n",
      "2021-05-25 12:15:55,183 epoch 5 - iter 3/3 - loss 0.01578107 - samples/sec: 25.48 - lr: 0.020000\n",
      "2021-05-25 12:15:55,184 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:55,184 EPOCH 5 done: loss 0.0158 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:15:55,185 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:15:55,186 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:55,239 epoch 6 - iter 1/3 - loss 0.00469666 - samples/sec: 23.66 - lr: 0.010000\n",
      "2021-05-25 12:15:55,281 epoch 6 - iter 2/3 - loss 0.00370215 - samples/sec: 24.24 - lr: 0.010000\n",
      "2021-05-25 12:15:55,322 epoch 6 - iter 3/3 - loss 0.01203016 - samples/sec: 24.63 - lr: 0.010000\n",
      "2021-05-25 12:15:55,323 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:55,324 EPOCH 6 done: loss 0.0120 - lr 0.0100000\n",
      "2021-05-25 12:15:55,325 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:15:55,325 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:55,378 epoch 7 - iter 1/3 - loss 0.00610898 - samples/sec: 24.25 - lr: 0.010000\n",
      "2021-05-25 12:15:55,420 epoch 7 - iter 2/3 - loss 0.00357580 - samples/sec: 24.19 - lr: 0.010000\n",
      "2021-05-25 12:15:55,461 epoch 7 - iter 3/3 - loss 0.00740720 - samples/sec: 24.39 - lr: 0.010000\n",
      "2021-05-25 12:15:55,463 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:55,463 EPOCH 7 done: loss 0.0074 - lr 0.0100000\n",
      "2021-05-25 12:15:55,464 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:15:55,465 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:55,516 epoch 8 - iter 1/3 - loss 0.02045113 - samples/sec: 25.27 - lr: 0.010000\n",
      "2021-05-25 12:15:55,555 epoch 8 - iter 2/3 - loss 0.01189839 - samples/sec: 25.85 - lr: 0.010000\n",
      "2021-05-25 12:15:55,597 epoch 8 - iter 3/3 - loss 0.00908704 - samples/sec: 24.40 - lr: 0.010000\n",
      "2021-05-25 12:15:55,598 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:55,598 EPOCH 8 done: loss 0.0091 - lr 0.0100000\n",
      "2021-05-25 12:15:55,599 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:15:55,599 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:55,654 epoch 9 - iter 1/3 - loss 0.00223496 - samples/sec: 23.06 - lr: 0.010000\n",
      "2021-05-25 12:15:55,695 epoch 9 - iter 2/3 - loss 0.00813806 - samples/sec: 25.32 - lr: 0.010000\n",
      "2021-05-25 12:15:55,735 epoch 9 - iter 3/3 - loss 0.00603264 - samples/sec: 25.19 - lr: 0.010000\n",
      "2021-05-25 12:15:55,736 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:55,737 EPOCH 9 done: loss 0.0060 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:15:55,737 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:15:55,738 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:55,787 epoch 10 - iter 1/3 - loss 0.00056619 - samples/sec: 26.03 - lr: 0.005000\n",
      "2021-05-25 12:15:55,829 epoch 10 - iter 2/3 - loss 0.00167098 - samples/sec: 24.29 - lr: 0.005000\n",
      "2021-05-25 12:15:55,875 epoch 10 - iter 3/3 - loss 0.00429777 - samples/sec: 21.86 - lr: 0.005000\n",
      "2021-05-25 12:15:55,877 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:15:55,877 EPOCH 10 done: loss 0.0043 - lr 0.0050000\n",
      "2021-05-25 12:15:55,878 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:15:56,530 Test data not provided setting final score to 0\n",
      "2021-05-25 12:15:56,538 loading file /tmp/financial_phrasebank/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cc103cb7c021400e82299526197c62e4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.409\n",
      "2021-05-25 12:16:10,082 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:16:13,779 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 7239.88it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:16:13,781 [b'neutral', b'negative', b'positive']\n",
      "2021-05-25 12:16:13,782 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:13,783 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:16:13,784 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:13,784 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:16:13,784 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:13,785 Parameters:\n",
      "2021-05-25 12:16:13,785  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:16:13,785  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:16:13,785  - patience: \"3\"\n",
      "2021-05-25 12:16:13,786  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:16:13,786  - max_epochs: \"10\"\n",
      "2021-05-25 12:16:13,786  - shuffle: \"True\"\n",
      "2021-05-25 12:16:13,786  - train_with_dev: \"False\"\n",
      "2021-05-25 12:16:13,787  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:16:13,787 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:13,787 Model training base path: \"/tmp/financial_phrasebank\"\n",
      "2021-05-25 12:16:13,787 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:13,788 Device: cuda:0\n",
      "2021-05-25 12:16:13,788 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:13,788 Embeddings storage mode: cpu\n",
      "2021-05-25 12:16:13,792 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:13,841 epoch 1 - iter 1/3 - loss 1.29203808 - samples/sec: 26.40 - lr: 0.020000\n",
      "2021-05-25 12:16:13,888 epoch 1 - iter 2/3 - loss 1.30565202 - samples/sec: 21.45 - lr: 0.020000\n",
      "2021-05-25 12:16:13,931 epoch 1 - iter 3/3 - loss 1.57933283 - samples/sec: 23.51 - lr: 0.020000\n",
      "2021-05-25 12:16:13,932 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:13,933 EPOCH 1 done: loss 1.5793 - lr 0.0200000\n",
      "2021-05-25 12:16:13,934 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:16:14,610 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:14,668 epoch 2 - iter 1/3 - loss 0.55709928 - samples/sec: 21.80 - lr: 0.020000\n",
      "2021-05-25 12:16:14,716 epoch 2 - iter 2/3 - loss 0.66018057 - samples/sec: 21.30 - lr: 0.020000\n",
      "2021-05-25 12:16:14,759 epoch 2 - iter 3/3 - loss 0.50495655 - samples/sec: 23.95 - lr: 0.020000\n",
      "2021-05-25 12:16:14,760 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:14,761 EPOCH 2 done: loss 0.5050 - lr 0.0200000\n",
      "2021-05-25 12:16:14,761 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:16:14,762 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:14,818 epoch 3 - iter 1/3 - loss 0.06017610 - samples/sec: 22.56 - lr: 0.020000\n",
      "2021-05-25 12:16:14,869 epoch 3 - iter 2/3 - loss 0.11279525 - samples/sec: 19.95 - lr: 0.020000\n",
      "2021-05-25 12:16:14,922 epoch 3 - iter 3/3 - loss 0.09436473 - samples/sec: 19.11 - lr: 0.020000\n",
      "2021-05-25 12:16:14,923 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:14,924 EPOCH 3 done: loss 0.0944 - lr 0.0200000\n",
      "2021-05-25 12:16:14,924 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:16:14,925 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:14,986 epoch 4 - iter 1/3 - loss 0.04936460 - samples/sec: 21.08 - lr: 0.020000\n",
      "2021-05-25 12:16:15,036 epoch 4 - iter 2/3 - loss 0.02731160 - samples/sec: 20.22 - lr: 0.020000\n",
      "2021-05-25 12:16:15,088 epoch 4 - iter 3/3 - loss 0.19035965 - samples/sec: 19.54 - lr: 0.020000\n",
      "2021-05-25 12:16:15,089 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:15,090 EPOCH 4 done: loss 0.1904 - lr 0.0200000\n",
      "2021-05-25 12:16:15,091 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:16:15,092 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:15,151 epoch 5 - iter 1/3 - loss 0.00177381 - samples/sec: 21.12 - lr: 0.020000\n",
      "2021-05-25 12:16:15,198 epoch 5 - iter 2/3 - loss 0.00238288 - samples/sec: 21.75 - lr: 0.020000\n",
      "2021-05-25 12:16:15,248 epoch 5 - iter 3/3 - loss 0.00996008 - samples/sec: 20.48 - lr: 0.020000\n",
      "2021-05-25 12:16:15,249 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:15,250 EPOCH 5 done: loss 0.0100 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:16:15,250 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:16:15,251 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:15,312 epoch 6 - iter 1/3 - loss 0.00204606 - samples/sec: 21.10 - lr: 0.010000\n",
      "2021-05-25 12:16:15,355 epoch 6 - iter 2/3 - loss 0.01076652 - samples/sec: 23.37 - lr: 0.010000\n",
      "2021-05-25 12:16:15,398 epoch 6 - iter 3/3 - loss 0.00780877 - samples/sec: 23.62 - lr: 0.010000\n",
      "2021-05-25 12:16:15,399 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:15,400 EPOCH 6 done: loss 0.0078 - lr 0.0100000\n",
      "2021-05-25 12:16:15,400 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:16:15,401 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:15,460 epoch 7 - iter 1/3 - loss 0.00190871 - samples/sec: 21.41 - lr: 0.010000\n",
      "2021-05-25 12:16:15,501 epoch 7 - iter 2/3 - loss 0.00557428 - samples/sec: 24.61 - lr: 0.010000\n",
      "2021-05-25 12:16:15,543 epoch 7 - iter 3/3 - loss 0.00938217 - samples/sec: 24.62 - lr: 0.010000\n",
      "2021-05-25 12:16:15,544 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:15,544 EPOCH 7 done: loss 0.0094 - lr 0.0100000\n",
      "2021-05-25 12:16:15,545 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:16:15,546 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:15,598 epoch 8 - iter 1/3 - loss 0.00119829 - samples/sec: 24.24 - lr: 0.010000\n",
      "2021-05-25 12:16:15,639 epoch 8 - iter 2/3 - loss 0.00128037 - samples/sec: 24.52 - lr: 0.010000\n",
      "2021-05-25 12:16:15,681 epoch 8 - iter 3/3 - loss 0.00488618 - samples/sec: 24.21 - lr: 0.010000\n",
      "2021-05-25 12:16:15,683 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:15,683 EPOCH 8 done: loss 0.0049 - lr 0.0100000\n",
      "2021-05-25 12:16:15,684 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:16:15,685 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:15,739 epoch 9 - iter 1/3 - loss 0.00113331 - samples/sec: 23.51 - lr: 0.010000\n",
      "2021-05-25 12:16:15,781 epoch 9 - iter 2/3 - loss 0.00405399 - samples/sec: 24.42 - lr: 0.010000\n",
      "2021-05-25 12:16:15,822 epoch 9 - iter 3/3 - loss 0.00324601 - samples/sec: 24.78 - lr: 0.010000\n",
      "2021-05-25 12:16:15,823 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:15,824 EPOCH 9 done: loss 0.0032 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:16:15,824 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:16:15,825 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:15,883 epoch 10 - iter 1/3 - loss 0.00111780 - samples/sec: 21.85 - lr: 0.005000\n",
      "2021-05-25 12:16:15,925 epoch 10 - iter 2/3 - loss 0.00230153 - samples/sec: 24.29 - lr: 0.005000\n",
      "2021-05-25 12:16:15,967 epoch 10 - iter 3/3 - loss 0.00217359 - samples/sec: 24.30 - lr: 0.005000\n",
      "2021-05-25 12:16:15,968 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:15,968 EPOCH 10 done: loss 0.0022 - lr 0.0050000\n",
      "2021-05-25 12:16:15,969 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:16:16,647 Test data not provided setting final score to 0\n",
      "2021-05-25 12:16:16,654 loading file /tmp/financial_phrasebank/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "63ff1ea2a75f4f4b8f3fd32993abd309",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.554\n",
      "2021-05-25 12:16:30,215 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:16:33,923 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 9923.43it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:16:33,926 [b'positive', b'negative', b'neutral']\n",
      "2021-05-25 12:16:33,927 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:33,928 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:16:33,929 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:33,930 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:16:33,930 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:33,931 Parameters:\n",
      "2021-05-25 12:16:33,931  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:16:33,931  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:16:33,931  - patience: \"3\"\n",
      "2021-05-25 12:16:33,931  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:16:33,932  - max_epochs: \"10\"\n",
      "2021-05-25 12:16:33,932  - shuffle: \"True\"\n",
      "2021-05-25 12:16:33,932  - train_with_dev: \"False\"\n",
      "2021-05-25 12:16:33,932  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:16:33,933 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:33,933 Model training base path: \"/tmp/financial_phrasebank\"\n",
      "2021-05-25 12:16:33,933 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:33,933 Device: cuda:0\n",
      "2021-05-25 12:16:33,934 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:33,934 Embeddings storage mode: cpu\n",
      "2021-05-25 12:16:33,937 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:33,991 epoch 1 - iter 1/3 - loss 1.47704351 - samples/sec: 23.66 - lr: 0.020000\n",
      "2021-05-25 12:16:34,035 epoch 1 - iter 2/3 - loss 1.14851230 - samples/sec: 22.71 - lr: 0.020000\n",
      "2021-05-25 12:16:34,079 epoch 1 - iter 3/3 - loss 1.18195820 - samples/sec: 23.09 - lr: 0.020000\n",
      "2021-05-25 12:16:34,080 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:34,081 EPOCH 1 done: loss 1.1820 - lr 0.0200000\n",
      "2021-05-25 12:16:34,081 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:16:34,764 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:34,825 epoch 2 - iter 1/3 - loss 0.70098358 - samples/sec: 20.67 - lr: 0.020000\n",
      "2021-05-25 12:16:34,873 epoch 2 - iter 2/3 - loss 0.48046508 - samples/sec: 21.38 - lr: 0.020000\n",
      "2021-05-25 12:16:34,916 epoch 2 - iter 3/3 - loss 0.40018753 - samples/sec: 23.51 - lr: 0.020000\n",
      "2021-05-25 12:16:34,917 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:34,918 EPOCH 2 done: loss 0.4002 - lr 0.0200000\n",
      "2021-05-25 12:16:34,918 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:16:34,919 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:34,977 epoch 3 - iter 1/3 - loss 0.24936408 - samples/sec: 21.81 - lr: 0.020000\n",
      "2021-05-25 12:16:35,020 epoch 3 - iter 2/3 - loss 0.15492204 - samples/sec: 23.17 - lr: 0.020000\n",
      "2021-05-25 12:16:35,065 epoch 3 - iter 3/3 - loss 0.27657163 - samples/sec: 22.68 - lr: 0.020000\n",
      "2021-05-25 12:16:35,066 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:35,067 EPOCH 3 done: loss 0.2766 - lr 0.0200000\n",
      "2021-05-25 12:16:35,068 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:16:35,068 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:35,124 epoch 4 - iter 1/3 - loss 0.01459855 - samples/sec: 22.25 - lr: 0.020000\n",
      "2021-05-25 12:16:35,166 epoch 4 - iter 2/3 - loss 0.00966776 - samples/sec: 24.30 - lr: 0.020000\n",
      "2021-05-25 12:16:35,209 epoch 4 - iter 3/3 - loss 0.18028196 - samples/sec: 23.68 - lr: 0.020000\n",
      "2021-05-25 12:16:35,210 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:35,211 EPOCH 4 done: loss 0.1803 - lr 0.0200000\n",
      "2021-05-25 12:16:35,211 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:16:35,212 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:35,266 epoch 5 - iter 1/3 - loss 0.00388216 - samples/sec: 23.85 - lr: 0.020000\n",
      "2021-05-25 12:16:35,307 epoch 5 - iter 2/3 - loss 0.00951286 - samples/sec: 24.45 - lr: 0.020000\n",
      "2021-05-25 12:16:35,348 epoch 5 - iter 3/3 - loss 0.02127222 - samples/sec: 24.85 - lr: 0.020000\n",
      "2021-05-25 12:16:35,350 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:35,350 EPOCH 5 done: loss 0.0213 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:16:35,351 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:16:35,352 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:35,407 epoch 6 - iter 1/3 - loss 0.01036518 - samples/sec: 22.66 - lr: 0.010000\n",
      "2021-05-25 12:16:35,450 epoch 6 - iter 2/3 - loss 0.00731384 - samples/sec: 23.68 - lr: 0.010000\n",
      "2021-05-25 12:16:35,489 epoch 6 - iter 3/3 - loss 0.00966531 - samples/sec: 25.77 - lr: 0.010000\n",
      "2021-05-25 12:16:35,490 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:35,491 EPOCH 6 done: loss 0.0097 - lr 0.0100000\n",
      "2021-05-25 12:16:35,491 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:16:35,492 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:35,545 epoch 7 - iter 1/3 - loss 0.01680582 - samples/sec: 23.73 - lr: 0.010000\n",
      "2021-05-25 12:16:35,586 epoch 7 - iter 2/3 - loss 0.00949669 - samples/sec: 24.62 - lr: 0.010000\n",
      "2021-05-25 12:16:35,628 epoch 7 - iter 3/3 - loss 0.00705883 - samples/sec: 24.38 - lr: 0.010000\n",
      "2021-05-25 12:16:35,629 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:35,630 EPOCH 7 done: loss 0.0071 - lr 0.0100000\n",
      "2021-05-25 12:16:35,630 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:16:35,631 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:35,686 epoch 8 - iter 1/3 - loss 0.00178796 - samples/sec: 22.95 - lr: 0.010000\n",
      "2021-05-25 12:16:35,727 epoch 8 - iter 2/3 - loss 0.00554474 - samples/sec: 24.84 - lr: 0.010000\n",
      "2021-05-25 12:16:35,767 epoch 8 - iter 3/3 - loss 0.00592947 - samples/sec: 25.09 - lr: 0.010000\n",
      "2021-05-25 12:16:35,769 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:35,769 EPOCH 8 done: loss 0.0059 - lr 0.0100000\n",
      "2021-05-25 12:16:35,770 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:16:35,771 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:35,821 epoch 9 - iter 1/3 - loss 0.01392245 - samples/sec: 25.88 - lr: 0.010000\n",
      "2021-05-25 12:16:35,862 epoch 9 - iter 2/3 - loss 0.00777394 - samples/sec: 24.59 - lr: 0.010000\n",
      "2021-05-25 12:16:35,902 epoch 9 - iter 3/3 - loss 0.00792434 - samples/sec: 25.40 - lr: 0.010000\n",
      "2021-05-25 12:16:35,903 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:35,904 EPOCH 9 done: loss 0.0079 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:16:35,905 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:16:35,905 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:35,960 epoch 10 - iter 1/3 - loss 0.00148433 - samples/sec: 22.90 - lr: 0.005000\n",
      "2021-05-25 12:16:36,002 epoch 10 - iter 2/3 - loss 0.00174796 - samples/sec: 24.47 - lr: 0.005000\n",
      "2021-05-25 12:16:36,042 epoch 10 - iter 3/3 - loss 0.00438558 - samples/sec: 25.12 - lr: 0.005000\n",
      "2021-05-25 12:16:36,043 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:36,044 EPOCH 10 done: loss 0.0044 - lr 0.0050000\n",
      "2021-05-25 12:16:36,044 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:16:36,720 Test data not provided setting final score to 0\n",
      "2021-05-25 12:16:36,728 loading file /tmp/financial_phrasebank/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b259fbc14ff74e91a3061a9148453477",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.551\n",
      "2021-05-25 12:16:50,346 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:16:54,062 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 10791.52it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:16:54,065 [b'neutral', b'negative', b'positive']\n",
      "2021-05-25 12:16:54,066 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:54,068 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:16:54,069 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:54,069 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:16:54,070 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:54,070 Parameters:\n",
      "2021-05-25 12:16:54,070  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:16:54,071  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:16:54,071  - patience: \"3\"\n",
      "2021-05-25 12:16:54,072  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:16:54,072  - max_epochs: \"10\"\n",
      "2021-05-25 12:16:54,073  - shuffle: \"True\"\n",
      "2021-05-25 12:16:54,073  - train_with_dev: \"False\"\n",
      "2021-05-25 12:16:54,073  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:16:54,074 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:54,074 Model training base path: \"/tmp/financial_phrasebank\"\n",
      "2021-05-25 12:16:54,074 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:54,074 Device: cuda:0\n",
      "2021-05-25 12:16:54,074 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:54,075 Embeddings storage mode: cpu\n",
      "2021-05-25 12:16:54,078 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:54,133 epoch 1 - iter 1/3 - loss 1.09389746 - samples/sec: 23.04 - lr: 0.020000\n",
      "2021-05-25 12:16:54,178 epoch 1 - iter 2/3 - loss 0.98965123 - samples/sec: 22.52 - lr: 0.020000\n",
      "2021-05-25 12:16:54,222 epoch 1 - iter 3/3 - loss 0.76306619 - samples/sec: 22.89 - lr: 0.020000\n",
      "2021-05-25 12:16:54,224 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:54,224 EPOCH 1 done: loss 0.7631 - lr 0.0200000\n",
      "2021-05-25 12:16:54,225 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:16:54,899 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:54,959 epoch 2 - iter 1/3 - loss 0.13108486 - samples/sec: 21.16 - lr: 0.020000\n",
      "2021-05-25 12:16:55,004 epoch 2 - iter 2/3 - loss 0.09518647 - samples/sec: 23.04 - lr: 0.020000\n",
      "2021-05-25 12:16:55,045 epoch 2 - iter 3/3 - loss 0.06596714 - samples/sec: 24.60 - lr: 0.020000\n",
      "2021-05-25 12:16:55,046 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:55,047 EPOCH 2 done: loss 0.0660 - lr 0.0200000\n",
      "2021-05-25 12:16:55,047 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:16:55,048 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:55,100 epoch 3 - iter 1/3 - loss 0.00796495 - samples/sec: 25.31 - lr: 0.020000\n",
      "2021-05-25 12:16:55,142 epoch 3 - iter 2/3 - loss 0.01196029 - samples/sec: 23.78 - lr: 0.020000\n",
      "2021-05-25 12:16:55,185 epoch 3 - iter 3/3 - loss 0.01134319 - samples/sec: 24.11 - lr: 0.020000\n",
      "2021-05-25 12:16:55,186 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:55,186 EPOCH 3 done: loss 0.0113 - lr 0.0200000\n",
      "2021-05-25 12:16:55,187 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:16:55,188 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:55,242 epoch 4 - iter 1/3 - loss 0.04017564 - samples/sec: 23.47 - lr: 0.020000\n",
      "2021-05-25 12:16:55,285 epoch 4 - iter 2/3 - loss 0.02803429 - samples/sec: 23.46 - lr: 0.020000\n",
      "2021-05-25 12:16:55,328 epoch 4 - iter 3/3 - loss 0.02259880 - samples/sec: 24.05 - lr: 0.020000\n",
      "2021-05-25 12:16:55,329 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:55,330 EPOCH 4 done: loss 0.0226 - lr 0.0200000\n",
      "2021-05-25 12:16:55,330 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:16:55,331 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:55,386 epoch 5 - iter 1/3 - loss 0.01225678 - samples/sec: 22.91 - lr: 0.020000\n",
      "2021-05-25 12:16:55,430 epoch 5 - iter 2/3 - loss 0.00698427 - samples/sec: 23.23 - lr: 0.020000\n",
      "2021-05-25 12:16:55,471 epoch 5 - iter 3/3 - loss 0.00573342 - samples/sec: 24.55 - lr: 0.020000\n",
      "2021-05-25 12:16:55,472 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:55,472 EPOCH 5 done: loss 0.0057 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:16:55,473 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:16:55,474 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:55,527 epoch 6 - iter 1/3 - loss 0.01342698 - samples/sec: 23.45 - lr: 0.010000\n",
      "2021-05-25 12:16:55,568 epoch 6 - iter 2/3 - loss 0.00821228 - samples/sec: 24.46 - lr: 0.010000\n",
      "2021-05-25 12:16:55,609 epoch 6 - iter 3/3 - loss 0.00671858 - samples/sec: 25.04 - lr: 0.010000\n",
      "2021-05-25 12:16:55,610 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:55,611 EPOCH 6 done: loss 0.0067 - lr 0.0100000\n",
      "2021-05-25 12:16:55,611 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:16:55,612 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:55,665 epoch 7 - iter 1/3 - loss 0.00109220 - samples/sec: 23.99 - lr: 0.010000\n",
      "2021-05-25 12:16:55,707 epoch 7 - iter 2/3 - loss 0.00297308 - samples/sec: 24.51 - lr: 0.010000\n",
      "2021-05-25 12:16:55,750 epoch 7 - iter 3/3 - loss 0.00315864 - samples/sec: 23.58 - lr: 0.010000\n",
      "2021-05-25 12:16:55,751 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:55,752 EPOCH 7 done: loss 0.0032 - lr 0.0100000\n",
      "2021-05-25 12:16:55,752 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:16:55,753 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:55,806 epoch 8 - iter 1/3 - loss 0.00085724 - samples/sec: 24.00 - lr: 0.010000\n",
      "2021-05-25 12:16:55,848 epoch 8 - iter 2/3 - loss 0.00167456 - samples/sec: 24.48 - lr: 0.010000\n",
      "2021-05-25 12:16:55,890 epoch 8 - iter 3/3 - loss 0.00289149 - samples/sec: 24.31 - lr: 0.010000\n",
      "2021-05-25 12:16:55,891 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:55,892 EPOCH 8 done: loss 0.0029 - lr 0.0100000\n",
      "2021-05-25 12:16:55,892 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:16:55,893 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:55,944 epoch 9 - iter 1/3 - loss 0.00747904 - samples/sec: 25.36 - lr: 0.010000\n",
      "2021-05-25 12:16:55,986 epoch 9 - iter 2/3 - loss 0.00681006 - samples/sec: 24.46 - lr: 0.010000\n",
      "2021-05-25 12:16:56,030 epoch 9 - iter 3/3 - loss 0.00487235 - samples/sec: 23.12 - lr: 0.010000\n",
      "2021-05-25 12:16:56,031 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:56,045 EPOCH 9 done: loss 0.0049 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:16:56,045 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:16:56,046 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:56,094 epoch 10 - iter 1/3 - loss 0.00188122 - samples/sec: 26.51 - lr: 0.005000\n",
      "2021-05-25 12:16:56,133 epoch 10 - iter 2/3 - loss 0.00212799 - samples/sec: 26.09 - lr: 0.005000\n",
      "2021-05-25 12:16:56,175 epoch 10 - iter 3/3 - loss 0.00228427 - samples/sec: 24.60 - lr: 0.005000\n",
      "2021-05-25 12:16:56,176 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:16:56,176 EPOCH 10 done: loss 0.0023 - lr 0.0050000\n",
      "2021-05-25 12:16:56,177 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:16:56,847 Test data not provided setting final score to 0\n",
      "2021-05-25 12:16:56,854 loading file /tmp/financial_phrasebank/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ccc03761246142828b61fa89b3d085dc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.342\n",
      "2021-05-25 12:17:10,699 loading file /home/yves/.flair/models/tars-base-v8.pt\n",
      "init TARS\n",
      "2021-05-25 12:17:14,429 Computing label dictionary. Progress:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:00<00:00, 10609.54it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:17:14,432 [b'negative', b'neutral', b'positive']\n",
      "2021-05-25 12:17:14,433 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:14,435 Model: \"TARSClassifier(\n",
      "  (document_embeddings): None\n",
      "  (decoder): None\n",
      "  (loss_function): None\n",
      "  (tars_model): TextClassifier(\n",
      "    (document_embeddings): TransformerDocumentEmbeddings(\n",
      "      (model): BertModel(\n",
      "        (embeddings): BertEmbeddings(\n",
      "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
      "          (position_embeddings): Embedding(512, 768)\n",
      "          (token_type_embeddings): Embedding(2, 768)\n",
      "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (encoder): BertEncoder(\n",
      "          (layer): ModuleList(\n",
      "            (0): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (1): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (2): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (3): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (4): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (5): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (6): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (7): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (8): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (9): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (10): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (11): BertLayer(\n",
      "              (attention): BertAttention(\n",
      "                (self): BertSelfAttention(\n",
      "                  (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "                (output): BertSelfOutput(\n",
      "                  (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                  (dropout): Dropout(p=0.1, inplace=False)\n",
      "                )\n",
      "              )\n",
      "              (intermediate): BertIntermediate(\n",
      "                (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "              )\n",
      "              (output): BertOutput(\n",
      "                (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "        (pooler): BertPooler(\n",
      "          (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (activation): Tanh()\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (decoder): Linear(in_features=768, out_features=2, bias=True)\n",
      "    (loss_function): CrossEntropyLoss()\n",
      "  )\n",
      "  (beta): 1.0\n",
      "  (weights): None\n",
      "  (weight_tensor) None\n",
      ")\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-25 12:17:14,435 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:14,436 Corpus: \"Corpus: 3 train + 0 dev + 0 test sentences\"\n",
      "2021-05-25 12:17:14,436 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:14,436 Parameters:\n",
      "2021-05-25 12:17:14,436  - learning_rate: \"0.02\"\n",
      "2021-05-25 12:17:14,437  - mini_batch_size: \"1\"\n",
      "2021-05-25 12:17:14,437  - patience: \"3\"\n",
      "2021-05-25 12:17:14,437  - anneal_factor: \"0.5\"\n",
      "2021-05-25 12:17:14,437  - max_epochs: \"10\"\n",
      "2021-05-25 12:17:14,437  - shuffle: \"True\"\n",
      "2021-05-25 12:17:14,438  - train_with_dev: \"False\"\n",
      "2021-05-25 12:17:14,438  - batch_growth_annealing: \"False\"\n",
      "2021-05-25 12:17:14,438 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:14,438 Model training base path: \"/tmp/financial_phrasebank\"\n",
      "2021-05-25 12:17:14,439 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:14,439 Device: cuda:0\n",
      "2021-05-25 12:17:14,439 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:14,439 Embeddings storage mode: cpu\n",
      "2021-05-25 12:17:14,443 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:14,492 epoch 1 - iter 1/3 - loss 0.28455985 - samples/sec: 26.52 - lr: 0.020000\n",
      "2021-05-25 12:17:14,536 epoch 1 - iter 2/3 - loss 0.55532333 - samples/sec: 22.98 - lr: 0.020000\n",
      "2021-05-25 12:17:14,581 epoch 1 - iter 3/3 - loss 0.70101776 - samples/sec: 22.48 - lr: 0.020000\n",
      "2021-05-25 12:17:14,582 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:14,582 EPOCH 1 done: loss 0.7010 - lr 0.0200000\n",
      "2021-05-25 12:17:14,582 BAD EPOCHS (no improvement): 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving best model\n",
      "2021-05-25 12:17:15,250 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:15,310 epoch 2 - iter 1/3 - loss 0.08467761 - samples/sec: 21.21 - lr: 0.020000\n",
      "2021-05-25 12:17:15,356 epoch 2 - iter 2/3 - loss 0.38372432 - samples/sec: 22.07 - lr: 0.020000\n",
      "2021-05-25 12:17:15,399 epoch 2 - iter 3/3 - loss 0.27040606 - samples/sec: 23.72 - lr: 0.020000\n",
      "2021-05-25 12:17:15,400 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:15,400 EPOCH 2 done: loss 0.2704 - lr 0.0200000\n",
      "2021-05-25 12:17:15,400 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:17:15,401 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:15,458 epoch 3 - iter 1/3 - loss 0.44373158 - samples/sec: 21.64 - lr: 0.020000\n",
      "2021-05-25 12:17:15,500 epoch 3 - iter 2/3 - loss 0.29286896 - samples/sec: 23.86 - lr: 0.020000\n",
      "2021-05-25 12:17:15,542 epoch 3 - iter 3/3 - loss 0.21473947 - samples/sec: 24.29 - lr: 0.020000\n",
      "2021-05-25 12:17:15,543 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:15,543 EPOCH 3 done: loss 0.2147 - lr 0.0200000\n",
      "2021-05-25 12:17:15,543 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:17:15,544 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:15,596 epoch 4 - iter 1/3 - loss 0.00170761 - samples/sec: 23.82 - lr: 0.020000\n",
      "2021-05-25 12:17:15,636 epoch 4 - iter 2/3 - loss 0.00516222 - samples/sec: 25.15 - lr: 0.020000\n",
      "2021-05-25 12:17:15,677 epoch 4 - iter 3/3 - loss 0.00908580 - samples/sec: 24.91 - lr: 0.020000\n",
      "2021-05-25 12:17:15,678 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:15,679 EPOCH 4 done: loss 0.0091 - lr 0.0200000\n",
      "2021-05-25 12:17:15,679 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:17:15,680 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:15,731 epoch 5 - iter 1/3 - loss 0.00291158 - samples/sec: 25.24 - lr: 0.020000\n",
      "2021-05-25 12:17:15,771 epoch 5 - iter 2/3 - loss 0.00487420 - samples/sec: 25.82 - lr: 0.020000\n",
      "2021-05-25 12:17:15,811 epoch 5 - iter 3/3 - loss 0.00352991 - samples/sec: 25.20 - lr: 0.020000\n",
      "2021-05-25 12:17:15,812 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:15,813 EPOCH 5 done: loss 0.0035 - lr 0.0200000\n",
      "Epoch     5: reducing learning rate of group 0 to 1.0000e-02.\n",
      "2021-05-25 12:17:15,813 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:17:15,814 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:15,867 epoch 6 - iter 1/3 - loss 0.00122718 - samples/sec: 23.59 - lr: 0.010000\n",
      "2021-05-25 12:17:15,909 epoch 6 - iter 2/3 - loss 0.00326641 - samples/sec: 24.56 - lr: 0.010000\n",
      "2021-05-25 12:17:15,951 epoch 6 - iter 3/3 - loss 0.00305591 - samples/sec: 24.33 - lr: 0.010000\n",
      "2021-05-25 12:17:15,952 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:15,952 EPOCH 6 done: loss 0.0031 - lr 0.0100000\n",
      "2021-05-25 12:17:15,953 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:17:15,953 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:16,009 epoch 7 - iter 1/3 - loss 0.00117394 - samples/sec: 22.78 - lr: 0.010000\n",
      "2021-05-25 12:17:16,051 epoch 7 - iter 2/3 - loss 0.00365677 - samples/sec: 24.09 - lr: 0.010000\n",
      "2021-05-25 12:17:16,092 epoch 7 - iter 3/3 - loss 0.00304420 - samples/sec: 24.96 - lr: 0.010000\n",
      "2021-05-25 12:17:16,093 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:16,093 EPOCH 7 done: loss 0.0030 - lr 0.0100000\n",
      "2021-05-25 12:17:16,094 BAD EPOCHS (no improvement): 2\n",
      "2021-05-25 12:17:16,095 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:16,147 epoch 8 - iter 1/3 - loss 0.00124271 - samples/sec: 24.34 - lr: 0.010000\n",
      "2021-05-25 12:17:16,190 epoch 8 - iter 2/3 - loss 0.00228257 - samples/sec: 23.16 - lr: 0.010000\n",
      "2021-05-25 12:17:16,231 epoch 8 - iter 3/3 - loss 0.00353697 - samples/sec: 24.94 - lr: 0.010000\n",
      "2021-05-25 12:17:16,233 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:16,233 EPOCH 8 done: loss 0.0035 - lr 0.0100000\n",
      "2021-05-25 12:17:16,234 BAD EPOCHS (no improvement): 3\n",
      "2021-05-25 12:17:16,234 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:16,285 epoch 9 - iter 1/3 - loss 0.00198273 - samples/sec: 25.52 - lr: 0.010000\n",
      "2021-05-25 12:17:16,326 epoch 9 - iter 2/3 - loss 0.00178080 - samples/sec: 25.32 - lr: 0.010000\n",
      "2021-05-25 12:17:16,366 epoch 9 - iter 3/3 - loss 0.00317438 - samples/sec: 25.31 - lr: 0.010000\n",
      "2021-05-25 12:17:16,367 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:16,367 EPOCH 9 done: loss 0.0032 - lr 0.0100000\n",
      "Epoch     9: reducing learning rate of group 0 to 5.0000e-03.\n",
      "2021-05-25 12:17:16,368 BAD EPOCHS (no improvement): 4\n",
      "2021-05-25 12:17:16,369 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:16,422 epoch 10 - iter 1/3 - loss 0.00121241 - samples/sec: 23.27 - lr: 0.005000\n",
      "2021-05-25 12:17:16,464 epoch 10 - iter 2/3 - loss 0.00159419 - samples/sec: 24.32 - lr: 0.005000\n",
      "2021-05-25 12:17:16,504 epoch 10 - iter 3/3 - loss 0.00231548 - samples/sec: 25.35 - lr: 0.005000\n",
      "2021-05-25 12:17:16,506 ----------------------------------------------------------------------------------------------------\n",
      "2021-05-25 12:17:16,506 EPOCH 10 done: loss 0.0023 - lr 0.0050000\n",
      "2021-05-25 12:17:16,507 BAD EPOCHS (no improvement): 1\n",
      "2021-05-25 12:17:17,176 Test data not provided setting final score to 0\n",
      "2021-05-25 12:17:17,183 loading file /tmp/financial_phrasebank/final-model.pt\n",
      "init TARS\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "381246ba18f44a3c9dfe433fbe49dd6f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Accuracy: 0.359\n"
     ]
    }
   ],
   "source": [
    "#results_fewshot = {}\n",
    "\n",
    "for dataset in data[3:]:\n",
    "    results_fewshot[dataset['name']] = []\n",
    "    for seed in range(10):\n",
    "\n",
    "        corpus = sample_training_corpus(dataset, seed)\n",
    "        train(corpus, dataset)\n",
    "        acc = evaluate(dataset)\n",
    "        results_fewshot[dataset['name']].append(acc)\n",
    "\n",
    "        print(\"Accuracy:\", acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "built-wallpaper",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f4fe5e21820>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuUAAAGhCAYAAADGG8oZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAACE5UlEQVR4nOzdd3xb1eH//9eRJe+Z2HHiDGdvEsgiJGRAQqBAwx5tgQIttKXtt1BaPpTdQhctPz58WloILVBaKKtlh51JyCZ7T2c4cex4b8k6vz8kK7bjISe25Djv5+Ohh3XvPffecyVZeuvo3HONtRYREREREQkfR7grICIiIiJyulMoFxEREREJM4VyEREREZEwUygXEREREQkzhXIRERERkTBTKBcRERERCTOFchERERGRMAtZKDfG/MgYs8oYU2WMebGFsncZYw4bY4qNMc8bY6JCVE0RERERkZALZUt5NvAY8HxzhYwxFwL3AjOATKA/8Mt2r52IiIiISJiELJRba/9rrX0bONpC0W8Df7fWbrLWFgCPAje3c/VERERERMLGGe4KNGIE8E6d6XVAujGmq7W2XqA3xtwO3A7w7LPPjr399ttDV0sREREROV2Ztt5gRwzl8UBRnena+wk0aGW31s4B5tROtn/VRERERETaXkccfaUUSKwzXXu/JAx1ERERERFpdx0xlG8CRteZHg3kNOy6IiIiIiLSWYRySESnMSYaiAAijDHRxpjGus+8BHzHGDPcGJMMPAC8GKp6ioiIiIiEWihbyh8AKvANd3iD//4Dxpg+xphSY0wfAGvtR8DjwHxgH5AFPBzCeoqIiIiIhJSxttOcH9lpDkRERKQz8nq9HDhwgLKysnBXRaRZLpeLbt26kZiY2FSR02L0FREREemE8vLyMMYwZMgQHI6OeFqbCFhrqaio4ODBgwDNBfM2pf8IERERCYnCwkLS09MVyKVDM8YQGxtLz549OXLkSMj2q/8KERERCYmamhpcLle4qyESlJiYGNxud8j2p1AuIiIiIWNMm3fFFWkXoX6tKpSLiIhIx/L44zB/fnBl58/3lRc5xSmUi4iISMcyfjxce23LwXz+fF+58ePbrSp79+7FGIPH42mx7OLFixkyZEi71eV0Y4xh586d4a5GyCiUi4iISMdy3nnw+uvNB/PaQP76677yJ6lv377ExMQQHx8fuGVnZ7dqG1OmTGHbtm1Bld23b1+9fRljiIuLC0wvXrwYgBdffBFjDK+99lq99RcsWIDD4SA+Pp6EhASGDBnCCy+8UK/MO++8w5lnnkliYiKpqamcf/757Nmzp1XHdCpqzRepjkShXERERDqe5oJ5GwfyWu+99x6lpaWBW0ZGRpttu2FA7NOnT719Aaxbty4wPWXKFAD+8Y9/0KVLF1566aXjtpmRkUFpaSnFxcU8+eST3HbbbYEvBTt37uSmm27iiSeeoKioiD179vDDH/6QiIiINjumlo5RWkehXERERDqmxoJ5OwXyYL3wwgsMGzaMhIQE+vfvz7PPPhtYtmDBAnr16hWY7tu3L7///e8ZNWoUcXFxrQ6tWVlZLFy4kDlz5vDxxx9z+PDhRssZY7j44ovp0qUL69evB2Dt2rX069ePGTNmYIwhISGBq666ij59+hy3fnZ2dr1W+9jY2HonOT7//PMMGzaMlJQULrzwQrKysurt++mnn2bQoEEMGjQIgOeee46BAwfSpUsXZs+e3ewvDjt37mTatGkkJSWRmprKddddV2/5Z599xqBBg0hOTuaHP/whtRe99Hq9PPbYY2RmZtKtWzduuukmioqKAJg6dSoAycnJxMfHs3Tp0hYf645AoVxEREQ6rrrB/KGHwhrIAbp168b7779PcXExL7zwAnfddRdfffVVk+X//e9/88EHH1BYWIjT2bprNr700kuMGzeOq666imHDhvHyyy83Ws7r9fLuu++Sl5fHwIEDARgzZgxbt27lrrvuYv78+YHW+MbUtrjX3q644gquv/56wNcF5je/+Q3//e9/yc3NZcqUKXzjG9+ot/7bb7/N8uXL2bx5M/PmzeMXv/gFr7/+OocOHSIzMzOwrcY8+OCDzJo1i4KCAg4cOMCPf/zjesvff/99Vq5cyfr163n99df5+OOPAV+3nhdffJH58+eze/duSktL+dGPfgTAokWLAN+4+KWlpZxzzjnNPcwdh7W2s9xERESkA9u8efOJr/zgg9aC7287yMzMtHFxcTYpKckmJSXZyy67zFpr7Z49eyxg3W53o+tddtll9n//93+ttdbOnz/f9uzZs942//73vwe1f8Du2LGj3ryBAwfaJ5980lpr7W9+8xs7atSowLL58+dbY4xNSkqykZGR1uFwBMrWWrp0qb3mmmtsamqqjYqKst/+9rdtSUlJs/X43e9+Z8eMGWPLy8uttdZedNFF9m9/+1tgeU1NjY2JibF79+4N1Pvzzz8PLL/11lvtz3/+88B0SUmJdTqdds+ePY3u78Ybb7S33Xab3b9/f6OPyeLFiwPT11xzjf3tb39rrbX2/PPPt08//XRg2datW63T6bRut7vF56w1mnnNtnmWVUu5iIiIdGzz58Nf/woPPuj7G+xwia309ttvU1hYSGFhIW+//XajZT788EMmTpxIly5dSE5OZu7cueTl5TW5zd69e59QXZYsWcKePXsCrczf/OY32bBhA2vXrg2UycjIoLCwkOLiYv7f//t/zJs3r942Jk6cyOuvv05ubi6LFy9m0aJF/PrXv25ynx9++CFPPfUUb7/9NjExMYCvC81PfvITkpOTSU5OpkuXLlhrA5egb3iM2dnZZGZmBqbj4+Pp2rUrBw8eZPHixYEuMiNGjADg8ccfx1rLhAkTGDFiBM8//3y9OnXv3j1wPzY2NtDi33A/mZmZeDwecnJymn9gO7DW/Y4iIiIiEkoN+5Cfd17YurBUVVVx1VVX8dJLL3HZZZfhcrm4/PLLA/2cG3OiF6D5xz/+gbWWM88887j5DedFRUXx+9//niFDhvD2229z+eWXH7e98ePHc+WVV7Jx48ZG97dt2za+/e1v89///rdeyO7duzf3338/3/rWt5qsa91jzMjIqNfnvKysjKNHj9KzZ0/69u17XDea7t2789xzzwHwxRdfMHPmTKZOnRrohtOUhvvZt28fTqeT9PT0el8YTiVqKRcREZGOqbGTOoMZLrGdVFdXU1VVRVpaGk6nkw8//JBPPvmkzfdTWVnJ66+/zpw5c1i7dm3g9qc//YlXXnml0RNGIyMjufvuu/nVr34F+ALuc889x5EjRwDYunUr7777LhMnTjxu3eLiYi677DJ+/etfc+6559Zb9v3vf5/f/va3bNq0CYCioiLeeOONJuv+jW98gxdeeIG1a9dSVVXFfffdx9lnn03fvn0bLf/GG29w4MABAFJSUjDG4HC0HE+/8Y1v8OSTT7Jnzx5KS0u57777uO6663A6naSlpeFwONi9e3eL2+lIFMpFRESk42lulJUwBfOEhAT+7//+j2uvvZaUlBReeeUVZs+e3eb7qe0+ctNNN9G9e/fA7dZbb8Xj8fDRRx81ut6tt97Kvn37eO+990hOTubdd9/ljDPOID4+nosuuogrrriCe+6557j1vvrqK7Zt28Zdd91VbxQWgCuuuIL/+Z//4frrrycxMZGRI0fy4YcfNln3mTNn8uijj3LVVVfRo0cPdu3axauvvtpk+ZUrV3L22WcTHx/P7Nmzeeqpp+jfv3+Lj9Gtt97KjTfeyNSpU+nXrx/R0dH86U9/AnzdXO6//34mT55McnIyy5Yta3F7HYFp7ieXU0ynORAREZHOaMuWLQwbNqzlgsEOexjm4RGl82vmNXti/ZKaoZZyERER6VhWrgwuaNe2mK9cGZp6ibQjnegpIiIiHUsjXSyaVHvyp8gpTi3lIiIiIiJhplAuIiIiIhJmCuUiIiIiImGmUC4iIiIiEmYK5SIiIiIiYaZQLiIiIqeGRx4Jdw1E2o1CuYiIiJwafvnLcNeg3eTk5DB16lQSEhK4++67w10dCQOFchERETmt1b20vMPhICYmJjD98ssvh6QOc+bMITU1leLiYp544omQ7LM97N27F2NM4PFLT0/njjvuwO12n/A2X3zxRc4999w2rGXHpFAuIiIip7XS0tLArU+fPrz33nuB6W9961uBch6Pp93qkJWVxfDhwzGm9Vdvb896nej2CwsLKS0tZcOGDSxdupSnn346ZPs+VSmUi4iIiDRiwYIF9OrVi9///vd0796dW265hYKCAi699FLS0tJISUnh0ksv5cCBA4F1pk+fzoMPPsjkyZNJSEhg1qxZ5OXlAVBZWckNN9xA165dSU5OZvz48eTk5HDzzTfzj3/8g8cff5z4+Hg+++wzqqqquPPOO8nIyCAjI4M777yTqqqqJuv1yCOPcM0113DDDTeQkJDAGWecwfbt2/ntb39Lt27d6N27N5988kmgnkVFRXznO9+hR48e9OzZkwceeICamhrA1zI9efJk7rrrLrp27cojjzzCzp07mTZtGklJSaSmpnLdddcF9Rh269aNCy64gM2bNwfm/e53v2PAgAEkJCQwfPhw3nrrrcCyhvu+7rrr+P73v8/SpUuJj48nOTn5hJ/Pjk6hXERERKQJhw8fJj8/n6ysLObMmYPX6+WWW24hKyuLffv2ERMTw49+9KN667zyyiu88MILHDlyhOrqav74xz8C8I9//IOioiL279/P0aNHeeaZZ4iJieHFF1/kW9/6Fvfccw+lpaXMnDmTX//61yxbtoy1a9eybt06VqxYwWOPPdZkvQDee+89brzxRgoKCjjrrLO48MIL8Xq9HDx4kIceeojvfe97gfVvvvlmnE4nO3fuZM2aNXzyySf87W9/Cyxfvnw5/fv3Jycnh/vvv58HH3yQWbNmUVBQwIEDB/jxj38c1OOXnZ3Nxx9/zMSJEwPzBgwYwOLFiykqKuLhhx/mhhtu4NChQ43u+1//+hfPPPMM55xzDqWlpRQWFgb/5J1iFMpFRESkY3nkETDm+Bs0Pr8dR2VxOBz88pe/JCoqipiYGLp27cpVV11FbGwsCQkJ3H///SxcuLDeOrfccguDBw8mJiaGa6+9lrVr1wLgcrk4evQoO3fuJCIigrFjx5KYmNjofl9++WUeeughunXrRlpaGg8//DD//Oc/m6wXwJQpU7jwwgtxOp1cc8015Obmcu+99+Jyubj++uvZu3cvhYWF5OTkMHfuXP73f/+XuLg4unXrxl133cWrr74a2H5GRgY//vGPcTqdxMTE4HK5yMrKIjs7m+jo6Bb7eKemppKcnEzPnj2Ji4vj6quvDiy75ppryMjIwOFwcN111zFo0CBWrFjR5L5PFwrlIiIi0rE88ghYe/wNGp/fjqE8LS2N6OjowHR5eTnf+973yMzMJDExkalTp1JYWBjo+gHQvXv3wP3Y2FhKS0sBuPHGG7nwwgu5/vrrycjI4J577mnyBMjs7GwyMzMD05mZmWRnZzdZL4D09PTA/ZiYGFJTU4mIiAhMg6//fFZWFm63mx49epCcnExycjLf+973OHLkSGD93r1719v2448/jrWWCRMmMGLECJ5//vlmH7e8vDwKCwspLy9n8uTJXHjhhYFlL730EmeeeWZg3xs3bgx08Wls36cLhXIRERGRJjQ88fKJJ55g27ZtLF++nOLiYhYtWgSArf3S0AyXy8XDDz/M5s2b+fLLL3n//fd56aWXGi2bkZFBVlZWYHrfvn1kZGQ0Wa/W6N27N1FRUYHgXFhYSHFxMZs2bWpy+927d+e5554jOzubZ599ljvuuIOdO3e2uK+YmBhuvvlmli1bRl5eHllZWdx22238+c9/5ujRoxQWFjJy5Mh6j1/DfZ/MsZ5KFMpFREREglRSUkJMTAzJycnk5+fzy1aMnT5//nw2bNhATU0NiYmJuFwuHI7Go9g3vvENHnvsMXJzc8nLy+NXv/oVN9xwQ5scQ48ePZg1axZ33303xcXFeL1edu3adVw3nLreeOONwAmtKSkpGGOarHtdVVVV/POf/6R79+507dqVsrIyjDGkpaUB8MILL7Bx48Zmt5Gens6BAweorq5uxVGeehTKRURERIJ05513UlFRQWpqKhMnTuSiiy4Ket3Dhw9z9dVXk5iYyLBhw5g2bRo33nhjo2UfeOABxo0bx6hRozjjjDMYM2YMDzzwQFsdBi+99BLV1dUMHz6clJQUrr766nonWza0cuVKzj77bOLj45k9ezZPPfUU/fv3b7J8cnJyYJzypUuX8u6772KMYfjw4dx9992cc845pKens2HDBiZPntxsXc8//3xGjBhB9+7dSU1NPeFj7uhMMD+3nCI6zYGIiIh0Rlu2bGHYsGEnvgFjjvUtFwmBZl6zbd6nRi3lIiIicmp4+OFw10Ck3SiUi4iIyKmhHUdZEQk3hXIRERERkTBTKBcRERERCTOFchERERGRMFMoFxEREREJM4VyEREREZEwUygXERGRjslaWL4crrkG4uLA4fD9vfZaWLFCY5ZLp6JQLiIiIh2P2w3f/Cacfz78979QXu4L4eXl8J//+OZ/85u+cp1ATk4OU6dOJSEhgbvvvjvc1QmLvXv3YozB4/GEuyphoVAuIiIiHYu1cNNN8O67vhDu9dZf7vVCWRm8846v3Em2mMfHxwduDoeDmJiYwPTLL798UtsO1pw5c0hNTaW4uJgnnngiJPtsD7XBuvbx69u3L7/73e/CWpdTJeQ7w10BERERkXpWrID33vMF8uZUVPjKrVwJEyac8O5KS0sD9/v27cvf/vY3Zs6ceVw5j8eD09k+0SkrK4vhw4djTOuv3t6e9TrR7RcWFuJ0Olm1ahXTpk1j7NixXHDBBe1Uw+OdKkG8LrWUi4iISMfyxBO+wB2Migpf+XawYMECevXqxe9//3u6d+/OLbfcQkFBAZdeeilpaWmkpKRw6aWXcuDAgcA606dP58EHH2Ty5MkkJCQwa9Ys8vLyAKisrOSGG26ga9euJCcnM378eHJycrj55pv5xz/+weOPP058fDyfffYZVVVV3HnnnWRkZJCRkcGdd95JVVVVk/V65JFHuOaaa7jhhhtISEjgjDPOYPv27fz2t7+lW7du9O7dm08++SRQz6KiIr7zne/Qo0cPevbsyQMPPEBNTQ0AL774IpMnT+auu+6ia9euPPLII+zcuZNp06aRlJREamoq1113XVCP4bhx4xgxYgRr164FwOv18thjj5GZmUm3bt246aabKCoqqrfO888/T0ZGBj169OCPf/xjYL7X6+V3v/sdAwYMoGvXrlx77bXk5+cDx1rF//73v9OnTx/OP/98pk6dCkBycjLx8fEsXbqUXbt2cf7559O1a1dSU1P51re+RWFhYSteFe1HoVxEREQ6lg8+OL7LSlO8Xl/5dnL48GHy8/PJyspizpw5eL1ebrnlFrKysti3bx8xMTH86Ec/qrfOK6+8wgsvvMCRI0eorq4OBMt//OMfFBUVsX//fo4ePcozzzxDTEwML774It/61re45557KC0tZebMmfz6179m2bJlrF27lnXr1rFixQoee+yxJusF8N5773HjjTdSUFDAWWedxYUXXojX6+XgwYM89NBDfO973wusf/PNN+N0Otm5cydr1qzhk08+4W9/+1tg+fLly+nfvz85OTncf//9PPjgg8yaNYuCggIOHDjAj3/846Aev2XLlrFx40YGDhwI+AL/iy++yPz589m9ezelpaXHPX7z589nx44dfPLJJ/z+97/ns88+A+BPf/oTb7/9NgsXLiQ7O5uUlBR++MMf1lt34cKFbNmyhY8//phFixYBvlb70tJSzjnnHKy1/OIXvyA7O5stW7awf/9+HnnkkaCOpd1ZazvLTURERDqwzZs3B1fQGGt9PcWDuzkcbVbHzMxM++mnn1prrZ0/f751uVy2oqKiyfJr1qyxycnJgelp06bZRx99NDD99NNP2wsvvNBaa+3f//53e84559h169Ydt51vf/vb9v777w9M9+/f337wwQeB6Y8++shmZmY2Wa+HH37Yzpw5MzD97rvv2ri4OOvxeKy11hYXF1vAFhQU2MOHD9vIyEhbXl4eKP/KK6/Y6dOnW2utfeGFF2zv3r3r1e/GG2+0t912m92/f3+Tj4W11u7Zs8cCNikpyUZHR1vA3n333dbr9VprrT3//PPt008/HSi/detW63Q6rdvtDqy7ZcuWwPKf//zn9tZbb7XWWjt06FD72WefBZZlZ2cft+6uXbuOq4vb7W6yvm+99ZY988wzm1zezGu2zbOsWspFRESkY4mJad/yrZCWlkZ0dHRgury8nO9973tkZmaSmJjI1KlTKSwsDHT9AOjevXvgfmxsbKDP+o033siFF17I9ddfT0ZGBvfccw/uJkaPyc7OJjMzMzCdmZlJdnZ2k/UCSE9PD9yPiYkhNTWViIiIwDT4+s9nZWXhdrvp0aMHycnJJCcn873vfY8jR44E1u/du3e9bT/++ONYa5kwYQIjRozg+eefb/Zxy8vLo7S0lCeeeIIFCxYEjrOx4/J4POTk5DS677rHnZWVxRVXXBGo87Bhw4iIiGhy3cbk5ORw/fXX07NnTxITE7nhhhsC3YvCTaFcREREOpZLLvGNSR4Mh8NXvp00PPHyiSeeYNu2bSxfvpzi4uJAFwkbxAgwLpeLhx9+mM2bN/Pll1/y/vvv89JLLzVaNiMjg6ysrMD0vn37yMjIaLJerdG7d2+ioqLIy8ujsLCQwsJCiouL2bRpU5Pb7969O8899xzZ2dk8++yz3HHHHezcubPZ/URERPDTn/6U6Oho/vKXvzR5XE6ns94Xiv3799dbXnvcvXv35sMPPwzUubCwkMrKSnr27NlovRt7jO677z6MMWzYsIHi4mL+9a9/BfXchYJCuYiIiHQsd98dfOt3dLSvfIiUlJQQExNDcnIy+fn5/PKXvwx63fnz57NhwwZqampITEzE5XLhaOLLxze+8Q0ee+wxcnNzycvL41e/+hU33HBDmxxDjx49mDVrFnfffTfFxcV4vV527drFwoULm1znjTfeCJzQmpKSgjGmybo3dO+99/L4449TWVnJN77xDZ588kn27NlDaWkp9913H9ddd1290V0effRRysvL2bRpEy+88ELgpNLvf//73H///YFQn5ubyzvvvNPkftPS0nA4HOzevTswr6SkhPj4eJKSkjh48CB/+MMfgjqGUFAoFxERkY5lwgT4+tdbDuYxMTB7NowfH5p6AXfeeScVFRWkpqYyceJELrrooqDXPXz4MFdffTWJiYkMGzaMadOmceONNzZa9oEHHmDcuHGMGjWKM844gzFjxvDAAw+01WHw0ksvUV1dzfDhw0lJSeHqq6/m0KFDTZZfuXIlZ599NvHx8cyePZunnnqK/v37B7WvSy65hJSUFJ577jluvfVWbrzxRqZOnUq/fv2Ijo7mT3/6U73y06ZNY+DAgcyYMYOf/exnzJo1C4Cf/OQnzJ49m1mzZpGQkMDEiRNZvnx5k/uNjY3l/vvvZ/LkySQnJ7Ns2TIefvhhvvrqK5KSkrjkkku48sorgzqGUDAdpcm+DXSaAxEREemMtmzZwrBhw4Ir7Hb7Lgz03nu+YQ/rjsbicPhayGfPhpdeAperfSosp71mXrMn3n+oCWopFxERkY7H5YJXXoF58+CqqyAuzhfG4+Lg6qthwQL4978VyKXT0BU9RUREpGMyxteV5fXXw10TkXanlnIRERERkTBTKBcRERERCTOFchEREQmZTjTAhHRy3ronF4eAQrmIiIiERHR0NEePHlUwlw7NWkt1dTUHDx4kLi4uZPvVkIgiIiISEm63mwMHDlBZWRnuqog0y+l0kpSURGpqalMXSWrzIREVykVEREREWkfjlIuIiIiIdDYK5SIiIiIiYaZQLiIiIiISZgrlIiIiIiJhFrJQbozpYox5yxhTZozJMsZ8s4lyUcaYZ4wxOcaYfGPMe8aYnqGqp4iIiIhIqIWypfxpoBpIB74F/NUYM6KRcj8BzgFGARlAAfCnUFVSRERERCTUQhLKjTFxwFXAg9baUmvtF8C7wI2NFO8HfGytzbHWVgKvAY2FdxERERGRTiFULeWDAY+1dnudeetoPGz/HZhsjMkwxsTia1X/sLGNGmNuN8asMsasmjNnTptXWkREREQkFJwh2k88UNxgXhGQ0EjZHcB+4CBQA2wAftTYRq21c4DaNK6LB4mIiIjIKSlULeWlQGKDeYlASSNlnwaigK5AHPBfmmgpFxERERHpDEIVyrcDTmPMoDrzRgObGil7JvCitTbfWluF7yTPCcaY1PavpoiIiIhI6IUklFtry/C1eP/KGBNnjJkMXAb8s5HiK4GbjDFJxhgXcAeQba3NC0VdRURERERCLZRDIt4BxABHgH8DP7DWbjLGTDHGlNYp9zOgEl/f8lzgYuCKENZTRERERCSkjLWd5vzITnMgIiIiItKhmbbeYChbykVEREREpBEK5SIiIiIiYaZQLiIiIiISZgrlIiIiIiJhplAuIiIiIhJmCuUiIiIiImGmUC4iIiIiEmYK5SIiIiIiYaZQLiIiIiISZgrlIiIiIiJhplAuIiIiIhJmCuUiIiIiImGmUC4iIq33yCPhroGISKdirLXhrkNb6TQHIiLS4RkDnefzQ0SktUxbb1At5SIiIiIiYaZQLiIiIiISZgrlIiIiIiJhplAuIiIiIhJmCuUiItK0Rx7xndTZ8AaNz9eoLCIiJ0Sjr4iInEa8Xi9VVVVUVFRQWVlJZWUlFRUVlJeXU1FRQVlZGWVlZZSWllJcXExxcTGFublc+8EHjM7KwuXx1GvN8QJup5ON/fsz97rrSOjShYSEBOLj44mPjycuLo7Y2FhiYmKIjY0lOjq63i0iIiJcD4WIyMlo89FXFMpFRDoYay1ut5uqqqrArbq6msrKSqqrqwPTLf2tW76qqorKykrcbndrK8N35s1j9N69RNXUNFmsKiKCdX378vfzzz/Wkh4Ep9NJVFQU0dHRREVFERkZSVRUVOBWO133b1Nl6k5HRkbicOjHYBFpN20eyp1tvUEREfHxer0UFhaSn59PUVERJSUllJWVUV5eHmiZrm2prqioCATnqqoq2rPBxOmMxOWKxOVy4XJG4nJFERkZRWTt36gYoiKjiY6OJWN/FmMO/BNnM4EcIKqmhjEHDlLYezjZPXtTVVVJVVUFVdWVuN1VVLurcFdX4fa4cbur8bircXuq8Xg8eDweysrK2vQYjTGBsF7bKh8TExP4GxsbS2xsLHFxcSQkJJCYmEiXLl3o0qWLWu9FJCwUykVE2sGnn37Km2++ecLrOxwRuJwuIpwuXE4XTv8twunCGeEMTDudLpwRtfed9eZF+Oe5nJE4/QE8IsKJaUVL9rC//QlHdXVwda6u5sx5HxP90P8XVHlrLTU1NXg81Xg8bn9Id/uDuhuPx01Njbv+vMD0sXk1NW7cHjc1Ht9f3zxP4BeC4uLioI8XYMqUKdxwww2tWkdE5GQplIuItIPFixcHVc4YQ2xMPDGx8cTGxBMdHUtMdByRkVH1g3fDmz+It3cXjS7LFuKw3qDKOqyXrssWBr1tY4z/i0TwH0Ver5eaGk+dYN4wqLv9rfFVVFaWU1FRRkVFGeUVpZSXl+L1Nt/iD7BkyRKuvfZaIiMjg66XiMjJUigXEWkHP/7xj/niiy84cuRIoPtKaWnpcX26rbWUlZdQVl5yQvtxOCL8wTYy8Lduy7qvldw3z+WKwuWK9HVRiYwmKjKaqOgYoqNimwz3juqq1tWnqrLR+dZaqqoqqKyqoKqqgurqSqqrq3C7q3G763RrqW01d7sD9+uG75oWutG0VkREBPHx8SQlJZGcnExqaipnn322ArmIhJxCuYhIO0hLS+OKK644bn51dXWgX3ltX/LaUVBqb3VP2Kydrjuv7n2vt4bq6hqqWxmeG3P+9Cvp0b1PvXneyCgimgjajfFGRdebLirO57N5b1JZWX7S9avV2AmgTd3qjvQSFRUVGAkmJiaGuLg4oqKiWtWdR0SkvSiUi4iEUO3IICkpKYF5Xq+X6urqeje3293o39r7tdPl5eWUlpZSUlJCaWlp4P6J2Ldv+3GhPH/iNLou+jSoLixe4+DoxGn15h0+vO+EA3lsbGxgeMXav3FxcURG+k5SDeZv3ZtO4BSRjkyhXESkHeTm5rJp0yYKCwsDY3+Xl5fXaw2vHbLQ4/GErF6+kVdcREZG+7uwRBEVGUNsXALDhow5rvyBa2+my/JFUFnR4rZtZCQHr7m53rxBA0cBkF9wBLe7murqSqqqK6muqgyc2NnUSDO1o9Tk5OS0/kAbERERcdyILLWt57XjqaekpNC/f3/69u3bJvsUEQmWQrmISBsrLS3lscceo7Iy+G4fTqeLiAgnzggnEU6n/75vnu8WUW9Z7c1XJsI37Ty2TmAkloj6J4i2tqtGydBRHJ10Hl2XzGu2G0tNVDRHJ59PydAz6s13OBwMGXxmk+v5RmDx4PZUU1NvhBVPYBSVmhoPHv/fGo/n+Hm1054G0/Xm+/qj13YZaskvfvELBXMRCSmFchGRNhYZGUmfPn3Yvn17k2XGnDmFfv2GB0J1h+3XbAzb7v0tQ373C7p+OR9TWXHcFT2tP5Bvu/e3rbpwkG/zJvCFoT1Za/F6a/B4POTmHWTh4veaLNujRw8SEhLatT4iIg3pip4iIu2krKyMI0eOsGjRIr788st6y7qn92b4sPFERUbjiowi0uW7oI/D0UH7PVtLwtYN1DxxH2fs24fL48HtdLK+Tx8ifvZbShu0kHcUXq8Xt6fad+Eif/eZA9l72Lrtq3rl+vTpw7XXXku3bt1ITEzsuF+SRKSj0BU9RUROFXFxcfTr14+CgoLjQvnhnP0cztl/3DoREU7/1TZ9wxe6nL6TFF3OSH94jzo2pGFUDNFRMcTGJhAdHdu+QdIYSoaN4uWZM49b9K12DuTWWqqqKykvL6Gyspyqqkr/kIqVVLur6gyt6LtKqNt9bNrjcbe8A2DUqFEMGjSoXY9DRKQ5aikXEQmB7Oxs9uzZQ35+PsXFxZSWlgZOAK3t51xRUdHkSY/B6N9vOGePn9muFxR6692/U15nTPXY2ASumP2ddtvfhk3LWb9h6UltIzo6ut4wiLW32rHJe/fuTb9+/dQ6LiKtoZZyEZFTUUZGBhkZGc2WqR0asXaElrrjmNfeLy8vp6ysjNWrVx839OHuPZsZPWoSsTHx7XYcDbvXtHd3m23b1jQ6f/LkySQmJtYbdzwmJqZeAK8dXaW9r3oqItIWFMpFRDoIh8NBVFQU1lo8Hg/GmMCIIUVFRRQWFnLkyBH27dtHWVlZYL2oyGiSk9MYPGh0uwZygLKyoman29q0qbPZtHkVBQVHKK8oDcxftWoVvXv3Jj09PTDmu8vlG12m7tCHCuQicqpQKBcRCaH9+/eTlZVFfn4+RUVFgQv+1O3O4vW2fKGe+Lgk+vcfTp9eA0lM7BLCrheG+r0F23e/aakZTJ86G4CysmIOZO9h1+6NFBTksnPnTnbu3Nl0TY0JjEEeFxcXuAhRYmIiKSkp9OzZk4EDB6rbioh0CArlIiIhsnbtWv7617+2ap3IyCji45KIj08iMSGFpOSupHbpTlxceEYIsQ2u7Nlwuj3FxSUyZNBohgwaTUVFGUfzcygozKW4uIDS0kJKy4rrXT3UWhv4otOU2bNnc8kll4Si+iIizVIoFxEJkdjYWKKjo1t1UaHq6iryq4+QX3Ck3nzfBYNc/gsKuQIXFvJdLMh/YaHaiwZFuIhwOnE5XTidkf4LC0XicrpwuaJwuly4XJFEuqKIiGjbjwWvt4bq2pFR/KOieDzuwMWC3J5qPG7fBYMaXjyoJvDXU2fesYsKtYXk5OQ22Y6IyMnS6CsiIiHk9XrJzc2lqKiIkpISysvLAydwVlVVUVVVRWVlZeB+VVUVbrc7cL+6uhq3O7hh/k5ERISTSFcUUVExviEXo2OJjY3H5YwEYOv2r6iurgqUj4yMYujgMXhq3FRUlFFZWUZlVUVg2MJghyQ8ES6XC5fLRVRUVKAPeWSkbwjJ2pM8o6Ki6p38GRsbG+jCkpaWhsvVvhctEpFOq81/qlQoFxE5xVhrcbvdVFdXN3qrDe+N3W8Y/GtHdqmsrKS8vDyo/uytYYypNzpKbUiOjo4OBOnaW8NgXXurnVc3gLtcLp3EKSLhpFDejE5zICIi4WCtpaqqivLy8sDJp7WjvlRXVze7rtPpJDk5mcTERBISEoiPjycuLk4joIhIZ6VQ3oxOcyAiIiIi0qG1eShX84WIiIiISJgplIuIiIiIhJlCuYiIiIhImCmUi4iIiIiEmUK5iIiIiEiYKZSLiIiIiISZQrmIiIiISJgplIuIiIiIhJlCuYiIiIhImCmUi4iIiIiEmUK5iIiIiEiYKZSLiIiIiISZQrmIiIiISJgplIuIiIiIhJlCuYiIiIhImCmUi4iIiIiEmUK5iIiIiEiYKZSLiIiIiISZQrmIiIiISJgplIuIiIiIhJlCuYiIiIhImCmUi4iIiIiEmUK5iIiIiEiYKZSLiIiIiIRZyEK5MaaLMeYtY0yZMSbLGPPNZsqOMcYsMsaUGmNyjDE/CVU9RURERERCzRnCfT0NVAPpwJnAB8aYddbaTXULGWNSgY+Au4A3gUigVwjrKSIiIiISUiFpKTfGxAFXAQ9aa0uttV8A7wI3NlL8p8DH1tqXrbVV1toSa+2WUNRTRERERNqZtbB8OVxzDcTFgcPh+3vttbBihW/5acjYEBy4MeYsYIm1NrbOvJ8B06y1X29Qdh6wARgPDASWAz+01u5rYTen5zMoIiLSwWRlZXHgwAEAIiIiGDlyJPHx8WGulXQIbjfcdBO8+y5UVoLXe2yZwwExMfD1r8NLL4HLFb56tsy09QZD1X0lHihuMK8ISGikbC9gDHABvnD+OPBvYHLDgsaY24HbAZ599lluv/32NqyyiIiItEZNTQ0ff/wx7777LnUb/bp06cJtt91G//79w1g7CTtrjwXy8vLjl3u9UFYG77zjK/fKK2DaPPt2WOFsKb8bmN5IS/k64Ctr7S3+6a5AHpBsrS1qZjdqKRcREQkDr9fL+vXreeedd8jOzgZgUFw08U4He8urKHDXYIzh7LPP5tJLLyUtLS3MNZawWL4cZszwBe+WxMXBvHkwYUL71+vEnLIt5dsBpzFmkLV2h3/eaGBTI2XXUz9gK2yLiIh0QB6PhxUrVvDpp58GwniyK4LL0lMYFB/jK+O1fJ5XxJcFJSxbtozly5czduxYZs2aRWZmZjirL6H2xBNQURFc2YoKX/nXXmvfOnUgIWkpBzDGvIovYH8X3+grc4FJjYy+cj7wH+A8fKH9cWCctXZKC7tQeBcREQkBj8fD4sWL+eijjygsLAQg0RnBlC4JjEuOx+U4vhHxaLWHhUeLWVtURm0v4mHDhjF79mx1azldxMU13m2lufKlpe1Xn5PT5i3loQzlXYDn8fUVPwrca619xRgzBfjQWhtfp+wPgAeAWOAL4A5r7f4WdqFQLiIi0s5ycnJ45plnAi3j3SKdnNs1kVGJsTj9/X/zqz3860AuudUe0iKd3NArjS6Rvh/nC90elhaUsrKwlGqv76N7+vTpXHvttURERITnoCQ0HI7WjazicEBNTfvV5+ScuqE8BDrNgYiIiHRE1dXVPPzww+Tn59PF5eTCbkkMi4/B0eBkvP/bfYjcag8WX3JJi3Ty//r3qFemosbL4qPFLCkoocbCRRddxBVXXBG6g5HQU0t5s0J2RU8RERE5te3Zs4f8/HySXRH8sF86IxJijwvkQCCQg6/FLLfac1yZmAgHs7olc31GKgCrVq1qx5pLh3DJJb7W72A4HL7ypxGFchEREQlKcnIy4Gvlbq6ZMC3SGVhe21LelEr/ONW125ZO7O67feOQByM62lf+NKJQLiIiIkFJT0+nV69eVHktWRXVTZa7oVdaIJjX9ilvyrZS32gcY8eObevqSkczYQJ8/evYFoK5jYmB2bNh/PgQVaxjCNWQiCIiInIKq6mpYcmSJRw+dAiA+Iim2/W6NNKHvCnx/pM7lyxZwqBBg+jdu/fJV1Y6JmOonDOHXZs3M3DzZlw1NTjqnNvoNQZ3RATbBwxgwDPPEHsaXTgIdKKniIiINKOyspLly5fz2WefceTIEQAmJsdzafeUNtl+iaeGOVk5FLh9o2yceeaZzJgxg0GDBmFOs1DW2VVVVfHnP/+Z7du2MaC4jK9nZTNgzTJc1VW4I6PYOeYc3u/bi10JMfTt25c777yTmGC7u4SeRl9pRqc5EBERkXDLz8/ns88+Y8mSJVRWVgKQ4opgVloyIxNimg3MzQ2J2JjKGi+f5xWxorCUGv+neUZGBjNnzmTixIkaKrETqKio4E9/+hO7du0iIiqWnmdfQmRc4nHl3BWlHFw+F09FCX369OEnP/kJ8fHxjWwx7BTKm9FpDkRERCRcrLV8/PHHvPfee3g8vlFTMmMimZiSwPCEGCKCaL0OZkjExpR4aljhH8O8tMZ3Amh6ejq33norffv2PYmjknAqKyvjqaeeIisrC2d0HBkTvkZkXFKT5d0VJWSv+BB3eQkZGRnceeedJCU1XT5MwhPKjTH9rLV72nrnbUyhXERE5CQtXbqUF198EYCRCTFM7ZpIRnRkq7bx4Nb99T6UDfDo0OD7inusZUNxOQuPFpNX7SE2Jobf/Pa3HbkrgzShvLycJ598kn379uGMSaDnhK/hik1ocT1PZRkHV3yEu6yQHj16cPfdd5OQ0PJ6IRS2ccofM8b8nzHmCv+VOUVERKQTqr1SZ4zDwfQTCORwfLho7VBvTmMYlRjLuV18Iay8ooLi4uJW10PCy+v1MmfOHPbt24crNpFeEy8JKpADOKPj6HX2xUTGJ3Po0CH++te/Bn656ayCHX3lJmAccB7wTWPMFmA+8KW1tqq9KiciIiKhde6557Js2TKKi4v5894cMmOiGJ0Yy4iEGOKcwfXtbnhh9GAvlG6t5VCVmw3F5awtLqPE4+vCMnbsWLp16xb8QUiH8Nlnn7FlyxYiIqPJmHARzui4esvd5cVkr/oUd3kRrtgkMsZdgCv2WD/ziKgYMsZ/jQNL32XXrl3MnTuX2bNnh/owQqbVfcqNMXHAZODrQDqwFPjIWrul7avXKuq+IiIi0gaKi4uZO3cuX375JVVVvrY3A/SNjWJEQgwjEmJJaCygW4tz3wEOvv85g7KyiPR4qHY62Z6ZSa9LZ+Lp0xMa9Em31nKgsppNJRVsKikPjMICvv7kF110ERMnTsQR7JUgpUPweDz8/Oc/p7y8nB5jLyCuW5/jyuxd8DqeipLAtDMmgb7Trz2uXEX+IQ4un0tkZCR/+MMfiI6Obte6B6nNu6+0apxyY0wMMBGYCqQCi4Fc4GfGmJXW2mfauoIiIiISWomJiVx//fVcfvnlfPXVV6xevZotW7awp7yKPeVVfJBTSP/YKCakxDMsPgaHMVBTQ8Ir/yFy01aS3R6Mv9EvyuNh5O7d2L8+T/WIoZR88yqIiKDMU8OKwlK+KiqrF8QTExM566yzmDBhAgMGDNCwiKeorKwsysvLccUlE5vW+PkEdQN5Y9O1Yrr0ICopjaqiXHbt2sWIESPavL4dQVCh3BgzHl/XlbHAZuATYJm1ttq//APgeUChXEREpJOIjo5m0qRJTJo0ifLyctavX89XX33Fpk2b2FVexa7yKrpHubimRxcGvvE2kRu34nC7j9uOsRZT7SZy4xYSXvkPn11yER/nFVPt9QX3pKQkxo4dy5gxYxgwYIBaxTuB2v7fEa7INvliFeGKrLfdzijYlvJvA/OAv1lr8xsutNaWGGOea9OaiYiISIcRGxvLxIkTmThxImVlZSxfvpxPP/2Uw/n5zF+9ieGbGg/kdTncHlwbt7IhcwDV3boxcuRIZsyYwdChQxXEO5mMjAwcDgeVRbm4y4vr9RWv5YpNxF1eXG+6MZ7KciryDwN06iu+BvUfYK39kbX2v40F8jplPmm7aomIiEhHFRcXx/nnn89NN90EwDlr1mLcwbVgGo+HmevXk5KSwne/+12GDx+uQN4JJSQkMH78eLCWnPWLsd7jT/fNGH8hrrhkMAZXXDIZ4y88roy1Xo5s9K0/evRounTpvIMABvVfYIy5zxgzosG8EcaYe9unWiIiItJRlZaW8sorr/DUU08BMHrfvkAf8pY4rGXUvn0UFBTw8MMPs2zZMrxeb3tWV8Lk6quvJjExkcqCw75gbus/z67YRDKnXsXAi24lc+pVx7WUW2vJ3fQl5bkHiIuL4/rrrw9l9UMu2K+mI4GGo6tsBUa1bXVERESko/J6vXz++efcf//9LFy4EKxlcpcEnK3s5xtZU0Ov6EiKiop44YUX+N3vfsfevXvbp9ISNomJifzwhz8kMjKK0kO7yFm/CBvkF7DaQF68fxsul4sf/OAHnbqVHIIP5dVAw/FnooHO29teREREAsrLy/nf//1fXn/9dSorKxkUF82P+qXztW7J4HK1bmMuJ7dnduPKHl1IcEaQlZXF73//e+bNm9cudZfw6du3Lz/5yf/zBfPsXRxeO7/Rrix1Wa+XIxsWUbx/Ky6XizvuuINBgwaFqMbhE2woXwP80BgTC+D/+33gq/aqmIiIiHQM1lrmzJnDtm3biI9w8K2eqXy7dxrpUb4RMaqGD8YGOcKGNYaq4UNwGMOYpDju7N+dSSnxeL1eXnvtNVasWNGehyJhMHDgQO66605iYmIoy9nL4TXzmmwxt9ZLzvpFlBzcSWRkFD/+8Y8ZPnx4iGscHsGG8r8DscArxph/Aa/4pzXiioiISCe3bt06tmzZQmyEg+9lpjMsIabe8orpk7GuIAd0czqpmDY5MBnlcHBxegqXdEsG4M0338Tdwigucurp378/D8fGMuroUcqO7CNnwyIaXsDS12VlKaWHdjEyN5eHY6IZMmRImGocesGOvlJqrf0lcCvwS+AWa+2j1tqydq2diIiIhJXX6+XDDz8EYHrXRFIijw/fnj69qB4xFG8LwdzrclI1cqjvyp4NTEyJp3uUi6KiIr788su2qbx0KCkXXMD3589nZG4updm7KNyzod7y4v3bKN6/leE5OfxgwQJSv/a1MNU0PFo1BpF/SMSdQJExxmGM0RhGIiIindj777/P3r17iY9wMC45rvFCxlDyzauoHjkMb6TruK4s1hisy0X1yGG+K3o20tXFGMP0rr7RN/773/9y8ODBNj8WCbPzziPizTf5wfz5DM7O5uj21VSXFgLgLi8hb+tyBmdnc8fChTj/8x8477zw1jfETMOfDhotZEwX4AfACKDef6S19rL2qVqrBTcWk4iIiLTIWsvcuXN59913McANvVIZEh/T0ko49x3kwPufMTgrC5fHg9vpZHvfTHpdOhNPn14t7vO17KNsLKkgIT6en9x5Z6e+WMxpa/58Ki+7jKenTSN7zGS6n3U+RzYspvuKBdyxYAEx7757KgTyk79MacMNBhnKHwSqgDeB3wL3At8EVllrP27rSp0ghXIREZE28vbbb/Phhx9igCu6d2FMU63kjXhw6/56H8oGeHRocOHa7bW8cjCPHWWVxMbEcOddd5GZmdmqukvHV/Luu3DddcyZOZPqG35G1CtPctsnH+P+17/octVV4a5eMNo8lAfb/WQY8H/W2t2AtdbuAf4PuLytKyQiIiLhtWDBAj788EMcwLUZXVsVyAHSIp2BxGL808FyOQzf6pnK8PgYyisq+NP//R/5+U1eUFxOUQmzZ/P+jTdy+2efMeFvf+C2Tz7mv9ddd6oE8nYRbCj3ArWDSpYZY5KASqBru9RKREREwuLAgQO8/vrrAFzRowtnJMa2ehuXpqcEAobDP90aTofhup5dGRAbRUlpKX//+9911c9OqGTcOBYOH86Mzz5g4fDhFI0ZE+4qhVWwoXwbMM5//yvgHuA+fCd9ioiISCdQXV3NCy+8QE1NDeOT4zgrqXUt5LXezymgNkJ7/dOtFWEM12Z0JT7Cwc6dO/nkk09OqC7ScSWsWsW0zZv5fMbFTNu8maSvTu/L3wQbyv8/YKP//nPAeiAL+GN7VEpERERCq7KykmeeeYYDBw7Q1eXkIv+44Scit9oT6FNu/dMnIs4ZwRU9fJdWf/vtt1myZMkJ10k6lqK33+bSf/6TOTNnsuK2e3jugllc+dprHH3zzXBXLWxaDOX+YQ9vx9ddBWtttbX2NWvti/4hEkVEROQUlp2dze9//3s2bdpEbISDb/VKJcpx4qMen0yf8oaGxMdwQVoS1lpeeuklXnnlFV1c6BRn580j8sYbmTNzJtljp+CMiuHQhOnMmTmTmJtvxs6bF+4qhkWL/3HWWi9wFhrdREREpNNZvnw5v/nNb8jOziYt0sntmd3oFuU6qW3W9im/dNWqE+pT3tC0ronMTk8hwsDChQv53e9+x9GjR09qmxIm8+fjvuIK/jJ9Ojt6Z9J18FgAugw8i52ZffnreedRfcUVMH9+mCsaesF+DX4H+KYx5sS/6oqIiEiHsmHDBp5//nncbjdnJcXy/b7ppEaeXCAHeOdwPjXA17/6ihr/9MmakBLP7ZnpdHU5OXDgAE8++STV1dUnvV0JHTtvHlWXX86fpk5le0YG6WdMwRWbAIAzOpb0UdPYntGTP0+dStXll592LebBhvJLgSuB140xLxhjnq+9tWPdREREpB0tXboUgLMSY7mqR9eT6rJS11F3TbPTJ6pndCTf75tOpMOQm5vLjh072mS70v4qKytZ+Ze/8Gd/IE8bMZn4Hv3qlYnr1of0UVPZ3tMXzJf+6U+Ul5eHqcahF2zL9xPtWgsREREJuZ49e7J69Wo2llTQJa+ISSkJREW0PpjHzFuMp3dP3IP6t1jWtWM3zv0HqTh/Sqv3c7iymk9yi6j2WhwOB+np6a3ehoTenj17+Pvf/05u166YCBfdR00lvnvfRssm9ByIIzKanRHz2e6p5r1HH+XWW29l0KBBoa10GAQVyq21G1suJSIiIqeSCy+8kNzcXJYuXcrnecV8mV/KpC7xTOqS0KpWc0/vniS+9BrFN12He1B/uric5LuPjbjSxeWLG64duwPlWuNwZTXz8orZXFoBQHR0NN/+9rdJTU1t1XYktDweD++//z4ff/wxXq+XyIQUup95PpHxyc2uF5fWi96TLuPwuvnk5+fxxBNPMGPGDC677DIiIyNDU/kwMNa2fP6mMeZbTS2z1r7cpjU6cToRVURE5ARs27aNd955h127dgGQ4Izg2owu9IuNDnobdQN3TmYf/nUgl0f//Bce/NEd3NArjfSsffWCezC81vJZbhGL80uwgNPp5Nxzz+Xiiy8mKSnpRA5VQmT//v28+OKLHDhwAIDkfiPpMmgsjojgT0+0Xi/5O9dQsHsdWEt6ejq33HIL/fr1a3nl9mdaLtLKDQYZyn/SYFYKMBJYZq3tKGOVK5SLiIicIGst27dv57///S979+4lymH4af8exDkjWlw39qN5xH0S/GgZZbPOo/yi81sst7yglPdyCjDGMG3aNIXxU4C1lgULFvDGm29S4/Hgik2g2xlTienS/YS3WVmYS876RbjLCnE4HFx++eVccMEFONroHIgTFJ5Q3uiKxowFplprn2zbKp0whXIREZGTVFFRwc9+9jM8Hg/f7dONvrFRrVrftWM38S+9xksXXMCtb7/D85dfxk2ffkppK1rIa/3n0FHWFJUzYcIEvvOd77RqXQm9mpoaXnnlFb744gsAEnsPJXXoBBzOxkf0cZcXk73qU9zlRbhik8gYdwGu2MRGy3prPBzdvoqivZsAGD9+PDfffDNOZ9gGBmzzUH4yXzHWABPbqiIiIiISXlVVVfz5z3/G4/HQxeWkV3Tr+++6B/XnpQsu4JqPPgbgmo8+5qULLmh1IAcYlRALwIoVK5h/Go5bfap5+eWX+eKLLzCOCNLPPI9uIyc3GcgBXyAvKwRrcZcVkr3q0ybLOiKcpA2bSI8xF2AiXKxcuZLnn3+eE21c7oiCCuXGmO4NbpnADUBu+1ZPREREQuX9999n586dJDojuKl3Kk7HiTUGLu+WzsLhwwFYOHw4y7ud2Cgpg+JjuDQ9GYDXXnuN7OzsE9qOtL+1a9eyZMkSjCOCjAlfI6FHy1/C3OVFzU43Ji69Dz3PvhiH08Xq1atZvnz5Cde5owm2pXwO8Kz/7xzgj8AI4H/bp1oiIiISarUn5WVEu4iPaLkveVOGZWczbfNmAKZt3sywkwjTfWKiiHQYrLUcOnTohLcj7au2y0qXwWOJSQnuS1hEZGyz002JTkql65AJACxZsqQVtezYgh0ScXZ7V0RERETCa/r06WzZsoWtpZX8fmc2IxNiGJEYy4DYKFxBnlTn2rGb73z2GXNmzuTu999nzsyZ3P7ZZ9T0Tgu6C0uR28OW0grWF5ezr8J31c5u3box3N/6Lh1PaWkpAFGJwQ9TWVNd0ex0c6KSUuvttzMIKpQbY/oDxdbavDrzUoEEa+2e9qqciIiIhM7o0aP5n//5H9566y22bdvGmuJy1hSX4zSG/rFRDIyLZmBcNGmRTow5vmtL7bCIL150ITu6pfPemDHsyMjgjYsu5OZmhkN0e73srahmV1klO8oqyalyB5ZFR0czefJkLr30UmJiYtr1+OXE9enThz179lCUtYmYLt0bfX0cx3qbn25qNWsDJ3z26dOntVXtsII9ZfWnwGMN5rn883/cpjUSERGRsOnXrx8//elPycnJYdWqVaxdu5Z9+/axvayS7WWVgG8c80Fx0YFbdISj3jjlkzP7sOdALh+MG0dapJPJE8+kuEeXeuOU51a5fdssrSCrogpPnfP1oqKiGDZsGGPGjGH06NFERwc/XrqEx6xZs1i2bBllOVnkbVlG6rCzMaaFX1eMo34Qb6k8vkCev2M1Jdk7cTqdXHTRRSdZ844j2HHKX7PWHnf5rabmh0nnOf1WRESkAykqKmLz5s1s2bKFLVu2UFxcHFgWYWDa0VyumPsRRTddB4MHNLmd8i3bSf/Xm/zrwlmsbHDyZ+/evRk+fDjDhg1j4MCBuFxNj9ohHdPGjRt5+umn8Xq9xHbrQ/qoqUS4mh5Ss/xoNtkrP/YFc+MgY/yFxHbNaLK811PNkY1fUHpoD8YYbrvtNsaOHdsehxKMsF086C/AE9baXXXmDQB+Zq39QVtX6gQplIuIiLQzay0HDhxg06ZNbNy4kZ07d3LBmjXs7daNfb16Mj45nqldE4mJONbqeaCiis/zitlRVsng7Gz6HjnCksmTGTFiBCNHjmT48OEkJCSE8aikrWzbto2//vWvVFRU4IyOI3309JO6cFCtyqJcctYuwF1eTFRUFLfddhtnnHFGG9T4hIUtlF8IXA/8BzgMdAeuBF631n7U1pU6QQrlIiIiIVZaWspXX33F0qVL2b17NwCxEQ5GJsQQYQyF7hq2lPpO4HO5XIwbN46zzz6bwYMHE3ESI7xIx5WXl8dzzz3H3r17AUPKwDPpMvDMlruzNMJaS+Hu9RzdsRqspVevXtx22210737yQf8khe+KnsaYc4ELgFQgD/jEWtuRxqFRKBcREQmjPXv28Oabb7Jz58568x0OBxdccAGzZs0iPj4+TLWTUPJ4PLz33nt8/PHHWGuJ6dKD9DPPwxkV/Mm6Ne4qctYuoDzPN1Tn+eefz5VXXtlRujaFL5SfAjrNgYiIiJyqvF4vX331FYWFhQAYYxgyZAi9evUKb8UkLLZs2cLzzz9PcXExzph4MsZdSGR8covructLyF71Me6yIuLi4rjlllvC3V2lobB1X7kdWGyt3VJn3jDgXGvtc21dqROkUC4iIiLSwRQVFfGXv/yFvXv3EhEVS6+Jl+KKbfocAk9lGQeWvY+nopRevXpxxx130LVr1xDWOChtHsqD7dwzDdjRYN5O/3wRERERkUYlJSVx9913M3jwYGqqyjm8bj62iTHJrbXkrF+Ep6KUvn378rOf/awjBvJ2EWwot42UddAO3xJEREREpHOJjIzk+9//PklJSVQV5lJ2ZF+j5SqOHqLiaDbx8fHccccdp9UFo4IN5ZuAG4z/tFn/328Cm9urYiIiIiLSecTFxTFjxgwAynIaD+VlR7IAmDZtGklJSSGrW0cQ7BU9nwMeAv5hjDkCdAPygUfbq2IiIiIi0rnUDmVYU1Xe6PKaqop65U4nQYVya22eMeZOYBCQBhQCZwNPAN9ur8qJiIiISOdx+PBhAJzRcY0uj4iOBeDQoUMhq1NH0ZpR3BOAIcA1wK+BAfha0EVEREREmlVTU8OSJb5L3MSm9W60TJx//tKlS3G73SGrW0fQbEu5McYJTABmAGOAQ8AifK3lj1trC9u7gnLiKioq+OKLLygoKAjMi4yMZOLEiaflz0IiIiISPosXLyYnJwdnTAJx3fo0WiamawaR8SkUFBTw+eefc9FFF4W4luHTUveVfwJe4HPgFWvtLgBjzMXtXTE5MdXV1WzdupU1a9awevVqqqqqjivz0UcfMXToUMaNG8cZZ5xx2p1IISIiIqGVn5/PW2+9BUDqkPEYR+OdNYwxpA6dQPaqj3n//fc566yzSE9PD2VVw6alUL4HGI6v20q2MSbHWlva/tWSYFVXV7N37162b9/O9u3b2bVrFx6PJ7A8vkc6Cb16+F78FioKCincvZctW7awZYvvWlA9e/ZkyJAhDBo0iEGDBpGQ0PSA/iIiIiKt4fV6+cc//kFlZSVx3foQ170vAO7yYrJXfYq7vAhXbBIZ4y7AFZtIbFovEjIGUJK9ixdeeIGf//znREREhPcgQqDFK3oaY7oB5wPn4eu2sgYYCdxhrT3a7jUM3mlxRc+amhp27drF1q1b2bZtG3v37q0XwgFiUruQlNmL5H6ZGIdh9ycLqSoqJiopkf6zphERGUnh3n0UZx2gJDsHW1NTb/0ePXowePBghg4dyrBhw06rMUJFRESkbc2fP59XX32VCFc0vadciTPKlyuyFv0Hd1lhoJwrLpnMqVcBUOOuYt/i/1JTVc4VV1zREbuxtPm1eloM5fUKGzMcX0A/F6gBPrPWvtDWlTpBnTqU7927l0WLFrFmzRrKy+sPIxTdJZn47t2I75FOfI90nNFRgWVb3nyfqsKiwHRUchLDrr40MO2tqaEsJ4+ywzmUHjpC2ZG8eiHd4XAwdOhQJk2axJgxY06Lb6qnFWthxQr44x9h7lyoqICYGLjkEvjZz2D8eDC6RpiIiJyYgoICHnr4Yaqrquh+1vnEd+8XWLbzo+d9n0O1jGHgRbcGJstzD5C96mOcTicPPfRQR+vG0uYfjsGOUw6AtXYzsNkYMweYiC+gSztyu9289NJLrFixIjAvKimRhF49SMjoTlz3NJxRUU2uX1VU3Oy0IyKChIx0EjJ8L3RvTQ3luUcpPZRDycFDlOXksXnzZjZv3kxGRgbf//73O9o/hZwotxtuugnefRcqK8Hrv+RxeTn85z++kP71r8NLL4HLFd66iojIKek///kP1VVVxKX3rRfIAVyxSfVbymPrn+MWm9aLhJ6DKDm4gzfeeIMf/ehHoahy2LSqpbyD6zQHUtebb77Jp59+isMZQeqwwXQZMoDo5OBPzGyppbwlnsoqCndncWTDFqpLSunRowcPPfQQjiZO0JBThLXwzW/6Anl54xdwAHyt5pddBq+8ohZzERFplYMHD/KrX/0K44igz9SrccXE11veVJ/yujxVFWQtfANb4+bee++lX7/6wT6M2vxDUcmqg6s9GTPzvMlknD2mVYEcoNekcYEznI3DQa9J41q1vjM6itThgxly5cWYiAgOHTpEYWFhq7YhHdCKFfDee80HcvB1Z3nvPVi5MjT1EhGRTmPevHkAJPYafFwgB3DFJpI59SoGXnQrmVOvOi6QAzijYkjKHAbA559/3r4VDjOF8g4uIyMDgEMr11GSfZjW/rJx4MtVWH+3BOv1cuDLVa2uQ1VJKfsXLcPW1BAXF0di4vH/NHKKeeIJX+AORkWFr7yIiEiQampq+OqrrwBI6jPspLZVu/7atWsbHeq5s2hVn3IJvSuvvJKsrCxycnLYNfdzoruk0HXIAFIG9m22L3mtlvqUN8V6vRTvP8jRbbso3p8N1uJyubj11ltxOvWyOeV98MGxPuQt8Xp95UVERIK0d+9eysvLccUlEZmQ0miZYLqvALhi4olKSqWqKI9du3YxfPjw9q5+WChddXApKSk88MADfPrpp8yfP5+S/AIOLl1F9oo1pA4bRI9xo3E0E5KjkhLr9ylPar6V21pL/o7dHF69HneZr2tDREQE48eP5+tf/zqpqaltc2ASXsG2kp9oeREROa0dPHgQgOjkbk2WyV71aeBET3dZIdmrPg0MidhQdHI3qoryOHDggEK5hE9kZCSXXHIJs2bNYu3atSxZsoQtW7aQu3Er1uul16TxTa7bf9a048Ypb07JwUPsX7QMgNTUVKZOnco555yjLiudjdPpG30lWBoKU0REWqGoyNcg6IyOa7KMu7yo2em6ardTXBzcL/6nopD1KTfGdDHGvGWMKTPGZBljvtlC+UhjzBZjzIFQ1bGjcrvd7Nu3j+XLl7N161by8/MDyzyVzfetikpMYNjVl/LdmiqGXX0pUYnNX62zps72Kioq2Lt3L8uXL2fbtm2UlZWd3IFIx3H55RDsCDoOB1xxRbtWR0RETj8Nh0BsOF2fb7CTTjRq4HFC2VL+NFANpANnAh8YY9ZZazc1Uf7nQC5w2lzz3VrL0aNHOXDgAAcPHuTgwYNkZ2eTk5ODt0H/34ioSFL6Z9J93JnNbrOquITdnyzkB397kX92SaP/rGnNBvPkAX3xVFeTt2kbZUUlfPXVV4ETNcDXnaZnz55kZGTQs2dPevXqRffu3dXP/FRz992+cciD+aIVHe0rLyIiEqSEBF/W8FQ1PcpXxrgLyF71KRcu+pyPp84gY9wFTZb1VJbV225nFJIkZYyJA64CRlprS4EvjDHvAjcC9zZSvh9wA/BT4LlQ1DFcSktLWbVqFZs3b2bXrl2UlpYeX8gYopISiemaQmxqV+LSU4lN6xoY6rA5uz6aR3Wxb5tVhUXs+mgew6+9rMnyxhjShg8hddhgqopKKMvJpTw3j4qjBVTkF1JQUEBBQQEbN24MrONyucjMzGTIkCGMHz+eHj16tP6BkNCaMMF3YaB33mm+v3hMDMye7buyp4iISJB69eoFQGXBkSbL1A6J+PUnH2frA39pdnuVhUfqbbczClXz5mDAY63dXmfeOqCpDs5/Au4Dmj27zBhzO3A7wLPPPsvtt9/eBlUNnQMHDvDHP/6RijqhKCI6ipguKcR0TSGmSzLRKclEJyc2ezJnc2oDeVPTTTHGEJ2cSHRyIl2HDAB8I7JUl5RSUVBEZX4hFfkFVOQXUF1cys6dO9m5cycffPAB119/Peedd94J1VdCxBjflTpvugneew9bUYGp82uMdTggOhoze7avnC4cJCIirdC3b19iYmKoKCukqqSAqEZGYKkdfQUga9F/mhx9xV1eQlVRLi6Xi4EDB7Z73cMlVKE8HmjYM7+IRrqmGGOuACKstW8ZY6Y3t1Fr7RxgTu3kyVcztDZs2BAI5FFJCWSeN5mYrl0wHTQAGYeDqKRE3wgufXsH5rsrKslevpqCnXsBWLlypUL5qcDlwvuvf7H95Zep/u1vGbJzJ1EeD1VOJ+v79GHbpZcy7Wc/o7fLFe6aiojIKcbpdDJ+/HgWLVpEUdYmuo0897gywY6+UrRvMwBnnXUW0dHR7VrvcApVKC8FGn71SQRK6s7wd3N5HLg4RPUKqylTprBhwwZ27dpFVVEJOz/4jP6zphPfI73N9hGZkEB1SUm96bbkqaxi14fzqMwvACAxMZErdFJgh1dSUsLy5ctZtGgROTk5MHUqXa/4Gpet+YplF01n9+48KisrWfzYYwwdOpRzzz2XM888E5cCuoiIBGnGjBksXryY4gM7SBkwGldM/QwSzOgrnqoKivZtDWyvMwtVKN8OOI0xg6y1O/zzRgMNT/IcBPQFFvtbiyOBJGPMYWCitXZvaKrb/rxeL5WVlcyaNYsPP/yQvXv34nV7OLJ+y0mF8nFzXmD8315sdNmzc/w/KtS5OuPK797MqttvOeH9Fe7dFwjkAJdeeikxMTF4PB6d/NnBVFVVsXbtWlasWMHmzZsDJw8nJ8cyY8YQJk0aQETEJYwCSkur+PTTLSxduputW7eydetWoqOjGTNmDBMmTGDIkCE4gh29RURETkvdu3dn/PjxrFixgvwdX5E+qn6vZVdsUqClvHa6oYKda7A1HkaNGkXfvn3bucbhZUI1tIwx5lV8XUy+i2/0lbnApLqjrxhjnEDdq9NMAv4MjAFyrbU1zeyiw3Vf8Xq9FBQUkJubS25uLnl5eRw5coQjR46Qk5ODu8E40RGRkfSdOYWEjO5tVofa0Vf+7/E/8P/u+XmLo6+0lruikt0fzaPiaEG9+cYYUlNTSU9PJy0trd4tNTVVLa4hlJ2dzeeff87KlSsDlyd2OAxDh3bn7LP7MXJkBgUF5Tz33Bfk5paQlpbAbbedS2pqPOXl1axencWKFXvZv//Yc5ySksKUKVOYNm0a8fHx4To0ERHp4HJzc3n44YepqanhW0fKmPr2y0Gvu+iKG3m5WywGeOihh8jIyGi/irZem/c1DmUo7wI8D1wAHAXutda+YoyZAnxorT3uk93fp/xf1tpgTrUNaygvLCxk165dHDhwIDCMYW5uLh6Pp8l1nNHRgRM643p0IyGj+wmf0NmUza+/Q3VxKc/OmcP3br+dyMT4ZkdfORHWWiry8ik5eIjyowVUHM1v9oRSYwwpKSmkp6fTo0cPevbsSb9+/cjIyOiw/elPRdXV1bzxxhssXrw4MK5r375dGTcukzPP7EV8/LF+eb/97UccOVKMtb5zOrt1S+QXv7io3vZycopZvXofq1dncfSob2iq6OhorrzySqZOnarnTkREGvXqq68yf/584rv3o/tZ5x+3/OdXnM0f3lp+3PycDYspObCdc845h5tvvjkENW2VNv/QC1n/AmttPnB5I/MX4zsRtLF1FgAdeuybffv28cILL5Cdnd3ocmdMNFGJCUQmJhCVEE9kYjzRSYlEJibgjIps9/qd6OgrrWGMITatK7FpXQPzvJ4aqktKqSwqprq4hKqSUqqLS6kqLqG6tIz8/Hzy8/PZsmVLYJ34+HiuuOIKzj33+JNBpPWef/551qxZQ0SEg4kT+zN16iDS0xu/Mmtubgm138+t9U03lJ6eyMUXj+Sii0awc+cR5s3bxtath3nllVew1jJ9+vR2PBoRETlVXXjhhSxatIjSw3vxVJY1e5XPWjXuKkqzd2GM4eKLT4tTDUN68aBO6c0336wXyGO6diFlUD/iu3cjKjGBiMjTs5uGwxlBdEoS0SnH9w/z1tTgLi2jPC+fgl1ZFO/zXbS1tLSUl19+mbFjxxITExPqKncq+/btY82aNURHu/jxj8+jZ8/kZsunpSXUaylPS2u6i5PDYRg8OJ3Bg9NZtmw3r766ivfff59zzz1X5xGIiMhxUlJSOOOMM1i7di1lR/aR1GdYi+uUH9mP9dYwZMgQunXrFoJahp8+QU/S9OnT2bZtW2C64mg+FUfziYiKIjopgajkJKKSalvJE067oO6tqaG6pIzqkhKqikqoKi6lqqiYyqJi3KXHX01yypQpCuQnqaioiH//+98AnH123xYDOcBtt517XJ/yYJx9dj8WLtzBoUNFvP7661xzzTU6X0BERI4zZMgQ1q5dS1VRXlDlK4t95YYOHdqe1epQFMpP0pgxY3jyySfZunUr27dvZ//+/Rw4cIDKykrKjlRRduT4F19EVBRRifFEJsTXCeu+v67YmLbtm+twgNfLe2PGHJtuY56qqmNdU0pKqSou9f0tKcFd2vTldR0OBxkZGfTu3Zt+/foxYsQIUlNTmywvzcvLy2PBggUsWrSIqqoqkpNjmDGjfd/MjDFcccWZPPPMIhYuXMiGDRu44IILOOecc/TlSkREAhITfd0na9zVQZX3+svVrnc6UChvA7GxsYwZM4Yx/uBrraWoqIjDhw9z+PDhwOgrR44cIS8vD3dVFeW5VZTnHj1uWyYiwhfUkxKITkzwXawnOZHo5CSc0VGtr5y/o/D748bVm24tr8dDZVExVYXFVBUV+1u9fbeaqqb/wYwxdOnShW7dugVGXunevTvdu3cnNTWViIiIE6qP+Fhr2bJlC59//jmbNm0KnNA5cmQGV189hsTE4ILxc899Eei+cuRIMc8998VxJ3o2ZfDgdH70o/N4/fVVHD6cz2uvvcbbb7/N2WefzYwZM+jeve1GExIRkVNTcbHvGpIRruPPp1ty3XePmxfhiqq33ulAobwdGGNITk4mOTn5uJ9dagN7Xl5evaESa++XlJRQWVhEZWHRcZdAdUZHEd0lhZiuKcSmdiEuPY3I+OZPlohKSqSqsKjedEtqqt2UHcmlPPcoFUcLqDhaQHVJ0yeIRkVFBQJ33WEP09LS6Nq1q4J3OykuLua5555j+/btAEREODjrrD5MnTqIPn26tGpbwZzo2Zz+/VO5555ZbNiQzeLFO9i5M5dFixaxePFipk2bxrXXXqvXgYjIaWzHDt9laqISj/9F/MvrbztuXlRi13rrnQ4UykOsbmAfOHDgccsrKioCreo5OTnk5OQEWtyrKqsozT5MafbhQPmoxASS+/chbeSwRlvS+8+axu5PFlJVVExUUiL9Z007rgyA9XrJ37GH/B27KMvJO65F3eFw0K1bt0Ard7du3QJjkCcmJmo4vDD497//zfbt23G5Ipg1axjnnDOA+PgT+DUF6NIljry80nrTreVwOBg9uhejR/fi8OFiFi3azpdf7mbBggVkZGQwbVrjrz0REencysrK2LBhAwCx3XoHtU5sWi8whq1bt1JYWEhycnI71rBjUCjvYGJiYujTpw99+vSpN7/2QkQHDhxg//797Nmzh127dlFRXELO2k0U7N7H0KsuwdGgNTIqMYFhV1/a4n4PLl1F3hbft1GHw0Fm377079+fPn360Lt3b9LT0zWyRgdTXe3rNuRwGOLjo3C5Tvx8gYa9mk728gXR0c5646DX1lVERE4/X375JW63m5iuGbhijo2C7S4vJnvVp7jLi3DFJpEx7gJcsb5f9CMio4nrlklZzl4WL17M17/+9XBVP2SUsk4RDoeDrl270rVrV0aPHo21lh07dvDss89SWlpKdXEJnorKFruzNKU059gJqddffz2TJk3SKBod3LXXXkteXh6HDx/mtddW89Zb6xg1qicTJvRl4MBuOBzB/3pRUFDW7HQwqqs9rF9/kBUr9rJjR04g2I8ePVqt5CIipymv18uCBQsASM4cXm9Z9qpPcZcVAuAuKyR71adkTr0qsDwpczhlOXtZtGgRX/va1zp942DnPrpOauPGjbz99tvs378fAONw0GPcmY0G8qrikuO6r0QlHj8Gda9J48ia9wXu8gpeeeUV3nvvPS644AJmzpypvsAdVHp6Og899BArV65k0aJF7Nq1i1Wrsli1Kotu3RKYNWs4Y8f2CaprUWvGKW+outrD559vZfHinZSX+1rEnU4no0ePZvr06QwePPiEj1FERE5tW7duJS8vD2dM/HFdV9zlRc1Ox3TpTmR8MsXFhaxfvz4woEZnZezJ/k7dcXSaA2nO5s2beeqppwCIiI6i65CBpA4b1GQL+ZY3369/omdyUpPdWbweD/k795C3eTuV+YUAzJgxg2uvvbZtD0LaxZEjR1i2bBlffvklBQUFAJx33mAuu+zMFtfNyys9bpzy1NRGL7Rbj8dTw1NPzWP/ft/+MjMzmTx5MuPGjSMu7sR+tRERkc7j5ZdfZtGiRaQMOJOug8fWW5a16D+BlnIAV1xyvZZygILdGzi6bQUTJkzgO9/5TiiqHKw2P5mu7Qetlna1b9++wP2Ufn1I6tMTV2zTw95VFRU3O12Xw+kkIaM7XQb1D8zbu3fviVdWQqpbt27Mnj2bX//611x55ZUALFy4A6+3/b6v7tlzNBDIf/rTn3Lfffcxbdo0BXIREQFgz549gP/EzQYyxl2AKy4ZjMEVl0zGuAuOKxOb2rPedjozdV85xUydOpUtW7b4fg7asoO8LTuIiIokIaM7Cb16kNCzR71W85aGRKypdlN66DAlBw9TcuAQVcXHhsJLSUnhmmuuaf+DkjZjrWXjxo0sXLgQgF69UoLqW36i45SnpycQGRlBdXUNH3zwAQkJCWRkZJz0cYiISOdQVOTLIHVP8Kzlik08rmX8+DLx9bbTmSmUn2JiY2O588472bt3LytXrmTdunXk5eVRuGcfhXt8rejRXVJI7teHLoP6NTokotfjoWB3FoW7sig9lIP1egPbj4mJYeTIkYwZM4bRo0erP/kpwuv1smHDBj766CN2794N+AL5LbdMCmr9Ex2nPDExhu9+91xeemkZ27Zt41e/+hXjxo3jwgsvpHfv4Ia9EhGRzqt20AhvjeeE1vd6fOtFRh5/0aHORqH8FGSMoV+/fvTr149rrrmG3NxcNm/ezObNm9m6dSuV+QUczi8g56v1pI4cytArL8Y4fD2VCvfsY8e7H+OprApsa8CAAQwbNozhw4fTt29fBfFTSH5+PsuWLWPJkiXk5flG0ImLi2LWrGGce+5AIiKC66F2Mid6Dh6czr33XsRHH21i6dLdrFy5kpUrVzJgwADOPfdcxowZQ3R0dMsbEhGRTicjI4OjR49SmZ9DZFxSvWXNDYlYq7IgB4AePXqErM7hohM9Oxm3283WrVtZunQpq1evBqD3lIl0HTIAd1k5m159G6wlMzOTKVOmcNZZZxEf3/IJfdJxlJaWsnr1alauXMnOnTup/R9OSYll6tRBTJrUn6io1g1neaInejZUUFDG/PnbWb58D1VVvtYNl8vFqFGjGD9+PCNHjtRQmyIip5HFixfzr3/9i6ikNHqd8/V6I4IFc6LnwRVzqTh6iKuvvpoLLji+z3kYtfmJnmop72RcLhepqamkph67jG3tTz/emprAVWG6du1KamoqMTFNnyQqHYe1li1btjB//nw2btyI19/lyOl0MHJkTyZM6MfQoek4HCd27nZBQTl5eaV4vZa8vFIKCspPKJSnpMRx5ZVnccklI1mzZj8rVuxl9+48Vq9ezerVq4mJiWH8+PGcf/75p0Wrh4jI6W7ChAm8/c47lBblUnpoNwkZAwLLWhoSsezIfiqOHiI6OppJk4LrjnkqU0t5J1BVVcWOHTvYvHkzGzZs4MiRI4FlSX17kzl9Eg7/gPu5G7eSvWJNoB95XFwcI0aMYPjw4QwfPpykpKRG9yHhU1lZyZw5c9i0aRPgu4Ln4MHpjBnTh1GjehIdffItz3ff/SY1NcfOLYiIcPDEE1ef9HYB8vPLWLNmP2vW7OPAgULA123qa1/7Gpdddlmb7EPa0eOPw/jxcN55LZedPx9WroR77mn/eonIKWPRokW8/PLLRERG0/vcK3FG+RoEm2spr6muYv+St/BUlnXEVnJQS7nUKi4uZtWqVaxbt46dO3fi8Rw7gSIiKpKkzN50HToQZ3QU297+qN6Jnkn9+nB0604Kd+2lrLiEFStWsGLFCgB69uzJqFGjGDt2rE7U6yCWLFkSCOSXXDKSiRP7k5DQtn206wbyxqZPRpcuccyYMZQZM4Zy6FARCxduZ9myPcydO5fx48drtJaObvx4uPZaeP315oP5/PnHyomI1HHuueeyevVqtm7dSs66BWSMvxBjHGSMu+C4PuXg+3X4yMbFeCrL6NevHzNmzAjzEYSGQvkpxuv18sYbb7BgwYJAFwaAmNQuJPqHRIxLTwuc2Fn34kFVhUXs/mQhw66+lB5jR9Fj7CgqC4soPnCIkgOHKDucw8GDBzl48CAffvghgwcP5tZbbyUlJSUsxyo+dcf83rPnKD16JDNsWPegT+IMRkSE47iW8rZWXl7Nnj15ZGXlA+BwOHQC6KngvPN8Qbu5YF43kAfToi4ipxWHw8Ett9zCo48+RunRbPJ3rKHr4LFNDolYuGcDZTlZxMTE8N3vfveEu2aeatR95RSzZs0annnmGQDi0tPoOnQQib174Gwi3Kz9+yuBfuQAGMOZ3/lmo2W9NTWUHTpC4Z59HN22E4BJkybx7W9/u20PQlrFWsuHH37I3LlzcbvdAMTFRTJqVC/OPLM3AwemnXSI3rHjCM88s4iaGi8REQ6+//2pDBrU7aTrXlnpZuPGbNau3c+WLYcDwT85OZkbbriBM84446T3ISHSVPBWIBeRIG3ZsoWnnnoKay09xs0iLu34X+Qr8g9zcMVcsJY77riD0aNHh6GmQVH3ldNdYmIiDocDr9dLee5RHC4nNVVVJPTqQVRSYr2zmqHliwfV8lRUUpJ9mOID2RTvOxiY36VLl/Y5EAmaMYaLL76YKVOmsGTJEpYtW8ahQ4dYunQ3S5fuJi4uilGjejJmTB8GDEg9oRaFlJRYUlPjyc0tITU1npSU2BOub1WVh02bslmzZj9bthzC4/EGjmPIkCFMmjSJsWPHahSWU01jLeYK5CLSCsOGDWP27Nm88847HFm/iN7nXoEz6tjnTU11FTnrFoC1XHjhhR05kLcLtZSfgnbt2sXcuXPZtGkTdZ8/V1wsCT17+Lqx9MogItJFVXHJcRcPikpMwHq9lOXkUrw/m5KDh6g4WlBvHz169GDmzJlMnjz5uKAv4WWt5eDBg4ERTXJycgLLkpJiGDcuk3PO6d+q0VN++9uP6o1T3q1bYlBX9Kxbpz178li6dA/r1h2gutp3jkPtOPhjx45lzJgxJCcnB71N6aBqg/gPfgB//asCuYi0itfr5amnnmLr1q3EpWfSY8zMwLKc9QspObiTfv368fOf/7yjXzelzcORQvkprKSkhI0bN7Jp0ya2bt1KScmxqzCaCAeJvXuSOnww8T3SA8G6qriEvM3bKdi5F09lZaC80+lk0KBBDB8+nDPOOEPD1Z0iagP6qlWrWLlyZeACQsbAGWf05JJLziA9vfFfR+r66U/fwOs99i/kcBj+v//vmqDqsHnzIebO3RAYWQWgX79+TJgwQUG8s3roIXj0UXjwQfjVr8JdGxE5xeTn5/PLX/6SyspKeoy7kLi0Xr5uK8s/wOVy8eCDD5Kenh7uarZEobwZneZAToTX6+XgwYNs2rSJ9evXs3v37kAruomIwDgM2GNjlgOkpqYyevRoRo4cycCBA0+LS9h2ZtZadu7cyZIlS1i5ciUej4eICAff+94UBg9u/s3tRFvKP/poEx995BsZJiEhgcmTJzN58mS6dTv5/ujSQamlXETawMcff8x///tfopJS6XXObLJXfEhF/iEuueQSZs+eHe7qBUOhvBmd5kDaQkFBAUuWLGHevHmUlZUF5jscDiZMmMB5551HZmamuqZ0UkVFRTz00ENUVlYyYkQPbrttSrPlT/SKnj//+X9wu2uYMGECN910k/qJd3YN+5CrT7mInKDq6mruu+8+SkpKSBsxidxNXxIdHc3vfve7U+XChgrlzeg0B9KWvF4v1dXVgemIiAgFp07s8OHDfPXVVyxfvpzDhw8DMHv2KM4/f2i77O+vf13Itm2+Pu2DBw/m7LPPZtSoUSQmttxlRk4xGn1FRNrYf/7zHz755JPA9LRp0/jmNxsfIa4DUihvRqc5EJHWyM7OZsWKFaxZsyYQxAESE6OZNWs4kycPaPEXkRNtKa+sdDN37kaWLt2N210D+E7uHDhwIGPGjGHcuHEK6J1BS8FbwVxETkBWVha/+c1vAtM//elPGTJkSBhr1CoK5c3oNAciEozs7GxeffVVtm3bFpgXGxvJiBEZnHlmL4YODf4CQyc7+kpFRTXr1h1g3boDbN9+JDAeucPh4JxzzuGaa645VX6OlIaCDdwK5iJyAlavXk1ubi4pKSlMmDDhVOpWq3HKRQTcbjdPPvkkxcXFuFwRjBuXyVln9WbAgBO7kFBtIAfftaaOHClu1foxMZFMnNifiRP7U1FRzebNh/nqq31s2pTNkiVLqKys5Pbbb291vaQDWLkyuKBdO475ypUK5SIStLFjx4a7Ch2GQrnIKcjhcFBTUxOYdjodeDxePJ6aEwrlvu15602fqOrqGqy1REYeG1+2bl3lFHPPPcGXPe88BXIRkROkUC5yCoqIiOCOO+7g5ZdfJjs7m8WLd7J48U4cDkPPnsn07duVvn27kpnZla5d41r8ObDuGOWNTTelpsbLoUNFZGUdZe/eo+zZc5S8vNJ6Zc444wy+8Y1vtO4ARURETjPqUy5yCrPWkpWVxZo1a9i6dStZWVk0/J+Oj4+ib9+u9OuXysCBafTqlXJca3qwfcqrqtzs3n2UXbuOsGfPUfbvz6e6un4reGRkJIMGDWLYsGGcddZZpKamtv2Bi4iIhJdO9GxGpzkQkRNVWVnJ3r172bVrF3v27GHv3r31rvQKEB3tYtiw7px1Vh9GjOhBRISj2dFXKiqqWb16H+vXH2DnztzjWtHT0tLo168f/fr1Y+DAgfTs2bOjXxpZRETkZCmUN6PTHIhIW7HWkpeXx65du9i5cyfbtm3jyJEjgeWpqfHcdNNE+vTp0uj6y5bt5q231lJV5bsSrDGGvn37MmjQIAYOHEj//v1JSEgIybGIiIh0IArlzeg0B3JSHn8cxo+H6dNhxQr44x9h7lyoqICYGLjkEvjZz3xlFizwjZTQmhO55JSXm5vLmjVrWLRoEbm5uXTpEsdDD13SSLkSfv3rDwHfhYEmT57MGWecQVxcXKirLCIi0tFoSERpwfjxcM01MHo0LFsGlZXg9Y+qUV4O//mPL6SffTasWwdvvBHe+krIpaamMmDAANatW0dubi4eT+Mjo3g8x0Zj6dGjBwMHDlQgFxERaSdqKe9srIWZM2HevJbLnn8+fPaZ78w+6fQqKytZsWIFixcvZt++fQDExLi4+eZJDBmS3ug6n366mblzN2GtxRjDGWecwdSpUxkxYsRJDZsoIiJyilP3lWZ0mgM5KcuXw4wZUFbWctm4OF94nzCh/eslYVNdXc1HH33EvHnzqKioACAuLpJzzunPeecNIS4uqtkTPQ8dKuLzz7eyZs3+wFjm3bp145JLLuHss88+la6+JiIi0lYUypvRaQ7kpFx7ra+LitfbclmHA66+Gl57rf3rJWHhdrt5/PHHAy3j/fp1ZfLkgYwe3QuX69gIKcEMiVhSUsmKFXv58stdHD3q+9J3/vnnc91114XugERERDqGNg/l+v25s/ngg+ACOfjKffBB+9ZHwiorKysQyG+5ZRI/+ckMxo3LrBfIwXdSZ+33c2t90w0lJEQzY8ZQ7rvva5x77gAAFi1adNy46CIiItJ6OtGzs/F3T2i38nJK6dGjB4mJiRQXF/PyyyvYtu0w55zTn169Uup1O0lLS6jXUp6WdvwwhzU1XrZsOczSpbvYtOkQAEOHDlX3FRERkTag7iudTVycb5SV1pQvLW25nJyycnNzef3111m/fn1gXvfuiYwf35cJE/qSkBDdbJ/y7OxCli/fy+rVWZSWVgHgdDqZNm0al19+OZGRkWE5LhERkTBSn/JmdJoDOSnqUy5NyM7OZvHixaxYsYJS/xcxh8Nw1lm9mTlzGD16JAXKWmvZvPkQn3++ld278wLzu3fvzqRJk5g0aZIuGiQiIqczhfJmdJoDOSmtGX0lNhbmz9foK6cZj8fDxo0b+fLLL9mwYQNerxdjDOPGZdKzZzLWWjZuzGbXrlwAoqOjmTBhApMmTaJv377qriIiIqJQ3qxOcyAnReOUSyvk5+fz8ccfs2jRIrwNfl2Ji4vja1/7GlOmTCE6OjpMNRQREemQFMqb0WkO5KTMn3/sip7Ll/tO5KwbthwOiI6GiROPXdHzvPPCV1/pEPbt28eKFSuoqfFd3TMmJobp06eTmJgY5pqJiIh0SArlzeg0B3JSHn8cxo+H6dNh5Ur44x9h7lxfOI+JgUsugZ/9zFdm/nxfmXvuCXetRURERE4lCuXN6DQHIiIiIiIdmi4eJCIiIiLS2SiUi4iIiIiEmUK5iIiIiEiYKZSLiIiIiISZQrmIiIiISJgplIuIiIiIhJlCuYiIiIhImCmUi4iIiIiEmUK5iIiIiEiYKZSLiIiIiISZQrmIiIiISJgplIuIiIiIhJlCuYiIiIhImCmUi4iIiIiEmUK5iIiIiEiYKZSLiIiIiISZQrmIiIiISJgplIuIiIiIhFnIQrkxposx5i1jTJkxJssY880myv3cGLPRGFNijNljjPl5qOooIiIiIhIOzhDu62mgGkgHzgQ+MMass9ZualDOADcB64EBwCfGmP3W2ldDWFcRERERkZAx1tr234kxcUABMNJau90/75/AQWvtvS2s+3/+ev64hd20/4GIiIiIiPgakdtUqLqvDAY8tYHcbx0wormVjDEGmAI0bE2vXX67MWaVMWbVnDlz2qyyIiIiIiKhFKruK/FAcYN5RUBCC+s9gu+LwwuNLbTWzgFq07haykVERETklBSqUF4KJDaYlwiUNLWCMeZH+PqWT7HWVrVj3UREREREwipU3Ve2A05jzKA680bTdLeUW4F7gRnW2gMhqJ+IiIiISNiE5ERPAGPMq/i6mHwX3+grc4FJDUdfMcZ8C3gCOM9au6UVu1D3FREREREJhVP2RE+AO4AY4Ajwb+AH1tpNxpgpxpjSOuUeA7oCK40xpf7bMyGsp4iIiIhISIWspTwEOs2BiIiIiEiHdkq3lIuIiIiISCMUykVEREREwkyhXEREREQkzBTKRURERETCTKFcRERERCTMFMpFRERERMJMoVxEREREJMwUykVEREREwkyhXEREREQkzBTKRURERETCTKFcRERERCTMFMpFRERERMJMoVxEREREJMwUykVEREREwkyhXEREREQkzBTKRURERETCTKFcRERERCTMFMpFRERERMJMoVxEREREJMwUykVEREREwkyhXEREREQkzBTKRU5Hjz8O8+cHV3b+fF95ERERaTcK5SKno/Hj4dprWw7m8+f7yo0fH5p6iYiInKYUykVOR+edB6+/3nwwrw3kr7/uKy8iIiLtRqFc5HTVXDBXIBcREQkphXKR01ljwVyBXEREJOSMtTbcdWgrneZAREKuNoj/4Afw178qkIuIiDTPtPUG1VIuIr4A/oMfwKOP+v4qkIuIiISUQrmI+FrK//pXePBB399gh0sUERGRNqFQLnK6q9uH/Fe/anlUFhEREWlzCuUip7PGTuoMZrhEERERaVMK5SKnq+ZGWVEwFxERCSmFcpHTUTDDHiqYi4iIhIxCucjpaOXK4IY9rA3mK1eGpl4iIiKnKY1TLiIiIiLSOhqnXERERESks1EoFxEREREJM4VyEREREZEwUygXEREREQkzhXIRERERkTBTKBcRERERCTOFchERERGRMFMoFxEREREJM4VyEREREZEwUygXEREREQkzhXIRERERkTBTKBcRERERCTOFchERERGRMFMoFxEREREJM4VyEREREZEwUygXEREREQkzhXIRERERkTBTKBcRERERCTOFchERERGRMFMoFxEREREJM4VyEREREZEwUygXEREREQkzhXIRERERkTBTKBcRERERCTOFchERERGRMFMoFxEREREJM4VyEREREZEwUygXEREREQkzhXIRERERkTBTKBcRERERCTOFchERERGRMAtZKDfGdDHGvGWMKTPGZBljvtlEOWOM+b0x5qj/9ntjjAlVPUVEREREQs0Zwn09DVQD6cCZwAfGmHXW2k0Nyt0OXA6MBizwKbAHeCZkNRURERERCSFjrW3/nRgTBxQAI6212/3z/gkctNbe26Dsl8CL1to5/unvALdZaye2sJv2PxAREREREWjzXhyhaikfDHhqA7nfOmBaI2VH+JfVLTeisY0aY27H17IOsAmoPPmqdlqpQF64KyEdnl4nEiy9ViQYep1IME7F18lH1tqL2nKDoQrl8UBxg3lFQEITZYsalIs3xhjboFnf35o+py0r2lkZY1ZZa8eFux7Ssel1IsHSa0WCodeJBEOvE59QnehZCiQ2mJcIlARRNhEobRjIRUREREQ6i1CF8u2A0xgzqM680fi6nDS0yb+spXIiIiIiIp1CSEK5tbYM+C/wK2NMnDFmMnAZ8M9Gir8E/NQY09MYkwHcDbwYinp2curmI8HQ60SCpdeKBEOvEwmGXieEaPQV8I1TDjwPXAAcBe611r5ijJkCfGitjfeXM8Dvge/6V/0b8D/qviIiIiIinVXIQrmIiIiIiDQuZFf0FBERERGRximUi3RgxhhrjIlvw+1NN8asamLZXGPMgJPc/mfGmLX+20Z//Ue1tExEROR0p1AeJGPMAmPMpeGux6nEHwBntcF2Zhtj/tAWdZKmWWsvttbuOsltzLTWnmmtPRN4ANhkrV3f0rLOyBjziDEmMkT7utMY0y0U+2qw373GmJGh3m8j9ejrv5hc3Xkn/SWzrek10a77rffYGmN+ZYy5LoT7b7fH2xhzuTFmizFmjTGmwhgT0x77aaEOa1var///8IQvAOR/Dv94ouufjPbY94nkRoVyCZoxJqKVq0wHTjqUW2vftdb+/GS3cwr7uf8NcZsx5qramcaYl40xq4wxG4wxbxljUvzzp/vLP2uMWW+MWWeMGdZwo8aYZGPMPGPMXf7pwIep/83kD8aYL4wxu40xv6uz3nBjzHJ/a/e/jDHLmnjjuRXfyd2NaW5ZZ/EwEJIABtwJhDyAdSB9OXZ1Z6BtvmS2A70m2k+9x9Za+5C19rUQ7v9O2u/x/h7wkLX2LGttjLW2op320yR/g0rI99sYY4zDPyhI52OtPe1uwM+Bp+tMpwM5wCXAUmANsAG4vk6ZBcAfgC+A3cDv6iwbCHwOrAe+Ai6qs+wi//bW+8sMbKFudwMr/essBc6ss8wC9/mX7wau8s+PBd4ANgPrgNf98/8NXOO/fw++q6NG+Kc3A4P9978NLAdWA/OAIf75NwOfAW8BG4Ez8b3xbQXW+uuY3MRxnAEcBo74y97rn3+T/7Fd799utzr7+hR411+3eUDPOsverLPtW/3Huc7/WKTjezP8zL/tDcCT4X6dtdFr1eJ7MwYYgm/kotrHLLVOucdqX5P4vgy5gbP80/cDL9dZtgrI9D/fV9fZxl5gZJ3X+2v4vrgn4bv88SD/stXADf7744Aa4NIG9e4OlANpjRxTk8s6yw142v/crfe//nv5H8O6/39P++9PAL7030/EN+LUCv+6T9VZpwfwpn/ZBuC+Os9vdZ3/y+HN1OtsYL7/OVwNXOKf/zfgJ3XKjcT3HmOAb+J7f1jjv81o7DXTwuPR3HEtAJ4AFgP7gZ8B3wC+9G//mjrbafT9FN+1LMr9x/9mI6/n5t6jG31f1Wui3V8TtwNb/PVbDwz1zx8CfOh/PtYBt7T0XDXy2CbjG0r5R/7ljwCvAnOBnfje287C9zmzC/hDnX00+pjWObZf4fts3ltn+0E/3ifwunkS30UVd/ufJwvEN1cf/7I/1nkMPwcy/fP74nvd/dr/3G0Dzq2z3qX4PiPW+ZePqvPYxwez7RaO5xHgdf9jvxX4D5BUZ9m//c/TVuADILbOsjeAT/zLUpqpR5N5APgf/3P7FfAe0D2Ies3gxLLhAvyfjcD1/rr2avbxaY83n45+A7rgC4y1L7AH/S/8FI692aUDB4CUOg9uUyFlOfAd//3h/mVp/hdGLv5/UOA7wPIW6pZW5/5MYFmdacuxN4HJwEH//SuAj+uUq63zd4Fn/Pc/xvchNxHfm84+//wp/hd+lH/6a8AS//2b8b0ZDKjzuBUCMf7pBMDZwj/fH+tMjwSygR7+6UeB1+rsq4JjXwge5tiH68117k/H96Za+48UD0QDdwHPNnwMTvWb/znvWWf6U2C2//5P8X2IbsD3ZvBRncdoY511ZtS+jvzLDvrLn9tgX3upH8rrBvbF+AJRIlAFOOosW8Pxofwe4L9NHFOTyzrTjTofYv7pL+r8/60C1vnn/wJ41H//b8CN/vsOfB9Qt9V57qf670f6n5MLGj53zdQn2f9c1f7/9cD3HpcMnAt8VafsExz7MtiVYyN1DQEONPGa+T7wqyb23dxxLeDYe2sGvnD9a/+yCbX7o5n3U//relUzr+dG36PrPE/Hva/qNdHur4miOvuNwte45MT3nlYb0BPwhcahdR6/Rp+rRh7bF6kfynfg++yOwBfiPvbvNw5f49GgIB/TP/rv98X3+Rjf8Ljb4XWzgGPhzjbYZ1P1qdto813g1TrlbJ3tfYtjn/mD8WWj2sciCkhoZL/NbTuYUH4ISPdPP1/nGGqfp2R8X/4+4dhr/RFgX4N9N1WPRvMAcAO+8dAd/ukfcKzBqrl6nWg2XIDvS849+L4kJLX0XDs5DVlr840x7wI3GmOeA27DF1zSgOf9Vx714AuhQ4Bl/lXfsNZ6gSJjzBZggDHmML4W5Bf8295sjFmL740WfG+ym/33XwD+YoxJsNaWNFG9scaY+/z79uL7J6nrVf/fZUCGMSYa3xvMMGPM0/heBB/4y3wO3GuMicLXKvMHfEE/C983boCv47tq6nL/r0EG3wuw1hf22E/ARfgC8UvGmE+A95s5jsacB8y11h7yTz/rr3vdfW3z3/8bvrDZ0CXAS9bawwDW2lIAY8wy4C5/3/OF+N5wOy3/+P4/ACZZa3ONMd+k/s/3lXXu10C9//UCfC2SF+MLBU1pbhu2hSregu8XqdYu68w+59j/33vAecaYXv55j/nLzAYmGGPu9k/HAgeMMXH4gmdanV9tE4Bh+EJEMCYB/YAP62zD4mtt/sIYk2CMOQNf6+U3gHP8ZQYA/zbG9MT3C0x3Y0z32v/BwIasfaaZfTd6XHWW1763ZhtjjuL7FQ18Aa2n/33ubJp4P23uoP3Lz6Tx9+j3/MWOe1+11lbS/k7n18Q84B/GmPeAD6y1u40xw/31f7VOfaL887b6p0/0ufrYWlsEYIxZj++1VAVUGWO24fs8z6blx/RV/7HtNcYU4Pts3Ur4NFWfrxljfoiv4aph1iu11r7vv78M3xcu8F1HZq61dod/m1X4GmEaam7bwXjfWpvjv/934E91ln1srS0EMMYsx/daqzXXWlu3z3pT9WgqD8zG9+vuV/7n14kv17RUr1ZnQ3xfLuDYl4mLrbXVzT0otRU6Xf0JeBnfN+Qt1todxpjP8XWfuNJaa40x2/G1wtZqLqScNP9JKm/i+5b+lfFd0fRgg2KVANbamtoXlf/NbAS+LxZfA35jjDnDWrvHGOPA97PJUnwfAC/h+wD4vHa3wPPW2oeaqFZp7R3/Pifia6E4H1htjLnIdoCT9ay1S40xZ+F7U7kRuBdfS09ncAvwmP8N4Sx8bwRn43szOer/0nVrK7ZXie+Kuq8bY54C7rT+r/UtsdYWG2M24fuAfsUYMwZfV6UAY8wkfC0GHzZcv7llp4F5+N6gs/B96fTia0U5C9+vWOD7f7zcWru77or+YGmB8dZa9wnu3wDrrbVTm1j+D3y/Si3A956Y5Z//b+Bua+3b/veTcuq/Lwa77+OOq46G763Hvc+1cn+tFer91TqdXxNXAuPxfZbMN8Z8H194ybO+k8GbcqLPVaOvsTrTTnytnS09pu2aA07AcfUxxmTi+/V/vD8HTAJeqVOuquE6we4siG2frIbHU/fk0kAeaa4ezeQBAzxmrW3t+Ux/5cSz4TJ/PTI5FtSbdNqe6Gmt3YCvf+7/4uuPBr6fTPb6H/QL8PVDbGk7Jfj6kH0bwH9C3Wh8T8QyYLQxZqi/+LeBNc20LkfjezL3+6fvCOZY/C0rNdbat/H9bJOG75sc+N70fwl8Zq3dj+9nx1kcC+XvATf5t4ExJsIYM7aJ/STg+8l3obX2YXz9zJs7y74YXwCrNR+42BjT3T99G/VbdCb7gyf4gui8Rrb5gb++6f46xRtjoo0x/YBia+2r+Lp1jPV/WHQGTmPMGuB94HvW2iPAR/j6Qm7H1xLwVWs26P/GfjW+n+LmtPKxugm40xizAV//3w3Ub224Bd+vGTWNrNvcss6mhPqv/6X43hsm4etO8Rm+D4vV/hYp8L3x32v8J1UbY1KNMf387xmL/eXxL+td53+p4f9aY74EBhljzquzjfHmWJPgS/i+bH0Xf6uyXzKwx3//Vnwtl63V6HG1chvNvZ82efwtvEeHml4Tvn04gf+/vfsJ1aIK4zj+/SnoTg1sU5sIhAhCWrW9qxaVLUIElSACSVxEVIIUhbXISyaJCyEQFIQgKCsXIlZm/zZJlCmRBBVFYRjUygivPS2e8+rccWbueIt3vL6/z+r9c2bmvMPMmWfOeea8t0fE5xExTaYp3E2mqlyQ9HCl7B2SlvVYbX3fXrMe+7RLn/09LsvIHPdzpW3f3HO5Y+Q1ehWApKUNI1HzXXfV/ZJuLq/brvVzaa1HRzxwGNiiK5MiLJW0uke9VnCNsWHFUXJU+0jpPO009B3e0PYBL5HBDuSJuFfSC2RCft8e4I3Aa8pZLGbI/L/zAKVxeb00QufJnKZGpRfyeeCkcgj3zZ7bvwuYLu3oYmBHRPxavvuAbDRHB9en5EM5v5RtfizpWeBwafSXkA9TfNGwneXAW8ppkRaRgeChjnq9TQbQX5G5XtOStgHvSQoyr/mxSvnPgFdKg3COvMOdJSJOSNoBvC/pH/KOfw055PikpEulbpvLcNKCFhGji+P22ucXgcbpviLiBDlEd9X72usZchRl5LbKMlO1dVbf/wjcUxqoO8letDOVsps6fk/rdzegXcBxSX8BUxHxp6STwExEXCyvb2L2BekJ4GXgVDlH/i6f/UC2M6+WmyHIIORR8lzZA+yXdAHYUEnxuCwi/pD0ILBT0m7yXP+ePH8iIn6S9A15Lq2v1ekd5dD4UbIz4yqlp/OWllG3rt/VS0nTamtPvwbOSjoDfBsRa2uLt7bRY+ZjIi0GDkhaQY4O/ExOBjAjaQ2wW9LWUu43YF3T+mtm7dse5dt07dMuc+7vcYmI05JGkz/8Tj442TYaUl3uO0mbgDdKPHCJvJk9XSkzr3XXfEKmKN1a1vPUHOWb6tpVjyma44GDklYCH5V4aRGwlytptG31mm9sOKrrcUmPkHHW2oj4sq3s6EGNiSRpH3A2IjwH9sDKAftAw8XUrjPKued3kkOBAM9FxLsDVsnMzBYASdvJB0afHrou16OJ7ClX5mp/SN75Pj5wdcwWlIg4Rg5zmpmZ2f9konvKhyLpPjJtpu6ZiDgy7vr8F8p/MGsK0A5FxIvjro/ZpCspcA81fHVveR7BJoyPCRsnxwXz56DczMzMzGxgN8rsFGZmZmZmC5aDcjMzMzOzgTkoNzMzMzMbmINyMzMzM7OB/QuuYOVFh/BKXgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x504 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import csv\n",
    "\n",
    "df = pd.DataFrame(results_fewshot)\n",
    "#sns.set(style=\"whitegrid\")\n",
    "\n",
    "f, ax = plt.subplots(figsize=(12, 7))\n",
    "\n",
    "# Show each distribution with both violins and points\n",
    "sns.violinplot(data=df, palette=\"Set3\", inner=\"points\",bw =.2, cut=2,linewidth=2, orient=\"v\")\n",
    "\n",
    "sns.despine(left=True)\n",
    "\n",
    "#f.suptitle(\"TARS Few-Shot Learning\", fontsize=18, fontweight='bold')\n",
    "ax.set_ylabel(\"Accuracy\", size = 12, alpha=0.7)\n",
    "ax.set_ylim([0, 1])\n",
    "#ax.set_ylabel(\"Model\",size = 14, alpha=0.7)\n",
    "for tick in ax.yaxis.get_major_ticks():\n",
    "    tick.label.set_fontsize(12) \n",
    "for tick in ax.xaxis.get_major_ticks():\n",
    "    tick.label.set_fontsize(11) \n",
    "\n",
    "    \n",
    "for i, result in enumerate(results):\n",
    "    ax.plot(i, results[data[i]['name']]['Flair TARS'], 'rx', markersize=12)\n",
    "    ax.plot(i, results[data[i]['name']]['Transformers Bart'], 'r+', markersize=12)\n",
    "    ax.plot(i, results[data[i]['name']]['Transformers Roberta'], 'ro', markersize=12)\n",
    "    \n",
    "ax.legend(['Flair TARS zero-shot', 'Transformers Bart', 'Transformers Roberta'], fontsize=12)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "complicated-optics",
   "metadata": {},
   "source": [
    "## Conclusions\n",
    "\n",
    "Without a doubt, zero-shot learning is an extraordinary application of transfer learning. Zero-shot classifiers predict the class of a text without having seen a single labeled example, and in some cases do so with a higher accuracy than supervised models that have been trained on hundreds of labeled training items. Their success is far from guaranteed &mdash; it depends on the particular task and a careful selection of class names &mdash; but in the right circumstances, these models can get you a long way towards accurate text classification.\n",
    "\n",
    "At the same time, the open-source zero-shot classifiers we tested out are no magic solutions, as it’s unlikely they are going to give you optimal performance on a specialized NLP task. For such applications, manually labeling a large number of examples, for example with a tool like <a href='https://www.tagalog.ai/'>Tagalog</a> still gives you the best chance of success. Even in those cases, however, zero-shot classification can prove useful, for example as a way to speed up manual labeling by suggesting potentially relevant labels to the annotators. It’s clear zero-shot and few-shot classification is here to stay, and can be a useful tool in any NLPer’s toolkit."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}