Housing Test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test algoritama na Housing setu podataka"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### California Housing test podataka\n",
    "Set podataka **california houses prices** sastoji se od $8$ značajki:\n",
    "\n",
    "1. medianIncome *(\"MedInc\")* - Prosječna zarada domaćinstava koja se nalaze u okolici (mjereno u 10k),\n",
    "\n",
    "2. housingMedianAge *(\"HouseAge)* - Prosječna starost kuća koje se nalaze u okolici (niži broj novija zgrada),\n",
    "\n",
    "3. averageRooms *(\"AveRooms\")* - Prosječan broj soba u kućama koje se nalaze u okolici,\n",
    "\n",
    "4. averageBedrooms *(\"AveBedrms\")* - Prosječan broj spavaćih soba u kućama koje se nalaze u okolici,\n",
    "\n",
    "5. population *(\"Population\")* - Prosječan broj ljudi koji žive u okolici,\n",
    "\n",
    "6. households *(\"AveOccup\")* - Prosječan broj ljudi po domaćinstvu,\n",
    "\n",
    "7. latitude *(\"Latitude\")* - Mjera koja kaže koliko je sjeverno kuća,\n",
    "\n",
    "8. longitude *(\"Longitude\")* - Mjera koja kaže kolo je zapadno kuća.\n",
    "\n",
    "Dakle temeljem ovih 8 značajki i cijene koliko pojedina kuća vrijedi, potrebno je kreirati takav model koji će predvidjeti cijenu neke nove kuće ako su za nju poznate prethodno navedene značajke. \n",
    "\n",
    "Više o podacima može se pronaći na poveznicama: [docs](https://www.kaggle.com/camnugent/california-housing-prices), [sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_california_housing.html).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Klasa za testiranje algoritama: *model*\n",
    "\n",
    "U svrhu lakšeg kreiranja neuronskih mreža i testiranja algoritama stvorena je klasa **model** koja služi kao svojevrsni API nad TensorFlow knjižicom. Cijela neuronska mreža zadaje se pomoću $9$ metoda:\n",
    "\n",
    "1. *setLayersNeurons* - prima listu koja ima definiran broj neurona po sloju počevši s prvim skrivenim slojem. Primjerice neuronska mreža koja ima topologiju 500,200,200 i izlaz 1 zadala bi se pomoću *setLayersNeurons([500,200,200,1])*.<br><br>\n",
    "\n",
    "2. *setActivations* - prima listu koja ima definirane aktivacijske funkcije. Primjerice ako želimo da prva dva sloja imaju Relu kao aktivacijsku funkciju, a zadnja dva da budu bez aktivacije, poziv metode je sljedeći: *setActivations(['relu','relu',None,None])* Implementirane aktivacijske funkcije jesu:\n",
    "    \n",
    "    * 'relu' - [Rectified Linear Unit](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)).\n",
    "    * 'elu'  - [Exponential linear unit](https://sefiks.com/2018/01/02/elu-as-a-neural-networks-activation-function/).\n",
    "    * 'sig' - [Sigmoid](https://en.wikipedia.org/wiki/Sigmoid_function).\n",
    "    * 'tanh' - [Hyperbolic Tangent Activation Function](https://theclevermachine.wordpress.com/tag/tanh-function/).\n",
    "    * None - Kada ne želimo imati aktivacijsku funkciju nad slojem.\n",
    "<br><br>   \n",
    "3. *setRugluarizations* - Metoda koja dodaje svakom sloju regularizaciju. Implementirane su dvije vrste regularizacije: *l1* i *l2* regularizacija. Uz svaku regularizaciju potrebno je dodati i regularizacijski parametar, stoga korištenje ove funkcije jest sljedeće: *setRegularization([['l2',0.001],['l2',0.001],['l1',0.01],None])* gdje se na prva dva sloja postavlja regularizacija *l2* s regularizacijskim parametrom $0.001$, na treći sloj se postavlja regularizacije *l1* s regularizacijskim parametrom $0.01$, dok četvrti sloj nema regularizacije.\n",
    "<br><br>\n",
    "4. *setBatchNormalization* - Metoda koja primjenjuje [Batch Normalization](https://towardsdatascience.com/batch-normalization-in-neural-networks-1ac91516821c) na slojeve, kao argument prima listu vrijednosti momentuma koji se koristi prilikom ažuriranja batch normalization matrice (*None* ukoliko nema normalizacije). Primjer pozivanja funkcije glasi: *setBatchNormalization([0.9, 0.9, 0.9, None])*\n",
    "<br><br>\n",
    "5. *setInitialisation* - Metoda koja postavlja inicijalne vrijednosti težina koje se treniraju. Implementirana je [He inicijalizacija](https://towardsdatascience.com/random-initialization-for-neural-networks-a-thing-of-the-past-bfcdd806bf9e). Korištenje jest vrlo jednostavno: *setInitialisation(['he_init','he_init','he_init',None])*. U slučaju kada se koristi *None*, težine su nasumično inicijalizirane.\n",
    "<br><br>\n",
    "6. *setLoss* - Metoda koja postavlja loss funkciju. Loss funkcija može biti: *softmax* ili *mse*, odnosno \n",
    "[softmax za klasifikaciju](https://deepnotes.io/softmax-crossentropy) ili [srednja kvadratna pogreška za regresiju](https://medium.freecodecamp.org/machine-learning-mean-squared-error-regression-line-c7dde9a26b93). Korištenje metode je sljedeće: *setLoss(\"softmax\")* i *setLoss(\"mse\")*.\n",
    "<br><br>\n",
    "\n",
    "7. *setOptimiser* - Metoda koja postavlja optimizator. Ugrađeni optimizatori su:\n",
    "\n",
    "    * Adam - *setOptimiser([\"adam\", $\\alpha$]=*\n",
    "    * AdaGrad - *setOptimiser([\"adaGrad\", $\\alpha$])*\n",
    "    * Gradient Descent - *setOptimiser([\"gradDes\", $\\alpha$])*\n",
    "    * Momentum - *setOptimiser([\"momentum\", $\\alpha$, $\\beta$])*\n",
    "    * RMSProp  - *setOptimiser([\"rmsProp\", $\\alpha$, $\\beta$])*\n",
    "<br><br>\n",
    "\n",
    "8. *setIoShape* - Metoda koja postavlja oblik ulaznih i izlaznih podataka kao i njihov tip. Za broj stupaca se tipično postavlja *None*. Metoda se poziva tako da se prvo navodi oblik ulaznog sloja, potom oblik izlaza, a zatim tipovi ulaznih i izlaznih podataka. Primjerice: *setIOshape((None,9),(None), tf.float32, tf.int32)* postavlja ulazni sloj koji će imati $9$ značajki tipa float, izlazni sloj će imati $1$ značajku i bit će tipa integer.\n",
    "<br><br>\n",
    "\n",
    "9. *setAlphaDecay* - Metoda omogućuje dodavanje raspada parametra $\\alpha$. Poziva se na sljedeći način: *setAlphaDecay($\\alpha_0$,$r$,$baza$)* (tipična baza je $1/10$).\n",
    "<br><br>\n",
    "\n",
    "10. *trainModel* - Metoda koja trenira model. Kao argumente prima redom: *ulazni parametri trening seta*, *izlaz trening seta*, *ulazni parametri validacijskog seta*, *izlaz validacijskog seta*, *broj epoha*, *veličina batch-a*, *early stopping kriterij* nakon kojeg se zaustavlja treniranje."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testirani modeli\n",
    "Svi modeli su imali iste parametre:\n",
    "\n",
    "1. Broj neurona i slojeva: 50, 50, 25, 10, 1,\n",
    "2. Aktivacije: relu, relu, relu, relu, None,\n",
    "3. Regularizacija: ['l2',0.001],['l2',0.001], ['l2',0.001],['l2',0.001],None,\n",
    "4. BatchNormalization: 0.9, 0.9, 0.9, 0.9, 0.9\n",
    "5. Inicijalizacija: He inicijalizacija, He inicijalizacija, He inicijalizacija, He inicijalizacija, He inicijalizacija\n",
    "6. Loss funkcija: MSE\n",
    "7. Ulazni sloj/izlazni sloj: 9 parametara tipa float/ 1 (vrijednost kuće) tipa float\n",
    "8. Trenirali su se s uključenim early stop algoritmom na 10000 epoha s veličinom batch-a 200 i early stopping kriterijem nakon 25 nepromijenjenih najboljih rezultata.\n",
    "9. Gradient descent i Momentum optimizator trenirali su se i sa AlphaDecay-om (parametri: 0.001,10000,1/10) \n",
    "10. Inicijalni $\\alpha = 0.001$\n",
    "\n",
    "Trening set se sastojao od $15480$ slika, a validacijski set od $5160$ slika."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from functools import partial\n",
    "\n",
    "class model:\n",
    "    \n",
    "    # Init variables\n",
    "    def __init__(self):\n",
    "        # Graph elements\n",
    "        self.neurons = None\n",
    "        self.activations = None\n",
    "        self.regularization = None\n",
    "        self.batchNormalisation = None\n",
    "        self.initialisation = None\n",
    "        self.loss = None\n",
    "        self.optimiser = None\n",
    "        self.decay = False\n",
    "    \n",
    "    # Set input/outputshape\n",
    "    def setIOshape(self, inputShape, outputShape, itype = tf.float32, otype = tf.float32):\n",
    "        self.inputShape = inputShape\n",
    "        self.outputShape = outputShape\n",
    "        self.itype= itype\n",
    "        self.otype= otype\n",
    "    \n",
    "    # Add neurons\n",
    "    def setLayersNeurons(self, neuronsPerLayer):\n",
    "        self.neurons = neuronsPerLayer\n",
    "\n",
    "    # Add activations\n",
    "    def setActivations(self, activationsPerLayers):\n",
    "        self.activations = activationsPerLayers\n",
    "    \n",
    "    # Add regularizations\n",
    "    def setRegularizations(self, regularizationPerLayer):\n",
    "        self.regularization = regularizationPerLayer\n",
    "        \n",
    "    # Add regularizations\n",
    "    def setBatchNormalization(self, batchNormalizationPerLayer):\n",
    "        self.batchNormalisation = batchNormalizationPerLayer\n",
    "    \n",
    "    # Add initialisations for weights\n",
    "    def setInitialisation(self, initialisationPerLayer):\n",
    "        self.initialisation = initialisationPerLayer\n",
    "    \n",
    "    # Set loss function\n",
    "    def setLoss(self, loss):\n",
    "        self.loss = loss\n",
    "    \n",
    "    # Set optimiser function\n",
    "    def setOptimiser(self, opt):\n",
    "        self.optimiser = opt\n",
    "               \n",
    "    # Set alphaDecay\n",
    "    def setAlphaDecay(self, initAlpha, decay_steps, decay_rate):\n",
    "        self.initial_learning_rate = initAlpha\n",
    "        self.decay_steps = decay_steps\n",
    "        self.decay_rate = decay_rate\n",
    "        self.decay = True\n",
    "        \n",
    "    # Reset default graph\n",
    "    def reset_graph(self, seed=42):\n",
    "        tf.reset_default_graph()\n",
    "        tf.set_random_seed(seed)\n",
    "        np.random.seed(seed)\n",
    "    \n",
    "    # Batch shuffeling\n",
    "    def shuffle_batch(self, X, y, batch_size):\n",
    "        rnd_idx = np.random.permutation(len(X))\n",
    "        n_batches = len(X) // batch_size\n",
    "        for batch_idx in np.array_split(rnd_idx, n_batches):\n",
    "            X_batch, y_batch = X[batch_idx], y[batch_idx]\n",
    "            yield X_batch, y_batch\n",
    "\n",
    "    # Train\n",
    "    def trainModel(self, xTrain, yTrain, xValid, yValid, epochs, batch_size, earlyStopping = 0):\n",
    "        # Building Graph\n",
    "        # Check for neurons\n",
    "        if (self.neurons == None):\n",
    "            print(\"Error, no neurons added - use setLayersNeuorns method\")\n",
    "            return(0)\n",
    "        \n",
    "        elementsSum = len(self.neurons) + len(self.activations) + len(self.regularization) + len(self.batchNormalisation) + len(self.initialisation) \n",
    "        if (len(self.neurons) != elementsSum / 5):\n",
    "            print(\"Error, invalid number of elements\")\n",
    "            return (0)\n",
    "        \n",
    "        # Reseting default graph\n",
    "        self.reset_graph()\n",
    "        \n",
    "        # Create input/outputPlaceholders and training flag\n",
    "        X = tf.placeholder(self.itype, shape = self.inputShape, name = 'X')\n",
    "        y = tf.placeholder(self.otype, shape = self.outputShape, name = 'y')\n",
    "        training = tf.placeholder_with_default(False, shape=(), name='training')\n",
    "        \n",
    "        \n",
    "        # Prepare dicts\n",
    "        # Initialisation\n",
    "        for i, elem in enumerate(self.initialisation):\n",
    "            if elem == \"he_init\":\n",
    "                self.initialisation[i] = tf.variance_scaling_initializer()\n",
    "            else:\n",
    "                self.initialisation[i] = None\n",
    "        \n",
    "        # Regularization\n",
    "        for i, elem in enumerate(self.regularization):\n",
    "            if (elem == None or len(elem) != 2):\n",
    "                continue\n",
    "            if (elem[0] != \"l1\" or elem[0]!= \"l2\"):\n",
    "                self.regularization[i] = None\n",
    "\n",
    "            if (elem[0] == \"l1\"):\n",
    "                self.regularization[i] = tf.contrib.layers.l1_regularizer(elem[1])\n",
    "            \n",
    "            if (elem[0] == \"l2\"):\n",
    "                self.regularization[i] = tf.contrib.layers.l2_regularizer(elem[1])\n",
    "        \n",
    "        # Batch normalisation\n",
    "        my_batch_norm_layer = partial(tf.layers.batch_normalization,training=training)\n",
    "        \n",
    "        # Activation function\n",
    "        for i, elem in enumerate(self.activations):\n",
    "            \n",
    "            if elem == 'relu':\n",
    "                self.activations[i] = partial(tf.nn.relu)\n",
    "            if elem == 'elu':\n",
    "                self.activations[i] = partial(tf.nn.elu)\n",
    "            if elem == 'sig':\n",
    "                self.activations[i] = partial(tf.nn.sigmoid)\n",
    "            if elem == 'tanh':\n",
    "                self.activations[i] = partial(tf.nn.tanh) \n",
    "            \n",
    "            if (elem not in ['relu', 'elu', 'sig', 'tanh']):\n",
    "                self.activations[i] = None\n",
    "            \n",
    "        # Build layers\n",
    "        layers = [X]\n",
    "        with tf.name_scope(\"dnn\"):        \n",
    "            \n",
    "            for i, numberOfNeurons in enumerate(self.neurons):\n",
    "                layer = tf.layers.dense(layers[i], self.neurons[i], kernel_initializer = self.initialisation[i]\n",
    "                                       , kernel_regularizer = self.regularization[i], name = \"hidden_\"+str(i+1))\n",
    "                if (self.activations[i] != None):\n",
    "                    if self.batchNormalisation[i] != None:\n",
    "                        layer = self.activations[i](my_batch_norm_layer(layer, momentum = self.batchNormalisation[i]))\n",
    "                    else:\n",
    "                        layer = self.activations[i](layer)\n",
    "                else:\n",
    "                    if self.batchNormalisation[i] != None:\n",
    "                        layer = my_batch_norm_layer(layer, momentum = self.batchNormalisation[i])\n",
    "                layers.append(layer)\n",
    "        \n",
    "        # Build loss\n",
    "        with tf.name_scope(\"loss\"):\n",
    "            if self.loss == \"softmax\":\n",
    "                base = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=layers[len(layers)-1])\n",
    "            \n",
    "            if self.loss == \"mse\":\n",
    "                base = tf.reduce_mean(tf.square(layers[len(layers)-1] - y), name = \"mse\")\n",
    "\n",
    "            base_loss = tf.reduce_mean(base, name = \"base_loss\")\n",
    "            reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)\n",
    "            loss = tf.add_n([base_loss] + reg_losses, name=\"loss\")\n",
    "            \n",
    "        # Build train\n",
    "        with tf.name_scope(\"train\"):\n",
    "            if (len(self.optimiser) != 2 and self.optimiser[0] not in [\"rmsProp\", \"momentum\"]):\n",
    "                print(\"Wrong set up for optimiser\")\n",
    "                return (0)\n",
    "            global_step = tf.Variable(0, trainable = False, name=\"global_step\")\n",
    "\n",
    "            if (self.optimiser[0] == \"gradDes\"):\n",
    "                if self.decay:\n",
    "                    learning_rate = tf.train.exponential_decay(self.initial_learning_rate, global_step, self.decay_steps, self.decay_rate)\n",
    "                else:\n",
    "                    learning_rate = self.optimiser[1]\n",
    "                optimizer = tf.train.GradientDescentOptimizer(learning_rate)\n",
    "                \n",
    "            if (self.optimiser[0] == \"adaGrad\"):\n",
    "                if self.decay:\n",
    "                    learning_rate = tf.train.exponential_decay(self.initial_learning_rate, global_step, self.decay_steps, self.decay_rate)\n",
    "                else:\n",
    "                    learning_rate = self.optimiser[1]\n",
    "                optimizer = tf.train.AdagradOptimizer(learning_rate)\n",
    "            \n",
    "            if (self.optimiser[0] == \"adam\"):\n",
    "                if self.decay:\n",
    "                    learning_rate = tf.train.exponential_decay(self.initial_learning_rate, global_step, self.decay_steps, self.decay_rate)\n",
    "                else:\n",
    "                    learning_rate = self.optimiser[1]\n",
    "                optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "            \n",
    "            if (self.optimiser[0] == \"momentum\"):\n",
    "                if self.decay:\n",
    "                    learning_rate = tf.train.exponential_decay(self.initial_learning_rate, global_step, self.decay_steps, self.decay_rate)\n",
    "                else:\n",
    "                    learning_rate = self.optimiser[1]\n",
    "                optimizer = tf.train.MomentumOptimizer(learning_rate,momentum = self.optimiser[2])\n",
    "            \n",
    "            if (self.optimiser[0] == \"rmsProp\"):\n",
    "                if self.decay:\n",
    "                    learning_rate = tf.train.exponential_decay(self.initial_learning_rate, global_step, self.decay_steps, self.decay_rate)\n",
    "                else:\n",
    "                    learning_rate = self.optimiser[1]\n",
    "                optimizer = tf.train.RMSPropOptimizer(learning_rate, momentum = self.optimiser[2])\n",
    "\n",
    "            training_op = optimizer.minimize(loss, global_step = global_step)\n",
    "\n",
    "        # Build accuracy\n",
    "        with tf.name_scope(\"eval\"):\n",
    "            if self.loss == \"softmax\":\n",
    "                correct = tf.nn.in_top_k(layers[len(layers)-1], y, 1)\n",
    "                accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))\n",
    "            \n",
    "            if self.loss == \"mse\":\n",
    "                accuracy = tf.reduce_mean(tf.square(layers[len(layers)-1]-y))\n",
    "                \n",
    "        # Traning setup\n",
    "        init = tf.global_variables_initializer()\n",
    "        extra_update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\n",
    "        saver = tf.train.Saver()\n",
    "\n",
    "        n_epochs = epochs\n",
    "        batch_size = batch_size\n",
    "        valid = []\n",
    "        train = []\n",
    "        \n",
    "\n",
    "        # For early stopping\n",
    "        if self.loss == \"softmax\":\n",
    "            best = 0\n",
    "        else:\n",
    "            best = 10000\n",
    "        epochBest = 0\n",
    "        \n",
    "        with tf.Session() as sess:\n",
    "            init.run()\n",
    "            for epoch in range(n_epochs):\n",
    "                for X_batch, y_batch in self.shuffle_batch(X_train, y_train, batch_size):\n",
    "                    sess.run([training_op, extra_update_ops],\n",
    "                             feed_dict={training: True, X: X_batch, y: y_batch})\n",
    "                accuracy_val = accuracy.eval(feed_dict={X: X_valid, y: y_valid})\n",
    "                accuracy_train = accuracy.eval(feed_dict={X: X_train, y: y_train})\n",
    "                \n",
    "                valid.append(accuracy_val)\n",
    "                train.append(accuracy_train)\n",
    "                #print(epoch, \"Validation accuracy:\", accuracy_val, \"Train accuracy:\", accuracy_train, best)\n",
    "                if self.loss == \"softmax\":\n",
    "                    if accuracy_val > best :\n",
    "                        best = accuracy_val\n",
    "                        epochBest = epoch\n",
    "                        save_path = saver.save(sess, \"./my_model_final.ckpt\")\n",
    "                else:\n",
    "                    if accuracy_val < best:\n",
    "                        best = accuracy_val\n",
    "                        epochBest = epoch\n",
    "                        save_path = saver.save(sess, \"./my_model_final.ckpt\")\n",
    "                if earlyStopping != 0 and epoch - epochBest > earlyStopping:\n",
    "                    #print(\"EARLY STOPPING\")\n",
    "                    break\n",
    "        return (valid,train,best,epochBest,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.datasets import fetch_california_housing\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Dohvaćanje podataka\n",
    "housing = fetch_california_housing()\n",
    "m, n = housing.data.shape\n",
    "\n",
    "#Skaliranje podataka\n",
    "scaler = StandardScaler()\n",
    "scaled_housing_data = scaler.fit_transform(housing.data)\n",
    "scaled_housing_data_plus_bias = np.c_[np.ones((m, 1)), scaled_housing_data]\n",
    "\n",
    "# Razdvajanje setova\n",
    "X_original, X_valid, y_original, y_valid = train_test_split(scaled_housing_data_plus_bias, housing.target.reshape(-1, 1), test_size=0.25, random_state=42)\n",
    "\n",
    "# Pohrana rezultata\n",
    "rmsPropBest = []\n",
    "rmsPropEpoch = []\n",
    "gradDescBest = []\n",
    "gradDescEpoch = []\n",
    "adaGradBest = []\n",
    "adaGradEpoch = []\n",
    "momentumBest = []\n",
    "momentumEpoch = []\n",
    "adamBest = []\n",
    "adamEpoch = []\n",
    "\n",
    "rmsPropValid = []\n",
    "rmsPropTrain = []\n",
    "gdValid = []\n",
    "gdTrain = []\n",
    "adaGradValid = []\n",
    "adaGradTrain = []\n",
    "momentumValid = []\n",
    "momentumTrain = []\n",
    "adamValid = []\n",
    "adamTrain = []\n",
    "\n",
    "\n",
    "\n",
    "# Treniranje\n",
    "for i in range(100):\n",
    "    print (\"Epoha: {}\".format(i))\n",
    "    X_train, X_valid, y_train, y_valid = train_test_split(scaled_housing_data_plus_bias, housing.target.reshape(-1, 1), test_size=0.25, random_state=i)\n",
    "    n, m = X_train.shape\n",
    "    myModel = model()\n",
    "    myModel.setLayersNeurons([50,50,25,10,1])\n",
    "    myModel.setActivations(['relu','relu','relu','relu', None])\n",
    "    myModel.setRegularizations([['l2',0.001],['l2',0.001], ['l2',0.001],['l2',0.001],None])\n",
    "    myModel.setBatchNormalization([0.9, 0.9, 0.9, 0.9, 0.9])\n",
    "    myModel.setInitialisation(['he_init','he_init','he_init','he_init','he_init'])\n",
    "    myModel.setLoss(\"mse\")\n",
    "    myModel.setOptimiser([\"adam\",0.001])\n",
    "    myModel.setIOshape((None,m),(None,1), tf.float32, tf.float32)\n",
    "\n",
    "    \n",
    "    myModel2 = model()\n",
    "    myModel2.setLayersNeurons([50,50,25,10,1])\n",
    "    myModel2.setActivations(['relu','relu','relu','relu', None])\n",
    "    myModel2.setRegularizations([['l2',0.001],['l2',0.001], ['l2',0.001],['l2',0.001],None])\n",
    "    myModel2.setBatchNormalization([0.9, 0.9, 0.9, 0.9, 0.9])\n",
    "    myModel2.setInitialisation(['he_init','he_init','he_init','he_init','he_init'])\n",
    "    myModel2.setAlphaDecay(0.001,10000,1/10)\n",
    "    myModel2.setLoss(\"mse\")\n",
    "    myModel2.setOptimiser([\"gradDes\",0.001])\n",
    "    myModel2.setIOshape((None,m),(None,1), tf.float32, tf.float32)\n",
    "    \n",
    "    myModel3 = model()\n",
    "    myModel3.setLayersNeurons([50,50,25,10,1])\n",
    "    myModel3.setActivations(['relu','relu','relu','relu', None])\n",
    "    myModel3.setRegularizations([['l2',0.001],['l2',0.001], ['l2',0.001],['l2',0.001],None])\n",
    "    myModel3.setBatchNormalization([0.9, 0.9, 0.9, 0.9, 0.9])\n",
    "    myModel3.setInitialisation(['he_init','he_init','he_init','he_init','he_init'])\n",
    "    myModel3.setLoss(\"mse\")\n",
    "    myModel3.setOptimiser([\"adaGrad\",0.001])\n",
    "    myModel3.setIOshape((None,m),(None,1), tf.float32, tf.float32)\n",
    "\n",
    "    myModel4 = model()\n",
    "    myModel4.setLayersNeurons([50,50,25,10,1])\n",
    "    myModel4.setActivations(['relu','relu','relu','relu', None])\n",
    "    myModel4.setRegularizations([['l2',0.001],['l2',0.001], ['l2',0.001],['l2',0.001],None])\n",
    "    myModel4.setBatchNormalization([0.9, 0.9, 0.9, 0.9, 0.9])\n",
    "    myModel4.setInitialisation(['he_init','he_init','he_init','he_init','he_init'])\n",
    "    myModel4.setAlphaDecay(0.001,10000,1/10)\n",
    "    myModel4.setLoss(\"mse\")\n",
    "    myModel4.setOptimiser([\"momentum\",0.001, 0.9])\n",
    "    myModel4.setIOshape((None,m),(None,1), tf.float32, tf.float32)\n",
    "\n",
    "    myModel5 = model()\n",
    "    myModel5.setLayersNeurons([50,50,25,10,1])\n",
    "    myModel5.setActivations(['relu','relu','relu','relu', None])\n",
    "    myModel5.setRegularizations([['l2',0.001],['l2',0.001], ['l2',0.001],['l2',0.001],None])\n",
    "    myModel5.setBatchNormalization([0.9, 0.9, 0.9, 0.9, 0.9])\n",
    "    myModel5.setInitialisation(['he_init','he_init','he_init','he_init','he_init'])\n",
    "    myModel5.setLoss(\"mse\")\n",
    "    myModel5.setOptimiser([\"rmsProp\",0.001,0.9])\n",
    "    myModel5.setIOshape((None,m),(None,1), tf.float32, tf.float32)\n",
    "    \n",
    "\n",
    "    valid1, train1, best1, epochBest1 = myModel.trainModel(X_train, y_train, X_valid, y_valid, 10000, 200, earlyStopping = 25)\n",
    "    valid2, train2, best2, epochBest2 = myModel2.trainModel(X_train, y_train, X_valid, y_valid, 10000, 200, earlyStopping = 25)\n",
    "    valid3, train3, best3, epochBest3 = myModel3.trainModel(X_train, y_train, X_valid, y_valid, 10000, 200, earlyStopping = 25)\n",
    "    valid4, train4, best4, epochBest4 = myModel4.trainModel(X_train, y_train, X_valid, y_valid, 10000, 200, earlyStopping = 25)\n",
    "    valid5, train5, best5, epochBest5 = myModel5.trainModel(X_train, y_train, X_valid, y_valid, 10000, 200, earlyStopping = 25)\n",
    "    \n",
    "    # Pohrana podataka\n",
    "    rmsPropValid.append(valid1)\n",
    "    rmsPropTrain.append(train1)\n",
    "    rmsPropBest.append(best1)\n",
    "    rmsPropEpoch.append(epochBest1)\n",
    "    \n",
    "    gdValid.append(valid2)\n",
    "    gdTrain.append(train2)\n",
    "    gradDescBest.append(best2)\n",
    "    gradDescEpoch.append(epochBest2)\n",
    "    \n",
    "    adaGradValid.append(valid3)\n",
    "    adaGradTrain.append(train3)\n",
    "    adaGradBest.append(best3)\n",
    "    adaGradEpoch.append(epochBest3)\n",
    "    \n",
    "    momentumValid.append(valid4)\n",
    "    momentumTrain.append(train4)\n",
    "    momentumBest.append(best4)\n",
    "    momentumEpoch.append(epochBest4)\n",
    "    \n",
    "    adamValid.append(valid5)\n",
    "    adamTrain.append(train5)\n",
    "    adamBest.append(best5)\n",
    "    adamEpoch.append(epochBest5)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grafovi konvergencije"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Učitavanje potrebnih knjižica\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Load data\n",
    "dataTrain = rmsPropTrain\n",
    "dataValid = rmsPropValid\n",
    "avgEpoch = np.mean(np.array(rmsPropEpoch))\n",
    "avgScore = np.mean(np.array(rmsPropBest))\n",
    "maxD = 0\n",
    "\n",
    "# Get max length\n",
    "for i in range(100):\n",
    "    if len(dataTrain[i]) > maxD:\n",
    "        maxD = len(dataTrain[i])\n",
    "   \n",
    "trainPoints = []\n",
    "validPoints = []\n",
    "\n",
    "# Get points\n",
    "for i in range(maxD):\n",
    "    trainPoint = 0\n",
    "    trainRep = 0\n",
    "    validPoint = 0\n",
    "    validRep = 0\n",
    "    \n",
    "    for j in range(100):\n",
    "        if len(dataTrain[j]) > i:\n",
    "            trainPoint += dataTrain[j][i]\n",
    "            trainRep += 1\n",
    "        if len(dataValid[j]) > i:\n",
    "            validPoint += dataValid[j][i]\n",
    "            validRep += 1\n",
    "    trainPoints.append(trainPoint/trainRep)\n",
    "    validPoints.append(validPoint/validRep)\n",
    "\n",
    "# Plot graph\n",
    "fig = plt.figure(\"Average\")\n",
    "plt.plot(range(maxD), trainPoints,'b',label=\"Train\")\n",
    "plt.plot(range(maxD),validPoints,'r',label=\"Valid\")\n",
    "plt.plot(np.ones(100)*avgEpoch, np.linspace(min(trainPoints),max(trainPoints),100), 'g--', label=\"Stop\")\n",
    "plt.title(\"RMSProp\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Score\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistička obrada podataka"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Učitavanje potrebnih knjižica\n",
    "import numpy as np\n",
    "import matplotlib.mlab as mlab\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "adaGradScore = np.array([3.5377717, 3.756132, 3.8414092, 3.8784597, 3.6417015, 3.661017, 3.821608, 3.876443, 3.847351, 3.3264523, 3.8323412, 3.7991445, 3.6358564, 3.7055511, 3.4977775, 3.859666, 3.9544687, 3.8616483, 3.6630762, 3.7617521, 3.7159967, 3.6688054, 3.6699855, 3.974916, 3.6163132, 3.7645693, 3.813519, 3.4751654, 3.5078292, 3.833887, 3.5527003, 3.6817458, 3.848904, 3.6620307, 3.89151, 3.5973225, 3.8503785, 3.754792, 3.5442836, 3.6148014, 4.0552907, 3.4481683, 3.8438046, 3.5001688, 3.9728303, 3.6669586, 3.8460844, 3.957142, 3.9264672, 3.347025, 3.7439866, 3.8272998, 3.8613887, 3.8332818, 3.7933533, 3.5557163, 3.7825034, 3.7477803, 3.9299226, 3.9952826, 3.9611108, 3.931929, 3.8432927, 3.5717077, 3.8013017, 3.4866416, 3.7633555, 3.8945498, 3.753362, 3.832951, 3.8551571, 3.7731586, 3.7697284, 3.797567, 3.787612, 3.8271832, 3.6563716, 3.7837474, 3.8455033, 3.7596128, 3.7613502, 3.884901, 3.7917995, 3.5050282, 3.6216574, 3.76245, 3.621797, 3.681921, 3.8025255, 3.5522943, 3.7051463, 3.7696464, 3.932058, 3.6936774, 3.4495292, 3.6672595, 3.5272775, 3.804337, 3.7350004, 3.871881])\n",
    "adaGradEpoch = np.array([202, 86, 76, 33, 162, 141, 75, 76, 74, 293, 86, 74, 153, 106, 198, 56, 30, 68, 124, 121, 130, 121, 84, 38, 195, 85, 64, 266, 228, 27, 232, 105, 79, 136, 67, 167, 50, 66, 172, 173, 30, 225, 49, 182, 25, 93, 43, 40, 28, 314, 99, 49, 46, 56, 87, 218, 80, 85, 26, 16, 21, 20, 27, 152, 99, 228, 70, 50, 103, 49, 44, 65, 46, 61, 79, 85, 138, 32, 84, 94, 76, 52, 68, 175, 108, 109, 189, 116, 81, 179, 95, 73, 46, 83, 186, 101, 206, 81, 137, 38])\n",
    "\n",
    "adamScore = np.array([0.32417157, 0.3236075, 0.31669715, 0.3234331, 0.3383628, 0.29094124, 0.2824574, 0.31537795, 0.3203576, 0.2863951, 0.30760592, 0.31439754, 0.31100094, 0.33154863, 0.29488206, 0.28510925, 0.2847425, 0.3036325, 0.35686594, 0.28584424, 0.35734564, 0.3026612, 0.2639545, 0.26700133, 0.3274023, 0.3079812, 0.31784797, 0.31448033, 0.31973773, 0.31328967, 0.3406593, 0.31198546, 0.2892943, 0.3033445, 0.3044309, 0.28795218, 0.3198657, 0.32269683, 0.30936855, 0.29700914, 0.32982007, 0.28271067, 0.29334688, 0.318613, 0.28108695, 0.29616547, 0.32502204, 0.3452332, 0.31639066, 0.30432656, 0.31218317, 0.29502794, 0.31452253, 0.35201618, 0.31210953, 0.32392386, 0.30951715, 0.32851955, 0.2856663, 0.27621236, 0.31520292, 0.28347623, 0.32140395, 0.29196933, 0.31453755, 0.2851151, 0.29347444, 0.30649784, 0.30142558, 0.2956203, 0.31236213, 0.31517443, 0.2816837, 0.33098647, 0.2874186, 0.29140204, 0.2943058, 0.33242828, 0.27634895, 0.30605254, 0.321146, 0.29049993, 0.28172693, 0.3212033, 0.31476814, 0.2972718, 0.27551717, 0.32316273, 0.32977644, 0.27361116, 0.3159537, 0.30150175, 0.30879086, 0.293877, 0.29113445, 0.31858087, 0.26887345, 0.32885072, 0.2955004, 0.2999737])\n",
    "adamEpoch = np.array([8, 18, 42, 12, 11, 32, 39, 36, 15, 84, 18, 58, 22, 10, 13, 57, 34, 25, 16, 17, 4, 87, 34, 51, 73, 11, 31, 4, 14, 8, 11, 17, 20, 19, 6, 45, 39, 14, 17, 12, 24, 59, 17, 15, 38, 7, 6, 11, 26, 11, 20, 19, 4, 7, 75, 24, 11, 36, 62, 45, 29, 24, 14, 37, 6, 28, 21, 12, 11, 23, 22, 32, 67, 9, 20, 8, 61, 38, 79, 12, 8, 66, 62, 20, 23, 29, 40, 14, 39, 60, 14, 32, 12, 24, 21, 3, 44, 36, 16, 53])\n",
    "\n",
    "gdScore = np.array([0.40343145, 0.3995014, 0.39679018, 0.37810802, 0.39395294, 0.39245, 0.4061372, 0.39538097, 0.385495, 0.40732184, 0.39901033, 0.39628386, 0.3935729, 0.40403333, 0.38848144, 0.39349768, 0.4640165, 0.38744423, 0.37974608, 0.3925574, 0.4050962, 0.39611825, 0.36157897, 0.37148148, 0.4006966, 0.3867069, 0.38846645, 0.3855416, 0.38297537, 0.36482897, 0.43375546, 0.3807426, 0.40070337, 0.37644392, 0.38361964, 0.3909691, 0.3861461, 0.38682568, 0.3791259, 0.39386222, 0.39573652, 0.38411364, 0.38425323, 0.38140243, 0.36409345, 0.38644892, 0.3918298, 0.38952842, 0.3921998, 0.39776242, 0.40449804, 0.37746486, 0.38892013, 0.40174055, 0.40954676, 0.40754405, 0.41206548, 0.40640643, 0.4047953, 0.3728906, 0.40148073, 0.3642799, 0.3826057, 0.37580577, 0.40339816, 0.38950244, 0.42415756, 0.40235814, 0.39622387, 0.39360327, 0.38017684, 0.4007698, 0.37753853, 0.3892596, 0.37608308, 0.45759633, 0.39648575, 0.40029955, 0.3803772, 0.3918096, 0.4068011, 0.41986558, 0.3841825, 0.38125318, 0.38147402, 0.39818507, 0.4134346, 0.37878206, 0.41287625, 0.3744243, 0.37956893, 0.40049177, 0.3916857, 0.37142944, 0.37624437, 0.38583326, 0.4208939, 0.4191676, 0.3898088, 0.3845271])\n",
    "gdEpoch = np.array([134, 218, 189, 169, 151, 130, 199, 236, 188, 202, 224, 193, 109, 144, 138, 87, 52, 146, 229, 159, 118, 180, 157, 132, 137, 97, 148, 114, 135, 165, 112, 191, 102, 105, 189, 168, 201, 104, 121, 164, 158, 150, 154, 158, 131, 125, 190, 194, 198, 171, 109, 197, 108, 108, 194, 145, 114, 174, 146, 203, 116, 234, 211, 175, 173, 144, 141, 154, 194, 160, 219, 210, 173, 176, 112, 104, 90, 222, 174, 178, 151, 148, 154, 189, 139, 188, 126, 167, 101, 217, 193, 120, 177, 177, 152, 147, 127, 152, 129, 95])\n",
    "\n",
    "momentumScore = np.array([0.30512246, 0.32050124, 0.3118204, 0.29860216, 0.32846767, 0.32377097, 0.3194519, 0.3188375, 0.32737195, 0.31416526, 0.31376815, 0.3155915, 0.31906888, 0.33468908, 0.2989422, 0.31819388, 0.3157418, 0.32492107, 0.30315226, 0.29696575, 0.32993317, 0.31338236, 0.28853416, 0.28785837, 0.32536113, 0.31178012, 0.29905328, 0.3069888, 0.31141117, 0.29798573, 0.3457167, 0.30721506, 0.3062942, 0.30499995, 0.30335963, 0.3079729, 0.30322728, 0.31084374, 0.30906832, 0.31301147, 0.31630963, 0.32588533, 0.30555665, 0.3049158, 0.28510648, 0.30788434, 0.33427083, 0.31363034, 0.31127074, 0.3030068, 0.32236198, 0.30453047, 0.3063938, 0.32014114, 0.34994724, 0.31897938, 0.31081125, 0.32688922, 0.32065347, 0.30915087, 0.33021173, 0.29745802, 0.3086957, 0.31498563, 0.30749908, 0.32109466, 0.31107074, 0.32902798, 0.32404593, 0.32046407, 0.29861194, 0.33787188, 0.2940089, 0.30977818, 0.3051475, 0.3120023, 0.3107265, 0.33036223, 0.29562786, 0.3249199, 0.32086077, 0.3154661, 0.300126, 0.30632794, 0.30394226, 0.3227191, 0.28800896, 0.3031807, 0.33319303, 0.28889045, 0.30301628, 0.3150718, 0.3172033, 0.29630026, 0.2998042, 0.32010755, 0.2913881, 0.3362923, 0.3105656, 0.3028073])\n",
    "momentumEpoch = np.array([95, 64, 92, 87, 40, 46, 77, 68, 43, 116, 108, 75, 35, 40, 112, 54, 92, 63, 82, 87, 38, 65, 67, 79, 49, 45, 136, 175, 58, 100, 46, 118, 55, 68, 102, 119, 88, 37, 60, 89, 76, 25, 64, 54, 69, 46, 23, 79, 67, 96, 55, 73, 58, 53, 31, 82, 97, 52, 71, 43, 59, 61, 52, 44, 73, 41, 126, 39, 73, 78, 120, 32, 127, 112, 41, 146, 66, 40, 102, 32, 74, 70, 103, 76, 60, 74, 148, 70, 39, 60, 63, 43, 73, 98, 90, 32, 75, 36, 61, 88])\n",
    "\n",
    "rmsScore = np.array([0.29006058, 0.28402218, 0.29099333, 0.283063, 0.30094597, 0.31597307, 0.28332794, 0.28437674, 0.28395766, 0.30384588, 0.28691795, 0.276377, 0.2808033, 0.27902743, 0.2672417, 0.28558615, 0.26566648, 0.27892727, 0.26648408, 0.28889012, 0.31122875, 0.28696412, 0.26855147, 0.2833428, 0.29345244, 0.27514333, 0.2713269, 0.28643894, 0.27636904, 0.27477035, 0.32560757, 0.27501276, 0.3043946, 0.2783086, 0.31523508, 0.2875137, 0.26740754, 0.2556993, 0.29588255, 0.28068423, 0.3005326, 0.27071145, 0.27468303, 0.2757961, 0.2589961, 0.27337775, 0.28602526, 0.27886572, 0.2755192, 0.26529616, 0.2837832, 0.27461398, 0.27821484, 0.2997016, 0.28743976, 0.28462982, 0.29634243, 0.28239188, 0.34044176, 0.27146813, 0.2917923, 0.2784095, 0.27055234, 0.26745185, 0.2688616, 0.3059669, 0.2770882, 0.2830725, 0.29811418, 0.28193155, 0.31134114, 0.27861995, 0.2575153, 0.29746437, 0.25985563, 0.3453521, 0.28055847, 0.3560163, 0.3752023, 0.28920022, 0.286063, 0.293109, 0.27619183, 0.28042608, 0.26896238, 0.2734545, 0.26086065, 0.28649902, 0.3061967, 0.28311342, 0.2707301, 0.29284564, 0.29340476, 0.29328632, 0.27273682, 0.2737857, 0.26837555, 0.3360302, 0.26765367, 0.27243727])\n",
    "rmsEpoch = np.array([56, 59, 67, 50, 63, 38, 64, 73, 99, 41, 73, 58, 74, 81, 81, 52, 108, 63, 121, 50, 38, 82, 90, 66, 80, 63, 68, 67, 58, 52, 46, 118, 69, 68, 47, 84, 87, 102, 66, 91, 55, 81, 62, 54, 85, 46, 52, 71, 67, 96, 54, 77, 59, 49, 110, 82, 74, 105, 56, 45, 59, 75, 90, 73, 69, 69, 80, 77, 106, 70, 51, 100, 67, 53, 93, 47, 66, 40, 62, 87, 63, 79, 76, 76, 60, 98, 77, 93, 51, 54, 63, 43, 45, 69, 79, 92, 54, 82, 61, 55])\n",
    "\n",
    "# Plot AdaGrad\n",
    "f = plt.figure(\"AdaGrad\")\n",
    "plt.title(\"AdaGrad\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Score\")\n",
    "plt.scatter(adaGradEpoch,adaGradScore, c='r', alpha = 0.25, marker=\"H\")\n",
    "plt.show()\n",
    "\n",
    "# Plot Adam\n",
    "f = plt.figure(\"Adam\")\n",
    "plt.title(\"Adam\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Score\")\n",
    "plt.scatter(adamEpoch,adamScore, c='b', alpha = 0.25, marker=\"D\")\n",
    "plt.show()\n",
    "\n",
    "\n",
    "# Plot Gradient Descent\n",
    "f = plt.figure(\"GradientDescent\")\n",
    "plt.title(\"GradientDescent\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Score\")\n",
    "plt.scatter(gdEpoch,gdScore, c='black', alpha = 0.25, marker=\"o\")\n",
    "plt.show()\n",
    "\n",
    "# Plot Momentum\n",
    "f = plt.figure(\"Momentum\")\n",
    "plt.title(\"Momentum\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Score\")\n",
    "plt.scatter(momentumEpoch,momentumScore, c='green', alpha = 0.25, marker=\"*\")\n",
    "plt.show()\n",
    "\n",
    "# Plot RMSProp\n",
    "f = plt.figure(\"RMSProp\")\n",
    "plt.title(\"RMSProp\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Score\")\n",
    "plt.scatter(rmsEpoch,rmsScore, c='hotpink', alpha = 0.25, marker=\"^\")\n",
    "plt.show()\n",
    "\n",
    "#Plot\n",
    "f = plt.figure(\"All algorithms\")\n",
    "\n",
    "plt.title(\"Compare of algorithms\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Score\")\n",
    "plt.scatter(rmsEpoch,rmsScore, c='hotpink', alpha = 0.25, marker=\"^\", label=\"RMSProp\")\n",
    "plt.scatter(momentumEpoch,momentumScore, c='green', alpha = 0.25, marker=\"*\", label=\"Momentum\")\n",
    "plt.scatter(gdEpoch,gdScore, c='black', alpha = 0.25, marker=\"o\", label = \"GD\")\n",
    "plt.scatter(adamEpoch,adamScore, c='b', alpha = 0.25, marker=\"D\", label = \"Adam\")\n",
    "plt.scatter(adaGradEpoch,adaGradScore, c='r', alpha = 0.25, marker=\"H\", label = \"AdaGrad\")\n",
    "plt.legend(loc='lower right')\n",
    "plt.show()\n",
    "\n",
    "#Plot\n",
    "f = plt.figure(\"All algorithms except adaGrads\")\n",
    "plt.title(\"Compare of algorithms without adaGrad\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Score\")\n",
    "plt.scatter(rmsEpoch,rmsScore, c='hotpink', alpha = 0.25, marker=\"^\", label=\"RMSProp\")\n",
    "plt.scatter(momentumEpoch,momentumScore, c='green', alpha = 0.25, marker=\"*\", label=\"Momentum\")\n",
    "plt.scatter(gdEpoch,gdScore, c='black', alpha = 0.25, marker=\"o\", label = \"GD\")\n",
    "plt.scatter(adamEpoch,adamScore, c='b', alpha = 0.25, marker=\"D\", label = \"Adam\")\n",
    "plt.legend(loc='lower right')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Učitavanje potrebnih knjižica\n",
    "import numpy as np\n",
    "import matplotlib.mlab as mlab\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Setovi podataka\n",
    "adaGradScore = np.array([3.5377717, 3.756132, 3.8414092, 3.8784597, 3.6417015, 3.661017, 3.821608, 3.876443, 3.847351, 3.3264523, 3.8323412, 3.7991445, 3.6358564, 3.7055511, 3.4977775, 3.859666, 3.9544687, 3.8616483, 3.6630762, 3.7617521, 3.7159967, 3.6688054, 3.6699855, 3.974916, 3.6163132, 3.7645693, 3.813519, 3.4751654, 3.5078292, 3.833887, 3.5527003, 3.6817458, 3.848904, 3.6620307, 3.89151, 3.5973225, 3.8503785, 3.754792, 3.5442836, 3.6148014, 4.0552907, 3.4481683, 3.8438046, 3.5001688, 3.9728303, 3.6669586, 3.8460844, 3.957142, 3.9264672, 3.347025, 3.7439866, 3.8272998, 3.8613887, 3.8332818, 3.7933533, 3.5557163, 3.7825034, 3.7477803, 3.9299226, 3.9952826, 3.9611108, 3.931929, 3.8432927, 3.5717077, 3.8013017, 3.4866416, 3.7633555, 3.8945498, 3.753362, 3.832951, 3.8551571, 3.7731586, 3.7697284, 3.797567, 3.787612, 3.8271832, 3.6563716, 3.7837474, 3.8455033, 3.7596128, 3.7613502, 3.884901, 3.7917995, 3.5050282, 3.6216574, 3.76245, 3.621797, 3.681921, 3.8025255, 3.5522943, 3.7051463, 3.7696464, 3.932058, 3.6936774, 3.4495292, 3.6672595, 3.5272775, 3.804337, 3.7350004, 3.871881])\n",
    "adaGradEpoch = np.array([202, 86, 76, 33, 162, 141, 75, 76, 74, 293, 86, 74, 153, 106, 198, 56, 30, 68, 124, 121, 130, 121, 84, 38, 195, 85, 64, 266, 228, 27, 232, 105, 79, 136, 67, 167, 50, 66, 172, 173, 30, 225, 49, 182, 25, 93, 43, 40, 28, 314, 99, 49, 46, 56, 87, 218, 80, 85, 26, 16, 21, 20, 27, 152, 99, 228, 70, 50, 103, 49, 44, 65, 46, 61, 79, 85, 138, 32, 84, 94, 76, 52, 68, 175, 108, 109, 189, 116, 81, 179, 95, 73, 46, 83, 186, 101, 206, 81, 137, 38])\n",
    "\n",
    "adamScore = np.array([0.32417157, 0.3236075, 0.31669715, 0.3234331, 0.3383628, 0.29094124, 0.2824574, 0.31537795, 0.3203576, 0.2863951, 0.30760592, 0.31439754, 0.31100094, 0.33154863, 0.29488206, 0.28510925, 0.2847425, 0.3036325, 0.35686594, 0.28584424, 0.35734564, 0.3026612, 0.2639545, 0.26700133, 0.3274023, 0.3079812, 0.31784797, 0.31448033, 0.31973773, 0.31328967, 0.3406593, 0.31198546, 0.2892943, 0.3033445, 0.3044309, 0.28795218, 0.3198657, 0.32269683, 0.30936855, 0.29700914, 0.32982007, 0.28271067, 0.29334688, 0.318613, 0.28108695, 0.29616547, 0.32502204, 0.3452332, 0.31639066, 0.30432656, 0.31218317, 0.29502794, 0.31452253, 0.35201618, 0.31210953, 0.32392386, 0.30951715, 0.32851955, 0.2856663, 0.27621236, 0.31520292, 0.28347623, 0.32140395, 0.29196933, 0.31453755, 0.2851151, 0.29347444, 0.30649784, 0.30142558, 0.2956203, 0.31236213, 0.31517443, 0.2816837, 0.33098647, 0.2874186, 0.29140204, 0.2943058, 0.33242828, 0.27634895, 0.30605254, 0.321146, 0.29049993, 0.28172693, 0.3212033, 0.31476814, 0.2972718, 0.27551717, 0.32316273, 0.32977644, 0.27361116, 0.3159537, 0.30150175, 0.30879086, 0.293877, 0.29113445, 0.31858087, 0.26887345, 0.32885072, 0.2955004, 0.2999737])\n",
    "adamEpoch = np.array([8, 18, 42, 12, 11, 32, 39, 36, 15, 84, 18, 58, 22, 10, 13, 57, 34, 25, 16, 17, 4, 87, 34, 51, 73, 11, 31, 4, 14, 8, 11, 17, 20, 19, 6, 45, 39, 14, 17, 12, 24, 59, 17, 15, 38, 7, 6, 11, 26, 11, 20, 19, 4, 7, 75, 24, 11, 36, 62, 45, 29, 24, 14, 37, 6, 28, 21, 12, 11, 23, 22, 32, 67, 9, 20, 8, 61, 38, 79, 12, 8, 66, 62, 20, 23, 29, 40, 14, 39, 60, 14, 32, 12, 24, 21, 3, 44, 36, 16, 53])\n",
    "\n",
    "gdScore = np.array([0.40343145, 0.3995014, 0.39679018, 0.37810802, 0.39395294, 0.39245, 0.4061372, 0.39538097, 0.385495, 0.40732184, 0.39901033, 0.39628386, 0.3935729, 0.40403333, 0.38848144, 0.39349768, 0.4640165, 0.38744423, 0.37974608, 0.3925574, 0.4050962, 0.39611825, 0.36157897, 0.37148148, 0.4006966, 0.3867069, 0.38846645, 0.3855416, 0.38297537, 0.36482897, 0.43375546, 0.3807426, 0.40070337, 0.37644392, 0.38361964, 0.3909691, 0.3861461, 0.38682568, 0.3791259, 0.39386222, 0.39573652, 0.38411364, 0.38425323, 0.38140243, 0.36409345, 0.38644892, 0.3918298, 0.38952842, 0.3921998, 0.39776242, 0.40449804, 0.37746486, 0.38892013, 0.40174055, 0.40954676, 0.40754405, 0.41206548, 0.40640643, 0.4047953, 0.3728906, 0.40148073, 0.3642799, 0.3826057, 0.37580577, 0.40339816, 0.38950244, 0.42415756, 0.40235814, 0.39622387, 0.39360327, 0.38017684, 0.4007698, 0.37753853, 0.3892596, 0.37608308, 0.45759633, 0.39648575, 0.40029955, 0.3803772, 0.3918096, 0.4068011, 0.41986558, 0.3841825, 0.38125318, 0.38147402, 0.39818507, 0.4134346, 0.37878206, 0.41287625, 0.3744243, 0.37956893, 0.40049177, 0.3916857, 0.37142944, 0.37624437, 0.38583326, 0.4208939, 0.4191676, 0.3898088, 0.3845271])\n",
    "gdEpoch = np.array([134, 218, 189, 169, 151, 130, 199, 236, 188, 202, 224, 193, 109, 144, 138, 87, 52, 146, 229, 159, 118, 180, 157, 132, 137, 97, 148, 114, 135, 165, 112, 191, 102, 105, 189, 168, 201, 104, 121, 164, 158, 150, 154, 158, 131, 125, 190, 194, 198, 171, 109, 197, 108, 108, 194, 145, 114, 174, 146, 203, 116, 234, 211, 175, 173, 144, 141, 154, 194, 160, 219, 210, 173, 176, 112, 104, 90, 222, 174, 178, 151, 148, 154, 189, 139, 188, 126, 167, 101, 217, 193, 120, 177, 177, 152, 147, 127, 152, 129, 95])\n",
    "\n",
    "momentumScore = np.array([0.30512246, 0.32050124, 0.3118204, 0.29860216, 0.32846767, 0.32377097, 0.3194519, 0.3188375, 0.32737195, 0.31416526, 0.31376815, 0.3155915, 0.31906888, 0.33468908, 0.2989422, 0.31819388, 0.3157418, 0.32492107, 0.30315226, 0.29696575, 0.32993317, 0.31338236, 0.28853416, 0.28785837, 0.32536113, 0.31178012, 0.29905328, 0.3069888, 0.31141117, 0.29798573, 0.3457167, 0.30721506, 0.3062942, 0.30499995, 0.30335963, 0.3079729, 0.30322728, 0.31084374, 0.30906832, 0.31301147, 0.31630963, 0.32588533, 0.30555665, 0.3049158, 0.28510648, 0.30788434, 0.33427083, 0.31363034, 0.31127074, 0.3030068, 0.32236198, 0.30453047, 0.3063938, 0.32014114, 0.34994724, 0.31897938, 0.31081125, 0.32688922, 0.32065347, 0.30915087, 0.33021173, 0.29745802, 0.3086957, 0.31498563, 0.30749908, 0.32109466, 0.31107074, 0.32902798, 0.32404593, 0.32046407, 0.29861194, 0.33787188, 0.2940089, 0.30977818, 0.3051475, 0.3120023, 0.3107265, 0.33036223, 0.29562786, 0.3249199, 0.32086077, 0.3154661, 0.300126, 0.30632794, 0.30394226, 0.3227191, 0.28800896, 0.3031807, 0.33319303, 0.28889045, 0.30301628, 0.3150718, 0.3172033, 0.29630026, 0.2998042, 0.32010755, 0.2913881, 0.3362923, 0.3105656, 0.3028073])\n",
    "momentumEpoch = np.array([95, 64, 92, 87, 40, 46, 77, 68, 43, 116, 108, 75, 35, 40, 112, 54, 92, 63, 82, 87, 38, 65, 67, 79, 49, 45, 136, 175, 58, 100, 46, 118, 55, 68, 102, 119, 88, 37, 60, 89, 76, 25, 64, 54, 69, 46, 23, 79, 67, 96, 55, 73, 58, 53, 31, 82, 97, 52, 71, 43, 59, 61, 52, 44, 73, 41, 126, 39, 73, 78, 120, 32, 127, 112, 41, 146, 66, 40, 102, 32, 74, 70, 103, 76, 60, 74, 148, 70, 39, 60, 63, 43, 73, 98, 90, 32, 75, 36, 61, 88])\n",
    "\n",
    "rmsScore = np.array([0.29006058, 0.28402218, 0.29099333, 0.283063, 0.30094597, 0.31597307, 0.28332794, 0.28437674, 0.28395766, 0.30384588, 0.28691795, 0.276377, 0.2808033, 0.27902743, 0.2672417, 0.28558615, 0.26566648, 0.27892727, 0.26648408, 0.28889012, 0.31122875, 0.28696412, 0.26855147, 0.2833428, 0.29345244, 0.27514333, 0.2713269, 0.28643894, 0.27636904, 0.27477035, 0.32560757, 0.27501276, 0.3043946, 0.2783086, 0.31523508, 0.2875137, 0.26740754, 0.2556993, 0.29588255, 0.28068423, 0.3005326, 0.27071145, 0.27468303, 0.2757961, 0.2589961, 0.27337775, 0.28602526, 0.27886572, 0.2755192, 0.26529616, 0.2837832, 0.27461398, 0.27821484, 0.2997016, 0.28743976, 0.28462982, 0.29634243, 0.28239188, 0.34044176, 0.27146813, 0.2917923, 0.2784095, 0.27055234, 0.26745185, 0.2688616, 0.3059669, 0.2770882, 0.2830725, 0.29811418, 0.28193155, 0.31134114, 0.27861995, 0.2575153, 0.29746437, 0.25985563, 0.3453521, 0.28055847, 0.3560163, 0.3752023, 0.28920022, 0.286063, 0.293109, 0.27619183, 0.28042608, 0.26896238, 0.2734545, 0.26086065, 0.28649902, 0.3061967, 0.28311342, 0.2707301, 0.29284564, 0.29340476, 0.29328632, 0.27273682, 0.2737857, 0.26837555, 0.3360302, 0.26765367, 0.27243727])\n",
    "rmsEpoch = np.array([56, 59, 67, 50, 63, 38, 64, 73, 99, 41, 73, 58, 74, 81, 81, 52, 108, 63, 121, 50, 38, 82, 90, 66, 80, 63, 68, 67, 58, 52, 46, 118, 69, 68, 47, 84, 87, 102, 66, 91, 55, 81, 62, 54, 85, 46, 52, 71, 67, 96, 54, 77, 59, 49, 110, 82, 74, 105, 56, 45, 59, 75, 90, 73, 69, 69, 80, 77, 106, 70, 51, 100, 67, 53, 93, 47, 66, 40, 62, 87, 63, 79, 76, 76, 60, 98, 77, 93, 51, 54, 63, 43, 45, 69, 79, 92, 54, 82, 61, 55])\n",
    "\n",
    "# Statistika\n",
    "print(\"AdaGrad\")\n",
    "print(\"Konveregencija: Mean: {}, Median: {}, Var: {}, Std: {}\".format(np.mean(adaGradScore), np.median(adaGradScore),\n",
    "                                                                     np.var(adaGradScore),np.std(adaGradScore)))\n",
    "print(\"Epohe: Mean: {}, Median: {}, Var: {}, Std: {}\".format(np.mean(adaGradEpoch), np.median(adaGradEpoch),\n",
    "                                                                     np.var(adaGradEpoch),np.std(adaGradEpoch)))\n",
    "print(\"-------------------------------------------------------------\")\n",
    "\n",
    "print(\"Adam\")\n",
    "print(\"Konveregencija: Mean: {}, Median: {}, Var: {}, Std: {}\".format(np.mean(adamScore), np.median(adamScore),\n",
    "                                                                     np.var(adamScore),np.std(adamScore)))\n",
    "print(\"Epohe: Mean: {}, Median: {}, Var: {}, Std: {}\".format(np.mean(adamEpoch), np.median(adamEpoch),\n",
    "                                                                     np.var(adamEpoch),np.std(adamEpoch)))\n",
    "print(\"-------------------------------------------------------------\")\n",
    "\n",
    "print(\"GD\")\n",
    "print(\"Konveregencija: Mean: {}, Median: {}, Var: {}, Std: {}\".format(np.mean(gdScore), np.median(gdScore),\n",
    "                                                                     np.var(gdScore),np.std(gdScore)))\n",
    "print(\"Epohe: Mean: {}, Median: {}, Var: {}, Std: {}\".format(np.mean(gdEpoch), np.median(gdEpoch),\n",
    "                                                                     np.var(gdEpoch),np.std(gdEpoch)))\n",
    "print(\"-------------------------------------------------------------\")\n",
    "\n",
    "print(\"Momentum\")\n",
    "print(\"Konveregencija: Mean: {}, Median: {}, Var: {}, Std: {}\".format(np.mean(momentumScore), np.median(momentumScore),\n",
    "                                                                     np.var(momentumScore),np.std(momentumScore)))\n",
    "print(\"Epohe: Mean: {}, Median: {}, Var: {}, Std: {}\".format(np.mean(momentumEpoch), np.median(momentumEpoch),\n",
    "                                                                     np.var(momentumEpoch),np.std(momentumEpoch)))\n",
    "print(\"-------------------------------------------------------------\")\n",
    "\n",
    "print(\"RMS\")\n",
    "print(\"Konveregencija: Mean: {}, Median: {}, Var: {}, Std: {}\".format(np.mean(rmsScore), np.median(rmsScore),\n",
    "                                                                     np.var(rmsScore),np.std(rmsScore)))\n",
    "print(\"Epohe: Mean: {}, Median: {}, Var: {}, Std: {}\".format(np.mean(rmsEpoch), np.median(rmsEpoch),\n",
    "                                                                     np.var(rmsEpoch),np.std(rmsEpoch)))\n",
    "print(\"-------------------------------------------------------------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pareto fronta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJztnXu8XFV5939PEkICQoIcTCAQExRtQZIjCTlEJOH1CiiBVj5V67VaqTY55yTYCrzAOUds66U1N0QRFUVr5fVSNa0KWlQElUBSEylYFCLITQmXAC0JyZl53j/WXsyaNWvfZvZczszv+/nsz8zesy9r75l5fms9z7PWElUFIYQQksSkdheAEEJI50OxIIQQkgrFghBCSCoUC0IIIalQLAghhKRCsSCEEJIKxYKQNiAiJ4vIb0Tkf0Tk7HaXh5A0KBakaxCRe0Rkd2SA/yAiXxCR5zTxWq9q4BSXAviEqj5HVb9VVLksBZSPkCooFqTbOFNVnwPgBACLAVyc9wQiMqXwUtXyfAC3x1xfRIT/TdJR8AdJuhJVfQDA9wC8BABE5C9E5Fci8pSI7BCRv7L7isipInK/iJwvIr8H8Plo++tFZJuI7BKRn4nIgmj7lwDMBfBvUSvmA9H2FSJye7T/j0Xkj0NlE5G7ARztHL9/tP/fi8hPATwN4GgROUJENonIYyJyl4i8xznHmIh8VUS+GN3T7SKyOKV8XxOR34vIEyLyExE5rtCHTrobVeXCpSsWAPcAeFX0/iiYmvuHovXXAXgBAAGwHMYgnxB9diqAcQAfBbA/gOkAXgrgYQADACYDeEd0/v39a0XrLwLwvwBeDWA/AB8AcBeAqWlljdZ/DOB3AI4DMCU6x08AfBLANAD9AHYCeEW0/xiAPQDOiMr3YQA3x50/2vYuAAdF97gewLZ2f2dcJs7ClgXpNr4lIrsA3ATgBgD/AACq+h1VvVsNNwD4PoBTnOPKAEZV9RlV3Q3gXACfVtXNqlpS1asBPAPgpJjrvhHAd1T1B6q6D8A/wYjOy3KU/QuqeruqjgOYDeBkAOer6h5V3QbgswDe7ux/k6p+V1VLAL4EYGHSyVX1KlV9SlWfgRGbhSIyI0f5SA9DsSDdxtmqOlNVn6+qfx0ZfojI6SJyc+TS2QVTI+9zjtupqnuc9ecDeH/kUtoVHXMUgCNirnsEgHvtiqqWAdwHYE6Ost/nne8xVX3K2Xavd77fO++fBjAtLt4iIpNF5CMicreIPAnT8gCqnwEhsVAsSNcjIvsD+AZMbX+Wqs4E8F0Yl5TFH375PgB/HwmPXQ5Q1a/E7P8gjMDYawqMuDyQo6juOR8E8FwROcjZNjfH+fzy/TmAswC8CsAMAPNsUXOUj/QwFAvSC0yF8dPvBDAuIqcDeE3KMZ8B8F4RGYiykw4Ukdc5xvsPMEFqy1cBvE5EXiki+wF4P4zb6mf1FFhV74uO/bCITIuC6+8G8M8ZT+GX76CoPI8COACRe46QrFAsSNcTuXKGYAz64zC17E0px2wB8B4An4iOuQvAO51dPgzg4shF9TeqeieAtwK4DMAjAM6ESePd20DR3wzTAngQwDdhYir/kfHYqvIB+CKMG+sBAHcAuLmBcpEeRFQ5+REhhJBk2LIghBCSCsWCEEJIKhQLQgghqVAsCCGEpNKKAdNaQl9fn86bN6/dxSCEkAnF1q1bH1HVw9L26xqxmDdvHrZs2dLuYhBCyIRCRO5N34tuKEIIIRmgWBBCCEmFYkEIISQVigUhhJBUKBaEEEJSoVgQQghJhWJBCCEkFYoFIYSQVCgWhBBCUqFYEEIISYViQQghJBWKBSGEkFQoFoQQQlKhWBBCCEmFYkEIISQVigUhhJBUKBa9jmryOiGEgGLR24yNAWvWVARC1ayPjbWzVISQDoRi0auoArt2ARs2VARjzRqzvmsXWxiEkCq6Zg5ukhMRYN06837DBrMAwPCw2S7SvrIRQjoO0S6pQS5evFi3bNnS7mJMPFSBSU4Ds1ymUBDSQ4jIVlVdnLYf3VC9jHU9ubgxjF6EAX9CglAsehU3RjE8bFoUw8PVMYxegwF/QmJhzKJXEQFmzqyOUdgYxsyZveeKcgP+gHkWrpiq9t4zIcSBMYtexzeCvWwU3daWhQF/0uVkjVlQLAhxYcCf9BgMcBOSFwb8CYmFYkEIwIA/ISk0VSxE5DQRuVNE7hKRCxL2e4OIqIgsjtbnichuEdkWLVc0s5yExAb8h4d7M+BPiEfTsqFEZDKAywG8GsD9AG4VkU2qeoe330EAhgFs9k5xt6r2N6t8hNQwNlYd4LeCQaEgpKktiyUA7lLVHaq6F8A1AM4K7PchAB8FsKeJZSEkG74wUCgIAdBcsZgD4D5n/f5o27OIyAkAjlLV7wSOny8ivxCRG0TklCaWkxBCSApt65QnIpMArAXwzsDHDwGYq6qPisgiAN8SkeNU9UnvHOcCOBcA5s6d2+QSE0JI79LMlsUDAI5y1o+MtlkOAvASAD8WkXsAnARgk4gsVtVnVPVRAFDVrQDuBvAi/wKqeqWqLlbVxYcddliTboMQQkgzxeJWAMeIyHwRmQrgTQA22Q9V9QlV7VPVeao6D8DNAFao6hYROSwKkENEjgZwDIAdTSwrIYSQBJrmhlLVcRFZBeA6AJMBXKWqt4vIpQC2qOqmhMOXAbhURPYBKAN4r6o+1qyyEkIISYbDfRBCSA/D4T4IIYQUBsWCEEJIKhQLQgghqVAsCCGEpEKxIKQeOFc36TEoFoTkhXN1kx6EYkFIHty5uq1g2Hkwdu1iC4N0LW0bG4qQCYkdthwwAmHn6+Zc3aTLYac8QuqBc3WTLoGd8ghpFpyrm/QgFAtC8sC5ukmPwpgFIXmIm6sb4FzdpKthzIKQenDn6g6tt4tOLRfpWBizIKSZdOJc3ez/QZoIxYKQboD9P0iTYcyCkE6gUfcR+3+QJsOWBSHtpij3kSsYlpBQhMa14lhXJAWKBSHtpEj3UZb+HyFhWrrULIx1kAQoFoS0E9sasH01Jk2q9OHI4z7K0v8jJEyrVwObN5tl9WrGOkgsTJ0lpBMoYviQsTFj4K3IWMM/c2alleCKgWVoyLxu3FjZxlhHz5A1dZZiQUi7CRnweo11lkB5SJgAjnXVo7CfBSETgXK52n1UKjU2fEha/49QXGP1arO4cOgS4kGxIL1NEVlA9Z5jbAw47zxgxgwjEGvXVq8XPXxIKK4xNGTcTxs3mvcc64rEwH4WpHfJ4uNv1jncYLMrFO76pILrcqFxrdavN8FtwLznWFckDlXtimXRokVKSGbKZdXhYZMjNDwcXm/2Odz97ZL12o3gn79cDm/rFrr53goAwBbNYGMZ4Ca9SxGB5UbPUUQWFImniNZjl8MANyFpZO3x3KxzZOlEVwS92ju7ng6PvfqsMkCxIL1LEca63nNk6URXBFmGEmmXgWz2dbN0ePR7t9uOibY87Mn+LBQL0psUYayTzuEaHbuvS9wkSkVmQbk1a7939uOPm/V2DWvequsmtfzcMqiaZ7JxY2XoE/ZkryZLYGMiLAxwk9yMjlYHlG3AeXS0dt+4IGnoHAMDZmnkvEVRKpmyuEH0oSGzjIw0HuSvhyKSC+q5lptEUCrVXnNoqHq/ViUctBlkDHC33cgXtVAsSF1kMdZpouIe4xqdog1hXmEZHa0Ig2sArXj4RtMVE/+eiiZ03WYKRei7iLt3d73LhUI1u1jQDUV6myw9nh9/vHbwPdc9YY+x79evb3xgQJ+8bhuNXFAbNwI33FD92ebNpgPeunWmfL6bxj9P0e4h2xlx7drq7TNmFJsJlubqS7t3gB0TXbIoykRY2LIgdZNUY0+qnfv7uTXjUim+hpq3tprVbeOft1RS7e+vLoffegi5adzPi3YPuef0y9bfb8pcNHHfbyP33qj7sIP6fqAT3FAATgNwJ4C7AFyQsN8bACiAxc62C6Pj7gTw2rRrUSxIXSS5mFyD4YuF66oJuTd8Q+gb51D8IomQYXONa9x5R0bCYjEw0D6/fej52PVWxQhCYmDdc2nfVZ5Yl/3cxcaK7DX831yLabtYAJgM4G4ARwOYCmA7gGMD+x0E4CcAbrZiAeDYaP/9AcyPzjM56XoUC5KbLDX2OAMa8uv7xnzhwupj3RZKPUaxXI435HEtjZDIuWVwDVfcMUUZMPc8vohZ4corov55Q+txhIz+0FB1GULnzhOcD7U4rTAODFR/H/Z9Pc+gATpBLJYCuM5ZvxDAhYH91gN4HYAfO2JRtS+A6wAsTboexaKHaaRJnzbkRpzBDRkH35iXSuHj6xWKUMsiqdx2/zSjFBegL7Jl4RrNuGvU44Lyz5u3ll7PbyftNxPazxeWUMuzkYpEA3SCWJwD4LPO+tsAfMLb5wQA34jeu2LxCQBvdfb7HIBzAtc4F8AWAFvmzp3bnCdJOpu8LoEQvpH3jWdc6qlfA40zIEnnz1q+OKOTdN68hjTpOo1kSPnnydrayhInSBPEkZHacxYVe8nyncb9LvyYVr3CXEDso+PFAqZD4I8BzNM6xcJd2LLoQfK6BNLOETLyvrEMGc80I1tETT3JnZFWu01aT7uOFcus/UbiCD3npNhA1kpAmqtt1qzq6wwNmXtpxNWTtWXh7h9yuYXEIs/vooiKknaGWCS6oQDMAPAIgHuiZQ+ABwEsphuKJFKU2ySL2Ng/pB9IHhmpPlec/9vt01CPmIXuuwiRDJ3X4rqD3OfrB8H9VkvSOe22pBac+z7P/fnntcvChWEBCbWU8jyrvGWLcx+GKhJZy1Xgb6ATxGIKgB1RgNoGuI9L2N9tWRznBbh3MMBNVDXeKDezduYHgeOCsSFDUW/tL834FlSrzHSeuFaBjclkaXlkqY27opO1EhDnkrM1+MHB2u227EnPt9Fn5pctFLOwZbOi4Vcs0sjbwomh7WJhyoAzAPw6yma6KNp2KYAVgX2fFYto/aLouDsBnJ52LYpFDxD68xWR6plkmItydSWt++QxRnnOGypX1nvzs5d8A5fkqsvagrNxB9dllFQJ8K+1ZEn1/oOD4d9HKAOsHrdalmcf912OjCTHlPKUo96KUkRHiEUrF4pFjxBXk2xWJ7K4azYrY8Xv+xDqC5G37GnrafeW1Lkvzp3iGn5VYxz7+yvuO9cw+sIQOl/c/dtOk24Lor+/VjhCbqAiXHhZSKuMJO2bdt5uaVm0cqFY9BB+bSoUbC4yV93W+hqswaXixkeyGoEkY5NUq3VJ623uuk5Cxjf0fLKMj2Vf43rJAybukCaYtmUSd46QAM2e3bCRbSvdFLNo9UKx6BHi/OdJtbW853cZGck23Eej+H9234CH+iBk7X3uC1B/v+r4uDkmS1aVK2JZWxZZs8B8V5J/blvOLJWAkGjNmlU5d6lUCXr7LY+JJBSWbsmGavVCsegBGqlNZWnup6WnFpVRE1eGpJTKUDA47VmEhNUay/5+1UsuqV5PqsGXSumCEIpZZDHIcbGnUKsx7Xn69zs4WBvkDsUyWjHSbjNoNG6lFAvSrdQzLk+WQGKS8fVdFkk9oEPree7jkkuqr5VkwLPGG/yyuymltvadNMaUe51Q57dQNpQf/E4S9JBY5BFlXxjTRMt3kWUdE6pLoViQ7iVPJkqoV29cimKc8Q359F1/e15XQJIwhVxCSeMmJcVS4pIB0mrv1vVjKZXSe4P7180Ss3D3j+sln9aBzi2X6zIcHa20hvxzx2VdWcFrdsC7w6BYkN7Gr1lm7fzkG9+kYHOjbjH/vG6WTigryh7nniMuLpAWA/HdMvaYOXNMS8pez7rili/P14Kqpy+CjSv461meoRUAt+/C4GB8p0hfIOvp2FmAC6gToFgQklSzjhOKJAOeNS6QtUbqC1Na/n/cYHxLloQNYVJ2Vah1sWpV9X27MRt/roms95e0bmkkUBvnwnJbC1my5ZJaaEWXucOgWBCiWmsE4gx6mmsozaefx9D41/NdTv5+ceWztWZbPjdWYMvn19T9LCDf7TU4WLtt9uxK6yPuGdSD//z99TznCX3Htl9HWm/tvIJfT4uynlZIi1ouFAtC0mqdcSmivlH0+yVkiXMk/dHz+vSTruW3fEL3lha78YXOd1lZ107esiatF1UzD33HftnTkhHqcSXmEZh67rWFLReKBeltQv5s31jGBYyT1uOukWRoihzJNU9MxT/Od2OF+liEWhbuOEZZat6hFGT33tw03CSRS8N93qFyz5pVyfxy3XG++Nc7/EeWFmW9rZB6Y2F1QLEgpN6JcbIIhi8EcW6qpD9+3tz+JNdVVjdYnFvLH3TP9px2YxZZruGf1w7z4V5nYMC4w4ro6JjWc9t10yW5FZNaj2n3mSaged1c9R5TJxQL0jxa5EsthLxlzdL8T6s5N+oTj7uPpJhKPee292Fr1qWSyYayc0DY9NP+frO9Hp++b7Bdo+63VtKyn5KejStKSUsR40LV21rIKuiNHFMHFAvSHLooC6SGLEagER93o3/8pB7m9Rq/UIqu6xJyWxyN3K8vCnFDh8e5B5PwW2tJrYw8424lkXVod798bFm0f6FYtIAW+1LbQpY/ad4/cpF/fP+YoobbTipfPb3m/fP5BruRHtshQkH8wcHwgIFpbrusrdGk/UJpyzY7izELikVP0MIaT9vIGrjM0lJo9I+fJ36StE8aafeTx4D6LRXfPRRK4bWLH/TPew+ukbbC4RvpJLddUXO6p8Vt0s7JbCiKRVfQIl9qW2hGy6LeP36rDEbRFYCkWrUfs/Br+XlGmU26H78cquluu9A4XPXW6NOeadaYRdJ6QVAsSHPo5pZFs2MWSev1lKVV91zveVXD2WI2bTgUP0gb/bbecljS3HZFuwwnQKWqcLEA8HIAfxG9PwzA/KzHtmKhWLSAXohZ1JMNNVFq/HE0+35CQukLwvh4/VldRZTHX2/UyE+gSlWhYgFgFMC/Afh1tH4EgJ9mObZVC8WiRXRzNpSlVXGCrGVpRe20VffjEsruandNvAgjP8EqVUWLxTYAAuAXzrZfZjm2VQvFooW0w7D0IhOodlo3fqWjnfdapJGfQJWqrGIxBdnYq6oqIgoAInJgxuNINyKSvE4aRxVYswbYsAEYHgbWrausA2a90567anWZ/PUQIp1zryLAzJmVMoiYV8Bsz1OGsbHq+7fnKuI+6nnOBZBVLL4qIp8GMFNE3gPgXQA+07xiEdLjiAAzZlQbrrVrzWd5DVcrGBsDdu2qlNUKwMyZ5rMkijTSjVKkkW9GpaqR59wgmcRCVf9JRF4N4EkALwYwoqo/aGrJCOllxsaAJ54wAmGNwnnnGQFpslHIjaoxYG5LwG0pZKn5NrMmnpdObTkX8Zwbu35qvGIygB9l8Wm1c2HMgnQNnRYgzRrwb3fMoRdownNGxpiFmH2TEZHrAfypqj7RPNlqjMWLF+uWLVvaXQxCisH141tcN02ryOP2UAUmTaqsl8udUyvvJgp+ziKyVVUXp+03KW2HiP8BcJuIfE5ENtql7tIRQpJx/faWVguF6/ZYs6ZawHbtMuvuvmvWVB9vjyHF0c7nnKX5AeAdoSXLsa1aGnJDMRWUdBqd4tbJO/xJJ7jNupUmPWcUmTqrqleLyFQAL4o23amq+4qXrjZQRHaBtieVre3XJs3BrcG3O23WtnBcd5h//U7KZupm2vycM4mFiJwK4GoA98B0zjtKRN6hqj9pXtFagNvMBurP4mhTKltbr02aRycZ3zi3hy8YnZTN1M208zlnaX4A2Argxc76iwBszXJsq5a63VCNNPfb2fxm07/7abd7lL+xngAFZ0P9UlUXpG1rJw1lQ2kD2QWuy8DSqqyVdl6b9AZsvXY9WbOhsorFVQDKAP452vQWAJNV9V0NlbJA6haLkMEdGgLWr68Y3DR3VCNi0yjtvDbpDRgX62qKTp19H4A7AAxFyx3RtomNH0gsl4GBAWDjRmD16opjas2a+FpUnE83gwjHni9pvZnXJiREp/ZoJi0lq1hMAbBBVf9UVf8UwEaYnt2JiMhpInKniNwlIhcEPn+viNwmIttE5CYROTbaPk9Edkfbt4nIFXluKjN+IBEwYgEAmzeb17i8ciAsNsPD1XnpeRgbqz4uSaiKvjYhhCSQdSDB6wG8CqZzHgBMB/B9AC+LO0BEJgO4HMCrAdwP4FYR2aSqdzi7/YuqXhHtvwLAWgCnRZ/drar9WW+kbvzsgvXrzevGjRX3TlwcoMislbyZWZ2UMUMI6Xqyxiy2+YY7tM37fCmAMVV9bbR+IQCo6odj9n8zgLer6ukiMg/Av6vqS7LeSKHDfeSNAxTl060nYE1/MiGkAYqOWfyviJzgnHwxgN0px8wBcJ+zfn+0zS/oShG5G8DHYOIhlvki8gsRuUFETgldQETOFZEtIrJl586dGW8lhbg4QLlcu1+lIH7B6rt2PUM8pF3bxl3i1gkhJANZxWI1gK+JyI0iciOAawCsKqIAqnq5qr4AwPkALo42PwRgrqq+FMB5AP5FRA4OHHulqi5W1cWHHXZYEYWJjwMsWlQRjLSgd6PXd2kk/jA2BixdWh2sX73abGPaIyEkB4liISInishsVb0VwB8B+H8A9gG4FsBvU879AICjnPUjo21xXAPgbABQ1WdU9dHo/VYAd6My1EjzCMUB1q4F+vuBbdvMfAKuoISC3vVSdMBaFXj8cROot9ldq1eb95s3m8/YwiCEZCWpxx6A/wTw3Oj9MgAPAngDgA8B+HrKsVMA7AAwH8BUANsBHOftc4zz/kxEPQkBHAbTjwMAjoYRmecmXa/Q+Sz8nqmlUmsGdSt63t5yWXVoqLrcgNnG3reEEC2oB7eIbFfVhdH7ywHsVNWxaD0xwB3tcwaA9TBptlep6t+LyKVR4TaJyAaYLKt9AB4HsEpVbxeRNwC4NNpeBjCqqv+WdK2mz2dRLgOTnWzhUqk6CF4URQes/WA90LqAPSGk48ka4E5LnZ0sIlNUdRzAKwGcm+NYqOp3AXzX2zbivB+OOe4bAL6Rdv6WMToKbNpUvW3RImDFCuCDHyz2WkV2gLIxCp/Vq6t7qLtweAdCSIC0qvFXANwgIt+GyX66EQBE5IUAOnbWvEIpl41QbNtmYhelUiWGsWlTbZZUp2CFYmM0R9XQkFmA6h7q/jFZJ7shhPQUia2DyG10PYDDAXxfKz6rSQAGm124jmDSJODgg4G+PiMQ1hXV12e2N8MVBTTuChIBDjnE9EgfGKh0NgRMgPuQQ8KtGJu6u2FDpb8HByckpOfJ1ClvItC0mEWoo5ylWUY0zhU0Y0a12yuLgNjv1x0U0V2PO6YTBidk7ISQplN0p7zexU2fdenvN9uLNl5JriDX7WW3j47WHu+X3x8mJE0oOmFwwjzjZBFCmk+WlKmJsBSaOusSmhypnvTZPBPZhK7Z3199Tft5f79J7VWtpPg2kmrbCZPddEo5COkBkDF1li2LNESM+yfUspgxI1vLIm8tOTTsx9atlU56kyaZVxtHscORLFrUWDA6bnDC4eHWDk7oXte9X8ZOCGkfWRRlIixNa1mUSpWObcPDtetptdx6aslxU72WSrWtG9vicNdtS6Ne2j2dp3td997YoiCkcMCWRQGMjQEvexlw880m7XTtWlOLv/lmk2GUpbadVEsOxTxsqyNufCoXm8LrsmVL4xlaWYPmzcQ+BxfO00FI26BYxKHO2Eq33FIxXhs3mvUlS2qDy3FYV5bL2rVmrCnfFZU2PtXAALBwodnXFwoAOPLI7OXKQjsCzUmCScEgpD1kaX5MhKUpbqiixlYqlcLuojRXlMvcuap9faqDg7Xl8ZfZsyuuqEZcN0UHmvO4t4oeJ4sQEgRFjA01kWhqP4u8Yyv5x9tasu826u83ges0t5ENXm/bZloV5TJw223x+y9aBNx6q3mfNlSHpvRlcMtvqSfQXM8wImllI4Q0DPtZ1IsrnpowtlJWkXXdSlu3Vn+2YkXyDHiWSZNMLKK/H9i+PVkoAODkk81raKgOd3iSsbHqewm5mOImZEoqb+izeoYRKXKcLEJIY2RpfkyEpRA3lOv6cLOeAOP+cddDrqik9VJJdcmSaneRPafvWolzwYyMpLugkrKjRkbM+shItYttYCDexRTKzBoYqL7/LC6ikEuPQ6UT0nbAbKicuLXfpUsrWU9AJaA8Y4bJihoYqA1Yj4zEB4JVzTlvucV8NjQEDA4Cl11mAubuRERJtfBPfjL9Pvygt9sPww6IeOWV1bX5zZsrWVpDQ9WuotWrqwPNQ0PVEyplbSXEjc7LHtmETAyyKMpEWAppWYRqv4ODlaByf7/q+LjqsmWqCxea7aOjZltfX3Wt3O+bYVsVfs3fDUa75fBr8/b8s2aZ606fXvlswQJT209rdYSWpNp+uWzOa1se7jPyr5cU8I5LFLDPY2Sk8e+OEFIXyNiyaLuRL2opLBvK7whml4ULK+Ixa1a1mNjPDjggbHitsfTdUEB8Bzq/A94pp1RcSuWyMbAHHKB6+OFGrNyMqyVL4g20n00Vt48rWL67yN6Te4zdFncvSVlcQ0NhwSSENJ2sYkE3lItqbUcwy/Llxv102WXAH/5Q2X7ZZSboDAAvfGH4WNs/Y2Cg9jM7r7fL6GhtB7ynngLOPLOSOfXBDwJPPAG85z3A+99v3EZnnmncWyedZD5ft652mJIbb6xe37ixMk+Hne/issuqOw+GJkpaurR63c7xbd1uFttP45BDgBNPrL3/2bOr5ze3S5ZBEgkhrSOLokyEpe6WRSiQ67uKkmrhdjn00PjPbG09S1+L8fHK9v5+1YsvrrigrBtseFh1+XJTHlsm+966h9zPQq4fP+Bua/d+GUNBez/Q764vWVLbogJUTzwx/rna7f49uMF59rMgpCmAbqgM+FlHNlso5C5yXVFJS5wLyK5bA+gKk/XZ2/JcckmtYe3rM9t9QQuJgjX89j6s8bbrtozlshGxOXPi7ycUixgdDV/XCoV7XJLA2mXlytr94kbZ5aizhBQKxSINxwD9/i8v0iefcAzSwECtj92NWSSJxuBgbdqtPafrm7eCYYXCthisQdy3L/4a9jxxhtgKwcCAMeBu7XxwsDqgHDqPvb+kXuZJcQvn5I2JAAAe2ElEQVQ/OJ9FMPzFFdQk4Qp9r0nrhJAqsopF78Ysos5mv33nBzHns2P44oxVldTRm26q9dFPm2Y+W78e2LEj/px2zKeNG83+dlyjzZsr+1g//tq1JrYwOgosXmymabVjIO23X/33ZuMH9po2BVfElP+JJyopq6GObtu3m7Jv3Wpe3QETVSv7hQb6A2o77bnH+PT1hbefd555Pi5pvcbbPWGSf59J903IRCOLokyEpe6YxbJl2o//1EW41dReV62qxAhCcYBSKTlNdWjI1NxD7q1QZzg3TtDfX9ui2Lcv7OsPtW6e97zq9VWrKseGOt65tfeQK2tkJNz5zt5f3HlWraot24IF2VoUCxdWu+zSWhahmFPRrqssrRWOZUUmKKAbKgNR/4gNGFRAdTuOrximadMqrigbnF6+PNkPf+ihJq5gz61abUj9+TDcJS34C1Sn7Pr7hAx0XDldo5Yn/uDehxUM+2rvy42LhM6ZJBQ2ScBNBkjqXe4b6FCAvtFe4llEYCLM7Ef3HImBYpGG84d+5LnH6FTs0dVYGza4vtEcGamt2VtDZw1of7/pvDc0VJ3hFGfIx8er1/2YyapV4cmP7LGlUnwsxT+X2yfC70Dorg8NhftluJlbbotifLzS6hocrL2nLLEL2wLzg/O+kQ4Z5FCLzw6nUo+xzCMCcRNWdUKMha0ekgDFIgv2TzQ+rmfjX3UO7tNynBHz3TGhWvLChdWZTLaTXqjV4BvhNDFJcn/Nnl1tXENlD7UaQm4ye39ueq5fbmt8rTD65bK93mfPDpclroOedf/FtQpChjVufvSk+81jLONEINSJMBTwz/L7a6YRnwitHtJWKBZZiWr9V+NtCqjeikVhg2PjFW7N26aghoy4b/iSzunGLMbHwwbfbzX4558927Q+kmrvvqG21/fdZKrJ2VbuoIoh4++u21Rht8URcpmFlqy1/7TzNJqG61/Dj0nVOwRKq4x4va0e0hNQLLLgGOpHjlumk2VcL8KHzGMJdbLzDZ+tjedJDR0crA6YW7dWltFgFy40ndtC5brkktpOdr4RL5UqIhAybFY4bMsh7b5sgD6t859qJW13dNQsfsts1aralkiaQQsZwVALzQ3I5zWWSdcI9SWx36nv2gsNh9JKI5631dMqGEtpOxSLrCxf/qxRO3XeDn3JoQ9UhMLGKw4/vNYQugMHjo/XBp/jWhZuHwnX3eC6NWzQOZSFFKpJ2xaPFYCFC8NlsvczMlIbT/D7eZTLphwrV8aLhd/C8D+3QX7V6swqKxyN1P5D+yTNaliPsUyq/fui5A7bbsXQrwAMDNS6mFphxDu1ZcFYSkdAschDZKjXrTNP5K6Vays/4qSa87RpZoC/vJ3OQgPn+YRqXFkMYuhz3z1lDVlSa8C2eJLcaW56bdKAhG6A39/XFZy+vtq4SZLhcI1NUossVL6sxjLr3CJu6yFpOJRQJaCZRrxTYxadWq4ehGJRBzt2mCfy8Y9r9R963z4jDHHG9eKLw8bXBritsbauF3/I7yxkNUJ+xpRfg3fHaLIxkpAIDA5Wyn/oobXncQPq7meHH14tMG5fj5Dw+H1NfMOc5blY/BaZzVrzx8uy667QxJ0ztJ7WuzxN1O0+rTKWnVqD79QWT49BsaiH0VFd0He/nnKK96ePGztpyhTVvXvjO7YNDFSEYmSkYrzy1J698lW5O0qlSkc8O6yHHwj3y+TGSOLEz12Pa1k897mVexwZqbi87PV9cY0T2yRDEWe07asbD7HrIWNss6Hss3BnKMxrSLMa+ZC70L+fVhrxTo0NdGospYegWOQl+qOOYEwnSUkf/oNjBPye0b4xtq6YUO3b9ekndXLLWrt1a8JJQei4FFV7Lb9FEZehtHdv9fq+fRVhsILh3m+oT0mcYPjC5rcSQoZ0+XLzumxZ9Si8NmV5+fLqZ+YH8t24Tb21+zQjn6VlkfY99wJsWXQEFIt6KJd1y5v+UQHVL+PNlT94XM/jKVOMkXKNQ8g/7ncsS/tz5KlxhgyTG0S3tf80H36c6CQNmuhnZtlU4tAgiHlaFlkCy7bF479aAUl6NnHXcstRT0wpdM2kmEUvw5hFx9ARYgHgNAB3ArgLwAWBz98L4DYA2wDcBOBY57MLo+PuBPDatGsVNVPevr1l3R+79f34R/N4/Hkf/MUGhP1ga1yNOa3ZnfdPFGcQy+VsPvwlS8LDiCxZUhGKWbOSR8F1n0XcWFZxghE3nEeSEU9K1bUtjaT0X99g+8FqN4U4jqQWQchd6GZDNWoIu6U10qmxlB6j7WIBYDKAuwEcDWAqgO2uGET7HOy8XwHg2uj9sdH++wOYH51nctL1CpuDe3hYF+MWfSV+UDFQ4+OqBx5YbVAWLDDG1Hd7JLUesrYs8uyX5IbyXVb2nNaQ+W4q/x6tiITGefJjGW7QPPS5fx9+X5OQoQgJqxXA0Ln94Ufi0nrdbLTQeFJpwfas40XZ8lqRi7vPPHSbge0W4ZvAdIJYLAVwnbN+IYALE/Z/M4DvhfYFcB2ApUnXKypmoYD+5XE/1UMPLWt5KCan3i6rVtW2GuJaBWkxC9/t4Wc1hf5EIaPru55i7rGqBRE3QKFf7qQe5jZobGMJoefW12daHrYsNnbgli+prHFxmDhBihskcc6cWledL259fdXuLLe1krXlV7Srha4b0gQ6QSzOAfBZZ/1tAD4R2G9l1HK4D8Ax0bZPAHirs8/nAJwTOPZcAFsAbJk7d27jTy1yH1x+6lcVUL3vd5FBO+KI2lq3nVHPdyuMjlb6F6hWG0XfVeUHbe0xodpunDEIdeDzO/y5+LX1tD4i5XJ19lSSwbb37Q60aFtmroD4E0CFOqrFGcUkYXNbGO6zsOfxs8Bsb/q4c8YZ5DyB2aKDuAwKk4KZMGLhfP7nAK7WHGLhLoW5oYaG9KdYqoDqpm+Xw6Ou+h3c/LiAPyjfkiWV/gSusRkZCY/N5BvWtNpj1qZ8yNCE5gYPGaK4GfXcmI47XevAQHXrxhWMLLXikLslKY05zni6biA3SO7HofxAftrQI2mxJ/+5p+2bxx2T59qEpNAJYpHXDTUJwBOhfVvihrKUy/rk+z6ggpJeiovNI7I+ft9Yur780ERGvssmbv7tkBH3x1XK6pf2DZpvbF3Dl9Sq8N1moTGw7BStSVlX7vWzdGZzy+/2m3DPERK3oaHwkCXuMwk956SKQJxBLrplkTf7jS0LUiCdIBZTAOyIAtQ2wH2ct88xzvszbaEBHOcFuHe0JMBtKZf1xfiVnobvVgyFHYTP/ZNmncgoVHv3a9l+bTE0BLb76m9XrTY6rr/eusr82r5t9fi1aF/ELrkkOZAeeg5xYhdXK7ZDnbsBbzeDyD1PXDmyBJD964eW0PeVJa23nphFO2MghGgHiIUpA84A8OsoJnFRtO1SACui9xsA3A6TOvsjV0wAXBQddyeA09OuVWTLQgcG9G/wMd0Pz+jjmFE7TIS/+C4Mv4NbqZTcQS4umOsbiqQaqF+jj8v1D8U3Qq0Nux7X4dAvZ2hiJvtZXLnsdfyOfEl9E0K9z91e8fa+4r7bUGZUyNWYZJDztATyzrSX1lrotmwo0nY6QixauRQZs1BAf44BBVS/+Oov1hoS3zCmBYnjFutm8Y1RKK3UFas8gdc0AxQXG/AFaGAgPG/F6Kgx1H7rJDSRkZtVlfQaJ5qN1Kz9ewmVxy5uC8c91jfw/vmTrp22b94YSNZrE5ICxaJeIvdNaXBYj8Tv9Az8u5k9z023DBlkXzCy9FiO69CXp+d1WuDVFye7j0vS8OhxMQJbHr9VEBcwd2vkcbGL0KCGflmdIeWryuf2d0n6bt3n7N5bqBXkP9dm0eo4BMWGOFAsGiEyJufjwwqovhC/1o9+pKy/fzCQueTOIeEbulDHtMHB8CirvmGKE4WkGmhay8LOC57FleXX3N3Fz2xyBxJMatX4A/25n7vT0bqLHSBRNZwZltdn7+8TmvUuKbictF4PrY5D0I1FPCgWjRD9gfZiin4Jb9FTcIMCqlOmlPWcY/9Lrzv7k1oad2rebp+Ccll18eJ4o+n6+PMEYuPcTCEXlF+7d9fdSZvK5Yprya67GUVxoue6jWz6b9xAhL67x5+aNnSNwcFq8bGTOdlge1oflLxGPcv+zTSyrTLgDJCTABSLeon5Q/0KL9b3n3C9HnpoWQHV+fNV/+7vVB94QCu9luMMoWsQL7mk+lohQxUX8M4TeA0Fgq1x989tO9PZFs+yZbVlj+uNbQXH77QImHkwrJjETX7krtv5M9xlypRagfL3cZ9hMwxvK4xsq1xDrXZ5kY6HYtEICQZnzx7Va65RfcUrzNObPLmsZx39S/0OTtfxwdVm31At26/Rq4ZdIG4g2w/EulN1euV6FvdPH0r1DW1fuLB6AEDfaC9cGL6nLMNv2P4mJ54YTtG1o/baVoN/3aRzJ7Wukox6PYa5m4xsnmA66XooFo2SwaD85jeq55+vOmuWaW0chXt1NdbqNfgzvQdzTWDcFwxr6ENxCz/zyE13DWVDhWId7vuQcYuLDfiiBphhTpJG3HWX6dNV99+/dntSHw0rGLYPS56sMj8gnZQR5hr1rKmsoe++G4xsN4keKQSKRQvZu1f1G18v6+n4jk7H/z77H5yFh/QsfFP/ARfoDw8+S5/Ec2qFIanzV9wfO5RBlTVQ7c77UCqpHn989fkvuih+xFhXwPIuq1aFB+tzxdAVgrgYiFt21dr7tttCRj1LyyNpzu2JbmQZsyABKBatxPnT7cUU3YqX6uV4n779jzbri15Ufta2TMK4Ho/t+mGcb1od1vAn1VZ9w2c7yaW1OuKMnjsqbNKSZZ88S9z5Qv0xsrQy/H4QaWm5IcEICXScMc0zrlUnw2wo4kGxaBUhw+F1qnv0kbJ+76xP6eiyHz6bWXUFzq3PsMUFin0Dao/3y6oaH7Nwt61cmS4ASQY91M8ky7zc1k0Xcpf191dm5/OHLfGNuv/qP1dfoNPEJE+KbafTqmA6mRBQLFpJUi9oSyQM45ikp+M7uh+e0Z8e845qQ+aKTlKfgrhaer2B2pUrw+e0AwX6tX+RdCGJEwM/aO3v09+vevHFlfIsWWJcUv4Q6UnDmrtG3k9RdsXcFT33+0pyY/nXJGSCQ7FoNUmGxBOCxx4t6wtmPKyz8aA+cNyr40eXTfKfh4xzWn+DkAC5LQg7LarNWlq4sDamEVrispb8Y2fNSo5FTJ9eKYc979BQJcjuDnAY9x0kGXkr4r5IuOtprT1CugyKRafhGf7bflnWA/fbo0uPvFf37HH2S6u9hvz5oYEC44QmNNHSnDlGIGw/EZupdfDBFQOetixYkF9M/OXQQ/XZ1kRocD9/JF7/OWUx8nGz59kh5osKALMVQiYIFItOxDMYX/uqCX6fe26O4930WtvvwnXPDAxUZ+7EubhUK4IyMlIZhsSur1pV6UVtjbi75HFN+WM+DQ7G988IXcttJaQ9lyxjPNkldP4iAsAMIpMJBMVignDhheZb+PSnMx7gZv24w2i4kxAlZQX5Li/X7eO/Dg1lc0P5QhAaQtxdX7Uqe2sjqysomhI3cT6MPBlTLnlbFExPJRMIisUEYXxc9bTTVPfbT/VnP8t4UJY0UNXaHtF2fKUskwkND9eOKGvFIM6g+66oJUsqomD7RrjH+62TUEDcnY0wydi68Qi7X5x4ZsmYaoSsLjFCOgCKxQTiscdUjz5a9fDDVR98MOfBce6UUBZTaMgR/3i75O1RvWpVbeC9XK6UY9myyvWsCy1tqBBXoEIZUO4zsK/19LFIG9SxHuK+F0I6jKxiMQmk7RxyCPCtbwFPPAGccw6wd2/GA1WBNWuqt61ZA4yPA5s2AY88AvT1VT6z6x//OCASPt4yeTKwcaN5v2ABsG9fclluvBG49trqbatXA7t2mev295vriQA/+xlw0knAZZcBQ0PJ51240DwgkfDnY2PmHuy5166t/nzdusqxImbd/3zSJPM6NpZclqzEfS+qxZyfkDZAsegQjj8e+PznjR0dHs5wgDVIGzaYA8pl87phA3DiicDrX28M9COPVB/3vvcZIXCP7+83n9lXlwULgLPOMuf0ed7zgMFB8377duCWW8z6kiVm28aNZrH7fPCD5nXSJCMAWYRixQqjohs3GuFxDa6q2bZhg7mXchlYtKj6HK6RTjLicWKUl6TvhYJBJjJZmh8TYZnIbiiX8883XovPfCbDzn7WjT84od8r23XLuOm1dggQd+RZ1wVk4w59fWbsqFA67YEHViYqCqX22lfXJeTOpeHOCxJa/GMtoaHY7VDwWTs5Fh1PyNJJk24p0iGAMYuJyfi46mteozp1qurPf57hgKSOd/5ijaqbNeQad9sZzw/8LllSm9Lqx0NsHw2/45tdQvNaDwzU9m+w82n4AmCNbdwUtCFRdFOBVauzoWwZssQq6smOcvfxp6plKi3pICgWE5hHHzWTKx1xhOpDD9ZhqPyxn9zZ7/r7Kz2i/ayhpMBvaH7suP3d1oRd/NkE/X4Q1pCPj8cLgDvqrT1PKEjujkrrj+kUGv4jiUb7TDCVlnQ4FIsJzvbtqgfs94yefPjd+syeHIYqLePHDpfh7+MbaNc4xqXY2hn2XOPnj4rrtyxCLjG37EmthbjhOLKIVz2GuihDz1Ra0sFQLCY65bJ+5bQvKKD61wt+ks1QhXpqh3pu232TxMJ19fj9E/wavCtiSf0d4gTJx7ptQm6oUNnjpo/1n0k9hrooQ89UWtKhUCy6gXJZ/3bRfyig+lH8rf4KL9anV/5Nek/mNLdJXEshbvY5N4hsRcfvyOb76EMBXn+0V1/A7L6h1oXbMztP2e05GzHURRzPlgXpUCgWXcK+vWV9Da6tsjOzZ6uedJLqm96kesEFqldcoXrttar//d+qTz+tyQHZkEH15t8ICkya79/Hv6bf2nBbK27AOTSRkxvf8M8TKrvfCklzfyVRZMuEMQvSgVAsuoHIsOzFFL0RJ+uX8Bb90NJ/13e/q6yvfKXqC15ghgnxK9lJYrJ7t9af2llPVpAlLc3XNaKhzCkbDwlNJ+uW3c08coXCTpbUjpgFBxYkHUxWsZjSvh4eJBHVZzt37Tc8jJevW4eXr1kDbHg9sGQY+IHpmVwqAQ89BNxzT2X57W/N6y23AF//uunQ7XL44WOYN08x7y2CefOAefME885Yj3nzBXP3ANOmIdxJzd/mr6tWb3PXx8aq1ydNArZuBc47z3RY27DBbB8ervSydntez5hhOue5PbDXrAFmzgTWr6/0SN+1y3Tgs721N282+w4MmFd7/MyZ6R3xRMx+tkxuD/Asx1v8e7fnKaojICEtQIywTHwWL16sW7ZsaXcximVszBg/a1isgMycmXloilIJePDBajFxl9/9LiQmiESkdpk7NxKTosqqaoTDUi6HRcjtFb1uXe26Pcbd1zI0VBEUu0+cqNWzTsgERkS2quri1P0oFh1Okw1Vo2Iyfz4w7/mKed/7FOZ9ax3mrlyB/S/7p3hj7t+bb9jj9s+7b5oAWQoQZEImMhQLUgj1iEkfduIAPI3ph0zDtLnPw/TpgmnTgOnTUXndXzF9602Y9svNmL7keExf8RpMu+7bmH7jdZj2qlMw/V1vxrTpUntM/4swDXswHbsx7cmdmH6AYPJk5+L1ClBai4WQLoViQVrC+LgjJr9V3PPOUTyEw7EH07D7je/Enj2C3buB3buBPXu818eexu59k7G7NBWq9RvlKVOMmEyfrpi2+3FMf+phTOs7CNNfeASm3X8Xpt//G0x7wRxMH1iAadMCAvT9b2Pajd/HPNyDM/A9CgXpKbKKBQPcpCGmTDFxjLlHKZb96xoATo1+9vYUo3sAoAqFYN++SESeVux5RqpFZbdi94YrsWfTddj92j/BnnPeit1f/gb2/Phm7D7pFdh96unRMYI9N99rzvHHLzbrR78Qj+ybij1PTcPuW6RGsMplAXA2gLPxSvyHEQsKBSE1UCxI4yS5c4Bk4ysCATB1qlkOPji0nwA/eQiYPxdY91Zzrne/AVhzEzDzFmDsDGffl3pxHQF0bvD6qsD4PsXu4Quw54rPVz5Ys4aCQYhHU8VCRE6DqWpOBvBZVf2I9/l5AP4SwDiAnQDepar3Rp+VANwW7fo7VV3RzLKSBigqxTSJPOmnaSm+djMU+31gDfa7YgMOzityhPQYTRMLEZkM4HIArwZwP4BbRWSTqt7h7PYLAItV9WkReR+AjwF4Y/TZblUNzMZDOpJW9CXIKAK5ztdskSOkS2hmy2IJgLtUdQcAiMg1AM4C8KxYqOqPnP1vBvDWJpaHNJuijXkrYIc5QjLRzGlV5wC4z1m/P9oWx7sBfM9ZnyYiW0TkZhE5O3SAiJwb7bNl586djZeY9CYTUeQIaTEdEeAWkbcCWAxgubP5+ar6gIgcDeCHInKbqt7tHqeqVwK4EjCpsy0rMCGE9BjNbFk8AOAoZ/3IaFsVIvIqABcBWKGqz9jtqvpA9LoDwI8BvLSJZSWEEJJAM8XiVgDHiMh8EZkK4E0ANrk7iMhLAXwaRigedrYfIiL7R+/7AJwMJ9ZBCCGktTTNDaWq4yKyCsB1MKmzV6nq7SJyKcyQuJsA/COA5wD4mhg/sU2R/WMAnxaRMoygfcTLoiKEENJCONwHIYT0MFmH+2imG4oQQkiXQLEghBCSCsWCEEJIKhQLQgghqVAsCCGEpEKxIIQQkgrFghBCSCoUC0IIIalQLAghhKRCsSCEEJIKxYIQQkgqXTM2lIjsBHBvjkP6ADzSpOJMFPgM+AwsfA69+wyer6qHpe3UNWKRFxHZkmXwrG6Gz4DPwMLnwGeQBt1QhBBCUqFYEEIISaWXxeLKdhegA+Az4DOw8DnwGSTSszELQggh2enllgUhhJCMUCwIIYSk0nNiISKnicidInKXiFzQ7vK0EhG5R0RuE5FtIrIl2vZcEfmBiPwmej2k3eUsEhG5SkQeFpH/crYF71kMG6Pfxi9F5IT2lbw4Yp7BmIg8EP0WtonIGc5nF0bP4E4ReW17Sl0sInKUiPxIRO4QkdtFZDja3lO/hUboKbEQkckALgdwOoBjAbxZRI5tb6lazv9R1X4nn/wCANer6jEAro/Wu4kvADjN2xZ3z6cDOCZazgXwqRaVsdl8AbXPAADWRb+FflX9LgBE/4c3ATguOuaT0f9mojMO4P2qeiyAkwCsjO61134LddNTYgFgCYC7VHWHqu4FcA2As9pcpnZzFoCro/dXAzi7jWUpHFX9CYDHvM1x93wWgC+q4WYAM0Xk8NaUtHnEPIM4zgJwjao+o6q/BXAXzP9mQqOqD6nqf0bvnwLwKwBz0GO/hUboNbGYA+A+Z/3+aFuvoAC+LyJbReTcaNssVX0oev97ALPaU7SWEnfPvfb7WBW5WK5y3I9d/wxEZB6AlwLYDP4WMtNrYtHrvFxVT4BpYq8UkWXuh2ryqHsql7oX7zniUwBeAKAfwEMAPt7e4rQGEXkOgG8AWK2qT7qf9fBvIRO9JhYPADjKWT8y2tYTqOoD0evDAL4J4174g21eR68Pt6+ELSPunnvm96Gqf1DVkqqWAXwGFVdT1z4DEdkPRii+rKr/Gm3u+d9CVnpNLG4FcIyIzBeRqTCBvE1tLlNLEJEDReQg+x7AawD8F8z9vyPa7R0Avt2eEraUuHveBODtUSbMSQCecFwUXYXnf/8TmN8CYJ7Bm0RkfxGZDxPgvaXV5SsaEREAnwPwK1Vd63zU87+FrExpdwFaiaqOi8gqANcBmAzgKlW9vc3FahWzAHzT/GcwBcC/qOq1InIrgK+KyLthhnj/szaWsXBE5CsATgXQJyL3AxgF8BGE7/m7AM6ACeo+DeAvWl7gJhDzDE4VkX4Yt8s9AP4KAFT1dhH5KoA7YDKIVqpqqR3lLpiTAbwNwG0isi3a9n/RY7+FRuBwH4QQQlLpNTcUIYSQOqBYEEIISYViQQghJBWKBSGEkFQoFoQQQlKhWBCSAxEpOSO1bity5GIRmeeODEtIJ9FT/SwIKYDdqtrf7kIQ0mrYsiCkAKK5Qj4WzRdyi4i8MNo+T0R+GA3Yd72IzI22zxKRb4rI9mh5WXSqySLymWjOhe+LyPS23RQhDhQLQvIx3XNDvdH57AlVPR7AJwCsj7ZdBuBqVV0A4MsANkbbNwK4QVUXAjgBgB1J4BgAl6vqcQB2AXhDk++HkEywBzchORCR/1HV5wS23wPgFaq6Ixqw7veqeqiIPALgcFXdF21/SFX7RGQngCNV9RnnHPMA/CCaiAcicj6A/VT175p/Z4Qkw5YFIcWhMe/z8IzzvgTGFUmHQLEgpDje6Lz+PHr/M5jRjQHgLQBujN5fD+B9gJnuV0RmtKqQhNQDay2E5GO6M2opAFyrqjZ99hAR+SVM6+DN0bZBAJ8Xkb8FsBOV0UuHAVwZjXZaghGOnh4Cm3Q2jFkQUgBRzGKxqj7S7rIQ0gzohiKEEJIKWxaEEEJSYcuCEEJIKhQLQgghqVAsCCGEpEKxIIQQkgrFghBCSCr/H9XrSFn0bGjOAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adam pareto %: 0.8\n",
      "Gd pareto %: 0.0\n",
      "Momentum pareto %: 0.0\n",
      "RMS pareto %: 0.2\n",
      "AdaGrad pareto %: 0.0\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "def plot_pareto_frontier(Xs, Ys, maxX=True, maxY=True):\n",
    "    '''Pareto frontier selection process'''\n",
    "    sorted_list = sorted([[Xs[i], Ys[i]] for i in range(len(Xs))], reverse=maxX)\n",
    "    pareto_front = [sorted_list[0]]\n",
    "    for pair in sorted_list[1:]:\n",
    "        if maxY:\n",
    "            if pair[1] >= pareto_front[-1][1]:\n",
    "                pareto_front.append(pair)\n",
    "        else:\n",
    "            if pair[1] <= pareto_front[-1][1]:\n",
    "                pareto_front.append(pair)\n",
    "    \n",
    "    '''Plotting process'''\n",
    "    plt.scatter(Xs,Ys, color=\"red\", marker=\"x\")\n",
    "    pf_X = [pair[0] for pair in pareto_front]\n",
    "    pf_Y = [pair[1] for pair in pareto_front]\n",
    "    plt.plot(pf_X, pf_Y, color = \"blue\")\n",
    "    plt.xlabel(\"Epoch\")\n",
    "    plt.ylabel(\"Score\")\n",
    "    plt.title(\"Pareto fronta\")\n",
    "    plt.show()\n",
    "    return(pf_X,pf_Y)\n",
    "\n",
    "# Setovi podataka\n",
    "adaGradScore = np.array([3.5377717, 3.756132, 3.8414092, 3.8784597, 3.6417015, 3.661017, 3.821608, 3.876443, 3.847351, 3.3264523, 3.8323412, 3.7991445, 3.6358564, 3.7055511, 3.4977775, 3.859666, 3.9544687, 3.8616483, 3.6630762, 3.7617521, 3.7159967, 3.6688054, 3.6699855, 3.974916, 3.6163132, 3.7645693, 3.813519, 3.4751654, 3.5078292, 3.833887, 3.5527003, 3.6817458, 3.848904, 3.6620307, 3.89151, 3.5973225, 3.8503785, 3.754792, 3.5442836, 3.6148014, 4.0552907, 3.4481683, 3.8438046, 3.5001688, 3.9728303, 3.6669586, 3.8460844, 3.957142, 3.9264672, 3.347025, 3.7439866, 3.8272998, 3.8613887, 3.8332818, 3.7933533, 3.5557163, 3.7825034, 3.7477803, 3.9299226, 3.9952826, 3.9611108, 3.931929, 3.8432927, 3.5717077, 3.8013017, 3.4866416, 3.7633555, 3.8945498, 3.753362, 3.832951, 3.8551571, 3.7731586, 3.7697284, 3.797567, 3.787612, 3.8271832, 3.6563716, 3.7837474, 3.8455033, 3.7596128, 3.7613502, 3.884901, 3.7917995, 3.5050282, 3.6216574, 3.76245, 3.621797, 3.681921, 3.8025255, 3.5522943, 3.7051463, 3.7696464, 3.932058, 3.6936774, 3.4495292, 3.6672595, 3.5272775, 3.804337, 3.7350004, 3.871881])\n",
    "adaGradEpoch = np.array([202, 86, 76, 33, 162, 141, 75, 76, 74, 293, 86, 74, 153, 106, 198, 56, 30, 68, 124, 121, 130, 121, 84, 38, 195, 85, 64, 266, 228, 27, 232, 105, 79, 136, 67, 167, 50, 66, 172, 173, 30, 225, 49, 182, 25, 93, 43, 40, 28, 314, 99, 49, 46, 56, 87, 218, 80, 85, 26, 16, 21, 20, 27, 152, 99, 228, 70, 50, 103, 49, 44, 65, 46, 61, 79, 85, 138, 32, 84, 94, 76, 52, 68, 175, 108, 109, 189, 116, 81, 179, 95, 73, 46, 83, 186, 101, 206, 81, 137, 38])\n",
    "\n",
    "adamScore = np.array([0.32417157, 0.3236075, 0.31669715, 0.3234331, 0.3383628, 0.29094124, 0.2824574, 0.31537795, 0.3203576, 0.2863951, 0.30760592, 0.31439754, 0.31100094, 0.33154863, 0.29488206, 0.28510925, 0.2847425, 0.3036325, 0.35686594, 0.28584424, 0.35734564, 0.3026612, 0.2639545, 0.26700133, 0.3274023, 0.3079812, 0.31784797, 0.31448033, 0.31973773, 0.31328967, 0.3406593, 0.31198546, 0.2892943, 0.3033445, 0.3044309, 0.28795218, 0.3198657, 0.32269683, 0.30936855, 0.29700914, 0.32982007, 0.28271067, 0.29334688, 0.318613, 0.28108695, 0.29616547, 0.32502204, 0.3452332, 0.31639066, 0.30432656, 0.31218317, 0.29502794, 0.31452253, 0.35201618, 0.31210953, 0.32392386, 0.30951715, 0.32851955, 0.2856663, 0.27621236, 0.31520292, 0.28347623, 0.32140395, 0.29196933, 0.31453755, 0.2851151, 0.29347444, 0.30649784, 0.30142558, 0.2956203, 0.31236213, 0.31517443, 0.2816837, 0.33098647, 0.2874186, 0.29140204, 0.2943058, 0.33242828, 0.27634895, 0.30605254, 0.321146, 0.29049993, 0.28172693, 0.3212033, 0.31476814, 0.2972718, 0.27551717, 0.32316273, 0.32977644, 0.27361116, 0.3159537, 0.30150175, 0.30879086, 0.293877, 0.29113445, 0.31858087, 0.26887345, 0.32885072, 0.2955004, 0.2999737])\n",
    "adamEpoch = np.array([8, 18, 42, 12, 11, 32, 39, 36, 15, 84, 18, 58, 22, 10, 13, 57, 34, 25, 16, 17, 4, 87, 34, 51, 73, 11, 31, 4, 14, 8, 11, 17, 20, 19, 6, 45, 39, 14, 17, 12, 24, 59, 17, 15, 38, 7, 6, 11, 26, 11, 20, 19, 4, 7, 75, 24, 11, 36, 62, 45, 29, 24, 14, 37, 6, 28, 21, 12, 11, 23, 22, 32, 67, 9, 20, 8, 61, 38, 79, 12, 8, 66, 62, 20, 23, 29, 40, 14, 39, 60, 14, 32, 12, 24, 21, 3, 44, 36, 16, 53])\n",
    "\n",
    "gdScore = np.array([0.40343145, 0.3995014, 0.39679018, 0.37810802, 0.39395294, 0.39245, 0.4061372, 0.39538097, 0.385495, 0.40732184, 0.39901033, 0.39628386, 0.3935729, 0.40403333, 0.38848144, 0.39349768, 0.4640165, 0.38744423, 0.37974608, 0.3925574, 0.4050962, 0.39611825, 0.36157897, 0.37148148, 0.4006966, 0.3867069, 0.38846645, 0.3855416, 0.38297537, 0.36482897, 0.43375546, 0.3807426, 0.40070337, 0.37644392, 0.38361964, 0.3909691, 0.3861461, 0.38682568, 0.3791259, 0.39386222, 0.39573652, 0.38411364, 0.38425323, 0.38140243, 0.36409345, 0.38644892, 0.3918298, 0.38952842, 0.3921998, 0.39776242, 0.40449804, 0.37746486, 0.38892013, 0.40174055, 0.40954676, 0.40754405, 0.41206548, 0.40640643, 0.4047953, 0.3728906, 0.40148073, 0.3642799, 0.3826057, 0.37580577, 0.40339816, 0.38950244, 0.42415756, 0.40235814, 0.39622387, 0.39360327, 0.38017684, 0.4007698, 0.37753853, 0.3892596, 0.37608308, 0.45759633, 0.39648575, 0.40029955, 0.3803772, 0.3918096, 0.4068011, 0.41986558, 0.3841825, 0.38125318, 0.38147402, 0.39818507, 0.4134346, 0.37878206, 0.41287625, 0.3744243, 0.37956893, 0.40049177, 0.3916857, 0.37142944, 0.37624437, 0.38583326, 0.4208939, 0.4191676, 0.3898088, 0.3845271])\n",
    "gdEpoch = np.array([134, 218, 189, 169, 151, 130, 199, 236, 188, 202, 224, 193, 109, 144, 138, 87, 52, 146, 229, 159, 118, 180, 157, 132, 137, 97, 148, 114, 135, 165, 112, 191, 102, 105, 189, 168, 201, 104, 121, 164, 158, 150, 154, 158, 131, 125, 190, 194, 198, 171, 109, 197, 108, 108, 194, 145, 114, 174, 146, 203, 116, 234, 211, 175, 173, 144, 141, 154, 194, 160, 219, 210, 173, 176, 112, 104, 90, 222, 174, 178, 151, 148, 154, 189, 139, 188, 126, 167, 101, 217, 193, 120, 177, 177, 152, 147, 127, 152, 129, 95])\n",
    "\n",
    "momentumScore = np.array([0.30512246, 0.32050124, 0.3118204, 0.29860216, 0.32846767, 0.32377097, 0.3194519, 0.3188375, 0.32737195, 0.31416526, 0.31376815, 0.3155915, 0.31906888, 0.33468908, 0.2989422, 0.31819388, 0.3157418, 0.32492107, 0.30315226, 0.29696575, 0.32993317, 0.31338236, 0.28853416, 0.28785837, 0.32536113, 0.31178012, 0.29905328, 0.3069888, 0.31141117, 0.29798573, 0.3457167, 0.30721506, 0.3062942, 0.30499995, 0.30335963, 0.3079729, 0.30322728, 0.31084374, 0.30906832, 0.31301147, 0.31630963, 0.32588533, 0.30555665, 0.3049158, 0.28510648, 0.30788434, 0.33427083, 0.31363034, 0.31127074, 0.3030068, 0.32236198, 0.30453047, 0.3063938, 0.32014114, 0.34994724, 0.31897938, 0.31081125, 0.32688922, 0.32065347, 0.30915087, 0.33021173, 0.29745802, 0.3086957, 0.31498563, 0.30749908, 0.32109466, 0.31107074, 0.32902798, 0.32404593, 0.32046407, 0.29861194, 0.33787188, 0.2940089, 0.30977818, 0.3051475, 0.3120023, 0.3107265, 0.33036223, 0.29562786, 0.3249199, 0.32086077, 0.3154661, 0.300126, 0.30632794, 0.30394226, 0.3227191, 0.28800896, 0.3031807, 0.33319303, 0.28889045, 0.30301628, 0.3150718, 0.3172033, 0.29630026, 0.2998042, 0.32010755, 0.2913881, 0.3362923, 0.3105656, 0.3028073])\n",
    "momentumEpoch = np.array([95, 64, 92, 87, 40, 46, 77, 68, 43, 116, 108, 75, 35, 40, 112, 54, 92, 63, 82, 87, 38, 65, 67, 79, 49, 45, 136, 175, 58, 100, 46, 118, 55, 68, 102, 119, 88, 37, 60, 89, 76, 25, 64, 54, 69, 46, 23, 79, 67, 96, 55, 73, 58, 53, 31, 82, 97, 52, 71, 43, 59, 61, 52, 44, 73, 41, 126, 39, 73, 78, 120, 32, 127, 112, 41, 146, 66, 40, 102, 32, 74, 70, 103, 76, 60, 74, 148, 70, 39, 60, 63, 43, 73, 98, 90, 32, 75, 36, 61, 88])\n",
    "\n",
    "rmsScore = np.array([0.29006058, 0.28402218, 0.29099333, 0.283063, 0.30094597, 0.31597307, 0.28332794, 0.28437674, 0.28395766, 0.30384588, 0.28691795, 0.276377, 0.2808033, 0.27902743, 0.2672417, 0.28558615, 0.26566648, 0.27892727, 0.26648408, 0.28889012, 0.31122875, 0.28696412, 0.26855147, 0.2833428, 0.29345244, 0.27514333, 0.2713269, 0.28643894, 0.27636904, 0.27477035, 0.32560757, 0.27501276, 0.3043946, 0.2783086, 0.31523508, 0.2875137, 0.26740754, 0.2556993, 0.29588255, 0.28068423, 0.3005326, 0.27071145, 0.27468303, 0.2757961, 0.2589961, 0.27337775, 0.28602526, 0.27886572, 0.2755192, 0.26529616, 0.2837832, 0.27461398, 0.27821484, 0.2997016, 0.28743976, 0.28462982, 0.29634243, 0.28239188, 0.34044176, 0.27146813, 0.2917923, 0.2784095, 0.27055234, 0.26745185, 0.2688616, 0.3059669, 0.2770882, 0.2830725, 0.29811418, 0.28193155, 0.31134114, 0.27861995, 0.2575153, 0.29746437, 0.25985563, 0.3453521, 0.28055847, 0.3560163, 0.3752023, 0.28920022, 0.286063, 0.293109, 0.27619183, 0.28042608, 0.26896238, 0.2734545, 0.26086065, 0.28649902, 0.3061967, 0.28311342, 0.2707301, 0.29284564, 0.29340476, 0.29328632, 0.27273682, 0.2737857, 0.26837555, 0.3360302, 0.26765367, 0.27243727])\n",
    "rmsEpoch = np.array([56, 59, 67, 50, 63, 38, 64, 73, 99, 41, 73, 58, 74, 81, 81, 52, 108, 63, 121, 50, 38, 82, 90, 66, 80, 63, 68, 67, 58, 52, 46, 118, 69, 68, 47, 84, 87, 102, 66, 91, 55, 81, 62, 54, 85, 46, 52, 71, 67, 96, 54, 77, 59, 49, 110, 82, 74, 105, 56, 45, 59, 75, 90, 73, 69, 69, 80, 77, 106, 70, 51, 100, 67, 53, 93, 47, 66, 40, 62, 87, 63, 79, 76, 76, 60, 98, 77, 93, 51, 54, 63, 43, 45, 69, 79, 92, 54, 82, 61, 55])\n",
    "\n",
    "scores = [*adamScore, *momentumScore, *rmsScore, *gdScore]\n",
    "epochs = [*adamEpoch, *momentumEpoch, *rmsEpoch, *gdEpoch]\n",
    "pX, pY = plot_pareto_frontier(Xs = epochs, Ys = scores, maxX=False, maxY=False)\n",
    "\n",
    "# Izračun udjela\n",
    "adam = 0\n",
    "gd = 0\n",
    "momentum = 0\n",
    "rms = 0\n",
    "\n",
    "for i,j in zip(adamScore, adamEpoch):\n",
    "    if i in pY and j in pX:\n",
    "        adam += 1\n",
    "        \n",
    "for i,j in zip(gdScore, gdEpoch):\n",
    "    if i in pY and j in pX:\n",
    "        gd += 1\n",
    "        \n",
    "for i,j in zip(momentumScore, momentumEpoch):\n",
    "    if i in pY and j in pX:\n",
    "        momentum += 1\n",
    "        \n",
    "for i,j in zip(rmsScore, rmsEpoch):\n",
    "    if i in pY and j in pX:\n",
    "        rms += 1\n",
    "print (\"Adam pareto %: {}\".format(adam/len(pX)))\n",
    "print (\"Gd pareto %: {}\".format(gd/len(pX)))\n",
    "print (\"Momentum pareto %: {}\".format(momentum/len(pX)))\n",
    "print (\"RMS pareto %: {}\".format(rms/len(pX)))\n",
    "print (\"AdaGrad pareto %: {}\".format(0.0))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}