{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "3f56e665",
      "metadata": {
        "id": "3f56e665"
      },
      "source": [
        "##Prepare data\n",
        "Based on Github: **csinva/gpt-paper-title-generator**\n",
        "\n",
        "PKL files are employed for data storage. Generated by the \"pickle\" module in Python, these files allow objects to be serialized onto disk and later deserialized into the program during runtime. Essentially, a PKL file comprises a byte stream representing the stored objects.\n",
        "\n",
        "Please make sure if file arxiv_metadata_2022.pkl is in google colab ide \"sample_data\" folder.\n",
        "\n",
        "##Stripped data\n",
        "Following the preparation using the get_metadata() function, our objective is to showcase solely the title column.\n",
        "\n",
        "Given the extensive number of rows in the Arxiv database, we narrow down our focus for neural network learning by selecting only the initial 1000 rows."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "705c004b",
      "metadata": {
        "id": "705c004b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "edcbd100-f56c-4b2a-f2db-dd962aeac06f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "File not exist, generating...\n",
            "46        data: Intelligent location of simultaneously a...\n",
            "49        data: Intelligent location of simultaneously a...\n",
            "303                 data: The World as Evolving Information\n",
            "670             data: Learning from compressed observations\n",
            "953       data: Sensor Networks with Random Links: Topol...\n",
            "                                ...                        \n",
            "128578           data: Matrix Completion from Noisy Entries\n",
            "128585    data: Regularization methods for learning inco...\n",
            "128705    data: From formulas to cirquents in computabil...\n",
            "128779    data: Characterising equilibrium logic and nes...\n",
            "128825    data: A Neural Network Classifier of Volume Da...\n",
            "Name: title, Length: 1000, dtype: object\n"
          ]
        }
      ],
      "source": [
        "import re\n",
        "import os\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "\n",
        "def clean(s):\n",
        "    s = s.replace('\\n', '').replace('\\t', '')\n",
        "    return re.sub(' +', ' ', s)\n",
        "\n",
        "def get_metadata():\n",
        "    path = './sample_data/arxiv_metatadata_2022_clean.pkl'\n",
        "    if os.path.exists(path):\n",
        "        df = pd.read_pickle(path)\n",
        "        print(\"File already exits\")\n",
        "    else:\n",
        "        print(\"File not exist, generating...\")\n",
        "        df = pd.read_pickle('./sample_data/arxiv_metadata_2022.pkl')\n",
        "        df = df.drop(columns=['versions', 'update_date'])\n",
        "        df = df.drop_duplicates(subset=['title', 'authors'], keep='first')\n",
        "\n",
        "        df['title'] = df['title'].apply(clean)\n",
        "        df = df[~df['title'].str.lower().str.startswith('comment')]\n",
        "        df['title_len'] = df['title'].str.split().apply(len)\n",
        "        df.to_pickle(path)\n",
        "\n",
        "    return df\n",
        "\n",
        "data = get_metadata().title\n",
        "data = data.iloc[0:1000]\n",
        "print(\"data: \"+data)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ef0b1743",
      "metadata": {
        "id": "ef0b1743"
      },
      "source": [
        "##Vectorizing text\n",
        "Converting strings into list of values. This function is based on TensorFlow tutorial.\n",
        "\n",
        "To convert the text vector into a stream of character indices we use the tf.data.Dataset.from_tensor_slices function. dataset.shuffle().batch() function is used to ensure that the model does not learn patterns based on the order of the examples in the dataset. Batching can improve computational efficiency because model is learning on data rather than individual data points."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5bb187e8",
      "metadata": {
        "id": "5bb187e8"
      },
      "outputs": [],
      "source": [
        "def split_target(chunk):\n",
        "    return chunk[:-1], chunk[1:]\n",
        "\n",
        "full_text = \" \".join(data.tolist())\n",
        "\n",
        "vocab = sorted(set(full_text))\n",
        "char_to_index = {char: index for index, char in enumerate(vocab)}\n",
        "sequences = tf.data.Dataset.from_tensor_slices(np.array([char_to_index[char] for char in full_text])).batch(101, drop_remainder=True)\n",
        "dataset = sequences.map(split_target)\n",
        "BUFFER_SIZE = 10000\n",
        "BATCH_SIZE = 10\n",
        "dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d74a747b",
      "metadata": {
        "id": "d74a747b"
      },
      "source": [
        "##Sequential model\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "942ef824",
      "metadata": {
        "id": "942ef824"
      },
      "outputs": [],
      "source": [
        "def create_model(vocab_size, embedding_dim, rnn_units, batch_size):\n",
        "    return tf.keras.Sequential([\n",
        "        tf.keras.layers.Embedding(vocab_size, embedding_dim, batch_input_shape=[batch_size, None]),\n",
        "        tf.keras.layers.GRU(rnn_units, return_sequences=True, stateful=True, recurrent_initializer='glorot_uniform'),\n",
        "        tf.keras.layers.Dense(vocab_size)\n",
        "    ])\n",
        "embedding_dim = 100\n",
        "rnn_units = 100\n",
        "\n",
        "model = create_model(vocab_size=len(vocab), embedding_dim=embedding_dim, rnn_units=rnn_units, batch_size=BATCH_SIZE)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e5136f66",
      "metadata": {
        "id": "e5136f66"
      },
      "source": [
        "##Loss function\n",
        "Loss function measures the difference between the predicted values of a model and the actual values of the target variable. The goal of the training is to minimize this loss function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d8d2417b",
      "metadata": {
        "id": "d8d2417b"
      },
      "outputs": [],
      "source": [
        "def loss(labels, logits):\n",
        "    return tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)\n",
        "\n",
        "model.compile(optimizer='adam', loss=loss, run_eagerly=True)\n",
        "checkpoint_prefix = os.path.join('./training_checkpoints', \"ckpt_{epoch}\")\n",
        "checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_prefix, save_weights_only=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "211df3d3",
      "metadata": {
        "id": "211df3d3"
      },
      "source": [
        "##Learning\n",
        "To ensure a reasonable processing time, we utilize minimum 50 epochs. Observing the loss function reveals a consistent decrease in its values over time."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3b488a36",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3b488a36",
        "outputId": "64c9e56f-0c13-45c5-aeee-6ec988ed296e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/75\n",
            "52/63 [=======================>......] - ETA: 0s - loss: 3.5989"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values. See the following logs for the specific values in question. To silence these warnings, use `status.expect_partial()`. See https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint#restorefor details about the status object returned by the restore function.\n",
            "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._iterations\n",
            "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._learning_rate\n",
            "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.1\n",
            "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.2\n",
            "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.3\n",
            "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.4\n",
            "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.5\n",
            "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.6\n",
            "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.7\n",
            "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.8\n",
            "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.9\n",
            "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.10\n",
            "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.11\n",
            "WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer._variables.12\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "63/63 [==============================] - 7s 91ms/step - loss: 3.5149\n",
            "Epoch 2/75\n",
            "63/63 [==============================] - 3s 40ms/step - loss: 2.8643\n",
            "Epoch 3/75\n",
            "63/63 [==============================] - 3s 39ms/step - loss: 2.5346\n",
            "Epoch 4/75\n",
            "63/63 [==============================] - 3s 42ms/step - loss: 2.4011\n",
            "Epoch 5/75\n",
            "63/63 [==============================] - 3s 38ms/step - loss: 2.2987\n",
            "Epoch 6/75\n",
            "63/63 [==============================] - 2s 38ms/step - loss: 2.2049\n",
            "Epoch 7/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 2.1191\n",
            "Epoch 8/75\n",
            "63/63 [==============================] - 3s 41ms/step - loss: 2.0447\n",
            "Epoch 9/75\n",
            "63/63 [==============================] - 3s 40ms/step - loss: 1.9778\n",
            "Epoch 10/75\n",
            "63/63 [==============================] - 3s 43ms/step - loss: 1.9181\n",
            "Epoch 11/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.8634\n",
            "Epoch 12/75\n",
            "63/63 [==============================] - 3s 46ms/step - loss: 1.8143\n",
            "Epoch 13/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.7676\n",
            "Epoch 14/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.7256\n",
            "Epoch 15/75\n",
            "63/63 [==============================] - 2s 36ms/step - loss: 1.6889\n",
            "Epoch 16/75\n",
            "63/63 [==============================] - 3s 40ms/step - loss: 1.6547\n",
            "Epoch 17/75\n",
            "63/63 [==============================] - 2s 36ms/step - loss: 1.6205\n",
            "Epoch 18/75\n",
            "63/63 [==============================] - 2s 36ms/step - loss: 1.5899\n",
            "Epoch 19/75\n",
            "63/63 [==============================] - 2s 36ms/step - loss: 1.5658\n",
            "Epoch 20/75\n",
            "63/63 [==============================] - 3s 38ms/step - loss: 1.5388\n",
            "Epoch 21/75\n",
            "63/63 [==============================] - 3s 50ms/step - loss: 1.5179\n",
            "Epoch 22/75\n",
            "63/63 [==============================] - 2s 36ms/step - loss: 1.4974\n",
            "Epoch 23/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.4771\n",
            "Epoch 24/75\n",
            "63/63 [==============================] - 3s 44ms/step - loss: 1.4581\n",
            "Epoch 25/75\n",
            "63/63 [==============================] - 3s 47ms/step - loss: 1.4410\n",
            "Epoch 26/75\n",
            "63/63 [==============================] - 2s 36ms/step - loss: 1.4215\n",
            "Epoch 27/75\n",
            "63/63 [==============================] - 2s 36ms/step - loss: 1.4077\n",
            "Epoch 28/75\n",
            "63/63 [==============================] - 2s 36ms/step - loss: 1.3927\n",
            "Epoch 29/75\n",
            "63/63 [==============================] - 3s 44ms/step - loss: 1.3778\n",
            "Epoch 30/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.3671\n",
            "Epoch 31/75\n",
            "63/63 [==============================] - 2s 36ms/step - loss: 1.3535\n",
            "Epoch 32/75\n",
            "63/63 [==============================] - 3s 49ms/step - loss: 1.3444\n",
            "Epoch 33/75\n",
            "63/63 [==============================] - 3s 48ms/step - loss: 1.3310\n",
            "Epoch 34/75\n",
            "63/63 [==============================] - 2s 36ms/step - loss: 1.3195\n",
            "Epoch 35/75\n",
            "63/63 [==============================] - 2s 36ms/step - loss: 1.3082\n",
            "Epoch 36/75\n",
            "63/63 [==============================] - 2s 36ms/step - loss: 1.2996\n",
            "Epoch 37/75\n",
            "63/63 [==============================] - 3s 46ms/step - loss: 1.2880\n",
            "Epoch 38/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.2819\n",
            "Epoch 39/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.2696\n",
            "Epoch 40/75\n",
            "63/63 [==============================] - 3s 40ms/step - loss: 1.2645\n",
            "Epoch 41/75\n",
            "63/63 [==============================] - 3s 41ms/step - loss: 1.2581\n",
            "Epoch 42/75\n",
            "63/63 [==============================] - 2s 38ms/step - loss: 1.2470\n",
            "Epoch 43/75\n",
            "63/63 [==============================] - 2s 36ms/step - loss: 1.2386\n",
            "Epoch 44/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.2305\n",
            "Epoch 45/75\n",
            "63/63 [==============================] - 3s 48ms/step - loss: 1.2235\n",
            "Epoch 46/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.2173\n",
            "Epoch 47/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.2067\n",
            "Epoch 48/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.1999\n",
            "Epoch 49/75\n",
            "63/63 [==============================] - 3s 42ms/step - loss: 1.1962\n",
            "Epoch 50/75\n",
            "63/63 [==============================] - 2s 36ms/step - loss: 1.1897\n",
            "Epoch 51/75\n",
            "63/63 [==============================] - 2s 36ms/step - loss: 1.1824\n",
            "Epoch 52/75\n",
            "63/63 [==============================] - 2s 38ms/step - loss: 1.1722\n",
            "Epoch 53/75\n",
            "63/63 [==============================] - 3s 40ms/step - loss: 1.1722\n",
            "Epoch 54/75\n",
            "63/63 [==============================] - 3s 50ms/step - loss: 1.1645\n",
            "Epoch 55/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.1603\n",
            "Epoch 56/75\n",
            "63/63 [==============================] - 3s 39ms/step - loss: 1.1540\n",
            "Epoch 57/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.1467\n",
            "Epoch 58/75\n",
            "63/63 [==============================] - 3s 45ms/step - loss: 1.1389\n",
            "Epoch 59/75\n",
            "63/63 [==============================] - 2s 38ms/step - loss: 1.1360\n",
            "Epoch 60/75\n",
            "63/63 [==============================] - 2s 38ms/step - loss: 1.1308\n",
            "Epoch 61/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.1246\n",
            "Epoch 62/75\n",
            "63/63 [==============================] - 3s 47ms/step - loss: 1.1199\n",
            "Epoch 63/75\n",
            "63/63 [==============================] - 2s 38ms/step - loss: 1.1151\n",
            "Epoch 64/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.1089\n",
            "Epoch 65/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.1044\n",
            "Epoch 66/75\n",
            "63/63 [==============================] - 3s 38ms/step - loss: 1.1029\n",
            "Epoch 67/75\n",
            "63/63 [==============================] - 3s 49ms/step - loss: 1.0930\n",
            "Epoch 68/75\n",
            "63/63 [==============================] - 2s 36ms/step - loss: 1.0910\n",
            "Epoch 69/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.0902\n",
            "Epoch 70/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.0857\n",
            "Epoch 71/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.0782\n",
            "Epoch 72/75\n",
            "63/63 [==============================] - 3s 48ms/step - loss: 1.0717\n",
            "Epoch 73/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.0698\n",
            "Epoch 74/75\n",
            "63/63 [==============================] - 2s 37ms/step - loss: 1.0666\n",
            "Epoch 75/75\n",
            "63/63 [==============================] - 2s 38ms/step - loss: 1.0650\n"
          ]
        }
      ],
      "source": [
        "EPOCHS = 75\n",
        "history = model.fit(dataset, epochs=EPOCHS, callbacks=[checkpoint_callback])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "43a06c2c",
      "metadata": {
        "id": "43a06c2c"
      },
      "outputs": [],
      "source": [
        "model = create_model(len(vocab), embedding_dim, rnn_units, batch_size=1)\n",
        "model.load_weights(tf.train.latest_checkpoint('./training_checkpoints'))\n",
        "model.build(tf.TensorShape([1, None]))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9e9eb5b6",
      "metadata": {
        "id": "9e9eb5b6"
      },
      "source": [
        "##Generating results\n",
        "To generate text, it is necessary to execute this model iteratively while maintaining awareness of its current state."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "49826da7",
      "metadata": {
        "id": "49826da7"
      },
      "outputs": [],
      "source": [
        "changeCharToIndex = {u:i for i, u in enumerate(vocab)}\n",
        "changeIndexToChar = np.array(vocab)\n",
        "\n",
        "def gen_title(model, start_string):\n",
        "    text_length = 50\n",
        "    input_eval = [changeCharToIndex[s] for s in start_string]\n",
        "    input_eval = tf.expand_dims(input_eval, 0)\n",
        "    text_generated = []\n",
        "    model.reset_states()\n",
        "    i = 0\n",
        "    while 1:\n",
        "        i += 1\n",
        "        temp = 0.05\n",
        "        pred = tf.squeeze(model(input_eval), 0) / temp\n",
        "        pred_id = tf.random.categorical(pred, num_samples=1)[-1,0].numpy()\n",
        "        input_eval = tf.expand_dims([pred_id], 0)\n",
        "        text_generated.append(changeIndexToChar[pred_id])\n",
        "        if (i > text_length and changeIndexToChar[pred_id] == \" \"):\n",
        "            break\n",
        "\n",
        "    return (start_string + ''.join(text_generated))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "153e2f0e",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "153e2f0e",
        "outputId": "5d2b0c84-e48a-4e2a-a682-db8e88809bff"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Neural Network Sequences Finding and the Computational Structure \n",
            "Statistics of a new generalization of the Computational Statistical \n",
            "Machine Learning Computational Structure of Semantic Web a Computational \n"
          ]
        }
      ],
      "source": [
        "print(gen_title(model, start_string=u\"Neural Network\"))\n",
        "print(gen_title(model, start_string=u\"Statistics\"))\n",
        "print(gen_title(model, start_string=u\"Machine Learning\"))"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.6"
    },
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 5
}