{"cells":[{"cell_type":"code","source":["!date\n","!python --version"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"zFtyj6SYAp3P","executionInfo":{"status":"ok","timestamp":1746676137737,"user_tz":-540,"elapsed":210,"user":{"displayName":"TOMA Naruaki","userId":"11747312442870110137"}},"outputId":"c363f870-722d-4b15-9589-1fe2f04bef6f"},"id":"zFtyj6SYAp3P","execution_count":2,"outputs":[{"output_type":"stream","name":"stdout","text":["Thu May  8 03:48:57 AM UTC 2025\n","Python 3.11.12\n"]}]},{"cell_type":"markdown","id":"2f313f12","metadata":{"id":"2f313f12"},"source":["# BoWとword2vec用の前処理\n","分かち書きに時間がかかる（當間環境で約20分）ため、実験コードを実行し直す都度分かち書きからやり直すのは時間がもったいない。そこでここでは分かち書きだけを行い、結果を（[pickle形式](https://docs.python.org/ja/3.13/library/pickle.html)）ファイルとして保存しておく。機械学習する側のコードでは、このファイルを読み込むだけの時間で実験を再開することができるようになる。\n","\n","本コードを実行すると、以下のファイルに前処理済みデータが保存される。\n","- 学習データ: `prepcoecced_train.pkl`\n","- 検証データ: `prepcoecced_val.pkl`\n","- テストデータ: `prepcoecced_test.pkl`\n","\n","pickleファイルの中身は「1件のデータが辞書型」であり、それをリストとして保存している。`sent1`は前提文、`sent2`は仮説文。`bow_input`がBoW用の分かち書き結果。`w2v_input`がword2vec用の分散ベクトル。`label`が教師ラベル（文字列）である。\n","```Python\n","[\n","  {\n","    \"sent1\": \"男の子が水たまりで遊んでいる。\",\n","    \"sent2\": \"子供が遊んでいる。\",\n","    \"bow_input\": \"男の子 が 水たまり で 遊ん で いる 。 [SEP] 子供 が 遊ん で いる 。\",\n","    \"w2v_input\": np.array([...]),  # shape: (600,)\n","    \"label\": \"entailment\"\n","  },\n","  ...\n","]\n","```\n","\n","実行後、作成されるファイルをダウンロードするのを忘れないようにしよう。忘れてしまったらまたこれを実行し直すことになります。"]},{"cell_type":"markdown","source":["## 前提\n","- [Github: JGLUE/datasets/jnil-v1.3/](https://github.com/yahoojapan/JGLUE/tree/main/datasets/jnli-v1.3) にある以下のファイルをダウンロードしておき、本ノートブックを実行する前にアップロードしておいてください。\n","    - train-v1.3.json（学習用データセット）\n","    - val-v1.3.json（検証用データセット）\n","    - test-v1.3.json（テスト用データセット）\n"],"metadata":{"id":"gHqOHwUMAC1C"},"id":"gHqOHwUMAC1C"},{"cell_type":"code","execution_count":3,"id":"36a52cee","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"36a52cee","executionInfo":{"status":"ok","timestamp":1746676148763,"user_tz":-540,"elapsed":11023,"user":{"displayName":"TOMA Naruaki","userId":"11747312442870110137"}},"outputId":"65a4a831-9d65-4ed4-a2e6-1185c4d62afc"},"outputs":[{"output_type":"stream","name":"stdout","text":["Collecting ginza\n","  Downloading ginza-5.2.0-py3-none-any.whl.metadata (448 bytes)\n","Collecting ja_ginza\n","  Downloading ja_ginza-5.2.0-py3-none-any.whl.metadata (5.8 kB)\n","Requirement already satisfied: spacy<4.0.0,>=3.4.4 in /usr/local/lib/python3.11/dist-packages (from ginza) (3.8.5)\n","Collecting plac>=1.3.3 (from ginza)\n","  Downloading plac-1.4.5-py2.py3-none-any.whl.metadata (5.9 kB)\n","Collecting SudachiPy<0.7.0,>=0.6.2 (from ginza)\n","  Downloading SudachiPy-0.6.10-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n","Collecting SudachiDict-core>=20210802 (from ginza)\n","  Downloading SudachiDict_core-20250129-py3-none-any.whl.metadata (2.5 kB)\n","Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (3.0.12)\n","Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (1.0.5)\n","Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (1.0.12)\n","Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.0.11)\n","Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (3.0.9)\n","Requirement already satisfied: thinc<8.4.0,>=8.3.4 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (8.3.6)\n","Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (1.1.3)\n","Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.5.1)\n","Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.0.10)\n","Requirement already satisfied: weasel<0.5.0,>=0.1.0 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (0.4.1)\n","Requirement already satisfied: typer<1.0.0,>=0.3.0 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (0.15.3)\n","Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (4.67.1)\n","Requirement already satisfied: numpy>=1.19.0 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.0.2)\n","Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.32.3)\n","Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.11.4)\n","Requirement already satisfied: jinja2 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (3.1.6)\n","Requirement already satisfied: setuptools in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (75.2.0)\n","Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (24.2)\n","Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.11/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (3.5.0)\n","Requirement already satisfied: language-data>=1.2 in /usr/local/lib/python3.11/dist-packages (from langcodes<4.0.0,>=3.2.0->spacy<4.0.0,>=3.4.4->ginza) (1.3.0)\n","Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.11/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<4.0.0,>=3.4.4->ginza) (0.7.0)\n","Requirement already satisfied: pydantic-core==2.33.2 in /usr/local/lib/python3.11/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<4.0.0,>=3.4.4->ginza) (2.33.2)\n","Requirement already satisfied: typing-extensions>=4.12.2 in /usr/local/lib/python3.11/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<4.0.0,>=3.4.4->ginza) (4.13.2)\n","Requirement already satisfied: typing-inspection>=0.4.0 in /usr/local/lib/python3.11/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<4.0.0,>=3.4.4->ginza) (0.4.0)\n","Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/dist-packages (from requests<3.0.0,>=2.13.0->spacy<4.0.0,>=3.4.4->ginza) (3.4.1)\n","Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/dist-packages (from requests<3.0.0,>=2.13.0->spacy<4.0.0,>=3.4.4->ginza) (3.10)\n","Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/dist-packages (from requests<3.0.0,>=2.13.0->spacy<4.0.0,>=3.4.4->ginza) (2.4.0)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/dist-packages (from requests<3.0.0,>=2.13.0->spacy<4.0.0,>=3.4.4->ginza) (2025.4.26)\n","Requirement already satisfied: blis<1.4.0,>=1.3.0 in /usr/local/lib/python3.11/dist-packages (from thinc<8.4.0,>=8.3.4->spacy<4.0.0,>=3.4.4->ginza) (1.3.0)\n","Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.11/dist-packages (from thinc<8.4.0,>=8.3.4->spacy<4.0.0,>=3.4.4->ginza) (0.1.5)\n","Requirement already satisfied: click>=8.0.0 in /usr/local/lib/python3.11/dist-packages (from typer<1.0.0,>=0.3.0->spacy<4.0.0,>=3.4.4->ginza) (8.1.8)\n","Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.11/dist-packages (from typer<1.0.0,>=0.3.0->spacy<4.0.0,>=3.4.4->ginza) (1.5.4)\n","Requirement already satisfied: rich>=10.11.0 in /usr/local/lib/python3.11/dist-packages (from typer<1.0.0,>=0.3.0->spacy<4.0.0,>=3.4.4->ginza) (13.9.4)\n","Requirement already satisfied: cloudpathlib<1.0.0,>=0.7.0 in /usr/local/lib/python3.11/dist-packages (from weasel<0.5.0,>=0.1.0->spacy<4.0.0,>=3.4.4->ginza) (0.21.0)\n","Requirement already satisfied: smart-open<8.0.0,>=5.2.1 in /usr/local/lib/python3.11/dist-packages (from weasel<0.5.0,>=0.1.0->spacy<4.0.0,>=3.4.4->ginza) (7.1.0)\n","Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.11/dist-packages (from jinja2->spacy<4.0.0,>=3.4.4->ginza) (3.0.2)\n","Requirement already satisfied: marisa-trie>=1.1.0 in /usr/local/lib/python3.11/dist-packages (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy<4.0.0,>=3.4.4->ginza) (1.2.1)\n","Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.11/dist-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy<4.0.0,>=3.4.4->ginza) (3.0.0)\n","Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.11/dist-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy<4.0.0,>=3.4.4->ginza) (2.19.1)\n","Requirement already satisfied: wrapt in /usr/local/lib/python3.11/dist-packages (from smart-open<8.0.0,>=5.2.1->weasel<0.5.0,>=0.1.0->spacy<4.0.0,>=3.4.4->ginza) (1.17.2)\n","Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.11/dist-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy<4.0.0,>=3.4.4->ginza) (0.1.2)\n","Downloading ginza-5.2.0-py3-none-any.whl (21 kB)\n","Downloading ja_ginza-5.2.0-py3-none-any.whl (59.1 MB)\n","\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m59.1/59.1 MB\u001b[0m \u001b[31m16.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hDownloading plac-1.4.5-py2.py3-none-any.whl (22 kB)\n","Downloading SudachiDict_core-20250129-py3-none-any.whl (72.1 MB)\n","\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m72.1/72.1 MB\u001b[0m \u001b[31m9.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hDownloading SudachiPy-0.6.10-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n","\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m63.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hInstalling collected packages: SudachiPy, plac, SudachiDict-core, ginza, ja_ginza\n","Successfully installed SudachiDict-core-20250129 SudachiPy-0.6.10 ginza-5.2.0 ja_ginza-5.2.0 plac-1.4.5\n"]}],"source":["# spacyインストール\n","!pip install ginza ja_ginza"]},{"cell_type":"code","execution_count":4,"id":"2a98c059","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"2a98c059","executionInfo":{"status":"ok","timestamp":1746677294448,"user_tz":-540,"elapsed":1145682,"user":{"displayName":"TOMA Naruaki","userId":"11747312442870110137"}},"outputId":"ab274066-3924-400f-e044-1b9a5671dc54"},"outputs":[{"output_type":"stream","name":"stderr","text":["100%|██████████| 20073/20073 [15:05<00:00, 22.16it/s]\n","100%|██████████| 2434/2434 [01:51<00:00, 21.83it/s]\n","100%|██████████| 2508/2508 [01:53<00:00, 22.12it/s]\n"]}],"source":["# ====================================\n","# ✅ 必要ライブラリ\n","# ====================================\n","import spacy\n","import pickle\n","import os\n","from tqdm import tqdm\n","import pandas as pd\n","import numpy as np\n","\n","# ====================================\n","# ✅ GiNZAのロード（時間がかかります）\n","# ====================================\n","nlp = spacy.load(\"ja_ginza\")\n","\n","# ====================================\n","# ✅ 入力ファイルの読み込み\n","# ====================================\n","def load_jsonl(filepath):\n","    with open(filepath, encoding=\"utf-8\") as f:\n","        return [eval(line.strip()) for line in f]\n","\n","def load_and_prepare(file_path):\n","    data = load_jsonl(file_path)\n","    return [(ex[\"sentence1\"], ex[\"sentence2\"], ex[\"label\"]) for ex in data]\n","\n","train_data = load_and_prepare(\"train-v1.3.json\")\n","val_data = load_and_prepare(\"valid-v1.3.json\")\n","test_data = load_and_prepare(\"test-v1.3.json\")\n","\n","# ====================================\n","# ✅ 文をspaCyで処理して保存\n","# ====================================\n","def preprocess_and_save(data, output_file):\n","    results = []\n","    for sent1, sent2, label in tqdm(data):\n","        # GiNZAで処理\n","        doc1 = nlp(sent1)\n","        doc2 = nlp(sent2)\n","\n","        # BoW用分かち書き（空白区切り文字列）\n","        tokens1 = [t.text for t in doc1 if not t.is_space]\n","        tokens2 = [t.text for t in doc2 if not t.is_space]\n","        bow_joined = \" \".join(tokens1 + [\"[SEP]\"] + tokens2)\n","\n","        # word2vec用ベクトル平均\n","        vecs1 = [t.vector for t in doc1 if t.has_vector and not t.is_space]\n","        vecs2 = [t.vector for t in doc2 if t.has_vector and not t.is_space]\n","        if vecs1:\n","            vec1_avg = np.mean(vecs1, axis=0)\n","        else:\n","            vec1_avg = np.zeros(nlp.vocab.vectors_length)\n","        if vecs2:\n","            vec2_avg = np.mean(vecs2, axis=0)\n","        else:\n","            vec2_avg = np.zeros(nlp.vocab.vectors_length)\n","        vec_concat = np.concatenate([vec1_avg, vec2_avg])\n","\n","        results.append({\n","            \"sent1\": sent1, # 前提文\n","            \"sent2\": sent2, # 仮説文\n","            \"bow_input\": bow_joined, # BoW用の分かち書きした文（[SEP]付きで結合）\n","            \"w2v_input\": vec_concat, # word2vec用の平均ベクトトル（2文の平均ベクトルを結合）\n","            \"label\": label\n","        })\n","\n","    # pickle保存\n","    with open(output_file, \"wb\") as f:\n","        pickle.dump(results, f)\n","\n","# ====================================\n","# ✅ 実行：train / val / test それぞれ保存\n","# ====================================\n","preprocess_and_save(train_data, \"preprocessed_train.pkl\")\n","preprocess_and_save(val_data, \"preprocessed_val.pkl\")\n","preprocess_and_save(test_data, \"preprocessed_test.pkl\")"]},{"cell_type":"markdown","source":["## 実行終了後、3つのpklファイルをダウンロードすること"],"metadata":{"id":"mApDBIS-AYpP"},"id":"mApDBIS-AYpP"},{"cell_type":"code","source":[],"metadata":{"id":"iq1oc2cFAb91","executionInfo":{"status":"ok","timestamp":1746677294500,"user_tz":-540,"elapsed":42,"user":{"displayName":"TOMA Naruaki","userId":"11747312442870110137"}}},"id":"iq1oc2cFAb91","execution_count":4,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"deep","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.6"},"colab":{"provenance":[]}},"nbformat":4,"nbformat_minor":5}