{ "cells": [ { "cell_type": "code", "source": [ "!date\n", "!python --version" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "7c9LVvDZVoQP", "outputId": "e9220659-7536-4b2a-bc05-a4e72805586d" }, "execution_count": 1, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Thu Jun 6 02:11:57 AM UTC 2024\n", "Python 3.10.12\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "rxL_gu_mVmth" }, "source": [ "# 係り受けを用いた目的語抽出＋ネットワーク描画の例\n", "自然言語処理における代表的な情報抽出には、(1) 表面的な文字そのものを指定する方法、(2) 品詞を指定する方法、(3) 係り受け関係を指定する方法、(4) 分散表現による距離を参照する方法がある。\n", "\n", "ここでは係り受け関係の例として「目的語 => 係り受け先」という関係を抽出してみよう。例えば「例題を取り入れて理解しやすくしてほしい。」という文においては「例題 => 取り入れ」がこの関係に相当する。この例では「例題」が目的語(UDではtoken.dep_ == obj)となり、「取り入れ」が係り受け先(UDではtoken.head)に相当する。\n", "\n", "- NOTE\n", " - ネットワーク描画のために networkx, pyviz を利用している。これらは pip install でインストール可能。\n", " - networkx はネットワーク描画や分析等に使われるライブラリ。本来ならこれだけで済むことが多いが、日本語には未対応のため pyviz (グラフ描画ライブラリ）も利用している。\n", " - [pyvis](https://pyvis.readthedocs.io/en/latest/index.html)注意点。\n", " - Google Colab ではやや取り扱いに難がある。具体的には pip install 後に一旦カーネルに再接続し直す必要がある。\n", " - Pythonスクリプトとして実行するか、ノートブック(ipynb)として実行するかによりコードがやや異なる。具体的にはコード内コメントを参照。\n", "- 参考\n", " - [Universal Dependency Relations](https://universaldependencies.org/u/dep/)\n", " - [日本語の構文解析における3つの「係り受け」](http://kanji.zinbun.kyoto-u.ac.jp/~yasuoka/publications/kakariuke.html)" ] }, { "cell_type": "code", "source": [ "# spacy, ginzaインストール\n", "!pip install -U ginza ja_ginza\n", "\n", "# matplotlib で日本語フォントを使うための環境構築\n", "!pip install japanize-matplotlib\n", "!apt-get -y install fonts-ipafont-gothic # 日本語フォント\n", "\n", "# ネットワーク分析\n", "!pip install networkx\n", "!pip install pyvis # グラフ描画ライブラリ。networkxと組み合わせて日本語化対応用。" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "rJfeBKeqVqPF", "outputId": "d76b1d43-8f76-4206-d511-03884b9ca07a" }, "execution_count": 2, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Collecting ginza\n", " Downloading ginza-5.2.0-py3-none-any.whl (21 kB)\n", "Collecting ja_ginza\n", " Downloading ja_ginza-5.2.0-py3-none-any.whl (59.1 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m59.1/59.1 MB\u001b[0m \u001b[31m12.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: spacy<4.0.0,>=3.4.4 in /usr/local/lib/python3.10/dist-packages (from ginza) (3.7.4)\n", "Collecting plac>=1.3.3 (from ginza)\n", " Downloading plac-1.4.3-py2.py3-none-any.whl (22 kB)\n", "Collecting SudachiPy<0.7.0,>=0.6.2 (from ginza)\n", " Downloading SudachiPy-0.6.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.6/2.6 MB\u001b[0m \u001b[31m24.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting SudachiDict-core>=20210802 (from ginza)\n", " Downloading SudachiDict_core-20240409-py3-none-any.whl (72.0 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m72.0/72.0 MB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (3.0.12)\n", "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (1.0.5)\n", "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (1.0.10)\n", "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.0.8)\n", "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (3.0.9)\n", "Requirement already satisfied: thinc<8.3.0,>=8.2.2 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (8.2.3)\n", "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (1.1.3)\n", "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.4.8)\n", "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.0.10)\n", "Requirement already satisfied: weasel<0.4.0,>=0.1.0 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (0.3.4)\n", "Requirement already satisfied: typer<0.10.0,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (0.9.4)\n", "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (6.4.0)\n", "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (4.66.4)\n", "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.31.0)\n", "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.7.3)\n", "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (3.1.4)\n", "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (67.7.2)\n", "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (24.0)\n", "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (3.4.0)\n", "Requirement already satisfied: numpy>=1.19.0 in /usr/local/lib/python3.10/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (1.25.2)\n", "Requirement already satisfied: language-data>=1.2 in /usr/local/lib/python3.10/dist-packages (from langcodes<4.0.0,>=3.2.0->spacy<4.0.0,>=3.4.4->ginza) (1.2.0)\n", "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<4.0.0,>=3.4.4->ginza) (0.7.0)\n", "Requirement already satisfied: pydantic-core==2.18.4 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<4.0.0,>=3.4.4->ginza) (2.18.4)\n", "Requirement already satisfied: typing-extensions>=4.6.1 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<4.0.0,>=3.4.4->ginza) (4.12.1)\n", "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<4.0.0,>=3.4.4->ginza) (3.3.2)\n", "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<4.0.0,>=3.4.4->ginza) (3.7)\n", "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<4.0.0,>=3.4.4->ginza) (2.0.7)\n", "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<4.0.0,>=3.4.4->ginza) (2024.6.2)\n", "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.10/dist-packages (from thinc<8.3.0,>=8.2.2->spacy<4.0.0,>=3.4.4->ginza) (0.7.11)\n", "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from thinc<8.3.0,>=8.2.2->spacy<4.0.0,>=3.4.4->ginza) (0.1.5)\n", "Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.10/dist-packages (from typer<0.10.0,>=0.3.0->spacy<4.0.0,>=3.4.4->ginza) (8.1.7)\n", "Requirement already satisfied: cloudpathlib<0.17.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from weasel<0.4.0,>=0.1.0->spacy<4.0.0,>=3.4.4->ginza) (0.16.0)\n", "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->spacy<4.0.0,>=3.4.4->ginza) (2.1.5)\n", "Requirement already satisfied: marisa-trie>=0.7.7 in /usr/local/lib/python3.10/dist-packages (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy<4.0.0,>=3.4.4->ginza) (1.1.1)\n", "Installing collected packages: SudachiPy, plac, SudachiDict-core, ginza, ja_ginza\n", "Successfully installed SudachiDict-core-20240409 SudachiPy-0.6.8 ginza-5.2.0 ja_ginza-5.2.0 plac-1.4.3\n", "Collecting japanize-matplotlib\n", " Downloading japanize-matplotlib-1.1.3.tar.gz (4.1 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.1/4.1 MB\u001b[0m \u001b[31m28.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n", "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from japanize-matplotlib) (3.7.1)\n", "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->japanize-matplotlib) (1.2.1)\n", "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->japanize-matplotlib) (0.12.1)\n", "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->japanize-matplotlib) (4.53.0)\n", "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->japanize-matplotlib) (1.4.5)\n", "Requirement already satisfied: numpy>=1.20 in /usr/local/lib/python3.10/dist-packages (from matplotlib->japanize-matplotlib) (1.25.2)\n", "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->japanize-matplotlib) (24.0)\n", "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->japanize-matplotlib) (9.4.0)\n", "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->japanize-matplotlib) (3.1.2)\n", "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib->japanize-matplotlib) (2.8.2)\n", "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib->japanize-matplotlib) (1.16.0)\n", "Building wheels for collected packages: japanize-matplotlib\n", " Building wheel for japanize-matplotlib (setup.py) ... \u001b[?25l\u001b[?25hdone\n", " Created wheel for japanize-matplotlib: filename=japanize_matplotlib-1.1.3-py3-none-any.whl size=4120257 sha256=7ff9483c41dc806854d65678a7cbf1c1f67330bd3a9b0067678465822372ecf4\n", " Stored in directory: /root/.cache/pip/wheels/61/7a/6b/df1f79be9c59862525070e157e62b08eab8ece27c1b68fbb94\n", "Successfully built japanize-matplotlib\n", "Installing collected packages: japanize-matplotlib\n", "Successfully installed japanize-matplotlib-1.1.3\n", "Reading package lists... Done\n", "Building dependency tree... Done\n", "Reading state information... Done\n", "The following additional packages will be installed:\n", " fonts-ipafont-mincho\n", "The following NEW packages will be installed:\n", " fonts-ipafont-gothic fonts-ipafont-mincho\n", "0 upgraded, 2 newly installed, 0 to remove and 45 not upgraded.\n", "Need to get 8,237 kB of archives.\n", "After this operation, 28.7 MB of additional disk space will be used.\n", "Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fonts-ipafont-gothic all 00303-21ubuntu1 [3,513 kB]\n", "Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fonts-ipafont-mincho all 00303-21ubuntu1 [4,724 kB]\n", "Fetched 8,237 kB in 1s (8,699 kB/s)\n", "Selecting previously unselected package fonts-ipafont-gothic.\n", "(Reading database ... 121918 files and directories currently installed.)\n", "Preparing to unpack .../fonts-ipafont-gothic_00303-21ubuntu1_all.deb ...\n", "Unpacking fonts-ipafont-gothic (00303-21ubuntu1) ...\n", "Selecting previously unselected package fonts-ipafont-mincho.\n", "Preparing to unpack .../fonts-ipafont-mincho_00303-21ubuntu1_all.deb ...\n", "Unpacking fonts-ipafont-mincho (00303-21ubuntu1) ...\n", "Setting up fonts-ipafont-mincho (00303-21ubuntu1) ...\n", "update-alternatives: using /usr/share/fonts/opentype/ipafont-mincho/ipam.ttf to provide /usr/share/fonts/truetype/fonts-japanese-mincho.ttf (fonts-japanese-mincho.ttf) in auto mode\n", "Setting up fonts-ipafont-gothic (00303-21ubuntu1) ...\n", "update-alternatives: using /usr/share/fonts/opentype/ipafont-gothic/ipag.ttf to provide /usr/share/fonts/truetype/fonts-japanese-gothic.ttf (fonts-japanese-gothic.ttf) in auto mode\n", "Processing triggers for fontconfig (2.13.1-4.2ubuntu5) ...\n", "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (3.3)\n", "Collecting pyvis\n", " Downloading pyvis-0.3.2-py3-none-any.whl (756 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m756.0/756.0 kB\u001b[0m \u001b[31m9.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: ipython>=5.3.0 in /usr/local/lib/python3.10/dist-packages (from pyvis) (7.34.0)\n", "Requirement already satisfied: jinja2>=2.9.6 in /usr/local/lib/python3.10/dist-packages (from pyvis) (3.1.4)\n", "Requirement already satisfied: jsonpickle>=1.4.1 in /usr/local/lib/python3.10/dist-packages (from pyvis) (3.0.4)\n", "Requirement already satisfied: networkx>=1.11 in /usr/local/lib/python3.10/dist-packages (from pyvis) (3.3)\n", "Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.10/dist-packages (from ipython>=5.3.0->pyvis) (67.7.2)\n", "Collecting jedi>=0.16 (from ipython>=5.3.0->pyvis)\n", " Downloading jedi-0.19.1-py2.py3-none-any.whl (1.6 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m57.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: decorator in /usr/local/lib/python3.10/dist-packages (from ipython>=5.3.0->pyvis) (4.4.2)\n", "Requirement already satisfied: pickleshare in /usr/local/lib/python3.10/dist-packages (from ipython>=5.3.0->pyvis) (0.7.5)\n", "Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.10/dist-packages (from ipython>=5.3.0->pyvis) (5.7.1)\n", "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from ipython>=5.3.0->pyvis) (3.0.45)\n", "Requirement already satisfied: pygments in /usr/local/lib/python3.10/dist-packages (from ipython>=5.3.0->pyvis) (2.16.1)\n", "Requirement already satisfied: backcall in /usr/local/lib/python3.10/dist-packages (from ipython>=5.3.0->pyvis) (0.2.0)\n", "Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.10/dist-packages (from ipython>=5.3.0->pyvis) (0.1.7)\n", "Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.10/dist-packages (from ipython>=5.3.0->pyvis) (4.9.0)\n", "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2>=2.9.6->pyvis) (2.1.5)\n", "Requirement already satisfied: parso<0.9.0,>=0.8.3 in /usr/local/lib/python3.10/dist-packages (from jedi>=0.16->ipython>=5.3.0->pyvis) (0.8.4)\n", "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.10/dist-packages (from pexpect>4.3->ipython>=5.3.0->pyvis) (0.7.0)\n", "Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=5.3.0->pyvis) (0.2.13)\n", "Installing collected packages: jedi, pyvis\n", "Successfully installed jedi-0.19.1 pyvis-0.3.2\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "DIDH0eoXVmti" }, "source": [ "## 利用ライブラリの用意、データセット準備\n", "事前に、[load_r_assesment.ipynb](./load_r_assesment.ipynb) でデータセットを作成し、pkl形式でファイル保存(r_assesment.pkl)しておく。今回は作成済みファイルをダウンロードして利用することにする。\n", "\n", "r_assesment.pklは授業評価アンケートの自由記述欄をpd.DataFrame形式で保存したもので、授業名(title)、学年(grade)、必修か否か(required)、質問番号(q_id)、コメント(comment)で構成される。" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "zq_Ua-1bVmtj", "outputId": "5ca768cb-aecb-463e-d886-853e2eaf2dcb" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ " % Total % Received % Xferd Average Speed Time Time Time Current\n", " Dload Upload Total Spent Left Speed\n", "100 34834 100 34834 0 0 11591 0 0:00:03 0:00:03 --:--:-- 11595\n" ] } ], "source": [ "!curl -O https://ie.u-ryukyu.ac.jp/~tnal/2022/dm/static/r_assesment.pkl" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "gwhRVnxpVmtk", "outputId": "7d7fc714-3b68-421e-a9f2-f3ece1dcd9eb" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ " title grade required q_id comment\n", "0 工業数学Ⅰ 1 True Q21 (1) 特になし\n", "1 工業数学Ⅰ 1 True Q21 (2) 正直わかりずらい。むだに間があるし。\n", "2 工業数学Ⅰ 1 True Q21 (2) 例題を取り入れて理解しやすくしてほしい。\n", "3 工業数学Ⅰ 1 True Q21 (2) 特になし\n", "4 工業数学Ⅰ 1 True Q21 (2) スライドに書く文字をもう少しわかりやすくして欲しいです。" ], "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titlegraderequiredq_idcomment
0工業数学Ⅰ1TrueQ21 (1)特になし
1工業数学Ⅰ1TrueQ21 (2)正直わかりずらい。むだに間があるし。
2工業数学Ⅰ1TrueQ21 (2)例題を取り入れて理解しやすくしてほしい。
3工業数学Ⅰ1TrueQ21 (2)特になし
4工業数学Ⅰ1TrueQ21 (2)スライドに書く文字をもう少しわかりやすくして欲しいです。
\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", " \n", "\n", "\n", "\n", " \n", "
\n", "
\n", "
\n" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "dataframe", "variable_name": "assesment_df", "summary": "{\n \"name\": \"assesment_df\",\n \"rows\": 170,\n \"fields\": [\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 16,\n \"samples\": [\n \"\\u5de5\\u696d\\u6570\\u5b66\\u2160\",\n \"\\u6280\\u8853\\u8005\\u306e\\u502b\\u7406\",\n \"\\u30a2\\u30eb\\u30b4\\u30ea\\u30ba\\u30e0\\u3068\\u30c7\\u30fc\\u30bf\\u69cb\\u9020\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"grade\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0,\n \"min\": 1,\n \"max\": 3,\n \"num_unique_values\": 3,\n \"samples\": [\n 1,\n 2,\n 3\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"required\",\n \"properties\": {\n \"dtype\": \"boolean\",\n \"num_unique_values\": 2,\n \"samples\": [\n false,\n true\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"q_id\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Q21 (2)\",\n \"Q22\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"comment\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 153,\n \"samples\": [\n \"\\u30fb\\u6559\\u79d1\\u66f8\\u304c\\u5fc5\\u8981\\u306a\\u306e\\u304b\\u5fc5\\u8981\\u3067\\u306a\\u3044\\u306e\\u304b\\u304c\\u66d6\\u6627\\u306a\\u307e\\u307e\\u6388\\u696d\\u304c\\u59cb\\u307e\\u308a\\u3001\\u975e\\u5e38\\u306b\\u4e0d\\u5b89\\u3060\\u3063\\u305f\\u305f\\u3081\\u3001\\u6559\\u79d1\\u66f8\\u304c\\u5fc5\\u9808\\u304b\\u305d\\u3046\\u3067\\u306a\\u3044\\u306e\\u304b\\u306f\\u6700\\u521d\\u306b\\u306f\\u3063\\u304d\\u308a\\u3057\\u3066\\u6b32\\u3057\\u3044\\u3002\\r\\n\\u30fb\\u8ab2\\u984c\\u3092\\u51fa\\u3059\\u3060\\u3051\\u51fa\\u3055\\u305b\\u3066\\u304a\\u3044\\u3066\\u3001\\u63a1\\u70b9\\u3082\\u305b\\u305a\\u3001\\u3069\\u3046\\u3044\\u3063\\u305f\\u89e3\\u7b54\\u304c\\u6b63\\u3057\\u3044\\u306e\\u304b\\u3068\\u3044\\u3063\\u305f\\u6307\\u91dd\\u3082\\u51fa\\u3059\\u306e\\u304c\\u3068\\u3066\\u3082\\u9045\\u3044\\u3002\\u8ab2\\u984c\\u306f\\u89e3\\u304f\\u3060\\u3051\\u3067\\u306f\\u77e5\\u8b58\\u306e\\u5b9a\\u7740\\u306b\\u3064\\u306a\\u304c\\u3089\\u306a\\u3044\\u3068\\u601d\\u3044\\u307e\\u3059\\u304c\\u3001\\u305d\\u3053\\u3089\\u3078\\u3093\\u306f\\u3069\\u3046\\u306a\\u3093\\u3067\\u3057\\u3087\\u3046\\u304b\\u3002\\r\\n\\u30fb\\u914d\\u5e03\\u8cc7\\u6599\\u3068\\u3057\\u3066\\u3001\\u904e\\u53bb\\u554f\\u3082\\u914d\\u5e03\\u3057\\u3066\\u304f\\u308c\\u308b\\u3068\\u3068\\u3066\\u3082\\u52a9\\u304b\\u308b\\u306a\\u3001\\u3068\\u601d\\u3044\\u307e\\u3059\\u3002\\u3054\\u691c\\u8a0e\\u304a\\u9858\\u3044\\u3057\\u307e\\u3059\\u3002\",\n \"\\u30fb\\u4e2d\\u9593\\u30c6\\u30b9\\u30c8\\u3092\\u5ef6\\u671f\\u3057\\u7d9a\\u3051\\u3001\\u6700\\u7d42\\u7684\\u306b\\u4e2d\\u9593\\u30fb\\u671f\\u672b\\u8a66\\u9a13\\u3092\\uff12\\u9031\\u7d9a\\u3051\\u3066\\u3084\\u308b\\u3053\\u3068\\u3068\\u306a\\u308a\\u3001\\u8a08\\u753b\\u6027\\u304c\\u6b20\\u3051\\u3066\\u3044\\u308b\\u3002\\r\\n\\u30fb\\u914d\\u5e03\\u8cc7\\u6599\\u306e\\u8aa4\\u5b57\\u8131\\u5b57\\u304c\\u591a\\u3059\\u304e\\u308b\\u3002\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}" } }, "metadata": {}, "execution_count": 4 } ], "source": [ "import collections\n", "import numpy as np\n", "import pandas as pd\n", "import spacy\n", "\n", "import networkx as nx\n", "import matplotlib.pyplot as plt\n", "import japanize_matplotlib\n", "\n", "nlp = spacy.load(\"ja_ginza\")\n", "\n", "assesment_df = pd.read_pickle('r_assesment.pkl')\n", "assesment_df.head()" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 420 }, "id": "PIMsHNoWVmtk", "outputId": "38e5072c-ec4a-4604-b470-13f5591311f2" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "sample = 例題を取り入れて理解しやすくしてほしい。\n" ] }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "\n", "\n", " 例題\n", " NOUN\n", "\n", "\n", "\n", " を\n", " ADP\n", "\n", "\n", "\n", " 取り入れ\n", " VERB\n", "\n", "\n", "\n", " て\n", " SCONJ\n", "\n", "\n", "\n", " 理解\n", " NOUN\n", "\n", "\n", "\n", " し\n", " AUX\n", "\n", "\n", "\n", " やすく\n", " AUX\n", "\n", "\n", "\n", " し\n", " VERB\n", "\n", "\n", "\n", " て\n", " SCONJ\n", "\n", "\n", "\n", " ほしい。\n", " AUX\n", "\n", "\n", "\n", " \n", " \n", " obj\n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " case\n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " advcl\n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " mark\n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " obl\n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " aux\n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " aux\n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " mark\n", " \n", " \n", "\n", "\n", "\n", " \n", " \n", " fixed\n", " \n", " \n", "\n", "" ] }, "metadata": {} } ], "source": [ "from spacy import displacy\n", "\n", "sample = assesment_df['comment'][2]\n", "print('sample = ', sample)\n", "\n", "doc = nlp(sample)\n", "displacy.render(doc, style=\"dep\", options={\"compact\":True})" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Rt6Au7QRVmtl", "outputId": "a4f0b284-8129-44be-a497-88cbcb451bd8" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "例題 取り入れ\n" ] } ], "source": [ "from spacy.symbols import obj\n", "\n", "for token in doc:\n", " if token.dep == obj:\n", " print(token.text, token.head.text)" ] }, { "cell_type": "markdown", "metadata": { "id": "bcxeVZdnVmtl" }, "source": [ "## 目的語とその係り受け先を抽出してみよう\n", "{'例題_取り入れ':1], ,,,}のように obj と token.head をアンダースコアで結んだ文字列を作り、その文字列が出現した回数も同時にカウントしてみる。" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Sfu76-EWVmtm", "outputId": "17ecaeaf-4546-4bb2-8cb2-f64c836e68aa" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Counter({'課題_出す': 3, '講義_受ける': 3, 'こと_学ぶ': 2, '講義_通す': 2, '話_聞く': 2, '予習_する': 2, '解答_出す': 2, '単位_落とす': 2, '知識_獲得': 2, '仮説_立てる': 2, '手法_使う': 2, '例題_取り入れる': 1, '文字_する': 1, '行列_扱う': 1, '演習_出す': 1, '時間_おく': 1, '何_話す': 1, '道徳_学ぶ': 1, '想い_考える': 1, '品性_伸ばす': 1, 'こと_学べる': 1, 'こと_知れる': 1, '考え方_持てる': 1, '興味_持つ': 1, '心_学べる': 1, '側面_見れる': 1, 'これ_考える': 1, '計画_立てる': 1, '将来_想像': 1, 'アドバイス_もらえる': 1, 'マナー_学べる': 1, 'ある_知る': 1, 'ニーズ_聞ける': 1, '友達_作る': 1, '作成_通す': 1, '将来_見通す': 1, '機会_設ける': 1, '対面_増やす': 1, '課題_変更': 1, 'プログラミング_学べる': 1, 'それ_完成': 1, '復習_こなせる': 1, '基礎_学ぶ': 1, '課題_見る': 1, 'コード_良い': 1, 'プログラミング_触る': 1, '予想_上回る': 1, '復習_する': 1, '言語_学ぶ': 1, '課題_やる': 1, 'プログラミング_学ぶ': 1, 'こと_覚える': 1, '知識_生かす': 1, '演習_通す': 1, '基礎_定着': 1, 'デンチュウ_動かす': 1, 'denchu_通す': 1, '方_知る': 1, 'コード_書く': 1, 'denchu_動かす': 1, 'プログラミング_楽しい': 1, 'こと_実践': 1, '事_落とし込む': 1, 'メモ書き_消す': 1, 'こと_書く': 1, 'こと_聞く': 1, '環境_作る': 1, '点数_教える': 1, '提出_行う': 1, 'コマンド_打つ': 1, '方_招待': 1, '資料_理解': 1, '授業_理解': 1, 'テスト_延期': 1, '試験_続ける': 1, 'まとめ方_する': 1, '内容_補強': 1, '不便_感ずる': 1, 'メール_読む': 1, 'チャンネル_開設': 1, '内容_アウトプット': 1, 'ペン_使う': 1, '説明_聞く': 1, '何_言う': 1, '点数_公開': 1, '点_送る': 1, '採点_願う': 1, '試験_実施': 1, '試験_行なう': 1, 'ら_用いる': 1, '勉強_する': 1, 'ところ_調べる': 1, '採点_使う': 1, 'プログラム_作る': 1, '内容_網羅': 1, '提出期間_過ぎる': 1, '試験_する': 1, '解決策_考える': 1, '問題_解決': 1, '試験_受ける': 1, '環境_整える': 1, '回答_出す': 1, '理解_測る': 1, 'こと_する': 1, '試験_もつ': 1, '間違い_指摘': 1, '授業_受ける': 1, '説明_書く': 1, '難易度_する': 1, '過去問_配布': 1, '要点_押さえる': 1, '不満_書く': 1, '部分_ミス': 1, 'こと_望む': 1, '自学_増やす': 1, '質問_する': 1, '説明_する': 1, '動き_理解': 1, '欲_言う': 1, '緊張感_持てる': 1, '課題_再提出': 1, '間隔_空ける': 1, '時間_作る': 1, '知識_利用': 1, '知識_使う': 1, '研究_踏まえる': 1, '講義_する': 1, '録画_残す': 1, '大切_学ぶ': 1, '成績_決定': 1, '資料_写す': 1, '使う_学ぶ': 1, '配分_間違える': 1, '結果_出す': 1, 'こと_使う': 1, 'こと_実感': 1, 'アイディア_出す': 1, '物事_伝える': 1, 'スキル_向上': 1, '書_紹介': 1, '力_活かす': 1, '情報_得る': 1, '時間_とる': 1, '顔_合わせる': 1, '説明_行く': 1, '手_焼く': 1, '報告会_行う': 1, 'javascript_用いる': 1, 'アプリケーション_開発': 1, 'コーディング_教える': 1, 'フレームワーク_試す': 1})\n" ] } ], "source": [ "from spacy.symbols import obj\n", "\n", "result = {}\n", "for text in assesment_df['comment']:\n", " doc = nlp(text)\n", " for token in doc:\n", " if token.dep == obj:\n", " target = token.lemma_ + '_' + token.head.lemma_\n", " if target not in result:\n", " result[target] = 1\n", " else:\n", " result[target] += 1\n", "\n", "print(collections.Counter(result))" ] }, { "cell_type": "markdown", "metadata": { "id": "xllb0fpQVmtm" }, "source": [ "## ネットワーク描画してみる\n", "「目的語=>係り受け先単語」の関係を「単語間動詞を接続したエッジ」として可視化してみよう。まずはpd.DataFrame形式で「係り受け元単語、係り受け先単語、出現回数」として保存し直す。" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 419 }, "id": "6O0fSdulVmtm", "outputId": "ad5570be-7926-4c13-b38c-82dff8440962" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ " from to weight\n", "0 例題 取り入れる 1\n", "1 文字 する 1\n", "2 行列 扱う 1\n", "3 演習 出す 1\n", "4 時間 おく 1\n", ".. ... ... ...\n", "146 報告会 行う 1\n", "147 javascript 用いる 1\n", "148 アプリケーション 開発 1\n", "149 コーディング 教える 1\n", "150 フレームワーク 試す 1\n", "\n", "[151 rows x 3 columns]" ], "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
fromtoweight
0例題取り入れる1
1文字する1
2行列扱う1
3演習出す1
4時間おく1
............
146報告会行う1
147javascript用いる1
148アプリケーション開発1
149コーディング教える1
150フレームワーク試す1
\n", "

151 rows × 3 columns

\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", " \n", "\n", "\n", "\n", " \n", "
\n", "
\n", "
\n" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "dataframe", "variable_name": "df", "summary": "{\n \"name\": \"df\",\n \"rows\": 151,\n \"fields\": [\n {\n \"column\": \"from\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 104,\n \"samples\": [\n \"\\u57fa\\u790e\",\n \"\\u89e3\\u6c7a\\u7b56\",\n \"\\u63d0\\u51fa\\u671f\\u9593\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"to\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 97,\n \"samples\": [\n \"\\u8abf\\u3079\\u308b\",\n \"\\u6d88\\u3059\",\n \"\\u884c\\u304f\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"weight\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0,\n \"min\": 1,\n \"max\": 3,\n \"num_unique_values\": 3,\n \"samples\": [\n 1,\n 2,\n 3\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}" } }, "metadata": {}, "execution_count": 8 } ], "source": [ "columns = ['from', 'to', 'weight']\n", "df = pd.DataFrame(columns=columns)\n", "\n", "index = 0\n", "nodes = []\n", "for key, value in result.items():\n", " node1, node2 = key.split('_')\n", " if node1 not in nodes:\n", " nodes.append(node1)\n", " if node2 not in nodes:\n", " nodes.append(node2)\n", " df.loc[index] = [node1, node2, value]\n", " index += 1\n", "\n", "df" ] }, { "cell_type": "markdown", "metadata": { "id": "8-8Kd9RnVmtn" }, "source": [ "## networkx + pyvizで描画\n", "- [networkx](https://networkx.org)はネットワークの描画・操作・解析などに使われるライブラリ。\n", "- [pyviz](https://pyviz.org)はグラフ描画ライブラリ。\n", "- グラフ描画するだけなら networkx + matplotlib でも可能だが、日本語には未対応で文字化けしてしまうため、ここではpyvizを採用している。" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "id": "YVbbTpP1Vmtn" }, "outputs": [], "source": [ "import networkx as nx\n", "from pyvis.network import Network\n", "\n", "G = nx.from_pandas_edgelist(df, source='from', target='to', edge_attr='weight')\n", "\n", "pyvis_G = Network(notebook=True, cdn_resources='in_line') # .pyで実行するならFalse\n", "pyvis_G.from_nx(G)\n", "pyvis_G.save_graph(\"mygraph.html\")\n", "#pyvis_G.show('mygraph.html')" ] }, { "cell_type": "markdown", "source": [ "## グラフ参照する手順\n", "- \"mygraph.html\" をダウンロード\n", "- ブラウザで新規タブを作成\n", "- mygraph.html をダブルクリック。もしくはブラウザ新規タブにドラッグ＆ドロップ。" ], "metadata": { "id": "91HYOVKYTtDf" } } ], "metadata": { "interpreter": { "hash": "880b2a8c90f9e6beae80b56829e3f671fedd58b6d14887184ddce26124cedfbd" }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.9" }, "colab": { "provenance": [] } }, "nbformat": 4, "nbformat_minor": 0 }