{ "cells": [ { "cell_type": "code", "source": [ "!date\n", "!python --version" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Rsu3rv5lhpUt", "outputId": "a387fb6f-7e53-4f19-cac0-01d4e0d846cb" }, "execution_count": 1, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Wed Jun 3 02:13:29 AM UTC 2026\n", "Python 3.12.13\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "v8IkNtWTYpm6" }, "source": [ "# トピックモデルによるクラスタリング\n", "トピックモデルとは文書中の単語出現分布を元に傾向（≒トピックらしきもの）を観察しようとするアプローチで、クラスタリングの一種である。なお、一般的なクラスタリング（例えば[k平均法](https://ja.wikipedia.org/wiki/K平均法)）では一つのサンプルが一つのクラスタに属するという前提でグルーピングを行うのに対し、トピックモデルでは一つのサンプルが複数のクラスタを内包しているという前提でグルーピングを行う。次の例を眺めるとイメージをつかみやすいだろう。\n", "\n", "- 例1: [トピックモデル入門：WikipediaをLDAモデル化してみた](https://recruit.gmo.jp/engineer/jisedai/blog/topic-model/)\n", "- 例2: [Wikipedia: Topic model](https://en.wikipedia.org/wiki/Topic_model)\n", "\n", "基本的には文書を BoW (CountVectrizor) やそれの重みを調整した TF-IDF 等の「文書単語行列」を作成し、ここから文書館類似度や単語間類似度を元に集約（≒次元削減）を試みる。文書単語行列の作成方法や次元削減方法、類似度の求め方などで様々なアルゴリズムが提案されている。ここでは (1) BowベースのLDAと、(2) TF-IDFベースのLDAを行い、それぞれどのようなトピックが出てくるのか眺めてみよう。\n", "\n", "なお、トピックモデルの注意点として、**トピックそのものは人手による解釈が求められる** 点が挙げられる。例えば先に上げた[トピックモデル入門：WikipediaをLDAモデル化してみた](https://recruit.gmo.jp/engineer/jisedai/blog/topic-model/)における図2（下図）では「政治」「スポーツ」「国際」といったトピックが並んでいるが、実際には「4-1. トピック観察」を行う必要がある。実際に観察してみよう。" ] }, { "cell_type": "code", "source": [ "# spacy, ginza インストール\n", "!pip install -U ginza ja_ginza\n", "\n", "# plotlyで作図した図をファイル出力するためのパッケージ\n", "#!pip install -U kaleido" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "PNA5onfzhrrq", "outputId": "47874e01-bdcf-4955-950b-4c1972e1b1a0" }, "execution_count": 2, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Collecting ginza\n", " Downloading ginza-5.2.0-py3-none-any.whl.metadata (448 bytes)\n", "Collecting ja_ginza\n", " Downloading ja_ginza-5.2.0-py3-none-any.whl.metadata (5.8 kB)\n", "Requirement already satisfied: spacy<4.0.0,>=3.4.4 in /usr/local/lib/python3.12/dist-packages (from ginza) (3.8.14)\n", "Collecting plac>=1.3.3 (from ginza)\n", " Downloading plac-1.4.5-py2.py3-none-any.whl.metadata (5.9 kB)\n", "Collecting SudachiPy<0.7.0,>=0.6.2 (from ginza)\n", " Downloading sudachipy-0.6.11-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (12 kB)\n", "Collecting SudachiDict-core>=20210802 (from ginza)\n", " Downloading sudachidict_core-20260428-py3-none-any.whl.metadata (2.7 kB)\n", "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (3.0.12)\n", "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (1.0.5)\n", "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (1.0.15)\n", "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.0.13)\n", "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (3.0.13)\n", "Requirement already satisfied: thinc<8.4.0,>=8.3.12 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (8.3.13)\n", "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (1.1.3)\n", "Requirement already satisfied: srsly<3.0.0,>=2.5.3 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.5.3)\n", "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.0.10)\n", "Requirement already satisfied: weasel<2.0.0,>=1.0.0 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (1.0.0)\n", "Requirement already satisfied: confection<2.0.0,>=1.3.2 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (1.3.3)\n", "Requirement already satisfied: typer<1.0.0,>=0.3.0 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (0.25.1)\n", "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (4.67.3)\n", "Requirement already satisfied: numpy>=1.19.0 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.0.2)\n", "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.32.4)\n", "Requirement already satisfied: pydantic<3.0.0,>=2.0.0 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (2.12.3)\n", "Requirement already satisfied: jinja2 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (3.1.6)\n", "Requirement already satisfied: setuptools in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (75.2.0)\n", "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.12/dist-packages (from spacy<4.0.0,>=3.4.4->ginza) (26.2)\n", "Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.12/dist-packages (from pydantic<3.0.0,>=2.0.0->spacy<4.0.0,>=3.4.4->ginza) (0.7.0)\n", "Requirement already satisfied: pydantic-core==2.41.4 in /usr/local/lib/python3.12/dist-packages (from pydantic<3.0.0,>=2.0.0->spacy<4.0.0,>=3.4.4->ginza) (2.41.4)\n", "Requirement already satisfied: typing-extensions>=4.14.1 in /usr/local/lib/python3.12/dist-packages (from pydantic<3.0.0,>=2.0.0->spacy<4.0.0,>=3.4.4->ginza) (4.15.0)\n", "Requirement already satisfied: typing-inspection>=0.4.2 in /usr/local/lib/python3.12/dist-packages (from pydantic<3.0.0,>=2.0.0->spacy<4.0.0,>=3.4.4->ginza) (0.4.2)\n", "Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.13.0->spacy<4.0.0,>=3.4.4->ginza) (3.4.7)\n", "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.13.0->spacy<4.0.0,>=3.4.4->ginza) (3.15)\n", "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.13.0->spacy<4.0.0,>=3.4.4->ginza) (2.5.0)\n", "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.12/dist-packages (from requests<3.0.0,>=2.13.0->spacy<4.0.0,>=3.4.4->ginza) (2026.5.20)\n", "Requirement already satisfied: blis<1.4.0,>=1.3.0 in /usr/local/lib/python3.12/dist-packages (from thinc<8.4.0,>=8.3.12->spacy<4.0.0,>=3.4.4->ginza) (1.3.3)\n", "Requirement already satisfied: click>=8.2.1 in /usr/local/lib/python3.12/dist-packages (from typer<1.0.0,>=0.3.0->spacy<4.0.0,>=3.4.4->ginza) (8.4.0)\n", "Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.12/dist-packages (from typer<1.0.0,>=0.3.0->spacy<4.0.0,>=3.4.4->ginza) (1.5.4)\n", "Requirement already satisfied: rich>=13.8.0 in /usr/local/lib/python3.12/dist-packages (from typer<1.0.0,>=0.3.0->spacy<4.0.0,>=3.4.4->ginza) (13.9.4)\n", "Requirement already satisfied: annotated-doc>=0.0.2 in /usr/local/lib/python3.12/dist-packages (from typer<1.0.0,>=0.3.0->spacy<4.0.0,>=3.4.4->ginza) (0.0.4)\n", "Requirement already satisfied: cloudpathlib>=0.7.0 in /usr/local/lib/python3.12/dist-packages (from weasel<2.0.0,>=1.0.0->spacy<4.0.0,>=3.4.4->ginza) (0.24.0)\n", "Requirement already satisfied: smart-open>=5.2.1 in /usr/local/lib/python3.12/dist-packages (from weasel<2.0.0,>=1.0.0->spacy<4.0.0,>=3.4.4->ginza) (7.6.1)\n", "Requirement already satisfied: httpx>=0.24.0 in /usr/local/lib/python3.12/dist-packages (from weasel<2.0.0,>=1.0.0->spacy<4.0.0,>=3.4.4->ginza) (0.28.1)\n", "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.12/dist-packages (from jinja2->spacy<4.0.0,>=3.4.4->ginza) (3.0.3)\n", "Requirement already satisfied: anyio in /usr/local/lib/python3.12/dist-packages (from httpx>=0.24.0->weasel<2.0.0,>=1.0.0->spacy<4.0.0,>=3.4.4->ginza) (4.13.0)\n", "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.12/dist-packages (from httpx>=0.24.0->weasel<2.0.0,>=1.0.0->spacy<4.0.0,>=3.4.4->ginza) (1.0.9)\n", "Requirement already satisfied: h11>=0.16 in /usr/local/lib/python3.12/dist-packages (from httpcore==1.*->httpx>=0.24.0->weasel<2.0.0,>=1.0.0->spacy<4.0.0,>=3.4.4->ginza) (0.16.0)\n", "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.12/dist-packages (from rich>=13.8.0->typer<1.0.0,>=0.3.0->spacy<4.0.0,>=3.4.4->ginza) (4.2.0)\n", "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.12/dist-packages (from rich>=13.8.0->typer<1.0.0,>=0.3.0->spacy<4.0.0,>=3.4.4->ginza) (2.20.0)\n", "Requirement already satisfied: wrapt in /usr/local/lib/python3.12/dist-packages (from smart-open>=5.2.1->weasel<2.0.0,>=1.0.0->spacy<4.0.0,>=3.4.4->ginza) (2.2.0)\n", "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.12/dist-packages (from markdown-it-py>=2.2.0->rich>=13.8.0->typer<1.0.0,>=0.3.0->spacy<4.0.0,>=3.4.4->ginza) (0.1.2)\n", "Downloading ginza-5.2.0-py3-none-any.whl (21 kB)\n", "Downloading ja_ginza-5.2.0-py3-none-any.whl (59.1 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m59.1/59.1 MB\u001b[0m \u001b[31m8.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hDownloading plac-1.4.5-py2.py3-none-any.whl (22 kB)\n", "Downloading sudachidict_core-20260428-py3-none-any.whl (72.2 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m72.2/72.2 MB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hDownloading sudachipy-0.6.11-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (1.6 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m24.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hInstalling collected packages: SudachiPy, plac, SudachiDict-core, ginza, ja_ginza\n", "Successfully installed SudachiDict-core-20260428 SudachiPy-0.6.11 ginza-5.2.0 ja_ginza-5.2.0 plac-1.4.5\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "zXJ0jzI1Ypm-" }, "source": [ "## データの準備\n", "これまで見てきたいつものやつ。" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "0DaZZaFJYpm-", "outputId": "fb392f64-c091-455b-b66d-8d90452269a3" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ " % Total % Received % Xferd Average Speed Time Time Time Current\n", " Dload Upload Total Spent Left Speed\n", "100 34834 100 34834 0 0 14587 0 0:00:02 0:00:02 --:--:-- 14593\n" ] } ], "source": [ "!curl -O https://ie.u-ryukyu.ac.jp/~tnal/2022/dm/static/r_assesment.pkl" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "qCxzwznFYpnA", "outputId": "e10559e7-4d80-4ed7-8faa-258db30179cb" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ " title grade required q_id comment\n", "0 工業数学Ⅰ 1 True Q21 (1) 特になし\n", "1 工業数学Ⅰ 1 True Q21 (2) 正直わかりずらい。むだに間があるし。\n", "2 工業数学Ⅰ 1 True Q21 (2) 例題を取り入れて理解しやすくしてほしい。\n", "3 工業数学Ⅰ 1 True Q21 (2) 特になし\n", "4 工業数学Ⅰ 1 True Q21 (2) スライドに書く文字をもう少しわかりやすくして欲しいです。" ], "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titlegraderequiredq_idcomment
0工業数学Ⅰ1TrueQ21 (1)特になし
1工業数学Ⅰ1TrueQ21 (2)正直わかりずらい。むだに間があるし。
2工業数学Ⅰ1TrueQ21 (2)例題を取り入れて理解しやすくしてほしい。
3工業数学Ⅰ1TrueQ21 (2)特になし
4工業数学Ⅰ1TrueQ21 (2)スライドに書く文字をもう少しわかりやすくして欲しいです。
\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", "
\n" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "dataframe", "variable_name": "assesment_df", "summary": "{\n \"name\": \"assesment_df\",\n \"rows\": 170,\n \"fields\": [\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 16,\n \"samples\": [\n \"\\u5de5\\u696d\\u6570\\u5b66\\u2160\",\n \"\\u6280\\u8853\\u8005\\u306e\\u502b\\u7406\",\n \"\\u30a2\\u30eb\\u30b4\\u30ea\\u30ba\\u30e0\\u3068\\u30c7\\u30fc\\u30bf\\u69cb\\u9020\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"grade\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0,\n \"min\": 1,\n \"max\": 3,\n \"num_unique_values\": 3,\n \"samples\": [\n 1,\n 2,\n 3\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"required\",\n \"properties\": {\n \"dtype\": \"boolean\",\n \"num_unique_values\": 2,\n \"samples\": [\n false,\n true\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"q_id\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Q21 (2)\",\n \"Q22\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"comment\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 153,\n \"samples\": [\n \"\\u30fb\\u6559\\u79d1\\u66f8\\u304c\\u5fc5\\u8981\\u306a\\u306e\\u304b\\u5fc5\\u8981\\u3067\\u306a\\u3044\\u306e\\u304b\\u304c\\u66d6\\u6627\\u306a\\u307e\\u307e\\u6388\\u696d\\u304c\\u59cb\\u307e\\u308a\\u3001\\u975e\\u5e38\\u306b\\u4e0d\\u5b89\\u3060\\u3063\\u305f\\u305f\\u3081\\u3001\\u6559\\u79d1\\u66f8\\u304c\\u5fc5\\u9808\\u304b\\u305d\\u3046\\u3067\\u306a\\u3044\\u306e\\u304b\\u306f\\u6700\\u521d\\u306b\\u306f\\u3063\\u304d\\u308a\\u3057\\u3066\\u6b32\\u3057\\u3044\\u3002\\r\\n\\u30fb\\u8ab2\\u984c\\u3092\\u51fa\\u3059\\u3060\\u3051\\u51fa\\u3055\\u305b\\u3066\\u304a\\u3044\\u3066\\u3001\\u63a1\\u70b9\\u3082\\u305b\\u305a\\u3001\\u3069\\u3046\\u3044\\u3063\\u305f\\u89e3\\u7b54\\u304c\\u6b63\\u3057\\u3044\\u306e\\u304b\\u3068\\u3044\\u3063\\u305f\\u6307\\u91dd\\u3082\\u51fa\\u3059\\u306e\\u304c\\u3068\\u3066\\u3082\\u9045\\u3044\\u3002\\u8ab2\\u984c\\u306f\\u89e3\\u304f\\u3060\\u3051\\u3067\\u306f\\u77e5\\u8b58\\u306e\\u5b9a\\u7740\\u306b\\u3064\\u306a\\u304c\\u3089\\u306a\\u3044\\u3068\\u601d\\u3044\\u307e\\u3059\\u304c\\u3001\\u305d\\u3053\\u3089\\u3078\\u3093\\u306f\\u3069\\u3046\\u306a\\u3093\\u3067\\u3057\\u3087\\u3046\\u304b\\u3002\\r\\n\\u30fb\\u914d\\u5e03\\u8cc7\\u6599\\u3068\\u3057\\u3066\\u3001\\u904e\\u53bb\\u554f\\u3082\\u914d\\u5e03\\u3057\\u3066\\u304f\\u308c\\u308b\\u3068\\u3068\\u3066\\u3082\\u52a9\\u304b\\u308b\\u306a\\u3001\\u3068\\u601d\\u3044\\u307e\\u3059\\u3002\\u3054\\u691c\\u8a0e\\u304a\\u9858\\u3044\\u3057\\u307e\\u3059\\u3002\",\n \"\\u30fb\\u4e2d\\u9593\\u30c6\\u30b9\\u30c8\\u3092\\u5ef6\\u671f\\u3057\\u7d9a\\u3051\\u3001\\u6700\\u7d42\\u7684\\u306b\\u4e2d\\u9593\\u30fb\\u671f\\u672b\\u8a66\\u9a13\\u3092\\uff12\\u9031\\u7d9a\\u3051\\u3066\\u3084\\u308b\\u3053\\u3068\\u3068\\u306a\\u308a\\u3001\\u8a08\\u753b\\u6027\\u304c\\u6b20\\u3051\\u3066\\u3044\\u308b\\u3002\\r\\n\\u30fb\\u914d\\u5e03\\u8cc7\\u6599\\u306e\\u8aa4\\u5b57\\u8131\\u5b57\\u304c\\u591a\\u3059\\u304e\\u308b\\u3002\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}" } }, "metadata": {}, "execution_count": 4 } ], "source": [ "import collections\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import spacy\n", "from wordcloud import WordCloud\n", "\n", "# Python 3.12 + Spacy 3.8 + Ginza 5.2 の構成だとそのままでは動作しないため、\n", "# 以下の設定を追加指定\n", "config = {\n", " \"components\": {\n", " \"compound_splitter\": {\n", " \"split_mode\": \"A\"\n", " }\n", " }\n", "}\n", "nlp = spacy.load(\"ja_ginza\", config=config)\n", "\n", "assesment_df = pd.read_pickle('r_assesment.pkl')\n", "assesment_df.head()" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 674 }, "id": "Stv68NvmYpnB", "outputId": "83491437-92f3-488e-827a-7d7776979f59" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ " title grade required q_id \\\n", "0 工業数学Ⅰ 1 True Q21 (1) \n", "1 工業数学Ⅰ 1 True Q21 (2) \n", "2 工業数学Ⅰ 1 True Q21 (2) \n", "3 工業数学Ⅰ 1 True Q21 (2) \n", "4 工業数学Ⅰ 1 True Q21 (2) \n", ".. ... ... ... ... \n", "165 データマイニング 3 False Q22 \n", "166 ICT実践英語Ⅰ 3 False Q22 \n", "167 知能情報実験Ⅲ 3 True Q21 (2) \n", "168 知能情報実験Ⅲ 3 True Q22 \n", "169 知能情報実験Ⅲ 3 True Q22 \n", "\n", " comment \\\n", "0 特になし \n", "1 正直わかりずらい。むだに間があるし。 \n", "2 例題を取り入れて理解しやすくしてほしい。 \n", "3 特になし \n", "4 スライドに書く文字をもう少しわかりやすくして欲しいです。 \n", ".. ... \n", "165 課題が難しいものが多く、時間を多くとってもらえたのは非常に良かったですがかなりきつかったです... \n", "166 オンラインなどで顔を合わせてやりたかったです。 \n", "167 unityの操作方法の説明などを最初に行ってもらえたらもう少しスムーズにできたのではないかと思う。 \n", "168 それぞれに任せるといった形で進められたものだったのでそれなりに進めやすかったですが、オンライ... \n", "169 モバイルアプリ班\\r\\nHTML/CSS，JavaScriptなどを用いてアプリケーションを... \n", "\n", " wakati \n", "0 特に なし \n", "1 正直 わかる ずらい むだ 間 ある \n", "2 例題 取り入れる 理解 する \n", "3 特に なし \n", "4 スライド 書く 文字 もう 少し わかる する \n", ".. ... \n", "165 課題 難しい もの 多い 時間 多い とる もらえる 非常 良い かなり きつい ござる \n", "166 オンライン 顔 合わせる やる \n", "167 unity 操作方法 説明 最初 行く もらえる もう 少し スムーズ できる 思う \n", "168 それぞれ 任せる いう 形 進める もの なり 進める オンライン 班 員 指導 全く する... \n", "169 モバイルアプリ 班 \\r\\n HTML CSS javascript 用いる アプリケーショ... \n", "\n", "[170 rows x 6 columns]" ], "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titlegraderequiredq_idcommentwakati
0工業数学Ⅰ1TrueQ21 (1)特になし特に なし
1工業数学Ⅰ1TrueQ21 (2)正直わかりずらい。むだに間があるし。正直 わかる ずらい むだ 間 ある
2工業数学Ⅰ1TrueQ21 (2)例題を取り入れて理解しやすくしてほしい。例題 取り入れる 理解 する
3工業数学Ⅰ1TrueQ21 (2)特になし特に なし
4工業数学Ⅰ1TrueQ21 (2)スライドに書く文字をもう少しわかりやすくして欲しいです。スライド 書く 文字 もう 少し わかる する
.....................
165データマイニング3FalseQ22課題が難しいものが多く、時間を多くとってもらえたのは非常に良かったですがかなりきつかったです...課題 難しい もの 多い 時間 多い とる もらえる 非常 良い かなり きつい ござる
166ICT実践英語Ⅰ3FalseQ22オンラインなどで顔を合わせてやりたかったです。オンライン 顔 合わせる やる
167知能情報実験Ⅲ3TrueQ21 (2)unityの操作方法の説明などを最初に行ってもらえたらもう少しスムーズにできたのではないかと思う。unity 操作方法 説明 最初 行く もらえる もう 少し スムーズ できる 思う
168知能情報実験Ⅲ3TrueQ22それぞれに任せるといった形で進められたものだったのでそれなりに進めやすかったですが、オンライ...それぞれ 任せる いう 形 進める もの なり 進める オンライン 班 員 指導 全く する...
169知能情報実験Ⅲ3TrueQ22モバイルアプリ班\\r\\nHTML/CSS，JavaScriptなどを用いてアプリケーションを...モバイルアプリ 班 \\r\\n HTML CSS javascript 用いる アプリケーショ...
\n", "

170 rows × 6 columns

\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", " \n", " \n", " \n", "
\n", "\n", "
\n", "
\n" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "dataframe", "variable_name": "assesment_df", "summary": "{\n \"name\": \"assesment_df\",\n \"rows\": 170,\n \"fields\": [\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 16,\n \"samples\": [\n \"\\u5de5\\u696d\\u6570\\u5b66\\u2160\",\n \"\\u6280\\u8853\\u8005\\u306e\\u502b\\u7406\",\n \"\\u30a2\\u30eb\\u30b4\\u30ea\\u30ba\\u30e0\\u3068\\u30c7\\u30fc\\u30bf\\u69cb\\u9020\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"grade\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0,\n \"min\": 1,\n \"max\": 3,\n \"num_unique_values\": 3,\n \"samples\": [\n 1,\n 2,\n 3\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"required\",\n \"properties\": {\n \"dtype\": \"boolean\",\n \"num_unique_values\": 2,\n \"samples\": [\n false,\n true\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"q_id\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Q21 (2)\",\n \"Q22\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"comment\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 153,\n \"samples\": [\n \"\\u30fb\\u6559\\u79d1\\u66f8\\u304c\\u5fc5\\u8981\\u306a\\u306e\\u304b\\u5fc5\\u8981\\u3067\\u306a\\u3044\\u306e\\u304b\\u304c\\u66d6\\u6627\\u306a\\u307e\\u307e\\u6388\\u696d\\u304c\\u59cb\\u307e\\u308a\\u3001\\u975e\\u5e38\\u306b\\u4e0d\\u5b89\\u3060\\u3063\\u305f\\u305f\\u3081\\u3001\\u6559\\u79d1\\u66f8\\u304c\\u5fc5\\u9808\\u304b\\u305d\\u3046\\u3067\\u306a\\u3044\\u306e\\u304b\\u306f\\u6700\\u521d\\u306b\\u306f\\u3063\\u304d\\u308a\\u3057\\u3066\\u6b32\\u3057\\u3044\\u3002\\r\\n\\u30fb\\u8ab2\\u984c\\u3092\\u51fa\\u3059\\u3060\\u3051\\u51fa\\u3055\\u305b\\u3066\\u304a\\u3044\\u3066\\u3001\\u63a1\\u70b9\\u3082\\u305b\\u305a\\u3001\\u3069\\u3046\\u3044\\u3063\\u305f\\u89e3\\u7b54\\u304c\\u6b63\\u3057\\u3044\\u306e\\u304b\\u3068\\u3044\\u3063\\u305f\\u6307\\u91dd\\u3082\\u51fa\\u3059\\u306e\\u304c\\u3068\\u3066\\u3082\\u9045\\u3044\\u3002\\u8ab2\\u984c\\u306f\\u89e3\\u304f\\u3060\\u3051\\u3067\\u306f\\u77e5\\u8b58\\u306e\\u5b9a\\u7740\\u306b\\u3064\\u306a\\u304c\\u3089\\u306a\\u3044\\u3068\\u601d\\u3044\\u307e\\u3059\\u304c\\u3001\\u305d\\u3053\\u3089\\u3078\\u3093\\u306f\\u3069\\u3046\\u306a\\u3093\\u3067\\u3057\\u3087\\u3046\\u304b\\u3002\\r\\n\\u30fb\\u914d\\u5e03\\u8cc7\\u6599\\u3068\\u3057\\u3066\\u3001\\u904e\\u53bb\\u554f\\u3082\\u914d\\u5e03\\u3057\\u3066\\u304f\\u308c\\u308b\\u3068\\u3068\\u3066\\u3082\\u52a9\\u304b\\u308b\\u306a\\u3001\\u3068\\u601d\\u3044\\u307e\\u3059\\u3002\\u3054\\u691c\\u8a0e\\u304a\\u9858\\u3044\\u3057\\u307e\\u3059\\u3002\",\n \"\\u30fb\\u4e2d\\u9593\\u30c6\\u30b9\\u30c8\\u3092\\u5ef6\\u671f\\u3057\\u7d9a\\u3051\\u3001\\u6700\\u7d42\\u7684\\u306b\\u4e2d\\u9593\\u30fb\\u671f\\u672b\\u8a66\\u9a13\\u3092\\uff12\\u9031\\u7d9a\\u3051\\u3066\\u3084\\u308b\\u3053\\u3068\\u3068\\u306a\\u308a\\u3001\\u8a08\\u753b\\u6027\\u304c\\u6b20\\u3051\\u3066\\u3044\\u308b\\u3002\\r\\n\\u30fb\\u914d\\u5e03\\u8cc7\\u6599\\u306e\\u8aa4\\u5b57\\u8131\\u5b57\\u304c\\u591a\\u3059\\u304e\\u308b\\u3002\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"wakati\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 153,\n \"samples\": [\n \"\\u6559\\u79d1\\u66f8 \\u5fc5\\u8981 \\u5fc5\\u8981 \\u66d6\\u6627 \\u307e\\u307e \\u6388\\u696d \\u59cb\\u307e\\u308b \\u975e\\u5e38 \\u4e0d\\u5b89 \\u305f\\u3081 \\u6559\\u79d1\\u66f8 \\u5fc5\\u9808 \\u305d\\u3046 \\u6700\\u521d \\u306f\\u3063\\u304d\\u308a \\r\\n \\u8ab2\\u984c \\u51fa\\u3059 \\u51fa\\u3059 \\u304a\\u304f \\u63a1\\u70b9 \\u3059\\u308b \\u3069\\u3046 \\u3044\\u3046 \\u89e3\\u7b54 \\u6b63\\u3057\\u3044 \\u3044\\u3046 \\u6307\\u91dd \\u51fa\\u3059 \\u3068\\u3066\\u3082 \\u9045\\u3044 \\u8ab2\\u984c \\u89e3\\u304f \\u77e5\\u8b58 \\u5b9a\\u7740 \\u3064\\u306a\\u304c\\u308b \\u601d\\u3046 \\u3089 \\u3078\\u3093 \\u3069\\u3046 \\r\\n \\u914d\\u5e03\\u8cc7\\u6599 \\u904e\\u53bb\\u554f \\u914d\\u5e03 \\u304f\\u308c\\u308b \\u3068\\u3066\\u3082 \\u52a9\\u304b\\u308b \\u601d\\u3046 \\u3054 \\u691c\\u8a0e \\u304a \\u9858\\u3046\",\n \"\\u4e2d\\u9593 \\u30c6\\u30b9\\u30c8 \\u5ef6\\u671f \\u7d9a\\u3051\\u308b \\u6700\\u7d42\\u7684 \\u4e2d\\u9593 \\u671f\\u672b \\u8a66\\u9a13 \\u9031 \\u7d9a\\u3051\\u308b \\u3084\\u308b \\u3053\\u3068 \\u306a\\u308b \\u8a08\\u753b\\u6027 \\u6b20\\u3051\\u308b \\u3044\\u308b \\r\\n \\u914d\\u5e03\\u8cc7\\u6599 \\u8aa4\\u5b57 \\u8131\\u5b57 \\u591a\\u3044 \\u3059\\u304e\\u308b\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}" } }, "metadata": {}, "execution_count": 5 } ], "source": [ "# 分かち書き\n", "poses = ['PROPN', 'NOUN', 'VERB', 'ADJ', 'ADV'] #名詞、動詞、形容詞、形容動詞\n", "\n", "assesment_df['wakati'] = ''\n", "for index, comment in enumerate(assesment_df['comment']):\n", " doc = nlp(comment)\n", " wakati_words = []\n", " for token in doc:\n", " if token.pos_ in poses:\n", " wakati_words.append(token.lemma_)\n", " wakati_text = ' '.join(wakati_words)\n", " assesment_df.at[index, 'wakati'] = wakati_text\n", "\n", "assesment_df" ] }, { "cell_type": "markdown", "metadata": { "id": "g85bcizRYpnB" }, "source": [ "## 文書ベクトルの作成\n", "ここでは CountVectorizer (Bag-of-Words) で作成してみよう。" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "LJNIMbe3YpnB", "outputId": "d94ad7dd-4269-414b-eae9-c7b958de7919" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "bow_tf_vector.shape = (170, 741)\n" ] } ], "source": [ "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n", "\n", "stop_words = ['こと', '\\r\\n', 'ため', '思う', 'いる', 'ある', 'する', 'なる']\n", "vectorizer = CountVectorizer(stop_words=stop_words)\n", "bow_tf_vector = vectorizer.fit_transform(assesment_df['wakati'])\n", "print('bow_tf_vector.shape = ', bow_tf_vector.shape)" ] }, { "cell_type": "markdown", "metadata": { "id": "Hv06M0vyYpnC" }, "source": [ "## LDAによるトピックモデル解析\n", "sklearnでは [LatentDirichletAllocation](https://scikit-learn.org/stable/modules/decomposition.html?highlight=lda#latent-dirichlet-allocation-lda) として用意されている。" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "id": "cQu3CNFLYpnC" }, "outputs": [], "source": [ "from sklearn.decomposition import LatentDirichletAllocation\n", "\n", "NUM_TOPICS = 5 #トピック数\n", "max_iter = 100 #LDAによる学習回数\n", "lda = LatentDirichletAllocation(n_components=NUM_TOPICS,\n", " max_iter=max_iter,\n", " learning_method='online',\n", " random_state=123) # シード値を指定すると結果を再現できる\n", "data_lda = lda.fit_transform(bow_tf_vector)" ] }, { "cell_type": "markdown", "source": [ "## トピックの観察\n" ], "metadata": { "id": "OIK13M90vjeq" } }, { "cell_type": "code", "source": [ "import plotly.graph_objects as go\n", "from plotly.subplots import make_subplots\n", "\n", "def plot_top_words(model, feature_names, n_top_words, title):\n", " \"\"\"\n", " LDA のトピックごと上位語を水平方向のバーで表示する Plotly 版\n", "\n", " Parameters\n", " ----------\n", " model : sklearn.decomposition.LatentDirichletAllocation\n", " すでに fit_transform 済みの LDA モデル\n", " feature_names : array‑like, shape (n_features,)\n", " model.get_feature_names_out() で得た語彙\n", " n_top_words : int\n", " 各トピックで表示したい単語数\n", " title : str\n", " 図全体のタイトル\n", " \"\"\"\n", " n_topics = model.components_.shape[0]\n", " n_cols = 5 # 列数は固定\n", " n_rows = int(np.ceil(n_topics / n_cols)) # トピック数に応じて行数を決定\n", "\n", " # サブプロット用の Figure を用意\n", " fig = make_subplots(\n", " rows=n_rows,\n", " cols=n_cols,\n", " shared_xaxes=False,\n", " horizontal_spacing=0.08,\n", " vertical_spacing=0.06,\n", " subplot_titles=[f\"Topic {i + 1}\" for i in range(n_topics)],\n", " )\n", "\n", " for topic_idx, topic in enumerate(model.components_):\n", " # 指定トピックの上位語と重み\n", " top_idx = topic.argsort()[-n_top_words:]\n", " top_features = [feature_names[i] for i in top_idx]\n", " weights = topic[top_idx]\n", "\n", " row = topic_idx // n_cols + 1\n", " col = topic_idx % n_cols + 1\n", "\n", " # 水平バーを追加\n", " fig.add_trace(\n", " go.Bar(\n", " x=weights,\n", " y=top_features,\n", " orientation=\"h\",\n", " marker=dict(line=dict(width=0)), # 枠線を消してすっきり\n", " ),\n", " row=row,\n", " col=col,\n", " )\n", "\n", " # y 軸を上から下に並べ替え（matplotlib の barh と同じ見た目）\n", " fig.update_yaxes(autorange=\"reversed\", row=row, col=col)\n", "\n", " # 図全体のレイアウト調整\n", " fig.update_layout(\n", " height=450 * n_rows,\n", " width=1700,\n", " title=dict(text=title, x=0.5, xanchor=\"center\", font=dict(size=40)),\n", " showlegend=False,\n", " margin=dict(t=120, l=20, r=20, b=20),\n", " )\n", "\n", " # サブプロットタイトル（各トピック）のフォントサイズを揃える\n", " fig.update_annotations(font_size=22)\n", "\n", " fig.show()\n", " #file_title = title.replace(' ', '_')\n", " #fig.write_image(f'{file_title}.png')" ], "metadata": { "id": "CK3DlukWxzI6" }, "execution_count": 8, "outputs": [] }, { "cell_type": "code", "source": [ "n_top_words = 10\n", "plot_top_words(lda, vectorizer.get_feature_names_out(), n_top_words, \"Topics in LDA model (TF)\")" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 487 }, "id": "aeyIxVwVx4Ey", "outputId": "3de3cdde-0481-4711-a7af-3cc9ec6f39cf" }, "execution_count": 9, "outputs": [ { "output_type": "display_data", "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", "\n", "" ] }, "metadata": {} } ] }, { "cell_type": "markdown", "source": [ "## 文書ベクトル2(TF-IDF）" ], "metadata": { "id": "e9CXnek_zCcR" } }, { "cell_type": "code", "source": [ "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n", "\n", "stop_words = ['こと', '\\r\\n', 'ため', '思う', 'いる', 'ある', 'する', 'なる']\n", "vectorizer2 = TfidfVectorizer(stop_words=stop_words)\n", "tfidf_vector = vectorizer2.fit_transform(assesment_df['wakati'])\n", "print('tfidf_vector.shape = ', tfidf_vector.shape)" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "45CIeMgWvn95", "outputId": "c4e1f6a6-9e05-4e74-f31a-f116f07e12ca" }, "execution_count": 10, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "tfidf_vector.shape = (170, 741)\n" ] } ] }, { "cell_type": "code", "source": [ "lda2 = LatentDirichletAllocation(n_components=NUM_TOPICS,\n", " max_iter=max_iter,\n", " learning_method='online',\n", " random_state=123) # シード値を指定すると結果を再現できる\n", "\n", "data_lda2 = lda2.fit_transform(bow_tf_vector)" ], "metadata": { "id": "YWvQ3Ie8vn0O" }, "execution_count": 11, "outputs": [] }, { "cell_type": "code", "source": [ "plot_top_words(lda2, vectorizer2.get_feature_names_out(), n_top_words, \"Topics in LDA model (TF-IDF)\")" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 487 }, "id": "liJQunL0vnhR", "outputId": "7eb6da6d-a66b-4ca3-eb8c-3c759b83b9ae" }, "execution_count": 12, "outputs": [ { "output_type": "display_data", "data": { "text/html": [ "\n", "\n", "\n", "
\n", "
\n", "\n", "" ] }, "metadata": {} } ] }, { "cell_type": "code", "source": [], "metadata": { "id": "cE98RagMtt8f" }, "execution_count": 12, "outputs": [] } ], "metadata": { "interpreter": { "hash": "880b2a8c90f9e6beae80b56829e3f671fedd58b6d14887184ddce26124cedfbd" }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.9" }, "colab": { "provenance": [] } }, "nbformat": 4, "nbformat_minor": 0 }