| \n", " | title | \n", "grade | \n", "required | \n", "q_id | \n", "comment | \n", "
|---|---|---|---|---|---|
| 0 | \n", "工業数学Ⅰ | \n", "1 | \n", "True | \n", "Q21 (1) | \n", "特になし | \n", "
| 1 | \n", "工業数学Ⅰ | \n", "1 | \n", "True | \n", "Q21 (2) | \n", "正直わかりずらい。むだに間があるし。 | \n", "
| 2 | \n", "工業数学Ⅰ | \n", "1 | \n", "True | \n", "Q21 (2) | \n", "例題を取り入れて理解しやすくしてほしい。 | \n", "
| 3 | \n", "工業数学Ⅰ | \n", "1 | \n", "True | \n", "Q21 (2) | \n", "特になし | \n", "
| 4 | \n", "工業数学Ⅰ | \n", "1 | \n", "True | \n", "Q21 (2) | \n", "スライドに書く文字をもう少しわかりやすくして欲しいです。 | \n", "
![](\"https://spacy.io/images/pipeline.svg\")
\n",
"\n",
"基本的な使い方は、予め学習済み言語モデルを用意し、用意したモデルに対して文章を与えるだけで一通りの解析を実行してくれる。上図は公式ページのusageから引用したもの。中間部分の青い点線で囲まれている ``nlp`` 部分が解析部分に相当する。用意したnlpに文章を入力すると自動でtagger(品詞推定)、parser(依存関係推定)、ner(固有表現推定)等の処理を一括して行う。解析結果はDoc形式として保存されている。コードで示すと以下のようになる。\n",
"\n",
"```Python\n",
"import spacy\n",
"nlp = spacy.load('ja_ginza') # 事前学習済みモデルを用意\n",
"doc = nlp('今回は自然言語処理ライブラリであるspaCyについて紹介します。') # nlpに文章を入力すると、その戻り値に解析結果が保存される。\n",
"```\n",
"\n",
"![](\"https://spacy.io/images/architecture.svg\")
\n",
"\n",
"上図はspacyによる処理の流れを示している。ここでは以下の流れを把握しよう。\n",
"- 上部のText(文章)をnlp(事前学習済み言語モデル)に与えると、赤枠のTokenizer等が実行され、結果が紫枠の[Doc形式](https://spacy.io/api/doc)で保存される。\n",
"- Doc形式は[Token形式](https://spacy.io/api/token)もしくは[Span形式](https://spacy.io/api/span)のシーケンスである。シーケンスになっているため、これに対してループ処理を実行することで文頭から文末までのTokenまたはSpanを一つずつ参照することで詳細を確認することができる。\n",
"- 分かち書き結果を利用する場合にはTokenを利用する。\n",
"- Token単位では扱い難い名詞句や固有表現を利用する場合にはSpanを利用する。\n",
"\n",
"一例をコードで示すと以下のようになる。\n",
"\n",
"```Python\n",
"import spacy\n",
"nlp = spacy.load('ja_ginza') # 事前学習済みモデルを用意。\n",
"doc = nlp('今回は自然言語処理ライブラリであるspaCyについて紹介します。') # nlpに文章を入力すると、その戻り値に解析結果が保存される。\n",
"for token in doc: # Token単位で処理結果を参照。\n",
" print(token.i, token.lemma_, token.pos_)\n",
"```\n",
"\n",
"上記コードを実行すると以下のような結果が得られる。文章が単語に分割されていることを確認しよう。\n",
"```\n",
"0 今回 NOUN\n",
"1 は ADP\n",
"2 自然言語処理 ADV\n",
"3 ライブラリ ADJ\n",
"4 だ AUX\n",
"5 ある VERB\n",
"6 spacy NOUN\n",
"7 に ADP\n",
"8 つく VERB\n",
"9 て SCONJ\n",
"10 紹介 VERB\n",
"11 する AUX\n",
"12 ます AUX\n",
"13 。 PUNCT\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vR8z7efqNYOp"
},
"source": [
"## 形態素解析による分かち書き\n",
"ここでは良く使う機能として分かち書き、原形処理、品詞推定、ストップワードを眺めるとともに、類語をまとめる例について紹介する。それ以外の[Token利用法についてはドキュメントを参照](https://spacy.io/api/token#attributes)しよう。\n",
"\n",
"- case 0: 形態素解析の基本\n",
"- case 1: 分かち書き, 原形処理(lemmatize) + カウント\n",
"- case 2: 分かち書き, 原形処理 + 品詞別カウント\n",
"- case 3: 分かち書き, 原形処理 + ストップワード + 品詞別カウント\n",
"- case 4: 分かち書き, 原形処理 + ストップワード + 手動類義語処理 + 品詞別カウント"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "z_G47aXzNYOp"
},
"source": [
"### case 0: 形態素解析の基本\n",
"nlpの解析結果をfor文で順序よく参照する。"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "_kIKfhvJNYOp",
"outputId": "a67e1b85-6786-43ca-f9d8-7f11eacf38f6",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"機械トラブルの操作で授業が止まることがあった。\r\n",
"新しいことが学べて大学って感じがした。\n",
"0 機械 機械 NOUN compound 1\n",
"1 トラブル トラブル NOUN nmod 3\n",
"2 の の ADP case 1\n",
"3 操作 操作 NOUN obl 7\n",
"4 で で ADP case 3\n",
"5 授業 授業 NOUN nsubj 7\n",
"6 が が ADP case 5\n",
"7 止まる 止まる VERB acl 8\n",
"8 こと こと NOUN nsubj 10\n",
"9 が が ADP case 8\n",
"10 あっ ある VERB ROOT 10\n",
"11 た た AUX aux 10\n",
"12 。 。 PUNCT punct 10\n",
"13 \r\n",
" \r\n",
" NOUN compound 14\n",
"14 新しい 新しい ADJ acl 15\n",
"15 こと こと NOUN nsubj 17\n",
"16 が が ADP case 15\n",
"17 学べ 学べる VERB acl 19\n",
"18 て て SCONJ mark 17\n",
"19 大学 大学 NOUN advcl 23\n",
"20 って って AUX aux 19\n",
"21 感じ 感じ NOUN nsubj 23\n",
"22 が が ADP case 21\n",
"23 し する VERB ROOT 23\n",
"24 た た AUX aux 23\n",
"25 。 。 PUNCT punct 23\n"
]
}
],
"source": [
"# case 0: 形態素解析の基本\n",
"# token.i = 分かち書きされた単語のインデックス\n",
"# token.text = 分かち書きされた単語そのもの\n",
"# token.lemma_ = 単語を原形処理したもの\n",
"# token.pos_ = 品詞推定結果\n",
"# token.dep_ = 係り受け情報\n",
"# token.head.i = 係り受け元単語のインデックス\n",
"\n",
"sample_text = assesment_df['comment'][10]\n",
"print(sample_text)\n",
"\n",
"doc = nlp(sample_text)\n",
"for token in doc:\n",
" print(token.i, token.text, token.lemma_, token.pos_, token.dep_, token.head.i)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "fn5XxEIRNYOq",
"outputId": "da2052f7-db13-4396-da8d-e4ec44715c31",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"subordinating conjunction\n",
"nominal subject\n"
]
}
],
"source": [
"# 品詞、係り受け情報等の用語は spacy.explain で補足情報を確認できる。\n",
"# 詳細はドキュメントページ参照。\n",
"# Universal POS tags: https://universaldependencies.org/u/pos/index.html\n",
"# Part-of-speech tags and dependencies: https://spacy.io/usage/spacy-101#annotations-pos-deps\n",
"\n",
"print(spacy.explain(\"SCONJ\"))\n",
"print(spacy.explain(\"nsubj\"))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "v9HEru2xNYOq"
},
"source": [
"### case 1: 分かち書き, 原形処理(lemmatize) + カウント\n",
"表記上は異なる同一語に活用する語がある。これらを表記によらず同一語として扱うために、原形処理結果(token.lemma_)を処理対象としよう。"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"id": "3qrz8V-nNYOq",
"outputId": "7b2aeb79-7146-4982-861d-a06ace89b2ca",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"[('た', 337), ('の', 310), ('が', 260), ('て', 233), ('。', 231), ('だ', 211), ('、', 193), ('を', 166), ('に', 162), ('と', 147)]\n"
]
}
],
"source": [
"# case 1: 分かち書き, 原形処理(lemmatize) + カウント\n",
"\n",
"def count_lemma(df, column):\n",
" '''分かち書き1：原形処理のみ。\n",
" args:\n",
" df (pd.DataFrame): 読み込み対象データフレーム。\n",
" column (str): データフレーム内の読み込み対象列名。\n",
" return\n",
" words_list ([token.lemma_,,,]): 原形処理済み単語のリスト。\n",
" '''\n",
" words_list = []\n",
" for comment in df[column]:\n",
" doc = nlp(comment)\n",
" for token in doc:\n",
" words_list.append(token.lemma_)\n",
" return words_list\n",
"\n",
"words_list = count_lemma(assesment_df, 'comment')\n",
"words_count = collections.Counter(words_list)\n",
"print(words_count.most_common(10))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dTBXb0AcNYOq"
},
"source": [
"### case 2: 分かち書き, 原形処理 + 品詞別カウント\n",
"多くの場合、データの全体像を観察するために処理結果をカウントする。分かち書き結果の場合には多くの文章で共通して現れる助詞が圧倒的多数となってしまうため、直接カウントするだけだとあまり意味のない結果になってしまう。そこで品詞別にカウントしてみることにしよう。"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"id": "ZInkifZzNYOq",
"outputId": "32a3f7e8-aa48-4ee7-e5cb-974f16c5080c",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"pos = PROPN\n",
"[('webclass', 5), ('和田', 2), ('沖縄', 1), ('工学部', 1), ('當間', 1), ('アルゴリズム', 1), ('解決策', 1), ('クソゲー', 1), ('キャンパス', 1), ('github', 1)]\n",
"pos = NOUN\n",
"[('こと', 73), ('授業', 44), ('課題', 39), ('試験', 38), ('\\r\\n', 34), ('講義', 31), ('内容', 22), ('ため', 21), ('問題', 20), ('説明', 19)]\n",
"pos = VERB\n",
"[('思う', 68), ('いる', 56), ('ある', 44), ('する', 35), ('なる', 31), ('わかる', 27), ('いう', 21), ('学ぶ', 20), ('できる', 18), ('出す', 17)]\n",
"pos = ADJ\n",
"[('良い', 25), ('ない', 24), ('よい', 21), ('多い', 19), ('難しい', 13), ('楽しい', 11), ('非常', 10), ('いい', 9), ('新しい', 6), ('大変', 5)]\n",
"pos = ADV\n",
"[('とても', 28), ('特に', 22), ('少し', 14), ('どう', 8), ('今後', 6), ('すぐ', 6), ('もう', 4), ('まだ', 4), ('初めて', 4), ('あまり', 4)]\n"
]
}
],
"source": [
"# case 2: 分かち書き, 原形処理 + 品詞別カウント\n",
"# 対象品詞\n",
"# 名詞(NOUN), 動詞(VERB), 形容詞(ADJ), 副詞(ADV)\n",
"# spacyによる品詞一覧: https://universaldependencies.org/u/pos/\n",
"\n",
"def count_lemma2(df, column, target_poses):\n",
" '''分かち書き2：原形処理し、品詞別にカウント。\n",
" args:\n",
" df (pd.DataFrame): 読み込み対象データフレーム。\n",
" column (str): データフレーム内の読み込み対象列名。\n",
" target_poses ([str]): カウント対象となる品詞名のリスト。\n",
" return\n",
" words_dict ({pos1:{token1.lemma_:i, token2.lemma_:j},\n",
" pos2:{token3.lemma_:k, token4.lemma_:l}}): 品詞(pos)別に、単語をカウント。\n",
" '''\n",
" words_dict = {}\n",
" for pos in target_poses:\n",
" words_dict[pos] = {}\n",
"\n",
" for comment in df[column]:\n",
" doc = nlp(comment)\n",
" for token in doc:\n",
" if token.pos_ in target_poses:\n",
" if token.lemma_ not in words_dict[token.pos_]:\n",
" words_dict[token.pos_].update({token.lemma_: 1})\n",
" else:\n",
" words_dict[token.pos_][token.lemma_] += 1\n",
" return words_dict\n",
"\n",
"target_poses = ['PROPN', 'NOUN', 'VERB', 'ADJ', 'ADV']\n",
"words_dict = count_lemma2(assesment_df, 'comment', target_poses)\n",
"\n",
"for pos in target_poses:\n",
" print('pos = ', pos)\n",
" words_count[pos] = collections.Counter(words_dict[pos])\n",
" print(words_count[pos].most_common(10))\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iBxN3Qo5NYOq"
},
"source": [
"### case 3: 分かち書き, 原形処理 + ストップワード + 品詞別カウント\n",
"品詞別カウントのお陰である程度は概要を掴みやすくなったが、それでも「こと」「ため」「いる」のような頻出語が上位に位置してしまう。見たくない単語（ストップワード）を指定し、削除してしまおう。"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "-357LsRANYOr",
"outputId": "a6e0c3aa-6354-4713-8af7-d43d004b9485",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"pos = PROPN\n",
"[('webclass', 5), ('和田', 2), ('沖縄', 1), ('工学部', 1), ('當間', 1), ('アルゴリズム', 1), ('解決策', 1), ('クソゲー', 1), ('キャンパス', 1), ('github', 1)]\n",
"pos = NOUN\n",
"[('授業', 44), ('課題', 39), ('試験', 38), ('講義', 31), ('内容', 22), ('問題', 20), ('説明', 19), ('なし', 18), ('資料', 16), ('先生', 15)]\n",
"pos = VERB\n",
"[('わかる', 27), ('いう', 21), ('学ぶ', 20), ('できる', 18), ('出す', 17), ('書く', 16), ('ござる', 13), ('つく', 13), ('理解', 12), ('見る', 11)]\n",
"pos = ADJ\n",
"[('良い', 25), ('ない', 24), ('よい', 21), ('多い', 19), ('難しい', 13), ('楽しい', 11), ('非常', 10), ('いい', 9), ('新しい', 6), ('大変', 5)]\n",
"pos = ADV\n",
"[('とても', 28), ('特に', 22), ('少し', 14), ('どう', 8), ('今後', 6), ('すぐ', 6), ('もう', 4), ('まだ', 4), ('初めて', 4), ('あまり', 4)]\n"
]
}
],
"source": [
"# case 3: 分かち書き, 原形処理 + ストップワード + 品詞別カウント\n",
"def count_lemma3(df, column, target_poses, stop_words):\n",
" '''分かち書き3：原形処理し、ストップワードを除き、品詞別にカウント。\n",
" args:\n",
" df (pd.DataFrame): 読み込み対象データフレーム。\n",
" column (str): データフレーム内の読み込み対象列名。\n",
" target_poses ([str]): カウント対象となる品詞名のリスト。\n",
" stop_words ([str]): 削除したい単語のリスト。\n",
" return\n",
" words_dict ({pos1:{token1.lemma_:i, token2.lemma_:j},\n",
" pos2:{token3.lemma_:k, token4.lemma_:l}}): 品詞(pos)別に、単語をカウント。\n",
" '''\n",
" words_dict = {}\n",
" for pos in target_poses:\n",
" words_dict[pos] = {}\n",
"\n",
" for comment in df[column]:\n",
" doc = nlp(comment)\n",
" for token in doc:\n",
" if token.lemma_ not in stop_words:\n",
" if token.pos_ in target_poses:\n",
" if token.lemma_ not in words_dict[token.pos_]:\n",
" words_dict[token.pos_].update({token.lemma_: 1})\n",
" else:\n",
" words_dict[token.pos_][token.lemma_] += 1\n",
" return words_dict\n",
"\n",
"target_poses = ['PROPN', 'NOUN', 'VERB', 'ADJ', 'ADV']\n",
"stop_words = ['こと', '\\r\\n', 'ため', '思う', 'いる', 'ある', 'する', 'なる']\n",
"words_dict = count_lemma3(assesment_df, 'comment', target_poses, stop_words)\n",
"\n",
"for pos in target_poses:\n",
" print('pos = ', pos)\n",
" words_count[pos] = collections.Counter(words_dict[pos])\n",
" print(words_count[pos].most_common(10))\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kQbe0mc5NYOr"
},
"source": [
"### case 4: 分かち書き, 原形処理 + ストップワード + 手動類義語処理 + 品詞別カウント\n",
"名詞別カウントを眺めてみると「授業,44回」「講義,31回」とあるが、これらは同一単語としてカウントしたほうが良いだろう。しかしながら直接spacyでそのように処理することは出来ない。ここではまとめてしまいたい単語群（類義語）とその代表語を手動で用意し、類義語が出現したら代表語に置き換えてしまうことで対処してみよう。"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "hSRPn8l1NYOr",
"outputId": "d03e19bd-c761-42fa-ff83-a8d3dc3b3d87",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"pos = PROPN\n",
"[('webclass', 5), ('和田', 2), ('沖縄', 1), ('工学部', 1), ('當間', 1), ('アルゴリズム', 1), ('解決策', 1), ('クソゲー', 1), ('キャンパス', 1), ('github', 1)]\n",
"pos = NOUN\n",
"[('授業', 75), ('課題', 39), ('試験', 38), ('内容', 22), ('問題', 20), ('説明', 19), ('なし', 18), ('資料', 16), ('先生', 15), ('お', 14)]\n",
"pos = VERB\n",
"[('わかる', 27), ('いう', 21), ('学ぶ', 20), ('できる', 18), ('出す', 17), ('書く', 16), ('ござる', 13), ('つく', 13), ('理解', 12), ('見る', 11)]\n",
"pos = ADJ\n",
"[('良い', 55), ('ない', 24), ('多い', 19), ('難しい', 13), ('楽しい', 11), ('非常', 10), ('新しい', 6), ('大変', 5), ('高い', 5), ('必要', 4)]\n",
"pos = ADV\n",
"[('とても', 50), ('少し', 14), ('どう', 8), ('今後', 6), ('すぐ', 6), ('もう', 4), ('まだ', 4), ('初めて', 4), ('あまり', 4), ('しっかり', 4)]\n"
]
}
],
"source": [
"# case 4: 分かち書き, 原形処理 + ストップワード + 手動類義語処理 + 品詞別カウント\n",
"\n",
"def reverse_dict(dict_with_list):\n",
" result = {}\n",
" for k, v_list in dict_with_list.items():\n",
" for v in v_list:\n",
" result[v] = k\n",
" return result\n",
"\n",
"def count_lemma4(df, column, target_poses, stop_words, similar_words):\n",
" '''分かち書き4：原形処理し、ストップワードを除き、類義語を代表語に置き換え、品詞別にカウント。\n",
" args:\n",
" df (pd.DataFrame): 読み込み対象データフレーム。\n",
" column (str): データフレーム内の読み込み対象列名。\n",
" target_poses ([str]): カウント対象となる品詞名のリスト。\n",
" stop_words ([str]): 削除したい単語のリスト。\n",
" similar_words ({similar_word1:representive_word1, similar_word2:representive_word1,,}):\n",
" 類義語辞書。keyをvalueに置き換える。\n",
" return\n",
" words_dict ({pos1:{token1.lemma_:i, token2.lemma_:j},\n",
" pos2:{token3.lemma_:k, token4.lemma_:l}}): 品詞(pos)別に、単語をカウント。\n",
" '''\n",
" words_dict = {}\n",
" for pos in target_poses:\n",
" words_dict[pos] = {}\n",
"\n",
" for comment in df[column]:\n",
" doc = nlp(comment)\n",
" for token in doc:\n",
" if token.lemma_ not in stop_words:\n",
" if token.lemma_ in similar_words.keys():\n",
" word = similar_words[token.lemma_]\n",
" else:\n",
" word = token.lemma_\n",
"\n",
" if token.pos_ in target_poses:\n",
" if word not in words_dict[token.pos_]:\n",
" words_dict[token.pos_].update({word: 1})\n",
" else:\n",
" words_dict[token.pos_][word] += 1\n",
" return words_dict\n",
"\n",
"target_poses = ['PROPN', 'NOUN', 'VERB', 'ADJ', 'ADV']\n",
"stop_words = ['こと', '\\r\\n', 'ため', '思う', 'いる', 'ある', 'する', 'なる']\n",
"similar_words = {'授業':['講義'], '良い':['よい', 'いい'], 'とても':['特に']}\n",
"similar_words = reverse_dict(similar_words)\n",
"words_dict = count_lemma4(assesment_df, 'comment', target_poses, stop_words, similar_words)\n",
"\n",
"for pos in target_poses:\n",
" print('pos = ', pos)\n",
" words_count[pos] = collections.Counter(words_dict[pos])\n",
" print(words_count[pos].most_common(10))\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d3mFs-rNNYOr"
},
"source": [
"### 授業別単語頻出傾向の観察\n",
"全コメントを対象とした頻出語は確認できたが、授業毎の違いはないのだろうか。固有名詞(PROPN)と名詞(NOUN)に限定して、授業毎にカウント＆積み上げ棒グラフとして描画してみよう。全体の流れは以下の通りである。\n",
"\n",
"- 頻出語上位20件を用意。\n",
"- 授業毎に頻出語が出現した回数をカウント。数えた結果を「単語x授業行列(pd.DataFrame形式)」として整形する。\n",
"- pd.DataFrameをmatplotlibで描画する。"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XO3dTFYlNYOr"
},
"source": [
"#### 頻出語上位20件を用意。"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "8dUwxPkpNYOr",
"outputId": "59a81eaf-24aa-404a-9499-6cf7b5163a27",
"colab": {
"base_uri": "https://localhost:8080/"
}
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"10 425 434\n"
]
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"[('授業', 75),\n",
" ('課題', 39),\n",
" ('試験', 38),\n",
" ('内容', 22),\n",
" ('問題', 20),\n",
" ('説明', 19),\n",
" ('なし', 18),\n",
" ('資料', 16),\n",
" ('先生', 15),\n",
" ('お', 14),\n",
" ('難易度', 12),\n",
" ('解答', 11),\n",
" ('実際', 11),\n",
" ('方', 10),\n",
" ('テスト', 10),\n",
" ('知識', 9),\n",
" ('機会', 9),\n",
" ('生徒', 9),\n",
" ('中間', 9),\n",
" ('期末', 9)]"
]
},
"metadata": {},
"execution_count": 12
}
],
"source": [
"# 講義名のユニーク名一覧\n",
"titles = assesment_df['title'].unique()\n",
"\n",
"# 固有名詞＋名詞の上位20単語一覧\n",
"top_n = 20\n",
"total_words = {**words_dict['PROPN'], **words_dict['NOUN']}\n",
"print(len(words_dict['PROPN']), len(words_dict['NOUN']), len(total_words))\n",
"top_n_words = collections.Counter(total_words).most_common(top_n)\n",
"top_n_words"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jm7n1zRHNYOr"
},
"source": [
"#### 授業毎に頻出語が出現した回数をカウント。"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"id": "HsSAy_SFNYOs",
"outputId": "75deecf4-0f4a-4414-c462-119c2a10b453",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
}
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
" 工業数学Ⅰ 技術者の倫理 工学基礎演習 プログラミングⅠ プログラミング演習Ⅰ アルゴリズムとデータ構造 情報ネットワークⅠ \\\n",
"授業 3 3 2 0 1 2 1 \n",
"課題 2 0 0 6 3 0 0 \n",
"試験 0 0 0 0 0 3 0 \n",
"内容 0 2 0 1 0 2 0 \n",
"問題 1 0 0 0 0 4 0 \n",
"説明 2 0 0 2 0 3 0 \n",
"なし 3 4 4 4 3 0 0 \n",
"資料 0 0 0 0 0 3 1 \n",
"先生 0 3 0 1 0 1 0 \n",
"お 1 2 0 2 1 0 3 \n",
"難易度 0 0 0 1 0 0 0 \n",
"解答 0 0 0 0 1 0 0 \n",
"実際 0 0 0 0 1 0 1 \n",
"方 0 0 1 0 1 0 2 \n",
"テスト 0 0 0 0 0 1 1 \n",
"知識 0 1 0 1 1 0 0 \n",
"機会 0 0 3 0 0 0 1 \n",
"生徒 0 0 0 0 0 1 0 \n",
"中間 0 0 0 0 0 0 2 \n",
"期末 0 0 0 0 0 0 0 \n",
"\n",
" コンピュータシステム データサイエンス基礎 ディジタル回路 情報処理技術概論 プロジェクトデザイン キャリアデザイン \\\n",
"授業 19 4 1 2 5 0 \n",
"課題 23 3 0 0 1 0 \n",
"試験 29 4 0 2 0 0 \n",
"内容 12 4 0 0 1 0 \n",
"問題 12 2 0 1 0 0 \n",
"説明 7 2 1 1 0 0 \n",
"なし 0 0 0 0 0 0 \n",
"資料 8 0 1 2 1 0 \n",
"先生 6 2 0 1 0 1 \n",
"お 4 1 0 0 0 0 \n",
"難易度 8 2 0 1 0 0 \n",
"解答 7 3 0 0 0 0 \n",
"実際 3 1 0 3 2 0 \n",
"方 0 2 0 2 2 0 \n",
"テスト 6 0 0 2 0 0 \n",
"知識 2 4 0 0 0 0 \n",
"機会 0 0 0 0 5 0 \n",
"生徒 8 0 0 0 0 0 \n",
"中間 7 0 0 0 0 0 \n",
"期末 6 0 0 3 0 0 \n",
"\n",
" データマイニング ICT実践英語Ⅰ 知能情報実験Ⅲ \n",
"授業 0 0 1 \n",
"課題 1 0 0 \n",
"試験 0 0 0 \n",
"内容 0 0 0 \n",
"問題 0 0 0 \n",
"説明 0 0 1 \n",
"なし 0 0 0 \n",
"資料 0 0 0 \n",
"先生 0 0 0 \n",
"お 0 0 0 \n",
"難易度 0 0 0 \n",
"解答 0 0 0 \n",
"実際 0 0 0 \n",
"方 0 0 0 \n",
"テスト 0 0 0 \n",
"知識 0 0 0 \n",
"機会 0 0 0 \n",
"生徒 0 0 0 \n",
"中間 0 0 0 \n",
"期末 0 0 0 "
],
"text/html": [
"\n",
" | \n", " | 工業数学Ⅰ | \n", "技術者の倫理 | \n", "工学基礎演習 | \n", "プログラミングⅠ | \n", "プログラミング演習Ⅰ | \n", "アルゴリズムとデータ構造 | \n", "情報ネットワークⅠ | \n", "コンピュータシステム | \n", "データサイエンス基礎 | \n", "ディジタル回路 | \n", "情報処理技術概論 | \n", "プロジェクトデザイン | \n", "キャリアデザイン | \n", "データマイニング | \n", "ICT実践英語Ⅰ | \n", "知能情報実験Ⅲ | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 授業 | \n", "3 | \n", "3 | \n", "2 | \n", "0 | \n", "1 | \n", "2 | \n", "1 | \n", "19 | \n", "4 | \n", "1 | \n", "2 | \n", "5 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "
| 課題 | \n", "2 | \n", "0 | \n", "0 | \n", "6 | \n", "3 | \n", "0 | \n", "0 | \n", "23 | \n", "3 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "
| 試験 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "3 | \n", "0 | \n", "29 | \n", "4 | \n", "0 | \n", "2 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 内容 | \n", "0 | \n", "2 | \n", "0 | \n", "1 | \n", "0 | \n", "2 | \n", "0 | \n", "12 | \n", "4 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 問題 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "4 | \n", "0 | \n", "12 | \n", "2 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 説明 | \n", "2 | \n", "0 | \n", "0 | \n", "2 | \n", "0 | \n", "3 | \n", "0 | \n", "7 | \n", "2 | \n", "1 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "
| なし | \n", "3 | \n", "4 | \n", "4 | \n", "4 | \n", "3 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 資料 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "3 | \n", "1 | \n", "8 | \n", "0 | \n", "1 | \n", "2 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 先生 | \n", "0 | \n", "3 | \n", "0 | \n", "1 | \n", "0 | \n", "1 | \n", "0 | \n", "6 | \n", "2 | \n", "0 | \n", "1 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "
| お | \n", "1 | \n", "2 | \n", "0 | \n", "2 | \n", "1 | \n", "0 | \n", "3 | \n", "4 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 難易度 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "8 | \n", "2 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 解答 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "7 | \n", "3 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 実際 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "1 | \n", "3 | \n", "1 | \n", "0 | \n", "3 | \n", "2 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 方 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "1 | \n", "0 | \n", "2 | \n", "0 | \n", "2 | \n", "0 | \n", "2 | \n", "2 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| テスト | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "1 | \n", "6 | \n", "0 | \n", "0 | \n", "2 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 知識 | \n", "0 | \n", "1 | \n", "0 | \n", "1 | \n", "1 | \n", "0 | \n", "0 | \n", "2 | \n", "4 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 機会 | \n", "0 | \n", "0 | \n", "3 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "5 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 生徒 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "8 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 中間 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "2 | \n", "7 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 期末 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "6 | \n", "0 | \n", "0 | \n", "3 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "