{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "LF1_UqO4kz4y",
"outputId": "f589e2c3-45e6-4f50-ca79-204459c4711e"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Tue Jun 23 09:31:12 AM UTC 2026\n"
]
}
],
"source": [
"!date"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "E55bNdPxksrk"
},
"source": [
"# 演習：ドメインの異なるコーパスで言語モデルを比較する\n",
"\n",
"## 概要\n",
"\n",
"**文学**（夏目漱石「坊っちゃん」）と**スポーツニュース**（livedoor ニュースコーパス）という\n",
"異なるドメインのコーパスから **トライグラム言語モデル（Laplace スムージング）** を構築し、\n",
"確率分布・補完候補・Perplexity がどう変わるかを実測します。\n",
"\n",
"| ドメイン | テキスト | 出典 | ライセンス |\n",
"|:---|:---|:---|:---|\n",
"| **文学** | 夏目漱石「坊っちゃん」（1906年） | [青空文庫](https://www.aozora.gr.jp/cards/000148/card752.html) | パブリックドメイン |\n",
"| **スポーツニュース** | livedoor ニュースコーパス Sports Watch | [RONDHUIT](https://www.rondhuit.com/download.html) | CC BY-ND 2.1 JP |\n",
"\n",
"**実行環境**：Google Colab 無料枠（CPU）で動作します。GPU は不要です。\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "crts-zO2ksrm",
"outputId": "7b9a478a-5302-4f4b-9208-bb3d33b93007"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.1/12.1 MB\u001b[0m \u001b[31m86.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m72.2/72.2 MB\u001b[0m \u001b[31m11.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m72.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25h\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
"You can now load the package via spacy.load('ja_core_news_sm')\n",
"\u001b[38;5;3m⚠ Restart to reload dependencies\u001b[0m\n",
"If you are in a Jupyter or Colab notebook, you may need to restart Python in\n",
"order to load all the package's dependencies. You can do this by selecting the\n",
"'Restart kernel' or 'Restart runtime' option.\n",
"インストール完了\n"
]
}
],
"source": [
"# ライブラリのインストール（初回のみ、数分かかります）\n",
"!pip install spacy plotly --quiet\n",
"!python -m spacy download ja_core_news_sm --quiet\n",
"print(\"インストール完了\")\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "v9jmllSNksrn",
"outputId": "f2d7995e-7fa5-45dc-a0d5-edf89ad04a48"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"インポート完了\n"
]
}
],
"source": [
"import re, math, zipfile, tarfile, urllib.request\n",
"from collections import Counter, defaultdict\n",
"\n",
"import spacy\n",
"import plotly.graph_objects as go\n",
"import plotly.express as px\n",
"from plotly.subplots import make_subplots\n",
"\n",
"print(\"インポート完了\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BNeEQ1G9ksrn"
},
"source": [
"## 1. コーパスのダウンロードと前処理\n",
"\n",
"2 つの異なるドメインのコーパスを公開データから直接取得します。\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "mVFS5BqVksrn",
"outputId": "d0119f16-1d49-4e71-9bfa-de57197ea4ee"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"坊っちゃん: 89,670 文字\n",
"冒頭サンプル: 【テキスト中に現れる記号について】\r\n",
"\r\n",
"：ルビ\r\n",
"（例）坊っちゃん\r\n",
"\r\n",
"：ルビの付く文字列の始まりを特定する記号\r\n",
"（例）夕方折戸の\r\n",
"\r\n",
"：入力者注 主に\n"
]
}
],
"source": [
"# 青空文庫「坊っちゃん」をダウンロード（ShiftJIS / zip）\n",
"AOZORA_URL = \"https://www.aozora.gr.jp/cards/000148/files/752_ruby_2438.zip\"\n",
"urllib.request.urlretrieve(AOZORA_URL, \"bocchan.zip\")\n",
"\n",
"with zipfile.ZipFile(\"bocchan.zip\") as zf:\n",
" fname = [n for n in zf.namelist() if n.endswith(\".txt\")][0]\n",
" raw = zf.read(fname).decode(\"shift_jis\", errors=\"replace\")\n",
"\n",
"def clean_aozora(text):\n",
" \"\"\"青空文庫のルビ・注記・ヘッダ・フッタを除去する\"\"\"\n",
" text = re.sub(r'^.*?-{5,}[^\\n]*\\n', '', text, flags=re.DOTALL)\n",
" text = re.split(r'\\n底本：', text)[0]\n",
" text = re.sub(r'《[^》]*》', '', text)\n",
" text = re.sub(r'[｜]', '', text)\n",
" text = re.sub(r'［＃[^］]*］', '', text)\n",
" text = re.sub(r'\\n{3,}', '\\n\\n', text)\n",
" return text.strip()\n",
"\n",
"bocchan_text = clean_aozora(raw)\n",
"print(f\"坊っちゃん: {len(bocchan_text):,} 文字\")\n",
"print(\"冒頭サンプル:\", bocchan_text[:80])\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "RybYPQuOksro",
"outputId": "76978fed-0143-492f-97f5-19a50cc00e96"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ダウンロード中（しばらくお待ちください）...\n",
"スポーツニュース: 901 記事, 620,431 文字\n",
"先頭サンプル: 原著作者のクレジットを表示し、ニュース記事の改変をしないことを条件に、記事全文を\n",
"自由に転載・引用が可能です。\n",
"\n",
"このディレクトリの記事ファイル内容の提供元：\n",
"\n"
]
}
],
"source": [
"# livedoor ニュースコーパス（Sports Watch カテゴリ）をダウンロード（約 20MB）\n",
"LIVEDOOR_URL = \"https://www.rondhuit.com/download/ldcc-20140209.tar.gz\"\n",
"print(\"ダウンロード中（しばらくお待ちください）...\")\n",
"urllib.request.urlretrieve(LIVEDOOR_URL, \"ldcc.tar.gz\")\n",
"\n",
"sports_texts = []\n",
"with tarfile.open(\"ldcc.tar.gz\", \"r:gz\") as tf:\n",
" for member in tf.getmembers():\n",
" if \"sports-watch\" in member.name and member.name.endswith(\".txt\"):\n",
" f = tf.extractfile(member)\n",
" if f is None:\n",
" continue\n",
" content = f.read().decode(\"utf-8\", errors=\"replace\")\n",
" lines = content.strip().split(\"\\n\")\n",
" # 先頭 2 行（URL・日時）を除いて本文のみ取得\n",
" body = \"\\n\".join(lines[2:]) if len(lines) > 2 else content\n",
" if len(body.strip()) > 30:\n",
" sports_texts.append(body.strip())\n",
"\n",
"sports_text = \"\\n\".join(sports_texts)\n",
"print(f\"スポーツニュース: {len(sports_texts)} 記事, {len(sports_text):,} 文字\")\n",
"print(\"先頭サンプル:\", sports_text[:80])\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "OWbvt9Jsksro"
},
"source": [
"## 2. spaCy でトークナイズ（形態素解析）\n",
"\n",
"日本語テキストを単語（形態素）に分割します。\n",
"spaCy の `ja_core_news_sm` モデルを使用します。\n",
"\n",
"- 空白・句読点は除外します\n",
"- 速度のため各コーパスは先頭 60,000 文字を使用します\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "tWrS-gj3ksro",
"outputId": "53ab61cb-8977-4f35-e44a-2a1255494c45"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"トークナイズ中（30 秒〜1 分程度かかります）...\n",
"\n",
"坊っちゃん : 34,909 トークン 語彙 4,096 語\n",
"スポーツ : 30,022 トークン 語彙 4,569 語\n",
"\n",
"先頭 20 トークン（坊っちゃん）: ['テキスト', '中', 'に', '現れる', '記号', 'に', 'つい', 'て', 'ルビ', '例', '坊っ', 'ちゃん', 'ルビ', 'の', '付く', '文字', '列', 'の', '始まり', 'を']\n",
"先頭 20 トークン（スポーツ） : ['原', '著作', '者', 'の', 'クレジット', 'を', '表示', 'し', 'ニュース', '記事', 'の', '改変', 'を', 'し', 'ない', 'こと', 'を', '条件', 'に', '記事']\n"
]
}
],
"source": [
"# spaCy モデルをロード\n",
"nlp = spacy.load(\"ja_core_news_sm\")\n",
"\n",
"# Sudachi（spaCy 日本語モデルの内部トークナイザ）の入力上限は約 49,149 bytes。\n",
"# 日本語 1 文字は UTF-8 で最大 3 bytes なので、1 チャンクあたり 12,000 文字に抑える。\n",
"CHUNK_SIZE = 12_000\n",
"EXCLUDE_CHARS = set(\"。、！？「」『』（）…―・\\n\\t \")\n",
"\n",
"def tokenize(text, max_chars=60_000):\n",
" \"\"\"テキストを形態素（表層形）のリストに変換する。\n",
" Sudachi の入力サイズ制限に対応するため CHUNK_SIZE 文字ずつ処理する。\"\"\"\n",
" text = text[:max_chars]\n",
" tokens = []\n",
" for start in range(0, len(text), CHUNK_SIZE):\n",
" chunk = text[start : start + CHUNK_SIZE]\n",
" doc = nlp(chunk)\n",
" tokens.extend(\n",
" t.text for t in doc\n",
" if not t.is_space\n",
" and not t.is_punct\n",
" and t.text.strip()\n",
" and t.text not in EXCLUDE_CHARS\n",
" )\n",
" return tokens\n",
"\n",
"print(\"トークナイズ中（30 秒〜1 分程度かかります）...\")\n",
"tokens_b = tokenize(bocchan_text)\n",
"tokens_s = tokenize(sports_text)\n",
"\n",
"print(f\"\\n坊っちゃん : {len(tokens_b):,} トークン 語彙 {len(set(tokens_b)):,} 語\")\n",
"print(f\"スポーツ : {len(tokens_s):,} トークン 語彙 {len(set(tokens_s)):,} 語\")\n",
"print(\"\\n先頭 20 トークン（坊っちゃん）:\", tokens_b[:20])\n",
"print(\"先頭 20 トークン（スポーツ） :\", tokens_s[:20])\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7xRouS0yksro"
},
"source": [
"## 3. トライグラム言語モデルの構築（Laplace スムージング）\n",
"\n",
"カウントがゼロの n-gram にも小さい確率を割り当て、ゼロ確率問題を回避します。"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "vlIXTGwYksro",
"outputId": "ceafb760-b8ad-497d-8e2f-1570ddc1196b"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"文学LM : バイグラム文脈 17,576 種 トライグラム 28,609 種\n",
"ニュースLM: バイグラム文脈 16,874 種 トライグラム 24,929 種\n",
"合算語彙サイズ |V| = 7,608\n"
]
}
],
"source": [
"def build_trigram_lm(tokens):\n",
" \"\"\"トークン列からトライグラム LM を構築する。\n",
"\n",
" Returns:\n",
" bi_counts : Counter -- C(w1, w2) バイグラムカウント（分母）\n",
" tri_counts : defaultdict -- C(w3 | w1, w2) トライグラムカウント（分子）\n",
" vocab : set -- 語彙\n",
" \"\"\"\n",
" bi_counts = Counter()\n",
" tri_counts = defaultdict(Counter)\n",
" vocab = set(tokens)\n",
"\n",
" for i in range(2, len(tokens)):\n",
" w1, w2, w3 = tokens[i-2], tokens[i-1], tokens[i]\n",
" bi_counts[(w1, w2)] += 1\n",
" tri_counts[(w1, w2)][w3] += 1\n",
"\n",
" return bi_counts, tri_counts, vocab\n",
"\n",
"\n",
"def trigram_prob(w1, w2, w3, bi_counts, tri_counts, vocab_size, k=1):\n",
" \"\"\"Laplace スムージング付き P(w3 | w1, w2) を返す\"\"\"\n",
" c_tri = tri_counts[(w1, w2)][w3]\n",
" c_bi = bi_counts[(w1, w2)]\n",
" return (c_tri + k) / (c_bi + k * vocab_size)\n",
"\n",
"\n",
"# 両コーパスのモデルを構築\n",
"bi_b, tri_b, vocab_b = build_trigram_lm(tokens_b)\n",
"bi_s, tri_s, vocab_s = build_trigram_lm(tokens_s)\n",
"VS = len(vocab_b | vocab_s) # クロスドメイン評価用の合算語彙サイズ\n",
"\n",
"print(f\"文学LM : バイグラム文脈 {len(bi_b):,} 種 トライグラム {sum(len(v) for v in tri_b.values()):,} 種\")\n",
"print(f\"ニュースLM: バイグラム文脈 {len(bi_s):,} 種 トライグラム {sum(len(v) for v in tri_s.values()):,} 種\")\n",
"print(f\"合算語彙サイズ |V| = {VS:,}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "smAb5pJTksrp"
},
"source": [
"## 4. 確率分布の比較\n",
"\n",
"### 4-1 頻出語（単語出現回数）\n",
"\n",
"2 つのドメインで頻出する語がどう違うかを可視化します。\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 471
},
"id": "fiyNmLvsksrp",
"outputId": "f46bc44a-a025-4310-8c2c-fa81972adc7f"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
"\n",
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"【坊っちゃん Top-10】: ['て', 'の', 'に', 'が', 'た', 'は', 'と', 'を', 'だ', 'で']\n",
"【スポーツ Top-10】: ['の', 'と', 'た', 'に', 'は', 'を', 'が', 'て', 'で', 'し']\n"
]
}
],
"source": [
"TOP_N = 20\n",
"cnt_b = Counter(tokens_b)\n",
"cnt_s = Counter(tokens_s)\n",
"top_b = cnt_b.most_common(TOP_N)\n",
"top_s = cnt_s.most_common(TOP_N)\n",
"\n",
"fig = make_subplots(\n",
" rows=1, cols=2,\n",
" subplot_titles=[\"坊っちゃん（文学）\", \"Sports Watch（ニュース）\"]\n",
")\n",
"fig.add_trace(\n",
" go.Bar(x=[w for w, _ in top_b], y=[c for _, c in top_b],\n",
" name=\"文学\", marker_color=\"#4682B4\"),\n",
" row=1, col=1\n",
")\n",
"fig.add_trace(\n",
" go.Bar(x=[w for w, _ in top_s], y=[c for _, c in top_s],\n",
" name=\"ニュース\", marker_color=\"#CD5C5C\"),\n",
" row=1, col=2\n",
")\n",
"fig.update_layout(title_text=\"頻出語 Top-20 比較\", showlegend=False, height=420)\n",
"fig.show()\n",
"\n",
"print(\"【坊っちゃん Top-10】:\", [w for w, _ in top_b[:10]])\n",
"print(\"【スポーツ Top-10】:\", [w for w, _ in top_s[:10]])\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "k2YOf4gTksrp"
},
"source": [
"### 4-2 頻出トライグラム\n",
"\n",
"各コーパスで最も多く出現したトライグラム（3 語の連続）を確認します。\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 537
},
"id": "JN8IaMHCksrp",
"outputId": "9660456a-1f94-491d-c36e-428b4fe68699"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
"\n",
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"def top_trigrams(tri_counts, n=15):\n",
" flat = Counter()\n",
" for (w1, w2), nexts in tri_counts.items():\n",
" for w3, cnt in nexts.items():\n",
" flat[(w1, w2, w3)] = cnt\n",
" return flat.most_common(n)\n",
"\n",
"tri_top_b = top_trigrams(tri_b, 15)\n",
"tri_top_s = top_trigrams(tri_s, 15)\n",
"\n",
"fig = go.Figure(data=[go.Table(\n",
" columnwidth=[50, 260, 50, 260],\n",
" header=dict(\n",
" values=[\"順位\", \"坊っちゃん 上位トライグラム\", \"順位\", \"スポーツ 上位トライグラム\"],\n",
" fill_color=\"#CCCCCC\",\n",
" align=\"center\",\n",
" font=dict(size=13)\n",
" ),\n",
" cells=dict(\n",
" values=[\n",
" list(range(1, 16)),\n",
" [f\"「{w1}」→「{w2}」→「{w3}」 ({cnt}回)\" for (w1, w2, w3), cnt in tri_top_b],\n",
" list(range(1, 16)),\n",
" [f\"「{w1}」→「{w2}」→「{w3}」 ({cnt}回)\" for (w1, w2, w3), cnt in tri_top_s],\n",
" ],\n",
" align=[\"center\", \"left\", \"center\", \"left\"],\n",
" font=dict(size=12),\n",
" height=28,\n",
" )\n",
")])\n",
"fig.update_layout(title=\"頻出トライグラム Top-15 比較\", height=520)\n",
"fig.show()\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RJGOaKEuksrq"
},
"source": [
"### 4-3 同じ 2 語文脈での次語分布\n",
"\n",
"**同じ 2 語を文脈として与えたとき**、2 つのモデルが次の語をどう予測するかを比較します。\n",
"両コーパスに共通して出現する文脈を自動選択します。\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
},
"id": "_iI8GZ2gksrq",
"outputId": "ada82ac2-7022-4671-cc47-62a34ad8ee80"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"比較に使う 2 語文脈（頻度順）: [('し', 'て'), ('し', 'た'), ('て', 'いる'), ('に', 'は')]\n"
]
},
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
"\n",
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
"\n",
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
"\n",
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
"\n",
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"def next_word_dist(w1, w2, tri_counts, bi_counts, vocab_size, top_k=8, k=1):\n",
" \"\"\"(w1, w2) を文脈とした次語の Top-K と Laplace 確率を返す\"\"\"\n",
" cands = {w3: (cnt + k) / (bi_counts[(w1, w2)] + k * vocab_size)\n",
" for w3, cnt in tri_counts[(w1, w2)].items()}\n",
" if not cands:\n",
" return []\n",
" return sorted(cands.items(), key=lambda x: x[1], reverse=True)[:top_k]\n",
"\n",
"\n",
"# 両コーパスに共通して頻出する 2 語文脈を自動選択\n",
"shared_bi = set(bi_b.keys()) & set(bi_s.keys())\n",
"good_contexts = sorted(\n",
" [ctx for ctx in shared_bi\n",
" if bi_b[ctx] >= 5 and bi_s[ctx] >= 5 # 両方で 5 回以上\n",
" and len(tri_b[ctx]) >= 3 and len(tri_s[ctx]) >= 3], # 次語の種類が 3 以上\n",
" key=lambda x: bi_b[x] + bi_s[x],\n",
" reverse=True\n",
")[:4]\n",
"\n",
"print(\"比較に使う 2 語文脈（頻度順）:\", good_contexts)\n",
"\n",
"for (w1, w2) in good_contexts:\n",
" dist_b = next_word_dist(w1, w2, tri_b, bi_b, VS)\n",
" dist_s = next_word_dist(w1, w2, tri_s, bi_s, VS)\n",
"\n",
" all_words = list(dict.fromkeys([w for w, _ in dist_b[:6]] + [w for w, _ in dist_s[:6]]))\n",
" probs_b = {w: p for w, p in dist_b}\n",
" probs_s = {w: p for w, p in dist_s}\n",
"\n",
" fig = go.Figure()\n",
" fig.add_trace(go.Bar(\n",
" name=\"文学LM\", x=all_words,\n",
" y=[probs_b.get(w, 0) for w in all_words],\n",
" marker_color=\"#4682B4\"\n",
" ))\n",
" fig.add_trace(go.Bar(\n",
" name=\"ニュースLM\", x=all_words,\n",
" y=[probs_s.get(w, 0) for w in all_words],\n",
" marker_color=\"#CD5C5C\"\n",
" ))\n",
" fig.update_layout(\n",
" title=f\"文脈「{w1}」→「{w2}」 次語の確率分布\",\n",
" xaxis_title=\"次語\", yaxis_title=\"P(次語 | 文脈)\",\n",
" barmode=\"group\", height=380\n",
" )\n",
" fig.show()\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "I7504Z0mksrq"
},
"source": [
"## 5. テキスト補完の比較（トライグラムベース）\n",
"\n",
"2 語の文脈から次の語を予測します（スマホ補完の原理）。\n",
"\n",
"- **ドメイン固有の文脈**：片方のコーパスにしか出現しない文脈\n",
"- **共通の文脈**：両コーパスに出現するが、後続語の分布が異なる文脈\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Ls7moOVVksrq",
"outputId": "85eaf58f-173a-4ec3-9a7e-80f7255fa9bf"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"=== ① 文学固有の文脈 ===\n",
" 文脈「おれ」「は」\n",
" 文学LM : ['こう', '何', 'この', '無論', '一']\n",
" ニュースLM: ['（この文脈のデータなし）']\n",
" 文脈「赤」「シャツ」\n",
" 文学LM : ['は', 'が', 'の', 'と', 'さん']\n",
" ニュースLM: ['（この文脈のデータなし）']\n",
" 文脈「と」「云っ」\n",
" 文学LM : ['た', 'て', 'たら', 'てる', 'たって']\n",
" ニュースLM: ['（この文脈のデータなし）']\n",
"\n",
"=== ② スポーツ固有の文脈 ===\n",
" 文脈「Sports」「Watch」\n",
" 文学LM : ['（この文脈のデータなし）']\n",
" ニュースLM: ['日本', '競馬', '浅田', 'TBS', '岡田']\n",
" 文脈「日本」「代表」\n",
" 文学LM : ['（この文脈のデータなし）']\n",
" ニュースLM: ['は', 'の', 'に', 'メンバー', 'が']\n",
" 文脈「です」「けど」\n",
" 文学LM : ['（この文脈のデータなし）']\n",
" ニュースLM: ['と', 'も', '90', 'ああ', '浅田']\n",
"\n",
"=== ③ 共通文脈（違いに注目） ===\n",
" 文脈「し」「て」\n",
" 文学LM : ['いる', 'い', 'も', 'くれ', 'くれる']\n",
" ニュースLM: ['いる', 'い', 'も', '知ら', 'いく']\n",
" 文脈「し」「た」\n",
" 文学LM : ['事', 'と', 'が', 'する', 'から']\n",
" ニュースLM: ['また', 'が', 'Sports', 'こと', 'と']\n",
" 文脈「て」「いる」\n",
" 文学LM : ['おれ', 'と', 'の', 'ん', 'から']\n",
" ニュースLM: ['の', 'と', 'ん', 'よう', 'から']\n"
]
}
],
"source": [
"def complete(w1, w2, tri_counts, bi_counts, vocab_size, top_k=5, k=1):\n",
" \"\"\"2 語文脈から次の語 Top-K と Laplace 確率を返す\"\"\"\n",
" cands = {w3: (cnt + k) / (bi_counts[(w1, w2)] + k * vocab_size)\n",
" for w3, cnt in tri_counts[(w1, w2)].items()}\n",
" if not cands:\n",
" return [(\"（この文脈のデータなし）\", 0.0)]\n",
" return sorted(cands.items(), key=lambda x: x[1], reverse=True)[:top_k]\n",
"\n",
"\n",
"# ① 文学コーパス固有の文脈\n",
"lit_only = sorted(\n",
" [ctx for ctx in bi_b\n",
" if bi_b[ctx] >= 3 and bi_s.get(ctx, 0) == 0 and len(tri_b[ctx]) >= 2],\n",
" key=lambda x: bi_b[x], reverse=True\n",
")[:3]\n",
"\n",
"# ② スポーツコーパス固有の文脈\n",
"sport_only = sorted(\n",
" [ctx for ctx in bi_s\n",
" if bi_s[ctx] >= 3 and bi_b.get(ctx, 0) == 0 and len(tri_s[ctx]) >= 2],\n",
" key=lambda x: bi_s[x], reverse=True\n",
")[:3]\n",
"\n",
"# ③ 共通文脈（次語の分布が異なるはず）\n",
"shared_ex = sorted(\n",
" [ctx for ctx in shared_bi\n",
" if bi_b[ctx] >= 3 and bi_s[ctx] >= 3\n",
" and len(tri_b[ctx]) >= 2 and len(tri_s[ctx]) >= 2],\n",
" key=lambda x: bi_b[x] + bi_s[x], reverse=True\n",
")[:3]\n",
"\n",
"for section, contexts, model_pair in [\n",
" (\"① 文学固有の文脈\", lit_only, (tri_b, bi_b, tri_s, bi_s)),\n",
" (\"② スポーツ固有の文脈\", sport_only, (tri_b, bi_b, tri_s, bi_s)),\n",
" (\"③ 共通文脈（違いに注目）\", shared_ex, (tri_b, bi_b, tri_s, bi_s)),\n",
"]:\n",
" print(f\"\\n=== {section} ===\")\n",
" tb, bb, ts, bs = model_pair\n",
" for (w1, w2) in contexts:\n",
" r_lit = complete(w1, w2, tb, bb, VS)\n",
" r_news = complete(w1, w2, ts, bs, VS)\n",
" print(f\" 文脈「{w1}」「{w2}」\")\n",
" print(f\" 文学LM : {[w for w, _ in r_lit]}\")\n",
" print(f\" ニュースLM: {[w for w, _ in r_news]}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pycPAUXPksrq"
},
"source": [
"## 6. クロスドメイン Perplexity 評価\n",
"\n",
"**仮説**：自分が学習したドメインのテキストに対して PPL は低く、他ドメインに対して高くなる。\n",
"\n",
"| 組み合わせ | 期待される PPL |\n",
"|:---|:---:|\n",
"| 文学LM × 文学テスト | 低い |\n",
"| 文学LM × ニューステスト | 高い |\n",
"| ニュースLM × 文学テスト | 高い |\n",
"| ニュースLM × ニューステスト | 低い |\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"id": "H1Wd6w-0ksrq"
},
"outputs": [],
"source": [
"def sentence_logprob(tokens, bi_counts, tri_counts, vocab_size, k=1):\n",
" \"\"\"トークン列の log P（トライグラム・Laplace スムージング）\n",
" 位置 2 以降を評価対象とする。短すぎる場合は (None, 0) を返す。\"\"\"\n",
" if len(tokens) < 3:\n",
" return None, 0\n",
" lp, n = 0.0, 0\n",
" for i in range(2, len(tokens)):\n",
" p = trigram_prob(tokens[i-2], tokens[i-1], tokens[i],\n",
" bi_counts, tri_counts, vocab_size, k)\n",
" lp += math.log(p)\n",
" n += 1\n",
" return lp, n\n",
"\n",
"\n",
"def perplexity(token_sents, bi_counts, tri_counts, vocab_size, k=1):\n",
" \"\"\"文リスト全体のトライグラム Perplexity\"\"\"\n",
" total_lp, total_n = 0.0, 0\n",
" for sent in token_sents:\n",
" lp, n = sentence_logprob(sent, bi_counts, tri_counts, vocab_size, k)\n",
" if lp is not None and n > 0:\n",
" total_lp += lp\n",
" total_n += n\n",
" return math.exp(-total_lp / total_n) if total_n > 0 else float(\"inf\")\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ngQ5FivMksrr",
"outputId": "818992de-1b88-4cc0-a896-4fa2c7ff25eb"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"テスト文を準備中...\n",
"文学テスト:\n",
" → 199 文\n",
"ニューステスト:\n",
" → 199 文\n"
]
}
],
"source": [
"# テスト文を準備（句点で分割し先頭 200 文をトークナイズ）\n",
"def make_test_sents(text, max_sents=200, max_chars_per_sent=500):\n",
" \"\"\"テキストを文に分割してトークナイズしたリストを返す\"\"\"\n",
" sents = [s.strip() for s in re.split(r'[。\\n]', text) if len(s.strip()) > 5]\n",
" sents = sents[:max_sents]\n",
" result = []\n",
" for s in sents:\n",
" toks = tokenize(s, max_chars=max_chars_per_sent)\n",
" if len(toks) >= 3:\n",
" result.append(toks)\n",
" return result\n",
"\n",
"print(\"テスト文を準備中...\")\n",
"print(\"文学テスト:\")\n",
"test_b = make_test_sents(bocchan_text)\n",
"print(f\" → {len(test_b)} 文\")\n",
"\n",
"print(\"ニューステスト:\")\n",
"test_s = make_test_sents(sports_text)\n",
"print(f\" → {len(test_s)} 文\")\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "4R7Ankseksrr",
"outputId": "a6618abe-349d-45b1-df74-da5cb52539e2"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Perplexity 計算中...\n",
"\n",
"=== クロスドメイン Perplexity ===\n",
" モデル × テストデータ → PPL 評価\n",
"-------------------------------------------------------\n",
" 文学LM × 文学テスト → 2861.8 ◎ 同ドメイン\n",
" 文学LM × ニューステスト → 6825.2 △ 異ドメイン\n",
" ニュースLM × 文学テスト → 6612.8 △ 異ドメイン\n",
" ニュースLM × ニューステスト → 3064.0 ◎ 同ドメイン\n"
]
}
],
"source": [
"print(\"Perplexity 計算中...\")\n",
"ppl = {\n",
" (\"文学LM\", \"文学テスト\"): perplexity(test_b, bi_b, tri_b, VS),\n",
" (\"文学LM\", \"ニューステスト\"): perplexity(test_s, bi_b, tri_b, VS),\n",
" (\"ニュースLM\",\"文学テスト\"): perplexity(test_b, bi_s, tri_s, VS),\n",
" (\"ニュースLM\",\"ニューステスト\"): perplexity(test_s, bi_s, tri_s, VS),\n",
"}\n",
"\n",
"print(\"\\n=== クロスドメイン Perplexity ===\")\n",
"print(f\"{'モデル':^12} × {'テストデータ':^14} → {'PPL':>8} 評価\")\n",
"print(\"-\" * 55)\n",
"for (model, test), v in ppl.items():\n",
" same = \"◎ 同ドメイン\" if (model.startswith(\"文学\") and test.startswith(\"文学\")) \\\n",
" or (model.startswith(\"ニュース\") and test.startswith(\"ニュース\")) else \"△ 異ドメイン\"\n",
" print(f\"{model:^12} × {test:^14} → {v:>8.1f} {same}\")\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 488
},
"id": "Z8vY9MrEksrr",
"outputId": "74bb64a6-e863-43c6-9da0-6d238abeaeff"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
"\n",
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"読み方：対角線（同ドメイン）の PPL が低く、\n",
"非対角線（異ドメイン）の PPL が高いほどドメインの差が明確に表れています。\n"
]
}
],
"source": [
"ppl_bb = ppl[(\"文学LM\", \"文学テスト\")]\n",
"ppl_bs = ppl[(\"文学LM\", \"ニューステスト\")]\n",
"ppl_sb = ppl[(\"ニュースLM\",\"文学テスト\")]\n",
"ppl_ss = ppl[(\"ニュースLM\",\"ニューステスト\")]\n",
"\n",
"z = [[ppl_bb, ppl_bs], [ppl_sb, ppl_ss]]\n",
"x_labels = [\"文学テスト\", \"ニューステスト\"]\n",
"y_labels = [\"文学LM\", \"ニュースLM\"]\n",
"\n",
"fig = go.Figure(data=go.Heatmap(\n",
" z=z, x=x_labels, y=y_labels,\n",
" colorscale=\"YlOrRd\", reversescale=False,\n",
" text=[[f\"{ppl_bb:.0f}\", f\"{ppl_bs:.0f}\"],\n",
" [f\"{ppl_sb:.0f}\", f\"{ppl_ss:.0f}\"]],\n",
" texttemplate=\"%{text}\",\n",
" textfont=dict(size=20),\n",
" hovertemplate=\"モデル: %{y}
テスト: %{x}
PPL = %{z:.1f}\",\n",
"))\n",
"fig.update_layout(\n",
" title=dict(\n",
" text=\"クロスドメイン Perplexity ヒートマップ
\"\n",
" \"値が低いほど良い（低 PPL = より自然に次語を予測できる）\",\n",
" font=dict(size=14)\n",
" ),\n",
" xaxis_title=\"テストデータ\",\n",
" yaxis_title=\"モデル\",\n",
" height=420, width=520,\n",
")\n",
"fig.show()\n",
"\n",
"print(\"\\n読み方：対角線（同ドメイン）の PPL が低く、\")\n",
"print(\"非対角線（異ドメイン）の PPL が高いほどドメインの差が明確に表れています。\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "h6oA6ztNksrr"
},
"source": [
"## 7. 語彙重複分析とドメイン特徴語\n",
"\n",
"2 つのコーパスの語彙がどの程度重なるかを分析し、\n",
"各ドメイン固有の語彙（ドメイン特徴語）を抽出します。\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 519
},
"id": "whr4pqvqksrr",
"outputId": "a7e36913-d904-4787-d43e-c09e4a01df55"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"=== 語彙重複分析 ===\n",
"坊っちゃん 語彙: 4,096 語\n",
"スポーツ 語彙: 4,569 語\n",
"共通語彙 : 1,057 語 （坊の 25.8%、スポの 23.1%）\n",
"文学のみ : 3,039 語\n",
"スポーツのみ : 3,512 語\n"
]
},
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
"\n",
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"vocab_b_only = vocab_b - vocab_s\n",
"vocab_s_only = vocab_s - vocab_b\n",
"vocab_shared = vocab_b & vocab_s\n",
"\n",
"print(\"=== 語彙重複分析 ===\")\n",
"print(f\"坊っちゃん 語彙: {len(vocab_b):,} 語\")\n",
"print(f\"スポーツ 語彙: {len(vocab_s):,} 語\")\n",
"print(f\"共通語彙 : {len(vocab_shared):,} 語 \"\n",
" f\"（坊の {len(vocab_shared)/len(vocab_b)*100:.1f}%、\"\n",
" f\"スポの {len(vocab_shared)/len(vocab_s)*100:.1f}%）\")\n",
"print(f\"文学のみ : {len(vocab_b_only):,} 語\")\n",
"print(f\"スポーツのみ : {len(vocab_s_only):,} 語\")\n",
"\n",
"fig = make_subplots(\n",
" rows=1, cols=2, specs=[[{\"type\": \"pie\"}, {\"type\": \"pie\"}]],\n",
" subplot_titles=[\"坊っちゃん語彙の構成\", \"スポーツ語彙の構成\"]\n",
")\n",
"fig.add_trace(go.Pie(\n",
" labels=[\"共通語彙\", \"文学のみ\"],\n",
" values=[len(vocab_shared), len(vocab_b_only)],\n",
" marker_colors=[\"#7EB5D6\", \"#4682B4\"], name=\"坊っちゃん\"\n",
"), row=1, col=1)\n",
"fig.add_trace(go.Pie(\n",
" labels=[\"共通語彙\", \"スポーツのみ\"],\n",
" values=[len(vocab_shared), len(vocab_s_only)],\n",
" marker_colors=[\"#F4A582\", \"#CD5C5C\"], name=\"スポーツ\"\n",
"), row=1, col=2)\n",
"fig.update_layout(title=\"語彙の重複率\", height=400)\n",
"fig.show()\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 491
},
"id": "C2OU6HZfksrr",
"outputId": "9a207322-6bd0-48df-d586-c7c638935201"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"坊っちゃん固有語（文学ドメイン） : ['おれ', '云う', '云っ', '赤', 'シャツ', '山嵐', '清', '野', '生徒', '校長', '男', 'うらなり', '居', '教師', '知れ']\n",
"スポーツ固有語（ニュースドメイン）: ['Sports', '選手', 'Watch', '1', '代表', '放送', '番組', 'けど', '位', '3', '試合', '監督', '戦', 'W杯', '2']\n"
]
},
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
"\n",
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"def domain_keywords(cnt_target, cnt_other, top_n=15):\n",
" \"\"\"対象コーパスに多く、他コーパスに少ない語を TF 比で抽出する\"\"\"\n",
" total_t = sum(cnt_target.values())\n",
" total_o = sum(cnt_other.values()) + 1\n",
" scores = {\n",
" w: (cnt / total_t) / (cnt_other.get(w, 0) / total_o + 1e-9)\n",
" for w, cnt in cnt_target.items()\n",
" }\n",
" return sorted(scores.items(), key=lambda x: x[1], reverse=True)[:top_n]\n",
"\n",
"kw_b = domain_keywords(cnt_b, cnt_s)\n",
"kw_s = domain_keywords(cnt_s, cnt_b)\n",
"\n",
"print(\"坊っちゃん固有語（文学ドメイン） :\", [w for w, _ in kw_b])\n",
"print(\"スポーツ固有語（ニュースドメイン）:\", [w for w, _ in kw_s])\n",
"\n",
"fig = make_subplots(\n",
" rows=1, cols=2,\n",
" subplot_titles=[\"坊っちゃん固有語（TF 比スコア）\", \"スポーツ固有語（TF 比スコア）\"]\n",
")\n",
"fig.add_trace(go.Bar(\n",
" x=[w for w, _ in kw_b], y=[s for _, s in kw_b],\n",
" marker_color=\"#4682B4\", name=\"文学\"\n",
"), row=1, col=1)\n",
"fig.add_trace(go.Bar(\n",
" x=[w for w, _ in kw_s], y=[s for _, s in kw_s],\n",
" marker_color=\"#CD5C5C\", name=\"スポーツ\"\n",
"), row=1, col=2)\n",
"fig.update_layout(title=\"ドメイン特徴語（TF 比スコア）\", showlegend=False, height=420)\n",
"fig.show()\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ukcy9A2iksrs"
},
"source": [
"## 振り返り問題\n",
"\n",
"**Q1**：ステップ4の頻出語グラフを見て、2つのドメインの頻出語はどう違いましたか？\n",
"その違いがトライグラム LM の補完結果にどう影響するかを説明してください。\n",
"\n",
"**Q2**：ステップ5で「共通文脈③」に対し、文学 LM とニュース LM の補完候補が\n",
"異なりました。なぜそうなるかを「トライグラムカウントの差異」で説明してください。\n",
"\n",
"**Q3**：ステップ6のヒートマップで、対角線（同ドメイン）の PPL が低くなりましたか？\n",
"この結果を「スマホ補完がユーザー固有のフレーズを学習すると精度が向上する」\n",
"という現象と関連づけて説明してください。\n",
"\n",
"**Q4**：Laplace スムージング（k=1）のとき、未出現トライグラムの確率を式で求め、語彙サイズ |V| = 10000\n",
"のときの具体的な値を計算してください。\n",
"\n",
"**Q5**：バイグラム LM とトライグラム LM を比べると、理論上はどちらが PPL が低い\n",
"（良い）と期待されますか？またトライグラムには「データスパース問題」の観点から\n",
"どのようなデメリットがありますか？\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "PsYjLHlFk1bM",
"outputId": "102b4452-e402-4deb-9d3b-ed84a13e12de"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Tue Jun 23 09:35:50 AM UTC 2026\n"
]
}
],
"source": [
"!date"
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}