{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.0" }, "colab": { "provenance": [], "gpuType": "T4" }, "accelerator": "GPU" }, "cells": [ { "cell_type": "markdown", "id": "cell-01", "metadata": { "id": "cell-01" }, "source": [ "# レポート課題4：RAGパラメータ比較実験\n", "\n", "---\n", "\n", "## 事前資料\n", "\n", "RAGの基礎概念・用語・今回の実装方針については先に **`guidance.ipynb`** を参照してください。\n", "\n", "> 📄 **[guidance.ipynb](guidance.ipynb)** — RAG基礎概念・チャンク分割・ROUGE-1・Top-K などの解説\n", "\n", "---\n", "\n", "## 課題趣旨\n", "\n", "LLMが回答する前に外部情報源を検索し、その情報に基づいて回答を生成する技術（RAG）を体験します。\n", "\n", "今回の知識ベースは架空企業「NALTOMA AI」の社内規程文書（英語）8 件です。\n", "TinyLlama はこれらの文書を学習していないため、**RAGなしでは正しく答えられません**。\n", "これにより「チャンクの内容が検索できれば答えられる・できなければ答えられない」という\n", "RAGの有効性を確認しつつ、また、必ず正答できるわけではないという限界についても観察する内容になっています。" ] }, { "cell_type": "markdown", "id": "cell-02", "metadata": { "id": "cell-02" }, "source": [ "## 全体の流れ\n", "\n", "| フェーズ | Level | 内容 | 実行頻度 |\n", "|---------|-------|------|----------|\n", "| **Phase 1：セットアップ** | — | Drive マウント・ライブラリインストール・モデル読み込み | セッション開始時に1回 |\n", "| **Phase 2：実験** | Level 1 | Medium 条件のベースライン確認 | 繰り返し可 |\n", "| | Level 2 | チャンクサイズ比較（Small / Medium / Large） | 繰り返し可 |\n", "| | Level 3 | Top-K 比較（K = 1, 2, 3, 5） | 繰り返し可 |\n", "| | Level 4 | 考察（記述） | — |\n", "\n", "### 実行手順\n", "\n", "1. Google Drive 直下に `2026dm-rep4` フォルダを作成する。\n", "2. **本ノートブック・`guidance.ipynb`・`naltoma_ai.tsv`** を同フォルダにアップロードする。\n", "3. GPUを指定する。\n", " - ダウンロード以外の処理時間は、CPU実行すると約1時間かかります。GPU実行すると約10分で終わります。GPUを使いたい場合には以下の手順を取ってください。なお、**GPUは使用上限があります**。使用上限に達した場合には数時間〜24時間程度、使用できなくなります。この場合は待つか、CPUに戻してから実行してください。\n", " - 「ランタイム」から「ランタイムのタイプの変更」を選ぶ。\n", " - 「T4 GPU」を選ぶと良い。選択肢の中では低スペックだが十分早い。\n", "4. Phase 1 のセルを上から順に実行する（初回はモデルのダウンロードに数分かかる）。\n", "5. Phase 2 の Level 1〜4 を順に取り組む。\n", "6. **セッション切断後は Phase 1（Drive マウント〜モデル読み込み）から再実行すること。**\n", "\n", "### 独自データセットを使う場合\n", "\n", "Step 0 の `DATASET_FILE` 変数の値を自分の TSV ファイル名に変更するだけで対応できます。\n", "TSV の形式：1行目はヘッダー（`doc_id`\\t`text`）、2行目以降が文書データ。\n", "\n", "### 再現性\n", "\n", "埋め込み・FAISS・TinyLlama（`do_sample=False`）はすべて決定論的に動作します。" ] }, { "cell_type": "markdown", "id": "cell-03", "metadata": { "id": "cell-03" }, "source": [ "---\n", "\n", "## Phase 1：セットアップ\n", "\n", "**新しいセッションを開くたびにここから実行すること。**" ] }, { "cell_type": "markdown", "id": "cell-04", "metadata": { "id": "cell-04" }, "source": [ "### Step 0：Google Drive マウント & キャッシュ設定" ] }, { "cell_type": "code", "id": "cell-05", "execution_count": null, "metadata": { "id": "cell-05" }, "outputs": [], "source": [ "from google.colab import drive\n", "drive.mount('/content/drive')\n", "\n", "import os\n", "os.environ['HF_HOME'] = '/content/drive/MyDrive/hf_cache'\n", "WORK_DIR = '/content/drive/MyDrive/2026dm-rep4'\n", "os.makedirs(WORK_DIR, exist_ok=True)\n", "\n", "# ── データセットファイル名 ─────────────────────────────────────────────────\n", "# 独自データを使う場合はここだけ変更する（TSV 形式: doc_id \\t text）\n", "DATASET_FILE = 'naltoma_ai.tsv'\n", "# ─────────────────────────────────────────────────────────────────────────\n", "\n", "print('Drive マウント完了')\n", "print(f'データセット: {DATASET_FILE}')" ] }, { "cell_type": "markdown", "id": "cell-06", "metadata": { "id": "cell-06" }, "source": [ "### Step 1：ライブラリインストール" ] }, { "cell_type": "code", "id": "cell-07", "execution_count": null, "metadata": { "id": "cell-07" }, "outputs": [], "source": [ "!pip install -q transformers accelerate sentence-transformers faiss-cpu rouge-score plotly" ] }, { "cell_type": "markdown", "id": "cell-08", "metadata": { "id": "cell-08" }, "source": [ "### Step 2：埋め込みモデルの読み込み" ] }, { "cell_type": "code", "id": "cell-09", "execution_count": null, "metadata": { "id": "cell-09" }, "outputs": [], "source": [ "from sentence_transformers import SentenceTransformer\n", "\n", "EMBED_MODEL_ID = 'sentence-transformers/all-MiniLM-L6-v2'\n", "embed_model = SentenceTransformer(EMBED_MODEL_ID, cache_folder=os.environ['HF_HOME'])\n", "print(f'埋め込みモデル読み込み完了: {EMBED_MODEL_ID}')" ] }, { "cell_type": "markdown", "id": "cell-10", "metadata": { "id": "cell-10" }, "source": [ "### Step 3：生成モデルの読み込み（TinyLlama）\n", "\n", "> **注意：** 初回はモデルのダウンロード（約 700 MB）が発生します。2回目以降は Drive キャッシュから読み込まれます。" ] }, { "cell_type": "code", "id": "cell-11", "execution_count": null, "metadata": { "id": "cell-11" }, "outputs": [], "source": [ "import torch\n", "from transformers import AutoTokenizer, AutoModelForCausalLM\n", "\n", "GEN_MODEL_ID = 'TinyLlama/TinyLlama-1.1B-Chat-v1.0'\n", "\n", "tokenizer = AutoTokenizer.from_pretrained(GEN_MODEL_ID)\n", "gen_model = AutoModelForCausalLM.from_pretrained(\n", " GEN_MODEL_ID,\n", " dtype=torch.float16,\n", " device_map='auto',\n", ")\n", "gen_model.eval()\n", "\n", "# max_length と max_new_tokens の競合を解消する\n", "if getattr(gen_model.generation_config, 'max_length', None) is not None:\n", " gen_model.generation_config.max_length = None\n", " print('generation_config.max_length をリセットしました')\n", "\n", "print(f'生成モデル読み込み完了: {GEN_MODEL_ID}')\n", "print(f'デバイス: {next(gen_model.parameters()).device}')" ] }, { "cell_type": "markdown", "id": "cell-12", "metadata": { "id": "cell-12" }, "source": [ "---\n", "\n", "## Phase 2：実験\n", "\n", "**セッションが新しい場合は Phase 1 を先に実行すること。**" ] }, { "cell_type": "markdown", "id": "cell-13", "metadata": { "id": "cell-13" }, "source": [ "### 知識ベース（ソース文書）\n", "\n", "TSV ファイルから読み込んだソース文書。デフォルトは「NALTOMA AI」社内規程 8 件（英語・各 430〜520 文字）。\n", "\n", "| 文書ID | 内容 |\n", "|--------|------|\n", "| remote_work | リモートワーク規程 |\n", "| expense_policy | 経費精算規程 |\n", "| code_review | コードレビュー規程 |\n", "| deployment | デプロイ規程 |\n", "| meeting_rooms | 会議室予約規程 |\n", "| api_policy | API利用規程 |\n", "| onboarding | オンボーディング規程 |\n", "| performance_review | 評価制度規程 |\n", "\n", "> **重要：** これらは架空の文書です。TinyLlama はこれらの規程を学習していないため、\n", "> RAG による検索なしでは正しく答えることができません。\n", "\n", "> **独自データを使う場合：** Step 0 の `DATASET_FILE` を変更するだけで、\n", "> 以降のセルはすべて自動的に新しいデータセットで動作します。" ] }, { "cell_type": "code", "id": "cell-14", "execution_count": null, "metadata": { "id": "cell-14" }, "outputs": [], "source": [ "import csv\n", "\n", "dataset_path = f'{WORK_DIR}/{DATASET_FILE}'\n", "documents = []\n", "with open(dataset_path, encoding='utf-8') as f:\n", " reader = csv.DictReader(f, delimiter='\\t')\n", " for row in reader:\n", " documents.append((row['doc_id'], row['text']))\n", "\n", "print(f'データセット : {DATASET_FILE}')\n", "print(f'ソース文書数 : {len(documents)}')\n", "for key, text in documents:\n", " print(f' [{key:20s}] {len(text)} 文字')\n", "print(f'\\n総文字数: {sum(len(t) for _, t in documents)}')" ] }, { "cell_type": "markdown", "id": "cell-15", "metadata": { "id": "cell-15" }, "source": [ "### クエリ（全実験で共通・変更不可）" ] }, { "cell_type": "code", "id": "cell-16", "execution_count": null, "metadata": { "id": "cell-16" }, "outputs": [], "source": [ "queries = [\n", " 'What are the core working hours for remote employees at NALTOMA AI?',\n", " 'What is the deadline for submitting expense reports?',\n", " 'How many approvals are required to merge a pull request?',\n", " 'What days and times do production deployments occur?',\n", " 'What is the API rate limit per team per minute?',\n", "]\n", "\n", "# 各クエリの参照回答（ROUGE-1 評価の基準として使用）\n", "# TinyLlama の出力ではなく、ソース文書に基づいて事前定義した正解文字列\n", "reference_answers = {\n", " 0: 'Remote employees must be available during core hours from 10:00 to 16:00 JST on weekdays.',\n", " 1: 'Business expenses must be submitted within 30 calendar days of purchase.',\n", " 2: 'Pull requests require a minimum of two approvals from team members who were not the PR author.',\n", " 3: 'Production deployments occur on Tuesdays and Thursdays at 02:00 JST.',\n", " 4: 'The API rate limit is 120 requests per minute per team token.',\n", "}\n", "\n", "print(f'クエリ数: {len(queries)}')\n", "for i, q in enumerate(queries):\n", " print(f' Q{i+1}: {q}')" ] }, { "cell_type": "markdown", "id": "cell-17", "metadata": { "id": "cell-17" }, "source": [ "### ユーティリティ関数\n", "\n", "以下の関数を Level 1〜3 で共通利用する。変更不可。" ] }, { "cell_type": "code", "id": "cell-18", "execution_count": null, "metadata": { "id": "cell-18" }, "outputs": [], "source": [ "import faiss\n", "import numpy as np\n", "from rouge_score import rouge_scorer as rouge_module\n", "\n", "def split_into_chunks(text, chunk_size, overlap=0, word_boundary=True):\n", " if word_boundary:\n", " words = text.split()\n", " chunks = []\n", " start = 0\n", " while start < len(words):\n", " chunk_words, total_len = [], 0\n", " for w in words[start:]:\n", " add = len(w) + (1 if chunk_words else 0)\n", " if total_len + add <= chunk_size:\n", " chunk_words.append(w)\n", " total_len += add\n", " else:\n", " break\n", " if not chunk_words:\n", " chunk_words = [words[start]]\n", " chunks.append(' '.join(chunk_words))\n", " start += len(chunk_words)\n", " return [c for c in chunks if len(c.strip()) > 5]\n", " else:\n", " chunks, start = [], 0\n", " while start < len(text):\n", " chunk = text[start:start + chunk_size].strip()\n", " if len(chunk) > 10:\n", " chunks.append(chunk)\n", " start += chunk_size - overlap\n", " return chunks\n", "\n", "def build_chunks_from_docs(documents, chunk_size, overlap=0, word_boundary=True):\n", " all_chunks = []\n", " for _, text in documents:\n", " all_chunks.extend(split_into_chunks(text, chunk_size, overlap, word_boundary))\n", " return all_chunks\n", "\n", "def build_index(chunks, embed_model):\n", " embeddings = embed_model.encode(chunks, convert_to_numpy=True).astype('float32')\n", " index = faiss.IndexFlatL2(embeddings.shape[1])\n", " index.add(embeddings)\n", " return index, embeddings\n", "\n", "def retrieve(query, index, chunks, embed_model, top_k=3):\n", " q_vec = embed_model.encode([query], convert_to_numpy=True).astype('float32')\n", " distances, indices = index.search(q_vec, top_k)\n", " return [{'chunk': chunks[i], 'distance': float(d)}\n", " for d, i in zip(distances[0], indices[0])]\n", "\n", "def rag_generate(query, retrieved_chunks, max_new_tokens=150):\n", " context = '\\n'.join([r['chunk'] for r in retrieved_chunks])\n", " messages = [\n", " {'role': 'system',\n", " 'content': 'You are a helpful assistant. '\n", " 'Answer questions based only on the provided context. '\n", " 'Be concise and direct.'},\n", " {'role': 'user',\n", " 'content': f'Context:\\n{context}\\n\\nQuestion: {query}'},\n", " ]\n", " prompt = tokenizer.apply_chat_template(\n", " messages, tokenize=False, add_generation_prompt=True\n", " )\n", " inputs = tokenizer(prompt, return_tensors='pt').to(gen_model.device)\n", " with torch.no_grad():\n", " outputs = gen_model.generate(\n", " **inputs,\n", " max_new_tokens=max_new_tokens,\n", " do_sample=False,\n", " pad_token_id=tokenizer.eos_token_id,\n", " )\n", " generated = outputs[0][inputs['input_ids'].shape[1]:]\n", " answer = tokenizer.decode(generated, skip_special_tokens=True).strip()\n", " return answer, prompt\n", "\n", "_scorer = rouge_module.RougeScorer(['rouge1'], use_stemmer=True)\n", "\n", "def rouge1(hypothesis, reference):\n", " return _scorer.score(reference, hypothesis)['rouge1'].fmeasure\n", "\n", "print('ユーティリティ関数を定義しました')" ] }, { "cell_type": "markdown", "id": "cell-19", "metadata": { "id": "cell-19" }, "source": [ "---\n", "\n", "### Level 1：ベースライン確認（Medium 条件）\n", "\n", "チャンクサイズ **120 文字**（Medium）で全社内規程文書を分割し、チャンクの状態を確認する。\n", "次に Q1・Q4 について検索と回答生成を実行して基準となる出力を観察する。" ] }, { "cell_type": "code", "id": "cell-20", "execution_count": null, "metadata": { "id": "cell-20" }, "outputs": [], "source": [ "CHUNK_MEDIUM = 120\n", "\n", "chunks_medium = build_chunks_from_docs(documents, chunk_size=CHUNK_MEDIUM)\n", "print(f'チャンクサイズ: {CHUNK_MEDIUM} 文字（単語境界分割）')\n", "print(f'総チャンク数 : {len(chunks_medium)}')\n", "print()\n", "print('── 先頭 5 チャンク ──')\n", "for i, c in enumerate(chunks_medium[:5]):\n", " print(f'[{i:02d}] ({len(c)} 文字) {c}')" ] }, { "cell_type": "code", "id": "cell-21", "execution_count": null, "metadata": { "id": "cell-21" }, "outputs": [], "source": [ "index_medium, _ = build_index(chunks_medium, embed_model)\n", "print(f'インデックス内ベクトル数: {index_medium.ntotal}')" ] }, { "cell_type": "code", "id": "cell-22", "execution_count": null, "metadata": { "id": "cell-22" }, "outputs": [], "source": [ "demo_queries = [queries[0], queries[3]] # Q1, Q4\n", "\n", "for q in demo_queries:\n", " retrieved = retrieve(q, index_medium, chunks_medium, embed_model, top_k=3)\n", " answer, _ = rag_generate(q, retrieved)\n", " print(f'Q: {q}')\n", " print(f'A: {answer}')\n", " print('検索ヒット（上位3件）:')\n", " for j, r in enumerate(retrieved):\n", " print(f' [{j+1}] (dist={r[\"distance\"]:.3f}) {r[\"chunk\"][:80]}')\n", " print()" ] }, { "cell_type": "markdown", "id": "cell-23", "metadata": { "id": "cell-23" }, "source": [ "### ✍️ Level 1 報告事項\n", "\n", "**(1)** Medium 条件のチャンク総数を報告せよ。\n", "\n", "**(2)** 先頭 5 チャンクを掲載し、どの文書の内容が含まれているかを確認せよ。単語境界分割によって文字が単語の区切りで終わっていることをコメントせよ。\n", "\n", "**(3)** Q1・Q4 それぞれについて、検索されたチャンク上位 3 件の内容（冒頭 60 文字程度）を掲載せよ。\n", "\n", "**(4)** 生成された回答を掲載し、参照回答と照合して正しく答えているかを簡潔にコメントせよ。\n", "特に、**RAG なしでは TinyLlama が答えられない理由**をあわせて述べよ。" ] }, { "cell_type": "markdown", "id": "cell-24", "metadata": { "id": "cell-24" }, "source": [ "---\n", "\n", "### Level 2：チャンクサイズ比較（Small / Medium / Large）\n", "\n", "3 種類のチャンクサイズでそれぞれインデックスを構築し、同一クエリに対する検索結果と回答品質を比較する。\n", "\n", "| 条件 | チャンクサイズ |\n", "|------|-------------|\n", "| Small | 50 文字 |\n", "| Medium | 120 文字 |\n", "| Large | 250 文字 |\n", "\n", "> **評価：** `reference_answers` を参照として ROUGE-1 を計算する（条件間の絶対比較）。\n", "> 数値だけでなく**ヒットチャンクの内容と実際の回答テキスト**を必ず確認すること。" ] }, { "cell_type": "code", "id": "cell-25", "execution_count": null, "metadata": { "id": "cell-25" }, "outputs": [], "source": [ "CHUNK_SMALL, CHUNK_LARGE = 50, 250\n", "conditions = {'Small': CHUNK_SMALL, 'Medium': CHUNK_MEDIUM, 'Large': CHUNK_LARGE}\n", "\n", "chunks_dict, index_dict = {}, {}\n", "for name, size in conditions.items():\n", " ch = build_chunks_from_docs(documents, chunk_size=size)\n", " idx, _ = build_index(ch, embed_model)\n", " chunks_dict[name] = ch\n", " index_dict[name] = idx\n", " print(f'{name:6s} (size={size:3d}): チャンク数 = {len(ch):3d}')" ] }, { "cell_type": "code", "id": "cell-26", "execution_count": null, "metadata": { "id": "cell-26" }, "outputs": [], "source": [ "TOP_K_FIXED = 3\n", "results_l2 = {}\n", "\n", "for name in conditions:\n", " results_l2[name] = {}\n", " for qi, q in enumerate(queries):\n", " retrieved = retrieve(q, index_dict[name], chunks_dict[name], embed_model, top_k=TOP_K_FIXED)\n", " answer, _ = rag_generate(q, retrieved)\n", " results_l2[name][qi] = {'answer': answer, 'hits': retrieved}\n", " print(f'{name} 完了')\n", "print('全条件の生成完了')" ] }, { "cell_type": "code", "id": "cell-27", "execution_count": null, "metadata": { "id": "cell-27" }, "outputs": [], "source": [ "import pandas as pd\n", "\n", "rouge_table = []\n", "for name in conditions:\n", " for qi in range(len(queries)):\n", " score = rouge1(results_l2[name][qi]['answer'], reference_answers[qi])\n", " rouge_table.append({'condition': name, 'query': f'Q{qi+1}', 'rouge1': score})\n", "\n", "df_rouge = pd.DataFrame(rouge_table)\n", "pivot = df_rouge.pivot(index='query', columns='condition', values='rouge1').round(3)\n", "print('ROUGE-1 スコア（reference_answers 参照）')\n", "print(pivot)\n", "print()\n", "print('条件別 平均 ROUGE-1:')\n", "print(df_rouge.groupby('condition')['rouge1'].mean().round(3))" ] }, { "cell_type": "code", "id": "cell-28", "execution_count": null, "metadata": { "id": "cell-28" }, "outputs": [], "source": [ "for qi in [0, 3]:\n", " print('=' * 70)\n", " print(f'クエリ Q{qi+1}: {queries[qi]}')\n", " print(f'参照回答 : {reference_answers[qi]}')\n", " for name in conditions:\n", " r = results_l2[name][qi]\n", " print(f'\\n [{name}]')\n", " print(f' 回答: {r[\"answer\"][:180]}')\n", " print(f' ヒット[1]: {r[\"hits\"][0][\"chunk\"][:70]}')\n", " print()" ] }, { "cell_type": "code", "id": "cell-29", "execution_count": null, "metadata": { "id": "cell-29" }, "outputs": [], "source": [ "import plotly.graph_objects as go\n", "from plotly.subplots import make_subplots\n", "\n", "cond_names = list(conditions.keys())\n", "colors = ['#42A5F5', '#66BB6A', '#FFA726']\n", "chunk_counts = [len(chunks_dict[n]) for n in cond_names]\n", "avg_rouge = df_rouge.groupby('condition')['rouge1'].mean().reindex(cond_names).values\n", "\n", "fig = make_subplots(rows=1, cols=2,\n", " subplot_titles=['Number of Chunks per Condition',\n", " 'Avg ROUGE-1 (vs reference_answers)'])\n", "\n", "for i, (name, cnt, rg) in enumerate(zip(cond_names, chunk_counts, avg_rouge)):\n", " fig.add_trace(go.Bar(x=[name], y=[cnt], name=name,\n", " marker_color=colors[i], showlegend=False,\n", " text=[cnt], textposition='outside'), row=1, col=1)\n", " fig.add_trace(go.Bar(x=[name], y=[round(rg,3)], name=name,\n", " marker_color=colors[i], showlegend=False,\n", " text=[round(rg,3)], textposition='outside'), row=1, col=2)\n", "\n", "fig.update_yaxes(range=[0, max(chunk_counts)*1.2], row=1, col=1)\n", "fig.update_yaxes(range=[0, 1.1], row=1, col=2)\n", "fig.update_layout(title_text='Level 2: Chunk Size Comparison', height=420)\n", "fig.show()\n", "fig.write_html(f'{WORK_DIR}/level2_chunk_comparison.html')\n", "print('図を保存しました')" ] }, { "cell_type": "markdown", "id": "cell-30", "metadata": { "id": "cell-30" }, "source": [ "### ✍️ Level 2 報告事項\n", "\n", "**(1)** Small / Medium / Large 各条件のチャンク総数を報告せよ。\n", "\n", "**(2)** ROUGE-1 スコアの表と条件別平均を掲載せよ。\n", "\n", "**(3)** 代表クエリ（Q1・Q4）について、条件ごとの回答テキストを掲載せよ（各 180 文字程度）。\n", "\n", "**(4)** Medium 条件と比較して Small 条件でスコアが変化したクエリを 1 件以上挙げ、**ヒットチャンクの内容を引用しながら**\n", "スコアが高く/低くなった理由を説明せよ。\n", "\n", "**(5)** チャンクサイズにかかわらず、Top-K 検索結果には異なる規程のチャンクが混入することがある。いずれかの条件でそのようなケースが観察された場合、具体的なチャンク内容を引用しながら回答品質への影響を考察せよ。" ] }, { "cell_type": "markdown", "id": "cell-31", "metadata": { "id": "cell-31" }, "source": [ "---\n", "\n", "### Level 3：Top-K 比較\n", "\n", "チャンクサイズを **Medium（120 文字）** で固定し、Top-K のみを変化させる。\n", "\n", "| K | 説明 |\n", "|---|------|\n", "| 1 | 最も類似度の高い 1 チャンクのみ |\n", "| 2 | 2 チャンク |\n", "| 3 | Level 2 のベースライン |\n", "| 5 | より多くのコンテキスト。ノイズ混入リスクも増す |" ] }, { "cell_type": "code", "id": "cell-32", "execution_count": null, "metadata": { "id": "cell-32" }, "outputs": [], "source": [ "K_VALUES = [1, 2, 3, 5]\n", "results_l3 = {}\n", "\n", "for k in K_VALUES:\n", " results_l3[k] = {}\n", " for qi, q in enumerate(queries):\n", " retrieved = retrieve(q, index_medium, chunks_medium, embed_model, top_k=k)\n", " answer, prompt = rag_generate(q, retrieved)\n", " results_l3[k][qi] = {\n", " 'answer' : answer,\n", " 'hits' : retrieved,\n", " 'prompt_len': len(prompt),\n", " }\n", " print(f'K={k} 完了')\n", "print('全 K の生成完了')" ] }, { "cell_type": "code", "id": "cell-33", "execution_count": null, "metadata": { "id": "cell-33" }, "outputs": [], "source": [ "rouge_l3 = []\n", "for k in K_VALUES:\n", " for qi in range(len(queries)):\n", " score = rouge1(results_l3[k][qi]['answer'], reference_answers[qi])\n", " plen = results_l3[k][qi]['prompt_len']\n", " rouge_l3.append({'K': k, 'query': f'Q{qi+1}', 'rouge1': score, 'prompt_len': plen})\n", "\n", "df_l3 = pd.DataFrame(rouge_l3)\n", "pivot3 = df_l3.pivot(index='query', columns='K', values='rouge1').round(3)\n", "print('ROUGE-1 スコア（reference_answers 参照）')\n", "print(pivot3)\n", "print()\n", "print('K 別 平均プロンプト文字数:')\n", "print(df_l3.groupby('K')['prompt_len'].mean().round(0))" ] }, { "cell_type": "code", "id": "cell-34", "execution_count": null, "metadata": { "id": "cell-34" }, "outputs": [], "source": [ "for qi in [1, 4]:\n", " print('=' * 70)\n", " print(f'クエリ Q{qi+1}: {queries[qi]}')\n", " print(f'参照回答 : {reference_answers[qi]}')\n", " for k in K_VALUES:\n", " r = results_l3[k][qi]\n", " hit_preview = ' | '.join([h['chunk'][:28]+'...' for h in r['hits']])\n", " print(f' [K={k}] {r[\"answer\"][:130]}')\n", " print(f' ヒット: {hit_preview}')\n", " print()" ] }, { "cell_type": "code", "id": "cell-35", "execution_count": null, "metadata": { "id": "cell-35" }, "outputs": [], "source": [ "avg_r1 = df_l3.groupby('K')['rouge1'].mean()\n", "avg_plen = df_l3.groupby('K')['prompt_len'].mean()\n", "\n", "fig = make_subplots(rows=1, cols=2,\n", " subplot_titles=['Avg ROUGE-1 vs K (reference_answers)',\n", " 'Avg Prompt Length vs K'])\n", "\n", "fig.add_trace(go.Scatter(x=K_VALUES, y=avg_r1.values,\n", " mode='lines+markers', line=dict(color='steelblue', width=2),\n", " marker=dict(size=8), showlegend=False), row=1, col=1)\n", "\n", "fig.add_trace(go.Scatter(x=K_VALUES, y=avg_plen.values,\n", " mode='lines+markers', line=dict(color='coral', width=2),\n", " marker=dict(size=8), showlegend=False), row=1, col=2)\n", "\n", "fig.update_xaxes(tickvals=K_VALUES)\n", "fig.update_yaxes(range=[0, 1.1], row=1, col=1)\n", "fig.update_layout(title_text='Level 3: Top-K Comparison (Medium chunks)', height=420)\n", "fig.show()\n", "fig.write_html(f'{WORK_DIR}/level3_topk_comparison.html')\n", "print('図を保存しました')" ] }, { "cell_type": "markdown", "id": "cell-36", "metadata": { "id": "cell-36" }, "source": [ "### ✍️ Level 3 報告事項\n", "\n", "**(1)** ROUGE-1 スコアの表と K 別平均プロンプト文字数を掲載せよ。\n", "\n", "**(2)** K を増やすことで ROUGE-1 が改善したクエリと改善しなかったクエリを各 1 件以上挙げ、\n", "ヒットチャンクの内容を引用しながら理由を考察せよ。\n", "\n", "**(3)** K=5 の場合、**質問と異なる規程文書** のチャンクが混入していると思われるケースはあったか？\n", "あった場合は具体的なチャンク内容と文書IDを示し、それが回答にどう影響したかを述べよ。\n", "\n", "**(4)** 「プロンプト長が増えることで生じうる問題」を、コンテキストウィンドウの観点から述べよ。" ] }, { "cell_type": "markdown", "id": "cell-37", "metadata": { "id": "cell-37" }, "source": [ "---\n", "\n", "### Level 4：考察\n", "\n", "Level 2・3 の実験結果全体を踏まえ、以下の問いに答えよ。コードの実行は不要。" ] }, { "cell_type": "markdown", "id": "cell-38", "metadata": { "id": "cell-38" }, "source": [ "### ✍️ Level 4 報告事項\n", "\n", "**必須（両方に答えること）**\n", "\n", "**(1)** 今回の実験条件の中で「最も実用的」と判断するチャンクサイズと Top-K の組み合わせを選び、\n", "実験結果に基づいて理由を述べよ。\n", "\n", "**(2)** 今回の知識ベースは「社内規程」という TinyLlama が学習していない文書であった。\n", "もし ML の教科書テキスト（TinyLlama が学習している内容）を知識ベースにした場合、\n", "実験結果の解釈がどう変わるか。RAGの有効性の評価という観点から述べよ。\n", "\n", "**選択（以下から 1 つ以上）**\n", "\n", "**(3)** 本課題で使用した ROUGE-1（`reference_answers` を参照とする手法）の問題点を具体的に述べよ。\n", "より適切な評価方法があるとすれば何か？\n", "- ヒント: BERTScore / コサイン類似度 / LLM-as-Judge / 情報抽出ベース\n", "\n", "**(4)** 今回の知識ベース（all-MiniLM-L6-v2）は英語テキストで、埋め込みモデルも英語中心に学習されている。\n", "日本語の社内規程を知識ベースにする場合、どのような問題が予想されるか。\n", "\n", "**(5)** RAG を使ってもモデルが誤った回答を生成してしまうケースを今回の実験から 1 件以上挙げ、\n", "「検索の失敗」と「生成モデルの限界」のどちらが主因かを分析せよ。" ] }, { "cell_type": "markdown", "id": "cell-39", "metadata": { "id": "cell-39" }, "source": [ "---\n", "\n", "## オプション：オーバーラップ実験\n", "\n", "以下は必須ではない。取り組んだ場合はレポートに記載すること（加点対象）。\n", "\n", "チャンクサイズ 120 文字・オーバーラップ 40 文字の条件（`word_boundary=False`）を追加し、\n", "Medium（オーバーラップなし・単語境界）と比較する。\n", "\n", "単にスコアを比較するだけでなく「なぜオーバーラップが効く（または効かない）か」を\n", "チャンクの内容から説明すること。" ] }, { "cell_type": "code", "id": "cell-40", "execution_count": null, "metadata": { "id": "cell-40" }, "outputs": [], "source": [ "CHUNK_OVERLAP = 40\n", "\n", "chunks_overlap = build_chunks_from_docs(\n", " documents, chunk_size=CHUNK_MEDIUM, overlap=CHUNK_OVERLAP, word_boundary=False\n", ")\n", "index_overlap, _ = build_index(chunks_overlap, embed_model)\n", "print(f'Overlap 条件 (word_boundary=False): チャンク数 = {len(chunks_overlap)}')\n", "print(f'Medium 条件 (word_boundary=True) : チャンク数 = {len(chunks_medium)}')\n", "\n", "results_ov = {}\n", "for qi, q in enumerate(queries):\n", " ret = retrieve(q, index_overlap, chunks_overlap, embed_model, top_k=3)\n", " ans, _ = rag_generate(q, ret)\n", " results_ov[qi] = {'answer': ans, 'hits': ret}\n", "\n", "print('\\nROUGE-1（reference_answers 参照）')\n", "for qi in range(len(queries)):\n", " s_ov = rouge1(results_ov[qi]['answer'], reference_answers[qi])\n", " s_med = rouge1(results_l2['Medium'][qi]['answer'], reference_answers[qi])\n", " print(f' Q{qi+1}: Overlap={s_ov:.3f} Medium={s_med:.3f}')" ] }, { "cell_type": "markdown", "id": "cell-41", "metadata": { "id": "cell-41" }, "source": [ "---\n", "\n", "## 提出物\n", "\n", "- **このノートブック（report4_2026.ipynb）**\n", " - Level 1〜4 のすべてのセルが実行済みの状態で提出すること\n", " - ✍️ マークの報告事項に対する回答・考察をノートブック内の Markdown セルに記入すること\n", " - オプションに取り組んだ場合はその結果と考察も記載すること\n", "- ノートブックでレポートを書きづらい場合には、別途自由形式で作成して提出しても良い。ただし実行結果を保存したノートブックは必ず提出すること。\n", "\n", "> `hf_cache`、生成された HTML ファイルは提出不要です。\n", "> 独自データセット TSV を使用した場合はそのファイルも提出すること。" ] }, { "cell_type": "code", "source": [], "metadata": { "id": "Ib4LXNoy1Zqo" }, "id": "Ib4LXNoy1Zqo", "execution_count": null, "outputs": [] } ] }