!date
!python --version
Fri Apr 11 03:29:22 AM UTC 2025
Python 3.11.12
14. 数値データに対する前処理コード例#
変更履歴
2024年5月: quilt 周りで不具合あり。そこでquiltは使用せず、別途データセットをファイルとして用意して利用するように修正。本コードを実行する際には、別に用意している
youtuber.xlsxをダウンロードしてから実行してください。2025年4月11日: matplotlib => plotlyに変更。
ref.
preprocess methods
data: YouTuberデータセット公開してみた
全体の流れ
データセットの準備。
数値データに対する前処理の例
手法1：バイナリ化
手法2：アドホックな離散化
手法3：統計的な離散化
手法4：ログスケール化
デフォルトとログスケールとの比較
手法5：標準化
手法6：min-maxスケーリング
特徴ベクトルに対する前処理の例
手法7：正規化
手法8：正規分布への写像
デフォルトとbox-cox写像との比較
14.1. 環境構築#
#!pip install quilt
#!quilt install haradai1262/YouTuber
14.2. データセットの準備#
#from quilt.data.haradai1262 import YouTuber
import numpy as np
import pandas as pd
#import matplotlib.pyplot as plt
import plotly.express as px
import plotly.graph_objects as go
#df = YouTuber.channels.UUUM()
#df = YouTuber.channel_videos.UUUM_videos()
df = pd.read_excel("youtuber.xlsx")
# check the descriptive statistics of numerical data
df.describe()
| viewCount | likeCount | favoriteCount | dislikeCount | commentCount | TopicIds | idx | |
|---|---|---|---|---|---|---|---|
| count | 6.627900e+04 | 64256.000000 | 66289.0 | 64256.000000 | 65997.000000 | 0.0 | 66289.000000 |
| mean | 4.545539e+05 | 3233.703390 | 0.0 | 296.430014 | 533.418807 | NaN | 235.730136 |
| std | 1.328105e+06 | 9768.090605 | 0.0 | 1633.734833 | 2253.437482 | NaN | 142.868881 |
| min | 0.000000e+00 | 0.000000 | 0.0 | 0.000000 | 0.000000 | NaN | 1.000000 |
| 25% | 3.907100e+04 | 266.000000 | 0.0 | 23.000000 | 55.000000 | NaN | 111.000000 |
| 50% | 1.214190e+05 | 776.000000 | 0.0 | 67.000000 | 150.000000 | NaN | 229.000000 |
| 75% | 3.512260e+05 | 2260.000000 | 0.0 | 201.000000 | 394.000000 | NaN | 357.000000 |
| max | 8.664236e+07 | 630051.000000 | 0.0 | 213677.000000 | 227598.000000 | NaN | 501.000000 |
# the description of data frame
df.head()
| id | title | description | liveBroadcastContent | tags | publishedAt | thumbnails | viewCount | likeCount | favoriteCount | ... | commentCount | caption | definition | dimension | duration | projection | TopicIds | relevantTopicIds | idx | cid | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | R7V5d94XkGQ | 【大食い】超高級寿司店で３人で食べ放題したらいくらかかるの!?【大トロ1カン2,000円】 | 提供：ポコロンダンジョンズ\n\n\n\niOS：https://bit.ly/2sGgOR... | none | ['ヒカキン', 'ヒカキンtv', 'hikakintv', 'hikakin', 'ひか... | 2018-06-30T04:00:01.000Z | https://i.ytimg.com/vi/R7V5d94XkGQ/default.jpg | 2244205.0 | 27703.0 | 0 | ... | 8647.0 | False | hd | 2d | PT21M16S | rectangular | NaN | ['/m/02wbm', '/m/019_rr', '/m/019_rr', '/m/02w... | 1 | UCZf__ehlCEBPop___sldpBUQ |
| 1 | 2R9_bkcWNd4 | 【女王集結】女性YouTuberたちと飲みながら本音トークしてみたら爆笑www | しばなんチャンネルの動画\n\n\n\nhttps://www.youtube.com/wa... | none | ['ヒカキン', 'ヒカキンtv', 'hikakintv', 'hikakin', 'ひか... | 2018-06-29T08:00:01.000Z | https://i.ytimg.com/vi/2R9_bkcWNd4/default.jpg | 1869268.0 | 30889.0 | 0 | ... | 8859.0 | False | hd | 2d | PT18M38S | rectangular | NaN | ['/m/04rlf', '/m/02jjt', '/m/02jjt'] | 2 | UCZf__ehlCEBPop___sldpBUQ |
| 2 | EU8S-zxS9PI | 【悪質】偽物ヒカキン許さねぇ…注意してください！【なりすまし】 | ◆チャンネル登録はこちら↓\n\n\n\nhttp://www.youtube.com/us... | none | ['ヒカキン', 'ヒカキンtv', 'hikakintv', 'hikakin', 'ひか... | 2018-06-27T08:38:55.000Z | https://i.ytimg.com/vi/EU8S-zxS9PI/default.jpg | 1724625.0 | 33038.0 | 0 | ... | 11504.0 | False | hd | 2d | PT6M12S | rectangular | NaN | ['/m/04rlf', '/m/02jjt', '/m/02jjt'] | 3 | UCZf__ehlCEBPop___sldpBUQ |
| 3 | 5wnfkIfw0jE | ツイッターのヒカキンシンメトリーBotが面白すぎて爆笑www | ◆チャンネル登録はこちら↓\n\n\n\nhttp://www.youtube.com/us... | none | ['ヒカキン', 'ヒカキンtv', 'hikakintv', 'hikakin', 'ひか... | 2018-06-25T07:46:07.000Z | https://i.ytimg.com/vi/5wnfkIfw0jE/default.jpg | 1109029.0 | 25986.0 | 0 | ... | 6852.0 | False | hd | 2d | PT6M31S | rectangular | NaN | ['/m/04rlf', '/m/02jjt', '/m/02jjt'] | 4 | UCZf__ehlCEBPop___sldpBUQ |
| 4 | -6duBsde_XM | 【放送事故】酒飲みながら東海オンエア×ヒカキンで質問コーナーやったらヤバかったwww | 提供：モンスターストライク\n\n\n\n▼キャンペーンサイトはこちら\n\n\n\nhtt... | none | ['ヒカキン', 'ヒカキンtv', 'hikakintv', 'hikakin', 'ひか... | 2018-06-21T08:00:00.000Z | https://i.ytimg.com/vi/-6duBsde_XM/default.jpg | 1759797.0 | 33923.0 | 0 | ... | 4517.0 | False | hd | 2d | PT27M7S | rectangular | NaN | ['/m/098wr', '/m/019_rr', '/m/02wbm', '/m/019_... | 5 | UCZf__ehlCEBPop___sldpBUQ |
5 rows × 21 columns
# column 'viewCount''
df['viewCount'].sort_values()
| viewCount | |
|---|---|
| 24426 | 0.0 |
| 14789 | 0.0 |
| 16400 | 0.0 |
| 12752 | 0.0 |
| 47228 | 0.0 |
| ... | ... |
| 45854 | NaN |
| 45856 | NaN |
| 45860 | NaN |
| 45864 | NaN |
| 45865 | NaN |
66289 rows × 1 columns
“NaN” = Not a Numberの略で直訳すると「数字ではない」。データセットでは欠落しており、データフレームとして読み込むと NaN として保存されることが多い。必要に応じて別途対応が必要。例えばこれらを説明変数として扱うと想定外の動作となることがある。
# drop samples including NaN & 0 on 'viewCount'
print('orig_num = ', len(df))
print('num of NaN = ', df['viewCount'].isna().sum())
df = df[df['viewCount'].notnull()]
print('after_num = ', len(df))
df = df[df['viewCount'] != 0]
print('after_num2 = ', len(df))
orig_num = 66289
num of NaN = 10
after_num = 66279
after_num2 = 66231
df['viewCount'].describe()
| viewCount | |
|---|---|
| count | 6.623100e+04 |
| mean | 4.548834e+05 |
| std | 1.328530e+06 |
| min | 2.000000e+00 |
| 25% | 3.919150e+04 |
| 50% | 1.215850e+05 |
| 75% | 3.515195e+05 |
| max | 8.664236e+07 |
# 1枚目: リニアスケール
fig1 = go.Figure()
fig1.add_trace(go.Histogram(x=df['viewCount'], nbinsx=100))
fig1.update_xaxes(title_text='ViewCount')
fig1.update_yaxes(title_text='Frequency')
fig1.update_layout(
title='histgram of viewCount (linear)',
height=500, width=700
)
fig1.show()
# 2枚目: ログスケール
fig2 = go.Figure()
fig2.add_trace(go.Histogram(x=df['viewCount'], nbinsx=100))
fig2.update_xaxes(title_text='ViewCount')
fig2.update_yaxes(title_text='Frequency (log)', type='log')
fig2.update_layout(
title='histgram of viewCount (log)',
height=500, width=700
)
fig2.show()
14.3. 数値データに対する前処理の例#
14.3.1. 手法1：バイナリ化(binarization)#
# preprocess method 1: binarization by mean value
THRESHOLD = df['viewCount'].mean()
new_column = df['viewCount'] > THRESHOLD
new_column = np.where(new_column == True, 1, 0)
temp = pd.DataFrame(df['viewCount'])
temp['binary'] = new_column
temp.head()
| viewCount | binary | |
|---|---|---|
| 0 | 2244205.0 | 1 |
| 1 | 1869268.0 | 1 |
| 2 | 1724625.0 | 1 |
| 3 | 1109029.0 | 1 |
| 4 | 1759797.0 | 1 |
14.3.2. 手法2：アドホックな離散化(ad-hoc discretization)#
# preprocess method 2: discretization 1, ad-hoc division
floor = 10000
new_column = np.floor_divide(df['viewCount'], floor)
temp['discret_floor'] = new_column
temp.head()
| viewCount | binary | discret_floor | |
|---|---|---|---|
| 0 | 2244205.0 | 1 | 224.0 |
| 1 | 1869268.0 | 1 | 186.0 |
| 2 | 1724625.0 | 1 | 172.0 |
| 3 | 1109029.0 | 1 | 110.0 |
| 4 | 1759797.0 | 1 | 175.0 |
# default (logスケール)
fig3 = go.Figure()
fig3.add_trace(go.Histogram(x=temp['viewCount'], nbinsx=100))
fig3.update_xaxes(title_text='viewCounts')
fig3.update_yaxes(title_text='Freqency (log)', type='log')
fig3.update_layout(
title='default',
height=500, width=700
)
fig3.show()
# discret_floor (logスケール)
fig4 = go.Figure()
fig4.add_trace(go.Histogram(x=temp['discret_floor'], nbinsx=100))
fig4.update_xaxes(title_text='discret_floor')
fig4.update_yaxes(title_text='Freqency (log)', type='log')
fig4.update_layout(
title='discret_floor',
height=500, width=700
)
fig4.show()
14.3.3. 手法3：統計的な離散化#
# preprocess method 3: discretization 2, quantilzation
discret_num = 4
ranges = np.linspace(0, 1, discret_num)
data = df['viewCount'].quantile(ranges)
data
| viewCount | |
|---|---|
| 0.000000 | 2.000000e+00 |
| 0.333333 | 5.913133e+04 |
| 0.666667 | 2.408843e+05 |
| 1.000000 | 8.664236e+07 |
new_column = pd.qcut(df['viewCount'], discret_num, labels=False)
temp['discret_quantile'] = new_column
temp
| viewCount | binary | discret_floor | discret_quantile | |
|---|---|---|---|---|
| 0 | 2244205.0 | 1 | 224.0 | 3 |
| 1 | 1869268.0 | 1 | 186.0 | 3 |
| 2 | 1724625.0 | 1 | 172.0 | 3 |
| 3 | 1109029.0 | 1 | 110.0 | 3 |
| 4 | 1759797.0 | 1 | 175.0 | 3 |
| ... | ... | ... | ... | ... |
| 66284 | 131489.0 | 0 | 13.0 | 2 |
| 66285 | 13271.0 | 0 | 1.0 | 0 |
| 66286 | 76266.0 | 0 | 7.0 | 1 |
| 66287 | 282447.0 | 0 | 28.0 | 2 |
| 66288 | 6900.0 | 0 | 0.0 | 0 |
66231 rows × 4 columns
# default (logスケール)
fig5 = go.Figure()
fig5.add_trace(go.Histogram(x=temp['viewCount'], nbinsx=100))
fig5.update_xaxes(title_text='viewCounts')
fig5.update_yaxes(title_text='Freqency (log)', type='log')
fig5.update_layout(
title='default',
height=500, width=700
)
fig5.show()
# discret_quantile (logスケール)
fig6 = go.Figure()
fig6.add_trace(go.Histogram(x=temp['discret_quantile'], nbinsx=discret_num*2))
fig6.update_xaxes(title_text='viewCounts (discret_quantile={})'.format(discret_num))
fig6.update_yaxes(title_text='Freqency (log)', type='log')
fig6.update_layout(
title='discret_quantile',
height=500, width=700
)
fig6.show()
14.3.4. 手法4：ログスケール化(log-scaling)#
# preprocess method 4: log-scaling
new_column = np.log10(df['viewCount'] + 1)
temp['log10'] = new_column
temp.head()
| viewCount | binary | discret_floor | discret_quantile | log10 | |
|---|---|---|---|---|---|
| 0 | 2244205.0 | 1 | 224.0 | 3 | 6.351063 |
| 1 | 1869268.0 | 1 | 186.0 | 3 | 6.271672 |
| 2 | 1724625.0 | 1 | 172.0 | 3 | 6.236695 |
| 3 | 1109029.0 | 1 | 110.0 | 3 | 6.044943 |
| 4 | 1759797.0 | 1 | 175.0 | 3 | 6.245463 |
14.3.5. デフォルトとログスケールとの比較#
# default
fig7 = go.Figure()
fig7.add_trace(go.Histogram(x=temp['viewCount'], nbinsx=100))
fig7.update_xaxes(title_text='viewCounts')
fig7.update_yaxes(title_text='Freqency')
fig7.update_layout(
title='default',
height=500, width=700
)
fig7.show()
# log10
fig8 = go.Figure()
fig8.add_trace(go.Histogram(x=temp['log10'], nbinsx=100))
fig8.update_xaxes(title_text='viewCounts')
fig8.update_yaxes(title_text='Freqency')
fig8.update_layout(
title='log10',
height=500, width=700
)
fig8.show()
14.3.6. 手法5：標準化(standardization)#
from sklearn import preprocessing
data = np.array(df['viewCount'].values, dtype='float64')
data = data.reshape(len(data), 1)
new_column = preprocessing.scale(data)
temp['standardization'] = new_column
temp.head()
| viewCount | binary | discret_floor | discret_quantile | log10 | standardization | |
|---|---|---|---|---|---|---|
| 0 | 2244205.0 | 1 | 224.0 | 3 | 6.351063 | 1.346854 |
| 1 | 1869268.0 | 1 | 186.0 | 3 | 6.271672 | 1.064632 |
| 2 | 1724625.0 | 1 | 172.0 | 3 | 6.236695 | 0.955757 |
| 3 | 1109029.0 | 1 | 110.0 | 3 | 6.044943 | 0.492387 |
| 4 | 1759797.0 | 1 | 175.0 | 3 | 6.245463 | 0.982231 |
mean = np.mean(new_column)
var = np.var(new_column)
print('mean = ', mean, ', var = ', var)
temp['standardization'].describe()
mean = 1.2873900181367037e-17 , var = 1.0
| standardization | |
|---|---|
| count | 6.623100e+04 |
| mean | 1.287390e-17 |
| std | 1.000008e+00 |
| min | -3.423971e-01 |
| 25% | -3.128985e-01 |
| 50% | -2.508795e-01 |
| 75% | -7.780379e-02 |
| max | 6.487481e+01 |
# default (logスケール)
fig9 = go.Figure()
fig9.add_trace(go.Histogram(x=temp['viewCount'], nbinsx=100))
fig9.update_yaxes(type='log')
fig9.update_xaxes(title_text='viewCounts')
fig9.update_yaxes(title_text='Freqency')
fig9.update_layout(
title='default',
height=500, width=700
)
fig9.show()