{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "887b09b8",
   "metadata": {},
   "source": [
    "# B2 非官方参考答案\n",
    "\n",
    "本实现已使用项目附件实际运行。原始框架保持不变；运行时需在同一目录放入对应 CSV，或者保留本阅读版的 answers/materials 目录结构。指标阈值与方案目标不代表官方评分标准。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "72d2f8a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "运行摘要\n",
      "{\n",
      "  \"valid_rows\": 5400,\n",
      "  \"quarantined_rows\": 0,\n",
      "  \"observed_dates\": 30,\n",
      "  \"train_rows\": 4500,\n",
      "  \"test_rows\": 900,\n",
      "  \"quality_train_rows\": 3600,\n",
      "  \"warmup_rows_removed\": 900,\n",
      "  \"train_last_date\": \"2024-05-25\",\n",
      "  \"test_first_date\": \"2024-05-26\",\n",
      "  \"index_ids_with_multiple_sectors\": 20,\n",
      "  \"weekend_dates_in_sample\": 8,\n",
      "  \"ks_comparisons\": 46\n",
      "}\n",
      "\n",
      "时段特征（前 6 行）\n",
      "index_id  date_only time_period  avg_pct_change\n",
      "    IDX1 2024-05-01   afternoon       -0.786667\n",
      "    IDX1 2024-05-01     morning       -0.126667\n",
      "    IDX1 2024-05-01        noon       -0.430000\n",
      "    IDX1 2024-05-02   afternoon        1.540000\n",
      "    IDX1 2024-05-02     morning       -0.720000\n",
      "    IDX1 2024-05-02        noon       -0.096667\n",
      "\n",
      "板块×阶段均值及上涨概率\n",
      "sector market_phase  avg_pct_change  up_probability\n",
      "    医疗           熊市       -1.695341        0.054496\n",
      "    医疗           牛市        0.926667        0.834646\n",
      "    医疗           震荡       -0.024521        0.457534\n",
      "    教育           熊市       -1.824103        0.028490\n",
      "    教育           牛市        1.034500        0.853125\n",
      "    教育           震荡        0.061534        0.482955\n",
      "    科技           熊市       -1.796554        0.052219\n",
      "    科技           牛市        1.039945        0.836066\n",
      "    科技           震荡        0.089425        0.526027\n",
      "    能源           熊市       -1.733196        0.027548\n",
      "    能源           牛市        0.996237        0.857527\n",
      "    能源           震荡        0.028291        0.490196\n",
      "    金融           熊市       -1.869209        0.031073\n",
      "    金融           牛市        0.984986        0.817439\n",
      "    金融           震荡       -0.039228        0.504451\n",
      "\n",
      "板块占比\n",
      "sector  overall_share  train_share  test_share\n",
      "    医疗       0.206111     0.208222    0.195556\n",
      "    教育       0.189444     0.189556    0.188889\n",
      "    科技       0.206296     0.202889    0.223333\n",
      "    能源       0.202222     0.200889    0.208889\n",
      "    金融       0.195926     0.198444    0.183333\n",
      "\n",
      "训练／测试 KS 分组结果\n",
      "   comparison          group_type   group  n_reference  n_test  ks_stat  p_value            note\n",
      "train_vs_test              sector      医疗          937     176 0.054259 0.755265 时间自相关与多重检验需额外处理\n",
      "train_vs_test              sector      教育          853     170 0.096959 0.129616 时间自相关与多重检验需额外处理\n",
      "train_vs_test              sector      科技          913     201 0.073134 0.324767 时间自相关与多重检验需额外处理\n",
      "train_vs_test              sector      能源          904     188 0.058911 0.629400 时间自相关与多重检验需额外处理\n",
      "train_vs_test              sector      金融          893     165 0.032794 0.997234 时间自相关与多重检验需额外处理\n",
      "train_vs_test        market_phase      熊市         1493     325 0.065238 0.197056 时间自相关与多重检验需额外处理\n",
      "train_vs_test        market_phase      牛市         1526     280 0.045151 0.701946 时间自相关与多重检验需额外处理\n",
      "train_vs_test        market_phase      震荡         1481     295 0.065723 0.228212 时间自相关与多重检验需额外处理\n",
      "train_vs_test sector+market_phase 医疗 / 熊市          306      61 0.111861 0.510298 时间自相关与多重检验需额外处理\n",
      "train_vs_test sector+market_phase 医疗 / 牛市          326      55 0.128054 0.390892 时间自相关与多重检验需额外处理\n",
      "train_vs_test sector+market_phase 医疗 / 震荡          305      60 0.094809 0.724012 时间自相关与多重检验需额外处理\n",
      "train_vs_test sector+market_phase 教育 / 熊市          288      63 0.203869 0.022626 时间自相关与多重检验需额外处理\n",
      "train_vs_test sector+market_phase 教育 / 牛市          266      54 0.085770 0.867031 时间自相关与多重检验需额外处理\n",
      "train_vs_test sector+market_phase 教育 / 震荡          299      53 0.151006 0.231796 时间自相关与多重检验需额外处理\n",
      "train_vs_test sector+market_phase 科技 / 熊市          310      73 0.095272 0.623328 时间自相关与多重检验需额外处理\n",
      "train_vs_test sector+market_phase 科技 / 牛市          309      57 0.119400 0.465002 时间自相关与多重检验需额外处理\n",
      "train_vs_test sector+market_phase 科技 / 震荡          294      71 0.097873 0.610745 时间自相关与多重检验需额外处理\n",
      "train_vs_test sector+market_phase 能源 / 熊市          295      68 0.072034 0.918096 时间自相关与多重检验需额外处理\n",
      "train_vs_test sector+market_phase 能源 / 牛市          307      65 0.068755 0.945103 时间自相关与多重检验需额外处理\n",
      "train_vs_test sector+market_phase 能源 / 震荡          302      55 0.120289 0.468577 时间自相关与多重检验需额外处理\n",
      "train_vs_test sector+market_phase 金融 / 熊市          294      60 0.137415 0.275544 时间自相关与多重检验需额外处理\n",
      "train_vs_test sector+market_phase 金融 / 牛市          318      49 0.073675 0.963851 时间自相关与多重检验需额外处理\n",
      "train_vs_test sector+market_phase 金融 / 震荡          281      56 0.156202 0.181354 时间自相关与多重检验需额外处理\n"
     ]
    }
   ],
   "source": [
    "\"\"\"B2 非官方参考实现：描述统计、过去日期特征、日期级划分与分组检验。\"\"\"\n",
    "from pathlib import Path\n",
    "import json\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy.stats import ks_2samp\n",
    "from sklearn.model_selection import TimeSeriesSplit\n",
    "\n",
    "BASE=Path(__file__).resolve().parent if '__file__' in globals() else Path.cwd()\n",
    "OUT=BASE/'B2_output';OUT.mkdir(exist_ok=True)\n",
    "path=next((p for p in [Path.cwd()/'B2.csv',BASE/'B2.csv',BASE.parent/'materials/B2.csv'] if p.is_file()),None)\n",
    "if path is None:raise FileNotFoundError('请将 B2.csv 与参考代码放在同一目录')\n",
    "df=pd.read_csv(path)\n",
    "df['datetime']=pd.to_datetime(df['date']+' '+df['time'],errors='coerce')\n",
    "df['pct_change']=pd.to_numeric(df['pct_change'],errors='coerce')\n",
    "bad=df[['datetime','pct_change','index_id','sector','market_phase']].isna().any(axis=1) | (df['pct_change']<=-100)\n",
    "quarantined=int(bad.sum());df.loc[bad].to_csv(OUT/'quarantined.csv',index=False)\n",
    "df=df.loc[~bad].copy()\n",
    "if df.duplicated(['index_id','datetime']).any():raise ValueError('存在同指数同时间重复记录，应确认后处理')\n",
    "df=df.sort_values(['datetime','index_id']).reset_index(drop=True)\n",
    "df['date_only']=df['datetime'].dt.normalize()\n",
    "df['hour']=df['datetime'].dt.hour\n",
    "df['time_period']=np.select([df['hour'].between(9,10),df['hour'].between(11,13)],['morning','noon'],default='afternoon')\n",
    "df['up']=(df['pct_change']>0).astype(int)\n",
    "\n",
    "# 原题要求的描述统计；不可将全量均值当作实时预测特征。\n",
    "period_index_change=df.groupby(['index_id','date_only','time_period'],as_index=False).agg(avg_pct_change=('pct_change','mean'))\n",
    "sector_phase_stats=df.groupby(['sector','market_phase'],as_index=False).agg(avg_pct_change=('pct_change','mean'),up_probability=('up','mean'))\n",
    "\n",
    "# 日收益按每个时点相对上一时点的收益复合。样例按记录日期排序；\n",
    "# 真实市场必须先核验交易日历、指数定义及百分比字段口径。\n",
    "daily=df.groupby(['index_id','date_only'],as_index=False).agg(daily_return=('pct_change',lambda s:((1+s/100).prod()-1)*100))\n",
    "daily=daily.sort_values(['index_id','date_only']).reset_index(drop=True)\n",
    "daily['trend_5']=daily.groupby('index_id')['daily_return'].transform(\n",
    "    lambda s:s.shift(1).rolling(5,min_periods=5).apply(lambda r:((1+r/100).prod()-1)*100,raw=True))\n",
    "df=df.merge(daily[['index_id','date_only','trend_5']],on=['index_id','date_only'],how='left',validate='many_to_one')\n",
    "index_sector_trend=df.groupby(['index_id','sector'],as_index=False).agg(avg_trend_5=('trend_5','mean'))\n",
    "\n",
    "def historical_mean_by_day(frame,keys,value,new_name):\n",
    "    \"\"\"同一天的记录均只读取更早日期，避免同时间跨指数数据泄漏。\"\"\"\n",
    "    g=frame.groupby(keys+['date_only'],as_index=False).agg(total=(value,'sum'),n=(value,'count'))\n",
    "    g=g.sort_values(keys+['date_only']).reset_index(drop=True)\n",
    "    previous_sum=g.groupby(keys)['total'].transform(lambda s:s.cumsum().shift(1))\n",
    "    previous_count=g.groupby(keys)['n'].transform(lambda s:s.cumsum().shift(1))\n",
    "    g[new_name]=previous_sum/previous_count\n",
    "    return frame.merge(g[keys+['date_only',new_name]],on=keys+['date_only'],how='left',validate='many_to_one')\n",
    "\n",
    "df=historical_mean_by_day(df,['index_id','time_period'],'pct_change','hist_period_mean')\n",
    "df=historical_mean_by_day(df,['sector','market_phase'],'pct_change','hist_sector_phase_mean')\n",
    "\n",
    "# 对日期进行 TimeSeriesSplit，同一天的所有观测一起划分。\n",
    "dates=np.sort(df['date_only'].unique())\n",
    "if len(dates)<6:raise ValueError('日期太少，无法使用 5 折时序划分')\n",
    "splits=list(TimeSeriesSplit(n_splits=5).split(dates))\n",
    "train_idx,test_idx=splits[-1]\n",
    "train_data=df[df['date_only'].isin(dates[train_idx])].copy()\n",
    "test_data=df[df['date_only'].isin(dates[test_idx])].copy()\n",
    "assert train_data['datetime'].max()<test_data['datetime'].min()\n",
    "assert set(train_data['date_only']).isdisjoint(set(test_data['date_only']))\n",
    "\n",
    "# 此实现假设逐日更新历史特征：测试日只能读取此前已完成日的数据，\n",
    "# 不更新模型参数。若预测整个未来窗口，应在预测起点冻结所有特征。\n",
    "# market_phase 还需在预测时已知；若是事后标注，应滞后或用历史数据构造。\n",
    "features=['hist_period_mean','hist_sector_phase_mean','trend_5']\n",
    "high_quality_train=train_data.dropna(subset=features).copy()\n",
    "high_quality_train.to_csv(OUT/'high_quality_train_index.csv',index=False,encoding='utf-8-sig')\n",
    "test_data.to_csv(OUT/'golden_test_index.csv',index=False,encoding='utf-8-sig')\n",
    "period_index_change.to_csv(OUT/'period_index_change.csv',index=False,encoding='utf-8-sig')\n",
    "sector_phase_stats.to_csv(OUT/'sector_phase_stats.csv',index=False,encoding='utf-8-sig')\n",
    "index_sector_trend.to_csv(OUT/'index_sector_trend.csv',index=False,encoding='utf-8-sig')\n",
    "\n",
    "def group_ks(reference,test,keys,label):\n",
    "    rows=[]\n",
    "    for key,g in test.groupby(keys):\n",
    "        values=key if isinstance(key,tuple) else (key,)\n",
    "        ref=reference\n",
    "        for field,value in zip(keys,values):ref=ref[ref[field]==value]\n",
    "        row={'comparison':label,'group_type':'+'.join(keys),'group':' / '.join(map(str,values)),\n",
    "             'n_reference':len(ref),'n_test':len(g)}\n",
    "        if len(ref)>=2 and len(g)>=2:\n",
    "            stat,pvalue=ks_2samp(ref['pct_change'],g['pct_change'],method='asymp')\n",
    "            row.update(ks_stat=float(stat),p_value=float(pvalue),note='时间自相关与多重检验需额外处理')\n",
    "        else:row.update(ks_stat=np.nan,p_value=np.nan,note='样本不足')\n",
    "        rows.append(row)\n",
    "    return pd.DataFrame(rows)\n",
    "\n",
    "ks_tables=[]\n",
    "for keys in [['sector'],['market_phase'],['sector','market_phase']]:\n",
    "    ks_tables.append(group_ks(train_data,test_data,keys,'train_vs_test'))\n",
    "    # 按原题提供全量对测试集的描述性比较。样本重叠，不作独立检验结论。\n",
    "    overlap=group_ks(df,test_data,keys,'overall_vs_test_descriptive')\n",
    "    overlap['note']='整体包含测试样本，p 值仅描述，不作独立推断'\n",
    "    ks_tables.append(overlap)\n",
    "ks_results=pd.concat(ks_tables,ignore_index=True)\n",
    "ks_results.to_csv(OUT/'ks_results.csv',index=False,encoding='utf-8-sig')\n",
    "shares=pd.DataFrame({'overall_share':df['sector'].value_counts(normalize=True),\n",
    "                    'train_share':train_data['sector'].value_counts(normalize=True),\n",
    "                    'test_share':test_data['sector'].value_counts(normalize=True)}).fillna(0).reset_index(names='sector')\n",
    "sector_changes=int((df.groupby('index_id')['sector'].nunique()>1).sum())\n",
    "weekend_dates=int(pd.Series(dates).dt.dayofweek.ge(5).sum())\n",
    "summary={'valid_rows':len(df),'quarantined_rows':quarantined,'observed_dates':len(dates),\n",
    "         'train_rows':len(train_data),'test_rows':len(test_data),\n",
    "         'quality_train_rows':len(high_quality_train),'warmup_rows_removed':len(train_data)-len(high_quality_train),\n",
    "         'train_last_date':str(train_data['date_only'].max().date()),'test_first_date':str(test_data['date_only'].min().date()),\n",
    "         'index_ids_with_multiple_sectors':sector_changes,'weekend_dates_in_sample':weekend_dates,\n",
    "         'ks_comparisons':len(ks_results)}\n",
    "(OUT/'summary.json').write_text(json.dumps(summary,ensure_ascii=False,indent=2),encoding='utf-8')\n",
    "print('运行摘要\\n'+json.dumps(summary,ensure_ascii=False,indent=2))\n",
    "print('\\n时段特征（前 6 行）\\n'+period_index_change.head(6).to_string(index=False))\n",
    "print('\\n板块×阶段均值及上涨概率\\n'+sector_phase_stats.to_string(index=False))\n",
    "print('\\n板块占比\\n'+shares.to_string(index=False))\n",
    "print('\\n训练／测试 KS 分组结果\\n'+ks_results[ks_results['comparison']=='train_vs_test'].to_string(index=False))\n",
    "\n",
    "def tbl(frame):return '<div class=\"table\">'+frame.to_html(index=False,border=0,float_format=lambda x:f'{x:.4f}')+'</div>'\n",
    "body='<h1>B2 参考代码运行结果</h1><p>非官方参考实现；由项目附件实际运行。未训练预测模型，不含模型准确率。</p>'\n",
    "body+='<h2>划分与质量</h2>'+tbl(pd.DataFrame(list(summary.items()),columns=['项目','实测值']))\n",
    "body+='<p>样例日期含周末，同一指数在样例中有多个板块标签。此处保留数据并按观测日期演示，真实市场需先核验交易日历及板块标签。</p>'\n",
    "body+='<h2>指数×日期×时段均值（示例）</h2>'+tbl(period_index_change.head(12))\n",
    "body+='<h2>板块×市场阶段均值</h2>'+tbl(sector_phase_stats)\n",
    "body+='<h2>指数×板块历史五日趋势（示例）</h2>'+tbl(index_sector_trend.head(12))\n",
    "body+='<h2>板块占比</h2>'+tbl(shares)\n",
    "body+='<h2>训练与测试分组 KS</h2>'+tbl(ks_results[ks_results['comparison']=='train_vs_test'])\n",
    "body+='<p>p≥0.05 不证明同分布。时序观测并非完全独立，多组检验也有多重比较问题；应结合效应量、样本量和按日期分块验证。</p>'\n",
    "body+='<h2>整体与测试描述性 KS（原题要求）</h2>'+tbl(ks_results[ks_results['comparison']=='overall_vs_test_descriptive'])\n",
    "body+='<p>整体包含测试样本，因此上述 p 值不能作为独立两样本推断。不要据此宣称测试集绝对代表整体，也不要通过混入未来样本来提高 p 值。</p>'\n",
    "body+='<h2>特征使用边界</h2><p>全量描述均值单独导出。预测特征只包含更早日期信息，五日趋势排除当日。前期历史不足的训练行被排除；逐日评估时允许读入此前已发生的数据，固定多日预测则必须冻结预测起点可见数据。</p>'\n",
    "body+='<p>market_phase 必须在预测时可观测。事后划定的牛熊阶段不能直接作为预测输入，应滞后或用历史规则构造；选择实际模型输入时要用明确的特征白名单，不能把原始结果列或未来标签全部送入模型。</p>'\n",
    "style='body{font:17px/1.8 system-ui,\"Microsoft YaHei\",sans-serif;color:#17263b;max-width:1000px;margin:auto;padding:20px}table{border-collapse:collapse;width:100%}td,th{border-bottom:1px solid #dce4ee;padding:10px;text-align:left}.table{overflow:auto}h1{font-size:1.6rem}h2{font-size:1.25rem}'\n",
    "(BASE/'B2_reference.html').write_text('<!doctype html><html lang=\"zh-CN\"><meta charset=\"utf-8\"><meta name=\"viewport\" content=\"width=device-width,initial-scale=1\"><title>B2 参考运行结果</title><style>'+style+'</style><body><a href=\"../index.html#b2\">返回学习资料</a>'+body+'</body></html>',encoding='utf-8')\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.13.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}