不多说，直接上干货！
现有一份数据如下。
下载日志数据并分析
到搜狗实验室下载用户查询日志
1) 介绍
搜索引擎查询日志库设计为包括约1个月(2008年6月)Sogou搜索引擎部分网页查询需求及用户点击情况的网页查询日志数据集合。为进行中文搜索引擎用户行为分析的研究者提供基准研究语料。
2) 格式说明
数据格式为:访问时间\t用户ID\t[查询词]\t该URL在返回结果中的排名\t用户点击的顺序号\t用户点击的URL
其中，用户ID是根据用户使用浏览器访问搜索引擎时的Cookie信息自动赋值，即同一次使用浏览器输入的不同查询对应同一个用户ID。
[kfk@bigdata-pro01 datas]$ more weblogs.log 00:00:00 2982199073774412 [360安全卫士] 8 3 download.it.com.cn/softweb/software/firewall/antivirus/20067/17938.html 00:00:00 07594220010824798 [哄抢救灾物资] 1 1 news.21cn.com/social/daqian/2008/05/29/4777194_1.shtml 00:00:00 5228056822071097 [75810部队] 14 5 www.greatoo.com/greatoo_cn/list.asp?link_id=276&title=%BE%DE%C2%D6%D0%C2%CE%C5 00:00:00 6140463203615646 [绳艺] 62 36 www.jd-cd.com/jd_opus/xx/200607/706.html 00:00:00 8561366108033201 [汶川地震原因] 3 2 www.big38.net/ 00:00:00 23908140386148713 [莫衷一是的意思] 1 2 www.chinabaike.com/article/81/82/110/2007/2007020724490.html 00:00:00 1797943298449139 [星梦缘全集在线观看] 8 5 www.6wei.net/dianshiju/????\xa1\xe9|????do=index 00:00:00 00717725924582846 [闪字吧] 1 2 www.shanziba.com/ 00:00:00 41416219018952116 [霍震霆与朱玲玲照片] 2 6 bbs.gouzai.cn/thread-698736.html 00:00:00 9975666857142764 [电脑创业] 2 2 ks.cn.yahoo.com/question/1307120203719.html 00:00:00 21603374619077448 [111aa图片] 1 6 www.fotolog.com.cn/tags/aa111 00:00:00 7423866288265172 [豆腐的制成] 3 13 ks.cn.yahoo.com/question/1406051201894.html 00:00:00 0616877776407358 [tudou.com+禁播电影] 2 9 topic.bindou.com/1487/ 00:00:00 3933365481995287 [最佳受孕时间] 6 3 ks.cn.yahoo.com/question/1407051001276.html 00:00:00 8242389147671512 [捷克民歌土风舞++教案] 2 3 shwamlys.blog.sohu.com/76558184.html 00:00:00 8248403977107859 [pdf] 1 1 download.it168.com/18/1805/13947/13947_3.shtml 00:00:00 6239728533154822 [喷嘭乐团lonely+day] 7 9 www.songtaste.com/song/266563/ 00:00:00 6551182914925117 [尼康相机报价] 6 4 product.it168.com/list/b/03050171_1.shtml 00:00:00 2345161545038265 [哄抢救灾物资] 2 1 pic.news.mop.com/gs/2008/0528/12985.shtml 00:00:00 06478452674857782 [1000] 4 1 www.1000dy.cn/ 00:00:00 23405445793591878 [剑荡天下+++展鹏] 4 5 74.53.27.3/~xx4ucom/bbs/simple/index.php?t31744.html 00:00:00 23457845712802688 [人妖] 4 1 play.zol.com.cn/detail/1659_1.html 00:00:00 4625224675315291 [印尼排华是怎么回事] 2 6 zhidao.baidu.com/question/47218514 00:00:00 048731119953599966 [东莞市塘厦新泰塑胶五金厂] 10 44 www.hi-b2b.com/product/17w47204/index.shtml 00:00:00 48530765688455246 [胡其美] 26 3 www.gotostreet.com/bbs/ShowPost.Asp?ID=121
将文件中的tab更换成逗号
cat weblog.log|tr "\t" "," > weblog2.log
[kfk@bigdata-pro01 datas]$ cat weblog.log|tr "\t" "," > weblog2.log [kfk@bigdata-pro01 datas]$ more weblog2.log ?00:00:00,2982199073774412,[360安全卫士],8 3,download.it.com.cn/softweb/software/firewall/antivirus/20067/17938.html 00:00:00,07594220010824798,[哄抢救灾物资],1 1,news.21cn.com/social/daqian/2008/05/29/4777194_1.shtml 00:00:00,5228056822071097,[75810部队],14 5,www.greatoo.com/greatoo_cn/list.asp?link_id=276&title=%BE%DE%C2%D6%D0%C2%CE%C5 00:00:00,6140463203615646,[绳艺],62 36,www.jd-cd.com/jd_opus/xx/200607/706.html 00:00:00,8561366108033201,[汶川地震原因],3 2,www.big38.net/ 00:00:00,23908140386148713,[莫衷一是的意思],1 2,www.chinabaike.com/article/81/82/110/2007/2007020724490.html 00:00:00,1797943298449139,[星梦缘全集在线观看],8 5,www.6wei.net/dianshiju/????\xa1\xe9|????do=index 00:00:00,00717725924582846,[闪字吧],1 2,www.shanziba.com/ 00:00:00,41416219018952116,[霍震霆与朱玲玲照片],2 6,bbs.gouzai.cn/thread-698736.html 00:00:00,9975666857142764,[电脑创业],2 2,ks.cn.yahoo.com/question/1307120203719.html 00:00:00,21603374619077448,[111aa图片],1 6,www.fotolog.com.cn/tags/aa111 00:00:00,7423866288265172,[豆腐的制成],3 13,ks.cn.yahoo.com/question/1406051201894.html 00:00:00,0616877776407358,[tudou.com+禁播电影],2 9,topic.bindou.com/1487/ 00:00:00,3933365481995287,[最佳受孕时间],6 3,ks.cn.yahoo.com/question/1407051001276.html 00:00:00,8242389147671512,[捷克民歌土风舞++教案],2 3,shwamlys.blog.sohu.com/76558184.html 00:00:00,8248403977107859,[pdf],1 1,download.it168.com/18/1805/13947/13947_3.shtml 00:00:00,6239728533154822,[喷嘭乐团lonely+day],7 9,www.songtaste.com/song/266563/ 00:00:00,6551182914925117,[尼康相机报价],6 4,product.it168.com/list/b/03050171_1.shtml 00:00:00,2345161545038265,[哄抢救灾物资],2 1,pic.news.mop.com/gs/2008/0528/12985.shtml 00:00:00,06478452674857782,[1000],4 1,www.1000dy.cn/ 00:00:00,23405445793591878,[剑荡天下+++展鹏],4 5,74.53.27.3/~xx4ucom/bbs/simple/index.php?t31744.html 00:00:00,23457845712802688,[人妖],4 1,play.zol.com.cn/detail/1659_1.html 00:00:00,4625224675315291,[印尼排华是怎么回事],2 6,zhidao.baidu.com/question/47218514 00:00:00,048731119953599966,[东莞市塘厦新泰塑胶五金厂],10 44,www.hi-b2b.com/product/17w47204/index.shtml 00:00:00,48530765688455246,[胡其美],26 3,www.gotostreet.com/bbs/ShowPost.Asp?ID=121
