通过redis扩展分布式存储fastdfs的数据对应及方案

本文涉及的产品
云数据库 Redis 版,社区版 2GB
推荐场景:
搭建游戏排行榜
简介:

前言:

   周天被朋友拖住了,他说最近要实现一个文件分享的网站,一个是图片,一个是特定的文件,比如压缩包、pdf这类的。 他是做idc的,手里的资源还是不少的, 想通过文件分享网站来导入点所谓的流量。哥年轻时跟他搞过一个php的淘宝图片站,结果有个卖衣服的站点,流量很大,成本太高了,再说也没钱再次投入了,也就放弃 ~   没赚多少钱,倒是赔给那几个淘宝卖家100多块钱。  想起往事,还算有趣的 ~

现在的我以前不是当初菜鸟的我了,google了半天,从别地方,拔下来一个html模板,而且还是很高端的样子,后端是用pyton的tornado框架改了改,跑了几个实例都是没有问题的 ~


实现的方法简单的

上传

建立表单,文件上传,然后写如到tmp目录的下,调用自己写的python fastdfs模块上传文件。也可以放到目录后,用inotify 来做文件的监控,触发文件后,就上传到fastdfs分布式存储里面。

下载

可以通过fastdfs本身的web做下载连接,但是为了更高的性能,推荐使用nginx fastdfs模块来实现的下载,给朋友也是这么做的。


(还没有正式的上线,嘿嘿)

subprocess 调用系统的fastdfs客户端

1
上传文件:/usr/local/bin/fdfs_upload_file  <config_file> <local_filename>


tornado的文件上传api

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
def post(self):
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
     upload_path=os.path.join(os.path.dirname(__file__), 'files' )  #文件的暂存路径
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
     file_metas=self.request.files[ 'file' ]    #提取表单中‘name’为‘file’的文件元数据
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
     for  meta  in  file_metas:
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
         filename=meta[ 'filename' ]
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
         filepath=os.path.join(upload_path,filename)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
         with  open(filepath, 'wb' as  up:      #有些文件需要已二进制的形式存储,实际中可以更改
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
             up.write(meta[ 'body' ])
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
         self.write( 'finished!' )
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
app=tornado.web.Application([
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
     (r '/file' ,UploadFileHandler),
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
])


wKiom1LJmfKi7TBIAAE4agppxMA452.jpg


一般来说,上传后得到的文件名字,是一串hash,和以前的名字比起来不太一样。


     举个例子,我明明上传的文件时 xiaorui.jpg,结果fastdfs接口给我返回的是 一堆的hash ,这是在让人有点小郁闷。

    我这里还只是图片的应用,后期文件功能出来后,让人看到的都是hash,你都不能通过hash想象出他的文件。

wKioL1LJogihA8GBAAA1TDtMvdE176.jpg


查看了uc那边的技术博客,扩展了一套简单的路子。


这边用到的是   nginx tornado  jquery bootstrap


1.   应用系统在上传文件到FastDFS成功时将原始文件名和“文件索引(FID)”保存下来,我这里会记录在redis里面。

2.  在浏览器下载的时用Nginx的域名和FID拼出url,在url后面增加一个参数,指定原始文件名。例如: xxx.xx.xx.xx/group2/M00/00/89/eQ6h3FKJf_PRl8p4AUz4wO8tqaA688.apk?attname=xiaorui.cc


3. 在Nginx上进行如下配置,这样Nginx就会截获url中的参数attname,在Http响应头里面加上字段 Content-Disposition “attachment;filename=$arg_attname”。



1
2
3
4
5
6
7
8
9
10
11
12
13
14
location /group1/M00 {
root /data/store/data1;
if  ($arg_attname ~  "^(.*).cc" ) {
     add_header Content-Disposition  "attachment;filename=$arg_attname" ;
}
ngx_fastdfs_module;
}
location /group2/M00 {
root /data/store/data2;
if  ($arg_attname ~  "^(.*).cc" ) {
     add_header Content-Disposition  "attachment;filename=$arg_attname" ;
}
ngx_fastdfs_module;
}


4.   浏览器发现响应头里面有Content-Disposition “attachment;filename=$arg_attname”时,就会把文件名显示成filename指定的名称。


大家可以测试下,原来就是给用户的url多出一个get参数,用nginx识别到之后,插入到header头信息Content-Disposition 协议。


到半夜为止,和哥们一块已经实现了该项目的大部门网络框架,当然前端的nginx


是用kvm虚拟机做的,毕竟7层对io需求不是很大。针对nginx做的轮训是在代码


层面实现的。


wKiom1LJq5KzMMsmAAC8ES5PABE643.jpg



好了,就这样吧。 这fastdfs真是个好东西 ! 特别适合以文件为载体的在线服


务,如图片站,文件分享等等 !






 本文转自 rfyiamcool 51CTO博客,原文链接:http://blog.51cto.com/rfyiamcool/1348839,如需转载请自行联系原作者




相关实践学习
基于Redis实现在线游戏积分排行榜
本场景将介绍如何基于Redis数据库实现在线游戏中的游戏玩家积分排行榜功能。
云数据库 Redis 版使用教程
云数据库Redis版是兼容Redis协议标准的、提供持久化的内存数据库服务,基于高可靠双机热备架构及可无缝扩展的集群架构,满足高读写性能场景及容量需弹性变配的业务需求。 产品详情:https://www.aliyun.com/product/kvstore &nbsp; &nbsp; ------------------------------------------------------------------------- 阿里云数据库体验:数据库上云实战 开发者云会免费提供一台带自建MySQL的源数据库&nbsp;ECS 实例和一台目标数据库&nbsp;RDS实例。跟着指引,您可以一步步实现将ECS自建数据库迁移到目标数据库RDS。 点击下方链接,领取免费ECS&amp;RDS资源,30分钟完成数据库上云实战!https://developer.aliyun.com/adc/scenario/51eefbd1894e42f6bb9acacadd3f9121?spm=a2c6h.13788135.J_3257954370.9.4ba85f24utseFl
相关文章
|
6天前
|
负载均衡 监控 NoSQL
Redis的几种主要集群方案
【5月更文挑战第15天】Redis集群方案包括主从复制(基础,读写分离,手动故障恢复)、哨兵模式(自动高可用,自动故障转移)和Redis Cluster(官方分布式解决方案,自动分片、容错和扩展)。此外,还有Codis、Redisson和Twemproxy等工具用于代理分片和负载均衡。选择方案需考虑应用场景、数据量和并发需求,权衡可用性、性能和扩展性。
48 2
|
6天前
|
存储 监控 负载均衡
保证Redis的高可用性是一个涉及多个层面的任务,主要包括数据持久化、复制与故障转移、集群化部署等方面
【5月更文挑战第15天】保证Redis高可用性涉及数据持久化、复制与故障转移、集群化及优化策略。RDB和AOF是数据持久化方法,哨兵模式确保故障自动恢复。Redis Cluster实现分布式部署,提高负载均衡和容错性。其他措施包括身份认证、多线程、数据压缩和监控报警,以增强安全性和稳定性。通过综合配置与监控,可确保Redis服务的高效、可靠运行。
31 2
|
6天前
|
存储 监控 NoSQL
Redis处理大量数据主要依赖于其内存存储结构、高效的数据结构和算法,以及一系列的优化策略
【5月更文挑战第15天】Redis处理大量数据依赖内存存储、高效数据结构和优化策略。选择合适的数据结构、利用批量操作减少网络开销、控制批量大小、使用Redis Cluster进行分布式存储、优化内存使用及监控调优是关键。通过这些方法,Redis能有效处理大量数据并保持高性能。
27 0
|
6天前
|
存储 监控 NoSQL
【Redis】分布式锁及其他常见问题
【Redis】分布式锁及其他常见问题
32 0
|
6天前
|
存储 NoSQL 关系型数据库
【Redis】Redis的特性和应用场景 · 数据类型 · 持久化 · 数据淘汰 · 事务 · 多机部署
【Redis】Redis的特性和应用场景 · 数据类型 · 持久化 · 数据淘汰 · 事务 · 多机部署
16 0
|
6天前
|
NoSQL Java Redis
【Redis】Redis实现分布式锁
【Redis】Redis实现分布式锁
8 0
|
6天前
|
NoSQL Redis 数据库
Redis实现数据持久性主要依赖两种机制
【5月更文挑战第15天】Redis持久化包括RDB快照和AOF日志。RDB通过定时内存数据快照生成文件,恢复速度快但可能丢失部分数据;AOF记录每次写操作,实时性好但文件大、恢复慢。混合持久化兼顾两者优点,提供数据安全与性能平衡。用户可按需选择或组合使用策略。
8 2
|
6天前
|
存储 缓存 NoSQL
【技术分享】求取列表需求的redis缓存方案
【技术分享】求取列表需求的redis缓存方案
17 0
|
6天前
|
NoSQL API Redis
Redis源码、面试指南(3)数据对象类型编码(下)
Redis源码、面试指南(3)数据对象类型编码
11 1
|
6天前
|
存储 NoSQL API
Redis源码、面试指南(3)数据对象类型编码(上)
Redis源码、面试指南(3)数据对象类型编码
17 2