应领导要求，针对公司业务添加squid 缓存。减轻前端Nginx的压力。简单的草图如下，squid和Nginx部署在同一台服务器上面，用户发送请求后，先通过Nginx前端进行处理，如果有数据直接返回给用户，如果请求的数据不存在，则转发给squid代理服务器，由squid向后端仓库服务器获取数据，最终在本地也缓存一份文件，再把数据返回给用户。
测试环境架构：
192.168.199.150 cdre1.xiazai.com cdre2.xiazai.com Nginx+Squid
192.168.199.131 www.ceshi.com 仓库(虚拟主机域名cdre1.ceshi.com,cdre2.ceshi.com,cdl1.ceshi.com)
192.168.199.1 本地客户端（用户）
在Squid服务器上添加域名解析/etc/hosts
#vim /etc/hosts ##此处一定要一个IP 对应一个域名，否则会出现解析错误！ 192.168.199.150 cdre1.ceshi.com
本地客户端添加域名解析：
C:\Windows\System32\drivers\etc\hosts 192.168.199.131 cdre2.ceshi.com 192.168.199.131 cdl1.ceshi.com (本篇文章暂不配置此主机) 192.168.199.131 cdre1.ceshi.com 192.168.199.150 cdre1.xiazai.com 192.168.199.150 cdre2.xiazai.com
安装并配置Nginx
1.下载Nginx源码包(192.168.199.150)
#yum install -y pcre pcre-devel zlib* #cd /tmp #wget http://nginx.org/download/nginx-1.6.3.tar.gz #tar -zxf nginx-1.6.3.tar.gz #cd nginx-1.6.3 #./configure --prefix=/usr/local/nginx --with-pcre #make && make install
2.配置nginx.conf以及虚拟主机文件
[root@luoji ~]# vim /usr/local/nginx/conf/nginx.conf
user nobody nobody;
worker_processes 1;
error_log /usr/local/nginx/logs/nginx_error.log crit;
pid /usr/local/nginx/logs/nginx.pid;
worker_rlimit_nofile 51200;
events
{
use epoll;
worker_connections 6000;
}
http
{
include mime.types;
default_type application/octet-stream;
server_names_hash_bucket_size 3526;
server_names_hash_max_size 4096;
log_format combined_realip '$remote_addr $http_x_forwarded_for [$time_local]'
'$host "$request_uri" $status'
'"$http_referer" "$http_user_agent"';
sendfile on;
tcp_nopush on;
keepalive_timeout 30;
client_header_timeout 3m;
client_body_timeout 3m;
send_timeout 3m;
connection_pool_size 256;
client_header_buffer_size 1k;
large_client_header_buffers 8 4k;
request_pool_size 4k;
output_buffers 4 32k;
postpone_output 1460;
client_max_body_size 10m;
client_body_buffer_size 256k;
client_body_temp_path /usr/local/nginx/client_body_temp;
proxy_temp_path /usr/local/nginx/proxy_temp;
fastcgi_temp_path /usr/local/nginx/fastcgi_temp;
fastcgi_intercept_errors on;
tcp_nodelay on;
gzip on;
gzip_min_length 1k;
gzip_buffers 4 8k;
gzip_comp_level 5;
gzip_http_version 1.1;
gzip_types text/plain application/x-javascript text/css text/htm application/xml;
include vhost/*.conf;
}以上配置文件可自行定义配置，最后一行添加了虚拟主机配置文件
3.建立虚拟目录及虚拟主机文件xiazai.conf
#mkdir /usr/local/nginx/conf/vhost
#cd vhost
#vim xiazai.conf
server {
listen 80;
server_name www.xiazai.com;
root /data/cdre1/;
access_log /dev/null;
error_log /dev/null;
location / {
if ( !-e $request_filename ) {
rewrite ^(.*)$ http://www.xiazai.com:81/$1; #增加跳转，将本地Nginx没有的请求路径转发到代理地址
}
}
}4.启动Nginx服务
#/usr/local/nginx/sbin/nginx -t ##检查配置文件 #/usr/local/nginx/sbin/nginx ##启动服务 #ps aux| grep nginx root 4584 0.0 0.1 25232 1812 ? Ss Dec29 0:00 nginx: master process /usr/local/nginx/sbin/nginx nobody 5217 0.0 0.1 25232 1648 ? S 11:52 0:00 nginx: worker process root 5269 0.0 0.0 103248 876 pts/0 S+ 14:48 0:00 grep nginx #netstat -alntp|grep :80 tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 4584/nginx
安装并配置Squid
依然在Nginx服务器上部署（192.168.199.150）
此处Squid是做反向代理加速web访问的。至于正向代理和透明代理，此处就不说了。
1.下面是我写的自动化安装脚本，如有不足之处请各位指出.
#!/bin/bash
#install squid
yum install -y wget gcc gcc-c++ net-snmp net-snmp-utils libtool lsof
check_ok() {
if [ $? != 0 ]
then
echo "it's error.please check the log"
exit 1
fi
}
ulimit -HSn 65536
echo "1024 40000" > /proc/sys/net/ipv4/ip_local_port_range
echo "ulimit -HSn 65536" >> /etc/rc.d/rc.local
echo "echo "1024 40000" > /proc/sys/net/ipv4/ip_local_port_range" >> /etc/rc.d/rc.local
groupadd squid -g 23
useradd -u 23 -g squid -s /sbin/nologin squid
cd /tmp
[ -f squid-3.5.12.tar.gz ] || wget http://www.squid-cache.org/Versions/v3/3.5/squid-3.5.12.tar.gz
tar -zxf squid-3.5.12.tar.gz
check_ok
cd squid-3.5.12
./configure --prefix=/usr/local/squid --enable-epoll --enable-htcp --enable-stacktraces --enable-storeio=ufs,aufs,diskd --enable-removal-policies=lru,heap --enable-icmp --enable-default-err-language=Simplify_Chinese --enable-err-languages="Simplify_Chinese English" --enable-cache-digests --enable-auth --enable-auth-basic="NCSA" --enable-useragent-log --enable-referer-log --enable-linux-netfilter --enable-delay-pools --enable-follow-x-forwarded-for --enable-kill-parent-hack --enable-gnuregex --enable-underscore --enable-arp-acl --enable-x-accelerator-vary --disable-ident-lookups --disable-ssl --disable-wccp --disable-internal-dns --disable-mempools --with-default-user=squid --with-pthreads --with-aio --with-large-files --with-filedescriptors=65535 --enable-snm
check_ok
make && make install
check_ok
ln -s /usr/local/squid/etc/squid.conf /etc/squid.conf
echo "######-----------------------it's the end-------------------------------####"
sleep 3
echo "Install is OK,Please config the file"2.上面的脚本是源码编译安装。下面开始配置squid.conf文件,我上面脚本中将配置文件软连接到了/etc/squid.conf处，所以直接编辑此文件就可以了。
#vim /etc/squid.conf
#
# Recommended minimum configuration:
#
http_port 81 accel vhost vport
cache_peer 192.168.199.131 parent 81 0 no-query no-digest originserver #name=abc
#cache_peer_domain abc www.123.com
cache_dir aufs /data/cache 5000 16 256
#设置用户及组
cache_effective_user squid
cache_effective_group squid
visible_hostname abc.com
#cache_peer_access abc allow all
# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl myip src 192.168.199.150
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl purge method PURGE
acl SSL_ports port 443
acl Safe_ports port 80 81 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
acl QUERY urlpath_regex cgi-bin \? \.php \.html
cache deny QUERY
#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
http_access deny !Safe_ports
http_access allow purge myip
# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports
# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager
# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost
#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#
# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost
# And finally deny all other access to this proxy
http_access allow all
# Squid normally listens to port 3128
#http_port 3128
# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /usr/local/squid/var/cache/squid 100 16 256
# Leave coredumps in the first cache dir
coredump_dir /usr/local/squid/var/cache/squid
#
# Add any of your own refresh_pattern entries above these.
#
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
refresh_pattern -i \.jpg$ 1440 50% 2880 ignore-reload
refresh_pattern -i \.png$ 1440 50% 2880 ignore-reload
refresh_pattern -i \.bmp$ 1440 50% 2880 ignore-reload
refresh_pattern -i \.gif$ 1440 50% 2880 ignore-reload
refresh_pattern -i \.jpeg$ 1440 50% 2880 ignore-reload
refresh_pattern -i \.swf$ 1440 50% 2880 ignore-reload
refresh_pattern -i \.js$ 1440 50% 2880 ignore-reload
refresh_pattern -i \.css$ 1440 50% 2880 ignore-reload
refresh_pattern -i \.zip$ 1440 50% 2880 ignore-reload
refresh_pattern -i \.exe$ 1440 50% 2880 ignore-reload
refresh_pattern -i \.rar$ 1440 50% 2880 ignore-reload
logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
cache_log /usr/local/squid/var/logs/cache.log
access_log /usr/local/squid/var/logs/access.log combined
pid_filename /usr/local/squid/var/logs/squid.pid
#日志轮询
logfile_rotate 12
maximum_object_size_in_memory 512 KB #可调节
minimum_object_size 1 MB #可调节
maximum_object_size 2500 MB #可调节
cache_mem 512 MB #可调节
memory_replacement_policy lru
cache_store_log none
#neicunchi 可调节
memory_pools on
memory_pools_limit 1024 MB
cache_mgr cache@ccc.com #邮箱可设置
via on
#显示客户端真实ip
forwarded_for on
log_icp_queries on #关闭icp队列在访问日志中
httpd_suppress_version_string off #禁止squid出错页面综合在网页最下方显示主机相关信息和squid版本信息
ipcache_size 1024
ipcache_low 90
ipcache_high 95
cache_swap_low 90
cache_swap_high 95
quick_abort_min -1 KB
quick_abort_max 32 KB
quick_abort_pct 95
#timeout
#peer_connect_timeout 20 seconds
#connect_timeout 20 seconds
#connect_timeout 1 minutes
#negative_ttl 0 minutes
#read_timeout 60 seconds
#read_timeout 15 minutes
#request_timeout 20 seconds
#read_timeout 15 minutes
collapsed_forwarding on
shutdown_lifetime 5 seconds
#关闭该项，就可以显示用户的整个请求内容
strip_query_terms off
#snmp
#snmp_port 3401
#acl snmppublic snmp_community show
#snmp_access allow snmppublic localhost
#snmp_access deny all
vary_ignore_expire on3.建立缓存目录并给予Squid属性
#mkdir -p /data/cache #chown -R squid.squid /data/cache #chown -R squid.squid /usr/local/squid/var
4.启动Squid服务
# /usr/local/squid/sbin/squid -z ##初始化缓存目录 # /usr/local/squid/sbin/squid ##启动服务 # /usr/local/squid/sbin/squid -k check ##检查配置文件是否有误 # /usr/local/squid/sbin/squid -krec ##如有修改，重新加载配置文件 # ps aux| grep squid root 5151 0.0 0.2 37688 2348 ? Ss 11:39 0:00 /usr/local/squid/sbin/squid squid 5153 1.5 1.1 47552 11948 ? Sl 11:39 2:58 (squid-1) root 5275 0.0 0.0 103248 876 pts/0 S+ 14:50 0:00 grep squid # netstat -alntp|grep :81 ##由于80端口已经被Nginx使用，所以此处定义Squid代理的端口为81 tcp 0 0 :::81 :::* LISTEN 5153/(squid-1)
5.squid常见用法：
-z 第一次时使用，初始化缓存目录
-k [参数]
check 检查配置文件
kill 杀掉进程，停止服务(不建议使用此命令关掉服务)
shutdown 最安全的停止服务，终止进程
清理缓存的方法：
$squid_dir/bin/squidclient -p PORT -m PURGE http://URL缓存地址
查看缓存命中率以及其他缓存信息：
squidclient-p80mgr:info#取得squid运行状态信息squidclient-p80mgr:mem#取得squid内存使用情况squidclient-p80mgr:objects#取得squid已经缓存的列表,使用时要小心，可能会造成squid崩溃：squidclient-p80mgr:diskd#取得squid的磁盘使用情况squidclient-p80 -m PURGE http://www.361way.com#强制更新某个url,即清理某连接的squid
仓库Nginx配置
（192.168.199.131）
1.下载安装与上面Nginx的安装方法相同。
2.配置虚拟主机
由于是数据仓库，所以数据都存储在此服务器，因此有对应的多个虚拟主机,此处虚拟主机的监听端口也设置成81与Squid代理的端口号保持一致。
#cd /usr/loca/nginx/conf/vhost
#vim ceshi.conf
server {
listen 81;
server_name cdl2.ceshi.com;
root /data/cdl2/;
access_log /dev/null;
error_log /dev/null;
}
server {
listen 81;
server_name cdre1.ceshi.com;
root /data/yxdown/pcgame/cdre1/;
access_log /dev/null;
error_log /dev/null;
}
server {
listen 81;
server_name cdre2.ceshi.com;
root /data/cdre2/;
access_log /dev/null;
error_log /dev/null;
}3.启动服务
# /usr/local/nginx/sbin/nginx -t # /usr/local/nginx/sbin/nginx # netstat -alntp | grep 81 tcp 0 0 0.0.0.0:81 0.0.0.0:* LISTEN 1112/nginx
客户端测试
1.在Nginx 前端添加测试文件以及后端仓库放置同样的文件
###192.168.199.150
# cd /data/cdre1
# touch {a,b,c,d}
# touch {1.rar,2.rar}
##192.168.199.131
# cd /data/cdre1
# touch {a,b,c,d,e} #此处多一个 e 文件
# touch {1.rar,2.rar,3.rar} #此处多一个 3.rar 文件2.本地192.168.199.1通过浏览器访问:
2.1 第一种情况也就是用户直接访问Nginx 返回200正确码，表示Nginx有数据返回。
http://cdre1.xiazai.com/a
2.2第二种情况当请求的文件Nginx前端没有，则通过Squid代理仓库下载所需文件，本地并缓存一份文件(忽略中的（1）测试文件下载多遍了)。
再看实际下载地址，已经变成了代理的仓库虚拟主机的地址，这表明代理成功了。
再查看本地缓存目录大小是否有变化：
# du -sh /data/cache #初始化目录大小 17M /data/cache # du -sh /data/cache/ #生成缓存后的大小，大小刚好是3.rar的大小 1.5G /data/cache/
至此结束！
以上配置实现了单台文件下载服务器的代理需求，如若要在一台下载服务器上面部署两个虚拟主机用于下载，则在Nginx端多配置一个cdre2的虚拟主机，rewrite则跳转到cdre2.ceshi.com:81.具体参照cdre1的配置修改即可实现！
折腾了差不多一周，终于实现了效果，当然此配置还有待完善的地方，如隐藏仓库的域名地址以及端口等，后续研究。
如果文中有错误之处欢迎各位指出和纠正。
本文由 Mr Gu 创作，采用 知识共享署名4.0 国际许可协议进行许可
本站文章除注明转载/出处外，均为本站原创或翻译，转载前请务必署名
最后编辑时间为: Aug 26, 2016 at 09:24 pm