stafwag Blog

staf wagemakers blog

Using Squid to Cache FreeBSD Packages

PKGNG config

I manage a few FreeBSD jails behind a squid proxy. pkgng is configured to use the proxy:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
root@rataplan:/root # cat /etc/pkg/FreeBSD.conf 
# $FreeBSD: releng/10.1/etc/pkg/FreeBSD.conf 263938 2014-03-30 15:29:54Z bdrewery $
#
# To disable this repository, instead of modifying or removing this file,
# create a /usr/local/etc/pkg/repos/FreeBSD.conf file:
#
#   mkdir -p /usr/local/etc/pkg/repos
#   echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf
#

pkg_env: {

        http_proxy: "http://xxx.xxx.xxx.xxx:3128"

}

FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}
root@rataplan:/root # 

SQUID config

Recompile

The squid proxy doesn’t cache to the FreeBSD packages. The squid pkgng package is compiled with “LAX_HTTP Do not enforce strict HTTP compliance” option disabled. which doesn’t allow you to override the cache headers sent by the remote site.

In order to cache the FreeBSD packages we need to recompile squid with “LAX_HTTPD” enabled.

Updating the ports

Physical system

If you use a physical FreeBSD system as your proxy run the “portsnap fetch” command.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@rataplan ~]# portsnap fetch
Looking up portsnap.FreeBSD.org mirrors... 7 mirrors found.
Fetching snapshot tag from ec2-eu-west-1.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Mon Jun 22 14:30:21 CEST 2015 to Tue Jun 23 08:39:41 CEST 2015.
Fetching 4 metadata patches... done.
Applying metadata patches... done.
Fetching 0 metadata files... done.
Fetching 417 patches. 
(417/417) 100.00%  done.                                       
done.
Applying patches... 
done.
Fetching 3 new ports or files... done.
[root@rataplan ~]# 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@rataplan ~]# portsnap extract
/usr/ports/.arcconfig
/usr/ports/.gitignore
/usr/ports/CHANGES
/usr/ports/CONTRIBUTING.md
/usr/ports/COPYRIGHT
/usr/ports/GIDs

<snip>

/usr/ports/x11/zenity/
Building new INDEX files... done.
[root@rataplan ~]# 

Jail

If you use an ezjail as your proxy run the “ezjail-admin update -P” command.

Build

Stop SQUID
1
2
3
root@stafproxy:/usr/ports/www/squid # /usr/local/etc/rc.d/squid stop
squid not running? (check /var/run/squid/squid.pid).
root@stafproxy:/usr/ports/www/squid # 
Make config
1
2
3
root@stafproxy:/usr/ports/www/squid # cd
root@stafproxy:/root # cd /usr/ports/www/squid
root@stafproxy:/usr/ports/www/squid # make config
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

       ┌─────────────────────────────── squid-3.5.5 ──────────────────────────────────┐
       │ ┌──────────────────────────────────────────────────────────────────────────┐ │  
       │ │ [ ] ARP_ACL         ARP/MAC/EUI based authentification                   │ │  
       │ │ [ ] AUTH_LDAP       Install LDAP authentication helpers                  │ │  
       │ │ [x] AUTH_NIS        Install NIS/YP authentication helpers                │ │  
       │ │ [ ] AUTH_SASL       Install SASL authentication helpers                  │ │  
       │ │ [ ] AUTH_SMB        Install SMB auth. helpers (req. Samba)               │ │  
       │ │ [ ] AUTH_SQL        Install SQL based auth (uses MySQL)                  │ │  
       │ │ [ ] CACHE_DIGESTS   Use cache digests                                    │ │  
       │ │ [ ] DEBUG           Build with extended debugging support                │ │  
       │ │ [ ] DELAY_POOLS     Delay pools (bandwidth limiting)                     │ │  
       │ │ [x] DOCS            Build and/or install documentation                   │ │  
       │ │ [ ] ECAP            Loadable content adaptation modules                  │ │  
       │ │ [ ] ESI             ESI support                                          │ │  
       │ │ [x] EXAMPLES        Build and/or install examples                        │ │  
       │ │ [ ] FOLLOW_XFF      Support for the X-Following-For header               │ │  
       │ │ [x] FS_AUFS         AUFS (threaded-io) support                           │ │  
       │ │ [x] FS_DISKD        DISKD storage engine controlled by separate service  │ │  
       │ │ [ ] FS_ROCK         ROCK storage engine                                  │ │  
       │ │ [x] HTCP            HTCP support                                         │ │  
       │ │ [ ] ICAP            the ICAP client                                      │ │  
       │ │ [ ] ICMP            ICMP pinging and network measurement                 │ │  
       │ │ [x] IDENT           Ident lookups (RFC 931)                              │ │  
       │ │ [x] IPV6            IPv6 protocol support                                │ │  
       │ │ [x] KQUEUE          Kqueue(2) support                                    │ │  
       │ │ [ ] LARGEFILE       Support large (>2GB) cache and log files             │ │  
       │ │ [x] LAX_HTTP        Do not enforce strict HTTP compliance                │ │  
       │ │ [ ] NETTLE          Nettle MD5 algorithm support                         │ │  
       │ │ [x] SNMP            SNMP support                                         │ │  
       │ │ [ ] SSL             SSL gatewaying support                               │ │  
       │ │ [ ] SSL_CRTD        Use ssl_crtd to handle SSL cert requests             │ │  
       │ │ [ ] STACKTRACES     Enable automatic backtraces on fatal errors          │ │  
       │ │ [ ] TP_IPF          Transparent proxying with IPFilter                   │ │  
       │ │ [ ] TP_IPFW         Transparent proxying with IPFW                       │ │  
       │ │ [ ] TP_PF           Transparent proxying with PF                         │ │  
       │ │ [ ] VIA_DB          Forward/Via database                                 │ │  
       │ └─────v(+)─────────────────────────────────────────────────────────82%─────┘ │  
       ├──────────────────────────────────────────────────────────────────────────────┤  
       │                       <  OK  >            <Cancel>                           │  
       └──────────────────────────────────────────────────────────────────────────────┘  
                                                                                         
Make install
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
root@stafproxy:/usr/ports/www/squid # make
===>  License GPLv2 accepted by the user
===>  Found saved configuration for squid-3.5.5
===>   squid-3.5.5 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by squid-3.5.5 for building
===>  Extracting for squid-3.5.5
=> SHA256 Checksum OK for squid3.5/squid-3.5.5.tar.xz.

<snip>

Making install in test-suite
install  -m 0644 /var/ports/basejail/usr/ports/www/squid/work/squid-3.5.5/helpers/basic_auth/DB/passwd.sql  /var/ports/basejail/usr/ports/www/squid/work/stage/usr/local/share/examples/squid
(cd /var/ports/basejail/usr/ports/www/squid/work/squid-3.5.5 && install  -m 0644 QUICKSTART README RELEASENOTES.html doc/debug-sections.txt /var/ports/basejail/usr/ports/www/squid/work/stage/usr/local/share/doc/squid)
/bin/mkdir -p /var/ports/basejail/usr/ports/www/squid/work/stage/var/squid/logs
/bin/rmdir /var/ports/basejail/usr/ports/www/squid/work/stage/var/run/squid
====> Compressing man pages (compress-man)
===> Staging rc.d startup script(s)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
root@stafproxy:/usr/ports/www/squid # make install clean
===>  Installing for squid-3.5.5                                                                                                                                                                                                             
===>   squid-3.5.5 depends on file: /usr/local/bin/perl5.20.2 - found                                                                                                                                                                        
===>  Checking if squid already installed                                                                                                                                                                                                    
===>   Registering installation for squid-3.5.5                                                                                                                                                                                              

<snip>

===> SECURITY REPORT: 
      This port has installed the following files which may act as network
      servers and may therefore pose a remote security risk to the system.
/usr/local/libexec/squid/basic_radius_auth
/usr/local/sbin/squid

      This port has installed the following startup scripts which may cause
      these network services to be started at boot time.
/usr/local/etc/rc.d/squid

      If there are vulnerabilities in these programs there may be a security
      risk to the system. FreeBSD makes no guarantee about the security of
      ports included in the Ports Collection. Please type 'make deinstall'
      to deinstall the port if this is a concern.

      For more information, and contact details about the security
      status of this software, see the following webpage: 
http://www.squid-cache.org/
pkg lock

Lock the squid package to prevent the upgrade from pkgng tree.

1
2
3
4
root@stafproxy:/usr/ports/www/squid # pkg lock squid
squid-3.5.5: lock this package? [y/N]: y
Locking squid-3.5.5
root@stafproxy:/usr/ports/www/squid #

View the locked pkgng packages

1
2
3
4
root@stafproxy:/usr/ports/www/squid # pkg lock -l
Currently locked packages:
squid-3.5.5
root@stafproxy:/usr/ports/www/squid # 

SQUID config

Update squid.conf

Edit the squid config:

1
2
root@stafproxy:/usr/ports/www/squid # cd /usr/local/etc/squid/
root@stafproxy:/usr/local/etc/squid # vi squid.conf

Add a “refresh_pattern” for “pkgmir.pkg.freebsd.org”:

1
2
3
4
5
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^http://pkgmir.pkg.freebsd.org/.*\.txz          1440    100%    10080 ignore-private ignore-must-revalidate override-expire ignore-no-cache
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320

Start squid

1
2
3
root@stafproxy:/usr/local/etc/squid # ../rc.d/squid start
Starting squid.
root@stafproxy:/usr/local/etc/squid # 

rc.conf

Make sure that the system is configured to start squid during the system startup.

1
2
3
4
5
6
7
8
root@stafproxy:/usr/local/etc/squid # cat /etc/rc.conf 
#
# squid
#

squid_enable="YES"

root@stafproxy:/usr/local/etc/squid # 

SQUID should cache the pkgng downloads now.

Have fun

Comments