Configuration of a Libre Hosting aka Announcing our public IPs to the Internets from an any place place¶

In technical terms: IPSec between OPNSense and OpenBSD to announce our ASN IP from an any place ISP place.

General presentation¶

Who we are¶

Aquilenet is a non profit organization since 2010 and a "do it yourself ISP", member of a Federation of similar ISP in France called FFDN. We are netneutrality builders, helping for more freedom and building networks using and participating to Libre Software.
We provide xDSL, VPN and we hope soon Fiber access, and a lot of services to our members (mail, nextcloud, hosting, VPS...) and others searx, Etherpad, Pastebin, Peertube, ...

The need: to create a Libre format hosting at our cool place, "la mezzanine".¶

To allow members of our non-profit organization to locate the hardware they want (like Nuc, Raspberry pi, towers, etc...) at our Libre hosting space called "la mezzanine" with a Public IPv6 and IPv4 from our ASN, we need to announce our IPs from another place than our data center.

How we do it¶

We have to tunnel all of our Libre Hosting networks to the Internets, and vice versa. This IP range, assigned to our ASN, is routed from our point of BGP announce in our data center, to the Libre Hosting.

We first tried OpenVPN but its userland application consumed too much resources for the required bandwidth (200Mbps). We use IPSec.

From our Libre Hosting we have an OPNSense Firewall and in our data center, two clustered OpenBSD.

A diagram to illustrate this:

Diagram explanation:

• (1) ASN & BGP

With our ASN with can do BGP to announce our IP addresses in our data center.

• (2) Firewalls in the data center

The job is done with a cluster of firewalls using the ®magic power© of OpenBSD and OpenBGPD.
For firewall redundancy we use CARP and pfsync.

• (3) La Mezzanine

La Mezzanine is our cool place where we meet up with other members and friends.
It has green power, thanks to the folks at a French electric utility cooperative company, Enercoop, who supplies only renewable energy.
Here, the firewall is a dedicated hardware running OPNSense, than can afford filters & IPSec on 500Mbps. We chose an appliance with AES-NI support, a Qotom-Q355G4 Fanless X86 with an Intel® Core™ i5-5250U processor, coming from Shenzen. It also deals with the PPPoE connection on a general public fiber provider (yes--we had to hack a little to remove this ISP blackbox).

• (4) Public IPv6 and v4 from the Mezzanine to Internet

This firewall is routing, through an IPSec tunnel (for each OpenBSD firewall), our public IP address range, dedicated to this place. These IP addresses allow for our friends to locate their servers / Arduino's & co right here, on the Internet within our network <3

---

Technical information, let's rock !¶

We configure IPSec with 2 × Phase-2 tunnels: one for IPv4, another for IPv6:

• Phase-1: IKEv1 AES (192 bits) + SHA1 + DH Group 14 (2048 bits)
• Phase-2: ESP IPv4 tunnel AES (256 bits) SHA512 Group 18 (8192 bits)
• Phase-2: ESP IPv6 tunnel AES (256 bits) SHA512 Group 18 (8192 bits)

OpenBSD configuration¶

We define OpenBSD's IPSec configuration file:

/etc/ipsec.conf

ike passive esp from any to $LIBRE_HOSTING_PUB_IP_V4/26 \
    peer $OPSense_Public_IP\
    main group modp2048\
    psk "My super secret shared key"

flow esp from any to $LIBRE_HOSTING_PUB_IP_V4/26 peer $OPSense_Public_IP
flow esp from any to $LIBRE_HOSTING_PUB_IP_V6/48 peer $OPSense_Public_IP

chmod 500 /etc/ipsec.conf

Launch the tunnel:

isakmpd -K  
ipsecctl -f /etc/ipsec.conf

To make it permanent add this to /etc/rc.conf.local:

isakmpd_flags="-K"
ipsec_rules=/etc/ipsec.conf
ipsec=YES

To verify the configuration:

ipsecctl -sa 

Enable some logging on Isakmpd:

sh -c "echo S \> /var/run/isakmpd.fifo"  
less /var/run/isakmpd.result

and:

isakmpd -d -DA=70 -K

Check IPSec flows:

ipsecctl -F

OPNSense configuration¶

• IPSec Phase-1

• IPSec Phase-2, IPv4

• IPSec Phase-2, IPv6

To get an IPv6 Phase-2 working with a IPv4 Phase-1 on Opnsense, better make sure you got this patch

• IPSec Advanced Settings

Important! On the LAN side, exclude the traffic targeted at the routed network from the tunnel, so all of its hosts may keep talking and responding to each other (example: ARP/ICMP request to the gateway)! Otherwise that traffic will go die at the other end of the IPSec tunnel (and the whole Planet could be endangered, too. Who knows?).

Hosted server settings¶

IP: The public IP/Mask
Gateway: OPNSense IP
mtu 1378

---

Addendum 20180825: FAILOVER IPSEC Between OpenBSD with SASyncd¶

https://www.packetmischief.ca/openbsd-ipsec-tunnel-guide/

OpenBSD has built-in support for deploying redundant IPsec gateways. Synchronization of SAs and flows is handled by the sasyncd(8) daemon.

• sasyncd.conf:

# track the external carp interface
interface carp100
# address of gateway #1b
peer 192.168.0.2
# 256 bit shared key (openssl rand -hex 32)
sharedkey da23169c9f49bb1fcdfab7d5565889754331b3f831b3723d273a06e9d780f2cc