Aquilenet » Services

# Configuration of a Libre Hosting aka Announcing our public IPs to the Internets from an any place place

In technical terms: IPSec between OPNSense and OpenBSD to announce our ASN IP from an any place  ISP place.

## General presentation

### Who we are

[Aquilenet](https://www.aquilenet.fr) is a non profit organization since 2010 and a "do it yourself ISP", member of a Federation of similar ISP in France called [FFDN](https://www.ffdn.org). We are netneutrality builders, helping for more freedom and building networks using and participating to Libre Software.

We provide xDSL, VPN and we hope soon Fiber access, and a lot of services to our members (mail, nextcloud, hosting, VPS...) and others [searx](https://searx.aquilenet.fr), [Etherpad](https://pad.aquilenet.fr), [Pastebin](https://pastebin.aquilenet.fr), [Peertube](https://tube.aquilenet.fr), ...

### The need: to create a Libre format hosting at our cool place, "la mezzanine".

To allow members of our non-profit organization to locate the hardware they want (like Nuc, Raspberry pi, towers, etc...) at our Libre hosting space called "la mezzanine" with a Public IPv6 and IPv4 from our [ASN](https://en.wikipedia.org/wiki/Autonomous_system_(Internet)), we need to announce our IPs from another place than our data center.

### How we do it

We have to tunnel all of our Libre Hosting networks to the Internets, and vice versa. This IP range, assigned to our [ASN](https://en.wikipedia.org/wiki/Autonomous_system_(Internet)), is routed from our point of [BGP](https://en.wikipedia.org/wiki/Border_Gateway_Protocol) announce in our data center, to the Libre Hosting.

We first tried OpenVPN but its userland application consumed too much resources for the required bandwidth (200Mbps). We use IPSec.

From our Libre Hosting we have an [OPNSense](https://opnsense.org/) Firewall and in our data center, two clustered [OpenBSD](https://www.openbsd.org/).

A diagram to illustrate this:

![](https://atelier.aquilenet.fr/attachments/download/550/Aquilenet-IPSec-Logical_Scheme.png)

Diagram explanation:

* (1) [ASN](https://en.wikipedia.org/wiki/Autonomous_system_(Internet)) & [BGP](https://en.wikipedia.org/wiki/Border_Gateway_Protocol)

With our [ASN](https://en.wikipedia.org/wiki/Autonomous_system_(Internet)) with can do BGP to announce our IP addresses in our data center.

* (2) Firewalls in the data center

The job is done with a cluster of firewalls using the ®magic power© of [OpenBSD](https://www.openbsd.org/) and [OpenBGPD](http://www.openbgpd.org/).

For firewall redundancy we use [CARP and pfsync](https://www.openbsd.org/faq/pf/carp.html).

* (3) La Mezzanine

La Mezzanine is our cool place where we meet up with other members and friends.

It has green power, thanks to the folks at a French electric utility cooperative company, [Enercoop](https://en.wikipedia.org/wiki/Enercoop), who supplies only renewable energy.

Here, the firewall is a dedicated hardware running [OPNSense](https://opnsense.org/), than can afford filters & IPSec on 500Mbps. We chose an appliance with [AES-NI](https://en.wikipedia.org/wiki/AES_instruction_set) support, a Qotom-Q355G4 Fanless X86 with an Intel® Core™ i5-5250U processor, coming from Shenzen. It also deals with the PPPoE connection on a general public fiber provider (yes--we had to hack a little to remove this ISP blackbox).

* (4) Public IPv6 and v4 from the Mezzanine to Internet

This firewall is routing, through an IPSec tunnel (for each OpenBSD firewall), our public IP address range, dedicated to this place. These IP addresses allow for our friends to locate their servers / Arduino's & co right here, on the Internet within our network <3

---

## Technical information, let's rock !

We configure IPSec with **2** × Phase-2 tunnels: one for IPv4, another for IPv6:

* Phase-1: IKEv1 AES (192 bits) + SHA1 + DH Group 14 (2048 bits)

* Phase-2: ESP IPv4 tunnel AES (256 bits) SHA512 Group 18 (8192 bits)

* Phase-2: ESP IPv6 tunnel AES (256 bits) SHA512 Group 18 (8192 bits) 

### [OpenBSD](https://www.openbsd.org/) configuration

We define [OpenBSD](https://www.openbsd.org/)'s IPSec configuration file:

**/etc/ipsec.conf**

~~~

ike passive esp from any to $LIBRE_HOSTING_PUB_IP_V4/26 \

    peer $OPSense_Public_IP\

    main group modp2048\

    psk "My super secret shared key"

flow esp from any to $LIBRE_HOSTING_PUB_IP_V4/26 peer $OPSense_Public_IP

flow esp from any to $LIBRE_HOSTING_PUB_IP_V6/48 peer $OPSense_Public_IP

~~~

~~~

chmod 500 /etc/ipsec.conf

~~~

**Launch the tunnel:**

~~~

isakmpd -K  

ipsecctl -f /etc/ipsec.conf

~~~

To make it permanent add this to /etc/rc.conf.local:

~~~

isakmpd_flags="-K"

ipsec_rules=/etc/ipsec.conf

ipsec=YES

~~~

To verify the configuration:

~~~

ipsecctl -sa 

~~~

Enable some logging on Isakmpd:

~~~

sh -c "echo S \> /var/run/isakmpd.fifo"  

less /var/run/isakmpd.result

~~~

and:

~~~

isakmpd -d -DA=70 -K

~~~

Check IPSec flows:

~~~

ipsecctl -F

~~~

## [OPNSense](https://opnsense.org/) configuration

* **IPSec Phase-1**

![](https://atelier.aquilenet.fr/attachments/download/556/OPNSense-Tunnel_Settings_IPsec_VPN-Phase1.png)

* **IPSec Phase-2, IPv4**

![](https://atelier.aquilenet.fr/attachments/download/553/OPNSense-Tunnel_Settings_IPsec-Phase2-IPv4.png)

* **IPSec Phase-2, IPv6**

To get an IPv6 Phase-2 working with a IPv4 Phase-1 on Opnsense, better make sure you got [this patch](https://github.com/opnsense/core/commit/a79b20c12759007f2079ffd6e0cad26d04b00808)

![](https://atelier.aquilenet.fr/attachments/download/554/OPNSense-Tunnel_Settings_IPsec-Phase2-IPv6.png)

* **IPSec Advanced Settings**

**Important!** On the LAN side, exclude the traffic targeted at the routed network from the tunnel, so *all* of its hosts may keep talking and responding to each other (example: ARP/ICMP request to the gateway)! Otherwise that traffic will go die at the other end of the IPSec tunnel (and the whole Planet could be endangered, too. Who knows?).![](https://atelier.aquilenet.fr/attachments/download/555/OPNSense-Tunnel_Settings_IPsec-Advanced_Settings.png)

## Hosted server settings

IP: The public IP/Mask

Gateway: OPNSense IP

mtu 1378

---

## ADUM 20180825: FAILOVER IPSEC Between OpenBSD with SASyncd

https://www.packetmischief.ca/openbsd-ipsec-tunnel-guide/

OpenBSD has built-in support for deploying redundant IPsec gateways. Synchronization of SAs and flows is handled by the sasyncd(8) daemon.

* sasyncd.conf:

~~~

# track the external carp interface

interface carp100

# address of gateway #1b

peer 192.168.0.2

# 256 bit shared key (openssl rand -hex 32)

sharedkey da23169c9f49bb1fcdfab7d5565889754331b3f831b3723d273a06e9d780f2cc

~~~