This page currently just contains some notes I am writing as I document my journey into DN42. This page is quite disorganised and not very well written yet.

DN42 is a “big dynamic VPN” where participants can connect and experiment with core internet technologies such as BGP which I discovered in May 2021. I decided to give it a try and have documented my progress below.

The very first thing you need to do in order to connect to DN42 is register your details and which ASN and IP addresses you want. On the “real” internet, this would be a long process but on DN42 you simply edit some files in a git repo. Assuming there isn't overlap between your network and someone elses, they are quick to merge your changes into the main branch which makes your registration “official”.

On the 24th May 2021, my changes had been approved and merged. I registered AS4242421469, fd37:c7f3:5bf7::/48, and 172.23.13.8/29 which means I can do whatever I want with these because they are “mine”. Anything I do with them is only visible on the DN42 network so the real internet is safe from any misconfigurations I may make.

The registration instructions on the git repo are easy enough to follow. My biggest issues came from authenticating and signing changes on the git server. I opted to use SSH keys and the process was as follows:

An SSH (or GPG) key is required to authenticate with the git server. I made my SSH keys like so:

ssh-keygen -t ed25519 -C "[email protected]"

Visit the keys page within the git server settings: https://git.dn42.dev/user/settings/keys (register/login required). Click on “Add Key” in the SSH section and paste in your public key which you just generated (.pub file).

You need to configure your SSH/git client to use your new key which is generally done with the ssh-add /path/to/key command on a Linux CLI. You may need to start ssh-agent first (instructions)

My first issue was getting an authentication error when using my new key to interact with their git server. Apparently, all users are given certain permissions which are only set after their first sign in but it can take a while. I had to log out and log back in then wait about 10 minutes before my key was usable.

The guide explained almost everything well enough for me. I did also take a look at other people's pull requests (both approved and declined) to see how things should be done. However, the “maintainer object” has an “auth” parameter which I wasn't very confident with because most examples I saw were for PGP. Here's what I did:

Open your .pub key which was generated earlier and copy the text but stop before your email address; this goes in the “auth” parameter. For example, my maintainer object looks like this:

mntner:             JHEWITT-MNT
admin-c:            JHEWITT-DN42
tech-c:             JHEWITT-DN42
mnt-by:             JHEWITT-MNT
auth:               ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJEkwOzurBs4rJ8D4ud9cSYQxEe29zrjma/i0j/oeqrK
source:             DN42

Make sure you never paste your private key anywhere!

The git maintainers want everybody to sign their pull requests to ensure it is being done by an authorised user. This also means that you have to keep your keys backed up because you need them to make future changes.

For SSH signing, the process is different. The first step is to find the full hash of your commit (make sure you follow the instructions on the guide to squash multiple commits into one). I used the git log command which resulted in this output:

commit c0292556d7a1271ee0ffc06d1913fd7dff574821 (HEAD -> jhewitt_net-20210523/AS4242421469, origin/jhewitt_net-20210523/AS4242421469)
Author: Joseph Hewitt <[email protected]>
Date:   Sun May 23 20:34:26 2021 +0100

    registering for dn42

“c0292556d7a1271ee0ffc06d1913fd7dff574821” is the full hash in this situation. The guide tells me to use this command to sign the commit hash:

echo "<commit hash>" | ssh-keygen -Y sign -f <private key file> -n dn42

The exact command I used was this:

echo "c0292556d7a1271ee0ffc06d1913fd7dff574821" | ssh-keygen -Y sign -f /path/to/key -n dn42

You can push your changes to their git server and navigate to it in a web browser. There should be a button which allows you to open a pull request. When issuing a pull request, you are given the option to include a message and this is where the output of the above command goes.

I used this as my message:

commit "c0292556d7a1271ee0ffc06d1913fd7dff574821" signed:

-----BEGIN SSH SIGNATURE-----
U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgkSTA7O6sGzisnwPi531xJhDER7
b3OuOZr+LSP+h6qsoAAAAEZG40MgAAAAAAAAAGc2hhNTEyAAAAUwAAAAtzc2gtZWQyNTUx
OQAAAECqzO6Eqsq2ODPA49K5PgbA9u62mjsD1CBkj307LPEljgag8tiWaKRwsLYckEc2Si
N7ci38hD5V3BwjVDWZxjsH
-----END SSH SIGNATURE-----

I included the (unsigned) hash too as recommended - this makes it easier for the maintainers. My changes were merged into the main branch the following morning.

This wasn't an issue with DN42 but it did waste hours of my time when trying to push my changes. I performed 1 push (my branch which was the same as the main branch at the time) with no issues at all. My next push which contained my changes got stuck immediately after the “Total 17 (delta 9), reused 0 (delta 0), pack-reused 0” line. I confirmed with my network monitoring that the connection to the git server was being interrupted.

I used this command to enable SSH debugging during the push:

GIT_SSH_COMMAND="ssh -vvv" git push

This outputted the following:

Compressing objects: 100% (17/17), done.
debug2: channel 0: read<=0 rfd 4 len 0
debug2: channel 0: read failed
debug2: channel 0: close_read
debug2: channel 0: input open -> drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug3: send packet: type 96
debug2: channel 0: input drain > closed
Writing objects: 100% (17/17), 1.88 KiB | 960.00 KiB/s, done.
Total 17 (delta 9), reused 0 (delta 0), pack-reused 0

This confirms a communication issue but I'm not familiar enough with SSH debugging to understand what exactly this error means; I just know the communication went bad. After a lot of troubleshooting, I eventually realised the problem was caused by the MTU being too high on the server I was using to push. Specifically, I was using a VM on my home server - the hypervisor has VLAN tagging enabled but the VM was configured to use an MTU of 1500 which meant the packets were too large when the VLAN tag got added. SSH was using the DF flag meaning that parts of the communication seemed to be getting dropped.

I'm still looking into exactly how this happened but this is the conclusion I currently have.

Once your registration has been approved, you will most likely want to connect to the network. This involves 2 steps. Firstly, you need to find a peer, and secondly you need to connect to that peer. I will describe how I achieved this below.

The DN42 Peer Finder tool can help you find a peer with low latency. I chose a peer with a 20ms latency and an open peering policy (essentially anyone can peer without asking for permission first). The peer I chose is Kioubit which appears to be one of the largest nodes on the DN42 network.

Kioubit allows you to register automatically using your SSH key (or presumably GPG key if that's what you used in your registration). I followed their instructions and within minutes I was given the required details to connect to their UK node via Wireguard. It was super easy and convenient, which explains why most DN42 routers seem to have a direct connection to them.

I'm personally sticking to Wireguard, but other VPN software can be used to gain access to DN42. I used the following commands to connect to the Kioubit UK server with Wireguard:

ip link add dev wg0 type wireguard
wg setconf wg0 tunnel.conf
ip addr add fe80::xxxx/64 dev wg0
ip addr add 172.23.13.8/32 peer 172.xx.xx.xx/32 dev wg0
ip link set wg0 up

Kioubit instructed me to use a particular IPv6 address whereas the IPv4 address is the first available address in my block (I own 172.23.13.8/29). The IPv4 peer IP was also given to me by Kioubit.

These are the contents of tunnel.conf:

[Interface]
PrivateKey = [redacted]
ListenPort = 51820

[Peer]
PublicKey = sLbzTRr2gfLFb24NPzDOpy8j09Y6zI+a7NkeVMdVSR8=
Endpoint = uk1.g-load.eu:21469
AllowedIPs = 0.0.0.0/0,::/0
PersistentKeepalive = 20

Most of this information was given to me by Kioubit and will vary based on whichever peer you are using. I generated my privatekey with the following command:

wg genkey | tee privatekey | wg pubkey > publickey

I had no problem connecting to their server once I had all of the commands right. Initially I had entered an IP address wrong and had some problems because of it.

Simply connecting to the VPN doesn't give you much access. I was able to ping a couple of things belonging to the Kioubit network but that's about as far as I could get. In order to reach the rest of the network, you need to announce your IP addresses so all the routers can find a path to you.

I opted to use Bird and BGP for this task. The DN42 wiki has a Bird guide to help you get started. I struggled with this step the most.

I used the configuration examples from the guide but found that my Bird instance wasn't even listening for connections. Eventually I discovered this was because I had used the wrong IP address in the “peers” config file which should look something like this:

# /etc/bird/peers4/kioubit
protocol bgp kioubit from dnpeers {
  neighbor 172.xx.xx.xx as 4242423914;
  import where dn42_import_filter(4,22,34);
  #export where dn42_import_filter(4,22,34);
  export all;
};

Bird noticed that the IP address I had used was unroutable (it was a 172.x address but not something the VPN could reach) so it just went into a kind of dormant mode where it wasn't doing anything.

Once I had copied the example config and used the correct IPs, my routing table started to get populated with various 172.x addresses belonging to people on the DN42 network. Unfortunately, my IPs were not being announced which meant I still wasn't actually connected.

The Bird guide recommends that BGP communities are used on this page so I followed their examples. Unfortunately, part of this example appears to be what caused my issues. Specifically this line:

export where dn42_import_filter(4,22,34);

Notice how it is commented out in my above snippet. The 3 numbers (4,22,34) correspond to my latency, bandwidth, and encryption type. These values are meant to help prioritise routes based on their capabilities but it seems like I made a mistake somewhere because it completely prevented my route from being announced.

I still need to investigate exactly what the situation is regarding this because I don't want to incorrectly announce my network capabilities.

Once my Bird setup was complete, I seemed to have a fully working connection to DN42. I found a Telegram bot which could perform traceroutes for me; I like this one because it was also showing hostnames so I could see which nodes my traffic was passing through without performing manual lookups:

traceroute to 172.23.13.8 (172.23.13.8), 30 hops max, 60 byte packets
 1  nl-ams02.rtr.jlu5.dn42 (172.20.229.118)  1.994 ms  1.933 ms  1.901 ms
 2  de-nbg01.rtr.jlu5.dn42 (172.20.229.117)  20.246 ms  20.332 ms  20.294 ms
 3  hub.de2.kioubit.dn42 (172.20.14.35)  21.036 ms  21.018 ms  20.999 ms
 4  edge.uk1.kioubit.dn42 (172.20.14.41)  27.248 ms  27.231 ms  27.182 ms
 5  172.23.13.8 (172.23.13.8)  47.922 ms  47.926 ms  48.136 ms

By looking at this traceroute, we can see it originated in The Netherlands, went to Germany via the jlu5 network, entered Kioubit's German node where it travelled to Kioubit UK, then to me.

This traceroute was performed by the jlu5 looking glass server in Australia:

 1  sg01.rtr.jlu5.dn42 (172.20.229.127) [*]  93.758 ms
 2  outpost.hk1.kioubit.dn42 (172.20.14.42) [*]  131.203 ms
 3  hub.de2.kioubit.dn42 (172.20.14.35) [*]  289.287 ms
 4  edge.uk1.kioubit.dn42 (172.20.14.41) [*]  304.108 ms
 5  172.23.13.8 (172.23.13.8) [*]  325.385 ms

After connecting to the network and being able to ping DN42 devices around the world, I decided it would make sense to now connect a PC through my router so I could browse some internal services.

I decided to use NAT and my DN42 router IP to access the network with another device as this is what I'm most familiar with. Many people will generally assign one IP per PC without NAT due to the large number of available IP addresses.

There are lots of guides online for how to use Ubuntu as a NAT router and it generally only involves a few iptables rules. I then configured a static route on my PC to point towards my DN42 router (but only for IPs belonging to DN42). This was enough to give me a basic connection to DN42 which “worked”.

Next time I connect a PC to DN42, I will probably use pfsense which is a router/firewall OS. This would give me a much cleaner interface and allow quicker configuration (with less chance of making mistakes).

I didn't spend much time on this part of the process because I am far more interested in playing around with BGP. My next step will be to rent another VPS and to announce another part of my IP prefix there so I can use different IPs in different locations.