Moving Dokku Apps Between Physical Servers with No (Very Little) Down Time

Moving Live Servers without Downtime... Risky my friend!

I built and run quite a large scale free cloud software that spans multiple independent servers (using a Microservice design pattern) and recently I was trying to optimise the operational management of all my servers. I am a big fan of Dokku and had cloud servers running "Microservice Apps" deployed via Dokku on Digital Ocean and also AWS EC2. The platforms were fragmented and I wanted to bring all my cloud servers together onto the same cloud platform (Digital Ocean or AWS) in order to make it easier to manage and gain operational visibility (centralised monitoring via dashboards where metrics all mean the same thing).

After looking at my options I decided to move all my servers to Digital Ocean. I always loved their platform and after they added streamlined (and free server monitoring) it made it a no-brainer (AWS EC2 monitoring is not seamless and the machines cost more as well). For me Digital Ocean always had the upper-edge on AWS when it came to user-friendless and provided a much better DX (Developer Experience). AWS is a lot more feature-rich when building distributed backend systems but Digital Ocean gives you a much better experience when it comes to cloud machine setup and management (Droplet vs EC2)

In this tutorial, I will show you how I moved a “Live” Microservice API running on AWS (an EC2 with Dokku Installed and the API Microservice deployed inside a Docker container) to the same setup in Digital Ocean (an Droplet with Dokku Installed and the API Microservice deployed inside a Docker container). As it was a live API, I wanted limited downtime as possible (I managed to do it with zero downtime - but as DNS updates are required you may want to schedule some downtime with your users before you attempt this). The API was also served over HTTPS so I had to bring up the HTTP and HTTPS endpoints as close together as possible. (I used the Dokku LetsEncrypt plugin for this as shown below)

Docker via Dokku in DigitalOcean

My setup was as follows:
Old Server: AWS - EC2 T2-Micro, Dokku 0.8.0 (older version)
New Server: Digital Ocean Droplet, Dokku 0.12.13 (I used the Digital Ocean one-click app images they already have for Dokku)


💥 💥 Firstly, I need to give you my usual Disclaimer for these kinds of risky tutorials :) There might be a much better way to do this but these are the steps I followed and got it down with zero downtime. I can’t guarantee this will work for you and attempting this might result in excessive downtime or lost data for you. Please do this at your own risk! 



Now let’s get to it:

1) Sign up for a new Digital Ocean Server (Droplet with Dokku Installed). I used the One-Click Dokku image they had. I already had an SSH key setup as I had other servers with DigitalOcean so I used the existing key during setup of the new Droplet. 

It's also a good idea to now upgrade the dokku version to the latest so we are up to date with all upgrades. You can follow instructions here: https://dokku.com/docs/getting-started/upgrading/

2) Once your server is up, open the Dokku config page (It’s usually found by hitting your new IP in the browser). In the config page, make sure your new server IP is added in the "hostname" field - do not use the domain you are using in the old server now. Don't select virtualhost naming for now (although I don’t think this will impact you - as you will need to turn on virtualhost in a step below). Finish this config setup as soon as possible or you risk someone discovering your IP and seeing your KEY details!

3) SSH into your new server and create a a Dokku app with the same name in the old server:

dokku apps:create my-app

4) Locally in your GIT controlled source code for the app, set a new GIT remote from your repo and point to your new sever:

git remote add dokku-new dokku@my-new-server-ip:my-app

5) Now push your latest master code to dokku-new. If you have any issues with the deployment you might want to enable tracing and debug by SSHing to your new Server and running:

dokku trace on

Possible Issues with SSH and Git push:

When you setup the server with Dokku, you most likely picked the Public Key SSH option and put in a Public Key. Now when you are trying to SSH in you will need to specify the local private key to do this. To keep things simple, I usually just update the ~/.ssh/config file to include the new Dokku server host/IP and private key. Something like this:

Host [server IP]

    Hostname [server IP]

    IdentityFile ~/.ssh/new_key_private

    IdentitiesOnly yes

If SSH does not work, you need to make sure the private/public keypair matches. I do this by running this cat ~/.ssh/authorized_keys in the box (might be hard to do if you have no SSH access, but a cloud terminal may help)

After this you will be able to SSH in fine. This should also work fine for "git push" commands to this IP Host as well - But i've seen the following error:

dokku@[server IP]: Permission denied (publickey).

fatal: Could not read from remote repository.

Please make sure you have the correct access rights

and the repository exists.

It seems that sometimes, Dokku loses the public key reference in the server as raised here

You can confirm this by SSHing into the box and running the command: dokku ssh-keys:list

You will NOT see any key.

To fix this, I just pushed the Public Key again like so (admin1 is just a name for the key):

cat ~/.ssh/new_key.pub | ssh root@[server IP] dokku ssh-keys:add admin1

Running dokku ssh-keys:list will show the key is now there and "git push" will work 

6) One common error that may occur with the above step is you will see an error similar to “pre-receive hook declined”. This may be caused by your code actually being successfully deployed but (in the case of a node.js app), it attempted to start the app in the new server but some Environment Variables driven dependency like a Database or Redis connection failed (as Environment Variables that hold the database endpoint or connection details do not exist in the new server). This will throw an error like above which will make you think the deployment failed. Another reason why it might fail is because as the deploy is happening, the server runs out of memory (happens a lot with low tier cloud servers). You need to view server metrics as deploy is happening to work this.

7) If this happens, then SSH into your box and set all the required Environment Variables. Your do this via the command:

dokku config:set my-app varName1=VarValue1 varName2=VarValue2




8) Once deployment has completed successfully, SSH in again and check to ensure all your Environment Variables are there and you will also notice some new Dokku Variables added.

9) I then enabled vhost/virtualhost on my new app via the command:

dokku domains:enable my-app

This will restart the app with vhost enabled and give you a IP:80 url instead of the default IP:RandomPort. Now when you can hit the url in the browser you can verify if your API is accessible. For me it was by hitting the Health Check URL I have on all my microservices. e.g. http://IP/health-check. Note that you may see a weird domain in the logs like 'http://http://IP/' - just ignore this as it will fix it self in the next step.

10) I then added the live domain to the app via the command:

dokku domains:add my-app myapplivedomain.com

After you run the above command you should see a success message that nginx reloaded with your new settings. Confirm this has happened now by hitting this command and seeing the response:

dokku domains:report my-app

11) If you restart your app now you should see the domain appear in the Dokku log:

.
.
=====> Application deployed:
http://myapplivedomain.com

12) [This is not needed if you put cloudflare in from of you server and enable FLEXIBLE SSL/TLS where traffic between cloudflare and the origin is via HTTP] - Now we should kickstart the HTTPS certificate part via the plugin dokku-letsencrypt, but we won't be able to complete it now as the DNS is not updated to new IP - but we should start it off now. We get HTTPS via the free letsencrypt service. Install the dokku plugin on your new server via:

sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git




* This article also points to some instruction on this process - https://medium.com/@pimterry/effortlessly-add-https-to-dokku-with-lets-encrypt-900696366890

13) I ran this command to set it up:
dokku config:set --no-restart my-app DOKKU_LETSENCRYPT_EMAIL=your@email.tld
* your@email.tld is something you need to control and monitor

14) Then I ran the command to start up the letsencrypt process:

dokku letsencrypt my-app

15) The above command will most likely fail with an error like:

Did you set correct path in -d myapplivedomain.com:path or --default_root? Is there a warning log entry about unsuccessful self-verification? Are all your domains accessible from the internet? Failing authorizations: https://acme-v01.api.letsencrypt.org/acme/authz/[some-verylong-code-here] Challenge validation has failed, see error log.

16) It's now time to swap the DNS nameservers for your domain and point to the new Server's IP. There will be some downtime from now on!! as we still have not successfully enabled letsencrypt for HTTPS on your new server (and your downstream apps are probably expecting the HTTPS urls).. but we have no choice as we can’t enable letsencrypt until the domain resolves to the new server IP.

17) Swap your DNS A and @ values to the new server IP (This may be different for each domain registrar)

18) At the same time do the letsencrypt “acme challenge” as well as seen in the error message above when we tried to enable letsencrypt. The acme challenge is something you may need to complete (not sure if this is really needed but I did it anyway). Basically you need to copy the [some-verylong-code-here] from the error message seen above as a TXT DNS record with the name _acme-challenge. The code needs to go into value like so:

TXT    _acme-challenge    some-verylong-code-here

19) After the DNS resolves, which should not take too long. Go back to the SSH console and re-run the command that failed previously:

dokku letsencrypt my-app

20) It should now work successfully... and you won't see that error anymore.

21) Its also a good idea to auto-renew your letsencrypt cert or your HTTPS endpoints may go down. Do that like so:

dokku letsencrypt:cron-job --add

22) Finally to verify it all went well, restart the dokku app. It should reload and show both your HTTP and HTTPS endpoint mapping set up:

.
.
=====> Application deployed:
http://myapplivedomain.com
https://myapplivedomain.com

23) You have now officially swapped over to the new server! 🙌💪💃💕


If this article helped you out and you are keen to give Digital Ocean a try (I highly recommend them) - then use this referrer link to get some free credit
https://m.do.co/c/63904110df0d

You should get 25$ (in October 2018 I believe it’s actually 100$!) and I’ll get some credit to run my servers as well :)

All the best and happy coding!