MULTI REGION DEPLOYMENTS

If you want to deploy using multiple regions you will need to set

MULTI_REGION="1"

in your template(s) for each region.

You will also need to set a primary region by having one of the regions you are deploying to set

PRIMARY_REGION="1"

and all the additional regions set

PRIMARY_REGION="0"

When making a multi region deployment you will need to provide the public DBaaS endpoint to all regions which are not the primary region. The primary region should be set such that the region for your webservers of your primary region is the same region/VPC/provider as the DBaaS database you are deploying.

Every region will need to provision its own reverse proxy machines because multi-region deployments are only possible when each individual region uses reverse proxies to forward traffic to the fleet of webservers for that region.

Here I have used one build machine running in the gb-lon to perform the builds for all regions of the multi region deployment but you could have a different build machine in/for each region and that would encapsulate each region nicely and make it simpler to have different teams responsible for primary, and secondary and even tertiary regions.


EXAMPLE 1 (PRIMARY REGION Linode running gb-lon)

Shown below are my template configurations if I want to deploy a primary region of gb-lon and a secondary region of nl-ams for my application on the linode platform.You need to take a similar approach with other providers if you want to deploy to multiple regions.

For this example, you first need to deploy a build machine in the gb-lon region which you can do by following this

TEMPLATE FOR THE PRIMARY REGION (gb-lon)

When you make a multi region deployment the advice is that you should set the BUILD_IDENTIFER to include the region that is being deployed to, for example, testdeploy-gb-lon for the gb-lon domain and testdeploy-nl-ams for the nl-ams domain under linode. I have one build machine in the gb-lon region.

Highlighted in red are the settings in the templates that you need to take particular care with when making a multi-region deployment

###############################################################################################
# Refer to: ${BUILD_HOME}/templatedconfigurations/specification.md ###############################################################################################
#This template is configured for temporal style builds

#####MANDATORY - the bare minimum set of values that you need to provide to have any chance of a successful build
#####NOT REQUIRED - isn't used by the Linode system

#####Application Settings#########
export APPLICATION="joomla" #MANDATORY
export JOOMLA_VERSION=""
export DRUPAL_VERSION=""
export APPLICATION_BASELINE_SOURCECODE_REPOSITORY=""
export BASELINE_DB_REPOSITORY=""
export APPLICATION_LANGUAGE="PHP"
export PHP_VERSION="8.4"
export BUILD_ARCHIVE_CHOICE="hourly"
export APPLICATION_NAME="Demo Application"

#####S3 Datastore Settings#######
export S3_ACCESS_KEY="JH56HJ78WE4T6U8OO90"
export S3_SECRET_KEY="Hgdj89K2w3eyrb1289sfjDewjk"
export S3_HOST_BASE="nl-ams-1.linodeobjects.com"
export S3_LOCATION="US" #For linode, this always needs to be set to "US"
export DIRECTORIES_TO_MOUNT="images" #This should always be unset for a virgin and baseline deployments
export PERSIST_ASSETS_TO_DATASTORE="1" #This should always be set to 0 for a virgin and baseline deployment

#####OS Settings#########
export BUILDOS="debian" # One of ubuntu|debian
export BUILDOS_VERSION="12" # 24.04 (or later for BUILDOS="ubuntu") | 12 (or later for BUILDOS="debian")

######Cloudhost Provider Settings#######
export TOKEN="hjdd738jdaq7fhwk2bdif8rhdnqi238fks92jdkfojshf93jsfhndjk67" #MANDATORY
export ACCESS_KEY="" #NOT REQUIRED
export SECRET_KEY="" #NOT REQUIRED
export CLOUDHOST_ACCOUNT_ID="demoaccount" #MANDATORY for Linode - this should be the account username that you login with

######DNS Settings##########
export DNS_USERNAME="dnsusername@email.com" #MANDATORY
export DNS_SECURITY_KEY="9aa34568735212367d137dfadf34daa26:H3aYfKLqW4gfJQP1qOPc_jkWPa-QjiodJAL0jCKM" #MANDATORY (note you need a API token and account ID here if you use cloudflare)
export DNS_CHOICE="cloudflare" #you will need to set your DNS nameservers according to this choice

#####Webserver Settings########
export WEBSITE_DISPLAY_NAME="Joomla Demo" #MANDATORY
export WEBSITE_NAME="codetesters" #MANDATORY
export WEBSITE_URL="www.codetesters.uk" #MANDATORY
export WEBSERVER_CHOICE="APACHE"
export REVERSE_PROXY_WEBSERVER="APACHE"
export NO_WEBSERVERS="1"
export MAX_WEBSERVERS="10"
export MOD_SECURITY="0"

#####Git settings#####
export GIT_USER="Templated User" #MANDATORY
export GIT_EMAIL_ADDRESS="templateduser@dummyemailZ123.com" #MANDATORY

#####Infrastructure Repository Settings#######
export INFRASTRUCTURE_REPOSITORY_PROVIDER="github"
export INFRASTRUCTURE_REPOSITORY_OWNER="wintersys-projects"
export INFRASTRUCTURE_REPOSITORY_USERNAME="wintersys-projects"

###### Application Repository Settings########
export APPLICATION_REPOSITORY_PROVIDER="github"
export APPLICATION_REPOSITORY_OWNER="adt-apps" #MANDATORY
export APPLICATION_REPOSITORY_USERNAME="adt-apps" #MANDATORY
export APPLICATION_REPOSITORY_TOKEN="github_pat_11BELT3NQ03fCHpjdjn7y3hjdhkf37DHHS8jfh38fjfy3o9qoskfogjJHHJDJkfhskfu3osdjf"

##### System Email Settings#########
export SYSTEM_EMAIL_PROVIDER=""
export SYSTEM_EMAIL_PROVIDER="2"
export SYSTEM_TOEMAIL_ADDRESS="webmaster@codetesters.uk"
export SYSTEM_FROMEMAIL_ADDRESS="webmaster@codetesters.uk"
export SYSTEM_EMAIL_USERNAME="gf72fdhkocnv28de7e8ifjjw8f2"
export SYSTEM_EMAIL_PASSWORD="hfjh47fi328rjfh28folmajfigj"
export EMAIL_NOTIFICATION_LEVEL="ERROR"

##### Database Settings######
export DB_PORT="2035"
export DATABASE_INSTALLATION_TYPE="DBaaS"
export DATABASE_DBaaS_INSTALLATION_TYPE="MySQL:DBAAS:mysql/8:gb-lon:g6-nanode-1:1:test-cluster:testdb:testdbuser:hfhuf83jfhfu73jd:436653:234531"
export BYPASS_DB_LAYER="0"

#####Server Settings #######
export REGION="gb-lon"
export DB_SERVER_TYPE="g6-nanode-1"
export WS_SERVER_TYPE="g6-nanode-1"
export AS_SERVER_TYPE="g6-nanode-1"
export AUTH_SERVER_TYPE="g6-nanode-1"
export RP_SERVER_TYPE="g6-nanode-1"
export MACHINE_TYPE="LINODE"
export SSH_PORT="1035"
export SERVER_TIMEZONE_CONTINENT="Europe"
export SERVER_TIMEZONE_CITY="London"
export USER="root"
export SYNC_WEBROOTS="0"
export USER_EMAIL_DOMAIN=""

#####Build Settings###### export PRODUCTION="1"
export DEVELOPMENT="0"
export NO_AUTOSCALERS="1"
export NO_REVERSE_PROXY="1"
export AUTHENTICATION_SERVER="0"
export BUILD_FROM_SNAPSHOT="0"

#####Security Settings#####
export ACTIVE_FIREWALLS="3"
export PUBLIC_KEY_NAME="AGILE_TOOLKIT_PUBLIC_KEY"
export SSL_GENERATION_METHOD="AUTOMATIC"
export SSL_GENERATION_SERVICE="LETSENCRYPT"
export SSL_LIVE_CERT="1"
export ALGORITHM="ed25519"
export BUILD_MACHINE_VPC="1"
export VPC_IP_RANGE="10.0.1.0/24"
export VPC_NAME="adt-vpc-gb-lon"

#####Multi Region Deployments#####
export MULTI_REGION="1"
export PRIMARY_REGION="1"
export DBaaS_PUBLIC_ENDPOINT=""

#####Build Style#######
export INPARALLEL="1"

export BUILD_IDENTIFIER="test-gb-lon"
export CLOUDHOST="linode"


Here is my template for the nl-ams region when I am deploying to a primary region of gb-lon using the template above and to a secondary region to nl-ams using the template below both (in this case) under Linode

###############################################################################################
# Refer to: ${BUILD_HOME}/templatedconfigurations/specification.md
###############################################################################################
#This template is configured for temporal style builds

#####MANDATORY - the bare minimum set of values that you need to provide to have any chance of a successful build
#####NOT REQUIRED - isn't used by the Linode system

#####Application Settings#########
export APPLICATION="joomla" #MANDATORY
export JOOMLA_VERSION=""
export DRUPAL_VERSION=""
export APPLICATION_BASELINE_SOURCECODE_REPOSITORY=""
export BASELINE_DB_REPOSITORY=""
export APPLICATION_LANGUAGE="PHP"
export PHP_VERSION="8.4"
export BUILD_ARCHIVE_CHOICE="hourly"
export APPLICATION_NAME="Demo Application"

#####S3 Datastore Settings#######
export S3_ACCESS_KEY="JH56HJ78WE4T6U8OO90"
export S3_SECRET_KEY="Hgdj89K2w3eyrb1289sfjDewjk"
export S3_HOST_BASE="nl-ams-1.linodeobjects.com"
export S3_LOCATION="US" #For linode, this always needs to be set to "US"
export DIRECTORIES_TO_MOUNT="images" #This should always be unset for a virgin and baseline deployments
export PERSIST_ASSETS_TO_DATASTORE="1" #This should always be set to 0 for a virgin and baseline deployment

#####OS Settings#########
export BUILDOS="debian" # One of ubuntu|debian
export BUILDOS_VERSION="12" # 24.04 (or later for BUILDOS="ubuntu") | 12 (or later for BUILDOS="debian")

######Cloudhost Provider Settings#######
export TOKEN="hjdd738jdaq7fhwk2bdif8rhdnqi238fks92jdkfojshf93jsfhndjk67" #MANDATORY
export ACCESS_KEY="" #NOT REQUIRED
export SECRET_KEY="" #NOT REQUIRED
export CLOUDHOST_ACCOUNT_ID="demoaccount" #MANDATORY for Linode - this should be the account username that you login with

######DNS Settings##########
export DNS_USERNAME="dnsusername@email.com" #MANDATORY
export DNS_SECURITY_KEY="9aa34568735212367d137dfadf34daa26:H3aYfKLqW4gfJQP1qOPc_jkWPa-QjiodJAL0jCKM" #MANDATORY (note you need a API token and account ID here if you use cloudflare)
export DNS_CHOICE="cloudflare" #you will need to set your DNS nameservers according to this choice

#####Webserver Settings########
export WEBSITE_DISPLAY_NAME="Joomla Demo" #MANDATORY
export WEBSITE_NAME="codetesters" #MANDATORY
export WEBSITE_URL="www.codetesters.uk" #MANDATORY
export WEBSERVER_CHOICE="APACHE"
export REVERSE_PROXY_WEBSERVER="APACHE"
export NO_WEBSERVERS="1"
export MAX_WEBSERVERS="10"
export MOD_SECURITY="0"

#####Git settings#####
export GIT_USER="Templated User" #MANDATORY
export GIT_EMAIL_ADDRESS="templateduser@dummyemailZ123.com" #MANDATORY

#####Infrastructure Repository Settings#######
export INFRASTRUCTURE_REPOSITORY_PROVIDER="github"
export INFRASTRUCTURE_REPOSITORY_OWNER="wintersys-projects"
export INFRASTRUCTURE_REPOSITORY_USERNAME="wintersys-projects"

###### Application Repository Settings########
export APPLICATION_REPOSITORY_PROVIDER="github"
export APPLICATION_REPOSITORY_OWNER="adt-apps" #MANDATORY
export APPLICATION_REPOSITORY_USERNAME="adt-apps" #MANDATORY
export APPLICATION_REPOSITORY_TOKEN="github_pat_11BELT3NQ03fCHpjdjn7y3hjdhkf37DHHS8jfh38fjfy3o9qoskfogjJHHJDJkfhskfu3osdjf"

##### System Email Settings#########
export SYSTEM_EMAIL_PROVIDER=""
export SYSTEM_EMAIL_PROVIDER="2"
export SYSTEM_TOEMAIL_ADDRESS="webmaster@codetesters.uk"
export SYSTEM_FROMEMAIL_ADDRESS="webmaster@codetesters.uk"
export SYSTEM_EMAIL_USERNAME="gf72fdhkocnv28de7e8ifjjw8f2"
export SYSTEM_EMAIL_PASSWORD="hfjh47fi328rjfh28folmajfigj"
export EMAIL_NOTIFICATION_LEVEL="ERROR"

##### Database Settings######
export DB_PORT="2035"
export DATABASE_INSTALLATION_TYPE="DBaaS" export DATABASE_DBaaS_INSTALLATION_TYPE="MySQL:DBAAS:mysql/8:gb-lon:g6-nanode-1:1:test-cluster:testdb:testdbuser:hfhuf83jfhfu73jd:436653:234531"
export BYPASS_DB_LAYER="1"

#####Server Settings #######
export REGION="nl-ams"
export DB_SERVER_TYPE="g6-nanode-1"
export WS_SERVER_TYPE="g6-nanode-1"
export AS_SERVER_TYPE="g6-nanode-1"
export AUTH_SERVER_TYPE="g6-nanode-1"
export RP_SERVER_TYPE="g6-nanode-1"
export MACHINE_TYPE="LINODE"
export SSH_PORT="1035"
export SERVER_TIMEZONE_CONTINENT="Europe"
export SERVER_TIMEZONE_CITY="London"
export USER="root"
export SYNC_WEBROOTS="0"
export USER_EMAIL_DOMAIN=""

#####Build Settings######
export PRODUCTION="1"
export DEVELOPMENT="0"
export NO_AUTOSCALERS="1"
export NO_REVERSE_PROXY="1"
export AUTHENTICATION_SERVER="0"
export BUILD_FROM_SNAPSHOT="0"

#####Security Settings#####
export ACTIVE_FIREWALLS="3"
export PUBLIC_KEY_NAME="AGILE_TOOLKIT_PUBLIC_KEY"
export SSL_GENERATION_METHOD="AUTOMATIC"
export SSL_GENERATION_SERVICE="LETSENCRYPT"
export SSL_LIVE_CERT="1"
export ALGORITHM="ed25519"
export BUILD_MACHINE_VPC="0"
export VPC_IP_RANGE="10.0.1.0/24"
export VPC_NAME="adt-vpc-nl-ams"

#####Multi Region Deployments#####
export MULTI_REGION="1"
export PRIMARY_REGION="0"
export DBaaS_PUBLIC_ENDPOINT="public-a47568393-akamai-prod-6748387-default.g2a.akamaidb.net"

#####Build Style#######
export INPARALLEL="1"

export BUILD_IDENTIFIER="test-nl-ams"
export CLOUDHOST="linode"

You will need to first deploy the primary region infrastrucuture (gb-lon) and once that is complete and online, deploy the secondary region and any further regions that you want to deploy for. The DBaaS instance is in the gb-lon region and the machines in nl-ams will connect to the database instance in the gb-lon domain across the Internet. As I understand it the traffic is encrypted by default but to be but don't take my word for it check it out with linode because it might be necessary to provide SSL certs when connecting to the DBaaS system across the Internet.

I then made a deployment to a third region (with a different vendor, exoscale) and this is what my template looked like when I made a deployment to a different vendor and had all the different regions tied together through the cloudflare DNS system and all having them access the shared Linode DBaaS system. Remember I chose to intitiate all these builds from the same build machine running on Linode.

###############################################################################################
# Refer to: ${BUILD_HOME}/templatedconfigurations/specification.md
###############################################################################################
#This template is configured for temporal style builds

#####MANDATORY - The bare minimum set of values that you need to provide to have any chance of a successful build
#####NOT REQUIRED - isn't used by the Exoscale

#####Application Settings#########
export APPLICATION="joomla" #MANDATORY
export JOOMLA_VERSION=""
export DRUPAL_VERSION=""
export APPLICATION_BASELINE_SOURCECODE_REPOSITORY=""
export BASELINE_DB_REPOSITORY=""
export APPLICATION_LANGUAGE="PHP"
export PHP_VERSION="8.4"
export BUILD_ARCHIVE_CHOICE="hourly"
export APPLICATION_NAME="Demo Application"

#####S3 Datastore Settings#######
export S3_ACCESS_KEY="GPIJ6HS1MY6LU7QE243V"
export S3_SECRET_KEY="Ru9aX4oBxK3W1Ga1eTNj4c96SV8AQzsk4p6KXdvT"
export S3_HOST_BASE="nl-ams-1.linodeobjects.com"
export S3_LOCATION="US" #For linode, this always needs to be set to "US"
export DIRECTORIES_TO_MOUNT="images" #This should always be unset for a virgin and baseline deployments
export PERSIST_ASSETS_TO_DATASTORE="1" #This should always be set to 0 for a virgin and baseline deployment

#####OS Settings#########
export BUILDOS="debian" # One of ubuntu|debian
export BUILDOS_VERSION="12" # 24.04 (or later for BUILDOS="ubuntu") | 12 (or later for BUILDOS="debian")

######Cloudhost Provider Settings#######
export TOKEN="" #NOT REQUIRED
export ACCESS_KEY="EXO7hfgu476gfvfnsu4nfbgy223dhf" #MANDATORY
export SECRET_KEY="jcbh846tfg2vdugihdhevuft2ojknbub374fbjbsksbd" #MANDATORY
export CLOUDHOST_ACCOUNT_ID="testcoders@yahoo.com" #MANDATORY for Exoscale - this should be the account email address that you login to the portal with

######DNS Settings##########
export DNS_USERNAME="dnsusername@email.com" #MANDATORY
export DNS_SECURITY_KEY="9aa34568735212367d137dfadf34daa26:H3aYfKLqW4gfJQP1qOPc_jkWPa-QjiodJAL0jCKM" #MANDATORY (note you need a API token and account ID here if you use cloudflare)
export DNS_CHOICE="cloudflare" #you will need to set your DNS nameservers according to this choice

#####Webserver Settings########
export WEBSITE_DISPLAY_NAME="Joomla Demo" #MANDATORY
export WEBSITE_NAME="codetesters" #MANDATORY
export WEBSITE_URL="www.codetesters.uk" #MANDATORY
export WEBSERVER_CHOICE="APACHE"
export REVERSE_PROXY_WEBSERVER="APACHE"
export NO_WEBSERVERS="1"
export MAX_WEBSERVERS="10"
export MOD_SECURITY="0"

#####Git settings#####
export GIT_USER="Templated User"
export GIT_EMAIL_ADDRESS="templateduser@dummyemailZ123.com"

#####Infrastructure Repository Settings#######
export INFRASTRUCTURE_REPOSITORY_PROVIDER="github"
export INFRASTRUCTURE_REPOSITORY_OWNER="wintersys-projects"
export INFRASTRUCTURE_REPOSITORY_USERNAME="wintersys-projects"

###### Application Repository Settings########
export APPLICATION_REPOSITORY_PROVIDER="github"
export APPLICATION_REPOSITORY_OWNER="adt-apps" #MANDATORY
export APPLICATION_REPOSITORY_USERNAME="adt-apps" #MANDATORY
export APPLICATION_REPOSITORY_TOKEN="github_pat_11BELT3NQ03fCHpjdjn7y3hjdhkf37DHHS8jfh38fjfy3o9qoskfogjJHHJDJkfhskfu3osdjf"

##### System Email Settings#########
export SYSTEM_EMAIL_PROVIDER=""
export SYSTEM_EMAIL_PROVIDER="2"
export SYSTEM_TOEMAIL_ADDRESS="webmaster@codetesters.uk"
export SYSTEM_FROMEMAIL_ADDRESS="webmaster@codetesters.uk"
export SYSTEM_EMAIL_USERNAME="gf72fdhkocnv28de7e8ifjjw8f2"
export SYSTEM_EMAIL_PASSWORD="hfjh47fi328rjfh28folmajfigj"
export EMAIL_NOTIFICATION_LEVEL="ERROR"

##### Database Settings######
export DB_PORT="2035"
export DATABASE_INSTALLATION_TYPE="DBaaS"
export DATABASE_DBaaS_INSTALLATION_TYPE="MySQL:DBAAS:mysql/8:gb-lon:g6-nanode-1:1:test-cluster:testdb:testdbuser:hfhuf83jfhfu73jd"
export BYPASS_DB_LAYER="1"

#####Server Settings #######
export REGION="ch-gva-2"
export DB_SERVER_TYPE="tiny"
export WS_SERVER_TYPE="tiny"
export AS_SERVER_TYPE="tiny"
export AUTH_SERVER_TYPE="tiny"
export RP_SERVER_TYPE="tiny"
export MACHINE_TYPE="EXOSCALE"
export SSH_PORT="1035"
export SERVER_TIMEZONE_CONTINENT="Europe"
export SERVER_TIMEZONE_CITY="London"
export USER="root"
export SYNC_WEBROOTS="1"
export USER_EMAIL_DOMAIN=""

#####Build Settings######
export PRODUCTION="1"
export DEVELOPMENT="0"
export NO_AUTOSCALERS="1"
export NO_REVERSE_PROXY="1"
export AUTHENTICATION_SERVER="0"
export BUILD_FROM_BACKUP="0"

#####Security Settings#####
export ACTIVE_FIREWALLS="3"
export PUBLIC_KEY_NAME="AGILE_TOOLKIT_PUBLIC_KEY"
export SSL_GENERATION_METHOD="AUTOMATIC"
export SSL_GENERATION_SERVICE="LETSENCRYPT"
export SSL_LIVE_CERT="1"
export ALGORITHM="rsa"
export BUILD_MACHINE_VPC="0"
export VPC_IP_RANGE="10.0.0.0/24" #MANDATORY
export VPC_NAME="adt-vpc"

#####Multi Region Deployments#####
export MULTI_REGION="1"
export PRIMARY_REGION="0"
export DBaaS_PUBLIC_ENDPOINT="a327564-akamai-prod-370219-default.g2a.akamaidb.net"

#####Build Style#######
export INPARALLEL="1"

export BUILD_IDENTIFIER="test-ch-gva"
export CLOUDHOST="exoscale"


EXAMPLE 2 (PRIMARY REGION Exoscale running ch-gva)

In this example, the Exoscale vendor is the PRIMARY_REGION with the machine marked "VM....." as the build machine from which the servers of all vendors are built and deployed from. In other words, its the machine on which the templates are configured. The DBaaS service used in this example is the DBaaS service offering from Exoscale and all the other vendors connect in to the DBaaS deployed through the primary domain mechanism across the Internet (performance seems to be OK as far as it goes).

The Digital Ocean servers are running as a secondary region.

The Linode servers are running as a 3rd region.

The Vultr servers are running as a 4th region

If you notice that there's a reverse proxy machine on for each vendor and as there are 4 vendors with one reverse proxy each we would expect there to be four ip addresses added to the DNS system and there are.

In a live deployment you would most likely want 2 reverse proxies per vendor for resilience. I might as well note here as well that you will see that only exoscale has a DB layer deployed which is a single point of failure (in terms of backups and so on) so in a live system you will most likely want a DB layer on a second region as well, which you can do by setting BYPASS_DB_LAYER to 1 ("1" because the primary domain is active already)

You can see the DBaaS system on Exoscale has had its ip-filter to allow the database (VPS) machine runnning on the exoscale server fleet (ip address: 185.19.28.170) to have access to the DBaaS system along with all the webserver ip addresses acrosss all the vendors. This list of IP addresses that have been granted access to our DBaaS system (the rest of the world is firewalled off) look as follows showing our 6 ip addresses that have been granted access to the DBaaS system:

So, for fun I scaled up the server fleet running on the Vultr vendor to 5 webservers which you can see here

And then you can see that the scaling up process opens up the ip-filter to allow the ip addresses of our new webservers as you can see here:

Here is shown 10 ip addresses that are allowed to access the database.

I then ran a one off apache bench test and here are the results:


RESILIENCE IN MULTI REGION DEPLOYMENTS

In the examples I have given, there is one build machine in one region and all the other regions on different vendors are built from that one build machine. If you want maximum resilience, then, what you need to do is have a build machine for each region that you are deploying to. That way a whole region can be taken offline (whilst other regions are kept online) with no difference in result to the end user. If a region has a problem then it can be swapped out and rebuilt or deployed to a different region if there is a regional issue and so on. This makes the way you are approaching your deployment resilient to regional outages and if you are deploying to different vendors, an outage at a particular provider.


MULTI REGION DBaaS DEPLOYMENTS

(Not currently supported but could be with some work (and money)- I don't have the money to deploy expensive cross regional databases for testing purposes 30 quid a month is about my budget for this project during its development.

If you really wanted to go to town it might be possible to use "multi region DBaaS deployments" by integrating AWS Aurora making your application accessible with low latency globally. The solution I have provided is only really suitable for specific geographical regions as it stands because the "database" is only located in a particular geographic location or region which means that if you go on holiday to Australia and you want to access an application local to the UK (in other words, a DBaaS solution local to the UK, for example). Without multi-region databases you will have high latency problems.

Amazon Aurora Global Database
Amazon Aurora Write Forwarding


RESILIENCE OF AUTHENTICATOR MACHINES IN MULTI-REGION DEPLOYMENTS

I only support one active authenticator type server at a time as the simplest design approach. That can seem like a single point of failure, but, what you can do if you don't want to risk that problem is deploy multiple backup authenticator machines in different regions but only have the IP address of one of those machines active at any one time in the DNS system. In other words, you can have "n" authenticator machines running but only "1" active in the DNS system (in other words, only one machine that is accessible by your customers through DNS lookup). If your active authenticator machine fails for some reason what you can do is switch over the authenticator machine that is active by altering the DNS record to be your secondary or tertiary authenticator machine whilst you work on resolving the problem with your original authentication machine. This will mean as little disruption as possible for your users.


AUTHENTICATION SERVER

You can deploy authentication servers to multiple regions but only the ip address of the authenticator in the primary region will have its ip address added to the DNS system. This is done for design reasons, in other words, you can't have more than one authenticator machine in the DNS system at once. The purpose for having authentications servers which are not used in secondary regions is for resilience. Having only one authentication server in the primary domain is a single point of failure and so by having backup autentication machines in other regions you can swap out the ip of the primary domain authenticator if it has a problem and swap in a secondary domain authenticator which will then do the same job.


MULTI-PROVIDER DATASTORE SYNCING

I haven't done anything to implement this, but, just to make a mental note that you are likely only relying on one provider as your datastore service and using that samee provider for all datastore activities even in multi-provider deployments. If you want complete resilience to catastrophy you might want to consider replicating your datastore buckets from one provider to other providers so that you have backups, for example that are multi-provider resilient. You can do this (replicate your datastore buckets to other providers) using rclone and how to do that is described here