Setting up mobile backup for my home broadband

As part of an interview for a Technical Product Manager job I had to create a video presentation how I would improve a product at home. I have chosen my home broadband and argued for enabling monitoring and mobile backup as an improvement. While I am monitoring my home broadband for quite some time I have always played with the idea to use my mobile data plan in case my fixed broadband is down.

I recently moved default gateway function to my monitoring Raspberry Pi with the help of a VLAN enabled switch - see my previous post about it. The mobile coverage in the electric room where my broadband modem installed is probably the weakest in the building so not an ideal place for a mobile backup solution. On the other hand upstairs at my working desk the mobile coverage is reasonable and I have my other Raspberry Pi that I use as my desktop PC usually. I also regularly charge my phone next to this Raspberry Pi which sparked the idea to extend USB tethering as a mobile backup solution for the entire home. For this I need to change the routing at both Raspberry Pi in case the fixed broadband is down so that all traffic is routed via USB tethering. For this I have found a failover script here that I have customised for my use case as follows on the Raspberry Pi next to my broadband modem:

jordana@pi4b2:~ $ nano failover.sh

#!/bin/bash

# Set defaults if not provided by environment

CHECK_DELAY=${CHECK_DELAY:-1}

CHECK_IP=${CHECK_IP:-8.8.8.8}

PRIMARY_IF=${PRIMARY_IF:-eth0.8}

PRIMARY_GW=${PRIMARY_GW:-192.168.0.1}

BACKUP_IF=${BACKUP_IF:-eth0}

BACKUP_GW=${BACKUP_GW:-192.168.0.12}

# Compare arg with current default gateway interface for route to healthcheck IP

gateway_if() {

  [[ "$1" = "$(ip route get "$CHECK_IP" | sed -rn 's/^.*dev ([^ ]*).*$/\1/p')" ]]

}

# Cycle healthcheck continuously with specified delay

while sleep "$CHECK_DELAY"

do

  # If healthcheck succeeds from primary interface

  if ping -I "$PRIMARY_IF" -c1 "$CHECK_IP" &>/dev/null

  then

    # Are we using the backup?

    if gateway_if "$BACKUP_IF"

    then # Switch to primary

      ip route delete default via "$BACKUP_GW" dev "$BACKUP_IF"

    fi

  # If healthcheck fails from primary interface

  else

    # Are we using the primary?

    if gateway_if "$PRIMARY_IF"

    then # Switch to backup

      ip route add default via "$BACKUP_GW" dev "$BACKUP_IF"

    fi

  fi

done

On my desk I also have a failover script running on the Raspberry Pi: 

jordana@dad-pi4g:~ $ nano failover.sh

#!/bin/bash

# Set defaults if not provided by environment

CHECK_DELAY=${CHECK_DELAY:-1}

CHECK_IP=${CHECK_IP:-8.8.8.8}

PRIMARY_IF=${PRIMARY_IF:-eth0}

PRIMARY_GW=${PRIMARY_GW:-192.168.0.1}

BACKUP_IF=${BACKUP_IF:-usb0}

BACKUP_GW=${BACKUP_GW:-192.168.0.1}

EMAIL=${EMAIL:-jordan.arpad@gmail.com}

# Compare arg with current default gateway interface for route to healthcheck IP

gateway_if() {

  [[ "$1" = "$(ip route get "$CHECK_IP" | sed -rn 's/^.*dev ([^ ]*).*$/\1/p')" ]]

}

switch_to_primary() {

  ip route delete default via "$PRIMARY_GW" dev "$PRIMARY_IF"

  ip route add default via "$PRIMARY_GW" dev "$PRIMARY_IF"

  date | mail -s "$HOSTNAME switched to primary" "$EMAIL"

}

switch_to_backup() {

  ip route delete default via "$PRIMARY_GW" dev "$PRIMARY_IF"

  ip route add default via "$PRIMARY_GW" dev "$PRIMARY_IF" metric 5000

  date | mail -s "$HOSTNAME switched to backup" "$EMAIL"

}

# Cycle healthcheck continuously with specified delay

while sleep "$CHECK_DELAY"

do

  # If healthcheck succeeds from primary interface

  if ping -I "$PRIMARY_IF" -c1 "$CHECK_IP" &>/dev/null

  then

    # Are we using the backup?

    if gateway_if "$BACKUP_IF"

    then switch_to_primary

    fi

  # If healthcheck fails from primary interface

  else

    sleep "$CHECK_DELAY"

    # If healthcheck succeeds from primary interface

    if ping -I "$PRIMARY_IF" -c1 "$CHECK_IP" &>/dev/null

    then

      # Are we using the backup?

      if gateway_if "$BACKUP_IF"

      then switch_to_primary

      fi

    # If healthcheck fails from primary interface AGAIN

    else

      # Are we using the primary?

      if gateway_if "$PRIMARY_IF"

      then switch_to_backup

      fi

    fi

  fi

done

I added routing and NAT to usb0 on the desktop Raspberry Pi also and whenever I have my phone connected with USB tethering to the mobile network the redundancy switching is working like a charm back and forth :) So I made these scripts running @reboot from crontab and configured notification emails from my desktop Raspberry.

UPDATE

I added email notification to the script as per above after installing ssmtp and mailutils and configured as follows. Please note that this only works with App Password added to your Google account so use it with care, at your own risk and consider creating a separate Google account for this purpose.

jordana@dad-pi4g:~ $ sudo cat /etc/ssmtp/ssmtp.conf

#

# Config file for sSMTP sendmail

#

# The person who gets all mail for userids < 1000

# Make this empty to disable rewriting.

root=

# The place where the mail goes. The actual machine name is required no 

# MX records are consulted. Commonly mailhosts are named mail.domain.com

mailhub=smtp.gmail.com:587

# Where will the mail seem to come from?

rewriteDomain=

# The full hostname

hostname=dad-pi4g

# Are users allowed to set their own From: address?

# YES - Allow the user to specify their own From: address

# NO - Use the system generated From: address

#FromLineOverride=YES

AuthUser=jordan.arpad73@gmail.com

AuthPass=

UseSTARTTLS=YES