Hosting simplified by a CDN

Before using a content delivery network (CDN) for the first time, I thought I would use them only when running a big site.

Last year I moved my personal website and our Meetup’s homepage to Netlify, a content delivery network (CDN). I was astounded about the minimal steps necessary to make it work. It simplified both the setup and the day-to-day operations for me.

This article takes you through the minimal, yet real world steps to set up the homepage of the Vue.js Frankfurt Meetup with Netlify. This includes the full build automation, deployment process and notifications via Telegram when new content is published. You should know about the basics of Linux scripting, HTTP Headers and git before reading this article. Vue.js’ Frankfurt Homepage relies on yarn or npm for build automation, therefore parts of this article rely on basic knowledge of one of the tools.

Content Delivery Networks ¶

At its core, a Content Delivery Network (CDN) provides an infrastructure to respond to HTTP requests of web site visitors. It hosts the content – usually static or cached files – on multiple server locations around the globe to deliver it with high availability, high bandwidth and low latency from the location closest to the visitor. As add-ons they also provide

• reverse proxy services to mix static and dynamic content
• caching for resources that change infrequently
• TLS encryption and certificate handling
• scaling for high traffic loads, and
• mitigation of DDoS attacks.

Over time, they evolved in a way that you don’t need to host your own servers to publish a web site any more. This article shows you how to do this.

Static Site Generators ¶

A static site generator creates files at build time that are then served unchanged (statically) to all visitors of the site. This includes HTML, CSS, JavaScript and resources like images. This concept gained popularity with Jekyll that integrated in GitHub pages: Site owners select a theme and write content in the Markdown format. GitHub runs Jekyll to create HTML files and serves them to the visitors.

The plugins supported for Jekyll by Github are minimal. You’ll get greater flexibility when creating the files outside of GitHub’s automated process and ask GitHub to serve the static files. Today there are multiple popular static site generators. There is still Jekyll based on Ruby with lots of plugins, new players like Hugo based on Go, and VuePress based on Vue.js. Frameworks like Nuxt and Next take this even further: They prepare pre-rendered pages that give an instant rendering to the visitor. Then they load all necessary JavaScript to make it an interactive single page application (SPA) that interacts with a back-end.

Getting started ¶

We built the homepage of the Vue.js Frankfurt Meetup with VuePress, a static site generator based on Vue.js. It stores all content in Markdown files (one file per page). You can add dynamic content by adding Vue components when needed. The initialisation of VuePress creates all the files needed for development and production mode.

The entire Vue.js Frankfurt website with content and build automation is available as source code on GitHub. It contains all the configuration files mentioned in this document.

To make changes to the site you first check out the GitHub repository. Then you run VuePress in the development mode where changes to a local Markdown file are automatically visible in your browser for a live preview. This resembles the edit mode of a content management system (CMS). Once the changes are complete, you commit the changes and push them to GitHub.

Setup of the Content Delivery Network ¶

Basic Setup and the first published Site ¶

To get started with Netlify you need a Git repository and a Domain. The Git repository can be hosted for example on GitHub, GitLab or Bitbucket. For Vue.js Frankfurt we host it on GitHub. We registered our domain vuejsfrankfurt.de at Strato, a German hosting company. There we edit the Domain Name Service (DNS) entries for our domain, but we host our content on Netlify.

DNS Entries, IPv6 ¶

Two DNS entries are necessary to forward all traffic to Netlify:

• An A record for vuejsfrankfurt.de
• An CNAME record for www.vuejsfrankfurt.de

Once I’ve enable IPv6 on the Netlify homepage, www.vuejsfrankfurt.de is reachable by IPv6 as well. This is great!

Publishing content on Netlify ¶

Now it’s time for Netlify to build the site and serve the first content. As a first step you register for a free account at Netlify and configure a link to the git repository with content for the static site generator.

Then Netlify needs to know about the build command and directory with the HTML content to be published. You can configure your site’s build settings via Netlify’s web user interface or via a configuration file. Our meetup uses the configuration file to track the changes in the version control system.

# netlify.toml
[build]
  command = "./build.sh"
  publish = "docs/.vuepress/dist"

#!/usr/bin/env bash
set -ex
yarn build

For a single command it’s straightforward to put the build command in the configuration file. But as we will extend the build in the upcoming chapters for example to check broken links, we start with a separated file. The x-setting will echo all commands on the console. The e-setting will exit the bash script once a command fails and return that exit code to the caller. See Explain Shell for a short description or Peter Bengthsson’s blog post for a longer explanation with examples.

In order for this to work the file needs to have executable permissions. You can add them after you’ve added the file to the git repository using the following command:

git update-index --chmod=+x build.sh

When building VuePress will translate the theme and all Markdown content to static HTML pages and JavaScript files.

Once the first build succeeds, your content is live. Congratulations!

Every subsequent push to the the master branch of the git repository will trigger a new build and a fresh deployment.

Let’s encrypt: SSL for our website ¶

When using Netlify, adding SSL to your site is only a few mouse clicks away. It uses Let’s encrypt, a well-known service for free certificates. Once the DNS settings have propagated, this is added within seconds. Let’s Encrypt certificates are valid for 90 days only, so the certificate needs to be updated every few weeks. Luckily, Netlify already takes care of that.

Redirects and 404-Page handling ¶

The _redirects file contains all the configurations about redirects and HTTP status codes. We want all links to the index.html to redirect to the start page, as they are the same page. If we would have had an old site, we could add redirects from the old relative URLs to the new relative URLs. We could even redirect to an external site for content we offer no more.

Netlify will return any file that exists. If Netlify can’t find the file, it returns the file 404.html. Using the /* match at the end of the configuration file you can redirect Netlify to any file you want for non-existent URLs. I’ve chosen to have the default configuration in the file to make it explicit.

# _redirects configuration file
# We placed the source in docs/.vuepress/public
# The build puts it into the dist folder where Netlify will find and use it. 

# Redirect default Netlify subdomain to primary domain
# (replace placeholders in pointy brackets with your values)
https://<alias>.netlify.com/* https://<www.yourdomain.tld>/:splat 301!

/index.html /

# any existing resource will be returned directly, any left-over wildcards will be 404
/* /404.html 404

Performance and Caching done right ¶

By default Netlify asks the browser to cache all pages and to re-validate them on every request if they have changed. Using this feature, visitors will always see the most recent content. But for the assets folder VuePress adds a content hash at the end of the file. Whenever the content changes (be it for an image or for a JavaScript resource), the file name will change as well and VuePress will reference the changed file name in every page that uses it.

Therefore, the browser doesn’t need to re-validate them anymore. We use the max-age attribute in the Cache-Control header to tell the browser about this. To pass Google’s Lighthouse test, you need to set max-age to at least 31,536,000 seconds (one year).

# _headers configuration file
# We placed the source in docs/.vuepress/public
# The build puts it into the dist folder where Netlify will find and use it. 

# The following files have content hashes, 
# therefore they can be used without re-validation
/assets/*
  Cache-Control: max-age=31536000, must-revalidate

With these settings your site will load faster for returning visitors.

Build Notifications and Functions ¶

Every push to the master branch triggers an update of the website. It takes a few seconds for the site to build, and then it is online. Waiting these few seconds is sometimes annoying, and as a developer you want to be sure the build succeeded.

Netlify allows you to configure hooks and functions that trigger on build events. I once set up IFTTT to capture the web hooks, but found it tedious to test and I couldn’t pass on all the information provided by Netlify. Then I found out about Netlify Functions.

The following scripts convert the Netlify event to a Telegram message and publishes it to a group chat. In this example there are notifications for the start of the build and successful and failed builds.

The steps for the setup:

1. Adding a new sub-project to the git repository. I called it lambda, but you can choose any name here.
2. Extending the build.sh with a new step to generate the compiled functions.
3. Extending the existing netlify.toml with the functions attribute where Netlify can find the compiled functions.
4. Placing a new (!) netlify.toml in the lambda directory to specify the distribution folder relative to the sub-folder. This build process will only see this file when compiling the lambdas.
5. Setting up a Telegram bot.
6. Adding the configuration elements API_TOKEN and CHAT_ID as environment variables in the Netlify web UI (these confidential information must not be stored in the git repository).

Step 1: The netlify-lambda GitHub repository describes how to set up the contents of the lambda folder. Keeping all of it inside its own folder prevents WebPack and library version conflicts between your project and netlify-lambda.

For sample contents of the lambda folder see the corresponding folder in the Vue.js Frankfurt GitHub repository.

In the lambda/src folder there is one JavaScript file for each hook following Netlify’s naming convention. Each function parses the data JSON element with detailed information provided by Netlify. It combines the essential bits to a message and sends it to the Telegram chat. It reports all errors on the console accessible from Netlify’s Web UI. See below two notifications as they appear in the Telegram client.

#!/usr/bin/env bash
set -ex

# add the following lines to build.sh
cd lambda
yarn install
yarn generate
cd ..

# netlify.toml in root folder with relative folder for deployment
[build]
  functions = "lambda/dist"

# netlify.toml in lambda folder with relative folder for building
[build]
  functions = "dist"

Step 5: Please have a look at the Telegram’s bots and their developer API documentation on how to setup a new bot. You won’t be able to interact with this bot as it will only send status information. Once you’ve registered the bot, you’ll receive an API_TOKEN. To get the CHAT_ID, add the bot as a manager to a group and allow it to read and post messages. Then post one message to the group using the Telegram client. Use the REST-API to get the CHAT_ID. Once you have the CHAT_ID, you can reduce the rights of the bot. The bot uses the CHAT_ID to post all notifications to this group.

POST https://api.telegram.org/{API_TOKEN}/getUpdates

HTTP/1.1 200 OK
Content-Type: application/json

{
  "ok": true,
  "result": [
    {
      "update_id": ...,
      "channel_post": {
        "chat": {
          "id": -100....,
          "title": "Vue.js Frankfurt's Web Site",
          "type": "channel"
        },
        "date": 1548088260
      }
    }
  ]
}

Step 6: Setup the environment variables in the Netlify Web UI in Settings > Build & Deploy > Continuous Deployment > Build environment variables.

Once you’ve completed this setup, you’ll receive two notifications for each build: One that the build has started, and another that it has completed.

Preview-Feature for Pull Requests ¶

We’ve also tried out the deploy preview feature: Netlify comments every pull request with a link to a preview page specific to the pull request.

With this feature you can review the changes of a pull request on a live site before you merge it.

Please be aware: On a public GitHub repository everyone can create a pull request. A malicious pull request might expose information you’ve passed to the build as environment variables (for example the API Key for the Telegram bot), as Netlify passes them to the build of the pull request. Therefore we’ve disabled this feature until Netlify offers a method to better protect API keys. See chapter “Limitations” below for more details. Update: As of March 26th 2020 you can choose a “Sensitive variable policy” to protect these variables, for example untrusted deploys (by pull requests) could be deployed without sensitive variables or after a manual approval.

Broken Link checker ¶

A single typo can break a link. Deleting a page might miss references pointing to it. Both will leave your visitor on a 404-Not-Found page.

Broken Link Checker (blc) is there to help: The following setup will check all internal links of the website. If one of the internal links is broken, the deployment will fail. We don’t check external links as we don’t want the build to fail when an external site is temporarily unavailable.

yarn add -dev broken-link-checker
yarn add -dev serve

{
  "scripts": {
    "serve": "serve -n docs/.vuepress/dist",
    "blc": "blc http://127.0.0.1:5000 -e -f -ro --filter-level 2",
    "blc-external": "blc http://127.0.0.1:5000 -ro --filter-level 2"
  }
}

#!/usr/bin/env bash
set -ex

# add the following lines to build.sh
# serve site on localhost and run link checker
# this call is optimized to run on the netlify server
# it is not wrapped by yarn, so that we get the PID to kill it later
node /opt/build/repo/node_modules/.bin/serve -n docs/.vuepress/dist &
child_id=$!
yarn blc
kill $child_id || true

The script above will serve the static content on port 5000. Then blc will check it. If it finds an error, it will exit with a non-zero error code. The e-setting will exit the bash with non-zero error code. This will make the build fail and no content will be deployed.

To check external links, run a local build, serve it and check the links.

yarn build
yarn serve &
yarn blc-external

Limitations running on Netlify ¶

After running for about 8 months on Netlify, here’s a summary of the limitations I found:

1. Storing secrets in environment variables.
   The best solution I found to store secrets like API_KEY and CHAT_ID for Telegram was using environment variables. This way the secrets are not stored in the git repository, but they are still accessible in plain text from the Netlify admin UI. Netlify will use the same environment variables for all branches and pull requests. A malicious script submitted by a pull request will expose your environment variables with this setup. I hope Netlify will provide the option to hide secrets from pull request builds in the future. Update: As of March 26th 2020 you can choose a “Sensitive variable policy” to protect these variables, for example untrusted deploys (by pull requests) could be deployed without sensitive variables or after a manual approval.

2. Paying Money for additional users.
   Netlify is only free as long there is a single user accessing the Netlify web UI. Having more users will cost you money. Sending notifications to Telegram to give the team a hint what’s wrong when a site doesn’t build successfully might defer this for some teams. Also the comments on the pull requests help to see a preview and a build status of a pull request.

3. Paying money for bandwidth.
   There is a soft limit of 100 GB per month for the free plan. I’m well below that, let’s see if this stays this way. They describe it as a soft limit; I assume they’ll disable the site if I exceed it too much they will get in touch to ask you to pay, before disabling it automatically (updated 05.02.2019).

4. Paying money when the number of functions called increases.
   Within the free plan, there are 125,000 executions and 100 hours execution time. As long as I use the functions only for notifications, I’ll stay well below. Once I use it to provide an API, the usage will increase. Reading “Scalable: Your plan will upgrade automatically to fit your usage.” is scary: Will I need to pay for the excess usage, or will they disable the service once I reach the limit?

5. Purging old versions
   All the old versions of the site exist, at least when I access them from the Netlify web console. The HTTP header X-Robots-Tag: noindex prevents search engines to index them. I read that you can contact the support to delete old content, for example, when you are obliged by legal reasons like General Data Protection Regulation (GDPR). I’d love to see a purge functionality in the API or on the web console.

6. Old files gone at redeployment
   Netlify deployments are atomic: They switch to a new set of files in an instant once the build is complete. For visitors of your site, a file that is referenced in an open browser might become unavailable. One thing that saves you is a Progressive Web App (PWA): This downloads at least all JavaScript resources at the beginning, and maybe other images depending on your configuration. For Nuxt 2.3 non-PWA I’ve added a workaround to reload the Single Page Application (SPA) if navigating to a new page is missing a JavaScript resource. Version Nuxt 2.4+ includes a fix.

A friend of mine is hosting his content on ZEIT – they apparently offer similar flexibility. They emphasise like Netlify using a CDN for static content and lambdas for dynamic content. I think it’s worth a look if you want to start a new project, but I’m happy with Netlify for now.

Summary ¶

In this article I’ve described how to setup a site on Netlify, configure the build process, how to check for broken links in the build process and how to send out notifications on Telegram about a started, completed or broken build.

Looking back, using Netlify was easier than setting up my own server. It was also easier than setting up a build process with Travis CI, something I’ve done previously for hosting pages on GitHub pages. The web UI allowed me to learn about Netlify’s features one by one. There is also an API and a command-line tool, but I’ve used neither.

The flexibility on how to create your HTML is amazing. It’s nice to have a web server running with SSL, HTTP/2 and IPv6, and that is managed and upgraded without manual intervention.

For a similar project and even a bigger project I would use Netlify again. It’s good to see it is possible to mix static with dynamic content when using single page applications: Either with Netlify Functions to call back end functionality, or using Netlify as a reverse proxy. If you want to learn more about a setup like this, you might want to read up on the JAMstack: JAM is an acronym for client side JavaScript, APIs and Markup.

You’ve seen real-life configurations that will give you an idea for your own projects. If you have questions, contact me via a direct message on twitter or via email, I’m happy to help.

(thank you to Wolfgang Werner and Mateusz Parzonka for reviewing drafts of this article)