Data-driven Edge Functions With Netlify and PolyScale

Netlify recently announced that Edge Functions are now available in public beta on the Netlify platform. Edge Functions are built on the Deno Deploy infrastructure and enable you to run serverless JavaScript/TypeScript functions from within Netlify’s Edge network.

With the addition of serverless Edge Functions, Netlify adds the ability to build dynamic microservices, all packaged up within the familiar Git-driven deployment workflow. Global deployments are the default as is scaling without infrastructure and DevOps overheads. Productivity++.

So the question then arises, what about data-driven functions? Once database connectivity is required for global state, transactions, complex data query etc, developers then need to think through how to scale their current database e.g. PostgreSQL or MySQL. TCP connections are not cheap latency wise and therefore scaling can get complex. Adding GraphQL or other HTTP based data abstractions can help with caching, but at the expense of development time.

The transition to multi-region or edge is often blocked by the inability to scale the data tier. After all, for data-driven apps, low latency static assets and code (business logic) are only a piece of the puzzle.

Netlify + PolyScale

PolyScale.ai provides a no-code, plug-and-play database edge cache. It is wire-protocol compatible with several databases (including MySQL and PostgreSQL) and intelligently caches data automatically based on query pattern analysis. Just update the database connection string to integrate.

PolyScale is serverless, highly scalable and will serve any cached query in ~1ms. With global Points of Presence (PoP’s), network latency is significantly reduced. It also automatically handles cache invalidation globally on data change (INSERT’s, UPDATE’s, DELETE’s).

Using PolyScale with Netlify is a powerful proposition for data-driven apps. Without the data tier cache, edge functions suffer the latency overheads of connecting back to the origin database, which may be geographically far away from the requesting edge function. In addition, that origin database can suffer from poor performance and scaling due to concurrency and connection limits exceeded by large numbers of ephemeral edge functions.

Having functions access a database through PolyScale ensures that queries are fast, low latency and massively concurrent, no matter where the edge functions are executed.🔥

Building A Data Driven Edge Function

With that context, let’s dive in and get a Netlify Edge Function connected to PostgreSQL via PolyScale.

Creating an Edge Function

Getting started with Edge Functions on Netlify is easy (see the docs here). All you need is an account with a Git provider (GitHub or GitLab) with a repository that you’ve setup to be deployed with Netlify.

Create your Edge Function under netlify/edge-functions/postgres.ts.

export default () => new Response("Success");

Next, create a netlify.toml file in your repository root and add the following configuration to make Netlify aware of your new function.

[[edge_functions]]
path = "/postgres"
function = "postgres"

You can test your functions locally by using the Netlify CLI.

Setting up PolyScale

Create a free PolyScale account and once logged in, select the New Cache button from the top right:

Here, enter the hostname and port of the origin database, and select the database type; PostgreSQL in this case.

Once created, PolyScale will return a unique cache id. This is a unique identifier for the cache and must be included as part of the database connection string. For PostgreSQL, this id must be embedded in the application_name property. Take note of this id for connecting in the next section.

Establishing a database connection

Since Netlify Edge Functions are built upon Deno, you can make use of any of the third party libraries currently made available by the community. Our example function will use the deno-postgres package to establish a connection to our PostgreSQL database instance.

The updated code looks like this:

import { Client } from "https://deno.land/x/postgres@v0.15.0/mod.ts";

export default async () => {
  const client = new Client({
    hostname: "{{DATABASE_HOST}}",
    port: "{{DATABASE_PORT}}",
    user: "{{DATABASE_USER}}",
    password: "{{DATABASE_PASSWORD}}",
    database: "{{DATABASE_DATABASE}}",
  });

  await client.connect();

  const d = await client.queryObject("SELECT 1");

  return new Response(JSON.stringify(d.rows));
};

We simply establish a TCP connection to the target database, perform a no-op query and return the result to the function caller.

Integrating PolyScale

Our function now makes connections from the edge directly to our target database. This isn’t ideal as the target database probably doesn’t exist in the region the edge function runs in and therefore suffers from the aforementioned latency and scaling overheads. To keep things fast and scalable for the data tier at the edge, we plug in PolyScale using our newly created cache.

For PostgreSQL, there are two changes to make to the connection parameters to integrate PolyScale:

1. Update the hostname to use PolyScale’s global DNS address (psedge.global). The PolyScale port for PostgreSQL is 5432.
2. Pass in an applicationName parameter containing the unique cache id from the PolyScale dashboard:

import { Client } from "https://deno.land/x/postgres@v0.15.0/mod.ts";

export default async () => {
  const client = new Client({
    hostname: "psedge.global",
    port: 5432,
    user: "{{DATABASE_USER}}",
    password: "{{DATABASE_PASSWORD}}",
    database: "{{DATABASE_DATABASE}}",
    applicationName: "{{CACHE_ID}}""
  });

  await client.connect();

  const d = await client.queryObject("SELECT 1");

  return new Response(JSON.stringify(d.rows));
};

With the connection parameter changes in place, our edge function now connects to the target database via PolyScale and data passing through is now being automatically inspected, cached and served from the edge.

Deploy and test

To deploy our new function simply commit all changes and run git push. Netlify will pick up the latest state and automatically deploy any functions specified in the netlify.toml file. Once deployment has finished visit {{deployment-url}}/postgres to see your newly created edge function in action.

Once the function has executed, switch to the PolyScale UI and select the Observability tab from the cache nav bar. Here you can inspect all query traffic that passes through the platform. The SQL and query count is shown under the heatmap graph:

After the function and database query has run ~14 times, PolyScale’s AI will start caching the response data and you will see cache hits. You can see the number of hits by switching to the Details view from the top navigation:

Conclusions

With the addition of Deno based Edge Functions from Netlify, developers can easily build dynamic services using the same unified Netlify developer experience. At PolyScale, we see the modern data architecture as both global and serverless. Combining PolyScale with Edge Functions offers devs a highly scalable, global architecture for building data-intensive applications.

If edge data intrigues you and you enjoy hacking on distributed systems, we’re hiring. We obsess about latency and work on hard problems with large abstractions to make developers’ day jobs simpler.