Data-driven Edge Functions With Netlify and PolyScale
Ben Hagan
May 02, 2022Netlify recently announced that Edge Functions are now available in public beta on the Netlify platform. Edge Functions are built on the Deno Deploy infrastructure and enable you to run serverless JavaScript/TypeScript functions from within Netlify’s Edge network.
With the addition of serverless Edge Functions, Netlify adds the ability to build dynamic microservices, all packaged up within the familiar Git-driven deployment workflow. Global deployments are the default as is scaling without infrastructure and DevOps overheads. Productivity++.
So the question then arises, what about data-driven functions? Once database connectivity is required for global state, transactions, complex data query etc, developers then need to think through how to scale their current database e.g. PostgreSQL or MySQL. TCP connections are not cheap latency wise and therefore scaling can get complex. Adding GraphQL or other HTTP based data abstractions can help with caching, but at the expense of development time.
The transition to multi-region or edge is often blocked by the inability to scale the data tier. After all, for data-driven apps, low latency static assets and code (business logic) are only a piece of the puzzle.
Netlify + PolyScale
PolyScale.ai provides a no-code, plug-and-play database edge cache. It is wire-protocol compatible with several databases (including MySQL and PostgreSQL) and intelligently caches data automatically based on query pattern analysis. Just update the database connection string to integrate.
PolyScale is serverless, highly scalable and will serve any cached query in ~1ms. With global Points of Presence (PoP’s), network latency is significantly reduced. It also automatically handles cache invalidation globally on data change (INSERT’s, UPDATE’s, DELETE’s).
Using PolyScale with Netlify is a powerful proposition for data-driven apps. Without the data tier cache, edge functions suffer the latency overheads of connecting back to the origin database, which may be geographically far away from the requesting edge function. In addition, that origin database can suffer from poor performance and scaling due to concurrency and connection limits exceeded by large numbers of ephemeral edge functions.
Having functions access a database through PolyScale ensures that queries are fast, low latency and massively concurrent, no matter where the edge functions are executed.🔥
Building A Data Driven Edge Function
With that context, let’s dive in and get a Netlify Edge Function connected to PostgreSQL via PolyScale.
Creating an Edge Function
Getting started with Edge Functions on Netlify is easy (see the docs here). All you need is an account with a Git provider (GitHub or GitLab) with a repository that you’ve setup to be deployed with Netlify.
Create your Edge Function under netlify/edge-functions/postgres.ts
.
export default () => new Response("Success");
Next, create a netlify.toml
file in your repository root and add the following
configuration to make Netlify aware of your new function.
[[edge_functions]]
path = "/postgres"
function = "postgres"
You can test your functions locally by using the Netlify CLI.
Setting up PolyScale
Create a free PolyScale account and once logged in, select the New Cache
button from the top right:
Here, enter the hostname
and port
of the origin database, and select the
database type; PostgreSQL in this case.
Once created, PolyScale will return a unique cache id. This is a unique
identifier for the cache and must be included as part of the database connection
string. For PostgreSQL, this id must be embedded in the application_name
property. Take note of this id for connecting in the next section.
Establishing a database connection
Since Netlify Edge Functions are built upon Deno, you can make use of any of the third party libraries currently made available by the community. Our example function will use the deno-postgres package to establish a connection to our PostgreSQL database instance.
The updated code looks like this:
import { Client } from "https://deno.land/x/postgres@v0.15.0/mod.ts";
export default async () => {
const client = new Client({
hostname: "{{DATABASE_HOST}}",
port: "{{DATABASE_PORT}}",
user: "{{DATABASE_USER}}",
password: "{{DATABASE_PASSWORD}}",
database: "{{DATABASE_DATABASE}}",
});
await client.connect();
const d = await client.queryObject("SELECT 1");
return new Response(JSON.stringify(d.rows));
};
We simply establish a TCP connection to the target database, perform a no-op query and return the result to the function caller.
Integrating PolyScale
Our function now makes connections from the edge directly to our target database. This isn’t ideal as the target database probably doesn’t exist in the region the edge function runs in and therefore suffers from the aforementioned latency and scaling overheads. To keep things fast and scalable for the data tier at the edge, we plug in PolyScale using our newly created cache.
For PostgreSQL, there are two changes to make to the connection parameters to integrate PolyScale:
- Update the
hostname
to use PolyScale’s global DNS address (psedge.global
). The PolyScale port for PostgreSQL is5432
. - Pass in an
applicationName
parameter containing the unique cache id from the PolyScale dashboard:
import { Client } from "https://deno.land/x/postgres@v0.15.0/mod.ts";
export default async () => {
const client = new Client({
hostname: "psedge.global",
port: 5432,
user: "{{DATABASE_USER}}",
password: "{{DATABASE_PASSWORD}}",
database: "{{DATABASE_DATABASE}}",
applicationName: "{{CACHE_ID}}""
});
await client.connect();
const d = await client.queryObject("SELECT 1");
return new Response(JSON.stringify(d.rows));
};
With the connection parameter changes in place, our edge function now connects to the target database via PolyScale and data passing through is now being automatically inspected, cached and served from the edge.
Deploy and test
To deploy our new function simply commit all changes and run git push
. Netlify
will pick up the latest state and automatically deploy any functions specified
in the netlify.toml
file. Once deployment has finished visit
{{deployment-url}}/postgres
to see your newly created edge function in action.
Once the function has executed, switch to the PolyScale UI and select the
Observability
tab from the cache nav bar. Here you can inspect all query
traffic that passes through the platform. The SQL and query count is shown under
the heatmap graph:
After the function and database query has run ~14 times, PolyScale’s AI will
start caching the response data and you will see cache hits. You can see the
number of hits by switching to the Details
view from the top navigation:
Conclusions
With the addition of Deno based Edge Functions from Netlify, developers can easily build dynamic services using the same unified Netlify developer experience. At PolyScale, we see the modern data architecture as both global and serverless. Combining PolyScale with Edge Functions offers devs a highly scalable, global architecture for building data-intensive applications.
If edge data intrigues you and you enjoy hacking on distributed systems, we’re hiring. We obsess about latency and work on hard problems with large abstractions to make developers’ day jobs simpler.