Upload Encrypted Files to S3
Hey folks, welcome back to another blog post! :)
I’m sorry for the delay since my last blog post, but I’ve been busy with creating my video course Build an MVP with Elixir and speaking at ElixirConf EU in Lisbon.
But now I’m back and will post new blog posts every two weeks again. So, before we dive into today’s blog post, first the ceremonial dad joke:
I don't have a dad body.
I have a father figure.
Anyways! This post is about how to encrypt and upload sensitive user files to Amazon’s S3, so let’s dive right in!
The Problem
When you ask users to upload their files to your service, you will probably store them in a storage service like S3. S3 and others are cheaper alternatives to storing the files in your database. They also make it easier to link and show the files in e.g. your UI.
However, if your users upload sensitive files, it’s paramount to store them in a way that no other than yourself and the user can access them. This is where file encryption comes in. So, whenever you save a file in e.g. S3, you only store an encrypted version of the file. This way, neither Amazon nor any attacker who gains access to your S3 bucket can read the files. Sounds great, no?
But how do set up the encryption, decryption, upload, and download functionality in Elixir and Phoenix LiveView? This blog post answers how :)
The Solution
The following demo project is quite extensive, but here’s what it does in a nutshell:
For uploading files
The demo project uses a LiveView to allow users to upload their passport as PDF, PNG, or JPG file. After an upload completes, it encrypts the file using the AES256 algorithm with a key that’s unique for each user. It then uploads the encrypted file to S3 using the ExAws library.
For downloading files
In the same LiveView, the demo project lists all files of the current user. When a user clicks on a file, the LiveView shows either an img or embed HTML tag based on whether the file is an image or a PDF. The new HTML tag fetches the image from a Phoenix Controller, which returns the decrypted file after downloading it from S3.
Let’s write some code!
The Config
We will use the ExAws library to upload and download the files to S3. Therefore, we need to add the following dependencies to our project:
# mix.exsdefp deps do
[
{:ex_aws, "~> 2.0"},
{:ex_aws_s3, "~> 2.0"},
{:hackney, "~> 1.9"},
{:sweet_xml, "~> 0.6"}
]
end
Next, we need to add the runtime configuration for our S3 Bucket. Here’s how:
# runtime.exsconfig :ex_aws,
access_key_id: [System.get_env("AWS_ACCESS_KEY_ID"), :instance_role],
secret_access_key: [System.get_env("AWS_SECRET_ACCESS_KEY"), :instance_role]
config :demo,
aws_file_bucket: System.get_env("AWS_FILE_BUCKET"), # <- The name of your bucket
aws_file_bucket_region: System.get_env("AWS_FILE_BUCKET_REGION") # <- The region of your bucket, e.g. "eu-west-3"
And that’s the configuration! If you start your server with iex -S mix phx.server, you should be able to make calls to your S3 bucket using ExAws. You can test this by running:
bucket = Application.get_env(:demo, :aws_file_bucket)region = Application.get_env(:demo, :aws_file_bucket_region)
ExAws.S3.list_objects(bucket) |> ExAws.request(region: region)
Why Unique Keys
As mentioned before, we want to encrypt every file with an encryption key that is unique per user. You could also encrypt all files with the same encryption key, but then you put all eggs in one basket. If an attacker gains access to your encryption key, they can decrypt and view all user files. That’s not ideal.
If we decrypt every file with an encryption key that is unique to a user, an attacker would need to extract all these encryption keys instead of only a single key, which diversifies the risk a bit. There are two other benefits though:
First, we store the encryption keys in our database and never on our servers as e.g. environment variables. So, even if an attacker gains access to our servers, they won’t be able to steal these keys since keys are kept only temporarily in the application memory.
Second, if we introduce a bug in our application that allows users to view the files of other users, they won’t be able to decrypt them if we always use the current_user for decryption. If we’d encrypt all files with the same key, we wouldn’t have this security check because we’d decrypt the file with the same key, regardless of who’s the currently logged-in user. But if we use the encryption key of the current user for decrypting the file, the result would be gibberish or worst-case, throw an error.
The only caveat of this implementation is that if we change the owner of a file, we need to download and decrypt the file with the old owner’s encryption key, and then encrypt and upload the file again with key of the new owner. That’s a bit more hassle than simply updating the file.owner_id field in the database, but it’s not that complicated either.
So, given all these benefits of having a unique key per user, let’s start implementing it!
Like what you read? Sign up for more!
The User Schema
First, let’s look at the schemas. This is the User-schema:
defmodule Demo.Accounts.User do use Ecto.Schema
import Ecto.Changeset
schema "users" do
field :encryption_key, :string
end
@doc false
def changeset(file_user, attrs) do
file_user
|> cast(attrs, [:encryption_key])
|> add_encryption_key()
|> validate_required([:encryption_key])
end
defp add_encryption_key(changeset) do
case get_field(changeset, :encryption_key) do
nil ->
key = GenerateSecureString.call(32)
put_change(changeset, :encryption_key, key)
_ ->
changeset
end
end
end
In our simplified User-schema, we only store one field: the encryption_key. We set this key when we create the user using the add_encryption_key/1 helper. If we don’t set the encryption key while creating the user, this will generate a new random key of 32 bytes using the GenerateSecureString-helper module. Let’s have a look at that next.
Generate Secrets
The GenerateSecureString module only has a single function: call/1. It uses the erlang :crypto-library to generate random bytes of a given number and encodes them to Base64. As a rule of thumb, you never want to store bytes directly into the database, but always want to encode them first using e.g. Base64. If not, you might lose some bytes during the transfer.
defmodule Demo.GenerateSecureString do def call(size) do
size
|> :crypto.strong_rand_bytes()
|> Base.encode64()
end
end
The File Schema
For every uploaded file, we create a File-schema in our database. This is it:
defmodule Demo.Files.File do use Ecto.Schema
import Ecto.Changeset
schema "files" do
field(:slug, Ecto.UUID, autogenerate: true)
field(:encryption_iv, :string)
field(:resource_path, :string)
field(:filename, :string)
field(:content_type, Ecto.Enum, values: [:pdf, :jpeg, :png])
belongs_to(:owner, Demo.Accounts.User)
timestamps()
end
def changeset(file, attrs) do
file
|> cast(attrs, [:owner_id, :encryption_iv, :filename, :resource_path, :content_type])
|> validate_required([:owner_id, :encryption_iv, :filename, :resource_path, :content_type])
end
end
In the File-schema, we generate a unique slug that allows us to fetch the file through our controller using an HTTP request like GET /files/#{file.slug}. We also store the encryption salt or encryption_iv. Next, we store the resource_path. That’s the path in our S3 bucket to the file, e.g. /files/another-random-slug. We need this path for downloading the file using ExAws.S3.get_object/3. We also store the filename and content_type. This makes it easier to display the file in our UI as either image or embedded PDF. Lastly, we store the relationship of the file to the User-schema as owner_id here. This allows us to use the encryption key of the user for decrypting the file.
And that’s it regarding the schemas! Next, we’ll look at the LiveView and the encryption and decryption services.
The LiveView
Our LiveView does a lot in this demo project. It allows the user to upload a passport file as PDF, PNG, or JPG. It displays all files of the user, and it embeds either an <img> or an <embed> element for displaying the files. Have a look at the complete code below and then we’ll discuss it afterward:
defmodule DemoWeb.UploadLive do use DemoWeb, :live_view
alias Demo.Files.EncryptAndUploadFile
@impl true
def render(assigns) do
~H"""
<div>
<h1>Upload</h1>
<form id="upload-form" phx-submit="save" phx-change="validate">
<.live_file_input upload={@uploads.passport} />
<button type="submit" phx-disable-with="Uploading...">Upload Passport</button>
</form>
<ul>
<li :for={file <- @files} id={"file-#{file.id}"} phx-click="show" phx-value-file_id={file.id}>
<%= file.filename %>
</li>
</ul>
<%= if @current_file && @current_file.content_type in [:jpeg, :png] do %>
<img src={~p"/files/#{@current_file.slug}"} />
<% end %>
<%= if @current_file && @current_file.content_type in [:pdf] do %>
<embed src={~p"/files/#{@current_file.slug}"} type="application/pdf" width="400" height="800" />
<% end %>
</div>
"""
end
on_mount({DemoWeb.UserAuth, :mount_current_user})
def mount(_params, _session, socket) do
socket =
socket
|> assign_files()
|> assign(:current_file, nil)
|> allow_upload(:passport, accept: ~w(.pdf .jpg .jpeg .png))
{:ok, socket}
end
def handle_event("validate", _params, socket), do: {:noreply, socket}
def handle_event("save", _params, socket) do
current_user = socket.assigns.current_user
result =
consume_uploaded_entries(socket, :passport, fn %{path: path}, entry ->
content = File.read!(path)
filename = entry.client_name
content_type = parse_content_type(entry)
result = EncryptAndUploadFile.call(current_user, content, filename, content_type)
case result do
{:ok, _file} -> {:ok, :ok}
_error -> {:postpone, {:error, error}}
end
end)
case result do
{:error, _error} -> {:noreply, put_flash(socket, :error, "Upload failed. Please try again.")}
:ok -> {:noreply, socket |> assign_files() |> put_flash(:info, "Upload successful!")}
end
end
defp parse_content_type(entry) do
case entry.client_type do
"image/png" -> :png
"image/jpeg" -> :jpeg
"application/pdf" -> :pdf
end
end
defp assign_files(socket) do
files = Files.list_files_for_user(socket.assigns.current_user)
assign(socket, :files, files)
end
end
Let’s go through this code from top to bottom. First, we create our heex template that shows a small form that accepts a live_file_upload input called :passport. Next, it lists all files of the currently logged-in user and if the user clicks on one of these files, it sends show event to the LiveView. Lastly, it embeds either an image or an embed element, which receives the URL path to the currently selected file through the src attribute.
You might wonder why we don’t embed the file directly from the LiveView by assigning its data Base64 encoded to an image HTML element. Although this approach works for images, it doesn’t work (well) for PDFs. It’s easier to let the HTML element make the HTTP request and render the returned value.
After the render/1 function, we have the usual mount/3 callback followed by the handle_event("validate", _params, socket) function. These functions are necessary for displaying the form and the files, but not that interesting. Let’s move on.
The juicy stuff begins in the event handler of the save event. Here, we consume the uploaded file and pass it on to the EncryptAndUploadFile service. In the consume_uploaded_entries function, we first read the file content from the temporary file that Phoenix created for us. We parse the content type from the defined types like application/pdf to something that makes it easier to work with in Elixir like the atom :pdf. You don’t necessarily need to do this, but it makes it easier to check the type of a file in your UI.
We’ll get to the EncryptAndUploadFile service later, but based on its result, we either return :ok or {:error, error} from the consume function and put a success or error flash on our socket which will show a notification toast in our UI.
Okay, this LiveView is quite complex, so I’d advise you to copy&paste it into your own application and to play around with it.
Now, let’s move on the controller that downloads, decrypts, and returns the files when the <img> or <embed> elements request it.
The Controller
defmodule DemoWeb.FileController do use DemoWeb, :controller
alias Demo.Files
alias Demo.Files.DownloadEncryptedFile
def show(conn, %{"slug" => slug}) do
current_user = conn.assigns.current_user
file = Files.get_file_by_slug!(slug)
{:ok, raw_content} = DownloadEncryptedFile.call(current_user, file)
conn
|> put_resp_header("content-type", convert_content_type(file))
|> send_resp(200, raw_content)
end
defp convert_content_type(file) do
case file.content_type do
:pdf -> "application/pdf"
:jpeg -> "image/jpeg"
:png -> "image/png"
end
end
end
The controller isn’t horribly complicated, but simply fetches a file from the database by the given slug, hands over the file to the DownloadEncryptedFile service, which we’ll get into later, and returns the decrypted file together with its content type as response header.
Moving on the last piece of the (web) puzzle, the Routes.
Like what you read? Sign up for more!
The Router
# router.ex live("/upload", UploadLive)
get("/files/:slug", FileController, :show)
Add these two routes to your router.ex and you should be able to upload and view your files.
Now, let’s move on to the really interesting bits: The encryption and decryption services.
Encrypt and Upload Files
Our EncryptAndUploadFile service takes care of creating a file schema, encrypting its contents, and uploading them to S3. Have a look at the code first. We’ll discuss it below:
defmodule Demo.Files.EncryptAndUploadFile do alias Demo.Files
alias Demo.Files.GenerateSecureString
alias Demo.Repo
@iv_size 16
def call(current_user, byte_content, filename, content_type) do
key = current_user.encryption_key
iv = GenerateSecureString.call(@iv_size)
resource_path = "/files/#{Ecto.UUID.generate()}"
Repo.transact(fn ->
with {:ok, file} <- create_file(current_user, resource_path, filename, content_type, iv),
encrypted_content <- encrypt(byte_content, key, iv),
{:ok, _result} <- upload(file, encrypted_content) do
{:ok, file}
end
end)
end
defp create_file(current_user, resource_path, filename, content_type, iv) do
Files.create_file(%{
owner_id: current_user.id,
resource_path: resource_path,
filename: filename,
content_type: content_type,
encryption_iv: iv
})
end
defp encrypt(content, key, iv) do
key = Base.decode64!(key)
iv = Base.decode64!(iv)
content = content |> Base.encode64() |> pad()
:aes_256_cbc
|> :crypto.crypto_one_time(key, iv, content, true)
|> Base.encode64()
end
defp upload(file, encrypted_content) do
get_bucket()
|> ExAws.S3.put_object(file.resource_path, encrypted_content)
|> ExAws.request(region: get_region())
end
defp pad(data, block_size \\ 16) do
to_add = block_size - rem(byte_size(data), block_size)
data <> String.duplicate(<<to_add>>, to_add)
end
defp get_bucket(), do: Application.get_env(:demo, :aws_file_bucket)
defp get_region(), do: Application.get_env(:demo, :aws_file_bucket_region)
end
Now, let’s go through the code from top to bottom. In the call/4 function, we first assemble all necessary fields to create a File-schema. We generate a new encryption salt and a random resource_path. Note that the slug used in the resource_path isn’t the same as the file.slug. This is a choice that I’ve made here deliberately. Imagine that an attacker gains access to your S3 bucket filenames. They might use the slug in the filename to try to fetch the files from the GET /files/{file_slug} route. If we use the same slug for the resource path and the file.slug field, they might succeed. But, if we have two different slugs, their requests to this route will return 404. So, yet another layer of security.
Now, with these variables, we first create a new File-record, encrypt its contents, and upload them to S3. The interesting part here is the encryption, so let’s look at that in more detail.
The encrypt/3 function first decodes the encryption key and salt from their Base64 encoding. Remember that we store these fields always as base64 to prevent byte loss. That’s why we also encode the raw byte content to base64 first so that S3 doesn’t lose any bytes either. After that, we add some padding to the encoded content. The padding is necessary, because our encryption algorithm needs an input that is divisible by its block size of 16 bytes. If we have content that isn’t divisible by 16, our encryption and decryption might fail and our data is lost.
In our case, we always add padding. Even if the content is exactly a multiple of 16, we still add another block of padding of 16 bytes. Otherwise, it would be hard to figure out whether our last byte is the amount of padding we added, or belongs to the actual data.
After we padded the data, we encrypt it using the :crypto.crypto_one_time/5 function, which is the replacement for the old block_encrypt/3 function. We encode the result to Base64 again to store it on S3. The true parameter means that we’re encrypting and not decrypting (which would be false). The rest is pretty straightforward. We use ExAws.S3 to upload the file to its resource_path and finally return {:ok, file}.
Now, you might have noticed the unusual Repo.transact/2 function, which wraps the creation of the File-record and the uploading of the file to S3. This function is a handy wrapper which Tom Konidas wrote about here. It’s a neat helper function around Repo.transction/1 that handles the return for us. It looks like this:
Repo.transact/2
# Taken from https://tomkonidas.com/repo-transact/defmodule Demo.Repo
def transact(fun, opts \\ []) do
transaction(
fn ->
case fun.() do
{:ok, value} -> value
:ok -> :transaction_commited
{:error, reason} -> rollback(reason)
:error -> rollback(:transaction_rollback_error)
end
end,
opts
)
end
end
And that’s it! Now you know how to encrypt and upload files to S3. Now, let’s look at the downloading and decryption part.
Download Files
Our DownloadEncryptedFile service downloads the file using its resource_path and decrypts it. Here’s how:
defmodule Demo.Files.DownloadEncryptedFile do def call(current_user, file) do
with {:ok, encrypted_content} <- download(file),
{:ok, content} <- decrypt(file, encrypted_content, current_user) do
{:ok, content}
end
end
defp download(file) do
get_bucket()
|> ExAws.S3.get_object(file.resource_path)
|> ExAws.request(region: get_region())
|> case do
{:ok, %{body: body}} -> {:ok, body}
error -> error
end
end
defp decrypt(file, encrypted_content, current_user) do
key = Base.decode64!(current_user.encryption_key)
iv = Base.decode64!(file.encryption_iv)
encrypted_content = Base.decode64!(encrypted_content)
:aes_256_cbc
|> :crypto.crypto_one_time(key, iv, encrypted_content, false)
|> unpad()
|> Base.decode64()
end
def unpad(data) do
to_remove = :binary.last(data)
:binary.part(data, 0, byte_size(data) - to_remove)
end
defp get_bucket(), do: Application.get_env(:demo, :aws_file_bucket)
defp get_region(), do: Application.get_env(:demo, :aws_file_bucket_region)
end
The service first downloads the file from its file.resource_path and then decrypts it. The decryption looks very similar to the encryption, but in reverse and with a false-flag in the :crypto.crypto_one_time/5 function for decryption. If you look at the unpad/1 function, you see how we remove the last bytes that we used for padding during the encryption.
The service returns raw bytes, which we can then return from our Phoenix Controller and show in our <img> or <embed> elements.
Demo Time!
And that’s it! If you put all these pieces together, you’ll get a working demo for encrypting and uploading, and downloading and decrypting of user files to S3.
Conclusion
And that’s it! I hope you enjoyed this article! If you have questions or comments, let’s discuss them on BlueSky. Follow me on BlueSky or subscribe to my newsletter below if you want to get notified when I publish the next blog post. Until the next time! Cheerio 👋