Filter by Value, Not by Path
In this post, I use the example of a parcel-shipping gateway integrating with multiple carriers.
This example is purely hypothetical and chosen to clearly illustrate the technical idea.
Integrating with N shipping carriers often means dealing with N different JSON (if you’re lucky!) responses.
Each carrier’s responses can sprinkle customer PII into your logs—until your CISO sees it.
Most teams try to predict where the PII will be:
-
"recipient.email"
for Carrier A -
"ship_to.contact.email"
for Carrier B -
"error.stacktrace[0].ctx.email"
when Carrier C blows up 😱
That game never ends.
Let’s flip the problem: we already know the values before we send any request.
So let’s redact by value, not by path.
Step 1 — Our Canonical Parcel Struct
defmodule Parcel do
@enforce_keys ~w(id name email phone)a
defstruct [:id, :name, :email, :phone, :address, :items]
end
Everything downstream is derived from %Parcel{}
.
Step 2 — Collect the Secrets Once
defmodule Secrets do
@doc """
Accepts a Parcel struct and returns a list of
values that are secrets that we don't want to see in our logs.
"""
@spec from_parcel(Parcel.t()) :: [String.t()]
def from_parcel(%Parcel{} = p) do
[
p.name,
p.email,
p.phone,
p.address.street,
p.address.city,
p.address.postcode
]
|> Enum.reject(&is_nil/1)
end
end
Step 3 — A Drop-in Logger Helper
defmodule SafeLog do
require Logger
@doc """
Logs a payload at the given level, safely redacting any known secret values.
If the payload is a map, it is encoded as JSON before logging.
If the payload is a printable string, it is used directly.
"""
@spec scrub_and_log(
:debug | :info | :warn | :error,
String.t(),
map() | String.t(),
[String.t()]
) :: :ok
def scrub_and_log(level, message, payload, secrets)
def scrub_and_log(level, message, payload, secrets) when is_map(payload) do
payload
|> Jason.encode!()
|> redact_and_log(level, message, secrets)
end
def scrub_and_log(level, message, payload, secrets) when is_binary(payload) do
if String.printable?(payload) do
redact_and_log(level, message, payload, secrets)
else
Logger.warn("SafeLog attempted to log a non-printable binary, skipping.")
:ok
end
end
@spec redact_and_log(
:debug | :info | :warn | :error,
String.t(),
String.t(),
[String.t()]
) :: :ok
defp redact_and_log(level, message, text, secrets) do
redacted =
secrets
|> Enum.uniq()
|> Enum.sort_by(&String.length/1, :desc)
|> Enum.reduce(text, fn secret, acc ->
String.replace(acc, secret, "[FILTERED]")
end)
Logger.log(level, fn -> "#{message} #{redacted}" end)
end
end
Step 4 — Use It in an Adapter
defmodule ShippingExpressAdapter do
@doc """
Creates a shipping label by sending a request to the ShippingExpress API.
Logs the request and response with PII safely scrubbed out,
using the known secret values extracted from the Parcel struct.
"""
@spec create_label(Parcel.t()) :: {:ok, map()} | {:error, any()}
def create_label(%Parcel{} = parcel) do
secrets = Secrets.from_parcel(parcel)
request = build_request(parcel)
SafeLog.scrub_and_log(:debug, "ShippingExpress request ⇢", request, secrets)
with {:ok, raw_resp} <- HTTPClient.post(url(), request),
{:ok, resp} <- Jason.decode(raw_resp) do
SafeLog.scrub_and_log(:debug, "ShippingExpress response ⇠", resp, secrets)
parse_label(resp)
end
end
end
No carrier-specific filter modules.
If ShippingExpress decides to throw "InternalError: phone=123-456"
somewhere, it still gets nuked.
Considerations
Concern | Mitigation |
---|---|
Large payloads? | Replacing many secrets in large payloads could introduce small slowdowns. Benchmark if you process very large logs. |
Secrets overlapping or partial matches? | Deduplicate secrets (Enum.uniq/1 ) and sort them by descending length (Enum.sort_by/2 ) before replacing. This ensures longer secrets are redacted first, avoiding partial leaks (e.g., preventing "john.smith@example.com" from becoming "[FILTERED].smith@example.com" ). |
Encoding issues | If sensitive values are encoded (e.g., Base64) before being logged, simple string replacement won't find them. Make sure sensitive fields are logged in plain form, or exclude them entirely. |
If you can think of scenarios where this approach might fail, I'd love to learn from your experience.
Conclusion
Third-party APIs are unpredictable — but the data you send is something you control.
Building your filters around known values, rather than brittle paths, can make your systems more robust and easier to trust.
Top comments (1)
Nice Ivor! I notice Jason.encode being used. I wonder if you can get away with Jason.encode_to_iodata. Especially because Logger can work with iodata. Essentially, is it possible to do the replacement on the iodata elements just in time? Just a gut feel. It may not work. And you will have to benchmark.