Forem: Philip Hern

how i automated dev.to and linkedin publishing so visibility stops depending on memory

Philip Hern — Sun, 05 Apr 2026 14:03:19 +0000

after i started writing more consistently, it became obvious that writing is only half the work; distribution is the other half. i wanted a system where i can publish from one canonical source and let automation push the same story to dev.to and linkedin.

quick answer

i set up two publish automations that watch my post changes and sync them to dev.to and linkedin. the first publish creates the post on each platform, and later edits update the same external post instead of creating duplicates. this gives me consistent visibility without adding manual publishing steps after every article.

who this is for

people who publish technical writing and keep forgetting cross-posting
creators who want one canonical source plus repeatable distribution
builders who care about discoverability as much as writing quality

why this matters

if distribution is manual, it eventually slips. then strong posts sit unread because i forgot to copy, paste, format, and re-share them across platforms. automation solves that by making visibility part of the same delivery path as the content itself.

this is the same pattern i described in a practical ai workflow: jira, github, and mcp, define one clear source of truth, then automate the handoff steps so i can spend more time on thinking and less time on clerical work.

step-by-step

1) define the starting point

i chose my site post as the only canonical source. every external platform receives content from that source, not from separate drafts. this keeps language, links, and updates aligned over time.

2) apply the change

i added automation for both targets:

trigger on post updates and support manual runs when i want a full backfill
create posts when no external mapping exists
update existing external posts when a mapping already exists
keep a small state map so each canonical url stays attached to one external post id

the practical result is that i can keep writing in one place and trust the sync layer to handle distribution. this complements the writing habits from my cursor setup, where reusable workflows remove repeated manual work.

3) validate the result

i test in three passes:

dry run to confirm detection and decisions without publishing
publish-all run to verify initial backfill behavior
normal change-trigger run to verify incremental updates on later edits

when all three pass, i know the pipeline is reliable enough for daily use.

faq

what was the biggest setup mistake?

token and redirect mismatches during oauth were the main failure point at first. once i aligned scopes, callback values, and secret placement, the automation became stable.

should i keep manual publishing as a fallback?

yes, especially while you are in early setup. after the workflow proves stable, manual publishing becomes a recovery path instead of a default habit.

references

the future of data engineering workflows with ai

Philip Hern — Fri, 03 Apr 2026 14:11:59 +0000

quick answer

the future of data engineering workflows with ai is about moving from manual coding to intelligent orchestration. ai agents will handle boilerplate code, pipeline generation, and data quality checks, allowing data engineers to focus on architecture, governance, and business value.

who this is for

audience: data engineers, analytics engineers, data architects, and technical leaders.
prerequisites: an understanding of modern data stack concepts and basic ai principles.
when to use this guide: when planning your data strategy and evaluating how to integrate ai into your engineering practices.

why this matters

the volume and complexity of data are growing faster than engineering teams can scale. relying solely on manual workflows leads to bottlenecks, technical debt, and delayed insights. embracing ai is not just about efficiency, it is a strategic imperative to remain competitive.

step-by-step

1) define the starting point

traditionally, data engineering has been a highly manual discipline. engineers spend countless hours writing sql, configuring orchestrators like airflow, and debugging failed pipelines. this approach is brittle and scales poorly as the organization grows.

2) apply the change

the integration of ai changes this paradigm. large language models can now generate complex sql queries, translate between dialects, and even suggest optimal data models based on source schemas. ai agents can monitor pipeline health, automatically retry transient failures, and alert engineers only when human intervention is necessary. this shift transforms the engineer from a coder into a system architect.

3) validate the result

the impact of this transformation is measurable. development cycles shorten, data quality improves through automated testing, and the overall reliability of the platform increases. engineers spend less time firefighting and more time building scalable, resilient architectures that drive business decisions.

faq

what is the most important caveat?

ai is a tool, not a replacement for fundamental engineering principles. you still need a strong understanding of data modeling, governance, and security to build a robust platform.

what should i do first?

start by identifying the most repetitive tasks in your workflow, such as writing documentation or basic transformations. experiment with ai tools to automate these specific areas before attempting to overhaul your entire architecture.

references

the modern data stack

how i use cursor and ai agents to write dbt tests and documentation

Philip Hern — Fri, 03 Apr 2026 14:07:49 +0000

quick answer

writing dbt tests and documentation is often the most neglected part of data engineering. i use cursor and custom ai agents to automate this process by reading my sql models, inferring the business logic, and generating the corresponding yaml files. this ensures high-quality data pipelines without the manual overhead.

who this is for

audience: data engineers, analytics engineers, and developers using dbt
prerequisites: basic knowledge of dbt, sql, and cursor
when to use this guide: when you want to scale your data engineering practices and reduce the time spent on writing boilerplate yaml

why this matters

documentation and testing are critical for data trust, but they are tedious to write manually. when these steps are skipped, data quality suffers and debugging becomes a nightmare. by automating this with ai, you get the benefits of rigorous testing and clear documentation while freeing up your time for higher-value architectural work.

step-by-step

1) define the starting point

most data engineers start with a raw sql model and a blank slate for their schema.yml file. the traditional approach requires manually typing out every column name, description, and test. this is prone to human error and inconsistency, plus almost always falls out of sync with current models with the first change.

2) apply the change

i use cursor to bridge this gap. by creating specific ai rules and skills, i can highlight a dbt model and ask the agent to generate the documentation. the agent reads the sql, understands the joins and transformations, and produces a complete yaml file with standard tests like not_null and unique. it can even infer complex relationships and suggest custom tests based on the data domain.

3) validate the result

once the ai generates the yaml, i review it for accuracy. i then run dbt test and dbt docs generate to ensure everything compiles correctly. the ai rarely makes syntax errors, so the validation step is mostly about confirming the business logic aligns with the documentation.

faq

what is the most important caveat?

you must still review the generated output. ai is excellent at scaffolding and inferring patterns, but it does not possess the full business context that you do.

what should i do first?

start by creating a simple cursor skill that defines your team's standards for dbt documentation. feed it a few examples of your best schema.yml files so it learns your preferred style.

references

dbt documentation

sharing is caring

Philip Hern — Thu, 02 Apr 2026 12:11:05 +0000

the value of early adoption

i have always found a unique kind of energy in being an early adopter. when a new tool emerges, especially something as transformative as cursor and artificial intelligence, diving in headfirst is not just about personal efficiency. it is about understanding the landscape before the map is fully drawn. by spending the hours required to become a high-level user, i build a deep familiarity with the edges of what the technology can do.

this mastery translates directly into value for my colleagues. when you understand the high-level nuance of a complex tool, you naturally become the point person for your team. people have onboarding questions, they hit roadblocks, and they need someone who has already navigated those early frustrations. being that resource is incredibly rewarding. it shifts my role from an individual contributor to a multiplier, helping the entire team elevate their workflow and avoid the pitfalls i have already solved.

the responsibility to share

this dynamic reminds me of a principle i have heard often over the years regarding the importance of using your voice and your platform to share. this is exactly why i started this website. i wanted a dedicated space to share my voice, my knowledge, my opinions, my experience, and the solutions i have discovered along the way.

when you hold onto knowledge, its impact is limited to your own output. when you share it, the impact scales infinitely. writing about these tools, documenting my workflows, and answering the nuanced questions my colleagues ask are all extensions of the same core belief. knowledge is meant to be distributed.

stepping into mentorship

i have reached a point of mastery and experience where the natural next step for me is to mentor others and deliberately increase my visibility and presence. it is no longer enough to simply be good at what i do behind the scenes. the real work now is in lifting others up.

in fact, i am starting to feel the weight of this realization. it feels almost selfish not to share what i have learned. when you spend years honing a craft or mastering a paradigm-shifting tool like ai-assisted development, you accumulate a wealth of invisible context. keeping that context locked away serves no one. stepping into a mentorship role, both directly with my colleagues and publicly through this platform, is how i honor the effort it took to gain that experience in the first place.

looking forward

my goal is to continue exploring the bleeding edge of these tools, but with a renewed focus on how i can translate those discoveries into accessible guidance for others. whether it is through answering a quick onboarding question about cursor, writing a detailed guide on this site, or simply being a sounding board for a colleague, the objective remains the same. i want to use my experience to make the path easier for those who follow.

related on this site

post title

what is art?

Philip Hern — Mon, 30 Mar 2026 23:38:30 +0000

thesis

i keep pondering lately, what are we actually defending when we say "ai art is not real art"?

i do not have a final position yet. i am writing this to think in public, not to close the debate.

context

while driving on a family vacation, i asked my wife to fulfill her duty as the passenger and dj some motown bangers. she searched on spotify and found something that seemed to fit the bill. most of the songs were recognizable, memories from my childhood, riding in the backseat listening to my parents' favorites. however, the first song on the playlist was by an artist called the 19s soulers, which was an artist i did not recognize. this was a user created playlist so not everything might fit perfectly into the motown mold i was asking for, and that was ok. the song started and it was SOOO good. TOO good. i had my suspicions, but the music caught me so hard that i completely forgot. i asked my boys in the back seat to look up the artist and they did not even search and just responded "AI DAD - IT IS AI". i felt so many conflicting emotions, including one of pride that my boys could tell the difference and they have some defense against being fooled.

the conversation around ai-generated images and music feels hotter every week, especially when a new ai music act gets attention or a contract. the reaction is often immediate and predictable outrage, fear, dismissal, and arguments about stolen style.

at the same time, many of us use ai to help write code, review pull requests, or shape architecture notes without the same emotional response. that contrast is interesting to me.

if i call code a craft, and sometimes an art form, then why does ai help feel acceptable there for so many people, but unacceptable when the ai helps write song lyrics? and if code can be expressive, why is the outrage concentrated in painting, illustration, and music.

my code has my fingerprints all over it, just as much as this website and the way i speak and write. it defintely qualifies as expressive. i make stylistic, logic, function, etc. choices that suit my style. how is this different from writing a book? but if you asked which one is acceptable to use ai and which one is not, i could guess your answer 99% of the time.

argument

i see a few possible reasons, and none of them feel complete on their own:

visual art and music are tied to identity in a very direct way
audiences often connect to the "maker story", not only the artifact
creative labor markets in those fields already felt fragile before ai
software teams have normalized tool-assisted output for decades
code is often judged by function first, while art is judged by intention and feeling

still, even with those differences, i cannot shake the inconsistency.

when i use ai in code, i still feel like the author because i set constraints, reject bad output, and own the result. i do not think that is very different from guiding a visual generator, editing outputs, and curating a final piece. maybe the difference is only social permission, not creative mechanics.

this question also links to my concern about ownership in the danger of trusting the ai agent, where speed is useful but responsibility still has to stay human.

tension or counterpoint

there is also a strong counterpoint i take seriously: in code, wrong answers fail in visible ways. tests fail, services break, users complain, and teams can trace accountability. in art, value is less binary, and that makes authorship feel more central and more vulnerable.

another counterpoint is economic, not philosophical. people may not be reacting to "is this art" at all. they may be reacting to "will this replace my livelihood".

both of those points feel real to me.

and i think the later point is one worth exploring because the wide-spread availability of ai has "democratized" creativity, technical endeavors, etc. for people who might have great ideas, but not the musical or technical skill to carry out the plan. well, now they do. and that instant competition that was not present before can certainly feel intimidating and encroaching.

i am currently mostly pro-ai, but with caution. we should have caution regarding how the models are being trained (and on what data) and regulated. we should exercise caution surrounding who is doing the regulating, as well. ai is a powerful assistant, and as we all know from spiderman, with great power comes great responsibility.

closing

i am left with questions, not conclusions.

maybe we value human touch most where we believe the human story is the product. maybe we accept ai more where we believe the product is utility. maybe those boundaries are changing and we are all reacting in real time.

for now, i am trying to keep the question open.....when ai is part of the process, what still makes something mine, yours, or ours?

related on this site

dbt tests

Philip Hern — Mon, 30 Mar 2026 23:34:36 +0000

we all know testing is valuable, but almost all dbt projects still underinvest in it. i am guilty of this, shipping the model, promising i will add tests later, then moving on to the next urgent request.

that pattern feels fast in the moment, but it is expensive over time. dbt tests are one of the easiest ways to protect trust in your data, and the setup cost is usually smaller than people expect.

quick answer

dbt tests are assertions about your data that run inside your transformation workflow. they verify assumptions like uniqueness, non-null keys, valid categorical values, and referential integrity. when a test fails, dbt surfaces the exact failing records so you can debug quickly. if you are new to the feature, start with the official dbt data tests documentation and add a few high-signal tests to your most consumed models first.

who this is for

analytics engineers who already build dbt models but still rely on manual spot checks
data teams that have recurring data quality incidents in dashboards or reports
anyone who wants a practical starting point instead of a perfect testing framework

why this matters

the cost of bad data is usually delayed, not immediate. a broken metric can sit in production for days before someone notices, and by then that number has already been used in a deck, decision, or leadership update.

the frustrating part is that most of these issues are predictable. duplicate primary keys, null foreign keys, unexpected status values, and invalid date ranges are all common failures. dbt tests can catch these early, near the model that introduced the issue.

i also think testing helps teams move faster, not slower. when tests are in place, i can refactor a model with more confidence because i have a safety net. without tests, every change feels risky and review cycles become slower because everyone is relying on intuition.

step-by-step

1) define the starting point

pick one model that is heavily consumed, for example an order fact table or a customer dimension. identify three assumptions that must always be true:

the model key is unique
important keys are never null
status fields only contain known values

then encode those assumptions directly in your yml.

2) apply the change

start with generic tests in your schema file.

version: 2

models:
  - name: fct_orders
    columns:
      - name: order_id
        tests:
          - not_null
          - unique
      - name: customer_id
        tests:
          - not_null
          - relationships:
              to: ref('dim_customers')
              field: customer_id
      - name: order_status
        tests:
          - accepted_values:
              values: ["placed", "shipped", "cancelled", "returned"]

this single block gives you strong baseline coverage:

not_null protects required fields
unique protects grain
relationships protects joins and referential integrity
accepted_values protects enum-like business states

next, add one singular test for a business rule that generic tests cannot express cleanly.

-- tests/orders_non_negative_amount.sql
SELECT
    order_id,
    order_amount
FROM {{ ref('fct_orders') }}
WHERE order_amount < 0

this test fails only when the query returns rows. singular tests are ideal for custom rules like range checks, cross-column logic, and impossible combinations.

3) validate the result

run tests in a tight loop while you are developing:

dbt test --select fct_orders

for a broader gate in CI or before merging, run:

dbt build --select fct_orders+

dbt build runs models and tests together, which is useful when you want to validate both transformation logic and data quality in one pass. the + after the model name tells dbt to also run any downstream models according to the dag to make sure dependencies also pass.

a practical prioritization rule

when time is limited, i prioritize tests in this order:

key integrity on high-consumption models (unique plus not_null)
foreign key integrity (relationships) on joins that power dashboards
controlled fields (accepted_values) where business logic depends on a finite set of values
one custom singular test for the highest-risk metric or business rule

this sequence catches a large share of real incidents with minimal setup.

faq

what should i test first in a mature project with little coverage?

start where breakage is most expensive, not where modeling is most elegant. choose one or two heavily consumed models and add key integrity plus relationships first. then add one singular test for the business rule that has caused the most historical pain.

do dbt tests slow down delivery?

they add some upfront work, but they usually reduce cycle time later. test failures are cheaper during development than after release, and tests make refactors safer because you can verify assumptions continuously instead of rediscovering issues in production.

references

dbt docs

Philip Hern — Mon, 30 Mar 2026 11:39:08 +0000

most data engineers i know will spend hours getting a model right, then skip the one step that makes it discoverable to everyone else. dbt docs are that step, and they are worth the effort.

quick answer

dbt docs is a built-in feature that generates a browsable website from your dbt project. it pulls descriptions from your yml files, renders a searchable model catalog, and draws a lineage graph showing how every model connects. running dbt docs generate followed by dbt docs serve gives you a local site instantly. the real payoff is that teammates who never open your sql files can still understand what each model does, what columns it exposes, and where the data comes from. this is especially useful to downstream consumers of your objects because they can see the exact format of the object, including column names and data types, etc.

who this is for

data engineers who use dbt but skip writing descriptions
analysts, product managers, or business users who need to understand available data without reading sql
team leads looking for a low-effort way to make data structures discoverable

why this matters

when you build a dbt project, the models represent real business concepts. customers, products, sales, inventory, whatever the domain is. the people who consume those models in dashboards or reports often have no involvement in building them and no reason to read raw sql.

without documentation, those consumers rely on tribal knowledge, slack messages, and guesswork. that does not scale. dbt docs solve this by turning the metadata you already maintain (yml files, project config, source definitions) into a navigable reference that anyone on the team can use. the effort to write a good description is small, and the compound value to the rest of the organization grows with every model you add.

i think of it like this: if i write a model and do not document it, the only person who truly understands it is me, and even that fades after a few months. if i write a two-sentence description and add column-level context, that knowledge lives in the project permanently and serves everyone who touches the data.

what dbt docs generates

when you run dbt docs generate, dbt produces two main artifacts in your target/ directory:

manifest.json contains the full project graph, including every model, source, seed, snapshot, and macro, along with their descriptions, tags, and configuration
catalog.json contains the schema-level metadata pulled from your warehouse, including column names, data types, and row counts

together with a bundled index.html, these files power a static site that you can open locally with dbt docs serve or host anywhere that serves static files.

the site includes

a searchable list of every model and data source in your project
model-level and column-level descriptions pulled from your yml files
the full sql compiled for each model (in each environment)
a lineage graph (dag - directed acyclic graph) that shows upstream sources, intermediate models, and downstream consumers for any selected node

the lineage graph is especially useful when someone asks "where does this column come from" or "what breaks if i change this source table". instead of tracing through sql files manually, the graph answers it visually.

how to document your models

dbt reads documentation from yml files that live alongside your models. you are probably already using these for configuration and source definitions, so adding descriptions is a natural extension.

model and column descriptions in yml

the most common approach is adding description fields directly in your yml files. here is what that looks like for a view in an information delivery layer:

version: 2

models:
  - name: ORDER_SUMMARY_V
    description: >
      aggregated view of customer orders with totals and status
      breakdowns, consumed by the reporting dashboard and the
      customer details page in the application.
    columns:
      - name: CUSTOMER_ID
        description: "unique identifier for the customer"
      - name: TOTAL_ORDERS
        description: "count of all orders placed by this customer"
      - name: TOTAL_SPEND
        description: "sum of order amounts across all completed orders"
      - name: LAST_ORDER_DATE
        description: "most recent order date for this customer"

every model and every column can have a description. the more specific you are, the more useful the generated docs become. "id" as a column description does not help anyone. "unique identifier for the customer, sourced from the application database" does.

source descriptions

sources benefit from the same treatment. when your project ingests raw data from an external system, describing those sources in your shared sources file makes the lineage graph meaningful from the very first node:

version: 2

sources:
  - name: RAW_ORDERS
    description: "raw order data ingested from the transactional database via kafka"
    database: ANALYTICS_DEV_DB
    schema: RAW_DATA
    tables:
      - name: ORDERS
        description: "one row per order, includes order status and timestamps"
      - name: ORDER_ITEMS
        description: "line items for each order, one row per product per order"

doc blocks for longer descriptions

when a model needs more than a sentence or two of context, dbt supports doc blocks. these are markdown files (.md) that live in your project and can be referenced from yml descriptions:

{% docs order_summary_description %}

this view surfaces the aggregated order history for each customer.
it joins the hub, satellite, and link tables from the refined data
layer to produce a single wide row per customer.

**grain:** one row per customer.

**consumers:** reporting dashboard, customer details api endpoint.

{% enddocs %}

then in your yml:

models:
  - name: ORDER_SUMMARY_V
    description: '{{ doc("order_summary_description") }}'

doc blocks are useful when the context is long enough that embedding it inline in yaml becomes awkward. they also let you reuse the same description across multiple references if needed.

a few useful options in dbt docs

persist_docs

by default, dbt docs only live in the generated static site. if you want the descriptions to also appear in your warehouse catalog (so someone querying snowflake information_schema can see them), you can enable persist_docs:

models:
  my_data_product:
    +persist_docs:
      relation: true
      columns: true

with this enabled, dbt run pushes your yml descriptions into the COMMENT property on the table or view and on each column in the warehouse. this is valuable because it means the documentation is available even outside the dbt docs site, directly in the database catalog that tools like snowflake and bi platforms already read.

the lineage graph

the generated site includes an interactive dag that visualizes every model, source, and their connections. you can click on any node to see its upstream dependencies and downstream consumers. this is one of the most powerful features in dbt docs because it makes the data flow tangible for people who do not read sql (or maybe they just do not have access to your workspace, but still have a need to understand it).

when you have a project with dozens or hundreds of models organized into layers (raw, refined, business, information delivery), the lineage graph shows how a raw source table flows through transformations into the final views that analysts query. it replaces the need for manually maintained architecture diagrams that go stale the moment someone adds a new model.

exposures

exposures let you document where your dbt models are consumed outside of dbt. dashboards, applications, api endpoints, anything downstream. defining them makes the lineage graph extend beyond the dbt project boundary:

exposures:
  - name: customer_dashboard
    type: dashboard
    description: "executive dashboard showing customer order trends and retention"
    depends_on:
      - ref('ORDER_SUMMARY_V')
      - ref('CUSTOMER_RETENTION_V')
    owner:
      name: analytics team
      email: analytics@example.com

exposures show up in the lineage graph as leaf nodes, making it clear which models are actively consumed and by what. this is helpful when you are deciding whether it is safe to refactor or deprecate a model.

hosting dbt docs on github pages

running dbt docs serve is great for local browsing, but the real value comes from hosting the site where the whole team can access it without installing dbt or cloning the repo. github pages is a straightforward, free option for this.

how it works

after dbt docs generate runs, the target/ directory contains everything needed to serve the site: index.html, manifest.json, and catalog.json. you copy those files to a branch or directory that github pages serves, and the docs are live.

a basic github actions workflow

here is a minimal workflow that generates the docs on every push to main and deploys them to github pages:

name: deploy dbt docs

on:
  push:
    branches: [main]

permissions:
  contents: read
  pages: write
  id-token: write

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    steps:
      - uses: actions/checkout@v4

      - name: set up python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: install dbt
        run: pip install dbt-snowflake

      - name: generate docs
        run: dbt docs generate --profiles-dir .
        working-directory: my_dbt_project
        env:
          DBT_SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }}
          DBT_SNOWFLAKE_USER: ${{ secrets.SNOWFLAKE_USER }}
          DBT_SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}

      - name: prepare pages artifact
        run: |
          mkdir -p pages
          cp my_dbt_project/target/index.html pages/
          cp my_dbt_project/target/manifest.json pages/
          cp my_dbt_project/target/catalog.json pages/

      - name: upload pages artifact
        uses: actions/upload-pages-artifact@v3
        with:
          path: pages

      - name: deploy to github pages
        id: deployment
        uses: actions/deploy-pages@v4

once this runs, your dbt docs are available at https://<org>.github.io/<repo>/ and automatically update every time someone merges to main. no one needs to install anything or run any commands to browse the documentation.

keep the credentials out of the repo

the workflow above uses github secrets for warehouse credentials. never commit profiles with real credentials. use environment variables or a ci-specific profiles.yml that references secrets, and make sure your .gitignore excludes any local profiles that contain actual passwords or tokens.

faq

do i need to write descriptions for every column?

you do not need to, but the columns that matter most to consumers deserve it. at minimum, describe the primary key, any business key, and any column whose meaning is not obvious from the name alone. over time, filling in the rest pays off as the team grows.

can i generate docs without connecting to the warehouse?

dbt docs generate pulls catalog metadata from the warehouse, so it does need a connection. however, if you already have a catalog.json from a previous run, you can serve the site locally with just those files.

how is this different from a wiki or confluence page?

dbt docs stay in sync with your code automatically. a wiki page about your data model goes stale the moment someone adds a column or renames a table. dbt docs regenerate from the source of truth (your yml files and your warehouse) every time you run the command, so the documentation and the code never drift apart.

references

what is sql, and why it still works

Philip Hern — Sat, 28 Mar 2026 12:48:02 +0000

i wanted this to be an entry point for the sql series, a clear "what is sql" reference you can scan quickly and return to later. even with constant technology shifts, relational databases and sql still anchor a huge part of modern software and analytics.

quick answer

sql is the language most relational databases use to define, read, update, and manage data. it is declarative, which means i describe the result i want and the database engine decides how to execute it efficiently. this model has held up for decades because sql is expressive, portable, and understandable. vendor dialects differ, but the core grammar is still recognizable almost everywhere.

who this is for

people new to data and databases who want a quick introduction
developers who use databases daily but never got the historical context
analysts and engineers deciding when sql is the right tool

what sql is

at its core, sql works on tables made of rows and columns. tables are connected by keys, and queries combine or reshape those tables to answer questions.

sql usually appears in four practical categories:

querying data with select, where, join, group by
modifying data with insert, update, delete
defining structures with create table, alter table, create view
controlling access and safety with grant, revoke

the most important mindset shift is this: sql is set-based and declarative, not loop-first and imperative.

a short history of sql

sql has been around for more than 50 years, and that long runway is one reason it feels "locked in" as a standard.

1970: edgar f. codd publishes the relational model at ibm
mid 1970s: ibm researchers develop sequel, which later becomes sql
1979: oracle ships one of the first commercial sql database implementations
1986: ansi publishes the first sql standard (i personally ONLY write ansi standard for maximum portability)
1990s to 2000s: major vendors expand features and performance, while the core language stays recognizable
today: sql powers operational databases, analytics warehouses, bi tools, and transformation frameworks

why sql is important

sql is still important for practical reasons, not nostalgia.

it is a common language shared by developers, analysts, and data engineers
it is standard enough that skills transfer across many systems
it lets database engines optimize queries with indexes and planners
it models relationships clearly through keys and joins
it supports transactional consistency for critical business data

for me, one of the biggest advantages is ease of learning. for common querying and reporting tasks, sql usually has a lower starting barrier than learning a full general-purpose programming language like python.

when and where to use sql

sql fits especially well in these environments:

application databases like postgresql, mysql, and sql server for transactional systems
analytics platforms like snowflake, where large-scale reporting and transformation matter most, i cover that context more in the difference between snowflake and the "other" databases
data transformation workflows where repeatable models and tests are required
reporting and business intelligence when teams need consistent, auditable metrics

sql is usually not the best first tool for heavy imperative algorithms, event orchestration, or specialized graph traversal.

if you want a practical comparison between snowflake and operational database systems, i break that down in the difference between snowflake and the "other" databases.

key features worth knowing early

joins to combine related entities through keys
aggregations (sum, count, avg) to move from row-level detail to summaries
window functions for ranking, running totals, and partitioned calculations
transactions for safe multi-step writes (begin, commit, rollback)
constraints (primary keys, foreign keys, uniqueness) to enforce data quality
views and ctes to make complex logic readable and reusable

a few simple examples

filter rows

SELECT
  order_id,
  order_date,
  total_amount
FROM orders
WHERE order_date >= DATE '2026-01-01';

join related data

SELECT
  c.customer_name,
  o.order_id,
  o.total_amount
FROM orders o
JOIN customers c ON c.customer_id = o.customer_id;

aggregate to a useful metric

SELECT
  customer_id,
  SUM(total_amount) AS total_spend
FROM orders
GROUP BY customer_id
ORDER BY total_spend DESC;

use a transaction for safe updates

BEGIN;
UPDATE accounts
SET balance = balance - 100
WHERE account_id = 1;
UPDATE accounts
SET balance = balance + 100
WHERE account_id = 2;
COMMIT;

faq

is sql the same thing as a database?

no. sql is a language, and a database engine is the system that executes it. two engines can both support sql while differing in extensions, performance behavior, and tooling.

what is the fastest way to get good at sql?

i suggest learning in this order: filtering, joins, aggregation, then window functions. if you can always explain the grain of one row in your result, your queries become much easier to trust.

when should i think twice about using sql?

if your core problem is orchestration, low-latency event handling, or algorithm-heavy logic, sql alone is usually not enough. in those cases, use sql for data access and pair it with application or workflow code.

references

the difference between snowflake and the "other" databases

Philip Hern — Sat, 28 Mar 2026 11:49:27 +0000

when you first step into data engineering, the sheer number of database options can be overwhelming. i spend a lot of my time working in snowflake, but it is definitely not the only tool in the shed.

to build a solid data platform, you have to understand where your data comes from and how different systems handle it. i want to break down how snowflake compares to two other popular systems you will encounter often: relational engines such as amazon rds and nosql patterns such as amazon dynamodb.

this is a basic guide written from a data engineer's perspective. i will point out the similarities, the differences, and when you should use each one.

the contenders

before we compare them, let us define what we are looking at:

snowflake: a cloud-native data warehouse built for analytics (olap - online analytical processing). storage and compute are separate, so you pay for each part independently
amazon rds: a managed relational database service that runs engines like postgresql, mysql, sql server, and oracle. it is built for transactional app workloads (oltp - online transactional processing)
amazon dynamodb: a fully managed nosql key-value and document store built for very low-latency lookups at high scale

similarities and differences

at a high level, snowflake and rds both use sql, which makes them feel familiar. dynamodb is api-first for key-value access. it also offers partiql support, but it does not behave like a relational sql engine with joins and broad ad hoc querying.

the biggest architectural difference is how they handle storage and compute.

feature	snowflake	amazon RDS	amazon dynamodb
primary use case	analytics and reporting (olap)	application backends (oltp)	high-scale key-value applications
query language	sql (warehouse dialect)	sql (engine-specific)	api-first (partiql support)
architecture	storage and compute decoupled	storage and compute coupled	distributed key-value store
data structure	tables and semi-structured data	relational tables	key-value and document items
indexes	micro-partition pruning metadata	user-managed indexes	partition and sort keys

snowflake does not require traditional user-managed indexes for most workloads. it uses micro-partitions and pruning metadata. in rds, you usually create and manage indexes manually to keep queries fast. dynamodb pushes you to define access patterns up front using partition and sort keys.

scalability: small, big, and the limits

scalability means something completely different depending on which database you are talking about.

snowflake

snowflake scales compute and storage independently. storage is backed by cloud object storage, so it is effectively unbounded for most teams. you can store petabytes of data without thinking about disks.

compute is handled by virtual warehouses, which come in t-shirt sizes from x-small to 6x-large. if a query is too slow, you can use a larger warehouse. if you have too many concurrent users, you can add clusters.

small: gigabytes of data running on an x-small warehouse
big: petabytes of data running on a 4x-large warehouse
limits: practically none for storage; compute is limited only by your budget

amazon RDS

rds usually scales write capacity vertically by moving to a larger instance class. depending on engine and change type, this can trigger a restart or failover window. you can also scale read traffic horizontally with read replicas.

small: a few gigabytes on a t3.micro instance
big: several terabytes on a massive m6g.16xlarge instance
limits: many engines cap around 64 terabytes per instance (roughly 64 tib), and you still hit single-instance ceilings on cpu, memory, and concurrent connections

amazon dynamodb

dynamodb scales horizontally and automatically. it partitions your data across many servers behind the scenes.

small: a few megabytes with single-digit read/write capacity units
big: hundreds of terabytes handling millions of requests per second
limits: very high scale for key-value access, but full-table scans and broad analytical queries are usually expensive and inefficient

ease of learning curve and use

as a data engineer, the learning curve dictates how fast you can deliver value.

snowflake (easiest for analysts)

if you know sql, you can use snowflake quickly. the learning curve is low because there is very little infrastructure to manage. you do not worry about vacuuming tables, managing indexes, or tuning memory pools. you mostly load data and query it.

amazon RDS (moderate)

rds is easy to start with because postgresql and mysql are industry standards. however, the learning curve gets steep when you hit scale. you have to read query plans, manage indexes, handle connection pooling, and tune database parameters.

amazon dynamodb (steepest for modeling)

dynamodb has a notoriously steep learning curve for data modeling. because it is not a relational join engine, you usually model around known access patterns (often with single-table design). if those patterns change later, you may need redesign work and backfills.

when to use each

choosing the right database is about matching the tool to the workload.

use snowflake when:

you need to analyze massive amounts of data
you are building dashboards, reports, or a data warehouse
you need to join data from many different sources
your queries read millions of rows at a time

use amazon RDS when:

you are building a standard web application
you need strong transactional guarantees (ACID compliance)
your queries typically look up or update a single row or a small batch of rows
your core workload is transactional and fits a single relational instance pattern, even as it grows into multi-terabyte range

use amazon dynamodb when:

you are building a shopping cart, session store, or gaming leaderboard
you need guaranteed single-digit millisecond response times
you have massive, unpredictable spikes in traffic
you do not need to run complex analytical queries

as a data engineer, your job is often to extract data from operational systems (like rds and dynamodb) and load it into an analytical system (like snowflake). that is what allows the business to understand what is happening.

references

adaptability

Philip Hern — Fri, 27 Mar 2026 16:45:05 +0000

thesis

i think adaptability is one of the most useful skills to develop, and its value keeps rising. ai and agents are changing workflows faster than most teams can document them. people who adjust quickly, while staying clear about outcomes, will keep moving when others stall.

context

widespread ai adoption is pushing change into almost every part of work. tasks are getting split differently, handoffs are being redefined, and the idea of "the normal way" has a shorter shelf life every month.

this pace is not only a work story. speed and connectivity shape family logistics, routines, and attention in daily life too. the same mental flexibility that helps at work helps at home, in travel, and during any week that does not look like the one before it.

argument

i am writing this from a very rural location, far from my normal office setup. i have no cell service except a satellite connection that can send slow texts, and the wifi is around 10 mbps down on a good day, shared across the whole family. that setup forced me to plan work in batches and communicate constraints early. it is a useful reminder that conditions change, but the work still has to get done.

for me, adaptability in practice looks like a few simple moves:

shifting to an offline-first workflow when connectivity is inconsistent
prioritizing high-leverage tasks before synchronous calls
communicating constraints early so expectations stay realistic
batching sync-heavy work into the most stable bandwidth windows
choosing tools that degrade gracefully instead of failing hard

this is also why i care so much about clear ownership when using ai. adaptable does not mean random. it means adjusting method while protecting intent, quality, and accountability in a rapidly changing workplace (and world). this is a theme i also touched on in the danger of trusting the ai agent.

tension or counterpoint

there is a valid concern here: constant adaptation can turn into reactive thrashing. not every new tool deserves adoption, and not every workflow change creates real value.

the balance i aim for is to adapt principles faster than (or at least as fast as) tools. if i stay anchored on outcomes, quality, and ownership, i can (hopefully) swap tactics without losing direction.

closing

i do not see adaptability as a soft trait anymore. i see it as operational leverage. it helps me keep momentum when the environment is unstable, when tools change quickly, or when ideal conditions are not available.

going forward, i expect this to matter even more. the people and teams that learn quickly, reframe constraints, and keep moving with clarity will have a durable advantage.

related on this site

the danger of trusting the ai agent

Philip Hern — Thu, 26 Mar 2026 13:31:57 +0000

i love how fast ai agents can move, but i have learned that speed without ownership can become expensive very quickly. if i let an agent operate in an area i do not understand deeply, i can end up with changes i cannot explain, verify, or recover confidently.

this is the darker side of the acceleration story i wrote about in from prototype to production: my early adopter view of ai: ai can compress execution time, but it can also greatly expand confusion if my boundaries are weak.

quick answer

if i trust the ai fully in a domain i do not understand, i am borrowing speed against future troubleshooting debt. the agent can still produce plausible progress, but when something breaks, i pay the bill because i do not have the mental model to debug with confidence. my rule now is simple: ai can do my work faster, but it should only do work i truly understand and could perform on my own.

who this is for

people using agents for coding, data, or operations tasks
anyone who has watched an agent "do a lot" but struggled to explain what actually changed
teams where responsibilities cross domains and ownership can get blurry

why this matters

automation bias is real. when the output looks polished, it is easy to accept it before understanding it. that is manageable in low-risk tasks, but dangerous in high-impact systems where small assumptions can trigger hard-to-trace regressions.

the hidden cost is confidence drift. git can say everything is clean while my own understanding says something feels off. when that gap appears, stress goes up, debugging slows down, and trust in the workflow drops.

and one other note to be fair is that in this case, i tried to use a faster, smaller agent to do work that i should have routed to a stronger, deeper thinking model. i was in a hurry and paid for it. remember the tried and true racing phrase "slow is smooth, smooth is fast".

the failure mode i hit

i asked an agent to diagnose an error in an area where i did not have strong depth. during the run, it created files to handle the issue. later, it removed those same files because they were not referenced anywhere. from git's perspective, it was a net zero diff.

from my perspective, it was not zero at all. i watched a long stream of activity, expected to see resulting changes, and then found a clean tree. i spent significant time retracing the run to understand what happened, and even after reconciling local and remote state i still had low confidence that everything was truly back to a known good place.

where i drew the line

the lesson for me is less "never trust ai" and more "do not outsource something you could not do yourself". if i cannot explain the system, i should not delegate high-autonomy changes in that system to an agent.

that lane discipline applies to people too. if work sits in another colleague's domain, i should route it to them, even if they use an agent themselves. the difference is not whether ai is involved. the difference is whether the person driving understands the domain deeply enough to verify and own the result.

the lane check i use now

before i hand work to an agent, i run five checks:

do i understand the domain well enough to review every change with confidence
if the agent makes a wrong assumption, can i detect it quickly
do i have explicit stop conditions and verification steps
will i review the actual diff and command output, not just the narrative in chat
if this crosses domain ownership, have i handed it to the right colleague

if i answer "no" to any of these, i narrow the scope or route the task to someone else.

faq

does this mean ai should only do trivial work?

no. ai can do serious work, but the owner still needs to understand the system and sign off on the result. complexity is fine, unowned complexity is the problem.

what do you do when git says clean but confidence is low?

i treat that as a process warning. i retrace the execution log, compare expected outcomes against actual artifacts, and document what happened before continuing. if i still cannot explain it clearly, i escalate to the domain owner instead of pushing forward on uncertainty.

references

plane wifi: when the cabin forced disconnect

Philip Hern — Tue, 24 Mar 2026 16:23:21 +0000

thesis

for a long time, a commercial flight was one of the few places where the world could not reach you unless you paid for a seatback phone nobody used. you were offline by default, not by discipline. now many cabins offer wifi strong enough to treat the plane like a slow office in the sky. that is a small technical change and a large cultural one, because it turns disconnection from a fact into a choice, and sometimes into an expectation. i used to think we should protect those disconnected spaces on purpose. today i am less sure of my own opinion, and this piece is an honest inventory of why.

context

the shift did not happen all at once. first it was email that barely worked, then messaging, then enough bandwidth that "i am on a plane" stopped being a credible excuse for silence in some workplaces. the option to connect is not the same as the obligation to connect, but options have a way of becoming norms. if the team assumes you can answer, the cost of opting out is social and professional, not just the price of the wifi pass.

i notice that i feel different about this than i did five or ten years ago. i used to romanticize the cabin as a rare enforced pause, a moving room where the only honest move was a book, a nap, or staring at clouds. now i sometimes have a different perspective since my brain is already racing and the seat is uncomfortable and sleep is not going to happen.

argument

the case for keeping flights disconnected

there is real relief in a few hours where slack cannot ping you and the news cannot refresh on reflex. the stream of messages and media is designed to feel urgent. a metal tube at thirty thousand feet used to interrupt that design for everyone equally. when you land, nothing has changed except you, and that can be restorative even when it is boring.

i still believe that boredom and idle time are inputs to thinking, not failures of entertainment. a flight without wifi can be a long, low-stakes walk for your attention, and that has value. it is the same thread i explored in brain defrag: time away from screens (and from "one more" with ai).

the case for wanting the connection

if you cannot sleep on planes, hours in the air can feel like time you are borrowing from your life and not getting back. in that frame, wifi looks less like an intrusion and more like a way to reclaim the block for work, reading you actually chose, or staying in touch with people you care about. the "wasted time" feeling is personal, but it is not irrational.

cabin class changes how physical that tradeoff is. if you are fortunate enough to sit in first or business, the tray, the elbow room, and the seat pitch can make typing tolerable for a while. in economy, the same work is often inconvenient for you and unfair to the person beside you who did not sign up to be your armrest and your privacy screen. connectivity does not create that squeeze, but it can intensify it when everyone tries to turn a narrow row into an office.

tension or counterpoint

the strongest counterpoint to my old "protect the disconnected cabin" instinct is that disconnection was never equally available. people with caregiving responsibilities, unpredictable schedules, or thin margins already paid a tax when flights were black holes on the calendar. wifi can reduce that tax. my nostalgia for a universal offline bubble was partly a privilege story dressed up as a wellness argument.

the strongest counterpoint to always-on flying is that the cabin is still a shared space, and not every kind of work belongs there without cost to neighbors and to your own nervous system. the technology says you can but your body and the person in the middle seat might wish you would not.

closing

i do not have a tidy answer about whether i like in-flight connectivity. some trips i am grateful for it. some trips i wish the excuse to be unreachable still existed without me having to defend it. what i know is that the choice is heavier than the toggle in the portal makes it look, and i will probably keep revisiting it every time the wheels leave the ground.

Forem: Philip Hern

how i automated dev.to and linkedin publishing so visibility stops depending on memory

quick answer

who this is for

why this matters

step-by-step

1) define the starting point

2) apply the change

3) validate the result

faq

what was the biggest setup mistake?

should i keep manual publishing as a fallback?

references

related reading

the future of data engineering workflows with ai

quick answer

who this is for

why this matters

step-by-step

1) define the starting point

2) apply the change

3) validate the result

faq

what is the most important caveat?

what should i do first?

references

related reading

how i use cursor and ai agents to write dbt tests and documentation

quick answer

who this is for

why this matters

step-by-step

1) define the starting point

2) apply the change

3) validate the result

faq

what is the most important caveat?

what should i do first?

references

related reading

sharing is caring

the value of early adoption

the responsibility to share

stepping into mentorship

looking forward

further reading

related on this site

what is art?

thesis

context

argument

tension or counterpoint

closing

further reading

related on this site

dbt tests

quick answer

who this is for

why this matters

step-by-step

1) define the starting point

2) apply the change

3) validate the result

a practical prioritization rule

faq

what should i test first in a mature project with little coverage?

do dbt tests slow down delivery?

references

related reading

dbt docs

quick answer

who this is for

why this matters

what dbt docs generates

the site includes

how to document your models

model and column descriptions in yml

source descriptions

doc blocks for longer descriptions

a few useful options in dbt docs