Forem: Arik

You're NOT doing everything wrong

Arik — Sat, 29 Nov 2025 17:53:41 +0000

Walk into any engineering community online and you’ll quickly find yourself surrounded by old-time gunslingers shooting well-meaning advice at you: "Stop writing classes like that.", "Don't over-abstract", "Stop putting business logic in your controllers." and on and on. Just search the keyword "stop" on this site and you'll see what I'm talking about.

The advice is usually correct. Technically. But it feels a lot like a virtuoso guitarist watching a beginner strum their first G chord and saying, "Actually, you’re holding that completely wrong. Also your timing is off, your pick angle is inefficient, and you're muting half the strings." That's not helpful to anyone.

The truth is, every single one of those senior engineers wrote code that would make them cringe today. They shipped bugs. They nested callbacks eight levels deep. They used global variables without batting an eyelash. But they also learned a lot of valuable lessons first-hand.

Of course they don't want others to make these same mistakes but unfortunately that's not how this works. You can't tell a baby to get up and start walking and "learn from you". Good engineering requires balancing an overwhelming amount of variables, each of which needs to be mastered separately.

The 10,000 hour rule

It's tempting to think there's a shortcut—a book, a bootcamp, or a YouTube video that will transform your engineering skills overnight. But as Peter Norvig, director of research at Google, pointed out in his classic essay Teach Yourself Programming in Ten Years, expertise isn't a weekend project. He warns against the "Teach Yourself (X Language) in 24 Hours" books by noting that while you might pick up superficial syntax in a day, writing meaningful programs, debugging in the wild, and grokking an ecosystem takes deliberate practice over a decade.

Norvig draws on research from psychologists like Anders Ericsson, who popularized the "10,000-hour rule": mastery in fields like chess, music, or programming demands about 10 years of focused effort—roughly 10 to 20 hours a week of pushing your limits, getting feedback, and iterating on errors. Even geniuses like Mozart who was a musical prodigy at age 4, took 13 more years before he began to produce world-class music. The Beatles? A solid decade of small-club gigs before Sgt. Pepper's.

The 10,000-hour figure isn’t magic (Ericsson himself later clarified it’s an average, not a universal law), but the core insight stands: real skill is built in the muscles and intuition, not dogma. You don’t truly understand why eight levels of nested callbacks are painful until you’ve personally lived through the day when the innermost one throws and you spend six hours unraveling the stack in your head. You don’t feel the cost of an over-abstracted "service layer" until you're the one onboarding a new teammate number seven who can’t find where the actual business rule lives. These lessons have to hurt a little, or they don’t stick.

That's why the well-meaning "stop doing X" comments so often bounce off. The junior isn't being stubborn, they just lack the scar tissue required to feel the warning signs yet. Telling them "don’t prematurely optimize your code" is like telling a teenager "don’t fall in love with someone who's bad for you." The advice is correct, but it’s useless until life supplies the missing context.

In programming terms, this means your first few thousand hours will be messy. You'll write inefficient loops, ignore edge cases, and probably invent your own wheel. But that's the point: those cringe-worthy commits are your deliberate practice. The solution? Do it anyway. Build large projects, read other people's code, refine. Rinse, repeat. Books help, but nothing beats building something that breaks—and fixing it yourself.

Imitate the greats

One of the fastest ways to compress those 10,000 hours is to stand on the shoulders of past giants. Every painter in art school spends months copying the old masters stroke-for-stroke before they’re allowed to "express themselves." Musicians transcribe solos note-for-note. Writers literally re-type entire novels by their heroes.

Programmers almost never do this, and it’s mind-boggling.

You will learn more in one week of carefully re-implementing a well-written open-source project than you will in six months of reading blog posts titled "10 Clean Code Principles You’re Probably Breaking."

Pick something small but legendary. If you need ideas check out this Github repo. Not because you're going to ship your own Redis tomorrow, but because the act of re-creating it forces you to confront thousands of tiny decisions that the original authors already solved elegantly. And more importantly, you'll have a benchmark against which you can compare your work to.

But what about AI?

OK, elephant in the room. You’re being told from every corner that AI is the new 10,000-hour cheat code. "Just describe the feature, and let Cursor/Claude/Grok spit out the code." And on the surface it sounds good: The CRUD API that used to take days to build, now can be done in twenty minutes.

But here’s the quiet part nobody wants to say out loud: AI is an extremely powerful painkiller, not a cure.

It removes the immediate sting of "how the heck do I do this?", but it also numbs the very feedback loop that builds real skill. When the model writes a 400-line "service" that technically works, you feel productive. You get the dopamine hit of green tests and a merged PR. But you didn’t actually grow the muscle that recognizes why that 400-line file is a maintenance nightmare waiting to happen.

There’s no lasting satisfaction in vibe coding. You're not going to remember that time Claude wrote you a perfect pagination helper six months from now. You will remember the weekend you fixed a race condition that drove you crazy for days. That pain lodged itself in your long-term memory along with the solution, and now your intuition quietly steers you away from foot-guns.

None of this means you should swear off AI. I use it every day. It’s an incredible pair-programming intern: fast, tireless, and full of ideas. But treat it like one. Review every line. Ask it "why did you do it this way?" and make it explain. When it suggests something that feels off, dig in—that discomfort is the scar tissue forming.

Your path to "Senior" status

The senior engineer leaving the comment "Can we make this function pure, with no side effects?" isn’t lying. In five years you’ll probably agree with them completely, and you might even leave the exact same comment on someone else's PR. That’s not hypocrisy, that’s growth.

You’re just doing what every great engineer before you did: writing the bad code that's required before you can write the good code.

One day—probably years from now—you’ll open a pull request from some junior who's proudly over-abstracted the login flow into seventeen interfaces and a strategy factory. You'll feel that familiar urge to write "stop doing it like that."
And if you took anything at all from this article, you’ll pause, smile, and write instead: "Let's get on a call. I want to share some ideas with you".

Inventing Blockchain: The tale of a frustrated programmer

Arik — Sun, 26 Jan 2025 18:14:37 +0000

Satoshi leaned back in his chair, staring at the blinking cursor on his laptop screen. His mother needed money—urgently. She lived halfway across the world, and her refrigerator had just kicked the bucket, spilling melted ice cream and spoiled milk onto the floor. "Just send me what you can, Satoshi," she’d said, her voice tight with worry. "I’ll figure it out."

That was two days ago.

Satoshi had done everything right—or so he thought. He’d logged into his bank account, initiated a transfer, and paid the exorbitant international fee without complaint. But the process wasn’t instant, not by a long shot. The bank had warned him it could take up to three business days to complete the transfer.

He clenched his fists. Three days? What century was this? He could send an email to his mom in seconds. He could video call her instantly. But money? Money still crawled across borders like a letter in the mail.

He’d even called customer service, hoping to speed things up. After navigating a labyrinth of hold music and robotic prompts, a weary-sounding representative informed him there was nothing to be done. "International transfers take time," she said flatly, as though the laws of physics themselves dictated it.

Satoshi grumbled to himself as he closed his laptop. His mother was stuck waiting, and there was nothing he could do to help. It wasn’t just frustrating—it felt wrong. It felt unnecessary. "Why does it have to be this way?" he muttered.

The problem with trust

That night, Satoshi couldn’t stop thinking about it. He knew the transfer delay wasn’t about physics or technology—it was about trust. His bank didn’t trust his mom’s bank. They relied on intermediaries, networks, and clearinghouses to vouch for each other. Every step of the process was a chain of people and institutions verifying the transaction. Each one took their cut, added their delay.

Trust. The word hung heavy in Satoshi’s mind. Banks existed because people couldn’t trust each other directly with their money. But what if there was a way to eliminate the middleman? What if you could create a system where people could transact directly, without relying on trust at all?

He scribbled the idea down in his notebook: "How to replace trust with math?"

The first idea: a centralized transparent ledger

By morning, Satoshi had a plan. If trust was the problem, transparency was the solution. What if there was a single, public ledger where anyone could log in and see every transaction? If the ledger was open for all to verify, no one could cheat.

It didn’t take long for Satoshi to set up a basic website. He registered a domain, spun up a small server, and wrote the backend code. Transactions were stored in a simple database. The frontend was barebones but functional: a webpage where anyone could view the full list of transactions.

let ledger = [];

// Add a transaction to the ledger
app.post("/add_transaction", (req, res) => {
  const transaction = req.body;
  ledger.push(transaction);
  res.json({ status: "success", transaction });
});

// View the entire ledger
app.get("/view_ledger", (req, res) => {
  res.json(ledger);
});

When a transaction was added, it would appear instantly on the public ledger. Satoshi imagined this would remove any suspicion or dishonesty. "If everyone can see everything," he thought, "then no one can cheat."

He showed the prototype to his long time friend Taylor, who nodded appreciatively. “Okay, this is cool. But doesn’t this just make you the middleman?” she asked.

Satoshi frowned. “What do you mean?”

“Well,” Taylor said, gesturing at the screen, “you’re running the server, right? You’re the one people have to trust now.”

The realization

Taylor’s words hung in the air. She was right. Satoshi’s transparent ledger was just a centralized system pretending to be trustless. Sure, anyone could view the transactions—but what if he decided to delete some? Or worse, alter them?

Satoshi tried to argue with himself. "I wouldn’t do that," he muttered, pacing around the room. But the truth was unavoidable: the system was only as trustworthy as its operator. And that operator was him.

“This isn’t the solution,” he said, slumping into his chair. “It’s just… another bank.”

Decentralization and the struggle for trust

Satoshi stared at his notebook, his frustration boiling over. The question gnawed at him: How do I get rid of myself?. His centralized ledger had replaced one middleman with another—himself. If he wanted to create a truly trustless system, he’d have to eliminate his role entirely.

But how?

The question haunted him. He couldn’t sleep that night, his mind cycling through possibilities. By morning, the answer came to him: make the ledger decentralized.

The next idea: A decentralized ledger

What if, instead of one server maintaining the ledger, everyone kept their own copy? Transactions would be broadcast to the entire network, and each participant would update their ledger independently. That way, no single person or entity would be in control.

Satoshi excitedly sketched out the architecture. He imagined a network of peers, each running a lightweight program. Transactions would propagate through the network, and every participant would maintain their own version of the truth.

Within hours, he was coding again, this time building a peer-to-peer prototype.

const net = require("net");

let ledger = [];

// Peer-to-peer server
const server = net.createServer((socket) => {
  socket.on("data", (data) => {
    const transaction = JSON.parse(data.toString());
    ledger.push(transaction);
    console.log("Transaction added:", transaction);
  });
});

server.listen(5000, () => {
  console.log("Peer listening on port 5000");
});

// Broadcast a transaction
function broadcastTransaction(transaction, peers) {
  peers.forEach((peer) => {
    const client = net.createConnection(peer.port, peer.host, () => {
      client.write(JSON.stringify(transaction));
      client.end();
    });
  });
}

He tested the system with Taylor, connecting their laptops over the network. Transactions flowed between their programs, and both of their ledgers updated in sync. It was a breakthrough.

Or so it seemed.

The trust problem: forks and fraud

Taylor, ever the skeptic, spotted the issue almost immediately. “Okay, so I send a transaction to you, and your ledger updates. But what happens if someone else sends me a conflicting transaction? Like if I spend the same $10 twice?”

Satoshi frowned. “Well… everyone will see both transactions, and they’ll just figure out which one is valid.”

Taylor raised an eyebrow. “How? What if different people update their ledgers in different orders? You could end up with two different versions of the truth.”

Satoshi’s stomach sank. She was right. Without a way to enforce a single, agreed-upon history, the network could split into forks. Different participants might believe in different versions of the ledger. Worse, malicious actors could exploit the system, flooding it with fake transactions to confuse the network.

He felt like he was back at square one. Decentralization had solved the problem of relying on a single middleman, but it had introduced a new issue: how do you get a group of untrustworthy participants to agree?

The spark of consensus

Satoshi knew he needed a way for the network to converge on a single version of the ledger. He spent days researching distributed systems, reading about Byzantine fault tolerance and quorum protocols. The solutions were complex, often relying on trusted leaders or voting systems. But trust was exactly what he wanted to eliminate.

The breakthrough came while reading an obscure article on cryptographic proofs. It described something called proof of work, a technique originally used to deter email spam. The idea was simple but powerful: participants would perform a computationally expensive task to prove their commitment.

Satoshi’s mind raced. What if transactions required proof of work? Instead of trusting votes or leaders, the network would trust the work itself.

Implementing Proof of Work

The proof-of-work concept transformed Satoshi’s approach. Each participant in the network would compete to solve a cryptographic puzzle. The first to solve it would broadcast their solution, along with the new transactions, to the rest of the network. If the solution was valid -- which would be easy to verify -- everyone would add the new transactions to their ledger.

He modified his code, implementing a simple proof-of-work algorithm. The puzzle was to find a number (a “nonce”) that, when combined with the transaction data and hashed, produced a hash starting with a specific number of zeros.

const crypto = require("crypto");

const difficulty = 4;

function proofOfWork(transaction) {
  let nonce = 0;
  let hash = "";
  do {
    nonce++;
    hash = crypto
      .createHash("sha256")
      .update(JSON.stringify(transaction) + nonce)
      .digest("hex");
  } while (!hash.startsWith("0".repeat(difficulty)));
  return { nonce, hash };
}

Testing the system

Taylor was impressed. “So, whoever solves the puzzle first gets to decide the next block of transactions?”

“Exactly,” Satoshi said, smiling for the first time in days. “The puzzle is so hard to solve that no one can fake it. But once it’s solved, it’s easy for everyone else to verify.”

They tested the system together, running separate instances of the program. The proof-of-work algorithm worked beautifully, ensuring that only one version of the ledger could advance at a time.

Inventing blockchain: the new problems

For a fleeting moment, Satoshi felt like he had solved it. Proof of work seemed like the perfect solution. The puzzle was hard enough to prevent fraud, yet easy to verify once solved. Transactions flowed through the network, and everyone agreed on the ledger.

But then the cracks began to show.

The energy Problem

Taylor’s laptop hummed loudly, its fan spinning at full speed. “Is this supposed to be happening?” she asked, pointing at the temperature warning on her screen.

Satoshi winced. He hadn’t thought much about the cost of computation. Proof of work relied on brute force, trying billions of combinations until it found the right one. As the difficulty increased, so did the energy required.

“You’re burning through electricity just to process one transaction,” Taylor said, raising an eyebrow. “Imagine if thousands of people were doing this at the same time. The whole network would become a furnace.”

Satoshi nodded grimly. He had optimized the code as much as possible, but proof of work was inherently resource-intensive. It was a price he hadn’t anticipated. “Maybe it’s worth it if it means we don’t need a middleman,” he said weakly.

Taylor didn’t look convinced.

The monopoly problem

Over the next few days, Satoshi expanded his tests to include more participants. At first, everything went smoothly. Friends joined the network, solving puzzles and adding transactions to their ledgers. But soon, another problem emerged: not everyone was solving puzzles at the same rate.

Participants with faster computers—or, worse, specialized hardware—were completing proofs of work faster than everyone else. They dominated the network, consistently broadcasting their solutions first and claiming control over the ledger.

“It’s just like the banks,” Taylor said, shaking her head. “The people with the most resources get to call the shots. It’s centralized all over again—just in a different way.”

Satoshi felt the weight of her words. He had replaced trust in institutions with trust in computation, but the result was the same: inequality. The rich got richer, and the small players were pushed aside.

The incentive problem

Even without the monopoly issue, Satoshi saw another flaw: why would anyone bother solving puzzles in the first place? Proof of work was costly. The energy required wasn’t free, and participants gained nothing tangible from their efforts.

Taylor pointed this out during another late-night discussion. “If it costs so much to solve these puzzles, why would anyone do it? What’s in it for them?”

Satoshi had no answer. He realized that without an incentive, the entire system would grind to a halt. Participants needed a reason to spend their resources cracking puzzles—something more compelling than altruism or ideology.

It was a sobering realization. His system wasn’t sustainable. If he wanted it to work in the real world, he needed to introduce an economic element.

The birth of The Block

Satoshi leaned back in his chair, rubbing his temples. The network needed a better structure, one that tied everything together. His eyes wandered to the ever-growing list of transactions on his screen. That’s when it hit him.

“Why are we solving puzzles for individual transactions?” he muttered. “Why not group them?”

If the system could bundle multiple transactions together into a single “block,” proof of work would only need to be done once for an entire group of transactions. This would immediately reduce the energy overhead—each puzzle would secure dozens or even hundreds of transactions, instead of just one.

He sketched it out in his notebook. Each block would contain:

A list of transactions.
A reference to the previous block’s proof (to chain them together).
A proof of work for the current block.

The result would be a chain of blocks—each securely linked to the one before it. Changing any block would invalidate the entire chain, making tampering nearly impossible. “It’s not just a ledger anymore,” he thought. “It’s… a blockchain.”

But blocks didn’t just solve the energy problem—they opened the door to a solution for another pressing issue: incentives. If participants competed to solve a block’s proof of work, the first to succeed could be rewarded for their effort. That reward could take the form of a new, native digital currency—coins created by the network itself.

He scribbled furiously in his notebook: “Reward miners with cryptocurrency for creating new blocks.”

The coins would serve as payment for the work miners did to secure the network, and they would have value because they were scarce and useful. Within the blockchain network, these coins could act as a medium of exchange, allowing participants to transact directly with one another for good and services.

To prevent abuse, the reward system must ensure that:

The reward is only issued when the block is mined and validated by the network.
All participants independently verify the reward transaction as part of the validation process.

Mitigating monopolization

The introduction of blocks didn’t just make the blockchain more efficient; it also helped reduce the risk of monopolization. Satoshi realized that smaller miners, unable to compete with powerful setups on their own, could band together by pooling their resources. In a mining pool, participants combined their computational power to increase their chances of solving a block’s proof-of-work puzzle. When the pool successfully mined a block, the reward was distributed proportionally among the contributors. Pooling allowed smaller miners to stay relevant in the network, ensuring they had a fair chance to earn rewards despite their limited hardware.

At the same time, the architecture of the blockchain made monopolization through fraud prohibitively difficult. Because each block was cryptographically linked to the one before it, tampering with a single block wasn’t enough to rewrite history. Changing even one transaction in an old block would invalidate its hash and all subsequent blocks in the chain. An attacker would need to remine the altered block and every block after it. The computational cost of such an attack grew exponentially with the number of blocks in the chain, making the system effectively tamper-proof.

Implementing the Blockchain

Satoshi started by implementing the blockchain itself. Each block would store a set of transactions, a proof of work, and a reference to the previous block. This reference linked the blocks together, creating a chain. If someone tried to alter an earlier block, it would invalidate the entire chain because the cryptographic link would break.

The first block, known as the “genesis block,” had no predecessor and was hardcoded into the system. Every subsequent block was mined by solving a proof-of-work puzzle, and its reference to the previous block’s proof ensured that the chain remained immutable.

const crypto = require("crypto");

let blockchain = [];
const rewardAmount = 50;

// Create the genesis block
function createGenesisBlock() {
  return {
    index: 0,
    timestamp: Date.now(),
    transactions: [],
    proof: "0",
    previousHash: "0",
  };
}

// Calculate the hash of a block
function calculateHash(block) {
  return crypto
    .createHash("sha256")
    .update(JSON.stringify(block.transactions) + block.nonce + block.previousHash)
    .digest("hex");
}

// Create a new block
function createBlock(transactions, previousHash, difficulty) {
  let nonce = 0;
  let hash = "";
  const rewardTransaction = { from: "Network", to: "Miner", amount: rewardAmount };

  // Add the reward transaction to the transactions list
  const blockTransactions = [...transactions, rewardTransaction];

  do {
    nonce++;
    hash = crypto
      .createHash("sha256")
      .update(JSON.stringify(blockTransactions) + nonce + previousHash)
      .digest("hex");
  } while (!hash.startsWith("0".repeat(difficulty)));

  return {
    index: blockchain.length,
    timestamp: Date.now(),
    transactions: blockTransactions,
    proof: hash,
    previousHash,
    nonce,
  };
}

// Validate a block
function validateBlock(block, previousBlock, difficulty) {
  if (block.previousHash !== previousBlock.proof) {
    return false; // Invalid link to previous block
  }
  const hash = calculateHash(block);
  if (!hash.startsWith("0".repeat(difficulty))) {
    return false; // Proof of work is invalid
  }
  if (block.transactions[block.transactions.length - 1].amount !== rewardAmount) {
    return false; // Invalid reward transaction
  }
  return true;
}

// Initialize the blockchain with the genesis block
blockchain.push(createGenesisBlock());

Testing the Blockchain

Satoshi leaned back in his chair, watching the blockchain grow. Each block carried its own proof of work, a list of transactions, and the miner’s reward. But as Taylor reviewed the system, she raised a concern.

“Wait,” she said, scrolling through the blockchain logs. “What’s stopping me from just giving myself a million coins in the reward transaction?”

Satoshi smirked. “Go ahead. Try.”

Taylor modified the code to include a reward of one million coins instead of the standard fifty. She mined a block and added it to her local chain. At first glance, the block looked valid—her version of the blockchain updated, and the oversized reward transaction showed up.

But when Taylor’s chain attempted to sync with the network, the other nodes rejected her block. “Invalid block,” the logs reported. Taylor frowned. “Why didn’t it work?”

Satoshi explained. “The reward transaction is just like any other transaction—it’s validated by the entire network. If the reward doesn’t follow the rules, the block is rejected. You can mine all the blocks you want, but no one else will accept them unless they’re valid.”

She nodded, impressed. “So the network enforces fairness?”

“Exactly,” Satoshi said. “No one—not even the miner—can cheat the system.”

A Vision for the future

As the weeks turned into months, Satoshi refined the system, documenting the protocol in meticulous detail. He imagined a world where people could transact directly, where intermediaries weren’t needed to enforce trust. Where governments, corporations, and even individuals could build applications on top of the blockchain to empower people.

It wasn’t just about money. The blockchain could track ownership, verify identities, manage contracts, and create entirely new forms of cooperation. Satoshi saw the possibilities stretching out before him, and for the first time in years, he felt hopeful about the world he lived in.

Making binary search slower

Arik — Sat, 18 Jan 2025 20:52:55 +0000

Binary search is frequently praised as one of the most efficient algorithms for finding an element in a sorted dataset. It’s fast—blazingly fast—and its efficiency grows more apparent as the size of the search space increases. But how can we truly appreciate its speed in a world where computers perform millions of operations per second?

Let’s flip the script: we’re going to intentionally slow down the algorithm by adding a one-second delay to each iteration. This simple twist will help us feel the power of binary search and understand how its logarithmic efficiency scales with the size of the dataset.

What is binary search?

Before diving into the code, let’s quickly recap binary search. Given a sorted list, binary search divides the list in half at each step, checking whether the target value lies in the left or right half. This process continues until the target is found or the search space is reduced to zero.

Here’s why it’s so efficient: with each step, the size of the search space is halved. For example:

A list of 1,000 elements requires at most 10 comparisons.
A list of 1 million elements? Just 20 comparisons.
A list of 1 billion elements? 30 comparisons.

The twist: slowing it down

To make the efficiency of binary search tangible, we’ll modify the algorithm by adding a one-second delay to each iteration. This way, the time taken will scale directly with the number of steps the algorithm performs. Watching the algorithm process datasets of different sizes will give us an intuitive understanding of how it scales.

Here’s the implementation:

import time
import sys

def binary_search(arr, target):
    """Performs a binary search with a 1-second delay per iteration."""
    low, high = 0, len(arr) - 1
    iterations = 0  # Track the number of iterations

    while low <= high:
        iterations += 1
        mid = (low + high) // 2
        print(f"Iteration {iterations}: Checking index {mid} (value: {arr[mid]})")

        # Artificial delay to simulate time passing
        time.sleep(1)

        if arr[mid] == target:
            print(f"Found {target} at index {mid} after {iterations} seconds.")
            return mid
        elif arr[mid] < target:
            low = mid + 1
        else:
            high = mid - 1

    print(f"{target} not found after {iterations} seconds.")
    return -1

if __name__ == "__main__":
    # Parse command-line arguments
    if len(sys.argv) != 3:
        print("Usage: python binary_search.py <array_size> <target_value>")
        sys.exit(1)

    try:
        array_size = int(sys.argv[1])
        target = int(sys.argv[2])
    except ValueError:
        print("Both array_size and target_value must be integers.")
        sys.exit(1)

    # Generate the sorted array
    array = list(range(array_size))

    print(f"Starting binary search for {target} in an array of size {array_size}...\n")
    # Run the binary search
    binary_search(array, target)

How to run the script

python binary_search.py <array_size> <target_value>

Examples

10 elements

% python3 script.py 10 4
Starting binary search for 4 in an array of size 10...

Iteration 1: Checking index 4 (value: 4)
Found 4 at index 4 after 1 seconds.

1000 elements

% python3 script.py 1000 572
Starting binary search for 572 in an array of size 1000...

Iteration 1: Checking index 499 (value: 499)
Iteration 2: Checking index 749 (value: 749)
Iteration 3: Checking index 624 (value: 624)
Iteration 4: Checking index 561 (value: 561)
Iteration 5: Checking index 592 (value: 592)
Iteration 6: Checking index 576 (value: 576)
Iteration 7: Checking index 568 (value: 568)
Iteration 8: Checking index 572 (value: 572)
Found 572 at index 572 after 8 seconds.

1 million elements

% python3 script.py 1000000 78246
Starting binary search for 78246 in an array of size 1000000...

Iteration 1: Checking index 499999 (value: 499999)
Iteration 2: Checking index 249999 (value: 249999)
Iteration 3: Checking index 124999 (value: 124999)
Iteration 4: Checking index 62499 (value: 62499)
Iteration 5: Checking index 93749 (value: 93749)
Iteration 6: Checking index 78124 (value: 78124)
Iteration 7: Checking index 85936 (value: 85936)
Iteration 8: Checking index 82030 (value: 82030)
Iteration 9: Checking index 80077 (value: 80077)
Iteration 10: Checking index 79100 (value: 79100)
Iteration 11: Checking index 78612 (value: 78612)
Iteration 12: Checking index 78368 (value: 78368)
Iteration 13: Checking index 78246 (value: 78246)
Found 78246 at index 78246 after 13 seconds.

1 billion elements

% python3 script.py 1000000000 708246
Starting binary search for 708246 in an array of size 1000000000...

Iteration 1: Checking index 499999999 (value: 499999999)
Iteration 2: Checking index 249999999 (value: 249999999)
Iteration 3: Checking index 124999999 (value: 124999999)
Iteration 4: Checking index 62499999 (value: 62499999)
Iteration 5: Checking index 31249999 (value: 31249999)
Iteration 6: Checking index 15624999 (value: 15624999)
Iteration 7: Checking index 7812499 (value: 7812499)
...
Iteration 25: Checking index 708248 (value: 708248)
Iteration 26: Checking index 708233 (value: 708233)
Iteration 27: Checking index 708240 (value: 708240)
Iteration 28: Checking index 708244 (value: 708244)
Iteration 29: Checking index 708246 (value: 708246)
Found 708246 at index 708246 after 29 seconds.

Why this matters

This experiment (hopefully) demonstrates the elegance of binary search. By slowing it down, we make its logarithmic behavior easy to observe and appreciate. No matter how large the dataset, the number of steps grows at a manageable rate.

Try It Yourself

Run the code, tweak the dataset sizes, and experiment with different target values. If you’re teaching others about algorithms or learning them yourself, this slowed-down version of binary search might be a good way to to give them a "feel" for it.

What are your thoughts on this approach? Do you have any other creative twists for understanding algorithms? Share your ideas in the comments below!

Building a webhook tester from scratch

Arik — Wed, 01 Jan 2025 20:07:00 +0000

Every now and then I find myself in need of a quick way to debug and test webhooks without the hassle of setting up an entire server. Whether it's validating a third-party API integration, troubleshooting event payloads, or simply experimenting with custom webhook workflows.

While there are plenty of solutions out there designed to test webhooks, they often come with significant limitations. Many of these tools are not self-hostable (or are difficult to install), they have stringent request limits (unless you’re willing to pay), they feel too complex, and they often store data on their servers, raising concerns about privacy and security, especially when I'm dealing with sensitive payloads.

So, I decided to build my own webhook tester with the following requirements:

Simple - No paywalls, sign-up or having to jump through a bunch of hoops to create a webhook.
Privacy-first - Nothing is stored on the servers. Requests go from the client to the browser that created the webhook.
Self-hostable - Easy to install without having to setup a database or other pieces of infrastructure.

Wirehook

Here’s how Wirehook looks in action. The interface is clean and straightforward: you get a unique URL to use as your webhook endpoint, and any incoming requests are displayed in real-time for you to view and analyze. And that's basically it.

If you'd like to run Wirehook locally on your machine (or on your servers), all you need is Docker installed. Once that's ready, just run the following command:

docker run -d -p 3000:3000 runabol/wirehook

This will spin up Wirehook on your local machine, accessible at http://localhost:3000. From there, you can create and test webhook endpoints just like you would with the hosted version.

Wirehook was built entirely using Next.js to keep it simple and self-contained. If you’re curious to see how it works or want to get involved, the full source code is available on GitHub.

Building scalable ML workflows

Arik — Sun, 22 Dec 2024 14:56:47 +0000

A little while back, I wrote a post introducing Tork, an open-source project I've been developing. In a nutshell, Tork is a general-purpose, distributed workflow engine suitable for various workloads. At my work, we primarily use it for CPU/GPU-heavy tasks such as processing digital assets (3D, videos, images etc.), as well as our CI/CD tool for our internal PaaS.

Recently, I've been thinking about how Tork could potentially be leveraged to run machine learning type workloads. I was particularly inspired by the Ollama project and wanted to see if I can do something similar, but using plain old Docker images rather than Ollama's Modelfile.

Given that ML workloads often consist of distinct, interdependent stages—such as data preprocessing, feature extraction, model training, and inference—it’s crucial to have an engine that can orchestrate these steps. These stages frequently require different types of compute resources (e.g., CPUs for preprocessing, GPUs for training) and can benefit greatly from parallelization.

Moreover, resiliency is a critical requirement when running machine learning workflows. Interruptions, whether due to hardware failures, network issues, or resource constraints, can result in significant setbacks, especially for long-running processes like model training.

These requirements are very similar to my other non-ML workloads, so I decided to put my all my theories to the test and see what it would take to execute a simple ML workflow on Tork.

The experiment

For this first experiment, let's try to execute a simple sentiment analysis inference task:

Download the latest Tork binary and untar it.

tar xvzf tork_0.1.109_darwin_arm64.tgz

Start Tork in standalone mode:

./tork run standalone

If all goes well, you should something like this:

...
10:36PM INF Coordinator listening on http://localhost:8000
...

Next, we need to build a docker image that contains the model and the necessary inference script. Tork tasks typically execute within a Docker container.

inference.py

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
import os

MODEL_NAME = os.getenv("MODEL_NAME")

def load_model_and_tokenizer(model_name):
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForSequenceClassification.from_pretrained(model_name)
    return tokenizer, model

def predict_sentiment(text, tokenizer, model):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True, max_length=512)

    with torch.no_grad():
        outputs = model(**inputs)

    predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
    predicted_label = torch.argmax(predictions, dim=1).item()
    confidence = predictions[0][predicted_label].item()

    return predicted_label, confidence

if __name__ == "__main__":
    tokenizer, model = load_model_and_tokenizer(MODEL_NAME)
    text = os.getenv("INPUT_TEXT")
    label, confidence = predict_sentiment(text, tokenizer, model)
    sentiment_map = {0: "Negative", 1: "Positive"}
    sentiment = sentiment_map[label]
    print(f"{sentiment}")

Dockerfile

FROM huggingface/transformers-pytorch-cpu:latest

WORKDIR /app

COPY inference.py .

# Pre-load the model during image build
RUN python3 -c "from transformers import AutoTokenizer, AutoModelForSequenceClassification; AutoTokenizer.from_pretrained('distilbert-base-uncased-finetuned-sst-2-english'); AutoModelForSequenceClassification.from_pretrained('distilbert-base-uncased-finetuned-sst-2-english')"

docker build -t sentiment-analysis .

Next, let's create the Tork job to run the inference.

sentiment.yaml

name: Sentiment analysis example
inputs:
  input_text: Today is a lovely day
  model_name: distilbert-base-uncased-finetuned-sst-2-english
output: "{{trim(tasks.sentimentResult)}}"
tasks:
  - name: Run sentiment analysis
    var: sentimentResult
    # the image we created in the previous step, 
    # but can be any image available from Docker hub
    # or any other image registry
    image: sentiment-analysis:latest
    run: |
      python3 inference.py > $TORK_OUTPUT
    env:
      INPUT_TEXT: "{{inputs.input_text}}"
      MODEL_NAME: "{{inputs.model_name}}"

Submit the job. Tork jobs execute asynchronously. Once a job is submitted you get back a job ID to track its progress:

JOB_ID=$(curl -s \
  -X POST \
  -H "content-type:text/yaml" \
  --data-binary @sentiment.yaml \
  http://localhost:8000/jobs | jq -r .id)

Poll the job's status and wait for it to complete:

while true; do 
  state=$(curl -s http://localhost:8000/jobs/$JOB_ID | jq -r .state)
  echo "Status: $state"
  if [ "$state" = "COMPLETED" ]; then; 
     break 
  fi 
  sleep 1
done

Inspect the job results:

curl -s http://localhost:8000/jobs/$JOB_ID | jq -r .result

Positive

Try changing the input_text in sentiment.yaml and re-submit the job for different results.

Next steps

Now that I got this basic proof of concept working on my machine I need to push that Docker image to a Docker registry so it is available for any Tork workers on my production cluster. But this seems like a viable approach.

The code for this article can be found on Github.

If you're interested in learning more about Tork:

Documentation: https://www.tork.run
Backend: https://github.com/runabol/tork
Web UI: https://github.com/runabol/tork-web

What is your Why?

Arik — Wed, 13 Sep 2023 02:06:27 +0000

I believe we all entered the field of programming for various reasons. It could be to earn a living, pursue a sought-after career, or simply because we love building stuff. Whatever the reason, we're here.

However, if I'm being perfectly honest, while all these are good reasons, none would provide me with more than the bare minimum level of happiness at what I do almost every day for many hours.

Yet, while I had my fair share of miserable days on the job like everyone else, more often than not, I am truly eager to do my thing at work.

So today, while I was running, I found myself in a bit of an introspective mood and wondered what is my personal Why? Why do I still love programming so much after all these years.

I always knew it had something to do with people. Seeing someone using something I wrote and maybe even liking it never ceases to give me a kick. But I felt there was a deeper desire.

After a little back and forth with myself I reduced it to something that felt really true for me: to reduce the suffering of someone else. Okay, I know it sounds a bit overly dramatic, but hear me out here for a minute.

Our profession is riddled with sharp objects we all occasionally bump into. People much smarter than myself say that it takes a very long time to even begin to master it, there is formidable math and sophisticated algorithms lurking at every corner, then there are new languages, tools, frameworks and paradigms jumping on us every other day that threaten to undermine everything we've learned for the past however many years.

So when I get to brighten someone's day through my work in even the smallest way, damn it it feels good.

It could be a user that with the help of something I wrote suddenly feels much more productive, or it could be as "small" as assisting a colleague by showing them how to use some tool that I take for granted, but is life-changing for them.

That's why I relish at the opportunity to spend that extra hour at making my API just a tiny bit simpler, or clean up and refactor some messy code, or write that extra page of documentation or test. Because someone (including myself) will experience just a tiny bit less frustration and pain down the road when they try to use it.

And whenever I get to see it first hand it gives me the energy to wake up the next day and do it all over again.

So now I'd like to invite you to find your personal Why, and to please share it with us.

Let's build a code execution engine

Arik — Sun, 10 Sep 2023 23:38:40 +0000

Have you ever wondered what happens behind the scenes when you hit "Run" on a code snippet in online development environments like Go Playground or OneCompiler?

By following along, at the end of this post, you will have the front-end and backend for a very bare-bones implementation that resembles something like this:

If you just want to see the code, you can find it here.

It's important to note that while skeletal, the implementation is by no means a "toy". We will address the most important considerations for building the core requirement of such a platform. Namely:

Security - we are letting users execute arbitrary code on our servers so we need a way to isolate the code execution in order to limit the possibility for abuse as much as possible.
Scalability - We need a way to scale our system as the number of users grow.
Limits - We want to limit the amount of resources we are allocating for a given code execution so it doesn't tax our servers as well as hurt other users' experience.

What are we going to use

In order to address all the considerations mentioned above, we are going to use Tork to do all the heavy lifting for us.

In a nutshell, Tork is a general purpose, distributed workflow engine that I've been working on for the past couple of months.

It uses Docker containers for the execution of workflow tasks which addresses point #1 and point #3 for us - we'll see exactly how in a minute.

It also supports a distributed setup to scale task processing to an arbitrary number of worker nodes which addresses point #2.

There are two ways we can go about this:

Download and install the vanilla Tork and write a new thin API service that will sit between the client and Tork -- because we don't necessarily want to expose Tork's native API to them in order to have tight control over which parameters are sent to Tork. The advantage with this approach is that I can write my "middleware" server in a language of my choice.
Extend Tork to expose our custom API endpoint and disable all other endpoints. This requires knowledge of Go programming.

For the purposes of this demo I'll be going with option #2.

OK, let's write some code

You'll need:

Docker installed on the machine that you're running the demo on.
Golang >= 1.19+

Create a new directory for the project:

mkdir code-execution-demo
cd code-execution-demo

Initialize the project:

go mod init example.com/code-execution-demo

Get the Tork dependency:

go get github.com/runabol/tork

Create a main.go file at the root of the project with the minimum boilerplate necessary to start Tork:

package main

import (
    "fmt"
    "os"

    "github.com/runabol/tork/cli"
    "github.com/runabol/tork/conf"
)

func main() {
   // Load the Tork config file (if exists) 
   if err := conf.LoadConfig(); err != nil {
     fmt.Println(err)
     os.Exit(1)
   }

   // Start the Tork CLI
   app := cli.New()
   if err := app.Run(); err != nil {
     fmt.Println(err)
     os.Exit(1)
   }
}

Start Tork:

go run main.go

If all goes well, you should see something like this:

 _______  _______  ______    ___   _ 
|       ||       ||    _ |  |   | | |
|_     _||   _   ||   | ||  |   |_| |
  |   |  |  | |  ||   |_||_ |      _|
  |   |  |  |_|  ||    __  ||     |_ 
  |   |  |       ||   |  | ||    _  |
  |___|  |_______||___|  |_||___| |_|
...

Let's use the RegisterEndpoint hook to register our custom endpoint:

package main

import (
    "fmt"
    "net/http"
    "os"

    "github.com/runabol/tork/cli"
    "github.com/runabol/tork/conf"
    "github.com/runabol/tork/middleware/web"
)

func main() {
    // removed for brevity

    app.RegisterEndpoint(http.MethodPost, "/execute",handler)

    // removed for brevity
}

func handler (c web.Context) error {
  return c.String(http.StatusOK, "OK")
}

Start Tork in standalone (not distributed) mode:

go run main.go run standalone

Call the new endpoint from another terminal window:

% curl -X POST http://localhost:8000/execute
OK

So far so good.

Let's assume the client is going to send us the following JSON object:

{
  "language":"python|bash|go|etc.",
  "code":"the source code to execute"
}

Let's write a struct that we can bind these values to:

type ExecRequest struct {
    Code     string `json:"code"`
    Language string `json:"language"`
}

func handler(c web.Context) error {
  req := ExecRequest{}
  if err := c.Bind(&req); err != nil {
    c.Error(http.StatusBadRequest, err)
    return nil
  }  

  return c.JSON(http.StatusOK,req)
}

At this point we just echo the request back to the user. But it's a good stepping stone to make sure the binding logic works. Let's try it:

% curl -X POST -H "content-type:application/json" -d '{"language":"bash","code":"echo hello world"}' http://localhost:8000/execute

{"code":"echo hello world","language":"bash"}

OK, next we need to convert the request to a Tork task:

func buildTask(er ExecRequest) (input.Task, error) {
        var image string
        var run string
        var filename string

        switch er.Language {
        case "":
                return input.Task{}, errors.Errorf("require: language")
        case "python":
                image = "python:3"
                filename = "script.py"
                run = "python script.py > $TORK_OUTPUT"
        case "go":
                image = "golang:1.19"
                filename = "main.go"
                run = "go run main.go > $TORK_OUTPUT"
        case "bash":
                image = "alpine:3.18.3"
                filename = "script"
                run = "sh ./script > $TORK_OUTPUT"
        default:
                return input.Task{}, errors.Errorf("unknown language: %s", er.Language)
        }

        return input.Task{
                Name:    "execute code",
                Image:   image,
                Run:     run,
                Files: map[string]string{
                        filename: er.Code,
                },
        }, nil
}

So we are doing three things here essentially:

Map the language field to a Docker image.
Write the code to an appropriate file in the container depending on the language.
Run the necessary command to execute the code in the container.

Let's use it in our handler:

task, err := buildTask(req)
if err != nil {
  c.Error(http.StatusBadRequest, err)
  return nil
}

And finally, let's submit the job:

input := &input.Job{
  Name:  "code execution",
  Tasks: []input.Task{task},
}   

job,err:= engine.SubmitJob(c.Request().Context(),input)
if err != nil {
  return err
}

fmt.Printf("job %s submitted!\n", job.ID)

Let's try to run our updated handler:

go run main.go run standalone

curl -X POST -H "content-type:application/json" -d '{"language":"bash","code":"echo hello world"}' http://localhost:8000/execute

If all goes well, you should see something like this in the logs:

job 5488620e9bc34e09b6ec3677ea28a067 submitted!

Next, we want to get the execution output so we can return it to the client. But since Tork operates asynchronously we need a way to tell Tork to let us know what the job is done (or failed).

This is where JobListener comes in:

result := make(chan string)

listener := func(j *tork.Job) {
  if j.State == tork.JobStateCompleted {
    result <- j.Execution[0].Result
  } else {
    result <- j.Execution[0].Error
  }
}

// pass the listener to the submit job call
job, err := engine.SubmitJob(c.Request().Context(), input, listener)
if err != nil {
  return err
}

return c.JSON(http.StatusOK, map[string]string{"output": <-result})

Since the job listener is not executing in the "main" thread/goroutine we need a way to pass it back to the main thread. Luckily, Golang has this really convenient thing called a channel which does exactly that.

OK, let's see if this works:

curl -X POST -H "content-type:application/json" -d '{"language":"bash","code":"echo hello world"}' http://localhost:8000/execute
{"output":"hello world\n"}

Nice!

Security

Let's update our Task definition to enforce a more strict set of security constraints:

input.Task{
  Name:  "execute code",
  Image: image,
  Run:   run,
  Limits: &input.Limits{
    CPUs:   ".5", // no more than half a CPU
    Memory: "20m", // no more than 20MB of RAM
  },
  Timeout:  "5s", // terminate container after 5 seconds
  Networks: []string{"none"}, // disable networking
  Files: map[string]string{
  filename: er.Code,
}

Let's disable Tork's built-in endpoints:

Create a file named config.toml in the root of your project with the following contents:

# config.toml
[coordinator.api]
endpoints.health = true
endpoints.jobs = false
endpoints.tasks = false
endpoints.nodes = false
endpoints.queues = false
endpoints.stats = false

Now when you start the project you should see that Tork picked up the config:

% go run main.go run standalone          
7:08PM INF Config loaded from config.tom
...

Frontend

Let's try to get the frontend to talk to our backend:

git clone git@github.com:runabol/code-execution-demo.git
cd code-execution-demo/frontend
npm i
npm run dev

And open http://localhost:3000

Scaling Up

The last point we are left to address is scalability.

There are many ways to tweak Tork for scalability, but for our purposes here we'll keep it simple and do the bare minimum of starting a RabbitMQ broker which will allow us to distribute the task processing:

Start a RabbitMQ broker. Example:

docker run \
  -d \
  --name=tork-rabbit \
  -p 5672:5672 \
  -p 15672:15672 \
  rabbitmq:3-management

Next, add the following to your config.toml:

[broker]
type = "rabbitmq"

[broker.rabbitmq]
url = "amqp://guest:guest@localhost:5672"

Stop Tork if it's currently running.

Start the Tork Coordinator:

go run main.go run coordinator

From a separate terminal window start a worker. You can also start additional workers if you like:

go run main.go run worker

Using curl or the frontend try to submit a code snippet.

Conclusion

Hope you enjoyed this tutorial as much as I did.

The full source code can be found on Github.

Building a distributed workflow engine from scratch

Arik — Fri, 25 Aug 2023 22:00:00 +0000

I've been somewhat obsessed with creating workflow engines for the better part of a decade. The idea of constructing a 'mega' machine from an army of smaller machines never seems to get old for me.

At their core, workflow engines are responsible for executing a series of tasks (typically referred to as a 'job', 'pipeline' or 'workflow') over a cluster of machines (typically referred to as 'workers' or nodes) as quickly and as efficiently as possible.

Building a workflow engine comes with a bunch of interesting challenges. Here's a short and highly non-exhaustive list:

What do you use to author workflows? Do you use a general-purpose programming language? a configuration-type language like JSON or YAML or do you roll your own DSL (Domain Specific Language)?
How do we decide which tasks go to which workers such that busy workers are not overloaded while others are idle?
How do we deal with the requirement to scale up or down the capacity in response to fluctuations in computational demand?
How do we deal with intermittent task failures?
How do we deal with worker crashes?
How do we deal with a situation where we have more tasks to execute than available capacity?

Let's put some rubber on the road

The first time I needed to actually build a workflow engine was while working for a video streaming startup. At the time, the company was outsourcing all its video processing needs to another company.

The existing process was slow, expensive and brittle. The company was regularly getting new content (movies, trailers, bonus video material, closed captions etc.) and we needed a way to quickly process this content in order to get it up on the service for customers to enjoy.

Moreover, the existing process was quite rigid and any changes to it (e.g. to introduce a new audio technology) took months or were simply not possible. I suggested to build a proof-of-concept that would allows us to bring the work in-house and luckily my managers were open to the idea.

At this point, you're probably asking yourself why would you want to be build one when there are a million open-source and commercial options out there?

True, there are many options out there. And we looked at a good number before we made the decision to build one ourselves. But at least at the time (circa 2014), many of the existing options were either not designed for a distributed environment, were designed more particularly for data-processing use cases, were seemingly abandoned, or simply felt over-engineered to our taste.

The initial iteration of the workflow engine allowed us to start processing 'low-risk' content such as trailers, and as we gained confidence in the new system we slowly phased out the old process completely.

Later on, when a co-worker left to another media company that needed a similar system, he asked me if I'd like to come over and do it all over again from scratch. Naturally, I agreed. Our 2.0 was similar in spirit but a lot of the lessons learned from the old design were fixed in the new design.

Meet Tork

After building two proprietary workflow engines for highly specialized use cases I had an itch to see if other companies - possibly with vastly different use cases - could also benefit from a similar system. So I decided to build an open-source version of it.

Tork is a Golang-based implementation which is very similar in spirit to its closed-source predecessors. It can run in a 'standalone' mode on a laptop or deployed to a huge cluster of machines depending on your need.

The main components of Tork are:

Coordinator: responsible for managing the lifecycle of jobs and tasks, routing tasks to the right workers and for dealing with task execution errors.
Worker: responsible for executing tasks according to the instructions of the Coordinator. Workers are stateless so it's easy to add and remove them as demand for capacity changes.
Broker: the means of communication between the Coordinator and worker nodes.
Datastore: holds the state for tasks and jobs.
Runtime: tasks are executed by means of a runtime which translates the task into actual executable. Currently only Docker is supported due to its ubiquity and large library of images, but there are plans to add support for Podman and WASM in the future.

Hello world

Tork jobs are authored in YAML:

# hello.yaml
---
name: hello job
tasks:
  - name: say hello
    image: ubuntu:mantic # docker image
    run: | # arbitrary script
      echo -n hello world 
  - name: say goodbye
    image: ubuntu:mantic
    run: |
      echo -n bye world

and submitted through the API:

JOB_ID=$(curl \
  -s \
  -X POST \
  --data-binary @hello.yaml \
  -H "Content-type: text/yaml" \
  http://localhost:8000/jobs | jq -r .id)

To query for a job's status:

curl -s http://localhost:8000/jobs/$JOB_ID | jq .

{
  "id": "ed0dba93d262492b8cf26e6c1c4f1c98",
  "state": "COMPLETED",
  ...
}

Tasks can also specify auto-retry count, timeout, enforce CPU and RAM limits and do other fun things such as conditionals, loops and parallel execution.

If you're interested in checking it out, you can find the project on Github:

Backend: https://github.com/runabol/tork
Web UI: https://github.com/runabol/tork-web

Making your relational database a little more graphy

Arik — Fri, 21 Jun 2019 01:25:34 +0000

When it comes to storing data, relational database had always been my go-to solution. They're reliable, they're mature, their Structured Query Language (or SQL) is quite powerful and if you design and treat them well, they generally provide great performance.

In relational databases, everything is a table. So for example if you want to store -- say a list of customers -- then you will probably create a customers table where these will be stored.

For each table in your database you would also define the pieces of information you want to store on each item in the table. These pieces of information are technically known as "fields".

Here's an example of a very simple customers table:

CREATE TABLE customers (
  id int,
  last_name varchar(255),
  first_name varchar(255),
  email_address varchar(255)
);

If you also want to store -- say, the customer orders -- you can create an orders table where each item (or row) in the table would be an order with a link (or relation) to the customer who made the order.

CREATE TABLE orders (
  id int,
  customer_id int,
  order_date date,
  order_status varchar(10),
  order_total decimal
);

That second field -- customer_id -- is the id of the customer in the customers table and is what creates the relation between items in the orders table to items in the customers table.

This is a great way to model your data. And in fact many many software projects are built in just this way.

More recently a new type of database started to become more mainstream: Graph Database. Graph databases take a slightly different approach to thinking/modeling your data.

Rather than thinking of your data as a series of tables, it encourages you to think about your data as a network of objects. So a typical example would be your Facebook/Linkedin network. You are connected to someone, that person is connected in turn to someone else and so on.

In graph databases there are really only two types of things:

Vertex/Node - the fundamental unit which make up the elements of your graph. So a "Friend" in the Facebook example would be a node/vertex.
Edge - acts as a link between nodes/vertices.

Here's an example of what this might look like:

So the yellow circles are nodes/vertices and the lines are the edges.

On the surface this might seem pretty similar to the relational database idea of relations. That's true so long as your network of objects isn't too "wide". Once you have many "degrees of relations" -- i.e. an object links to an object which links to another one and so on -- SQL queries can become somewhat unwieldy to deal with.

This was all interesting to me when I first learned about it, but because I didn't have any practical application for it at the time, I parked it at the back of my head.

And then one day I was faced with an interesting problem I hadn't had to deal with in the past: building a Content Management System. Let's leave aside for a second the build vs. not build a CMS discussion. This is not the point of this article. Let's just say that we had no choice but to build one.

This thing with CMSes though, is that unlike traditional customers/orders type databases, they typically have a much richer set of objects. So in our case, being in the television industry, we had to store films and episodes and series and videos and things like that. But we also wanted to drive the entire experience on the screen straight from our CMS.

That means that we wanted to give our editors the power to define how the homepage would look like for various types of users at a certain point in time. Or we wanted them to schedule certain collection of movies that would appear on the homepage (think Netflix type of experience). Or we wanted certain collection of movies to be driven automatically based on the user interaction with the website.

Looking over this data model it was clear that we had a fairly complex network of objects in our hands. It was at this point that I recalled that graph database thing I parked at the back of my head. This seems like a perfect fit for it.

I was sold on the idea of using a "graphy" approach but I wasn't sold about the idea of using anything but my venerable relational databases. None of the "real" graph databases seemed to have the level of maturity and robustness I learned to love in the many years of using a relational database.

So I wondered if there was a way for me to eat my cake and have it too. Or use a relational database but as a graph database. There were a couple of things I had to solve in order to make this happen:

How do I store the data?
How do I query the data without writing complicated and therefore brittle SQL queries.
What about performance?

The first problem ended up being relatively easy to solve:

create table node (
  id          varchar(64) primary key,
  node_type   varchar(64) not null,
  created_at  timestamp not null default current_timestamp,
  modified_at timestamp,
  properties  jsonb not null,
  deleted     boolean not null default false
);

create table edge (
  id           varchar(64) primary key,
  edge_type    varchar(64) not null,
  created_at   timestamp not null default current_timestamp,
  modified_at  timestamp,
  from_node_id varchar(64) not null references node (id),
  to_node_id   varchar(64) not null references node (id),
  properties   jsonb not null,
  deleted      boolean not null default false
);

Using just two tables I was able to get any conceivable object stored in the node table and be able to arbitrarily connect it to any other object using the edge table. Also, using PostgreSQL's jsonb table I did away with the need to alter my schema ever again to add custom properties/fields.

To solve the second problem, I wrote a pretty thin wrapper around my SQL library to allow me to query my graph of objects in a "graphy" way:

 // list all actors for the home alone movie
 Traversal<Node> t = g.nodes().type("Movie").eq("title","Home Alone").to("Actor");

For the third problem, I had to use a little bit of caching magic on the application side. But since the data model is really simple (2 tables) it ended up being pretty straightforward to accomplish.

The solution I just described here ended up being used to power every aspect of our website and mobile apps: https://watch.smithsonianchannel.com

The power of abstraction

Arik — Thu, 06 Dec 2018 02:13:18 +0000

If someone had asked me "what is the number 1, absolutely most important concept in all of software development?" I would answer it using one word: "abstraction".

Abstraction is all about reducing an idea to its absolute essence.

Let's look at an example: take the activity of running. If you take a closer look at what is actually happening in your body while you run, your head would spin: a myriad of muscles are keeping your body balanced, your heart is pumping blood like crazy, your lungs are working full speed to keep your blood oxygenated, your sweat glands are pouring sweat on your skin to keep you cool and on and on.

But to you it's simply running. Your mind simply abstracted, or removed the non-essential details from this concept of running so you can easily think about it.

Let's look at another example. Here's some code:

var splitString = str.split("");
var reverseArray = splitString.reverse();
var joinArray = reverseArray.join("");
return joinArray;

Now let's add abstraction:

function reverseString (str) {
  var splitString = str.split("");
  var reverseArray = splitString.reverse();
  var joinArray = reverseArray.join("");
  return joinArray;
}

In the first example we have to read every line of code in sequence and keep track of what's happening in order to understand it. If this was part of a larger piece of code we would potentially have to do that over and over again throughout the lifetime of the software.

In the second example however, we give a name to that piece of code which lets us ignore the internal details and keep the essential concept in mind: reversing a string. Aha!

This idea is so central to computers (and life, really) that when you think about it for a moment you realize that the hardware and software you use every day are composed of "layers" (or abstractions) of software. From silicon to machine code, to assembly code, to the operating system, to your application code, each layer can be easily reasoned about without exploding our heads with details.

Creating clear abstraction in software is the essence of software design. It's the difference between understandable, easy to maintain software and a nightmare to deal with.

Tell-tales of the expert programmer

Arik — Sat, 27 Oct 2018 19:52:54 +0000

Early on in my programming career, I was very lucky to have had the opportunity to work with an expert programmer who took me under his wing. (I prefer using the word "expert" over the word "senior", because the word "senior" is too muddied up with vague notions about what it means for someone to be a "senior" programmer).

Being a young programmer and with very little experience under my belt, I didn't consider my mentor to be particularly exceptional. Sure, he was doing a great job. But having nothing to compare it against, I just figured this is how all programmers perform. Little did I know.

Over the next 12 years I've crossed paths with many programmers -- some great, some not so much. Nonetheless, having worked with programmers of varying degrees of ability allowed me to put that initial experience into perspective. Moreover, I began noticing certain key characteristics, common to almost all expert programmers I had run into:

Craftsmanship

Expert programmers treat their work like the age-old fine craftsmen: quality is king. While average engineers churn out reams and reams of code, expert programmers write code that reads more like poetry than code. Their code has an actual aesthetic quality to it and they believe that "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away."-Antoine de Saint-Exupery

Responsibility

Expert programmers really own their work. Expert programmers don't wait around for "sprint planning", to be assigned tasks or for someone else to write "stories" for them. They simply take ownership of the problem and get to work. This also extends to not blindly following rote requirements and just do what their told -- thus passing their responsibility elsewhere . Instead, they constantly challenge the status quo in order to get to the bottom of the situation and to deliver the right solution. They believe in giving their customers what they need, not what they think they want.

Expert programmers make mistakes like any other programmers. But rather than blaming the stars, the weather and the dog next door for their faulty code, they get down to business to solve the bug. They do not dismiss bugs as a "one-off glitch" or (god forbid) write workarounds around them. Rather, they find the root cause of the problem and eliminate it.

Agility

Interestingly, and almost counter-intuitively, expert programmers move very quickly from problem to solution. Despite spending more time on properly designing, writing and testing their solution they usually deliver much quicker than the average programmer. This is because expert programmers rarely start from scratch. Their software is typically built in a way that promotes reusability and extensibility.

Another advantage of the Expert programmer is that they recognize that the vast majority of problems were already solved time and again by other programmers. These solutions can often be found in the form of code libraries, write-ups and best practices. While average programmers will try to solve the problem with the tools and knowledge they have, expert programmers will seek other viewpoints, even if it means getting into uncharted waters.

Passion

Expert programmers are some of the most passionate individuals I have ever met. They love what they do and their job seems to me more like play than work. As a result, programming is not something they put aside when their work day is over. They usually spend significant amounts of their free time reading about, practicing and improving their craft. It is this diligence of practice that often has the expert programmer appear as exceptionally gifted. What usually passes as talent is actually consistent study and practice over a long period of time.

Parting Thoughts

If you run into one of these expert programmers, I have but only one piece of advice to give you: "leach" on to them like your life depends on it and learn everything you can learn. No book can replace the one-on-one experience and the gains attendant to apprenticing under an expert programmer.

Also, if you have a related story to share I'd love to hear about it.

Why I love Java

Arik — Tue, 23 Oct 2018 02:21:04 +0000

Java is unarguably one of the most popular programming languages in the world today. Companies like Google, Amazon and Netflix -- to name a few big names -- are using Java to build large portions of their infrastructure and backend services. But despite its huge popularity (or perhaps because of it) Java's reputation among programmers is something of a mixed bag. Some seem to genuinely like it while others hate it.

Personally, as someone who has been using the language for more than a decade professionally, I find it to be an excellent language. No, it isn't perfect. And the programming world -- being young as it is -- is still learning from its own mistakes. But all other things being equal I think that Java got a lot more right than wrong.

Following are some of the things I think Java got right:

It's a productive language

As a programmer I get paid to solve problems. In minimum time and maximum quality. My software is expected to be robust (bug free), performant and maintainable (easily extended). I typically write large, distributed and fairly complex server-side software.

Java is a purely Object Oriented language. To the uninitiated, this means that you design your code around unit of codes called "objects" which loosely resemble real-world object (or concepts). So if you were to write, say, a library management software you'll most likely have these code objects representing books, members, staff but also more abstract concepts such as genre and loan.

This happens to be a very neat way to organize software (and your thoughts). Instead of thinking about your system as a big line-by-line, step-by-step algorithm, you think of it as a set of interacting objects. If done right, each one of these objects can be reasoned about independently from other objects. Each object can be tested in isolation from others and each one can be extended without necessarily affecting the entire system.

Java did not invent the idea of Object Oriented Programming. In fact OOP dates back to the late 60's, but the Java language designers did a great job implementing the idea in the language in such a way that it is practical and productive for the programmer.

The Java Virtual Machine

One of Java's most convenient features is the Java Virtual Machine (or JVM). The JVM essentially acts as the translator between your Java code and the particular operating system that your code is running on. This is the origin of the once-famous Java marketing slogan "write once, run anywhere". Without the JVM, you'd have to compile your code for every operating system separately. the JVM guarantees that your Java code runs identically on Linux, Mac or Windows.

The Java language designers also envisioned a world where other languages can execute on the JVM besides Java. To that end they've specified a low-level language (called "bytecode") which is the language that Java complies into and the stuff that actually gets executed on the JVM. Today there are dozens of languages targeting the JVM as a result.

Automatic Memory Management

If you ever used a language such as C or C++ then you are intimately familiar with the painstaking effort involved in manually managing memory allocation (and deallocations) in your programs. This is an area, so rife with bugs that the designers of the Java Programming Language decided to do away with it completely. Rather than giving the programmer the opportunity to shoot themselves in the foot they decided to let Java do all the memory management for them -- automatically.

In Java, the programmer simply creates whatever objects they need (without explicitly worrying about allocating memory for it) and Java will automatically reclaim that memory once the object is no longer used by the program.

Standing on the shoulders of giants

Java was designed and implemented by some of the brightest people in the Computer Science world. People such a Doug Steele, Joshua Bloch, Mark Reinhold and Brian Goetz are true masters of their craft and are a constant source of inspiration and learning for me. Not just about Java per se, but more importantly, about writing better software in general.

Huge community

As of this writing, according to Wikipedia, there are 9 million Java developers around the world. This is a huge advantage because for just about every problem you run into as a developer there is a high likelihood that someone, somewhere had already solved a variation of this problem or that very same problem. Usually a simple online search will reveal multiple sources that can be used to solve or at the very least approach the problem.

Fantastic frameworks & libraries

Java has an enormous array of frameworks and libraries to choose from. From logging, to web development, to database connectivity, to messaging clients, there's a library (or several) to help solve just about any problem. A particular favorite of mine is the Spring Framework which had become the defacto framework for any sort of work involving web (HTTP) technologies.

Writing web application using "raw" Java would be a tedious task, to say the least. Moreover, the designers of Spring noticed that many programmers are trying to solve the same (or very similar) problems and each one is solving these problems slightly differently. Spring (and its vast portfolio of sub-projects) are meant to give the programmer a leg-up by giving them a solid set of building blocks to build upon for their projects.

Closing thoughts

There are many fine languages to choose from out there. Java is just one of them. This article is in no way trying to claim that "Java is the best" or that "Java is better than X language". It is simply meant as an acknowledgement to the fine work that made Java possible.