Forem: Isaac Lyman

50 for FOSS needs your help

Isaac Lyman — Mon, 16 Sep 2024 16:21:00 +0000

tl;dr: I'm organizing a monthly community event in support of open-source software. You can join up at 50forFOSS.org.

How it started

It all began with a Mastodon post:

This was my most popular Mastodon post ever. By the end of the day it had been boosted across 28 instances. A lot of people must have been feeling the same thing I was: an urge to protect the fragile and extraordinary world of free, open-source software.

This wasn't the first time I'd brought it up. A year ago, I commented:

For years, I've been concerned about the state of Free and Open-Source Software (FOSS). I depend on it for work, entertainment, and productivity at home. But I'm constantly finding out that libraries in my 'node_modules' can't be updated because they've been abandoned, or an app I just learned about is already obsolete because the creator burned out, or a project I want to use isn't compatible with my hardware because the maintainer doesn't have time to work on it.

Think about all the open-source software you use every day:

Applications on your computer.
Web apps you access online.
Plugins and extensions in your web browser.
Mods for your video games.
Apps on your smartphone.
Add-ons for your productivity and notetaking tools.
Web services on your home server or NAS.
Dependency files in your code repositories.
Even the Linux distribution you use.

FOSS is an incredibly valuable resource, something that's totally unique to our industry. But its continued existence isn't guaranteed. We can all do more to keep it alive.

The future of truly great software is in principled, counterculture communities. And we can start building those communities today.

The pledge

50 for FOSS is a monthly pledge that anyone can sign.

Here's how it works: on the first Friday of each month, pick a FOSS project and give the maintainer(s) $50.

Then, if you'd like, share your donation on social media with the hashtag #50forFOSS.

There's no middleman, no nonprofit organization, no gatekeepers or committees. It's just you, your wallet, and a FOSS maintainer who's about to have a very good day.

Why do I believe this will work? Because I know there are millions of software developers out there who, like me, truly appreciate the open-source work of their peers—but, like me, haven't known how to show it.

50 for FOSS is about open-source community and appreciation. It's a habit that's easy to adopt. And for some maintainers, it will make all the difference in the world.

How do I join?

Visit 50forFOSS.org to learn more, get all your questions answered, and sign the pledge.

Once a month, I'll send out an email reminder and spotlight a couple of open-source projects the community loves. It will be up to you to choose a project and send them your donation.

The first ever 50 for FOSS Day will be October 4, 2024.

I'll see you there.

#50forFOSS

I've spent 10 years building my portfolio. What was it all for?

Isaac Lyman — Fri, 16 Feb 2024 16:49:00 +0000

They tell you if you want to be competitive as a developer, you have to go above and beyond. They tell you to work outside of work: contribute to open-source projects, learn new programming languages and frameworks, build apps, write a blog, amass followers on social media.

Since 2014, when I took my first programming job, I've been doing all of those things. I'm active on GitHub. I maintain a popular open-source tool. I ran a software micro-startup for three years. I know Rust, Dart, Vue and React, none of which I've ever used at work. I've published three mobile apps and a handful of websites. Between my blog, the DEV community, Medium, and the Stack Overflow Blog, I've reached hundreds of thousands of readers. I even wrote a book.

Want to know how job interviews go for me?

The same way they go for everyone else.

First, I send in an application online. Then I spend about a week trying to get in touch with someone at the company via email or LinkedIn, since online applications are useless. Then there's a phone call with a recruiter. Then one or two technical interviews, which half the time are just vocabulary quizzes (the industry sucks at interviewing). Then they make me an offer, or not—and if they do, it's strictly based on years of experience. I don't know for sure, but I'd guess the average hiring manager spends about ten seconds skimming through the second and third pages of my resume, where my non-work projects live.

In ten years, I think there's been one time when my side projects materially changed the direction of an interview. (And they didn't end up making me an offer, anyway.)

If this sounds like a lot of moaning and complaining, well, fair enough. That's how I feel today. I don't think it's unreasonable to be disappointed when you've put in the work and the promised results are nowhere to be seen.

On the positive side, I don't think all that time has been wasted. I like coding. Nothing is a waste of time if you enjoy it. Besides, I've learned a lot by doing it on my own, with no safety net or guardrails; a lot of my seniority at work is due to the intuition I've built with side projects.

But I've got a new attitude going forward. I'm done saying "this is for my portfolio" or "this will help me get my next job." It's not, and it won't. From now on, I'm coding because I like doing it and I want to. It may not look any different from the outside, but I'll be operating with different expectations. And if, on occasion, I want to abandon a project or spend a year doing Sudoku after work instead of pushing commits, I will.

Has your experience been different? Have your side projects turned into job opportunities? Let's hear your side of it in the comments.

Solving the CIA Kryptos Code (Part 4)

Isaac Lyman — Fri, 26 Jan 2024 22:55:00 +0000

You can find all the code for this series on GitHub.

Kryptos 4 is, I repeat, unsolved. We're very unlikely to solve it today. Enthusiastic codebreakers have been putting together clues and proposing theories for decades. At least half of them are like this:

But hey, since we've got some code ready to go, we might as well try.

Setup

All the code we've written so far is available on GitHub. I won't repeat it here. If you want to play along, you can clone the repo and use ts-node to run the files.

Kryptos 4

The source text of Kryptos 4 is:

OBKR
UOXOGHULBSOLIFBBWFLRVQQPRNGKSSO
TWTQSJQSSEKZZWATJKLUDIAWINFBNYP
VTTMZFPKWGDKZXTJCDIGKUHUAUEKCAR

Thanks to a few clues from the sculptor, we know the decrypted message includes the phrases "BERLINCLOCK" and "EASTNORTHEAST" at certain positions. With that knowledge, we can write a function to rate possible solutions:

export function ratePossibleK4Solution(attempt: string): number {
  attempt = attempt.toUpperCase();
  const reversed = attempt.split('').reverse().join('');

  let score = 0;

  if (attempt.includes('EASTNORTHEAST') || reversed.includes('EASTNORTHEAST')) {
    score++;
  }

  if (
    attempt.slice(21).startsWith('EASTNORTHEAST') ||
    reversed.slice(21).startsWith('EASTNORTHEAST')
  ) {
    score++;
  }

  if (attempt.includes('BERLINCLOCK') || reversed.includes('BERLINCLOCK')) {
    score++;
  }

  if (
    attempt.slice(63).startsWith('BERLINCLOCK') ||
    reversed.slice(63).startsWith('BERLINCLOCK')
  ) {
    score++;
  }

  return score;
}

A score of 4 would be incredible, but even 2 would be exciting. As I've written code for Kryptos 1-3, I've made several little mistakes that sometimes resulted in a decrypted but offset message, like VEWATERILO instead of ILOVEWATER. So getting the hinted phrases to turn up at all could turn out to be a complete solution.

Attempt 1

Let's try all the ciphers we've built so far:

Vigenere with a standard alphabet
Vigenere with the special KRYPTOS alphabet
Keyed columnar tranposition
Route transposition
Routed keyed columnar transposition


const results = [];

// VIGENERE > PALIMPSEST

const vigStandard1 = vigDecrypt(encryptedK4, 'PALIMPSEST', standardAlphabet);
results.push({
  description: 'Vigenere | PALIMPSEST | Standard alphabet',
  score: ratePossibleK4Solution(vigStandard1),
  decrypted: vigStandard1,
});

const vigReversed1 = vigEncrypt(encryptedK4, 'PALIMPSEST', standardAlphabet);
results.push({
  description: 'Vigenere (reversed) | PALIMPSEST | Standard alphabet',
  score: ratePossibleK4Solution(vigReversed1),
  decrypted: vigReversed1,
});

const vigKryptos1 = vigDecrypt(
  encryptedK4,
  'PALIMPSEST',
  generateAlphabet('KRYPTOS')
);
results.push({
  description: 'Vigenere | PALIMPSEST | KRYPTOS alphabet',
  score: ratePossibleK4Solution(vigKryptos1),
  decrypted: vigKryptos1,
});

const vigReversedKryptos1 = vigEncrypt(
  encryptedK4,
  'PALIMPSEST',
  generateAlphabet('KRYPTOS')
);
results.push({
  description: 'Vigenere (reversed) | PALIMPSEST | KRYPTOS alphabet',
  score: ratePossibleK4Solution(vigReversedKryptos1),
  decrypted: vigReversedKryptos1,
});

// VIGENERE > ABSCISSA

const vigStandard2 = vigDecrypt(encryptedK4, 'ABSCISSA', standardAlphabet);
results.push({
  description: 'Vigenere | ABSCISSA | Standard alphabet',
  score: ratePossibleK4Solution(vigStandard2),
  decrypted: vigStandard2,
});

const vigReversed2 = vigEncrypt(encryptedK4, 'ABSCISSA', standardAlphabet);
results.push({
  description: 'Vigenere (reversed) | ABSCISSA | Standard alphabet',
  score: ratePossibleK4Solution(vigReversed2),
  decrypted: vigReversed2,
});

const vigKryptos2 = vigDecrypt(
  encryptedK4,
  'ABSCISSA',
  generateAlphabet('KRYPTOS')
);
results.push({
  description: 'Vigenere | ABSCISSA | KRYPTOS alphabet',
  score: ratePossibleK4Solution(vigKryptos2),
  decrypted: vigKryptos2,
});

const vigReversedKryptos2 = vigEncrypt(
  encryptedK4,
  'ABSCISSA',
  generateAlphabet('KRYPTOS')
);
results.push({
  description: 'Vigenere (reversed) | ABSCISSA | KRYPTOS alphabet',
  score: ratePossibleK4Solution(vigReversedKryptos2),
  decrypted: vigReversedKryptos2,
});

// COLUMNAR > KRYPTOS

const columnar1 = columnarDecrypt(encryptedK4, 'KRYPTOS');
results.push({
  description: 'Keyed columnar | KRYPTOS',
  score: ratePossibleK4Solution(columnar1),
  decrypted: columnar1,
});

const columnarReversed1 = columnarEncrypt(encryptedK4, 'KRYPTOS');
results.push({
  description: 'Keyed columnar (reversed) | KRYPTOS',
  score: ratePossibleK4Solution(columnarReversed1),
  decrypted: columnarReversed1,
});

// ROUTE > (86, 7)

const route1 = routeDecrypt(encryptedK4, 86, 7);
results.push({
  description: 'Route | 86,7',
  score: ratePossibleK4Solution(route1),
  decrypted: route1,
});

const routeReversed1 = routeEncrypt(encryptedK4, 86, 7);
results.push({
  description: 'Route (reversed) | 86,7',
  score: ratePossibleK4Solution(routeReversed1),
  decrypted: routeReversed1,
});

// ROUTED COLUMNAR > (86, KRYPTOS)
const routedColumnar1 = routedColumnarDecrypt(encryptedK4, 86, 'KRYPTOS');
results.push({
  description: 'Routed columnar | 86,KRYPTOS',
  score: ratePossibleK4Solution(routedColumnar1),
  decrypted: routedColumnar1,
});

const routedColumnarReversed1 = routedColumnarEncrypt(encryptedK4, 86, 'KRYPTOS');
results.push({
  description: 'Routed columnar (reversed) | 86,KRYPTOS',
  score: ratePossibleK4Solution(routedColumnarReversed1),
  decrypted: routedColumnarReversed1,
});

// LOG RESULTS

console.table(
  results.map((result) => ({
    ...result,
    decrypted: result.decrypted.slice(0, 15) + '...',
  }))
);

That's 14 swings and 14 misses:

┌─────────┬────────────────────────────────────────────────────────┬───────┬──────────────────────┐
│ (index) │                      description                       │ score │      decrypted       │
├─────────┼────────────────────────────────────────────────────────┼───────┼──────────────────────┤
│    0    │      'Vigenere | PALIMPSEST | Standard alphabet'       │   0   │ 'ZBZJIZFKOOFLQKC...' │
│    1    │ 'Vigenere (reversed) | PALIMPSEST | Standard alphabet' │   0   │ 'DBVZGDPSYAJLMAA...' │
│    2    │       'Vigenere | PALIMPSEST | KRYPTOS alphabet'       │   0   │ 'YRCFPYMQADMDLLG...' │
│    3    │ 'Vigenere (reversed) | PALIMPSEST | KRYPTOS alphabet'  │   0   │ 'BILJGBTJNMXXZUW...' │
│    4    │       'Vigenere | ABSCISSA | Standard alphabet'        │   0   │ 'OASPMWFOGGCJTAW...' │
│    5    │  'Vigenere (reversed) | ABSCISSA | Standard alphabet'  │   0   │ 'OCCTCGPOGIMNJKG...' │
│    6    │        'Vigenere | ABSCISSA | KRYPTOS alphabet'        │   0   │ 'XKQMSZMXSSIBNKZ...' │
│    7    │  'Vigenere (reversed) | ABSCISSA | KRYPTOS alphabet'   │   0   │ 'FJSDDETFQVRKWFE...' │
│    8    │               'Keyed columnar | KRYPTOS'               │   0   │ 'OSDRMOIBSINZLAK...' │
│    9    │         'Keyed columnar (reversed) | KRYPTOS'          │   0   │ 'OOOFRTSAINZKIUO...' │
│   10    │                     'Route | 86,7'                     │   0   │ 'OLGSIMCKFSKWFIU...' │
│   11    │               'Route (reversed) | 86,7'                │   0   │ 'OKOKOFRTSAINZKI...' │
│   12    │             'Routed columnar | 86,KRYPTOS'             │   0   │ 'RDOTPJJCLBVQXQE...' │
│   13    │       'Routed columnar (reversed) | 86,KRYPTOS'        │   0   │ 'OKOKOFRTSAINZKI...' │
└─────────┴────────────────────────────────────────────────────────┴───────┴──────────────────────┘

Attempt 2

What if we try all three ciphers in sequence?

const results = [];

// K1-K3 forwards

(function () {
  const vigKryptos1 = vigDecrypt(
    encryptedK4,
    'PALIMPSEST',
    generateAlphabet('KRYPTOS')
  );
  const vigKryptos2 = vigDecrypt(
    vigKryptos1,
    'ABSCISSA',
    generateAlphabet('KRYPTOS')
  );
  const routedColumnar = routedColumnarDecrypt(vigKryptos2, 86, 'KRYPTOS');

  results.push({
    description: 'K1-K3 ciphers',
    score: ratePossibleK4Solution(routedColumnar),
    decrypted: routedColumnar,
  });
})();


// K3-K1 backwards

(function () {
  const routedColumnar = routedColumnarEncrypt(encryptedK4, 86, 'KRYPTOS');

  const vigKryptosReversed2 = vigEncrypt(
    routedColumnar,
    'ABSCISSA',
    generateAlphabet('KRYPTOS')
  );
  const vigKryptosReversed1 = vigEncrypt(
    vigKryptosReversed2,
    'PALIMPSEST',
    generateAlphabet('KRYPTOS')
  );

  results.push({
    description: 'K3-K1 ciphers (reversed)',
    score: ratePossibleK4Solution(vigKryptosReversed1),
    decrypted: vigKryptosReversed1,
  });
})();

// LOG RESULTS

console.table(
  results.map((result) => ({
    ...result,
    decrypted: result.decrypted.slice(0, 15) + '...',
  }))
);

No luck!

┌─────────┬────────────────────────────┬───────┬──────────────────────┐
│ (index) │        description         │ score │      decrypted       │
├─────────┼────────────────────────────┼───────┼──────────────────────┤
│    0    │      'K1-K3 ciphers'       │   0   │ 'HKACCRSEVYDCGBO...' │
│    1    │ 'K3-K1 ciphers (reversed)' │   0   │ 'BMKUWQJOHRJTQUH...' │
└─────────┴────────────────────────────┴───────┴──────────────────────┘

Attempt 3

All right, last one. Let's try each possible combination of two ciphers from (K1, K2, K3).

const results = [];

// Vigenere PALIMPSEST + Vigenere ABSCISSA

(function () {
  const vigPalimpsest = vigDecrypt(encryptedK4, 'PALIMPSEST', generateAlphabet('KRYPTOS'));
  const vigAbscissa = vigDecrypt(vigPalimpsest, 'ABSCISSA', generateAlphabet('KRYPTOS'));

  results.push({
    description: 'K1 + K2 ciphers',
    score: ratePossibleK4Solution(vigAbscissa),
    decrypted: vigAbscissa,
  });
})();


(function () {
  const vigAbscissa = vigEncrypt(encryptedK4, 'ABSCISSA', generateAlphabet('KRYPTOS'));
  const vigPalimpsest = vigEncrypt(vigAbscissa, 'PALIMPSEST', generateAlphabet('KRYPTOS'));

  results.push({
    description: 'K2 + K1 ciphers (reversed)',
    score: ratePossibleK4Solution(vigPalimpsest),
    decrypted: vigPalimpsest,
  });
})();

// Vigenere PALIMPSEST + routed columnar

(function () {
  const vigKryptos = vigDecrypt(encryptedK4, 'PALIMPSEST', generateAlphabet('KRYPTOS'));
  const routedColumnar = routedColumnarDecrypt(vigKryptos, 86, 'KRYPTOS');

  results.push({
    description: 'K1 + K3 ciphers',
    score: ratePossibleK4Solution(routedColumnar),
    decrypted: routedColumnar,
  });
})();

(function () {
  const routedColumnar = routedColumnarEncrypt(encryptedK4, 86, 'KRYPTOS');
  const vigKryptos = vigEncrypt(routedColumnar, 'PALIMPSEST', generateAlphabet('KRYPTOS'));

  results.push({
    description: 'K3 + K1 ciphers (reversed)',
    score: ratePossibleK4Solution(vigKryptos),
    decrypted: vigKryptos,
  });
})();

// Vigenere ABSCISSA + routed columnar

(function () {
  const vigKryptos = vigDecrypt(encryptedK4, 'ABSCISSA', generateAlphabet('KRYPTOS'));
  const routedColumnar = routedColumnarDecrypt(vigKryptos, 86, 'KRYPTOS');

  results.push({
    description: 'K2 + K3 ciphers',
    score: ratePossibleK4Solution(routedColumnar),
    decrypted: routedColumnar,
  });
})();

(function () {
  const routedColumnar = routedColumnarEncrypt(encryptedK4, 86, 'KRYPTOS');
  const vigKryptos = vigEncrypt(routedColumnar, 'ABSCISSA', generateAlphabet('KRYPTOS'));

  results.push({
    description: 'K3 + K2 ciphers (reversed)',
    score: ratePossibleK4Solution(vigKryptos),
    decrypted: vigKryptos,
  });
})();

// LOG RESULTS

console.table(
  results.map((result) => ({
    ...result,
    decrypted: result.decrypted.slice(0, 15) + '...',
  }))
);

And as you might have predicted: no cigar.

┌─────────┬──────────────────────────────┬───────┬──────────────────────┐
│ (index) │         description          │ score │      decrypted       │
├─────────┼──────────────────────────────┼───────┼──────────────────────┤
│    0    │      'K1 + K2 ciphers'       │   0   │ 'UNPPHVFGKYFRYEA...' │
│    1    │ 'K2 + K1 ciphers (reversed)' │   0   │ 'IWWZYHDWKKTAHRP...' │
│    2    │      'K1 + K3 ciphers'       │   0   │ 'UAGXMCGLYLJLQHQ...' │
│    3    │ 'K3 + K1 ciphers (reversed)' │   0   │ 'RDQFBHDXANDUOIB...' │
│    4    │      'K2 + K3 ciphers'       │   0   │ 'QPZIQBCPENJFLHV...' │
│    5    │ 'K3 + K2 ciphers (reversed)' │   0   │ 'OECSOLDQBWGWPSV...' │
└─────────┴──────────────────────────────┴───────┴──────────────────────┘

Bummer! There's still an infinity of untried possibilities, though: different keys, different vigenere alphabets, different orders of operations, different ciphers altogether. Someday a lucky codebreaker will manage to brute-force it. Go ahead and clone the repo if you want to give it a try.

Best of luck!

Solving the CIA Kryptos Code (Part 3)

Isaac Lyman — Fri, 26 Jan 2024 22:54:00 +0000

You can find all the code for this series on GitHub.

Kryptos 3 is going to require more code than anything else so far. It's encoded using both the types of transposition we covered in Part 2, plus a twist.

It's written out with a rectangle size of 86 and a block size of 7.
The columns are stacked.
A keyed columnar cipher is applied with the key 'KRYPTOS'. (KRYPTOS is seven characters long, which is why our block size is 7.)
The route transposition is completed: each line is read top to bottom, from the leftmost characters to the rightmost characters.
The encrypted message is reversed. (That's the twist.)

We'll need to combine both pieces of transposition code to make this happen.

Updating the route transposition code

You'll notice the columnar transposition is (rudely) inserted right in the middle of the route transposition! We need to break the route transposition into steps: one step to arrange the message in stacked columns, then another step to read it out (and vice versa).

function chunk<T>(array: T[], chunkSize: number): T[][] {
  const result: T[][] = [];
  for (let i = 0; i < array.length; i += chunkSize) {
      const chunk = array.slice(i, i + chunkSize);
      result.push(chunk);
  }
  return result;
}

function routeEncryptArrange(message: string, rectangleSize: number, blockSize: number): string[] {
  const messageChars = message.split('');
  const messageLines = chunk(messageChars, rectangleSize);
  const chunkedLines = messageLines.map(line => chunk(line, blockSize));

  const allChunks = chunkedLines.flat();
  const finalChunk = allChunks.pop();
  const mainChunkSizes = allChunks.map(chunk => chunk.length);
  const oddChunkSize = mainChunkSizes.find(chunkSize => chunkSize !== blockSize);

  if (
    typeof oddChunkSize === 'number' &&
    Array.isArray(finalChunk) &&
    finalChunk.length !== oddChunkSize &&
    finalChunk.length !== blockSize
  ) {
    if (finalChunk.length < oddChunkSize) {
      finalChunk.push(...'Q'.repeat(oddChunkSize - finalChunk.length));
    } else {
      finalChunk.push(...'Q'.repeat(blockSize - finalChunk.length));
    }
  }

  const stackedLines: string[][] = [];

  const blocksPerLine = Math.ceil(rectangleSize / blockSize);
  for (let blockIx = 0; blockIx < blocksPerLine; blockIx++) {
    const numberOfRows = chunkedLines.length;
    for (let rowIx = 0; rowIx < numberOfRows; rowIx++) {
      const encryptedRowIx = rowIx + (blockIx * numberOfRows);
      stackedLines[encryptedRowIx] = chunkedLines[rowIx][blockIx];
    }
  }

  return stackedLines.filter(line => Array.isArray(line)).map(line => line.join(''));
}

function routeEncryptReadOut(stackedLines: string[], blockSize: number): string {
  const encryptedChars: string[] = [];
  for (let charIx = 0; charIx < blockSize; charIx++) {
    for (let rowIx = 0; rowIx < stackedLines.length; rowIx++) {
      if (typeof stackedLines[rowIx] !== 'string') {
        continue;
      }

      const char = stackedLines[rowIx][charIx];
      if (typeof char === 'string') {
        encryptedChars.push(char);
      }
    }
  }

  return encryptedChars.join('');
}

function routeDecryptReadIn(encrypted: string, rectangleSize: number, blockSize: number): string[] {
  const numberOfRows = Math.ceil(encrypted.length / rectangleSize);
  const chunksPerRow = Math.ceil(rectangleSize / blockSize);
  const lastRowLength = encrypted.length % rectangleSize;
  const lastRowChunks = Math.ceil(lastRowLength / blockSize);
  const totalChunks = (chunksPerRow * (numberOfRows - 1)) + lastRowChunks;

  const lastChunkSize = lastRowLength % blockSize;
  const oddChunkSize = rectangleSize % blockSize;

  let numberOfShortVLines: number;
  if (oddChunkSize !== blockSize) {
    numberOfShortVLines = blockSize - oddChunkSize;
  } else if (lastChunkSize !== blockSize) {
    numberOfShortVLines = blockSize - lastChunkSize;
  } else {
    numberOfShortVLines = 0;
  }

  const numberOfLongVLines = blockSize - numberOfShortVLines;

  const longVLineLength = totalChunks;
  const shortVLineLength = totalChunks -
    ((oddChunkSize !== blockSize ? numberOfRows - 1 : 0) +
     (lastChunkSize !== blockSize ? 1 : 0));

  const stackedLines: string[][] = [];
  let encryptedIx = 0;

  for (let vLineIx = 0; vLineIx < blockSize; vLineIx++) {
    const currentVLineLength = vLineIx < numberOfLongVLines ?
      longVLineLength :
      shortVLineLength;

    for (let rowIx = 0; rowIx < currentVLineLength; rowIx++) {
      stackedLines[rowIx] ??= [];

      stackedLines[rowIx][vLineIx] = encrypted[encryptedIx++];
    }
  }

  return stackedLines.filter(line => Array.isArray(line)).map(line => line.join(''));
}

function routeDecryptUnarrange(stackedLines: string[], rectangleSize: number, blockSize: number): string {
  const messageLength = stackedLines.join('').length;
  const numberOfRows = Math.ceil(messageLength / rectangleSize);
  const blocksPerRow = Math.ceil(rectangleSize / blockSize);
  const lastRowLength = messageLength % rectangleSize;
  const lastRowBlocks = Math.ceil(lastRowLength / blockSize);

  const messageLines: string[][] = [];
  let stackedLineIx = 0;

  for (let columnIx = 0; columnIx < blocksPerRow; columnIx++) {
    const rowsInBlock = columnIx < lastRowBlocks ? numberOfRows : numberOfRows - 1;
    for (let rowIx = 0; rowIx < rowsInBlock; rowIx++) {
      messageLines[rowIx] ??= [];
      messageLines[rowIx].push(stackedLines[stackedLineIx++]);
    }
  }

  return messageLines.flat().join('');
}

const rectangleSize = 15;
const blockSize = 4

const arrangedToEncrypt = routeEncryptArrange(
  'THEFITNESSGRAMPACERTESTISAMULTISTAGEAEROBICCAPACITYTEST',
  rectangleSize,
  blockSize
);
const encrypted = routeEncryptReadOut(
  arrangedToEncrypt,
  blockSize
);
console.log(encrypted);
// > TAIPITGTSIRSAUCHCSATEEYSSOTMLCEETCNSATGABQPTAFRAIETEERMI

const arrangedToDecrypt = routeDecryptReadIn(
  encrypted,
  rectangleSize,
  blockSize
);
const decrypted = routeDecryptUnarrange(
  arrangedToDecrypt,
  rectangleSize,
  blockSize
);
console.log(decrypted);
// > THEFITNESSGRAMPACERTESTISAMULTISTAGEAEROBICCAPACITYTESTQ

Updating the columnar transposition code

Now we need to modify the columnar transposition code so it can take an array of prearranged strings instead of a joined, unspaced string. You may have noticed how the columnar transposition and the route transposition have the same final step: reading out lines of characters from top to bottom, left to right. We can omit that step from the columnar transposition, since the second step of the route transposition will handle it.

function chunk<T>(array: T[], chunkSize: number): T[][] {
  const result: T[][] = [];
  for (let i = 0; i < array.length; i += chunkSize) {
      const chunk = array.slice(i, i + chunkSize);
      result.push(chunk);
  }
  return result;
}

function columnarEncrypt(stackedLines: string[], key: string): string[] {
  const keyChars = key.split('');
  const sortedKey = keyChars.toSorted();
  const positionNumbers = keyChars.map(char => sortedKey.indexOf(char));

  const encryptedRows: (string | null)[][] = Array(stackedLines.length)
    .fill(null)
    .map(_ => Array(key.length).fill(null).map(_ => null));

  for (const orderedColumnIx in keyChars) {
    const messageColumnIx = positionNumbers.indexOf(Number(orderedColumnIx));

    for (const rowIx in stackedLines) {
      encryptedRows[rowIx][orderedColumnIx] = stackedLines[rowIx][messageColumnIx] ?? ' ';
    }
  }

  return encryptedRows.map(row => row.join(''));
}

function columnarDecrypt(stackedLines: string[], key: string): string[] {
  const keyChars = key.split('');
  const sortedKey = keyChars.toSorted();
  const positionNumbers = keyChars.map(char => sortedKey.indexOf(char));

  const encryptedRows: (string | null)[][] = Array(stackedLines.length)
    .fill(null)
    .map(_ => Array(key.length).fill(null).map(_ => null));

  for (const messageColumnIx in keyChars) {
    const orderedColumnIx = positionNumbers.indexOf(Number(messageColumnIx));

    for (const rowIx in stackedLines) {
      encryptedRows[rowIx][orderedColumnIx] = stackedLines[rowIx][messageColumnIx] ?? ' ';
    }
  }

  return encryptedRows.map(row => row.join(''));
}

const key = 'HYDRATE';
const message = [
  'ILOVEWA',
  'TER'
]

const encrypted = columnarEncrypt(message, key);
console.log(encrypted);
// > ["EOAIVWL", " R T  E"]

const decrypted = columnarDecrypt(encrypted, key);
console.log(decrypted);
// > ["ILOVEWA", "TER    "]

Why are there spaces in this version? Because position is important. R has to be in the second column after encryption, not the first, or the decryption process will break.

Putting it all together

It would be great if we could just call these functions in order to encrypt and decrypt Kryptos 3. However, there's one more obstacle. Since we're dealing with shuffled-up columns after the columnarEncrypt method is called, the longest vertical lines won't always be the leftmost ones. We need to inspect the key to know which lines are long and which ones are short. Check out routeDecryptReadIn:

function chunk<T>(array: T[], chunkSize: number): T[][] {
  const result: T[][] = [];
  for (let i = 0; i < array.length; i += chunkSize) {
      const chunk = array.slice(i, i + chunkSize);
      result.push(chunk);
  }
  return result;
}

function columnarEncrypt(stackedLines: string[], key: string): string[] {
  const keyChars = key.split('');
  const sortedKey = keyChars.toSorted();
  const positionNumbers = keyChars.map(char => sortedKey.indexOf(char));

  const encryptedRows: (string | null)[][] = Array(stackedLines.length)
    .fill(null)
    .map(_ => Array(key.length).fill(null).map(_ => null));

  for (const orderedColumnIx in keyChars) {
    const messageColumnIx = positionNumbers.indexOf(Number(orderedColumnIx));

    for (const rowIx in stackedLines) {
      encryptedRows[rowIx][orderedColumnIx] = stackedLines[rowIx][messageColumnIx] ?? ' ';
    }
  }

  return encryptedRows.map(row => row.join(''));
}

function columnarDecrypt(stackedLines: string[], key: string): string[] {
  const keyChars = key.split('');
  const sortedKey = keyChars.toSorted();
  const positionNumbers = keyChars.map(char => sortedKey.indexOf(char));

  const encryptedRows: (string | null)[][] = Array(stackedLines.length)
    .fill(null)
    .map(_ => Array(key.length).fill(null).map(_ => null));

  for (const messageColumnIx in keyChars) {
    const orderedColumnIx = positionNumbers.indexOf(Number(messageColumnIx));

    for (const rowIx in stackedLines) {
      encryptedRows[rowIx][orderedColumnIx] = stackedLines[rowIx][messageColumnIx] ?? ' ';
    }
  }

  return encryptedRows.map(row => row.join(''));
}

function routeEncryptArrange(message: string, rectangleSize: number, blockSize: number): string[] {
  const messageChars = message.split('');
  const messageLines = chunk(messageChars, rectangleSize);
  const chunkedLines = messageLines.map(line => chunk(line, blockSize));

  const allChunks = chunkedLines.flat();
  const finalChunk = allChunks.pop();
  const mainChunkSizes = allChunks.map(chunk => chunk.length);
  const oddChunkSize = mainChunkSizes.find(chunkSize => chunkSize !== blockSize);

  if (
    typeof oddChunkSize === 'number' &&
    Array.isArray(finalChunk) &&
    finalChunk.length !== oddChunkSize &&
    finalChunk.length !== blockSize
  ) {
    if (finalChunk.length < oddChunkSize) {
      finalChunk.push(...'Q'.repeat(oddChunkSize - finalChunk.length));
    } else {
      finalChunk.push(...'Q'.repeat(blockSize - finalChunk.length));
    }
  }

  const stackedLines: string[][] = [];

  const blocksPerLine = Math.ceil(rectangleSize / blockSize);
  for (let blockIx = 0; blockIx < blocksPerLine; blockIx++) {
    const numberOfRows = chunkedLines.length;
    for (let rowIx = 0; rowIx < numberOfRows; rowIx++) {
      const encryptedRowIx = rowIx + (blockIx * numberOfRows);
      stackedLines[encryptedRowIx] = chunkedLines[rowIx][blockIx];
    }
  }

  return stackedLines.filter(line => Array.isArray(line)).map(line => line.join(''));
}

function routeEncryptReadOut(stackedLines: string[], blockSize: number): string {
  const encryptedChars: string[] = [];
  for (let charIx = 0; charIx < blockSize; charIx++) {
    for (let rowIx = 0; rowIx < stackedLines.length; rowIx++) {
      if (typeof stackedLines[rowIx] !== 'string') {
        continue;
      }

      const char = stackedLines[rowIx][charIx];
      if (typeof char === 'string' && char !== ' ') {
        encryptedChars.push(char);
      }
    }
  }

  return encryptedChars.join('');
}

function routeDecryptReadIn(encrypted: string, rectangleSize: number, key: string): string[] {
  const numberOfRows = Math.ceil(encrypted.length / rectangleSize);
  const blockSize = key.length;
  const chunksPerRow = Math.ceil(rectangleSize / blockSize);
  const lastRowLength = encrypted.length % rectangleSize;
  const lastRowChunks = Math.ceil(lastRowLength / blockSize);
  const totalChunks = (chunksPerRow * (numberOfRows - 1)) + lastRowChunks;

  const lastChunkSize = (lastRowLength % blockSize) || blockSize;
  const oddChunkSize = rectangleSize % blockSize;

  let numberOfShortVLines: number;
  if (oddChunkSize !== blockSize) {
    numberOfShortVLines = blockSize - oddChunkSize;
  } else if (lastChunkSize !== blockSize) {
    numberOfShortVLines = blockSize - lastChunkSize;
  } else {
    numberOfShortVLines = 0;
  }

  const numberOfLongVLines = blockSize - numberOfShortVLines;

  const longVLineLength = totalChunks;
  const shortVLineLength = totalChunks -
    ((oddChunkSize !== blockSize ? numberOfRows - 1 : 0) +
     (lastChunkSize !== blockSize ? 1 : 0));

  const stackedLines: string[][] = [];
  let encryptedIx = 0;

  // New code to inspect the key so we can detect long vs.
  //  short lines.
  const keyChars = key.split('');
  const sortedKey = keyChars.toSorted();
  const positionNumbers = keyChars.map(char => sortedKey.indexOf(char));

  for (let vLineIx = 0; vLineIx < blockSize; vLineIx++) {
    const transposedVLineIx = positionNumbers.indexOf(vLineIx);
    const currentVLineLength = transposedVLineIx < numberOfLongVLines ?
      longVLineLength :
      shortVLineLength;

    // Updated to add spaces in the empty spots on a short line
    for (let rowIx = 0; rowIx < longVLineLength; rowIx++) {
      stackedLines[rowIx] ??= [];

      stackedLines[rowIx][vLineIx] = rowIx < currentVLineLength ? encrypted[encryptedIx++] : ' ';
    }
  }

  return stackedLines.filter(line => Array.isArray(line)).map(line => line.join(''));
}

function routeDecryptUnarrange(stackedLines: string[], rectangleSize: number, blockSize: number): string {
  // Remove any empty space, we don't need it anymore
  stackedLines = stackedLines.map(line => line.replace(/ /g, ''));
  const messageLength = stackedLines.join('').length;
  const numberOfRows = Math.ceil(messageLength / rectangleSize);
  const blocksPerRow = Math.ceil(rectangleSize / blockSize);
  const lastRowLength = messageLength % rectangleSize;
  const lastRowBlocks = Math.ceil(lastRowLength / blockSize);

  const messageLines: string[][] = [];
  let stackedLineIx = 0;

  // For each column of blocks...
  for (let columnIx = 0; columnIx < blocksPerRow; columnIx++) {
    const rowsInBlock = columnIx < lastRowBlocks ? numberOfRows : numberOfRows - 1;
    // For each row in the column...
    for (let rowIx = 0; rowIx < rowsInBlock; rowIx++) {
      messageLines[rowIx] ??= [];
      messageLines[rowIx].push(stackedLines[stackedLineIx++]);
    }
  }

  return messageLines.flat().join('');
}

const message = 'THEREAREMOREHYDROGENATOMSINAWATERMOLECULETHANSTARSINTHESOLARSYSTEM';
const key = 'PLANETS';
const rectangleSize = 15;

const arrangedToEncrypt = routeEncryptArrange(message, rectangleSize, key.length);
console.log(arrangedToEncrypt);
/* >
[
  "THEREAR", "ROGENAT", "TERMOLE", "STARSIN",
  "SYSTEMQ", "EMOREHY", "OMSINAW", "CULETHA",
  "THESOLA", "D", "A", "N",
  "R"
]
*/

const columnarEncrypted = columnarEncrypt(arrangedToEncrypt, key);
console.log(columnarEncrypted);
/* >
[
  "EEHRTRA", "GNOERTA", "ROEMTEL", "ASTRSNI",
  "SEYTSQM", "OEMREYH", "SNMIOWA", "LTUECAH",
  "EOHSTAL", "    D  ", "    A  ", "    N  ",
  "    R  "
]
*/

const encrypted = routeEncryptReadOut(columnarEncrypted, key.length);
console.log(encrypted);
/* >
EGRASOSLEENOSEENTOHOETYMMUHREMRTRIESTRTSSEOCTDANRRTENQYWAAAALIMHAHL
*/

const arrangedToDecrypt = routeDecryptReadIn(encrypted, rectangleSize, key);
console.log(arrangedToDecrypt);
/* >
[
  "EEHRTRA", "GNOERTA", "ROEMTEL", "ASTRSNI",
  "SEYTSQM", "OEMREYH", "SNMIOWA", "LTUECAH",
  "EOHSTAL", "    D  ", "    A  ", "    N  ",
  "    R  "
]
*/

const columnarDecrypted = columnarDecrypt(arrangedToDecrypt, key);
console.log(columnarDecrypted);
/* >
[
  "THEREAR", "ROGENAT", "TERMOLE", "STARSIN",
  "SYSTEMQ", "EMOREHY", "OMSINAW", "CULETHA",
  "THESOLA", "D      ", "A      ", "N      ",
  "R      "
]
*/

const decrypted = routeDecryptUnarrange(columnarDecrypted, rectangleSize, key.length);
console.log(decrypted);

/* >
THEREAREMOREHYDROGENATOMSINAWATERMOLECULETHANSTARSINTHESOLARSYSTEMQ
*/

Seems stable! There's a padding Q at the end, but that's expected.

Let's see if it can handle Kryptos 3.

Solving Kryptos 3

The source text of Kryptos 3 is:

ENDYAHROHNLSRHEOCPTEOIBIDYSHNAIA
CHTNREYULDSLLSLLNOHSNOSMRWXMNE
TPRNGATIHNRARPESLNNELEBLPIIACAE
WMTWNDITEENRAHCTENEUDRETNHAEOE
TFOLSEDTIWENHAEIOYTEYQHEENCTAYCR
EIFTBRSPAMHHEWENATAMATEGYEERLB
TEEFOASFIOTUETUAEOTOARMAEERTNRTI
BSEDDNIAAHTTMSTEWPIEROAGRIEWFEB
AECTDDHILCEIHSITEGOEAOSDDRYDLORIT
RKLMLEHAGTDHARDPNEOHMGFMFEUHE
ECDMRIPFEIMEHNLSSTTRTVDOHW?

Let's plug it in! Don't forget that it's reversed.

function chunk<T>(array: T[], chunkSize: number): T[][] {
  const result: T[][] = [];
  for (let i = 0; i < array.length; i += chunkSize) {
      const chunk = array.slice(i, i + chunkSize);
      result.push(chunk);
  }
  return result;
}

function columnarEncrypt(stackedLines: string[], key: string): string[] {
  const keyChars = key.split('');
  const sortedKey = keyChars.toSorted();
  const positionNumbers = keyChars.map(char => sortedKey.indexOf(char));

  const encryptedRows: (string | null)[][] = Array(stackedLines.length)
    .fill(null)
    .map(_ => Array(key.length).fill(null).map(_ => null));

  for (const orderedColumnIx in keyChars) {
    const messageColumnIx = positionNumbers.indexOf(Number(orderedColumnIx));

    for (const rowIx in stackedLines) {
      encryptedRows[rowIx][orderedColumnIx] = stackedLines[rowIx][messageColumnIx] ?? ' ';
    }
  }

  return encryptedRows.map(row => row.join(''));
}

function columnarDecrypt(stackedLines: string[], key: string): string[] {
  const keyChars = key.split('');
  const sortedKey = keyChars.toSorted();
  const positionNumbers = keyChars.map(char => sortedKey.indexOf(char));

  const encryptedRows: (string | null)[][] = Array(stackedLines.length)
    .fill(null)
    .map(_ => Array(key.length).fill(null).map(_ => null));

  for (const messageColumnIx in keyChars) {
    const orderedColumnIx = positionNumbers.indexOf(Number(messageColumnIx));

    for (const rowIx in stackedLines) {
      encryptedRows[rowIx][orderedColumnIx] = stackedLines[rowIx][messageColumnIx] ?? ' ';
    }
  }

  return encryptedRows.map(row => row.join(''));
}

function routeEncryptArrange(message: string, rectangleSize: number, blockSize: number): string[] {
  const messageChars = message.split('');
  const messageLines = chunk(messageChars, rectangleSize);
  const chunkedLines = messageLines.map(line => chunk(line, blockSize));

  const allChunks = chunkedLines.flat();
  const finalChunk = allChunks.pop();
  const mainChunkSizes = allChunks.map(chunk => chunk.length);
  const oddChunkSize = mainChunkSizes.find(chunkSize => chunkSize !== blockSize);

  if (
    typeof oddChunkSize === 'number' &&
    Array.isArray(finalChunk) &&
    finalChunk.length !== oddChunkSize &&
    finalChunk.length !== blockSize
  ) {
    if (finalChunk.length < oddChunkSize) {
      finalChunk.push(...'Q'.repeat(oddChunkSize - finalChunk.length));
    } else {
      finalChunk.push(...'Q'.repeat(blockSize - finalChunk.length));
    }
  }

  const stackedLines: string[][] = [];

  const blocksPerLine = Math.ceil(rectangleSize / blockSize);
  for (let blockIx = 0; blockIx < blocksPerLine; blockIx++) {
    const numberOfRows = chunkedLines.length;
    for (let rowIx = 0; rowIx < numberOfRows; rowIx++) {
      const encryptedRowIx = rowIx + (blockIx * numberOfRows);
      stackedLines[encryptedRowIx] = chunkedLines[rowIx][blockIx];
    }
  }

  return stackedLines.filter(line => Array.isArray(line)).map(line => line.join(''));
}

function routeEncryptReadOut(stackedLines: string[], blockSize: number): string {
  const encryptedChars: string[] = [];
  for (let charIx = 0; charIx < blockSize; charIx++) {
    for (let rowIx = 0; rowIx < stackedLines.length; rowIx++) {
      if (typeof stackedLines[rowIx] !== 'string') {
        continue;
      }

      const char = stackedLines[rowIx][charIx];
      if (typeof char === 'string' && char !== ' ') {
        encryptedChars.push(char);
      }
    }
  }

  return encryptedChars.join('');
}

function routeDecryptReadIn(encrypted: string, rectangleSize: number, key: string): string[] {
  const numberOfRows = Math.ceil(encrypted.length / rectangleSize);
  const blockSize = key.length;
  const chunksPerRow = Math.ceil(rectangleSize / blockSize);
  const lastRowLength = encrypted.length % rectangleSize;
  const lastRowChunks = Math.ceil(lastRowLength / blockSize);
  const totalChunks = (chunksPerRow * (numberOfRows - 1)) + lastRowChunks;

  const lastChunkSize = (lastRowLength % blockSize) || blockSize;
  const oddChunkSize = rectangleSize % blockSize;

  let numberOfShortVLines: number;
  if (oddChunkSize !== blockSize) {
    numberOfShortVLines = blockSize - oddChunkSize;
  } else if (lastChunkSize !== blockSize) {
    numberOfShortVLines = blockSize - lastChunkSize;
  } else {
    numberOfShortVLines = 0;
  }

  const numberOfLongVLines = blockSize - numberOfShortVLines;

  const longVLineLength = totalChunks;
  const shortVLineLength = totalChunks -
    ((oddChunkSize !== blockSize ? numberOfRows - 1 : 0) +
     (lastChunkSize !== blockSize ? 1 : 0));

  const stackedLines: string[][] = [];
  let encryptedIx = 0;

  // New code to inspect the key so we can detect long vs.
  //  short lines.
  const keyChars = key.split('');
  const sortedKey = keyChars.toSorted();
  const positionNumbers = keyChars.map(char => sortedKey.indexOf(char));

  for (let vLineIx = 0; vLineIx < blockSize; vLineIx++) {
    const transposedVLineIx = positionNumbers.indexOf(vLineIx);
    const currentVLineLength = transposedVLineIx < numberOfLongVLines ?
      longVLineLength :
      shortVLineLength;

    // Updated to add spaces in the empty spots on a short line
    for (let rowIx = 0; rowIx < longVLineLength; rowIx++) {
      stackedLines[rowIx] ??= [];

      stackedLines[rowIx][vLineIx] = rowIx < currentVLineLength ? encrypted[encryptedIx++] : ' ';
    }
  }

  return stackedLines.filter(line => Array.isArray(line)).map(line => line.join(''));
}

function routeDecryptUnarrange(stackedLines: string[], rectangleSize: number, blockSize: number): string {
  // Remove any empty space, we don't need it anymore
  stackedLines = stackedLines.map(line => line.replace(/ /g, ''));
  const messageLength = stackedLines.join('').length;
  const numberOfRows = Math.ceil(messageLength / rectangleSize);
  const blocksPerRow = Math.ceil(rectangleSize / blockSize);
  const lastRowLength = messageLength % rectangleSize;
  const lastRowBlocks = Math.ceil(lastRowLength / blockSize);

  const messageLines: string[][] = [];
  let stackedLineIx = 0;

  // For each column of blocks...
  for (let columnIx = 0; columnIx < blocksPerRow; columnIx++) {
    const rowsInBlock = columnIx < lastRowBlocks ? numberOfRows : numberOfRows - 1;
    // For each row in the column...
    for (let rowIx = 0; rowIx < rowsInBlock; rowIx++) {
      messageLines[rowIx] ??= [];
      messageLines[rowIx].push(stackedLines[stackedLineIx++]);
    }
  }

  return messageLines.flat().join('');
}

const encrypted = 'ENDYAHROHNLSRHEOCPTEOIBIDYSHNAIACHTNREYULDSLLSLLNOHSNOSMRWXMNETPRNGATIHNRARPESLNNELEBLPIIACAEWMTWNDITEENRAHCTENEUDRETNHAEOETFOLSEDTIWENHAEIOYTEYQHEENCTAYCREIFTBRSPAMHHEWENATAMATEGYEERLBTEEFOASFIOTUETUAEOTOARMAEERTNRTIBSEDDNIAAHTTMSTEWPIEROAGRIEWFEBAECTDDHILCEIHSITEGOEAOSDDRYDLORITRKLMLEHAGTDHARDPNEOHMGFMFEUHEECDMRIPFEIMEHNLSSTTRTVDOHW?';
const key = 'KRYPTOS';
const rectangleSize = 86;

const unreversed = encrypted.split('').reverse().join('');
const arrangedToDecrypt = routeDecryptReadIn(unreversed, rectangleSize, key);
const columnarDecrypted = columnarDecrypt(arrangedToDecrypt, key);
const decrypted = routeDecryptUnarrange(columnarDecrypted, rectangleSize, key.length);
console.log(decrypted);

/* >
?SLOWLYDESPARATLYSLOWLYTHEREMAINSOFPASSAGEDEBRISTHATENCUMBEREDTHELOWERPARTOFTHEDOORWAYWASREMOVEDWITHTREMBLINGHANDSIMADEATINYBREACHINTHEUPPERLEFTHANDCORNERANDTHENWIDENINGTHEHOLEALITTLEIINSERTEDTHECANDLEANDPEEREDINTHEHOTAIRESCAPINGFROMTHECHAMBERCAUSEDTHEFLAMETOFLICKERBUTPRESENTLYDETAILSOFTHEROOMWITHINEMERGEDFROMTHEMISTXCANYOUSEEANYTHINGQ
*/

The message is:

? SLOWLY DESPARATLY SLOWLY THE REMAINS OF PASSAGE DEBRIS
THAT ENCUMBERED THE LOWER PART OF THE DOORWAY
WAS REMOVED WITH TREMBLING HANDS
I MADE A TINY BREACH IN THE UPPER LEFT HAND CORNER
AND THEN WIDENING THE HOLE A LITTLE
INSERTED THE CANDLE AND PEERED IN
THE HOT AIR ESCAPING FROM THE CHAMBER
CAUSED THE FLAME TO FLICKER
BUT PRESENTLY DETAILS OF THE ROOM WITHIN
EMERGED FROM THE MIST X
CAN YOU SEE ANYTHING Q

Presumably, the question mark should go at the end of the passage, not the beginning.

There you go! Kryptos 3, the message that stumped codebreakers for years, decoded in under 200 lines of TypeScript.

In the next post, we'll throw a few things at Kryptos 4 and see how it holds up.

Acknowledgments

Thanks to the following, who made this series possible:

The NSA: The CIA Kryptos Sculpture (Declassified)
UC San Diego: The Kryptos Sculpture Solutions
Wikipedia: Kryptos

Solving the CIA Kryptos Code (Part 2)

Isaac Lyman — Fri, 26 Jan 2024 22:53:00 +0000

You can find all the code for this series on GitHub.

In the previous post we solved Kryptos 1 and 2. Kryptos 3 is going to be a bit tougher. It doesn't use our trusty Vigenere cipher. And it doesn't use just one cipher, either; it uses two. We'll learn them separately in this post, then use them together in the next one.

Keyed columnar transposition

We need to learn a transposition cipher, which is where the letters in a message are rearranged but not changed. There are many kinds of transposition ciphers, but for starters, we'll do a keyed columnar transposition cipher. Again, you need a message and a key. Let's use "ILOVEWATER" and "HYDRATE" again.

To start, you put the key in alphabetical order and note the position of each letter. "A" is the earliest alphabetical letter in HYDRATE, so it's 0. "D" is the next alphabetical letter, so it's 1. And so on:

Column #	0	1	2	3	4	5	6
Alphabetized key	A	D	E	H	R	T	Y
Original key	H	Y	D	R	A	T	E
Position numbers	3	6	1	4	0	5	2

Our numeric key is 3614052. Since the key, HYDRATE, is 7 characters long, let's write out our message in rows of 7 characters:

Position numbers	3	6	1	4	0	5	2
Message line 1	I	L	O	V	E	W	A
Message line 2	T	E	R

Now all we have to do is order the columns by their position number and write out the encrypted message:

Position numbers	0	1	2	3	4	5	6
Message line 1	E	O	A	I	V	W	L
Message line 2		R		T			E

The encrypted message, which you read out a column at a time, is "EORAITVWLE".

To decrypt it, first we look at the length of the message: 10 characters. The key, HYDRATE, is 7 characters. We have to multiply 7 by 2 to get enough space for 10 characters, so we know there will be 2 rows, and since 14 minus 10 is 4, there will be 4 blank cells at the end of the message.

Let's write out the key and the position numbers, with two blank lines for our message, and block out the last four cells.

Key	H	Y	D	R	A	T	E
Position numbers	3	6	1	4	0	5	2
Message line 1
Message line 2				X	X	X	X

Now we can take our encrypted message, "EORAITVWLE", and start writing it out in order of the "Position numbers" row. The first letter "E" goes under 0, and then that column is full. The second letter "O" goes under 1, and since there's a space for another letter, the third letter "R" goes in that column as well. Then we move onto the 2 column, and so on.

Key	H	Y	D	R	A	T	E
Position numbers	3	6	1	4	0	5	2
Message line 1	I	L	O	V	E	W	A
Message line 2	T	E	R	X	X	X	X

And we've decrypted our original message, "ILOVEWATER".

Here's how we'd do this in TypeScript:

// Breaks an array into sub-arrays of size `chunkSize`
function chunk<T>(array: T[], chunkSize: number): T[][] {
  const result: T[][] = [];
  for (let i = 0; i < array.length; i += chunkSize) {
      const chunk = array.slice(i, i + chunkSize);
      result.push(chunk);
  }
  return result;
}

function columnarEncrypt(message: string, key: string): string {
  // First, sort the key and get the alphabetical position number of each character
  const keyChars = key.split('');
  const sortedKey = keyChars.toSorted();
  const positionNumbers = keyChars.map(char => sortedKey.indexOf(char));

  // Then break the message into rows the same length as the key
  const messageChars = message.split('');
  const messageRows = chunk(messageChars, keyChars.length);

  // Create a two-dimensional array with the same number of rows
  //  as the message and the same number of columns as the key
  const encryptedRows: (string | null)[][] = Array(messageRows.length)
    .fill(null)
    .map(_ => Array(key.length).fill(null).map(_ => null));

  // For each column of the to-be-encrypted message...
  for (const orderedColumnIx in keyChars) {
    // Find out what column its characters are currently in
    const messageColumnIx = positionNumbers.indexOf(Number(orderedColumnIx));

    // And put them in order
    for (const rowIx in messageRows) {
      encryptedRows[rowIx][orderedColumnIx] = messageRows[rowIx][messageColumnIx] ?? null;
    }
  }

  // Finally, read out the encrypted message a column at a time
  let encrypted = '';
  for (let ix = 0; ix < key.length; ix++) {
    for (const row of encryptedRows) {
      if (row[ix] === null) {
        continue;
      }

      encrypted += row[ix];
    }
  }
  return encrypted;
}

function columnarDecrypt(encrypted: string, key: string): string {
  // Again, sort the key and get position numbers
  const keyChars = key.split('');
  const sortedKey = keyChars.toSorted();
  const positionNumbers = keyChars.map(char => sortedKey.indexOf(char));

  // Figure out how many rows are needed
  const numberOfRows = Math.ceil(encrypted.length / key.length);

  // And how many empty cells there will be
  const numberOfEmpties = (numberOfRows * key.length) % encrypted.length;

  // Create a two-dimensional array with the correct number of rows
  //  and the same number of columns as the key
  const messageRows = Array(numberOfRows)
    .fill(null)
    .map(row => Array(key.length).fill(null));

  // Work backwards to fill in the number of empty cells on the last row
  const lastRow = messageRows[messageRows.length - 1];
  for (let emptyIx = 1; emptyIx <= numberOfEmpties; emptyIx++) {
    lastRow[lastRow.length - emptyIx] = '';
  }

  // Then, starting with column 0...
  let orderedColumnIx = 0;
  // For each character in the encrypted message...
  for (const encryptedChar of encrypted) {
    // Find out the position number of the current column
    const messageColumnIx = positionNumbers.indexOf(orderedColumnIx);

    // Put the current character in the first available row of that column
    const availableRowIx = messageRows.findIndex(row => row[messageColumnIx] === null);
    const availableRow = messageRows[availableRowIx];
    availableRow[messageColumnIx] = encryptedChar;

    // If all rows are full, increment the column
    if (messageRows.every(row => row[messageColumnIx] !== null)) {
      orderedColumnIx++;
    }
  }

  // Finally, flatten and join the decrypted message
  return messageRows.flat().join('');
}

const key = 'HAMLET';
const message = 'OMYOFFENCEISRANKITSMELLSTOHEAVEN';

const encrypted = columnarEncrypt(message, key);
console.log(encrypted); // > MNAMONFIILAOERSTEOEKLEYCNEHFSTSV

const decrypted = columnarDecrypt(encrypted, key);
console.log(decrypted); // > OMYOFFENCEISRANKITSMELLSTOHEAVEN

This is already quite a bit more complex than the Vigenere cypher. But we're not ready to solve Kryptos 3 yet. Not even close!

Route transposition

Another type of transposition is called route transposition. The kind we'll do right now involves wrapping the text at a certain line length, chunking each line, stacking the chunks, and reading out one vertical line at a time.

Instead of a cipher key, this algorithm uses two numbers: a rectangle size and a block size. Let's say our rectangle size is 15 and our block size is 4. To encode the message "THE FITNESSGRAM PACER TEST IS A MULTI STAGE AEROBIC CAPACITY TEST", first we'll write it out in lines of 15 characters:

THEFITNESSGRAMP
ACERTESTISAMULT
ISTAGEAEROBICCA
PACITYTEST

Then we divide each line into blocks of 4 characters:

THEF ITNE SSGR AMP
ACER TEST ISAM ULT
ISTA GEAE ROBI CCA
PACI TYTE ST

For reasons that will be clear later, we only want to have two sizes of blocks. Right now we have too many: most blocks are 4 characters, but the ones at the end of the first three lines are 3 characters and the very last one is 2. The easiest way to fix that is to add a "padding character" (a character that isn't part of the message) to the last line. We'll use Q, since it's an uncommon letter and unlikely to confuse the message.

THEF ITNE SSGR AMP
ACER TEST ISAM ULT
ISTA GEAE ROBI CCA
PACI TYTE STQ

Much better. You can see how we have four vertical columns of blocks. Let's stack those columns up:

THEF
ACER
ISTA
PACI

ITNE
TEST
GEAE
TYTE

SSGR
ISAM
ROBI
STQ

AMP
ULT
CCA

Finally, we read out the characters from top to bottom, starting with the first letter of each row (TAIPI etc.), then the second letter of each row (HCSAT etc.), and so on.

TAIP ITGT SIRS AUC HCSA TEEY SSOT MLC EETC NSAT GABQ PTA FRAI ETEE RMI

Remove the spaces, and we have the encrypted message TAIPITGTSIRSAUCHCSATEEYSSOTMLCEETCNSATGABQPTAFRAIETEERMI.

To decrypt this message, we can start by laying it out the same way we did the original message:

TAIP ITGT SIRS AUC
HCSA TEEY SSOT MLC
EETC NSAT GABQ PTA
FRAI ETEE RMI

We'll throw this away in a moment, but it shows the "shape" of the final decryption step. More importantly, if you stack the columns up like we did before, you'll see the leftmost three lines from top to bottom have 15 characters in them and the fourth only has 11. (Note that this isn't necessarily the same as counting the number of characters in each row above—it's just a coincidence they're the same for this message!)

We could also obtain this information mathematically:

There are 56 characters in the message. 56 divided by 15 is 3 remainder 11, so there are four rows, with 15 characters in the first three rows and 11 characters in the fourth.
15 divided by 4 is 3 remainder 3, so in each of the first three rows there will be three blocks of 4 and one block of 3.
11 divided by 4 is 2 remainder 3, so in the fourth row there will be two blocks of 4 and one block of 3.
In total, there are 15 blocks: 11 blocks of 4 characters and 4 blocks of 3 characters.
4 minus 3 is 1, so there are 3 long (vertical) lines and 1 short line. The long lines will have 15 characters–the same as the total number of blocks—and the short line will have 11 characters, the same as the number of blocks with the larger number of characters.

Now we can throw away the layout above and start over. We'll write the encrypted message in four vertical lines (three lines of 15 and one line of 11).

THEF
ACER
ISTA
PACI

ITNE
TEST
GEAE
TYTE

SSGR
ISAM
ROBI
STQ

AMP
ULT
CCA

Now we'll place the columns next to each other:

THEF ITNE SSGR AMP
ACER TEST ISAM ULT
ISTA GEAE ROBI CCA
PACI TYTE STQ

And if we remove the spaces, we get the original message: THEFITNESSGRAMPACERTESTISAMULTISTAGEAEROBICCAPACITYTEST.

Here's some TypeScript to manage all this:

// Breaks an array into sub-arrays of size `chunkSize`
function chunk<T>(array: T[], chunkSize: number): T[][] {
  const result: T[][] = [];
  for (let i = 0; i < array.length; i += chunkSize) {
      const chunk = array.slice(i, i + chunkSize);
      result.push(chunk);
  }
  return result;
}

function routeEncrypt(message: string, rectangleSize: number, blockSize: number): string {
  const messageChars = message.split('');

  // Lay out the message in lines of length `rectangleSize`
  const messageLines = chunk(messageChars, rectangleSize);

  // Then break each line into chunks of length `blockSize`
  const chunkedLines = messageLines.map(line => chunk(line, blockSize));

  // Check to see if padding characters are needed at the end
  const allChunks = chunkedLines.flat();
  const finalChunk = allChunks.pop();
  const mainChunkSizes = allChunks.map(chunk => chunk.length);
  const oddChunkSize = mainChunkSizes.find(chunkSize => chunkSize !== blockSize);

  // Only add padding if the "odd chunks" (chunks at the ends of lines)
  //  are shorter than regular chunks, the final chunk isn't the same size
  //  as the odd chunks, and the final chunk isn't the same size as a
  //  regular chunk
  if (
    typeof oddChunkSize === 'number' &&
    Array.isArray(finalChunk) &&
    finalChunk.length !== oddChunkSize &&
    finalChunk.length !== blockSize
  ) {
    if (finalChunk.length < oddChunkSize) {
      finalChunk.push(...'Q'.repeat(oddChunkSize - finalChunk.length));
    } else {
      finalChunk.push(...'Q'.repeat(blockSize - finalChunk.length));
    }
  }

  // We could skip the intermediate step of "stacking" the chunks—that's just
  //  a convenience to help visualize the process—but we'll need it intact
  //  for Part 3.
  const stackedLines: string[][] = [];

  // For each block column...
  const blocksPerLine = Math.ceil(rectangleSize / blockSize);
  for (let blockIx = 0; blockIx < blocksPerLine; blockIx++) {
    // For each row...
    const numberOfRows = chunkedLines.length;
    for (let rowIx = 0; rowIx < numberOfRows; rowIx++) {
      // We'll have numberOfRows rows for each block column
      const encryptedRowIx = rowIx + (blockIx * numberOfRows);
      stackedLines[encryptedRowIx] = chunkedLines[rowIx][blockIx];
    }
  }

  // Now we read out from top to bottom, starting at the left
  const encryptedChars: string[] = [];
  for (let charIx = 0; charIx < blockSize; charIx++) {
    for (let rowIx = 0; rowIx < stackedLines.length; rowIx++) {
      if (!Array.isArray(stackedLines[rowIx])) {
        continue;
      }

      const char = stackedLines[rowIx][charIx];
      if (typeof char === 'string') {
        encryptedChars.push(char);
      }
    }
  }

  // Join and return the encrypted message
  return encryptedChars.join('');
}

function routeDecrypt(encrypted: string, rectangleSize: number, blockSize: number): string {
  // Do the math to determine the total number of chunks
  const numberOfRows = Math.ceil(encrypted.length / rectangleSize);
  const chunksPerRow = Math.ceil(rectangleSize / blockSize);
  const lastRowLength = encrypted.length % rectangleSize;
  const lastRowChunks = Math.ceil(lastRowLength / blockSize);
  const totalChunks = (chunksPerRow * (numberOfRows - 1)) + lastRowChunks;

  // If there are chunk(s) of a different size, either the
  //  "odd chunks" or the final chunk, find them so we know
  //  how many long/short vertical lines there are
  const lastChunkSize = (lastRowLength % blockSize) || blockSize;
  const oddChunkSize = rectangleSize % blockSize;

  let numberOfShortVLines: number;
  if (oddChunkSize !== blockSize) {
    numberOfShortVLines = blockSize - oddChunkSize;
  } else if (lastChunkSize !== blockSize) {
    numberOfShortVLines = blockSize - lastChunkSize;
  } else {
    numberOfShortVLines = 0;
  }

  const numberOfLongVLines = blockSize - numberOfShortVLines;

  // Determine the length of the short and long lines
  const longVLineLength = totalChunks;
  const shortVLineLength = totalChunks -
    ((oddChunkSize !== blockSize ? numberOfRows - 1 : 0) +
     (lastChunkSize !== blockSize ? 1 : 0));

  const stackedLines: string[][] = [];
  let encryptedIx = 0;

  // For each vertical line...
  for (let vLineIx = 0; vLineIx < blockSize; vLineIx++) {
    // The long lines come first, then the short lines
    const currentVLineLength = vLineIx < numberOfLongVLines ?
      longVLineLength :
      shortVLineLength;

    // For each row in the current vertical line length...
    for (let rowIx = 0; rowIx < currentVLineLength; rowIx++) {
      stackedLines[rowIx] ??= [];

      // Place the next character from the encrypted message
      stackedLines[rowIx][vLineIx] = encrypted[encryptedIx++];
    }
  }

  // Now we'll place the blocks next to each other.
  // First, we need to figure out how many blocks are in each row
  const blocksPerRow = Math.ceil(rectangleSize / blockSize);
  const lastRowBlocks = Math.ceil(lastRowLength / blockSize);

  const messageLines: string[][] = [];
  let stackedLineIx = 0;

  // For each column of blocks...
  for (let columnIx = 0; columnIx < blocksPerRow; columnIx++) {
    const rowsInBlock = columnIx < lastRowBlocks ? numberOfRows : numberOfRows - 1;

    // For each row in the column...
    for (let rowIx = 0; rowIx < rowsInBlock; rowIx++) {
      messageLines[rowIx] ??= [];

      // Grab the next line from the stack and place it
      messageLines[rowIx].push(stackedLines[stackedLineIx++].join(''));
    }
  }

  return messageLines.flat().join('');
}

const rectangleSize = 15;
const blockSize = 4

const encrypted = routeEncrypt(
  'THEFITNESSGRAMPACERTESTISAMULTISTAGEAEROBICCAPACITYTEST',
  rectangleSize,
  blockSize
);
console.log(encrypted);
// > TAIPITGTSIRSAUCHCSATEEYSSOTMLCEETCNSATGABQPTAFRAIETEERMI

const decrypted = routeDecrypt(encrypted, rectangleSize, blockSize);
console.log(decrypted);
// > THEFITNESSGRAMPACERTESTISAMULTISTAGEAEROBICCAPACITYTESTQ

Solved! Now you know all the operations you need to decode Kryptos 3, which we'll do in the next post.

Acknowledgments

Thanks to the following, who made this series possible:

The NSA: The CIA Kryptos Sculpture (Declassified)
UC San Diego: The Kryptos Sculpture Solutions
Wikipedia: Kryptos, Transposition cipher

Solving the CIA Kryptos Code (Part 1)

Isaac Lyman — Fri, 26 Jan 2024 22:52:00 +0000

You can find all the code for this series on GitHub.

On the campus of CIA headquarters in Virginia, United States, there's a 34-year-old sculpture called Kryptos. It somewhat resembles a heavy copper flag waving in the wind. Cut out of its surface are 1,736 uppercase ASCII characters split into four encrypted messages and a Vigenere cipher table.

The first three messages were decrypted in the 1990s, first by the NSA and CIA, who didn't publicize their success, and then by a private-sector computer scientist, who did. To this day, the fourth message hasn't been cracked.

This raises a few interesting points. First, the U.S. government paid a quarter of a million dollars to have secret messages installed on their property, then paid at least two different teams of codebreakers to figure out what the messages said. I can't explain why, but I find that funny. Second, some or several government agencies probably have solved the fourth cipher and just haven't bothered to let us know, partially because secret-keeping is the modus operandi of the NSA/CIA, and partially because it's free advertising—they can get a wave of public interest (and a handful of new recruits) every few years just by dropping another "clue." And third, we don't know for certain that the fourth message doesn't include the phrase "CIA SUX" or "HOMER SIMPSON FOR PRESIDENT". It probably doesn't, but it could.

Anyway, let's write some TypeScript code capable of decrypting and encrypting the first three messages, then make an attempt to solve the fourth one. (Keep your expectations nice and low, I don't have a Ph.D. in cryptanalysis.)

Getting started: Basic vigenere

The first Kryptos message uses a Vigenere cipher. A standard Vigenere is pretty straightforward: you assign the numbers 0-25 to the letters A-Z, such that A=0, B=1, C=2, and so on. You translate your source message into these numbers. Then you pick a decryption key; any sequence of letters will do. You also translate the decryption key into numbers. And finally, for each index of the source message, you go to the same index in the decryption key and add both numbers together, then translate back to a letter. When you have a number greater than 25, you loop back around to 0—that is, modulo 26.

For all you visual learners, here's how you'd translate "ILOVEWATER" with the key "HYDRATE":

Message	I	L	O	V	E	W	A	T	E	R
a=0 number (A)	8	11	14	21	4	22	0	19	4	17
Key	H	Y	D	R	A	T	E	H	Y	D
a=0 number (B)	7	24	3	17	0	19	4	7	24	3
Add A + B	15	35	17	38	4	41	4	26	28	20
Modulo 26	15	9	17	12	4	15	4	0	2	20
0=a letters	P	J	R	M	E	P	E	A	C	U

So the encrypted message is PJRMEPEACU. To decrypt it, you'd go the opposite direction:

Encrypted message	P	J	R	M	E	P	E	A	C	U
a=0 number (C)	15	9	17	12	4	15	4	0	2	20
Key	H	Y	D	R	A	T	E	H	Y	D
a=0 number (D)	7	24	3	17	0	19	4	7	24	3
Subtract C - D	8	-15	14	-5	4	-4	0	-7	-22	17
If negative, add 26	8	11	14	21	4	22	0	19	4	17
0=a letters	I	L	O	V	E	W	A	T	E	R

As you can see, the Vigenere cipher isn't particularly strong, but most people wouldn't be able to crack it without the key.

Here's how you'd encrypt and decrypt in TypeScript:

const alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ';

function vigEncrypt(message: string, key: string): string {
  const messageChars = message.toUpperCase().split('');


  // For each character in the message...
  let messageIx = 0;
  const encryptedChars = messageChars.map((messageChar) => {
    // Traditionally you'd remove spaces and symbols before encrypting to
    //  make it harder to crack, but you don't have to.
    if (!alphabet.includes(messageChar)) {
      return messageChar;
    }

    // Find the same index in the key, looping around if needed
    const keyIx = messageIx % key.length;
    messageIx++;
    // Get the character at that index
    const keyChar = key[keyIx];
    // Find its "a=0" number
    const keyNumber = alphabet.indexOf(keyChar);
    // Find the "a=0" number of the original message character as well
    const messageNumber = alphabet.indexOf(messageChar);
    // Add the numbers together, looping around if >26
    const encryptedNumber = (messageNumber + keyNumber) % 26;
    // Get the letter corresponding to that number
    const encryptedChar = alphabet[encryptedNumber];
    // That's our encrypted character!
    return encryptedChar;
  });

  // Join all the encrypted characters together into one message
  return encryptedChars.join('');
}

function vigDecrypt(encrypted: string, key: string): string {
  const encryptedChars = encrypted.toUpperCase().split('');

  // Now we simply reverse what we did above
  let encryptedIx = 0;
  const messageChars = encryptedChars.map((encryptedChar) => {
    if (!alphabet.includes(encryptedChar)) {
      return encryptedChar;
    }

    const keyIx = encryptedIx % key.length;
    encryptedIx++;
    const keyChar = key[keyIx];
    const keyNumber = alphabet.indexOf(keyChar);
    const encryptedNumber = alphabet.indexOf(encryptedChar);
    let messageNumber = encryptedNumber - keyNumber;
    // Special handling for negative numbers...
    if (messageNumber < 0) {
      messageNumber = alphabet.length + messageNumber;
    }

    const messageChar = alphabet[messageNumber];
    return messageChar;
  });

  // Join all the decrypted characters together into one message
  return messageChars.join('');
}

const sourceMessage = 'PROGRAMMINGISFUN';
const key = 'EXCEPTFORJAVA';

const encrypted = vigEncrypt(sourceMessage, key);
console.log(encrypted); // > TOQKGTRAZWGDSJRP

const decrypted = vigDecrypt(encrypted, key);
console.log(decrypted); // > PROGRAMMINGISFUN

Nothing too crazy, right? But we're not there yet—Kryptos 1 has a trick up its sleeve.

Kryptos 1: Advanced vigenere

If you were sending an encrypted message and wanted to stump someone who's already familiar with Vigenere ciphers, what could you change to throw them off? You could pick a longer key or mess with the math, but Kryptos does something even more fun: it screws up the alphabet.

The Kryptos alphabet is KRYPTOSABCDEFGHIJLMNQUVWXZ. That is, you start with the word KRYPTOS, then go through the alphabet from A to Z, skipping any letters that were already present in KRYPTOS. This means there's effectively a second key: the key that displaces and rearranges the alphabet. We can swap out our alphabet in the code above to build a Kryptos 1 encoder/decoder.

The text of Kryptos 1 is:

EMUFPHZLRFAXYUSDJKZLDKRNSHGNFIVJ
YQTQUXQBQVYUVLLTREVJYQTMKYRDMFD

And the key is "palimpsest". Let's plug this into our TypeScript.

const alphabet = 'KRYPTOSABCDEFGHIJLMNQUVWXZ';

function vigDecrypt(encrypted: string, key: string): string {
  const encryptedChars = encrypted.toUpperCase().split('');

  let encryptedIx = 0;
  const messageChars = encryptedChars.map((encryptedChar) => {
    if (!alphabet.includes(encryptedChar)) {
      return encryptedChar;
    }

    const keyIx = encryptedIx % key.length;
    encryptedIx++;
    const keyChar = key[keyIx];
    const keyNumber = alphabet.indexOf(keyChar);
    const encryptedNumber = alphabet.indexOf(encryptedChar);
    let messageNumber = encryptedNumber - keyNumber;

    if (messageNumber < 0) {
      messageNumber = alphabet.length + messageNumber;
    }

    const messageChar = alphabet[messageNumber];
    return messageChar;
  });

  return messageChars.join('');
}

const encrypted = `EMUFPHZLRFAXYUSDJKZLDKRNSHGNFIVJ
YQTQUXQBQVYUVLLTREVJYQTMKYRDMFD`;
const key = 'PALIMPSEST';

const decrypted = vigDecrypt(encrypted, key);
console.log(decrypted);
/* >
BETWEENSUBTLESHADINGANDTHEABSENC
EOFLIGHTLIESTHENUANCEOFIQLUSION
*/

The message is "BETWEEN SUBTLE SHADING AND THE ABSENCE OF LIGHT LIES THE NUANCE OF IQLUSION". The last word is an intentional misspelling of "ILLUSION".

What if we want to generate the special Kryptos alphabet instead of hard-coding it? That's easy enough:

const alphabetKey = 'KRYPTOS';
const standardAlphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ';
const alphabet = alphabetKey + standardAlphabet.split('').filter(ch => !alphabetKey.includes(ch)).join('');

function vigDecrypt(encrypted: string, key: string): string {
  const encryptedChars = encrypted.toUpperCase().split('');

  let encryptedIx = 0;
  const messageChars = encryptedChars.map((encryptedChar) => {
    if (!alphabet.includes(encryptedChar)) {
      return encryptedChar;
    }

    const keyIx = encryptedIx % key.length;
    encryptedIx++;
    const keyChar = key[keyIx];
    const keyNumber = alphabet.indexOf(keyChar);
    const encryptedNumber = alphabet.indexOf(encryptedChar);
    let messageNumber = encryptedNumber - keyNumber;

    if (messageNumber < 0) {
      messageNumber = alphabet.length + messageNumber;
    }

    const messageChar = alphabet[messageNumber];
    return messageChar;
  });

  return messageChars.join('');
}

const encrypted = `EMUFPHZLRFAXYUSDJKZLDKRNSHGNFIVJ
YQTQUXQBQVYUVLLTREVJYQTMKYRDMFD`;
const key = 'PALIMPSEST';

const decrypted = vigDecrypt(encrypted, key);
console.log(decrypted);
/* >
BETWEENSUBTLESHADINGANDTHEABSENC
EOFLIGHTLIESTHENUANCEOFIQLUSION
*/

Only a couple of extra lines, and now we can play with different alphabet keys and cipher keys.

Kryptos 2: Vigenere (again)

Kryptos 2 is encrypted using the same method as Kryptos 1. The only difference is the key: instead of "palimpsest", it uses "abscissa".

The text of Kryptos 2 is:

VFPJUDEEHZWETZYVGWHKKQETGFQJNCE
GGWHKK?DQMCPFQZDQMMIAGPFXHQRLG
TIMVMZJANQLVKQEDAGDVFRPJUNGEUNA
QZGZLECGYUXUEENJTBJLBQCRTBJDFHRR
YIZETKZEMVDUFKSJHKFWHKUWQLSZFTI
HHDDDUVH?DWKBFUFPWNTDFIYCUQZERE
EVLDKFEZMOQQJLTTUGSYQPFEUNLAVIDX
FLGGTEZ?FKZBSFDQVGOGIPUFXHHDRKF
FHQNTGPUAECNUVPDJMQCLQUMUNEDFQ
ELZZVRRGKFFVOEEXBDMVPNFQXEZLGRE
DNQFMPNZGLFLPMRJQYALMGNUVPDXVKP
DQUMEBEDMHDAFMJGZNUPLGEWJLLAETG

Let's plug it in!

const alphabetKey = 'KRYPTOS';
const standardAlphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ';
const alphabet = alphabetKey + standardAlphabet.split('').filter(ch => !alphabetKey.includes(ch)).join('');

function vigDecrypt(encrypted: string, key: string): string {
  const encryptedChars = encrypted.toUpperCase().split('');

  let encryptedIx = 0;
  const messageChars = encryptedChars.map((encryptedChar) => {
    if (!alphabet.includes(encryptedChar)) {
      return encryptedChar;
    }

    const keyIx = encryptedIx % key.length;
    encryptedIx++;
    const keyChar = key[keyIx];
    const keyNumber = alphabet.indexOf(keyChar);
    const encryptedNumber = alphabet.indexOf(encryptedChar);
    let messageNumber = encryptedNumber - keyNumber;

    if (messageNumber < 0) {
      messageNumber = alphabet.length + messageNumber;
    }

    const messageChar = alphabet[messageNumber];
    return messageChar;
  });

  return messageChars.join('');
}

const encrypted = `VFPJUDEEHZWETZYVGWHKKQETGFQJNCE
GGWHKK?DQMCPFQZDQMMIAGPFXHQRLG
TIMVMZJANQLVKQEDAGDVFRPJUNGEUNA
QZGZLECGYUXUEENJTBJLBQCRTBJDFHRR
YIZETKZEMVDUFKSJHKFWHKUWQLSZFTI
HHDDDUVH?DWKBFUFPWNTDFIYCUQZERE
EVLDKFEZMOQQJLTTUGSYQPFEUNLAVIDX
FLGGTEZ?FKZBSFDQVGOGIPUFXHHDRKF
FHQNTGPUAECNUVPDJMQCLQUMUNEDFQ
ELZZVRRGKFFVOEEXBDMVPNFQXEZLGRE
DNQFMPNZGLFLPMRJQYALMGNUVPDXVKP
DQUMEBEDMHDAFMJGZNUPLGEWJLLAETG`;
const key = 'ABSCISSA';

const decrypted = vigDecrypt(encrypted, key);
console.log(decrypted);
/* >
ITWASTOTALLYINVISIBLEHOWSTHATPO
SSIBLE?THEYUSEDTHEEARTHSMAGNET
ICFIELDXTHEINFORMATIONWASGATHER
EDANDTRANSMITTEDUNDERGRUUNDTOANU
NKNOWNLOCATIONXDOESLANGLEYKNOWA
BOUTTHIS?THEYSHOULDITSBURIEDOUT
THERESOMEWHEREXWHOKNOWSTHEEXACTL
OCATION?ONLYWWTHISWASHISLASTMES
SAGEXTHIRTYEIGHTDEGREESFIFTYSE
VENMINUTESSIXPOINTFIVESECONDSNO
RTHSEVENTYSEVENDEGREESEIGHTMINU
TESFORTYFOURSECONDSWESTIDBYROWS
*/

The message is:

IT WAS TOTALLY INVISIBLE HOWS THAT POSSIBLE?
THEY USED THE EARTHS MAGENTIC FIELD X
THE INFORMATION WAS GATHERED AND TRANSMITTED UNDERGRUUND
TO AN UNKNOWN LOCATION X
DOES LANGLEY KNOW ABOUT THIS?
THEY SHOULD ITS BURIED OUT THERE SOMEWHERE X
WHO KNOWS THE EXACT LOCATION?
ONLY WW THIS WAS HIS LAST MESSAGE X
THIRTY EIGHT DEGREES FIFTY SEVEN MINUTES
SIX POINT FIVE SECONDS NORTH
SEVENTY SEVEN DEGREES EIGHT MINUTES FOURTY FOUR SECONDS WEST
ID BY ROWS

However, it's now known that the sculptor left out a letter for aesthetic reasons. The last snippet EWJLLAETG should be ESWJLLAETG, so the last line of the message is "X LAYER TWO", not "ID BY ROWS". Not that it makes the message any less cryptic.

Ready for part 2? It continues in the next post.

Acknowledgments

Thanks to the following, who made this series possible:

The NSA: The CIA Kryptos Sculpture (Declassified)
UC San Diego: The Kryptos Sculpture Solutions
Wikipedia: Kryptos, Vigenere

Should more developers be starting software companies?

Isaac Lyman — Tue, 05 Dec 2023 19:06:00 +0000

A software developer has, in theory, the ability to walk into an outhouse with an old laptop and emerge a few days later with something worth millions of dollars. If you can build software, you can create value out of thin air. Software has no material components.

^{It's been rightly pointed out that software does have material components, namely a working computer, which over half the world doesn't have at home. What I mean is that software has no material cost per unit. Once you've built the software, you can reproduce it indefinitely without further expense.}

However, it rarely works out like this. Most side projects never make a single dollar, let alone compensate the developer for their time, let alone create any income on top of that. Why is this the case?

Some would say it's about having the right idea, being smart, or working hard. None of this is true. The millionaire software tycoons of the world are no smarter or more hardworking than you. They don't even have better ideas—in fact, their "ideas" routinely run high-functioning companies into the ground. Many of them are considerably less smart, hardworking, and innovative than you are.

Others say it's about finding the right market. This is closer to the truth. A significant percentage of the world's business is conducted via spreadsheets emailed back and forth as email attachments. Much of it is less structured than that. Some of it depends on fax. The world still has a vast appetite for software, and most developers don't know where that appetite is strongest.

But that's not the whole story. You may learn there's a desperate need for up-to-date software in the landscape supply industry, but what are you going to do about it if you've never worked in landscaping? You could apply yourself to market research and try to figure things out, but it will be years before you have any intuition for the pain points your software should solve. You'll almost always be more efficient solving problems you're intimately familiar with.

This is the first big problem of software entrepreneurship. The common denominator all developers share is (obviously) software development. And since developers understand the developer market, they tend to come up with business ideas that are relevant to developers. As a result, the "software for developers" market is anomalously well-served. If there were a "wooden tools for carpenters" market, I'd expect that to be saturated as well.

Most software opportunities are located outside of tech. But most software workers haven't had a career outside of tech.

The second biggest commonality among developers is the life experience of a middle-class urban office worker. This entails a lot of bland consumer problems, like getting stuck in traffic, finding affordable housing, feeling isolated, and filling a few hours of free time each day. This gave us Zoom, Zillow, Facebook, and Netflix, respectively, plus several competitors in each category. The success of these companies is deceptive; for every Facebook and Netflix, there are a thousand other companies that tried to solve the same problems and failed. Some shipped better products and still failed.

We're biased by experience to idolize companies we've personally interacted with. Who among us has never watched a show on Netflix? But again, this is not where the opportunity is. Everyone knows what it's like to watch TV. Everyone knows what it's like to be a person. If you build software that any person could conceivably be interested in, you're competing in an over-understood and thus overserved market. Do we really need 50+ streaming TV services? I'll go out on a limb and say we don't.

Consumer (B2C) software in general isn't where the money is. It's exciting because we're all familiar with it in our day-to-day lives, but it's driven by irrational forces and incredibly difficult to sell. If you have name recognition, connections, and unlimited capital (or the absurd luck of a two-time lottery winner) maybe you can compete. But if you want to play on easy mode, corporate (B2B) software is where it's at.

Most people won't pay a dollar for their favorite app. But their employers will shell out thousands (or millions) for software that solves problems specific to their industry. When you think of Amazon, you think of an "everything store" and 2-day shipping. But most of their revenue comes from AWS, a cloud application platform for businesses. When you think of Google, you think of a search engine. But most of their revenue comes from advertising by businesses.

If you want to compete with Amazon, you don't build an e-commerce store; you build cloud services. If you want to compete with Google, you don't build a search engine; you build an ad delivery network. Of the five companies in the FAANG acronym, only Netflix had no B2B offering in the first half of 2023. Throw in Microsoft, Salesforce, Adobe, and PayPal, and Netflix still stood alone. And even that couldn't last. Netflix now sells ad spots to businesses.

So what does it take to build a successful software business? After a couple ill-fated stints as a software entrepreneur, I now believe in a few essential elements:

A very boring idea. Your friends and family should be stifling yawns as you tell them what you're building. If their eyes light up, that's a sign you've come up with something that's either wildly unrealistic (innovation usually happens in small steps) or too relevant to the average person (so probably already software'd to death). If you're still set on building "the next Facebook" or whatever, try asking people if they'll commit to paying $20 for it. $20 is the bare minimum you can charge per customer if you want to build a successful company without relying on outrageous amounts of luck.
Deep understanding of the market. As a software engineer, your ideal cofounder is someone who's spent 20 or 30 years working in an industry where the software choices are non-competitive or under-competitive. Even five or ten years of outside experience can be enough. Your cofounder doesn't necessarily need to have a software idea—they just need to have experience managing a complex process and plenty of complaints about the way it's typically done.
A sales professional. This is your other cofounder: someone who could sell saltwater to a dolphin. Sales is a far more complex and specialized field than most devs give it credit for. You can't just build great software and wait for customers to show up with their wallets open. They won't. "If you build it, they will come" has always been a lie. You can't just buy some Google Ads and wait for clicks; you'll spend 20 times as much as you get back. You can't just launch on ProductHunt and tell everyone to upvote you; you'll harvest a few hundred email addresses, maybe, but no one will buy. Sustainable growth only comes with the unrelenting effort of a skilled salesperson. I don't care how good your idea is, you will fail without sales.

These aren't the same criteria I had when I started my career—they're all things I've picked up along the way. And they may still change, but I've used them to assess every startup idea I've heard over the last couple years, and they've proven to be pretty good.

What else does it take to build a successful software company? I've only scratched the surface, really. A healthy and productive culture, careful growth, superb employee retention, trustworthy customers, rational decision-making, well-aligned owners and investors...I could list things all day and so could you, probably. But for my money, it all starts with a boring idea and the right cofounders.

My top home software upgrades

Isaac Lyman — Thu, 23 Nov 2023 03:42:00 +0000

I've made some big changes to my personal tech ecosystem over the last few years and discovered some great applications along the way. Here are a few standouts.

Kagi

Paid for by the user

Replacement for: Google search, which I've used since at least 2004.

If you've noticed your Google results have been getting steadily worse—or if you're getting sick of having to scroll down below the fold to even see your results—you should sign up for a free trial of Kagi (rhymes with "doggy"), a search engine whose biggest con is also its biggest pro: you have to pay for it.

It's a pretty good pro, though. Google gets paid by advertisers, so they want what advertisers want (tons of ads, links to sites that run ads, results that prioritize online shopping). Kagi gets paid by you, so they want what you want (no ads, relevant results only). Considering how much time I spend searching for things online, the fee of $10/month is pretty reasonable.

So far I've only gone back to Google a few times. Google is better if you're shopping for a PS5. But Kagi has held its own in every other respect, especially programming questions. And it makes me happy to know that the only person paying for my search results is me.

Immich

Open-source, self-hosted

Replacement for: Google Photos, which I've used since 2011, and iCloud, which I've used since 2021.

Google Photos pulled a very predictable bait-and-switch on their "unlimited free photo storage" offering in 2021, and iCloud uses several dirty tricks (storing low-quality versions of photos on your iPhone to "optimize storage;" making it impossible to download all your iCloud photos at once) to lock up your pictures in the cloud so you'll feel forced to pay for expanded storage.

I'm never going to have fewer photos than I have now, so I saw where this was going: a never-ending spiral of higher and higher monthly storage fees. So I bought a cheap home server with a couple terabytes of disk space, installed Docker, and spun up an Immich instance. And it's been great. The mobile app syncs my photos behind the scenes and they're all accessible from a LAN server, no Internet connection required.

SyncThing

Open-source, self-hosted

Replacement for: Dropbox, which I've used since 2010.

Dropbox was pretty cool when it first launched. The app was slick, it was the only game in town, and there were tons of ways to get extra free storage. I relied on it all the way through college.

These days the free tier is more restrictive and the company seems to have lost its way. (What do they even do anymore? Why don't their ads make any sense? Why did they make a Microsoft Word overlay that's so hard to turn off? Darned if I know.) I was getting tired of cleaning out old files every time I wanted to move a 1 GB zip archive around, so I set up SyncThing on my home server and switched all my files over in about 30 minutes. It's been a flawless plug-and-play replacement ever since.

SyncThing is free, but if you want file access on your iPhone you'll need Mobius Sync, which costs $5—once, not monthly.

Firefox

Open-source

Replacement for: Google Chrome, which I've used since 2008.

I wish I had some big noble reason for switching to Firefox, like privacy or stickin' it to the man. But if we're being honest, I switched because Chrome crashed while I was at work one day and I couldn't get it to start up again. So I used Microsoft Edge for its one true purpose: downloading Firefox.

Firefox is great, though, and I have no regrets. It's neck and neck with Chrome in the latest benchmarks and, as a bonus, isn't planning to force-delete your adblocker next year.

OTP Auth

Paid for by the user

Replacement for: Google Authenticator, which I've used since at least 2015.

Every 2-factor authentication app is essentially the same. However, Google Authenticator has an extra special feature: it stores your auth keys in a proprietary format which prevents you from exporting them anywhere else unless you're willing to clone a community-supported tool from GitHub and mess around on the command line for a while.

Which I was.

OTP Auth is a great replacement and it works with your Apple Watch, so you can get your two-factor codes without even pulling out your phone. If that isn't luxury, I don't know what is.

Mastodon

Open-source, decentralized

Replacement for: Twitter, which I've used since 2015.

Mastodon is literally just "what if Twitter/Threads/Bluesky/Tumblr worked like email, so you could join whatever server you want and still follow all your friends?" Super simple.

Like Kagi, this is a case of incentive alignment. If a big tech company runs your social media, they're going to sell you out as often as they can, for as much money as they can. If Steve (who you met at a conference one time) runs your social media, he's probably too busy at his day job to be thinking about how to sell your data.

(You should tip Steve, he does a great job.)

Obsidian

Self-hosted

Replacement for: Edward, a writing app I launched in 2018 and discontinued in 2021.

Running a B2C app for novelists was great fun but extremely unprofitable. Luckily, after shutting it down I discovered Obsidian, a local-first Markdown editor with an impressive plugin ecosystem. And the one missing feature (word count in the file tree) was easy to implement on my own.

Now I do most of my writing in Obsidian, which gets better with every release. And it's great being able to keep all my files on my computer in an open format, so even if the app dies someday, I won't lose anything.

What's next?

I'd love to use even more self-hosted, open-source, and/or paid-for-by-the-user products in 2024. (Does anyone know how to leave Gmail and take your email address with you? ...No? Oh well.)

What are your favorites?

The 3D object model of software construction

Isaac Lyman — Mon, 20 Nov 2023 18:14:00 +0000

Software projects are most efficient when, at any given time, each contributor is focused on a different concern. Like objects in 3D space, projects can be said to have surface area and volume:

Surface area represents the number of opportunities to contribute—that is, how many people could efficiently work on the project at the same time.
Volume represents the knowledge space of the project (measured, perhaps, in the total number of productive person-hours spent on it). Divided by the number of contributors, it also measures the theoretical limit (relative to the project as a whole) on any single contributor's ability to make an extraordinary impact.

As projects grow, they inevitably increase across both measures—but not evenly. A perfect sphere has the smallest possible surface-area-to-volume ratio: that's a project where a few well-tenured contributors will outperform any other team configuration, meaning that hiring wouldn't increase productivity or efficiency. A flattened cuboid, on the other hand, has a large surface-area-to-volume ratio: there are many different opportunities to contribute, and hiring will usually increase speed of delivery, but increasing headcount will eventually yield diminishing returns as the space becomes too limited for each contributor to spread their wings. Thus, projects with the same scope may have very different shapes.

^{For teams that practice pair programming, two developers can be counted as a single, highly efficient contributor.}

Following are four theories about software development that proceed from this analogy.

1. Starting a project

Theory: Every starting-from-zero project should have exactly two people assigned to it for the first eight weeks. Those two people are an engineer and a product manager.

A brand new software project, like a marble, is very small and has a minimal surface-area-to-volume ratio. One motivated developer can make incredible progress in a very short time; another developer would only slow the project down or end up waiting around while the first developer works on initial project setup, architecture, and creation of patterns.

What one programmer can do in one month, two programmers can do in two months.

— Fred Brooks

To mitigate the risk of the developer making progress in the wrong direction, the product manager's role is to map out the scope of the project and continuously negotiate the feature set. They also lead out on collaborating with pilot clients, testing and validating early iterations, and building a prioritized backlog.

Only when a project actually does something is it reasonable to assign more people. Ideally, it should do several things. It should also have had a chance to fail if any of its core assumptions were faulty. Every project is an experiment until proven otherwise. Part of the logic of assigning minimal staff to it for a while is to avoid overspending on something that won't yield fruit.

2. Growing the team

Theory: Individual contributors have the clearest picture of what roles should be added to a project.

It's important to be thoughtful about adding more people to a project as it grows. Headcount is often assigned to projects based on extraneous criteria such as profitability, corporate strategy, or contract deadlines. This approach to staffing is based on wishful thinking, not sound methodology.

Adding [human resources] to a late software project makes it later.

— Brooks's Law

The 3D object model of software development insists that we assign headcount for one reason only: efficiency. This can mean different things in context, but at the very least, adding a role of type X when there's an existing bottleneck of type Y won't do much for productivity.

Every development cycle has bottlenecks, but many are only visible from the ground.

If a product manager is working frequent overtime, it may appear that another product manager is needed to partner with them. But maybe they're spending all their time fielding support tickets because the software has a high rate of defects. In this case, another product manager would be less helpful than a QA engineer.
If features are spending too long in development, it may seem like the solution is to hire more developers. But the problem could be that developers are having to wait several days for UI prototypes to be finished, or that the features are underspecified and they have to jockey for time with an overscheduled product manager to nail down the details. In these cases, more designers or product managers, respectively, are needed.

To discover these bottlenecks, it's essential to talk to the people closest to them: individual contributors. Any hiring decision made without this information is no better than a guess.

3. Building connected projects

Theory: The size, composition, and hierarchy of a software team is a software-architectural decision, not a human resource decision.

Choices about the structure of a software team invariably affect the projects they ship.

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.

— Conway's Law

When you split a product team in two, you're incentivizing them to create two different products. Whether those products appear unified to the end user is a consequence of how well the teams communicate with each other. When you combine two teams into one, you're incentivizing them (albeit in competition with other constraints) to combine their products. When you form a centralized "core UI" or "platform" team, you're incentivizing those teams to ship discrete products to other teams, which become end users—for better and for worse. These incentives can be overcome, but only by well-defined rules and continuous incentives.

Teams, by nature, create and defend their own opportunities to contribute (surface area) and their own tight-knit knowledge space (volume). When their projects must interact, the quality and reliability of the interaction largely depends on how difficult it is for contributors on different projects to ask each other for favors.

Teams working on interdependent projects can only be effectively organized by someone who has a mental model of each project's structure, shape, and interaction points. Trying to force teams of one's preferred shape onto projects with completely different shapes is, in a real sense, no different than trying to put a round peg in a square hole.

4. Building long-term value

Theory: Growth is never value-neutral; it either adds consumer value or subtracts from it.

A project with a simple shape is easy for a customer to understand and use, but that doesn't mean it provides ideal value, or even any value at all.

Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.

— Zawinski's Law

Some customers prefer software that does one thing well, but most will gravitate to a one-stop shop. Nearly all software products are pathologically incapable of communicating with each other, so one mediocre product that manages an entire process typically requires less effort to use than five spectacular products that each manage a single step.

In the 3D object model of software, consumer value can be thought of in terms of congruence between a customer's process (also a 3D object of sorts) and a project's knowledge space. Each customer may have a different process, but if they're rational, they'll prefer whatever software has the most similar shape and the same or larger size. Processes appear continuous when you're in them; you can't easily persuade a customer to slice their process into multiple pieces for the sake of adopting your software. Therefore, every process should be considered holistically.

Planning new features for a project is a matter of two driving goals: matching the average customer process more closely (shape), and increasing the knowledge space of the project so it can accommodate a greater number of differences between processes (volume). An increase in volume without a change in shape creates value for all customers, since processes tend to evolve over time, but every change in shape either makes the project more congruent with the average customer process or less so. This is one reason why customer collaboration and functional empathy are so important.

Changes in shape and volume can come in many forms. Running ads in a product, for example, is a risky proposition because "viewing ads" isn't part of anyone's process. The presence of ads always decreases customer value, which isn't to say it's always self-defeating; the revenue opportunity may justify the addition of unsightly vertices to the project's shape. But if a project's entire revenue comes from advertising, it's destined to take on the shape of its advertisers' process rather than its users', which will make it less competitive. Collecting and selling user data is risky for the same reasons.

The Platonic ideal of any project is something that perfectly matches its users' needs without any unnecessary volume. This has never been achieved. But it can only be approximated when the user and the payer are the same person.

A project that completely covers its target market's processes, variations included, will nonetheless face pressure to grow. There may be opportunities to do so by addressing new customers with very similar needs. However, there are more and easier opportunities to grow in the short term by sacrificing customer value for profit. When a project's only remaining options are to stop growing or sell off its own competitiveness for parts, it becomes a sinking ship. Vendor lock-in and lack of competition are the only forces that can keep its customers on board after that.

Modeling success

Like any model, the 3D object model is imperfect and reductive. There are many factors outside of it that contribute to project success or failure: sales and marketing, willingness to pay, market power, pricing, organizational competence, turnover, and so on. But it's a useful tool for thinking about software in terms of the internal (what it does, how it's used, and who builds it) rather than the external (what we believe about it and how it sells). This is important because the internal factors are more often ignored, yet more easily controlled.

So software engineering isn't working out for you

Isaac Lyman — Thu, 09 Nov 2023 19:00:00 +0000

I used to be solidly on team "anyone can code." That changed when I talked to my therapist about it.

ME: I'm a big believer in neuroplasticity. I think anyone can learn to do anything, it's just a matter of how long it takes.

THERAPIST (visibly uncomfortable): I don't think that's fair.

ME: Why not?

THERAPIST: Well, for example, my partner and I have very different skills. I'm no good at some things she's really good at, like math, and it wouldn't matter how long I try to be good at them, I'd be wasting my time. People are just different.

ME: Hmmm. It is true that people are different.

THERAPIST: You can't choose how someone else spends their time, but if you go in with the attitude that everyone can be good at everything, you're putting unrealistic expectations on people who may not be cut out for something.

I gravitated to "anyone can code" because it sounded egalitarian and humane, a battle cry against gatekeepers and elitists. But that was false empathy. "Anyone can code" sounds like a motivational poster, but in practice it behaves like a standard of judgment. If anyone can code, but you meet someone who really can't—despite years of studious effort—how do you resolve that without putting them down? "They just need more time" is a nice thought, but it won't hold up forever. No one should waste years of their career on something they're not good at. It isn't fair to them.

Maybe that person is you. You did the four-year degree, or the bootcamp, or Codecademy, and you floated around looking for contracts for a little while or joined a tech company. You've been paying attention and asking questions and trying to wrap your head around business requirements. You've been doing this for a year or more. But at the end of the day, it's not clicking. You look at code and all you see is a solid wall of gibberish. Even if you spend all day chipping away at it, you won't understand what it's doing. And forget contributing; unless someone tells you where to go and what to do, you can't figure out how to make things happen.

^{NOTE: For all you devs with imposter syndrome: if you've been at this for less than a year, or you regularly fix bugs and complete small tasks without guidance, I'm not talking about you. Go find something else to worry about.}

If this sounds familiar, the first thing you need to know is it's okay. Not everyone can code. There are lots of smart and driven people who can't code. Coding is its own thing. If you're not good at it, that doesn't mean anything about you.

The other thing you should know: you don't need to throw away all the time you've spent trying. Maybe your next job title isn't "Software Engineer." But being almost a software engineer is a powerful credential that can set you up for success in a lot of other careers.

Let's talk about a few of those.

Project Manager

Job description: A project manager coordinates the efforts of a team of developers, making sure they reach project goals on time and on budget. They translate those goals into features and tasks, then prioritize and assign them to their team. Along the way they collaborate daily with developers to understand the cost and impact of each task, resolve any ambiguities in the specifications, and figure out who's best suited to build and own each piece of the project. They also facilitate communication with QA, UX, and upper management.

Why you, why now: You'll be more effective as a project manager if you know a few things about code. You'll be better at knowing what's feasible and what's not, estimating the size of tasks, and suggesting ideas to help meet customer needs without introducing too much complexity to the project. You'll also be able to understand a lot of developer-speak that many project managers struggle with. You'll recognize when something's wrong with the development process and know the right terminology to talk about it. You don't have to know any code to be a great project manager, but it helps a lot.

Quality Assurance

Job description: QA Analysts test software to find bugs, performance issues, unmet requirements, and other deficiencies. They write test cases as features are built, then re-test them regularly to make sure new code doesn't break what came before. Some QA Analysts are also QA Engineers, writing end-to-end tests with tools like Selenium and Cypress to test applications quickly and automatically. QA sometimes has the reputation of being a "stepping stone" to software development, or a "less-than" career in terms of prestige and salary, but this is changing. Smart companies recognize that QA is a unique profession with its own skillset and significance, and a great QA is just as valuable as a great developer.

Why you, why now: QA Analysts with a background in code write better bug reports. They know what kinds of mistakes can cause bugs, what else they should test to help narrow down the problem, and where to look for additional context. A well-written bug report can make the difference between a bug that takes all day to fix and one that only takes a few minutes. Basic coding skills are even more beneficial to a QA Engineer, since automated tests are typically written in (or supplemented by) code. But even if software engineering isn't right for you, you may be a great QA Engineer; automated tests use a smaller code vocabulary and are less abstract than applications.

Technical Writer

Job description: Technical writers leverage a deep understanding of applications and APIs to write documentation, release notes, video scripts, courses, and even marketing materials. These make it possible for end users and others in the organization to understand and use their team's software. Technical writing is multidisciplinary: it sits at the intersection of technical skills and communication skills, making it a great fit for people who take a lot of notes and are good at explaining things.

Why you, why now: The better you understand something, the better you can explain it to someone else. You'll need to spend time talking to developers or project managers to understand what they're building. If you already know some coding terminology, it'll save you a lot of time. You'll also have the ability to understand not just what, but why. This will help you communicate more clearly and avoid setting false expectations for readers.

UX Designer / UI Designer

Job description: UX designers work directly with users to understand how they use a product, then translate that information into specific changes the development team can implement to improve the product's design and flow. Depending on where you work, "UI designer" and "UX designer" may be synonyms, but UI designers are typically more focused on an application's branding, style, and appearance, while UX designers specialize in understanding workflows and gathering user feedback.

Why you, why now: If you have an artistic bent, transitioning to a career in design is a great way to put your programming knowledge to use. Designers who know code are better at understanding what kinds of UI elements and changes are easy or hard to implement, so they can make more efficient, practical decisions and help engineering teams deliver faster. They can also suggest improvements that may not be obvious on the surface, but make applications more unified and maintainable under the hood.

CTO / Engineering Director

Job description: Managers and executives in technical organizations are in charge of vision, strategy, policy, and resourcing. In other words, they make decisions that affect teams and careers over the long term, so the people working for them can focus on decisions that affect products and interactions in the short term. A good leader knows how to find obstacles and bottlenecks before they become showstoppers, and they can be trusted to fix them.

Why you, why now: As a leader with a programming background, you'll know what to expect from programmers and how to get the best out of them. You'll understand the rhythm of development, so you'll be able to sense when something is wrong. You'll also have instant rapport with the people who make your product happen; programmers are more likely to trust someone who speaks their language, and in leadership, trust is everything.

And a thousand others...

Job description: Science, healthcare, law, public policy, sales, administration, and just about everything else.

Why you, why now: Learning to code isn't just about vocabulary and syntax rules. First and foremost, it's about learning to think methodically, understand abstract concepts, work with constraints, come up with creative solutions, see details without missing the big picture, and make your own path from A to Z. These are helpful skills in most careers. You'll never regret learning how to see things the way a computer sees them.

In defense of learning on company time

Isaac Lyman — Mon, 23 Oct 2023 16:00:00 +0000

Based on my own experience and interactions with many other developers, I'm going on the record with a possibly controversial opinion:

If you're a dev, especially one in the first five years of your career, you should be spending at least 10% of your on the clock time—at least 4 hours of your 40 hour work week—studying the fundamentals of programming. Put aside your assigned work for those 4 hours. Schedule them in your work calendar. Mark yourself unavailable on Slack. No matter how busy you are, protect that time.

Everybody wins here. The time you spend improving your fundamental skills will pay itself off almost immediately, then yield continuing returns for the rest of your career.

Why is this controversial? Because many developers would say it should be 25%.

To clarify, I'm not talking about "just-in-time" learning (looking up the APIs and patterns you need to finish a task at work). I'm talking about focused, academic, non-urgent study. Books, tutorials, courses, videos. Stuff that you could survive without, maybe indefinitely. Stuff that will help you swim instead of just treading water.

What are the fundamentals of programming?

For the programming language(s) you specialize in:

Syntax
Standard library
Best practices
Variable scope and lifetime
Memory management
Asynchrony, scheduling, threading, and concurrency
Compilation toolchain
Code navigation & IDE features
Automated testing
Debugging
Top 10 packages

For programming in general:

Data structures
Big O notation
Design patterns
Software planning and architecture
Quality assurance
Functional programming concepts
Recursion
Information hiding/encapsulation
Continuous integration/continuous delivery
Threat modeling
UX and user behavior
Documentation
Test-driven and interface-driven development

I'm sure I missed a few. Comments are open for additions/corrections.

Have we reached Peak Programming Languages?

Isaac Lyman — Wed, 19 Apr 2023 18:03:03 +0000

I’m relatively young on the programming scene (I wasn't around when FORTRAN came out or anything) so this is a hubristic thing to say, but it feels like we’ve reached Peak Programming Languages. For anything you want to build and any way you want to build it, there's a mature and popular language already available. Typically the community around it is robust and the syntax is delightful. I mean, just try writing a few lines of C# or Dart and tell me it doesn't feel nice.

I made this flowchart recommending a language based on what you're building and the features you value most in a programming language. I left out a lot of languages (including 8 of the top 20) and for every leaf node on the chart, there's another option or three I could have chosen instead. But it's my chart, so you get my opinions.

Full-size link

It’s an unprecedented environment for new general-purpose languages to be competing in. People are definitely still creating them, but in 2023 more than ever, you might wonder why. Do any of them have a hope of gaining market share? Is it worth taking a chance on a new language, even one with some cool ideas or novel syntax, when you could use an established alternative with an enormous developer community and open-source ecosystem? What niche remains so painfully unfilled that we would all jump ship for a newcomer?

Of course, there are plenty of reasons to create a programming language other than trying to take the world by storm. It's fun, for one. It breeds new ideas that often get adopted into existing programming languages, for another. And if nothing else, it's one of the best ways to build your understanding of programming. We shouldn't stop making languages. If anything, we should be making more of them. But some of these new programming languages seem to be making a genuine effort to grow and establish a user base, and while I don't begrudge them that, I again ask why? I could happily use C#, Dart, Rust, and TypeScript for everything I build for the rest of my life. I can create anything I want, any way I want. I am happy.

I probably sound like this:

640K [of RAM] ought to be enough for anybody.

~Bill Gates, 1981 (allegedly)

All right, then correct me. What's still missing? What innovations have I overlooked? What brand-new language is going to be a huge deal in five years?

Maybe we haven't hit the Peak. That's up for debate. But we sure are spoiled for choice. It's an incredible time to be writing code.