Forem: TheLinuxMan

I Built a Corrupt Archive Recovery Engine in Rust — Because Every Tool I Tried Just Gave Up

TheLinuxMan — Mon, 09 Mar 2026 14:17:04 +0000

It started with a 2 GB backup archive and a familiar error:

$ unzip backup.zip
error [backup.zip]:  reported length of central directory is
  -14 bytes too long. Aborting.

$ 7z x backup.7z
ERROR: Can not open output file : Data Error

$ tar xzf data.tar.gz
gzip: data.tar.gz: unexpected end of file
tar: Child returned status 1

Three tools. Three failures. Every single file — gone.

Except they weren't gone. The archive metadata was damaged, but the actual file data — photos, documents, source code — was almost entirely intact. The tools just didn't care enough to find it.

So I built Helix Salvager — a corrupt archive recovery engine in Rust that uses a fail-forward architecture to save every file it possibly can.

The Core Idea: Stop Aborting, Start Skipping

Traditional tools treat an archive like a chain. One broken link = entire chain fails.

Helix Salvager treats each file inside an archive as independent. If file #3 of 10 is corrupt, you still get #1, #2, #4 through #10.

Standard:  File1 ✓ → File2 ✓ → File3 ✗ → ABORT  (0 files saved)
Helix:     File1 ✓ → File2 ✓ → File3 ✗ → File4 ✓ → ... → File10 ✓  (9 files saved)

That's the philosophy. The implementation is where it gets interesting.

Three Engines. One Pipeline.

┌──────────────────────────────────────────────────────────┐
│                  CORRUPT ARCHIVE INPUT                   │
│          (ZIP, 7z, gzip, bzip2, xz, tar, raw)           │
└──────────────────────┬───────────────────────────────────┘
                       │
         ┌─────────────┼─────────────┐
         ▼             ▼             ▼
   ┌───────────┐ ┌───────────┐ ┌───────────┐
   │  Engine A  │ │  Engine B  │ │  Engine C  │
   │ Fail-Fwd  │ │  Zombie   │ │  Magic    │
   │ Extractor │ │  LZMA     │ │  Carver   │
   └─────┬─────┘ └─────┬─────┘ └─────┬─────┘
         └─────────────┼─────────────┘
                       ▼
         ┌─────────────────────────┐
         │     RECOVERED FILES     │
         │  SHA-256 deduplication  │
         │  Per-type breakdown     │
         └─────────────────────────┘

Engine A — Fail-Forward Extraction

The ZIP format stores each file entry with its own local header. My extractor wraps every decompression attempt in catch_unwind isolation — if decompressing one entry panics or errors, the engine logs it and continues to the next.

CRC mismatches are also bypassed entirely. Yes, you might get a file with a few corrupt bytes. But you get the file — which is infinitely better than nothing.

Engine B — Zombie LZMA Decoder

7z archives are the hardest to recover because LZMA decompression is stateful — corruption early in the stream can make everything after it undecodable.

My solution: Shannon entropy validation + chunked retry.

Entropy Classification:
  H < 1.5    →  Padding/Empty      (discard)
  1.5–7.85   →  Structured Data    (keep — this is your content)
  H > 7.85   →  Random Noise       (mark as tainted)

When full-stream decompression fails, the Zombie decoder slides forward byte-by-byte, trying to find the next decodable region. Every output byte gets a taint flag — clean vs. reconstructed — so you know exactly which parts of your recovered file to trust.

Engine C — Magic Header Carver

When archive metadata is completely destroyed, Engine C goes raw. It uses an Aho-Corasick multi-pattern automaton to scan every byte of the input for 29 known file signatures — in a single O(n) pass.

Category	Types
Images	JPEG, PNG, GIF, BMP, WebP, TIFF, ICO, PSD
Documents	PDF
Audio	WAV, MP3, FLAC, OGG
Video	MP4, AVI
Executables	ELF, PE/EXE, WASM
Archives	ZIP, RAR, 7z, TAR

Every match goes through structure validation (PNG chunk integrity, JPEG marker sequences, ELF header fields) and SHA-256 deduplication. The same JPEG bytes matching at multiple offsets only gets extracted once.

Benchmarks Against Standard Tools

Tested on intentionally corrupted archives:

Scenario	`unzip`	`7z`	Helix Salvager
1 dead sector in 7-file ZIP	ABORT	ABORT	6/7 files ✓
20% sectors zeroed	ABORT	ABORT	3/7 files ✓
Central directory destroyed	ABORT	0 files	5/7 files ✓
Truncated at 50%	ABORT	ABORT	4/7 files ✓
Heavy bitrot (100 bit flips)	ABORT	ABORT	2/7 files ✓

Not perfect — but something is always better than nothing.

The Test Suite

179 tests, 0 failures, 0 clippy warnings

──────────────────────────────────────────────────
  29  unit tests (core engine)
  35  integration tests (real-life archive files)
  21  real-world corruption simulation tests
  12  stress tests (edge cases, zip bombs, 100MB)
  42  regression tests
   1  doc test
──────────────────────────────────────────────────

I tested against real corrupt files, not synthetic ones — including archives from corkami/pocs (Ange Albertini's legendary file format PoCs), real GNU hello tarballs, 7-Zip's official XZ distribution, and W3C PDFs and PNGs.

Corruption patterns: sector death, bitrot, USB transfer errors, NAND flash degradation, truncation, header destruction, TCP packet reordering, power loss mid-write, and combinations of all of the above.

Using It

CLI:

git clone https://github.com/vedLinuxian/helix-salvager.git
cd helix-salvager
cargo build --release

# Recover to directory
./target/release/salvager recover broken.zip -o ./recovered/

# Recover as ZIP
./target/release/salvager recover damaged.7z -o recovered.zip --zip

# Inspect without extracting
./target/release/salvager inspect suspicious_file.bin

# JSON output for scripting
./target/release/salvager recover data.zip -o ./out/ --json --quiet

Web UI:

./start.sh
# Drag-and-drop at http://localhost:3000

Rust library:

use salvager_core::SalvageEngine;

let data = std::fs::read("corrupt_archive.zip")?;
let engine = SalvageEngine::new();

let report = engine.salvage(&data, Some(&|phase, pct| {
    println!("[{pct}%] {phase}");
}));

println!("Recovered {} files ({} bytes)",
    report.files_salvaged, report.bytes_recovered);

Docker:

docker build -t helix-salvager .
docker run -p 3000:3000 helix-salvager

What's Next

The roadmap I'm most excited about:

WASM build — Run the entire recovery engine in the browser, no server needed
Streaming/mmap mode — Process multi-gigabyte files without loading into RAM
RAR native extraction — True RAR v4/v5 header parsing (not just magic carving)
Confidence scoring — Per-file recovery percentage so you know what to trust
Python bindings — pip install helix-salvager

Why Rust?

Short answer: because this is exactly the domain Rust was made for.

Long answer: corrupt data recovery means touching malformed bytes, invalid lengths, and decompressors that were never designed to handle what you're throwing at them. Rust's ownership model and catch_unwind isolation let me do aggressive fault-tolerance without risking undefined behavior in the recovery engine itself. Zero unsafe blocks in the core library. 0 clippy warnings. Production-grade from day one.

Links

GitHub: github.com/vedLinuxian/helix-salvager
Website: vedlinuxian.github.io/helix-salvager
License: MIT / Apache-2.0 (dual)

If this has ever happened to you — an archive that every tool refused to open — give it a try. And if you find a corruption pattern it can't handle, open an issue. That's exactly how the test suite grows.

Built with 🧬 by Ved Prakash Pandey

NumPy: The Superhero Library Python Deserves (But Maybe Didn't Know It Needed)

TheLinuxMan — Fri, 19 Jul 2024 04:16:55 +0000

Introduction: Meet NumPy, Your New Best Friend
The Origin Story: Why NumPy Exists
NumPy's Superpowers: Key Features
Saving the Day: Real-World Applications
The Dark Timeline: A World Without NumPy
Becoming a NumPy Superhero: Code Examples
Conclusion: With Great Power Comes Great Computation

Introduction: Meet NumPy, Your New Best Friend

Hey there, Python enthusiast! 👋 Are you tired of wrestling with lists when you're trying to do serious number crunching? Do you break out in a cold sweat when someone mentions "multi-dimensional arrays"? Well, have I got news for you! Let me introduce you to NumPy, the superhero library that's here to save your mathematical day!

NumPy (which stands for "Numerical Python", not "New Pie" as I initially thought) is like that friend who's really good at math and is always eager to help with your homework. Except in this case, your homework is data analysis, scientific computing, or any task that involves working with large arrays or matrices. And trust me, NumPy is WAY faster at this than your human math-whiz friend (no offense, Dave).

The Origin Story: Why NumPy Exists

Picture this: it's the late 1990s. Python is gaining popularity, but scientists and engineers are pulling their hair out trying to do complex numerical operations efficiently. Lists just aren't cutting it. They're slow, memory-hungry, and about as suited for linear algebra as a potato is for smartphone.

Enter our heroes: a bunch of super-smart folks who decided Python needed some serious numerical computing muscle. Thus, NumPy was born! It brought to Python the power of arrays and matrices from languages like MATLAB, but with the added coolness of being, well, Python.

NumPy's Superpowers: Key Features

ndarray: The mighty multi-dimensional array object. It's like a list, but on steroids.
Broadcasting: Performing operations on arrays of different shapes. It's basically mathematical magic.
Vectorization: Write less, compute more. Because who has time to write loops?
Speed: It's faster than a speeding bullet (or at least faster than a Python list).
Memory efficiency: Uses less memory than traditional Python sequences.
Support for a wide range of mathematical operations: Linear algebra, Fourier transforms, random number generation - you name it, NumPy's probably got it.

Saving the Day: Real-World Applications

NumPy isn't just for impressing your fellow code nerds (although it's great for that too). It's used in a ton of real-world applications:

Data Analysis: It's the foundation for pandas, the go-to library for data manipulation in Python.
Machine Learning: Many ML libraries, like scikit-learn, are built on NumPy.
Image and Audio Processing: Because images are just big 3D arrays, right?
Financial Analysis: Crunching those numbers faster than you can say "stonks".
Scientific Simulations: From particle physics to climate modeling, NumPy's got you covered.

The Dark Timeline: A World Without NumPy

Imagine, if you will, a dystopian future where NumPy was never created. shudders In this bleak world:

Data scientists spend most of their time writing nested loops instead of actually analyzing data.
Machine learning models train slower than a sloth on vacation.
Researchers can't simulate complex systems efficiently, leading to a severe lack of cool science gifs on the internet.
Financial models run so slowly that by the time they finish, the stock market has already crashed (and recovered).
Image processing is done pixel by painstaking pixel, leading to a critical shortage of cat memes.

Scary, right? Thank goodness we don't live in that world!

Becoming a NumPy Superhero: Code Examples

Enough chit-chat! Let's see NumPy in action. Don't worry if you're new to this - we'll start simple and work our way up to the fancy stuff.

First, let's import NumPy:

import numpy as np  # Because typing 'numpy' every time is just too much work

Creating Arrays: The Building Blocks of NumPy Greatness

# Creating a 1D array
arr1 = np.array([1, 2, 3, 4, 5])
print(arr1)  # Output: [1 2 3 4 5]

# Creating a 2D array
arr2 = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print(arr2)
# Output:
# [[1 2 3]
#  [4 5 6]
#  [7 8 9]]

# Creating an array of zeros (for when you feel empty inside)
zeros = np.zeros((3, 3))
print(zeros)
# Output:
# [[0. 0. 0.]
#  [0. 0. 0.]
#  [0. 0. 0.]]

# Creating an array of ones (for when you're feeling more positive)
ones = np.ones((2, 4))
print(ones)
# Output:
# [[1. 1. 1. 1.]
#  [1. 1. 1. 1.]]

# Creating an array with a range of values
range_arr = np.arange(0, 10, 2)  # Start, stop, step
print(range_arr)  # Output: [0 2 4 6 8]

# Creating an array with linearly spaced values
linear_space = np.linspace(0, 1, 5)  # Start, stop, num of points
print(linear_space)  # Output: [0.   0.25 0.5  0.75 1.  ]

Array Operations: Math at the Speed of Light (Almost)

# Element-wise operations
arr1 = np.array([1, 2, 3, 4])
arr2 = np.array([5, 6, 7, 8])

print(arr1 + arr2)  # Output: [ 6  8 10 12]
print(arr1 * arr2)  # Output: [ 5 12 21 32]

# Broadcasting
arr3 = np.array([1, 2, 3, 4])
scalar = 5
print(arr3 * scalar)  # Output: [ 5 10 15 20]

# Matrix multiplication
matrix1 = np.array([[1, 2], [3, 4]])
matrix2 = np.array([[5, 6], [7, 8]])
print(np.dot(matrix1, matrix2))
# Output:
# [[19 22]
#  [43 50]]

# Transposing
transposed = matrix1.T
print(transposed)
# Output:
# [[1 3]
#  [2 4]]

Indexing and Slicing: Dicing Your Data Like a Pro Chef

arr = np.array([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])

# Getting a specific element
print(arr[1, 2])  # Output: 7

# Slicing
print(arr[:, 1])  # Output: [ 2  6 10]
print(arr[1:, 2:])  # Output: [[ 7  8] [11 12]]

# Fancy indexing
fancy_index = np.array([0, 2])
print(arr[fancy_index])
# Output:
# [[ 1  2  3  4]
#  [ 9 10 11 12]]

# Boolean indexing
bool_index = arr > 5
print(arr[bool_index])  # Output: [ 6  7  8  9 10 11 12]

Statistical Operations: Impress Your Friends with Quick Maths

arr = np.array([1, 2, 3, 4, 5])

print(np.mean(arr))   # Output: 3.0
print(np.median(arr)) # Output: 3.0
print(np.std(arr))    # Output: 1.4142135623730951

# Aggregating 2D arrays
arr_2d = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print(np.sum(arr_2d, axis=0))  # Sum of each column
# Output: [12 15 18]
print(np.sum(arr_2d, axis=1))  # Sum of each row
# Output: [ 6 15 24]

Reshaping: Because Sometimes You Need to Bend Your Data (Not Break It)

arr = np.arange(12)
print(arr)  # Output: [ 0  1  2  3  4  5  6  7  8  9 10 11]

# Reshape to 2D array
arr_2d = arr.reshape((3, 4))
print(arr_2d)
# Output:
# [[ 0  1  2  3]
#  [ 4  5  6  7]
#  [ 8  9 10 11]]

# Reshape to 3D array
arr_3d = arr.reshape((2, 3, 2))
print(arr_3d)
# Output:
# [[[ 0  1]
#   [ 2  3]
#   [ 4  5]]
#  [[ 6  7]
#   [ 8  9]
#   [10 11]]]

# Flattening an array
flattened = arr_3d.flatten()
print(flattened)  # Output: [ 0  1  2  3  4  5  6  7  8  9 10 11]

Random Number Generation: For When You're Feeling Lucky

# Generate random integers
random_ints = np.random.randint(0, 10, 5)
print(random_ints)  # Output: [3 7 9 1 4] (your results may vary)

# Generate random floats
random_floats = np.random.random(5)
print(random_floats)  # Output: [0.42 0.71 0.23 0.94 0.12] (your results may vary)

# Generate a random normal distribution
normal_dist = np.random.normal(0, 1, 1000)  # mean=0, std=1, size=1000
print(np.mean(normal_dist), np.std(normal_dist))
# Output: something close to 0 and 1

Conclusion: With Great Power Comes Great Computation

And there you have it, folks! You've just taken your first steps into the wider world of NumPy. From creating arrays to performing complex mathematical operations, NumPy is here to make your Python number-crunching life easier, faster, and dare I say, more fun!

Remember, NumPy is like a Swiss Army knife for numerical computing in Python. It might seem a bit overwhelming at first, but once you get the hang of it, you'll wonder how you ever lived without it. It's the kind of library that makes you want to go out and find large datasets just so you can use it more.

So go forth, young Padawan, and may the NumPy be with you! 🚀🐍📊

P.S. If anyone asks where you learned all this, just tell them it came to you in a dream. It's way cooler than admitting you read a whitepaper, right?

The Ultimate Guide to Python Lists: From Newbie to Ninja

TheLinuxMan — Fri, 19 Jul 2024 03:57:17 +0000

Hey there, code wrangler! 👋 Ready to dive into the wild world of Python lists? Buckle up, because we're about to turn you from a list newbie into a list ninja! 🥷

What the Heck is a List Anyway?
Creating Lists: Your First Rodeo
List Methods: Your Swiss Army Knife
Slicing and Dicing: Become a List Surgeon
List Comprehensions: One-Liners That Pack a Punch
Nested Lists: Inception, but with Data
Looping Like a DJ: Iterating Over Lists
Jedi Mind Tricks: Advanced List Techniques
Wrapping Up: You're a List Ninja Now!

What the Heck is a List Anyway?

Imagine you're packing for a trip to a Python convention (yes, that's a thing, you nerd! 🤓). You've got a suitcase, and you're tossing in all sorts of items: your laptop, some snacks, a rubber duck for debugging (don't judge), and maybe a few spare semicolons for your Java-loving friends.

That suitcase? That's basically a Python list. It's a container that can hold multiple items, keep them in order, and let you add or remove stuff whenever you want. It's like Mary Poppins' bag, but for data!

Creating Lists: Your First Rodeo

Let's start by creating some lists. It's easier than learning to ride a bike, and you're less likely to scrape your knee!

# Your conference packing list
packing_list = ["laptop", "charger", "rubber duck", "snacks", "Python t-shirt"]

# List of excuses for when your code doesn't work
excuses = ["It worked on my machine", "Must be a cosmic ray bit flip", "I was holding it wrong"]

# Empty list for your hopes and dreams (just kidding!)
hopes_and_dreams = []

# Accessing items (zero-indexed, because programmers count from 0)
print(packing_list[0])  # Output: laptop
print(excuses[-1])      # Output: I was holding it wrong (last item)

# Modifying items
packing_list[2] = "debugger"  # Sorry, rubber duck. You're fired.

See? Creating lists is a piece of cake. Or should I say, a slice of pie? (Python... Pi... get it? 😉)

List Methods: Your Swiss Army Knife

Now, let's talk about list methods. These are like the special moves in a fighting game, but instead of "Hadouken," you're yelling "append()" at your screen.

todo_list = ["Learn Python", "Write bug-free code", "Achieve world domination"]

# Adding items
todo_list.append("Take a nap")
todo_list.insert(1, "Drink coffee")

print(todo_list)
# Output: ['Learn Python', 'Drink coffee', 'Write bug-free code', 'Achieve world domination', 'Take a nap']

# Removing items
todo_list.remove("Write bug-free code")  # Let's be realistic here
last_item = todo_list.pop()  # Removes and returns the last item
print(f"Removed: {last_item}")  # Output: Removed: Take a nap

# Other useful methods
todo_list.sort()  # Alphabetical order, because even world domination needs organization
print(todo_list.count("Drink coffee"))  # How many coffee breaks? (Output: 1)
todo_list.reverse()  # Reverse the list, just to keep things spicy

print(todo_list)
# Output: ['Learn Python', 'Drink coffee', 'Achieve world domination']

# Finding items
print(todo_list.index("Drink coffee"))  # Output: 1 (second item, remember we start at 0!)

# Clearing the list
todo_list.clear()  # Ah, sweet procrastination

There are more methods, but these are the heavy hitters. Use them wisely, and may the odds be ever in your favor!

Slicing and Dicing: Become a List Surgeon

Slicing lists is like being a surgeon, but instead of "Scalpel, please," you're saying "Give me elements 2 through 5." Let's slice and dice!

numbers = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

print(numbers[2:5])   # Output: [2, 3, 4]
print(numbers[:5])    # Output: [0, 1, 2, 3, 4]
print(numbers[5:])    # Output: [5, 6, 7, 8, 9]
print(numbers[::2])   # Output: [0, 2, 4, 6, 8]
print(numbers[::-1])  # Output: [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

# You can also modify slices
numbers[1:4] = [10, 20, 30]
print(numbers)  # Output: [0, 10, 20, 30, 4, 5, 6, 7, 8, 9]

Remember: list[start:stop:step]. It's like a time machine for your list. Where we're going, we don't need loops!

List Comprehensions: One-Liners That Pack a Punch

List comprehensions are like the Chuck Norris of Python features. They're brief, powerful, and slightly intimidating until you get to know them.

# Generate a list of squares
squares = [x**2 for x in range(10)]
print(squares)  # Output: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

# List of even squares
even_squares = [x**2 for x in range(10) if x % 2 == 0]
print(even_squares)  # Output: [0, 4, 16, 36, 64]

# Convert Celsius to Fahrenheit
celsius = [0, 10, 20, 30, 40]
fahrenheit = [(9/5) * temp + 32 for temp in celsius]
print(fahrenheit)  # Output: [32.0, 50.0, 68.0, 86.0, 104.0]

# Flattening a matrix
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
flattened = [num for row in matrix for num in row]
print(flattened)  # Output: [1, 2, 3, 4, 5, 6, 7, 8, 9]

List comprehensions: because ain't nobody got time for verbose loops!

Nested Lists: Inception, but with Data

Nested lists are like Inception: it's lists within lists. But instead of dreams, we're dealing with data. And instead of Leonardo DiCaprio, we have... well, you!

# A 3x3 tic-tac-toe board
tic_tac_toe = [
    [' ', 'X', 'O'],
    ['X', 'O', 'X'],
    ['O', ' ', ' ']
]

# Accessing elements
print(tic_tac_toe[1][1])  # Output: O (center of the board)

# Modifying elements
tic_tac_toe[2][2] = 'X'  # Player X makes a move

# Print the board (don't worry, we'll make this prettier later)
for row in tic_tac_toe:
    print(row)

# Output:
# [' ', 'X', 'O']
# ['X', 'O', 'X']
# ['O', ' ', 'X']

# Creating a multiplication table (because who doesn't love math?)
mult_table = [[i * j for j in range(1, 6)] for i in range(1, 6)]
for row in mult_table:
    print(row)

# Output:
# [1, 2, 3, 4, 5]
# [2, 4, 6, 8, 10]
# [3, 6, 9, 12, 15]
# [4, 8, 12, 16, 20]
# [5, 10, 15, 20, 25]

Nested lists: perfect for game boards, matrices, or planning your eventual takeover of neighboring dimensions.

Looping Like a DJ: Iterating Over Lists

Iterating over lists is like being a DJ - you're going through your tracks (list items), doing something with each one. Let's drop some sick beats... I mean, loops!

playlist = ["Stayin' Alive", "YMCA", "Macarena", "Gangnam Style"]

# Basic loop (The Classic)
for song in playlist:
    print(f"Now playing: {song}")

# Enumerate (The Track Number Special)
for index, song in enumerate(playlist, start=1):
    print(f"Track {index}: {song}")

# While loop (The Old School)
i = 0
while i < len(playlist):
    print(f"Song {i + 1} of {len(playlist)}: {playlist[i]}")
    i += 1

# List comprehension (The One-Liner Wonder)
uppercase_songs = [song.upper() for song in playlist]
print(uppercase_songs)

# Zip (The Mashup)
ratings = [5, 4, 3, 5]
for song, rating in zip(playlist, ratings):
    print(f"{song}: {'★' * rating}")

# Output:
# Stayin' Alive: ★★★★★
# YMCA: ★★★★
# Macarena: ★★★
# Gangnam Style: ★★★★★

Remember: with great looping power comes great responsibility. And potential infinite loops. Don't be that DJ.

Jedi Mind Tricks: Advanced List Techniques

Ready to take your list skills to the next level? These techniques are so advanced, they make Yoda look like a padawan.

List unpacking (The Magician's Trick)

   first, *middle, last = [1, 2, 3, 4, 5]
   print(first, middle, last)  # Output: 1 [2, 3, 4] 5

List as a stack (The Last-In-First-Out Lifesaver)

   stack = []
   stack.append("Learn Python")
   stack.append("Learn list tricks")
   stack.append("????")
   stack.append("PROFIT!")

   while stack:
       print(f"TODO: {stack.pop()}")

   # Output:
   # TODO: PROFIT!
   # TODO: ????
   # TODO: Learn list tricks
   # TODO: Learn Python

List as a queue (The First-In-First-Out Time Machine)

   from collections import deque
   queue = deque(["Wake up", "Drink coffee", "Code", "Sleep"])

   while queue:
       print(f"Now doing: {queue.popleft()}")

   # Output:
   # Now doing: Wake up
   # Now doing: Drink coffee
   # Now doing: Code
   # Now doing: Sleep

Sorting custom objects (The Overachiever's Gambit)

   class Jedi:
       def __init__(self, name, midi_chlorian_count):
           self.name = name
           self.midi_chlorian_count = midi_chlorian_count

   jedi_council = [
       Jedi("Yoda", 17700),
       Jedi("Mace Windu", 12000),
       Jedi("Obi-Wan Kenobi", 13400),
       Jedi("Anakin Skywalker", 27700)
   ]

   sorted_jedi = sorted(jedi_council, key=lambda j: j.midi_chlorian_count, reverse=True)
   for jedi in sorted_jedi:
       print(f"{jedi.name}: {jedi.midi_chlorian_count}")

   # Output:
   # Anakin Skywalker: 27700
   # Yoda: 17700
   # Obi-Wan Kenobi: 13400
   # Mace Windu: 12000

Filtering with filter() (The Bouncer at Club Python)

   numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
   even_numbers = list(filter(lambda x: x % 2 == 0, numbers))
   print(even_numbers)  # Output: [2, 4, 6, 8, 10]

List flattening (The Pancake Maker)

   nested_list = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
   flat_list = [item for sublist in nested_list for item in sublist]
   print(flat_list)  # Output: [1, 2, 3, 4, 5, 6, 7, 8, 9]

Master these techniques, and you'll be manipulating lists like a Jedi Master manipulates the Force. Use them wisely, young Pythonista!

Wrapping Up: You're a List Ninja Now!

Congrats, grasshopper! You've journeyed from the humble beginnings of creating your first list to performing Jedi-level list manipulation. You've learned to slice, dice, loop, comprehend, and even flatten lists like a pro.

Remember, with great power comes great responsibility. Use your newfound list powers for good - like organizing your Netflix queue or sorting your collection of rubber ducks.

Now go forth and conquer the Python world, one list at a time! And remember, when in doubt, just append() it out! 🐍💻🥷

P.S. If anyone asks, you learned all this through years of meditation and drinking mountain dew, not from some random whitepaper on the internet. We've got to keep some mystery alive!
``

Advanced Python Concepts: A Comprehensive Guide

TheLinuxMan — Thu, 18 Jul 2024 01:54:30 +0000

Advanced Python Concepts: A Comprehensive Guide

Introduction
Decorators
Generators and Iterators
Context Managers
Metaclasses
Conclusion

1. Introduction

Python is a versatile and powerful programming language that offers a wide range of advanced features. This whitepaper explores four key advanced concepts: decorators, generators and iterators, context managers, and metaclasses. These features allow developers to write more efficient, readable, and maintainable code. While these concepts may seem complex at first, understanding and utilizing them can significantly enhance your Python programming skills.

2. Decorators

Decorators are a powerful and flexible way to modify or enhance functions or classes without directly changing their source code. They are essentially functions that take another function (or class) as an argument and return a modified version of that function (or class).

2.1 Basic Decorator Syntax

The basic syntax for using a decorator is:

@decorator_function
def target_function():
    pass

This is equivalent to:

def target_function():
    pass
target_function = decorator_function(target_function)

2.2 Creating a Simple Decorator

Let's create a simple decorator that logs the execution of a function:

def log_execution(func):
    def wrapper(*args, **kwargs):
        print(f"Executing {func.__name__}")
        result = func(*args, **kwargs)
        print(f"Finished executing {func.__name__}")
        return result
    return wrapper

@log_execution
def greet(name):
    print(f"Hello, {name}!")

greet("Alice")

Output:

Executing greet
Hello, Alice!
Finished executing greet

2.3 Decorators with Arguments

Decorators can also accept arguments. This is achieved by adding another layer of function:

def repeat(times):
    def decorator(func):
        def wrapper(*args, **kwargs):
            for _ in range(times):
                result = func(*args, **kwargs)
            return result
        return wrapper
    return decorator

@repeat(3)
def say_hello():
    print("Hello!")

say_hello()

Output:

Hello!
Hello!
Hello!

2.4 Class Decorators

Decorators can also be applied to classes:

def singleton(cls):
    instances = {}
    def get_instance(*args, **kwargs):
        if cls not in instances:
            instances[cls] = cls(*args, **kwargs)
        return instances[cls]
    return get_instance

@singleton
class DatabaseConnection:
    def __init__(self):
        print("Initializing database connection")

# This will only print once, even if called multiple times
db1 = DatabaseConnection()
db2 = DatabaseConnection()

Decorators are a powerful tool for modifying behavior and adding functionality to existing code without changing its structure.

3. Generators and Iterators

Generators and iterators are powerful features in Python that allow for efficient handling of large datasets and creation of custom iteration patterns.

3.1 Iterators

An iterator is an object that can be iterated (looped) upon. It represents a stream of data and returns one element at a time. In Python, any object that implements the __iter__() and __next__() methods is an iterator.

class CountDown:
    def __init__(self, start):
        self.count = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.count <= 0:
            raise StopIteration
        self.count -= 1
        return self.count

for i in CountDown(5):
    print(i)

Output:

3.2 Generators

Generators are a simple way to create iterators using functions. Instead of using the return statement, generators use yield to produce a series of values.

def fibonacci(n):
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b

for num in fibonacci(10):
    print(num, end=" ")

Output:

0 1 1 2 3 5 8 13 21 34

3.3 Generator Expressions

Generator expressions are a concise way to create generators, similar to list comprehensions but with parentheses instead of square brackets:

squares = (x**2 for x in range(10))
print(list(squares))

Output:

[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Generators are memory-efficient because they generate values on-the-fly instead of storing them all in memory at once.

4. Context Managers

Context managers provide a convenient way to manage resources, ensuring proper acquisition and release of resources like file handles or network connections.

4.1 The `with` Statement

The most common way to use context managers is with the with statement:

with open('example.txt', 'w') as file:
    file.write('Hello, World!')

This ensures that the file is properly closed after writing, even if an exception occurs.

4.2 Creating Context Managers Using Classes

You can create your own context managers by implementing the __enter__() and __exit__() methods:

class DatabaseConnection:
    def __enter__(self):
        print("Opening database connection")
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        print("Closing database connection")

    def query(self, sql):
        print(f"Executing SQL: {sql}")

with DatabaseConnection() as db:
    db.query("SELECT * FROM users")

Output:

Opening database connection
Executing SQL: SELECT * FROM users
Closing database connection

4.3 Creating Context Managers Using `contextlib`

The contextlib module provides utilities for working with context managers, including the @contextmanager decorator:

from contextlib import contextmanager

@contextmanager
def tempdirectory():
    print("Creating temporary directory")
    try:
        yield "temp_dir_path"
    finally:
        print("Removing temporary directory")

with tempdirectory() as temp_dir:
    print(f"Working in {temp_dir}")

Output:

Creating temporary directory
Working in temp_dir_path
Removing temporary directory

Context managers help ensure that resources are properly managed and cleaned up, reducing the risk of resource leaks and making code more robust.

5. Metaclasses

Metaclasses are classes for classes. They define how classes behave and are created. While not commonly used in everyday programming, metaclasses can be powerful tools for creating APIs and frameworks.

5.1 The Metaclass Hierarchy

In Python, the type of an object is a class, and the type of a class is a metaclass. By default, Python uses the type metaclass to create classes.

class MyClass:
    pass

print(type(MyClass))  # <class 'type'>

5.2 Creating a Simple Metaclass

Here's an example of a simple metaclass that adds a class attribute to all classes it creates:

class AddClassAttribute(type):
    def __new__(cls, name, bases, dct):
        dct['added_attribute'] = 42
        return super().__new__(cls, name, bases, dct)

class MyClass(metaclass=AddClassAttribute):
    pass

print(MyClass.added_attribute)  # 42

5.3 Metaclass Use Case: Singleton Pattern

Metaclasses can be used to implement design patterns, such as the Singleton pattern:

class Singleton(type):
    _instances = {}
    def __call__(cls, *args, **kwargs):
        if cls not in cls._instances:
            cls._instances[cls] = super().__call__(*args, **kwargs)
        return cls._instances[cls]

class Database(metaclass=Singleton):
    def __init__(self):
        print("Initializing Database")

# This will only print once
db1 = Database()
db2 = Database()
print(db1 is db2)  # True

5.4 Abstract Base Classes

The abc module in Python uses metaclasses to implement abstract base classes:

from abc import ABC, abstractmethod

class Animal(ABC):
    @abstractmethod
    def make_sound(self):
        pass

class Dog(Animal):
    def make_sound(self):
        return "Woof!"

# This would raise an error:
# animal = Animal()

dog = Dog()
print(dog.make_sound())  # Woof!

Metaclasses are a powerful feature that allows you to customize class creation and behavior. While they're not needed for most programming tasks, understanding metaclasses can give you deeper insight into Python's object system and can be useful for creating advanced frameworks and APIs.

6. Conclusion

This whitepaper has explored four advanced Python concepts: decorators, generators and iterators, context managers, and metaclasses. These features provide powerful tools for writing more efficient, readable, and maintainable code. While they may seem complex at first, mastering these concepts can significantly enhance your Python programming skills and open up new possibilities in your software development projects.

Remember that while these advanced features are powerful, they should be used judiciously. Clear, simple code is often preferable to overly clever solutions. As with all aspects of programming, the key is to use the right tool for the job and to always prioritize code readability and maintainability.

Comprehensive Python Data Structures Cheat sheet

TheLinuxMan — Thu, 18 Jul 2024 01:45:00 +0000

Comprehensive Python Data Structures Cheat sheet

Lists
Tuples
Sets
Dictionaries
Strings
Arrays
Stacks
Queues
Linked Lists
Trees
Heaps
Graphs
Advanced Data Structures

Lists

Lists are ordered, mutable sequences.

Creation

empty_list = []
list_with_items = [1, 2, 3]
list_from_iterable = list("abc")
list_comprehension = [x for x in range(10) if x % 2 == 0]

Common Operations

# Accessing elements
first_item = my_list[0]
last_item = my_list[-1]

# Slicing
subset = my_list[1:4]  # Elements 1 to 3
reversed_list = my_list[::-1]

# Adding elements
my_list.append(4)  # Add to end
my_list.insert(0, 0)  # Insert at specific index
my_list.extend([5, 6, 7])  # Add multiple elements

# Removing elements
removed_item = my_list.pop()  # Remove and return last item
my_list.remove(3)  # Remove first occurrence of 3
del my_list[0]  # Remove item at index 0

# Other operations
length = len(my_list)
index = my_list.index(4)  # Find index of first occurrence of 4
count = my_list.count(2)  # Count occurrences of 2
my_list.sort()  # Sort in place
sorted_list = sorted(my_list)  # Return new sorted list
my_list.reverse()  # Reverse in place

Advanced Techniques

# List as stack
stack = [1, 2, 3]
stack.append(4)  # Push
top_item = stack.pop()  # Pop

# List as queue (not efficient, use collections.deque instead)
queue = [1, 2, 3]
queue.append(4)  # Enqueue
first_item = queue.pop(0)  # Dequeue

# Nested lists
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
flattened = [item for sublist in matrix for item in sublist]

# List multiplication
repeated_list = [0] * 5  # [0, 0, 0, 0, 0]

# List unpacking
a, *b, c = [1, 2, 3, 4, 5]  # a=1, b=[2, 3, 4], c=5

Tuples

Tuples are ordered, immutable sequences.

Creation

empty_tuple = ()
single_item_tuple = (1,)  # Note the comma
tuple_with_items = (1, 2, 3)
tuple_from_iterable = tuple("abc")

Common Operations

# Accessing elements (similar to lists)
first_item = my_tuple[0]
last_item = my_tuple[-1]

# Slicing (similar to lists)
subset = my_tuple[1:4]

# Other operations
length = len(my_tuple)
index = my_tuple.index(2)
count = my_tuple.count(3)

# Tuple unpacking
a, b, c = (1, 2, 3)

Advanced Techniques

# Named tuples
from collections import namedtuple
Point = namedtuple('Point', ['x', 'y'])
p = Point(11, y=22)
print(p.x, p.y)

# Tuple as dictionary keys (immutable, so allowed)
dict_with_tuple_keys = {(1, 2): 'value'}

Sets

Sets are unordered collections of unique elements.

Creation

empty_set = set()
set_with_items = {1, 2, 3}
set_from_iterable = set([1, 2, 2, 3, 3])  # {1, 2, 3}
set_comprehension = {x for x in range(10) if x % 2 == 0}

Common Operations

# Adding elements
my_set.add(4)
my_set.update([5, 6, 7])

# Removing elements
my_set.remove(3)  # Raises KeyError if not found
my_set.discard(3)  # No error if not found
popped_item = my_set.pop()  # Remove and return an arbitrary element

# Other operations
length = len(my_set)
is_member = 2 in my_set

# Set operations
union = set1 | set2
intersection = set1 & set2
difference = set1 - set2
symmetric_difference = set1 ^ set2

Advanced Techniques

# Frozen sets (immutable)
frozen = frozenset([1, 2, 3])

# Set comparisons
is_subset = set1 <= set2
is_superset = set1 >= set2
is_disjoint = set1.isdisjoint(set2)

# Set of sets (requires frozenset)
set_of_sets = {frozenset([1, 2]), frozenset([3, 4])}

Dictionaries

Dictionaries are mutable mappings of key-value pairs.

Creation

empty_dict = {}
dict_with_items = {'a': 1, 'b': 2, 'c': 3}
dict_from_tuples = dict([('a', 1), ('b', 2), ('c', 3)])
dict_comprehension = {x: x**2 for x in range(5)}

Common Operations

# Accessing elements
value = my_dict['key']
value = my_dict.get('key', default_value)

# Adding/Updating elements
my_dict['new_key'] = value
my_dict.update({'key1': value1, 'key2': value2})

# Removing elements
del my_dict['key']
popped_value = my_dict.pop('key', default_value)
last_item = my_dict.popitem()  # Remove and return an arbitrary key-value pair

# Other operations
keys = my_dict.keys()
values = my_dict.values()
items = my_dict.items()
length = len(my_dict)
is_key_present = 'key' in my_dict

Advanced Techniques

# Dictionary unpacking
merged_dict = {**dict1, **dict2}

# Default dictionaries
from collections import defaultdict
dd = defaultdict(list)
dd['key'].append(1)  # No KeyError

# Ordered dictionaries (Python 3.7+ dictionaries are ordered by default)
from collections import OrderedDict
od = OrderedDict([('a', 1), ('b', 2), ('c', 3)])

# Counter
from collections import Counter
c = Counter(['a', 'b', 'c', 'a', 'b', 'b'])
print(c.most_common(2))  # [('b', 3), ('a', 2)]

Strings

Strings are immutable sequences of Unicode characters.

Creation

single_quotes = 'Hello'
double_quotes = "World"
triple_quotes = '''Multiline
string'''
raw_string = r'C:\Users\name'
f_string = f"The answer is {40 + 2}"

Common Operations

# Accessing characters
first_char = my_string[0]
last_char = my_string[-1]

# Slicing (similar to lists)
substring = my_string[1:4]

# String methods
upper_case = my_string.upper()
lower_case = my_string.lower()
stripped = my_string.strip()
split_list = my_string.split(',')
joined = ', '.join(['a', 'b', 'c'])

# Other operations
length = len(my_string)
is_substring = 'sub' in my_string
char_count = my_string.count('a')

Advanced Techniques

# String formatting
formatted = "{} {}".format("Hello", "World")
formatted = "%s %s" % ("Hello", "World")

# Regular expressions
import re
pattern = r'\d+'
matches = re.findall(pattern, my_string)

# Unicode handling
unicode_string = u'\u0061\u0062\u0063'

Arrays

Arrays are compact sequences of numeric values (from the array module).

Creation and Usage

from array import array
int_array = array('i', [1, 2, 3, 4, 5])
float_array = array('f', (1.0, 1.5, 2.0, 2.5))

# Operations (similar to lists)
int_array.append(6)
int_array.extend([7, 8, 9])
popped_value = int_array.pop()

Stacks

Stacks can be implemented using lists or collections.deque.

Implementation and Usage

# Using list
stack = []
stack.append(1)  # Push
stack.append(2)
top_item = stack.pop()  # Pop

# Using deque (more efficient)
from collections import deque
stack = deque()
stack.append(1)  # Push
stack.append(2)
top_item = stack.pop()  # Pop

Queues

Queues can be implemented using collections.deque or queue.Queue.

Implementation and Usage

# Using deque
from collections import deque
queue = deque()
queue.append(1)  # Enqueue
queue.append(2)
first_item = queue.popleft()  # Dequeue

# Using Queue (thread-safe)
from queue import Queue
q = Queue()
q.put(1)  # Enqueue
q.put(2)
first_item = q.get()  # Dequeue

Linked Lists

Python doesn't have a built-in linked list, but it can be implemented.

Simple Implementation

class Node:
    def __init__(self, data):
        self.data = data
        self.next = None

class LinkedList:
    def __init__(self):
        self.head = None

    def append(self, data):
        if not self.head:
            self.head = Node(data)
            return
        current = self.head
        while current.next:
            current = current.next
        current.next = Node(data)

Trees

Trees can be implemented using custom classes.

Simple Binary Tree Implementation

class TreeNode:
    def __init__(self, value):
        self.value = value
        self.left = None
        self.right = None

class BinaryTree:
    def __init__(self, root):
        self.root = TreeNode(root)

    def insert(self, value):
        self._insert_recursive(self.root, value)

    def _insert_recursive(self, node, value):
        if value < node.value:
            if node.left is None:
                node.left = TreeNode(value)
            else:
                self._insert_recursive(node.left, value)
        else:
            if node.right is None:
                node.right = TreeNode(value)
            else:
                self._insert_recursive(node.right, value)

Heaps

Heaps can be implemented using the heapq module.

Usage

import heapq

# Create a heap
heap = []
heapq.heappush(heap, 3)
heapq.heappush(heap, 1)
heapq.heappush(heap, 4)

# Pop smallest item
smallest = heapq.heappop(heap)

# Create a heap from a list
my_list = [3, 1, 4, 1, 5, 9]
heapq.heapify(my_list)

Graphs

Graphs can be implemented using dictionaries.

Simple Implementation

class Graph:
    def __init__(self):
        self.graph = {}

    def add_edge(self, u, v):
        if u not in self.graph:
            self.graph[u] = []
        self.graph[u].append(v)

    def bfs(self, start):
        visited = set()
        queue = [start]
        visited.add(start)
        while queue:
            vertex = queue.pop(0)
            print(vertex, end=' ')
            for neighbor in self.graph.get(vertex, []):
                if neighbor not in visited:
                    visited.add(neighbor)
                    queue.append(neighbor)

Advanced Data Structures

Trie

class TrieNode:
    def __init__(self):
        self.children = {}
        self.is_end = False

class Trie:
    def __init__(self):
        self.root = TrieNode()

    def insert(self, word):
        node = self.root
        for char in word:
            if char not in node.children:
                node.children[char] = TrieNode()
            node = node.children[char]
        node.is_end = True

    def search(self, word):
        node = self.root
        for char in word:
            if char not in node.children:
                return False
            node = node.children[char]
        return node.is_end

Disjoint Set (Union-Find)

class DisjointSet:
    def __init__(self, vertices):
        self.parent = {v: v for v in vertices}
        self.rank = {v: 0 for v in vertices}

    def find(self, item):
        if self.parent[item] != item:
            self.parent[item] = self.find(self.parent[item])
        return self.parent[item]

    def union(self, x, y):
        xroot = self.find(x)
        yroot = self.find(y)
        if self.rank[xroot] < self.rank[yroot]:
            self.parent[xroot] = yroot
        elif self.rank[xroot] > self.rank[yroot]:
            self.parent[yroot] = xroot
        else:
            self.parent[yroot] = xroot
            self.rank[xroot] += 1

This comprehensive cheatsheet covers a wide range of Python data structures, from the basic built-in types to more advanced custom implementations. Each section includes creation methods, common operations, and advanced techniques where applicable.
0

Forem: TheLinuxMan

I Built a Corrupt Archive Recovery Engine in Rust — Because Every Tool I Tried Just Gave Up

The Core Idea: Stop Aborting, Start Skipping

Three Engines. One Pipeline.

Engine A — Fail-Forward Extraction

Engine B — Zombie LZMA Decoder

Engine C — Magic Header Carver

Benchmarks Against Standard Tools

The Test Suite

Using It

What's Next

Why Rust?

Links

NumPy: The Superhero Library Python Deserves (But Maybe Didn't Know It Needed)

Table of Contents

Introduction: Meet NumPy, Your New Best Friend

The Origin Story: Why NumPy Exists

NumPy's Superpowers: Key Features

Saving the Day: Real-World Applications

The Dark Timeline: A World Without NumPy

Becoming a NumPy Superhero: Code Examples

Creating Arrays: The Building Blocks of NumPy Greatness

Array Operations: Math at the Speed of Light (Almost)

Indexing and Slicing: Dicing Your Data Like a Pro Chef

Statistical Operations: Impress Your Friends with Quick Maths

Reshaping: Because Sometimes You Need to Bend Your Data (Not Break It)

Random Number Generation: For When You're Feeling Lucky

Conclusion: With Great Power Comes Great Computation

The Ultimate Guide to Python Lists: From Newbie to Ninja

Table of Contents

What the Heck is a List Anyway?

Creating Lists: Your First Rodeo

List Methods: Your Swiss Army Knife

Slicing and Dicing: Become a List Surgeon

List Comprehensions: One-Liners That Pack a Punch

Nested Lists: Inception, but with Data

Looping Like a DJ: Iterating Over Lists

Jedi Mind Tricks: Advanced List Techniques

Wrapping Up: You're a List Ninja Now!

Advanced Python Concepts: A Comprehensive Guide

Advanced Python Concepts: A Comprehensive Guide

Table of Contents

1. Introduction

2. Decorators

2.1 Basic Decorator Syntax

2.2 Creating a Simple Decorator

2.3 Decorators with Arguments

2.4 Class Decorators

3. Generators and Iterators

3.1 Iterators

3.2 Generators

3.3 Generator Expressions

4. Context Managers

4.1 The with Statement

4.2 Creating Context Managers Using Classes

4.3 Creating Context Managers Using contextlib

5. Metaclasses

5.1 The Metaclass Hierarchy

5.2 Creating a Simple Metaclass

5.3 Metaclass Use Case: Singleton Pattern

5.4 Abstract Base Classes

6. Conclusion

Comprehensive Python Data Structures Cheat sheet

Comprehensive Python Data Structures Cheat sheet

Table of Contents

Lists

Creation

Common Operations

Advanced Techniques

Tuples

Creation

Common Operations

Advanced Techniques

Sets

Creation

Common Operations

Advanced Techniques

Dictionaries

Creation

Common Operations

4.1 The `with` Statement

4.3 Creating Context Managers Using `contextlib`