Forem: Yu Qian Yang

How I Used Bit Manipulation to Speed Up Float-to-Int Conversion in a Storage Engine

Yu Qian Yang — Wed, 15 Apr 2026 02:36:14 +0000

In a columnar time-series database, one of the most effective compression tricks is
deceptively simple: if a float value is actually an integer, store it as one.

Why Integers Compress Better Than Floats

Integer compression algorithms like Delta-of-Delta, ZigZag, and Simple8b work by
exploiting predictable bit patterns — small deltas between adjacent values, values
that fit in fewer than 64 bits, and so on. They can pack multiple values into a
single 64-bit word.

Floats don't cooperate with these schemes. Even 1.0 and 2.0 have completely
different IEEE 754 bit representations (0x3FF0000000000000 and 0x4000000000000000).
Their XOR is large, their delta is meaningless as an integer, and bit-packing is useless.

So when a column is declared as FLOAT but actually contains values like 12.0,
18.0, 25.0 — which happens more often than you'd expect, either because the schema
was designed generically or because the upstream system always emits .0 values — you're
leaving significant compression headroom on the table.

The fix: detect these integer-valued floats at encode time, convert them losslessly
to integers, and route them through the integer compression path.

A temperature sensor that reports 21.0, 21.5, 22.0 is a good example. Multiply
by 10 and you get 210, 215, 220 — plain integers with small, predictable deltas.
Delta-of-Delta or Simple8b will compress these far more efficiently than any
float-specific scheme.

The challenge: before converting, you need to check whether the scaled value can be
losslessly represented as an integer. The naive check — std::isnan + range comparison —
works but it's slower than it needs to be on the hot encoding path.

Here's the faster approach I implemented, using nothing but bit manipulation.

The Setup: Scaling Floats to Integers

The encoding scheme works in two steps:

Scale: multiply the float by 10^scale (configurable per column)
Convert: cast the scaled value to integer using std::lround

For example, with scale = 2:

1.23 → 1.23 * 100 = 123.0 → 123
45.678 → 45.678 * 100 = 4567.8 → overflow risk or precision loss

Step 2 only makes sense if the scaled value actually fits in the target integer type.
That's the overflow check.

The Overflow Check

The function takes a pointer to the raw float bytes and the target integer width in bytes.
It returns non-zero if the value would overflow.

Called before every conversion — if it fires, skip the integer path and fall back to
float encoding.

The key insight: you can determine whether a float overflows a given integer type
purely from the float's exponent bits, without doing any arithmetic.

Here's why.

IEEE 754 in One Paragraph

A double-precision float is stored as 64 bits:

[ sign: 1 bit ][ exponent: 11 bits ][ fraction: 52 bits ]

The value is: 1.fraction × 2^(exponent − 1023)

The 1023 is the bias — it allows the 11-bit exponent field to represent negative
exponents. The real exponent is stored_exponent − 1023.

For 32-bit floats: 8 exponent bits, bias 127, fraction 23 bits.

Extracting the Exponent

For a double:

uint64_t bits;
memcpy(&bits, src, 8);                        /* safe type-pun, no UB */
int16_t real_exp = (int16_t)((bits >> 52) & 0x07ff) - 1023;

Step by step:

memcpy into a uint64_t — reinterpret the 8 bytes as a 64-bit integer (no arithmetic, just bits)
>> 52 — shift right past the 52 fraction bits, bringing the exponent to the low end
& 0x07ff — mask off the sign bit, keep only the 11 exponent bits
- 1023 — subtract the bias to get the real exponent

For a float:

uint32_t bits;
memcpy(&bits, src, 4);
int16_t real_exp = (int16_t)((bits >> 23) & 0xff) - 127;

Same logic: shift past 23 fraction bits, mask 8 exponent bits, subtract bias 127.

The Overflow Condition

Once you have the real exponent, the overflow check is one comparison:

is_overflow = real_exp > int_typewidth * 8 - 2;

Where does - 2 come from?

−1 for the sign bit: a signed integer of N bits can hold values up to 2^(N-1) - 1
−1 for the implicit leading 1: in IEEE 754, the fraction is 1.fraction, not 0.fraction

So a float with real exponent E represents a value with E + 1 significant bits
(the implicit 1 plus E fraction bits). For it to fit in a signed N-bit integer, you
need E + 1 ≤ N - 1, which simplifies to E ≤ N - 2.

Full implementation in C:

#include <stdint.h>
#include <string.h>

/* Returns 1 if the double at src overflows a signed integer of int_bytes bytes. */
static inline int double_overflow_check(const char *src, int int_bytes)
{
    uint64_t bits;
    memcpy(&bits, src, 8);
    int16_t real_exp = (int16_t)((bits >> 52) & 0x07ff) - 1023;
    return real_exp > int_bytes * 8 - 2;
}

/* Returns 1 if the float at src overflows a signed integer of int_bytes bytes. */
static inline int float_overflow_check(const char *src, int int_bytes)
{
    uint32_t bits;
    memcpy(&bits, src, 4);
    int16_t real_exp = (int16_t)((bits >> 23) & 0xff) - 127;
    return real_exp > int_bytes * 8 - 2;
}

Total cost: one memcpy, one shift, one AND, one subtract, one compare.
No floating-point arithmetic, no branches on the value itself.

How It Fits into the Encoder

The encoder scales the value first, then calls the overflow check on the scaled result:

double scaled = orig * scaler;               /* scale: e.g. orig * 100.0 */
if (double_overflow_check((char *)&scaled, sizeof(int64_t)))
    return ENCODE_OVERFLOW;                  /* fall back to float encoding */

int64_t result = llround(scaled);            /* safe: overflow already ruled out */

The scale factor is stored in the column header so the decoder can reverse the
operation: decoded = (double)stored_integer / pow(10, scale).

Why Not Just Use `std::isnan` + Range Check?

The conventional approach:

if (std::isnan(value)) return false;
if (value > INT64_MAX || value < INT64_MIN) return false;
return true;

This involves floating-point comparisons, which on many architectures require the
value to be loaded into a float register before comparison. On a hot encoding path
processing millions of values, the difference adds up.

The bit manipulation approach operates entirely on integer registers. The float's
bytes are reinterpreted as an integer — no floating-point unit involved until the
final std::lround conversion, which only happens when you've already confirmed
no overflow.

What This Enables

This check is the entry gate for the full encoding chain:

float column
    ↓
check_float_overflow   ← this article
    ↓ (passes)
float → integer cast
    ↓
Delta+ZigZag encoding
    ↓
Simple8b bit-packing

Without a cheap overflow gate, the chain can't run on untrusted float data. With it,
each value costs one check before entering the integer compression path — which can
achieve far better compression ratios than float-specific schemes on "integer-like"
time-series data.

What's Next

This article is part of a series on compression engineering in time-series databases:

Part 1: Runtime adaptive compression — how the system selects the best algorithm without scanning all data (published)
Part 3: Chained encoding — the full float-to-integer → Delta+ZigZag → Simple8b pipeline
Part 4: An improved floating-point compression algorithm based on ELF

I'm currently available for freelance work on backend systems, storage engineering,
and systems integration. Feel free to reach out.

How I Built a Runtime Adaptive Compression System That Selects the Best Algorithm Without Scanning All Data

Yu Qian Yang — Tue, 14 Apr 2026 13:31:07 +0000

When I was working on a columnar time-series database built on Greenplum, we faced a
problem that every time-series storage engineer eventually hits:

No single compression algorithm works best for all data.

A sensor reading temperature every second looks nothing like a financial tick stream.
A column of integer IDs compresses completely differently from a column of floating-point
measurements. And in the real world, the same column can change character over time —
steady signal for hours, then suddenly a burst of noise.

The naive answer is: let the user pick an algorithm. But that puts the burden on the user,
and in practice they almost always pick wrong — or just leave the default.

So I designed automode: a runtime analyzer that samples incoming data, detects its
statistical character, and selects the best encoding chain automatically.

Here's how it works.

The Encoding Chain

Before talking about the analyzer, I need to briefly explain what it's selecting between.

We built a set of encoding algorithms, each optimized for a specific data pattern:

Run-Length Encoding — all values are identical
Delta-of-Delta — slowly changing values (timestamps, smooth sensors)
Delta + ZigZag — values oscillating around a baseline
Bit Packing — small integers that can be bit-packed
Gorilla — floating-point values with small XOR between adjacent values

The insight that made automode possible: these algorithms operate on fundamentally
different statistical features of data. If I can identify the dominant feature of a
data stream, I can map it directly to the right algorithm.

The Design Problem

The obvious approach — scan all the data and compute statistics — doesn't work in a
storage engine. The whole point of compression is to be fast. Scanning every value
before encoding defeats the purpose.

I needed a way to characterize a data stream with minimal overhead.

The solution: sample windows.

Instead of scanning the full column, I divide the input stream into fixed-size windows
and analyze only a small number of items from each window. Within each window, the
sample start position is randomized to avoid being fooled by local patterns.

Why randomize? Consider this sequence:

[1 1 1 1 1 1 1 1 1 1 1 1 2 3 2 3 2 3 2 3 2 3 2 3]

If you always sample from the start, you'd classify this as "all equal" and pick
Run-Length Encoding. But if you look at the full stream, it's clearly an alternating
pattern — RLE would be terrible. Random sampling within the window gives a much more
representative picture.

The Five Features

Each sampled window is analyzed for five statistical features:

All Equal — all sampled values are identical
Values bitwise small — the values themselves fit in few bits
1st-order delta small — differences between adjacent values are small
2nd-order delta small — differences between differences are small (delta-of-delta)
Float XOR small — XOR of adjacent float values is small (Gorilla pattern)

For each window, the analyzer computes 0th, 1st, and 2nd order differentials, plus
XOR pairs for float data. "Small" is defined bitwise: a value is small if its
significant bit length is below a configurable threshold. This makes the check
robust against occasional spike values — a single outlier doesn't ruin the
classification.

Strong vs. Weak Features

Not all features are equal. I introduced a Strong/Weak classification:

Strongest: All Equal → Run-Length Encoding
Strong: 2nd-order delta small → Delta-of-Delta
Strong: Float XOR small → Gorilla
Weak: Values bitwise small → Bit Packing
Weak: 1st-order delta small → Delta + ZigZag

Why does this matter? Consider a case where Delta-of-Delta and Bit Packing score
similarly. Delta-of-Delta is fundamentally better for smoothly varying time-series
data, even with a slightly lower score. The weight system ensures strong methods
are preferred when they're competitive.

One Edge Case Worth Noting

The alternating pattern [A B A B A B] is a trap.

It looks like "values bitwise small", but Bit Packing performs poorly on alternating
sequences. A general-purpose compressor actually handles this pattern better.

I added an explicit check: if odd-indexed and even-indexed items within the sample
are each internally equal, it's an alternating pattern — skip the Bit Packing
classification and fall back to general compression.

This kind of edge case only appears when you test against real production data. In our
case, it showed up in sensor data from a deployment at a large financial institution.

Feature → Algorithm Mapping

Once features are aggregated across all sample windows, the dominant feature maps to
an algorithm:

All Equal → Run-Length Encoding
2nd-order delta small → Delta-of-Delta
Values bitwise small → Bit Packing
1st-order delta small → Delta + ZigZag
Float XOR small → Gorilla
No clear pattern → General Compression (fallback)

For floating-point columns, the encoding chain adds a float-to-integer conversion pass
first — converting floats that happen to be integers (like 1.0, 2.0) into actual
integers before passing to the integer encoding path. This is worth a separate article.

Two Modes

The system supports two optimization priorities:

Compression-rate mode: maximize storage efficiency
Speed mode: maximize decompression speed

In practice, compression-rate mode is used for cold storage and analytics workloads,
and speed mode for hot data that's frequently queried. The same feature detection runs
in both cases — only the final algorithm selection weights differ.

Results

The system is deployed in production at scale, including a large-scale installation
at a major financial institution managing thousands of PBs of time-series data. In
testing, automode consistently matched or outperformed manually-tuned single-algorithm
compression across diverse real-world datasets.

The key insight that made it work: you don't need to scan all the data to understand
its character. A few dozen samples, taken randomly from fixed-size windows, are
enough to make a reliable classification — as long as you handle the edge cases.

What's Next

This article is part of a series on compression engineering in time-series databases:

Part 2: Float-to-integer encoding — how I used IEEE 754 bit manipulation to detect convertibility with near-zero overhead
Part 3: Chained encoding — a unified float compression pipeline
Part 4: An improved floating-point compression algorithm based on ELF

I'm currently available for freelance work on backend systems, storage engineering,
and systems integration. Feel free to reach out.

Forem: Yu Qian Yang

How I Used Bit Manipulation to Speed Up Float-to-Int Conversion in a Storage Engine

Why Integers Compress Better Than Floats

The Setup: Scaling Floats to Integers

The Overflow Check

IEEE 754 in One Paragraph

Extracting the Exponent

The Overflow Condition

How It Fits into the Encoder

Why Not Just Use std::isnan + Range Check?

What This Enables

What's Next

How I Built a Runtime Adaptive Compression System That Selects the Best Algorithm Without Scanning All Data

The Encoding Chain

The Design Problem

The Five Features

Strong vs. Weak Features

One Edge Case Worth Noting

Feature → Algorithm Mapping

Two Modes

Results

What's Next

Why Not Just Use `std::isnan` + Range Check?