Forem: TAHRI Ahmed R.

Revived the promise made six years ago for Requests 3

TAHRI Ahmed R. — Tue, 02 Apr 2024 05:16:45 +0000

Requests is undeniably unmissable for anyone willing to start in the Python ecosystem. Yet the community is broadly unaware of how sad the situation has become.

The incident

Due to dramatics circumstances, it was partially locked and put in "maintenance mode" only, meaning
that it would no longer evolve (features speaking).

Some of you may have eared, more than six years ago that Requests was working toward a major version, namely the V3.
There was even a completed fundraiser with a substantial amount, around $30k.

At the time, there was a plan for bringing Async, and HTTP/2 support (most notably). You'll guess that none of exposed promises ever landed.

Impacts

This incident led to several impacts.

Maintainers felt too much pressure on making steps toward a major, some of them said: "We just can't deliver the promises, and it's going hard to explain why".
All the back pressure landed at urllib3.
Abruptly started to refuse any changes, saying: "Requests is feature complete, and it's popularity makes it nearly impossible to fix bugs that may be considered feature to this point".
Newer project started ditching HTTP client Requests by learning to use a new one.
Some large scale projects bound to Requests did not migrate because of the potential high cost doing it.

Let's be clear, this article is not about saying "Requests maintainers are bad" but rather saying we don't agree, and everyone is entitled to his choices.

For many years now, Requests has been frozen. Being left in a vegetative state and not evolving, this blocked millions of developers from using more advanced features.

The Journey into Waking Up

Initially, like many other reasonable person, I said to myself "Let's wait, Requests will eventually unblock the situation!"
Yet the years are still passing and nothing.

Let's realize something, Internet Explorer 11 just celebrated its tenth anniversary last year, and you’ll never guess that it
has support for HTTP/2 integrated! Yet Requests is unable to do so.

I said to myself: "Why are we waiting for this indefinitely? They don't owe us anything! There's a reasonable chance we are waiting... trapped in a circular reference.. Let's act."

Then, I tried to get a firm grip on urllib3 base code, contributing this and there until I was ready to kick things up with a
proof of concept that would have put urllib3 far ahead. Without any breaking changes.
I was delusional. This was a bit of a shock, but six months passed between my initial kick off and my formal give up, and here's why in a nutshell:

Every bits of idea brought to the table was dismissed, and never seriously analysed.

This felt as good as taking the coldest wind on your face while exiting your warm home. It's the hard truth of the OSS,
it's not a democracy and it's not ruled by pure logic, but humans.

No far latter on, urllib3 called out a fundraiser to help with HTTP/2 support (only) and claimed around $40k to be able to deliver. They probably meant well, but[...]

This was the last straw for me, knowing the past mistakes, and actually the best incentive to break the cycle. Everything point to another year before having the slightest sign of a PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n into our production pipes. They clearly lack the time to move forward (or incentives? or both?). No worries, as developers, we should share this burden for the greater good.

From my perspective (my opinion only), the OSS ecosystem should not fund raise money for a decade old protocol, and most of time should deliver prior to asking. Entities raising money with no solid proof of concept, that's rare!

This felt like a bit of a hostage situation, but it could only be me seeing it this way.

The Solution

So many years and various expertise have been brought to Requests and urllib3.. Do we throw all of it into the garbage disposal? Engineers should be reasonable scientists and not zeled revisionists.

We don't have to reinvent the wheel all over again, HTTP client Requests is well established and really pleasant in its usage.
We believe that Requests has the most inclusive and developer friendly interfaces.

So how to propose an attractive continuity for Requests? How hard could this be to (i) bring a true upgrade experience and (ii) keeping the compatibility as high as possible?

Sometime the OSS ecosystem act like it is trying to land a rover on the Moon, let's be more realistic and bring our vision down to firm grounds.

There's a chance offered to us to rehabilitate Requests (and urllib3) and give a fresh breath of air to us, the developers. Without having to learn something as basic as handling a HTTP client.

So, the work started, without looking back.

Just around my intent to give up on urllib3 evolution, we proposed a fork of both urllib3 and Requests, namely urllib3-future and Niquests.

They are (truly) drop-in replacements, at the time of writing, those forks are six-months old! And the whole work around this is roughly a year old. We listened to what the community needed by... triaging all the issues on major repositories (Requests, urllib3, and dependants of them) so that the upgrade would be truly useful.

Are you seeing it coming? Great! Let's look at the feature table comparison against requests, httpx and aiohttp!

Feature	niquests	requests	httpx	aiohttp
`HTTP/1.1`	✅	✅	✅	✅
`HTTP/2`	✅	❌	✅	❌
`HTTP/3 over QUIC`	✅	❌	❌	❌
`Synchronous`	✅	✅	✅	❌
`Asynchronous`	✅	❌	✅	✅
`Thread Safe`	✅	✅	❌	N/A
`Task Safe`	✅	N/A	✅	✅
`OS Trust Store`	✅	❌	❌	❌
`Multiplexing`	✅	❌	Limited	❌
`DNSSEC`	✅	❌	❌	❌
`Customizable DNS Resolution`	✅	❌	❌	✅
`DNS over HTTPS`	✅	❌	❌	❌
`DNS over QUIC`	✅	❌	❌	❌
`DNS over TLS`	✅	❌	❌	❌
`Multiple DNS Resolver`	✅	❌	❌	❌
`Network Fine Tuning & Inspect`	✅	❌	Limited	Limited
`Certificate Revocation Protection`	✅	❌	❌	❌
`Session Persistence`	✅	✅	✅	✅
`In-memory Certificate CA & mTLS`	✅	❌	Limited	Limited
`SOCKS 4/5 Proxies`	✅	✅	✅	❌
`HTTP/HTTPS Proxies`	✅	✅	✅	✅
`TLS-in-TLS Support`	✅	✅	✅	✅
`Direct HTTP/3 Negotiation`	✅	N/A	N/A	N/A
`Happy Eyeballs`	✅	❌	❌	✅
`Package / SLSA Signed`	✅	❌	❌	✅

And you would think... performance..? are bad?

Look at the performance comparison against them!

Scenario: Fetch a thousand requests using 10 tasks or threads, each with a hundred requests using a single pool of 10 connection.

High-Level APIs

Client	Average Delay to Complete	Notes
requests	987 ms	ThreadPoolExecutor. HTTP/1.1
httpx	735 ms	Asyncio. HTTP/2
niquests	470 ms	Asyncio. HTTP/2

Simplified APIs

Client	Average Delay to Complete	Notes
requests core	643 ms	ThreadPoolExecutor. HTTP/1.1
httpx core	550 ms	Asyncio. HTTP/2
aiohttp	220 ms	Asyncio. HTTP/1.1
niquests core	210 ms	Asyncio. HTTP/2

Did you give up on HTTP/2 due to performance concerns? Think again! Multiplexing and response laziness open up a wide range
of possibilities!

And everything you are witnessing is in fact "Requests"-extended and you have nothing to learn again to get started. We kept 98% of the test cases from both Requests and urllib3, then extended them to cover newer features.

We are looking forward to see you at the Niquests repository! Let's say it's a promotional offer, get everything and more for $̶4̶0̶k̶ $0!

#1 You depends on Requests? Discover Niquests instead.
- Drop-in replacement, simple patch needed, tiny effort required.
#2 You depends on urllib3? Discover urllib3.future instead.
- Drop-in replacement, no patch required, atom-like sized effort required.

By visiting the GitHub repository and documentation you will be able to learn more about this, and hopefully make a decisive switch!

10 reasons you should quit your HTTP client

TAHRI Ahmed R. — Wed, 15 Nov 2023 11:47:52 +0000

I have been both secretly, and publicly working in the HTTP client field for an entire year, here is what my newly acquired
expertise got us. This will shock you, so be prepared. Seatbelt on! Let's travel back to the future.

As a Python programmer, we should be aware that it is soon to be 2024, but most of the community lives like it's 2015.
And we don't always know it. That's insane.

Let us crawl out of the pit we inadvertently dug ourselves, together. We've almost all started by using Requests, and now
we are trapped and kind of forced out sometimes. The thing is... Habits die hard!

I was so lazy to migrate my projects from Requests, that I decided to tackle the next Requests myself.

🏆 (I) HTTP/2 by default, HTTP/3 broadly available!

👴 Internet Explorer 11 just celebrated its tenth anniversary this year, and you'll never guess that it has
support for HTTP/2 integrated (not activated by default^*).

Saying that while the Python ecosystem has no HTTP client supporting HTTP/2, and HTTP/3 by default in November 2023.

How about a client that supports them all? With great incentives! 👇

*: Doesn't this remind you of a particular HTTP client? HTTP/1 by default, optional HTTP/2?

🔀 (II) Multiplexed Connection

We've just almost murdered Async, and made you watch!

Beware that reading down further will follow by a " Cannot be unseen " in your head.

🚨 Some ideas are just sticky, like " faster HTTP " requires Async or your code will be slower. Wrong!

Do you, too, work in a company with programs that exceed 100k lines involving synchronous HTTP requests? No way that the
company would allow us to rewrite the whole thing to async? budget and mental sanity reasons? We are here to save the day.

Let's do a quick MCQ!

According to you, how long does this take to run? We are fetching this resource. The controller behind does a sleep(3) and then returns a valid JSON output.

from requests import Session

responses = []

with Session() as s:
    responses.append(s.get("https://pie.dev/delay/3"))
    responses.append(s.get("https://pie.dev/delay/3"))
    print([r.status_code for r in responses])

A) 6 seconds
B) 3 seconds
C) 12 seconds

✅ ¡spuoɔǝs 9 sɐʍ ɹǝʍsuɐ ǝɥ⊥

Again!

from niquests import Session

responses = []

with Session(multiplexed=True) as s:
    responses.append(s.get("https://pie.dev/delay/3"))
    responses.append(s.get("https://pie.dev/delay/3"))
    print([r.status_code for r in responses])

A) 6 seconds
B) 3 seconds
C) 12 seconds

✅ ¡spuoɔǝs Ɛ sɐʍ ɹǝʍsuɐ ǝɥ⊥

With multiplexed enabled you have just transferred the whole async/await to the remote peer! Isn't awesome?!
No hidden magic trick under the carpet, no async, no threads!

Did you wonder how you would do this without Niquests? Look 👉

There's a non-negligible chance you'd rather write 6 lines with a mere multiplexed=True!

The great thing here is that we made it completely transparent for our fellow developers. You are literally one toggle away from leveraging one or many multiplexed connections!

🔒 (III) Enhanced Security

TLS 1.2+ was just the starting block

Earlier this year, some HTTP clients decided to enforce TLS 1.2 as the minimum supported. We decided to go all in,
not only did we enforce TLS 1.2, but we also enforced the recommended cipher suite that Mozilla regularly updates so that
we remain safe against sophisticated types of attacks. Quantum CPUs are a thing, right?

OCSP!

No HTTP client is, in fact, capable of providing a basic level of insurance that the peer certificate is in fact (still)
valid. With the growing size of issued Let's Encrypt (short-lived) certificates, more insurance is needed.
It's not bulletproof as it follows the fail-safe strategy that most browsers do unless strict-mode is enabled.

The dependency chain is certified! SLSA!

Almost all the dependencies in the chain are verifiable at any moment.
They all benefit from reproducible build.

Each of our releases comes with SLSA provenance data (multiple.intoto.jsonl), which can be used to verify the source and provenance of the binaries with the slsa-verifier tool.

slsa-verifier verify-artifact ./niquests-3.2.3-py3-none-any.whl --provenance-path multiple.intoto.jsonl --source-uri github.com/jawah/niquests --source-tag

Certificates are validated against a trustable source 👇 (Spoiler: Your OS)

🪪 (IV) System Root CAs

Almost everyone is using Certifi or has seen it pass through your usual pip install ... routine.
In a nutshell, Certifi is a package that ships with a static collection of certificate authorities or root CAs if you
prefer.

Certifi was a mistake, a huge one. A workaround is great to move on, but only for a short period of time.

Here's the cons:

Security concerns

This list of trusted CA is so critical that you should be scared if anything would happen,
It is almost never updated by users, so your local copy is most likely too old and may contain revoked certificate.

Does not work at work!

Well, Certifi does not ship with your company's certificates! So requesting internal services may come with additional
painful extra steps! Also for a local development environment that uses mkcert for example!

This is why you should migrate!

Our implementation takes away all of those concerns by looking at your OS trust store. Which is constantly updated in the background and protected properly.

⚡ (V) Faster

You may get up to 3X faster in your usual synchronous context if you ever switch to Niquests.
Without multiplexing, well, it is roughly aligned to competitors, except with Requests, we are faster in all comparable scenarios.

HTTP/3 is a bit less performant as of today, but still much faster than any competitor in a synchronous context.
We are actively working toward making it faster than HTTP/2.

Keep in mind that ~Re~Niquests have a lot of features and that we made every effort conceivable to keep them!

🧠 (VI) In-memory certificate

This one should come with enthusiasm to many advanced users! Python did not allow end-users to load client mTLS certificates and
associated key without having it in a file. And Requests did not allow to pass on your root CAs without a file either.

That's a thing of the past now, we elaborated a clever way to work around this limitation.

🍰 (VII) Object-oriented headers

We should control ourselves when eating sugar, not when pushing the " sugar syntax " as a programmer.

Who did not, at the very least, waste a bit of time accessing HTTP headers from a response? Well!

import niquests

r = niquests.get("https://1.1.1.1")

Now how should we access the max_age from header Nel or Report-To or Strict-Transport-Security ASAP?

r.oheaders.nel.max_age
# and
r.oheaders.report_to.max_age
# also
r.oheaders.strict_transport_security.max_age

You are very much awake, yes, this is now that easy! Pseudo-oriented object headers are long overdue.

😫 (VIII) Fixed known (painful) issues

We've heard you, we know how painful some encounters are, especially with bugs!
Here is an extract of what we've fixed for you! (in addition to the In-memory client certificate)

An invalid content-type definition would cause the charset to be evaluated to True, thus making the program crash.
Given proxies could be mutated when environment proxies were evaluated and injected. This package should not modify your inputs.
Fixed Transfer-Encoding wrongfully added to headers when the body is actually of length 0.
Function proxy_bypass_registry for Windows may be fooled by insufficient control on our end.
Unattended override of manually provided Authorization if .netrc existed with an eligible entry.

🚦 (IX) Type annotated

Knows you are doing right from the start.

Powerful IDEs like PyCharm can easily leverage our definitions to guide you through the code.

Just spotted something weird.

...And here's why!

📝 Yes, actually typeshed already has some definitions for Requests, but they are (most of them) incomplete.

☎️ (X) Plain better ~customer~"developer"-care

It's a cliché linked to small entities, but, it can make all the difference.
You won't felt wronged or thrown away because I am deeply convinced that knowledge is best acquired by good confrontations.

It's a promise, we'll never say to go away to Stackoverflow. We will assume this position forever. For us, even the most insignificant Q&A is like a real commit to the project.
We have a great turnover for responses to your concerns, proposals or criticisms.
Reports are good, always. Never be scared of filling in an issue even if you are not sure of anything. Even less to submit a PR.

➕ Bonus

Main feature matrix

Client / Features Matrix	(Async)	Multiplexed	HTTP/1.1	HTTP/2	HTTP/3
requests	❌	❌	✅	❌	❌
aiohttp	✅	❌	✅	❌	❌
httpx	✅	❌	✅	✅***	❌
niquests	✅**	✅	✅	✅	✅

Compatibility

Niquests is compatible with Python, and PyPy 3.7+, and we remain attached to not dropping any interpreter version unless strictly required to!

Versioning

All releases on the same Major version will be guaranteed to be backward compatible. Unless forced to.

Did you mention Async?

Yes! Niquests support Async, while it's not true Async**, it serves the purpose of having non-blocking code in your event loop! You may combine Async with the usage of a multiplexed connection.

Drop-in replacement for Requests

You have nothing to do (most of the time) to switch to Niquests. A simple CTRL+R on import requests to import niquests as requests suffice! or on from requests import to from niquests import.
Any trouble migrating? Come and let us know what happened. We will find a way, trust me.

Much more you don't know about

Many topics (e.g. improvements) haven't been shared in this article, come and discover all the great things we delivered!

FYI Niquests is more resilient to most bot detectors provided you'd put the right headers (e.g. UA, Sec-CH-UA, etc..)

**: We ported sync_to_async used in asgiref, as Niquests is mostly thread-safe. It is transparent for the end-user.

***: Not enabled by default.

📆 What's next?

With your support, we can project ourselves much more than 2024!

Advanced proxy combination

Today we support tunneling from HTTP/1.1, meaning we can to HTTP/1.1 -> HTTP/2 but not HTTP/2 -> HTTP/2 or even less
HTTP/3 -> HTTP/2 or 3.

Support for DoH, and DoQ

DNS over QUIC is soon to be a must-have.

ECH when using QUIC.
True Async top to bottom

Will take some time. Not urgent.

Advanced scheduling for multiplexed connections

Support for request priority weight.

Faster! Faster!

We'll never settle. This is just the starting block.

—

Source: https://github.com/jawah/niquests
PyPI: https://pepy.tech/project/niquests
Docs: https://niquests.readthedocs.io/en/latest/

— Thank you for reading!

In the end, I got what I wanted, migrated all my projects, enabled multiplexing where it was pertinent, and gained a substantial boost.

Don't screw up reading Headers, Keep It Simple, Stupid ! HTTP or IMAP !

TAHRI Ahmed R. — Tue, 17 Mar 2020 14:01:04 +0000

Why ? Indeed Why ?!

No matters your religion, IMAP4 or HTTP, you should not worries about accessing easily header and associated attributes, adjectives or values.

But even with modern language like Python we still have those :

charset = headers['Content-Type'].split(';')[-1].split('=')[-1]

I have seen so much chunk of code like this, trying to deal with them, often in a risky way to take care of more important part of the code. In face of that I say no more !

Do not forget that headers are not 1 TO 1. One header can be repeated multiple time and attribute can have multiple value within the same header. So representing them by using a ´dict()´ is not a good idea even if using case insensible dict !

Project Kiss-Headers

That is why I created a project that would remove the pain of dealing with them no matter where they came from.

It was absolutely clear that they was something missing in the python community. From the project ´psf/requests´ to ´tornado´, each one have it own implementation, each one are also incomplete. The best so far I have seen is the one from ´httpx´ project.

Mine is not perfect, but has its amount of sweetness. Beginning from auto-completion capability in python interpreter or in your beloved IDE. It should dramatically reduce stress and pain when encountering them.

Ousret / kiss-headers

Python package for HTTP/1.1 style headers. Parse headers to objects. Most advanced available structure for http headers.

HTTP Headers, the Complete Toolkit 🧰

^{Object-oriented headers. Kind of structured headers.}

❓ Why

No matter if you are currently dealing with code using HTTP or IMAP (message, email), you should not worry about easily accessing and exploiting headers.

I have seen so many chunks of code trying to deal with these headers; often I saw this implementation:

# No more of that!
charset = headers['Content-Type'].split(';')[-1].split('=')[-1].replace('"', '')
# That too..
response = get(
    "https://httpbin.org/headers",
    headers={
        "user-agent": "custom/2.22",
        "referer": "https://www.google.com",
        "accept": "text/html",
        "accept-language": "en-US",
        "custom-header-xyz": "hello; charset=utf-8"
    }
)

Scroll down and see how you could do it from now on.

🔪 Features

kiss-headers is a basic library that allow you to handle headers as…

View on GitHub

Imaging being capable of doing those things :

End note

I am pretty confident that those kind of libraries that serve reducing time spent on the little things would actually help make the life of a programmer easier. And spent a lot more time on something else.

So, do not hesitate, when you are encountering a case that respect those criteria : (i) stupid (ii) repeatably reinventing the wheel (iii) everyone is facing this issue at least once. (iv) break the loop by contributing.

Thank you :)

Make ✨ 🍰 ✨ requests act like jQuery when passing nested objects to data/payload

TAHRI Ahmed R. — Sun, 23 Feb 2020 19:11:37 +0000

Once upon a time

I, once was stuck when using python-requests in order to poke a PHP backend.

You may currently face this issue as Requests does not support dict with depth greater than 1 on parameter data for a x-www-form-urlencoded based request.

PHP/jQuery $.params support this syntax out of the box. As requests is not about supporting the way PHP/jQuery $.params work internally, this PR maybe doomed.

To reproduce

from requests import post
from pprint import pprint

if __name__ == '__main__':

    response = post(
        'https://api.ipify.org?format=json',
        {
            'auth_user': 'test',
            'auth_pwd': 'test',
            'json_data': {
                'operation': 'core/get',
                'class': 'ServiceChange',
                'key': 'SELECT ServiceChange WHERE id=429',
                'output_fields': 'request_state',
                'h': {
                    't': 1
                }
            }
        }
    )

    pprint(
        response.request.body
    )

What happen now

print(response.request.body)
# auth_user=test&auth_pwd=test&json_data=operation&json_data=class&json_data=key&json_data=output_fields&json_data=h

Something should happen when passing nested dict. This behavior is wrong. We loss data between passing and sending. At least raise a warning or whatnot.

✨ What should have happened

print(response.request.body)
# auth_user=test&auth_pwd=test&json_data%5Boperation%5D=core%2Fget&json_data%5Bclass%5D=ServiceChange&json_data%5Bkey%5D=SELECT+ServiceChange+WHERE+id%3D429&json_data%5Boutput_fields%5D=request_state&json_data%5Bh%5D%5Bt%5D=1

Why ?

[x] requests is on a higher level than urllib, this does not make sense in urllib, but can in requests
[x] PHP and Ruby on Rails support this syntax out of the box, and is used by more than 30 % of the remote backends.
[x] This PR would make it more beginner friendly, advanced users that are looking for test=foo&test=baz are more likely to encode their payload manually. Not the other way around.
[x] This change does not break or change actual behavior expect for the multivalued param case which is a very rare case in my opinion.

How to use this patch ?

Simply use my fork for the time being, you could support my PR too.

Patch : tree/patch-1
PR : https://github.com/psf/requests/pull/5253

Be more productive 🚀, less robotic 🤖 when inbox need too much attention

TAHRI Ahmed R. — Sun, 23 Feb 2020 17:14:15 +0000

I worked in a french company that is using iTop internally.
This project was born out of a specific need which hinted at the possibility of a much more open and generic case.

A company may face this problem:

How to manage an interoperability of services with others based solely on mails ?

At first they were using 'incoming mail' embedded module. Defined by Combodo/iTop as "This extension runs in the background to scan the defined mail inbox(es) and either create or update tickets based on the content of the incoming emails".

With the old solution (Incoming Mail):

Identification is very limited and restricted
It is mandatory to create IMAP files for each task
Scanner actions are limited to simple operations

Hermes offers a complete solution to what iTop/Incoming Mail cannot provide. A better way to identify mail and a more versatile way to react to it.

There is absolutely no need to know how to code in order to get started.

Right now, it is available in french only. It is open source and free to use. This project is "i18n ready", that mean it can be translated as easily as iTop was. That does not mean it is not translatable by Google.

It is as easy as typing docker-compose up in shell to get started !

Every contribution and feedbacks are welcome.
Project available on GitHub : https://github.com/Ousret/hermes

If needed, "Google-provided" english translation

How I used Python to solve declareless encoding madness

TAHRI Ahmed R. — Wed, 02 Oct 2019 09:36:12 +0000

There is a very old issue regarding "encoding detection" in a text file that has been partially resolved by a program like Chardet.

Nowadays, one could argue that this issue is not actually one. Indeed, most standards are providing a way to declare the encoding, like in HTTP specifications.

But the reality is different, a huge part of the internet still have content with an unknown encoding. One could point out subrip subtitle (SRT) for instance.

This is why a popular package like Requests list Chardet as a requirement to guess apparent encoding on remote resources.

But there is a catch. First of all, libraries like Chardet are unreliable, unmaintained and sometime even disliked publicly by their owner.

Nearly all popular libraries are using the same idea, for each code page or encoding they want to detect they create a specific probe. They are trying to identify originating encoding.

The first thing I did not like is the idea of single prober per encoding table that could lead to hard coding specifications. Secondly, I am convinced that we should not care about the originating encoding, that because two different tables can produce two identical files.

This is where I came with an alternative. A crazy one : Using brute-force to get sense out of a given content. For each encoding there is.

Exclude encoding that does not fit content at all
Measure observable chaos once opened, identify things like "жГЪСЭ Ян"
Measure coherence by letter appearances frequencies

By not creating specific probe per encoding I was able to provide detection for around 90 encodings ! That's more than twice compared to Chardet and it is actually more reliable.

So I present to you Charset Normalizer. The real first universal charset detector. Or if currently busy or whatnot, try it online via this url

How I used brute-force where I least expected it

TAHRI Ahmed R. — Tue, 03 Sep 2019 08:08:58 +0000

There is a very old issue regarding "encoding detection" in a text file that has been partially resolved by a program like Chardet. I did not like the idea of single prober per encoding table that could lead to hard coding specifications.

I wanted to challenge the existing methods of discovering originating encoding.

You could consider this issue as obsolete because of current norms :

You should indicate used charset encoding as described in standards

But the reality is different, a huge part of the internet still have content with an unknown encoding. (One could point out subrip subtitle (SRT) for instance)

This is why a popular package like Requests embed Chardet to guess apparent encoding on remote resources.

You should know that :

You should not care about the originating charset encoding, that because two different table can produce two identical files.
BOM (byte order mark) is not universal and concern only a tiny number of encodings and not only Unicode !

I'm brute-forcing on three premises (in that order) :

Binaries fit encoding table
Chaos
Coherence

Chaos : I opened hundred of text files, written by humans, with the wrong encoding table. I observed, then I established some ground rules about what is obvious when it seems like a mess. I know that my interpretation of what is chaotic is very subjective, feel free to contribute to improve or rewrite it.

Coherence : For each language there is on earth (the best we can), we have computed letter appearance occurrences ranked. So I thought that those intel are worth something here. So I use those records against the decoded text to check if I can detect intelligent design.

So I present to you Charset Normalizer. The Real First Universal Charset Detector.

Feel free to help though testing or contributing.