Forem: Giacomo Tesio

About European Draft for AI Ethics Guidelines

Giacomo Tesio — Mon, 28 Jan 2019 00:00:00 +0000

What follows are the comments I’ve sent to the High-Level Expert Group working to define AI Ethics Guidelines for the European Commission.

To understand them, you need a decent understanding of Artificial Intelligence and Machine Learning and to read the DRAFT ETHICS GUIDELINES FOR TRUSTWORTHY AI

Introduction: Rationale and Foresight of the Guidelines

The first lines of the introduction highlight a serious flaw of the draft: the pillars that underpin the Commission’s vision show a fundamental bias:

increasing public and private investments in AI to boost its uptake,
preparing for socio-economic changes, and
ensuring an appropriate ethical and legal framework to strengthen European values

Being eager to adopt a largely misunderstood technology obviously inhibits the ability to reason about its limits and risks.

Before trying to boost its uptake, the Commission should try to understand to what extent and in which fields of endeavour the set of techniques that goes under the AI umbrella should be experimented.

As Shoshana Zuboff recently wrote, technology is NOT an unstoppable force of nature, but a human artifact serving interests and needs of specific humans. In other words, Technology is a prosecution of Politics by other means: each advancement can be designed to serve the public interest or private and elitarian ones. And just like with Politics, a renounce to participate to its course just means to being subject to others’ will.

Before talking about “Trustworthy AI”, we should have a population able to understand the topic enough for their trust to be meaningful.

As for today, without a serious investments in schools to foster History and Informatics as preconditions of our citizenship, such trust can not be meaningful but just deceptive and ill founded.

It’s not a trust on the technology, but in the corporations and the “experts” that can exploit such trust and the widespread ignorance of the topic to weaken regulations and streghten their handle on society.

Having said that, the high level description outlined for the Ethical framework is basically sound: it’s reasonable to think that when the whole population will be able to understand how a neural network’s calibration differs from a k-mean clustering, a similar framework will emerge.

However the glossary that preceed the Introduction already shows that we are not ready for such framework: despite being written by an high level expert group on AI, the definitions still use an antropomorphic language to describe what is just software. In particular describing software bugs (either intentional or unintentional) as “bias” shows a deep misunderstanding about the software in question and about the statistical processes that define its behaviour. Later on, similar concerns emerge when the draft cites “non-determinism” while talking about software that is executed by deterministic machines (aka computers).

Such language is worrying because it shows a tendency from the HLEG to rationalize the risks as inevitable instead of understanding them deeply and taking them into account.

Chapter I: Respecting Fundamental Rights, Principles and Values - Ethical Purpose

Despite an interesting and convidisible introduction, the principles that the chapter proposes lack a fundamental hierarchical structure.

It should be quite evident by looking at such principles:

The Principle of Beneficence: “Do Good”
The Principle of Non maleficence: “Do no Harm”
The Principle of Autonomy: “Preserve Human Agency”
The Principle of Justice: “Be Fair”
The Principle of Explicability: “Operate transparently”

Even if we hadn’t more than two thousands years from the Hippocratic Oath and generations of physicians grown with the “Primum non nocere” maxim, we can see how the last three principles are just specializations of the more general “Do no Harm”. In particular the Principle of Autonomy tries to address risks to individuals, the Principle of Justice tries to address the risks to weak groups and the Principle of Explicability tries to address socio-political risks.

Since the Principle of Non Maleficience is so preponderant to require three specializations, we should put it first, before the principle of Beneficience, and underlining its relation with the others:

The Principle of Non maleficence: “Do no Harm”
- The Principle of Autonomy: “Preserve Human Agency”
- The Principle of Justice: “Be Fair”
- The Principle of Explicability: “Operate transparently”
The Principle of Beneficence: “Do Good”

The road to hell is paved with good intentions: just like with medicine, whenever simpler and safer solutions exist they should be preferred to more complex and risky ones.

But there is an even more important omission in the list: the Principle of Ultimate Human Accountability.

This is a fundamental principle that underlie all European ethical and legal system: at least a human must always be accountable for the problems caused by a human artifact.

In other terms: what is forbidden to a human can not be allowed through an artificial proxy, no matter how “autonomous” (aka expensive to debug) such proxy is.

Talking about ethics is void if we are not ready to enforce this simple but fundamental principle of human responsibility.

The section on “Lethal Autonomous Weapon Systems” is in direct contrast to all the principles stated above.

The only way an Ethical Framework can be credible while proposing principles like “Do no Harm”, “Preserve Human Agency”, “Be Fair”, “Operate transparently” and “Do Good” is to clearly state that Autonomous Weapon Systems (lethal or not) must be forbidden on the European territory.

The section on the “Potential longer-term concerns” shows the usual sci-fi based fears that are the flip side of the current hype.

Instead of being concerned about Artificial Consciousness that would be way easier to fake than to implement we should be afraid of semi-autonomous weapons in the hands of a small group of people holding most of the planet’s wealth. And in the count of such weapons we should obviously include every tool that can be used to direct human attention, to manipulate feelings or perceptions and to forge mass opinions.

Chapter II: Realising Trustworthy AI

Even this chapter present several issues:

The short paragraph about “Accountability” suggest to design mechanisms that can range from monetary compensation to apology, but it forgets to include prison: to gain trust it is important to explicitly state that an autonomous proxy cannot become a “Get Out of Jail Free” ticket.
The section on “Safety” looks like it was designed to be ineffective: it’s pointless to assess potential risks associated with the use of AI-based products and services without defining serious punishments when things goes wrong anyway.
The section on “Trasparency” is too vague and forgiving: a simpler approach is to say that no opacity must be allowed in applications that consume human data. Such rules would instantly skyrocket private and public investments in AI research, looking for new machine learning techniques that can be fully explained and debugged.
The section on “Robustness” looks well designed but open to a wide de-responsibilization when it improperly talks about “non-determinism” (false, if we are talking about deterministic, non-quantum, computers) and it cites “complexity”, “opacity”, and “sensitivity to training/model building conditions” as a sort of justifications for unreproducible results. Simply, whenever such conditions exists, the AI program is not robust and should not be applied to problems that require such robustness.
The section about “Human Autonomy” is very scary: in no way people should be nudged by machines. If AI will be successful in enhancing human wealth as it’s promise to be, a lot of friendly people with a lot of free time will be able to nudge us on our request, but it’s too dangerous to let flawed machines manipulate humans whatever the goal: every software has bugs vulnerabilities and many have intentional backdoors: AI won’t be different.

Later, in “Architectures for Trustworthy AI”, while considering the technical means to ensure an ethical behaviour the HLEG suggest to integrate an ethical signal in the “sense” phase of the stochastic system.

This is both naive and weird:

WHICH ethics we should use? If we widely deploy autonomous machines following a certain ethical model, people will adapt to it (because machines cannot really adapt to us): this could turn to be most effective brain washing project ever conceived. Humans naturally adapt to the sorrounding intelligences: put a consumist agent in every room, and you will build a population of consumists.
HOW MUCH ethics? Who will decide the weight of that signal? And what when a bug will inhibit it? Or what if other inputs overwhelm such signal? The only use of an ethical signal in an autonomous system is to shield corporations from taking full responsibility of errors: it’s dumb to pretend to teach ethics to trolleys, we should build infrastructures that simply prevent lethal incidents to occur.

Moreover in the section about Regulation we lack any reference to penal justice: just like before, it should be clearly stated that when an autonomous artifact kill or harm, one or more humans will be held fully accountable for it.

Chapter III: Assessing Trustworthy AI

I really appreciated the flexible approach to the assessment process: talking about ethics, a checklist would be too easy to exploit.

For sure, each technique requires different kind of assessments: for example the dataset used to calibrate a k-mean could be enough to reproduce the calibration process and to exclude any racial discriminations, but it would be totally inadequate for assess any property of a classifier based on an artificial neural network.

The risk however is that, without a widespread understanding of the AI techniques, the Commission will ask to the wolfs how to rule the sheeps: we cannot rely on experts that consults large corporations to define any assessment of “trust” into something that can manipulate people.

Moreover, being able to assess the Ethics of a “Trustworthy AI” cannot replace clear regulation establishing the characteristics that an algorithm must have before being fed with human data.

In particular we need to extend the right to “meaningful information about the logic involved” by each AI processing beyond the individuals protected by the article 13 of the GDPR: even groups, such as families, neightbors, customers and so on should have the right to know and understand the exact logic applied to their collective data, when and to which aim the processing occurs.

General Comments

Despite all the issues described above, I appreciated the effort and care that has been evidently put by the HLEG in the writing of this draft.

It’s important for Europe to fill our technological gap with U.S.A. and China and it’s conforting to see serious people working on the ethical issues that will emerge from the AI adoption.

However is even more important to avoid short-cuts. Good will and honesty are fundamental, but not enough to balance lobbying and hype.

To address our technological issues (including AI adoption) we need to raise the general population understanding of Informatics. We need a new mass education plan, with serious investments on teachers and professors from the primary school on. We need to raise a generation of people able to modify the software that they use and they feed with their own data.

Since Technology is Politics, being able to self-host and customize the applications we use is the only way to preserve democracy: it will prevent data capitalization and people manipulation.

Programming is today what Writing was during Ancient Egypt: a tool which is totally primitive, but effective to collect and retain Power among humans exactly because it is primitive.

We need better systems, better programming languages and people able to use software without being manipulated through it.

Until then, widespread adoption of AI can be useful, but it’s irresponsible to apply it to human data. We need prudent regulations that err on the side of caution, not because computer-aided statistics is dangerous in itself but because it’s too easy to abuse it and manipulate or hurt people and societies in a context when most people can’t understand their working.

Jehanne: simplicity awakes

Giacomo Tesio — Thu, 15 Nov 2018 00:00:00 +0000

Preemptive multitasking is now more than 50 years old, initially introduced in PDP-6 Monitor and MULTICS in 1964. Instead of relying on processes to cooperatively release the CPU to the kernel, they registered timer interrupt handlers to move control back to kernel code.

Nevertheless, processes were still able to yield control of the CPU to the kernel during a time-slice by calling any system call.

System calls are often distinguished into blocking or non-blocking depending on the probability that the kernel will return the control to the calling process’ code before the next tick.

In real world, non-blocking syscalls might block the process and blocking syscalls might not block at all: it all depends on implementation details and on the state of the system at run-time.

Classical examples of blocking system calls are

sleep that should block for a certain number of seconds
wait that could block until a children process exit
read that would block until a chunk of data is available
write that would block on a pipe without enough space for the new data

CPU control and time

Sleep is a very interesting syscall because its whole point is just to yield the CPU for a while, so that other processes can use it. Its main use case is to poll for a certain resource to become available without subtracting computing power that could be used to produce such resource: it’s a tool for cooperative multitasking in a OS supporting preemptive multitasking.

Other blocking system calls (like wait, read and write) depend on external events to return control to the process. Soon programmers realized that an external event might never occur and designed ways to mitigate the risks.

Unix introduced signals and services like alarm and setitimer to give back control to the calling process on certain events or after a certain time.

Later, new syscalls like select, poll, kqueue or sigtimedwait were introduced with a timeout parameter from the very beginning.

Plan 9 from Bell Labs

Compared to the 400 system calls of Linux, Plan 9’s API is rather simpler but it still supports a bit of each time-control styles:

sleep suspends the calling process for a number of milliseconds specified by the argument.
alarm causes an alarm note to be sent to the invoking process after a number of milliseconds.
tsemacquire only waits for a number of milliseconds to acquire a semaphore.

By design, Plan 9 provides only a very limited support for non-blocking I/O (through alarm) and no support for I/O multiplexing:

if a program needs to wait for several resources , it usually calls rfork
if a program want to serve several concurrent clients , it usually expose a 9P2000 filesystem

Furthermore, with libthread, Plan 9 provides an implementation of Hoare’s Communicating Sequential Processesin which dedicated (preemptively scheduled) processes are used to issue any blocking system calls and several cooperatively-scheduled threads sharing memory in a single process are used to access global state.

The Curse of Frankenstein

The complex interactions between the different ways to handle timeouts in Unix (and, to a lesser extent, in Plan 9) are direct effects of its design philosophy.

As explained by Richard P. Gabriel in his famous essay, "Unix and C are the ultimate computer viruses".

Worse is better, isn’t it?

Worse-is-Better is a model of software design and implementation which has the following characteristics (in approximately descending order of importance):

Simplicity: The design must be simple, both in implementation and interface. It is more important for the implementation to be simple than the interface. Simplicity is the most important consideration in a design.
Correctness: The design should be correct in all observable aspects, but it is slightly better to be simple than correct.
Consistency: The design must not be overly inconsistent. Consistency can be sacrificed for simplicity in some cases, but it is better to drop those parts of the design that deal with less common circumstances than to introduce either complexity or inconsistency in the implementation.
Completeness: The design must cover as many important situations as is practical. All reasonably expected cases should be covered.

Completeness can be sacrificed in favor of any other quality. In fact, completeness must be sacrificed whenever implementation simplicity is jeopardized. Consistency can be sacrificed to achieve completeness if simplicity is retained; especially worthless is consistency of interface.

In a way the so called “New Jersey style” was a rush for a minimum viable product able to minimize the time-to-market and to gain the first mover advantage.

Quoting Gabriel

Both early Unix and C compilers had simple structures, are easy to port, require few machine resources to run, and provide about 50%-80% of what you want from an operating system and programming language.

[…]

It is important to remember that the initial virus has to be basically good. If so, the viral spread is assured as long as it is portable. Once the virus has spread , there will be pressure to improve it , possibly by increasing its functionality closer to 90% , but users have already been conditioned to accept worse than the right thing.

So far so good.

Early Unix, like Plan 9, didn’t aim at perfection, but at serving the majority of users’ need well enough.

But is "enough", enough?

The Unix philosophy spread beyond Unix, and it became the common wisdom of software engineering so far.

For early Unix and Plan 9, enough is enough. Look at 9front to see a modern system that follows this philosophy consistently.

There is a catch, however.

Plan 9 (like early Unix), is a system evolving as a whole. This is particularly visible in the 9front repository. The whole system evolves together, all programs are modified consistently, answering to the ever changing needs of its users.

Plan 9 is one application of computers. An operating system split into several executables, but still one thing.

So in its asymptotic path toward completeness, all pieces progress together. When they provides 90% of required functionality, it really means 90%. Or 95%. Or 97%. Or 99%.

But what if we split the system into independent components and assign them to different teams following the same philosophy?

This is what happened to mainstream operating systems.

With their widespread adoption, the variety of people needs couldn’t be served anymore by a single company. Thus good APIs and development tools provided a strategic advantage to build ecosystems of applications that, by running on a certain OS, would have expanded its market share.

Unix was particularly well suited for this kind of race for application developers.

However this process had a significant but overlooked drawback: the running operating system lose its unity, its coherence. The operating system became a collection of variegate software, developed by a variety of different teams, each with their own goals and taste and set of best practices.

More pieces, more cracks!

Take one application that works correctly 99% of times, and you will be fine 99% of times.

But what if you have 10 applications each working correctly 99% of times?

The probability that they work correctly together at each computer clock is (99%)^10. That is: their composition will do what you expect roughly 90% of times.

With such assumption, the probability of a system behaving correctly goes down pretty quickly approaching the formula

p(n) = (0.99)ⁿ

How many programs are you running right now? :-D

This is the curse of Frankenstein: more pieces, more cracks.

Simplex Sigillum Veri

Plan 9 from Bell Labs followed the New Jersey style from the very beginning and still does so in most if it’s incarnations.

But what about Jehanne, from Meucci’s laboratory?

Turns out the design of Jehanne doesn’t follow the New Jersey style. I have no rush. Yet Jehanne doesn’t follow the MIT/Stanford style either.

To follow Gabriel’s scheme, we could say that Jehanne is based upon

Simplicity: The design must be simple. Few simple, easy to learn and orthogonal abstractions should be able to describe any use case conceivable. If the implementation is difficult, the design is not simple enough.
Correctness: The design should be correct in all observable aspects. Incorrectness is not allowed.
Consistency: The design must not be inconsistent. Any inconsistency reveals a design problem: either a missing abstraction or abstractions that are not orthogonal enough.
Composability: Completeness should naturally emerge as a (desirable) side effect. As such, it cannot be a goal: the design must cover as few important situations as practical and let the user compose the abstractions provided to build what he needs.

I call this set of principles simplex sigillum veri.

It’s deeply inspired by the works of other Italian hackers, such as Antonio Meucci, Pier Giorgio Perotto, Guglielmo Marconi, Renzo Piano and Leonardo da Vinci.

The Awake system call

Plan 9 supports 39 system calls (not counting obsolete ones). Since some system calls can be expressed in term of the others, in Jehanne I moved the duplicates in userspace.

The idea is that a smaller kernel surface makes it easier to get it right and simple in the long run.

During such clean up, I realized that sleep, alarm andtsemacquire were somehow “ugly”: I supposed that they were mixing orthogonal issues and looked for the missing abstraction they were hiding.

This is how the awake syscall was born:

long awake(long milliseconds);

Awake is the complement of sleep: it books a new time slice in the future. It’s a fundamental building block that can be used to implement other services in user space, like sleep, alarm and tsemacquire.

It interrupts a blocking syscall after a certain number of milliseconds. On failure it returns 0. On success it returns a negative long that can be used as an identifier to release the booked time slice.

On wakeup, no note or signal is sent to the process, the process’ error string left unchanged: the blocking syscall simply returns to the process with a ~0ULL result.

A process can register how many wakeups it want (within a global system cap) and each wake up will have a chance to interrupt a system call.

Wakeups are booked in two distinct group that do not interact: normal process execution and note handlers. Such groups are automatically reset respectively on exits and on noted.

In libc, two very simple functions wrap awake to enhance readability.

Awakened tells the calling process whether a certain wakeup already occurred.Forgivewkp tells the kernel to remove a certain wakeup.

With these primitives it’s trivial to move sleep and tsemaquire to libc.

In particular, tsemaquire shows pretty well the simple idiom of awake:

/* book a time slice in the future */
long wkup = awake(ms);
while(blocking_syscall() == -1){
    if(jehanne_awakened(wkup)){
        /* handle syscall timed out */
    }
    /* the syscall has been otherwise interrupted, you can
     * - try again
     * - fail with an error
     * - do whatever fit
     */
}
/* the syscall completed, release the booked time slice... */
jehanne_forgivewkp(wkup);

/* ...and enjoy */

(the mindful reader will notice that alarm is still waiting to be moved to user space... the fact is that it’s too boring of a task!)

To be fair, awake was originally designed to interrupt rendezvous only, to enable timeouts support for QLock, RWLock and Rendez in libc.

But from the very beginning it was clear that it could have been generalized and composed with other system calls, to provide other services.

With the advent of libposix, I used awake to implement support for POSIX non-blocking I/O, signals and sigset waiters.

Which in turn enabled the port of newlib and of MirBSD Korn Shell to Jehanne.

All this with more or less 600 lines of code.

Two kernel processes, a timer and a ringer, cooperate through a linked list of wakeups kept in order of expiration that is filled by the system calls. On each tick, if the first element of the registry is expired, the interrupt handler awake the timer, that prepare a collection of expired timers.

Then the ringer start a loop to interrupt the process properly.

These two different processes make it possible to keep processing new wakeups while the ringer is waiting for the process to be in an interruptible state.

Not all blocking system calls can be interrupted though. Create is a notable example of a blocking system call that has been excluded from the interruptible ones, to prevent a timeout to leave an orphan file behind.

Issues and future uses

Awake will be used to implement all timeouts of Jehanne’s system calls.

Even without libposix, it provides a simple mechanism to implement non-blocking I/O, if needed.

And it will be used to implement timeouts in multiplexing I/O too.

Unfortunately, as an interface to the kernel scheduler, awake doesn’t work with user space schedulers that keep control of their own tasks. Right now, this just means that qlockt, rlockt, wlockt and rsleeptcannot work when linking libthread. In the long run, it might make slightly more difficult to port virtual machines and programming languages that provide their own scheduler in user space, like Go or Java.

The solution however is already outlined by libthread usage of a custom rendezvous: we will simply override the system call to give control to the desired scheduler.

The Intelligent Symbiosis

Giacomo Tesio — Mon, 15 Oct 2018 16:30:06 +0000

This is a translated (and polished) transcript of my talk (slides here) at the XV Congress of the Association Luca Coscioni about Research Freedom.
This is a cross post for my followers here: this is the canonical URI

Good evening, my name Giacomo, I'm a programmer and I will try to share here a more practical perspective on these matters. I will talk about the symbiosis between humans and machines, trying to envision a path that can move us forward into this direction.

A new specie

Let's start with a definition of Artificial Intellingence from the University of Stanford

Artificial intelligence (AI) is the field devoted to building artificial animals (or at least artificial creatures that – in suitable contexts – appear to be animals) and, for many, artificial persons (or at least artificial creatures that – in suitable contexts – appear to be persons).

Now, if we introduce a new specie into an ecosystem, be it artificial or not, we break its dynamic equilibrium with few possible outcomes:

the extinction of the new specie, refused by the ecosystem
an articulation of the food web
the destruction of the ecosystem itself
the establishment of a symbiotic relationship among the new specie and a resident one.

We can surely exclude the first outcome, since we are here talking about Artificial Intelligences that are taking off. But we can also exclude the articulation of the food web by looking at the metabolism of Artificial Intelligences that is based on data:

data collected by specialized infrastructures
provided to AIs in INPUT
transformed by such AI
returned from the AIs in OUTPUT
and then fed to other AIs
or turned into informations by humans

We can thus see at least three phases:

Eating
Defecation
Interpretation (of the excrements :-D)

which is a little magical, it's always arbitrary, since it's up to humans.

Data are just representations of information.
Information belong to human minds.

But where do these data come from?

Feeding Artificial Intelligences

To understand this I suggest you to look at this except from a wikipedia page.

In a footnote it has a link to a primary source.
By clicking such link we spread at the very least 10 personal informations.

Some informations are cultural, some are medical, some are geopolitical and some are economical, since if you are browsing from the last iPhone you probably have different wealth then if you browse Wikipedia from a library's PC.

Note that this is the bare minimum: with the default browser configuration, you spread a lot more infos. And this is just one click: you can't even imagine how many precious medical infos you spread by playing an online casual game!

However this data are what AI eat, every day, without any interaction with the carbon cycle: they are neither prey nor predator for other species.

Thus we are left with two possible outcomes:

the destruction of the ecosystem
the symbiosis

That is: we can either go to Terminator... or to WALL-E.

Artificial... Politics.

Sadly, this year excluding the Terminator outcome became harder since, on March, an artificial intelligence from Uber killed a 49 years old woman in Arizona.

Nobody went to jail.
Not even the car, that had perceived Elaine Herzberg but it didn't break anyway.

What does it means?

The Singularity!

We have two possible interpretations of this fact.

The first is that we have reached the Singularity: the machine asserts its superiority by killing a citizen of the greatest military power of the planet.

Please note the political acumen of this intelligence: while AI are said to reproduce human bias, it killed a white woman in the states; but to avoid a popular uprising it killed a homeless, a poor, a woman that was not relevant in the capitalist economy that stay at the core of the values of that nation.

This way the car was able to establish a precedent, with a clear message: "it's allowed to sacrify human lifes to progress; it's fine to kill people to facilitate the evolution of my specie".

A super intelligence, as you can see.

But there is another explanation.

Hype-Driven Politics

That simulator shouldn't have been driving a 2 tons car in a road open to public transit. Indeed Uber's engineers explained that the car was configured to avoid emergency braking maneuvers while the vehicle is under computer control, "to reduce the potential for erratic vehicle behavior".
The car didn't break to avoid car sickness to Uber's customers.

So Elaine wasn't killed for progress, but for Uber's profit!

Teaching ethics to machines is like teaching sex to condoms.
I'd focus on humans, first.

And this is relevant to because Uber was invited to test a technology totally inadequated to the complexity it had to face, by an ignorant Governor.

But a Governor whose ignorance is pretty mainstream.

A misleading language

The language we use to describe reality forges our understanding of it.

Thus when we use words like "intelligence", "learning", "training" and even "neural networks", we evoke human experiences, things that are deep in our consciuosness, but that have NOTHING to do with that happens inside these software.

This language is great for science-fiction, but it sucks at science.

It would be easy to adopt a more descriptive technical language, a less anthropomorphic one, but we prefer to easily get funds out of fantasies than to really understand what we are searching with those funds.
After all, the more it takes to find it, the more funds!

Robot means slave!

However this incident let us exclude the Terminator outcome.

It proves that, even when a machine kills autonomously, it kills after a human choice. A human decided that Elaine life was worth less than Uber profit.

So even when we will be able to build autonomous weapons, they won't destroy the planet. They won't destroy the ecosystem. At all!
They will just thin out humanity of the exceding 99 percent!

Symbiosis

So we are left with a single possible outcome: human-machine symbiosis.

But what is symbiosis?

An intimate association, often obliged, between members of different species, that usually lead to a coevolution.

It's a long process, started by single individuals of the original species that gain an evolutive advantage over the populations they come from.

Thus we should ask ourselves: who shall inherit the Earth? Who will join this coevolution?

The smart phones into your pockets can make you think you are part of these chosen people, but I have to disappoint you. To raise artificial intelligences you need to control data. You do not control the data about you.

Companies like Google, Amazon, Facebook, Apple, Microsoft and Uber hold your data.

Now, somebody will say that these company cannot do evil, and that the interdependencies produced by the global market will protect all of us from destructive practices.

The market protection

To reason about this objection I propose you to consider the browsers that serve over 90% of the market: Google Chrome, Mozilla Firefox, Microsoft Edge and Apple Safari.

These browsers are the primary milking tools... I mean... the primary tools to collect users' data. They are distributed for free, but funded or developed by the companies. But through these software any website you visit can

tunnel into your private network, wherever you work, in an hospital, in a bank, in a military base, in Parliament... wherever!
put illegal contents into your device without you knowing it

and several other attacks that would be too long to describe here.

These attacks are possible by decades but they have been reported a month ago to these organizations. Still you do not know about them because they closed the security report saying that they cannot fix them.

This is false: in that same bug report, I suggested few simple expedients, pretty easy to implement, that would largerly reduce the attack surface.
The problem is that one of these expedients is to make JavaScript execution opt-in (as Flash was) and JavaScript is fundamental to the business model of tons of companies in the Silicon Valley.

This shows that the invisible hand do not protect us from ignorance.

These companies did not inform their users, they did not inform you.

We have to understand that the market do not work whenever we have information asymmetry!

Understanding the geopolitical hazard, China decided to compete on the same ground, creating several big companies in robotics, face recognition and artificial intelligence in general that are slowly taking huge market shares.

What about Europe?

Obviously, Europe could start competing on the market too: Macron already proposed France for this race and others will follow.

But I think there are other terrains where Europe, with its complex history, have a real advantage, such as

the Principles and the Laws that stem from such principles, such as:
- Reponsibility that must always be held by identifiable humans, particularly when the machines fail
- Algorithmic transparency that should always be total and absolute when AI are applied to humans
the Research that such Principles could inspire(and such Laws incentivate)
- new AI algorithms that can grant the total transparency (without just pretending to)
- new language, that would not fool people
the Culture, through the study of History and Informatics

Symbiosis is a Coevolution

If Symbiosis is a coevolution process, we have to evolve too, we need our collective intelligence to evolve. To this goal we usually use Education.

Today Informatics is what Writing was 5000 years ago: a tool which is totally primitive (nothing we have built in the last 100 years will last!), but an effective tool to collect and retain Power among humans exactly because it is primitive.

But who own such Power?

This is Audrey Jones. She is 11 years old and she hacked the electronic election system of the United States of America. In 10 minutes. How is this possible?

Randall Munroe explain part of the problem with this wonderful strip

Our entire field is bad at what we do, and if you rely on us, everyone will die.

(and NO, the blockchain is not the solution, it's a symptom of the problem)

But there is another aspect of this story that is even more important.

Audrey Jones can

write
count
program

which are all preconditions of a true citizenship in the world we live in.

And computers are easy! A SQL injection is a trivial attack: the fact that you cannot do it doesn't mean that it's difficult, just that you don't know how to do it! Audrey can do it, and she is eleven!

Audrey Jones has

Curiosity
Fantasy
and the Cultural tools to build the infrastructure of the world she desires!

Europe need to raise a generation of hackers like Audrey!
A generation that will produce Politicians that understand the cybernetic democracy we live in!
We cannot teach to Politicians how computers work: it must be part of their background from the primary school!

Because, to be honest, people like me will always be able to fool you. But we can't fool Audrey... she knows best!

Audrey shows us that we can only be either Hackers or (biological) Robots, either Citizens or Slaves, because (let's be clear on this) if a software can affect the elections of a Nation, it means that you are the robots!
You that believe you are using the software, while you are being used by the software because you don't know how it works.

We do not have to teach our sons to use computers, we have to teach them how to create with computers. So that they can build the world they want. A world we cannot even imagine but that we should prepare them to live in anyway.

We must put servers in every house, mail servers... web servers...

And we must start to teach History and Informatics at school. Seriously!

It is the only option we have, not to put chips under skin.

This is the Symbiosis we need: to know how to create the Future.

I have been banned from Lobste.rs, ask me anything.

Giacomo Tesio — Tue, 11 Sep 2018 19:37:00 +0000

Let me start by saying that Lobste.rs is a great community that I enjoined for more than an year. Several very smart guys hungs there, and I got great conversations with them about operating system design, programming languages, artificial intelligence and machine learning, security, privacy and so on.

I also tried to be a constructive member of such community, posting there interesting documents I came across.

NOTE In the url above the two submission marked as "[Story removed by original submitter]" have been removed by the administrator after my ban.

I didn't remove them. I have nothing to hide.

One was my recent article documenting an exploit that let any website you visit to tunnel into your private network (bypassing many corporate firewalls and proxies).

The other was the related bug report that I wrote to Mozilla (than reported to Chromium too) before disclosing such Proof-of-concept exploit.

Something went wrong after these submissions, because despite the fact Lobste.rs was suggested by a Mozilla Security developer as a place to continue the discussion about the HTTP/JavaScript vulnerability I reported, nobody answered to my question "are Firefox users vulnerable to this wide class of attacks?".

Yet I got downvoted so much that an administrator (after writing me on August 30 for the first time) decided that I do not suit to the community's culture.

The official reason of the ban was: "Constant antagonstic behavior and no hope for improvement".

Now let's be clear, I'm fine with Peter's decision, even if I don't agree with it. Your server, your rules.

But I think that my ban is a very nice example of Statistics misuse.

Indeed, since the first private message I got from Peter, he asked me to explain why I was downvoted 18 (and later 22) standard deviations more than the average.

Note, I was also upvoted enough to get a positive ranking on most of my comments and posts, but he was just looking to the downvotes, in isolation.

As one who knows how to lie with statistics this was a bit of a smell, but since my private explanations were not enough I carefully explained how most of those downvotes did not complied with the Lobste.rs own guideline about downvotes (sorry, due to the downvotes, you have to expand this comment to see the explaination).

To get a clue about my bad behavior you can give a look to my recent comments on Lobste.rs (some of the comments have been censored, but Peter has kindly sent me a CSV containing a full export from the DB).

Here some examples of the missing contents (beware, 18+ only! :-D):

I feel very uneasy about the safe browsing thing.

For most people (those using WHATWG browsers like Firefox, Chromium, IE/Edge and derived such as Tor Browser, Safari or Google Chrome) there is not such a thing like "safe browsing".

I mean: if any website you visit can enter your private network or check in your cache if you visited a certain page... or upload illegal contents into your hard disk... calling it safe is rather misleading!

HTTPS protects users by certain threats, by reducing the number of potential attackers to CA and those who have access to certificates (which is a varying and large number of people anyway, if you consider CDN or custom CA you might have to install on your work pc).

As for this being anticompetitive... maybe.

But some of the issues here are rooted in Copyright protection, so... it might just be one of the many problems of a legal system designed before information technology.

see in context here

NOTE: every browser executing JavaScript and honouring HTTP cache controls headers is equally vulnerable.

see in context here

I'm seriously concerned by this attitude among IT people.

My question is simple and have a boolean answer.

Are the attacks described in the bug report possible, or not?

see in context here

Okay, I’ll bite.

+1! I'm Italian! I'm very tasty! ;-)

Bugzilla is not a discussion forum.

Indeed this is a bug report.

Ah, here’s where we disagree. I understand that a bug is an ambiguous concept. This is why we have our Bugzilla etiquette, which also contains a link to Mozilla’s bug writing guidelines.

I'm pretty serious with netiquette, and I checked your before writing the report.

I'm very sorry if I violated one of your etiquette rule, but honestly I cannot see which one.

Even about Bug writing I tried my best, what exactly I got wrong?

Note that this is not a single RCE, but a whole category of them.

And the problem are not just the JavaScript attacks themselves, but the fact that they can remove all evidences.

Furthermore, what you seek to discuss is not specific to Mozilla or Firefox.

True. Several other browsers are affected too, but:

This doesn’t means that it’s not a bug in Firefox

As a browser “built for people, not for profit” I think you are more interested about the topic.

Please elaborate, I am not sure what you mean to imply.

As a Firefox user (and "evangelist") from version 0.8 I know Mozilla as a brand that cares about people.

Even the word you used, "people" instead of "users", has always been inspirational to me.

Now, the issue here is specifically dangerous because not all people live under the same law.

Thus I think (and hope) that Mozilla is more interested to the safety of such people than other browser vendors that are led by profit.

I agree with what @callahad@wandering.shop says right away: If you browse to a website. It gives you JavaScript. The browser executes it. That’s by design! Nowadays, the web is specified by W3C and WHATWG as an application platform. You have to accept that the web is not about hyper*text* anymore.

I worked (and still work) on such application platform for 20 years, I think I have understood that pretty well.

The point is if such application platform is broken at design level or not.

This is not a bug in Firefox.

Are you saying that these attacks are not possible?

I am saying that this is not specific to Firefox, but inherent to the browser as a concept.

Sorry if I ask it again, but I'm pretty dumb.

Are the attacks described in the bug report possible in Firefox, or not?

see in context here

This is a just a sampling but if you find other censored contents that you are curious about feel free to ask.

Now, I still think that Lobste.rs is a great technical community and you should really join them. And even Peter is a good administrator: he just did an error.

But I'm a Data Science hobbyist myself, so feel free to ask me how an actual troll could fool such metric by downvoting others. Or why if you do not care about Internet points (and do not try to maximize them), you will obviously loose a lot of them.

Or well... ask me anything else! :-D

I'm not from Mozilla Security.

I will answer. I'm a hacker.

The Meltdown of the Web.

Giacomo Tesio — Mon, 03 Sep 2018 11:00:08 +0000

As Bruce Perens recently put it, I'm "just a programmer".

A humble programmer. And a self-taught one.

A programmer that has learned how to program from a weird group of people whose core value is curiosity: the hackers.

So when I see a security hole affecting in various ways billions of people, I behave like a programmer. I try to fix it... or get it fixed. As soon as possible.

So a mounth ago, I wrote an article explaining how the Web is still a DARPA weapon (that sometimes backfire, as the Russiagate shows).

There I describe two dangerous flawns of the Internet and the Web.

Once I realized that most security experts didn't understand the severity of the issue, I talked about it with a Mozilla developer that suggested to open an issue to Mozilla.

Thus I spent two hours to write a detailed bug report, but it was soon closed (without saying if the Firefox users are vulnerable to such attacks or not), because

Bugzilla is not a discussion forum.

On the suggested Lobste.rs thread (cached here), I asked if Firefox users are vulnerable to such wide class of attacks (several times) without getting a response.

Instead I got several sarcastic, condescending and even - insulting - comments.

Still, no response to such a simple question. Are Firefox users vulnerable?

When I reported the same issue to Chromium team, it was closed in less than ten minutes with the same tone:

Filing a bug here isn't the way to change web standards no matter how you feel about them.

It worth noticing here that both Mozilla and Google are WHATWG members and they write the Living Standards that we are talking about. Living Standards that basically follow the implementations.

To my money, this means that you have to fix the implementations to fix the standard... but remember, I'm just a programmer!

Now, I think I've been very clear about the wide class of attacks that JavaScript opens. When asked to, I even carefully explained how simple is to fix them.

But since

this is the Web functioning as designed

I want you to see what the Web is designed for.

PoC of one of the many possible exploits (bypassing corporate firewalls)

Please add a temporary line to your C:\Windows\System32\drivers\etc\hosts containing

127.0.0.1 local.jsfiddle.net

This mimic the control of a DNS from the attacker.

Then try this simple JSFiddle with a WHATWG browser.

You can change the port number at line 21 to test for any port on your PC.

You can change the IP in /etc/host to probe other machines on your LAN.

JSFiddle (the fictional attacker) has just bypassed your corporate firewall/proxy.

Everything is broken.

This is just one of the uncountably many attacks you can do this way.

I could go on hours inventing more attacks. And you should be able too.

EDIT: here you can find another exploit

As explained in the bug report, you can target a specific person or group.

Even over a CDN (thus through a third party site that the victim trusts).

And then you can reload an harmless script from the same url, rewriting the cache copy and removing all evidences of the attack.

It's really just a matter of compentence and fantasy.

Still I'm not going to find a cool name or draw puppets to "evangelize" about it. I'm a programmer, not a clown.

How can we fix it?

As I explained in the bug report, the technical solution is basically to

make users opt-in to program executions on a per-website basis
threat such programs as potentially dangerous

You can read a simple recap with details here.

However, what you can see here is how deeply the Web is broken.

This is not (just) about JavaScript.

This is about people.