Forem: Karkael

When the cloud isn't enough (triple modular redundancy)

Karkael — Mon, 28 Oct 2024 18:42:27 +0000

You're using a single source for your applications, and you're wrong.

Dear administrator, I'm humbled to make this appeal to you, after yet another failure by our team to ensure the availability of our application. I'm going back to my engineering classes, taking the opportunity to optimize, cut costs and harmonize.

This article is aimed at all applications that require a high level of availability or whose data is sensitive.

Triple modular redundancy, engineering curriculum

https://en.wikipedia.org/wiki/Triple_modular_redundancy

During your engineering courses, you must have studied processes for reinforcing your product: emergency stops, mistake-proofing, monitoring tools. Today, remember triple modular redundancy: a critical service must be available from at least 3 sources.

For your application, this means you need to think about:

provide a service from 3 sources,
duplicate the data in 3 sources,
allow a balance from one to the other.

Standards, ISO-27001

https://en.wikipedia.org/wiki/ISO/IEC_27001

When you're working with large entities or internationally, you need to ensure your customers' your ability to follow standards such as ISO-27001, which will enable you to work with industries in particular. These standards were introduced in 2005 and are regularly revised.

ISO-27001 is based on 3 pillars:

Confidentiality of information,
Information integrity,
Information availability.

To obtain this certification, having a single data source and a single application source won't be enough: you'll have to apply triple modular redundancy. You can't depend solely on the server running on your premises, nor can you depend solely on a cloud.

The benefits of Clouds and internal servers

The advantages of a cloud:

✅ faster start-up,
✅ no hardware management,
✅ outsourced hardware technical skills (fewer employees),
✅ standards already applied (e.g. redundancy),
✅ flexibility and on-demand dimensions,
✅ service close to the customer.

The advantages of an in-house server:

✅ complete, self-supervised management,
✅ fixed costs,
✅ information security assured,
✅ easy migrations,
✅ self-managed breakdowns,
✅ controlled liability,
✅ evolutions as needed,
✅ servers developed to suit the customer, with no superfluous extras.

In this tangle of benefits, it's important to understand:

that a cloud will help you get started and take good practices early in your application development ⭐,
that it's in your interest to favor an in-house application and data service (as soon as possible) ⭐⭐,
that you will need to switch to cloud in case of in-house problems and program a balance ⭐⭐⭐,
that you'll need to enable service redundancy on the cloud ⭐⭐⭐⭐.

By choosing a triple redundancy of your service (by having at least 3 internal servers and clouds), you'll even be able to determine the qualitative data in case of conflict between two services: the two servers that give the same answer are right, the server that answers differently must correct its corrupted data ⭐⭐⭐⭐⭐.

Successfully balancing

Encourage in-house use of the application: its fixed cost means you can budget correctly, while the cloud can take over in the event of overload or breakdown. Cloud costs are harder to predict. When only your in-house server is running, you save a lot of money on the cloud.

Once you've decided how to disperse your services (how many internal servers and cloud servers) and once you've defined the type of service you need (FaaS, SaaS, PaaS, IaaS, On-Premise), then you can determine the cloud closest to your needs, start development with it, try to get the same level of service internally, and finally look for another cloud with the same level of service to get triple redundancy.

Consider working with at least one cloud that can provide services as close to the customer as possible, particularly to avoid additional download costs.

Available tools

Containers and management

For simple development, I'd recommend pairs like Kubernetes and Docker, which can easily export a similar configuration between the local work environment, the internal server and all the clouds that enable this kind of IaaS.

Developing with Kubernetes ensures that all servers are the same, and reduces the risk of getting different results (such as corruption).

Software-as-a-Service

For a simple, inexpensive web application, you'll need a domain name, HTTP server, file server and database, take your pick:

A registrar that allows balancing on its CNAME (OVH, Gandi, ...).
AWS: lambda, S3, RDS
Azure: Functions, Blob Storage, Database
Google: Cloud Run, Cloud CDN, Cloud SQL
A multitude of equivalents for in-house services.

Developing compatible SaaS is more difficult, but it does help to reduce runtime costs.

Cleaning, synchronization and security

This redundancy calls for exceptional rigor and costly management. You'll also need to ensure that the perimeters are secure from one another, and that synchronization does not propagate attacks. You'll need to ensure 4 mandatory services:

determine the server closest to the client, to favor nearby downloads,
always copy all data to other cloud and internal servers,
check integrity regularly (e.g. daily at midnight UTC with hash keys) and deal with data corruption,
produce synchronization reports.

Block-chain

In the spirit of redundancy and data integrity, block-chain proposes a systemic approach to determining a secure, decentralized source. In reality, we're not far from a block-chain.

Depending on your data format, you can use blockchain. My guess is that most applications will need services that handle data in very different formats (user, file, checkout, auth), and your CDN or PostgreSQL servers deserve synchronization services at their scale.

Conclusion

Make sure you're not dependent on a single data source. An in-house server and a cloud can work in a complementary way. You'll get a secure, robust application that's ready to work with global companies.

Quand le Cloud n'est pas suffisant (triple redondance modulaire)

Karkael — Mon, 28 Oct 2024 18:30:54 +0000

Vous utilisez une seule source pour vos applications, et vous avez tort.

Très cher administrateur, c'est avec humilité que je te fais cet appel, après un nouvel échec de notre équipe à assurer la disponibilité de notre application. Je reviens vers mes cours d'ingénieur, j'en profite pour optimiser, diminuer les coûts et harmoniser.

Cet article s'adresse à toutes les applications qui ont besoin d'un fort taux de disponibilité ou dont les données sont sensibles.

Triple redondance modulaire, cursus ingénieur

https://en.wikipedia.org/wiki/Triple_modular_redundancy

Durant vos cours d'ingénieur, vous avez forcément étudié des procédés pour renforcer votre produit: arrêts d'urgence, détrompeurs, outils surveillance. Aujourd'hui, rappelez-vous de la triple redondance modulaire: un service critique doit être disponible par au moins 3 sources.

Pour votre application, cela signifie que vous devez penser à:

fournir un service par 3 sources,
dupliquer la données en 3 sources,
permettre une balance de l'un à l'autre.

Normes, ISO-27001

https://en.wikipedia.org/wiki/ISO/IEC_27001

Lorsque vous travaillez avec de grandes entités ou à l'international, vous devez assurer à vos clients la capacité à suivre des normes comme l'ISO-27001, qui vous permettra notamment de travailler avec les industries. Ce sont des normes qui sont apparues en 2005 et révisées régulièrement.

L'ISO-27001 repose sur 3 pilliers:

La confidentialité de l'information,
L'intégrité de l'information,
La disponibilité de l'information.

Pour obtenir cette certification, avoir une seule source de donnée et une seule source applicative ne sera pas suffisante, vous allez devoir appliquer la triple redondance modulaire. Vous ne pouvez pas dépendre que du serveur qui tourne dans vos locaux, vous ne pouvez pas non plus dépendre que d'un cloud.

Les apports des Clouds et des Serveurs internes

Les avantages d'un cloud:

✅ démarrage plus rapide,
✅ pas de gestion matérielle,
✅ compétences techniques matérielles sous-traitées (moins de salariés),
✅ des normes déjà appliquées (ex: redondance),
✅ flexibilité et dimensions à la demande,
✅ service au plus près du client.

Les avantages d'un serveur interne:

✅ gestion entière supervisée soi-même,
✅ coûts fixes,
✅ sécurité de l'information assurée,
✅ migrations faciles,
✅ pannes gérées soi-même,
✅ responsabilité maîtrisée,
✅ évolutions au besoin,
✅ serveurs développé pour correspondre au client, sans superflus.

Dans cet enchevêtrement d'avantages, il faut comprendre:

qu'un cloud vous aidera à démarrer et prendre des bonnes pratiques tôt dans le développement de votre application ⭐,
que (dès que possible) vous avez intérêt à favoriser un service interne de l'application et des données ⭐⭐,
que vous aurez besoin de cloud en cas de problème en interne et programmer une balance ⭐⭐⭐,
que vous devrez permettre une redondance du service sur le cloud ⭐⭐⭐⭐.

En choisissant une triple redondance de votre service (en ayant au moins 3 serveurs internes et clouds), vous allez même pouvoir déterminer la donnée qualitative en cas de conflit entre deux services: les deux serveurs qui donnent la même réponse ont raison, le serveur qui répond différemment doit corriger sa donnée corrompue ⭐⭐⭐⭐⭐.

Réussir la balance

Favorisez l'utilisation de l'application en interne: son coût fixe vous permet de prévoir un budget correctement, tandis que le cloud peut prendre le relais en cas de surcharge ou de panne. Le coût d'un cloud est plus difficile à prévoir. Lorsque seul votre serveur interne tourne, vous faites de grosses économies sur le cloud.

Quand vous aurez choisi comment disperser vos services (combien de serveurs internes et serveurs clouds) et quand vous aurez défini le type de service nécessaire (FaaS, SaaS, PaaS, IaaS, On-Premise), alors vous pourrez déterminer le cloud le plus proche de vos besoins, démarrer le développement avec lui, tenter d'obtenir le même niveau de service en interne, et enfin chercher un autre cloud avec le même niveau de service pour obtenir la triple redondance.

Pensez à travailler avec au moins un cloud qui peut mettre à disposition des services au plus proche du client, notamment pour éviter les surcoûts liés au téléchargement.

Les outils disponibles

Conteneur et gestion

Dans le cadre d'un développement simple, je recommande plutôt des couples comme Kubernetes et Docker, qui peuvent facilement exporter une configuration similaire entre l'environnement de travail local, le serveur interne et tous les clouds qui permettent ce genre de IaaS.

Développer avec Kubernetes permet d'assurer que tous les serveurs sont les mêmes, et diminue les risques d'obtenir des résultats différents (comme une corruption).

Software-as-a-Service

Dans le cadre d'une application web simple et peu coûteuse, vous avez besoin d'un nom de domaine, d'un serveur HTTP, d'un serveur de fichiers et d'une base de donnée, choisissez:

Un registar qui permet la balance sur ses CNAME (OVH, Gandi, ...).
AWS: lambda, S3, RDS
Azure: Functions, Blob Storage, Database
Google: Cloud Run, Cloud CDN, Cloud SQL
Une multitude d'équivalents pour les services internes.

Développer des SaaS compatibles est plus difficile, mais ça permet de diminuer les coûts sur l'exécution.

Nettoyage, synchronisation et sécurité

Cette redondance impose une rigueur exceptionnelle et avec un management coûteux. Vous allez aussi vous assurer que les périmètres sont sécurisés les uns par rapport aux autres et que les synchronisations ne propagent pas d'attaque. Vous devrez vous assurer de 4 services obligatoires:

déterminer le serveur le plus proche du client pour favoriser les téléchargements proches,
toujours copier toutes les données dans les autres serveurs cloud et internes,
s'assurer de l'intégrité régulièrement (ex: tous les jours à minuit UTC avec des clefs de hash) et régler les corruptions de données,
produire des rapports de synchronisation.

Block-chain

Dans l'esprit de la redondance et de l'intégrité des données, la block-chain propose une approche systémique pour déterminer une source sûre et décentralisée. En réalité, on n'est pas loin d'une block-chain.

Selon votre format de donnée, vous pouvez faire appel à la block-chain. J'estime que la plupart des applications auront besoin de services qui gèrent des données dont les formats seront très différents les uns des autres (user, file, checkout, auth), et vos serveurs CDN ou PostgreSQL méritent des services de synchronisation à leur échelle.

Conclusion

Assurez-vous rapidement que vous n'êtes pas dépendant d'une seule source de données. Un serveur interne et un cloud peuvent travailler de façon complémentaire. Vous obtiendrez une application sécurisée, robuste et prête pour travailler avec les entreprises internationales.

When the cost of quality (COQ) overwhelms the project

Karkael — Mon, 23 Sep 2024 14:16:24 +0000

As I'm looking for a new job as a web developer, I'm looking at my reasons for leaving a wonderful company with a senior position to join another project. Let me tell you how the extra cost of quality made me lose my footing.

Companies naturally evolves from feature-oriented to quality-oriented

Software development is successful when the customer is satisfied and the company has sufficient revenue. There is a break in the life of the project: it's when there are enough customers to provide support services. In this case, the company chooses to focus on the quality of the most lucrative functions, and to redistribute expenses towards building customer loyalty. Congratulations, you're part of a sustainable company!

Here is an exhaustive list of practices to focus on quality:

The basics: hire skilled employees, use ticketing & versionning flow, follow standard practices of your choosen tools.
Contribute for continuous improvment (plan, try, measure, standardize).
Add or switch for some better tools (read benchmarks).
Add automated tests (unit tests, end-to-end tests, non-regression testing).
Add automated workers (linters, code-analysers, automated code-reviewers).
Delegate checks to machines (CI/CD).
Forcing more manual tests or hire manual-testers.
Dedicated teams (release, cybersecurity, compliance), audits, formations.
Run AI in your project.

There's a big difference in quality rules between a business-to-business software with 150 users and thousands of functions, and a retail application for 100k simultaneous customers and only one purchase function.

Your company will add more quality rules, sometimes after a critical problem, sometimes in anticipation of problems. The more rules you add, the more expensive quality becomes. You can then calculate the cost to obtain quality (COQ):

Cost of bad code =
  Human time for resolving issues + Bad reputation

Cost of good code =
  Human time for quality + Cost of dedicated machines

Cost to obtain quality =
  Cost of bad code + Cost of good code

Why the hell any company would choose bad-quality?

They don't! No company would allow crappy code with security issues. No, they wouldn't. But you have to recognize that quality is clearly not the priority when you have no customer, no budget and the concept is still evolving.

You then need to hire people who are quick to develop a saleable product, you need to focus on highly anticipated features that will set you apart from the competition or improve your public image.

Force developer have good practices

A company that is making progress and maintaining good relations with its customers will evolve its profile to match those customers. The company will also evolve the profile of its developers in this direction, by imposing rules that are of interest. Developers take on board the new rules, change their habits and acquire reflexes for their working day.

Customers expectations -> developers new reflexes

But don't lock the development

However, companies can become over-costly on quality and forget to develop new products. As a result, marketing-commercial-product plans dilate and new products are delayed.

A company with quality overruns can be identified by a decrease in technical work time, a lengthening plan, rules that prevent novelty, difficulty with change, long processes, interminable meetings with little added value.

A good practice is following continuous improvement.

To get started, define your actual status:

by listing what matter for current customers,
by listing opportunities,
by measuring time to market,
by measuring quality.

The next steps are to plan numerous improvements, select and implement a few at a time, then check that they're working for you. An improvement could be adding good-practices or reducing processus.

Example of quality surcharge on web-development

It's time to share my personal experience with you. Several companies have followed this path and ended up, in spite of myself, being unable to develop new functions in their applications, as a result of an accumulation of rules, which often took them out of Agile good practice, in order to have over-control, or to protect themselves from a non-existent risk, or finally because the company was no more expecting new features.

The company I work for started with a CI/CD and unit tests on main features. Now we changed for a custom CI/CD with AWS, running unit tests for 75% coverage, running non-regression tests for 40% user journeys, with 20% employees dedicated for QA, 1/3 week for release production with 90% employees of the company (product, quality, developers, infra). The cost of quality is actually 80% of the budget (employees and AWS CI/CD) and there is no more time for new feature here. Actually, the customers are satisfied by product, but are awaiting for some expected features, and quality issues are from product (no more technical issues).

It's time to change company mind and find a better fit for customer expectations.

Conclusion

In order to control the cost of obtaining quality, we need to ensure customer loyalty and develop our customer base, and adapt quality, costs and time to market. It's not uncommon to fall into the trap of over-quality. In IT, however, you have to keep inventing new things to ensure the longevity of your business, and controlled risk-taking remains the best strategy.