Forem: Otto Kekalainen

Heartbleed and XZ Backdoor Learnings: Open Source Infrastructure Can Be Improved Efficiently With Moderate Funding

Otto Kekalainen — Sun, 07 Apr 2024 16:47:51 +0000

The XZ Utils backdoor, discovered last week, and the Heartbleed security vulnerability ten years ago, share the same ultimate root cause. Both of them, and in fact all critical infrastructure open source projects, should be fixed with the same solution: ensure baseline funding for proper open source maintenance.

Open source software is the foundation of much of our digital infrastructure. From web servers to encryption libraries to operating systems, open source code powers systems that millions rely on daily. The open source model has proven tremendously successful at producing innovative, reliable, and widely used software.

However, the Heartbleed vulnerability in OpenSSL and the recent backdoor discovered in the XZ Utils compression library have highlighted potential weaknesses in how open source software is funded, developed, and maintained. These incidents showed that even very widely used open source projects can have serious, undiscovered bugs due to lack of resources.

Learnings from Heartbleed

Today, April 7th, 2024, marks the 10-year anniversary since CVE-2014-0160 was published. This security vulnerability known as "Heartbleed" was a flaw in the OpenSSL cryptography software, the most popular option to implement Transport Layer Security (TLS). In more layman's terms, if you type https:// in your browser address bar, chances are high that you are interacting with OpenSSL.

The fallout from Heartbleed was immense, prompting widespread panic among developers, businesses, and users alike. About one-fifth of all web servers in the world at the time were believed to be vulnerable to the attack, allowing theft of the servers' private keys and users' session cookies and passwords.

The software bug existed in OpenSSL's codebase for over two years before being discovered. While code reviews were in place, the bug wasn't spotted and went into OpenSSL's source code repository on New Year's Eve, December 31st, 2011. At the time, the OpenSSL project was maintained by a small 4-person team with limited funding and basically working as volunteers, driven by just the importance of their mission.

This was the ultimate root cause – a piece of software that had started as a hobby project (just like Linux) grew over time and became part of the Internet infrastructure, but there was no mechanism to ensure resources would grow to be able to maintain it well long-term.

In April 2014, the Linux Foundation Executive Director Jim Zemlin seized the opportunity to get visibility and managed to get Amazon Web Services, Cisco, Dell, Facebook, Fujitsu, Google, IBM, Intel, Microsoft, NetApp, Qualcomm, Rackspace, and VMware to all pledge to commit at least $100,000 a year for at least three years to the Core Infrastructure Initiative. The initiative continued for many years and eventually transformed into the Open Source Security Foundation. Also due to Heartbleed, the European Commission launched the EU-Free and Open Source Software Auditing project and spent at least a million euros on auditing OpenSSL, the Apache Server, KeePass, and other security-critical open source software.

This relatively modest funding, along with code audits and process improvements, allowed OpenSSL to become more secure and sustainable. Today the OpenSSL project is thriving: it is FIPS 140-2 certified and has a healthy base of both financial and code contributors.

Learnings from the XZ / liblzma library backdoor

While there are surely still more details to uncover in the coming weeks, when the news broke about the XZ compression software backdoor (CVE-2024-3094), it was immediately clear that it happened because XZ had become hugely popular and widely used but was still maintained by one single overworked person as a spare time project. A well-resourced malicious actor was able to manipulate and pressure the maintainer to give them commit access, and thus the software supply chain was compromised. We should not blame the original maintainer, but rather everyone else for not realizing how widely used XZ was, yet going by with very little support and resources.

A huge number of applications depend on XZ. Right now the priority should be to offer help to maintain it properly, both upstream and at various downstreams, such as in Linux distributions, so the whole software supply chain is secured. It does not require a massive effort – just having a couple more maintainers to share the maintenance and review work should go a long way.

Would we be better off with closed-source software?

In both cases, the vulnerabilities were fixed quickly because the world had access to the source code of the affected software. This is a major advantage of open source software: it allows anyone to inspect the code and find potential vulnerabilities, and submit fixes to them.

In the case of Heartbleed, Google's security team reported it to OpenSSL first, but the Finnish national NCSC-FI has records of local cybersecurity company Codenomicon reporting it independently. In the case of XZ, a Microsoft employee and PostgreSQL developer Andres Freund found the backdoor while doing performance regression testing in a Debian Linux development version. It was a huge fluke of luck that the XZ backdoor didn't go in any actual Linux distribution releases. Next time we might not be as lucky, so more reviews, testing, and validation are needed. It will need resources, but at least public review is possible – thanks to this infrastructure-level software being open source.

Public scrutiny, testing, and validation are not possible for closed-source software. In fact, if closed-source code gets backdoored, it will go unnoticed for a much longer time. For example, the 2020 U.S. government data breach was possible due to multiple backdoors and flaws that went undetected for a long time in closed-source software from SolarWinds, Microsoft, VMware, and Zerologon. In theory, companies always have money (unless they are bankrupt), but in practice, the pressure to channel that money into software review and testing varies wildly, and working without exposure to public scrutiny often incentivizes companies to skimp on security to maximize profits.

Thus, I firmly believe in open source software having a better overall security posture as long as there are reasonable resources. And if the source code is public, anybody can audit how active the maintenance is and thus also the fact if maintenance is funded itself is a public and auditable property of open source.

Pledge for funding and participation

Both Heartbleed and the XZ backdoor incident underscore the critical role that open source software plays in powering the digital infrastructure of today's world. Such important and widely used projects shouldn't be struggling to get by. It's time for companies to step up and provide reasonable funding to the projects they depend on.

You don't need billions to meaningfully improve open source security – the OpenSSL example shows that even modest funding increases can have an outsized impact. A tiny slice of the corporate IT budget pie could go a long way. Additionally, some of the government defense spending should be funneled into key open source software projects that our society relies on.

The incidents of Heartbleed and the XZ backdoor serve as sobering reminders of the vulnerabilities that may exist within our open source infrastructure today. However, they also present an opportunity for positive change. By investing in the security and maintenance of open source projects through moderate funding and support, we can enhance the resilience of our digital infrastructure and ensure a safer and more secure internet for all.

See original post at optimizedbyotto.com/post/what-heartbleed-xz-utils-had-in-common/

Communication Is the Key to Efficiency in a Software Engineering Organization

Otto Kekalainen — Sun, 31 Mar 2024 19:00:48 +0000

For a software engineering organization to be efficient, it is key that everyone is an efficient communicator. Everybody needs to be calibrated in what to communicate, to whom and how to ensure information spreads properly in the organization. Having smart people with a lot of knowledge results in progress only if information flows well in the veins of the organization.

This does not mean that everyone needs to communicate everything – on the contrary, it is also the responsibility of every individual to make sure there is the right amount of communication, not too much and not too little. From an individual point of view, it is also important to be a good communicator, both verbally and in writing, as that defines to a large degree how professionally others will perceive you.

Reflecting on the principles below may help both your organization and you personally to become a more efficient communicator.

Communicate Early and Exactly

Foster a culture where people share small updates early. When you introduce a change, describe it immediately. Don't accept "I will document it later" from yourself or others. People are interested in the change when they learn about it, and should be able to immediately read up on git commit messages, ticket communications, etc. When you make a change that affects the workflow of others, announce it immediately. Don't wait until other people run into problems and start asking questions.

When you announce a change, be exact. If the change has a commit id or a URL, or if there is a document that describes it in detail, reference it. Avoid abbreviations and spell out the names of things to avoid misunderstandings. Use the same name consistently when referencing the same thing. Don't be vague if being exact requires just a couple seconds more of effort. If you know something, spell it out and don't force other people to guess. In a larger organization, it might even make sense to have a written vocabulary to ensure that people understand the daily jargon and assign the same meaning to the words used.

Keep in mind that you, the announcer, are one person, but your audience consists of many people: if you take additional time and effort to be precise, you may save a great deal of repeated effort by many others to determine precisely what you were referring to. When you see other people putting effort into making easy-to-understand, brief, and crisp communication, thank them for it.

Use the Right Channels

Always keep communication in context. If a piece of code does something that needs an explanation, do it in the inline comments rather than in an external document. For example, if the user of a software application needs guidance, don't offer it on a completely separate system that is hard to discover, but instead offer it via the user interface of the software itself, or in a man page or a README that a user is likely to come across when trying to use the software. If there is a bug that needs to be debugged and fixed, discuss it in the issue tracker about the bug itself, not in a separate communication channel elsewhere. If there is code review open on GitHub or GitLab, don't discuss it on Slack or equivalent, but do it in the code review comments – as it was designed for. Always communicate about something as closely as possible to the subject of the message.

Prefer asynchronous channels over synchronous channels. Chat systems like Slack or Zulip are better than a phone call. A well-written email is better than scheduling a 30-minute meeting. In an async channel, people can process incoming communication and respond in their own time. Sometimes a response requires some research and is not possible in real-time anyway. Having to juggle schedules can also be a wasteful use of time compared to working through a queue of tasks. Interrupting software engineers is very costly, as it can take tens of minutes before one gets back into "flow" and back to cracking a hard engineering problem. Also, as many teams today work across many time zones, you might need to wait until the next day for the reply anyway.

When using Slack and similar chat software, try to pick the most appropriate channel. Avoid private one-on-one discussions unless the matter is personal or confidential. The discussion in a public channel is more inclusive and allows others to join spontaneously or to be pinged in the discussion to join it. In Slack and similar chat systems that have threads, use them correctly to make it easier for participants to follow up on the topic while at the same time keeping the main channel cleaner. Avoid using @here on channels with more than 20 participants. Get into the habit of using Shift+Enter to write multiple paragraphs in one message instead of multiple short ones in succession which might cause unnecessary notifications.

In chat systems, do not send people messages that only say "Hello" . Get straight to the point.

Have a chat when it is appropriate. If you feel there is miscommunication and you can't resolve it async with well-written and thoughtful messages, a short chat 1:1 or with a small group of people can bring a lot of clarity. An in-person meeting or video chat usually works best, as both parties can read each other's cues to see that they understand and can follow the topic.

Teams Exist to Channel the Flow of Information in the Veins of an Organization

Team members interact mostly with others inside their team. It is not the responsibility of individual team members to know what people in other teams are doing. If something noteworthy is happening or is planned to happen, it is the responsibility of the team lead to communicate that upwards and laterally along the organizational lines. The team lead is also responsible for the inward flow of information and making team members aware of things that are relevant to the team.

The reason teams exist is to limit the flow of information. Most organizations are divided into 5–15 person teams simply because if teams were very large with 20 or more people, the overhead of everybody communicating with everybody would eat up too much time.

With this in mind, please be considerate and try to avoid approaching individual engineers in other teams too often. Channel communication through managers and architects, who are responsible for gatekeeping and prioritizing things. In a large organization, if you notice that people are reaching out to you personally all the time, just politely refer those requests to your manager.

In particular, when doing cross-org communication for large groups of people, think about the signal vs noise ratio. Free flow of information may sound like a noble principle, but a lot of information does not necessarily convert into real knowledge sharing. Write and share summaries instead of raw information. Be deliberate in selecting who should know what and when.

Make Meetings Intentionally Efficient

Principles for good meetings:

If you organize a meeting, make sure it has an agenda in the meeting invite so attendees know what the meeting is about, and have a chance to prepare for the meeting in advance. The agenda also allows people to make a better-informed decision if they can skip the meeting or not
If the meeting makes decisions, those should be written down somewhere (e.g. design doc, issue, ticket, meeting minutes). People tend to forget, so there must be some way to recall what the meeting decided.
Don’t invite too many people. If there are more than 5 attendees in a 30-minute meeting, there will be no genuine discussions as there would be less than 5 minutes of speaking time per participant.
Don’t attend all possible meetings. If dozens of people are invited to the meeting and it seems like an announcement event rather than a discussion, maybe just skip it and read the announcement documents instead.

Practice Efficient Status/Progress Communication

The purpose of progress information is to allow others to learn the state of an issue and allow them to adapt their own work in relation to that issue. Issue status information also helps the author themself to remember where and in what state they left something, essentially being communication to their future self.

Good principles to follow:

Avoid duplication. If an issue tracking system is in use in an organization, and an issue tracker entry has been filed, maintain it and do not disperse the information out in multiple places. Focus your energy on making sure the issue tracker is up-to-date so followers of that issue don't need to ask about status or search for separate updates in email or old chat messages.
Keep status information current. There is no point in a status-tracking system if the statuses are out-of-date. On the other hand, there is no need to update the status daily. A good rule of thumb is to update important status information immediately when it happens and less important statuses perhaps bi-weekly or monthly, depending on what the normal cadence of reviewing and prioritizing work is.
Annotate status changes. If an issue was closed but there is no comment whatsoever on why it was closed, it will raise more questions than it answers. When updating the status of issues and in particular when closing them, add a comment on what changed and why the status changed. Use to your convenience the feature present in most issue tracking software (e.g. GitLab and GitHub) that they automatically close issues when a commit with a closing note lands on the mainline git branch, and those status updates are automatically annotated with a link to the change, including date and author.
No news is bad news. In the context of status information and progress communication, people tend to view a lack of communication as a sign of a lack of progress. If something is blocked and there is no progress, a quick message noting "no progress" is better than silence and letting people stare at issues with no updates. Eventually people will start to worry and will reach out for updates, so skipping status updates to save effort might not actually save any effort.
Remember the purpose. At the end of the day, progress is more important than communication. If a task is small and you work on it alone, issues/status reporting may be omitted completely. If you find yourself spending more effort on communication about an issue than working on the issue itself, something is wrong with the overall process and you should review it.

Give and Get Feedback

Be honest. Engineering is about building stuff that works, and a lot of effort goes into making sure stuff actually works. That requires a constant loop of testing and feedback. If you spot something that is off, report it. Don't waste time on sugar-coating when communicating from a professional engineer to another, but just report the problem you've spotted and do it exactly.

Be grateful for all the feedback you get. Thank the person for taking the time to give feedback. The more you get, the better. Never scold another engineer for giving you feedback you don't like. Stay professional and just read/listen and try to understand it.

Not all feedback is valid, and as a professional you choose what feedback you act on. If you don't understand the feedback, ask for clarification. Engineering is, and should be, full of debate about what is the wrong or right solution, so that the chances of landing on the actually right solution are maximized. Be professional and don't take those discussions emotionally. Engage in them, and make sure all data points are analyzed. Intentionally challenge your own bias and preconceptions, and try to reach the best conclusion you can.

Less Is More

Delete stuff that is outdated or obsolete. Remove weasel words and duplicate texts. If you come across some documentation that is clearly outdated but might be needed for archival purposes, then add a banner to it stating that it is no longer up-to-date and kept only for archival purposes (in particular in a wiki where anybody can contribute to maintaining the contents). False information may cause more harm than no information.

Avoid link rot. If a document is moved from one place to another, delete the old version and replace it with a link to the new one.

Always shorten and simplify code when you can do so without sacrificing other desirable qualities: correctness, readability, maintainability, and consistency with the rest of the codebase. For example, if you are writing or maintaining 10 unit tests which are 20 lines of code each, but differ only in a couple of inputs and outputs, then combine them into a single parameterized test.

Layering and abstractions are valuable techniques for writing reusable and correct code. However, too much abstraction can make code difficult to understand and reason about. An integrated development environment (IDE) can be a useful tool for quickly navigating a code base. However, if you have to use an IDE to follow the code's logic, it is a telltale sign that the code is way too abstracted.

A Good Coder Is Also Good at Writing in Human Languages

The primary goal for code is to be easy to understand and follow. Optimize for readability and maintainability. Do not optimize for speed or build layers of abstractions upfront. Instead, do such things only later on if and when you really need to.

Principles to make code easy to understand:

Start with good naming: Files, functions, variables should all be named in such a way that one can guess from the name what they do or contain. Don’t be afraid of changing a name if you realize that something else would describe it much better. Functionality and contents evolve – so should the naming.
Use the project's coding conventions. A suboptimal but consistent convention is better than mixing multiple conventions in one code base. Use correct indentation, line length, white space, etc. to enhance the readability of your code. Keep the flow easy to follow.
Add inline comments in places that require some additional explanation of what the code does or why the code was written in a particular way. Good inline comments prevent the code from being deleted or refactored by somebody else, or by yourself a year later when you can no longer recall the details.
Longer or higher-level documentation should go into README files. The convention of using README files in code repositories is a great application of coupling code and documentation. A README is easy to discover in a code repository, and very likely to get updated by the same commits that update the code itself. If a code repository completely lacks a README file, the code is most likely not intended to be long-lived and should be avoided.

Software Engineers Need to Excel at Making Git Commits

Last but not least - remember that good software engineers write both great code as well as brilliant git commit messages. The more senior the engineer, the more they value both the code and the description of it, and the whole feedback cycle that eventually leads to making the world a better place – or at least one piece of software better. Practice this skill even if it requires a bit of extra effort initially.

If You Are Not a Native Speaker, Invest in Improving Your English Skills

Most of the world's population are not native English speakers – including myself. Yet, as English is the lingua franca of the software profession world, we all need to put effort into becoming more fluent in English. The best way to become fluent is to simply force yourself to read, write, listen, and speak English, and to do it in a way where you intentionally try to improve your English. Personally, I, for example, watch YouTube videos in well-articulated English, such as the British comedy panel show QI, or listen to podcasts attentively, trying to pick up new expressions that help articulate ideas and opinions more accurately, and in general to expand my vocabulary.

High-Quality Communication Facilitates High-Quality Engineering

From an organizational point of view, it doesn't matter how many amazingly smart engineers you hire if there are not proper mechanisms in place to ensure that the right amount of relevant information flows between the experts. Efficient communication is vital also for growing junior engineers quickly. You don't want to have any engineers wasting time trying to solve problems that have already been solved. The whole organization will be vastly more productive if everyone is able to find information, easily and quickly at the time they need it.

Achieving it is not hard nor expensive – it just requires setting a couple of ground rules, reflecting on their meaning, and executing them consistently. Managers play a vital role in achieving a strong communication culture by leading with their example and showing that good communication is valued across the organization.

This was posted originally at optimizedbyotto.com/post/efficient-communication-software-engineering-org

8 Writing Tips for Software Professionals

Otto Kekalainen — Sun, 24 Mar 2024 16:47:47 +0000

Original post at optimizedbyotto.com/post/writing-tips-for-software-professionals

People usually associate advanced software engineering with gray-bearded experts with vast knowledge of how computers and things like compiler internals work. However, having technical knowledge is just the base requirement to work in the field. In my experience, the greatest minds in the field are not just experts in knowledge, but also extremely efficient communicators, particularly in writing.

Following these 8 principles can help you maximize your efficiency in written communication:

1. Less is more

In a workplace setting, the ability to summarize something in three sentences is far more valuable than the ability to write fancy-looking research papers. Forget school assignments with minimum lengths – in reality, you need to put in effort to specifically keep it short.

2. Start with the solution or the ask

Unless you are a professional novel writer building up an arc of drama, your readers are most likely not captivated enough to read all of your text fully. Therefore, you need to put forward your main suggestion or request as early in the text as possible. In ideal cases, the main message you want to convey is already in the title.

3. Show the facts, with examples

If you are an expert, people will value your opinions. But it is always much more compelling if they are delivered with supporting facts, numbers, timelines, and references. Ideally, there is a reliable source to refer to or an indicator or statistic to look at, but a couple of anecdotal case examples also work well as both evidence and as a concrete story to showcase cause and effect.

4. Always quantify

A number is always more expressive than an adjective. Instead of a vague "expensive", just write "500 USD/h" if the price is known. Don't state that something is "significantly faster" as it does not actually mean anything. Saying, for example, "travel time decreased to 5 hours (down 30% from 7 hours)" paints a much more accurate picture.

5. Include links and references

Instead of a verbal reference like "read the report for more," make a service to readers and include a direct URL they can simply click. When describing a system or a problem, include the documentation link or issue tracker identifier.

6. Explain why it matters

After stating facts, ask yourself "so what?". Cater to readers who are not fully familiar with the domain by being explicit on why something matters and what it means, in as concrete terms as possible.

7. Ask feedback from one person

Before sending out a text to a large group of recipients, ask one person to read it first. If your main message does not get across, iterate on your text until at least one person understands it in the way you intended. If the text has great significance, you might continue to ask for feedback from two or three more people, but remember that everyone has an opinion, and there is no guarantee that getting more opinions will converge on one opinion. Asking multiple people for opinions is not directly bad, but perhaps wasteful, as it quickly leads to diminishing returns.

8. Sleep on it

When it comes to your own text, the most important opinion is your own. A good way to figure out what you really want and value is to write a text, put it away, and then return to it one or more days later and ask yourself if you still really agree with it.

Sender is responsible for delivery

Last but not least, remember it is the responsibility of the broadcaster to make sure the message was received. Don't assume people received and saw your message, or that they read it, or that they understood what they read. You need to put in the effort into preparing your message and following up on how it was received.

Writing well is also a way to show respect for the reader's intellect and time. Think about it this way: If you send a message to a hundred people and expect them to spend 6 minutes each reading it, you are spending 600 minutes (10 hours) of organizational time. If you spend 15 minutes extra to polish your message so it can be read and understood in just 2 minutes, you save the organization almost a full workday (600 minutes vs. 15 + 200 minutes equals 385 minutes or 6 ½ hours less).

It does not matter how good your idea is if the text describing it is bad. If you practice writing well, people near and far will become more receptive to your ideas.

What are your tips?

Are you a seasoned professional who masters written communication? What are your tips? Please comment below!