Forem: Jeremy Katz

How Google manages open source

Jeremy Katz — Thu, 23 Jul 2020 14:36:56 +0000

Many people know that Google uses a single repository, the monorepo, to store all internal source code. The Google monorepo has been blogged about, talked about at conferences, and written up in Communications of the ACM.

Most of this has focused on how the monorepo impacts Google developer productivity and the ability to have software written by one team and used by many other teams. But I haven’t seen as much written about how it also impacts the way teams within Google consume and use open source.

The benefits of the Google monorepo

Just like internal code, third-party open source code is also imported into the monorepo under a /third_party prefix. There are a number of benefits to this approach:

Single version. Much like with internally developed libraries at Google, importing open source code into the monorepo ensures that the same version of a library is used in all applications rather than having a spaghetti of versions to understand and support across the many applications within Google.
Ease of updates. With a single version of the code in one place, updating an open source library either for normal maintenance or because of a critical security issue is much easier. You just have to update the project copy in /third_party and every application in the Google monorepo now gets built with that new version. You do, though, have to ensure that you haven’t broken the build of anything else in the monorepo.
Dependency clarity. By having a single location where every dependency is stored, Google engineers can easily see which things within the monorepo depend on a given open source library. Thus when doing an update for a security vulnerability, the developers who own the individual applications can easily be notified that they need to deploy new binaries with the fixed dependency.
Simplified licensing review. Licensing reviews can be done in a single location rather than requiring a new review any time an application wants to depend on a new-to-it library. As you can imagine, at Google scale, vast numbers of open source projects have already had their licenses reviewed and approved for use inside of Google.

It turns out that these same benefits that Google gets from a monorepo can also be valuable to most other engineering organizations using open source—even though not operating at Google scale. But most engineering organizations don’t have the human power or financial resources to ensure that they get them on their own.

After all, one of the main benefits of using open source to begin with is having access to a lot of common infrastructure components without having to write them from scratch yourself.

But most development teams still need to have a high degree of confidence that the software that they are using is being properly maintained. They need confirmation that it is licensed in a way that is acceptable to the organization, and they need to know that it is secure, or be notified when there are vulnerabilities.

At a basic level, most developers would love to have access to “known good” components like Google’s developers get when pulling from the monorepo, rather than the dependency roulette of bringing in new open source components without any sort of sanity check.

How to manage open source like Google

Every organization could benefit from managing open source like Google does. Fortunately, the Tidelift Subscription makes it easy for you to create customized catalogs of open source components that provide many of the benefits of Google's approach, without the need to maintain your own fork or invest in creating and maintaining your own monorepo.

With the Tidelift Subscription, you’ll be able to see the catalog of open source packages and releases you use across all of your applications. You can approve new packages as developers need them with workflow automation—developers request packages, and managers or architects review and approve.

You can disallow certain packages or package releases based on known security vulnerabilities or licensing concerns. Or you can centrally flag that a vulnerability that is largely theoretical in nature can be ignored not just once, but by every development team without requiring each one to painstakingly review the vulnerability and assess it on their own to pass some pre-deployment scanner test.

Partnering with open source maintainers

We even take it a step further by partnering with the maintainers of many open source packages to help ensure that they are well maintained, have clear licensing, and get timely security fixes as vulnerabilities are discovered. This is a win-win, because the more subscribers who use a project, the more its maintainers get paid, which means they have even more time and incentive to keep their projects well maintained and up to date.

As a Tidelift subscriber, you can set your own policies for how you would like to use open source projects within your organization—or you can just choose to accept our guidance entirely.

Customizing your catalogs

A catalog of managed open source within Tidelift can be consumed in lots of different ways.

Your developers can ensure that they are using appropriate packages and versions with our command line tool and request new ones as they discover a need.
You can add a check as a part of your continuous integration pipeline to ensure that nothing is built that uses components that haven’t been vetted.
You can plug into a central artifact manager (such as JFrog Artifactory) to only allow approved components to be downloaded.

Each option can be used individually or, for the most effective deployment, use all three!

If you are interested in learning more about best practices for managing open source dependencies, we can help. Talk to one of our experts or read more about the Tidelift approach to managed open source here.

Photo by Joseph Barrientos on Unsplash

Spring cleaning: 3 tips for getting your application development house in order

Jeremy Katz — Tue, 26 May 2020 18:20:31 +0000

Despite some indications to the contrary where I live in the northeast US, it is finally spring in the northern hemisphere—which many people traditionally view as the best time to take on do-it-yourself projects and improvements around the house that have been piling up over the winter.

Painting the front staircase. Finally organizing the garage. Tackling that garden overhaul.

When it comes to your job, spring can also be a good chance to invest some time and energy in improvements to the applications that your development team is building and how you work on them. There are always plenty of back-burnered items that you’ve been meaning to get around to.

Things like ensuring you have a process to respond to security problems. Understanding how to answer obscure licensing questions for the components you use. Making an inventory list of all of the open source components you use like your boss asked you for six months ago. Not to mention, of course, plenty of things which are specific to your own application architecture.

Here’s a starting point for your to-do list:

1. Inventory what you have

Make a list of all of the third-party SaaS services that are being used across your entire organization. Are you paying for software or seats that you’re not using? Do you have good security controls in place, for example through a single sign-on provider like Okta or OneLogin?
Make a list of your in-house applications. Where are they in their lifecycle? Who’s on point for maintaining them? Are they aligned with your broader IT strategy? For example, can they be migrated to the public cloud, or do you need to start thinking about re-architecture or retirement of these services for the next growth period ahead?
Make a list of your open source components. Now that you’ve inventoried your applications, dig into them to discover what third-party open source they are using.

2. Clean it up to eliminate inefficiency

Cut unneeded expenses. With your newfound visibility and an idea of your near-term priorities, cancel SaaS services which you aren’t using anymore and ensure access to these services is locked down and centrally administered.

Communicate the plan. Now that you’ve got a plan around your internal application development, come up with a communications plan and roll it out to the application developers on your team starting with the high level strategy. Be sure every application has a clear owner. They’ll be happy to hear about the big picture and understand how their work fits into it.
Centralize your open source management process. Now that you’ve got a list of your current open source components and a roadmap for where you want to go, finally tackle that task of creating a central repository of known-good open source components that will meet your standards today and in the future. To get the heavy lift off your to-do list, consider partnering with the independent open source maintainers behind the projects that you use via the Tidelift Subscription.

3. Planning the next steps

Plan for your internal apps. Create a database (or just a spreadsheet or wiki page!) of your custom-built applications with key attributes such as the primary owner and reference to your company’s security, licensing, and maintenance policies.

Plan your open source management policy. Establish a baseline policy for open source components that go into your applications. How will you handle security vulnerabilities? What open source licenses work for your organization? Do you want to favor actively maintained packages over found source code? (You can learn more about how to work through these questions with these resources from Tidelift.)

And there you go—done and dusted. By taking time to get your arms around the application development resources you’re already committed to and making a plan for the future, you will be in a better place to have your development team working on the important things—like building applications that matter to your customers.

Photo credit: JEShoots

New to working from home? Here’s how to make remote work work.

Jeremy Katz — Tue, 07 Apr 2020 13:10:13 +0000

Over the past few weeks, companies employing millions of workers have had to figure out how they can make remote work work. Organizations across the technology industry and beyond have moved to working from home as a temporary solution for keeping employees and communities healthy.

When we started Tidelift a few years ago, we knew that we wanted to build a distributed team. Remote work wasn’t foreign to our four founders—we all have backgrounds working on open source software, where the contributions come from people all around the world, and collaboration without co-location is key.

So we’ve been incredibly intentional about designing our culture around the remote work experience because we know the way we communicate and interact on a daily basis affects our happiness, productivity, and success as a business.

While we got many things right from the beginning, there are other areas where we have done poorly but learned and improved over time. Here are some of the key lessons I can share from our experience.

Create a consistent meeting experience

If one person is on video, everyone is on video from their own machines. We didn’t always work this way, but when we tried this, it was instantly a better experience for everyone. It’s critical, particularly for teams that aren’t used to working from home, that everyone is able to see everyone else’s face.

We even use this same technique in interviews—we have pairs of folks talking with candidates at the same time. If your team is even partially remote, this is the single biggest tip I can give you to make the experience better.

Design group social opportunities

One example of a virtual social activity is our daily “water cooler.” The basic idea is that we set aside 15 minutes where we all join a shared video call, with no agenda. We shoot the breeze, talk about the newest board games, what we watched on TV last night, what the dog is doing right now, introduce a new employee, whatever.

We have conversations about so many fun things and it helps us all be humans together. Different people’s interests are represented and the conversation meanders and even sometimes gets quiet. But that’s okay. We appreciate the chance to be a little social and break the remote work solitude for a few minutes, while seeing people we wouldn’t otherwise get a chance to see and talk with.

If your team is new to working from home, creating a chance to talk as a group about topics that have nothing to do with your business can help maintain a sense of normalcy during a tough time.

Engineer 1:1 social situations

Not everyone is extroverted and wants to jump into a conversation with dozens of people. So we use donut.ai to facilitate random pairings between two people so they can learn more about each other in a smaller setting. Some of our team members do this while walking to show coworkers cool parts of their neighborhood or home working space.

Standardize communication

Pick one preferred communication mechanism for the company. It’s important to have a primary place to go for information and communication, and we have chosen to use Slack as our primary tool. The ability to have low-latency discussion is an important distinction over email and gets us closer to the way people interact in person.

Slack also helps us “default to open,” where as many conversations as possible are happening on public channels where every employee can access them, as opposed to email, where only the people who are copied can follow along. Side benefit: this also makes it possible for new employees to catch up on discussions that happened before they arrived.

We use Zoom for video calls and have a standing expectation that if you’ve been talking in circles about something on Slack for more than 10-15 minutes, you start a video chat to increase the bandwidth and reduce the latency even more.

Along these lines, another good policy is to ensure people don’t have to read everything. One of the criticisms of using Slack heavily is “there is so much to read.” But for us, Slack is the virtual office. Conversations happen all around in a physical office that you may miss, so the same will happen in your virtual office.

Ensure decisions are made remote-first

Many of us have worked in organizations that supported remote work, yet all of the important decisions were made by people who were co-located in an office. We learned from this not to marginalize people who were working from home. Now it’s more important than ever, not least for employee morale, not to make decisions in smaller groups than are normal for your company under typical working conditions.

Focus on written communication

Written communication becomes the default when working from home, so you have to focus on doing it well—and prioritize the time required to communicate in a distributed team. This can come out in ways that aren’t obvious: writing design docs around features that will be built, descriptive text for designs to try to flesh out the details, or writing project plans to coordinate across people and teams.

Know that text is lossy and your teammates mean well. We all are occasionally short and in text, communication can seem even more abrupt. Remember that we’re all in this together and gently let someone know (in private) if you see them coming across more harshly than they certainly intended.

Those are some of the most important things we’ve learned along the way about making remote work successful. We hope it’s helpful to you as you think about how your company can approach designing a productive home work environment for the coming months.

All the above said, remember that work can be challenging right now for people. While we have been set up for remote work from the beginning, most of us have a degree of stress and anxiety about world events that will make some days harder than others.

Also, in addition to just the normal of working from home, we are now working from home with partners, kids, or roommates in the house as well, which adds other complications. And as a parent, there may be an aspect of trying to give kids a sense of normalcy and learning.

So be empathetic to your coworkers and don’t write any current challenges off as just the result of working from home. I think that my friend Mark said it well last week:

Mark Imbriaco @ Home

@markimbriaco

I've worked from home for almost 20 years before this job. I'm very comfortable working from home. And I'm struggling right now to maintain any sort of focus or feel like I'm getting much useful done.

But you know what? I'd be struggling exactly the same way in the office.

16:04 PM - 23 Mar 2020

24 184

Why coordinated security vulnerability disclosure policies are important

Jeremy Katz — Thu, 23 Jan 2020 16:20:10 +0000

We believe that working with maintainers to create coordinated security vulnerability policies is important. Why? Here’s one story to illustrate.

Last year, a new security vulnerability was found in the urllib3 library—a powerful HTTP client for Python. If you are using Python, then you’re probably using urllib3.

When one of the core developers of Python 3, Christian Heimes, discovered this security vulnerability, he followed the disclosure policy on the urllib3 GitHub page, which gave instructions on how to notify the maintainers via Tidelift. Tidelift works with all of our participating maintainers to set up coordinated security vulnerability disclosure policies for their projects, which helps avoid risky zero-day security vulnerability scenarios.

Tidelift then took the following measures:

We worked with MITRE to coordinate the allocation of a CVE for the vulnerability. CVEs provide an industry standard way to refer to a vulnerability across vendors.
Next, we collaborated with the urllib3 maintainers to implement a fix and have it tested by the original reporter.
We alerted our subscribers about the existence of this new vulnerability.
In addition to the information on the security vulnerability’s existence, we also gave subscribers information on which new releases would resolve the vulnerability in their codebases.
We linked the release notes for users to understand any other changes present in the urllib3 update.

This process—which historically has often taken months with many open source projects—all occurred within 1 day.

If the package hadn’t had a maintainer watching over it, a scenario like this might require that your team spend time forking the library, patching it yourselves, and crossing your fingers that an official patch would be released before you descend into dependency hell.

This is where Tidelift helps. Tidelift ensures that there are maintainers standing behind covered packages who have the financial incentives to fix problems quickly once they are discovered.

In the case of urllib3, all of this was handled before our customers even knew there was an issue. This same scenario has been repeated a number of times since we launched our security vulnerability disclosure process in December 2018.

"Tidelift has made the process of offering a comprehensive vulnerability disclosure process simple for the urllib3 team,” said co-maintainer of urllib3, Seth Larson. “This makes delivering secure code and responding quickly to vulnerabilities easy even for a small team."

It's the end of Python 2. Are we prepared?

Jeremy Katz — Mon, 28 Oct 2019 13:45:35 +0000

In just a few short months, Python 2 will officially reach the end of its supported life. 💀 This means that anyone building applications in Python will need to have moved to Python 3 if they want to keep getting updates including, importantly, fixes for any security vulnerabilities in the core of Python or in the standard library. How did we get here?

Python 3 was initially released on December 3, 2008 and included a variety of major compatibility-breaking changes. Overall, these changes are welcomed by Pythonistas and remove a lot of hacks and workarounds that had evolved over time. One of my favorites is that things like dict.items() no longer return a list, so you don’t have to use dict.iteritems() to get a lower memory and more performant way to iterate over dictionary items.

Others, while still welcome, are more challenging from a compatibility perspective as they bring along syntax changes to the core language. This meant that many of the Python libraries that we use for building applications weren’t ready for Python 3. Django, Flask, urllib3, etc... none were ready for the initial release of Python 3. But they now are and have been for quite a while. The efforts to support multiple Python versions have been great but can’t continue forever.

This isn’t the first time this kind of event has happened in the Python world, though. Way back in October 2000, Python 2 came out. This major release of Python had a number of incompatible changes that impacted developers, especially surrounding how one worked with strings and Unicode.

At that time I was working for Red Hat and maintaining Anaconda, the installer for Red Hat Linux. We had decided that migrating all of the Python usage within Red Hat to Python 2 was a priority. There were many less Python modules back then and a small group of us (employed by Red Hat!) were able to do the work to update the modules we shipped to support Python 2. We sent patches upstream, in some cases taking over upstream maintenance of the module, and were able to help move the world forward to Python 2.

But today is different. There are now over 200,000 Python libraries. It’s not practical for one company to help drive all of the changes in the ecosystem to support this new and incompatible release. And the vast majority of the Python packages out there are maintained by volunteers—people who are doing this in their spare time and as a labor of love.

This challenge of how to migrate successfully from Python 2 to Python 3 is exactly the sort of situation where having an incentive for maintainers to support a new version and work with the incompatibilities would be so much better. It’s a perfect example of why we need to pay the maintainers of the open source libraries that all of our applications depend upon. With strong financial incentives in place, the speed and comprehensiveness of our preparation for Python 3 could have been accelerated.

For users, major incompatible changes like those involved in the migration to Python 3 are an important part of keeping software vibrant, alive, and performant. But without being a psychic, it is simply impossible to understand how the world will change and evolve and require modifications to our software.

Let’s extend continuous integration to our open source dependencies

Jeremy Katz — Thu, 18 Jul 2019 14:47:09 +0000

Over the past 5-10 years, the software development world has fully embraced the idea of continuous integration.

It’s not a new idea at all—continuous integration can easily be tracked back twenty years to Kent Beck’s book Extreme Programming where it was one of the twelve practices to help a software development team develop higher quality software while also having a good quality of life.

The general idea behind continuous integration is that you want to set up systems where you are constantly testing changes immediately as they are introduced into the build. Daily builds, which had been the previous Best You Could Hope For, were no longer a fast enough feedback loop to allow for the development of quality software.

Prior to Kent Beck’s book, developers avoided doing this integration because of the conflicts and problems that were uncovered. Extreme Programming instead suggested that “if it hurts, do it more often.” This approach leads to an environment where you find problems faster and also get a lot better at identifying problems and being able to fix them quickly.

But with our move to the cloud and increased focus on developer productivity, it’s become a practice adopted not just by niche early adopters but instead a defacto part of software engineering best practices. By continuously integrating our code, we are forced to do things such as having our code in a version control system, automating our builds, having automated testing as part of our builds, and having everyone commit early and often. So if you’re in a company and have a change pending, you’re regularly pulling in the changes from the rest of your team as you do your development and ensuring that the code continues to work.

That’s well and good while we’re talking about the code being built inside your organization. But as we have covered previously, the application code being developed within your organization is only 20% of the application code. If you’re like most development teams, there are a wide swath of frameworks and dependencies that you’re using which are open source. And we treat these libraries as precious and unable to be upgraded. An upgrade of these components is treated with a fear that is far beyond that which we attribute to the rest of the code being built by our team.

Why is that? 🤔🤔🤔

Some of it just stems from history. When you purchase proprietary software, often you purchase a specific version of that software. Upgrading to a new version requires you to work with procurement to buy the new version. That friction and the relative lack of frequency of updates have meant that we just aren’t used to doing these sorts of upgrades on a regular basis for code that is not our own.

Another—and perhaps larger part—is just a matter of comfort. We don’t know when the changes happen to code we don’t control. We don’t have a good way to understand why there are changes. We don’t know the people who made the changes. So the default assumption is that they aren’t making changes that will benefit us.

It’s time for us to work differently

Let’s start upgrading our dependencies early and often. When an open source package we depend on is updated, we should update our application to use the new version. By bringing in the changes more often, the cost of integrating each change will be lower, just like the cost of integrating the code from other parts of your organization is lower.

And then, when there is a major security vulnerability, the cost will be far lower to your organization. First because the amount of change will be reduced and thus the amount of work to validate that there aren’t problems for you. But just as importantly, because as an organization, you will be more used to the process of integrating the updates.

I’d be remiss if I didn’t point out that Tidelift helps to make this process even easier. With a Tidelift Subscription, you will get information directly from the maintainers about what has changed in each new version of the packages that you depend on.

We also work with the maintainers to help provide recommendations about what version you should upgrade to given what you currently use. And in cases where there are incompatibilities between versions of different software that you depend on, we will work with the maintainers to resolve the incompatibility.

The result? Open source libraries you can depend on the same way you depend on commercial software.