Forem: We Got POP

Turning AWS Log Insights into metrics

Robert Rees — Tue, 13 Apr 2021 11:25:24 +0000

Leo Giordani one of POP's developers has written up a detailed blog on his recent work creating a Cloudwatch metrics dashboard. The dashboard helps visualise our application's use of Celery tasks and to provide alerts where problems in the system occur such as tasks backing up as demand spikes.

Leo's post pretty much steps through the entire process involved including setting up a DynamoDB table in Terraform and creating the Python Lambda to does the work of converting the queries into time-series metrics.

This is a new technique for us and we're looking at how we might use this idea of polling, transforming and writing CloudWatch information more in the future.

Getting Black and isort to agree on formatting

Robert Rees — Wed, 30 Dec 2020 10:46:08 +0000

We have been using Black for a while to standardise our code formatting and reduce unproductive time in code reviews discussing syntax.

When we introduced it we had disabled our isort formatting due to an issue but when it came time to reintroduce it I found that black and isort were engaged in a formatting war both trying to change the formatting of a few files whenever they were run.

isort now has some advice on compatibility that wasn't available at the time (and which as a result I haven't tried).

This is the configuration file I ended up using for our code.

Automated code formatting has been a massively time saver in terms of code review and consistency across the codebase. If you're not already doing it then it is definitely one of the better new year resolutions you could make.

Experimenting with team autonomy

Robert Rees — Tue, 24 Dec 2019 16:47:43 +0000

Once upon a time things were fairly simple, we were a company that specialised in providing a UK-specific comprehensive technology service that allowed casting agents to run their entire business online.

Back then our team was between three to five developers and our development process looked a lot like Scrum with two-week sprints with a planning session at the start of the sprint and a product-facilitated retrospective at the end of the sprint.

Fast-forward three years and we now have a team of around fifteen developers, four related but different business lines across two continents and a code base that is fast becoming bigger than anyone can keep in their head even at a conceptual level.

So how have we changed?

Giving teams autonomy and accountability

This year I wanted to very consciously make a change to the way we worked as a development organisation. Senior leaders were very involved in the management of projects, in my view too involved. The effort required to provide enough information "up" the chain to allow effective decisions to come "down" meant that project bureaucracy was starting to take up as much time as the actual delivery work. Management was in danger of becoming a friction instead of facilitating work.

New projects now started with a few key people being assigned who understood the problem and do an initial investigation. That work would then lead to a proposal about what the team was that was required to deliver a project in terms of skills and number of people.

For management the goal was then to get the right mix and number of people working on the project or to be brave and cancel the project because it was beyond our capabilities.

Giving up on potentially rewarding projects because of the ability to fully commit to them is a hard and useful constraint. I'd like to say we always made the hard choices but that wouldn't be true. Instead I'd say that when it came to hard choices we made far more hard choices in 2019 than we did the year before. Improving is important if we want to get to a more ideal situation.

Governing work, not leading it

Once the right team is in place the next challenge is answer "How should we work?". The answer to this question lies with the team. The right way of working for a three week project of opportunity is not the same as a six-month infrastructure migration where system stability is at risk.

I firmly believe that self-organisation is important to create an appropriate amount of process. The minute someone outside the project attempts to define the process you run the risk of under or over-specifying what needs to happen with risks on either side.

Instead of trying to define process we have been trying to define governance. At the start of the project there was an outcome, often defined by leadership; a plan for achieving the outcome, defined by the team pioneers and then there is the feedback from trying to execute the plan.

This feedback is really the heart of governance. Is our outcome realistic and achievable? Is our plan the most likely way to achieve that plan? What feedback to do we have that supports our outcome happening and the way our plan works? What feedback implies problems with these?

We have set a weekly or fortnightly cadence of governance meetings that review the work of the team and how successful the team's work has been at achieving the outcome we have been seeking.

The team knows the governance meetings will happen. Anyone is free to attend but generally the pioneers tend to be the people who spend most time in them as the people who have most context and originally proposed the plan. This governance is the key to accountability, the team knows they need to demonstrate that they are getting feedback and acting on it to improve their chances of success.

The governance meetings also allow the company leadership to provide feedback on things happening outside of the project that might help or hinder it. For example, early warning of a partnership that might provide new opportunities or a key sale that could be influenced by the work the team is doing.

It is also a chance for the team to talk about what they need to deal with negative feedback. Commonly this will be about skills the team needs or how sensitive timelines are. If a team is asking for help and the request is reasonable then governance is also where leadership is held to account. No team can be successful if they don't have what they need to succeed.

The team should also be in position to challenge the idea that the team should exist or the problem they are working is the right one to address. The possibility of an internal pivot to a new problem or opportunity should always exist instead of forcing the team to just follow orders.

What has stayed the same?

We still have a fortnightly retrospective, this is attended by all developers and topics can be from anything that anyone is doing. We use the cloud service Retrium to facilitate and record our retrospectives and anyone from the team can volunteer to facilitate the session in the room.

We also still have a daily standup at 10am with all the developers with optional attendance from product and design. People can listen remotely but post their update to Slack ahead of the standup so that we're only trying to get the audio to work one way. Resolving issues for people working away from the office is handled via Slack.

We've had to adjust the reporting format several times this year to make the standup happen at pace and in truth I'm not sure it will be able to survive an increase in the size of the team.

Has it worked?

Well from a selfish point of view I am working significantly less this year than I did last year. That means I've been able to take on a whole different kind of work and focus more on strategy.

General feedback from the teams has been good. Some people feel that there should be more structure and that there should be some common structures between projects. In the development team we definitely have some standard practices and technology that cuts across teams so it makes sense that some of the process should be standard as well.

The most common feedback is that people feel that they are free to get on with their work the way they think is best. However sometimes they feel uncertain about how to do that and they wonder if they are doing too much or too little work.

Addressing this feedback feels like an issue of mentoring and supporting people to me. Remembering to tell people when they are doing okay and giving them some space and time to talk about their doubts (although in general the best people to discuss this with are your team members, which brings up back to team culture).

In terms of delivery output we did much more and grew more in 2019 than any previous years. In just one example we went from a team not existing to it generating £500K of annually recurring revenue in the year. We've not done that before.

If we can keep that empowerment of people to respond to their immediate situation effectively while feeling supported by the people around them I think we'll have a great workplace.

What's next?

POP as a whole is still under the crucial Dunbar's number so in many ways our culture can continue to be face to face and with a reliance on immediate communication to resolve issues.

As our US business grows however we will not be able to rely on this and we will have to start to evolve a more sophisticate mixture of asynchronous communications and principles of action.

I'm honestly not sure what the best solution is but I think we will probably look at loosely collaborating units which themselves have strong internal relationships and communication.

We'll have to look at who does this well and what we can learn from them. If you think you have something to share on this, please do. It's a tough challenge that I think a lot about when I think about where we are going next.

Negotiating security requirements with clients

Robert Rees — Tue, 06 Aug 2019 16:37:53 +0000

Sometimes it feels that clients with an enterprise-level security checklist want their cake and too eat it too. The requirements can feel like an impossible mountain to climb.

There is often room for negotiation though. If answering no to a question then explain why. When making assessments, different factors may be weighted differently according to the service you are offering. Only the most difficult clients expect everything you offer to be exactly what they want initially.

If a security feature is not currently present but you are planning to implement it in the future, don't be afraid to share your plan. It can be reassuring to know that you even have a plan to change things rather than being unaware of the potential issue.

In some ways it is more important for you and your team to have a strong vision of what security story you want to tell and be delivering that rather than being reactive to different client requirements with no coherence to the changes you are making to your product.

I think it is also okay to make clear that the implementation of some security features is dependent on commercial agreements. If one client is very adamant that they need something then perhaps they need to pay to ensure that element is prioritised over other things that might matter to other clients.

Finally the truth is that security very rarely have the final word. If your product meets a unique need then often you can have an exception to given security requirements if you have an enthusiastic internal sponsor.

This can be particularly important if the parent corporate rules are set for a different kind of business to the one you are working in. For example a few organisations we've discussed with have wanted to apply the rules they use for their Active Directory users to freelancers or independent third-parties. Discussing the origin of the rule and giving some practical examples of the impact it would have on the work being down led to a more nuanced version of the policy.

Generally the best security comes from collaboration that is based on trust and openness. It should be a normal part of the process to discuss and review requirements.

Automate all the security!

Robert Rees — Thu, 11 Jul 2019 11:04:00 +0000

I recently gave a talk at FullStack 2019 about what we have been doing to automate security to help

My slides are available and the talk is available through the Skillsmatter site (login required).

Since the talk was relatively short I've also been posting bits of it here on Dev.To as an experiment. I'm hoping to also go into some of the details the talk had to omit in the sake of time.

POP's cloud-based security services

Robert Rees — Thu, 11 Jul 2019 06:32:23 +0000

As part of our general security automation toolkit we use a number of third-party cloud-based services that allow us to take advantage of expertise and specialisation in those partners.

The services we use are:

Probely
Ghost Inspector
Buildkite
Sentry

Probely

Probely is a security scanning service that offers a variety of different scans. Lightning scans check endpoints for basic issues and we run these against production and staging environments before the work day starts and just before it ends.

We use the OWASP extension to the more in-depth scan and we run that several times a week. This scan is more akin to a manual penetration test (and there is a human element to the process so it isn't fully automated) and when we are asked for the result of our last penetration test I now supply the most recent PDF export of the Probley report. So far this has kept auditors and researchers happy.

When we first ran Problely it discovered a problem in our AWS and application setup and we felt it was important to get a clean report so we resolved the issue and generally we try to resolve any issues it raises within two weeks of its discovery. Recently though we found that very few Problely customers have zero issues in their scans and clients have been equally suspicious. If I had known that then I might have kept in a low-priority issue.

Having corrected various HTTP proxy issues the most pernicious issues that Problely now finds for us our XSS style issues. This has changed some of the ways we use our features now.

Ghost Inspector

We started using Ghost Inspector as a way of improving our integration testing and removing some of the headless tests that happen in our test runner system.

It has been such a life saver in terms of catching regressions that it would terrifying not to have it or an equivalent system. It's rare now that we ship a regression all the way to production that we have to revert. The vast majority are all caught in the testing process.

Ghost Inspector allows you to capture test scenarios via a browser plugins but the resulting CSS selectors tend to be very specific and brittle. The system comes into its own if you actually program the steps yourself.

Test steps can be abstracted into modules and reshared. The system has a pretty good screenshot comparison tools. A complete history of runs and failures (again providing a great audit trail). Videos are available of what happened during the test run. It also provides a fake email system which is something we thought we were going to have to build ourselves.

Ghost Inspector has changed the way we've approached some of the ways we now develop the system. We have screens that are specifically designed to expose information to Ghost Inspector for verification. We've massively improved the structure of our DOM to be more machine readable.

We also have some classic hot points in our code like primary keys used as page parameters where it would be too hard to switch to opaque identifiers so instead we have regression tests that try to access different pages by direct access to confirm that the access control system is working as intended. This is the only way barring a big rewrite that we could guarantee this potential security hole is not in fact a problem.

We've also seen that a lot of our clients use manual testing process when checking our platform for vulnerabilities. Often they share their process when they do discover bugs and we've been able to turn those into Ghost Inspector tests. We've been able to build a suite of tests based on the different expertise of the various companies we've worked with. It is now like we have a little team of virtual security researchers on our side!

Buildkite

Buildkite is the glue that sticks all our automation together. While advertised as a continuous integration (CI) platform is actually a general kind of task automation system and is useful for any kind of scheduled or triggered task.

Buildkite does manage all our CI builds but it also handles deployments, refreshes of our pre-production environment and other tasks.

Within its pipelines it will deploy software and then use Ghost Inspector and verify the results of the deployment.

Before we switched to Buildkite we were using [Jenkins. This is a venerable tool but there were two major issues for us with trying to run our own CI service.

Firstly the truth is that running Jenkins effectively is hard. It relies on filesystem access, to get auditable controls you need to run plugins which need to be updated and maintained, the pipeline functionality is a late addition to the system rather than a core element.

Secondly after attending a few secops conferences it was clear that Jenkins was one of the top targets they look for when looking for bug bounties (the number one is unsurprisingly Wordpress).

This means we had a high-profile target that we didn't have confidence in our ability to secure. By not using Jenkins a number of drive-by hackers will just pass you by and move on to other organisations that are. We needed to make a change.

One of the key differences between Buildkite and CI systems like CircleCI is that you bring your own agents to Buildkite. We can run jobs in our own AWS accounts and assign our own security policies to the agents. This means we can choose to completely isolate one cluster of agents without affecting the more privileged permissions of clusters that do deployments to ECS that need a lot of permissions.

We do also use CircleCI if what we want is pure CI or lightweight automation and the contents of the repo is low risk.

Sentry

Sentry is primarily an exception reporting and aggregation service. However because it also records who in your team has looked at an exception it means it can be used to confirm that you make periodic reviews of the errors that occur on your system.

You should probably use an error aggregation service anyway but it is worth looking at how you can use them to also provide a way of delivering security objectives, for example explicitly ignoring common but irrelevant errors and looking for unexpected issues.

Why is it worthwhile automating security?

Robert Rees — Wed, 10 Jul 2019 07:50:25 +0000

I sometimes think you can divide clients into three groups:

Those who care about security
Those who want to tick the boxes on security
Those who don't know yet that they care about security

The good news is that creating an automated security infrastructure helps all three.

If you client cares about security they want proof that you are doing what you say you are doing and that you regularly review and monitor it. Automation provides all the reporting and proof without you having to do anything about it.

Those same reports allow you to tick the boxes with people. If you didn't have them then you'd have to go through a manual process to dig up the information and supply it adhoc as the review process requires.

Before we had some of the automation that we do now that's exactly what I had to do. It could be painful to find the right information, sometimes it would be out of date or would reveal a problem that needed to be resolved before we could respond because we weren't reviewing things frequently enough.

Often all of this would be happening under time pressure because the security review was linked to a sales opportunity we were trying to secure (generally the entertainment industry loves working to deadline and hates to do anything it doesn't have to). That made everything more stressful than it needs to be.

Once you have automation in place I've found that we've reduced some of our review processes with potential clients to a few days. We probably spend more time discussing cloud infrastructure than the security side of things.

Finally what about those clients who don't yet care about security. In my experience those clients are happy to take risks with what they are doing but when things go wrong it often turns out that they did have security expectations about the service you were providing but failed to articulate them because often they don't understand how technology works that well.

The good news is that automation helps them too as the cost to build automation is really all in the setup. Operationally we've been able to throw and umbrella over our clients and marginal additional cost. All you need is one client who really cares about security and you can make a step improvement for all of them.