Forem: Sam Texas

Data is like deep fryer refuse, not oil. People closest to the data can refine it.

Sam Texas — Wed, 07 Aug 2024 09:40:28 +0000

Data is Not the New Oil - More Like Deep Fryer Refuse

The Grim Reality

Let's burst that bubble: Your data is far from being the light, sweet crude you're dreaming of. Think less "Spindletop gushes riches" and more "last night's deep fryer leftovers." That goopy, lumpy mess of data you've got? It's filled with impurities and inconsistencies. Seriously, if data were oil, yours would make Exxon weep.

A Sober Reckoning

Now, let's take a hard look at where you stand. I’m betting only a handful in your organization, probably the tech-savvy folks buried in the trenches, can distinguish the gold from the garbage. Clarity often begins at the bottom of the org chart. Yet, leaders frequently miss the forest for the trees—they don’t know where to dig, how to engage, or even what questions to ask.

Our increasing awareness of our business mortality hits home. That tranquil period of Zero Interest Rate Policy (ZIRP) was a lullaby we now must wake from. It’s time to face reality.

No Mary Poppins for Data Cleanup

Tossing your data into the AI pot won't magically transform it into a treasure trove of insights. Best case, you'll get a machine that's slightly better than your average high school senior at writing grammatically correct sentences. Worst case, it's all smoke and mirrors with no lasting value.

Rolling Up Your Sleeves

Here’s the deal: the transformation begins with your own people. Engage with those individual contributors at the bottom of the org chart—the ones who are knee-deep in where the data is born. They'll help structure, clean, and contextualize your data, turning that lumpy fryer oil into high-grade crude.

And let's not sugarcoat it: it’s a refining process. Skipping this part is like trying to ace a college course without ever cracking open a book. The students who walk into the lecture prepared are the ones who truly grasp the material.

In short, there’s no snapping of fingers for pristine data sets. It’s a grind, a journey that starts from the ground up. But trust me, this groundwork sets the stage for any AI wizardry you may want to pull off down the line.

Startups and scale-ups are not supposed to fight fair

Sam Texas — Mon, 15 Jul 2024 14:13:33 +0000

We have playbooks and templates purpose built for situations and seasons you have not yet seen, but you will

The road from scrappy startup to enterprise player isn’t paved but filled with hurdles and unexpected surprises. A seasoned CTO comes armed with playbooks, templates, and built-in Spidey-sense that is tailor-made for situations you've never seen but inevitably will:

Testing hypotheses : Patterns and tactics for rapid prototyping isn’t just a buzzword; it’s survival.
Rapid iteration : Iterate until you can tattoo "Perfect is the enemy of good" on your forearm.
Maturing from proof-of-concept to production readiness : It's like graduating from a paper plane to a spacecraft.

These templates help you see around corners and over the horizon, covering scaling, security, and compliance—the "unknown unknowns" you’d rather not think about.

We have networks built up over years in multiple industries and across multiple disciplines

It’s not just financial capital that keeps your startup afloat; there’s also human and social capital. A Fractional CTOs network is stretches wide and deep across industry, government, regulatory, and even those murky in-between places.

A Fractional CTO's network is stretches wide and deep across industry, government, regulatory, and even those murky in-between places.

That social equity and trust is earned in drops and spent (or lost) in buckets. A seasoned operator knows how to allocate and deploy it wisely and effectively – to the benefit of the business and the team.

Don't sleep on this. You don't get far by money alone.

🐈 🐈 🐈 We have lived multiple lives and made pivots before landing in your Slack channel

You’re not just getting a tech leader; you might be getting a professional copywriter, marketer, behavioral psychologist, and even a part-time therapist. Seriously.

✍ Copywriting: Stuck on that landing page copy? Your Fractional CTO might just moonlight as a Mad Men alum.

🧠 Behavioral insights: Gleaning user behavior to refine product-market fit.

🚨 Crisis management: Parenting-level de-escalation skills—trust me, they come in handy.

It make sense when you think about it – there is a lot in that Mary Popping bag than just the one thing. Make sure you ask about it, there could be a lot more that is not mentioned on that CV/resume.

We Leading Like Missionaries, Executing Like Mercenaries

Hourly consultants come in all tactical and no soul, like Seal Team Six. But a Fractional CTO brings that “Moses with tactical gear” swagger. They’re in for the mission, not just the hourly rate.

Seal Team Six vibes : Task assassins who move the needle.

Moses archetype: Guiding your team through the wastelands of market uncertainty to the promised land of product-market fit.

Broader vision: Catching weekend calls to soothe your 2 AM product existential crises.

Bang for Your Buck

We’re no longer in a ZIRP market where every pitch deck filled with Blockchain buzzwords gets funded. A Fractional CTO provides value that far outstrips the cost.

Affordable expertise : Tap into a seasoned leader's wisdom for a fraction (pun intended) of the full-time cost.

Life-stage advantages : Many Fractional CTOs opt for this setup due to other life obligations—kids, aging parents, you name it.

Win-win : It’s a mutually beneficial deal, allowing startups to leverage high-caliber talent without breaking the bank.

5 Things I Will Never Do Again as a CTO or Senior manager

Sam Texas — Tue, 02 Jul 2024 14:16:31 +0000

Becoming a good manager is not only about learning what to do. It is also what not to do. These are some of my hardest-won lessons as I write this as a diary to myself.

I will never again Be the Sole Agent of Culture Creation or Change

As a senior manager or CTO, I've learned that driving company culture isn't my primary responsibility. While I can and should influence the engineering culture within my team, overall company culture should be shaped and driven by the CEO. In a previous role as a non-co-founding CTO, I made the mistake of trying to take on the mantle of culture agent for the entire company. This role belongs to the CEO, and if they don't take it up, it's not my responsibility to fill that gap. My focus should remain on fostering a positive and productive culture within the engineering department, aligning with the broader vision set by the leadership.

I will never again Start Greenfield Projects with Microservices

For greenfield projects, I’ve learned the hard way that starting with microservices can lead to unnecessary complexity and overhead. Instead, I will always begin with a templated monolith and iterate from there. This approach allows for faster development and easier debugging in the early stages. As the project grows and the need for scalability becomes more apparent, we can then consider breaking the monolith into microservices. This strategy ensures we don't prematurely optimize and can adapt more flexibly to the project’s evolving requirements.

I will never again seek out 10x engineers

I’ve come to believe that the concept of a "10x engineer" is more myth than reality—hallucinations of Silicon Valley fever dreams and Substack newsletters. Instead of chasing these unicorns, I prefer to take a "Moneyball" approach, akin to Brad Pitt's character in the movie, and focus on building a team of solid performers who can execute consistently and predictably. This approach ensures that the organization can effectively absorb their output, and customers can handle the changes and deliveries we make at a steady, reliable pace. It’s about building a balanced team where each member contributes to a cohesive, well-functioning unit rather than relying on the perceived exceptional output of a few individuals.

I will never again Dictate How Things Should Be Done

I’ve learned that as a CTO, my role is not to dictate how my team should accomplish tasks, but to clearly define and enforce the desired outcomes. My focus should be on setting clear goals and objectives, ensuring everyone understands what success looks like. This empowers the team to leverage their expertise and creativity to find the best path to achieve those outcomes. By emphasizing results over methods, I foster a more innovative and motivated team that is aligned with our overall vision and goals.

Additionally, I understand that my team might not always know how to achieve these outcomes. Therefore, I support them by providing learning and development opportunities. This could involve buying them books, enrolling them in courses, or offering other resources that help them build the skills needed to reach our goals. Supporting my team in this way ensures they have the knowledge and tools necessary to succeed, and it demonstrates my commitment to their professional growth.

I will never again Blur the Lines Between Professional and Personal Relationships

I’ve learned the importance of maintaining professional distance with my team members. I will never again become close friends or deeply engage with them on a personal level. While building rapport and understanding is crucial, it is equally important to preserve professional boundaries.

From a psychological perspective, blurring these lines can lead to various issues. Personal friendships within a professional context can create biases, favoritism, and conflict of interest, which can undermine team dynamics and decision-making. It can also make it challenging to provide objective feedback or make tough decisions that may impact those personal relationships.

Moreover, maintaining professional distance helps in preserving authority and respect. When the boundaries are clear, it becomes easier to lead effectively, as team members are less likely to question decisions based on personal feelings.

In essence, while it’s important to be empathetic and supportive as a leader, it’s equally crucial to keep the relationship professional to ensure fairness, objectivity, and the ability to lead without personal entanglements clouding judgment.

Docker, Django, Traefik, and IntercoolerJS is My Go-To Stack for Building a SaaS in 2021

Sam Texas — Sat, 30 Jan 2021 21:25:00 +0000

This article has been updated. It was previously dated for 2020. Very little has changed, but I call it out where needed.

I recently published some thoughts on Django being a great framework for applications. This post expands on that to include the other pieces of infrastructure from development to production environments.

I have used this stack (or ones that look a lot like it) to build small SaaS apps in 2018, 2019, 2020, and now in 2021.

My entire stack

Linux Server/VM Hosted anywhere (I like Azure, Digital Ocean, or Scaleway) (edit: 2021 – I'm in the process of moving everything over to a dedicated server at Hetzner.)
Docker. Just plain docker
Traefik for reverse proxy and TLS with LetsEncrypt
Postgresql running in docker
Django in a container
Intercoolerjs for easy and slick Ajax-like front-end work (Edit: 2021 - the creator of intercooler has released HTMX, the successor to IntercoolerJS.)
Sentry for catching production bugs (easy three-lines added to your config)
Bitbucket pipelines for CI/CD (Edit: 2021 - I no longer bother with CI / CD for personal projects. It it too much tooling for no benefit)
ZeroTier for VPN/ControlPlane

For smaller projects I run tests locally in docker containers and then push directly into production. I don't bother with full CI/CD because I don't need the complexity of it all. That said, I do like Bitbucket Pipelines.

Woah! that is a lot to unpack here. Let's visualize this another way.

Virtual Machine
- Docker
- Django
  - Volume mounted data disks
- Workers (Long-running django commands)
  - Volume mounted data disks
- Postgres
  - Volume-mounted data disks
- Traefik
- Zero-Tier
- SSH

Was this helpful? Let me know...

Hosting

Your stuff needs a home (yes, even in the "serverless" world. LOL). My personal preferences are Azure, Digital Ocean (affiliate link), or Scaleway. They each offer enough compute, networking options, storage, and basic services to build out proofs-of-concept or whatever you might need.

Another honorable mention here is Hetzner. They offer a good level of hardware, service and price.

The Virtual Machines

For those side projects and many enterprise applications, scale is not an issue. That means that I will not serve thousands of simultaneous users or handling terabytes of data. Therefore I can get by on the smaller offerings – usually under $20/month. Even Azure (the most expensive of the three) offer their burstable VMs. Generally I like to go with Scaleway's Developer line of servers.

_ Notice that Kubernetes is missing from my stack? When scale is not an issue then you don't need Kubernetes. _

Docker. Just plain Docker.

I do not rely on the OS vendor (Ubuntu) to make sure that I run the latest Docker on new VMs. Therefore I use the nice little curl|bash technique.

curl -s https://get.docker.com | sudo bash

This cure one-liner will get the best and most recent version for your machine running.

Traefik for Reverse Proxy

Traefik has been a God-send since I found it. Nginx is great, but it was not built for the Docker universe. Traefik has two killer features that have saved me hours upon hours:

Automatic TLS with LetsEncrypt. Literally set-it-and-forget-it. With the right API keys and DNS provider you can also do verification with DNS.
Automatic no-reload configuration using docker labels. When you spin up new services Traefik will pickup the changes automatically because it listens to all Docker-related events. This makes it incredibly convenient to add, remove, or merge services as needed without any hassle.

My only comment on Traefik is that there is a bit of a learning curve. You have to decide how you want to configure it (config file, command line options, yaml, or docker labels, or use a combination!)

Another note here: I have already published my production configuations for Traefik here.

Postgres for Database

Tried and True, PostgreSQL has never let me down. I usually attach one of these containers to a project that needs it without any difficulty. I simply spin up the container, bind the ports, and then bind the data volume to my host disk. Done and Done.

docker-compose.yml

version: '3.1'

services:

  db:
    container_name: postgres
    hostname: postgres
    image: postgres:11
    restart: always
    environment:
      POSTGRES_PASSWORD: secretsonly
    volumes:
      - ./data:/var/lib/postgresql/data
    ports:
      - 5432:5432
    networks:
      - web

networks:
    web:
        external: true

Dockerized Django for Web

Docker deploys nicely in a container, and I have been doing it for a few years now. The benefits of matching your development environment to your production one cannot be overstated, and I have Docker to thank for that.

Edit: 2021, I've got a reference Django project up that I use as my template for new projects: https://github.com/simplecto/django-reference-implementation

Django commands for Asynchronous tasks

Also, for asynchronous tasks I simply use custom Django commands which are a part of the standard framework. The pattern here is a simple while loop with a sleep() period. It polls the database for relevant actions and then does it's thing.

Edit: 2021 - This has paid off in a big way. I have been running a website screenshot project for over a year now with this pattern. It takes about 1500 website screenshots per day. All of it is scheduled and managed by a Django command.

That project lives here: https://github.com/simplecto/screenshots

IntercoolerJS, because who needs the complexity?

I have a lot to say about this, but that will have to go into a series of posts. The tldr; here that I use this lovely little javascript library along with jQuery (yeah, it is 2020, and I still use jQuery) to make parts of my applications feel like single-page-apps but not really.

IntercoolerJS keeps that old-school "Ajax" (remember that word) goodness and allows me to update the DOM with HTML from the backend. It is seamless, smooth, and really convenient for things like logins and small form updates.

I strongly suggest you check it out: Learn more about IntercoolerJS

Edit 2021: The creator has released HTMX, the successor to IntercoolerJS.

Sentry to catch production errors

I make mistakes. Lots of mistakes--but there is no need to show them to my users, right? Sentry gives me an easy and convenient way to capture production bugs as they happen. Some cool things about them:

Open source-ish (relicensed here), so you can host yourself if that is your thing.
Easy few lines added to your settings.py file in Django and that is it.
Tight integration to your Git repos and Issue tracking systems for full production defect traceability.

Another nice thing is that you can disable it for development.

Check out Sentry here.

Bitbucket Pipelines for CI/CD

Edit: 2021 - I no longer use CI/CD for deployment of my one-man projects. It is too much tooling and complexity. I simply run tests with PyCharm and ship directly to production from my development machine.

There are so many CI/CD offerings out there today, but I have been happy with Bitbucket's offering called Pipelines. They offer you a few hundred minutes free every month with the option to top-up as needed for a small fee. I have rarely had problems, and I really enjoy their YAML configuraiton / directive files.

Using the bitbucket-pipelines.yml file I have been able to do full end-to-end testing by spinning up multiple docker containers, loading the databases, and running hundreds of test in just a few minutes. This was key in speeding things up in our team and enabling 5+ pushes to production per day.

Checkout Bitbucket here.

ZeroTier for VPN/Control Plane

Finally, we come to the bit of tech that is largely optional but nice to have. Zerotier is a unique kind of network/VPN that I use to link all my personal machines. It works though firewalls (at home, in office) and offers an easy 1 minute setup.

Using ZeroTier in my last company we were able to remove the SSH Jump Servers which caused a headache in terms of key management and shared bandwidth on a single machine.

Zerotier work on Linux, Mac, Windows, Android, and iPhone, so you are pretty much covered.

The one downside to ZeroTier is that I don't entirely understand how it works. It is a lot like MacOS or iPhone in that it "just works" as expected and I rarely (never) have problems. That is a strength from a user experience perspective but a WTF? from a CTO perspective.

Conclusion

Hopefully this deeper-dive will prompt some interest and curiosity about Docker, Django, Traefik, and especially IntercoolerJS. It is simple, easy to work with, and allows you to grow out of it when the time comes.

Edit 2021 - As you can see not a lot has changed. I've been able to add more code to my public repositories that flesh out the ideas expressed here over a year ago.

Get More Secure Websites From Your Traefik Configuration using Mozilla's Observatory

Sam Texas — Tue, 24 Mar 2020 23:55:10 +0000

Thanks to /u/TrollW00t on Reddit for bringing Mozilla Observatory to my attention. In a previous article I talked about improved security ratings from Qualys SSL Labs, but I did not go far enough!

My rating for simplecto.com from Mozilla Observatory was an F!

The creators of Firefox certainly know a lot about browser security, so their Observatory app is a welcome tool. Here we will go step-by step and outline the changes to my config that took it from F to a B.

A ratings are possible but only with Content-Security-Policy, but that can wait for another post.

Mozilla Observatory's 11 tests

Below are the secure headers settings I use now in my deployments:

# Adding in secure headers
- traefik.http.middlewares.securedheaders.headers.forcestsheader=true
- traefik.http.middlewares.securedheaders.headers.sslRedirect=true
- traefik.http.middlewares.securedheaders.headers.STSPreload=true
- traefik.http.middlewares.securedheaders.headers.ContentTypeNosniff=true
- traefik.http.middlewares.securedheaders.headers.BrowserXssFilter=true
- traefik.http.middlewares.securedheaders.headers.STSIncludeSubdomains=true
- traefik.http.middlewares.securedheaders.headers.stsSeconds=63072000
- traefik.http.middlewares.securedheaders.headers.frameDeny=true
- traefik.http.middlewares.securedheaders.headers.browserXssFilter=true
- traefik.http.middlewares.securedheaders.headers.contentTypeNosniff=true

The settings above instruct the browser to make it harder for XSS attacks, SSL downgrade attacks, and prevent the iFrame-ing of content into other pages.

Selenium Can Make Hi-DPI, Retina Style Screenshots (Firefox and Chrome)

Sam Texas — Tue, 17 Mar 2020 20:36:15 +0000

With just a single line of code you can modify your Selenium browser to operate in (x.x) density mode. This means you can take higher-DPI screenshots as you might on retina screens.

Firefox Code Sample

my_dpi = 2.0

profile = webdriver.FirefoxProfile()
profile.set_preference("layout.css.devPixelsPerPx", str(my_dpi))

Yes kids, it is this simple. All you need to do is set a preference on your profile.

NOTE: The gotcha here is that you have to cast the float to a string using str(). That is non-obvious and will save you a ton of time in debugging. Otherwise it is silently ignored.

Chrome Code Sample

Chrome is a little different. There is not a preferences profile that you tweak with various settings. In this case you simple pass in a command line option when launching the browser.

my_dpi = 2.0
options = ChromeOptions()
    # ... other options..
    options.add_argument(f"--force-device-scale-factor={my_dpi}")

Full Code Sample

This is an export from my Jupyter Notebook. Note, you will need to pip install selenium as well as have the firefox-geckodriver installed with brew or whatever linux flavor.

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.common.exceptions import NoSuchElementException, WebDriverException
from time import sleep

my_dpi = 2.0

profile = webdriver.FirefoxProfile()
profile.set_preference("layout.css.devPixelsPerPx", str(my_dpi))

options = Options()
options.headless = True
driver = webdriver.Firefox(options=options, firefox_profile=profile)
driver.set_page_load_timeout(60)
driver.set_window_size(1440, 800)

driver.get('https://www.simplecto.com')

driver.save_screenshot("simplecto-dpi2.0.png")
driver.quit()

NOTE : This was not tested on Windows. Sorry not sorry.

You can try DPI at 1.0 and 2.0 to see the sizable difference in disk space and memory required. This becomes a little more work for a CPU when running full-size page screenshots.

Never Trust Free Services Without an Escape Plan

Sam Texas — Thu, 05 Mar 2020 12:47:11 +0000

I get it.

You are a startup and your are scraping by, hustling, and squeezing every drop out of every lemon you can find.

Or, you are just a single developer experimenting with all the free services out there.

You are trying to save money at every turn until:

you land that whale of a customer
you land that whale of an investor
you actually understand the parts of your business that make money and simply decide to put money into that

Either way, you are taking advantage of free services like Amazon's Free Tier (TM), Google's free tier, Sendgrid's Free Tier and so on.

Build with the escape plan in mind

Whenever, where ever, however you sign up for these services – always keep a plan B in mind. This plan B can be a few things:

#1 Do not count on anyone's good will

You eventually have to put your money where your services are hosted...so to speak. Are you ready for that moment, and what will the terms be? If you are lucky they they will grandfather you in. Which means, "hey you were early and helped us beta test this bad boy. Keep the free tier as a thank you." I don't see that happen too often.

Next, you have the option to move to a paid plan with a discount. This might suit you, it might not. But we are cheap bastards, and that $8/month is a real pinch.

So now that leaves you with the final option. Pay the price or GTFO (Get the f*ck out). Fair enough, freeloaders cost money, and everyone has to eat.

#2 Set aside some cash for when it time to pay the devil

You can simply start paying and keep your infrastructure/services in place without disruption. (This is the grown-up thing to do) It is also a good idea if you buy-in to propreitary serivices offered by the likes of cloud providers like AWS, Azure, and CGP. Cloud providers are all the devil, and this is the very deal they want you to make.

No evil here, but know the deal.

#3 Only procure services that are built on open source

Many companies run an open source business model which means that you can run their software in your own infrastrucure at zero-cost (no counting the cost of your time or hosting). This can be a win, especially in regulated environments like healthcare and banking.

It also frees you from having to scramble to find a comparable or compatible service to the existing features you get for free now. Sentry.io is a great example of this model. They offer a compelling paid product but also offer the same experience and software as a download. Should you need to move away from the hosted version it is very easy to host yourself.

#4 Only procure with open standards

This escape plan there also applies to service like SMTP (email sending). You can move from Mailgun to sendgrid to your own with some DNS and API key changes. No big deal.

This also applies to the likes of container hosting. Across the board the providers offer some kinds of Kubernetes or Docker support. Keeping your deployments and infrastructure within those environments goes a long way to ensure that you are not too locked in to that provider should the tables turn.

#5 Think ahead, have the fire-drill in place

Build and deploy absolutely knowing this is going to happen. When it does, then you already know what needs doing:

adjust the dns
swap the api keys
generate new certs
etc, etc, etc

And then execute! If you are lucky then they providers will give you plenty of time to plan and allocate resources to roll over. In case they don't then it does not hurt to have this punch list in place.

Conclusion

Don't be surprised when the free is no longer free. Be prepared either with cash in hand or a solid plan to pull up stakes and leave.

Let's Go Shopping on Github, or 5 Tips to Pick the Right Software

Sam Texas — Wed, 26 Feb 2020 08:35:00 +0000

There are literally thousands of open-source projects posted in GitHub. Depending on your use-cases, there may be tens if not more projects which deliver functionality to your project or company.

How, then, can you be sure to pick the right ones?

Check the License

Check the kind of open-source license (if any) this project is using. It might not be the best choice for you depending on your cases. Learn more about choosing projects with compatible licenses here:

https://tldrlegal.com/

This can become a complex topic depending on your Legal situation. That said, during a due diligence (pre-investment, acquisition, or vendor) you will face questions about what software you use and what licenses you are subject to).

Check compatibility with your stack

Make sure that the code is compatible with your current stack:

Supported OS
Language version support (Eg- python 2.7 vs 3.x, or Swift 2,3,4, PHP 5.x, 7.x, etc)
Will it work with your chosen database (if required).
Browser support
Hardware and system requirements (Eg- do you have enough cores, ram, and disk space?)

Note: this library may have it’s own dependencies, which triggers its of list of compatibility requirements. This is why it could be best to use a dependency manager like Pip, Composer, NPM, Maven, or whatever depending on your chosen language.

Another issue for Python users out there: Python2 vs Python3 – it is 2020 and there are still some dependencies stuck on v2. Further complicating things could be that some libraries will only work in Python3 or 3.6 or greater. Beware.

Check the health of the project

Is the project alive, on life-support, or in the graveyard?

This is probably the most important metric I use when picking. Please note, this is subjective. Over time you will develop a “feel” for this kind of thing and set your own limits.

Number of stars indicate interest from the broader community
Number of open issues indicate engaged developers and users
Number of closed issues indicate some kind of resolution processes are in place (eg — bugs are getting fixed and features delivered)
Check the documentation. Is it up to date with the current code base?
Date of the last public release
Is the project under stewardship (eg - Apache incubated) or is it simply sponsored by a company or single developer (aka BDFL)?

Caveat: This generally holds true for me, but it is quite possible that some projects are considered stable and will not need updates for long periods of time.

Does it trigger any new hardware or infrastructure dependencies?

It might require deploying new backing services such as storage, caching, or authentication services. This can significantly increase your complexity or costs of the project.

One such anecdote is that we wanted a thumbnail-generating library for a website project. It just so happens that managing thumnails at scale brings significant speed, memory, and caching challenges. The chosen library offered more functionality but at the cost of adding caching services. This was not budgetd for (mentally or financially), and the library was scrapped.

What are the security risks?

Check the CVE and NVS databases for any unresolved security issues. This will have major implications of you maintain risk assessments, compliance records, or security audits.

Use Traefik 2 with Nginx, Apache, or CaddyServer to Serve Static Files

Sam Texas — Mon, 24 Feb 2020 06:09:00 +0000

Setup a simple HTTP static file server behind Traefik. My specific use case is to serve static files (eg - PDF, excel, etc) from my blog that runs Ghost. However, the software itself does not support that. There are a number of HTTP servers, but we will choose three of the most popular ones.

These work well because they provide a nice drop-in to solve static file serving because it is small, fast, and will work well in a default configuration.

Now we will explore two ways to serve up files.

Hosting on a subdomain (eg - files.example.com)
Hosting on the blog's domain but in a subfolder (eg - example.com/static)

Use docker-compose and docker labels to configure Traefik's routing and middlewares.

In both scenarios:

Use the images from Docker Hub.
Bind-mount a local folder to host the files for the web server. See volumes: in the docker-compose.

On its own domain

You must have the DNS entry, files.example.com in this case, to point at the server. An example file download URL would be:

https://files.example.com/sales-pitch.pdf

All-in-one config (docker-compose)

This is a docker-compose.yml with configurations for all three servers. Simply comment out the ones you don't want and un-comment the ones you do. Bobs- your uncle.

    static:

        # nginx config
        image: nginx
        volumes:
            - ./files:/usr/share/nginx/html:ro

# uncomment to use Aapche HTTPD Server
# image: httpd:2.4-alpine
# volumes:
# - ./files:/usr/local/apache2/htdocs:ro

# Uncomment to use Caddeserver
# image: caddy/caddy:scratch
# command: file-server --root /files --listen 0.0.0.0:80
# volumes:
# - ./files:/files:ro

        container_name: static-files
        restart: unless-stopped
        networks:
            - traefik
        labels:
            # Match on the hostname and the path
            - traefik.http.routers.nginx.rule=Host(`files.example.com`)
            - traefik.http.routers.nginx.tls=true
            - traefik.http.routers.nginx.tls.certresolver=le
            - traefik.http.services.nginx.loadbalancer.server.port=80

            # tell Traefik which middlewares we want to use on this container
            - traefik.http.routers.nginx.middlewares=gzip

On a specific URI path (/static) in an existing domain

It is only a matter of three lines that will enable this to live under /static Path. An example file download URL would be:

https://blog.example.com/static/sales-pitch.pdf

Just the parts you need

# Match on the hostname and the path
- traefik.http.routers.nginx.rule=(Host(`blog.example.com`) && Path(`/static`))

# Define a new middleware to strip the URL prefix before sending it to nginx
- traefik.http.middlewares.nginx-stripprefix.stripprefix.prefixes=/static

# tell Traefik which middlewares we want to use on this container
- traefik.http.routers.nginx.middlewares=gzip@docker,nginx-stripprefix@docker

Full docker-compose.yml

    static:

        # nginx config
        image: nginx
        volumes:
            - ./files:/usr/share/nginx/html:ro

# uncomment to use Aapche HTTPD Server
# image: httpd:2.4-alpine
# volumes:
# - ./files:/usr/local/apache2/htdocs:ro

# Uncomment to use Caddeserver
# image: caddy/caddy:scratch
# command: file-server --root /files --listen 0.0.0.0:80
# volumes:
# - ./files:/files:ro

        container_name: static-files
        restart: unless-stopped
        networks:
            - traefik
        labels:
            # Match on the hostname and the path
            - traefik.http.routers.static-files.rule=(Host(`blog.example.com`) && Path(`/static`))
            - traefik.http.routers.static-files.tls=true
            - traefik.http.routers.static-files.tls.certresolver=le
            - traefik.http.services.static-files.loadbalancer.server.port=80

            # Define a new middleware to strip the URL prefix before sending it to static-files
            - traefik.http.middlewares.static-files-stripprefix.stripprefix.prefixes=/static

            # tell Traefik which middlewares we want to use on this container
            - traefik.http.routers.static-files.middlewares=gzip@docker,static-files-stripprefix@docker

Things to note

The CaddyServer config needed a command: attribute added to tell it to activate its fileserver feature.
The mountpoints for volumes: is different per server
Only official images are used. Your milage may vary if you stray from these...

Other guides

I previously made a guide on hosting your files using a nice Golang service called filebrowser. Read more about Ghost and Filebrowse here.

Conclusion

Using the above snippets you should be able to add Nginx, Apache HTTPD, or CaddyServer to your docker / Traefik deployments and have simple, no fuss static file serving.

I should also mention that is snipped is part of my production stack template that use for most Simple CTO projects. More on that later

3 Things to Do Before DDoS'ing Yourself with a CDN/Application Gateway

Sam Texas — Mon, 17 Feb 2020 12:25:48 +0000

Every now and then I get back into the Azure ecosystem to see what is there. I recently came across their Front Door product which looked like competitor to CloudFlare or a rehash of their Application Gateway. To be honest there are so many products that do so many things it all kind of blends together.

This is not a post about Front Door. It only plays a supporting role in this little story of how many DDoS events are often self-imposed.

Anyway, I thought the Front Door product looked interesting, so begin to setup a test against my little screenshot side project.

It takes a few minutes to deploy a Front Door, but once it was up I tried it out.

504 Gateway errors...

Uh-oh.

I head into the weblogs on Traefik and see that I'm getting hammered from a number of IPs in the same B-Class subnet (147.xx.00.00/16). It occurs to me that the Front Door edge servers are attempting to pull content from the site, but this little VM and application cannot keep up.

Reason 1 - No caching (Appliction architecture)

There is no caching in the application. Traefik does not bother with caching, so the issue is in my little Django app. There is no caching configured, nor does it calculate ETags or offer any kind of 304 responses. That means every page request is it hammering Postgres, fetching the data, and building the pages. This should not be a big problem, except that I'm storing images as bytes in the database (because reasons). That means pushing more data than expected from one service to django, assembling the page, and sending it back. Oops.

Mitigation:

Add in-memory page caching for home page.
Added 304 calculation based on most recently added screenshot. This is called conditional-view processing.

Reason 2 - No rate-limiting on the Reverse Proxy (Traefik)

A quick calculation showed that the Front Door was attempting to make about 25 requests per second in an attempt to acquire the content. I could configure Traefik to rate limit based on IP, subnet, and a few other factors as well. Handing back errors to the CDN is not ideal. However, rate-limiting in general is not a bad idea.

Mitigation :

Add rate-limiting for reasonable human usage of the site. (eg - about 15 reqs/second, including all loading of assets)

Reason 3 - The CDN was set to aggressively health check improper endpoint

As you can imagine getting the CDN configured is also tricky. I am first to admit that I don't know what I'm doing all the time. Upon looking into it I could see that the CDN was simply doing HEAD requests on the home page, which caused quite a load on my little server. The smarter thing to do here would have been to offer up an endpoint that simply return 200OK (or something that did not load my systems too heavily). This became too much when 40+ edge servers are requesting the same data every 30 seconds.

Mitigation :

Add a /health-check endpoint to only return 200OK on HEAD and GET requests. Configure Front Door to hit that instead
Decrease frequency of healthcheck from every 30 seconds to 300 seconds.

Conclusion

A CDN/App gateway by itself can do more harm than good. It is not a turn-key operation, and you should be prepared before kicking off a project like that. Consider making a risk/gap analysis and subsequent checklist before undertaking such a project. My checklist follows:

Make sure the app offers a light-weight health check endpoint
Make sure the CDN hits that health-check endpoint
Setup sane rate-limiting on your reverse proxy and provide helpful gateway errors for when people see it. (Reddit does a pretty good job here)
Measure the performance of your site before / after the implementation. Was it worth all the trouble? Could you have just bought a bigger server for the same benefit and fewer moving parts?

Improve Traefik's HTTPS Encryption with Qualys SSL Labs and testssl.sh

Sam Texas — Fri, 14 Feb 2020 12:18:13 +0000

tldr; Encryption (and HTTPS) is a complicated beast, but we have to do our best to make sure that our sites run securely. With the help of tools like Qualys SSL Labs [1] or the open-source testssl.sh [2] I update my production Traefik installations to run with the most secure configurations as possible.

Disclaimer : I am not an encryption expert and will be the first to admit that there is a bit of "cargo culting." I am making the changes necessary to my configurations such that the SSL labs gives an "A" grade or better, and that I see no RED flags from testssl.sh. I provide no advice here but rather share the configuration and outcomes.

Before we begin

Before you start disabling old encryption protocial it is a good idea to know who your users are and what software stacks they use to connect with you. Older stacks such as IE11 on older Windows or Safari on older Macs may break. See all the red? Yeah...

SSL Labs report (after changes)

My current config

Below you see the "B" rating for the SSL configuration for simplecto.com. This is in part because the current configuration allows for older, less secure versions of TLS.

Old rating for simplecto.com on SSL Labs

My current config that earned the "B" rating

This is my current config. There are no options set for encryption settings, cyphers, protocols, etc. I simply take the defaults from Traefik. This may or may not be sufficient for you to pass a security or compliance audit.

I use a single docker-compose.yml file to deploy which keeps complexity to a minimum.

version: '3'

services:

    traefik:
        container_name: traefik
        image: traefik:v2.1
        command:
            - "--providers.docker=true"
            - "--entryPoints.web.address=:80"
            - "--entryPoints.websecure.address=:443"
            - "--certificatesResolvers.le.acme.email=secret_email@domain.com"
            - "--certificatesResolvers.le.acme.storage=acme.json"
            - "--certificatesResolvers.le.acme.tlsChallenge=true"
            - "--certificatesResolvers.le.acme.httpChallenge=true"
            - "--certificatesResolvers.le.acme.httpChallenge.entryPoint=web"

        restart: always
        ports:
            - 80:80
            - 443:443
            - 8080:8080
        networks:
            - web
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock
            - ./acme.json:/acme.json
        labels:

            # Redirect all HTTP to HTTPS permanently
            - traefik.http.routers.http_catchall.rule=HostRegexp(`{any:.+}`)
            - traefik.http.routers.http_catchall.entrypoints=web
            - traefik.http.routers.http_catchall.middlewares=https_redirect
            - traefik.http.middlewares.https_redirect.redirectscheme.scheme=https
            - traefik.http.middlewares.https_redirect.redirectscheme.permanent=true

networks:
    web:
        external: true

What needs to change to get the "A"?

I went searching for answers and quickly found myself on the Containous community forums [3]. It was a post there that held the answers.

I must tell Traefik that I want TLS 1.3 (still experimental) and that the minimum version to support is TLSv1.2. Oh, and remember that pristine single-file config? Yeah, well, now I need to bind-mount some static files into the container. No biggie, but slightly more complex.

The new traefik_conf.yaml

This yaml file tells Traefik to:

Make the minimum supported version of TLS 1.2 and to enable 1.3.
Only accept connections where the hostname is specified (SNI)
Support a specific list of Cyphers (the cargo-cult part for me)

tls:
  options:
    default:
      minVersion: VersionTLS12
      sniStrict : true
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

    mintls13:
      minVersion: VersionTLS13

Now that we have this, we need to get it into the container and tell Traefik to use it.

There are two lines we need to add to our docker-compose.yml file:

"--providers.file.filename=/traefik_conf.yaml" under command:
./traefik_conf.yaml:/traefik_conf.yaml under volumes:

The two lines you need (in context of the docker-compose.yml)

Now all we need to do is docker-compose up -d and re-test!

Wohoo! – Ok, A-grade achieved.

Test locally with testssl.sh

As an alternative to SSL Labs there is a nice open-source tool called testssl.sh. You can run this locally and see way more information that you might know what to do with. You can test on any docker-enabled machine with:

docker run --rm -ti drwetter/testssl.sh example.com

There is quite a bit of output, but suffice to say you will get more than you are looking for :-)

Conclusion

With a small configuration change to Traefik we were able to move our SSL Labs overall rating from B to A. We had to add support for TLS1.3 and remove support for TLS 1.0 and 1.1.

Reference

Building my own Screenshot-as-a Service with Docker, Django, and Selenium

Sam Texas — Thu, 13 Feb 2020 08:47:00 +0000

tldr; There are no shortage of API or services for getting website screenshots. However, this is a good experiment to explore what it takes to make high-fidelity screenshots. Let me be clear--this is largely a "solved problem" as demonstrated by multiple companies in the space (see appendix). There are even some open-source projects solving pieces of it at a time.

This distraction satisfies a years-long curiosity I have had about making website screenshots in an automated way.

tldr part2; Show me the code!

Ok, you pushy bastards. The whole Django project is here:

https://github.com/simplecto/screenshots

Some thoughts and guiding principles of this project

Simplicity (hah, I know there are Yaks that need shaving), meaning no CI/CD, no Kubernetes, deploy to prod right from dev machine, and other "sins."
How does execution look when I am not at all concerned with scale of any kind?
Optimize only for developer productivity?
Slow your roll and document things (check that README.md, yo)
Challenge old truths like, "storing images in the database is bad" and "using a database as a queue is bad."
How far can we push cheap, commodity VMs at a place like Scaleway?
How far can this single server scale vertically before needing a 2nd server?

System Architecture

Below is the current system architecture used.

In case you missed it, most of my projects (personal and professional) follow a similar pattern of system and application architecture: linked here

Docker
Django for web
Django Commands for async tasks
Postgresql
Traefik for reverse proxy and HTTPS
All on a single Ubuntu Linux VM.

Scale is not an issue

I always preface this with "scale is not an issue" when starting new projects. You have no customers, no users--nothing! So, it makes sense that we should not distract ourselves with the complexity of distributed systems, remote storages, or other things.

Should we need them we can revisit this conversation.

Simple deployment

Everything you see lives on a single system all running in plain old Docker. I use a docker-compose.yml file to keep deployment simple.

Learn more about my setup for reverse proxy configuration with LetsEncrypt and Traefik.

If you can Docker and Django then you can Screenshot

That may already sound technically heavy, but it is nothing compared to the massive stacks of AWS Lambda, DynamoDB, Cloud Front, S3, and Cloud Formation whatever just to take pictures of web pages. My goal is to spin up a dev environment in a few minutes without any need for third-party apis.

Are old truths still truths?

My reasononing is that what was true in 2010 is not necessarily true in 2020. Compute, bandwidth, and storage are far cheaper and plentiful. For example, 1TB of data today is not that much from a capacity perspective. However, that is potentially 10 million images at 100Kb each. (Screenshots, yo).

Developer time is far more expensive than CPU, Bandwidth, or Storage. Why then do we let, or even encourage developers, to spend so much time on optimizations when it is more efficient to buy a bigger server?

Appendix: Alternative Screenshot Service Providers

See? I was not kidding. This list is current as of February 2020.

CloudBrowser
GeoScreenshot
LambdaTest
Page2Images
PagePeeker API
PageScreen
PageScreenShot
Screen.rip
Screenshot Bin API
ScreenshotAPI.net
ScreenshotsCloud API
Thumbalizr
WebCargo
WebShrinker
WhoAPI
apercite.fr
browserstack.com
browshot.com
grabz.it
apiflash.com
capture.techulus.in/
https://screenshot-v2.now.sh/
linkpeek.com
pagelr.com
paste.pics
restpack.io/screenshot
screencloud.net
screenshotlayer.com
screenshotmachine.com
shrinktheweb.com
site-shot.com
snapito.com
snipboard.io
stillio.com
thum.io
thumbnail.ws API
web-capture.net

TRY IT!! screenshot.simplecto.com