Forem: Tom Houlé

What changed in our stack at store2be in 2018 - A retrospective

Tom Houlé — Tue, 05 Feb 2019 14:53:55 +0000

As our team and codebase grow, we sometimes find the need to cast a look back at how things changed, and remember fondly the code we wrote and deleted, the abstractions we built and scrapped, how our code used to be, and how much fun it was to get it to where it is now.

We are building a platform to make live marketing scalable, measurable and targeted — as such we have multiple apps for the various actors of the platform and one API. We want to make sure we can keep growing and adapting the platform to changing needs, maintaining agility and stability in the process. This is our high-level mindset when making decisions regarding our tech stack. This post is a recap of how it has worked for us in 2018.

store2be developer, early 2018 (photo © Tambako The Jaguar, flickr)

Frontend - the React apps

We introduced Typescript in our React apps at the end of 2017, and we are only now getting close to completely typed frontend apps. Before Typescript, we had suffered from churn in our JS architecture as we experimented with more scalable approaches to state management and asynchronous actions - we had our own abstractions on top of redux-saga and even open sourced our way to deal with boilerplate in redux, redux-belt. These abstractions were constantly evolving, and it was a challenge to evolve core abstractions in very stateful apps without introducing type errors which translated directly to crashes for our users. There are enough Typescript success stories on the internet, so I will not elaborate too much on this.

Good tooling is something we appreciate, so in addition to Typescript we adopted prettier, TSLint (in addition to ESLint), jest (replacing jasmine), and snapshot testing. All of these lifted a lot of mental burden: no more worrying about or manually fiddling with code formatting, automatically enforced conventions, more certainty that the UI does not change by surprise, etc. Linting, testing and a lot more is enforced in CI so we can focus on the hard problems during code reviews.

We liked the experience so much we implemented a backend service for email templating in typescript and wrote about it.

Another notable and much-appreciated addition to our toolbox was react-storybook for our shared UI components.

An unexpected journey - GraphQL in our apps

Despite the numerous changes to the frontend workflow, architectural issues were still making our lives difficult.

Our RESTful API structure had not evolved much: most of our endpoints were mapping directly to database tables, with little additional logic. With more resources and more complex business logic, that meant a lot of business logic had to be implemented in the frontend and merely validated by the API.
State management and synchronization in the React apps was getting too complex - some operations and queries required several network calls and dispatching multiple redux actions. Some critical operations were not atomic, when they should have been.
Onboarding new developers to our redux-saga based abstractions and all the related conventions proved challenging, as documentation was not as comprehensive as it should have been and old conventions coexisted with new ones.
With most of the data in our apps coming from the API, we felt that manually writing Typescript interfaces for all these resources was a lot of error prone manual work. We thought about generating them from our OpenAPI/Swagger documentation, but the experiment wasn't very conclusive. Coupled with the type information loss from the redux-saga generators, we felt that we were not realizing Typescript's potential.
Documentation was maintained manually as a Swagger specification. This was also a lot of error-prone, manual work.

Faced with these problems, we decided to try and address them by progressively introducing a new GraphQL API, in the same Rails codebase as the existing RESTful API. Here is what we observed:

The additional layer gave us more room and structure to move business logic back to the backend.
Data fetching in the frontend is now a lot less complex, with queries mapping more or less 1-to-1 with views, and mutations to forms and buttons. We are not using any redux-based abstraction for CRUD anymore, and the overall amount of state management code has decreased. State is (or appears) also more localized to where it belongs in the component tree. We found this easier to reason about.
The simplicity, but also the adherence to a well-adopted and documented standard for data fetching made onboarding easier. There are a lot of good GraphQL resources for frontend developers (the official guide, to mention just one). In our experience picking up the technology was easy for everyone.
The tooling is amazing, it lets us leverage Typescript's strengths by generating precise types for our API data. We use apollo-client and apollo-codegen, they are well-documented, worked fine and the caching layer is the cherry on top.
Our API documentation is now in much better shape, because most of it is automated, keeping it in sync with the implementation is trivial, and GraphiQL offers a good interface to explore it.

It took some time to have a viable minimum implementation, but once this was deployed, it became clear very quickly that we wanted to go in that direction.

Two more pain-points that we did not identify before migrating, but improved as a result:

Debuggability. GraphiQL has been a boon to identify where errors come from. With multiple requests that cannot be reproduced easily, it was sometimes hard to distinguish between backend and frontend issues (the network tools do not let you tweak and reproduce the requests easily). With queries you can share directly as a link — GraphiQL can take queries from URL query parameters — we saved a lot of debugging time.
Evolvability. The way fields can be added and deprecated progressively has radically decreased the frequency of necessary breaking changes and made deployment smoother as a result. We use eslint-plugin-graphql to warn us when an app is using a field that has been deprecated.

store2be developer discovering GraphiQL (photo © unknown)

GraphQL in our Rails API

Adopting GraphQL was, in retrospect, an easy win in the frontend. It did require some changes, but towards a more standard workflow with a wide array of tools to make our lives easier. It was not the same in the API. GraphQL APIs pose a different set of problems and require different solutions, compared to purely RESTful APIs. To list a few of the challenges we faced:

Code organization. We naturally started by separating the API we expose to users and the one we use for our internal tools, since we had separate sets of endpoints. It seemed like the easiest way to separate what admins and normal users should see and be able to do, and code-sharing would still be possible. For a variety of reasons, this turned out to be a bad decision and we ended up merging the two APIs after a few months. This is only one example of the difficulties we had with proper code organization and reuse in the GraphQL API. We now have a good set of conventions and abstractions but it took a fair amount of exploration.
Testing. Some of the abstractions in graphql-ruby are not very friendly to test, so we had to come up with a test-friendly code structure.
Limiting access and visibility. Nothing particularly challenging here, but this also needs to be learned.
Proper use of more "advanced" features like interfaces
Avoiding the n+1 query problem - there is one well-supported, well-documented and straightforward solution, graphql-batch (by Shopify), but this is one more thing to learn and implement.

It took us a few months to settle on satisfactory abstractions. Resources like the Shopify GraphQL design tutorial were helpful. In general, having someone with prior knowledge or significant interest/experience advocating GraphQL good practices and tools helps.

Now that the abstractions are well defined, we find it easier to onboard developers to the API, and we work more with plain ruby classes instead of Rails-specific methods in models and controllers.

In short, the change was received enthusiastically by the whole team once the initial hurdles were behind us, and we are very happy with it.

Backend and infrastructure - events, k8s

We started implementing an event log for our platform, for auditing, debugging and in the future for decoupling tasks. The primary feature at the moment is a timeline in our internal app, but it opens up a lot of perspectives and challenges for 2019.

Another change we have been working on is the migration to Kubernetes. We strictly adhere to infrastructure as code and have all of our infrastructure defined as terraform scripts and helm charts. We want to complete the migration this year. We discovered a lot of tools on the way, but one fun tool that bears mentioning is click; it acts as a kind of REPL/terminal UI for your kubernetes cluster.

The Tech Handbook

At the end of 2017 our team was still at a scale where every developer had a relatively complete knowledge of the codebase in their head, but this could not last. We started and grew our internal development team documentation, what we call the Tech Handbook. It takes the shape of a GitHub repository with simple markdown files — with indexes and ToCs — so the editing workflow is comfortable for everyone. We discovered that as the codebase grows, this is not a nice-to-have but a necessity.

Sign-off

store2be developer, early 2019 (photo © HuffingtonPost)

We made rather dramatic changes to our stack in 2018 and did not compromise the delivery of actual features in the process. There was no hard, breaking change, migration or rewrite, only improvements. Let us hope the same goes for 2019.

If you would like to know more, feel free to reach out to us on twitter.

Hi there, we’re store2be, a Berlin based startup that builds a SaaS enabled marketplace for short term retails space. If you like what we are posting you might wanna check out the store2be tech page or follow our Medium channel.

graphql_client 0.6, to the browser and beyond

Tom Houlé — Sun, 30 Dec 2018 14:11:26 +0000

graphql_client is getting a new release today! The most visible new feature is a new graphql_client_web crate that makes calling a GraphQL API from Rust code running in browsers boilerplate-free.

Another notable change is the move into the graphql-rust organization on GitHub, where we can pool documentation and branding efforts with the other GraphQL crates. Thanks to the graphql-rust team for their support with and feedback on graphql_client!

Web client

Calling a GraphQL API in WebAssembly code inside a browser is now as simple as this:

Add graphql_client_web as a dependency.
Construct a client with Client::new("<GraphQL endpoint URL>").
Call it with a normal graphql_client query struct and the query variables struct.
You get a Rust future that returns a strongly typed response or a client error.

This is how it looks (copy-pasted directly from the test suite):

#[derive(GraphQLQuery)]
#[graphql(
    schema_path = "tests/countries_schema.json",
    query_path = "tests/Germany.graphql"
)]
struct Country;

#[wasm_bindgen_test(async)]
fn test_country() -> impl Future<Item = (), Error = JsValue> {
    Client::new("https://countries.trevorblades.com/")
        .call(
            Country,
            country::Variables {
                country_code: "CN".to_owned(),
            },
        )
        .map(|response| {
            let continent_name = response
                .data
                .expect("response data is not null")
                .country
                .expect("country is not null")
                .continent
                .expect("continent is not null")
                .name
                .expect("country is on a continent");

            assert_eq!(continent_name, "Asia");
        })
        .map_err(|err| {
            panic!("{:?}", err);
        })
}

Network (via the fetch API) and serialization/deserialization are taken care of by graphql_client_web.

This crate works with the wasm-bindgen ecosystem only (web-sys, js-sys), support for stdweb could be investigated if there is demand for it.

This is a first release, so expect some minor breaking changes, especially around error handling.

CLI

@h-michael has been pushing for code generation using the CLI, and it is shaping up rapidly with new features such as:

Support for query documents defining multiple queries
Optional formatting of the output code with rustfmt

Other fixes and improvements

There were a few other new features, bug fixes and breaking (naming) changes. You can take a look at the changelog for the full list of changes.

Next steps

The library is now in a very usable state, and we can start thinking about a 1.0 release. There are many enhancement ideas in the issues, but the core abstractions are stable, so we should not be too afraid to take that step once the documentation is in better shape.

Thanks

As always, many warm thanks to everyone who contributed to make the crate happen!

Announcing graphql-client 0.5

Tom Houlé — Sun, 07 Oct 2018 17:37:48 +0000

Today's release has by far the most improvements since graphql-client was initially release. This is all thanks to those who started using and contributing to the library.

In addition to the new features, the project on GitHub has moved from my personal account to the graphql-rust organization.

What graphql_client does, in short

Given a GraphQL query, and the schema of the API you are querying, the library generates a Rust module that gives you:

Correct interaction (request/response shapes)
Strongly typed responses
Strongly typed query variables
Some degree of validation (behaviour for deprecated fields is configurable for example)

I gave a short talk on the library at the Berlin Rust and Tell meetup this week (slides).

What is in 0.5

0.4 was more than a month ago. Since then users and contributors have stepped up, reported bugs (which were fixed) and contributed a lot of awesome features. Here are a few highlights.

Aliases in queries are now supported. Beyond the ergonomics, this is an important feature since it allows to query the same field on an object multiple times with different arguments, as shown in the official guide. Thanks a lot to @mathstuf!
Configurable behaviour when a query uses a field marked as deprecated in the schema. By default, you will get warnings, but you can configure the library to fail to compile if a query uses a deprecated field, or completely disable the warnings.
@h-michael has split the codegen out of the derive crate, to implement code generation via the CLI.
For several classes of items that we generate, we only generate those that are actually used by the query. This way, you do not need to define mappings for every scalar in the schema you are querying - even for small queries - anymore. This also improves compile times a lot in some scenarios. (#116)
Top-level exported types have been renamed. Types no longer have a GraphQL prefix. e.g. GraphQLQuery -> Query, GraphQLError -> Error.

Those are just the highlights - there have been a lot of other significant performance and correctness improvements, and the corner-cases in the spec that we do not support are getting very rare. See the changelog for the complete list.

Very special thanks to @h-michael, @mathstuf, @legneato and @scrogson for their work!

Next steps

The GraphQL specification is nearly completely supported, but there are minor points that need work.

Here are a few exciting features we want to work on next:

Currently code generation works with a custom derive, but we want to implement CLI and build script based flows as well. This should enable more code sharing, therefore reduced code size.
Implement platform-specific client libraries to further reduce boilerplate. The recent publication of the web-sys crate enables the creation a browser-native client built on top of it - work has already started.
A normalized cache implementation similar to what is implemented by apollo-client.
Real query validation, with better error messages. We are exploring sharing code with juniper since the spec specifies validation in detail.

Fixing a silent logger with OSÂ debugging tools

Tom Houlé — Fri, 25 Aug 2017 13:20:13 +0000

One of the features we wanted in the recent push to release a 0.2Â version of Papers was debugging output uploaded to an S3 bucket for generated documents. One of the major things we want in that debug archive is the logs from the service itself - this means having a second sink for our logs on a per-document-generation basis, ideally with a higher log level. Fortunately, this is possible in a boilerplate-free way with slog 2.0, since loggers are now also sinks.

So we decided to update to slog 2 and rework our logs configuration, which is just a few lines anyway. There is even a nice high level wrapper around slog called sloggers for logging out to files or the terminal with nice colors.

It went very smoothly and the result was impeccable, but one does not simply containerize an app that logs to the terminal: in our test deployment, the service was running but there was absolutely no terminal output.

After checking that yes, release binaries do log in the same docker image locally, and investigating several suspects, including fluentd and slog behaving differently in release builds (this is documented behaviour), I resorted to one of my favourite debugging tools: strace.

strace <your program> runs your program and prints whatever system calls it makes to the terminal, allowing you to inspect all IO operations. For Papers, you will get something like that (excerpt):

open("/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\5\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1985472, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5f6ca3d000
mmap(NULL, 3823824, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f5f6b57b000
mprotect(0x7f5f6b718000, 2093056, PROT_NONE) = 0
mmap(0x7f5f6b917000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19c000) = 0x7f5f6b917000
mmap(0x7f5f6b91d000, 14544, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f5f6b91d000
close(3)                                = 0
open("/usr/lib/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`]\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1120216, ...}) = 0
mmap(NULL, 3215384, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f5f6b269000
mprotect(0x7f5f6b37a000, 2093056, PROT_NONE) = 0
mmap(0x7f5f6b579000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x110000) = 0x7f5f6b579000
close(3)                                = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5f6ca3a000
arch_prctl(ARCH_SET_FS, 0x7f5f6ca3a8c0) = 0
mprotect(0x7f5f6b917000, 16384, PROT_READ) = 0
mprotect(0x7f5f6b579000, 4096, PROT_READ) = 0
mprotect(0x7f5f6bb36000, 4096, PROT_READ) = 0
mprotect(0x7f5f6bd50000, 4096, PROT_READ) = 0
mprotect(0x7f5f6bf5c000, 4096, PROT_READ) = 0
mprotect(0x7f5f6c160000, 4096, PROT_READ) = 0
mprotect(0x7f5f6c5b7000, 122880, PROT_READ) = 0
mprotect(0x7f5f6c842000, 20480, PROT_READ) = 0
mprotect(0x55873cd6f000, 1167360, PROT_READ) = 0
mprotect(0x7f5f6ca6f000, 4096, PROT_READ) = 0
munmap(0x7f5f6ca41000, 185208)          = 0
set_tid_address(0x7f5f6ca3ab90)         = 13360
set_robust_list(0x7f5f6ca3aba0, 24)     = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x7f5f6bb3d740, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f5f6bb497e0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0x7f5f6bb3d7e0, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f5f6bb497e0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
readlink("/etc/malloc.conf", 0x7ffcda2ec870, 4096) = -1 ENOENT (No such file or directory)
brk(NULL)                               = 0x55873d340000
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5f6b069000
munmap(0x7f5f6b069000, 2097152)         = 0
mmap(NULL, 4190208, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5f6ae6a000
munmap(0x7f5f6ae6a000, 1662976)         = 0
munmap(0x7f5f6b200000, 430080)          = 0
open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 3
read(3, "0-3\n", 8192)                  = 4
close(3)                                = 0
rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f5f6b5ae940}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5f6ae00000
open("/proc/self/maps", O_RDONLY|O_CLOEXEC) = 3
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(3, "55873c5e7000-55873cb6f000 r-xp 0"..., 1024) = 1024
read(3, "libc-2.25.so\n7f5f6b917000-7f5f6b"..., 1024) = 1024
read(3, "pthread-2.25.so\n7f5f6bd52000-7f5"..., 1024) = 1024
read(3, "so.1.1\n7f5f6c5b7000-7f5f6c5d5000"..., 1024) = 1024
read(3, "000-7ffcda2f0000 rw-p 00000000 0"..., 1024) = 316
close(3)                                = 0
sched_getaffinity(13360, 32, [0, 1, 2, 3]) = 16
mmap(0x7ffcd9af0000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ffcd9af0000
rt_sigaction(SIGSEGV, {sa_handler=0x55873ca20260, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK|SA_SIGINFO, sa_restorer=0x7f5f6bb497e0}, NULL, 8) = 0
rt_sigaction(SIGBUS, {sa_handler=0x55873ca20260, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK|SA_SIGINFO, sa_restorer=0x7f5f6bb497e0}, NULL, 8) = 0
sigaltstack(NULL, {ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5f6ca6d000
sigaltstack({ss_sp=0x7f5f6ca6d000, ss_flags=0, ss_size=8192}, NULL) = 0
getcwd("/home/tom/src/papers", 512)     = 21
open("/home/tom/src/papers/.env", O_RDONLY|O_CLOEXEC) = 3
ioctl(3, FIOCLEX)                       = 0
read(3, "PAPERS_AWS_ACCESS_KEY=<REDACTED>"..., 8192) = 228
getrandom("", 0, GRND_NONBLOCK)         = 0
getrandom("\x3d\xfd\x3c\x08\x48\x6d\x03\x13", 8, GRND_NONBLOCK) = 8
getrandom("\x80\x11\x02\x43\x2e\xa6\x6f\x1b", 8, GRND_NONBLOCK) = 8
read(3, "", 8192)                       = 0
close(3)                                = 0
stat("/home/tom/.terminfo", 0x7ffcda2eb7d0) = -1 ENOENT (No such file or directory)
stat("/etc/terminfo", 0x7ffcda2eb7d0)   = -1 ENOENT (No such file or directory)
stat("/lib/terminfo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/lib/terminfo/s/screen", {st_mode=S_IFREG|0644, st_size=1587, ...}) = 0
open("/lib/terminfo/s/screen", O_RDONLY|O_CLOEXEC) = 3
ioctl(3, FIOCLEX)                       = 0
read(3, "\32\1*\0+\0\20\0i\1z\2screen|VT 100/ANSI X"..., 8192) = 1587
close(3)                                = 0
stat("/home/tom/.terminfo", 0x7ffcda2eb790) = -1 ENOENT (No such file or directory)
stat("/etc/terminfo", 0x7ffcda2eb790)   = -1 ENOENT (No such file or directory)
stat("/lib/terminfo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/lib/terminfo/s/screen", {st_mode=S_IFREG|0644, st_size=1587, ...}) = 0
open("/lib/terminfo/s/screen", O_RDONLY|O_CLOEXEC) = 3
ioctl(3, FIOCLEX)                       = 0
read(3, "\32\1*\0+\0\20\0i\1z\2screen|VT 100/ANSI X"..., 8192) = 1587
close(3)                                = 0
ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
futex(0x7f5f6c161048, FUTEX_WAKE_PRIVATE, 2147483647) = 0
mmap(NULL, 2101248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f5f6abff000
mprotect(0x7f5f6abff000, 4096, PROT_NONE) = 0
clone(child_stack=0x7f5f6adfee30, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f5f6adff9d0, tls=0x7f5f6adff700, child_tidptr=0x7f5f6adff9d0) = 13361
epoll_create1(EPOLL_CLOEXEC)            = 3
pipe2([4, 5], O_NONBLOCK|O_CLOEXEC)     = 0
epoll_ctl(3, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLET, {u32=4294967295, u64=18446744073709551615}}) = 0
socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 6
setsockopt(6, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0

It looks scary at first, but you absolutely don't need to know what every system call does to gain insight from it (if you wonder, they all have manpages so you can go man futex).

You can follow the program linearly, and it's easy to spot things like the moment we call dotenv in config.rs In this strace output it's:

open("/home/tom/src/papers/.env", O_RDONLY|O_CLOEXEC) = 3

We also have, at the end of the trace, the moment where the server starts listening (it's binding to a TCP socket). epoll is used by the tokio event loop, so our logger configuration is happening before that, and after dotenv.

A lot of calls in-between are for reading the terminfo database to figure out the capabilities of the terminal.

This was missing in the strace output inside the container in our kubernetes cluster.

...

And that was it: when we set the TERM environment variable, our log output is back.

This is because when not specified, kubernetes does not allocate a TTY to containers, so TERM is not set, and since sloggers does not set up terminal output without it, nothing got printed.

Our solution was to set TERM to screen, although it can also be set to xterm or dumb. Another possible solution would be to configure the deployment to allocate a TTY to the container (similar to the -t flag in docker run and docker exec), but this is not necessary for simple logging.

This post is a testimony to the usefulness of the OS debugging tools and strace especially. There is a very good series of posts by Julia Evans on the many tracing tools for Linux, I would recommend them to anyone writing server-side software:

Releasing Papers, a PDF generation service based on LaTeX, written in Rust

Tom Houlé — Fri, 26 May 2017 07:36:19 +0000

The store2be team is happy to announce the first public release of Papers, an HTTP service that receives LaTeX templates, variables and assets needed for the generated LaTex file, and generates good looking PDFs. It is designed to be stateless, safe, reliable, easy to use and to deploy. We even have a second binary to test your templates locally before deploying them.

What? Why?

We needed a PDF generation service at store2be for a new feature in our advertising space manager. We investigated four options:

Use a pure Ruby PDF library like prawn
Generate PDFs from HTML and CSS with something like wkhtmltopdf
Use a paying service like DocRaptor
Write it ourselves

Writing templates as code takes too long for the number of templates we are likely to write, and generating from HTML ourselves did not yield very nice documents, so the remaining solutions were an external service or Papers. Given the small scope of the project, what we would learn from the experiment of using a different programming ecosystem (we only used Ruby in the backend before), and the much better looking PDFs produced by TeX, we went ahead and wrote Papers.

How do I use it?

You will need three things:

Have the Papers service running somewhere so you can make HTTP calls to it
Have the (Tera) templates and assets hosted somewhere (we use S3/CloudFront)
Have an HTTP endpoint in your app where the PDF or errors will be posted

Generating a PDFÂ is then a simple matter of posting a JSON object describing your wish to the /submit endpoint of your Papers service. For example:

{
  "assets_urls": [
    "https://static.cook.example.com/spaghetti.jpg",
    "https://static.cook.example.com/pesto.jpg"
  ],
  "callback_url": "https://cook.example.com/illustrated-recipes/generated&token=18e8d495ac34c541b5a5167bffcbac96",
  "template_url": "https://static.cook.example.com/templates/pesto_spaghetti.tex"
}

In order to deploy Papers easily, we prepared a docker image. There is also a Kubernetes example in the repository.

Our experience with Rust

Overall, Rust was a very good fit for the goals we had.

We wanted a crash-free service, and the explicit error handling made that possible with a high level of confidence.
We wanted the service to be stable and not leak memory, disk space or file descriptors. Once again, it was very helpful that Rust gives you the tools to apply the same logic that makes memory safety without garbage collection possible for other resources, like files or temporary directories. And we get fine-grained concurrency control with tokio as well.
Since most of the overhead of the service comes from LaTeX itself, we wanted it to be as lightweight as possible, mostly memory-wise, and there Rust is ideal again compared to Ruby.
We wanted easy deployment and administration, and having a compiled binary makes it very easy.

Writing a small service was also a way for our team to assess Rust as a tool for other projects in the future. Our perceptions have adjusted in more than a few ways from this experience.

We set out with a perception of Rust as the cool new thing, and expected to get high level abstractions from it. The previous experience with Rust in the team ranged from having read the Book to having completed small programs, and learning curve was a bigger deal than we thought. Rust does force you to think at a lower level and handle more details than something like Ruby or Python: you have to make decisions about errors, even if you want to ignore them, and even if there are ways to simplify memory management problems (Rc for example), they are opt-in.

It was hard to accept that you sometimes have to write code to make the program compile and does not contribute at all towards solving the business problem (in our minds as programmers used to high-level languages), for example:

fn make_pasta(&self) -> Box<Future<Item=Satisfaction, Error=OvercookedError>> {
  let logger = self.logger.clone();
  Box::new(cook_pasta.and_then(move |pasta| {
    warn!(logger, "Pasta is very hot, be careful");
    eat_pasta(pasta)
  }))
}

The line where we clone the logger so it can be moved into the Future is purely there to satisfy the borrow checker, and while it makes complete sense once you "get" the concept ownership, it still feels like clutter in the context of high level code.

The cargo ecosystem, and especially cargo doc, is fantastic. Period.

Library-wise, we found good quality libraries for everything. The elephant in the room now for server programming is of course the work-in-progress conversion of the ecosystem to asynchronous APIs based on futures and tokio, which was inconvenient but not a show stopper. We went for hyper with the master branch, as there was no higher level alternative that worked with futures.

In general, futures were hard. There is a clear rightward drift, and error handling differs from synchronous code (in a bad way). The interaction with the event loop also has a learning curve of its own. To be fair, things are going to improve a lot with time and a few language features that are already planned, in particular impl Trait, trait aliases and, albeit more distant, async/await syntax. The current need to box everything or write complicated types nudged us in the direction of big, monolithic functions when we have to return a Future.

Compile times are long. Coming from interpreted languages, this is the most frustrating part of developing in Rust. They are getting better though, and incremental compilation has yielded promising results for us.

Our conclusion is not very original: Rust is a good tool for the use-cases where it makes sense, but we wouldn't use it as a language to write a classical web app (yet).

What's next?

The service is usable in production for small volumes of PDFs (depending on the size, a dozen per minute seems reasonable) and behind a firewall. We do plan to make it faster, and the first obvious step in that direction would be caching the assets. We also need a smarter way to manage the latex processes, and benchmarks to measure and improve performance.

There are already many other features that go in the direction of more configurability and better debugging experience planned for the next releases.

Epilogue

If you read this far, we'd be really interested in feedback on our endeavour and on the code if you have some. Just shoot us an email at tech@store2be.com :)

For comprehensive documentation, you can head to the README.

This is of course all open source, released under the MIT license, and available on Github. We are very much open to contributions.

Kubernetes - Redirect HTTP to HTTPS with ELB and the nginx ingress controller

Tom Houlé — Tue, 04 Apr 2017 13:44:53 +0000

TL;DR Read the last paragraph for the most recent way of implementing HTTPS with an nginx ingress controller while letting ELB handle the certificates.

In our exploratory work with Kubernetes, one of the first hurdles was configuring a proper HTTP ingress. At first, we wanted three properties from our setup:

All unencrypted http traffic is immediately redirected to https.
The ingress controller does not handle TLS certificates. instead we have TLS termination at the ELB level.
The redirection is handled by the ingress controller, not the individual ingress resources, since we always want HTTP traffic to be encrypted, and it limits the potential for errors.

It turns out that combining those three was more complicated than expected, since ELB does not do HTTP redirects, and neither trÃ¦fik nor nginx ingress controllers supported redirecting unless they handled TLS termination themselves.

Our solution

The DNS (Route 53) and ELB configurations were the simplest.

We first created a LoadBalancer service for our ingress:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: kube-system
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <the arn for our wildcard TLS cert>
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
spec:
  selector:
    k8s-app: nginx-ingress-lb
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 80

When applied, the service will create the ELB load balancer for you.

Our DNS configuration is very simple: a “gateway” domain that points to the load balancer. The other subdomains are just CNAME records that point to it. The “gateway” domain is updated manually as a part of the cluster creation, and we just add the CNAME records when needed. We plan on trying out external-dns for the “gateway” domain to automate this in the future.

The last part was less ideal, since the nginx ingress controller from kubernetes (there is another implementation by the nginx people) did not support redirecting to https unless it is configured to handle the https traffic itself (which we really don't want).

Fortunately, the controller lets you completely override its nginx config file, so we just copied the default file from the source, which is of course not ideal, and added what we wanted, which is a 301 redirect based on the X-Forwarded-Proto header set by ELB:

if ( $http_x_forwarded_proto = http ) {
  rewrite ^(.*) https://$host$1 permanent;
}

We can then just put the modified config template in a ConfigMap (in our case nginx-config in the kube-system namespace) and pass it to nginx by placing this in the container section of the Deployment:

args:
  - /nginx-ingress-controller
  - --default-backend-service=kube-system/default-http-backend
  - --publish-service=kube-system/nginx-service
  - --configmap=kube-system/nginx-config

How to do it today

This story is a testimony to how fast things are moving in the k8s ecosystem. We came up with this solution on the nginx ingress controller version 0.9.0-beta.1, exactly two weeks ago. Now the current version is 0.9.0-beta.3 and the ingress.kubernetes.io/force-ssl-redirect annotation was added, making the same configuration achievable by adding it in each ingress resource.

This option ticks 2 of our 3 boxes, and we found that the last point, having this setting centralized, was not important enough to justify the hack.

Hi there, we're store2be, a Berlin based startup that builds a SaaS enabled marketplace for short term retails space. If you like what we are posting you might wanna check out the store2be tech page or follow our Medium channel.