Forem: Stream

Hello, we're Stream and we proudly support CodeLand!

Tess Gordon — Thu, 23 Jul 2020 11:54:38 +0000

Hello! We're Stream, helping you build scalable activity feeds and in-app chat messaging in a few hours instead of weeks.

We provide product teams with everything that they need to add activity feeds or chat to their app. This includes a UI kit, scalable APIs, frontend components for React, React Native, Flutter, iOS or Android, integrated machine learning/AI, and a management dashboard.

Stream enables our customers to get to market faster, grow without worrying about scalability, and ship a higher quality user experience.

Here at Stream, we are all about sparking new ideas and encouraging collaboration. We deeply value the developer community, as we learn from and contribute to it every day. Therefore, we proudly support CodeLand and its efforts to create and provide networked learning opportunities in this space.

Come work with us!

We are hiring! There are tons of open positions at Stream right now, and we will post many more in the future as we continue to grow.

Check out these open roles:

Engineer turned CEO, Ask Me Anything!

Thierry — Thu, 11 Apr 2019 02:12:36 +0000

7 years professionally as an engineer. Started hobbying with code since I was 13.
Have been the CEO of Stream now for 4 years.

How a Go Program Compiles down to Machine Code

Thierry — Tue, 25 Sep 2018 06:00:26 +0000

Here at Stream, we use Go extensively, and it has drastically improved our productivity. We have also found that by using Go, the speed is outstanding and since we started using it, we have implemented mission-critical portions of our stack, such as our in-house storage engine powered by gRPC, Raft, and RocksDB.

Today we are going to look at the Go 1.11 compiler and how it compiles down your Go source code to an executable to gain an understanding of how the tools we use everyday work. We will also see why Go code is so fast and how the compiler helps. We will take a look at three phases of the compiler:

The scanner, which converts the source code into a list of tokens, for use by the parser.
The parser, which converts the tokens into an Abstract Syntax Tree to be used by code generation.
The code generation, which converts the Abstract Syntax Tree to machine code.

Note: The packages we are going to be using ( go/scanner , go/parser , go/token , go/ast , etc.) are not used by the Go compiler, but are mainly provided for use by tools to operate on Go source code. However, the actual Go compiler has very similar semantics. It does not use these packages because the compiler was once written in C and converted to Go code, so the actual Go compiler is still reminiscent of that structure.

Scanner

The first step of every compiler is to break up the raw source code text into tokens, which is done by the scanner (also known as lexer). Tokens can be keywords, strings, variable names, function names, etc. Every valid program “word” is represented by a token. In concrete terms for Go, this might mean we have a token “package”, “main”, “func” and so forth.

Each token is represented by its position, type, and raw text in Go. Go even allows us to execute the scanner ourselves in a Go program by using the go/scanner and go/token packages. That means we can inspect what our program looks like to the Go compiler after it has been scanned. To do so, we are going to create a simple program that prints all tokens of a Hello World program.

The program will look like this:

View the code on Gist.

We will create our source code string and initialize the scanner.Scanner struct which will scan our source code. We call Scan() as many times as we can and print the token’s position, type, and literal string until we reach the End of File ( EOF ) marker.

When we run the program, it will print the following:

View the code on Gist.

Here we can see what the Go parser uses when it compiles a program. What we can also see is that the scanner adds semicolons where those would usually be placed in other programming languages such as C. This explains why Go does not need semicolons: they are placed intelligently by the scanner.

Parser

After the source code has been scanned, it will be passed to the parser. The parser is a phase of the compiler that converts the tokens into an Abstract Syntax Tree (AST). The AST is a structured representation of the source code. In the AST we will be able to see the program structure, such as functions and constant declarations.

Go has again provided us with packages to parse the program and view the AST: go/parser and go/ast. We can use them like this to print the full AST:

View the code on Gist.

Output:

View the code on Gist.

In this output, you can see that there is quite some information about the program. In the Decls fields, there is a list of all declarations in the file, such as imports, constants, variables, and functions. In this case, we only have two: our import of the fmt package and the main function.

To digest it further, we can look at this diagram, which is a representation of the above data, but only includes types and in red the code that corresponds to the nodes:

The main function is composed of three parts: the name, the declaration, and the body. The name is represented as an identifier with the value main. The declaration, specified by the Type field, would contain a list of parameters and return type if we had specified any. The body consists of a list of statements with all lines of our program, in this case only one.

Our single fmt.Println statement consists of quite a few parts in the AST. The statement is an ExprStmt , which represents an expression, which can, for example, be a function call, as it is here, or it can be a literal, a binary operation (for example addition and subtraction), a unary operation (for instance negating a number) and many more. Anything that can be used in a function call’s arguments is an expression.

Our ExprStmt contains a CallExpr , which is our actual function call. This again includes several parts, most important of which are Fun and Args. Fun contains a reference to the function call, in this case, it is a SelectorExpr , because we select the Println identifier from the fmt package. However, in the AST it is not yet known to the compiler that fmt is a package, it could also be a variable in the AST.

Args contains a list of expressions which are the arguments to the function. In this case, we have passed a literal string to the function, so it is represented by a BasicLit with type STRING.

It is clear that we are able to deduce a lot from the AST. That means that we can also inspect the AST further and find for example all function calls in the file. To do so, we are going to use the Inspect function from the ast package. This function will recursively walk the tree and allow us to inspect the information from all nodes.

To extract all function calls, we are going to use the following code:

View the code on Gist.

What we are doing here is looking for all nodes and whether they are of type *ast.CallExpr , which we just saw represented our function call. If they are, we are going to print the name of the function, which was present in the Fun member, using the printer package.

The output for this code will be:

fmt.Println

This is indeed the only function call in our simple program, so we indeed found all function calls.

After the AST has been constructed, all imports will be resolved using the GOPATH, or for Go 1.11 and up possibly modules. Then, types will be checked, and some preliminary optimizations are applied which make the execution of the program faster.

Code generation

After the imports have been resolved and the types have been checked, we are certain the program is valid Go code and we can start the process of converting the AST to (pseudo) machine code.

The first step in this process is to convert the AST to a lower-level representation of the program, specifically into a Static Single Assignment (SSA) form. This intermediate representation is not the final machine code, but it does represent the final machine code a lot more. SSA has a set of properties that make it easier to apply optimizations, most important of which that a variable is always defined before it is used and each variable is assigned exactly once.

After the initial version of the SSA has been generated, a number of optimization passes will be applied. These optimizations are applied to certain pieces of code that can be made simpler or faster for the processor to execute. For example, dead code, such as if (false) { fmt.Println(“test”) } can be eliminated because this will never execute. Another example of an optimization is that certain nil checks can be removed because the compiler can prove that these will never false.

Let’s now look at the SSA and a few optimization passes of this simple program:

View the code on Gist.

As you can see, this program has only one function and one import. It will print 2 when run. However, this sample will suffice for looking at the SSA.

Note: Only the SSA for the main function will be shown, as that is the interesting part.

To show the generated SSA, we will need to set the GOSSAFUNC environment variable to the function we would like to view the SSA of, in this case main. We will also need to pass the -S flag to the compiler, so it will print the code and create an HTML file. We will also compile the file for Linux 64-bit, to make sure the machine code will be equal to what you will be seeing here. So, to compile the file we will run:

$ GOSSAFUNC=main GOOS=linux GOARCH=amd64 go build -gcflags “-S” simple.go

It will print all SSA, but it will also generate a ssa.html file which is interactive so we will use that.

When you open ssa.html, a number of passes will be shown, most of which are collapsed. The start pass is the SSA that is generated from the AST; the lower pass converts the non-machine specific SSA to machine-specific SSA and genssa is the final generated machine code.

The start phase’s code will look like this:

View the code on Gist.

This simple program already generates quite a lot of SSA (35 lines in total). However, a lot of it is boilerplate and quite a lot can be eliminated (the final SSA version has 28 lines and the final machine code version has 18 lines).

Each v is a new variable and can be clicked to view where it is used. The b** ’s are blocks, so in this case, we have three blocks: b1 ***,* b2 , and b3. ** b1 will always be executed. b2 and b3 are conditional blocks, which can be seen by the If v19 → b2 b3 (likely) at the end of b1 . We can click the v19 in that line to view where v19 is defined. We see it is defined as IsSliceInBounds v14 v15*, and by viewing the Go compiler source code we can see thatIsSliceInBounds **checks that* 0 <= arg0 <= arg1 . We can also click v14 and v15 to view how they are defined and we will see that v14 = Const64 [0] ***;* Const64 is a constant 64-bit integer. v15 is defined as the same but as 1. So, we essentially have 0 <= 0 <= 1 , which is obviously true.

The compiler is also able to prove this and when we look at the opt phase (“machine-independent optimization”), we can see that it has rewritten v19 as ConstBool [true]. This will be used in the opt deadcode phase, where b3 is removed because v19 from the conditional shown before is always true.

We are now going to take a look at another, simpler, optimization made by the Go compiler after the SSA has been converted into machine-specific SSA, so this will be machine code for the amd64 architecture. To do so, we are going to compare lower to lowered deadcode. This is the content of the lower phase:

View the code on Gist.

In the HTML file, some lines are greyed out, which means they will be removed or changed in one of the next phases. For example, v15 (MOVQconst [1]) is greyed out. By further examining v15 by clicking on it, we see it is used nowhere else, and MOVQconst is essentially the same instruction as we saw before, Const64 , only machine-specific for amd64. So, we are setting v15 to 1. However, v15 is used nowhere else, so it is useless (dead) code and can be eliminated.

The Go compiler applies a lot of these kinds of optimization. So, while the first generation of SSA from the AST might not be the fastest implementation, the compiler optimizes the SSA to a much faster version. Every phase in the HTML file is a phase where speed-ups can potentially happen.

If you are interested to learn more about SSA in the Go compiler, please check out the Go compiler’s SSA source. Here, all operations, as well as optimizations, are defined.

Conclusion

Go is a very productive and performant language, supported by its compiler and its optimization. To learn more about the Go compiler, the source code has a great README.

If you would like to learn more about why Stream uses Go and in particular why we moved from Python to Go, please check out our blog post on switching to Go.

The post How a Go Program Compiles down to Machine Code appeared first on The Stream Blog.

Stream & Go: News Feeds for Over 300+ Million End Users

Nick Parsons — Sun, 08 Jul 2018 18:27:10 +0000

Stream is an API that enables developers to build news feeds and activity streams (try the API. We are used by over 500 companies and power the feeds of more than 300 million end users. Companies such as Product Hunt, Under Armour, Powerschool, Bandsintown, Dubsmash, Compass, and Fabric (Google) rely on Stream to power their news feeds. In addition to the API, the founders of Stream also wrote the most widely used open source solution for building scalable feeds.

Here’s what Stream looks like today:

Number of servers: 180
Feed updates per month: 34 billion
Average API response time: 12ms
Average real-time response time: 2ms
Regions: 4 (US-East, EU-West, Tokyo, and Singapore)
1B API requests per month (~20k/minute)

Given that most of our customers are engineers, we often talk about our stack. Here’s a high-level overview:

Go is our primary programming language.
We use a custom solution built on top of RocksDB + Raft for our primary database (we started out on Cassandra but wanted more control over performance). PostgreSQL stores configs, API keys, etc.
OpenTracing and Jaeger handle tracing, StatsD and Grafana provide detailed monitoring and we use the ELK stack for centralized logging.
Python is our language of choice for machine learning, DevOps and our website https://getstream.io (Django, DRF & React).
Stream uses a combination of fanout-on-write and fanout-on-read. This results in fast read performance when a users open their feed, as well as fast propagation times when a famous user posts an update.

Tommaso Barguli and Thierry Schellenbach are the developers who started Stream nearly 3 years ago. We founded the company in Amsterdam, participated in Techstars NYC 2015 and opened up our Boulder, Colorado office in 2016. It’s been quite a crazy ride in a fairly short amount of time! With over 15 developers and a host of supporting roles, including sales and marketing, the team feels enormous compared to the early days.

The Challenge: News Feeds & Activity Streams

The nature of follow relationships makes it hard to scale feeds. Most of you will remember Facebook’s long load times, Twitter’s fail whale or Tumblr’s year of technical debt. Feeds are hard to scale since there is no clear way to shard the data. Follow relationships connect everyone to everyone else. This makes it difficult to split data across multiple machines. If you want to learn more about this problem, check out these papers:

Twitter’s tech back in the days with 150 million active users
LinkedIn’s ranked feeds
Yahoo/Princeton research paper

At a very high level there are 3 different ways to scale your feeds:

Fanout-on-write: basically precompute everything. It’s expensive, but it's easy to shard the data.
Fanout-on-read: hard to scale, but more affordable and new activities show up faster.
Combination of the above two approaches: better performance and reduced latency, but increased code complexity.

Stream uses a combination of fanout-on-write and fanout-on-read. This allows us to effectively support both customers with highly connected graphs, as well as customers with a more sparse dataset. This is important since the ways in which our customers use Stream are very different. Have a look at these screenshots from Bandsintown, Unsplash, and Product Hunt:

Switching from Python to Go

After years of optimizing our existing feed technology, we decided to make a larger leap with 2.0 of Stream. While the first iteration of Stream was powered by Python and Cassandra, for Stream 2.0 of our infrastructure we switched to Go. The main reason why we switched from Python to Go is performance. Certain features of Stream such as aggregation, ranking and serialization were very difficult to speed up using Python.

We’ve been using Go since March 2017 and it’s been a great experience so far. Go has greatly increased the productivity of our development team. Not only has it improved the speed at which we develop, it’s also 30x faster for many components of Stream.

The performance of Go greatly influenced our architecture in a positive way. With Python we often found ourselves delegating logic to the database layer purely for performance reasons. The high performance of Go gave us more flexibility in terms of architecture. This led to a huge simplification of our infrastructure and a dramatic improvement of latency. For instance, we saw a 10 to 1 reduction in web-server count thanks to the lower memory and CPU usage for the same number of requests.

Initially, we struggled a bit with package management for Go. However, using Dep together with the VG package contributed to creating a great workflow.

If you’ve never tried Go, you’ll want to try this online tour: https://tour.golang.org/welcome/1

Go as a language is heavily focused on performance. The built-in PPROF tool is amazing for finding performance issues. Uber’s Go-Torch library is great for visualizing data from PPROF and will be bundled in PPROF in Go 1.10.

Switching from Cassandra to RocksDB & Raft

1.0 of Stream leveraged Cassandra for storing the feed. Cassandra is a common choice for building feeds. Instagram, for instance, started, out with Redis but eventually switched to Cassandra to handle their rapid usage growth. Cassandra can handle write-heavy workloads very efficiently.

Cassandra is a great tool that allows you to scale write capacity simply by adding more nodes, though it is also very complex. This complexity made it hard to diagnose performance fluctuations. Even though we had years of experience with running Cassandra, it still felt like a bit of a black box. When building Stream 2.0 we decided to go for a different approach and build Keevo. Keevo is our in-house key-value store built upon RocksDB, gRPC, and Raft.

RocksDB is a highly performant embeddable database library developed and maintained by Facebook’s data engineering team. RocksDB started as a fork of Google’s LevelDB that introduced several performance improvements for SSD. Nowadays RocksDB is a project on its own and is under active development. It is written in C++ and it’s fast. Have a look at how this benchmark handles 7 million QPS. In terms of technology, it’s much more simple than Cassandra. This translates into reduced maintenance overhead, improved performance and, most importantly, more consistent performance. It’s interesting to note that LinkedIn also uses RocksDB for their feed.

Our infrastructure is hosted on AWS and is designed to survive entire availability zone outages. Unlike Cassandra, Keevo clusters organize nodes into leaders and followers. When a leader (master) node becomes unavailable the other nodes in the same deployment will start an election and pick a new leader. Electing a new leader is a fast operation and barely impacts live traffic.

To do this, Keevo implements the Raft consensus algorithm using Hashicorp’s Go implementation. This ensures that every bit stored in Keevo is stored on 3 different servers and operations are always consistent. This site does a great job of visualizing how Raft works: https://raft.github.io/

Not Quite Microservices

By leveraging Go and RocksDB we’re able to achieve great feed performance. The average response time is around 12ms. The architecture lies somewhere between a monolith and a microservice. Stream runs on the following 7 services:

Stream API
Keevo
Real-time & Firehose
Analytics
Personalization & Machine learning
Site & Dashboard
Async Workers

To see all our services divided into stacks, head over https://stackshare.io/stream/stacks.

Personalization & Machine Learning

Almost all large apps with feeds use machine learning and personalization. For instance, LinkedIn prioritizes the items in your feed. Instagram’s explore feed displays pictures outside of the people you follow that you might be interested in. Etsy uses a similar approach to optimize e-commerce conversion. Stream supports the following 5 use cases for personalization:

Documentation for building personalized feeds.

All of these personalization use cases rely on combining feeds with analytics and machine learning. For the machine learning side, we generate the models using Python. The models are different for each of our enterprise customers. Typically we’ll use one of these amazing libraries:

LightFM — How to implement recommender systems
XGBoost
Scikit learn
Numpy, Pandas, and Dask
Jupyter Notebook (For development)
Mesa Framework

Analytics

Analytics data is collected using a tiny Go-based server. In the background, it will spawn go-routines to rollup the data as needed. The resulting metrics are stored in Elastic. In the past, we looked at Druid, which seems like a solid project. For now, we could get away with a simpler solution though.

Dashboard & Site

The dashboard is powered by React and Redux. We also use React and Redux for all of our example applications:

The site, as well as the API for the site, is powered by Python, Django and Django Rest Framework. Stream is sponsoring Django Rest Framework since it’s a pretty great open source project. If you need to build an API quickly there is no better tool than DRF and Python.

We use Imgix to resize the images on our site. For us, Imgix is cost-efficient, fast and overall a great service. Thumbor is a good open source alternative.

Real-time

Our real-time infrastructure is based on Go, Redis and the excellent gorilla websocket library. It implements the Bayeux protocol. In terms of architecture, it’s very similar to the node based Faye library.

It was interesting to read the “Ditching Go for Node.js” post on Hacker News. The author moves from Go to Node to improve performance. We actually did the exact opposite and moved from Node to Go for our real-time system. The new Go-based infrastructure handles 8x the traffic per node.

Devops, Testing & Multiple Regions

In terms of DevOps, the provisioning and configuration of instances is fully automated using a combination of:

CloudFormation
Cloud-Init
Puppet
Boto & Fabric

Because our infrastructure is defined in the code it has become trivial to launch new regions. We heavily use CloudFormation. Every single piece of our stack is defined in a CloudFormation template. If needed we are able to spawn a new dedicated shard in a few minutes. In addition, AWS Parameter Store is used to hold application settings. Our largest deployment is in US-East, but we also have regions in Tokyo, Singapore and Dublin.

A combination of Puppet and Cloud-init is used to configure our instances. We run our self-contained Go’s binaries directly on the EC2 instance without any additional containerization layer.

Releasing new versions of our services are done by Travis. Travis first runs our test suite. Once it passes, it publishes a new release binary to GitHub. Common tasks such as installing dependencies for the Go project, or building a binary are automated using plain old Makefiles. (We know, crazy old school, right?) Our binaries are compressed using UPX.

Tool highlight: Travis

Travis has come a long way over the past years. I used to prefer Jenkins in some cases since it was easier to debug broken builds. With the addition of the aptly named “debug build” button, Travis is now the clear winner. It’s easy to use and free for open source, with no need to maintain anything.

Next, we use Fabric to do a rolling deploy to our AWS instances. If anything goes wrong during the deploy it will halt the deploy. We take stability very seriously:

A high level of test coverage is required.
Releases are created by Travis (making it hard to deploy without running tests).
Code is reviewed by at least 2 team members.
Our extensive QA integration test suite evaluates if all 7 components still work.

We’ve written about our experience with testing our Go codebase. When things do break we do our best to be transparent about the ongoing issue:

StatusPage
Twilio for 24/7 phone support
VictorOps for notifying our team
#firefighting channel on Slack
Intercom for customer support

VictorOps is a recent addition to our support stack. It’s made it very easy to collaborate on ongoing issues.

Tool Highlight: VictorOps

The best part about VictorOps is how they use a timeline to collaborate amongst team members. VictorOps is an elegant way to keep our team in the loop about outages. It also integrates well with Slack. This setup enables us to quickly react to any problems that make it into production, work together and resolve them faster.

The vast majority of our infrastructure runs on AWS:

ELB for load balancing
RDS for reliable Postgres hosting
ElastiCache for Redis hosting
Route53 for our DNS
CloudFront for our CDN
AWS ElasticSearch for ELK and tracing with Jaeger

The DevOps responsibilities are shared across our team. While we do have one dedicated DevOps engineer, all our developers have to understand and own the entire workflow.

Monitoring

Stream uses OpenTracing for tracing and Grafana for beautiful dashboards. The tracking for Grafana is done using StatsD. The end result is this beauty:

We track our errors in Sentry and use the ELK stack to centralize our logs.

Tool Highlight: OpenTracing + Jaegar

One new addition to the stack is OpenTracing. In the past, we used New Relic, which works like a charm for Python but isn’t able to automatically measure tracing information for Go. OpenTracing with Jaeger is a great solution that works very well for Stream. It also has, perhaps, the best logo for a tracing solution:

Closing Thoughts

Go is an absolutely amazing language and has been a major win in terms of performance and developer productivity. For tracing we use OpenTracing and Jaeger. Our monitoring is running on StatsD and Graphite. Centralized logging is handled by the ELK stack.

Stream’s main database is a custom solution built on top of RocksDB and Raft. In the past, we used Cassandra, which we found hard to maintain and which didn’t give us enough control over performance when compared to RocksDB.

We leverage external tools and solutions for everything that’s not a core competence. Redis hosting is handled by ElastiCache, Postgres by RDS, email by Mailgun, test builds by Travis and error reporting by Sentry.

Thank you for reading about our stack! If you’re a user of Stream, please be sure to add Stream to your stack on StackShare. If you’re a talented individual, come work with us! And finally, if you haven’t tried out Stream yet, take a look at this quick tutorial for the API.

React Fragments – the end of the wrapper div

Ken Hoff — Mon, 18 Jun 2018 08:30:55 +0000

Hi there! I’m Ken. I’m a Developer Advocate over at GetStream.io, where we build personalized and scalable activity feeds.

For the last several months, I’ve been working on Winds 2.0, an open-source RSS reader and podcast listening app. It’s built in Node.js, Electron, Redux and React, and as of this writing, has over 5,000 stars on GitHub. If you’d like to check it out, head on over to https://getstream.io/winds/, or check out the source code at https://github.com/GetStream/winds.

In Winds, we had a couple of unique frontend situations requiring the use of React Fragments. React Fragments are a neat little feature that were released late last year with React v16.2.0 — it’s a pretty small feature, but just knowing about the existence of them can save developers a huge headache when running into some very specific layout and styling situations.

Okay, so what’s a React Fragment?

Let’s back up a little — I’m sure that every React developer will have run into this at some point in their career (or will, very soon):

class App extends React.Component {
    render() {
        return (
            <p>I would</p>
            <p>really like</p>
            <p>to render</p>
            <p>an array</p>
        );
    }
}

Looks fine to me! But when we run it through the JSX transpiler…

  Failed to compile.

    ./src/App.js
    Syntax error: Adjacent JSX elements must be wrapped in an enclosing tag (6:8)

        4 |         return (<p>I would
        5 |         </p>
        6 |         <p>
          |         ^
        7 |             really like
        8 |         </p>
        9 |         <p>

Our JSX transpiler doesn’t like that 🙁

( What’s happening behind the curtain here? JSX is turning all of our <div>s and <MyComponent>s into React.createElement() calls — when the JSX transpiler sees multiple elements instead of a single element, it doesn’t know what tag name to render with. See React.createElement in the React documentation.)

So, what do we do? The same thing that we do every time we need to wrap a couple elements together, Pinky — wrap it in a <div>! Just like web developers have been doing since the invention of the <div> tag, another nested <div> in the DOM won’t hurt anything (probably).

class App extends React.Component {
    render() {
        return (
            <div>
                <p>I would</p>
                <p>really like</p>
                <p>to render</p>
                <p>an array</p>
            </div>
        );
    }
}

Right, problem solved. But it turns out, there’s one more way to render this group of content on the page inside of a single React Component – by having the render method return an array of nodes.

class App extends React.Component {
    render() {
        return [
            <p>I would</p>,
            <p>really like</p>,
            <p>to render</p>,
            <p>an array</p>
        ];
    }
}

If we return an array of elements, then React will transpile and render this just fine, without a wrapper <div>. Neat!

(Remember how the JSX transpiler is just turning the <div> and <MyComponent> tags into React.createElement() calls? In this case, the transpiler is just putting together an array of those calls and affixing them directly to the parent element as children, as opposed to an array of uncontained elements that it can’t find a parent for. This feature got introduced with React v16.0.0.)

See, here’s the thing – Dan Abramov and the super duper smart folks on the React team looked at this and said:

“Okay, so you can render an array of elements in two different ways – either by introducing an extra <div> into the DOM, or by using some clunky non-JSX syntax. That doesn’t make for a good developer experience!”

So, in v16.2.0, they released support for React Fragments.

Okay, now what’s a React Fragment?

Here’s the correct way to use a React Fragment:

class App extends React.Component {
    render() {
        return (
            <React.Fragment>
                <p>I would</p>
                <p>really like</p>
                <p>to render</p>
                <p>an array</p>
            </React.Fragment>
        );
    }
}

Check this out – we write this just like we would the <div>-wrapper method, but it’ll behave functionally equivalent to the array-render method, just with some nice JSX syntax. This will render those paragraph elements as an array, without any kind of wrapper <div>.

There’s also an alternate, more concise syntax for using React Fragments:

class App extends React.Component {
    render() {
        return (
            <>
                <p>I would</p>
                <p>really like</p>
                <p>to render</p>
                <p>an array</p>
            </>
        );
    }
}

Depending on your tooling, linters, build pipeline, etc, this might not work for you – the release notes say that wider support is on the way, but I’ve noticed create-react-app doesn’t support it yet.

Okay, but when do I actually use them?

Whenever you need to get rid of a wrapper <div>.

That’s it – if you ever find yourself in a situation where a wrapper <div> is screwing up the layout of your React components, use a React Fragment.

So, whenever you want to turn this:

<div class="app">

    (...a bunch of other elements)

    <div> (my react component)
        <ComponentA></ComponentA>
        <ComponentB></ComponentB>
        <ComponentC></ComponentC>
    </div>

    (...a bunch more elements)

</div>

Into this:

<div class="app">

    (...a bunch of other elements)

    <ComponentA></ComponentA>
    <ComponentB></ComponentB>
    <ComponentC></ComponentC>

    (...a bunch more elements)

</div>

Example: 2×2 CSS grid

In Winds 2.0, we made pretty heavy use of CSS Grid. This is one of the general layouts that you’ll see when looking through podcasts or RSS feeds:

If you don’t know CSS Grid yet, don’t worry – this bit of CSS will give you a quick idea of how things are laid out:

.grid {
    display: grid;
    grid-template-areas: 
        'topnav header' 
        'subnav content';
    grid-gap: 1em;
}

Okay, let’s unpack this:

In the upper left, we’ve got our brand / top-level navigation bits.
In the lower left, we’ve got our “sub-navigation” – this can respond to a couple changes in global and local state, like “active” states, tabs, or collapsing navigation.
On the right side, we’ve got the content that we’d like to show on the screen — in Winds, this is something like an RSS feed or article header, paired with an article list or article contents. These two sections will be a single React component – the props for both components change based on URL navigation.

All of these components interact with global (redux + URL) and local state slightly differently. This view is structured so that we’ve got three React components as siblings:

<div className="grid">
    <TopNav />
    <SubNav />
    <ContentComponent />
</div>

But, we want four elements actually rendered to the page:

<div class="grid">
    <div class="topnav"  />
    <div class="subnav"  />
    <div class="header"  />
    <div class="content" />
</div>

This...presents kind of a problem without React Fragments. Imagine that we’re creating the component that wraps the two right sections of our 2×2 grid view, the ContentComponent:

class ContentComponent extends React.Component {
    render() {
        return (
            <div>
                <div className="header"/>
                <div className="content"/>
            </div>
        );
    }
}

If we wrap the rendered content in <div>s, then we’ll get the following rendered output:

<div class="grid">
    <div class="topnav"  />
    <div class="subnav"  />
    <div>
        <div class="header"  />
        <div class="content" />
    </div>
</div>

This won’t work – it will totally screw up the CSS grid. From the browser’s point of view, only 3 items are present inside the grid, and one of them doesn’t have a grid-area style set.

Remember when we’re supposed to use React Fragments? Whenever we want to get rid of a <div>. If we wrap our ContentComponent in React Fragments instead of a <div>:

class ContentComponent extends React.Component {
    render() {
        return (
            <React.Fragment>
                <div className="header"/>
                <div className="content"/>
            </React.Fragment>
        );
    }
}

Then we’ll see a much different rendered output:

<div class="grid">
    <div class="topnav"  />
    <div class="subnav"  />
    <div class="header"  />
    <div class="content" />
</div>

And that works exactly as expected! No wrapper <div> is rendered, our 4 elements are rendered from 3 React components, the browser sees all elements with the correct grid-area style, and our CSS Grid is rendered correctly.

Neat! What now?

React Fragments aren’t the most significant feature that’s shown up in React recently, but they are tremendously helpful in some specific situations. Just by knowing about the existence of React Fragments, you’ll save many hours of google-fu-induced headaches. They let us render an array of elements / components in a JSX-y way, which can solve a lot of layout and styling issues with tables, lists, and CSS grids.

If you’re interested in seeing some of these React Fragments in a production app, check out the source code for Winds 2.0 over at https://github.com/GetStream/winds — or, you can download and use Winds 2.0 over at https://getstream.io/winds/.

Until next time – cheers!

The post React Fragments – the end of the wrapper div appeared first on The Stream Blog.

Winds 2.0: It’s Time to Revive RSS

Thierry — Tue, 22 May 2018 09:21:07 +0000

I love using RSS to follow the programming and tech news I care about. Unfortunately, the number of sites supporting RSS has been in rapid decline over the last few years. The reader ecosystem is slowly degrading as well. Facebook and Twitter’s dominance is now delivering its final blow to the open standard of RSS.

It’s time to turn the tide. Stream is releasing Winds 2.0, an open-source Podcast & RSS Reader. You can download Winds here or head over to Github for the code. (Also, Winds is currently trending on Product Hunt)

Many power users of RSS are developers, designers and journalists. Together, we hope to tip the balance and #ReviveRSS. With the help of the community we can build an RSS experience that’s polished, bug free and popular enough to convince publications to improve their support for open standards and RSS.

Winds is not perfect yet, we’re working on it though and with your help we might just manage to # ReviveRSS

Star Winds on Github
Share your feedback on Winds’ features and roadmap on Product Hunt
Tweet about Winds & RSS
Ask your favorite publications to fix and/or implement RSS support
Write about Winds and RSS
Contribute to Winds on Github

A huge thank you to:

Hackernoon for sharing this message
Dev.to for getting the community involved
Product Hunt for the product feedback
My colleagues at Stream for building and sponsoring Winds
Algolia for sponsoring Winds
MongoDB for helping with promotion

And of a course many thanks to the devrel team at Stream for taking what started out as an example app and turning it into such a beautiful polished experience!

Download Winds or checkout the code on Github.

The post Winds 2.0: It’s Time to Revive RSS appeared first on The Stream Blog.

Stream & Go: News Feeds for Over 300 Million End Users

Thierry — Mon, 29 Jan 2018 18:56:01 +0000

Stream is an API that enables developers to build news feeds and activity streams (try the API). We are used by over 500 companies and power the feeds of more than 300 million end users. Companies such as Product Hunt, Under Armour, Powerschool, Bandsintown, Dubsmash, Compass and Fabric (Google) rely on Stream to power their news feeds. In addition to the API, the founders of Stream also wrote the most widely used open source solution for building scalable feeds.

Here’s what Stream looks like today:

Number of servers: 180
Feed updates per month: 34 billion
Average API response time: 12ms
Average real-time response time: 2ms
Regions: 4 (US-East, EU-West, Tokyo and Singapore)
1B API requests per month (~20k/minute)

Given that most of our customers are engineers, we often talk about our stack. Here’s a high level overview:

Go is our primary programming language.
We use a custom solution built on top of RocksDB + Raft for our primary database (we started out on Cassandra but wanted more control over performance). PostgreSQL stores configs, API keys, etc.
OpenTracing and Jaeger handle tracing, StatsD and Grafana provide detailed monitoring and we use the ELK stack for centralized logging.
Python is our language of choice for machine learning, devops and our website https://getstream.io (Django, DRF & React).
Stream uses a combination of fanout-on-write and fanout-on-read. This results in fast read performance when a users open their feed, as well as fast propagation times when a famous user posts an update.

Tommaso Barguli and I (@tschellenbach) are the developers who started Stream nearly 3 years ago. We founded the company in Amsterdam, participated in Techstars NYC 2015 and opened up our Boulder, Colorado office in 2016. It’s been quite a crazy ride in a fairly short amount of time! With over 15 developers and a host of supporting roles, including sales and marketing, the team feels enormous compared to the early days.

The Challenge: News Feeds & Activity Streams

At a very high level there are 3 different ways to scale your feeds:

Fanout-on-write: basically precompute everything. It’s expensive, but its easy to shard the data.
Fanout-on-read: hard to scale, but more affordable and new activities show up faster.
Combination of the above two approaches: better performance and reduced latency, but increased code complexity.

Switching from Python to Go

After years of optimizing our existing feed technology we decided to make a larger leap with 2.0 of Stream. While the first iteration of Stream was powered by Python and Cassandra, for Stream 2.0 of our infrastructure we switched to Go. The main reason why we switched from Python to Go is performance. Certain features of Stream such as aggregation, ranking and serialization were very difficult to speed up using Python.

Initially we struggled a bit with package management for Go. However, using Dep together with the VG package contributed to creating a great workflow.

If you’ve never tried Go, you’ll want to try this online tour: https://tour.golang.org/welcome/1

Go as a language is heavily focused on performance. The built-in PPROF tool is amazing for finding performance issues. Uber’s Go-Torch library is great for visualizing data from PPROF and will be bundled in PPROF in Go 1.10.

Switching from Cassandra to RocksDB & Raft

1.0 of Stream leveraged Cassandra for storing the feed. Cassandra is a common choice for building feeds. Instagram, for instance started, out with Redis but eventually switched to Cassandra to handle their rapid usage growth. Cassandra can handle write heavy workloads very efficiently.

Cassandra is a great tool that allows you to scale write capacity simply by adding more nodes, though it is also very complex. This complexity made it hard to diagnose performance fluctuations. Even though we had years of experience with running Cassandra, it still felt like a bit of a black box. When building Stream 2.0 we decided to go for a different approach and build Keevo. Keevo is our in-house key-value store built upon RocksDB, gRPC and Raft.

RocksDB is a highly performant embeddable database library developed and maintained by Facebook’s data engineering team. RocksDB started as a fork of Google’s LevelDB that introduced several performance improvements for SSD. Nowadays RocksDB is a project on its own and is under active development. It is written in C++ and it’s fast. Have a look at how this benchmark handles 7 million QPS. In terms of technology it’s much more simple than Cassandra. This translates into reduced maintenance overhead, improved performance and, most importantly, more consistent performance. It’s interesting to note that LinkedIn also uses RocksDB for their feed.

Our infrastructure is hosted on AWS and is designed to survive entire availability zone outages. Unlike Cassandra, Keevo clusters organizes nodes into leaders and followers . When a leader (master) node becomes unavailable the other nodes in the same deployment will start an election and pick a new leader. Electing a new leader is a fast operation and barely impacts live traffic.

To do this, Keevo implements the Raft consensus algorithm using Hashicorp’s Go implementation. This ensures that every bit stored in Keevo is stored on 3 different servers and operations are always consistent. This site does a great job of visualizing how Raft works: https://raft.github.io/

Not Quite Microservices

To see all our services divided into stacks, head over here.

Personalization & Machine Learning

Almost all large apps with feeds use machine learning and personalization. For instance, LinkedIn prioritizes the items in your feed. Instagram’s explore feed displays pictures outside of the people you follow that you might be interested in. Etsy uses a similar approach to optimize ecommerce conversion. Stream supports the following 5 use cases for personalization:

Documentation for building personalized feeds.

All of these personalization use cases rely on combining feeds with analytics and machine learning. For the machine learning side we generate the models using Python. The models are different for each of our enterprise customers. Typically we’ll use one of these amazing libraries:

LightFM — How to implement recommender systems
XGBoost
Scikit learn
Numpy, Pandas and Dask
Jupyter Notebook (For development)
Mesa Framework

Analytics

Analytics data is collected using a tiny Go-based server. In the background, it will spawn go-routines to rollup the data as needed. The resulting metrics are stored in Elastic. In the past, we looked at Druid, which seems like a solid project. For now, we could get away with a simpler solution though.

Dashboard & Site

The dashboard is powered by React and Redux. We also use React and Redux for all of our example applications:

Cabin: A tutorial series on building a social network with React & Redux
Winds: Open source RSS & Podcast app (Winds 2.0 is coming out very soon)

The site, as well as the API for the site, is powered by Python, Django and Django Rest Framework. Stream is sponsoring Django Rest Framework since it’s a pretty great open source project. If you need to build an API quickly there is no better tool than DRF and Python.

We use Imgix to resize the images on our site. For us, Imgix is cost efficient, fast and overall a great service. Thumbor is a good open source alternative.

Real time

Our real time infrastructure is based on Go, Redis and the excellent gorilla websocket library. It implements the Bayeux protocol. In terms of architecture it’s very similar to the node based Faye library.

It was interesting to read the “Ditching Go for Node.js” post on Hacker News. The author moves from Go to Node to improve performance. We actually did the exact opposite and moved from Node to Go for our real time system. The new Go-based infrastructure handles 8x the traffic per node.

Devops, Testing & Multiple Regions

In terms of devops the provisioning and configuration of instances is fully automated using a combination of:

Because our infrastructure is defined in code it has become trivial to launch new regions. We heavily use CloudFormation. Every single piece of our stack is defined in a CloudFormation template. If needed we are able to spawn a new dedicated shard in a few minutes. In addition, AWS Parameter Store is used to hold application settings. Our largest deployment is in US-East, but we also have regions in Tokyo, Singapore and Dublin.

A combination of Puppet and Cloud-init is used to configure our instances. We run our self-contained Go’s binaries directly on the EC2 instance without any additional containerization layer.

Releasing new versions of our services is done by Travis. Travis first runs our test suite. Once it passes, it publishes a new release binary to GitHub. Common tasks such as installing dependencies for the Go project, or building a binary are automated using plain old Makefiles. (We know, crazy old school, right?) Our binaries are compressed using UPX.

Tool highlight: Travis

Travis has come a long way over the past years. I used to prefer Jenkins in some cases since it was easier to debug broken builds. With the addition of the aptly named “debug build” button, Travis is now the clear winner. It’s easy to use and free for open source, with no need to maintain anything.

Next we use Fabric to do a rolling deploy to our AWS instances. If anything goes wrong during the deploy it will halt the deploy. We take stability very seriously:

A high level of test coverage is required.
Releases are created by Travis (making it hard to deploy without running tests).
Code is reviewed by at least 2 team members.
Our extensive QA integration test suite evaluates if all 7 components still work.

We’ve written about our experience with testing our Go codebase. When things do break we do our best to be transparent about the ongoing issue:

StatusPage
Twilio for 24/7 phone support
VictorOps for notifying our team
#firefighting channel on Slack
Intercom for customer support

VictorOps is a recent addition to our support stack. It’s made it very easy to collaborate on ongoing issues.

Tool Highlight: VictorOps

The best part about VictorOps is how they use a timeline to collaborate amongst team members. VictorOps is an elegant way to keep our team in the loop about outages. It also integrates well with Slack. This setup enables us to quickly react to any problems that make it into production, work together and resolve them faster.

The vast majority of our infrastructure runs on AWS:

ELB for load balancing
RDS for reliable Postgres hosting
ElastiCache for Redis hosting
Route53 for our DNS
Cloudfront for our CDN
AWS ElasticSearch for ELK and tracing with Jaeger

The devops responsibilities are shared across our team. While we do have one dedicated devops engineer, all our developers have to understand and own the entire workflow.

Monitoring

Stream uses OpenTracing for tracing and Grafana for beautiful dashboards. The tracking for Grafana is done using StatsD. The end result is this beauty:

We track our errors in Sentry and use the ELK stack to centralize our logs.

Tool Highlight: OpenTracing *+ * Jaegar

One new addition to the stack is OpenTracing. In the past we used New Relic, which works like a charm for Python, but isn’t able to automatically measure tracing information for Go. OpenTracing with Jaeger is a great solution that works very well for Stream. It also has, perhaps, the best logo for a tracing solution:

Closing Thoughts

Stream’s main database is a custom solution built on top of RocksDB and Raft. In the past we used Cassandra, which we found hard to maintain and which didn’t give us enough control over performance when compared to RocksDB.

We leverage external tools and solutions for everything that’s not a core competence. Redis hosting is handled by ElastiCache, Postgres by RDS, email by Mailgun, test builds by Travis and error reporting by Sentry.

Thank you for reading about our stack! If you’re a user of Stream, please be sure to add Stream to your stack on StackShare. If you’re a talented individual, come work with us! And finally, if you haven’t tried out Stream yet, take a look at this quick tutorial for the API.

This post was originally written by Thierry Schellenbach, CEO at GetStream.io. The original post can be found at https://stackshare.io/stream/stream-and-go-news-feeds-for-over-300-million-end-users.