Forem: Joe Buckle

Creating a web API with Lua using Nginx OpenResty

Joe Buckle — Sun, 02 Feb 2020 16:58:55 +0000

I'm a bit of an old fashioned, pragmatic developer and tend to stick with frameworks and patterns that I know are reliable - a 'stick in the mud' if you will.

This dive is deviating from my usual path.

I had never considered Lua to be a viable language to create a backend for a web application. Of course I'm not totally oblivious to the language itself - I do know it's used mostly for embedding in other software and game engines - but that's it really.

I came across the OpenResty project whilst on a learning journey of Nginx web server.

Writing code directly on the web server rather than passing the request to some interpreter is a very interesting concept for me.
Lua runs directly inside the Nginx worker which means a very small barrier between the webserver and the application code.
Additionally, Lua is known for being immensely fast compared to interpreted languages particularly when using the LuaJIT compiler.

Nginx is capable of handling a very high number of concurrent connections at a very low memory footprint.

However, there doesn't appear to be that much excitement about it.
It's been around for some time (at least since 2016) but it does appear to be lacking in developer adoption. Saying that, some big names are using it, including Cloudflare and Tumbler.

Despite that - the performance gain from this sort of backend has peaked my interest.

The easiest way (perhaps the only way) to get started with this is by installing OpenResty.
This provides the Nginx web server with the Lua module.

If you're familiar with Nginx this example will make sense to you. The location directive is being passed a Lua script as a file.
The output from that script is expected to be in JSON in this configuration:

location ~ ^/api(.*)$ {
  default_type 'text/json';
  add_header 'Content-Type' 'application/json';
  content_by_lua_file /etc/openresty/sites/api.lua; # < Points to Lua file
}

You can also pass arbitrary code into content_by_lua and content_by_lua_block.

Now, Lua is a reasonable simple language to pick up if you're used to Ruby or Python. I hadn't used it before but I was still able to quickly write a script that allows creation of API endpoints and parses the body and parameters.

There is an object you can access in Lua called ngx. This provides you with the data passed into Nginx from the request such as the body or the path etc...

Example:

ngx.var.request_method -- POST, GET.. whatever
ngx.req.get_body_data() -- The data passed in from the request
ngx.var.uri -- The request path

The script I wrote allows me to set the Method and Endpoints that are allowed.

--[[ api.lua --]]

-- Helper functions
function strSplit(delim,str)
    local t = {}

    for substr in string.gmatch(str, "[^".. delim.. "]*") do
        if substr ~= nil and string.len(substr) > 0 then
            table.insert(t,substr)
        end
    end

    return t
end

-- Read body being passed
-- Required for ngx.req.get_body_data()
ngx.req.read_body();
-- Parser for sending JSON back to the client
local cjson = require("cjson")
-- Strip the api/ bit from the request path
local reqPath = ngx.var.uri:gsub("api/", "");
-- Get the request method (POST, GET etc..)
local reqMethod = ngx.var.request_method
-- Parse the body data as JSON
local body = ngx.req.get_body_data() ==
        -- This is like a ternary statement for Lua
        -- It is saying if doesn't exist at least
        -- define as empty object
        nil and {} or cjson.decode(ngx.req.get_body_data());

Api = {}
Api.__index = Api
-- Declare API not yet responded
Api.responded = false;
-- Function for checking input from client
function Api.endpoint(method, path, callback)
    -- If API not already responded
    if Api.responded == false then
        -- KeyData = params passed in path
        local keyData = {}
        -- If this endpoint has params
        if string.find(path, "<(.-)>")
        then
            -- Split origin and passed path sections
            local splitPath = strSplit("/", path)
            local splitReqPath = strSplit("/", reqPath)
            -- Iterate over splitPath
            for i, k in pairs(splitPath) do
                -- If chunk contains <something>
                if string.find(k, "<(.-)>")
                then
                    -- Add to keyData
                    keyData[string.match(k, "%<(%a+)%>")] = splitReqPath[i]
                    -- Replace matches with default for validation
                    reqPath = string.gsub(reqPath, splitReqPath[i], k)
                end
            end
        end

        -- return false if path doesn't match anything
        if reqPath ~= path
        then
            return false;
        end
        -- return error if method not allowed
        if reqMethod ~= method
        then
            return ngx.say(
                cjson.encode({
                    error=500,
                    message="Method " .. reqMethod .. " not allowed"
                })
            )
        end

        -- Make sure we don't run this again
        Api.responded = true;

        -- return body if all OK
        body.keyData = keyData
        return callback(body);
    end

    return false;
end

Then I can call Api.endpoint() following to create my endpoints:

--[[ api.lua --]]
Api.endpoint('POST', '/test',
    function(body)
        return ngx.say(
            cjson.encode(
                {
                    method=method,
                    path=path,
                    body=body
                }
            )
        );
    end
)

Api.endpoint('GET', '/test/<id>/<name>',
    function(body)
        return ngx.say(
            cjson.encode(
                {
                    method=method,
                    path=path,
                    body=body,
                }
            )
        );
    end
)

So this has got me started. Obviously the logic inside these endpoints would be much more complex and usually have to connect to databases - all of which OpenResty supports.

One thing I did come across was the Lapis Framework created by Leafo who I know mostly from creating the PHP compilers for Less and SCSS.

Github for code the code here - https://github.com/bambattajb/openresty-api-example,

Embedded UI Components with Preact

Joe Buckle — Tue, 24 Dec 2019 10:34:14 +0000

As a front-end developer it's quite common to be tasked with creating a new UI component for an application.

You're given access to some backend API and you have to crack on and build a UI that sits somewhere inside the website as a widget, page or whatever.

The component is complex enough that you need to manage view states, so you'd quite like a system for that.
You also like using the cleaner ES6 syntax and you're already familiar with React and JSX and all that jazz.

Now, you don't have access to the applications overarching build tools - you just need to find a way of passing some reasonably sized and efficiently written code over to the application maintainers who will factor it into their monolith 😊.

I tend to see these UI components as their own small compartmentalised applications but I am fully conscious of the increased size when shipping seemingly unnecessary frameworks in packaged code.

So React is out of the question because it's > 30kb.

I eventually discovered Preact which claimed to come in at 3kb and supports all of the Virtual DOM and State Management features of React.

I totally signed up and was not disappointed. Just set up a Webpack build environment and away you go.

How is Preact so small?

React has its own Synthetic Event System for handling events and Preact uses the browsers native addEventListener.
This could lead to a user experience penalty.

React is trying very hard (30kb + hard) to ensure a consistent feel in whatever browser it's being run in; whereas Preact has chosen to use unpredictable browser-native event listeners for a massively reduced packaged size.

More on the 'subtle' differences here - Main Differences.

What's missing?

Well, not much!

Proptypes (they aren't used all the time anyway)
Children

Conclusion

Preact is a great solution to my problem but it does look as though it might come with some UI performance trade-offs leading to an inconsistent UX.

That said, if you chose to build and embedded UI component without a framework you most likely wouldn't create your own synthetic event system for seamlessly handling differences in native events.

Preact is totally winning for me in this context 👍

I love WordPress, but I don't like its ecosystem

Joe Buckle — Sat, 21 Dec 2019 15:24:21 +0000

Come gather round friends and I'll tell you a tale
Of when I claimed to be a WordPress guru
But was then made to use off-the-shelf themes and plugins
And that claim I sought to undo

I've been using WordPress to solve content management problems for clients in the digital publishing industry for around 10 years and know its core features pretty well having spent countless hours traversing through the codex and reading a plethora of problem-solving developer articles.

I am an advocate for the software and push it as a low-cost, easy to use, extensible and familiar content editing & publishing platform for digital publishing teams.

From a developer perspective the core provides a very good and efficient Query Engine, Database Abstraction Layer, Rest API Framework, User / Roles management and loads more.

It also provides a decent admin interface that is intuitive and adaptable enough that I can tailor it to specific business requirements.

What's not to like?

Well, last year I assumed the role of "WordPress Guru".

Up until then I had been using WordPress as a toolkit for building a unique set of features for my clients. I would be writing a lot of specific business logic around this toolkit just as you would a framework.

At the time of accepting this role I didn't know which sites I'd be working on and just assumed that, armed with this knowledge, I should be able to handle it right?

But I was wrong!

You see, there is, in my opinion, a "Dark Side" to WordPress - its Theme and Plugin culture.

It's not like I was unfamiliar with this and whenever I'd read up or hear of this culture I thought it was individuals or small businesses on small budgets looking to get a website up as cheap as possible and that it shouldn't apply to serious businesses or organisations.

I was wrong again!

I used the term "Dark Side" because I would be dealing with content managers that have a business requirement for something that is more flexible than the feature-restrictive theme or plugin. At some point this system was sold to them either by someone in-house or a developer that is now long gone.

The first problem is you have a client that doesn't understand "why" this feature is difficult to implement (they're right - it shouldn't be, if this system was built properly).
The second is the developer trying to shoehorn features into this system knowing it's not really fit for purpose thus making their ongoing maintenance work more difficult.

Sometimes you have to do things like "modify core code" of plugins only to learn of a "major security update" some weeks later that you now have to factor in.

Sites end up becoming a maintenance and security headache.

Lesson learned

I will never claim to be a "WordPress Guru" again as putting me in an environment where I have to build or extend sites that use WooThemes or Divi (for example) then I am lost!

Someone who is a "WordPress Guru" would understand the entire ecosystem of available WordPress products - not just its core system.

My advice for publishers is if you are looking at purchasing a Theme or Plugin to handle a feature you're needing make sure it does EVERYTHING you're ever going to need.

Otherwise consult a developer - there are some good quality plugins that allow themselves to be extended by using custom hooks and filters, perhaps enough so to fill the void in requirements.