Forem: Max

React to Vue Conversion with Claude, Gemini, and ChatGPT

Max — Fri, 16 May 2025 05:19:31 +0000

This post covers converting my book reading tracker app from React to Vue using GitHub Copilot Pro in VSCode.

Project size:

8 JSX components
8 asset files (TailwindCSS, images)
4 WASM files
index.js

The app has only 3 screens, so it is quite an uncomplicated project.

Online React to Vue Converters

All online tools offering React to Vue conversion that came up on page one in Google produced unusable garbage.
It was not good enough even for a boilerplate.

I DO NOT recommend any of these tools:

GitHub Copilot

GitHub Copilot is the default AI Chat choice for VSCode.
You may also consider using Cursor or Claude Code.

If you are new to using Copilot Chat window in VSCode, there is one customization I highly recommend - disable Send on Enter setting to enter multiline prompts.

Open Keyboard Shortcuts (Ctrl-K + Ctrl-S)
Search for workbench.action.chat.submit
Change from Enter to Shift+Enter or any other key combination you like

More info:

Converting the Entire Project 🙄

Attempts to convert the entire project in one go failed.
All the models clearly prefer the user to break it down into smaller steps, e.g. create scaffolding, improve it, add linters, NPM packages and then convert the components one by one.

I could have asked the LLM for the action plan, correct it and then ask to execute.
Doing the conversion in small steps seemed like a better approach.

Converting a Single React Component 🎉

My NavBar component had only 71 lines of code and used a single dependency, useAuth0 from @auth0/auth0-react for authentication.

Full file: https://github.com/rimutaka/bookwormfood/blob/react-to-vue-conv/vue/src/components/conv/navbar.js

Prompt

It took me a few attempts to come up with a prompt that produced a meaningful conversion with navbar.js as the context:

Convert the current react component into a VueJS component to go into an existing app structure.
Assume that any functions, objects or APIs called from the component exist somewhere.
Copy comments as-is.
Use TypeScript, VueJS composition API.

I asked ChatGPT to improve it. The new prompt was more readable, but made no difference to the end result - all models produced the same output for both.

Convert the provided React component into a Vue 3 Single File Component that fits into an existing Vue application structure.

Use TypeScript and the VueJS Composition API.
Preserve all comments exactly as they appear in the original code.
Assume that any external functions, objects, or APIs referenced in the component are available elsewhere in the project.
Ensure the resulting Vue component is idiomatic and ready for integration into a typical Vue app.

Claude 3.7 Sonnet

The worst result of them all:

didn't follow the composition API structure
left placeholders
unusual formatting

Vue file: https://github.com/rimutaka/bookwormfood/blob/react-to-vue-conv/vue/src/components/conv/Claude37Sonnet.vue

Claude 3.5 Sonnet

A much better result with a few minor issues:

window.location.port == 80 should be "80" in TypeScript
process.env.VUE_APP_BUILD_TS should be import.meta.env.VITE_APP_BUILD_TS
inconsistent use of '' and ""

Vue file: https://github.com/rimutaka/bookwormfood/blob/react-to-vue-conv/vue/src/components/conv/Claude35Sonnet.vue

Gemini 2.5 Pro

A very similar result to Claude 3.5 Sonnet.

The <template> part was nearly identical with only syntactical and formatting differences for the rest of the code.

Vue file: https://github.com/rimutaka/bookwormfood/blob/react-to-vue-conv/vue/src/components/conv/Gemini25Pro.vue

ChatGPT-4.1

A nearly identical result to Claude 3.5 Sonnet and Gemini 2.5 Pro.

Vue file: https://github.com/rimutaka/bookwormfood/blob/react-to-vue-conv/vue/src/components/conv/ChatGPT41.vue

Conclusion

No clear winner - I used Claude, Gemini, and ChatGPT interchangeably.

The LLMs did about 80% of the work.
The converted code was mostly correct, but not production-ready.

The full conversion took me 2 days including unrelated bug fixes.

From this React:
https://github.com/rimutaka/bookwormfood/tree/54f949b...

To this Vue:
https://github.com/rimutaka/bookwormfood/tree/28e52c2...

Question

Could one of these brilliant golden retrievers on acid completely replace me for this task?

Not yet.

AWS Lambda with CloudFront: CORS, authorization and caching examples

Max — Wed, 14 Aug 2024 01:50:09 +0000

This post is a collection of examples with different configuration options for invoking AWS Lambda via CloudFront.
It demonstrates how configuration changes affect the lambda function invocations and request/response headers.

Example 1: CORS not required, no authorization headers, no caching

This simple GET request does not require the CORS protocol or authorization:

const response = await fetch("https://d9tskged4za87.cloudfront.net/sync.html");

Function URL configuration:

Auth type: AWS_IAM
CORS disabled (irrelevant as no CORS protocol is involved)

CloudFront configuration:

origin access with signed requests
CachingDisabled policy
OPTIONS caching setting is irrelevant since caching is disabled via CachingDisabled policy
Managed-AllViewerExceptHostHeader request policy
No response policy

How it works:

CloudFront replaces the value of Host and passes all other headers onto the lambda
the lambda function handler is invoked for all HTTP request types
all headers generated by the lambda function are forwarded back to the client

Request headers

Browser to CloudFront:

GET /sync.html HTTP/2
Host: d9tskged4za87.cloudfront.net
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
Origin: https://localhost:8080
DNT: 1
Connection: keep-alive
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
Priority: u=0
Pragma: no-cache
Cache-Control: no-cache

Headers in the payload passed to the lambda function:

{
  "accept": "*/*",
  "accept-encoding": "gzip, deflate, br, zstd",
  "accept-language": "en-US,en;q=0.5",
  "cache-control": "no-cache",
  "cloudfront-forwarded-proto": "https",
  "cloudfront-is-android-viewer": "false",
  "cloudfront-is-desktop-viewer": "true",
  "cloudfront-is-ios-viewer": "false",
  "cloudfront-is-mobile-viewer": "false",
  "cloudfront-is-smarttv-viewer": "false",
  "cloudfront-is-tablet-viewer": "false",
  "cloudfront-viewer-address": "2406:2d40:728d:fa10::c4a:42838",
  "cloudfront-viewer-asn": "14593",
  "cloudfront-viewer-city": "Auckland",
  "cloudfront-viewer-country": "NZ",
  "cloudfront-viewer-country-name": "New Zealand",
  "cloudfront-viewer-country-region": "AUK",
  "cloudfront-viewer-country-region-name": "Auckland",
  "cloudfront-viewer-http-version": "2.0",
  "cloudfront-viewer-latitude": "-36.85060",
  "cloudfront-viewer-longitude": "174.76790",
  "cloudfront-viewer-postal-code": "1010",
  "cloudfront-viewer-time-zone": "Pacific/Auckland",
  "cloudfront-viewer-tls": "TLSv1.3:TLS_AES_128_GCM_SHA256:sessionResumed",
  "dnt": "1",
  "host": "mq75dt64puwxk3u6gjw2rhak4m0bcmmi.lambda-url.us-east-1.on.aws",
  "origin": "https://localhost:8080",
  "pragma": "no-cache",
  "priority": "u=0",
  "sec-fetch-dest": "empty",
  "sec-fetch-mode": "cors",
  "sec-fetch-site": "cross-site",
  "user-agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0",
  "via": "2.0 03e8784cc6fbcd65ff743e9f537e8e88.cloudfront.net (CloudFront)",
  "x-amz-cf-id": "Wtw2gvr0...o3nk-DQ==",
  "x-amz-content-sha256": "e3b0c44...852b855",
  "x-amz-date": "20240812T024912Z",
  "x-amz-security-token": "IQoJb3...yM6wdjMEg==",
  "x-amz-source-account": "512295225992",
  "x-amz-source-arn": "arn:aws:cloudfront::512295225992:distribution/E3FGXRC3VXQ2IF",
  "x-amzn-tls-cipher-suite": "ECDHE-RSA-AES128-GCM-SHA256",
  "x-amzn-tls-version": "TLSv1.2",
  "x-amzn-trace-id": "Root=1-66b97828-7217f26e68fb64c806f4daf5",
  "x-forwarded-for": "2406:2d40:728d:fa10::c4a",
  "x-forwarded-port": "443",
  "x-forwarded-proto": "https"
}

Request headers of interest:

browser to CloudFront Host: d9tskged4za87.cloudfront.net was replaced by CloudFront
CloudFront to Lambda: host: mq75dt64puwxk3u6gjw2rhak4m0bcmmi.lambda-url.us-east-1.on.aws
origin: https://localhost:8080 is the same for both

Response headers

Lambda's response:

{
  "statusCode": 200,
  "headers": {
    "x-foo-header": "bar",
    "content-type": "text/html; charset=utf-8"
  },
  "body": "Hello from client-sync",
  "isBase64Encoded": false,
  "cookies": []
}

CloudFront to the browser:

HTTP/2 200 
content-type: text/html; charset=utf-8
content-length: 22
date: Mon, 12 Aug 2024 02:49:14 GMT
x-amzn-requestid: 28b96338-a3a6-4a6e-aad2-8b7d25a4fe13
x-foo-header: bar
vary: Origin
x-amzn-trace-id: root=1-66b97828-7217f26e68fb64c806f4daf5;parent=717724cd0dc027c2;sampled=0;lineage=a964c7ca:0
x-cache: Miss from cloudfront
via: 1.1 03e8784cc6fbcd65ff743e9f537e8e88.cloudfront.net (CloudFront)
x-amz-cf-pop: LAX3-C3
x-amz-cf-id: Wtw2gvr045gDD6eqIC6s_rDEnugktHpg0YUjmrp2jRuN6PNo3nk-DQ==
X-Firefox-Spdy: h2

Response headers of interest:

x-foo-header: bar custom header was passed from the lambda function to the browser
x-cache: Miss from cloudfront CloudFront header is always Miss ... because the caching is disabled

Example 2: CORS with custom authorization header and no caching

This request requires the CORS protocol, but the authorization is done through a custom header:

const response = await fetch(
  "https://d9tskged4za87.cloudfront.net/sync.html",
  {
    headers: {
      "X-Books-Authorization": "dummy-book-auth4",
    },
  },
);

In Function URL:

Auth type: AWS_IAM
CORS enabled

In CloudFront (unchanged):

origin access with signed requests
CachingDisabled policy
OPTIONS caching is disabled
Managed-AllViewerExceptHostHeader request policy
No response policy

Flow changes:

The browser sends the HTTP OPTIONS request first to obtain the CORS headers:
CORS headers are added by AWS Lambda as part of the lambda handler invocation
The browser sends a GET request after checking the CORS headers

OPTIONS request/response

CORS-related request headers for HTTP OPTIONS request:

OPTIONS /sync.html HTTP/2
Access-Control-Request-Method: GET
Access-Control-Request-Headers: x-books-authorization
Origin: https://localhost:8080

CloudFront OPTIONS response has four CORS headers generated by AWS Lambda without invoking the lambda function handler:

access-control-allow-origin: https://localhost:8080
access-control-allow-headers: authorization,content-type,x-books-authorization
access-control-allow-methods: GET,POST
access-control-allow-credentials: true

GET request/response

The OPTIONS request is followed by a GET request with a custom X-Books-Authorization header:

GET /sync.html HTTP/2
X-Books-Authorization: dummy-book-auth4
Origin: https://localhost:8080
...

The lambda function handler receives the X-Books-Authorization header inside the GET request payload:

{
  "x-books-authorization": "dummy-book-auth4",
  ...
}

The lambda's response is identical to the previous GET example with no CORS:

{
  "statusCode": 200,
  "headers": {
    "x-foo-header": "bar",
    "content-type": "text/html; charset=utf-8"
  },
  "body": "Hello from client-sync",
  "isBase64Encoded": false,
  "cookies": []
}

CloudFront GET response has two CORS headers added to the Lambda's response by AWS Lambda (not the handler):

access-control-allow-origin: https://localhost:8080
access-control-allow-credentials: true

Example 3: CORS with Authorization header and caching

This request uses the Authorization header and requires the CORS protocol:

const response = await fetch(
  "https://d9tskged4za87.cloudfront.net/sync.html",
  {
    headers: {
      "Authorization": "dummy-auth",
    },
  },
);

In Function URL:

Auth type: NONE
CORS enabled

In CloudFront:

origin access with unsigned requests
custom caching policy that includes the Authorization header
OPTIONS caching is enabled
Managed-AllViewerExceptHostHeader request policy
No response policy

Flow changes:

the Lambda function is public (Auth type: NONE)
CloudFront no longer uses the Authorization header (no request signing)
repeat requests with the same value for the Authorization header are served from the CloudFront cache
repeat OPTIONS requests are served from the CloudFront cache

All the request/response headers are nearly identical to the previous example, except for the common Authorization header replacing the custom X-Books-Authorization header.
The Authorization header is passed to the lambda function handler as part of the payload.

OPTIONS request/response:

identical to the previous example

GET request headers

From browser to CloudFront:

GET /sync.html HTTP/2
Host: d9tskged4za87.cloudfront.net
Authorization: dummy-auth
Origin: https://localhost:8080
...

The Authorization header was added to the lambda handler payload:

{
  "authorization": "dummy-auth",
  ...
}

Caching

Initial OPTIONS/GET requests:

both OPTIONS and GET return x-cache: Miss from cloudfront header
AWS Lambda (not the handler) responds to OPTIONS request
the lambda handler is invoked for the GET request

Repeat OPTIONS/GET requests with the same Authorization value:

both return x-cache: Hit from cloudfront header
no lambda invocations

Repeat OPTIONS/GET requests with different Authorization values:

OPTIONS returns x-cache: Hit from cloudfront header
GET returns x-cache: Miss from cloudfront header
the lambda handler is invoked for the GET request

AWS Lambda with CloudFront configuration guide

Max — Wed, 14 Aug 2024 01:28:32 +0000

This post explores different configuration options for invoking AWS Lambda via CloudFront to demonstrate how different CloudFront and Lambda Function URL settings affect CORS and authorization headers.

Overview

AWS Lambda Functions can be invoked via a public AWS Lambda Function URL with an HTTP call from a browser script, e.g.

const lambdaResponse = await fetch(
  "https://mq75dt64puwxk3u6gjw2rhak4m0bcmmi.lambda-url.us-east-1.on.aws",
  {
    headers: {
      Authorization: `Bearer ${token}`,
    },
  },
);

There are two possible downsides to this type of invocation:

you cannot use a custom domain
no server-side caching

Calling your lambdas via CloudFront solves both of these issues.

Configuration complexity

There is quite a bit of configuration to be done upfront to make a Lambda function work with CloudFront:

Lambda Function URL access control and CORS
CloudFront origin and behaviors
CloudFront caching policy
CloudFront origin request policy
CloudFront response headers policy

This guide complements the official AWS documentation with examples and shortcuts.

Core concepts

This section explains CloudFront terms and concepts that affect Lambda, CORS and the Authorization header.

General

CloudFront forwards always forwards uncached HTTP OPTIONS requests to the Lambda URL
CloudFront can cache OPTIONS responses
CloudFront can drop or overwrite some of the request and response headers
Authorization, Host and other headers are used by AWS in communication between CloudFront and Lambda and may conflict with the same headers used by the web client

CORS headers can be added in three different places:

by the code inside the lambda function
by AWS if Lambda Function URL CORS were configured
by CloudFront via Response Headers Policy

Origin access control settings tell CloudFront if it should sign requests sent to the Lambda Function URL.
Signing requests takes over the Authorization header and drops the value sent by the client app.

Caching policy settings of a Behavior tell CloudFront which headers to use as caching keys:

headers included in the caching key are passed onto the lambda
you have to include your authorization key to avoid serving one user's response to another user
even if you include the Authorization header in the key it may be taken over by the AWS signature configured in the Origin access control
you do not have to have a caching policy for the CORS protocol to work
AWS recommends CachingDisabled policy for Lambda URLs

Origin request policy settings tell CloudFront which headers to forward to the lambda:

not all headers are forwarded by default
you should not forward the Host header (it returns an error at runtime if you do)
CloudFront may drop or replace some headers
this policy depends on the options selected in the Caching Policy
AllViewerExceptHostHeader policy works fine with CORS and is the default choice for Lambda URLs

Response headers policy tells CloudFront what headers to add to the response:

you can choose None if the lambda function handles the CORS response
you can choose a suitable managed CORS policy from the list to complement or overwrite the lambda's response
you can create a custom policy to complement or overwrite the lambda's response

Header quotas set limits to how much data you can put into custom headers:

Max length of a header value: 1,783 characters
Max length of all headers: 10,240 characters

Lambda Function URL configuration for CloudFront

Let's assume that you already have a working Lambda function with a configured URL for public access and we need to make it work with CloudFront.

Lambda URL access control config

The following config lets CloudFront access a Lambda function via its URL in the most secure way.

We will add the CloudFront IAM policy to the Lambda at a later stage.

Setting your function URL access control to Auth type: NONE allows anyone, including CloudFront to invoke it.

Lambda URL CORS headers

Enabling CORS headers in Lambda Function URL settings intercepts all HTTP OPTIONS requests sent to the function and returns the configured response.

Disabling CORS headers forwards HTTP OPTIONS requests to the function handler.

CORS headers can be configured in CloudFront's Response Headers Policy.

CloudFront can cache HTTP OPTIONS responses regardless of which of the above strategies of handling CORS you choose.

CloudFront configuration

A Lambda function should be added as an Origin of a CloudFront distribution.

In this example, client-sync lambda origin is linked to /sync.html path. For example, if a client app makes a fetch() call to https://d9tskged4za87.cloudfront.net/sync.html the request will be forwarded to client-sync lambda for processing.

Origin configuration

origin domain: copy-paste the domain of the Lambda Function URL, e.g. mq75dt64puwxk3u6gjw2rhak4m0bcmmi.lambda-url.us-east-1.on.aws
origin path: leave blank
name: give it an informative name, spaces are OK; it will not be used as an ID

Origin access control affects the use of Authorization header

Create a new Origin Access Policy that can be shared between multiple lambda functions.

Signing CloudFront requests to the lambda function URL also uses the Authorization header which can create a conflict if the web client uses the Authorization header for its authorization.

Do not sign requests

CloudFront passes the Authorization header from the browser to the lambda
the lambda must have Auth type: NONE and a public access policy enabled

Sign requests

Authorization header from the browser is dropped
CloudFront uses the Authorization header to sign requests to the lambda
Authorization header is excluded from the lambda's payload
the lambda can have Auth type: AWS_IAM and no public access

Do not override authorization header
This is a combination of the two previous policies:

sign if there is no Authorization header coming from the browser
do not sign if there is one coming from the browser
the lambda must have Auth type: NONE and a public access policy enabled for this option to work

The Origin Access policy can be edited later from the sidebar under Security / Origin access menu.

Lambda access policy

Copy-paste and run the template generated by the console:

If you go back to your lambda function after running the aws lambda add-permission command, you should see the newly added policy:

The policy should have your account and distribution IDs:

Behavior configuration

Create a new behavior with the path to the endpoint that invokes the Lambda (1) and the name of origin you created for the lambda earlier (2) as in the following example:

The address of this endpoint would be https://[CF distr ID].cloudfront.net/sync.html or https://[custom domain]/sync.html.

You must select an option with the HTTP OPTIONS method for CORS to work.

Caching OPTIONS requests prevents sending repeat OPTIONS requests to the lambda. This option depends on the selection of the caching policy discussed further in this guide.

In the policies section of the form select:

Cache policy CachingDisabled
Origin request policy AllViewerExceptHostHeader
Response headers policy:
- select None if the Lambda Function URL CORS option was configured, or
- create a new policy (more on this is further in the document), select None for now anyway

Policy selection

The above configuration should be sufficient to correctly process CORS and authorization requests to a lambda function via CloudFront.

CloudFront policies and CORS headers

Caching policies

Caching disabled

AWS recommends disabling caching for Lambda functions because most of the requests are unique and should not be cached.

Some headers, including the Authorization header, are removed by CloudFront if caching is disabled.

Caching enabled with Authorization header

Authorization header gets special treatment from AWS:

CloudFront does not pass it to lambda functions for GET/HEAD requests unless it is included in the caching policy
CloudFront forwards it for DELETE, PATCH, POST, and PUT HTTP methods

The above rule is affected by a different setting:

AWS uses the Authorization header for IAM authentication, so even if you have Auth type: AWS_IAM in the Function URL settings the Authorization header will not reach the lambda no matter what the caching policy says

One workaround for passing Authorization header to lambdas is to:

disable Lambda Function URL IAM authentication with Auth type: NONE,
add Authorization header to the caching policy.

If disabling Lambda Function URL IAM authentication is not an option:

keep Auth type: AWS_IAM in the Function URL settings
use a custom header to pass the authentication value from your web client to your lambda, e.g. x-myapp-auth: Bearer some-long-JWT-value
always add the custom authorization header to the caching policy to prevent responses to one authenticated user served to someone else

Origin request policy

Use AllViewerExceptHostHeader or create a custom policy that excludes the Host header from being forwarded to lambda.

The AllViewer- part of the name of that origin request policy is misleading because CloudFront removes and repurposes some of the headers, e.g. Authorization and many others.

Response headers policy

Use this policy only if you want CloudFront to add CORS and other headers to the response on top or instead of what the lambda returns in its response. For example, this sample policy adds all the necessary CORS headers to let scripts running in your local DEV environment (https://localhost:8080) call the lambda function:

Use one of the AWS-managed CORS policies that add the Access-Control-Allow-Origin: * header to allow scripts from any domain to access your lambda function.

References

AWS Lambda

CloudFront

CORS

AWS Lambda Function URL with CORS explained by example

Max — Wed, 14 Aug 2024 00:25:55 +0000

This post explores different configuration options for invoking AWS Lambda functions via a URL.

Overview

Why we need Cross-Origin Resource Sharing (CORS) for Lambda Function URLs

AWS Lambda Functions can be invoked via a public AWS Lambda Function URL with an HTTP call from a browser script, e.g.

const lambdaResponse = await fetch(
  "https://mq75dt64puwxk3u6gjw2rhak4m0bcmmi.lambda-url.us-east-1.on.aws/",
  {
    headers: {
      Authorization: `Bearer ${token}`,
    },
  },
);

Since the calling script and the lambda URL are running on different domains, the web browser would require the right CORS headers (also called CORS protocol) to be present for the call to succeed.

The CORS protocol for the above fetch() involves sending an HTTP OPTIONS request to the lambda with these three CORS headers to confirm that the lambda accepts requests from the script's domain:

Access-Control-Request-Method: GET
Access-Control-Request-Headers: authorization
Origin: https://localhost:8080

The lambda is expected to respond with this set of headers:

Access-Control-Allow-Methods: GET
Access-Control-Allow-Headers: authorization
Access-Control-Allow-Origin: https://localhost:8080

Option 1: Returning CORS from the lambda function handler

It is not hard to return a few headers from your lambda handler. For example, these 4 lines of Rust code do the job:

    let mut headers = HeaderMap::new();
    headers.append("Access-Control-Allow-Origin", HeaderValue::from_static("https://localhost:8080"));
    headers.append("Access-Control-Allow-Methods", HeaderValue::from_static("GET"));
    headers.append("Access-Control-Allow-Headers",HeaderValue::from_static("authorization"));

On the other hand, it is an extra coding, testing and maintenance effort. Any changes to the CORS settings would require code changes as well.

If your client app sends one OPTIONS request for every GET/POST it doubles the number of invocations.
See the Access-Control-Max-Age header to reduce repeat OPTIONS calls by the same client.

Returning CORS from the lambda function handler may be a good option if you need extra control over processing HTTP OPTIONS requests.

Option 2: Configuring CORS headers in AWS Lambda settings

Function URLs can be configured to let AWS handle CORS preflight requests (HTTP OPTIONS method) and add necessary headers to all the other HTTP methods after your lambda returns its response.

This is a simpler and more reliable option that does not require any changes to the function handler.

Custom domain names for Lambda URLs

It is not possible to use custom domain names with Lambda URLs because the web server that invokes the lambda relies on the Host HTTP header to identify which lambda to invoke, e.g. host: mq75dt64puwxk3u6gjw2rhak4m0bcmmi.lambda-url.us-east-1.on.aws.

CNAME example

lambda URL: mq75dt64puwxk3u6gjw2rhak4m0bcmmi.lambda-url.us-east-1.on.aws
CNAME record: lambda.example.com CNAME mq75dt64puwxk3u6gjw2rhak4m0bcmmi.lambda-url.us-east-1.on.aws
host header sent to AWS: host: lambda.example.com

The web server handling that request at the AWS end would not know which lambda function to invoke by looking at host: lambda.example.com and would return an error or time out.

Use ApiGateway or CloudFront as the proxy if using lambda Function URLs is not an option.

Access control configuration for Lambda Function URL

Let's assume that our function is exposed to the internet and does not require AWS IAM authentication.

The following config allows public access to the function via its URL.

You need both the Auth type: NONE setting and a resource-based policy statement to allow public access.
Having one setting without the other results in 403 Permission Denied.

AWS automatically adds the required FunctionURLAllowPublicAccess access policy to the Lambda function when you choose Auth type: NONE in the console.

Request/response examples for different configuration options in detail

This section contains examples of HTTP headers exchanged between the web browser, AWS and the lambda function to help us understand how different configuration options affect the headers.

Example 1: Lambda URL request/response without CORS protocol

Enable public access to your lambda function URL as explained earlier
Do not enable CORS settings
Send a request to the lambda URL displayed in the config screen

Your Function URL config should look similar to this:

A sample lambda URL call that does not require the CORS protocol:

const lambdaResponse = await fetch("https://mq75dt64puwxk3u6gjw2rhak4m0bcmmi.lambda-url.us-east-1.on.aws/");

Your request may look similar to this example:

GET / HTTP/1.1
Host: mq75dt64puwxk3u6gjw2rhak4m0bcmmi.lambda-url.us-east-1.on.aws
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Priority: u=4
Pragma: no-cache
Cache-Control: no-cache

The lambda function receives all the headers sent by the browser as part of the request payload and some additional AWS headers (see 6 headers starting with x- at the end of the list):

{
  "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8",
  "accept-encoding": "gzip, deflate, br, zstd",
  "accept-language": "en-US,en;q=0.5",
  "authorization": "foo-bar",
  "cache-control": "no-cache",
  "dnt": "1",
  "host": "mq75dt64puwxk3u6gjw2rhak4m0bcmmi.lambda-url.us-east-1.on.aws",
  "pragma": "no-cache",
  "priority": "u=4",
  "sec-fetch-dest": "document",
  "sec-fetch-mode": "navigate",
  "sec-fetch-site": "same-origin",
  "sec-fetch-user": "?1",
  "upgrade-insecure-requests": "1",
  "user-agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0",
  "x-amzn-tls-cipher-suite": "TLS_AES_128_GCM_SHA256",
  "x-amzn-tls-version": "TLSv1.3",
  "x-amzn-trace-id": "Root=1-66b44063-116ea8667f4b70e405b1b19a",
  "x-forwarded-for": "222.154.108.14",
  "x-forwarded-port": "443",
  "x-forwarded-proto": "https"
}

Function URL configuration with CORS: not enabled option invokes the lambda for all HTTP methods, including OPTIONS and passes all browser headers to the lambda.

Use this configuration if your lambda handles responses to the HTTP OPTIONS method with the CORS protocol.

Example 2: CORS headers added by lambda handler

This example has the same Lambda URL configuration as in the previous example:

This time we include an optional Authorization header to trigger the CORS protocol:

const lambdaResponse = await fetch("https://mq75dt64puwxk3u6gjw2rhak4m0bcmmi.lambda-url.us-east-1.on.aws/",
  { headers: { Authorization: `Bearer ${token}` } },
);

The new request/response flow:

a browser script running on https://localhost:8080 attempts to fetch() from the lambda's URL
the browser initiates the CORS protocol by sending an HTTP OPTIONS request to the lambda
the lambda replies with the necessary CORS headers
the browser sends the GET request

The HTTP OPTIONS request would be very similar to the previous example with the addition of a few CORS headers:

origin: https://localhost:8080
access-control-request-headers: authorization
access-control-request-method: GET

This OPTIONS request can be translated as the browser asking our lambda: can I send you a GET request with the authorization header from a web page located at https://localhost:8080?

If the lambda replies Yes, the browser sends the GET request. If the lambda replies something other than Yes, the GET request is never sent and the script gets a CORS error.

Our lambda would see this list of headers for the above OPTIONS request (see 3 lines at the end for CORS):

{
  "accept": "*/*",
  "accept-encoding": "gzip, deflate, br, zstd",
  "accept-language": "en-US,en;q=0.5",
  "cache-control": "no-cache",
  "dnt": "1",
  "host": "mq75dt64puwxk3u6gjw2rhak4m0bcmmi.lambda-url.us-east-1.on.aws",
  "pragma": "no-cache",
  "priority": "u=4",
  "sec-fetch-dest": "empty",
  "sec-fetch-mode": "cors",
  "sec-fetch-site": "cross-site",
  "user-agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0",
  "x-amzn-tls-cipher-suite": "TLS_AES_128_GCM_SHA256",
  "x-amzn-tls-version": "TLSv1.3",
  "x-amzn-trace-id": "Root=1-66b42e1d-56f894f802ecd9bc345ef57a",
  "x-forwarded-for": "222.154.108.14",
  "x-forwarded-port": "443",
  "x-forwarded-proto": "https",
  "origin": "https://localhost:8080",
  "access-control-request-headers": "authorization",
  "access-control-request-method": "GET"
}

Assuming that our lambda function can handle CORS protocol and is happy with the request, the response would have the necessary CORS headers starting with access-control-allow-* to tell the browser that the lambda is happy to receive the GET request, as in this example (see 4 lines at the end for CORS):

HTTP/1.1 200 OK
Date: Thu, 08 Aug 2024 18:18:41 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 5
Connection: keep-alive
x-amzn-RequestId: 9f50a878-597b-4fd9-8a07-717e078a2ab0
X-Amzn-Trace-Id: root=1-66b50c01-207a612c605445d738495a39;parent=5ff8d4e2feb9297b;sampled=0;lineage=a964c7ca:0
access-control-allow-origin: https://localhost:8080
access-control-allow-headers: authorization
access-control-allow-methods: GET
access-control-allow-credentials: true

The browser receives the above response and follows with HTTP GET asking the lambda to do some work, e.g. return some data.
The GET request contains the Authorization header with a truncated JWT token (see the last line):

GET / HTTP/1.1
Host: mq75dt64puwxk3u6gjw2rhak4m0bcmmi.lambda-url.us-east-1.on.aws
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
Origin: https://localhost:8080
DNT: 1
Connection: keep-alive
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
Priority: u=0
Pragma: no-cache
Cache-Control: no-cache
Authorization: Bearer eyJhbGci...ba3mp4OQ

The lambda does its work and returns some payload with the following headers:

HTTP/1.1 200 OK
Date: Thu, 08 Aug 2024 22:44:38 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 22
Connection: keep-alive
x-amzn-RequestId: 7307ef70-161c-431c-a7ac-b6ef4838549e
Vary: Origin
X-Amzn-Trace-Id: root=1-66b54a56-2822789c217e2f0847d6b03f;parent=0e288bd87a583781;sampled=0;lineage=a964c7ca:0
Access-Control-Allow-Origin: https://localhost:8080
Access-Control-Allow-Credentials: true

The above response has to contain Access-Control-Allow-Origin: https://localhost:8080 and Access-Control-Allow-Credentials: true headers for the browser to accept it (2 lines at the end).

Example 3: CORS request/response headers added by AWS

This example shows how CORS headers can be configured out of the lambda handler:

allow HTTP GET/POST requests containing Authorization and other headers from https://localhost:8080
allow sending credentials

The client calls the lambda URL with the same fetch() as before:

const lambdaResponse = await fetch("https://mq75dt64puwxk3u6gjw2rhak4m0bcmmi.lambda-url.us-east-1.on.aws/",
  { headers: { Authorization: `Bearer ${token}` } },
);

The browser does the same CORS protocol as before with the same CORS headers:

Unlike the previous example where the OPTIONS request was handled by the lambda handler, no lambda invocation takes place for HTTP OPTIONS requests which are handled by AWS.

This response was generated by AWS and contains the settings configured in the CORS section of the Function URL:

HTTP/1.1 200 OK
Date: Thu, 08 Aug 2024 18:53:07 GMT
Content-Type: application/json
Content-Length: 0
Connection: keep-alive
x-amzn-RequestId: 215426b9-920c-4d1f-b994-54ccd29b2612
Vary: Origin
Access-Control-Allow-Origin: https://localhost:8080
Access-Control-Allow-Headers: authorization,content-type,x-books-authorization
Access-Control-Allow-Methods: GET,POST
Access-Control-Allow-Credentials: true

The last four lines of response tell the browser it may continue with more GET/POST requests.

Unlike the previous example, the lambda handler was not involved in handling the CORS protocol - it was processed by AWS outside of the function handler.
This is an easier way than handling CORS inside the lambda handler.

A few "gotchas"

This section lists a few minor things that can suck up a lot of your time.

Add one header per line in the CORS configuration form

Since the HTTP headers are sent as a comma-separated list it seems logical to enter the entire list in a single Allow headers box (red highlight).
AWS will let you save the invalid config and produce an incorrect response to the OPTIONS request later.

Enter one header per line (green highlight). Remember that header names are case-insensitive.

Don't allow `localhost` CORS in production

It is not trivial to exploit this, but it is possible if the site has XSS vulnerabilities or a reverse proxy is involved.

See https://stackoverflow.com/questions/39042799/cors-localhost-as-allowed-origin-in-production for detailed explanations.

Adding and deleting FunctionURLAllowPublicAccess access policy

FunctionURLAllowPublicAccess policy is added by AWS when you choose Auth type: NONE for the Function URL.

It is not removed if you change to Auth type: AWS_IAM, but the public access is no longer available. See AWS docs for more details.

Removing FunctionURLAllowPublicAccess policy while Auth type: NONE blocks public access to the lambda's URL.

Header double-up

AWS adds access-control-allow-origin and access-control-allow-credentials CORS headers regardless of their presence in the response from the lambda.

If our lambda returned CORS headers and the Function URL was configured to return CORS as well, the response would become invalid because access-control-allow-origin and access-control-allow-credentials would be included twice (see 4 lines at the end):

HTTP/1.1 200 OK
Date: Thu, 08 Aug 2024 18:53:09 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 22
Connection: keep-alive
Vary: Origin
x-amzn-RequestId: f1d1d6ce-1208-450d-8c61-e5a51f878090
X-Amzn-Trace-Id: root=1-66b51414-3998ea28145bd0d8661c065f;parent=7df2ad2039925c82;sampled=0;lineage=a964c7ca:0
access-control-allow-headers: x-books-authorization,authorization,content-type
access-control-allow-methods: GET, OPTIONS, POST
access-control-allow-origin: https://localhost:8080
access-control-allow-origin: https://localhost:8080
access-control-allow-credentials: true
access-control-allow-credentials: true

References

An awesome AWS Lambda debugging tool I used to experiment and capture requests and responses: Github / Crates.io
AWS Lambda CORS docs: https://docs.aws.amazon.com/lambda/latest/dg/urls-configuration.html?icmpid=docs_lambda_help#urls-cors
CORS overview on MDN: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
CORS protocol spec: https://fetch.spec.whatwg.org/#http-cors-protocol
CORS request headers
Required CORS response headers
Optional CORS response headers

Passwordless login to Postgres from VSCode SQLTools extension using IDENT/PEER method

Max — Wed, 31 Jan 2024 19:16:29 +0000

This post explains how to connect to PostgreSQL using the current Linux user account with IDENT/PEER authentication method.

Postgres version: 16
SQLTools extension version: 0.28.1

This may apply to other Postgres and SQLTools versions. Replace the version number in the paths and URLs as needed.

About IDENT/PEER Authentication methods

IDENT/PEER authentication method maps the currently logged-in Linux user name to the Postgres user name without a need for storing the password in Postgres or VSCode configuration files.

With IDENT/PEER auth, instead of receiving the login and password from the user, Postgres gets the OS user name of the caller and maps it to the user name stored in its pg_authid system catalog.
This method is enabled by default in /etc/postgresql/16/main/pg_hba.conf file:

# Database administrative login by Unix domain socket
local   all             postgres                                peer

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     peer

For the PEER auth to work we need to connect via sockets, not TCP/IP.
A socket is a logical endpoint for communication that bypasses the networking hardware.

Postgres has a default socket at either: /run/postgresql or /var/run/postgresql.
The exact location is configured in /etc/postgresql/16/main/postgresql.conf file:

unix_socket_directories = '/var/run/postgresql'

References:

Install Postgress with default settings

Skip this section if you already have Postgres installed.

# add the PostgreSQL 16 repository
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'

# import the repository signing key
curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/postgresql.gpg

sudo apt update

# install PostgreSQL 16
sudo apt install postgresql-16

# start and enable PostgreSQL service
sudo systemctl start postgresql
sudo systemctl enable postgresql

# check the installed version
psql --version

You should be able to connect to Postgres via psql utility with this command:

sudo -u postgres psql

References:

Add your current Linux user to Postgres

Postgres creates a new Linux user called postgres to match the default superuser name in Postgres.
For IDENT/PEER authentication to work you need to create a Postgres user matching your OS username:

sudo -u postgres createuser --superuser [your_linux_username]

You should be able to connect to Postgres without a need for sudo -u postgres from then on. For example, running psql -d postgres command should print something like this:

psql (16.1 (Ubuntu 16.1-1.pgdg23.04+1))
Type "help" for help.

postgres=#

Note that postgres=# prompt means you are a superuser. Otherwise, it would be postgres=> for less privileged Postgres users.

References:

Connecting VS Code to Postgres

Install SQLTools extension for VSCode if you have not done so.

Do not attempt to use the UI provided by the extension to configure PEER access.
Open settings.json and add the following section:

"sqltools.connections": [
    {
      "name": "PGSQL PEER TEST",
      "server": "/run/postgresql",
      "driver": "PostgreSQL",
      "database": "postgres"
  }
]

where

name - whatever you name it
server - leave as in the example for a default installation on Linux
driver - leave as is
database - use your DB name, if you have one, but "postgres" DB is a good option for the test because it's guaranteed to be there

Save the settings and you should see a new connection option appear in the SQLTools sidebar.

Try connecting to the server:

click on Connect icon against the connection you've just added
you should get a popup saying The extension wants to sign in using SQLTools Driver Credentials
click on Allow and enter the password for the current Linux user

The extension saves the password via VSCode SecretStorage API which is protected by the OS keyring. It is much more secure than storing it withing VSCode settings.

You can choose to save it permanently or only for the duration of the VSCode session by clicking on the Key icon in the password dialog. You will have to enter the password again after restarting VSCode if you choose not to save it.

References:

Chrome extension example with Rust and WASM

Max — Mon, 08 Jan 2024 01:42:47 +0000

This post explains the inner workings of a Chrome extension built with Rust and compiled into WASM.

Target audience: Rust programmers interested in building cross-browser extensions with WASM.

About the extension: Github, Chrome Webstore, Mozilla Webstore

What you will learn:

toolchain
architecture of the extension
how WASM, background and content scripts communicate with each other
intercepting session tokens to impersonate the user
debugging
making it work for Chrome and Firefox
listing in Google and Mozilla addon stores

You will get the best value for your time if you look at the source code and read this post side by side.

Toolchain

The main tool to build, test and publish WASM: https://rustwasm.github.io/docs/wasm-pack/

cargo install wasm-pack

See build.sh for an example of build command.

cargo installs wasm-bindgen on the first compilation to facilitate high-level interactions between wasm modules and JavaScript.

References:

https://rustwasm.github.io/wasm-bindgen/ - Rust/WASM book with examples, explanations and details references. Read about:
- Hello, World!
- console.log
- Without a bundler
https://rustwasm.github.io/wasm-bindgen/api/web_sys/index.html - a Rust wrapper around Web APIs. APIs of interest:
- Service Workers API
- Fetch API
https://rustwasm.github.io/wasm-bindgen/api/js_sys/index.html - a Rust wrapper around JS types and objects

Architecture of the extension

This extension consists of several logical modules:

manifest - the definition of the extension for the browser
background script - a collection of JS scripts and WASM code that run in a separate browser process regardless of the current web page
content script - the part of the extension that runs inside the current web page, but it is not used in this extension
popup - a window displayed by the browser when the user clicks on the extension button in the toolbar

Folders

extension: the code of the extension as it is loaded into the browser, including JS and WASM
media: various media files for publishing the extension, not essential
wasm_mod: Rust code to be compiled into WASM
wasm_mod/samples: Spotify request/response captures, not essential

Manifest V3

We have to use separate manifest V3 files for different browsers because of incompatible features:

extension/manifest_cr.json: Chrome version
extension/manifest_ff.json: Firefox version

Browser-specific manifests are renamed into manifest.json by build.sh as explained in Debugging and Packaging sections of this document.

List of manifest properties to pay attention to:

action/show_matches - when the browser should make the extension button active
action/default_popup - what should happen when the user clicks on the extension button
background/service_worker - the name of the script to run as a background service worker
content_security_policy - declares what the extension can do, e.g. load scripts or WASM

Other properties are either self-explanatory or not as important.

Manifest V3 docs: MDN / Chrome

Background script

extension/js/background.js acts as a Service Worker. The name "background script" is historical and can be used interchangeably with "service worker".

What background.js does:

loads and initializes WASM module
listens to messages from WASM and the popup page (extension/js/popup.html)
sends error messages to the popup page (extension/js/popup.js)
captures session token (captureSessionToken())
fetches user details (fetchUserDetails())
extracts the current playlist ID from the active tab URL (getPlaylistIdFromCurrentTabUrl())
calls WASM functions in response to user actions (add_random_tracks())
controls the extension icon in the toolbar (toggleToolbarBadge())

background.js is loaded when the browser is started and performs:

WASM module initialisation
adding listeners for messaging and token capture

Once running, background.js and its WASM part continue to run independently of the page it was started from. A service worker will continue running even if the user navigates elsewhere or closes the tab that started it because the Service Worker lifecycle is separate from that of the document.

Popup page

The manifest file instructs the browser to open a popup window when the extension is activated by the user, e.g. with a click on the extension icon in the toolbar.

This line of the manifest defines what popup window to open when the user clicks on the toolbar button:

"default_popup": "js/popup.html"

The popup lives only while it is being displayed. It cannot host any long-running processes and it does not retain the state from one activation to the other. All DOMContentLoaded events fire every time it is activated.

What popup.js does:

attaches event handlers to its buttons and links
handles on-click for links because browsers do not open link URLs from popups (see chrome.tabs.create())
listens to messages from background.js and WASM (see chrome.runtime.onMessage.addListener())
displays an activity log from the messages it receives from WASM and background.js

Any inline JS in popup.html is ignored. Any JS code or event handlers have to be loaded externally with <script type="module" src="popup.js"> because only src scripts are allowed by content_security_policy/extension_pages entry of the manifest.

Content script

Content scripts interact with the UI and DOM. They run in the same context as the page and have access to Window object as their main browser context.

This extension does not interact with the Spotify tab and does not need a content script.

Messaging between the scripts

Communication between different scripts (background, popup, WASM) is done via asynchronous message passing. A script A sends a message to a shared pool hoping that the intended recipient is listening and can understand the message. This means that if there are multiple listeners, all of them get the message notification. The sender does not get notifications for the messages it sends out.

It is possible to invoke functions from a different script within the same extension, but the script will run in the context of the caller. For example, popup.js can call a background.js function when the user clicks on a button inside the popup. The invocation will work in the context of the popup and die as soon as the popup is closed.

On the other hand, background.js is a long-running process. It listens to messages sent to it from other scripts. E.g. chrome.runtime.onMessage.addListener() function checks the message payload and acts depending on the message contents.

This extension relies on messaging, not direct invocation.

Key messaging concepts:

messages are objects
the only metadata attached to the message is the sender's details
structure the message to include any additional metadata, e.g. the type of the payload
always catch chrome.runtime.sendMessage() errors because there is no guarantee of delivery and an unhandled error will fail your entire script
an error is always raised if you send a message and there is no active listener, e.g. if you expect a popup to listen, but the user closed it
message senders cannot receive their own messages, so if you send and listen within the same script, there will be no message notification to self, only to other listeners

More on message passing between different scripts: https://developer.chrome.com/docs/extensions/develop/concepts/messaging.

Messaging examples

From background.js to popup.js

background.js sends error messages out to anyone who listens. It handles non-delivery errors by ignoring them.

popup.js listens and displays them if popup.html is open. If the popup is not open, there is no listener and the sender gets an error. These errors should be handled for the rest of the sender's script to work.

chrome.runtime.sendMessage("Already running. Restart the browser if stuck on this message.").then(onSuccess, onError);

where both onSuccess and onError do nothing to prevent the error from bubbling up to the top.

From popup.js to background.js

popup.js sends a message out when the user clicks Add tracks button.

background.js listens and invokes WASM to handle the user request.

The popup script could call a function in the background script to invoke WASM or even call WASM directly, but the popup lives only if open. Also, it wouldn't have access to the token stored in the context of the long-running background.js process.

So, we have a long-running background script that lives independently of the tabs or the popup. When a message from the popup arrives, background.js calls WASM and continues running even if the caller no longer exists.

Sending messages from WASM to JS scripts

WASM sends out messages via report_progress() function located in wasm_mod/src/progress.js script. That function is imported into wasm_mod/src/lib.rs as

#[wasm_bindgen(module = "/src/progress.js")]
extern "C" {
    pub fn report_progress(msg: &str);
}

and is called from other Rust functions as a native Rust function.

The progress reporting from WASM to the popup delivers messages in near-real-time while the WASM process continues running.

WASM

wasm_mod folder contains the WASM part of the extension.

Inside Cargo.toml

Cargo file: wasm_mod/Cargo.toml

crate-type = ["cdylib", "rlib"]

cdylib: used to create a dynamic system library, e.g. .so or .dll, but for WebAssembly target it creates a *.wasm file without a start function
rlib: optional, used to create an intermediate "Rust library" for unit testing with wasm-pack test

More on targets: https://doc.rust-lang.org/reference/linkage.html

Dependencies

wasm-bindgen: contains the runtime support for #[wasm_bindgen] attribute, JsValue interface and other JS bindings (crate docs)
js-sys: bindings to JS types, e.g. Array, Date and Promise (crate docs)
web-sys: raw API bindings for Web APIs, 1:1, e.g. the browser Window object is web_sys::Window with all the methods, properties and events available from JS (crate docs)

Remember to add WebAPI classes and interfaces as web-sys features in your Cargo.toml. E.g. if you want to use Window class in Rust code it has to be added to web-sys features first.

More about how Rust binds to browser and JS APIs: https://rustwasm.github.io/wasm-pack/book/tutorials/npm-browser-packages/template-deep-dive/cargo-toml.html

Calling WASM from JS example

background.js makes a call to hello_wasm() from lib.rs that logs a greeting in the browser console for demo purposes.

`hello_wasm()` sequence diagram

How `console.log()` is called from Rust

To log something into the browser console our Rust code has to access the logging function of WebAPI provided by the browser.
lib.rs imports WebAPI console.log() function and makes it available in Rust:

#[wasm_bindgen]
extern "C" {
    #[wasm_bindgen(js_namespace=console)]
    fn log(s: &str);
}

The low-level binding is done by web-sys and js-sys crates with the help of wasm-bindgen and wasm-pack.

How Rust's `hello_wasm()` is called from JS

To call a Rust/WASM function from JS we need to export it with #[wasm_bindgen].

lib.rs exports hello_wasm() Rust function and makes it available to background.js:

#[wasm_bindgen]
pub fn hello_wasm() {
    log("Hello from WASM!");
}

wasm-bindgen and wasm-pack generate wasm_mod.js file every time we run the build. That file is the glue between WASM and JS. Some of its high-level features:

initializes the WASM module
exports wasm.hello_wasm() function for background.js
type and param checking

How WASM is initialized

Every time the browser activates the extension, it runs the following code from background.js:

import initWasmModule, { hello_wasm } from './wasm/wasm_mod.js';
(async () => {
    await initWasmModule();
    hello_wasm(); 
})();

wasm-bindgen and wasm-pack generate the initialization routine and place it in wasm_mod.js file as the default export.

lib.rs

lib.rs is the top-level file for our WASM module. It exposes Rust functions and binds to JS interfaces.

Main entry point

The main entry point that does the work of adding tracks to the current playlist is pub async fn add_random_tracks(...).

Browser context

The code will need access to some global functions that are only available through an instance of a browser context or runtime. The two main classes that serve this purpose are WorkerGlobalScope and Window.

The important difference between the two is that:

WorkerGlobalScope is available to Service Workers (background scripts running in their own process independent of a web page)
Window is available to content scripts running in the same process as the web page

Both classes are very similar, but have slightly different sets of properties and methods.

lib.rs obtains a reference to the right browser runtime inside async fn get_runtime() and passes it to other functions.

get_runtime() attempts to get a reference to WorkerGlobalScope first. If that fails it tries to get a reference to Window object. One works in Chrome and the other in Firefox.

Reporting progress back to the UI

Our WASM code may take minutes to run. It reports its progress and failures back to the UI via browser messaging API.

pub fn report_progress(msg: &str) in lib.rs is a proxy for report_progress() in progress.js.
It is called from various locations in our Rust code to send progress and error messages to popup.js.

To make a custom JS function available to Rust code we created wasm_mod/src/progress.js with the following export:

export function report_progress(msg) {
  chrome.runtime.sendMessage(msg).then(handleResponse, handleError);
}

and matched it by creating this import in lib.rs:

#[wasm_bindgen(module = "/src/progress.js")]
extern "C" {
    pub fn report_progress(msg: &str);
}

wasm-bindgen and wasm-pack generate necessary bindings and copy progress.js to extension/js/wasm/snippets/wasm_mod-bc4ca452742d1bb1/src/progress.js.

other .rs files

The rest of .rs files contain vanilla Rust that can be compiled into WASM.

client.rs - a single high-level function that brings everything together
api_wrappers.rs - a collection of wrappers for Spotify API, called from client.rs
constants.rs - shared constants, utility functions
models.rs - Rust structures for Spotify requests and responses

Token capture

The extension needs some kind of credentials to communicate with Spotify on behalf of the user. One way of achieving this without asking the user is to capture the current session token. For that to work, the user has to navigate to a Spotify web page and log in there.

A frequent Spotify user would already be logged in, because Spotify maintains an active session on all *.spotify.com pages.

This extension captures all request headers sent to https://api-partner.spotify.com/pathfinder/v1/query endpoint, including the tokens. The headers are stored in local variables and are copied into requests made by the extension to mimic the Spotify app.

captureSessionToken() function from background.js does the header extraction when onBeforeSendHeaders event is triggered:

chrome.webRequest.onBeforeSednHeaders.addListener(captureSessionToken, { urls: ['https://api-partner.spotify.com/pathfinder/v1/query*'] }, ["requestHeaders"])

onBeforeSendHeaders listener reads, but does not modify the headers on https://api-partner.spotify.com/pathfinder/v1/query*.
The * at the end of the URL is necessary for the pattern to work.
["requestHeaders"] param instructs the browser to include all the headers in the request details object passed onto captureSessionToken handler. Only the URL and a few common headers are included If that param is omitted.
host_permissions": ["*://*.spotify.com/*"] in manifest.json is required for onBeforeSendHeaders to work.

All extracted headers are stored in headers variable inside background.js. The tokens are stored in auth and token vars. None of the headers or tokens are persisted in storage.

The tokens are passed onto WASM as function parameters:

#[wasm_bindgen]
pub async fn add_random_tracks(
    auth_header_value: &str,
    token_header_value: &str,
    playlist_id: &str,
    user_uri: &str,
)

Debugging

Extensions can be loaded from source code in Firefox and Chrome for testing and debugging.

Firefox

run . build.sh to build the WASM code
go to about:debugging#/runtime/this-firefox
click on Load temporary Add-on button
Firefox opens a file selector popup asking to select manifest.json file. Remember to rename manifest_ff.json into manifest.json.

If the extension was loaded correctly, its details should appear inside the Temporary Extensions section on the same page.

If you keep losing this obscure about:debugging#/runtime/this-firefox URL, there is an alternative way of getting to the debugging page in Firefox:

click on extensions icon in the toolbar
click on Manage extensions
click on settings gear-like icon at the top right of the page
click on Debug Add-ons

Chrome

The Chrome process is very similar to the one in Firefox.

run . build.sh to build the WASM code
go to chrome://extensions/
Turn on Developer mode toggle at the top-right corner of the page
Cick on Load unpacked and select the folder with manifest.json. Remember to rename manifest_cr.json into manifest.json.

If the extension was loaded correctly, its details should appear in the list of extensions on the same page. See Chrome docs for more info.

If you forget what the direct URL is (chrome://extensions/), there is an alternative way of getting to the debugging page in Chrome:

click on ... icon in the Chrome toolbar to open the Chrome options menu
click on Extensions / Manage Extensions
Chrome will open chrome://extensions/ page

Making changes and reloading

Changes to code located inside wasm_mod/src folder require running build.sh script to rebuild the WASM code.

Changes to JS, CSS or asset files inside extension folder do not require a new build and are picked up by the browser when you reload the extension.

Click on Reload icon for the extension to load the latest changes.

For example, changing lib.rs or progress.js requires a rebuild and reload. Changing popup.js only requires a reload.

Viewing logs

Different modules log messages into different consoles. If you are not seeing your log message, you are probably looking at the wrong console.

Content scripts output log messages (e.g. via console.log()) to the same log as the web page. View them in DevTools / Console tab for the web page.

Background scripts, including WASM, send messages to a separate console log.

in Chrome: click on Inspect views link in the extension details panel (e.g. Inspect views service worker (Inactive))

in Firefox: click on Inspect button in the extension details panel

If you follow the steps above, Firefox should open a new DevTools window with Console, Network and other tabs.

Popup windows (e.g. popup.html) in Firefox log messages into the same console as the background scripts.

The popup window in Chrome also logs into the same console as the background script, but you have to enable it with a separate step.

Right-click on the open popup and select Inspect. It will open a new DevTools window with both background and popup logs. Closing the popup window kills the DevTools window as well.

Network requests and responses from background.js and WASM appear in the background DevTools window.

Cross-browser compatibility

Most of the extension code works in both, Firefox and Chrome. There are a few small differences that have to be kept separate: manifest, global context and host_permissions.

manifest.json

background.js lives under different manifest property names:

Firefox: background/scripts
Chrome: background/service_worker

Firefox complains about minimum_chrome_version, offline_enabled and show_matches.

Chrome rejects browser_specific_settings required by Firefox.

See the full list of manifest differences between Firefox and Chrome with this CLI command:

git diff --no-index --word-diff extension/manifest_cr.json extension/manifest_ff.json

This project is configured to have two manifests in separate files: manifest_cr.json and manifest_ff.json. Rename one of them manually into manifest.json for local debugging and then revert to the browser-specific file when finished. build.sh script renames them to manifest.json on the fly during packaging.

If a package has no manifest.json it is not recognized as an extension by the browser.

Global context

Firefox and Chrome use different global context class names to access methods such as fetch():

WorkerGlobalScope
Window

Both classes are part of the standardized WebAPI. The difference is in how they are used in content and background scripts.

Chrome

web_sys::window() function returns Some(Window) for content scripts and None for Service Workers
js_sys::global().dyn_into::<WorkerGlobalScope>() returns Ok(WorkerGlobalScope) for Service Workers and Err for content scripts

Firefox

web_sys::window() function returns Some(Window) for both, context and Service Worker scripts
js_sys::global().dyn_into::<WorkerGlobalScope>() always returns None

It looks like Firefox is not compliant with the standard or even MDN, but maybe I missed something and got this part wrong.

See get_runtime() function in lib.rs for more implementation details. Also, see Window and WorkerGlobalScope docs on MDN.

Host permissions

Both Chrome and Firefox have this entry in the manifest:

"host_permissions": [
    "*://*.spotify.com/*"
]

The format is the same, but Chrome grants this permission on install and Firefox treats it as an optional permission that has to be requested at runtime.

The extension code handles this discrepancy gracefully at the cost of some complexity. More info:

the implementation of btn_add click listener in popup.js has detailed comments
MDN docs: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/permissions/request
Firefox's strange behavior: https://stackoverflow.com/questions/47723297/firefox-extension-api-permissions-request-may-only-be-called-from-a-user-input

Listing WASM extensions in Google and Mozilla addon stores

There are no special listing requirements for extensions containing WASM code.

Both Google and Mozilla may request the source code and build instructions for the WASM part, which may delay the review process. This extension is 100% open source and was always approved within 24hrs.

Packaging

build.sh script packages the extension into chrome.zip and firefox.zip files. The packaging steps include:

building the WASM code with wasm-pack
removing unnecessary files
renaming browser-specific manifest files to manifest.json
zipping up extension folder into chrome.zip and firefox.zip with the right manifest

wasm-pack copies .js files from wasm_mod/src folder to extension/js/wasm/snippets/wasm_mod/src if they are used for binding to Rust. E.g. wasm_mod/src/progress.js is copied to extension/js/wasm/snippets/wasm_mod-bc4ca452742d1bb1/src/progress.js.

See detailed packaging instructions for Firefox and Chrome for more info.

Listing

The listing form has no WASM-specific questions or options. Make sure you do not have any unused permissions in the manifest and give clear explanations why you need the permissions you request.

Useful links

Firefox

Example: https://addons.mozilla.org/en-US/firefox/addon/spotify-playlist-builder-addon/
Sign up here: https://addons.mozilla.org/en-US/developers/
List your extensions: https://addons.mozilla.org/en-US/developers/addons
Uploading listing images may only be available after the extension is approved.

Chrome

Example: https://chromewebstore.google.com/detail/spotify-playlist-builder/kmbnbjbfpnchgmmkbeidpllpamcahljn
Sign up and list your extensions: https://chrome.google.com/webstore/devconsole

Listing process

Sign up, create a new listing, describe the extension and upload the .zip for review
Check any errors and warnings generated by the store website after uploading the .zip
Approval time for both stores is about 24 hours, from personal experience.
Any change in the code or the listing requires a store review before it is published

Both stores allow listing extensions without them appearing in the public directory while users with the extension URL can still access it.

Rust's lazy_static! usage benchmarks and code deep dive

Max — Sat, 02 Apr 2022 08:44:30 +0000

lazy_static is one of the foundational crates in the Rust ecosystem.
It lets us use static variables without an explicit initialization call.
I used it many times without giving its performance implications much thought. Putting it inside some deeply nested loop got me worried if all that lazy-static magic has some hidden cost.

The crate's docs explain the mechanics behind lazy_static! macro as:

The Deref implementation uses a hidden static variable that is guarded by an atomic check on each access.

That sounds innocuous enough, but I still have questions:

Is there any noticeable performance cost incurred by the atomic check on each access?
If lazy_static is used in a sub-module, will it be re-initialized on every call to a function from that module?
Is it any slower than initializing a variable manually and passing it to other functions as a parameter?

Without understanding the implementation details of lazy_static I figured it would be easier to benchmark it than to dig through its source code.

How to run

grab project's source: git clone https://github.com/rimutaka/empirical.git
benchmarks: cargo +nightly bench
tests: cargo +nightly test --benches

Results

$ cargo +nightly bench

test bad_rust_local           ... bench:      40,608 ns/iter (+/- 9,239)
test lazy_static_backref      ... bench:          27 ns/iter (+/- 1)
test lazy_static_external_mod ... bench:          27 ns/iter (+/- 0)
test lazy_static_inner        ... bench:          27 ns/iter (+/- 1)
test lazy_static_local        ... bench:          27 ns/iter (+/- 5)
test lazy_static_reinit       ... bench:          26 ns/iter (+/- 1)
test once_cell_lazy           ... bench:          26 ns/iter (+/- 2)
test vanilla_rust_local       ... bench:          27 ns/iter (+/- 0)

The results looked pretty neat. The only outlier was a piece of bad code I put in the benches intentionally to set the baseline.

TL;DR: `lazy_static!` is fine, but `once_cell` may be better for new projects.

Benchmarks in detail

`bad_rust_local()`

This bench does something obviously stupid - it recompiles the regex within the loop.

b.iter(|| {
    let compiled_regex = regex::Regex::new(LONG_REGEX).unwrap(); // <-- don't place this inside a loop
    let is_match = compiled_regex.is_match(TEST_EMAIL);
    test::black_box(is_match);
});

With 40,608 ns/iter it gives us the baseline for the recompilation cost.

`vanilla_rust_local()`

There was NO noticeable performance cost incurred by the atomic check on each access.

vanilla_rust_local bench compiled the regex once and took exactly the same 27 ns/iter as the benches using lazy_static.

let compiled_regex = regex::Regex::new(LONG_REGEX).unwrap(); // <-- compiled once only
b.iter(|| {
    let is_match = compiled_regex.is_match(TEST_EMAIL);
    test::black_box(is_match);
});

`lazy_static_external_mod()`, `lazy_static_inner()`, `lazy_static_local()`, `lazy_static_backref()`

These benches relied on lazy_static with the only difference in where it was declared:

lazy_static_local: at the root level
lazy_static_inner: at a sub-module level (same file)
lazy_static_external_mod: at a module placed in a separate file
lazy_static_backref: at the root level, used in a sub-module

The lazy_static declarations were identical in all cases:

lazy_static! {
    pub(crate) static ref COMPILED_REGEX: regex::Regex = regex::Regex::new(LONG_REGEX).unwrap();
}

The placement of lazy_static! { ... } declaration made no difference:

The static variable was initialized once only
All these benches took ~27 ns/iter each.

`lazy_static_reinit()`

It is possible to initialize the static variable before or after its first use by calling

lazy_static::initialize(&STATIC_VAR_NAME);

There was no additional performance cost for calling initialize for the first time or any number of times after that.
This is inline with the documentation that states:

Takes a shared reference to a lazy static and initializes it if it has not been already.

`once_cell`

once_cell is just as elegant as lazy_static! and is about 20% faster in a 32-thread test. It performed on the par with lazy_static! in my basic tests.

The static declaration is a single line of code:

static COMPILED_REGEX_ONCE_CELL: once_cell::sync::Lazy<regex::Regex> =
    once_cell::sync::Lazy::new(|| regex::Regex::new(LONG_REGEX).unwrap());

and the usage of the static variable is exactly the same as with lazy_static!:

    b.iter(|| {
        let is_match = COMPILED_REGEX_ONCE_CELL.is_match(TEST_EMAIL);
        test::black_box(is_match);
    });

There is an RFC to merge once_cell into std::lazy making it part of the standard library. It may be a more future-proof choice if you are starting a new project.

`lazy_static` alternatives that DO NOT work

Declaring a static variable

pub(crate) static STATIC_REGEX: regex::Regex = regex::Regex::new(LONG_REGEX).unwrap();

ERROR: calls in statics are limited to constant functions, tuple structs and tuple variants rustc E0015

Declaring a const function

const fn static_regex() -> regex::Regex {
    regex::Regex::new(LONG_REGEX).unwrap()
}

ERROR: calls in constant functions are limited to constant functions, tuple structs and tuple variants rustc E0015

`lazy_static` in depth

This section goes deep into the source code of `lazy_static` to really understand how it works.

The demo code in this project is split into several parts:

benches: the source for the benchmarks in this post
examples/expansion_base.rs: a minimal implementation to get expanded code from lazy_static! macro
examples/expanded.rs: the expanded code generated by lazy_static! macro from expansion_base.rs
src/main.rs: a self-contained implementation based on the expanded code

Your IDE will be unhappy with some parts of the code if you are on stable channel. Switch to/from nightly with these commands to get rid of the IDE warnings:

rustup default nightly
rustup default stable

Macro's code

Running cargo expand --example expansion_base outputs the code generated for the crate. You can find the full output in examples/expanded.rs file.

In short, it looks something like this:

#![feature(prelude_import)]
#[prelude_import]
use std::prelude::rust_2021::*;
#[macro_use]
extern crate std;
extern crate lazy_static;
use lazy_static::lazy_static;
#[allow(missing_copy_implementations)]
#[allow(non_camel_case_types)]
#[allow(dead_code)]
struct COMPILED_REGEX {
    __private_field: (),
}
#[doc(hidden)]
static COMPILED_REGEX: COMPILED_REGEX = COMPILED_REGEX {
    __private_field: (),
};
impl ::lazy_static::__Deref for COMPILED_REGEX {
    type Target = regex::Regex;
    fn deref(&self) -> &regex::Regex {
        #[inline(always)]
        fn __static_ref_initialize() -> regex::Regex {
            regex::Regex::new(".*").unwrap()
        }
        #[inline(always)]
        fn __stability() -> &'static regex::Regex {
            static LAZY: ::lazy_static::lazy::Lazy<regex::Regex> = ::lazy_static::lazy::Lazy::INIT;
            LAZY.get(__static_ref_initialize)
        }
        __stability()
    }
}
impl ::lazy_static::LazyStatic for COMPILED_REGEX {
    fn initialize(lazy: &Self) {
        let _ = &**lazy;
    }
}
fn main() {
    let _x = COMPILED_REGEX.is_match("abc");
}

The snippet above depends on some functions provided by lazy_static and that's where most of the magic happens.

I distilled it to a simpler version that does not have lazy_static as a dependency at all. Skim through it and go to the explanations that follow:

struct Lazy<T: Sync>(Cell<Option<T>>, Once);
unsafe impl<T: Sync> Sync for Lazy<T> {}

struct CompiledRegex {
    __private_field: (),
}

static COMPILED_REGEX: CompiledRegex = CompiledRegex {
    __private_field: (),
};

impl Deref for CompiledRegex {
    type Target = regex::Regex;
    fn deref(&self) -> &regex::Regex {
        static LAZY: Lazy<regex::Regex> = Lazy(Cell::new(None), Once::new());

        LAZY.1.call_once(|| {
            LAZY.0.set(Some(regex::Regex::new(LONG_REGEX).unwrap()));
        });

        unsafe {
            match *LAZY.0.as_ptr() {
                Some(ref x) => x,
                None => {
                    panic!("attempted to dereference an uninitialized lazy static. This is a bug");
                }
            }
        }
    }
}

There are a few key features in the last snippet to pay attention to:

struct Lazy holds generic struct CompiledRegex that is instantiated as static COMPILED_REGEX that actually holds the compiled regex.
That long chain is unravelled inside impl Deref for CompiledRegex to give us the compiled regex as a static variable
std::sync::Once::call_once() is used to initialize the regex once only
match *LAZY.0.as_ptr() gets us the initialized regex from deep inside the chain of structs

If the above code is still a bit confusing, look up src/main.rs for the full version with detailed comments.

Running the program with cargo run will execute this demo code from src/main.rs using the snippet above for the lazy-static part:

fn main() {
    println!("Program started");
    println!("{TEST_EMAIL} is valid: {}", COMPILED_REGEX.is_match(TEST_EMAIL));
    println!("{TEST_NOT_EMAIL} is valid: {}", COMPILED_REGEX.is_match(TEST_NOT_EMAIL));
}

and produce this output:

Program started

Derefencing CompiledRegex

CompiledRegex initialized

name@example.com is valid: true

Derefencing CompiledRegex

Hello world! is valid: false

As you can see, COMPILED_REGEX is initialized once only and is dereferenced every time COMPILED_REGEX variable is used.

Q.E.D.? :)

193 hours of music for coding and writing

Max — Mon, 31 Jan 2022 23:36:01 +0000

I find it easier to write and code when the background music has no lyrics I can understand. These words are written to some groovy sounds of a trumpet from my Coding Time playlist on Spotify.

Duration: 193h+
Style: eclectic
Includes: world, classical, jazz, electronica, instrumental, indie
Excludes: rap, heavy or overly emotional sounds, anything with lyrics in English

What are you favorite focus playlists or albums?

Goodbye AWS OpenSearch, hello self-hosted ElasticSearch on EC2

Max — Sun, 30 Jan 2022 23:55:07 +0000

An AWS OpenSearch to EC2-hosted ElasticSearch migration guide

AWS ElasticSearch Service used to be a quick and easy option to add ElasticSearch to a project already hosted on AWS. It was forked into AWS OpenSearch and is now only nominally related to ElasticSearch. That change created a dilemma to stay with this new AWS service or make a move elsewhere to stay with ElasticSearch.

The future of OpenSearch doesn't look bright. AWS OpenSearch project on github has tanked since AWS took over while ElasticSearch project is keeping up a steady pace.

The other consideration against AWS OpenSearch is its high cost compared to other options.

For example, using a mix of on-demand and spot EC2 instances running ElasticSearch can cost just 1/3 of the comparable OpenSearch configuration.

Target EC2 configuration

Mixing EC2 on-demand and spot instances is very cost-effective for non-mission-critical projects. If the ES cluster is configured correctly, the worst case scenario of all spot instances going down is degraded performance with no data loss. It is possible only if the shards are allocated so that there is at least one copy in the on-demand part of the cluster. This setting is addressed later in this guide.

The number of nodes can be easily scaled with addition of more master and data nodes from the same AMI. Both types can be run as either spot or on-demand as long as there are enough on-demand instances to hold at least one copy of every shard.

ElasticSearch fear factor

ElasticSearch installation and configuration may look daunting at first. I was definitely intimidate by the JVM and all the "cluster-node-master-whatnot terminology". It turned out to be easy to install and not too hard to configure. I did have to read a lot of their docs to fill in the blanks in their official installation guide. The rest of this post contains the exact steps I had to go through to create my ElasticSearch cluster and migrate all the data from AWS OpenSearch to it.

I am not scared of you, ElasticSearch, any more!

ES Node AMI

The first task is to build an ElasticSearch Node AMI (ES Node AMI) to serve as a blueprint for any node type. The size of the instance is not important at this stage. It will be easy to resize after creation of the AMI.

ElasticSearch version: 7.16
AWS AMI: Ubuntu 20.04LTS
Storage: 50GB root + 60GB data GP3 SSD (storage sizing guide)
Static IP: required
Misc:
- Hostname type: Resource name
- Enhanced monitoring: Yes

We only need to set up a single machine to create a reusable ES Node AMI. Any differences between ES nodes will be in the configuration applied after the AMI is launched.

Start by launching a brand new Ubuntu 20.04 instance on EC2 with the above config.

ES installation

This guide is based on the official ES install guide for Debian Package. It was used to build a 3-node production cluster with a single master on-demand and 2 data spot instances.

Important note about Shell Scripts

Most scripts in this guide are formatted as blocks and can be copy-pasted as such into the CLI. It is better to copy-paste them line by line because there is no error handling and some steps require manual input.

# update the OS
sudo apt-get update && sudo apt-get upgrade

# ES pre-requisites
sudo apt-get install unzip
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list

# install ES
sudo apt-get update
sudo apt-get install elasticsearch

ES is now installed, but not enabled.

Storage configuration

Attach the 2nd disk to the ES data directory to make it portable. If the instance dies or has to be replaced the index data can be salvaged by re-attaching the volume to another ES EC2 instance.

Mount the 2nd EBS drive to /var/lib/elasticsearch folder. Make sure the permissions of the mount are set exactly as in the target folder (drwxr-s--- 2 elasticsearch elasticsearch).

sudo mkfs -t xfs /dev/nvme1n1
sudo mount /dev/nvme1n1 /var/lib/elasticsearch
sudo chown elasticsearch:elasticsearch /var/lib/elasticsearch
sudo chmod 2750 /var/lib/elasticsearch

Mount the drive on boot by adding it to fstab.

Update fstab:

echo "UUID=$(sudo blkid | grep -oP '(?<=/dev/nvme1n1: UUID=")[a-z0-9\-]+') /var/lib/elasticsearch xfs defaults 0 2" | sudo tee -a /etc/fstab

Verify it is still a valid fstab file. The instance may fail to boot if its corrupt:

sudo blkid
cat /etc/fstab

fstab should look similar to this example:

ubuntu@i-0c4ba3e0d781e7f34:~$ sudo blkid
/dev/nvme0n1p1: LABEL="cloudimg-rootfs" UUID="436cf32d-5e3d-46ca-b557-f870c8a25794" TYPE="ext4" PARTUUID="24ca9e81-01"
/dev/loop0: TYPE="squashfs"
/dev/loop1: TYPE="squashfs"
/dev/loop2: TYPE="squashfs"
/dev/loop3: TYPE="squashfs"
/dev/loop4: TYPE="squashfs"
/dev/nvme1n1: UUID="b90fdf46-25a9-48a2-a7e6-82986da39306" TYPE="xfs"
ubuntu@i-0c4ba3e0d781e7f34:~$ cat /etc/fstab
LABEL=cloudimg-rootfs   /    ext4   defaults,discard    0 1
UUID=b90fdf46-25a9-48a2-a7e6-82986da39306 /var/lib/elasticsearch xfs defaults 0 2

Reboot to make sure the VM comes back to life and there are no old file handles pointing at the mounting folder.

sudo shutdown -r 0

Install Kibana and plugins

Install ES S3 plugin to backup and restore using AWS S3 for snapshot storage. It is needed to restore a backup from AWS OpenSearch. Do not try to restart ES as instructed by the plugin. We'll do that later.

sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install repository-s3

Install Kibana:

sudo apt-get install kibana

Useful shortcuts and troubleshooting

A detailed overview of the distro layout can be summarized with a few frequently used commands:

Config files

ES config: sudo nano /etc/elasticsearch/elasticsearch.yml
Kibana config: sudo nano /etc/kibana/kibana.yml

Log files

ES log: sudo tail -n 100 /var/log/elasticsearch/stm-prod.log
Kibana log: sudo tail -n 100 /var/log/kibana/kibana.log
systemctl log: tail -n 100 /var/log/syslog

Folders

ES tools: /usr/share/elasticsearch/bin
ES path.data: /var/lib/elasticsearch
ES path.logs: /var/log/elasticsearch

CLI `curl` calls

Check ES is running: curl -X GET "localhost:9200/?pretty"
Over HTTPS locally: curl --insecure -X GET "https://localhost:9200/?pretty"
Over HTTPS anywhere: curl -X GET "https://master.es.stackmuncher.com:9200/?pretty"
Check Kibana is running: curl --insecure --verbose -L http://localhost:5601/
Delete all Kibana indexes: curl --user $STM_ES_BASIC_AUTH -XDELETE https://master.es.stackmuncher.com:9200/.kibana
View all ES indexes: curl --user $STM_ES_BASIC_AUTH https://master.es.stackmuncher.com:9200/_cat/indices

ES log is placed in the file named after the cluster. Initially it is called elasticsearch.log. As soon as the cluster is renamed into stm-prod, the log name changes to stm-prod.log.

systemctl log is useful for troubleshooting failed start commands. ES start-up errors are likely to be found there. Kibana always starts OK and then places any errors in its own /var/log/kibana/kibana.log

$STM_ES_BASIC_AUTH env variable should be set to elastic:your_elastic_autogenerated_pwd. The password will be generated at a later step.

Networking

ES instances need to be accessible by clients from within the VPC and from outside the VPC. Create a single public access point at master.es.stackmuncher.com with 600 TTL in Route53 for both access modes.

public hosted zone: e.g. master.es.stackmuncher.com A 34.232.124.28
private hosted zone: e.g. master.es.stackmuncher.com CNAME i-0c4ba3e0d781e7f34.ec2.internal (replace i-0c4ba3e0d781e7f34.ec2.internal with the instance ID)

Test it within and outside of the VPC with nslookup master.es.stackmuncher.com. It should resolve to a private and static IPs respectively.

AWS Lambda Functions need to have access to the VPC of the ES instance.

Initial config

ES configuration is dependant on the use case. I suggest to get something going using this example and then tailor it to your needs.

Reference:

Recommended 1-2-3 node cluster configurations: https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability-cluster-small-clusters.html
Discovery settings: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-settings.html
Node settings: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html

ES configuration is done in .yml config files and via PUT _cluster/settings commands once the cluster is up and running.

Basic security config

There are 3 levels of security config in ES. A production set up for migration requires the full monty with basic, transport and HTTPs for ES and Kibana.

Also, it's a good idea to password-protect the keystore. This can be done after the cluster is fully operational.

Reference:

We'll need 2 sets of certificates:

self-issued for transport level to enable ES nodes securely talk to each other
a trusted cert for external requests to ES API and Kibana front-end

Enable and launch ES:

sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

Bootstrap ES security

This step will generate random passwords for ES built-in accounts and can only be run once. Save the login/passwords it outputs into a file.

echo "xpack.security.enabled: true" | sudo tee -a /etc/elasticsearch/elasticsearch.yml
echo "discovery.type: single-node" | sudo tee -a /etc/elasticsearch/elasticsearch.yml
echo "network.host: 0.0.0.0" | sudo tee -a /etc/elasticsearch/elasticsearch.yml
sudo systemctl restart elasticsearch.service
sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto

discovery.type: single-node setting is needed to stop ES accepting other nodes. network.host: 0.0.0.0 will make it listen on all IPs, internal and external.

Kibana security config reference: https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html#add-built-in-users

Add the password for kibana_system account when requested:

echo 'elasticsearch.username: "kibana_system"' | sudo tee -a /etc/kibana/kibana.yml
echo "server.host: 0.0.0.0" | sudo tee -a /etc/kibana/kibana.yml
sudo /usr/share/kibana/bin/kibana-keystore add elasticsearch.password

Enable and start Kibana:

sudo systemctl enable kibana.service
sudo systemctl start kibana.service

Wait for a minute or so and check the logs to make sure Kibana started OK.

Check the logs: sudo tail -n 100 /var/log/kibana/kibana.log
This request should return some HTML: curl --insecure --verbose -L http://localhost:5601/

Generate transport-level certificates

Transport-level certificates are self-issued certificates that are trusted by all nodes. They can be used for external communications if all your clients can be forced to trust them. It may be easier to get a LetsEncrypt cert and be trusted everywhere.

Reference: https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-basic-setup.html#generate-certificates

Generate the certificate authority file: sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca
- it is saved as /usr/share/elasticsearch/elastic-stack-ca.p12
Generate the certificate: sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
- it is saved as /usr/share/elasticsearch/elastic-certificates.p12
Copy the cert to its default location:

sudo cp /usr/share/elasticsearch/elastic-certificates.p12 /etc/elasticsearch/elastic-certificates.p12
sudo chmod 440 /etc/elasticsearch/elastic-certificates.p12

Enable transport-level certificates

Certificates are generated and enabled once only because all nodes are built from the same AMI and share the certs.

echo "xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12"  | sudo tee -a /etc/elasticsearch/elasticsearch.yml

sudo /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password

sudo /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

sudo systemctl restart elasticsearch.service

HTTPs with LetsEncrypt

Communications between ES and external clients should be secured with a publicly trusted CA. LetsEncrypt is free, but comes with 2 problems: must be renewed every 3 months and the web server must provide the full chain version of the certificate because some intermediaries there are not commonly trusted.

Generate a LetsEncrypt wildcard certificate on a local machine:

Issue the cert: sudo certbot certonly --manual for *.es.stackmuncher.com
Local location: /etc/letsencrypt/live/es.stackmuncher.com with shortcuts pointing at /etc/letsencrypt/archive/es.stackmuncher.com/

Copy the certificate files to the ES instance:

sudo scp -i ~/.ssh/dev-ld-useast1.pem  /etc/letsencrypt/live/es.stackmuncher.com/fullchain.pem ubuntu@master.es.stackmuncher.com:/home/ubuntu/fullchain.pem

sudo scp -i ~/.ssh/dev-ld-useast1.pem  /etc/letsencrypt/live/es.stackmuncher.com/privkey.pem ubuntu@master.es.stackmuncher.com:/home/ubuntu/privkey.pem

sudo scp -i ~/.ssh/dev-ld-useast1.pem  /etc/letsencrypt/live/es.stackmuncher.com/chain.pem ubuntu@master.es.stackmuncher.com:/home/ubuntu/chain.pem

scp can't get to the ultimate destination of the certificates on the ES instance, so they are copied into the home directory on the instance first and then moved locally within the instance:

sudo mv ~/fullchain.pem /etc/elasticsearch/fullchain.pem
sudo chown root:elasticsearch /etc/elasticsearch/fullchain.pem
sudo chmod 440 /etc/elasticsearch/fullchain.pem
sudo mv ~/privkey.pem /etc/elasticsearch/privkey.pem
sudo chown root:elasticsearch /etc/elasticsearch/privkey.pem
sudo chmod 440 /etc/elasticsearch/privkey.pem
sudo mv ~/chain.pem /etc/elasticsearch/chain.pem
sudo chown root:elasticsearch /etc/elasticsearch/chain.pem
sudo chmod 440 /etc/elasticsearch/chain.pem

Add the certificates to ES config:

echo "xpack.security.http.ssl.enabled: true" | sudo tee -a /etc/elasticsearch/elasticsearch.yml
echo "xpack.security.http.ssl.key: /etc/elasticsearch/privkey.pem" | sudo tee -a /etc/elasticsearch/elasticsearch.yml
echo "xpack.security.http.ssl.certificate: /etc/elasticsearch/fullchain.pem" | sudo tee -a /etc/elasticsearch/elasticsearch.yml
echo "xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/chain.pem" | sudo tee -a /etc/elasticsearch/elasticsearch.yml
sudo systemctl restart elasticsearch.service

Test ES access via HTTPs internally and externally by running curl -X GET "https://master.es.stackmuncher.com:9200/?pretty". In both cases it should return an error similar to this example, which means ES is working and HTTP SSL is configured correctly:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "missing authentication credentials for REST request [/?pretty]",
        "header" : {
          "WWW-Authenticate" : [
            "Basic realm=\"security\" charset=\"UTF-8\"",
            "Bearer realm=\"security\"",
            "ApiKey"
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "missing authentication credentials for REST request [/?pretty]",
    "header" : {
      "WWW-Authenticate" : [
        "Basic realm=\"security\" charset=\"UTF-8\"",
        "Bearer realm=\"security\"",
        "ApiKey"
      ]
    }
  },
  "status" : 401
}

Configure kibana

Detailed Kibana configuration: https://www.elastic.co/guide/en/kibana/current/settings.html

Copy LetsEncrypt certificates from the ES location to Kibana:

sudo cp /etc/elasticsearch/fullchain.pem /etc/kibana/fullchain.crt
sudo chown root:kibana /etc/kibana/fullchain.crt
sudo chmod 440 /etc/kibana/fullchain.crt
sudo cp /etc/elasticsearch/privkey.pem /etc/kibana/privkey.key
sudo chown root:kibana /etc/kibana/privkey.key
sudo chmod 440 /etc/kibana/privkey.key

Add the certificates to Kibana config:

echo "elasticsearch.hosts: https://master.es.stackmuncher.com:9200" | sudo tee -a /etc/kibana/kibana.yml
echo "server.publicBaseUrl: https://master.es.stackmuncher.com:5601" | sudo tee -a /etc/kibana/kibana.yml
echo "server.ssl.certificate: /etc/kibana/fullchain.crt" | sudo tee -a /etc/kibana/kibana.yml
echo "server.ssl.key: /etc/kibana/privkey.key" | sudo tee -a /etc/kibana/kibana.yml
echo "server.ssl.enabled: true" | sudo tee -a /etc/kibana/kibana.yml
sudo systemctl restart kibana.service

Kibana should now be available at https://master.es.stackmuncher.com:5601/. Login as elastic user with its auto-generated password.

If Kibana is not working, troubleshoot with tail -n 100 /var/log/syslog and sudo tail -n 100 /var/log/kibana/kibana.log.

Misc settings

Add any tweaks specific to the set up. E.g. StackMuncher doesn't use GeoIP and ML:

echo "ingest.geoip.downloader.enabled: false
xpack.ml.enabled: false" | sudo tee -a /etc/elasticsearch/elasticsearch.yml
sudo systemctl restart elasticsearch.service

Anonymous access to ES

AWS OpenSearch access control is different from how it is commonly done in ES. The easiest migration approach is to retain the anonymous access for now, but restrict the anonymous user privileges.

Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/anonymous-access.html

Create a role describing anonymous user rights in Kibana:

POST /_security/role/stm_apps
{
  "run_as": [ "_es_anonymous_user" ],
  "indices": [
    {
      "names": [ "stm*","dev*","stats*" ],
      "privileges": [ "read","write","create" ]
    }
  ]
}

GET /_security/role/stm_apps

Map built in _es_anonymous_user to stm_apps role:

echo "xpack.security.authc:
  anonymous:
    roles: stm_apps" | sudo tee -a /etc/elasticsearch/elasticsearch.yml

sudo systemctl restart elasticsearch.service

Anonymous access tests

This call from any machine with sufficient network access to the VM should fail because _es_anonymous_user is not allowed to create new indexes:

curl -X POST "https://master.es.stackmuncher.com:9200/dev/_doc" -H 'Content-Type: application/json' -d'{ "foo": "bar" }'

Expected error: ...action [indices:admin/auto_create] is unauthorized for user [_anonymous] with roles [stm_apps]...

Create dev index in Kibana by running PUT /dev and repeat the curl request - it should succeed with dev index in place.
Clean up with DELETE /dev.

This call should always fail for _es_anonymous_user: curl https://master.es.stackmuncher.com:9200/_cat/indices

Create ES Node AMI

The ES instance is now in a state ready to be reused for creating multiple master and data nodes by spinning more instances within the same VPC.

Optionally, remove comment lines starting with # from ES config to make it more readable:

sudo sed -i '/^#/d' /etc/elasticsearch/elasticsearch.yml

Kibana config can also be cleaned up with sudo sed -i '/^#/d' /etc/kibana/kibana.yml.

Save /etc/elasticsearch/elasticsearch.yml and /etc/kibana/kibana.yml for reference.

Create an AMI

Launching ES nodes

An instance launched from ES Node AMI starts as a single node due to discovery.type: single-node setting and lack of cluster-level config. It needs to be re-configured to join a cluster as a master or a data node.

Master node config

This script will turn a standalone ES node into a master:

sudo systemctl stop elasticsearch.service
sudo sed -i '/discovery.type: single-node/d' /etc/elasticsearch/elasticsearch.yml
echo 'cluster.name: stm-prod
node.name: es-master
discovery.seed_hosts: ["master.es.stackmuncher.com"]

# comment out one of the two reservation options
# this setting affects shard allocation
node.attr.reservation: persistent
# node.attr.reservation: spot

cluster.initial_master_nodes: ["es-master"]
cluster.routing.allocation.awareness.attributes: reservation' | sudo tee -a /etc/elasticsearch/elasticsearch.yml
sudo systemctl start elasticsearch.service
sudo systemctl restart kibana.service

Open Kibana at https://master.es.stackmuncher.com:5601/ and check the cluster state with GET /_cluster/state. It should be named appropriately and have only one node:

{
  "cluster_name" : "stm-prod",
  "cluster_uuid" : "voHWCBEdRO-shwxv0ngiaw",
  "version" : 207,
  "state_uuid" : "x7vj0CmvSlmmHY4ASeCTcA",
  "master_node" : "V4EZprKlTleq9DzgIfIm1A",
  "blocks" : { },
  "nodes" : {
    "V4EZprKlTleq9DzgIfIm1A" : {
      "name" : "es-master",
      "ephemeral_id" : "OWTuL7nIRGedVd19oTM0Hw",
      "transport_address" : "172.31.7.40:9300",
      "attributes" : {
        "xpack.installed" : "true",
        "transform.node" : "true",
        "reservation" : "persistent"
      },
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "remote_cluster_client",
        "transform"
      ]
    }
  }
  ...

Use node.attr.reservation: persistent for instances that are unlikely to go offline unexpectedly (reserved or on-demand instances) and node.attr.reservation: spot for any kind of spot instances. ES will be configured to allocate shards so that even if all spot instances drop off there will still be enough replicas to recover and re-balance.

If you are trying to recover an instance and Kibana won't start due to its shard loss:

kill Kibana indexes with curl --user $STM_ES_BASIC_AUTH -XDELETE https://master.es.stackmuncher.com:9200/.kibana*
restart Kibana with sudo systemctl restart kibana.service

The master node is now ready for the production snapshot restore from AWS OpenSearch.

Data node config

A data node performs fewer functions than a master and has to be configured differently.

Remove Kibana as it's not needed on data nodes. Also, any prior node info should be deleted for a new node to establish itself:

sudo systemctl stop elasticsearch.service
sudo systemctl disable kibana.service
sudo systemctl stop kibana.service
sudo rm /etc/kibana/kibana.yml
sudo rm -rf /var/lib/elasticsearch/nodes
sudo ls -al /var/lib/elasticsearch

Expected output:

drwxr-s---  2 elasticsearch elasticsearch    6 Jan 27 08:52 .
drwxr-xr-x 42 root          root          4096 Jan 26 22:05 ..

WARNING

WARNING: modify the node name in this script below.

WARNING

sudo sed -i '/discovery.type: single-node/d' /etc/elasticsearch/elasticsearch.yml
echo 'cluster.name: stm-prod
node.name: es-data-1
node.roles: [ data ]
discovery.seed_hosts: ["master.es.stackmuncher.com"]

# comment out one of the two reservation options
# this setting affects shard allocation
# node.attr.reservation: persistent
node.attr.reservation: spot' | sudo tee -a /etc/elasticsearch/elasticsearch.yml
sudo systemctl start elasticsearch.service

Open Kibana at https://master.es.stackmuncher.com:5601/ and check cluster state with GET _cat/nodes. It should have the newly added node:

172.31.9.51 9 65 16 0.28 0.32 0.23 d          - es-data-1
172.31.7.40 8 75  4 0.05 0.07 0.10 cdfhimrstw * es-master

Migrate production data from AWS OpenSearch to EC2 ES

Backup and restore to S3 requires S3 Repo Plugin. It is installed on AWS OpenSearch and is part of our ES Node AMI.

Backup / restore is done via snapshot repositories.

Use these snapshot commands to list and recreate a repository (replace IDs as needed):

GET /_snapshot
DELETE _snapshot/snap-syd
PUT _snapshot/snap-syd
{"type":"s3","settings":{"bucket":"stm-prod-sdvevr3i65bzgp4zuh6b","endpoint":"s3.amazonaws.com"}}

A repository must be registered at both ends (AWS OpenSearch and EC2 ES) for the transfer to work.

Stop all ES clients to prevent any updates until the new ES cluster is operational.

Start a backup on AWS OpenSearch:

PUT /_snapshot/snap-syd/es_20220125_1044?wait_for_completion=false

Check the backup progress:

GET /_snapshot/snap-syd/es_20220125_1044

Start a restore on EC2 ES:

POST /_snapshot/snap-syd/es_20220125_1044/_restore?wait_for_completion=false
{"indices": "dev_*,search_log,stats,stm_*"}

Check the restore progress and shard allocation:

GET _cat/recovery
GET _cat/shards

All shards should be distributed such that there is at least one copy on persistent nodes and one copy on spot nodes with no UNASSIGNED shards.

Modify replication settings by adjusting the number of replicas if the number of nodes in the new cluster is different from what it was on AWS OpenSearch:

PUT _all/_settings
{
  "index.number_of_replicas": 2,
  "unassigned.node_left.delayed_timeout": "5m"
}

Wait for all shards to be allocated and GET _cluster/health to return green cluster status.

Change the config in all ES clients to point them at https://master.es.stackmuncher.com:9200 .

Congratulations - you are now running your own ElasticSearch cluster.

How to display multiple blank lines in Markdown

Max — Wed, 17 Nov 2021 00:58:33 +0000

Ever wished you could space out your blocks of text in markdown a bit more without using non-markdown tags?

According to markdown rules multiple empty lines are collapsed into a single empty line or just a start of a new paragraph, depending on the flavour.

Example

*Line 1*





*Line 2*

is collapsed into

Line 1

Line 2

However, this seemingly identical example

*Line 1*
 

 

 

*Line 2*

retains its white space:

Line 1
 

Line 2

Both examples look the same to a naked eye in markdown and have no special markup.

The difference in the second example is in an invisible Unicode character   followed by a new line.

It fools the rendering code into adding a visually empty <p> </p> tag giving you an extra empty line. You can add as many of them as you need alternating with an empty line.

According to emptycharacter.com there are 16 types of those "invisible characters". I found that Medium Mathematical Space (U+205F) works best here, on GitHub, Reddit or sites produced by Hugo static site generator. It looks like this:

Happy
 

              writing
 

Quick tip: if Rust stopped formatting your code ...

Max — Sun, 24 Oct 2021 09:06:43 +0000

Sometimes Rust Analyzer stops formatting my VS Code document and there is no message or explanation why.

This is just one ugly example - lines 46-53 are all over the place.

No matter how many times I press Ctrl+Shift+I the IDE would not fix the formatting or tell me what's wrong.

Turns out there is a simple way to find out why Rust Analyzer fails to format the document: cargo fmt. It will tell you exactly what's broken.

In this case it was a comment on line 44 that got in the way. The document started formatting again as soon as I removed it.

Beware, cargo fmt will attempt to reformat the entire project. Read more about it in https://github.com/rust-lang/rustfmt or run cargo fmt -- --help for options.

I went back to several other files that were not formatting properly, ran cargo fmt on them and quickly fixed the problems. Happy coding!

How much can you earn on UpWork before you give up?

Max — Thu, 16 Sep 2021 20:00:05 +0000

A brief analysis of actual UpWork earnings for software developers

TL;DR; The sooner you give up, the more you earn elsewhere.

In case you don't know what UpWork is, it is a leading freelance marketplace. They are a publicly listed company with $338M revenue in 2020. Anyone can post a job there, and most professionals can offer their services at the cost of 20% of their earnings. The platform provides no vetting for employers and minimal vetting for freelancers leaving it to the feedback system to weed out bad players. It is one of the better ones out there.

I used to hire on UpWork for our startup. It was an easy way to get things done on a shoestring budget. The results were a mixed bag. Most of the work was either a complete throw-away or prototype-level quality.
We got what we paid for, but there were a few outliers.

High-quality freelancers are outliers

Despite considering bids only from freelancers with excellent feedback and 90%+ Job Success Score our hit rate was about 1 in 10 for "acceptable" and 1 in 20 for "great". We were lucky to hire some really good freelancers from remote (to us) places like Moldova, Macedonia and Turkey. They were up and coming, had reasonable rates and appreciated the constant stream of work we were sending them.

Paying more doesn't get you more quality

There seemed to be a pretty low ceiling on the quality of the workforce there. Assuming that our job ads were well-written, simply offering more money didn't bring better quality bids. None of the higher-cost freelancers we tried ended up on our regulars list.

Freelancer's lifetime value

I recently got in touch with 11 out of 13 UpWork freelancers we hired on a regular basis. To my surprise, none of them was still using UpWork or any other freelance marketplace. Only 3 continued with casual contract work mostly through referrals from past clients.

They all had a different story to tell, but the common theme was that the jobs on UpWork were hard to get, many employers had no idea what they were doing and the pay was below the market rate.

So how much do freelancers make before they move off the platform?

I looked at the total earnings of 100 freelancers in search results for Rust.

Assuming you don't give up before landing that very first job, the chances are:

1/3 quits before earning $1,000
1/4 quits before earning $10,000
1/3 quits before reaching $100,000
the rest stays for a long haul and makes UpWork a significant source of their income

High-earner list

It's not all doom and gloom. You can make a decent living in some categories.

9 out of 100 freelancers in my search for a Rust engineer had over $100k of total earnings.

$1,000,000 club - do you want to be a member?

Evgeniy T. from Catalonia tops our list with $1M+ of lifetime earnings. We don't know exactly how much he earned, but let's assume it is a million.

Earning one million dollars sounds very inspirational. Who doesn't want to be a millionaire! Let's break it down:

$1,000,000 / 18,076 hrs worked = $55 per hour
period over 10 years (started in 2011)
$117k per year if adjusted for inflation and UpWork fees

If we disregard the intangible values such as freedom this type of work brings,

$117k per year for someone with 10 years of full-stack experience is a rather low figure in today's world.

The same high-earners list doesn't look as impressive on an annual basis, except maybe entry #3:

$185,288 / 5 years
$210,023 / 12 years
$277,418 / 2 years
$281,584 / 12 years
$295,363 / 6 years
$425,345 / 11 years
$673,153 / 12 years
$921,241 / 9 years
$1,000,000 / 10 years

In the end, UpWork is no better or worse than any other career path. It does a good job of putting you in front of employers. If you are good and want to be poached, you most likely will be, despite what the Terms and Conditions say.

Hey, are you open to offers of better employment or more contract work?

I am building a software engineer search engine for devs like you and me to be discovered by employers looking for our skills. It is a community-first and fully open-source project that takes no commission or charges placement fees. Head to https://stackmuncher.com/about/developers/ and let interesting and well-paid work find you

Forem: Max

React to Vue Conversion with Claude, Gemini, and ChatGPT

Online React to Vue Converters

GitHub Copilot

Converting the Entire Project 🙄

Converting a Single React Component 🎉

Prompt

Claude 3.7 Sonnet

Claude 3.5 Sonnet

Gemini 2.5 Pro

ChatGPT-4.1

Conclusion

Question

AWS Lambda with CloudFront: CORS, authorization and caching examples

Example 1: CORS not required, no authorization headers, no caching

Request headers

Response headers

Example 2: CORS with custom authorization header and no caching

OPTIONS request/response

GET request/response

Example 3: CORS with Authorization header and caching

GET request headers

Caching

AWS Lambda with CloudFront configuration guide

Overview

Configuration complexity

Core concepts

Lambda Function URL configuration for CloudFront

Lambda URL access control config

Lambda URL CORS headers

CloudFront configuration

Origin configuration

Origin access control affects the use of Authorization header

Lambda access policy

Behavior configuration

CloudFront policies and CORS headers

Caching policies

Caching disabled

Caching enabled with Authorization header

Origin request policy

Response headers policy

References

AWS Lambda Function URL with CORS explained by example

Overview

Why we need Cross-Origin Resource Sharing (CORS) for Lambda Function URLs

Option 1: Returning CORS from the lambda function handler

Option 2: Configuring CORS headers in AWS Lambda settings

Custom domain names for Lambda URLs

Access control configuration for Lambda Function URL

Request/response examples for different configuration options in detail

Example 1: Lambda URL request/response without CORS protocol

Example 2: CORS headers added by lambda handler

Example 3: CORS request/response headers added by AWS

A few "gotchas"

Add one header per line in the CORS configuration form

Don't allow localhost CORS in production

Adding and deleting FunctionURLAllowPublicAccess access policy

Header double-up

References

Passwordless login to Postgres from VSCode SQLTools extension using IDENT/PEER method

About IDENT/PEER Authentication methods

Install Postgress with default settings

Add your current Linux user to Postgres

Connecting VS Code to Postgres

Chrome extension example with Rust and WASM

Toolchain

Architecture of the extension

Folders

Manifest V3

Background script

Popup page

Content script

Messaging between the scripts

Messaging examples

From background.js to popup.js

From popup.js to background.js

Sending messages from WASM to JS scripts

WASM

Inside Cargo.toml

Dependencies

Don't allow `localhost` CORS in production

`hello_wasm()` sequence diagram

How `console.log()` is called from Rust

How Rust's `hello_wasm()` is called from JS

TL;DR: `lazy_static!` is fine, but `once_cell` may be better for new projects.

`bad_rust_local()`

`vanilla_rust_local()`

`lazy_static_external_mod()`, `lazy_static_inner()`, `lazy_static_local()`, `lazy_static_backref()`

`lazy_static_reinit()`

`once_cell`

`lazy_static` alternatives that DO NOT work

`lazy_static` in depth

This section goes deep into the source code of `lazy_static` to really understand how it works.

CLI `curl` calls