Forem: ohaddahan

Tips for safer dependency management in JavaScript environments

ohaddahan — Sun, 21 Jan 2024 16:56:57 +0000

Just about anyone who worked on the JavaScript ecosystem has faced annoying issues due to version mismatch , dependencies updates etc. , I want to share a few small tips that helped me mitigate some of the issues.

Tip #1 save-exact

npm install <package-name> --save-exact

This will install the package exact version instead of default range.

"dependencies": { "axios": "1.6.5" }

Due note that package-lock.json will still contain ranges. One way to mitigate this is to manually edit the file, a simple regex will enforce exact versions on the dependencies too.

(if some dependencies requirements can't be fulfilled with strict nested versions, you can iterate until you find a set of matching versions and make them static)

For example running the code below in vim editor.

%s/: "\^/: "/g

You can set it as the default behavior by running.

npm config set save-exact=true

Which will set save-exact=true in your .npmrc.

Tip #2 avoid named releases

One way to help synchronize multiple developers working on the same repository is using nvm (or similar) with the appropriate rc file such as .nvmrc.

nvm supports named versions which might look nicer on a first look but generates various issues, since these versions aren't static.

Causing issues with compatibility with various dependencies , and other random issues such as running nvm use but it no longer finds the version, since it was updated and you need to reinstall it locally.

It's even more problematic once you take into account CI/CD since they reinstall the environment from scratch each time and it will probably differ from local developers environment that haven't yet update to the latests named version update.

Hence I highly recommend using a version number and simply change it as needed, knowingly.

Tip #3 use engines

Basically an expansion on #2 , explicitly name the node / npm version (strict, not a range!) to avoid more unexpected and unwanted surprises.

Tip #4 use engine-strict

Avoid installing potential packages that might cause issues.

It's better to resolve an issue early on, with little overhead at the start than fixing it later when it's a critical part of your system.

Reducing egress costs for dummies (R2 vs S3)

ohaddahan — Sun, 21 Jan 2024 16:41:33 +0000

One day I was checking CloudFlare dashboard I noticed that one of the endpoints we were using to deliver new versions has huge bandwidth spikes.

Each new version we released was automatically downloaded by thousands of clients, resulting in terra bytes of egress costs to S3.

At the current rate this was around 3,000$/month.

Furthermore, our plan was to move to a daily release, which will make this problem even worse and jump to ~50,000$/month.

I immediately tried to think how to resolve this issue.

Since all the content we deliver on a new update is identical copies of the same new version, caching jumped into mind.

But unfortunately CloudFront costs weren't much better at around 2,500$/month.

Being a big fan of CloudFlare I knew of R2 and quickly calculated that using R2 will reduce the costs to effectively zero.

Implementing this was quite easy, simply store and serve from R2 instead of S3.

Another bonus was faster download speed.

How to secure SSH server

ohaddahan — Sun, 21 Jan 2024 16:35:19 +0000

Disable root login

Create new user useradd -m username.
Set password passwd username.
Optional: Add user to sudoers usermod -aG sudo username.
Edit /etc/ssh/ssh_config or /etc/ssh/sshd_config and add:

# Authentication:
PermitRootLogin no
AllowUsers username

Might need to look for other config files being included that might override this setting (grep -r "PermitRootLogin" /etc/ssh/).

Harden SSH

Disable empty password:

PermitEmptyPasswords no

Limit the number of authentication tries per connection:

MaxAuthTries 3

Changed to ssh version 2:

Include /etc/ssh/sshd_config.d/*.conf
Protocol 2

Disable plain text authentication

Connecting with SSH key:

UsePAM no
PasswordAuthentication no

ssh-keygen

Restart SSH service

Restart ssh service sudo systemctl restart ssh or sudo systemctl restart sshd.

Prevent brute force attacks

Install fail2ban or sshguard to ban IPs that fail to authenticate after a certain number of attempts.

References

Using data/computer science to solve a real life auto-complete issue

ohaddahan — Fri, 29 Nov 2019 23:58:43 +0000

Using data science to solve a real life auto complete issue

Problem background

In one of the applications I maintain, we had to create an auto complete with 20,000 options.

There is no typo, 20,000 is the real number.

So I looked up how to do an auto complete and I implemented it with a datalist, something that looked roughly
like this.

<input list="datalist" name="datalist">
<datalist id="datalist">
  <option>1</option>
  <option>2</option>
  ...
  ...
  <option>19,999</option>
  <option>20,000</option>
</datalist>

The problem was, that this is VERY SLOW, each time a user inserted a character, it will trigger an iteration on
the entire 20,000 options and check for each one of them which is slow, especially since I actually only need to
compare the prefix and datalist compares if the input string is contained in the options and not if they start with it.

For example:

<option>dog food</option>
<option>cat food</option>
<option>dogs and cats</option>
<option>cats and dogs</option>

In this case, inserting dog will match all but cat food while the user is interested only in options starting with dog.

This is an O(n) time complexity and it wasn't good enough.

Trie Tree

Trying to find a solution, I was thinking to myself, auto complete need doesn't care about any string that isn't
starting with the current input. Hence if I'll restructure my data in the form of a tree I'll be able to start
my check from the relevant place and not iterate over all options each time.

As it turns out, I didn't need to reinvent the wheel, people already thought about it, created it long before me.
This type of structure is called a trie tree.

A nice visualisation cant be found at trie tree visualization.
(Keep in mind that the find method implemented in the visualization only check for exact match which isn't our case)

Fortunately there are plenty of simple trie tree implementation.
And I saved myself lots of time writing it from scratch.

End Result

The end result can be seen in sandbox example.
Inserting any string into the original datalist input will take significantly more time
and will show too many unrelated results. Using the trie based auto complete is considerably faster and doesn't show
unneeded data, two wins with one change! :)

<label>Data List</label>
<input list="dropdown_menu" name="example">
<datalist id="dropdown_menu"></datalist>
<br>
<label>Trie Tree</label>
<input id="trie_tree" onkeyup="updateTrie()">
<ul id="trie_menu"></ul>

// Here comes code from https://gist.github.com/tpae/72e1c54471e88b689f85ad2b3940a8f0
var trie = new Trie();

function makeid(length) {
   var result           = '';
   var characters       = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
   var charactersLength = characters.length;
   for ( var i = 0; i < length; i++ ) {
      result += characters.charAt(Math.floor(Math.random() * charactersLength));
   }
   return result;
}


var datalist = document.getElementById('dropdown_menu');
for (var i = 0; i < 10000 ; i++) {
  var value = makeid(30);
  var option = document.createElement('option');
  option.value = value;
  option.setAttribute("value", value);
  var t = document.createTextNode(value);
  option.appendChild(t);
  datalist.appendChild(option);
  trie.insert(value.toString());
}

function removeAllChildNodes(el) {
  while (el.firstChild) {
    el.removeChild(el.firstChild);
  }
}

function updateTrie() {
    var trie_el = document.getElementById('trie_tree');
    var trie_menu = document.getElementById('trie_menu');
    removeAllChildNodes(trie_menu);
    var text = trie_el.value;
    var options = trie.find(text);  
    for (var i = 0 ; i < options.length ; i++) {
      var tmp_text = options[i];
      var li = document.createElement('li');
      li.innerHTML = tmp_text;      
      trie_menu.appendChild(li);
    }
}

React based Chrome extension with one click

ohaddahan — Mon, 14 Oct 2019 19:42:01 +0000

The Goals

Build a Chrome extension
Base the extension on React

Instructions

git clone https://github.com/ohaddahan/how-to-build-react-chrome-extension
cd how-to-build-react-chrome-extension
./scripts/run.sh
Load the unpacked extension into Chrome

The details

./scripts/run.sh is a fairly simple shell script to follow , here is how it works:

Setup and use nvm , this part optional , and you comment this section if you like. I left it since I think it's easier to avoid version mismatches with nvm
Create the React application by running npx create-react-app ${extName} --use-npm where extName is an environment variable with default value react-chrome-extension.
Install dependencies npm install npm-run-all webpack-cli axios @material-ui/core chalk.
Copy contentScript.js and background.js sample files from my repo into the new extension src directory.
Copy over the scripts and icons.
Run node scripts/prep.js (will elaborate on it later).
Finally it will build our extension using npm run build

Why all these extra scripts?

Let me elaborate on the various issues I encountered while getting this to work.

package.json compatibility issues:
- Changes done to pacakge.json by ./scripts/prep.js:

  packageJSON['scripts']  = {
     "start": "react-scripts start",
     "eject": "react-scripts eject",
     "test":  "react-scripts test",
     "prebuild": "rm -fr build ; node scripts/validate.js",
     "build": "npm-run-all build:*",
     "build:app": "INLINE_RUNTIME_CHUNK=false react-scripts build",
     "build:files":  "node ./scripts/build.js",
     "prezip": "rm -fr *.zip",
     "zip": "npm-run-all zip:*",
     "zip:build": "cd build; zip -r ../build.zip * -x '*.DS_Store'",
     "zip:src": "zip -r src.zip src package.json README.md public -x '*.DS_Store'",
     "release": "npm-run-all build zip"
   };
   packageJSON['eslintConfig']  = { "extends": "react-app" };

There were no changes done to start , eject and test.

prezip , zip , zip:build , zip:src and release are just regular helpers needed for any Chrome extension.

prebuild first cleans up the previous build and then runs ./scripts/validate.js
all this script does is check that package.json and manifiest.json are valid JSON files and that the required
Chrome settings exist.
(some of the settings I used aren't really mandatory but are a very common case and you can remove them if you have such a special case
icons and browser_action)
The use of npm-run-all is purely for ease of use to run multiple targets from one target.
Since we're using create-react-app it will use react-scripts build which doesn't expect to be used
as an extension. The first thing we need to change is adding INLINE_RUNTIME_CHUNK=false before react-scripts build.
This is done inside package.json , under scripts key , you can see the result in the final file create by scripts/run.sh.
If we won't use this flag Chrome will not run React properly and raise a
Content Security Policy error since React will try to inline JavaScript code.
- package.json at scripts key , "build:app": "INLINE_RUNTIME_CHUNK=false react-scripts build"
- Create React App - Advanced Configuration > By default, Create React App will embed the runtime script into index.html during the production build. When set to false, the script will not be embedded and will be imported as usual. This is normally required when dealing with CSP.
- Content Security Policy (CSP) > Inline JavaScript will not be executed. This restriction bans both inline <script> blocks and inline event handlers (e.g. <button onclick="...">). The first restriction wipes out a huge class of cross-site scripting attacks by making it impossible for you to accidentally execute script provided by a malicious third-party. It does, however, require you to write your code with a clean separation between content and behavior (which you should of course do anyway, right?). An example might make this clearer. You might try to write a Browser Action's popup as a single popup.html containing
script/build.js is the last helper script we have, it simply runs npx webpack on the contentScript.js
and background.js , after that it copies them into the build directory since react-scripts won't do it for us.
- manifest.json compatibility issues:
We add version and manifest_version which create-react-app doesn't add but Chrome demands.

We also add icons which isn't mandatory but comes as an incompatible format from create-react-app.
We remove background_color, display, start_url and theme_color which come from create-react-app but

aren't compatible with Chrome.
We also add browser_action , content_scripts , permissions , content_security_policy , background

which aren't mandatory but most likely needed by most and is missing.

Background and Content Scripts

Included are simple background.js and contentScript.js which send a GET request to https://postman-echo.com/get.
The reason I included them is mainly for completeness and as a reference for others on how they work with a React-Chrome-Extension.
Furthermore they are also an example of chrome.runtime

What does it actually do?

This sample extension will add a div and a button inside it , as seen in the image below.
Once you click on the button it will send a GET request to https://postman-echo.com/get and print the response.

How to load and debug it?

Load the unpacked extension , make sure you're in developer mode and you select the build directory.
For more details see :
How to use React.js to create a cross-browser extension in 5 minutes
Chrome - Getting Started Tutorial and
Enable background scripts , click on background page (Inactive) and it will turn to background page.
And open DevTools for the background scripts, without this you won't see any console.log or
any other action done in the background
Inside your regular Chrome , open DevTools and you'll see the prints coming from contentScript.js.
The response from https://postman-echo.com/get inside DevTools.
The background DevTools as you can see by the title.
The prints from background.js and response from https://postman-echo.com/get inside DevTools.

Final thoughts

While trying to do this I was trying to follow a few other examples as shown in the references section.
Unfortunately none had all the elements I needed or explained all the details I elaborate on here.
I hope this will help anyone else who would like to create a React-Chrome-Extension.
(I'm sure my code can get better, open an issue on GitHub and I'll happily fix it)

References

Algorithm Optimisation: A Real Example - My insights

ohaddahan — Wed, 02 Oct 2019 20:21:52 +0000

@steadbytes

This week I read Algorithm Optimisation: A Real Example and I recommend you all to read it too.
It's a good article.

I do think some items can be improved and I'll elaborate on them here.

When dealing with a large range , plots should be logarithmic to improve visibility. In Algorithm Optimisation: A Real Example the plots are in raw values , making the different curves and points on the curves unusable.
When benchmarking , we should remove outliers , the samples set being used by Algorithm Optimisation: A Real Example [1000 , 10000 , 100000 , 500000 , 1000000 , 2000000 , 5000000 , 10000000 , 20000000 , 50000000 , 100000000 , 200000000 , 500000000]
Sample points 1000 , 10000 are too small and are too susceptible to noise (the buffer size used is 4096). From my tests these sample points vary too much and their results doesn't reflect the 10X size difference.
When benchmarking , we should run the same test multiple times , one iteration isn't enough. For example , the machine used to run might be under a different load and if we only run once we won't be able to mitigate this effect. Hence I used Benchmark IPS to run each test multiple times.

Furthermore , while the actual solution given by Algorithm Optimisation: A Real Example is very good and show out of box thinking.
I suggest a different , much simpler solution that also have better performance.

Algorithm Optimisation: A Real Example still requires a malloc call the size of the file being read.
My proposed solution is instead of using the innovative linked list , we simply allocate all the memory needed beforehand and read the file directly into out result variable.

while ((read_len = read(fd, *data + len, BUFSIZE)) > 0) {
      len += read_len;
    }

This gives us almost 2X on our largest sample point of 100000000.

Code gist

Nested data structure extractors

ohaddahan — Sat, 21 Sep 2019 12:42:56 +0000

Wanted to share a few helpful methods I use to traverse and extract data from nested data structures, usually 3rd party API responses.

input_data = {
  x: {
    z: {
      first_name: 'hello',
      last_name: 'world'
    }
  },
  q: [
    1,
    2,
    {
      z: {
        first_name: 'hello',
        last_name: 'bye'
      }
    }
  ]
}

First extraction method fetch_all_by_key , the purpose of this method is to extract all the elements with the same key (for example getting a list of similar objects in a response):



def fetch_all_by_key(element: nil, key: nil, container: [])
  if element.is_a?(Array)
    element.each { |i| fetch_all_by_key(element: i, key: key, container: container) }
  elsif element.is_a?(Hash)
    container << element.fetch(key) unless element.fetch(key, nil).nil?
    element.each_pair { |_, v| fetch_all_by_key(element: v, key: key, container: container) }
  end
  container
end

Usage:

fetch_all_by_key(element: input_data, key: :q)
[
    [0] [
        [0] 1,
        [1] 2,
        [2] {
            :z => {
                :first_name => "hello",
                 :last_name => "bye"
            }
        }
    ]
]


fetch_all_by_key(element: input_data, key: :z)
[
    [0] {
        :first_name => "hello",
         :last_name => "world"
    },
    [1] {
        :first_name => "hello",
         :last_name => "bye"
    }
]

The next method, is basically the same as the previous one, but will only fetch the first element which fits our requirement (when you care only about getting any element that fits your key):


def deep_fetch(element: nil, key: nil)
  ret_value = tmp = nil
  if element.is_a?(Array)
    element.each do |i|
      tmp = deep_fetch(element: i, key: key)
      break unless tmp.nil?
    end
  elsif element.is_a?(Hash)
    if element.has_key?(key)
      tmp = element[key]
    else
      element.each_pair do |_, v|
        tmp = deep_fetch(element: v, key: key)
        break unless tmp.nil?
      end
    end
  end
  ret_value = tmp unless tmp.nil?
  ret_value
end

deep_fetch(element: input_data, key: :q)
[
    [0] 1,
    [1] 2,
    [2] {
        :z => {
            :first_name => "hello",
             :last_name => "bye"
        }
    }
]

 deep_fetch(element: input_data, key: :z)
{
    :first_name => "hello",
     :last_name => "world"
}
2.6.3 :096

The last method uses the previous method to enforce order (sometimes the same key can appear in two different hierarchical paths hence you want to have more control over what you're fetching):


def multi_deep_fetch(element: nil, keys: [])
  while key = keys.shift
    element = deep_fetch(element: element, key: key)
  end
  element
end


multi_deep_fetch(element: input_data, keys: [:q])
[
    [0] 1,
    [1] 2,
    [2] {
        :z => {
            :first_name => "hello",
             :last_name => "bye"
        }
    }
]



 multi_deep_fetch(element: input_data, keys: [:y, :z])
nil

 multi_deep_fetch(element: input_data, keys: [:x, :z])
{
    :first_name => "hello",
     :last_name => "world"
}

multi_deep_fetch(element: input_data, keys: [:q, :z])
{
    :first_name => "hello",
     :last_name => "bye"
}