Forem: Martin Splitt

Reading local files with JavaScript

Martin Splitt — Mon, 09 Dec 2019 10:16:28 +0000

For security and privacy reasons web apps do not have direct access to the files
on the user's device. If you need to read one or multiple local files, you can do
this through the usage of a file input and a FileReader. In this post we will take a look
at how this works through a few examples.

TL;DR

JavaScript does not have direct access to the local files due to security and privacy.
We can offer the user the possibility to select files via a file input element that we can then process.
The file input has a files property with the selected file(s).
We can use a FileReader to access the content of the selected file(s).

How it works

As JavaScript in the browser cannot access the local files from the user's device,
we need to provide the user with a way to select one or multiple files for us to use.
This can be done with the HTML file input element:

<input type="file" id="fileInput">

If we want to allow the selection of multiple files, we can add the multiple attribute:

<input type="file" id="fileInput" multiple>

We can either use the change event of the input field to respond to a file selection
or add another UI element to let the user explicitly start the processing of the selected file.

Also note: The selection of a file with the input element does not upload the file anywhere,
the only thing that happens is that the file becomes available to the JavaScript on the page.

The file input has a files property that is a list (as there could be multiple selected files) of File objects.

<input type="file" id="fileInput">
<script>
    document.getElementById('fileInput').addEventListener('change', function selectedFileChanged() {
      console.log(this.files); // will contain information about the file that was selected.
    });
</script>

A File object looks like this:

{
  name: 'test.txt',         // the name of the selected file
  size: 1024,               // the size in bytes
  type: 'text/plain',       // the assumed file type based on file extension. This might be incorrect.
  lastModified: 1234567890, // timestamp of the last change according to the user's system
  lastModifiedDate: 'Thu Jul 04 2019 09:22:51 GMT+0200 (Central European Summer Time)' // a date object for the last modified timestamp
}

Now we have the ability to select a file and see the metadata, but how can we access the file content?
To get the actual content of a selected file, we need a FileReader.

A file reader takes a File object and offers us methods to get access to the data as:

a string of text data
a data URL
a string of binary data
an ArrayBuffer containing raw binary data

In the following examples, we will use the readAsText and readAsDataURL methods to show the content of text and image files.

Example one: Reading text files

To show the file content as text, we change the change event handler:

document.getElementById('fileInput').addEventListener('change', function selectedFileChanged() {
  if (this.files.length === 0) {
    console.log('No file selected.');
    return;
  }

  const reader = new FileReader();
  reader.onload = function fileReadCompleted() {
    // when the reader is done, the content is in reader.result.
    console.log(reader.result);
  };
  reader.readAsText(this.files[0]);
});

First we make sure that there is a file that can be read. If the user cancels or otherwise
closes the file selection dialog without selecting a file, we have nothing to read and exit our function.

We then proceed to create a FileReader. The reader works asychronously in order
to not block the main thread and UI updates which is important when reading large files (like videos).

The reader emits a 'load' event (similar to images for example) that tells our code that the reading is finished.
The reader keeps the file content in its result property. The data in this property depends on which method we used to read the file.

In our example we read the file with the readAsText method, so the result will be a string of text.

Example two: Displaying an image from the user's device

If we want to display images, reading the file into a string isn't very helpful.
Conveniently the FileReader has a readAsDataURL method that reads the file into
an encoded string that can be used as the source for an <img> element. The code is pretty much the same as previously,
with the exceptions that we read the file with readAsDataURL and display the result as an image:

document.getElementById('fileInput').addEventListener('change', function selectedFileChanged() {
  if (this.files.length === 0) {
    console.log('No file selected.');
    return;
  }

  const reader = new FileReader();
  reader.onload = function fileReadCompleted() {
      const img = new Image();          // creates an <img> element
      img.src = reader.result;          // loads the data URL as the image source
      document.body.appendChild(img);   // adds the image to the body
  };
  reader.readAsDataURL(this.files[0]);
});

JavaScript in parallel - web workers explained

Martin Splitt — Fri, 29 Nov 2019 08:20:41 +0000

This is a repost from my personal blog

TL;DR

JavaScript is single-threaded and long-running scripts make the page unresponsive
Web Workers allow running JavaScript in separate threads, communicating with the main thread using messages.
Messages that transfer large amount of data in TypedArrays or ArrayBuffers cause large memory cost due to data being cloned
Using transfers mitigates the memory cost of cloning, but makes the data inaccessible to the sender
All code can be found in this repository
Depending on the type of work that our JavaScript carries out, navigator.hardwareConcurrency might help us spread the work across processors.

An example application

As an example we want to build a web application that constructs a table where each entry denotes if the number that belongs to it is prime or not.

We will use an ArrayBuffer to hold our booleans for us and we will be bold and make it 10 megabyte large.

Now this just serves to have our script do some heavy lifting - it isn't a very useful thing, but I may use techniques described here in future posts dealing with binary data of different sorts (images, audio, video for instance).

Here we will use a very naive algorithm (there are much better ones available):



function isPrime(candidate) {
  for(var n=2; n <= Math.floor(Math.sqrt(candidate)); n++) {
    // if the candidate can be divided by n without remainder it is not prime
    if(candidate % n === 0) return false
  }
  // candidate is not divisible by any potential prime factor so it is prime
  return true
}

Here is the rest of our application:

index.html



<!doctype html>
<html>
<head>
  <style>
    /* make the page scrollable */
    body {
      height: 300%;
      height: 300vh;
    }
  </style>
<body>
  <button>Run test</button>
  <script src="app.js"></script>
</body>
</html>

We make the page scrollable to see the effect of our JavaScript code in a moment.

app.js



document.querySelector('button').addEventListener('click', runTest)

function runTest() {
  var buffer = new ArrayBuffer(1024 * 1024 * 10) // reserves 10 MB
  var view = new Uint8Array(buffer) // view the buffer as bytes
  var numPrimes = 0

  performance.mark('testStart')
  for(var i=0; i<view.length;i++) {
    var primeCandidate = i+2 // 2 is the smalles prime number
    var result = isPrime(primeCandidate)
    if(result) numPrimes++
    view[i] = result
  }
  performance.mark('testEnd')
  performance.measure('runTest', 'testStart', 'testEnd')
  var timeTaken = performance.getEntriesByName('runTest')[0].duration

  alert(`Done. Found ${numPrimes} primes in ${timeTaken} ms`)
  console.log(numPrimes, view)
}

function isPrime(candidate) {
  for(var n=2; n <= Math.floor(Math.sqrt(candidate)); n++) {
    if(candidate % n === 0) return false
  }
  return true
}

We are using the User Timing API to measure time and add our own information into the timeline.

Now I let the test run on my trusty "old" Nexus 7 (2013):

Okay, that's not very impressive, is it?
Making matters worse is that the website stops reacting to anything during these 39 seconds - no scrolling, no clicking, no typing. The page is frozen.

This happens because JavaScript is single-threaded and in a single thread only one thing can happen at the same time. Making matters worse, pretty much anything that is concerned with interactions for our page (so browser code for scrolling, entering text etc.) runs on the same thread.

So is it that we just cannot do any heavy lifting?

Web Workers to the rescue

No. This is just the kind of work we can use Web Workers.

A Web Worker is a JavaScript file from the same origin as our web application that will be running in a separate thread.

Running in a separate thread means:

it will run in parallel
it will not make the page unresponsive by blocking the main thread
it will not have access to the DOM or any variable or function in the main thread
it can use the network and communicate with the main thread using messages

So how do we keep our page responsive while the prime-searching work goes on? Here is the procedure:

We start a worker and send the ArrayBuffer to it
The worker does its job
When the worker is done, it sends the ArrayBuffer and the number of primes it found back to the main thread

Here is the updated code:

app.js



document.querySelector('button').addEventListener('click', runTest)

function runTest() {
  var buffer = new ArrayBuffer(1024 * 1024 * 10) // reserves 10 MB
  var view = new Uint8Array(buffer) // view the buffer as bytes

  performance.mark('testStart')
  var worker = new Worker('prime-worker.js')
  worker.onmessage = function(msg) {
    performance.mark('testEnd')
    performance.measure('runTest', 'testStart', 'testEnd')
    var timeTaken = performance.getEntriesByName('runTest')[0].duration
    view.set(new Uint8Array(buffer), 0)
    alert(`Done. Found ${msg.data.numPrimes} primes in ${timeTaken} ms`)
    console.log(msg.data.numPrimes, view)
  }
  worker.postMessage(buffer)
}

prime-worker.js



self.onmessage = function(msg) {
  var view = new Uint8Array(msg.data),
      numPrimes = 0
  for(var i=0; i<view.length;i++) {
    var primeCandidate = i+2 // 2 is the smalles prime number
    var result = isPrime(primeCandidate)
    if(result) numPrimes++
    view[i] = result
  }
  self.postMessage({
    buffer: view.buffer,
    numPrimes: numPrimes
  })
}

function isPrime(candidate) {
  for(var n=2; n <= Math.floor(Math.sqrt(candidate)); n++) {
    if(candidate % n === 0) return false
  }
  return true
}

And here is what we get when run again on my Nexus 7:

Well, uhm, did all that ceremony give us anything then? After all now it is even slower!

The big win here wasn't making it faster, but try scrolling the page or otherwise interacting... it stays responsive at all times! With the calculation being ferried off to its own thread, we don't get in the way of the main thread taking care of responding to the user.

But before we move on to make things faster, we shall figure out an important detail on how postMessage works.

The cost of cloning

As mentioned earlier the main thread and worker thread are separated so we need to shuttle data between them using messages

But how does that actually move data between them? The answer for the way we have done it before is structured cloning.

This means we are copying our 10 megabyte ArrayBuffer to the worker and then copy the ArrayBuffer from the worker back.

I assumed this would total in 30 MB memory usage: 10 in our original ArrayBuffer, 10 in the copy sent to the worker and another 10 in the copy that is sent back.

Here is the memory usage before starting the test:

And here right after the test:

Wait, that is 50 megabytes more. As it turns out:

we start with 10mb for the ArrayBuffer
the cloning itself* creates another +10mb
the clone is copied to the worker, +10mb
the worker clones its copy again, +10mb
the cloned copy is copied to the main thread, +10mb

*) I am not exactly sure why the clone is not moved to the target thread instead of being copied, but the serialialisation itself seems to be incurring the unexpected memory cost

Transferables save the day

Luckily for us there is a different way of transferring data between the threads in the optional second parameter of postMessage, called the transfer list.

This second parameter can hold a list of Transferable objects that will be excluded from cloning and will be moved or transferred instead.

Transferring an object, however, neutralises it in the source thread, so for instance our ArrayBuffer won't contain any data in the main thread after it's been transferred to the worker and its byteLength will be zero.
This is to avoid the cost of having to implement mechanisms to deal with a bunch of issues that can happen when multiple threads access shared data.

Here is the adjusted code using transfers:

app.js



worker.postMessage(buffer, [buffer])

prime-worker.js



  self.postMessage({
    buffer: view.buffer,
    numPrimes: numPrimes
  }, [view.buffer])

And here are our numbers:

So we got a little faster than the cloning worker, close to the original main-thread-blocking version. How are we doing in terms of memory?

So having started with 40mb and ending up with a little more than 50mb sounds about right.

More workers = more speed?

So up until now we have

unblocked the main thread
removed the memory overhead from cloning

Can we speed it up as well?

We could split the range of numbers (and our buffer) among multiple workers, run them in parallel and merge the results:

app.js

Instead of launching a single worker, we are about to launch four. Each worker will receive a message instructing it with the offset to begin with and how many numbers to check.

When a worker finishes, it reports back with

an ArrayBuffer containing the information about which entries are prime
the amount of primes it found
its original offset
its original length

We then copy the data from the buffer into the target buffer, sum up the total number of primes found.

Once all workers completed, we display the final results.



document.querySelector('button').addEventListener('click', runTest)

function runTest() {
  const TOTAL_NUMBERS = 1024 * 1024 * 10
  const NUM_WORKERS = 4
  var numbersToCheck = TOTAL_NUMBERS, primesFound = 0
  var buffer = new ArrayBuffer(numbersToCheck) // reserves 10 MB
  var view = new Uint8Array(buffer) // view the buffer as bytes

  performance.mark('testStart')
  var offset = 0
  while(numbersToCheck) {
    var blockLen = Math.min(numbersToCheck, TOTAL_NUMBERS / NUM_WORKERS)
    var worker = new Worker('prime-worker.js')
    worker.onmessage = function(msg) {
      view.set(new Uint8Array(msg.data.buffer), msg.data.offset)
      primesFound += msg.data.numPrimes

      if(msg.data.offset + msg.data.length === buffer.byteLength) {
        performance.mark('testEnd')
        performance.measure('runTest', 'testStart', 'testEnd')
        var timeTaken = performance.getEntriesByName('runTest')[0].duration
        alert(`Done. Found ${primesFound} primes in ${timeTaken} ms`)
        console.log(primesFound, view)
      }
    }

    worker.postMessage({
      offset: offset,
      length: blockLen
    })

    numbersToCheck -= blockLen
    offset += blockLen
  }
}

prime-worker.js

The worker creates a Uint8Array view large enough to hold the length bytes as ordered by the main thread.

The prime checks start at the desired offset and finally data is transferred back.



self.onmessage = function(msg) {
  var view = new Uint8Array(msg.data.length),
      numPrimes = 0
  for(var i=0; i<msg.data.length;i++) {
    var primeCandidate = i+msg.data.offset+2 // 2 is the smalles prime number
    var result = isPrime(primeCandidate)
    if(result) numPrimes++
    view[i] = result
  }
  self.postMessage({
    buffer: view.buffer,
    numPrimes: numPrimes,
    offset: msg.data.offset,
    length: msg.data.length
  }, [view.buffer])
}

function isPrime(candidate) {
  for(var n=2; n <= Math.floor(Math.sqrt(candidate)); n++) {
    if(candidate % n === 0) return false
  }
  return true
}

And here are the result:

So this solution took approximately half the time with quite some memory cost (40mb base memory usage + 10mb for the target buffer + 4 x 2.5mb for the buffer in each worker + 2mb overhead per worker.

Here is the timeline of the application using 4 workers:

We can see that the workers run in parallel, but we aren't getting a 4x speed-up as some workers take longer than others. This is the result of the way we divided the number range: As each worker needs to divide each number x by all numbers from 2 to √x, the workers with larger numbers need to do more divisions, i.e. more work. This can surely minimised by dividing the numbers in a way that ends up spreading the operations more equally among them. I will leave this as an exercise to you, the keen reader ;-)

Another question is: Could we just throw more workers at it?

Here is the result for 8 workers:

Well, this got slower! The timeline shows us why this happened:

We see that, aside from minor overlaps, no more than 4 workers are active at the same time.

This will depend on the system and worker characteristics and isn't a hard-and-fast number.

A system can only do so much at the same time and work is usually either I/O-bound (i.e. limited by network or file throughput) or CPU-bound (i.e. limited by running calculations on the CPU).

In our case each worker hogs the CPU for our calculations. As my Nexus 7 has four cores, it can deal with four of our fully CPU-bound workers simultaneously.

Usually you will end up with a mix of CPU- and I/O bound workloads or problems that aren't easy to chunk in smaller workloads, so the number of workers is sometimes a little hard to judge. If you are looking to find out how many logical CPUs are available, you can use navigator.hardwareConcurrency though.

Wrap up

This was quite a lot to take in, so let's recap!

We found out that JavaScript is single-threaded and runs on the same thread as the browser tasks to keep our UI fresh and snappy.

We then used Web Workers to offload our work to separate threads and used `postMessage* to.communicate between the threads.

We noticed that threads don't scale infinitely, so some consideration for the amount of threads we run is advised.

When doing so we found out that data gets cloned by default which easily incurs more memory toll than meets the eye.

We fixed that via transferring data which is a viable option for certain types of data, referred to as Transferables.

Understanding CORS

Martin Splitt — Tue, 12 Nov 2019 17:32:24 +0000

This is a cross-posting from my blog

TL;DR

The browser enforces the Same-origin policy to avoid getting responses from websites that do not share the same origin.
The Same-origin policy does not prevent requests being made to other origins, but disables access to the response from JavaScript.
CORS headers allow access to cross-origin responses.
CORS together with credentials require caution.
CORS is a browser-enforced policy. Other applications aren't affected by it.

Our example

I will only show the request handling code here, but the full example is available on Github.

Let's start with an example. Say we have an amazing website with a public API available at http://good.com:8000/public:

app.get('/public', function(req, res) {
  res.send(JSON.stringify({
    message: 'This is public'
  }));
})

We also have a simple login feature, where users enter a shared secret and a cookie is set, identifying them as authenticated:

app.post('/login', function(req, res) {
  if(req.body.password === 'secret') {
    req.session.loggedIn = true
    res.send('You are now logged in!')
  } else {
    res.send('Wrong password.')
  }
})

We use this to protect some private data we made available to our users at /private.

app.get('/private', function(req, res) {
  if(req.session.loggedIn === true) {
    res.send(JSON.stringify({
      message: 'THIS IS PRIVATE'
    }))
  } else {
    res.send(JSON.stringify({
      message: 'Please login first'
    }))
  }
})

Requesting our API via AJAX from other domains

Now our API isn't particularly well-designed or fancy, but we could allow others to fetch data from our /public URL. Say, our API lives at good.com:300/public and our client is hosted on thirdparty.com, the client might run the following code:

fetch('http://good.com:3000/public')
  .then(response => response.text())
  .then((result) => {
    document.body.textContent = result
  })

But that doesn't work in our browser!

Let's take a look at the network tab for http://thirdparty.com:

The request was successful, but the result was not available. The reason can be found in the JavaScript console:

Aha! We are missing the Access-Control-Allow-Origin header. But why do we need it and what is it good for?

The Same-Origin Policy

The reason why we won't get the response in JavaScript is the Same-Origin Policy. This policy was aimed at making sure that a website can't read the result from a request made to another website and is enforced by the browser.

For instance: If you are on example.org you would not want that website to make a request to your banking website and fetch your account balance and transactions.

The Same-Origin Policy prevents exactly that.

The "origin" in this case is composed from

the protocol (e.g. http)
the host (e.g. example.com)
the port (e.g. 8000)

So http://example.org and http://www.example.org and https://example.org are three different origins.

A note on CSRF

Note that there is a class of attacks, called Cross Site Request Forgery that is not mitigated by the Same-Origin Policy.

In a CSRF attack, the attacker makes a request to a third party page in the background, for instance by sending a POST request to your bank website. If you have a valid session with your bank, any website can make a request in the background that will be carried out unless your bank uses counter measures against CSRF.

Note that despite the Same-Origin Policy being in effect, our example request from thirdparty.com was successfully carried out to good.com - we just could not access the results. For CSRF we don't need the result...

For example, an API that allows sending emails by doing a POST request would send an email, if we give it the right data - an attacker doesn't care about the result, they care about the email being sent which it will regardless of the ability to see the API response.

Enabling CORS for our public API

Now we do want to allow the JavaScript on third party sites (such as thirdparty.com) to access our API responses. To do so, we can enable the CORS header as the error said:

app.get('/public', function(req, res) {
  res.set('Access-Control-Allow-Origin', '*')
  res.send(...)
})

Here we are setting the Access-Control-Allow-Origin header to * which means: Any host is allowed to access this URL and the response in the browser:

Non-simple requests and preflights

The previous example was a so-called simple request. Simple requests are GET or POST requests with a few allowed headers and header values.

Now thirdparty.com changes the implementation a little to get the JSON:

fetch('http://good.com:3000/public', {
  headers: {
    'Content-Type': 'application/json'
  }
})
  .then(response => response.json())
  .then((result) => {
    document.body.textContent = result.message
  })

But this breaks thirdparty.com again!
This time the network panel shows us the reason:

Any request that is using a method that isn't GET or POST or uses a content type that isn't

text/plain
application/x-www-form-urlencoded
multipart/form-data

Any other header that isn't allowed for simple requests requires a preflight request.

This mechanism is meant to allow web servers to decide if they want to allow the actual request. The browser sets the Access-Control-Request-Headers and Access-Control-Request-Method headers to tell the server what request to expect and the server answers with corresponding headers.

Our server doesn't answer with these headers yet, so we need to add them:

app.get('/public', function(req, res) {
  res.set('Access-Control-Allow-Origin', '*')
  res.set('Access-Control-Allow-Methods', 'GET, OPTIONS')
  res.set('Access-Control-Allow-Headers', 'Content-Type')
  res.send(JSON.stringify({
    message: 'This is public info'
  }))
})

Now thirdparty.com can access the response again.

Credentials and CORS

Now let's assume that we are logged in on good.com and can access the /private URL with the sensitive information.

With all our CORS settings, can another site, say evil.com get this sensitive information?

Let's see:

fetch('http://good.com:3000/private')
  .then(response => response.text())
  .then((result) => {
    let output = document.createElement('div')
    output.textContent = result
    document.body.appendChild(output)
  })

No matter if we are logged in to good.com or not, we will see "Please login first".

The reason is that the cookie from good.com will not be sent when the request comes from another origin, in this case evil.com.
We can ask the browser to send the cookies along, even when it's a cross-origin domain:

fetch('http://good.com:3000/private', {
  credentials: 'include'
})
  .then(response => response.text())
  .then((result) => {
    let output = document.createElement('div')
    output.textContent = result
    document.body.appendChild(output)
  })

But again this won't work in the browser. That is great news, actually.

Imagine any website could make authenticated requests - the request will be made but won't send the actual cookie and the response is inaccessible.

So, we don't want evil.com to be able to access this private data - but what if we want thirdparty.com to have access to /private?
In this case we need to set the Access-Control-Allow-Credentials header to true:

app.get('/private', function(req, res) {
  res.set('Access-Control-Allow-Origin', '*')
  res.set('Access-Control-Allow-Credentials', 'true')
  if(req.session.loggedIn === true) {
    res.send('THIS IS THE SECRET')
  } else {
    res.send('Please login first')
  }
})

But this will still not work. It's a dangerous practice to allow every authenticated cross-origin requests.

The browser does not allow us to make this mistake this easily.

When we want to allow thirdparty.com access to /private we can specify this origin in the header:

app.get('/private', function(req, res) {
  res.set('Access-Control-Allow-Origin', 'http://thirdparty.com:8000')
  res.set('Access-Control-Allow-Credentials', 'true')
  if(req.session.loggedIn === true) {
    res.send('THIS IS THE SECRET')
  } else {
    res.send('Please login first')
  }
})

Now http://thirdparty:8000 has access to the private data as well, while evil.com is locked out.

Allowing multiple origins

Now we have allowed one origin to do cross origin requests with authentication data. But what if we have multiple third parties?

In this case, we probably want to use a whitelist:

const ALLOWED_ORIGINS = [
  'http://anotherthirdparty.com:8000',
  'http://thirdparty.com:8000'
]

app.get('/private', function(req, res) {
  if(ALLOWED_ORIGINS.indexOf(req.headers.origin) > -1) {
    res.set('Access-Control-Allow-Credentials', 'true')
    res.set('Access-Control-Allow-Origin', req.headers.origin)
  } else { // allow other origins to make unauthenticated CORS requests
    res.set('Access-Control-Allow-Origin', '*')        
  }

  // let caches know that the response depends on the origin
  res.set('Vary', 'Origin');

  if(req.session.loggedIn === true) {
    res.send('THIS IS THE SECRET')
  } else {
    res.send('Please login first')
  }
})

Again: Do not directly send req.headers.origin as the CORS origin header. This would allow any website access to authenticated requests to your site.
There may be exceptions to this rule, but think at least twice before implementing CORS with credentials without the whitelist.

Summary

In this article we've looked at the Same-Origin Policy and how we can use CORS to allow cross-origin requests when required.

This requires server- and client-side settings and depending on the request will cause a preflight request.

Additional care should be taken when dealing with authenticated cross-origin requests. A whitelist can help to allow multiple origins without risking to leak sensitive data (that is protected behind an authentication).