is LDAP still relevant in terms of being a security source provider?

Guo-Guang — Tue, 05 Jun 2018 08:27:33 +0000

I am going to work on a new project which is related to security. Currently, I am studying where to store user credential like password and token. What we have now is to store this sensitive information in MongoDB along with a user profile. Well, I am not sure if this is a good design or not. But, I would like to make a change by separating the credential and profile data to different storage like LDAP and MySQL or even MongoDB. The main difference is to store users' credential in LDAP and store profile data in either MySQL or MongoDB. Thinking of performance and security level an LDAP can provide, do you think the LDAP is still a relevant technology in such a case? Or, how do you use LDAP?

Have you played the puppeteer?

Guo-Guang — Thu, 31 May 2018 14:44:59 +0000

I got a chance recently working on a UI test automation project which is on top of the Google puppeteer. It's a tool highly customizable for developers to control the behavior of Google Chromium browser. The details of puppeteer can be found here.

We have a Google Chrome web extension project and the tests before each release a series of tests will be performed by interns. We have tried to maintain 100% coverage by executing the unit tests for the project. However, the integration tests like hitting APIs and monitor how the web extension UI behaves according to web pages or websites the web extension is on is still done manually. Indeed, manual tests are still important. But, we sometimes found out that the data collected by interns through manual tests are not that accurate for any kinds of reasons. We have surveyed UI test automation solutions like Selenium with WebDrivers and others but we ended up with Google puppeteer since we only support Google chrome at this stage.

The APIs of Google Puppeteer is pretty straightforward. Starting from creating a browser is very simple like codes below

const browser = await puppeteer.launch({
    headless, // if true then no UI will be shown
    ignoreDefaultArgs: true,
    ignoreHTTPSErrors: true,
    slowMo, // delay executing each actions performed on web broswer
    args, // other args
  })
return browser;

Due to the test target is the browser extension, I need to specify where the browser extension is so that the Google Chromium browser can locate the assets and then start the installation. extPath is the absolute path to the folder holding browser extensions assets.

  const args = new Array();
  args.push(`--disable-extensions-except=${extPath}`);
  args.push(`--load-extension=${extPath}`);
  const b = await puppeteer.launch({
    headless,
    ignoreDefaultArgs: true,
    ignoreHTTPSErrors: true,
    slowMo,
    args,
  });
  return b;

The codes above will show you the Google Chromium browser with the web extension icon on the top-right corner and probably open another tab landing to the website the web extension specify in menifest.json.

There are few key components composing the Google web extensions. Among which, the main UI commonly called popup.html is one of the test targets in the project. To the best of my knowledge, there no is API currently able to trigger the click event of the extension icon or button sitting on top-right corner to show the main UI which is the popup.html. However, there is a way around the issue, which is directly visiting the popup.html page. Therefore, you need to create a Page and specify the URL you want to visit. In codes below, there is a variable called extId. If you don't have this id you probably cannot load the popup.html.

  const page = await browser.newPage(); // create a new tab
  const chromeExtPath = `chrome-extension://${extId}/popup.html`; // where to find extId?
  await page.goto(chromeExtPath, { waitUntil: 'domcontentloaded', });
  if (bringToFront) await page.bringToFront();
  await page.reload();
  return page;

For the normal installation, you should find out the web extensions IDs at this directory ~/Library/Application Support/Google/Chrome/Default/Extensions if you're on the Macbook. However, you won't find any IDs in this directory related to the unpacked extensions.

➜  ls -l ~/Library/Application\ Support/Google/Chrome/Default/Extensions
total 0
drwx------@ 3 guoguang  staff  96 May 21 10:33 bmnlcjabgnpnenekpadlanbbkooimhnj
drwx------@ 3 guoguang  staff  96 May 31 12:57 chhjbpecpncaggjpdakmflnfcopglcmi
drwx------@ 3 guoguang  staff  96 May 26 13:37 djjjmdgomejlopjnccoejdhgjmiappap
drwx------@ 3 guoguang  staff  96 May 30 13:59 jifpbeccnghkjeaalbbjmodiffmgedin
drwx------@ 3 guoguang  staff  96 May 31 12:55 kbfnbcaeplbcioakkpcpgfkobkghlhen
drwx------@ 3 guoguang  staff  96 May 18 14:01 niloccemoadcdkdjlinkgdfekeahmflj
drwx------@ 3 guoguang  staff  96 May 16 10:31 nmmhkkegccagdldgiimedpiccmgmieda
drwx------@ 3 guoguang  staff  96 May 16 10:31 pkedcjkdefgpdelpbcmbmeomcjbeemfm

There are two methods address the missing extension id. First, load the unpacked extension in Google Chrome browser and also turn on Developer mode to show extension ids. However, this must go through the manual process, which is a bit problematic if you want to have a fully automated testing process. The second way is to generate the extension key and id on the fly. Following this thread on StackOverflow about how to create the key and id I have a shell script generating key and id as follows. You probably need to enlarge the key size to 4096 while creating the pem.key so as to make the web extension id and key works out for the extensions. Btw, the Google Chromium version I am using is 68.0.3419.0 Developer Build 64-bit.

  # genereate the web extension key.
  ret=$(openssl rsa -in ./extensions/key.pem -pubout -outform DER | openssl base64 -A)
  # based the key above to generate the **extension id**
  ret=$(openssl rsa -in ./extensions/key.pem -pubout -outform DER | shasum -a 256 | head -c32 | tr 0-9a-f a-p)

Not only the popup.html but how extensions interacting with web pages is also the test target. One important function of our web extension is to change its icon if a user is visiting our partners' website. In the beginning, I was trying the Message Passing to sync up testing data between web extension and web pages. I found that I need to change codes a lot in the extensions especially for the onMessage and onExtenalMesasge parts. To minimize the changes, I then used the Firebase as the medium to sync up the testing data. This probably is not the best choice but it really works for us currently.

  firebase.initializeApp(JSON.parse(data));
  firebaseDB = firebase.database();
...
  firebaseDB.ref('urlPatternTest/').update({
    uuid: testKey,
    host: '',
    icon: '',
    ts: (new Date()).getTime(),
    key: '',
  }, err => {
    if (err) reject(err);
    else resolve();
  });

The rest is about codes how we doing UI automation tests like switching tab, login/logout and etc.

async function selectTab(page, tabName) {
  const selector = 'div#buttonBox :first-child'; // querySelector
  let handle = await page.waitForSelector(selector); // blokc until the expected dom element loaded
  const text = await page.evaluate(sel => document.querySelector(sel).innerText, selector, handle); // evalute the selector and return the data
  if (text === tabName) {
    handle = await page.$(selector);
    await handle.click();
  } else {
    handle = await page.$('div#buttonBox :last-child');
    await handle.click();
  }
}
...
....
  let selector = 'input[type="email"]';
  await sgPage.waitForSelector(selector);
  const emailHandle = await sgPage.$(selector);
  await emailHandle.type(USER_EMAIL); // key in user email

  selector = 'input[type="password"]';
  const pwdHandle = await sgPage.$(selector);
  if (_.isNil(pwdHandle)) throw new TypeError('found no password input element!');
  await pwdHandle.type(USER_PWD); // key in use pwd
...

I really enjoy using Puppeteer so far since its simplicity getting UI test done with its full-fledged APIs and ways of intercepting data (HTML, image, URL, cookies and etc). Right now, we are still working on the project by adding more sophisticated test scenarios. Not sure if you will use Puppeteer in the near future. Any comments are so welcome!

The Twelve-Factor App

Guo-Guang — Fri, 18 May 2018 09:55:48 +0000

Just read this article https://12factor.net and found out dev.to has a series of podcast on 12factor. I will manage to listen to podcast. While reading, I am also checking how many factors I have applied completely, partially, or even none on projects I have involved. Surprisingly, all these 12 factors, in my opinion, can be shown on projects. However, it is just the matter of how much we enforce each factor in projects.

Taking the codebase factor. We have tried to follow microservices architecture to organize projects. However, I still can see the duplicated functions or codes among projects. The bad thing about the duplications is that some of functions or APIs provided by a service are under very low utilization for any reasons, which are still maintained by developers. However, you cannot directly remove those functions from a service since there could be some coupling inside.

Another factor I think we can enhance is to minimize the deployment difference for dev, staging, QA, and PROD, which is Dev/prod parity factor. The problem we are facing in this factor is that the app must detect environment the app exists and then get or load proper configuration values to complete the app initialisation. This detection adds extras a bit meaningless test codes like mocking staging, or QA environments.

how to correctly identify a web page?

Guo-Guang — Fri, 11 May 2018 10:01:06 +0000

why need to identify a certain page? A Google browser extension needs to react to a certain page loaded from a website. The current way of identifying the specific page is to match its URL by using a regular expression generated according to a set of ULRs given by Ops people.

Using regex has some drawbacks. Ops or marketing people don't know regex. Therefore, it always relies on engineers to generate the regex if there are contracts signed with partners. We are in EC field and it is at least thousands of partners in each country running our services. So far we run our service in 7 countries.

Instead of using regex, I am thinking if I could use Solr or ElasticSearch to index the URL with different weights on the specific terms in the URLs. Hope to learn from how you probably address such a problem.

Please recommend a restful API framework in Golang

Guo-Guang — Wed, 02 May 2018 05:45:55 +0000

Forem: Guo-Guang

is LDAP still relevant in terms of being a security source provider?

Have you played the puppeteer?

The Twelve-Factor App

how to correctly identify a web page?

Please recommend a restful API framework in Golang