Category: Javascript

Everybody likes an API. It is the clean version of an integration: documented endpoints, structured responses, credentials meant for software, and hopefully somebody else’s problem when the implementation changes.

But plenty of useful systems do not have one. Or they have an API that covers part of the product but not the screen a team actually needs. The data is there, behind a normal login, and somebody is opening a browser and copying it into a spreadsheet.

That is a good place for browser automation.

We have been looking at a practical combination of Playwright and Codex for this kind of work. Playwright drives a real Chrome browser. Codex can work with a browser exposed over the Chrome DevTools Protocol (CDP), which makes it useful for exploring an application and helping build the automation. The important piece is that the login stays human: someone who is authorized to use the site signs in and completes MFA themselves. The automated job uses the resulting browser session; it does not try to get around the login.

A concrete example is recording live odds from sportsbooks such as FanDuel or BetMGM. If an organization is allowed to collect and use the data, that is a much better job for a computer than for someone watching pages and updating a sheet all day.

The browser is sometimes the integration

This is not an argument to scrape everything. If there is an official API that does the job, use it. It is almost always less fragile and easier to support.

But sometimes the browser is the only interface available to the user. A person can log in, look at a live market, and see the numbers, but there is no supported endpoint for getting the same information into an internal system. In that situation, the browser can be the boundary between the site and your workflow.

There are obvious limits. The account needs to be authorized, and the intended use needs to comply with the site’s terms, contracts, and applicable law. That matters especially for sportsbooks, where access and permitted use can vary by operator and jurisdiction. This is not a way to bypass MFA, CAPTCHAs, rate limits, or other controls.

Getting a logged-in browser session

Playwright is a browser automation framework. It can launch Chromium or Chrome, navigate pages, click buttons, fill forms, and read what the page renders.

The feature that makes authenticated automation workable is a persistent browser profile. Instead of starting fresh every time, Playwright launches Chrome with the same profile directory. That lets it retain browser state such as cookies and sessions when the site permits it.

The first run is simple: launch a visible browser, log in normally, handle the two-factor prompt, and make sure the page you need is available. After that, the job can reopen that same profile. It can run visibly while somebody is building or debugging it, then run headlessly when it is ready to collect data on a schedule.

import { chromium } from "playwright";

const userDataDir = "/secure/path/to/browser-profile";
const context = await chromium.launchPersistentContext(userDataDir, {
  channel: "chrome",
  headless: false, // first run: let the account owner log in
});

const page = await context.newPage();
await page.goto("https://example.com/login");

// The authorized user completes sign-in and MFA in this browser window.

That profile directory is sensitive. It may contain an active session, so treat it like a credential: keep it in approved storage, restrict who can access it, and do not put it in source control or logs. Also plan for it to expire. The site decides how long a session lasts. If the job finds a login page again, it should stop and let a person reauthenticate.

Asking Codex to do the tedious part

The page is usually the part that makes these projects annoying. Modern web applications load data after the page appears, change the DOM as markets update, and use selectors that are not obvious until you can inspect the live application.

This is where Codex helps. With the authenticated browser connected over CDP, you can ask it to inspect the page, find the market and selection elements, and build the Playwright script around what is actually there. It is much faster than guessing at selectors from a screenshot or trying to reverse engineer an undocumented backend.

A reasonable first prompt is something like:

Set up Playwright with a persistent Chrome profile and launch it visibly so I can log in. Once I have an authenticated session, use that session to inspect the live odds page and build a script that records the event, market, selection, displayed odds, and collection time.

Codex is helping with the implementation; it is not replacing the account holder. Keep that line clear. It should not be asked to find credentials, solve MFA, or work around controls the site has put in place.

A live-odds collector

Say the job is to keep an internal, timestamped record of odds for a set of games and markets.

First decide what an observation is. At a minimum, it will probably include the sportsbook, event, market, selection, displayed odds, the source page, and the time the value was seen. That last field matters: a number collected at 2:00 PM is not the same thing as a number collected five minutes later.

Next, use the visible browser session to get to the right market and let Codex inspect the rendered page. The job needs to know when the market is actually loaded, how a suspended or unavailable price is represented, and which labels or attributes are stable enough to use as selectors. Prefer user-facing labels where possible over a long chain of generated CSS classes that will disappear in the next redesign.

Once that is understood, the collector can launch the same persistent profile in headless mode, visit only the pages it needs, validate what it finds, and write normalized records to a database or queue.

const context = await chromium.launchPersistentContext(userDataDir, {
  channel: "chrome",
  headless: true,
});
const page = await context.newPage();

await page.goto(targetMarketUrl, { waitUntil: "domcontentloaded" });
await page.getByRole("heading", { name: /live odds/i }).waitFor();

const collectedAt = new Date().toISOString();
const observations = await page.locator("[data-market]").evaluateAll((markets) =>
  markets.map((market) => ({
    market: market.getAttribute("data-market"),
    text: market.textContent?.trim(),
    collectedAt,
  }))
);

The code above is deliberately generic. The real selectors should come from the target site and should be tested against its actual states. The useful outcome is not just a script that reads a page once. It is a small internal data source that dashboards, reports, or models can rely on without each one having to understand the sportsbook’s UI.

The unglamorous stuff is what makes it work

A browser job will change when the website changes. That is normal. The difference between a useful integration and a fragile script is how it behaves on a bad day.

Keep the browser profile locked down. Save where each observation came from and when it was collected. Alert when the job suddenly gets no results, sees a login screen, or returns far fewer records than normal. And give somebody a straightforward way to rerun the visible browser and refresh the session.

That is enough to turn a manual task into something dependable without pretending the website is an API.

For companies sitting on useful data behind logins, this is a practical option: a person handles the authorization once, Playwright keeps the browser state, and Codex speeds up the work of turning what is on the screen into structured data.

Through the beginning of my career in software engineering, I’ve found it challenging yet helpful to be exposed to multiple different languages and practices. This quickly teaches what languages, frameworks, and solutions can be used for the different problems that arise on a day-to-day basis. However, most of my professional experience lies with backend development in PHP as well as some recent front end development using jQuery and a bit of Angular. With a combination of PHP, Symfony, and jQuery, I’ve been fortunate to build features that I hope both end users and admins alike will appreciate. Many of these features however, are mostly based on relatively simple forms -- the user enters their data using textboxes, dropdowns, radio buttons, etc. and the backend does all of the complicated work.

With one of our new projects here at Setfive, we wanted a rich and reactive user interface for the sign-up portion, while Symfony would handle the rest. We could have just used Angular for the whole thing and come up with something similar, but using Angular would mean we’d need the entire page in Angular -- we didn’t want to lose our Symfony functionality for one component of the page. Using only jQuery was another option, but jQuery can quickly get complicated, hard to read, and unmaintable.  Like with many of the problems we encounter here at Setfive, a little bit of research led to a viable solution.

Introducing: Vue.js -- a JavaScript framework that allows you to build rich JS components similar in design to Angular but without using the framework for the entire page. That last aspect was especially appealing to us. With Vue.js, you can build a single component with whatever functionality you need, and drop it into the HTML at basically any spot in your project and the component can work independently of everything else on that page. Thus this component appears on the page alongside anything else we have on there from our Symfony/Twig side.

Similar to Angular, Vue.js allows you to use asynchronous functions to quickly send info back and forth via GET and POST requests and notably change the information on the page almost instantly. For our project we would be able to do exactly what we wanted to do: create a sign-up section that would allow you to quickly select different sources from different categories, easily view what you can select/have selected, enter your email, and submit -- all in an interactive and easy to use display. The idea was perfect, but the setup not so much.

I imagine that in a scenario where you are creating a project using mostly JavaScript and HTML/CSS, adding a basic Vue component to the project would be pretty easy as the documentation for Vue is solid. However, the process of adding a component to a Symfony project is a bit different, specifically because using Webpack Encore with Symfony is understandably not something that the Vue.js documentation covers. A quick search on the internet found me the perfect guide for integrating Vue.js into a Symfony project. Combining this guide with the actual Vue and Symfony documentation, I was able to get the crucial ‘Hello world’ to display on the page and we were ready to get going.

Similar to the Twig templates we normally use in our Symfony projects, Vue provides a few crucial features that allowed the bulk of this component to work as designed:

• ‘For loops’ allowed us to display a drop down menu of categories, and a reactive list of sources based on which category was chosen.
• ‘If statements’ allowed us to display errors and a confirmation message upon submitting
• With on-click functions, we were able to instantly move sources back and forth between the list of ones they select and ones not yet selected. 
• The input models allowed us to easily associate string variables with the different textbox inputs, allowing us to add a ‘search’ feature. 

 As someone who is not an expert in JavaScript, figuring all of this out took a lot of debugging. Luckily, the guide mentioned before had a lot of what I needed in it as well as StackOverflow and random other guides and sites found throughout my search. With the instant reactivity of Vue.js, debugging was made easy and I was able to roll through my development without having to spend much if any time waiting for data to load/finish. The main pieces that caused trouble involved making sure items displayed correctly based on what was already selected. That sort of validation of the display had to all be done manually in JavaScript. The validation of submitted data and the usage of that data can be done easily via Symfony, allowing us to use Vue solely to make a reactive and easy to use user interface that we hope will entice people to sign up for this product. 

One of the biggest pros of software development in my eyes is the satisfaction of designing and successfully building something new. Creating something, big or small, in an unfamiliar language/framework heightens that satisfaction even more. A lot of times though, the road to the end product in a new language/framework is filled with tons of frustration and confusion (especially for a relatively new developer). However, building this component using Vue was easier than expected once past my struggles with the initial set up. Thanks to the documentation from Vue and Symfony, StackOverflow, as well as the guide here by Krasimir Hristozov, I was able to create something that works well in a totally new framework without all of the trouble that can normally come with that process. Vue.js is described on their website as a blazing fast, “adoptable ecosystem” that can be used quickly by anyone who knows HTML, CSS, and JavaScript and I can attest to that. For anyone who needs a relatively small reactive component in their project, where you know jQuery will get unnecessarily complicated, Vue.js is the way to go.

Have you ever tried anything other than JavaScript or Typescript for front end and what was the result?

I tried Scala.js a couple of years ago, before TypeScript was popular. The results were mixed, at best. My experience was not recent, but, if I recall correctly, my two biggest stumbling blocks were tooling support and interop with standard JavaScript libraries.

On the tooling side, the major challenge was that scala.js needs to be compiled via the JVM based Scala compiler in order to produce JavaScript. That made integrating with JavaScript build tools like Gulp or Grunt fairly difficult. On top of that, I was sort of stuck writing scala.js in Eclipse since it had the best Scala plugin support. What that meant is that I had to give up IntelliJ’s awesome web development capabilities.

Integrating with third party libraries is a struggle for all of the “compile to JavaScript” solutions. A few years back, scala.js had a pretty rough integration story. Scalajs still relies on typed “facades,” Write facade types for JavaScript APIs to for integration. The problem faced is that there is not “any” type of Scala where you can skirt needing actual type information. These two issues highlight how well TypeScript is implemented in regards to tooling and 3rd party library support. Today, (2019) I think it that TypeScript is really the only effective option for compiling to JavaScript.

The main reason I decided to get into computer science was because my father used to be a programmer. Now, he has moved into the project management field, but still oversees different types of large scale computer science projects. He works from home a lot and has never seemed like he was very busy or overly stressed about work, so my hope is that getting into the computer science field will lead me down a similar path. Whenever I would complain about school programming projects, he would always tell me how much larger and more complex it gets in real world programming projects but I never really thought much of it.

After working on software for some time, I can now understand what he was talking about – my programming went from projects made up of a couple classes consisting of three or four functions, to a project made up of 50+ classes with I don’t even know how many functions as well as entities, endpoints, html files, css files, and a database with multiple related tables. To say it was a large increase in complexity would be a huge understatement. One thing I learned very quickly is that when you are working with large amounts of data, efficient programming is incredibly important and can be the difference between a webpage taking 10 seconds to load and the page loading almost instantly.

One of the most important aspects of efficient programming is the concept of Big O notation – a way to classify the speed at which your program will run and the memory it will take up. The smaller the Big O, the better. For example, if you have a loop running over a string of length n, your Big O notation will be order n, or O(n) in Big O notation – the loop needs to iterate n times to complete. However, if you can do whatever you need to do without a loop, you can save a lot of memory and time. This would not really matter in a loop of length 20, something you may see in a college project. However, if you are running a loop over an array of length 10,000, you will see a serious increase in the time it takes for your loop to complete as computers do not give instantaneous responses! The idea is to avoid the use of loops if you ever can, though this is not always possible.

A more common problem arises when using nested loops. If you want to count the letters in an array of strings, you need one loop to run over the array of strings, and another loop within that one to run over the string you are currently on. In other words, your loop needs to run n*m times – n strings with m letters in each string. For simplification, this is known as order n*n, or order n^2. Nested loops should be avoided if ever possible, as again the difference between a loop running 1000 times and 1000*1000 times is quite literally exponential. The more nested loops you add, the longer a program will take to run in a field where the difference between a 2 second load time and a 4 second load time is huge. With the example above of counting the number of letters in an array of strings, this can be done with a nested for loop:

const arrayOfStrings: string[] = [];
arrayOfStrings.push('hello');
arrayOfStrings.push('world');
arrayOfStrings.push('how are');
arrayOfStrings.push('you');

let charCounter = 0;
for (let i = 0; i < arrayOfStrings.length; i ++) {
   for (let k = 0; k < arrayOfStrings[i].length; k++) {
       charCounter++;
   }
}
console.log(charCounter);

However, with a little creativity, you can avoid using nested loops much of the time. For example, instead of pushing all the elements into an array, you can increase your charCounter2 variable by the length of each string as you add them:

const arrayOfStrings2: string[] = [];
let charCounter2 = 0;
arrayOfStrings2.push('hello');
charCounter2 += 'hello'.length;
arrayOfStrings2.push('world');
charCounter2 += 'world'.length;
arrayOfStrings2.push('how are');
charCounter2 += 'how are'.length;
arrayOfStrings2.push('you');
charCounter2 += 'you'.length;

console.log(charCounter2);

This will eliminate the nested for loop, and greatly reduce your runtime in cases of large arrays. Each .length call runs at order n, thus giving you an order of n + n + n + n, simplified to be just order n. The nested for loop would run at order n^2 – if n = 1000 elements, the runtime difference would be 4000 vs 1,000,000. As n gets larger and larger, this difference becomes increases more and more while the load time of your webpage would reduce considerably.

Recently, I ran into a substantial Big O problem in my code. When trying to count the occurences of each word in an array of paragraphs, I had a loop within a loop within a loop to give the correct output. This seemed totally fine when testing with 4 or 5 small paragraphs and I was just happy to get it working. The first loop iterated over the array of paragraphs (denoted as Review[]), and the next iterated over each of those paragraphs (review.body). The third loop iterated over my variable storing the current word counts to see if that word previously occurred – if not then it was added, and if it was then that word’s count incremented by 1.

export class Wordcloud {

   public getWordCloudCounts(reviews: Review[]): IWordCount[] {
       const wordsWithCounts: IWordCount[] = new Array();
       const numReviews = reviews.length;
       wordsWithCounts.push({
           word: 'ThisIsAPlaceHolderThatWillBeRemoved',
           count: 0,
       });

       for (let i = 0; i < numReviews; i++) {
           const splitter = reviews[i].body.split(' ');
           let found: boolean = false;
           for (let j = 0; j < splitter.length; j++) {
               if (splitter[j].endsWith('.')) {
                   splitter[j] = splitter[j].slice(0, -1);
               }
               for (let k = 0; k < wordsWithCounts.length; k++) {
                   if (splitter[j] === wordsWithCounts[k].word) {
                       wordsWithCounts[k].count++;
                       found = true;
                   }
                   else if (found === false && k === wordsWithCounts.length - 1) {
                       wordsWithCounts.push({
                           word: splitter[j],
                           count: 0,
                       });
                   }
               }
               found = false;
           }
       }
       wordsWithCounts.reverse();
       wordsWithCounts.pop();
       console.log(wordsWithCounts);
       return wordsWithCounts;

   }
}

interface IWordCount {
   word: string;
   count: number;

However, when I used it with the actual arrays of thousands of longer paragraphs, it took 10-15 seconds complete which was way too long. With some help from colleagues, I discovered associative arrays. In a normal array, you would have to loop through each element in the array to see if the element you are checking exists in that array. If it does not, then you must iterate over every element in the array to check. With associative arrays on the other hand, checking to see if an element is within the associative array is much simpler. When you add an element, a hash string is generated based on that element. Therefore, when you check to see if an element is in an associative array, your computer computes the same exact hash and knows exactly where to look to see if that hash already exists. Thus eliminating an entire for loop brought my function down from order n^3, to order n^2 and reduced the load time by 8-12 seconds. If n=1000, the amount of iterations would drop from 1,000,000,000 to 1,000,000!

getWordCloudCounts(reviews: string[]): {} {
 const words = {};
 const numReviews = reviews.length;
 const wordCounts: IWordCount[] = [];

 const stopWords = StopWords.get();

 for (let i = 0; i < numReviews; i++) {
   reviews[i] = reviews[i].toString();
   const splitter = reviews[i].toLowerCase().split(' ');
   for (let j = 0; j < splitter.length; j++) {
     if (splitter[j].endsWith('.') || splitter[j].endsWith(',') || splitter[j].endsWith('?') || splitter[j].endsWith('!')
       || splitter[j].endsWith(';') || splitter[j].endsWith(':') || splitter[j].endsWith('"') || splitter[j].endsWith('\'')
       || splitter[j].endsWith('&') || splitter[j].endsWith(')')) {
       splitter[j] = splitter[j].slice(0, -1);
     }
     if (splitter[j].startsWith('"') || splitter[j].startsWith('\'') || splitter[j].startsWith('&') || splitter[j].startsWith(' ')
       || splitter[j].startsWith('(')) {
       splitter[j] = splitter[j].slice(1);
     }
     if (splitter[j].endsWith('."')) {
       splitter[j] = splitter[j].slice(0, -1);
       splitter[j] = splitter[j].slice(0, -1);
     }
     if ((!(splitter[j] in words) && splitter[j].length > 3)) {
       words[splitter[j]] = 1;
     }
     else if (splitter[j].length > 3){
       words[splitter[j]]++;
     }
   }
 }

 return words;

The word cloud generator now only takes a few seconds to create beautiful word clouds as opposed to 10-15 seconds:

When it comes to web development, load time is very important. Many times if I try to click on something on my phone and it takes more than a couple seconds to load, I just immediately exit out due to impatience. When you create a website, you do not want users exiting out because your site takes a few seconds to load even with a solid internet connection. It is important to start practicing efficient programming early even when working with small amounts of data.

This will help you avoid situations similar to mine, where you have to figure out how to write efficient programs with data sets of thousands of elements and will save you from a few infinite loops that immediately crash your computer! Efficient programming is a major key throughout all of computer science, but is especially important when it comes to a user interface and a user’s experience!

Last week my girlfriend Diane was looking for some help scraping pollen data from a couple of sites. The code was simple enough to hammer out but how was I going to deliver it? Diane is fairly tech savy but even so asking her to install nodejs and run a command line app was going to be a bit much. After considering options like a Java Swing app or Qt+nodejs I decided to give Electron a shot. Just want to see the code? It’s available here, pollen-scraper.

Electron is a cross platform application runtime which basically “runs” code inside a Chrome browser alongside nodejs. In practice you can use nodejs libraries with your favorite JavaScript framework too build applications that run anywhere that Chrome will. Several popular companies including Slack and Spotify have desktop clients powered by Electron. Pretty much perfect for my use case. So what was using Electron for the first time like?

Getting started is easy

One of the frustrating aspects of “enterprise” cross platform frameworks is that it takes a long time to even get something up on the screen. Between complex build systems and custom layout languages, it generally takes awhile to get something on the screen using something like Qt or Swing. With Electron getting started was as simple as cloning https://github.com/electron/electron-quick-start-typescript, firing off an “npm install”, and after a “npm start” I had a working cross platform UI on the screen. Additionally, since Electron leverages web technologies it was also straightforward to add Bootstrap and AngularJS to the project.

The NodeJS ecosystem

As mentioned above, Electron applications can use any nodejs library which makes the environment incredibly powerful right out of the box. For example, I was able to leverage turfjs along with Google’s geocoder to find the closest city to an arbitrary zip code in Japan. Being able to tap into the npm/nodejs ecosystem also makes it possible to deliver high value applications quickly since you’re able to focus on business differentiators not plumbing.

Debugging

Since its built on Chrome, you get access to Chrome’s DevTools within Electron. And you can also enable remote debugging with a launch flag to make it possible to connect to your Electron instance remotely. In addition, for production apps you’d be able to drop in something like Rollbar to track JavaScript errors on the client side to help you debug and resolve issues for clients in the wild.

All in all, my first foray with Electron was a pretty positive experience. With a couple of beers and an afternoon of work I was able to deliver a cross platform application which saved my girlfriend’s team a dramatic amount of time.